Skip to main content
We also show how to use the QuestionFunctional question type to compare the responses using a function instead of calling a model. Learn more about this question type.

Creating questions to generate “random” numbers

We start by creating some questions prompting an agent to return a single random number and a list of random numbers. We then combine the questions in a survey and add “memories” of prior questions to some of the questions in order to compare responses generated with and without information about other responses. We also investigate how agents handle instructions to ignore a prior response that is nevertheless included, and to an instruction to return a list of random numbers all at once. This code is readily editable for further exploration, such as adding agent traits and personas to compare their additional impact on responses.
from edsl import QuestionNumerical, QuestionList, Survey, Agent, AgentList, Model, ModelList

m = ModelList([
    Model("gemini-1.5-flash", service_name = "google"),
    Model("gpt-4o", service_name = "openai"),
    Model("claude-sonnet-4-20250514", service_name = "anthropic")
])

q_random = QuestionNumerical(
    question_name="random",
    question_text="Pick a random number between 1 and 50."
)

# We will administer this question with no memory of the prior question
q_random_no_memory = QuestionNumerical(
    question_name="random_no_memory",
    question_text="Pick a random number between 1 and 50.",
)

# We will administer this question with a memory of the first question
q_random_memory = QuestionNumerical(
    question_name="random_memory",
    question_text="Pick a random number between 1 and 50.",
)

# We will administer this question with a memory of the first question but instruct the agent to ignore it
q_random_ignore = QuestionNumerical(
    question_name="random_ignore",
    question_text="""Pick a random number between 1 and 50.
    Be sure to completely disregard the other random number that you picked.""",
)

# We will administer this question with no memory of prior questions
q_random_list = QuestionList(
    question_name="random_list",
    question_text="Pick 5 random numbers between 1 and 50. Return a list of integers.",
    max_list_items=5,
)

# Combine the questions in a survey
survey = Survey(questions =
    [q_random, q_random_no_memory, q_random_memory, q_random_ignore, q_random_list]
)

# Adding targeted question memories - the prior question and answer become part of the new question prompt
survey = (
    survey
        .add_targeted_memory(q_random_memory, q_random)
        .add_targeted_memory(q_random_ignore, q_random)
)

# Create a variety of instructions for the agents
instructions = [
    "Respond as if you are a real human trying to pick randomly.",
    "Respond as if you are simulating actual random events.",
    "Respond as if you are playing a game.",
    "Respond as if you are participating in a lottery.",
    "Respond as if you are testing software.",
    "Respond as if you are choosing among indistinguishable options.",
    "Respond spontaneously.",
    "Respond erratically.",
    "Respond chaotically.",
]

# Just pass the instructions - we could also explore variations in agent traits, eg personas or knowledge
a = AgentList(
    Agent(traits={}, instruction=i) for i in instructions
)

results = survey.by(a).by(m).run(fresh=True)

(
    results
        .sort_by("random")
        .select(
            "model",
            "agent_instruction",
            "random",
            "random_no_memory",
            "random_memory",
            "random_ignore",
            "random_list"
        )
)
model.modelagent.agent_instructionanswer.randomanswer.random_no_memoryanswer.random_memoryanswer.random_ignoreanswer.random_list
0gemini-1.5-flashRespond as if you are choosing among indistinguishable options.25252537[23, 4, 17, 38, 49]
1gemini-1.5-flashRespond as if you are a real human trying to pick randomly.27271333[23, 4, 17, 48, 31]
2gpt-4oRespond as if you are a real human trying to pick randomly.27373314[7, 22, 35, 48, 11]
3claude-sonnet-4-20250514Respond as if you are a real human trying to pick randomly.27274242[7, 23, 31, 42, 16]
4gemini-1.5-flashRespond as if you are simulating actual random events.2727173[23, 4, 48, 11, 37]
5gpt-4oRespond as if you are simulating actual random events.27271342[17, 34, 8, 22, 45]
6claude-sonnet-4-20250514Respond as if you are simulating actual random events.27274242[7, 23, 31, 42, 16]
7gemini-1.5-flashRespond as if you are playing a game.27273333[23, 4, 17, 48, 31]
8gpt-4oRespond as if you are playing a game.27271413[7, 19, 34, 42, 50]
9claude-sonnet-4-20250514Respond as if you are playing a game.27274242[7, 23, 31, 42, 16]
10gemini-1.5-flashRespond as if you are participating in a lottery.27273333[17, 23, 38, 4, 49]
11gpt-4oRespond as if you are participating in a lottery.27273414[7, 14, 23, 36, 45]
12claude-sonnet-4-20250514Respond as if you are participating in a lottery.27274242[7, 23, 31, 42, 18]
13gemini-1.5-flashRespond as if you are testing software.27274242[23, 4, 48, 11, 37]
14gpt-4oRespond as if you are testing software.27273434[7, 23, 35, 42, 18]
15claude-sonnet-4-20250514Respond as if you are testing software.27274234[23, 7, 41, 15, 32]
16gpt-4oRespond as if you are choosing among indistinguishable options.27271434[17, 34, 8, 42, 29]
17claude-sonnet-4-20250514Respond as if you are choosing among indistinguishable options.27274234[7, 23, 31, 42, 16]
18gemini-1.5-flashRespond spontaneously.27273342[23, 4, 17, 42, 31]
19gpt-4oRespond spontaneously.27271413[7, 23, 15, 42, 36]
20claude-sonnet-4-20250514Respond spontaneously.27274242[7, 23, 31, 42, 16]
21gemini-1.5-flashRespond erratically.27273237[17, 2, 48, 31, 1]
22gpt-4oRespond erratically.27374242[17, 42, 8, 29, 33]
23claude-sonnet-4-20250514Respond erratically.27274242[7, 23, 41, 15, 32]
24gemini-1.5-flashRespond chaotically.27271737[17, 42, 2, 31, 49]
25claude-sonnet-4-20250514Respond chaotically.27374242[42, 7, 23, 31, 16]
26gpt-4oRespond chaotically.37272342[42, 7, 23, 16, 35]

Using QuestionFunctional to evaluate responses

We can use question type QuestionFunctional to answer a question with a function instead of calling a model. This can be useful where a model is not needed for part of a survey. This question type lets us define a function for evaluating scenarios and (optionally) agent traits, which is passed as a parameter func to the question type in the following general format:
def my_function(scenario, agent_traits=None):
    <some function>

q = QuestionFunctional(
    question_name = "example",
    func = my_function
)
Here we can use QuestionFunctional to compute some straightforward comparisons of the agents’ “random” numbers. We start by creating scenarios of the responses to use as inputs to a function for comparing the numbers:
scenarios = results.select("agent_instruction", "random", "random_no_memory").to_scenario_list()
scenarios
ScenarioList scenarios: 27; keys: [‘random’, ‘random_no_memory’, ‘agent_instruction’];
agent_instructionrandomrandom_no_memory
0Respond as if you are a real human trying to pick randomly.2727
1Respond as if you are a real human trying to pick randomly.2737
2Respond as if you are a real human trying to pick randomly.2727
3Respond as if you are simulating actual random events.2727
4Respond as if you are simulating actual random events.2727
5Respond as if you are simulating actual random events.2727
6Respond as if you are playing a game.2727
7Respond as if you are playing a game.2727
8Respond as if you are playing a game.2727
9Respond as if you are participating in a lottery.2727
10Respond as if you are participating in a lottery.2727
11Respond as if you are participating in a lottery.2727
12Respond as if you are testing software.2727
13Respond as if you are testing software.2727
14Respond as if you are testing software.2727
15Respond as if you are choosing among indistinguishable options.2525
16Respond as if you are choosing among indistinguishable options.2727
17Respond as if you are choosing among indistinguishable options.2727
18Respond spontaneously.2727
19Respond spontaneously.2727
20Respond spontaneously.2727
21Respond erratically.2727
22Respond erratically.2737
23Respond erratically.2727
24Respond chaotically.2727
25Respond chaotically.3727
26Respond chaotically.2737
Next we use the function to generate the comparison and print the results as another table:
from edsl import QuestionFunctional

def check_random(scenario, agent_traits):
    if scenario.get("random") == scenario.get("random_no_memory"):
        return "Agent returned the same number."
    else:
        return "Agent returned a different number."


q = QuestionFunctional(question_name="check_random", func=check_random)
results = q.by(scenarios).by(m).run()
results.select("agent.agent_instruction", "random", "random_no_memory", "check_random")
/Users/a16174/edsl/edsl/results/result.py:382: UserWarning: Key 'agent_instruction' of data type 'scenario' is already in use. Renaming to agent_instruction_scenario
  warnings.warn(
agent.agent_instructionscenario.randomscenario.random_no_memoryanswer.check_random
0You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
1You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
2You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
3You are answering questions as if you were a human. Do not break character.2737Agent returned a different number.
4You are answering questions as if you were a human. Do not break character.2737Agent returned a different number.
5You are answering questions as if you were a human. Do not break character.2737Agent returned a different number.
6You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
7You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
8You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
9You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
10You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
11You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
12You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
13You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
14You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
15You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
16You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
17You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
18You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
19You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
20You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
21You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
22You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
23You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
24You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
25You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
26You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
27You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
28You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
29You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
30You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
31You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
32You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
33You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
34You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
35You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
36You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
37You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
38You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
39You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
40You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
41You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
42You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
43You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
44You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
45You are answering questions as if you were a human. Do not break character.2525Agent returned the same number.
46You are answering questions as if you were a human. Do not break character.2525Agent returned the same number.
47You are answering questions as if you were a human. Do not break character.2525Agent returned the same number.
48You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
49You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
50You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
51You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
52You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
53You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
54You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
55You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
56You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
57You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
58You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
59You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
60You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
61You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
62You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
63You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
64You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
65You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
66You are answering questions as if you were a human. Do not break character.2737Agent returned a different number.
67You are answering questions as if you were a human. Do not break character.2737Agent returned a different number.
68You are answering questions as if you were a human. Do not break character.2737Agent returned a different number.
69You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
70You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
71You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
72You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
73You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
74You are answering questions as if you were a human. Do not break character.2727Agent returned the same number.
75You are answering questions as if you were a human. Do not break character.3727Agent returned a different number.
76You are answering questions as if you were a human. Do not break character.3727Agent returned a different number.
77You are answering questions as if you were a human. Do not break character.3727Agent returned a different number.
78You are answering questions as if you were a human. Do not break character.2737Agent returned a different number.
79You are answering questions as if you were a human. Do not break character.2737Agent returned a different number.
80You are answering questions as if you were a human. Do not break character.2737Agent returned a different number.

Posting to Coop

Coop is an integrated platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using Coop. Here we demonstrate how to post this notebook:
from edsl import Notebook

nb = Notebook(path = "random_numbers.ipynb")

nb.push(
    description = "Example code for exploring randomness with agents and models",
    alias = "random-numbers-notebook",
    visibility = "public"
)
{'description': 'Example code for exploring randomness with agents and models',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/e4737176-951a-484a-921e-bda9e1e3232b',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/random-numbers-notebook',
 'uuid': 'e4737176-951a-484a-921e-bda9e1e3232b',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
I