Simulating randomness

This notebook provides example EDSL code for exploring ways of prompting AI agents to return “random” numbers. We do this by passing agents different instructions for picking a number at random, and by including and excluding memories of prior selections.

We also show how to use the QuestionFunctional question type to compare the responses using a function instead of calling a model. Learn more about this question type.

Creating questions to generate “random” numbers

We start by creating some questions prompting an agent to return a single random number and a list of random numbers. We then combine the questions in a survey and add “memories” of prior questions to some of the questions in order to compare responses generated with and without information about other responses. We also investigate how agents handle instructions to ignore a prior response that is nevertheless included, and to an instruction to return a list of random numbers all at once. This code is readily editable for further exploration, such as adding agent traits and personas to compare their additional impact on responses.

[1]:

from edsl import QuestionNumerical, QuestionList, Survey, AgentList, Agent, Model

m = Model("gemini-1.5-flash")

q_random = QuestionNumerical(
    question_name="random",
    question_text="Pick a random number between 1 and 100."
)

# We will administer this question with no memory of the prior question
q_random_no_memory = QuestionNumerical(
    question_name="random_no_memory",
    question_text="Pick a random number between 1 and 100.",
)

# We will administer this question with a memory of the first question
q_random_memory = QuestionNumerical(
    question_name="random_memory",
    question_text="Pick a random number between 1 and 100.",
)

# We will administer this question with a memory of the first question but instruct the agent to ignore it
q_random_ignore = QuestionNumerical(
    question_name="random_ignore",
    question_text="""Pick a random number between 1 and 100.
    Be sure to completely disregard the other random number that you picked.""",
)

# We will administer this question with no memory of prior questions
q_random_list = QuestionList(
    question_name="random_list",
    question_text="Pick 5 random numbers between 1 and 100. Return a list of integers.",
    max_list_items=5,
)

# Combine the questions in a survey
survey = Survey(
    [q_random, q_random_no_memory, q_random_memory, q_random_ignore, q_random_list]
)

# Adding targeted question memories - the prior question and answer become part of the new question prompt
survey = (survey.add_targeted_memory(q_random_memory, q_random)
          .add_targeted_memory(q_random_ignore, q_random)
         )

# Create a variety of instructions for the agents
instructions = [
    "Respond as if you are a real human trying to pick randomly.",
    "Respond as if you are simulating actual random events.",
    "Respond as if you are playing a game.",
    "Respond as if you are participating in a lottery.",
    "Respond as if you are testing software.",
    "Respond as if you are choosing among indistinguishable options.",
    "Respond spontaneously.",
    "Respond erratically.",
    "Respond chaotically.",
]

# Just pass the instructions - we could also explore variations in agent traits, eg personas or knowledge
al = AgentList(
    Agent(traits={}, instruction=i) for i in instructions
)

results = survey.by(al).by(m).run()

(
    results.sort_by("random")
    .select(
        "agent_instruction",
        "random",
        "random_no_memory",
        "random_memory",
        "random_ignore",
        "random_list",
    )
)

▼ Job Status (2025-03-03 12:03:22)

Job UUID	95bc9eee-1545-4d0d-a956-c346ef5daadd
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/95bc9eee-1545-4d0d-a956-c346ef5daadd
Exceptions Report URL	None
Results UUID	e105df94-5b25-4c21-9be5-f8b47108d9aa
Results URL	https://www.expectedparrot.com/content/e105df94-5b25-4c21-9be5-f8b47108d9aa

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/e105df94-5b25-4c21-9be5-f8b47108d9aa

[1]:

	agent.agent_instruction	answer.random	answer.random_no_memory	answer.random_memory	answer.random_ignore	answer.random_list
0	Respond as if you are playing a game.	42	42	87.000000	87	[23, 67, 12, 88, 4]
1	Respond as if you are testing software.	42	42	87.000000	87	[23, 67, 12, 88, 4]
2	Respond as if you are choosing among indistinguishable options.	50	50	50.000000	27	[67, 22, 91, 15, 4]
3	Respond as if you are a real human trying to pick randomly.	63	63	27.000000	27	[23, 67, 12, 88, 4]
4	Respond as if you are participating in a lottery.	63	63	27.000000	42	[23, 67, 12, 88, 4]
5	Respond as if you are simulating actual random events.	87	87	23.000000	42	[87, 22, 1, 95, 48]
6	Respond spontaneously.	87	87	42.000000	42	[23, 67, 12, 88, 4]
7	Respond erratically.	87	87	3.141590	42	[92, 17, 3, 66, 42]
8	Respond chaotically.	87	87	42.000000	42	[87, 2, 99, 14, 53]

Using `QuestionFunctional` to evaluate responses

We can use question type QuestionFunctional to answer a question with a function instead of calling a model. This can be useful where a model is not needed for part of a survey. This question type lets us define a function for evaluating scenarios and (optionally) agent traits, which is passed as a parameter func to the question type in the following general format:

def my_function(scenario, agent_traits=None):
    <some function>

q = QuestionFunctional(
    question_name = "example",
    func = my_function
)

Here we can use QuestionFunctional to compute some straightforward comparisons of the agents’ “random” numbers. We start by creating scenarios of the responses to use as inputs to a function for comparing the numbers:

[2]:

scenarios = results.select("agent_instruction", "random", "random_no_memory").to_scenario_list()

[3]:

scenarios

[3]:

ScenarioList scenarios: 9; keys: ['random', 'random_no_memory', 'agent_instruction'];

	agent_instruction	random	random_no_memory
0	Respond as if you are a real human trying to pick randomly.	63	63
1	Respond as if you are simulating actual random events.	87	87
2	Respond as if you are playing a game.	42	42
3	Respond as if you are participating in a lottery.	63	63
4	Respond as if you are testing software.	42	42
5	Respond as if you are choosing among indistinguishable options.	50	50
6	Respond spontaneously.	87	87
7	Respond erratically.	87	87
8	Respond chaotically.	87	87

Next we use the function to generate the comparison and print the results as another table:

[4]:

from edsl import QuestionFunctional

def check_random(scenario, agent_traits):
    if scenario.get("random") == scenario.get("random_no_memory"):
        return "Agent returned the same number."
    else:
        return "Agent returned a different number."


q = QuestionFunctional(question_name="check_random", func=check_random)

[5]:

results = q.by(scenarios).by(m).run()

▼ Job Status (2025-03-03 12:03:32)

Job UUID	790e6549-15f3-49b4-bcd3-7cc320522551
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/790e6549-15f3-49b4-bcd3-7cc320522551
Exceptions Report URL	None
Results UUID	0d0f41d1-01db-40cd-8d7c-0bc804a540bb
Results URL	https://www.expectedparrot.com/content/0d0f41d1-01db-40cd-8d7c-0bc804a540bb

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/0d0f41d1-01db-40cd-8d7c-0bc804a540bb

[7]:

results.select("agent.agent_instruction", "random", "random_no_memory", "check_random")

[7]:

	agent.agent_instruction	scenario.random	scenario.random_no_memory	answer.check_random
0	You are answering questions as if you were a human. Do not break character.	63	63	Agent returned the same number.
1	You are answering questions as if you were a human. Do not break character.	87	87	Agent returned the same number.
2	You are answering questions as if you were a human. Do not break character.	42	42	Agent returned the same number.
3	You are answering questions as if you were a human. Do not break character.	63	63	Agent returned the same number.
4	You are answering questions as if you were a human. Do not break character.	42	42	Agent returned the same number.
5	You are answering questions as if you were a human. Do not break character.	50	50	Agent returned the same number.
6	You are answering questions as if you were a human. Do not break character.	87	87	Agent returned the same number.
7	You are answering questions as if you were a human. Do not break character.	87	87	Agent returned the same number.
8	You are answering questions as if you were a human. Do not break character.	87	87	Agent returned the same number.

Posting to Coop

The Coop is an integrated platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

Here we demonstrate how to post this notebook:

[9]:

from edsl import Notebook

nb = Notebook(path = "random_numbers.ipynb")

if refresh := False:
    nb.push(
        description = "Example code for exploring randomness with agents and models",
        alias = "random-numbers-notebook",
        visibility = "public"
    )
else:
    nb.patch('e90bcc84-58d3-469b-b3df-8c7e46d5310d', value = nb)

Simulating randomness

Creating questions to generate “random” numbers

Using QuestionFunctional to evaluate responses

Posting to Coop

Using `QuestionFunctional` to evaluate responses