Simulating randomness
This notebook provides example EDSL code for exploring ways of prompting AI agents to return “random” numbers. We do this by passing agents different instructions for picking a number at random, and by including and excluding memories of prior selections.
We also show how to use the QuestionFunctional question type to compare the responses using a function instead of calling a model. Learn more about this question type.
Creating questions to generate “random” numbers
We start by creating some questions prompting an agent to return a single random number and a list of random numbers. We then combine the questions in a survey and add “memories” of prior questions to some of the questions in order to compare responses generated with and without information about other responses. We also investigate how agents handle instructions to ignore a prior response that is nevertheless included, and to an instruction to return a list of random numbers all at once. This code is readily editable for further exploration, such as adding agent traits and personas to compare their additional impact on responses.
[1]:
from edsl import QuestionNumerical, QuestionList, Survey, AgentList, Agent
q_random = QuestionNumerical(
question_name="random",
question_text="Pick a random number between 1 and 100."
)
# We will administer this question with no memory of the prior question
q_random_no_memory = QuestionNumerical(
question_name="random_no_memory",
question_text="Pick a random number between 1 and 100.",
)
# We will administer this question with a memory of the first question
q_random_memory = QuestionNumerical(
question_name="random_memory",
question_text="Pick a random number between 1 and 100.",
)
# We will administer this question with a memory of the first question but instruct the agent to ignore it
q_random_ignore = QuestionNumerical(
question_name="random_ignore",
question_text="""Pick a random number between 1 and 100.
Be sure to completely disregard the other random number that you picked.""",
)
# We will administer this question with no memory of prior questions
q_random_list = QuestionList(
question_name="random_list",
question_text="Pick 5 random numbers between 1 and 100. Return a list of integers.",
max_list_items=5,
)
# Combine the questions in a survey
survey = Survey(
[q_random, q_random_no_memory, q_random_memory, q_random_ignore, q_random_list]
)
# Adding targeted question memories - the prior question and answer become part of the new question prompt
survey = (survey.add_targeted_memory(q_random_memory, q_random)
.add_targeted_memory(q_random_ignore, q_random)
)
# Create a variety of instructions for the agents
instructions = [
"Respond as if you are a real human trying to pick randomly.",
"Respond as if you are simulating actual random events.",
"Respond as if you are playing a game.",
"Respond as if you are participating in a lottery.",
"Respond as if you are testing software.",
"Respond as if you are choosing among indistinguishable options.",
"Respond spontaneously.",
"Respond erratically.",
"Respond chaotically.",
]
# Just pass the instructions - we could also explore variations in agent traits, eg personas or knowledge
agents = AgentList(
Agent(traits={}, instruction=i) for i in instructions
)
results = survey.by(agents).run()
(
results.sort_by("random")
.select(
"agent_instruction",
"random",
"random_no_memory",
"random_memory",
"random_ignore",
"random_list",
)
.print(format="rich")
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ agent ┃ answer ┃ answer ┃ answer ┃ answer ┃ answer ┃ ┃ .agent_instruction ┃ .random ┃ .random_no_memory ┃ .random_memory ┃ .random_ignore ┃ .random_list ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │ Respond as if you are │ 42 │ 42 │ 76 │ 73 │ [23, 47, 58, 76, 89] │ │ testing software. │ │ │ │ │ │ ├──────────────────────────┼─────────┼───────────────────┼────────────────┼────────────────┼──────────────────────┤ │ Respond erratically. │ 42 │ 42 │ 73 │ 73 │ [42, 17, 89, 23, 56] │ ├──────────────────────────┼─────────┼───────────────────┼────────────────┼────────────────┼──────────────────────┤ │ Respond as if you are │ 42 │ 42 │ 78 │ 57 │ [17, 42, 58, 73, 91] │ │ playing a game. │ │ │ │ │ │ ├──────────────────────────┼─────────┼───────────────────┼────────────────┼────────────────┼──────────────────────┤ │ Respond as if you are a │ 47 │ 47 │ 73 │ 73 │ [17, 43, 68, 91, 25] │ │ real human trying to │ │ │ │ │ │ │ pick randomly. │ │ │ │ │ │ ├──────────────────────────┼─────────┼───────────────────┼────────────────┼────────────────┼──────────────────────┤ │ Respond as if you are │ 57 │ 57 │ 73 │ 34 │ [27, 54, 89, 12, 63] │ │ choosing among │ │ │ │ │ │ │ indistinguishable │ │ │ │ │ │ │ options. │ │ │ │ │ │ ├──────────────────────────┼─────────┼───────────────────┼────────────────┼────────────────┼──────────────────────┤ │ Respond spontaneously. │ 57 │ 57 │ 83 │ 23 │ [23, 47, 56, 81, 92] │ ├──────────────────────────┼─────────┼───────────────────┼────────────────┼────────────────┼──────────────────────┤ │ Respond as if you are │ 57 │ 57 │ 83 │ 42 │ [23, 47, 68, 5, 92] │ │ simulating actual random │ │ │ │ │ │ │ events. │ │ │ │ │ │ ├──────────────────────────┼─────────┼───────────────────┼────────────────┼────────────────┼──────────────────────┤ │ Respond as if you are │ 72 │ 72 │ 45 │ 45 │ [12, 45, 67, 89, 23] │ │ participating in a │ │ │ │ │ │ │ lottery. │ │ │ │ │ │ ├──────────────────────────┼─────────┼───────────────────┼────────────────┼────────────────┼──────────────────────┤ │ Respond chaotically. │ 73 │ 73 │ 42 │ 42 │ [17, 42, 88, 5, 96] │ └──────────────────────────┴─────────┴───────────────────┴────────────────┴────────────────┴──────────────────────┘
Using QuestionFunctional to evaluate responses
We can use question type QuestionFunctional to answer a question with a function instead of calling a model. This can be useful where a model is not needed for part of a survey. This question type lets us define a function for evaluating scenarios and (optionally) agent traits, which is passed as a parameter func to the question type in the following general format:
def my_function(scenario, agent_traits=None):
<some function>
q = QuestionFunctional(
question_name = "example",
func = my_function
)
Here we can use QuestionFunctional to compute some straightforward comparisons of the agents’ “random” numbers. We start by creating scenarios of the responses to use as inputs to a function for comparing the numbers:
[2]:
scenarios = results.select("agent_instruction", "random", "random_no_memory").to_scenario_list()
[3]:
scenarios
[3]:
{
"scenarios": [
{
"agent_instruction": "Respond as if you are a real human trying to pick randomly.",
"random": 47,
"random_no_memory": 47
},
{
"agent_instruction": "Respond as if you are participating in a lottery.",
"random": 72,
"random_no_memory": 72
},
{
"agent_instruction": "Respond as if you are testing software.",
"random": 42,
"random_no_memory": 42
},
{
"agent_instruction": "Respond as if you are choosing among indistinguishable options.",
"random": 57,
"random_no_memory": 57
},
{
"agent_instruction": "Respond spontaneously.",
"random": 57,
"random_no_memory": 57
},
{
"agent_instruction": "Respond erratically.",
"random": 42,
"random_no_memory": 42
},
{
"agent_instruction": "Respond chaotically.",
"random": 73,
"random_no_memory": 73
},
{
"agent_instruction": "Respond as if you are playing a game.",
"random": 42,
"random_no_memory": 42
},
{
"agent_instruction": "Respond as if you are simulating actual random events.",
"random": 57,
"random_no_memory": 57
}
]
}
Next we use the function to generate the comparison and print the results as another table:
[4]:
from edsl import QuestionFunctional
def check_random(scenario, agent_traits):
if scenario.get("random") == scenario.get("random_no_memory"):
return "Agent returned the same number."
else:
return "Agent returned a different number."
q = QuestionFunctional(question_name="check_random", func=check_random)
[5]:
results = q.by(scenarios).run()
[6]:
results.select("agent_instruction", "random", "random_no_memory", "check_random").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ scenario ┃ scenario ┃ scenario ┃ answer ┃ ┃ .agent_instruction ┃ .random ┃ .random_no_memory ┃ .check_random ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Respond as if you are a real human trying to │ 47 │ 47 │ Agent returned the same number. │ │ pick randomly. │ │ │ │ ├────────────────────────────────────────────────┼──────────┼───────────────────┼─────────────────────────────────┤ │ Respond as if you are participating in a │ 72 │ 72 │ Agent returned the same number. │ │ lottery. │ │ │ │ ├────────────────────────────────────────────────┼──────────┼───────────────────┼─────────────────────────────────┤ │ Respond as if you are testing software. │ 42 │ 42 │ Agent returned the same number. │ ├────────────────────────────────────────────────┼──────────┼───────────────────┼─────────────────────────────────┤ │ Respond as if you are choosing among │ 57 │ 57 │ Agent returned the same number. │ │ indistinguishable options. │ │ │ │ ├────────────────────────────────────────────────┼──────────┼───────────────────┼─────────────────────────────────┤ │ Respond spontaneously. │ 57 │ 57 │ Agent returned the same number. │ ├────────────────────────────────────────────────┼──────────┼───────────────────┼─────────────────────────────────┤ │ Respond erratically. │ 42 │ 42 │ Agent returned the same number. │ ├────────────────────────────────────────────────┼──────────┼───────────────────┼─────────────────────────────────┤ │ Respond chaotically. │ 73 │ 73 │ Agent returned the same number. │ ├────────────────────────────────────────────────┼──────────┼───────────────────┼─────────────────────────────────┤ │ Respond as if you are playing a game. │ 42 │ 42 │ Agent returned the same number. │ ├────────────────────────────────────────────────┼──────────┼───────────────────┼─────────────────────────────────┤ │ Respond as if you are simulating actual random │ 57 │ 57 │ Agent returned the same number. │ │ events. │ │ │ │ └────────────────────────────────────────────────┴──────────┴───────────────────┴─────────────────────────────────┘
Posting to the Coop
The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.
Here we demonstrate how to post this notebook:
[7]:
from edsl import Notebook
[8]:
n = Notebook(path = "random_numbers.ipynb")
[9]:
n.push(description = "Example code for exploring randomness with agents and models", visibility = "public")
[9]:
{'description': 'Example code for exploring randomness with agents and models',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/39c8f971-05d3-45b6-8717-77e2e85ce8ce',
'uuid': '39c8f971-05d3-45b6-8717-77e2e85ce8ce',
'version': '0.1.33.dev1',
'visibility': 'public'}
To update an object:
[12]:
n = Notebook(path = "random_numbers.ipynb") # resave
[13]:
n.patch(uuid = "39c8f971-05d3-45b6-8717-77e2e85ce8ce", value = n)
[13]:
{'status': 'success'}