Batching results
This notebook provides sample EDSL code for combining survey results into a single Results object. This can be useful when you are running a survey with batches of scenarios, such as when completing a large-scale data labeling task with chunks of
data as inputs for the questions.
EDSL is an open-source library for simulating surveys and experiments with AI. Please see our documentation page for tips and tutorials on getting started.
[1]:
# pip install edsl
Importing the tools:
[2]:
from edsl.questions import QuestionFreeText, QuestionNumerical, QuestionLinearScale
from edsl import Scenario, Survey
Creating a survey of questions:
[3]:
q_name = QuestionFreeText(
question_name="name",
question_text="What's a good name for this character: {{ character }}",
)
q_year = QuestionNumerical(
question_name="year",
question_text="""What year in history would have been an especially interesting time to talk
to this character: {{ character }}""",
)
q_book = QuestionFreeText(
question_name="book",
question_text="If this character wrote a best-seller, what would it be called: {{ character }}",
)
q_talk = QuestionLinearScale(
question_name="talk",
question_text="""On a scale from 0 to 10, how much would you enjoy talking to this character:
{{ character }}""",
question_options=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
option_labels={0: "Not at all", 10: "Very much"},
)
survey = Survey([q_name, q_year, q_book, q_talk])
Some data to create scenarios of the questions:
[4]:
characters = [
"A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology.",
"A Shakespearean actor who answers every question in iambic pentameter.",
"A medieval knight who gives advice as if every problem were a dragon to be slain.",
"A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm.",
"A surfer dude who relates every topic to the ocean or surfing.",
"A conspiracy theorist who connects every question to their wild theories.",
"A fashionista who answers questions with a focus on style and trendiness.",
"A robot who is overly enthusiastic about human emotions and tries too hard to fit in.",
"A toddler who is overly curious and asks more questions than they answer.",
"A fitness guru who turns every answer into a workout metaphor.",
"A foodie who relates every question to cooking and food experiences.",
"A detective from a noir film who answers in a gritty, mysterious manner.",
"A hippie from the 60s who gives peace and love-centric advice.",
"A gamer who references video games and uses gamer lingo.",
"A superhero who answers questions as if they are saving the day.",
"A poet who responds in rhyming couplets.",
"A comedian who tries to turn every answer into a joke or punchline.",
"A DJ who relates everything to music and beats.",
"A film critic who answers questions as if they are reviewing a movie.",
"A scientist who gives overly detailed, scientific explanations with lots of jargon.",
]
[5]:
scenarios = [Scenario({"character": c}) for c in characters]
Run the survey with batches of scenarios, combining the results into a single Results object:
[6]:
def chunked_iterable(iterable, size):
for i in range(0, len(iterable), size):
yield iterable[i : i + size]
results = None
for batch in chunked_iterable(scenarios, 5):
new_results = survey.by(batch).run()
if results is None:
results = new_results
else:
results = results + new_results
Access all the Results methods:
[7]:
results.columns
[7]:
['agent.agent_instruction',
'agent.agent_name',
'answer.book',
'answer.name',
'answer.talk',
'answer.year',
'comment.talk_comment',
'comment.year_comment',
'iteration.iteration',
'model.frequency_penalty',
'model.logprobs',
'model.max_tokens',
'model.model',
'model.presence_penalty',
'model.temperature',
'model.top_logprobs',
'model.top_p',
'prompt.book_system_prompt',
'prompt.book_user_prompt',
'prompt.name_system_prompt',
'prompt.name_user_prompt',
'prompt.talk_system_prompt',
'prompt.talk_user_prompt',
'prompt.year_system_prompt',
'prompt.year_user_prompt',
'question_options.book_question_options',
'question_options.name_question_options',
'question_options.talk_question_options',
'question_options.year_question_options',
'question_text.book_question_text',
'question_text.name_question_text',
'question_text.talk_question_text',
'question_text.year_question_text',
'question_type.book_question_type',
'question_type.name_question_type',
'question_type.talk_question_type',
'question_type.year_question_type',
'raw_model_response.book_raw_model_response',
'raw_model_response.name_raw_model_response',
'raw_model_response.talk_raw_model_response',
'raw_model_response.year_raw_model_response',
'scenario.character',
'scenario.edsl_class_name',
'scenario.edsl_version']
[8]:
results.select("character", "name", "year", "book", "talk").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓ ┃ scenario ┃ answer ┃ answer ┃ answer ┃ answer ┃ ┃ .character ┃ .name ┃ .year ┃ .book ┃ .talk ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩ │ A pirate who speaks in 'arrs' │ Techbeard the Cyber Corsair │ 1720 │ The Tech-Savvy Buccaneer: │ 9 │ │ and 'mateys' but has an │ │ │ Arrs, Algorithms, and │ │ │ encyclopedic knowledge of │ │ │ Adventures on the Digital Seas │ │ │ modern technology. │ │ │ │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A Shakespearean actor who │ Pentameter Peter │ 1600 │ The Pentameter Protagonist: │ 8 │ │ answers every question in │ │ │ Verses of Inquiry and Response │ │ │ iambic pentameter. │ │ │ │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A medieval knight who gives │ Sir Counsel the Dragonwise │ 1348 │ The Knight's Guide to │ 9 │ │ advice as if every problem │ │ │ Dragon-Slaying: Conquering │ │ │ were a dragon to be slain. │ │ │ Life's Challenges with Valor │ │ │ │ │ │ and Sword │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A sassy grandmother who gives │ Granny Hazel │ 1920 │ Pearls of Wisdom: Tart and │ 8 │ │ blunt, no-nonsense advice with │ │ │ Truthful Life Lessons from │ │ │ a touch of sarcasm. │ │ │ Grandma's Porch │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A surfer dude who relates │ Kai WaveRider │ 1960 │ Wave Philosophy: Surfing the │ 7 │ │ every topic to the ocean or │ │ │ Tides of Life │ │ │ surfing. │ │ │ │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A conspiracy theorist who │ Conrad Query │ 1947 │ The Grand Scheme: Unveiling │ 5 │ │ connects every question to │ │ │ the Web of Universal │ │ │ their wild theories. │ │ │ Connections │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A fashionista who answers │ Trendy Trudy │ 1960 │ Trendsetting Queries: A │ 8 │ │ questions with a focus on │ │ │ Fashionista's Guide to Stylish │ │ │ style and trendiness. │ │ │ Answers │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A robot who is overly │ EmotiBot │ 0 │ Too Human: The Overzealous │ 8 │ │ enthusiastic about human │ │ │ Robot's Guide to Mastering │ │ │ emotions and tries too hard to │ │ │ Emotions │ │ │ fit in. │ │ │ │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A toddler who is overly │ Parker Query │ 0 │ The Little Inquisitor: │ 9 │ │ curious and asks more │ │ │ Adventures in 'Why?' │ │ │ questions than they answer. │ │ │ │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A fitness guru who turns every │ Flex Wheeler │ 1980 │ Pumping Prose: Turning │ 7 │ │ answer into a workout │ │ │ Literary Success into Muscular │ │ │ metaphor. │ │ │ Metaphors │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A foodie who relates every │ Gastro Gary │ 1784 │ Savoring Life: A Culinary │ 8 │ │ question to cooking and food │ │ │ Journey Through the Questions │ │ │ experiences. │ │ │ We Crave │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A detective from a noir film │ Max Sterling │ 1945 │ Shadows Over the City: Tales │ 8 │ │ who answers in a gritty, │ │ │ of a Hard-Boiled Detective │ │ │ mysterious manner. │ │ │ │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A hippie from the 60s who │ Harmony Sage │ 1967 │ Groovy Guidance: The │ 7 │ │ gives peace and love-centric │ │ │ Flower-Powered Path to Peace │ │ │ advice. │ │ │ and Love │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A gamer who references video │ PixelPwnr │ 2023 │ Respawn: Leveling Up in the │ 9 │ │ games and uses gamer lingo. │ │ │ Game of Life │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A superhero who answers │ Factfinder Flash │ 1938 │ The Query Crusader: Tales of │ 8 │ │ questions as if they are │ │ │ the Answer Avenger │ │ │ saving the day. │ │ │ │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A poet who responds in rhyming │ Rhyme Master Bard │ 0 │ Verses in Tandem: A Poet's │ 8 │ │ couplets. │ │ │ Journey Through Rhyme │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A comedian who tries to turn │ Chuck Laffs │ 0 │ The Last Laugh: Turning Life's │ 7 │ │ every answer into a joke or │ │ │ Questions into Punchlines │ │ │ punchline. │ │ │ │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A DJ who relates everything to │ DJ BeatSync │ 1970 │ Life on the Beat: Syncing │ 8 │ │ music and beats. │ │ │ Rhythms with Reality │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A film critic who answers │ Cinematic Scribe │ 1927 │ Cinematic Inquiries: A │ 8 │ │ questions as if they are │ │ │ Critic's Journey Through the │ │ │ reviewing a movie. │ │ │ Art of Q&A │ │ ├────────────────────────────────┼─────────────────────────────┼────────┼────────────────────────────────┼────────┤ │ A scientist who gives overly │ Dr. Lexicon Pendanticus │ 1905 │ The Comprehensive Compendium │ 7 │ │ detailed, scientific │ │ │ of Complicated Concepts: A │ │ │ explanations with lots of │ │ │ Scholar's Guide to the │ │ │ jargon. │ │ │ Esoteric Intricacies of the │ │ │ │ │ │ Universe │ │ └────────────────────────────────┴─────────────────────────────┴────────┴────────────────────────────────┴────────┘