Batching results

This notebook provides sample EDSL code for combining survey results into a single Results object. This can be useful when you are running a survey with batches of scenarios, such as when completing a large-scale data labeling task with chunks of data as inputs for the questions.

Technical setup

Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Creating questions

We start by creating a survey of questions. EDSL comes with many question types that we can choose from based on the form of the response that we want to get back from the model. We can use a {{ placeholder }} for data or content that we want to add to questions later:

from edsl import QuestionFreeText, QuestionNumerical

q_name = QuestionFreeText(
    question_name="name",
    question_text="What's a good name for this character: {{ character }}",
)

q_year = QuestionNumerical(
    question_name="year",
    question_text="""What year in history would have been an especially interesting time to talk
    to this character: {{ character }}""",
)

q_book = QuestionFreeText(
    question_name="book",
    question_text="If this character wrote a best-seller, what would it be called: {{ character }}",
)

Constructing a survey

We pass a list of questions to a Survey object in order to administer them together, and add any desire logic or rules for how the questions should be presented (e.g., skip/stop rules or “memories” of other questions). Learn more about constructing surveys.

from edsl import Survey

survey = Survey(questions = [q_name, q_year, q_book])

Adding context to questions

Next we create Scenario objects representing the data or content to be added to the questions. EDSL has a variety of methods for generating scenarios from different data sources (PDFs, CSVs, docs, images, tables, dicts, etc.). Here we import a list of values to use:

characters = [
    "A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology.",
    "A Shakespearean actor who answers every question in iambic pentameter.",
    "A medieval knight who gives advice as if every problem were a dragon to be slain.",
    "A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm.",
    "A surfer dude who relates every topic to the ocean or surfing.",
    "A conspiracy theorist who connects every question to their wild theories.",
    "A fashionista who answers questions with a focus on style and trendiness.",
    "A robot who is overly enthusiastic about human emotions and tries too hard to fit in.",
    "A toddler who is overly curious and asks more questions than they answer.",
    "A fitness guru who turns every answer into a workout metaphor.",
    "A foodie who relates every question to cooking and food experiences.",
    "A detective from a noir film who answers in a gritty, mysterious manner.",
    "A hippie from the 60s who gives peace and love-centric advice.",
    "A gamer who references video games and uses gamer lingo.",
    "A superhero who answers questions as if they are saving the day.",
    "A poet who responds in rhyming couplets.",
    "A comedian who tries to turn every answer into a joke or punchline.",
    "A DJ who relates everything to music and beats.",
    "A film critic who answers questions as if they are reviewing a movie.",
    "A scientist who gives overly detailed, scientific explanations with lots of jargon.",
]

from edsl import ScenarioList

scenarios = ScenarioList.from_list("character", characters)

We can inspect the scenarios that have been created:

scenarios

ScenarioList scenarios: 20; keys: ['character'];

	character
0	A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology.
1	A Shakespearean actor who answers every question in iambic pentameter.
2	A medieval knight who gives advice as if every problem were a dragon to be slain.
3	A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm.
4	A surfer dude who relates every topic to the ocean or surfing.
5	A conspiracy theorist who connects every question to their wild theories.
6	A fashionista who answers questions with a focus on style and trendiness.
7	A robot who is overly enthusiastic about human emotions and tries too hard to fit in.
8	A toddler who is overly curious and asks more questions than they answer.
9	A fitness guru who turns every answer into a workout metaphor.
10	A foodie who relates every question to cooking and food experiences.
11	A detective from a noir film who answers in a gritty, mysterious manner.
12	A hippie from the 60s who gives peace and love-centric advice.
13	A gamer who references video games and uses gamer lingo.
14	A superhero who answers questions as if they are saving the day.
15	A poet who responds in rhyming couplets.
16	A comedian who tries to turn every answer into a joke or punchline.
17	A DJ who relates everything to music and beats.
18	A film critic who answers questions as if they are reviewing a movie.
19	A scientist who gives overly detailed, scientific explanations with lots of jargon.

Running a survey

We run the survey by adding any agent personas that we have created to answer the questions (in this example, none) and specifying language models to generate the responses. If no model is specified the default model (currently, GPT-4o) is used. Here we specifyit for demonstration purposes, and then call the run() method to administer the survey. This generates a dataset of Results that we can access with built-in methods for analysis.

from edsl import Model

model = Model("gpt-4o")

results = survey.by(scenarios).by(model).run()

▼ Job Status (2024-12-28 07:25:28)

Job UUID	5ee25ddd-66f1-4e32-bbe8-d3f412c4011e
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/5ee25ddd-66f1-4e32-bbe8-d3f412c4011e
Error Report URL	None
Results UUID	a49e5078-727a-4806-b5be-38eb9143c138
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/a49e5078-727a-4806-b5be-38eb9143c138

Batching scenarios

If for any reason we want to batch the scenarios when running the survey and combine the results, this can be done in the following manner:

def chunked_iterable(iterable, size):
    for i in range(0, len(iterable), size):
        yield iterable[i : i + size]


results = None

for batch in chunked_iterable(scenarios, 5):
    new_results = survey.by(batch).by(model).run()
    if results is None:
        results = new_results
    else:
        results = results + new_results

▼ Job Status (2024-12-28 07:25:44)

Job UUID	a5a1f026-84b3-40c9-8d34-2c48940d8967
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/a5a1f026-84b3-40c9-8d34-2c48940d8967
Error Report URL	None
Results UUID	586d107c-2ded-4ade-be89-346fc124fa08
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/586d107c-2ded-4ade-be89-346fc124fa08

▼ Job Status (2024-12-28 07:25:59)

Job UUID	d5098931-69dc-47bf-a3d6-8c5689ef183e
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/d5098931-69dc-47bf-a3d6-8c5689ef183e
Error Report URL	None
Results UUID	b22723f7-c9c8-49b9-8908-5576031b094b
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/b22723f7-c9c8-49b9-8908-5576031b094b

▼ Job Status (2024-12-28 07:26:14)

Job UUID	c8b3c97c-f3f8-407e-b7da-07a5d3a16363
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/c8b3c97c-f3f8-407e-b7da-07a5d3a16363
Error Report URL	None
Results UUID	0731f3ab-26f4-4f7f-88de-f35277d8dac8
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/0731f3ab-26f4-4f7f-88de-f35277d8dac8

▼ Job Status (2024-12-28 07:26:28)

Job UUID	cd32d003-f8d9-4583-9def-4b3f0ba95569
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/cd32d003-f8d9-4583-9def-4b3f0ba95569
Error Report URL	None
Results UUID	14295069-d7f9-41db-af30-64bc19741f6c
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/14295069-d7f9-41db-af30-64bc19741f6c

To see a list of the components of the results:

results.columns

	0
0	agent.agent_instruction
1	agent.agent_name
2	answer.book
3	answer.name
4	answer.year
5	comment.book_comment
6	comment.name_comment
7	comment.year_comment
8	generated_tokens.book_generated_tokens
9	generated_tokens.name_generated_tokens
10	generated_tokens.year_generated_tokens
11	iteration.iteration
12	model.frequency_penalty
13	model.logprobs
14	model.max_tokens
15	model.model
16	model.presence_penalty
17	model.temperature
18	model.top_logprobs
19	model.top_p
20	prompt.book_system_prompt
21	prompt.book_user_prompt
22	prompt.name_system_prompt
23	prompt.name_user_prompt
24	prompt.year_system_prompt
25	prompt.year_user_prompt
26	question_options.book_question_options
27	question_options.name_question_options
28	question_options.year_question_options
29	question_text.book_question_text
30	question_text.name_question_text
31	question_text.year_question_text
32	question_type.book_question_type
33	question_type.name_question_type
34	question_type.year_question_type
35	raw_model_response.book_cost
36	raw_model_response.book_one_usd_buys
37	raw_model_response.book_raw_model_response
38	raw_model_response.name_cost
39	raw_model_response.name_one_usd_buys
40	raw_model_response.name_raw_model_response
41	raw_model_response.year_cost
42	raw_model_response.year_one_usd_buys
43	raw_model_response.year_raw_model_response
44	scenario.character

We can inspect them:

results.select("model", "character", "name", "year", "book")

	model.model	scenario.character	answer.name	answer.year	answer.book
0	gpt-4o	A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology.	How about "Techbeard the Cyber Corsair"? This name combines the classic pirate element with a nod to their expertise in modern technology.	1717	Arrr and Algorithms: A Pirate's Guide to the Digital Seas
1	gpt-4o	A Shakespearean actor who answers every question in iambic pentameter.	A fitting name for such a character could be "Percival Versewright." This name suggests a classical, theatrical flair while also hinting at his unique talent for speaking in iambic pentameter.	1600	The Bard's Cadence: Echoes of the Stage
2	gpt-4o	A medieval knight who gives advice as if every problem were a dragon to be slain.	Sir Draconis Counsel.	1099	Slaying Life's Dragons: A Knight's Guide to Conquering Challenges
3	gpt-4o	A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm.	A great name for this character could be "Mabel Sharp." The name "Mabel" has a classic, grandmotherly feel, while "Sharp" hints at her wit and straightforward nature.	1945	Straight Talk & Sass: Grandma's Guide to Life
4	gpt-4o	A surfer dude who relates every topic to the ocean or surfing.	A good name for this character could be "Rip Tide Ryder." This name captures the essence of surfing with "Rip Tide," a common ocean term, and "Ryder," which evokes the adventurous spirit of a surfer.	1966	Riding the Wave: Life Lessons from the Ocean
5	gpt-4o	A conspiracy theorist who connects every question to their wild theories.	How about the name "Cipher Sage"? This name suggests a sense of mystery and knowledge, while also hinting at the character's tendency to interpret everything through the lens of their intricate theories.	1969	Threads of Truth: Unraveling the Hidden Web
6	gpt-4o	A fashionista who answers questions with a focus on style and trendiness.	How about "Chic Clarissa"? It captures her fashionable flair and expertise in style and trends.	1920	Chic Queries: The Stylish Guide to Life's Burning Questions
7	gpt-4o	A robot who is overly enthusiastic about human emotions and tries too hard to fit in.	A good name for this character could be "EmotiBot." This name highlights the character's fascination with emotions while playfully suggesting its robotic nature. Alternatively, you could consider names like "Eagertron" or "Feelix," which also capture the essence of its enthusiastic attempts to understand and fit in with humans.	2023	Heartwired: A Robot's Journey to Feel
8	gpt-4o	A toddler who is overly curious and asks more questions than they answer.	A good name for this character could be "Quincy Quest." The name "Quincy" has a playful and inquisitive sound, while "Quest" highlights their adventurous and curious nature.	1776	Title: "Why, Why, Why: Adventures in Curiosity"
9	gpt-4o	A fitness guru who turns every answer into a workout metaphor.	How about "Flex McMetaphor"? It's catchy and highlights both the character's fitness focus and their unique way of communicating through workout metaphors.	1980	Flex Your Mind: Turning Life's Challenges into Strength Training
10	gpt-4o	A foodie who relates every question to cooking and food experiences.	A great name for this character could be "Gourmet Gabe" or "Culinary Clara." These names capture their passion for food and their tendency to connect everything back to culinary experiences.	1765	Life's Recipe: Stirring Up Answers One Dish at a Time
11	gpt-4o	A detective from a noir film who answers in a gritty, mysterious manner.	A fitting name for a detective in a noir film who exudes grit and mystery might be "Jack Shadows." This name evokes a sense of enigma and toughness, perfectly aligning with the classic noir aesthetic.	1940	The Shadows Whisper Truths
12	gpt-4o	A hippie from the 60s who gives peace and love-centric advice.	A good name for your character could be "Harmony Moonbeam." This name captures the essence of the 60s hippie culture and reflects the character's focus on peace and love.	1969	Groovy Vibes: A Journey to Peace and Love
13	gpt-4o	A gamer who references video games and uses gamer lingo.	How about "Pixel Paladin"? This name captures the essence of a gamer who is deeply immersed in video games and uses gaming terminology, while also suggesting a sense of adventure and skill.	1983	Level Up: Unlocking Life's Achievements with Gamer Wisdom
14	gpt-4o	A superhero who answers questions as if they are saving the day.	A good name for this character could be "The Responder." This name captures both their superhero essence and their mission to save the day by answering questions.	1938	Answer Avenger: Rescuing the World One Question at a Time
15	gpt-4o	A poet who responds in rhyming couplets.	A fitting name for a poet who speaks in rhyming couplets could be "Rhymeheart Bard." This name captures their poetic nature and their talent for crafting harmonious verses.	1609	The Couplets of My Soul: A Rhyming Odyssey
16	gpt-4o	A comedian who tries to turn every answer into a joke or punchline.	A good name for this comedic character could be "Punchline Pete" or "Jester Jess." These names capture the essence of someone who consistently turns conversations into humorous exchanges.	1920	Punchlines & Page-Turners: Laughing Through Life One Joke at a Time
17	gpt-4o	A DJ who relates everything to music and beats.	A good name for a DJ character who relates everything to music and beats could be "Rhythm Riff." This name captures the essence of their musical perspective and their knack for connecting everything back to rhythm and sound.	1973	Rhythm & Resonance: Life's Playlist
18	gpt-4o	A film critic who answers questions as if they are reviewing a movie.	A fitting name for this character could be "Cinephile Critique." This name captures their passion for film and their unique approach to answering questions as if they are reviewing a movie.	1927	Scenes from the Silver Screen: Life Reviewed with Cinematic Flair
19	gpt-4o	A scientist who gives overly detailed, scientific explanations with lots of jargon.	A fitting name for your character could be "Dr. Lexicon J. Theorist." This name suggests a deep familiarity with language and theory, aligning with the character's tendency to provide detailed, jargon-heavy explanations.	1905	Decoding the Universe: A Deep Dive into the Intricacies of Science

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

Here we post the scenarios, survey and results from above, and this notebook:

from edsl import Notebook

n = Notebook(path = "batching_results.ipynb")

n.push(description = "Example code for batching scenarios and combining results", visibility = "public")

{'description': 'Example code for batching scenarios and combining results',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/8f6cd656-0184-45ea-b878-d5b6b567d3c9',
 'uuid': '8f6cd656-0184-45ea-b878-d5b6b567d3c9',
 'version': '0.1.39.dev2',
 'visibility': 'public'}

To update an object at Coop:

n = Notebook(path = "batching_results.ipynb") # resave

n.patch(uuid = "8f6cd656-0184-45ea-b878-d5b6b567d3c9", value = n)

{'status': 'success'}