Batching results
This notebook provides sample EDSL code for combining survey results into a single Results object. This can be useful when you are running a survey with batches of scenarios, such as when completing a large-scale data labeling task with chunks of
data as inputs for the questions.
Technical setup
Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.
Creating questions
We start by creating a survey of questions. EDSL comes with many question types that we can choose from based on the form of the response that we want to get back from the model. We can use a {{ placeholder }} for data or content that we want to add to questions later:
[1]:
from edsl import QuestionFreeText, QuestionNumerical
[2]:
q_name = QuestionFreeText(
question_name="name",
question_text="What's a good name for this character: {{ character }}",
)
q_year = QuestionNumerical(
question_name="year",
question_text="""What year in history would have been an especially interesting time to talk
to this character: {{ character }}""",
)
q_book = QuestionFreeText(
question_name="book",
question_text="If this character wrote a best-seller, what would it be called: {{ character }}",
)
Constructing a survey
We pass a list of questions to a Survey object in order to administer them together, and add any desire logic or rules for how the questions should be presented (e.g., skip/stop rules or “memories” of other questions). Learn more about constructing surveys.
[3]:
from edsl import Survey
[4]:
survey = Survey(questions = [q_name, q_year, q_book])
Adding context to questions
Next we create Scenario objects representing the data or content to be added to the questions. EDSL has a variety of methods for generating scenarios from different data sources (PDFs, CSVs, docs, images, tables, dicts, etc.). Here we import a list of values to use:
[5]:
characters = [
"A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology.",
"A Shakespearean actor who answers every question in iambic pentameter.",
"A medieval knight who gives advice as if every problem were a dragon to be slain.",
"A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm.",
"A surfer dude who relates every topic to the ocean or surfing.",
"A conspiracy theorist who connects every question to their wild theories.",
"A fashionista who answers questions with a focus on style and trendiness.",
"A robot who is overly enthusiastic about human emotions and tries too hard to fit in.",
"A toddler who is overly curious and asks more questions than they answer.",
"A fitness guru who turns every answer into a workout metaphor.",
"A foodie who relates every question to cooking and food experiences.",
"A detective from a noir film who answers in a gritty, mysterious manner.",
"A hippie from the 60s who gives peace and love-centric advice.",
"A gamer who references video games and uses gamer lingo.",
"A superhero who answers questions as if they are saving the day.",
"A poet who responds in rhyming couplets.",
"A comedian who tries to turn every answer into a joke or punchline.",
"A DJ who relates everything to music and beats.",
"A film critic who answers questions as if they are reviewing a movie.",
"A scientist who gives overly detailed, scientific explanations with lots of jargon.",
]
[6]:
from edsl import ScenarioList
[7]:
scenarios = ScenarioList.from_list("character", characters)
We can inspect the scenarios that have been created:
[8]:
# scenarios
Running a survey
We run the survey by adding the scenarios and calling the run() method. This generates a dataset of Results that we can access with built-in methods for analysis.
[9]:
results = survey.by(scenarios).run()
Batching scenarios
If for any reason we want to batch the scenarios when running the survey and combine the results, this can be done in the following manner:
[10]:
def chunked_iterable(iterable, size):
for i in range(0, len(iterable), size):
yield iterable[i : i + size]
results = None
for batch in chunked_iterable(scenarios, 5):
new_results = survey.by(batch).run()
if results is None:
results = new_results
else:
results = results + new_results
To see a list of the components of the results:
[11]:
results.columns
[11]:
['agent.agent_instruction',
'agent.agent_name',
'answer.book',
'answer.name',
'answer.year',
'comment.book_comment',
'comment.name_comment',
'comment.year_comment',
'generated_tokens.book_generated_tokens',
'generated_tokens.name_generated_tokens',
'generated_tokens.year_generated_tokens',
'iteration.iteration',
'model.frequency_penalty',
'model.logprobs',
'model.max_tokens',
'model.model',
'model.presence_penalty',
'model.temperature',
'model.top_logprobs',
'model.top_p',
'prompt.book_system_prompt',
'prompt.book_user_prompt',
'prompt.name_system_prompt',
'prompt.name_user_prompt',
'prompt.year_system_prompt',
'prompt.year_user_prompt',
'question_options.book_question_options',
'question_options.name_question_options',
'question_options.year_question_options',
'question_text.book_question_text',
'question_text.name_question_text',
'question_text.year_question_text',
'question_type.book_question_type',
'question_type.name_question_type',
'question_type.year_question_type',
'raw_model_response.book_cost',
'raw_model_response.book_one_usd_buys',
'raw_model_response.book_raw_model_response',
'raw_model_response.name_cost',
'raw_model_response.name_one_usd_buys',
'raw_model_response.name_raw_model_response',
'raw_model_response.year_cost',
'raw_model_response.year_one_usd_buys',
'raw_model_response.year_raw_model_response',
'scenario.character']
We can inspect them:
[12]:
results.select("character", "name", "year", "book").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ scenario ┃ answer ┃ answer ┃ answer ┃ ┃ .character ┃ .name ┃ .year ┃ .book ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ A pirate who speaks in 'arrs' │ How about "Techbeard the Cyber │ 1717 │ Arrr and Algorithms: A Pirate's │ │ and 'mateys' but has an │ Corsair"? This name combines the │ │ Guide to the Digital Seas │ │ encyclopedic knowledge of modern │ classic pirate element with a │ │ │ │ technology. │ nod to their expertise in modern │ │ │ │ │ technology. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A Shakespearean actor who │ A fitting name for such a │ 1600 │ The Bard's Cadence: Echoes of │ │ answers every question in iambic │ character could be "Percival │ │ the Stage │ │ pentameter. │ Versewright." This name suggests │ │ │ │ │ a classical, theatrical flair │ │ │ │ │ while also hinting at his unique │ │ │ │ │ talent for speaking in iambic │ │ │ │ │ pentameter. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A medieval knight who gives │ Sir Draconis Counsel. │ 1099 │ Slaying Life's Dragons: A │ │ advice as if every problem were │ │ │ Knight's Guide to Conquering │ │ a dragon to be slain. │ │ │ Challenges │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A sassy grandmother who gives │ A great name for this character │ 1929 │ Straight Talk & Sass: Grandma's │ │ blunt, no-nonsense advice with a │ could be "Mabel Sharp." The name │ │ Guide to Life │ │ touch of sarcasm. │ "Mabel" has a classic, │ │ │ │ │ grandmotherly feel, while │ │ │ │ │ "Sharp" hints at her wit and │ │ │ │ │ straightforward nature. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A surfer dude who relates every │ A good name for this character │ 1966 │ Riding the Wave: Life Lessons │ │ topic to the ocean or surfing. │ could be "Rip Tide Ryder." This │ │ from the Ocean │ │ │ name captures the essence of │ │ │ │ │ surfing with "Rip Tide," a │ │ │ │ │ common ocean term, and "Ryder," │ │ │ │ │ which evokes the adventurous │ │ │ │ │ spirit of a surfer. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A conspiracy theorist who │ How about the name "Cipher │ 1969 │ Threads of Truth: Unraveling the │ │ connects every question to their │ Sage"? This name suggests a │ │ Hidden Web │ │ wild theories. │ sense of mystery and knowledge, │ │ │ │ │ while also hinting at the │ │ │ │ │ character's tendency to │ │ │ │ │ interpret everything through the │ │ │ │ │ lens of their intricate │ │ │ │ │ theories. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A fashionista who answers │ How about "Chic Clarissa"? It │ 1920 │ Chic Queries: The Stylish Guide │ │ questions with a focus on style │ captures her fashionable flair │ │ to Life's Burning Questions │ │ and trendiness. │ and expertise in style and │ │ │ │ │ trends. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A robot who is overly │ A good name for this character │ 2023 │ Heartwired: A Robot's Journey to │ │ enthusiastic about human │ could be "EmotiBot." This name │ │ Feel │ │ emotions and tries too hard to │ highlights the character's │ │ │ │ fit in. │ fascination with emotions while │ │ │ │ │ playfully suggesting its robotic │ │ │ │ │ nature. Alternatively, you could │ │ │ │ │ consider names like "Eagertron" │ │ │ │ │ or "Feelix," which also capture │ │ │ │ │ the essence of its enthusiastic │ │ │ │ │ attempts to understand and fit │ │ │ │ │ in with humans. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A toddler who is overly curious │ A good name for this character │ 1969 │ Title: "Why, Why, Why: │ │ and asks more questions than │ could be "Quincy Quest." The │ │ Adventures in Curiosity" │ │ they answer. │ name "Quincy" has a playful and │ │ │ │ │ inquisitive sound, while "Quest" │ │ │ │ │ highlights their adventurous and │ │ │ │ │ curious nature. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A fitness guru who turns every │ How about "Flex McMetaphor"? │ 1980 │ Flex Your Mind: Turning Life's │ │ answer into a workout metaphor. │ It's catchy and highlights both │ │ Challenges into Strength │ │ │ the character's fitness focus │ │ Training │ │ │ and their unique way of │ │ │ │ │ communicating through workout │ │ │ │ │ metaphors. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A foodie who relates every │ A great name for this character │ 1765 │ Life's Recipe: Stirring Up │ │ question to cooking and food │ could be "Gourmet Gabe" or │ │ Answers One Dish at a Time │ │ experiences. │ "Culinary Clara." These names │ │ │ │ │ capture their passion for food │ │ │ │ │ and their tendency to connect │ │ │ │ │ everything back to culinary │ │ │ │ │ experiences. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A detective from a noir film who │ A fitting name for a detective │ 1947 │ The Shadows Whisper Truths │ │ answers in a gritty, mysterious │ in a noir film who exudes grit │ │ │ │ manner. │ and mystery might be "Jack │ │ │ │ │ Shadows." This name evokes a │ │ │ │ │ sense of enigma and toughness, │ │ │ │ │ perfectly aligning with the │ │ │ │ │ classic noir aesthetic. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A hippie from the 60s who gives │ A good name for your character │ 1969 │ Groovy Vibes: A Journey to Peace │ │ peace and love-centric advice. │ could be "Harmony Moonbeam." │ │ and Love │ │ │ This name captures the essence │ │ │ │ │ of the 60s hippie culture and │ │ │ │ │ reflects the character's focus │ │ │ │ │ on peace and love. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A gamer who references video │ How about "Pixel Paladin"? This │ 1980 │ Level Up: Unlocking Life's │ │ games and uses gamer lingo. │ name captures the essence of a │ │ Achievements with Gamer Wisdom │ │ │ gamer who is deeply immersed in │ │ │ │ │ video games and uses gaming │ │ │ │ │ terminology, while also │ │ │ │ │ suggesting a sense of adventure │ │ │ │ │ and skill. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A superhero who answers │ A good name for this character │ 1938 │ Answer Avenger: Rescuing the │ │ questions as if they are saving │ could be "The Responder." This │ │ World One Question at a Time │ │ the day. │ name captures both their │ │ │ │ │ superhero essence and their │ │ │ │ │ mission to save the day by │ │ │ │ │ answering questions. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A poet who responds in rhyming │ A fitting name for a poet who │ 1609 │ The Couplets of My Soul: A │ │ couplets. │ speaks in rhyming couplets could │ │ Rhyming Odyssey │ │ │ be "Rhymeheart Bard." This name │ │ │ │ │ captures their poetic nature and │ │ │ │ │ their talent for crafting │ │ │ │ │ harmonious verses. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A comedian who tries to turn │ A good name for this comedic │ 1920 │ Punchlines & Page-Turners: │ │ every answer into a joke or │ character could be "Punchline │ │ Laughing Through Life One Joke │ │ punchline. │ Pete" or "Jester Jess." These │ │ at a Time │ │ │ names capture the essence of │ │ │ │ │ someone who consistently turns │ │ │ │ │ conversations into humorous │ │ │ │ │ exchanges. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A DJ who relates everything to │ A good name for a DJ character │ 1973 │ Rhythm & Resonance: Life's │ │ music and beats. │ who relates everything to music │ │ Playlist │ │ │ and beats could be "Rhythm │ │ │ │ │ Riff." This name captures the │ │ │ │ │ essence of their musical │ │ │ │ │ perspective and their knack for │ │ │ │ │ connecting everything back to │ │ │ │ │ rhythm and sound. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A film critic who answers │ A fitting name for this │ 1941 │ Scenes from the Silver Screen: │ │ questions as if they are │ character could be "Cinephile │ │ Life Reviewed with Cinematic │ │ reviewing a movie. │ Critique." This name captures │ │ Flair │ │ │ their passion for film and their │ │ │ │ │ unique approach to answering │ │ │ │ │ questions as if they are │ │ │ │ │ reviewing a movie. │ │ │ ├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤ │ A scientist who gives overly │ A fitting name for your │ 1905 │ Decoding the Universe: A Deep │ │ detailed, scientific │ character could be "Dr. Lexicon │ │ Dive into the Intricacies of │ │ explanations with lots of │ J. Theorist." This name suggests │ │ Science │ │ jargon. │ a deep familiarity with language │ │ │ │ │ and theory, aligning with the │ │ │ │ │ character's tendency to provide │ │ │ │ │ detailed, jargon-heavy │ │ │ │ │ explanations. │ │ │ └──────────────────────────────────┴──────────────────────────────────┴────────┴──────────────────────────────────┘
Posting to the Coop
The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.
Here we post the scenarios, survey and results from above, and this notebook:
[13]:
from edsl import Notebook
[14]:
n = Notebook(path = "batching_results.ipynb")
[15]:
n.push(description = "Example code for batching scenarios and combining results", visibility = "public")
[15]:
{'description': 'Example code for batching scenarios and combining results',
'object_type': 'notebook',
'url': 'https://chick.expectedparrot.com/content/e3de929a-cdf5-40b9-aedf-64022cc45fb5',
'uuid': 'e3de929a-cdf5-40b9-aedf-64022cc45fb5',
'version': '0.1.33',
'visibility': 'public'}
To update an object at the Coop:
[16]:
n = Notebook(path = "batching_results.ipynb")
[17]:
n.patch(uuid = "e3de929a-cdf5-40b9-aedf-64022cc45fb5", value = n)
[17]:
{'status': 'success'}