Batching results

This notebook provides sample EDSL code for combining survey results into a single Results object. This can be useful when you are running a survey with batches of scenarios, such as when completing a large-scale data labeling task with chunks of data as inputs for the questions.

Technical setup

Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Creating questions

We start by creating a survey of questions. EDSL comes with many question types that we can choose from based on the form of the response that we want to get back from the model. We can use a {{ placeholder }} for data or content that we want to add to questions later:

from edsl import QuestionFreeText, QuestionNumerical

q_name = QuestionFreeText(
    question_name="name",
    question_text="What's a good name for this character: {{ character }}",
)

q_year = QuestionNumerical(
    question_name="year",
    question_text="""What year in history would have been an especially interesting time to talk
    to this character: {{ character }}""",
)

q_book = QuestionFreeText(
    question_name="book",
    question_text="If this character wrote a best-seller, what would it be called: {{ character }}",
)

Constructing a survey

We pass a list of questions to a Survey object in order to administer them together, and add any desire logic or rules for how the questions should be presented (e.g., skip/stop rules or “memories” of other questions). Learn more about constructing surveys.

from edsl import Survey

survey = Survey(questions = [q_name, q_year, q_book])

Adding context to questions

Next we create Scenario objects representing the data or content to be added to the questions. EDSL has a variety of methods for generating scenarios from different data sources (PDFs, CSVs, docs, images, tables, dicts, etc.). Here we import a list of values to use:

characters = [
    "A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology.",
    "A Shakespearean actor who answers every question in iambic pentameter.",
    "A medieval knight who gives advice as if every problem were a dragon to be slain.",
    "A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm.",
    "A surfer dude who relates every topic to the ocean or surfing.",
    "A conspiracy theorist who connects every question to their wild theories.",
    "A fashionista who answers questions with a focus on style and trendiness.",
    "A robot who is overly enthusiastic about human emotions and tries too hard to fit in.",
    "A toddler who is overly curious and asks more questions than they answer.",
    "A fitness guru who turns every answer into a workout metaphor.",
    "A foodie who relates every question to cooking and food experiences.",
    "A detective from a noir film who answers in a gritty, mysterious manner.",
    "A hippie from the 60s who gives peace and love-centric advice.",
    "A gamer who references video games and uses gamer lingo.",
    "A superhero who answers questions as if they are saving the day.",
    "A poet who responds in rhyming couplets.",
    "A comedian who tries to turn every answer into a joke or punchline.",
    "A DJ who relates everything to music and beats.",
    "A film critic who answers questions as if they are reviewing a movie.",
    "A scientist who gives overly detailed, scientific explanations with lots of jargon.",
]

from edsl import ScenarioList

scenarios = ScenarioList.from_list("character", characters)

We can inspect the scenarios that have been created:

# scenarios

Running a survey

We run the survey by adding the scenarios and calling the run() method. This generates a dataset of Results that we can access with built-in methods for analysis.

results = survey.by(scenarios).run()

Remote inference activated. Sending job to server...
Job sent to server. (Job uuid=615eefa8-fd0d-4dda-8642-3a1e0b8a35a5).

View job progress here:
https://www.expectedparrot.com/home/remote-job-progress/615eefa8-fd0d-4dda-8642-3a1e0b8a35a5

Job completed and Results stored on Coop: https://www.expectedparrot.com/content/d1d687a8-bce6-4f94-b9aa-392cbc8e3d0d.

Batching scenarios

If for any reason we want to batch the scenarios when running the survey and combine the results, this can be done in the following manner:

def chunked_iterable(iterable, size):
    for i in range(0, len(iterable), size):
        yield iterable[i : i + size]


results = None

for batch in chunked_iterable(scenarios, 5):
    new_results = survey.by(batch).run()
    if results is None:
        results = new_results
    else:
        results = results + new_results

Remote inference activated. Sending job to server...
Job sent to server. (Job uuid=ab07ffd1-16cb-45db-a44a-f834c32e25ad).

View job progress here:
https://www.expectedparrot.com/home/remote-job-progress/ab07ffd1-16cb-45db-a44a-f834c32e25ad

Job completed and Results stored on Coop: https://www.expectedparrot.com/content/51f1a934-7a88-49f5-937d-a30a8b897058.
Remote inference activated. Sending job to server...
Job sent to server. (Job uuid=73e90ebc-a8c8-4dc3-a100-163ecf6f21c0).

View job progress here:
https://www.expectedparrot.com/home/remote-job-progress/73e90ebc-a8c8-4dc3-a100-163ecf6f21c0

Job completed and Results stored on Coop: https://www.expectedparrot.com/content/9ef7e4ec-0c49-41aa-b006-9d276e9e9fbc.
Remote inference activated. Sending job to server...
Job sent to server. (Job uuid=e742711a-06eb-43e6-aaa1-a76671a3e305).

View job progress here:
https://www.expectedparrot.com/home/remote-job-progress/e742711a-06eb-43e6-aaa1-a76671a3e305

Job completed and Results stored on Coop: https://www.expectedparrot.com/content/03879fd8-91cf-40e6-ad1b-972a3ac785f3.
Remote inference activated. Sending job to server...
Job sent to server. (Job uuid=1fc53ff7-dab2-46dc-85af-b9dff7f68373).

View job progress here:
https://www.expectedparrot.com/home/remote-job-progress/1fc53ff7-dab2-46dc-85af-b9dff7f68373

Job completed and Results stored on Coop: https://www.expectedparrot.com/content/e6a55ffc-a6f0-4ce3-a346-bc3f0824d0c2.

To see a list of the components of the results:

results.columns

['agent.agent_instruction',
 'agent.agent_name',
 'answer.book',
 'answer.name',
 'answer.year',
 'comment.book_comment',
 'comment.name_comment',
 'comment.year_comment',
 'generated_tokens.book_generated_tokens',
 'generated_tokens.name_generated_tokens',
 'generated_tokens.year_generated_tokens',
 'iteration.iteration',
 'model.frequency_penalty',
 'model.logprobs',
 'model.max_tokens',
 'model.model',
 'model.presence_penalty',
 'model.temperature',
 'model.top_logprobs',
 'model.top_p',
 'prompt.book_system_prompt',
 'prompt.book_user_prompt',
 'prompt.name_system_prompt',
 'prompt.name_user_prompt',
 'prompt.year_system_prompt',
 'prompt.year_user_prompt',
 'question_options.book_question_options',
 'question_options.name_question_options',
 'question_options.year_question_options',
 'question_text.book_question_text',
 'question_text.name_question_text',
 'question_text.year_question_text',
 'question_type.book_question_type',
 'question_type.name_question_type',
 'question_type.year_question_type',
 'raw_model_response.book_cost',
 'raw_model_response.book_one_usd_buys',
 'raw_model_response.book_raw_model_response',
 'raw_model_response.name_cost',
 'raw_model_response.name_one_usd_buys',
 'raw_model_response.name_raw_model_response',
 'raw_model_response.year_cost',
 'raw_model_response.year_one_usd_buys',
 'raw_model_response.year_raw_model_response',
 'scenario.character']

We can inspect them:

results.select("character", "name", "year", "book")

scenario.character	answer.name	answer.year	answer.book
A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology.	How about "Techbeard the Cyber Corsair"? This name combines the classic pirate element with a nod to their expertise in modern technology.	1717	Arrr and Algorithms: A Pirate's Guide to the Digital Seas
A Shakespearean actor who answers every question in iambic pentameter.	A fitting name for such a character could be "Percival Versewright." This name suggests a classical, theatrical flair while also hinting at his unique talent for speaking in iambic pentameter.	1600	The Bard's Cadence: Echoes of the Stage
A medieval knight who gives advice as if every problem were a dragon to be slain.	Sir Draconis Counsel.	1099	Slaying Life's Dragons: A Knight's Guide to Conquering Challenges
A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm.	A great name for this character could be "Mabel Sharp." The name "Mabel" has a classic, grandmotherly feel, while "Sharp" hints at her wit and straightforward nature.	1945	Straight Talk & Sass: Grandma's Guide to Life
A surfer dude who relates every topic to the ocean or surfing.	A good name for this character could be "Rip Tide Ryder." This name captures the essence of surfing with "Rip Tide," a common ocean term, and "Ryder," which evokes the adventurous spirit of a surfer.	1966	Riding the Wave: Life Lessons from the Ocean
A conspiracy theorist who connects every question to their wild theories.	How about the name "Cipher Sage"? This name suggests a sense of mystery and knowledge, while also hinting at the character's tendency to interpret everything through the lens of their intricate theories.	1969	Threads of Truth: Unraveling the Hidden Web
A fashionista who answers questions with a focus on style and trendiness.	How about "Chic Clarissa"? It captures her fashionable flair and expertise in style and trends.	1920	Chic Queries: The Stylish Guide to Life's Burning Questions
A robot who is overly enthusiastic about human emotions and tries too hard to fit in.	A good name for this character could be "EmotiBot." This name highlights the character's fascination with emotions while playfully suggesting its robotic nature. Alternatively, you could consider names like "Eagertron" or "Feelix," which also capture the essence of its enthusiastic attempts to understand and fit in with humans.	2023	Heartwired: A Robot's Journey to Feel
A toddler who is overly curious and asks more questions than they answer.	A good name for this character could be "Quincy Quest." The name "Quincy" has a playful and inquisitive sound, while "Quest" highlights their adventurous and curious nature.	1776	Title: "Why, Why, Why: Adventures in Curiosity"
A fitness guru who turns every answer into a workout metaphor.	How about "Flex McMetaphor"? It's catchy and highlights both the character's fitness focus and their unique way of communicating through workout metaphors.	1980	Flex Your Mind: Turning Life's Challenges into Strength Training
A foodie who relates every question to cooking and food experiences.	A great name for this character could be "Gourmet Gabe" or "Culinary Clara." These names capture their passion for food and their tendency to connect everything back to culinary experiences.	1765	Life's Recipe: Stirring Up Answers One Dish at a Time
A detective from a noir film who answers in a gritty, mysterious manner.	A fitting name for a detective in a noir film who exudes grit and mystery might be "Jack Shadows." This name evokes a sense of enigma and toughness, perfectly aligning with the classic noir aesthetic.	1940	The Shadows Whisper Truths
A hippie from the 60s who gives peace and love-centric advice.	A good name for your character could be "Harmony Moonbeam." This name captures the essence of the 60s hippie culture and reflects the character's focus on peace and love.	1969	Groovy Vibes: A Journey to Peace and Love
A gamer who references video games and uses gamer lingo.	How about "Pixel Paladin"? This name captures the essence of a gamer who is deeply immersed in video games and uses gaming terminology, while also suggesting a sense of adventure and skill.	1983	Level Up: Unlocking Life's Achievements with Gamer Wisdom
A superhero who answers questions as if they are saving the day.	A good name for this character could be "The Responder." This name captures both their superhero essence and their mission to save the day by answering questions.	1938	Answer Avenger: Rescuing the World One Question at a Time
A poet who responds in rhyming couplets.	A fitting name for a poet who speaks in rhyming couplets could be "Rhymeheart Bard." This name captures their poetic nature and their talent for crafting harmonious verses.	1609	The Couplets of My Soul: A Rhyming Odyssey
A comedian who tries to turn every answer into a joke or punchline.	A good name for this comedic character could be "Punchline Pete" or "Jester Jess." These names capture the essence of someone who consistently turns conversations into humorous exchanges.	1920	Punchlines & Page-Turners: Laughing Through Life One Joke at a Time
A DJ who relates everything to music and beats.	A good name for a DJ character who relates everything to music and beats could be "Rhythm Riff." This name captures the essence of their musical perspective and their knack for connecting everything back to rhythm and sound.	1973	Rhythm & Resonance: Life's Playlist
A film critic who answers questions as if they are reviewing a movie.	A fitting name for this character could be "Cinephile Critique." This name captures their passion for film and their unique approach to answering questions as if they are reviewing a movie.	1927	Scenes from the Silver Screen: Life Reviewed with Cinematic Flair
A scientist who gives overly detailed, scientific explanations with lots of jargon.	A fitting name for your character could be "Dr. Lexicon J. Theorist." This name suggests a deep familiarity with language and theory, aligning with the character's tendency to provide detailed, jargon-heavy explanations.	1905	Decoding the Universe: A Deep Dive into the Intricacies of Science

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

Here we post the scenarios, survey and results from above, and this notebook:

from edsl import Notebook

n = Notebook(path = "batching_results.ipynb")

n.push(description = "Example code for batching scenarios and combining results", visibility = "public")

{'description': 'Example code for batching scenarios and combining results',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/ce552a6a-a190-473a-927b-ee5409d0da83',
 'uuid': 'ce552a6a-a190-473a-927b-ee5409d0da83',
 'version': '0.1.38.dev1',
 'visibility': 'public'}