Batching results

This notebook provides sample EDSL code for combining survey results into a single Results object. This can be useful when you are running a survey with batches of scenarios, such as when completing a large-scale data labeling task with chunks of data as inputs for the questions.

Technical setup

Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Creating questions

We start by creating a survey of questions. EDSL comes with many question types that we can choose from based on the form of the response that we want to get back from the model. We can use a {{ placeholder }} for data or content that we want to add to questions later:

[1]:
from edsl import QuestionFreeText, QuestionNumerical
[2]:
q_name = QuestionFreeText(
    question_name="name",
    question_text="What's a good name for this character: {{ character }}",
)

q_year = QuestionNumerical(
    question_name="year",
    question_text="""What year in history would have been an especially interesting time to talk
    to this character: {{ character }}""",
)

q_book = QuestionFreeText(
    question_name="book",
    question_text="If this character wrote a best-seller, what would it be called: {{ character }}",
)

Constructing a survey

We pass a list of questions to a Survey object in order to administer them together, and add any desire logic or rules for how the questions should be presented (e.g., skip/stop rules or “memories” of other questions). Learn more about constructing surveys.

[3]:
from edsl import Survey
[4]:
survey = Survey(questions = [q_name, q_year, q_book])

Adding context to questions

Next we create Scenario objects representing the data or content to be added to the questions. EDSL has a variety of methods for generating scenarios from different data sources (PDFs, CSVs, docs, images, tables, dicts, etc.). Here we import a list of values to use:

[5]:
characters = [
    "A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology.",
    "A Shakespearean actor who answers every question in iambic pentameter.",
    "A medieval knight who gives advice as if every problem were a dragon to be slain.",
    "A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm.",
    "A surfer dude who relates every topic to the ocean or surfing.",
    "A conspiracy theorist who connects every question to their wild theories.",
    "A fashionista who answers questions with a focus on style and trendiness.",
    "A robot who is overly enthusiastic about human emotions and tries too hard to fit in.",
    "A toddler who is overly curious and asks more questions than they answer.",
    "A fitness guru who turns every answer into a workout metaphor.",
    "A foodie who relates every question to cooking and food experiences.",
    "A detective from a noir film who answers in a gritty, mysterious manner.",
    "A hippie from the 60s who gives peace and love-centric advice.",
    "A gamer who references video games and uses gamer lingo.",
    "A superhero who answers questions as if they are saving the day.",
    "A poet who responds in rhyming couplets.",
    "A comedian who tries to turn every answer into a joke or punchline.",
    "A DJ who relates everything to music and beats.",
    "A film critic who answers questions as if they are reviewing a movie.",
    "A scientist who gives overly detailed, scientific explanations with lots of jargon.",
]
[6]:
from edsl import ScenarioList
[7]:
scenarios = ScenarioList.from_list("character", characters)

We can inspect the scenarios that have been created:

[8]:
# scenarios

Running a survey

We run the survey by adding the scenarios and calling the run() method. This generates a dataset of Results that we can access with built-in methods for analysis.

[9]:
results = survey.by(scenarios).run()

Batching scenarios

If for any reason we want to batch the scenarios when running the survey and combine the results, this can be done in the following manner:

[10]:
def chunked_iterable(iterable, size):
    for i in range(0, len(iterable), size):
        yield iterable[i : i + size]


results = None

for batch in chunked_iterable(scenarios, 5):
    new_results = survey.by(batch).run()
    if results is None:
        results = new_results
    else:
        results = results + new_results

To see a list of the components of the results:

[11]:
results.columns
[11]:
['agent.agent_instruction',
 'agent.agent_name',
 'answer.book',
 'answer.name',
 'answer.year',
 'comment.book_comment',
 'comment.name_comment',
 'comment.year_comment',
 'generated_tokens.book_generated_tokens',
 'generated_tokens.name_generated_tokens',
 'generated_tokens.year_generated_tokens',
 'iteration.iteration',
 'model.frequency_penalty',
 'model.logprobs',
 'model.max_tokens',
 'model.model',
 'model.presence_penalty',
 'model.temperature',
 'model.top_logprobs',
 'model.top_p',
 'prompt.book_system_prompt',
 'prompt.book_user_prompt',
 'prompt.name_system_prompt',
 'prompt.name_user_prompt',
 'prompt.year_system_prompt',
 'prompt.year_user_prompt',
 'question_options.book_question_options',
 'question_options.name_question_options',
 'question_options.year_question_options',
 'question_text.book_question_text',
 'question_text.name_question_text',
 'question_text.year_question_text',
 'question_type.book_question_type',
 'question_type.name_question_type',
 'question_type.year_question_type',
 'raw_model_response.book_cost',
 'raw_model_response.book_one_usd_buys',
 'raw_model_response.book_raw_model_response',
 'raw_model_response.name_cost',
 'raw_model_response.name_one_usd_buys',
 'raw_model_response.name_raw_model_response',
 'raw_model_response.year_cost',
 'raw_model_response.year_one_usd_buys',
 'raw_model_response.year_raw_model_response',
 'scenario.character']

We can inspect them:

[12]:
results.select("character", "name", "year", "book").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ scenario                          answer                            answer  answer                           ┃
┃ .character                        .name                             .year   .book                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ A pirate who speaks in 'arrs'     How about "Techbeard the Cyber    1717    Arrr and Algorithms: A Pirate's  │
│ and 'mateys' but has an           Corsair"? This name combines the          Guide to the Digital Seas        │
│ encyclopedic knowledge of modern  classic pirate element with a                                              │
│ technology.                       nod to their expertise in modern                                           │
│                                   technology.                                                                │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A Shakespearean actor who         A fitting name for such a         1600    The Bard's Cadence: Echoes of    │
│ answers every question in iambic  character could be "Percival              the Stage                        │
│ pentameter.                       Versewright." This name suggests                                           │
│                                   a classical, theatrical flair                                              │
│                                   while also hinting at his unique                                           │
│                                   talent for speaking in iambic                                              │
│                                   pentameter.                                                                │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A medieval knight who gives       Sir Draconis Counsel.             1099    Slaying Life's Dragons: A        │
│ advice as if every problem were                                             Knight's Guide to Conquering     │
│ a dragon to be slain.                                                       Challenges                       │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A sassy grandmother who gives     A great name for this character   1929    Straight Talk & Sass: Grandma's  │
│ blunt, no-nonsense advice with a  could be "Mabel Sharp." The name          Guide to Life                    │
│ touch of sarcasm.                 "Mabel" has a classic,                                                     │
│                                   grandmotherly feel, while                                                  │
│                                   "Sharp" hints at her wit and                                               │
│                                   straightforward nature.                                                    │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A surfer dude who relates every   A good name for this character    1966    Riding the Wave: Life Lessons    │
│ topic to the ocean or surfing.    could be "Rip Tide Ryder." This           from the Ocean                   │
│                                   name captures the essence of                                               │
│                                   surfing with "Rip Tide," a                                                 │
│                                   common ocean term, and "Ryder,"                                            │
│                                   which evokes the adventurous                                               │
│                                   spirit of a surfer.                                                        │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A conspiracy theorist who         How about the name "Cipher        1969    Threads of Truth: Unraveling the │
│ connects every question to their  Sage"? This name suggests a               Hidden Web                       │
│ wild theories.                    sense of mystery and knowledge,                                            │
│                                   while also hinting at the                                                  │
│                                   character's tendency to                                                    │
│                                   interpret everything through the                                           │
│                                   lens of their intricate                                                    │
│                                   theories.                                                                  │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A fashionista who answers         How about "Chic Clarissa"? It     1920    Chic Queries: The Stylish Guide  │
│ questions with a focus on style   captures her fashionable flair            to Life's Burning Questions      │
│ and trendiness.                   and expertise in style and                                                 │
│                                   trends.                                                                    │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A robot who is overly             A good name for this character    2023    Heartwired: A Robot's Journey to │
│ enthusiastic about human          could be "EmotiBot." This name            Feel                             │
│ emotions and tries too hard to    highlights the character's                                                 │
│ fit in.                           fascination with emotions while                                            │
│                                   playfully suggesting its robotic                                           │
│                                   nature. Alternatively, you could                                           │
│                                   consider names like "Eagertron"                                            │
│                                   or "Feelix," which also capture                                            │
│                                   the essence of its enthusiastic                                            │
│                                   attempts to understand and fit                                             │
│                                   in with humans.                                                            │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A toddler who is overly curious   A good name for this character    1969    Title: "Why, Why, Why:           │
│ and asks more questions than      could be "Quincy Quest." The              Adventures in Curiosity"         │
│ they answer.                      name "Quincy" has a playful and                                            │
│                                   inquisitive sound, while "Quest"                                           │
│                                   highlights their adventurous and                                           │
│                                   curious nature.                                                            │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A fitness guru who turns every    How about "Flex McMetaphor"?      1980    Flex Your Mind: Turning Life's   │
│ answer into a workout metaphor.   It's catchy and highlights both           Challenges into Strength         │
│                                   the character's fitness focus             Training                         │
│                                   and their unique way of                                                    │
│                                   communicating through workout                                              │
│                                   metaphors.                                                                 │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A foodie who relates every        A great name for this character   1765    Life's Recipe: Stirring Up       │
│ question to cooking and food      could be "Gourmet Gabe" or                Answers One Dish at a Time       │
│ experiences.                      "Culinary Clara." These names                                              │
│                                   capture their passion for food                                             │
│                                   and their tendency to connect                                              │
│                                   everything back to culinary                                                │
│                                   experiences.                                                               │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A detective from a noir film who  A fitting name for a detective    1947    The Shadows Whisper Truths       │
│ answers in a gritty, mysterious   in a noir film who exudes grit                                             │
│ manner.                           and mystery might be "Jack                                                 │
│                                   Shadows." This name evokes a                                               │
│                                   sense of enigma and toughness,                                             │
│                                   perfectly aligning with the                                                │
│                                   classic noir aesthetic.                                                    │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A hippie from the 60s who gives   A good name for your character    1969    Groovy Vibes: A Journey to Peace │
│ peace and love-centric advice.    could be "Harmony Moonbeam."              and Love                         │
│                                   This name captures the essence                                             │
│                                   of the 60s hippie culture and                                              │
│                                   reflects the character's focus                                             │
│                                   on peace and love.                                                         │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A gamer who references video      How about "Pixel Paladin"? This   1980    Level Up: Unlocking Life's       │
│ games and uses gamer lingo.       name captures the essence of a            Achievements with Gamer Wisdom   │
│                                   gamer who is deeply immersed in                                            │
│                                   video games and uses gaming                                                │
│                                   terminology, while also                                                    │
│                                   suggesting a sense of adventure                                            │
│                                   and skill.                                                                 │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A superhero who answers           A good name for this character    1938    Answer Avenger: Rescuing the     │
│ questions as if they are saving   could be "The Responder." This            World One Question at a Time     │
│ the day.                          name captures both their                                                   │
│                                   superhero essence and their                                                │
│                                   mission to save the day by                                                 │
│                                   answering questions.                                                       │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A poet who responds in rhyming    A fitting name for a poet who     1609    The Couplets of My Soul: A       │
│ couplets.                         speaks in rhyming couplets could          Rhyming Odyssey                  │
│                                   be "Rhymeheart Bard." This name                                            │
│                                   captures their poetic nature and                                           │
│                                   their talent for crafting                                                  │
│                                   harmonious verses.                                                         │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A comedian who tries to turn      A good name for this comedic      1920    Punchlines & Page-Turners:       │
│ every answer into a joke or       character could be "Punchline             Laughing Through Life One Joke   │
│ punchline.                        Pete" or "Jester Jess." These             at a Time                        │
│                                   names capture the essence of                                               │
│                                   someone who consistently turns                                             │
│                                   conversations into humorous                                                │
│                                   exchanges.                                                                 │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A DJ who relates everything to    A good name for a DJ character    1973    Rhythm & Resonance: Life's       │
│ music and beats.                  who relates everything to music           Playlist                         │
│                                   and beats could be "Rhythm                                                 │
│                                   Riff." This name captures the                                              │
│                                   essence of their musical                                                   │
│                                   perspective and their knack for                                            │
│                                   connecting everything back to                                              │
│                                   rhythm and sound.                                                          │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A film critic who answers         A fitting name for this           1941    Scenes from the Silver Screen:   │
│ questions as if they are          character could be "Cinephile             Life Reviewed with Cinematic     │
│ reviewing a movie.                Critique." This name captures             Flair                            │
│                                   their passion for film and their                                           │
│                                   unique approach to answering                                               │
│                                   questions as if they are                                                   │
│                                   reviewing a movie.                                                         │
├──────────────────────────────────┼──────────────────────────────────┼────────┼──────────────────────────────────┤
│ A scientist who gives overly      A fitting name for your           1905    Decoding the Universe: A Deep    │
│ detailed, scientific              character could be "Dr. Lexicon           Dive into the Intricacies of     │
│ explanations with lots of         J. Theorist." This name suggests          Science                          │
│ jargon.                           a deep familiarity with language                                           │
│                                   and theory, aligning with the                                              │
│                                   character's tendency to provide                                            │
│                                   detailed, jargon-heavy                                                     │
│                                   explanations.                                                              │
└──────────────────────────────────┴──────────────────────────────────┴────────┴──────────────────────────────────┘

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

Here we post the scenarios, survey and results from above, and this notebook:

[13]:
from edsl import Notebook
[14]:
n = Notebook(path = "batching_results.ipynb")
[15]:
n.push(description = "Example code for batching scenarios and combining results", visibility = "public")
[15]:
{'description': 'Example code for batching scenarios and combining results',
 'object_type': 'notebook',
 'url': 'https://chick.expectedparrot.com/content/e3de929a-cdf5-40b9-aedf-64022cc45fb5',
 'uuid': 'e3de929a-cdf5-40b9-aedf-64022cc45fb5',
 'version': '0.1.33',
 'visibility': 'public'}

To update an object at the Coop:

[16]:
n = Notebook(path = "batching_results.ipynb")
[17]:
n.patch(uuid = "e3de929a-cdf5-40b9-aedf-64022cc45fb5", value = n)
[17]:
{'status': 'success'}