Creating question variants
EDSL comes with a variety of features for efficiently generating different versions of questions in surveys. This notebook demonstrates methods for doing this with Scenario objects.
What is a Scenario?
A Scenario is a dictionary of one or more key/value pairs representing data or content to be added to questions; a ScenarioList is a list of Scenario objects. Scenario keys are used as question parameters that get replaced with the values when the scenarios are added to the questions, allowing you to create variants of questions efficiently. For example:
from edsl import QuestionFreeText, ScenarioList, Scenario
q = QuestionFreeText(
question_name = "favorite",
question_text = "What is your favorite {{ scenario.thing }}?"
)
s = ScenarioList(
Scenario({"thing": t}) for t in ["flower", "pizza topping", "chatbot"]
)
Using scenarios
Scenarios can be added to questions when constructing a survey or when running it. Functionally, the same question context is delivered to agents and models whether they are added during or after survey construction. The difference is how the information is arranged in the results that are generated by the models.
Methods
Adding scenarios at survey construction: loop
Each question type (QuestionMultipleChoice, QuestionFreeText, etc.) has a loop() method that generates a copy of the question for each scenario in a ScenarioList that is passed to it, returning a list of the questions that are generated. The loop() method is used when survey questions are being constructed. The typical workflow is:
Construct a (single)
Questionwith one or more parametersConstruct a
ScenarioListCall the
loop()method on the question and pass it the scenario listPass the list of the questions to a
Survey
From the example above this looks like:
from edsl import Survey
questions = q.loop(s)
survey = Survey(questions)
results = survey.run()
When the survey is run, the results that are generated will include columns for each question and answer; there are no scenario columns in the results (unless scenarios are also added when the survey is run).
Running a survey with scenarios: by
Scenarios can also be passed to a question or survey at the time that it is run. This is done by calling the by() method on a survey, passing it the scenarios, and then calling the run() method. The typical workflow is:
Construct a question (or survey of multiple questions) with one or more parameters
Construct scenarios
Call the
by()method on the question or survey and pass it the scenariosCall the
run()method to administer the question or survey
From the example above this looks like:
results = q.by(s).run()
(If any agents or models have been created and specified, they would also be added in separated by() calls. See details on designing agents and selecting language models.)
Example: Looping a question with scenarios
The loop() method is called on a Question object, and takes a ScenarioList of values to be inserted in copies of the question. We can optionally use the scenario key in the question name as well (so long as it is Pythonic); otherwise, unique identifiers are added to the original question name.
We start by constructing a question that takes a parameter:
[1]:
from edsl import QuestionFreeText
q = QuestionFreeText(
question_name = "features",
question_text = "What are the features of this sailboat model: {{ scenario.sailboat_model }}"
)
Next we create a scenario list to pass to the loop() method. EDSL comes with many methods for generating scenarios from different data sources, such as PDFs, CSVs, docs, tables, images, etc. For example, we can use the from_list() method to construct a scenario list from a list. Learn about other methods for generating scenarios.
[2]:
from edsl import ScenarioList
s = ScenarioList.from_list("sailboat_model", ['Laser', 'Sunfish', 'Optimist', 'Finn'])
s
[2]:
ScenarioList scenarios: 4; keys: ['sailboat_model'];
| sailboat_model | |
|---|---|
| 0 | Laser |
| 1 | Sunfish |
| 2 | Optimist |
| 3 | Finn |
Next we call the loop() method with the scenario list to create a list of the copies of the question, and verify that formatted questions have been generated:
[3]:
questions = q.loop(s)
questions
[3]:
[Question('free_text', question_name = """features_0""", question_text = """What are the features of this sailboat model: Laser"""),
Question('free_text', question_name = """features_1""", question_text = """What are the features of this sailboat model: Sunfish"""),
Question('free_text', question_name = """features_2""", question_text = """What are the features of this sailboat model: Optimist"""),
Question('free_text', question_name = """features_3""", question_text = """What are the features of this sailboat model: Finn""")]
We can pass the questions to a Survey and then run it:
[4]:
from edsl import Survey
survey = Survey(questions)
results = survey.run()
▼
Job Status (2025-03-06 04:56:06)
| Job UUID | 70b71552-dcfe-4163-92c4-1c8b5361b51a |
| Progress Bar URL | https://www.expectedparrot.com/home/remote-job-progress/70b71552-dcfe-4163-92c4-1c8b5361b51a |
| Exceptions Report URL | None |
| Results UUID | cb212ab1-be63-421f-9404-41fdcd7eea87 |
| Results URL | https://www.expectedparrot.com/content/cb212ab1-be63-421f-9404-41fdcd7eea87 |
✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/cb212ab1-be63-421f-9404-41fdcd7eea87
We can check the columns of dataset of Results that have been generated, and see that there are sets of columns for each question identifiable by question name (but no scenario columns):
[5]:
results.columns
[5]:
| 0 | |
|---|---|
| 0 | agent.agent_index |
| 1 | agent.agent_instruction |
| 2 | agent.agent_name |
| 3 | answer.features_0 |
| 4 | answer.features_1 |
| 5 | answer.features_2 |
| 6 | answer.features_3 |
| 7 | cache_keys.features_0_cache_key |
| 8 | cache_keys.features_1_cache_key |
| 9 | cache_keys.features_2_cache_key |
| 10 | cache_keys.features_3_cache_key |
| 11 | cache_used.features_0_cache_used |
| 12 | cache_used.features_1_cache_used |
| 13 | cache_used.features_2_cache_used |
| 14 | cache_used.features_3_cache_used |
| 15 | comment.features_0_comment |
| 16 | comment.features_1_comment |
| 17 | comment.features_2_comment |
| 18 | comment.features_3_comment |
| 19 | generated_tokens.features_0_generated_tokens |
| 20 | generated_tokens.features_1_generated_tokens |
| 21 | generated_tokens.features_2_generated_tokens |
| 22 | generated_tokens.features_3_generated_tokens |
| 23 | iteration.iteration |
| 24 | model.frequency_penalty |
| 25 | model.inference_service |
| 26 | model.logprobs |
| 27 | model.max_tokens |
| 28 | model.model |
| 29 | model.model_index |
| 30 | model.presence_penalty |
| 31 | model.temperature |
| 32 | model.top_logprobs |
| 33 | model.top_p |
| 34 | prompt.features_0_system_prompt |
| 35 | prompt.features_0_user_prompt |
| 36 | prompt.features_1_system_prompt |
| 37 | prompt.features_1_user_prompt |
| 38 | prompt.features_2_system_prompt |
| 39 | prompt.features_2_user_prompt |
| 40 | prompt.features_3_system_prompt |
| 41 | prompt.features_3_user_prompt |
| 42 | question_options.features_0_question_options |
| 43 | question_options.features_1_question_options |
| 44 | question_options.features_2_question_options |
| 45 | question_options.features_3_question_options |
| 46 | question_text.features_0_question_text |
| 47 | question_text.features_1_question_text |
| 48 | question_text.features_2_question_text |
| 49 | question_text.features_3_question_text |
| 50 | question_type.features_0_question_type |
| 51 | question_type.features_1_question_type |
| 52 | question_type.features_2_question_type |
| 53 | question_type.features_3_question_type |
| 54 | raw_model_response.features_0_cost |
| 55 | raw_model_response.features_0_one_usd_buys |
| 56 | raw_model_response.features_0_raw_model_response |
| 57 | raw_model_response.features_1_cost |
| 58 | raw_model_response.features_1_one_usd_buys |
| 59 | raw_model_response.features_1_raw_model_response |
| 60 | raw_model_response.features_2_cost |
| 61 | raw_model_response.features_2_one_usd_buys |
| 62 | raw_model_response.features_2_raw_model_response |
| 63 | raw_model_response.features_3_cost |
| 64 | raw_model_response.features_3_one_usd_buys |
| 65 | raw_model_response.features_3_raw_model_response |
| 66 | scenario.scenario_index |
We can access built-in methods for analyzing results, e.g., printing a table:
[6]:
results.select("answer.*")
[6]:
| answer.features_3 | answer.features_1 | answer.features_2 | answer.features_0 | |
|---|---|---|---|---|
| 0 | The Finn is a well-known single-handed sailing dinghy that has been used in competitive sailing, including the Olympics. Here are some of its features: 1. **Design**: The Finn is a single-handed dinghy with a classic design that has been refined over the years for optimal performance. It is known for its sleek hull and efficient sail plan. 2. **Hull**: Typically made from fiberglass or carbon fiber, the hull is designed for speed and stability. It is approximately 4.5 meters (14.8 feet) in length. 3. **Rigging**: The Finn features a single mast with a fully battened mainsail. The rigging is designed to be adjustable, allowing sailors to tweak the sail shape for different wind conditions. 4. **Sail**: The sail area is around 10 square meters (108 square feet). The sail is usually made from durable materials like Dacron or Mylar. 5. **Weight**: The boat is relatively lightweight, with a minimum weight of around 107 kilograms (236 pounds), which makes it responsive and agile on the water. 6. **Centerboard**: The Finn is equipped with a retractable centerboard, which helps with stability and maneuverability. 7. **Control Systems**: The boat has sophisticated control systems, including adjustable vang, outhaul, and Cunningham, which allow fine-tuning of the sail shape. 8. **Performance**: Known for its excellent upwind performance and ability to handle a wide range of wind conditions, the Finn is a favorite among competitive sailors. 9. **Olympic History**: The Finn class has been part of the Olympic sailing program for many years, making it a prestigious and highly competitive class. 10. **Community**: There is a strong international community of Finn sailors, with numerous regattas and championships held worldwide. These features make the Finn a challenging and rewarding boat to sail, especially for those interested in racing and honing their sailing skills. | The Sunfish sailboat is a popular and classic small sailing dinghy known for its simplicity, ease of use, and widespread popularity. Here are some of its key features: 1. **Design**: The Sunfish has a distinctive and simple design with a flat, lightweight hull. It is a lateen-rigged sailboat, which means it uses a triangular sail with a spar along the top edge. 2. **Dimensions**: The Sunfish is approximately 13 feet 9 inches (4.2 meters) in length and has a beam (width) of about 4 feet 1 inch (1.25 meters). 3. **Weight**: The hull typically weighs around 120 pounds (54 kilograms), making it relatively easy to transport and launch. 4. **Sail Area**: The sail area is about 75 square feet (7 square meters), providing enough power for spirited sailing while still being manageable for beginners. 5. **Capacity**: It is generally designed for one or two people, making it ideal for solo sailing or casual sailing with a friend. 6. **Construction**: Traditionally, Sunfish sailboats have been made from fiberglass, which offers durability and low maintenance. 7. **Ease of Use**: The Sunfish is known for being easy to rig and sail, making it a popular choice for beginners and sailing schools. Its simplicity also makes it a favorite for experienced sailors looking for a straightforward sailing experience. 8. **Portability**: Due to its lightweight design, the Sunfish can be easily transported on a trailer, car top, or even by hand with a dolly. 9. **Versatility**: The Sunfish can be sailed in a variety of conditions, from calm lakes to coastal waters, and is suitable for both leisurely sailing and competitive racing. 10. **Community and Racing**: There is a strong community of Sunfish sailors, and the boat is used in many competitive racing events around the world, with a well-established class association supporting these activities. The Sunfish's enduring popularity is a testament to its practical design and the enjoyment it offers to sailors of all skill levels. | The Optimist is a small, single-handed sailing dinghy intended for use by children up to the age of 15. It is one of the most popular sailing dinghies in the world for young sailors. Here are some of its key features: 1. **Size and Dimensions**: - Length: Approximately 2.3 meters (7 feet 6 inches). - Beam (width): About 1.13 meters (3 feet 8 inches). - Sail area: Around 3.5 square meters (37.5 square feet). 2. **Design**: - The Optimist has a pram hull, which means it has a flat bow. - The boat is designed to be stable and forgiving, making it ideal for beginners. - It has a single sail, which is rigged on a free-standing mast. 3. **Construction**: - Typically made of fiberglass, though wooden versions exist. - Lightweight, allowing for easy handling by children. 4. **Rigging**: - The rig is simple, consisting of a mast, boom, and sail. - The sail is often made of durable materials like Dacron. - The mast is unstayed, meaning it doesn't have supporting wires (stays). 5. **Steering and Control**: - Controlled using a tiller and rudder system. - The sail is controlled with a mainsheet. 6. **Safety and Accessibility**: - The Optimist is designed to be unsinkable, thanks to built-in buoyancy. - It is easy to right if capsized, which is an important feature for young sailors learning to handle a boat. 7. **Competitive Sailing**: - The Optimist is used in many youth sailing programs and competitions worldwide. - It is recognized by the International Sailing Federation (World Sailing) as an international class. 8. **Popularity**: - The class has a strong global presence, with many clubs and organizations dedicated to Optimist sailing. - It is often the first step for young sailors before moving on to larger and more complex boats. These features make the Optimist an excellent choice for introducing children to the sport of sailing, providing both a platform for learning and a pathway to competitive sailing. | The Laser sailboat, now officially known as the ILCA Dinghy, is a popular single-handed racing dinghy. Here are some of its key features: 1. **Design and Hull**: - The Laser has a simple, yet efficient design with a sleek fiberglass hull. - The hull length is approximately 4.2 meters (about 13.8 feet) with a beam of 1.39 meters (4.56 feet). - It is lightweight, typically around 59 kg (130 lbs), making it easy to handle both on and off the water. 2. **Rigging and Sails**: - The boat features a cat-rigged sail plan with a single sail. - It has a choice of three different sail sizes to accommodate various sailor weights and skill levels: the Standard (full rig), Radial, and 4.7 rigs. - The Standard rig has a sail area of 7.06 square meters (76 sq ft), the Radial has 5.76 square meters (62 sq ft), and the 4.7 has 4.7 square meters (50.6 sq ft). 3. **Mast and Boom**: - The mast is a two-piece design, allowing for easy transportation and storage. - The rigging is straightforward, with minimal adjustments needed, making it ideal for both beginners and experienced sailors. 4. **Performance**: - The Laser is known for its excellent performance in a wide range of wind conditions. - It is highly responsive and capable of planing, which makes it exciting and challenging to sail. - The boat is designed to be physically demanding, rewarding skill and fitness. 5. **Class and Competition**: - The Laser is one of the most popular one-design racing classes in the world, with strict class rules to ensure fairness in competition. - It is an Olympic class boat, used in both men's and women's single-handed events. 6. **Accessibility and Popularity**: - Due to its simplicity, affordability, and widespread availability, the Laser is accessible to sailors of all ages and skill levels. - It has a large international community and numerous local, national, and international racing events. The Laser's design emphasizes simplicity and performance, making it a timeless choice for sailors seeking both recreational and competitive experiences. |
Running a question with scenarios
If we instead want to add the scenarios to the question when it is run, we simply add them with the by() method. This will re-administer a question for each scenario:
[7]:
results = q.by(s).run()
▼
Job Status (2025-03-06 04:56:15)
| Job UUID | cea5558b-8c80-4ccd-9115-337e4706a2ba |
| Progress Bar URL | https://www.expectedparrot.com/home/remote-job-progress/cea5558b-8c80-4ccd-9115-337e4706a2ba |
| Exceptions Report URL | None |
| Results UUID | 6faf9cac-1bb2-481c-a9bd-142a258768f8 |
| Results URL | https://www.expectedparrot.com/content/6faf9cac-1bb2-481c-a9bd-142a258768f8 |
✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/6faf9cac-1bb2-481c-a9bd-142a258768f8
The results now include columns for the single question but with a separate row for each scenario:
[8]:
results.columns
[8]:
| 0 | |
|---|---|
| 0 | agent.agent_index |
| 1 | agent.agent_instruction |
| 2 | agent.agent_name |
| 3 | answer.features |
| 4 | cache_keys.features_cache_key |
| 5 | cache_used.features_cache_used |
| 6 | comment.features_comment |
| 7 | generated_tokens.features_generated_tokens |
| 8 | iteration.iteration |
| 9 | model.frequency_penalty |
| 10 | model.inference_service |
| 11 | model.logprobs |
| 12 | model.max_tokens |
| 13 | model.model |
| 14 | model.model_index |
| 15 | model.presence_penalty |
| 16 | model.temperature |
| 17 | model.top_logprobs |
| 18 | model.top_p |
| 19 | prompt.features_system_prompt |
| 20 | prompt.features_user_prompt |
| 21 | question_options.features_question_options |
| 22 | question_text.features_question_text |
| 23 | question_type.features_question_type |
| 24 | raw_model_response.features_cost |
| 25 | raw_model_response.features_one_usd_buys |
| 26 | raw_model_response.features_raw_model_response |
| 27 | scenario.sailboat_model |
| 28 | scenario.scenario_index |
[9]:
results.select("sailboat_model", "features") # results.select("scenario.*", "answer.*") is equivalent here
[9]:
| scenario.sailboat_model | answer.features | |
|---|---|---|
| 0 | Laser | The Laser sailboat, now officially known as the ILCA Dinghy, is a popular single-handed racing dinghy. Here are some of its key features: 1. **Design and Hull**: - The Laser has a simple, yet efficient design with a sleek fiberglass hull. - The hull length is approximately 4.2 meters (about 13.8 feet) with a beam of 1.39 meters (4.56 feet). - It is lightweight, typically around 59 kg (130 lbs), making it easy to handle both on and off the water. 2. **Rigging and Sails**: - The boat features a cat-rigged sail plan with a single sail. - It has a choice of three different sail sizes to accommodate various sailor weights and skill levels: the Standard (full rig), Radial, and 4.7 rigs. - The Standard rig has a sail area of 7.06 square meters (76 sq ft), the Radial has 5.76 square meters (62 sq ft), and the 4.7 has 4.7 square meters (50.6 sq ft). 3. **Mast and Boom**: - The mast is a two-piece design, allowing for easy transportation and storage. - The rigging is straightforward, with minimal adjustments needed, making it ideal for both beginners and experienced sailors. 4. **Performance**: - The Laser is known for its excellent performance in a wide range of wind conditions. - It is highly responsive and capable of planing, which makes it exciting and challenging to sail. - The boat is designed to be physically demanding, rewarding skill and fitness. 5. **Class and Competition**: - The Laser is one of the most popular one-design racing classes in the world, with strict class rules to ensure fairness in competition. - It is an Olympic class boat, used in both men's and women's single-handed events. 6. **Accessibility and Popularity**: - Due to its simplicity, affordability, and widespread availability, the Laser is accessible to sailors of all ages and skill levels. - It has a large international community and numerous local, national, and international racing events. The Laser's design emphasizes simplicity and performance, making it a timeless choice for sailors seeking both recreational and competitive experiences. |
| 1 | Sunfish | The Sunfish sailboat is a popular and classic small sailing dinghy known for its simplicity, ease of use, and widespread popularity. Here are some of its key features: 1. **Design**: The Sunfish has a distinctive and simple design with a flat, lightweight hull. It is a lateen-rigged sailboat, which means it uses a triangular sail with a spar along the top edge. 2. **Dimensions**: The Sunfish is approximately 13 feet 9 inches (4.2 meters) in length and has a beam (width) of about 4 feet 1 inch (1.25 meters). 3. **Weight**: The hull typically weighs around 120 pounds (54 kilograms), making it relatively easy to transport and launch. 4. **Sail Area**: The sail area is about 75 square feet (7 square meters), providing enough power for spirited sailing while still being manageable for beginners. 5. **Capacity**: It is generally designed for one or two people, making it ideal for solo sailing or casual sailing with a friend. 6. **Construction**: Traditionally, Sunfish sailboats have been made from fiberglass, which offers durability and low maintenance. 7. **Ease of Use**: The Sunfish is known for being easy to rig and sail, making it a popular choice for beginners and sailing schools. Its simplicity also makes it a favorite for experienced sailors looking for a straightforward sailing experience. 8. **Portability**: Due to its lightweight design, the Sunfish can be easily transported on a trailer, car top, or even by hand with a dolly. 9. **Versatility**: The Sunfish can be sailed in a variety of conditions, from calm lakes to coastal waters, and is suitable for both leisurely sailing and competitive racing. 10. **Community and Racing**: There is a strong community of Sunfish sailors, and the boat is used in many competitive racing events around the world, with a well-established class association supporting these activities. The Sunfish's enduring popularity is a testament to its practical design and the enjoyment it offers to sailors of all skill levels. |
| 2 | Optimist | The Optimist is a small, single-handed sailing dinghy intended for use by children up to the age of 15. It is one of the most popular sailing dinghies in the world for young sailors. Here are some of its key features: 1. **Size and Dimensions**: - Length: Approximately 2.3 meters (7 feet 6 inches). - Beam (width): About 1.13 meters (3 feet 8 inches). - Sail area: Around 3.5 square meters (37.5 square feet). 2. **Design**: - The Optimist has a pram hull, which means it has a flat bow. - The boat is designed to be stable and forgiving, making it ideal for beginners. - It has a single sail, which is rigged on a free-standing mast. 3. **Construction**: - Typically made of fiberglass, though wooden versions exist. - Lightweight, allowing for easy handling by children. 4. **Rigging**: - The rig is simple, consisting of a mast, boom, and sail. - The sail is often made of durable materials like Dacron. - The mast is unstayed, meaning it doesn't have supporting wires (stays). 5. **Steering and Control**: - Controlled using a tiller and rudder system. - The sail is controlled with a mainsheet. 6. **Safety and Accessibility**: - The Optimist is designed to be unsinkable, thanks to built-in buoyancy. - It is easy to right if capsized, which is an important feature for young sailors learning to handle a boat. 7. **Competitive Sailing**: - The Optimist is used in many youth sailing programs and competitions worldwide. - It is recognized by the International Sailing Federation (World Sailing) as an international class. 8. **Popularity**: - The class has a strong global presence, with many clubs and organizations dedicated to Optimist sailing. - It is often the first step for young sailors before moving on to larger and more complex boats. These features make the Optimist an excellent choice for introducing children to the sport of sailing, providing both a platform for learning and a pathway to competitive sailing. |
| 3 | Finn | The Finn is a well-known single-handed sailing dinghy that has been used in competitive sailing, including the Olympics. Here are some of its features: 1. **Design**: The Finn is a single-handed dinghy with a classic design that has been refined over the years for optimal performance. It is known for its sleek hull and efficient sail plan. 2. **Hull**: Typically made from fiberglass or carbon fiber, the hull is designed for speed and stability. It is approximately 4.5 meters (14.8 feet) in length. 3. **Rigging**: The Finn features a single mast with a fully battened mainsail. The rigging is designed to be adjustable, allowing sailors to tweak the sail shape for different wind conditions. 4. **Sail**: The sail area is around 10 square meters (108 square feet). The sail is usually made from durable materials like Dacron or Mylar. 5. **Weight**: The boat is relatively lightweight, with a minimum weight of around 107 kilograms (236 pounds), which makes it responsive and agile on the water. 6. **Centerboard**: The Finn is equipped with a retractable centerboard, which helps with stability and maneuverability. 7. **Control Systems**: The boat has sophisticated control systems, including adjustable vang, outhaul, and Cunningham, which allow fine-tuning of the sail shape. 8. **Performance**: Known for its excellent upwind performance and ability to handle a wide range of wind conditions, the Finn is a favorite among competitive sailors. 9. **Olympic History**: The Finn class has been part of the Olympic sailing program for many years, making it a prestigious and highly competitive class. 10. **Community**: There is a strong international community of Finn sailors, with numerous regattas and championships held worldwide. These features make the Finn a challenging and rewarding boat to sail, especially for those interested in racing and honing their sailing skills. |
Posting to the Coop
The Coop is a new platform for creating, storing and sharing LLM-based research. We can post surveys, agents, results and notebooks, such as this one. Learn more about using the Coop.
[ ]:
from edsl import Notebook
nb = Notebook(path = "question_loop_scenarios.ipynb")
if refresh := False:
nb.push(
description = "New question method `loop` for creating questions with scenarios",
alias = "question-loop-scenarios",
visibility = "public"
)
else:
nb.patch("https://www.expectedparrot.com/content/RobinHorton/question-loop-scenarios", value = nb)