Creating question variants

EDSL comes with a variety of features for efficiently generating different versions of questions in surveys. This notebook demonstrates methods for doing this with Scenario objects.

What is a Scenario?

A Scenario is a dictionary of one or more key/value pairs representing data or content to be added to questions; a ScenarioList is a list of Scenario objects. Scenario keys are used as question parameters that get replaced with the values when the scenarios are added to the questions, allowing you to create variants of questions efficiently. For example:

from edsl import QuestionFreeText, ScenarioList, Scenario

q = QuestionFreeText(
    question_name = "favorite",
    question_text = "What is your favorite {{ thing }}?"
)

s = ScenarioList(
    Scenario({"thing": t}) for t in ["flower", "pizza topping", "chatbot"]
)

Using scenarios

Scenarios can be added to questions when constructing a survey or when running it. Functionally, the same question context is delivered to agents and models whether they are added during or after survey construction. The difference is how the information is arranged in the results that are generated by the models.

Methods

Adding scenarios at survey construction: loop

Each question type (QuestionMultipleChoice, QuestionFreeText, etc.) has a loop() method that generates a copy of the question for each scenario in a ScenarioList that is passed to it, returning a list of the questions that are generated. The loop() method is used when survey questions are being constructed. The typical workflow is:

  • Construct a (single) Question with one or more parameters

  • Construct a ScenarioList

  • Call the loop() method on the question and pass it the scenario list

  • Pass the list of the questions to a Survey

From the example above this looks like:

from edsl import Survey

questions = q.loop(s)
survey = Survey(questions)
results = survey.run()

When the survey is run, the results that are generated will include columns for each question and answer; there are no scenario columns in the results (unless scenarios are also added when the survey is run).

Running a survey with scenarios: by

Scenarios can also be passed to a question or survey at the time that it is run. This is done by calling the by() method on a survey, passing it the scenarios, and then calling the run() method. The typical workflow is:

  • Construct a question (or survey of multiple questions) with one or more parameters

  • Construct scenarios

  • Call the by() method on the question or survey and pass it the scenarios

  • Call the run() method to administer the question or survey

From the example above this looks like:

results = q.by(s).run()

(If any agents or models have been created and specified, they would also be added in separated by() calls. See details on designing agents and selecting language models.)

Example: Looping a question with scenarios

The loop() method is called on a Question object, and takes a ScenarioList of values to be inserted in copies of the question. We can optionally use the scenario key in the question name as well (so long as it is Pythonic); otherwise, unique identifiers are added to the original question name.

We start by constructing a question that takes a parameter:

[1]:

from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "features",
    question_text = "What are the features of this sailboat model: {{ sailboat_model }}"
)

Next we create a scenario list to pass to the loop() method. EDSL comes with many methods for generating scenarios from different data sources, such as PDFs, CSVs, docs, tables, images, etc. For example, we can use the from_list() method to construct a scenario list from a list. Learn about other methods for generating scenarios.

[2]:

from edsl import ScenarioList

s = ScenarioList.from_list("sailboat_model", ['Laser', 'Sunfish', 'Optimist', 'Finn'])
s

[2]:

ScenarioList scenarios: 4; keys: ['sailboat_model'];

sailboat_model
Laser
Sunfish
Optimist
Finn

Next we call the loop() method with the scenario list to create a list of the copies of the question, and verify that formatted questions have been generated:

[3]:

questions = q.loop(s)
questions

[3]:

[Question('free_text', question_name = """features_0""", question_text = """What are the features of this sailboat model: Laser"""),
 Question('free_text', question_name = """features_1""", question_text = """What are the features of this sailboat model: Sunfish"""),
 Question('free_text', question_name = """features_2""", question_text = """What are the features of this sailboat model: Optimist"""),
 Question('free_text', question_name = """features_3""", question_text = """What are the features of this sailboat model: Finn""")]

We can pass the questions to a Survey and then run it:

[4]:

from edsl import Survey

survey = Survey(questions)
results = survey.run()
▼ Remote Job Log (2024-12-14 10:47:58)
Remote inference activated. Sending job to server...
Your survey is running at the Expected Parrot server...
Job sent to server. (Job uuid=b7bff505-2acf-4051-a70c-3462c3b8b7c3).
Job status: queued - last update: 2024-12-14 10:47:38 AM
Job status: queued - last update: 2024-12-14 10:47:41 AM
Job status: running - last update: 2024-12-14 10:47:45 AM
Job status: running - last update: 2024-12-14 10:47:48 AM
Job status: running - last update: 2024-12-14 10:47:51 AM
Job status: running - last update: 2024-12-14 10:47:54 AM

We can check the columns of dataset of Results that have been generated, and see that there are sets of columns for each question identifiable by question name (but no scenario columns):

[5]:

results.columns

[5]:
0
agent.agent_instruction
agent.agent_name
answer.features_0
answer.features_1
answer.features_2
answer.features_3
comment.features_0_comment
comment.features_1_comment
comment.features_2_comment
comment.features_3_comment
generated_tokens.features_0_generated_tokens
generated_tokens.features_1_generated_tokens
generated_tokens.features_2_generated_tokens
generated_tokens.features_3_generated_tokens
iteration.iteration
model.frequency_penalty
model.logprobs
model.max_tokens
model.model
model.presence_penalty
model.temperature
model.top_logprobs
model.top_p
prompt.features_0_system_prompt
prompt.features_0_user_prompt
prompt.features_1_system_prompt
prompt.features_1_user_prompt
prompt.features_2_system_prompt
prompt.features_2_user_prompt
prompt.features_3_system_prompt
prompt.features_3_user_prompt
question_options.features_0_question_options
question_options.features_1_question_options
question_options.features_2_question_options
question_options.features_3_question_options
question_text.features_0_question_text
question_text.features_1_question_text
question_text.features_2_question_text
question_text.features_3_question_text
question_type.features_0_question_type
question_type.features_1_question_type
question_type.features_2_question_type
question_type.features_3_question_type
raw_model_response.features_0_cost
raw_model_response.features_0_one_usd_buys
raw_model_response.features_0_raw_model_response
raw_model_response.features_1_cost
raw_model_response.features_1_one_usd_buys
raw_model_response.features_1_raw_model_response
raw_model_response.features_2_cost
raw_model_response.features_2_one_usd_buys
raw_model_response.features_2_raw_model_response
raw_model_response.features_3_cost
raw_model_response.features_3_one_usd_buys
raw_model_response.features_3_raw_model_response

We can access built-in methods for analyzing results, e.g., printing a table:

[6]:

results.select("answer.*")

[6]:
answer.features_2 answer.features_3 answer.features_1 answer.features_0
The Optimist is a small, single-handed sailing dinghy intended for use by children up to the age of 15. It is one of the most popular sailing dinghies worldwide for youth sailing. Here are some of its key features: 1. **Design**: The Optimist has a pram bow and a flat, square stern. Its hull is relatively flat and wide, which provides stability, making it ideal for beginners. 2. **Size and Weight**: The boat is about 2.3 meters (7 feet 6 inches) long and 1.13 meters (3 feet 8 inches) wide. It typically weighs around 35 kg (77 lbs) without the rigging. 3. **Sail**: The Optimist has a single, triangular sail with a total sail area of approximately 3.5 square meters (37.5 square feet). The sail is usually made of Dacron, a durable polyester material. 4. **Rigging**: It features a simple rigging system with a sprit rig, which is easy to handle and adjust, making it suitable for young sailors. 5. **Construction**: Optimists are typically made from fiberglass, although some older models may be constructed from wood. 6. **Buoyancy**: The boat is equipped with buoyancy tanks to ensure it remains afloat even if it capsizes. 7. **Ease of Use**: The design is straightforward, with a tiller for steering and a centerboard for stability, making it easy for children to learn the basics of sailing. 8. **Competitive Sailing**: The Optimist is widely used in competitive youth sailing, with many clubs and organizations hosting races and regattas specifically for this class. 9. **Portability**: Its small size and light weight make it easy to transport and store, often carried on top of a car or in a small trailer. The Finn is a single-handed, cat-rigged sailboat designed for racing, particularly known for its use in Olympic competitions. Here are some features of the Finn sailboat: 1. **Design**: The Finn has a classic design with a single sail, making it a cat-rigged dinghy. It features a deep, heavy hull that provides stability and performance in various wind conditions. 2. **Length and Beam**: The Finn is approximately 4.5 meters (14.8 feet) long with a beam (width) of about 1.5 meters (4.9 feet). 3. **Sail Area**: It has a large sail area for its size, typically around 10 square meters (107 square feet), which allows it to be fast and responsive. 4. **Hull Material**: Modern Finns are often constructed using fiberglass or composite materials, providing a good balance of strength and weight. 5. **Mast and Rigging**: The Finn features a flexible mast that can be adjusted to optimize performance in different wind conditions. The rigging is designed to allow for precise sail control. 6. **Weight**: The hull weight is typically around 107 kg (236 lbs), which contributes to its stability and handling. 7. **Performance**: The Finn is known for its excellent upwind performance and ability to handle a wide range of wind conditions, making it a challenging and rewarding boat to sail. 8. **Olympic History**: The Finn class has a rich history in the Olympics, having been used in the Games from 1952 until 2020. It is known for its demanding physical requirements and tactical racing. The Sunfish is a popular and iconic small sailing dinghy known for its simplicity, ease of use, and wide appeal among both beginner and experienced sailors. Here are some of its key features: 1. **Design and Build**: - The Sunfish has a distinctive hull shape with a flat deck and a single, lateen-rigged sail. - It is typically constructed from fiberglass, making it lightweight and durable. 2. **Dimensions**: - Length: Approximately 13 feet 9 inches (4.2 meters). - Beam (width): About 4 feet 1 inch (1.2 meters). - Hull weight: Around 120 pounds (54 kg), which makes it easy to transport and launch. 3. **Sail**: - The Sunfish features a single sail with a lateen rig, which is triangular and supported by a yard and boom. - The sail area is about 75 square feet (7 square meters). 4. **Capacity**: - It is generally designed for one or two people, making it perfect for solo sailing or a small crew. 5. **Ease of Use**: - The Sunfish is known for its simplicity, with minimal rigging and controls, making it accessible for beginners. - Its design allows for easy righting in the event of a capsize. 6. **Performance**: - It is well-suited for a variety of wind conditions and is known for its agility and speed. - The Sunfish can be used for both recreational sailing and racing. 7. **Portability**: - Due to its lightweight hull, the Sunfish can be easily transported on a car roof rack or a small trailer. 8. **Maintenance**: - The fiberglass construction requires minimal maintenance, making it a low-maintenance option for sailors. The Laser sailboat, now officially known as the ILCA Dinghy, is a popular one-design racing sailboat recognized for its simplicity and performance. Here are some key features of the Laser: 1. **Design and Construction:** - **Hull:** The Laser features a lightweight, fiberglass hull that is both durable and easy to handle. The hull is typically 4.2 meters (13 feet 9 inches) long. - **Beam:** The beam (width) of the boat is about 1.39 meters (4 feet 6 inches). - **Weight:** The hull weight is approximately 59 kg (130 lbs), making it easy to transport and launch. 2. **Rigging:** - **Mast and Boom:** The Laser has a simple, two-piece aluminum mast and a boom that supports the sail. - **Sail:** It uses a single sail, which is a cat-rigged, fully battened, and typically made of Dacron. The sail area varies slightly depending on the rig version (Standard, Radial, or 4.7). 3. **Variants:** - **Laser Standard:** Features a larger sail area of about 7.06 square meters (76 square feet), suitable for heavier sailors. - **Laser Radial:** Has a smaller sail area of 5.76 square meters (62 square feet), designed for lighter sailors, including women and youth. - **Laser 4.7:** Features an even smaller sail area of 4.7 square meters (50 square feet), ideal for younger or lighter sailors. 4. **Performance:** - The Laser is known for its excellent performance in a wide range of wind conditions, making it a favorite for competitive racing. - It is highly responsive and can plane easily in stronger winds. 5. **One-Design Class:** - The Laser is a strict one-design class, meaning that all boats are built to the same specifications. This ensures that races are won based on skill rather than equipment advantages. 6. **Popularity:** - The Laser is one of the most popular single-handed dinghies in the world, with a large international racing community and numerous regattas, including being an Olympic class.

Running a question with scenarios

If we instead want to add the scenarios to the question when it is run, we simply add them with the by() method. This will re-administer a question for each scenario:

[7]:

results = q.by(s).run()
▼ Remote Job Log (2024-12-14 10:48:30)
Remote inference activated. Sending job to server...
Your survey is running at the Expected Parrot server...
Job sent to server. (Job uuid=4f4d94e9-162c-46f7-b56d-7483dc386df8).
Job status: queued - last update: 2024-12-14 10:48:13 AM
Job status: queued - last update: 2024-12-14 10:48:17 AM
Job status: running - last update: 2024-12-14 10:48:20 AM
Job status: running - last update: 2024-12-14 10:48:23 AM
Job status: running - last update: 2024-12-14 10:48:26 AM

The results now include columns for the single question but with a separate row for each scenario:

[8]:

results.columns

[8]:
0
agent.agent_instruction
agent.agent_name
answer.features
comment.features_comment
generated_tokens.features_generated_tokens
iteration.iteration
model.frequency_penalty
model.logprobs
model.max_tokens
model.model
model.presence_penalty
model.temperature
model.top_logprobs
model.top_p
prompt.features_system_prompt
prompt.features_user_prompt
question_options.features_question_options
question_text.features_question_text
question_type.features_question_type
raw_model_response.features_cost
raw_model_response.features_one_usd_buys
raw_model_response.features_raw_model_response
scenario.sailboat_model 
[9]:

results.select("sailboat_model", "features")  # results.select("scenario.*", "answer.*") is equivalent here

[9]:
scenario.sailboat_model answer.features
Laser The Laser sailboat, now officially known as the ILCA Dinghy, is a popular one-design racing sailboat recognized for its simplicity and performance. Here are some key features of the Laser: 1. **Design and Construction:** - **Hull:** The Laser features a lightweight, fiberglass hull that is both durable and easy to handle. The hull is typically 4.2 meters (13 feet 9 inches) long. - **Beam:** The beam (width) of the boat is about 1.39 meters (4 feet 6 inches). - **Weight:** The hull weight is approximately 59 kg (130 lbs), making it easy to transport and launch. 2. **Rigging:** - **Mast and Boom:** The Laser has a simple, two-piece aluminum mast and a boom that supports the sail. - **Sail:** It uses a single sail, which is a cat-rigged, fully battened, and typically made of Dacron. The sail area varies slightly depending on the rig version (Standard, Radial, or 4.7). 3. **Variants:** - **Laser Standard:** Features a larger sail area of about 7.06 square meters (76 square feet), suitable for heavier sailors. - **Laser Radial:** Has a smaller sail area of 5.76 square meters (62 square feet), designed for lighter sailors, including women and youth. - **Laser 4.7:** Features an even smaller sail area of 4.7 square meters (50 square feet), ideal for younger or lighter sailors. 4. **Performance:** - The Laser is known for its excellent performance in a wide range of wind conditions, making it a favorite for competitive racing. - It is highly responsive and can plane easily in stronger winds. 5. **One-Design Class:** - The Laser is a strict one-design class, meaning that all boats are built to the same specifications. This ensures that races are won based on skill rather than equipment advantages. 6. **Popularity:** - The Laser is one of the most popular single-handed dinghies in the world, with a large international racing community and numerous regattas, including being an Olympic class.
Sunfish The Sunfish is a popular and iconic small sailing dinghy known for its simplicity, ease of use, and wide appeal among both beginner and experienced sailors. Here are some of its key features: 1. **Design and Build**: - The Sunfish has a distinctive hull shape with a flat deck and a single, lateen-rigged sail. - It is typically constructed from fiberglass, making it lightweight and durable. 2. **Dimensions**: - Length: Approximately 13 feet 9 inches (4.2 meters). - Beam (width): About 4 feet 1 inch (1.2 meters). - Hull weight: Around 120 pounds (54 kg), which makes it easy to transport and launch. 3. **Sail**: - The Sunfish features a single sail with a lateen rig, which is triangular and supported by a yard and boom. - The sail area is about 75 square feet (7 square meters). 4. **Capacity**: - It is generally designed for one or two people, making it perfect for solo sailing or a small crew. 5. **Ease of Use**: - The Sunfish is known for its simplicity, with minimal rigging and controls, making it accessible for beginners. - Its design allows for easy righting in the event of a capsize. 6. **Performance**: - It is well-suited for a variety of wind conditions and is known for its agility and speed. - The Sunfish can be used for both recreational sailing and racing. 7. **Portability**: - Due to its lightweight hull, the Sunfish can be easily transported on a car roof rack or a small trailer. 8. **Maintenance**: - The fiberglass construction requires minimal maintenance, making it a low-maintenance option for sailors.
Optimist The Optimist is a small, single-handed sailing dinghy intended for use by children up to the age of 15. It is one of the most popular sailing dinghies worldwide for youth sailing. Here are some of its key features: 1. **Design**: The Optimist has a pram bow and a flat, square stern. Its hull is relatively flat and wide, which provides stability, making it ideal for beginners. 2. **Size and Weight**: The boat is about 2.3 meters (7 feet 6 inches) long and 1.13 meters (3 feet 8 inches) wide. It typically weighs around 35 kg (77 lbs) without the rigging. 3. **Sail**: The Optimist has a single, triangular sail with a total sail area of approximately 3.5 square meters (37.5 square feet). The sail is usually made of Dacron, a durable polyester material. 4. **Rigging**: It features a simple rigging system with a sprit rig, which is easy to handle and adjust, making it suitable for young sailors. 5. **Construction**: Optimists are typically made from fiberglass, although some older models may be constructed from wood. 6. **Buoyancy**: The boat is equipped with buoyancy tanks to ensure it remains afloat even if it capsizes. 7. **Ease of Use**: The design is straightforward, with a tiller for steering and a centerboard for stability, making it easy for children to learn the basics of sailing. 8. **Competitive Sailing**: The Optimist is widely used in competitive youth sailing, with many clubs and organizations hosting races and regattas specifically for this class. 9. **Portability**: Its small size and light weight make it easy to transport and store, often carried on top of a car or in a small trailer.
Finn The Finn is a single-handed, cat-rigged sailboat designed for racing, particularly known for its use in Olympic competitions. Here are some features of the Finn sailboat: 1. **Design**: The Finn has a classic design with a single sail, making it a cat-rigged dinghy. It features a deep, heavy hull that provides stability and performance in various wind conditions. 2. **Length and Beam**: The Finn is approximately 4.5 meters (14.8 feet) long with a beam (width) of about 1.5 meters (4.9 feet). 3. **Sail Area**: It has a large sail area for its size, typically around 10 square meters (107 square feet), which allows it to be fast and responsive. 4. **Hull Material**: Modern Finns are often constructed using fiberglass or composite materials, providing a good balance of strength and weight. 5. **Mast and Rigging**: The Finn features a flexible mast that can be adjusted to optimize performance in different wind conditions. The rigging is designed to allow for precise sail control. 6. **Weight**: The hull weight is typically around 107 kg (236 lbs), which contributes to its stability and handling. 7. **Performance**: The Finn is known for its excellent upwind performance and ability to handle a wide range of wind conditions, making it a challenging and rewarding boat to sail. 8. **Olympic History**: The Finn class has a rich history in the Olympics, having been used in the Games from 1952 until 2020. It is known for its demanding physical requirements and tactical racing.

Posting to the Coop

The Coop is a new platform for creating, storing and sharing LLM-based research. We can post surveys, agents, results and notebooks, such as this one. Learn more about using the Coop.

[10]:

from edsl import Notebook

[11]:

n = Notebook(path = "question_loop_scenarios.ipynb")

[12]:

info = n.push(description = "New question method `loop` for creating questions with scenarios", visibility = "public")
info

[12]:

{'description': 'New question method `loop` for creating questions with scenarios',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/6313d66a-0ba8-4c99-8447-d2e444972e86',
 'uuid': '6313d66a-0ba8-4c99-8447-d2e444972e86',
 'version': '0.1.39.dev1',
 'visibility': 'public'}

To update an object at the Coop:

[13]:

n = Notebook(path = "question_loop_scenarios.ipynb")

[14]:

n.patch(uuid = info["uuid"], value = n)

[14]:

{'status': 'success'}