Creating question variants

EDSL comes with a variety of features for efficiently generating different versions of questions in surveys. This notebook demonstrates methods for doing this with Scenario objects.

What is a `Scenario`?

A Scenario is a dictionary of one or more key/value pairs representing data or content to be added to questions; a ScenarioList is a list of Scenario objects. Scenario keys are used as question parameters that get replaced with the values when the scenarios are added to the questions, allowing you to create variants of questions efficiently. For example:

from edsl import QuestionFreeText, ScenarioList, Scenario

q = QuestionFreeText(
    question_name = "favorite",
    question_text = "What is your favorite {{ scenario.thing }}?"
)

s = ScenarioList(
    Scenario({"thing": t}) for t in ["flower", "pizza topping", "chatbot"]
)

Using scenarios

Scenarios can be added to questions when constructing a survey or when running it. Functionally, the same question context is delivered to agents and models whether they are added during or after survey construction. The difference is how the information is arranged in the results that are generated by the models.

Methods

Adding scenarios at survey construction: `loop`

Each question type (QuestionMultipleChoice, QuestionFreeText, etc.) has a loop() method that generates a copy of the question for each scenario in a ScenarioList that is passed to it, returning a list of the questions that are generated. The loop() method is used when survey questions are being constructed. The typical workflow is:

Construct a (single) Question with one or more parameters
Construct a ScenarioList
Call the loop() method on the question and pass it the scenario list
Pass the list of the questions to a Survey

From the example above this looks like:

from edsl import Survey

questions = q.loop(s)
survey = Survey(questions)
results = survey.run()

When the survey is run, the results that are generated will include columns for each question and answer; there are no scenario columns in the results (unless scenarios are also added when the survey is run).

Running a survey with scenarios: `by`

Scenarios can also be passed to a question or survey at the time that it is run. This is done by calling the by() method on a survey, passing it the scenarios, and then calling the run() method. The typical workflow is:

Construct a question (or survey of multiple questions) with one or more parameters
Construct scenarios
Call the by() method on the question or survey and pass it the scenarios
Call the run() method to administer the question or survey

From the example above this looks like:

results = q.by(s).run()

(If any agents or models have been created and specified, they would also be added in separated by() calls. See details on designing agents and selecting language models.)

Example: Looping a question with scenarios

The loop() method is called on a Question object, and takes a ScenarioList of values to be inserted in copies of the question. We can optionally use the scenario key in the question name as well (so long as it is Pythonic); otherwise, unique identifiers are added to the original question name.

We start by constructing a question that takes a parameter:

[1]:

from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "features",
    question_text = "Briefly describe the features of this sailboat model: {{ scenario.sailboat_model }}"
)

Next we create a scenario list to pass to the loop() method. EDSL comes with many methods for generating scenarios from different data sources, such as PDF, PNG, CSV, docs, tables, dicts, lists, etc. We can use the from_source() method to construct a scenario list and specify the data type (“csv”, “pdf”, “list”, etc.). Learn about other methods for generating scenarios.

[2]:

from edsl import ScenarioList

s = ScenarioList.from_source("list", "sailboat_model", ['Laser', 'Sunfish', 'Optimist', 'Finn'])
s

[2]:

ScenarioList scenarios: 4; keys: ['sailboat_model'];

	sailboat_model
0	Laser
1	Sunfish
2	Optimist
3	Finn

Next we call the loop() method with the scenario list to create a list of the copies of the question, and verify that formatted questions have been generated:

[3]:

questions = q.loop(s)
questions

[3]:

[Question('free_text', question_name = """features_0""", question_text = """Briefly describe the features of this sailboat model: Laser"""),
 Question('free_text', question_name = """features_1""", question_text = """Briefly describe the features of this sailboat model: Sunfish"""),
 Question('free_text', question_name = """features_2""", question_text = """Briefly describe the features of this sailboat model: Optimist"""),
 Question('free_text', question_name = """features_3""", question_text = """Briefly describe the features of this sailboat model: Finn""")]

We can pass the questions to a Survey and then run it:

[4]:

from edsl import Survey

survey = Survey(questions = questions)

results = survey.run()

⌃ Job Status 🦜

Completed (1 completed, 0 failed)

Job Links

Results

Progress Report

Content

Remote Jobs

Remote Cache

Identifiers

Results UUID:

18af41f2...f014

Use Results.pull(uuid) to fetch results.

Job UUID:

782f1ff7...2e4b

Use Jobs.pull(uuid) to fetch job.