Skip logic & scenarios
This notebook provides example EDSL code for using a language model to simulate a survey that uses skip logic in different ways to determine which questions and variations of questions are administered based on responses to other questions in the survey.
In the first example below we construct a survey of questions and then add a rule to skip a particular question based on the response to a prior question.
In the second example we create different versions of questions and combine them in a survey. Then we add rules to skip the certain versions of the questions based on a response to another version of a question.
EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.
Example 1
In the first example below we construct questions, combine them in a survey, and add a rule to skip the second question based on the response to the first question. Then we create scenarios for the questions that will be added to the questions when the survey is run. The effect of this is that the second question will be skipped based on the response to the first question for each individual scenario.
Import the tools:
[1]:
from edsl import QuestionYesNo, QuestionNumerical, QuestionMultipleChoice, Survey, ScenarioList, Scenario, AgentList, Agent, Model
Construct questions:
[2]:
q1 = QuestionYesNo(
question_name = "recent_purchase",
question_text = "In the last year have you or anyone in your household purchased any {{ item }}?",
)
q2 = QuestionNumerical(
question_name = "amount",
question_text = "In the last year, how much did your household spend on {{ item }} (in USD)?"
)
q3 = QuestionMultipleChoice(
question_name = "next_purchase",
question_text = "When do you next expect to purchase {{ item }}?",
question_options = [
"Never",
"Within the next month",
"Within the next year",
"I do not know"
]
)
Combine the questions in a survey to administer them together:
[3]:
survey = Survey([q1, q2, q3])
Add a rule to skip q2 based on the response to q1:
[4]:
survey = survey.add_skip_rule(q2, "recent_purchase == 'No'")
Create scenarios for the “item” in each question:
[5]:
s = ScenarioList(
Scenario({"item":item}) for item in ["electronics", "phones"]
)
Create agents to answer the questions:
[6]:
a = AgentList(
Agent({"annual_income":income, "age":age}) for income in ["under $100,000", "$100,000-250,000", "above $250,000"] for age in [30, 50, 70]
)
Select a model to generate the responses:
[7]:
m = Model("gemini-1.5-flash")
Run the survey with the scenarios, agent and model:
[8]:
results = survey.by(s).by(a).by(m).run()
| Job UUID | f9db513e-8a2f-4da7-9705-7816f1086346 |
| Progress Bar URL | https://www.expectedparrot.com/home/remote-job-progress/f9db513e-8a2f-4da7-9705-7816f1086346 |
| Error Report URL | None |
| Results UUID | c752ea5f-8824-451c-9bc0-bdb9e87b911c |
| Results URL | None |
Inspect the responses. We can see by a “None” response that a question was skipped:
[9]:
(
results
.sort_by("annual_income", "age", "item")
.select("model", "annual_income", "age", "item", "recent_purchase", "amount", "next_purchase")
)
[9]:
| model.model | agent.annual_income | agent.age | scenario.item | answer.recent_purchase | answer.amount | answer.next_purchase | |
|---|---|---|---|---|---|---|---|
| 0 | gemini-1.5-flash | $100,000-250,000 | 30 | electronics | Yes | 2500.000000 | Within the next year |
| 1 | gemini-1.5-flash | $100,000-250,000 | 30 | phones | Yes | 1200.000000 | Within the next year |
| 2 | gemini-1.5-flash | $100,000-250,000 | 50 | electronics | Yes | 2500.000000 | Within the next year |
| 3 | gemini-1.5-flash | $100,000-250,000 | 50 | phones | Yes | 1200.000000 | Within the next year |
| 4 | gemini-1.5-flash | $100,000-250,000 | 70 | electronics | Yes | 500.000000 | Within the next year |
| 5 | gemini-1.5-flash | $100,000-250,000 | 70 | phones | No | nan | Within the next year |
| 6 | gemini-1.5-flash | above $250,000 | 30 | electronics | Yes | 5000.000000 | Within the next year |
| 7 | gemini-1.5-flash | above $250,000 | 30 | phones | Yes | 0.000000 | Within the next year |
| 8 | gemini-1.5-flash | above $250,000 | 50 | electronics | Yes | 15000.000000 | Within the next year |
| 9 | gemini-1.5-flash | above $250,000 | 50 | phones | Yes | 1200.000000 | Within the next year |
| 10 | gemini-1.5-flash | above $250,000 | 70 | electronics | Yes | 0.000000 | Within the next year |
| 11 | gemini-1.5-flash | above $250,000 | 70 | phones | No | nan | Never |
| 12 | gemini-1.5-flash | under $100,000 | 30 | electronics | Yes | 0.000000 | Within the next year |
| 13 | gemini-1.5-flash | under $100,000 | 30 | phones | No | nan | Within the next year |
| 14 | gemini-1.5-flash | under $100,000 | 50 | electronics | No | nan | Within the next year |
| 15 | gemini-1.5-flash | under $100,000 | 50 | phones | No | nan | Within the next year |
| 16 | gemini-1.5-flash | under $100,000 | 70 | electronics | No | nan | Within the next year |
| 17 | gemini-1.5-flash | under $100,000 | 70 | phones | No | nan | Never |
Example 2
In the next example, we use the same scenarios to create versions of the questions before we combine them in a survey. This allows us to add a skip rule based on a question/scenario combination, as opposed to skipping a question for all scenarios:
[10]:
q1 = QuestionYesNo(
question_name = "recent_purchase_{{ item }}",
question_text = "In the last year have you or anyone in your household purchased any {{ item }}?",
)
q2 = QuestionNumerical(
question_name = "amount_{{ item }}",
question_text = "In the last year, how much did your household spend on {{ item }} (in USD)?"
)
q3 = QuestionMultipleChoice(
question_name = "next_purchase_{{ item }}",
question_text = "When do you next expect to purchase {{ item }}?",
question_options = [
"Never",
"Within the next month",
"Within the next year",
"I do not know"
]
)
[11]:
questions = q1.loop(s) + q2.loop(s) + q3.loop(s)
questions
[11]:
[Question('yes_no', question_name = """recent_purchase_electronics""", question_text = """In the last year have you or anyone in your household purchased any electronics?""", question_options = ['No', 'Yes']),
Question('yes_no', question_name = """recent_purchase_phones""", question_text = """In the last year have you or anyone in your household purchased any phones?""", question_options = ['No', 'Yes']),
Question('numerical', question_name = """amount_electronics""", question_text = """In the last year, how much did your household spend on electronics (in USD)?""", min_value = None, max_value = None),
Question('numerical', question_name = """amount_phones""", question_text = """In the last year, how much did your household spend on phones (in USD)?""", min_value = None, max_value = None),
Question('multiple_choice', question_name = """next_purchase_electronics""", question_text = """When do you next expect to purchase electronics?""", question_options = ['Never', 'Within the next month', 'Within the next year', 'I do not know']),
Question('multiple_choice', question_name = """next_purchase_phones""", question_text = """When do you next expect to purchase phones?""", question_options = ['Never', 'Within the next month', 'Within the next year', 'I do not know'])]
Combine the questions in a survey to administer them together:
[12]:
survey = Survey(questions)
Here we add different rules specifying that questions with one scenario (phones) should be administered or skipped based on the answer to a question with another scenario (electronics):
[13]:
survey = (
survey
.add_skip_rule("recent_purchase_phones", "recent_purchase_electronics == 'No'")
.add_skip_rule("amount_phones", "recent_purchase_electronics == 'No'")
.add_skip_rule("next_purchase_phones", "recent_purchase_electronics == 'No'")
)
Run the survey with the scenarios, agents and model:
[14]:
results = survey.by(a).by(m).run()
| Job UUID | f58d69cd-7cd8-463b-ba7c-c0bca9e952c2 |
| Progress Bar URL | https://www.expectedparrot.com/home/remote-job-progress/f58d69cd-7cd8-463b-ba7c-c0bca9e952c2 |
| Error Report URL | https://www.expectedparrot.com/home/remote-inference/error/c470ca6b-310d-4e83-8fa0-08b71f979cfd |
| Results UUID | 3cd20420-37be-4e20-8baf-137b5d2211b0 |
| Results URL | None |
There is no “scenario” field in results because the scenarios were already added to questions:
[15]:
(
results
.sort_by("annual_income", "age")
.select("model", "annual_income", "age", "recent_purchase_electronics", "amount_electronics", "next_purchase_electronics", "recent_purchase_phones", "amount_phones", "next_purchase_phones")
)
[15]:
| model.model | agent.annual_income | agent.age | answer.recent_purchase_electronics | answer.amount_electronics | answer.next_purchase_electronics | answer.recent_purchase_phones | answer.amount_phones | answer.next_purchase_phones | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | gemini-1.5-flash | $100,000-250,000 | 30 | Yes | 2500.000000 | Within the next year | Yes | 1200.000000 | Within the next year |
| 1 | gemini-1.5-flash | $100,000-250,000 | 50 | Yes | 2500.000000 | Within the next year | Yes | 1200.000000 | Within the next year |
| 2 | gemini-1.5-flash | $100,000-250,000 | 70 | Yes | 500.000000 | Within the next year | No | 1200.000000 | Within the next year |
| 3 | gemini-1.5-flash | above $250,000 | 30 | Yes | 5000.000000 | Within the next year | Yes | 0.000000 | Within the next year |
| 4 | gemini-1.5-flash | above $250,000 | 50 | Yes | 15000.000000 | Within the next year | Yes | 1200.000000 | Within the next year |
| 5 | gemini-1.5-flash | above $250,000 | 70 | Yes | 0.000000 | Within the next year | No | 0.000000 | Never |
| 6 | gemini-1.5-flash | under $100,000 | 30 | Yes | 0.000000 | Within the next year | No | 600.000000 | Within the next year |
| 7 | gemini-1.5-flash | under $100,000 | 50 | No | nan | Within the next year | nan | nan | nan |
| 8 | gemini-1.5-flash | under $100,000 | 70 | No | 0.000000 | Within the next year | nan | nan | nan |
Posting to the Coop
Here we post this notebook to the Coop. Learn more.
[16]:
from edsl import Notebook
[17]:
n = Notebook(path = "skip_logic_scenarios.ipynb")
[18]:
info = n.push(description = "Using skip logic with question scenarios", visibility = "public")
info
[18]:
{'description': 'Using skip logic with question scenarios',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/22cfeb7c-0689-46c9-aba9-776b5531b521',
'uuid': '22cfeb7c-0689-46c9-aba9-776b5531b521',
'version': '0.1.39.dev2',
'visibility': 'public'}
Updating an object at the Coop:
[19]:
n = Notebook(path = "skip_logic_scenarios.ipynb") # resave
[20]:
n.patch(uuid = info["uuid"], value = n)
[20]:
{'status': 'success'}