Question types
This notebook provides sample EDSL code for creating different types of questions and using an AI agent to answer them.
EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.
Selecting question types
EDSL comes with many common question types that we can choose from based on the form of the response that we want to get back from the model:
Free text
Multiple choice
Checkbox
Linear scale
Yes/No
Numerical
List
Rank
Top K
Budget
Extract
Learn more about selecting question types.
Free Text
A free_text question prompts the respondent to provide a short unstructured response.
[1]:
from edsl import QuestionFreeText
q_ft = QuestionFreeText(
question_name="q_ft",
question_text="What improvements would you like to see in options for clothes shopping?",
)
Multiple Choice
A multiple_choice question prompts the respondent to select a single option from a given set of options.
[2]:
from edsl import QuestionMultipleChoice
q_mc = QuestionMultipleChoice(
question_name="q_mc",
question_text="How often do you shop for clothes?",
question_options=["Rarely or never", "Annually", "Seasonally", "Monthly", "Daily"],
)
Checkbox
A checkbox question prompts the respondent to select one or more of the given options, which are returned as a list.
[3]:
from edsl import QuestionCheckBox
q_cb = QuestionCheckBox(
question_name="q_cb",
question_text="""Which of the following factors are important to you in making decisions about clothes shopping?
Select all that apply.""",
question_options=[
"Price",
"Quality",
"Brand Reputation",
"Style and Design",
"Fit and Comfort",
"Customer Reviews and Recommendations",
"Ethical and Sustainable Practices",
"Return Policy",
"Convenience",
"Other",
],
min_selections=1, # This is optional
max_selections=3, # This is optional
)
Linear Scale
A linear_scale question prompts the respondent to choose from a set of numerical options.
[4]:
from edsl import QuestionLinearScale
q_ls = QuestionLinearScale(
question_name="q_ls",
question_text="On a scale of 0-10, how much do you typically enjoy clothes shopping?",
question_options=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
option_labels = {0:"Not at all", 10:"Very much"}
)
Yes / No
A yes_no question requires the respondent to respond “yes” or “no”. Response options are set by default and not modifiable. To include other options use a multiple choice question.
[5]:
from edsl import QuestionYesNo
q_yn = QuestionYesNo(
question_name="q_yn",
question_text="Have you ever felt excluded or frustrated by the standard sizes of the fashion industry?",
)
Numerical
A numerical question prompts the respondent to provide a response that is a number.
[6]:
from edsl import QuestionNumerical
q_nu = QuestionNumerical(
question_name="q_nu",
question_text="Estimate the amount of money that you spent on clothing in the past year (in $USD).",
)
List
A list question prompts the respondent to provide a response in the form of a list. This can be a convenient way to reformat free text questions.
[7]:
from edsl import QuestionList
q_li = QuestionList(
question_name="q_li",
question_text="What considerations are important to you in shopping for clothes?",
)
Rank
A rank question prompts the respondent to rank the items in a given list.
[8]:
from edsl import QuestionRank
q_rk = QuestionRank(
question_name="q_rk",
question_text="Rank the following items in terms of how much you enjoy shopping for them.",
question_options=[
"Hats",
"Scarves",
"Shirts",
"Jackets",
"Pants",
"Shoes"
],
num_selections=5 # optional
)
Top K
A top_k question prompts the respondent to identify a specified number of options from a list. This type is similar to checkbox and rank types.
[9]:
from edsl import QuestionTopK
q_tk = QuestionTopK(
question_name="q_tk",
question_text="Which of the following payment methods do you prefer?",
question_options=[
"Credit card",
"Debit card",
"Cash",
"Check",
"Smartphone app",
"Other"
],
min_selections=2, # min and max must be equal
max_selections=2
)
Budget
A budget question prompts the respondent to allocation a specified sum among a set of options.
[10]:
from edsl import QuestionBudget
q_bg = QuestionBudget(
question_name="q_bg",
question_text="""Estimate the percentage of your total time spent shopping for clothes in each of the
following modes.""",
question_options=[
"Online",
"Malls",
"Freestanding stores",
"Mail order catalogs",
"Other",
],
budget_sum=100,
)
Extract
A question type thatprompts the respondent to provide a response in the form of a dictionary, where the keys and example values are provided.
[11]:
from edsl import QuestionExtract
q_ex = QuestionExtract(
question_name="q_ex",
question_text="""Consider all of the articles of clothing in your closet.
Identify the categories of clothing that are most and least valuable to you.""",
answer_template={"most_valuable": "socks", "least_valuable": "shoes"},
)
Designing AI agents
Next we can design one or more AI agents for a language model to use in answering the questions. This is done by creating an Agent object and passing it a dictionary of relevant traits. Learn more about designing agents to use with surveys.
[12]:
from edsl import AgentList, Agent
agents = AgentList([
Agent(
name="Fashion designer", # optional
traits={"persona": "You are an expert in fashion design."},
instruction="You are answering a survey about your personal experiences shopping for clothes."
),
Agent(
name="College student", # optional
traits={"persona": "You are an undergraduate student."},
instruction="You are answering a survey about your personal experiences shopping for clothes."
)
])
Selecting language models
EDSL works with many popular language models that we can select to generate the responses. Learn more about working with language models.
To see a list of all available models:
[13]:
from edsl import ModelList, Model
[14]:
# Model.available()
Here we select models to use with our survey. (If none are selected, the default model GPT 4 preview is used.)
[15]:
models = ModelList(
Model(m) for m in ["gpt-4o", "claude-3-5-sonnet-20240620"]
)
Administering questions
We administer a question to a language model by adding any agents and models with the by() method, and then calling the run() method:
[16]:
result_ft = q_ft.by(agents).by(models).run()
▼
Job Status (2025-03-03 09:33:11)
| Job UUID | e815b28c-1696-44a8-9561-9e63fb0a7993 |
| Progress Bar URL | https://www.expectedparrot.com/home/remote-job-progress/e815b28c-1696-44a8-9561-9e63fb0a7993 |
| Exceptions Report URL | None |
| Results UUID | f830c7c2-de86-4559-a9ac-19ba75f10fa0 |
| Results URL | https://www.expectedparrot.com/content/f830c7c2-de86-4559-a9ac-19ba75f10fa0 |
✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/f830c7c2-de86-4559-a9ac-19ba75f10fa0
We can use built-in methods for inspecting the response:
[17]:
result_ft.select("model", "agent_name", "q_ft").print(format="rich")
[17]:
| model.model | agent.agent_name | answer.q_ft | |
|---|---|---|---|
| 0 | gpt-4o | Fashion designer | As someone with expertise in fashion design, there are several improvements I would like to see in clothes shopping options: 1. **Sustainability and Ethical Practices**: More focus on sustainable materials and ethical production processes. Brands should provide transparency about their supply chains and the environmental impact of their products. 2. **Customization and Personalization**: Enhanced options for customization to cater to individual preferences and body types. This could include more made-to-measure services or customizable design elements. 3. **Technology Integration**: Better integration of technology, such as virtual try-ons and AI-driven recommendations, to improve the online shopping experience and reduce returns. 4. **Inclusive Sizing**: A wider range of sizes that truly reflect the diversity of body shapes and sizes, ensuring everyone can find stylish and well-fitting clothing. 5. **Local and Artisanal Options**: Increased availability of locally made and artisanal products, supporting small designers and offering unique, high-quality pieces. 6. **Improved In-Store Experience**: More engaging and interactive in-store experiences, perhaps through workshops or events that connect consumers with the design process. 7. **Efficient Returns and Exchanges**: Streamlined processes for returns and exchanges, making it easier for customers to find the right fit or style without hassle. These improvements would not only enhance the shopping experience but also promote a more responsible and inclusive fashion industry. |
| 1 | claude-3-5-sonnet-20240620 | Fashion designer | As an expert in fashion design, I would like to see several improvements in clothes shopping options: 1. Better size inclusivity: More brands should offer a wider range of sizes, from petite to plus-size, ensuring that fashion is accessible to all body types. 2. Sustainable materials: Increased use of eco-friendly fabrics and production methods to reduce the fashion industry's environmental impact. 3. Customization options: More opportunities for customers to personalize their clothing, such as choosing fabrics, colors, or small design details. 4. Improved virtual try-on technology: Enhanced augmented reality and 3D modeling to allow customers to see how clothes will look on their specific body type when shopping online. 5. Transparency in production: Clear information about where and how clothes are made, including labor practices and materials sourcing. 6. Modular clothing systems: Designs that allow for easy mixing and matching, or pieces that can be worn multiple ways to increase versatility. 7. Integration of technology: Smart fabrics and wearable tech that enhance functionality without sacrificing style. 8. More diverse representation: In marketing and design, to showcase how clothing looks on various body types, skin tones, and ages. 9. Rental and subscription services: Expanded options for accessing high-quality, designer pieces without the need for permanent ownership. 10. Improved fit technology: Better use of data and AI to recommend sizes and styles based on individual body measurements and preferences. These improvements would not only enhance the shopping experience but also address some of the current challenges in the fashion industry related to sustainability, inclusivity, and personalization. |
| 2 | gpt-4o | College student | As an undergraduate student, some improvements I would like to see in clothes shopping include: 1. **Affordable Pricing**: More student discounts or budget-friendly options would be great, especially since managing finances on a student budget can be challenging. 2. **Sustainable Options**: An increase in eco-friendly and sustainable clothing options that are also affordable would be beneficial. 3. **Inclusive Sizing**: A wider range of sizes to accommodate all body types, ensuring everyone can find clothes that fit well. 4. **Convenient Returns**: Easier and free return policies, especially for online shopping, would make it less risky to try new styles or sizes. 5. **Trendy yet Comfortable Styles**: More options that combine current trends with comfort, ideal for both campus life and social activities. 6. **Virtual Try-Ons**: Improved virtual try-on technology to better visualize how clothes will look and fit before purchasing online. 7. **Local and Unique Brands**: More visibility for local or less mainstream brands that offer unique styles, giving students a chance to express individuality. 8. **Customizable Options**: The ability to customize certain aspects of clothing, like colors or designs, to better suit personal style preferences. |
| 3 | claude-3-5-sonnet-20240620 | College student | As an undergraduate student, I'd like to see a few improvements in clothes shopping options: 1. More affordable, trendy clothing choices that fit a student budget. Maybe more student discounts or budget-friendly lines from popular brands. 2. Easier online shopping experiences with better size guides and return policies. It can be hard to know how things will fit when ordering online. 3. More sustainable and ethically-made clothing options that are still reasonably priced. Many students care about environmental and social issues. 4. Expanded size ranges to be more inclusive of different body types. 5. More versatile pieces that can transition from casual campus wear to internship or job interview appropriate attire. 6. Pop-up shops or events on campus to make shopping more convenient and social for students. 7. Loyalty programs or rewards that cater specifically to students and our shopping habits. These improvements would make clothes shopping easier and more enjoyable for someone like me on a student budget and schedule. |
Creating a survey
We can also compile all the questions into a Survey to administer them at once:
[18]:
from edsl import Survey
survey = Survey(questions=[q_mc, q_cb, q_ls, q_yn, q_nu, q_li, q_rk, q_tk, q_bg, q_ex])
We administer a survey the same way that we do an individual question:
[19]:
results = survey.by(agents).by(models).run()
▼
Job Status (2025-03-03 09:33:22)
| Job UUID | 1be59269-5d32-43e9-b07d-06a2959aa1ba |
| Progress Bar URL | https://www.expectedparrot.com/home/remote-job-progress/1be59269-5d32-43e9-b07d-06a2959aa1ba |
| Exceptions Report URL | None |
| Results UUID | 6e133f5d-2b20-44f0-a5be-c253574f2a56 |
| Results URL | https://www.expectedparrot.com/content/6e133f5d-2b20-44f0-a5be-c253574f2a56 |
✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/6e133f5d-2b20-44f0-a5be-c253574f2a56
We can see a list of all the components of the results that have been generated:
[20]:
results.columns
[20]:
| 0 | |
|---|---|
| 0 | agent.agent_index |
| 1 | agent.agent_instruction |
| 2 | agent.agent_name |
| 3 | agent.persona |
| 4 | answer.q_bg |
| 5 | answer.q_cb |
| 6 | answer.q_ex |
| 7 | answer.q_li |
| 8 | answer.q_ls |
| 9 | answer.q_mc |
| 10 | answer.q_nu |
| 11 | answer.q_rk |
| 12 | answer.q_tk |
| 13 | answer.q_yn |
| 14 | cache_keys.q_bg_cache_key |
| 15 | cache_keys.q_cb_cache_key |
| 16 | cache_keys.q_ex_cache_key |
| 17 | cache_keys.q_li_cache_key |
| 18 | cache_keys.q_ls_cache_key |
| 19 | cache_keys.q_mc_cache_key |
| 20 | cache_keys.q_nu_cache_key |
| 21 | cache_keys.q_rk_cache_key |
| 22 | cache_keys.q_tk_cache_key |
| 23 | cache_keys.q_yn_cache_key |
| 24 | cache_used.q_bg_cache_used |
| 25 | cache_used.q_cb_cache_used |
| 26 | cache_used.q_ex_cache_used |
| 27 | cache_used.q_li_cache_used |
| 28 | cache_used.q_ls_cache_used |
| 29 | cache_used.q_mc_cache_used |
| 30 | cache_used.q_nu_cache_used |
| 31 | cache_used.q_rk_cache_used |
| 32 | cache_used.q_tk_cache_used |
| 33 | cache_used.q_yn_cache_used |
| 34 | comment.q_bg_comment |
| 35 | comment.q_cb_comment |
| 36 | comment.q_ex_comment |
| 37 | comment.q_li_comment |
| 38 | comment.q_ls_comment |
| 39 | comment.q_mc_comment |
| 40 | comment.q_nu_comment |
| 41 | comment.q_rk_comment |
| 42 | comment.q_tk_comment |
| 43 | comment.q_yn_comment |
| 44 | generated_tokens.q_bg_generated_tokens |
| 45 | generated_tokens.q_cb_generated_tokens |
| 46 | generated_tokens.q_ex_generated_tokens |
| 47 | generated_tokens.q_li_generated_tokens |
| 48 | generated_tokens.q_ls_generated_tokens |
| 49 | generated_tokens.q_mc_generated_tokens |
| 50 | generated_tokens.q_nu_generated_tokens |
| 51 | generated_tokens.q_rk_generated_tokens |
| 52 | generated_tokens.q_tk_generated_tokens |
| 53 | generated_tokens.q_yn_generated_tokens |
| 54 | iteration.iteration |
| 55 | model.frequency_penalty |
| 56 | model.inference_service |
| 57 | model.logprobs |
| 58 | model.max_tokens |
| 59 | model.model |
| 60 | model.model_index |
| 61 | model.presence_penalty |
| 62 | model.temperature |
| 63 | model.top_logprobs |
| 64 | model.top_p |
| 65 | prompt.q_bg_system_prompt |
| 66 | prompt.q_bg_user_prompt |
| 67 | prompt.q_cb_system_prompt |
| 68 | prompt.q_cb_user_prompt |
| 69 | prompt.q_ex_system_prompt |
| 70 | prompt.q_ex_user_prompt |
| 71 | prompt.q_li_system_prompt |
| 72 | prompt.q_li_user_prompt |
| 73 | prompt.q_ls_system_prompt |
| 74 | prompt.q_ls_user_prompt |
| 75 | prompt.q_mc_system_prompt |
| 76 | prompt.q_mc_user_prompt |
| 77 | prompt.q_nu_system_prompt |
| 78 | prompt.q_nu_user_prompt |
| 79 | prompt.q_rk_system_prompt |
| 80 | prompt.q_rk_user_prompt |
| 81 | prompt.q_tk_system_prompt |
| 82 | prompt.q_tk_user_prompt |
| 83 | prompt.q_yn_system_prompt |
| 84 | prompt.q_yn_user_prompt |
| 85 | question_options.q_bg_question_options |
| 86 | question_options.q_cb_question_options |
| 87 | question_options.q_ex_question_options |
| 88 | question_options.q_li_question_options |
| 89 | question_options.q_ls_question_options |
| 90 | question_options.q_mc_question_options |
| 91 | question_options.q_nu_question_options |
| 92 | question_options.q_rk_question_options |
| 93 | question_options.q_tk_question_options |
| 94 | question_options.q_yn_question_options |
| 95 | question_text.q_bg_question_text |
| 96 | question_text.q_cb_question_text |
| 97 | question_text.q_ex_question_text |
| 98 | question_text.q_li_question_text |
| 99 | question_text.q_ls_question_text |
| 100 | question_text.q_mc_question_text |
| 101 | question_text.q_nu_question_text |
| 102 | question_text.q_rk_question_text |
| 103 | question_text.q_tk_question_text |
| 104 | question_text.q_yn_question_text |
| 105 | question_type.q_bg_question_type |
| 106 | question_type.q_cb_question_type |
| 107 | question_type.q_ex_question_type |
| 108 | question_type.q_li_question_type |
| 109 | question_type.q_ls_question_type |
| 110 | question_type.q_mc_question_type |
| 111 | question_type.q_nu_question_type |
| 112 | question_type.q_rk_question_type |
| 113 | question_type.q_tk_question_type |
| 114 | question_type.q_yn_question_type |
| 115 | raw_model_response.q_bg_cost |
| 116 | raw_model_response.q_bg_one_usd_buys |
| 117 | raw_model_response.q_bg_raw_model_response |
| 118 | raw_model_response.q_cb_cost |
| 119 | raw_model_response.q_cb_one_usd_buys |
| 120 | raw_model_response.q_cb_raw_model_response |
| 121 | raw_model_response.q_ex_cost |
| 122 | raw_model_response.q_ex_one_usd_buys |
| 123 | raw_model_response.q_ex_raw_model_response |
| 124 | raw_model_response.q_li_cost |
| 125 | raw_model_response.q_li_one_usd_buys |
| 126 | raw_model_response.q_li_raw_model_response |
| 127 | raw_model_response.q_ls_cost |
| 128 | raw_model_response.q_ls_one_usd_buys |
| 129 | raw_model_response.q_ls_raw_model_response |
| 130 | raw_model_response.q_mc_cost |
| 131 | raw_model_response.q_mc_one_usd_buys |
| 132 | raw_model_response.q_mc_raw_model_response |
| 133 | raw_model_response.q_nu_cost |
| 134 | raw_model_response.q_nu_one_usd_buys |
| 135 | raw_model_response.q_nu_raw_model_response |
| 136 | raw_model_response.q_rk_cost |
| 137 | raw_model_response.q_rk_one_usd_buys |
| 138 | raw_model_response.q_rk_raw_model_response |
| 139 | raw_model_response.q_tk_cost |
| 140 | raw_model_response.q_tk_one_usd_buys |
| 141 | raw_model_response.q_tk_raw_model_response |
| 142 | raw_model_response.q_yn_cost |
| 143 | raw_model_response.q_yn_one_usd_buys |
| 144 | raw_model_response.q_yn_raw_model_response |
| 145 | scenario.scenario_index |
We can filter, sort, select and print any of the components:
[22]:
(
results.filter("model.model == 'gpt-4o'")
.sort_by("agent_name")
.select("agent_name", "q_mc", "q_cb", "q_ls", "q_yn", "q_nu")
.print(pretty_labels={
"agent.agent_name": "Agent",
"answer.q_mc": q_mc.question_text,
"answer.q_cb": q_cb.question_text,
"answer.q_ls": q_ls.question_text,
"answer.q_yn": q_yn.question_text,
"answer.q_nu": q_nu.question_text,
}, format="rich")
)
[22]:
| Agent | How often do you shop for clothes? | Which of the following factors are important to you in making decisions about clothes shopping? Select all that apply. | On a scale of 0-10, how much do you typically enjoy clothes shopping? | Have you ever felt excluded or frustrated by the standard sizes of the fashion industry? | Estimate the amount of money that you spent on clothing in the past year (in $USD). | |
|---|---|---|---|---|---|---|
| 0 | College student | Seasonally | ['Price', 'Style and Design', 'Fit and Comfort'] | 6 | Yes | 300 |
| 1 | Fashion designer | Seasonally | ['Quality', 'Style and Design', 'Ethical and Sustainable Practices'] | 8 | Yes | 5000 |
Please see our documentation page for more details on all of the above objects and methods.