Using EDSL to sense check data
This notebook provides example code for sense checking survey data using EDSL, an open-source library for simulating surveys, experiments and market research with AI agents and large language models.
Contents
Using a set of responses to a survey about online marketplaces as an example, we demonstrate EDSL methods for:
Evaluating survey questions (e.g., for clarity and improvements)
Analyzing each respondent’s set of answers (e.g., to summarize or identify sentiment, themes, etc.)
Reviewing each answer individually (e.g., to evaluate its relevance or usefulness)
Coop
We also show how to post EDSL questions, surveys, results and notebooks (like this one) to the Coop: a new platform for creating and sharing LLM-based research.
How EDSL works
EDSL is a flexible library that can be used to perform a broad variety of research tasks. A typical workflow consists of the following steps:
Construct questions in EDSL
Add data to the questions (e.g., for data labeling tasks)
Use an AI agent to answer the questions
Select a language model to generate the answers
Analyze results in a formatted dataset
Technical setup
Before running the code below please ensure that you have completed setup:
Install the EDSL library.
Create a Coop account and activate remote inference OR store your own API Keys for language models that you want to use.
Our Starter Tutorial provides examples of EDSL basic components.
Example data
Our example data is a CSV consisting of several questions and a few rows of responses. Here we store it at the Coop and then re-import it:
[1]:
from edsl.scenarios.FileStore import CSVFileStore
[2]:
fs = CSVFileStore("marketplace_survey_results.csv")
info = fs.push()
print(info)
{'description': 'File: marketplace_survey_results.csv', 'object_type': 'scenario', 'url': 'https://www.expectedparrot.com/content/ba9148ae-ab7c-4630-baa4-046f8d237b28', 'uuid': 'ba9148ae-ab7c-4630-baa4-046f8d237b28', 'version': '0.1.39.dev1', 'visibility': 'unlisted'}
[3]:
csv_file = CSVFileStore.pull(info["uuid"])
Creating questions about the data
There are many questions we might want to ask about the data, such as:
Does this survey question have any logical or syntactical problems? {{ question }}
What is the overall sentiment of this respondent’s answers? {{ responses }}
Is this answer responsive to the question that was asked? {{ question }} {{ answer }}
Question types
EDSL comes with many common question types that we can select from based on the form of the response that we want to get back from the model: multiple choice, checkbox, linear scale, free text, etc. Learn more about EDSL question types.
Here we construct Question objects for the questions that we want to ask about the data, using {{ placeholders }} for the information that we will add to the questions in the steps that follow:
[4]:
from edsl import QuestionFreeText, QuestionMultipleChoice, QuestionYesNo
[5]:
q_logic = QuestionFreeText(
question_name = "logic",
question_text = "Describe any logical or syntactical problems in the following survey question: {{ question }}"
)
[6]:
q_sentiment = QuestionMultipleChoice(
question_name = "sentiment",
question_text = "What is the overall sentiment of this respondent's survey answers? {{ responses }}",
question_options = ["Very unsatisfied", "Somewhat unsatisfied", "Somewhat satisfied", "Very satisfied"]
)
[7]:
q_responsive = QuestionYesNo(
question_name = "responsive",
question_text = "Is this answer responsive to the question that was asked? Question: {{ question }} Answer: {{ answer }}"
)
Adding survey data to the questions
Next we’ll add our data to our questions. This can be done efficiently by creating a ScenarioList representing the data. The individual Scenario objects in the list can be constructed in a variety of ways depending on the information that we want to include in a particular question.
We start by calling the from_csv() method to create a ScenarioList for the data in its original form. We can see that this generates a Scenario dictionary for each respondent’s set of answers with key/value pairs for the individual questions and answers:
[8]:
from edsl import ScenarioList
[9]:
sl = ScenarioList.from_csv(csv_file.to_tempfile()) # replace with CSV file name if importing a local file
sl
[9]:
ScenarioList scenarios: 3; keys: ['Is there anything else you would like to share about your experience with us?', 'What is one feature you would like to see added to improve your shopping experience?', 'How do you feel about the current product search and filtering options?', 'Respondent ID', 'Can you describe a recent experience where you were dissatisfied with our service?', 'What do you like most about using our online marketplace?'];
| Respondent ID | What do you like most about using our online marketplace? | What is one feature you would like to see added to improve your shopping experience? | Can you describe a recent experience where you were dissatisfied with our service? | How do you feel about the current product search and filtering options? | Is there anything else you would like to share about your experience with us? |
|---|---|---|---|---|---|
| 101 | The wide variety of products and the ease of use. | It would be great to have a personalized recommendation system based on my browsing history. | I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it. | The search and filtering options are intuitive and work well for me. | No, keep up the great work! |
| 102 | I enjoy the simplicity of the interface. | A feature that helps compare similar products side by side would be useful. | No complaints here. | I find the product search to be pretty effective. | I think the sky is a beautiful shade of purple today. |
| 103 | The platform is user-friendly and offers a vast selection of products. | Would love to see an option to save and compare different products. | My delivery was late by a few days, which was frustrating. | It’s okay. | No. |
Evaluating the questions
For our first question we want to create a Scenario for each survey question:
[10]:
from edsl import QuestionFreeText, Survey
q_logic = QuestionFreeText(
question_name = "logic",
question_text = "Describe any logical or syntactical problems in the following survey question: {{ question }}"
)
q_improved = QuestionFreeText(
question_name = "improved",
question_text = "Please draft an improved version of the survey question. Return only the revised question text."
)
survey = Survey([q_logic, q_improved]).add_targeted_memory(q_improved, q_logic)
The survey questions are the parameters of the ScenarioList created above:
[11]:
questions = list(sl.parameters)
questions
[11]:
['Is there anything else you would like to share about your experience with us?',
'What is one feature you would like to see added to improve your shopping experience?',
'How do you feel about the current product search and filtering options?',
'Respondent ID',
'Can you describe a recent experience where you were dissatisfied with our service?',
'What do you like most about using our online marketplace?']
We can pass them to the from_list() method to create a new ScenarioList, specifying that the key for each Scenario will be question in order to match the parameter of our logic question:
[12]:
sl_questions = ScenarioList.from_list("question", questions)
sl_questions
[12]:
ScenarioList scenarios: 6; keys: ['question'];
| question |
|---|
| Is there anything else you would like to share about your experience with us? |
| What is one feature you would like to see added to improve your shopping experience? |
| How do you feel about the current product search and filtering options? |
| Respondent ID |
| Can you describe a recent experience where you were dissatisfied with our service? |
| What do you like most about using our online marketplace? |
We add the scenarios to the survey when we run it:
[13]:
results = survey.by(sl_questions).run()
This generates a dataset of Results that we can access with built-in methods for analysis:
[14]:
results.select("question", "logic", "improved")
[14]:
| scenario.question | answer.logic | answer.improved |
|---|---|---|
| Is there anything else you would like to share about your experience with us? | The survey question "Is there anything else you would like to share about your experience with us?" is generally clear and straightforward, but there are a few considerations that could be addressed to improve its effectiveness: 1. **Open-Ended Nature**: The question is open-ended, which can be beneficial for gathering detailed feedback but may also result in responses that are difficult to categorize or analyze. Depending on the survey's goals, it might be helpful to provide some guidance on the type of information being sought. 2. **Vagueness**: The phrase "anything else" is quite broad and might lead to a wide range of responses. If the survey aims to gather specific types of feedback, it could be helpful to specify or narrow down the focus. 3. **Contextual Clarity**: Without context, respondents might be unsure about what aspects of their experience they should focus on. Adding context or examples could help guide responses. 4. **Emotional Tone**: The question assumes a neutral tone, which is generally appropriate, but if the survey aims to capture emotional responses, it might benefit from a more empathetic or inviting wording. | Could you please share any additional thoughts or specific feedback about your experience with us, such as areas for improvement or aspects you particularly enjoyed? |
| Is there anything else you would like to share about your experience with us? | The survey question "Is there anything else you would like to share about your experience with us?" is generally clear and straightforward, but there are a few considerations that could be addressed to improve its effectiveness: 1. **Open-Ended Nature**: The question is open-ended, which can be beneficial for gathering detailed feedback but may also result in responses that are difficult to categorize or analyze. Depending on the survey's goals, it might be helpful to provide some guidance on the type of information being sought. 2. **Vagueness**: The phrase "anything else" is quite broad and might lead to a wide range of responses. If the survey aims to gather specific types of feedback, it could be helpful to specify or narrow down the focus. 3. **Contextual Clarity**: Without context, respondents might be unsure about what aspects of their experience they should focus on. Adding context or examples could help guide responses. 4. **Emotional Tone**: The question assumes a neutral tone, which is generally appropriate, but if the survey aims to capture emotional responses, it might benefit from a more empathetic or inviting wording. | Could you please share any additional thoughts or specific feedback about your experience with us, such as areas for improvement or aspects you particularly enjoyed? |
| What is one feature you would like to see added to improve your shopping experience? | The survey question "What is one feature you would like to see added to improve your shopping experience?" is generally clear and straightforward, but there are a few potential issues to consider: 1. **Limitation to One Feature**: The question restricts respondents to mentioning only one feature. This might limit the feedback you receive, as respondents may have multiple suggestions or ideas that could be valuable. Consider allowing respondents to list more than one feature or providing an option to elaborate further. 2. **Ambiguity in "Feature"**: The term "feature" is somewhat vague and may be interpreted differently by different respondents. Some might think of website or app features, while others might consider in-store features or customer service improvements. Clarifying the context or providing examples could help ensure more focused and useful responses. 3. **Assumption of Improvement**: The question assumes that adding a new feature is the best way to improve the shopping experience. However, some respondents might feel that improving existing features, services, or processes could be more beneficial. Consider rephrasing to allow for suggestions related to both new and existing features. 4. **Lack of Specificity**: The question does not specify the type of shopping experience (e.g., online, in-store, or both). If the survey is aimed at a specific type of shopping experience, it would be helpful to include that context to gather more relevant feedback. | What features or improvements would you suggest to enhance your online or in-store shopping experience? Please feel free to list multiple suggestions. |
| How do you feel about the current product search and filtering options? | The survey question "How do you feel about the current product search and filtering options?" is generally clear, but it has a few potential issues that could be improved: 1. **Ambiguity**: The question might be too broad or vague. "Feel" is a subjective term and can lead to varied interpretations. Respondents might not know if they should focus on emotional responses (like satisfaction or frustration) or practical ones (like ease of use or effectiveness). 2. **Lack of Specificity**: It combines two different elements—product search and filtering options—into one question. If respondents have different opinions on these two aspects, they may find it difficult to provide a single, coherent answer. 3. **Open-ended Format**: While open-ended questions can provide rich data, they can also lead to varied responses that are difficult to analyze quantitatively. If the survey aims to gather quantitative data, it might be better to use a closed-ended question with specific response options. 4. **Assumes Familiarity**: The question assumes that the respondent has used the product search and filtering options. If someone hasn't used these features, they might not be able to provide a meaningful response. To address these issues, you might consider breaking the question into two separate questions and providing specific response options. For example: 1. "How satisfied are you with the current product search options?" - Very satisfied - Satisfied - Neutral - Dissatisfied - Very dissatisfied 2. "How satisfied are you with the current filtering options?" - Very satisfied - Satisfied - Neutral - Dissatisfied - Very dissatisfied | 1. "How satisfied are you with the current product search options?" - Very satisfied - Satisfied - Neutral - Dissatisfied - Very dissatisfied 2. "How satisfied are you with the current filtering options?" - Very satisfied - Satisfied - Neutral - Dissatisfied |
| Respondent ID | The survey question "Respondent ID" has a few issues: 1. **Lack of Clarity**: The phrase "Respondent ID" is not a question. It lacks a clear query or prompt for the respondent to answer. In a survey, questions should be formulated to elicit specific information from respondents. 2. **Missing Instruction**: There is no instruction on what the respondent is supposed to do with "Respondent ID." Is it asking them to provide their ID, or is it just a placeholder for the survey administrator to fill in? This ambiguity can lead to confusion. 3. **Assumption of Prior Knowledge**: It assumes that respondents know what a "Respondent ID" is and whether they have one. If this is not explained elsewhere in the survey, respondents might not understand what is being asked of them. 4. **Syntactical Structure**: As a standalone item, "Respondent ID" does not have a syntactical structure that forms a complete sentence or question, which can make it seem out of place in a survey context. | Please enter your Respondent ID below: |
| Can you describe a recent experience where you were dissatisfied with our service? | The survey question "Can you describe a recent experience where you were dissatisfied with our service?" has a few potential issues: 1. **Assumption of Dissatisfaction**: The question assumes that the respondent has had a recent experience of dissatisfaction. If a respondent has not had such an experience, they might find it difficult to answer the question, which could lead to frustration or confusion. 2. **Lack of Neutrality**: The question is negatively biased as it specifically asks about dissatisfaction. It doesn't provide an opportunity for respondents to share positive or neutral experiences, which could skew the feedback towards negative responses. 3. **Open-ended Nature**: While open-ended questions can provide detailed feedback, they can also be difficult to analyze systematically. Respondents might provide varying levels of detail, making it challenging to categorize and quantify the responses. | Have you had any recent experiences with our service that you would like to share, whether positive, negative, or neutral? If so, please describe them. |
| What do you like most about using our online marketplace? | The survey question "What do you like most about using our online marketplace?" is generally clear and straightforward, but there are a few potential issues to consider: 1. **Assumption of Positive Experience**: The question assumes that the respondent has a positive experience with the online marketplace and has something they like about it. This may not be the case for all users, and those who have a neutral or negative experience might find it difficult to answer. 2. **Lack of Options for Negative Feedback**: By focusing solely on what the respondent likes, the question does not provide an opportunity for users to express dissatisfaction or suggest improvements. Including a way to capture negative feedback can provide a more comprehensive understanding of user experiences. 3. **Open-Ended Nature**: While open-ended questions can provide rich qualitative data, they can also lead to varied responses that are difficult to analyze quantitatively. Depending on the survey's goals, it might be beneficial to include some structured response options or follow-up questions. 4. **Ambiguity in Scope**: The term "online marketplace" might be interpreted differently by respondents. It could refer to the website, the mobile app, the range of products, customer service, or other aspects. Clarifying what aspects are being referred to could help in getting more precise answers. | What aspects of our online marketplace do you find most beneficial, and are there any areas where you think we could improve? |
Evaluating respondents’ collective answers
Next we can create a ScenarioList for each respondent’s answers to use with our question about sentiment:
[15]:
sl_responses = ScenarioList.from_list("responses", sl['scenarios'])
sl_responses
[15]:
ScenarioList scenarios: 3; keys: ['responses'];
| responses |
|---|
| {'Respondent ID': '101', 'What do you like most about using our online marketplace?': 'The wide variety of products and the ease of use.', 'What is one feature you would like to see added to improve your shopping experience?': 'It would be great to have a personalized recommendation system based on my browsing history.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it.', 'How do you feel about the current product search and filtering options?': 'The search and filtering options are intuitive and work well for me.', 'Is there anything else you would like to share about your experience with us?': 'No, keep up the great work!'} |
| {'Respondent ID': '102', 'What do you like most about using our online marketplace?': 'I enjoy the simplicity of the interface.', 'What is one feature you would like to see added to improve your shopping experience?': 'A feature that helps compare similar products side by side would be useful.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'No complaints here.', 'How do you feel about the current product search and filtering options?': 'I find the product search to be pretty effective.', 'Is there anything else you would like to share about your experience with us?': 'I think the sky is a beautiful shade of purple today.'} |
| {'Respondent ID': '103', 'What do you like most about using our online marketplace?': 'The platform is user-friendly and offers a vast selection of products.', 'What is one feature you would like to see added to improve your shopping experience?': 'Would love to see an option to save and compare different products.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'My delivery was late by a few days, which was frustrating.', 'How do you feel about the current product search and filtering options?': 'It’s okay.', 'Is there anything else you would like to share about your experience with us?': 'No.'} |
Next we add these scenarios to our sentiment question (and any others we want to add) and run it:
[16]:
from edsl import QuestionMultipleChoice, QuestionLinearScale, Survey
q_sentiment = QuestionMultipleChoice(
question_name = "sentiment",
question_text = "What is the overall sentiment of this respondent's survey answers? {{ responses }}",
question_options = ["Very unsatisfied", "Somewhat unsatisfied", "Somewhat satisfied", "Very satisfied"]
)
q_recommend = QuestionLinearScale(
question_name = "recommend",
question_text = "On a scale from 1 to 5, how likely do you think this respondent is to recommend the company to a friend? {{ responses }}",
question_options = [1, 2, 3, 4, 5],
option_labels = {1:"Not at all likely", 5:"Very likely"}
)
survey = Survey([q_sentiment, q_recommend])
[18]:
results = survey.by(sl_responses).run()
[19]:
results.select("responses", "sentiment", "recommend")
[19]:
| scenario.responses | answer.sentiment | answer.recommend |
|---|---|---|
| {'Respondent ID': '101', 'What do you like most about using our online marketplace?': 'The wide variety of products and the ease of use.', 'What is one feature you would like to see added to improve your shopping experience?': 'It would be great to have a personalized recommendation system based on my browsing history.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it.', 'How do you feel about the current product search and filtering options?': 'The search and filtering options are intuitive and work well for me.', 'Is there anything else you would like to share about your experience with us?': 'No, keep up the great work!'} | Somewhat satisfied | 5 |
| {'Respondent ID': '102', 'What do you like most about using our online marketplace?': 'I enjoy the simplicity of the interface.', 'What is one feature you would like to see added to improve your shopping experience?': 'A feature that helps compare similar products side by side would be useful.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'No complaints here.', 'How do you feel about the current product search and filtering options?': 'I find the product search to be pretty effective.', 'Is there anything else you would like to share about your experience with us?': 'I think the sky is a beautiful shade of purple today.'} | Somewhat satisfied | 5 |
| {'Respondent ID': '103', 'What do you like most about using our online marketplace?': 'The platform is user-friendly and offers a vast selection of products.', 'What is one feature you would like to see added to improve your shopping experience?': 'Would love to see an option to save and compare different products.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'My delivery was late by a few days, which was frustrating.', 'How do you feel about the current product search and filtering options?': 'It’s okay.', 'Is there anything else you would like to share about your experience with us?': 'No.'} | Somewhat satisfied | 3 |
Evaluating individual answers
Next we create a ScenarioList for each individual question and answer to use with our question about the responsiveness of each answer. We can use the unpivot() method to expand the scenarios by desired identifiers (e.g., respondent ID):
[20]:
sl_qa = sl.unpivot(id_vars = ["Respondent ID"])
sl_qa
[20]:
ScenarioList scenarios: 15; keys: ['variable', 'Respondent ID', 'value'];
| Respondent ID | variable | value |
|---|---|---|
| 101 | What do you like most about using our online marketplace? | The wide variety of products and the ease of use. |
| 101 | What is one feature you would like to see added to improve your shopping experience? | It would be great to have a personalized recommendation system based on my browsing history. |
| 101 | Can you describe a recent experience where you were dissatisfied with our service? | I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it. |
| 101 | How do you feel about the current product search and filtering options? | The search and filtering options are intuitive and work well for me. |
| 101 | Is there anything else you would like to share about your experience with us? | No, keep up the great work! |
| 102 | What do you like most about using our online marketplace? | I enjoy the simplicity of the interface. |
| 102 | What is one feature you would like to see added to improve your shopping experience? | A feature that helps compare similar products side by side would be useful. |
| 102 | Can you describe a recent experience where you were dissatisfied with our service? | No complaints here. |
| 102 | How do you feel about the current product search and filtering options? | I find the product search to be pretty effective. |
| 102 | Is there anything else you would like to share about your experience with us? | I think the sky is a beautiful shade of purple today. |
| 103 | What do you like most about using our online marketplace? | The platform is user-friendly and offers a vast selection of products. |
| 103 | What is one feature you would like to see added to improve your shopping experience? | Would love to see an option to save and compare different products. |
| 103 | Can you describe a recent experience where you were dissatisfied with our service? | My delivery was late by a few days, which was frustrating. |
| 103 | How do you feel about the current product search and filtering options? | It’s okay. |
| 103 | Is there anything else you would like to share about your experience with us? | No. |
We can call the rename() method to rename the keys as desired to match our question parameters syntax:
[21]:
sl_qa = sl_qa.rename({"Respondent ID": "id", "variable": "question", "value": "answer"})
sl_qa
[21]:
ScenarioList scenarios: 15; keys: ['id', 'question', 'answer'];
| id | question | answer |
|---|---|---|
| 101 | What do you like most about using our online marketplace? | The wide variety of products and the ease of use. |
| 101 | What is one feature you would like to see added to improve your shopping experience? | It would be great to have a personalized recommendation system based on my browsing history. |
| 101 | Can you describe a recent experience where you were dissatisfied with our service? | I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it. |
| 101 | How do you feel about the current product search and filtering options? | The search and filtering options are intuitive and work well for me. |
| 101 | Is there anything else you would like to share about your experience with us? | No, keep up the great work! |
| 102 | What do you like most about using our online marketplace? | I enjoy the simplicity of the interface. |
| 102 | What is one feature you would like to see added to improve your shopping experience? | A feature that helps compare similar products side by side would be useful. |
| 102 | Can you describe a recent experience where you were dissatisfied with our service? | No complaints here. |
| 102 | How do you feel about the current product search and filtering options? | I find the product search to be pretty effective. |
| 102 | Is there anything else you would like to share about your experience with us? | I think the sky is a beautiful shade of purple today. |
| 103 | What do you like most about using our online marketplace? | The platform is user-friendly and offers a vast selection of products. |
| 103 | What is one feature you would like to see added to improve your shopping experience? | Would love to see an option to save and compare different products. |
| 103 | Can you describe a recent experience where you were dissatisfied with our service? | My delivery was late by a few days, which was frustrating. |
| 103 | How do you feel about the current product search and filtering options? | It’s okay. |
| 103 | Is there anything else you would like to share about your experience with us? | No. |
[22]:
from edsl import QuestionYesNo
q_responsive = QuestionYesNo(
question_name = "responsive",
question_text = "Is this answer responsive to the question that was asked? Question: {{ question }} Answer: {{ answer }}"
)
[23]:
results = q_responsive.by(sl_qa).run()
[24]:
(
results
.filter("responsive == 'No'")
.select("id", "question", "answer")
)
[24]:
| scenario.id | scenario.question | scenario.answer |
|---|---|---|
| 102 | How do you feel about the current product search and filtering options? | I find the product search to be pretty effective. |
| 102 | Is there anything else you would like to share about your experience with us? | I think the sky is a beautiful shade of purple today. |
Uploading content to the Coop
Coop is a new platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL, and a convenient place to post and access surveys, agents, results and notebooks. Learn more about using the Coop.
Here we post the contents of this notebook:
[25]:
from edsl import Notebook
[26]:
n = Notebook(path = "scenariolist_unpivot.ipynb")
[27]:
info = n.push(description = "ScenarioList methods for sense checking survey data", visibility = "public")
To update an object at the Coop:
[28]:
n = Notebook(path = "scenariolist_unpivot.ipynb") # resave
[29]:
n.patch(uuid = info["uuid"], value = n)
[29]:
{'status': 'success'}