Skip to main content

​
Contents

Using a set of responses to a survey about online marketplaces as an example, we demonstrate EDSL methods for:
1
Evaluating survey questions (e.g., for clarity and improvements)
2
Analyzing each respondent’s set of answers (e.g., to summarize or identify sentiment, themes, etc.)
3
Reviewing each answer individually (e.g., to evaluate its relevance or usefulness)

​
Expected Parrot

We also show how to post EDSL questions, surveys, results and notebooks (like this one) to Expected Parrot: an integrated platform for creating and sharing LLM-based research.

​
How EDSL works

EDSL is a flexible library that can be used to perform a broad variety of research tasks. A typical workflow consists of the following steps:
  • Construct questions
  • Add data to the questions (e.g., for data labeling tasks)
  • Use an AI agent to answer the questions
  • Select a language model to generate the answers
  • Analyze results in a formatted dataset

​
Technical setup

Before running the code below please ensure that you have completed setup: Our Starter Tutorial provides examples of EDSL basic components.

​
Example data

Our example data is a CSV consisting of several questions and a few rows of responses. Here we store it at Expected Parrot and then re-import it. To post an object:
This returns details of the object we can use to retrieve it:

​
Creating questions about the data

There are many questions we might want to ask about the data, such as:
  • Does this survey question have any logical or syntactical problems? {{ *question* }}
  • What is the overall sentiment of this respondent’s answers? {{ *responses* }}
  • Is this answer responsive to the question that was asked? {{ *question* }} {{ *answer* }}

​
Question types

EDSL comes with many common question types that we can select from based on the form of the response that we want to get back from the model: multiple choice, checkbox, linear scale, free text, etc. Learn more about EDSL question types. Here we construct Question objects for the questions that we want to ask about the data, using {{ placeholders }} for the information that we will add to the questions in the steps that follow:

​
Adding survey data to the questions

Next we’ll add our data to our questions. This can be done efficiently by creating a ScenarioList representing the data. The individual Scenario objects in the list can be constructed in a variety of ways depending on the information that we want to include in a particular question. We start by calling the from_source() method to create a ScenarioList for the data in its original form. We can see that this generates a Scenario dictionary for each respondent’s set of answers with key/value pairs for the individual questions and answers:
ScenarioList scenarios: 3; keys: [‘Respondent ID’, ‘What do you like most about using our online marketplace?’, ‘What is one feature you would like to see added to improve your shopping experience?’, ‘Is there anything else you would like to share about your experience with us?’, ‘How do you feel about the current product search and filtering options?’, ‘Can you describe a recent experience where you were dissatisfied with our service?’];

​
Evaluating the questions

For our first question we want to create a Scenario for each survey question:
The survey questions are the parameters of the ScenarioList created above:
[‘Respondent ID’, ‘What do you like most about using our online marketplace?’, ‘What is one feature you would like to see added to improve your shopping experience?’, ‘Is there anything else you would like to share about your experience with us?’, ‘How do you feel about the current product search and filtering options?’, ‘Can you describe a recent experience where you were dissatisfied with our service?’] We can pass them to the from_source() method to create a new ScenarioList, specifying that the key for each Scenario will be question in order to match the parameter of our logic question:
ScenarioList scenarios: 6; keys: [‘question’]; We select a model to use, and then add the scenarios to the survey when we run it:
This generates a dataset of Results that we can access with built-in methods for analysis:
Learn more about working with results.

​
Evaluating respondents’ collective answers

Next we can create a ScenarioList for each respondent’s answers to use with our question about sentiment:
ScenarioList scenarios: 3; keys: [‘Respondent ID’, ‘What do you like most about using our online marketplace?’, ‘What is one feature you would like to see added to improve your shopping experience?’, ‘Is there anything else you would like to share about your experience with us?’, ‘How do you feel about the current product search and filtering options?’, ‘Can you describe a recent experience where you were dissatisfied with our service?’];
ScenarioList scenarios: 3; keys: [‘responses’]; Next we add these scenarios to our sentiment question (and any others we want to add) and run it:

​
Evaluating individual answers

Next we create a ScenarioList for each individual question and answer to use with our question about the responsiveness of each answer. We can use the unpivot() method to expand the scenarios by desired identifiers (e.g., respondent ID):
ScenarioList scenarios: 15; keys: [‘variable’, ‘value’, ‘Respondent ID’]; We can call the rename() method to rename the keys as desired to match our question parameters syntax:
ScenarioList scenarios: 15; keys: [‘question’, ‘id’, ‘answer’];

​
Uploading content to Expected Parrot

Expected Parrot is a new platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL, and a convenient place to post and access surveys, agents, results and notebooks. Learn more about posting content. Here we post the contents of this notebook: