Skip to main content
EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please see instructions on getting started, and tips and tutorials at our documentation page.

​
Create questions

We start by creating questions about a set of course evaluations for an agent to answer. EDSL comes with a variety of question types that we can choose from based on the form of the response that we want to get back from a model (multiple choice, linear scale, checkbox, free text, numerical, dictionary, etc.). We can use a {{ placeholder }} in the question texts to parameterize them with each evaluation. This allows us to create different “scenarios” of the questions that we can administer at once. We start by importing some question types and composing questions in the relevant templates (see examples of all types in the docs):

​
Construct a survey

Next we combine our questions into a survey. This allows us to administer the questions asynchronously (by default), or according to any desired survey logic or rules that we want to add, such as skip/stop rules or giving an agent “memories” of other questions in the survey. Here we create a simple asynchronous survey by passing the list of questions to a Survey object:

​
Add data to the questions

Next we identify the data to be analyzed and create Scenario objects for it. This allows us to efficiently readminister questions with different data. We can create data using EDSL or import it from other sources (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). Here we use some mock evaluations for an Econ 101 course stored as a list of texts:

​
Design AI agents

Next we design agents with relevant traits and personas for a language model to use in answering the questions. This can be useful if we want to compare responses among different audiences. We do this by passing dictionaries of traits to Agent objects. We can also choose whether to give an agent additional instructions for answering the survey (independent of individual question texts). Please see documentation for more details and example code for creating agents to use with surveys. Here we create a persona for the professor of the course and pass it some special instructions:

​
Select language models

EDSL works with many popular language models that we can use to generate responses for our survey. A current list of available models, performance and pricing is available here. We select models to use with a survey by creating Model objects for them (if no model is specified, gpt-5.2 is currently used by default):

​
Run the survey

Next we add the scenarios and agent to the survey, and then run it with the specified model. This will generate a dataset of Results that we can store and begin analyzing:

​
Analyzing results

EDSL comes with built-in methods for analyzing results in data tables, dataframes, SQL queries and other formats. We can print a list of all the components that can be accessed:
Here we select just the responses to the questions and display them in a table:
We can do a quick tally of the sentiments:
We can also transform the results into a dataframe:
5 rows × 69 columns We can also use pandas methods by first converting:

​
Use responses to construct new questions

We can use the responses to our initial questions to construct more questions about the texts. For example, we can prompt a model to condense the individual lists of themes and areas for improvement into short lists, and then use the new lists to quantify the topics across the set of evaluations. Here we take the lists of themes in each evaluation, flatten them into a (duplicative) list, and then create a new question prompting a model to condense it for us:
Next we construct a question to condense the list into a new list:
Now we run the question and select the new list. Note that we can choose whether we want to use the agent for this question by not adding it to the question when we run it:
Now we can create a question to identify all the themes in the list that appear in each evaluation (our new list becomes the list of answer options):
Here we run the question and show a table listing all the themes for each evaluation in the results:
ScenarioList scenarios: 22; keys: [‘evaluation’, ‘theme’];
We can do the same thing with the areas of improvement:

​
Other examples

Please check out the EDSL Docs for examples of other methods and templates for use cases!

​
Posting to Expected Parrot

Expected Parrot is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about creating an account and the platform. We can post any EDSL object to Expected Parrot by calling the push() method on it, including this notebook:
To upate an object at Expected Parrot: