Skip to main content
This workflow consists of the following steps:
1
Import data into EDSL
2
Create questions about the data
3
Design an AI agent to answer the questions
4
Select a language model to generate responses
5
Analyze results as a formatted dataset
This workflow can be visualized as follows:
general_survey.png

​
Conducting agent-specific tasks

We can add a layer of complexity to this generalized flow by creating different AI agents for subsets of the data to be reviewed. For example, we can design agents with specific “expertise” to review only the data that is relevant to that expertise. This can be useful if our data is sorted (or sortable) in some way that is important to our task. We can also use EDSL to prompt a language model to sort the data as needed. This modified workflow can be visualized as follows:
general_survey.png

​
Example task: Evaluating job posts

Using a dataset of job posts as an example, in the steps below we create AI agents with expertise in the relevant job categories and then prompt them to evaluate relevant job posts in a variety of ways. The steps are:
1
Import a dataset of job categories and job posts.
2
Construct questions about the job posts and combine them in a survey.
3
Design AI agents with job category expertise.
4
Administer the survey to each agent with job posts for the relevant category.
5
Inspect the results using built-in methods for analysis.

​
Validating results

EDSL also provides methods for launching your surveys with human respondents to compare and validate LLM results. Learn more about these features.

​
Technical setup

Before running the code below please see instructions on getting started. An introductory data labeling example notebook is also available at our documentation page.

​
Import the tools

We start by selecting question types and survey components that we will use. Please see the EDSL Docs for examples of all question types and details on these basic components.

​
Import data

Next we import a dataset for review, using Scenario objects to represent the individual data that will be added to each of our data labeling questions. EDSL allows us to create data or import it from other sources (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). For purposes of demonstration, we create a dataframe, post it to Expected Parrot with the FileStore module and then retrieve it. Note that FileStore works with many file types and automatically infers the file type. Learn more about posting and retrieving files.
Here we post the file to Expected Parrot and get the information for the object:
Next we retrieve the data file and use it to create scenarios (replace this code with the UUID of any file you want to use):
ScenarioList scenarios: 20; keys: [”, ‘job_title’, ‘job_category’, ‘job_post’];

​
Construct questions about the data

Next we construct questions to ask about the job posts, selecting question types based on the form of the response that we want to get back from the language model (multiple choice, linear scale, free text, numerical, etc.–see examples of all question types). We include a {{ placeholder }} for the scenario keys in order to parameterize each question with each job post and category when we run the survey:

​
Combining questions into a Survey

Next we combine our questions into a survey that will be administered to the AI agents. By default, the questions will be administered asynchronously. If desired, we can also specify survey rules (skip/stop logic) and within-survey memories of prior questions and responses. See the EDSL Docs for details on methods for applying survey rules.

​
Creating personas for Agents

Next we draft personas for AI agents that will answer the questions. For each job category we construct an AI agent that is an expert in the category. Agents are constructed by passing a dictionary of traits to an Agent object. Learn more about designing AI agents to answer surveys. To get the set of job categories from the scenarios:
Next we use them to create an agent for each job category:

​
Selecting language models

EDSL works with many popular language models that we can select to generate the agents’ responses to the survey. Information about current models is available here. To check a list of service providers:
Here we specify a model to use to generate responses (if we do not specify a model, gpt-5.2 is used by default):

​
Running the survey

We administer a survey by appending the components with the by() method and then calling run() method. In the simplest case where we want a single agent or list of agents to answer all questions with the same scenarios, this takes the following form:
Here we have individual agents answer the questions only for category-specific job posts, and then combine the results:

​
Accessing Results

In the previous step we created Results for individual agents’ responses and combined them. Next we show how to inspect and analyze results with built-in methods. We can identify the column names to select the fields that we want to inspect:
We can select individual fields in a variety of ways:
We can apply some labels to our table for readability. Note that each question field also automatically includes a <question>_comment field for any commentary by the LLM on the question:
We can also access results as a SQL table (called self) with the .sql() method, and optionally removing the column name prefixes ‘agent’, ‘model’, ‘prompt’, etc.:

​
Posting content at Expected Parrot

We can post any EDSL objects to Expected Parrot, including this notebook:
We can also post this Notebook:
Learn more about the platform to conduct LLM-based research.