Skip to main content
Before running the code below please see instructions on getting started using EDSL.

​
Overview

Using a dataset of mock customer service tickets as an example, we demonstrate how to:
1

Import data into EDSL as scenarios
2

Create questions about the data
3

Design an AI agent to answer the questions
4

Select a language model to generate responses
5

Analyze results as a formatted dataset
This workflow can be visualized as follows:
general_survey.png

​
Selecting data for review

First we identify some data for review. Data can be created in EDSL or imported from other sources (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). For purposes of demonstration we import a set of hypothetical customer tickets for a transportation app: 
tickets = [
    "I just realized I left my phone in the car on my last ride. Can you help me get it back?",
    "I'm unhappy with my recent experience. The driver was very rude and unprofessional.",
    "I was charged more than the estimated fare for my trip yesterday. Can you explain why?",
    "The car seat provided was not properly installed, and I felt my child was at risk. Please ensure driver training.",
    "My driver took a longer route than necessary, resulting in a higher fare. I request a fare adjustment.",
    "I had a great experience with my driver today! Very friendly and efficient service.",
    "I'm concerned about the vehicle's cleanliness. It was not up to the standard I expect.",
    "The app keeps crashing every time I try to book a ride. Please fix this issue.",
    "My driver was exceptional - safe driving, polite, and the car was spotless. Kudos!",
    "I felt unsafe during my ride due to the driver's erratic behavior. This needs to be addressed immediately.",
    "The driver refused to follow my preferred route, which is shorter. I'm not satisfied with the service.",
    "Impressed with the quick response to my ride request and the driver's professionalism.",
    "I was charged for a ride I never took. Please refund me as soon as possible.",
    "The promo code I tried to use didn't work. Can you assist with this?",
    "There was a suspicious smell in the car, and I'm worried about hygiene standards.",
    "My driver was very considerate, especially helping me with my luggage. Appreciate the great service!",
    "The app's GPS seems inaccurate. It directed the driver to the wrong pick-up location.",
    "I want to compliment my driver's excellent navigation and time management during rush hour.",
    "The vehicle didn't match the description in the app. It was confusing and concerning.",
    "I faced an issue with payment processing after my last ride. Can you look into this?",
]

​
Constructing questions about the data

Next we create some questions about the data. EDSL provides a variety of question types that we can choose from based on the form of the response that we want to get back from the model (multiple choice, free text, checkbox, linear scale, etc.). Learn more about question types.
Note:Note that we use a {{ placeholder }} in each question text in order to parameterize the questions with the individual ticket contents in the next step:
from edsl import (
    QuestionMultipleChoice,
    QuestionCheckBox,
    QuestionFreeText,
    QuestionYesNo,
    QuestionLinearScale,
)

q_issues = QuestionCheckBox(
    question_name="issues",
    question_text="Check all of the issues mentioned in this ticket: {`{ scenario.ticket }`}",
    question_options=[
        "safety",
        "cleanliness",
        "driver performance",
        "GPS/route",
        "lost item",
        "other",
    ],
)

q_primary_issue = QuestionFreeText(
    question_name="primary_issue",
    question_text="What is the primary issue in this ticket? Ticket: {`{ scenario.ticket }}",
)

q_accident = QuestionMultipleChoice(
    question_name="accident",
    question_text="If the primary issue in this ticket is safety, was there an accident where someone was hurt? Ticket: {`{ scenario.ticket }}",
    question_options=["Yes", "No", "Not applicable"],
)

q_sentiment = QuestionMultipleChoice(
    question_name="sentiment",
    question_text="What is the sentiment of this ticket? Ticket: {`{ scenario.ticket }}",
    question_options=[
        "Very positive",
        "Somewhat positive",
        "Neutral",
        "Somewhat negative",
        "Very negative",
    ],
)

q_refund = QuestionYesNo(
    question_name="refund",
    question_text="Does the customer ask for a refund in this ticket? Ticket: {`{ scenario.ticket }}",
)

q_priority = QuestionLinearScale(
    question_name="priority",
    question_text="On a scale from 0 to 5, what is the priority level of this ticket? Ticket: {`{ scenario.ticket }}",
    question_options=[0, 1, 2, 3, 4, 5],
    option_labels={0: "Lowest", 5: "Highest"},
)

​
Building a survey

We combine the questions into a survey in order to administer them together: from edsl import Survey survey = Survey( questions=[ q_issues, q_primary_issue, q_accident, q_sentiment, q_refund, q_priority, ] ) Survey questions are administered asynchronously by default. Learn more about adding conditional logic and memory to your survey. Here we inspect them: 
survey

Survey # questions: 6; question_name list: ['issues', 'primary_issue', 'accident', 'sentiment', 'refund', 'priority'];
question_textquestion_optionsquestion_typequestion_nameoption_labels
0Check all of the issues mentioned in this ticket: [‘safety’, ‘cleanliness’, ‘driver performance’, ‘GPS/route’, ‘lost item’, ‘other’]checkboxissuesnan
1What is the primary issue in this ticket? Ticket: nanfree_textprimary_issuenan
2If the primary issue in this ticket is safety, was there an accident where someone was hurt? Ticket: [‘Yes’, ‘No’, ‘Not applicable’]multiple_choiceaccidentnan
3What is the sentiment of this ticket? Ticket: [‘Very positive’, ‘Somewhat positive’, ‘Neutral’, ‘Somewhat negative’, ‘Very negative’]multiple_choicesentimentnan
4Does the customer ask for a refund in this ticket? Ticket: [‘No’, ‘Yes’]yes_norefundnan
5On a scale from 0 to 5, what is the priority level of this ticket? Ticket: [0, 1, 2, 3, 4, 5]linear_scalepriority{0: 'Lowest', 5: 'Highest'}

​
Designing AI agents

A key feature of EDSL is the ability to create personas for AI agents that the language models are prompted to use in generating responses to the questions. This is done by passing a dictionary of traits to Agent objects: 
from edsl import Agent

agent = Agent(
    traits={
        "persona": "You are an expert customer service agent.",
        "years_experience": 15,
    }
)
agent

Agent
keyvalue
0traits:personaYou are an expert customer service agent.
1traits:years_experience15

​
Selecting language models

EDSL allows us to select the language models to use in generating results. See the model pricing page <>__ for pricing and performance information for available models. Here we select gpt-4o (if no model is specified, the default model is used – run Model() to verify the current default model): 
from edsl import Model

model = Model("gpt-4o", service_name = "openai")
model

LanguageModel
keyvalue
0modelgpt-4o
1parameters:temperature0.500000
2parameters:max_tokens1000
3parameters:top_p1
4parameters:frequency_penalty0
5parameters:presence_penalty0
6parameters:logprobsFalse
7parameters:top_logprobs3
8inference_serviceopenai

​
Adding data to the questions

We add the contents of each ticket into each question as an independent “scenario” for review. This allows us to create versions of the questions for each job post and deliver them to the model all at once: 
from edsl import ScenarioList

scenarios = ScenarioList.from_list("ticket", tickets)
scenarios

ScenarioList scenarios: 20; keys: ['ticket'];
ticket
0I just realized I left my phone in the car on my last ride. Can you help me get it back?
1I’m unhappy with my recent experience. The driver was very rude and unprofessional.
2I was charged more than the estimated fare for my trip yesterday. Can you explain why?
3The car seat provided was not properly installed, and I felt my child was at risk. Please ensure driver training.
4My driver took a longer route than necessary, resulting in a higher fare. I request a fare adjustment.
5I had a great experience with my driver today! Very friendly and efficient service.
6I’m concerned about the vehicle’s cleanliness. It was not up to the standard I expect.
7The app keeps crashing every time I try to book a ride. Please fix this issue.
8My driver was exceptional - safe driving, polite, and the car was spotless. Kudos!
9I felt unsafe during my ride due to the driver’s erratic behavior. This needs to be addressed immediately.
10The driver refused to follow my preferred route, which is shorter. I’m not satisfied with the service.
11Impressed with the quick response to my ride request and the driver’s professionalism.
12I was charged for a ride I never took. Please refund me as soon as possible.
13The promo code I tried to use didn’t work. Can you assist with this?
14There was a suspicious smell in the car, and I’m worried about hygiene standards.
15My driver was very considerate, especially helping me with my luggage. Appreciate the great service!
16The app’s GPS seems inaccurate. It directed the driver to the wrong pick-up location.
17I want to compliment my driver’s excellent navigation and time management during rush hour.
18The vehicle didn’t match the description in the app. It was confusing and concerning.
19I faced an issue with payment processing after my last ride. Can you look into this?

​
Running the survey

We run the survey by adding the scenarios, agent and model with the by() method and then calling the run() method: 
results = survey.by(scenarios).by(agent).by(model).run()
This generates a formatted dataset of Results that includes information about all the components, including the prompts and responses. We can see a list of all the components: 
results.columns

​
Analyzing results

EDSL comes with built-in methods for analyzing results. Here we filter, sort, select and print components in a table: 
(
    results
    .filter("priority in [4, 5]")
    .sort_by("issues", "sentiment")
    .select("ticket", "issues", "primary_issue", "accident", "sentiment", "refund", "priority")
)
We can apply some lables to our table: 
(
    results.select(
        "ticket",
        "issues",
        "primary_issue",
        "accident",
        "sentiment",
        "refund",
        "priority",
    ).print(
        pretty_labels={
            "scenario.ticket": "Ticket",
            "answer.issues": "Issues",
            "answer.primary_issue": "Primary issue",
            "answer.accident": "Accident",
            "answer.sentiment": "Sentiment",
            "answer.refund": "Refund request",
            "answer.priority": "Priority",
        }
    )
)
EDSL also comes with methods for accessing results as a dataframe or SQL table: 
df = (
    results
    .select(
        "issues",
        "primary_issue",
        "accident",
        "sentiment",
        "refund",
        "priority"
    )
    .to_pandas(remove_prefix=True)
)
df
We can also access results as a SQL table: 
results.sql("""
select ticket, issues, primary_issue, accident, sentiment, refund, priority
from self
order by priority desc
""")
To export results to a CSV file: 
results.to_csv("data_labeling_example.csv")

​
Posting content to the Coop

We can post any EDSL objects to Coop, and share them publicly, privately or unlisted (by default). The above results were automatically posted to Coop; we can also post them manually: 
# results.push(
#     description = "Customer service tickets data labeling example",
#     alias = "customer-service-tickets-results-example",
#     visibility="public"
# )

# survey.push(
#     description = "Customer service tickets data labeling example survey",
#     alias = "customer-service-tickets-survey-example",
#     visibility="public"
# )
To post this notebook: 
# from edsl import Notebook

# nb = Notebook("data_labeling_example.ipynb")

# nb.push(
#     description = "Data labeling example",
#     alias = "data-labeling-example-notebook",
#     visibility = "public"
# )
To update an object at Coop: 
from edsl import Notebook


nb = Notebook("data_labeling_example.ipynb") # resave

nb.patch("https://www.expectedparrot.com/content/RobinHorton/data-labeling-example-notebook", value = nb)