NPS survey in EDSL

This notebook provides sample EDSL code for simulating a Net Promoter Score (NPS) survey with AI agents and large language models. In the steps below we show how to construct an EDSL survey, create personas for AI agents to answer the questions, and then administer the survey to them. We also demonstrate some built-in methods for inspecting and analyzing the dataset of results that is generated when an EDSL survey is run.

The following questions are used in the sample survey:

On a scale from 0-10, how likely are you to recommend our company to a friend or colleague? (0=Not at all likely, 10=Very likely) Please tell us why you gave a rating.

How satisfied are you with the following experience with our company? Product quality Customer support Purchasing experience

Is there anything specific that our company can do to improve your experience?

Technical setup

Before running the code below, ensure that you have (1) installed the EDSL library and (2) created a Coop account to activate remote inference or stored your own API keys for language models that you want to use with EDSL. Please also see our tutorials and documentation page on getting started using the EDSL library.

Constructing questions

We start by selecting appropriate question types for the above questions. EDSL comes with a variety of common question types that we can choose from based on the form of the response that we want to get back from the model. The first quesiton is linear scale; we import the class type and then construct a question in the relevant template:

from edsl import QuestionLinearScale

q_recommend = QuestionLinearScale(
    question_name = "recommend",
    question_text = "On a scale from 0-10, how likely are you to recommend our company to a friend or colleague?",
    question_options = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    option_labels = {0:"Not at all likely", 10:"Very likely"}
)

Each question type other than free text automatically includes a “comment” field for the model to provide commentary on its response to the main question. When we run the survey, we can check that it has effectively captured the follow-on question from above–Please tell us why you gave a rating–and modify or add questions as needed.

For the next question, we use a {{ placeholder }} for an “experience” that we will insert when repeating the base question:

from edsl import QuestionMultipleChoice

q_satisfied = QuestionMultipleChoice(
    question_name = "satisfied",
    question_text = "How satisfied are you with the following experience with our company: {{ experience }}",
    question_options = [
        "Extremely satisfied",
        "Moderately satisfied",
        "Neither satisfied nor dissatisfied",
        "Moderately dissatisfied",
        "Extremely dissatisfied"
    ]
)

The third question is a simple free text question that we can choose whether to administer once or individually for each “experience” question. In the steps that follow we show how to apply survey logic to achieve this effect:

from edsl import QuestionFreeText

q_improve = QuestionFreeText(
    question_name = "improve",
    question_text = "Is there anything specific that our company can do to improve your experience?"
)

Creating variants of questions with scenarios

Next we want to create a version of the “satisfied” question for each “experience”. This can be done with Scenario objects–dictionaries of key/value pairs representing the content to be added to questions. Scenarios can be automatically generated from a variety of data sources (PDFs, CSVs, images, tables, etc.). Here we have import a simple list:

from edsl import ScenarioList, Scenario

experiences = ["Product quality", "Customer support", "Purchasing experience"]

s = ScenarioList(
    Scenario({"experience":e}) for e in experiences
)

We could also use a specific method for creating scenarios from a list:

s = ScenarioList.from_list("experience", experiences)

We can check the scenarios that have been created:

s

ScenarioList scenarios: 3; keys: ['experience'];

	experience
0	Product quality
1	Customer support
2	Purchasing experience

To create the question variants, we pass the scenario list to the question loop() method, which returns a list of new questions. We can see that each question has a new unique name and a question text with the placeholder replaced with an experience:

satisfied_questions = q_satisfied.loop(s)
satisfied_questions

[Question('multiple_choice', question_name = """satisfied_0""", question_text = """How satisfied are you with the following experience with our company: Product quality""", question_options = ['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied']),
 Question('multiple_choice', question_name = """satisfied_1""", question_text = """How satisfied are you with the following experience with our company: Customer support""", question_options = ['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied']),
 Question('multiple_choice', question_name = """satisfied_2""", question_text = """How satisfied are you with the following experience with our company: Purchasing experience""", question_options = ['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied'])]

We can also use the loop() method to create copies of the “improve” question in order to present it as a follow-up question to each of the “satisfied” questions that have been parameterized with experiences. Here, we’re simply duplicating the base question without a scenario {{ placeholder }} because we will instead add a “memory” of the relevant “satisfied” question when administering each copy of it:

improve_questions = q_improve.loop(s)
improve_questions

[Question('free_text', question_name = """improve_0""", question_text = """Is there anything specific that our company can do to improve your experience?"""),
 Question('free_text', question_name = """improve_1""", question_text = """Is there anything specific that our company can do to improve your experience?"""),
 Question('free_text', question_name = """improve_2""", question_text = """Is there anything specific that our company can do to improve your experience?""")]

Creating a survey

Next we pass a list of all the questions to a Survey in order to administer them together:

questions = [q_recommend] + satisfied_questions + improve_questions

from edsl import Survey

survey = Survey(questions)

Adding survey logic

In the next step we add logic to the survey specifying that each “improve” question should include a “memory” of a “satisfied” question (the question and answer that was provided):

for i in range(len(s)):
    survey = survey.add_targeted_memory(f"improve_{i}", f"satisfied_{i}")

We can inspect the survey details:

survey

Survey # questions: 7; question_name list: ['recommend', 'satisfied_0', 'satisfied_1', 'satisfied_2', 'improve_0', 'improve_1', 'improve_2'];

	question_name	question_text	option_labels	question_type	question_options
0	recommend	On a scale from 0-10, how likely are you to recommend our company to a friend or colleague?	{0: 'Not at all likely', 10: 'Very likely'}	linear_scale	[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
1	satisfied_0	How satisfied are you with the following experience with our company: Product quality	nan	multiple_choice	['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied']
2	satisfied_1	How satisfied are you with the following experience with our company: Customer support	nan	multiple_choice	['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied']
3	satisfied_2	How satisfied are you with the following experience with our company: Purchasing experience	nan	multiple_choice	['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied']
4	improve_0	Is there anything specific that our company can do to improve your experience?	nan	free_text	nan
5	improve_1	Is there anything specific that our company can do to improve your experience?	nan	free_text	nan
6	improve_2	Is there anything specific that our company can do to improve your experience?	nan	free_text	nan

AI agent personas

EDSL comes with a variety of methods for designing AI agents to answer surveys. An Agent is constructed by passing a dictionary of relevant traits with optional additional instructions for the language model to reference in generating responses for the agent. Agents can be constructed from a variety of data sources, including existing survey data (e.g., a dataset of responses that were provided to some other questions). We can also use an EDSL question to draft some personas for agents. Here, we ask for a list of them:

from edsl import QuestionList

q_personas = QuestionList(
    question_name = "personas",
    question_text = "Draft 5 personas for diverse customers of landscaping business with varying satisfaction levels."
)

We can run this question alone and extract the response list (more on working with results below):

personas = q_personas.run().select("personas").to_list()[0]
personas

▼ Job Status (2024-12-28 15:44:51)

Job UUID	e11fda90-5a22-4af9-bdfd-8b9235594184
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/e11fda90-5a22-4af9-bdfd-8b9235594184
Error Report URL	None
Results UUID	ccf837e8-b1d3-4ae1-b783-806a293247e4
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/ccf837e8-b1d3-4ae1-b783-806a293247e4

['John, a retired veteran who loves his garden and is highly satisfied with the personalized service',
 'Emily, a busy professional who is moderately satisfied but wishes for more eco-friendly options',
 'Raj, a young tech entrepreneur who is dissatisfied due to inconsistent appointment scheduling',
 'Maria, a single mother who is very satisfied with the affordable pricing and flexible payment plans',
 'Grace, an elderly woman who is dissatisfied because of slow response times to her queries']

Next we pass the personas to create a set of agents:

from edsl import AgentList, Agent

a = AgentList(
    Agent(traits = {"persona":p}) for p in personas
)

Selecting language models

EDSL works with many popular large language models that we can select to use with a survey. To see a list of available models:

from edsl import Model

# Model.available()

To select a model to use with a survey we pass a model name to a Model:

m = Model("gemini-1.5-flash")

If we want to compare responses for several models, we can use a ModelList instead:

from edsl import ModelList

m = ModelList(
    Model(model) for model in ["gemini-1.5-flash", "gpt-4o"]
)

Note: If no model is specified when running a survey, the default model GPT-4o is used (as above when we generated personas).

Running a survey

We administer the survey by adding the agents and models with the by() method and then calling the run() method:

results = survey.by(a).by(m).run()

▼ Job Status (2024-12-28 15:47:52)

Job UUID	99b2b457-d27f-4ff5-9b1c-aea15eea2167
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/99b2b457-d27f-4ff5-9b1c-aea15eea2167
Error Report URL	None
Results UUID	7777e1b5-4832-47bb-b69e-dd98440fd05f
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/7777e1b5-4832-47bb-b69e-dd98440fd05f

This generates a dataset of Results that includes a response for each agent/model that was used. We can access the results with built-in methods for analysis. To see a list of all the components of the results:

results.columns

	0
0	agent.agent_instruction
1	agent.agent_name
2	agent.persona
3	answer.improve_0
4	answer.improve_1
5	answer.improve_2
6	answer.recommend
7	answer.satisfied_0
8	answer.satisfied_1
9	answer.satisfied_2
10	comment.improve_0_comment
11	comment.improve_1_comment
12	comment.improve_2_comment
13	comment.recommend_comment
14	comment.satisfied_0_comment
15	comment.satisfied_1_comment
16	comment.satisfied_2_comment
17	generated_tokens.improve_0_generated_tokens
18	generated_tokens.improve_1_generated_tokens
19	generated_tokens.improve_2_generated_tokens
20	generated_tokens.recommend_generated_tokens
21	generated_tokens.satisfied_0_generated_tokens
22	generated_tokens.satisfied_1_generated_tokens
23	generated_tokens.satisfied_2_generated_tokens
24	iteration.iteration
25	model.frequency_penalty
26	model.logprobs
27	model.maxOutputTokens
28	model.max_tokens
29	model.model
30	model.presence_penalty
31	model.stopSequences
32	model.temperature
33	model.topK
34	model.topP
35	model.top_logprobs
36	model.top_p
37	prompt.improve_0_system_prompt
38	prompt.improve_0_user_prompt
39	prompt.improve_1_system_prompt
40	prompt.improve_1_user_prompt
41	prompt.improve_2_system_prompt
42	prompt.improve_2_user_prompt
43	prompt.recommend_system_prompt
44	prompt.recommend_user_prompt
45	prompt.satisfied_0_system_prompt
46	prompt.satisfied_0_user_prompt
47	prompt.satisfied_1_system_prompt
48	prompt.satisfied_1_user_prompt
49	prompt.satisfied_2_system_prompt
50	prompt.satisfied_2_user_prompt
51	question_options.improve_0_question_options
52	question_options.improve_1_question_options
53	question_options.improve_2_question_options
54	question_options.recommend_question_options
55	question_options.satisfied_0_question_options
56	question_options.satisfied_1_question_options
57	question_options.satisfied_2_question_options
58	question_text.improve_0_question_text
59	question_text.improve_1_question_text
60	question_text.improve_2_question_text
61	question_text.recommend_question_text
62	question_text.satisfied_0_question_text
63	question_text.satisfied_1_question_text
64	question_text.satisfied_2_question_text
65	question_type.improve_0_question_type
66	question_type.improve_1_question_type
67	question_type.improve_2_question_type
68	question_type.recommend_question_type
69	question_type.satisfied_0_question_type
70	question_type.satisfied_1_question_type
71	question_type.satisfied_2_question_type
72	raw_model_response.improve_0_cost
73	raw_model_response.improve_0_one_usd_buys
74	raw_model_response.improve_0_raw_model_response
75	raw_model_response.improve_1_cost
76	raw_model_response.improve_1_one_usd_buys
77	raw_model_response.improve_1_raw_model_response
78	raw_model_response.improve_2_cost
79	raw_model_response.improve_2_one_usd_buys
80	raw_model_response.improve_2_raw_model_response
81	raw_model_response.recommend_cost
82	raw_model_response.recommend_one_usd_buys
83	raw_model_response.recommend_raw_model_response
84	raw_model_response.satisfied_0_cost
85	raw_model_response.satisfied_0_one_usd_buys
86	raw_model_response.satisfied_0_raw_model_response
87	raw_model_response.satisfied_1_cost
88	raw_model_response.satisfied_1_one_usd_buys
89	raw_model_response.satisfied_1_raw_model_response
90	raw_model_response.satisfied_2_cost
91	raw_model_response.satisfied_2_one_usd_buys
92	raw_model_response.satisfied_2_raw_model_response

For example, we can filter, sort and display columns of results in a table:

(
    results
    .filter("model.model == 'gemini-1.5-flash'")
    .sort_by("recommend", reverse=True)
    .select("model","persona","recommend", "recommend_comment")
)

	model.model	agent.persona	answer.recommend	comment.recommend_comment
0	gemini-1.5-flash	John, a retired veteran who loves his garden and is highly satisfied with the personalized service	10	Honestly, I've been so pleased with the personalized attention I've received. It's a breath of fresh air compared to the usual impersonal corporate stuff. Wouldn't hesitate to recommend you to anyone.
1	gemini-1.5-flash	Maria, a single mother who is very satisfied with the affordable pricing and flexible payment plans	10	Honestly, I'd give you guys a 10 out of 10! As a single mom, being able to afford things and having flexible payment options is a lifesaver. I really appreciate it.
2	gemini-1.5-flash	Emily, a busy professional who is moderately satisfied but wishes for more eco-friendly options	7	Honestly, I've been pretty happy with your services so far. They've gotten the job done, but I do wish there were more sustainable options available. That's my main hesitation.
3	gemini-1.5-flash	Raj, a young tech entrepreneur who is dissatisfied due to inconsistent appointment scheduling	3	Honestly, the constant rescheduling is driving me crazy. I *want* to like your company – the product itself is great – but the unreliable appointments are a huge negative. It's just too much hassle.
4	gemini-1.5-flash	Grace, an elderly woman who is dissatisfied because of slow response times to her queries	2	Honestly, I'm finding it terribly difficult to even get a simple question answered. The wait times are just inexcusable. I'm not sure I'd want to put a friend through that.

(
    results
    .filter("model.model == 'gemini-1.5-flash'")
    .sort_by("satisfied_0")
    .select("satisfied_0", "satisfied_0_comment", "improve_0")
)

	answer.satisfied_0	comment.satisfied_0_comment	answer.improve_0
0	Extremely satisfied	My tomatoes have never been so plump and juicy! The quality of your gardening supplies is top-notch, just like the personalized service I received.	Well, honestly, I'm just about as happy as a clam in my garden! The product quality was top-notch, no complaints there at all. You guys really went above and beyond. If I had to nitpick – and I'm really stretching here – maybe a little more detail in the initial instructions could have saved me a phone call. But that's a minor thing. The personalized service I received made all the difference. I felt like a valued customer, not just another number. That's what keeps me coming back, you know? So, keep doing what you're doing!
1	Extremely satisfied	Honestly, for the price, the quality is amazing! I'm a single mom, so budgeting is everything, and I've never felt like I'm sacrificing quality for affordability.	Honestly? Keeping things affordable and flexible is the biggest thing for me. As a single mom, every penny counts. So, if you could keep those payment plans as straightforward and easy to understand as they are now, that would be amazing. I've never had any trouble, and that peace of mind is worth a lot. Beyond that, I'm really happy with everything!
2	Moderately dissatisfied	Honestly, the product itself is okay, but the constant rescheduling of appointments to discuss features and address bugs is a huge headache. It impacts my ability to plan, and that frustration bleeds over into how I feel about the product as a whole.	Ugh, look, the product itself is okay, I'll give you that. It's not *bad*, but it's not amazing either. The *real* problem is scheduling. Seriously, it's a nightmare. One minute I'm getting an email saying my appointment's confirmed, the next it's been rescheduled, then cancelled, then rescheduled again... it's a chaotic mess. I'm constantly having to rearrange my whole day because of your scheduling inconsistencies. If you could just get that sorted – make the scheduling process reliable and actually *stick* to the appointments – that would drastically improve my experience. That's way more important to me than minor tweaks to the product itself.
3	Moderately dissatisfied	Honestly, the product itself is alright, but it took them *forever* to even get it to me. That's what really soured the experience.	Oh, honey, "moderately dissatisfied" is putting it mildly when it comes to the product quality. Frankly, I've had to wait *ages* for even the simplest of issues to be addressed. What I really need is faster response times. I'm not asking for miracles, just a bit of promptness. If a problem arises, I shouldn't have to wait weeks, or even days, to get a response. A simple acknowledgment that my query has been received would be a start! Then, a reasonable timeframe for a solution, not this endless waiting game. Is that too much to ask?
4	Moderately satisfied	The product quality is okay; it does what it's supposed to. But I wish there were more sustainable options available. I'm always looking for ways to reduce my environmental impact, and that's a factor I consider when making purchases.	Honestly? I'm pretty happy with the product quality, but I'd love to see you guys offer more sustainable packaging options. It's something I'm increasingly thinking about, and it would really make a difference for me. Little things, like less plastic or recycled materials, would go a long way. I know it's probably a bigger undertaking, but it's something I'd really appreciate.

(
    results
    .filter("model.model == 'gemini-1.5-flash'")
    .select("satisfied_1", "satisfied_1_comment", "improve_1")
    .print(pretty_labels = {
        "answer.satisfied_1": "Customer service: satisfaction",
        "comment.satisfied_1_comment": "Customer service: comment",
        "answer.improve_1": "Customer service: improvements"
    })
)

	Customer service: satisfaction	Customer service: comment	Customer service: improvements
0	Extremely satisfied	Honestly, I've been a customer for years, and the personalized service I've received, especially from customer support, has been top-notch. They always go the extra mile, and it's a breath of fresh air in this day and age. Makes a fella feel appreciated, you know?	Well, now that's a thoughtful question. Honestly, I've been so pleased with the personalized attention I've received. It's a breath of fresh air, you know? In this day and age, it feels like you're dealing with robots half the time. So, to be perfectly frank, I can't think of anything specific you need to *improve*.
1	Moderately satisfied	Honestly, the customer support was fine. They resolved my issue, but it took a few tries and a bit longer than I'd have liked. I wish they had more eco-friendly options for contacting them, like email instead of just phone.	Honestly? It's a bit of a mixed bag. Your customer support was alright – helpful enough when I needed it, but it wasn't exactly *amazing*. What would really bump things up for me would be more eco-conscious options. I'm always looking for companies that are doing their part for the environment, and that includes things like packaging, shipping methods, and even the overall sustainability of your products. If you could offer more choices in that area, it would definitely make me a much happier customer. I know it's a big ask, but it's something I'm increasingly prioritizing.
2	Moderately dissatisfied	Ugh, honestly, the appointment scheduling was a nightmare. I had to reschedule twice because of conflicts, and the whole process felt clunky and unprofessional. It's a shame, because the actual support I *did* get was pretty good. But the scheduling hassle really soured the whole experience.	Ugh, honestly? The biggest thing is the scheduling. It's a nightmare. I've had appointments moved, cancelled, rescheduled... it's a total mess. I understand things happen, but the lack of consistency and the way it's handled is incredibly frustrating. If you could improve the reliability of appointments and maybe offer more options – like different time slots, or even a wider range of days – that would make a huge difference. I spend way too much time just trying to *get* an appointment, let alone actually *using* your customer support. That's time I could be spending on, you know, *actually running my business*.
3	Extremely satisfied	Honestly, as a single mom, I'm always juggling so much. The customer support team was incredibly understanding and helpful – they really went above and beyond to work with my schedule and payment plan. I can't say enough good things!	Honestly? Keeping things affordable and flexible is the biggest thing. As a single mom, every penny counts, and knowing I can adjust my payments if something unexpected comes up... that's a lifesaver. So, maybe just keeping a close eye on those things and making sure those options remain available would be amazing. I don't need anything fancy, just reliable and budget-friendly.
4	Extremely dissatisfied	Honestly, the wait times were atrocious. I spent more time on hold than I did actually talking to someone! It's simply unacceptable.	Oh, honey, "improve my experience"? Where do I even begin? "Extremely dissatisfied" doesn't even begin to cover it. I've been waiting for ages – *ages* – for simple answers. I'm talking days, sometimes weeks, for a response to an email! Days! I'm not asking for the moon, just a prompt and courteous reply.

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL, allowing you to access objects from your workspace or Coop account interface. Learn more about creating an account and using the Coop. Here we post the survey and results publicly:

survey.push(description = "Example NPS survey", visibility = "public")

{'description': 'Example NPS survey',
 'object_type': 'survey',
 'url': 'https://www.expectedparrot.com/content/a9cfac3d-3d7f-414e-b58d-f245f714ecaf',
 'uuid': 'a9cfac3d-3d7f-414e-b58d-f245f714ecaf',
 'version': '0.1.39.dev2',
 'visibility': 'public'}

We can also post a notebook, such as this one:

from edsl import Notebook

n = Notebook(path = "nps_survey.ipynb")

info = n.push(description = "Notebook for simulating an NPS survey")
info

{'description': 'Notebook for simulating an NPS survey',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/de151af7-d9a2-40fd-9e74-198a32ca94da',
 'uuid': 'de151af7-d9a2-40fd-9e74-198a32ca94da',
 'version': '0.1.39.dev2',
 'visibility': 'unlisted'}

To update an object at the Coop:

n = Notebook(path = "nps_survey.ipynb") # resave

n.patch(uuid = info["uuid"], visibility = "public", value = n)

{'status': 'success'}