NPS survey in EDSL
This notebook provides sample EDSL code for simulating a Net Promoter Score (NPS) survey with AI agents and large language models. In the steps below we show how to construct an EDSL survey, create personas for AI agents to answer the questions, and then administer the survey to them. We also demonstrate some built-in methods for inspecting and analyzing the dataset of results that is generated when an EDSL survey is run.
The following questions are used in the sample survey:
On a scale from 0-10, how likely are you to recommend our company to a friend or colleague? (0=Not at all likely, 10=Very likely) Please tell us why you gave a rating.
How satisfied are you with the following experience with our company? Product quality Customer support Purchasing experience
Is there anything specific that our company can do to improve your experience?
Technical setup
Before running the code below, ensure that you have (1) installed the EDSL library and (2) created a Coop account to activate remote inference or stored your own API keys for language models that you want to use with EDSL. Please also see our tutorials and documentation page on getting started using the EDSL library.
Constructing questions
We start by selecting appropriate question types for the above questions. EDSL comes with a variety of common question types that we can choose from based on the form of the response that we want to get back from the model. The first quesiton is linear scale; we import the class type and then construct a question in the relevant template:
[1]:
from edsl import QuestionLinearScale
[2]:
q_recommend = QuestionLinearScale(
question_name = "recommend",
question_text = "On a scale from 0-10, how likely are you to recommend our company to a friend or colleague?",
question_options = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
option_labels = {0:"Not at all likely", 10:"Very likely"}
)
Each question type other than free text automatically includes a “comment” field for the model to provide commentary on its response to the main question. When we run the survey, we can check that it has effectively captured the follow-on question from above–Please tell us why you gave a rating–and modify or add questions as needed.
For the next question, we use a {{ placeholder }} for an “experience” that we will insert when repeating the base question:
[3]:
from edsl import QuestionMultipleChoice
[4]:
q_satisfied = QuestionMultipleChoice(
question_name = "satisfied",
question_text = "How satisfied are you with the following experience with our company: {{ experience }}",
question_options = [
"Extremely satisfied",
"Moderately satisfied",
"Neither satisfied nor dissatisfied",
"Moderately dissatisfied",
"Extremely dissatisfied"
]
)
The third question is a simple free text question that we can choose whether to administer once or individually for each “experience” question. In the steps that follow we show how to apply survey logic to achieve this effect:
[5]:
from edsl import QuestionFreeText
[6]:
q_improve = QuestionFreeText(
question_name = "improve",
question_text = "Is there anything specific that our company can do to improve your experience?"
)
Creating variants of questions with scenarios
Next we want to create a version of the “satisfied” question for each “experience”. This can be done with Scenario objects–dictionaries of key/value pairs representing the content to be added to questions. Scenarios can be automatically generated from a variety of data sources (PDFs, CSVs, images, tables, etc.). Here we have import a simple list:
[7]:
from edsl import ScenarioList, Scenario
[8]:
experiences = ["Product quality", "Customer support", "Purchasing experience"]
s = ScenarioList(
Scenario({"experience":e}) for e in experiences
)
We could also use a specific method for creating scenarios from a list:
[9]:
s = ScenarioList.from_list("experience", experiences)
We can check the scenarios that have been created:
[10]:
s
[10]:
{
"scenarios": [
{
"experience": "Product quality"
},
{
"experience": "Customer support"
},
{
"experience": "Purchasing experience"
}
]
}
To create the question variants, we pass the scenario list to the question loop() method, which returns a list of new questions. We can see that each question has a new unique name and a question text with the placeholder replaced with an experience:
[11]:
satisfied_questions = q_satisfied.loop(s)
satisfied_questions
[11]:
[Question('multiple_choice', question_name = """satisfied_0""", question_text = """How satisfied are you with the following experience with our company: Product quality""", question_options = ['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied']),
Question('multiple_choice', question_name = """satisfied_1""", question_text = """How satisfied are you with the following experience with our company: Customer support""", question_options = ['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied']),
Question('multiple_choice', question_name = """satisfied_2""", question_text = """How satisfied are you with the following experience with our company: Purchasing experience""", question_options = ['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied'])]
We can also use the loop() method to create copies of the “improve” question in order to present it as a follow-up question to each of the “satisfied” questions that have been parameterized with experiences. Here, we’re simply duplicating the base question without a scenario {{ placeholder }} because we will instead add a “memory” of the relevant “satisfied” question when administering each copy of it:
[12]:
improve_questions = q_improve.loop(s)
improve_questions
[12]:
[Question('free_text', question_name = """improve_0""", question_text = """Is there anything specific that our company can do to improve your experience?"""),
Question('free_text', question_name = """improve_1""", question_text = """Is there anything specific that our company can do to improve your experience?"""),
Question('free_text', question_name = """improve_2""", question_text = """Is there anything specific that our company can do to improve your experience?""")]
Creating a survey
Next we pass a list of all the questions to a Survey in order to administer them together:
[13]:
questions = [q_recommend] + satisfied_questions + improve_questions
[14]:
from edsl import Survey
[15]:
survey = Survey(questions)
Adding survey logic
In the next step we add logic to the survey specifying that each “improve” question should include a “memory” of a “satisfied” question (the question and answer that was provided):
[16]:
for i in range(len(s)):
survey = survey.add_targeted_memory(f"improve_{i}", f"satisfied_{i}")
We can inspect the survey details:
[17]:
# survey
AI agent personas
EDSL comes with a variety of methods for designing AI agents to answer surveys. An Agent is constructed by passing a dictionary of relevant traits with optional additional instructions for the language model to reference in generating responses for the agent. Agents can be constructed from a variety of data sources, including existing survey data (e.g., a dataset of responses that were provided to some other questions). We
can also use an EDSL question to draft some personas for agents. Here, we ask for a list of them:
[18]:
from edsl import QuestionList
[19]:
q_personas = QuestionList(
question_name = "personas",
question_text = "Draft 5 personas for diverse customers of landscaping business with varying satisfaction levels."
)
We can run this question alone and extract the response list (more on working with results below):
[20]:
personas = q_personas.run().select("personas").to_list()[0]
personas
[20]:
["Karen, a meticulous homeowner who is highly satisfied with the precision and attention to detail in her garden's upkeep",
'Bob, a busy professional who is moderately satisfied but wishes the service was more flexible with scheduling',
'Samantha, a new homeowner who is delighted with the transformative landscaping makeover she received',
'Greg, an environmentally-conscious customer who is dissatisfied with the lack of sustainable options',
'Tina, a retiree who is neutral because she appreciates the friendly service but finds it slightly overpriced']
Next we pass the personas to create a set of agents:
[21]:
from edsl import AgentList, Agent
[22]:
a = AgentList(
Agent(traits = {"persona":p}) for p in personas
)
Selecting language models
EDSL works with many popular large language models that we can select to use with a survey. To see a list of available models:
[23]:
from edsl import Model
[24]:
# Model.available()
To select a model to use with a survey we pass a model name to a Model:
[25]:
m = Model("gemini-pro")
If we want to compare responses for several models, we can use a ModelList instead:
[26]:
from edsl import ModelList
[27]:
m = ModelList(
Model(model) for model in ["gemini-pro", "gpt-4o"]
)
Note: If no model is specified when running a survey, the default model GPT 4 preview is used (as above when we generated personas).
Running a survey
We administer the survey by adding the agents and models with the by() method and then calling the run() method:
[28]:
results = survey.by(a).by(m).run()
This generates a dataset of Results that includes a response for each agent/model that was used. We can access the results with built-in methods for analysis. To see a list of all the components of the results:
[29]:
# results.columns
For example, we can filter, sort and display columns of results in a table:
[30]:
(results
.filter("model.model == 'gemini-pro'")
.sort_by("recommend", reverse=True)
.select(
"model",
"persona",
"recommend", "recommend_comment"
)
.print(format="rich")
)
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ model ┃ agent ┃ answer ┃ comment ┃ ┃ .model ┃ .persona ┃ .recommend ┃ .recommend_comment ┃ ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ gemini-pro │ Samantha, a new homeowner who is │ 10 │ The landscaping makeover completely │ │ │ delighted with the transformative │ │ transformed my backyard and made it a │ │ │ landscaping makeover she received │ │ beautiful and relaxing space for my │ │ │ │ │ family and friends to enjoy. I'm so happy │ │ │ │ │ with the results that I've already │ │ │ │ │ recommended your company to several │ │ │ │ │ people. │ ├────────────┼───────────────────────────────────────────┼────────────┼───────────────────────────────────────────┤ │ gemini-pro │ Karen, a meticulous homeowner who is │ 10 │ Karen is a meticulous homeowner who is │ │ │ highly satisfied with the precision and │ │ highly satisfied with the precision and │ │ │ attention to detail in her garden's │ │ attention to detail in her garden's │ │ │ upkeep │ │ upkeep. She is very likely to recommend │ │ │ │ │ the company to a friend or colleague │ │ │ │ │ because she is very satisfied with the │ │ │ │ │ company's services. │ ├────────────┼───────────────────────────────────────────┼────────────┼───────────────────────────────────────────┤ │ gemini-pro │ Tina, a retiree who is neutral because │ 7 │ I appreciate the friendly service, but I │ │ │ she appreciates the friendly service but │ │ find the prices to be a bit high. │ │ │ finds it slightly overpriced │ │ │ ├────────────┼───────────────────────────────────────────┼────────────┼───────────────────────────────────────────┤ │ gemini-pro │ Bob, a busy professional who is │ 7 │ I'm generally happy with the service, but │ │ │ moderately satisfied but wishes the │ │ I wish it was more flexible with │ │ │ service was more flexible with scheduling │ │ scheduling. │ ├────────────┼───────────────────────────────────────────┼────────────┼───────────────────────────────────────────┤ │ gemini-pro │ Greg, an environmentally-conscious │ 2 │ I am not likely to recommend your company │ │ │ customer who is dissatisfied with the │ │ to a friend or colleague because I am │ │ │ lack of sustainable options │ │ dissatisfied with the lack of sustainable │ │ │ │ │ options. I am an │ │ │ │ │ environmentally-conscious customer and I │ │ │ │ │ believe that companies have a │ │ │ │ │ responsibility to reduce their │ │ │ │ │ environmental impact. │ └────────────┴───────────────────────────────────────────┴────────────┴───────────────────────────────────────────┘
[31]:
(results
.filter("model.model == 'gemini-pro'")
.sort_by("satisfied_0")
.select("satisfied_0", "satisfied_0_comment", "improve_0")
.print(format="rich")
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ answer ┃ comment ┃ answer ┃ ┃ .satisfied_0 ┃ .satisfied_0_comment ┃ .improve_0 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Extremely satisfied │ The quality of the products used in my │ I am absolutely thrilled with the product │ │ │ landscaping makeover was exceptional. The │ quality and the transformative │ │ │ plants were healthy and vibrant, and the │ landscaping makeover I received from your │ │ │ materials used for the hardscaping were │ company. The results have exceeded my │ │ │ durable and attractive. I am confident │ expectations, and I couldn't be happier │ │ │ that my new landscape will last for many │ with the outcome. │ │ │ years to come. │ │ ├─────────────────────────┼───────────────────────────────────────────┼───────────────────────────────────────────┤ │ Extremely satisfied │ As a meticulous homeowner, I am highly │ As a meticulous homeowner who takes great │ │ │ satisfied with the product quality of my │ pride in my garden, I am extremely │ │ │ garden's upkeep. The precision and │ satisfied with the quality of your │ │ │ attention to detail are impeccable, │ products. Your attention to detail and │ │ │ resulting in a stunning and vibrant │ commitment to precision have been evident │ │ │ outdoor space. │ in every interaction I've had with your │ │ │ │ company. │ │ │ │ │ │ │ │ However, there is one area where I │ │ │ │ believe you could further enhance my │ │ │ │ experience: │ │ │ │ │ │ │ │ **Personalized Recommendations:** │ │ │ │ │ │ │ │ I would appreciate it if you could │ │ │ │ provide personalized recommendations │ │ │ │ based on my specific gardening needs and │ │ │ │ preferences. For example, I would find it │ │ │ │ helpful to receive tailored suggestions │ │ │ │ for plants that would thrive in my local │ │ │ │ climate and soil conditions, as well as │ │ │ │ tips on how to optimize their growth. │ ├─────────────────────────┼───────────────────────────────────────────┼───────────────────────────────────────────┤ │ Moderately dissatisfied │ I am moderately dissatisfied with the │ Yes, there are a few things your company │ │ │ product quality because I am concerned │ can do to improve my experience: │ │ │ about the environmental impact of the │ │ │ │ products. I would like to see more │ * **Offer more sustainable products.** │ │ │ sustainable options available. │ I'm particularly interested in products │ │ │ │ that are made from recycled materials, │ │ │ │ are biodegradable, or are otherwise │ │ │ │ environmentally friendly. │ │ │ │ * **Make it easier to find sustainable │ │ │ │ products.** I shouldn't have to dig │ │ │ │ through your website or store to find │ │ │ │ products that meet my needs. You should │ │ │ │ have a dedicated section for sustainable │ │ │ │ products, or at least make it easy to │ │ │ │ filter products by their environmental │ │ │ │ impact. │ │ │ │ * **Educate your customers about │ │ │ │ sustainability.** Many people don't │ │ │ │ realize the impact that their purchases │ │ │ │ have on the environment. You can help to │ │ │ │ change that by providing information │ │ │ │ about the environmental benefits of │ │ │ │ sustainable products. │ ├─────────────────────────┼───────────────────────────────────────────┼───────────────────────────────────────────┤ │ Moderately satisfied │ The products are generally of good │ As a retiree, I'm always looking for ways │ │ │ quality, but I find them to be a bit │ to save money. While I appreciate the │ │ │ overpriced. │ friendly service your company provides, I │ │ │ │ find the prices to be a bit steep. If you │ │ │ │ could offer more discounts or promotions, │ │ │ │ I would be more likely to make repeat │ │ │ │ purchases. Additionally, I would │ │ │ │ appreciate it if you could expand your │ │ │ │ product line to include more affordable │ │ │ │ options. │ ├─────────────────────────┼───────────────────────────────────────────┼───────────────────────────────────────────┤ │ Moderately satisfied │ I've been generally happy with the │ Sure, there are a few things that your │ │ │ quality of the products I've purchased │ company can do to improve my experience: │ │ │ from your company. The materials are │ │ │ │ high-quality, and the construction is │ * **Offer more flexible scheduling │ │ │ solid. However, I do wish there was more │ options.** I'm a busy professional, and │ │ │ flexibility with scheduling deliveries. I │ it can be difficult to find time to │ │ │ often have to rearrange my schedule to │ schedule appointments. If you could offer │ │ │ accommodate your delivery times, which │ more flexible scheduling options, such as │ │ │ can be inconvenient. │ evening or weekend appointments, it would │ │ │ │ be much easier for me to get the service │ │ │ │ I need. │ │ │ │ * **Improve communication.** I've had a │ │ │ │ few instances where I've had to │ │ │ │ reschedule appointments at the last │ │ │ │ minute, and I've found that the │ │ │ │ communication from your company has been │ │ │ │ lacking. If you could improve │ │ │ │ communication, it would help me to avoid │ │ │ │ these situations in the future. │ └─────────────────────────┴───────────────────────────────────────────┴───────────────────────────────────────────┘
[32]:
(results
.filter("model.model == 'gemini-pro'")
.select("satisfied_1", "satisfied_1_comment", "improve_1")
.print(pretty_labels = {
"answer.satisfied_1": "Customer service: satisfaction",
"comment.satisfied_1_comment": "Customer service: comment",
"answer.improve_1": "Customer service: improvements"
},
format="rich")
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Customer service: satisfaction ┃ Customer service: comment ┃ Customer service: improvements ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Extremely satisfied │ I am extremely satisfied with the │ I'm absolutely thrilled with the │ │ │ customer support I received. The team │ exceptional customer support I've │ │ │ was very helpful and responsive to my │ received from your company throughout │ │ │ questions and concerns. They went │ my landscaping project. The team has │ │ │ above and beyond to make sure I was │ been incredibly responsive, proactive, │ │ │ happy with my experience. │ and attentive to my every need. I │ │ │ │ couldn't have asked for a better │ │ │ │ experience. │ │ │ │ │ │ │ │ However, one area where I believe │ │ │ │ there could be a slight improvement is │ │ │ │ in providing more detailed updates on │ │ │ │ the progress of the project. While I │ │ │ │ appreciate the regular check-ins, I │ │ │ │ would find it even more helpful to │ │ │ │ receive more specific information │ │ │ │ about the tasks being completed and │ │ │ │ the estimated timeline for completion. │ │ │ │ This would give me a clearer │ │ │ │ understanding of the project's │ │ │ │ progress and allow me to plan │ │ │ │ accordingly. │ ├────────────────────────────────┼───────────────────────────────────────┼────────────────────────────────────────┤ │ Moderately dissatisfied │ I am moderately dissatisfied with the │ I'm moderately dissatisfied with the │ │ │ customer support I received because I │ customer support I received. I would │ │ │ was not able to get the information I │ like to see more sustainable options │ │ │ needed about the company's │ offered by your company. For example, │ │ │ sustainability practices. I am very │ I would like to see more products made │ │ │ concerned about the environment and I │ from recycled materials, and I would │ │ │ want to make sure that I am doing │ like to see more options for repairing │ │ │ business with companies that are │ products instead of replacing them. I │ │ │ committed to sustainability. I was │ would also like to see more │ │ │ hoping to learn more about the │ information about your company's │ │ │ company's environmental policies and │ sustainability initiatives. │ │ │ practices, but I was not able to get │ │ │ │ that information from customer │ │ │ │ support. │ │ ├────────────────────────────────┼───────────────────────────────────────┼────────────────────────────────────────┤ │ Extremely satisfied │ I am incredibly pleased with the │ As a meticulous homeowner, I am │ │ │ customer support I have received. The │ extremely satisfied with the customer │ │ │ representatives are always courteous, │ support I have received from your │ │ │ knowledgeable, and prompt in │ company. The representatives have │ │ │ responding to my inquiries. They have │ always been knowledgeable and helpful, │ │ │ consistently gone above and beyond to │ and they have gone above and beyond to │ │ │ assist me with my gardening needs, │ ensure that my needs are met. I am │ │ │ providing personalized advice and │ particularly impressed with the │ │ │ recommendations that have helped me │ company's commitment to precision and │ │ │ maintain a thriving garden. │ attention to detail, which is evident │ │ │ │ in all aspects of their customer │ │ │ │ service. │ │ │ │ │ │ │ │ However, there is one area where I │ │ │ │ believe your company could improve: │ │ │ │ the speed of response to customer │ │ │ │ inquiries. While I understand that │ │ │ │ your team is busy, I have sometimes │ │ │ │ had to wait several days for a │ │ │ │ response to my emails or phone calls. │ │ │ │ I would appreciate it if your company │ │ │ │ could prioritize responding to │ │ │ │ customer inquiries more quickly, as │ │ │ │ this would help me to resolve any │ │ │ │ issues or concerns I have in a timely │ │ │ │ manner. │ ├────────────────────────────────┼───────────────────────────────────────┼────────────────────────────────────────┤ │ Moderately satisfied │ I appreciate the friendly service, │ I appreciate the friendly customer │ │ │ but I find the prices to be a bit │ support, but I do find the prices to │ │ │ high. │ be a bit high. If you could lower the │ │ │ │ prices, I would be more likely to use │ │ │ │ your services more often. │ ├────────────────────────────────┼───────────────────────────────────────┼────────────────────────────────────────┤ │ Moderately satisfied │ I'm generally happy with the customer │ As a busy professional, I often find │ │ │ support I've received, but I wish │ it difficult to schedule appointments │ │ │ there were more flexibility with │ during traditional business hours. │ │ │ scheduling appointments. │ Offering more flexible scheduling │ │ │ │ options, such as evening or weekend │ │ │ │ appointments, would greatly improve my │ │ │ │ experience with your company's │ │ │ │ customer support. │ └────────────────────────────────┴───────────────────────────────────────┴────────────────────────────────────────┘
Posting to the Coop
The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL, allowing you to access objects from your workspace or Coop account interface. Learn more about creating an account and using the Coop. Here we post the survey and results publicly:
[33]:
survey.push(description = "Example NPS survey", visibility = "public")
[33]:
{'description': 'Example NPS survey',
'object_type': 'survey',
'url': 'https://www.expectedparrot.com/content/581044a0-1a97-410b-be5d-9f1b52655112',
'uuid': '581044a0-1a97-410b-be5d-9f1b52655112',
'version': '0.1.33.dev1',
'visibility': 'public'}
[34]:
results.push(description = "Results of example NPS survey", visibility = "public")
[34]:
{'description': 'Results of example NPS survey',
'object_type': 'results',
'url': 'https://www.expectedparrot.com/content/1322aa3f-1816-41d2-815c-9417f20a2499',
'uuid': '1322aa3f-1816-41d2-815c-9417f20a2499',
'version': '0.1.33.dev1',
'visibility': 'public'}
We can also post a notebook, such as this one:
[35]:
from edsl import Notebook
[36]:
n = Notebook(path = "nps_survey.ipynb")
[37]:
n.push(description = "Notebook for simulating an NPS survey")
[37]:
{'description': 'Notebook for simulating an NPS survey',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/0f12162f-474f-446e-9456-32b14bc87591',
'uuid': '0f12162f-474f-446e-9456-32b14bc87591',
'version': '0.1.33.dev1',
'visibility': 'unlisted'}
To update an object at the Coop:
[40]:
n = Notebook(path = "nps_survey.ipynb")
[41]:
n.patch(uuid = "0f12162f-474f-446e-9456-32b14bc87591", visibility = "public", value = n)
[41]:
{'status': 'success'}
[ ]: