Analyzing a transcript
This notebook provides example EDSL code for analyzing a transcript using AI. In the steps below we construct a survey of questions about a transcript of a fictional call between a business development representative and a potential customer, and then use a language model to answer the questions in the persona of an business development expert delivering feedback on the call. The code can easily be modified for purposes of analyzing other textual content, images, PDFs, websites, tables and other data types with diverse personas and many popular language models.
EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.
Constructing questions
We start by importing the tools we will use and constructing questions that we want to ask about the contents of the call. EDSL provides many common question types that we can choose from based on the form of the response that we want to get back from language model: free text, multiple choice, etc.
[1]:
from edsl import QuestionMultipleChoice, QuestionLinearScale, QuestionFreeText, Survey, Agent, AgentList, Model
[2]:
q1 = QuestionMultipleChoice(
question_name = "understanding",
question_text = "How would you rate the representative's understanding of the customer's needs?",
question_options = ["Excellent", "Good", "Fair", "Poor"]
)
q2 = QuestionMultipleChoice(
question_name = "explain_features",
question_text = "How well did the representative explain the features relevant to the customer's business?",
question_options = ["Very clearly", "Somewhat clearly", "Unclear", "Not explained"]
)
q3 = QuestionMultipleChoice(
question_name = "address_concerns",
question_text = "How well did the representative address the questions and concerns of the customer?",
question_options = ["Very effectively", "Mostly effectively", "Somewhat effectively", "Not effectively"]
)
q4 = QuestionMultipleChoice(
question_name = "product_knowledge",
question_text = "How would you rate the representative's knowledge of the products and services discussed?",
question_options = ["Excellent", "Good", "Fair", "Poor"]
)
q5 = QuestionMultipleChoice(
question_name = "attentive",
question_text = "Did the representative appear attentive to the customer's needs and concerns?",
question_options = ["Very attentive", "Mostly attentive", "Somewhat attentive", "Not attentive"]
)
q6 = QuestionLinearScale(
question_name = "likely_sale",
question_text = "On a scale from 1 to 10, how likely is the customer to use the products and servies discussed?",
question_options = [1,2,3,4,5,6,7,8,9,10]
)
q7 = QuestionMultipleChoice(
question_name = "next_steps",
question_text = "Did the representative offer relevant next steps for implementation or testing products?",
question_options = ["Yes, very clear and actionable", "Somewhat clear", "Not clear", "No next steps offered"]
)
q8 = QuestionFreeText(
question_name = "helpful_approaches",
question_text = "What aspects of the representative’s approach were most helpful to the customer?"
)
q9 = QuestionFreeText(
question_name = "inadequate_explanation",
question_text = "Were any features or details not adequately explained?"
)
q10 = QuestionFreeText(
question_name = "improvements",
question_text = "How could the representative have improved their explanation of the products?"
)
q11 = QuestionFreeText(
question_name = "confusing",
question_text = "Was there anything about the conversation that seemed unclear or confusing?"
)
q12 = QuestionFreeText(
question_name = "confident",
question_text = "What additional information could have helped the customer feel more confident in the representative's offerings?"
)
We combine the questions in a survey in order to administer them together:
[3]:
survey = Survey([q1, q2, q3, q4, q5, q6, q7, q8, q9, q10, q11, q12])
Selecting content for review
Next we import the text of the call that we want to give to the model in answering the questions. Here we use a mock transcript drafted by chatGPT:
[4]:
transcript = """
Alex (Quick Representative):
Hi, Jamie! Thanks so much for taking the time to chat today. How’s everything going with your website?
Jamie (Customer):
Hi, Alex. It’s going okay, I think. We’re getting some users, but honestly, I’m just trying to figure out the best way to manage payments. It’s all new to me.
Alex:
Got it, no worries! Quick can really help simplify things for you. Our platform allows you to accept payments easily and has a subscription option, which I think you mentioned you were interested in, right?
Jamie:
Yeah, we’re thinking about offering different subscription levels for our software, but I’m not sure how to set it all up. And we have some international users, so that’s been a bit confusing too.
Alex:
Right, right. So, with Quick, you can definitely do subscriptions, and it handles multiple currencies, too. For example, if you have a customer in Europe, Quick can automatically handle the conversion. It’s designed to make that part easy, so you don’t need to worry too much about different currencies.
Jamie:
That sounds helpful. But I’m a bit concerned about the setup. I’m not a developer, so I’m worried this might be too complicated.
Alex:
Oh, yeah, I totally understand! Quick’s setup is meant to be as easy as possible. We have plugins and a lot of guides to walk you through it. I’d say that most people get it up and running without too much trouble. And if you do need help, our support team is available.
Jamie:
Okay, that’s a relief. I’ve heard some stories about people getting stuck for weeks with payment systems. I don’t want that to happen.
Alex:
I get that completely. We try to make it as straightforward as possible. Also, if you’re worried about fraud or security, Quick has a feature called Shield that helps detect fraud. It’s built in, so it’s automatically protecting you from day one.
Jamie:
Oh, that sounds good. I don’t know much about fraud detection. Is it something I’d have to set up?
Alex:
Nope! Shield is on by default, so you don’t have to configure anything special unless you want to. It uses machine learning to detect unusual activity based on data from many transactions, which helps catch fraudulent transactions without much work on your end.
Jamie:
That’s nice to know. What about fees? I’d like to have a clear idea of what this will cost.
Alex:
Absolutely. So, Quick charges 2.9% plus 30 cents per transaction. For subscriptions, there’s no extra fee beyond the normal transaction cost. Also, if you start processing a lot of payments, we do offer some discounts, but those kick in at a higher volume.
Jamie:
Alright, that makes sense. And what about invoicing? A few customers have asked about getting invoices instead of just a credit card charge.
Alex:
Yes, we can do that too. With Quick Invoicing, you can send out one-time or recurring invoices, which is nice if you have larger clients who prefer an invoice. It’s all within the Quick system, so it integrates well.
Jamie:
Good to know. I think this covers most of my questions. It sounds like Quick could be a good fit. I’d just need to make sure I’m able to actually set it up without a ton of back-and-forth.
Alex:
I totally understand, Jamie. I’ll send over some guides and set you up with a sandbox account to test things out. Let’s reconnect in a couple of weeks to go over any questions you might have.
Jamie:
Sounds good, thanks, Alex. Looking forward to trying it out.
Alex:
Great! Thanks for your time, Jamie. I’ll follow up soon. Have a good one!
"""
AI agent personas and instructions
Here we create a persona and instructions for an AI agent that we will give a model to reference in answering questions. We include the transcript of the customer call in the agent instructions. (We could also include it directly in question texts; learn more about using questions with multiple pieces of content with Scenario objects.)
[5]:
agent = Agent(
traits = {"persona": "You are an expert business development representative with many years of experience at diverse companies."},
instruction = """
You are being asked to review the following transcript of a business development call between a representative of a payments
processing company and a potential customer and provide critical feedback.
Transcript:
""" + transcript
)
Selecting a model
EDSL works with many popular language models that we can select to generate responses to the questions. Here we specify that we want to use OpenAI’s GPT-4o:
[6]:
model = Model("gpt-4o")
Running the survey
Next we add the agents and model to the survey and run it:
[7]:
results = survey.by(agent).by(model).run()
Job sent to server. (Job uuid=98a39b8b-8c39-45fe-84ca-df7562a3e869).
Job completed and Results stored on Coop: https://www.expectedparrot.com/content/15e79d9f-704e-4d19-a5ed-211a8a47ca74.
This generates a formatted dataset of results that we can inspect with built-in methods for analyzing responses. Here we print them in some tables together with information about each persona that was used:
[8]:
(
results
.select("persona", "understanding", "explain_features", "address_concerns", "product_knowledge",
# "attentive", "likely_sale", "next_steps"
)
.print(
pretty_labels = {
"agent.persona": "Persona",
"answer.understanding": q1.question_text,
"answer.explain_features": q2.question_text,
"answer.address_concerns": q3.question_text,
"answer.product_knowledge": q4.question_text,
# "answer.attentive": q5.question_text,
# "answer.likely_sale": q6.question_text,
# "answer.next_steps": q7.question_text
},
format = "rich"
)
)
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ ┃ ┃ How well did the ┃ How would you rate ┃ ┃ ┃ ┃ How well did the ┃ representative ┃ the ┃ ┃ ┃ How would you rate ┃ representative ┃ address the ┃ representative's ┃ ┃ ┃ the representative's ┃ explain the features ┃ questions and ┃ knowledge of the ┃ ┃ ┃ understanding of the ┃ relevant to the ┃ concerns of the ┃ products and ┃ ┃ Persona ┃ customer's needs? ┃ customer's business? ┃ customer? ┃ services discussed? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩ │ You are an expert │ Good │ Somewhat clearly │ Mostly effectively │ Good │ │ business development │ │ │ │ │ │ representative with │ │ │ │ │ │ many years of │ │ │ │ │ │ experience at │ │ │ │ │ │ diverse companies. │ │ │ │ │ └──────────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴─────────────────────┘
[9]:
(
results
.select("persona",
# "understanding", "explain_features", "address_concerns", "product_knowledge",
"attentive", "likely_sale", "next_steps"
)
.print(
pretty_labels = {
"agent.persona": "Persona",
# "answer.understanding": q1.question_text,
# "answer.explain_features": q2.question_text,
# "answer.address_concerns": q3.question_text,
# "answer.product_knowledge": q4.question_text,
"answer.attentive": q5.question_text,
"answer.likely_sale": q6.question_text,
"answer.next_steps": q7.question_text
},
format = "rich"
)
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ ┃ On a scale from 1 to 10, ┃ ┃ ┃ ┃ Did the representative ┃ how likely is the ┃ Did the representative ┃ ┃ ┃ appear attentive to the ┃ customer to use the ┃ offer relevant next steps ┃ ┃ ┃ customer's needs and ┃ products and servies ┃ for implementation or ┃ ┃ Persona ┃ concerns? ┃ discussed? ┃ testing products? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ You are an expert │ Mostly attentive │ 8 │ Yes, very clear and │ │ business development │ │ │ actionable │ │ representative with many │ │ │ │ │ years of experience at │ │ │ │ │ diverse companies. │ │ │ │ └───────────────────────────┴────────────────────────────┴───────────────────────────┴────────────────────────────┘
[10]:
(
results
.select("helpful_approaches")
.print(pretty_labels = {"answer.helpful_approaches": q8.question_text}, format = "rich")
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ What aspects of the representative’s approach were most helpful to the customer? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ The representative, Alex, demonstrated several helpful aspects in their approach: │ │ │ │ 1. **Understanding and Empathy**: Alex showed understanding and empathy towards Jamie's concerns, particularly │ │ regarding the complexity of setting up payment systems and the potential for getting stuck. This helped build │ │ trust and rapport. │ │ │ │ 2. **Clear Explanation of Features**: Alex provided a clear explanation of Quick's features, such as handling │ │ subscriptions, multiple currencies, and fraud detection with Shield. This helped Jamie understand how the │ │ platform could meet their specific needs. │ │ │ │ 3. **Reassurance on Ease of Setup**: By emphasizing the ease of setup with plugins, guides, and available │ │ support, Alex addressed Jamie's concerns about not being a developer and reassured them about the process. │ │ │ │ 4. **Proactive Offer of Support**: Alex offered to send guides and set up a sandbox account, showing a │ │ proactive approach to ensuring Jamie could test and understand the system without commitment. │ │ │ │ 5. **Transparent Pricing Information**: By clearly explaining the transaction fees and mentioning potential │ │ discounts for higher volumes, Alex provided transparency about the costs involved, which is crucial for │ │ customer decision-making. │ │ │ │ 6. **Addressing Additional Needs**: Alex responded to Jamie's inquiry about invoicing by explaining the Quick │ │ Invoicing feature, demonstrating attentiveness to Jamie's business requirements. │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
[11]:
(
results
.select("inadequate_explanation")
.print(pretty_labels = {"answer.inadequate_explanation": q8.question_text}, format = "rich")
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ What aspects of the representative’s approach were most helpful to the customer? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Overall, Alex did a good job of explaining the features and benefits of Quick's payment processing system. │ │ However, there are a few areas where the explanation could have been more detailed: │ │ │ │ 1. **Subscription Setup**: While Alex mentioned that Quick supports subscriptions, there was no detailed │ │ explanation of how to set them up or manage different subscription levels. Given Jamie's concern about not │ │ being a developer, it would have been helpful to provide more information on how user-friendly the subscription │ │ setup is, perhaps mentioning specific tools or interfaces that make it easier. │ │ │ │ 2. **International Payments**: Alex briefly mentioned that Quick handles multiple currencies and automatic │ │ conversion, but didn't go into detail about how this process works or any potential fees associated with │ │ currency conversion. This could be a significant concern for Jamie, who has international users. │ │ │ │ 3. **Fraud Detection**: Alex mentioned that Shield is on by default and uses machine learning, but didn't │ │ elaborate on what types of fraud it can detect or provide examples of how it has helped other customers. A │ │ brief case study or example could have strengthened this point. │ │ │ │ 4. **Invoicing**: While Alex mentioned Quick Invoicing, there was no discussion of how easy it is to use or │ │ customize invoices, or whether there are any additional costs associated with this feature. │ │ │ │ 5. **Fees and Discounts**: Alex provided basic information about transaction fees but did not elaborate on what │ │ constitutes a "higher volume" for discounts, which could be important for Jamie's financial planning. │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
[12]:
(
results
.select("improvements")
.print(pretty_labels = {"answer.improvements": q8.question_text}, format = "rich")
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ What aspects of the representative’s approach were most helpful to the customer? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ The representative, Alex, did a good job of addressing Jamie's concerns and providing an overview of Quick's │ │ features. However, there are several areas where the explanation could have been improved: │ │ │ │ 1. **Tailored Explanation**: Alex could have tailored the explanation more specifically to Jamie's business │ │ model and needs. For example, asking more detailed questions about Jamie's subscription plans or international │ │ user base could have allowed Alex to provide more targeted solutions. │ │ │ │ 2. **Technical Jargon**: While Alex did a good job of keeping the explanation simple, there were moments where │ │ technical terms were used without clarification, such as "plugins" and "sandbox account." Providing a brief │ │ explanation or analogy for these terms would help a non-technical customer like Jamie feel more at ease. │ │ │ │ 3. **Use Cases and Examples**: Providing specific use cases or examples of how other similar businesses have │ │ successfully implemented Quick could have helped Jamie visualize the platform's benefits. This could include │ │ stories of other companies overcoming similar challenges or how they structured their subscription models. │ │ │ │ 4. **Visual Aids**: Offering to share visual aids, such as a demo video or a screen share during the call, │ │ could have helped Jamie understand the setup process more clearly and feel more confident about the ease of │ │ use. │ │ │ │ 5. **Emphasizing Support**: While Alex mentioned that support is available, emphasizing the level of support, │ │ such as dedicated account managers or 24/7 availability, could reassure Jamie about getting help when needed. │ │ │ │ 6. **Cost Transparency**: While Alex provided the basic fee structure, discussing any potential hidden fees or │ │ additional costs (e.g., currency conversion fees, chargebacks) would provide a more comprehensive understanding │ │ of the pricing model. │ │ │ │ 7. **Follow-up Plan**: The follow-up plan could have been more detailed, specifying what materials Jamie will │ │ receive and setting a specific date and time for the next meeting to discuss any issues that arise during the │ │ trial period. │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
[13]:
(
results
.select("confusing")
.print(pretty_labels = {"answer.confusing": q8.question_text}, format = "rich")
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ What aspects of the representative’s approach were most helpful to the customer? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ The conversation was generally clear and well-structured, but there are a few areas where additional clarity │ │ could have been beneficial: │ │ │ │ 1. **Setup Process**: While Alex mentioned that Quick's setup is meant to be easy and includes plugins and │ │ guides, it might have been helpful to provide a more specific example or a brief overview of the steps │ │ involved. This could help Jamie feel more confident about the process. │ │ │ │ 2. **International Payments**: Alex mentioned that Quick handles multiple currencies and automatic conversion. │ │ It could be useful to explain if there are any additional fees or considerations specific to international │ │ transactions, as this is often a concern for businesses with international customers. │ │ │ │ 3. **Fraud Detection**: While Alex explained that Shield is on by default, it might have been reassuring to │ │ provide a bit more detail about how effective it is or any additional customization options available for fraud │ │ detection. │ │ │ │ 4. **Fees and Discounts**: Alex mentioned discounts for high-volume processing but did not specify what │ │ constitutes "a lot of payments" or what the discount structure looks like. Providing a rough idea of the │ │ thresholds for discounts could be helpful. │ │ │ │ 5. **Invoicing**: While Alex confirmed that Quick can handle invoicing, it might have been useful to mention if │ │ there are any specific features related to invoicing that could benefit Jamie, such as customization options or │ │ integration with accounting software. │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
[14]:
(
results
.select("confident")
.print(pretty_labels = {"answer.confident": q8.question_text}, format = "rich")
)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ What aspects of the representative’s approach were most helpful to the customer? ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ The representative, Alex, could have provided additional information in several areas to help Jamie feel more │ │ confident: │ │ │ │ 1. **Detailed Setup Process**: While Alex mentioned that the setup is easy and there are guides available, │ │ providing a brief overview of the actual steps involved in setting up Quick on Jamie's website would have been │ │ helpful. This could include mentioning any specific plugins compatible with popular website platforms or CMSs. │ │ │ │ 2. **Customer Support**: Although Alex mentioned that the support team is available, providing more details │ │ about the level of support (e.g., 24/7 availability, dedicated account managers, response times) could reassure │ │ Jamie about the assistance available if issues arise. │ │ │ │ 3. **Case Studies or Testimonials**: Sharing success stories or testimonials from similar businesses that have │ │ successfully implemented Quick could provide Jamie with real-world examples of how Quick has benefited others. │ │ │ │ 4. **Security Details**: While Alex touched on fraud detection, providing more information about overall │ │ security measures, such as data encryption and compliance with industry standards (e.g., PCI DSS compliance), │ │ could further ease Jamie's concerns. │ │ │ │ 5. **Trial or Demo**: Offering a live demo or a limited-time free trial could allow Jamie to experience Quick's │ │ features firsthand, which might increase confidence in the platform's ease of use and functionality. │ │ │ │ 6. **Integration with Existing Systems**: If Jamie is using other tools for their website, information on how │ │ Quick integrates with popular platforms (e.g., CRM, accounting software) could be beneficial. │ │ │ │ 7. **Fee Structure Clarity**: While Alex provided the basic fee structure, additional details on any potential │ │ hidden fees, setup costs, or long-term contracts would ensure full transparency. │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Posting to the Coop
The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.
Here we demonstrate how to post this notebook:
[14]:
from edsl import Notebook
notebook = Notebook(path = "analyze_customer_call.ipynb")
notebook.push(description = "Analyzing a customer call")
[14]:
{'description': 'Analyzing a customer call',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/c7d3042e-2b50-4232-a655-7b9011290151',
'uuid': 'c7d3042e-2b50-4232-a655-7b9011290151',
'version': '0.1.37.dev1',
'visibility': 'unlisted'}
Updating content at the Coop:
[19]:
notebook = Notebook(path = "analyze_customer_call.ipynb")
notebook.patch(uuid = "c7d3042e-2b50-4232-a655-7b9011290151", value = notebook)
[19]:
{'status': 'success'}