Analyzing a transcript

This notebook provides example EDSL code for analyzing a transcript using AI. In the steps below we construct a survey of questions about a transcript of a fictional call between a business development representative and a potential customer, and then use a language model to answer the questions in the persona of an business development expert delivering feedback on the call. The code can easily be modified for purposes of analyzing other textual content, images, PDFs, websites, tables and other data types with diverse personas and many popular language models.

EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Constructing questions

We start by importing the tools we will use and constructing questions that we want to ask about the contents of the call. EDSL provides many common question types that we can choose from based on the form of the response that we want to get back from language model: free text, multiple choice, etc.

from edsl import QuestionMultipleChoice, QuestionLinearScale, QuestionFreeText, Survey, Agent, AgentList, Model

q1 = QuestionMultipleChoice(
    question_name = "understanding",
    question_text = "How would you rate the representative's understanding of the customer's needs?",
    question_options = ["Excellent", "Good", "Fair", "Poor"]
)
q2 = QuestionMultipleChoice(
    question_name = "explain_features",
    question_text = "How well did the representative explain the features relevant to the customer's business?",
    question_options = ["Very clearly", "Somewhat clearly", "Unclear", "Not explained"]
)
q3 = QuestionMultipleChoice(
    question_name = "address_concerns",
    question_text = "How well did the representative address the questions and concerns of the customer?",
    question_options = ["Very effectively", "Mostly effectively", "Somewhat effectively", "Not effectively"]
)
q4 = QuestionMultipleChoice(
    question_name = "product_knowledge",
    question_text = "How would you rate the representative's knowledge of the products and services discussed?",
    question_options = ["Excellent", "Good", "Fair", "Poor"]
)
q5 = QuestionMultipleChoice(
    question_name = "attentive",
    question_text = "Did the representative appear attentive to the customer's needs and concerns?",
    question_options = ["Very attentive", "Mostly attentive", "Somewhat attentive", "Not attentive"]
)
q6 = QuestionLinearScale(
    question_name = "likely_sale",
    question_text = "On a scale from 1 to 10, how likely is the customer to use the products and servies discussed?",
    question_options = [1,2,3,4,5,6,7,8,9,10]
)
q7 = QuestionMultipleChoice(
    question_name = "next_steps",
    question_text = "Did the representative offer relevant next steps for implementation or testing products?",
    question_options = ["Yes, very clear and actionable", "Somewhat clear", "Not clear", "No next steps offered"]
)
q8 = QuestionFreeText(
    question_name = "helpful_approaches",
    question_text = "What aspects of the representative’s approach were most helpful to the customer?"
)
q9 = QuestionFreeText(
    question_name = "inadequate_explanation",
    question_text = "Were any features or details not adequately explained?"
)
q10 = QuestionFreeText(
    question_name = "improvements",
    question_text = "How could the representative have improved their explanation of the products?"
)
q11 = QuestionFreeText(
    question_name = "confusing",
    question_text = "Was there anything about the conversation that seemed unclear or confusing?"
)
q12 = QuestionFreeText(
    question_name = "confident",
    question_text = "What additional information could have helped the customer feel more confident in the representative's offerings?"
)

We combine the questions in a survey in order to administer them together:

survey = Survey([q1, q2, q3, q4, q5, q6, q7, q8, q9, q10, q11, q12])

Selecting content for review

Next we import the text of the call that we want to give to the model in answering the questions. Here we use a mock transcript drafted by chatGPT:

transcript = """
Alex (Quick Representative):
Hi, Jamie! Thanks so much for taking the time to chat today. How’s everything going with your website?

Jamie (Customer):
Hi, Alex. It’s going okay, I think. We’re getting some users, but honestly, I’m just trying to figure out the best way to manage payments. It’s all new to me.

Alex:
Got it, no worries! Quick can really help simplify things for you. Our platform allows you to accept payments easily and has a subscription option, which I think you mentioned you were interested in, right?

Jamie:
Yeah, we’re thinking about offering different subscription levels for our software, but I’m not sure how to set it all up. And we have some international users, so that’s been a bit confusing too.

Alex:
Right, right. So, with Quick, you can definitely do subscriptions, and it handles multiple currencies, too. For example, if you have a customer in Europe, Quick can automatically handle the conversion. It’s designed to make that part easy, so you don’t need to worry too much about different currencies.

Jamie:
That sounds helpful. But I’m a bit concerned about the setup. I’m not a developer, so I’m worried this might be too complicated.

Alex:
Oh, yeah, I totally understand! Quick’s setup is meant to be as easy as possible. We have plugins and a lot of guides to walk you through it. I’d say that most people get it up and running without too much trouble. And if you do need help, our support team is available.

Jamie:
Okay, that’s a relief. I’ve heard some stories about people getting stuck for weeks with payment systems. I don’t want that to happen.

Alex:
I get that completely. We try to make it as straightforward as possible. Also, if you’re worried about fraud or security, Quick has a feature called Shield that helps detect fraud. It’s built in, so it’s automatically protecting you from day one.

Jamie:
Oh, that sounds good. I don’t know much about fraud detection. Is it something I’d have to set up?

Alex:
Nope! Shield is on by default, so you don’t have to configure anything special unless you want to. It uses machine learning to detect unusual activity based on data from many transactions, which helps catch fraudulent transactions without much work on your end.

Jamie:
That’s nice to know. What about fees? I’d like to have a clear idea of what this will cost.

Alex:
Absolutely. So, Quick charges 2.9% plus 30 cents per transaction. For subscriptions, there’s no extra fee beyond the normal transaction cost. Also, if you start processing a lot of payments, we do offer some discounts, but those kick in at a higher volume.

Jamie:
Alright, that makes sense. And what about invoicing? A few customers have asked about getting invoices instead of just a credit card charge.

Alex:
Yes, we can do that too. With Quick Invoicing, you can send out one-time or recurring invoices, which is nice if you have larger clients who prefer an invoice. It’s all within the Quick system, so it integrates well.

Jamie:
Good to know. I think this covers most of my questions. It sounds like Quick could be a good fit. I’d just need to make sure I’m able to actually set it up without a ton of back-and-forth.

Alex:
I totally understand, Jamie. I’ll send over some guides and set you up with a sandbox account to test things out. Let’s reconnect in a couple of weeks to go over any questions you might have.

Jamie:
Sounds good, thanks, Alex. Looking forward to trying it out.

Alex:
Great! Thanks for your time, Jamie. I’ll follow up soon. Have a good one!
"""

AI agent personas and instructions

Here we create a persona and instructions for an AI agent that we will give a model to reference in answering questions. We include the transcript of the customer call in the agent instructions. (We could also include it directly in question texts; learn more about using questions with multiple pieces of content with Scenario objects.)

agent = Agent(
        traits = {"persona": "You are an expert business development representative with many years of experience at diverse companies."},
        instruction = """
        You are being asked to review the following transcript of a business development call between a representative of a payments
        processing company and a potential customer and provide critical feedback.
        Transcript:
        """ + transcript
)

Selecting a model

EDSL works with many popular language models that we can select to generate responses to the questions. Here we specify that we want to use OpenAI’s GPT-4o:

model = Model("gpt-4o")

Running the survey

Next we add the agents and model to the survey and run it:

results = survey.by(agent).by(model).run()

▼ Job Status (2025-02-07 20:26:12)

Job UUID	1b6fcb68-d462-4efb-9a36-c27911081e9e
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/1b6fcb68-d462-4efb-9a36-c27911081e9e
Exceptions Report URL	None
Results UUID	8cb11227-cec8-44b5-80b6-6065fa7b6d79
Results URL	https://www.expectedparrot.com/content/8cb11227-cec8-44b5-80b6-6065fa7b6d79

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/8cb11227-cec8-44b5-80b6-6065fa7b6d79

This generates a formatted dataset of results that we can inspect with built-in methods for analyzing responses. Here we print them in some tables together with information about each persona that was used:

(
    results
    .select("persona", "understanding", "explain_features", "address_concerns", "product_knowledge",
            # "attentive", "likely_sale", "next_steps"
           )
    .print(
        pretty_labels = {
            "agent.persona": "Persona",
            "answer.understanding": q1.question_text,
            "answer.explain_features": q2.question_text,
            "answer.address_concerns": q3.question_text,
            "answer.product_knowledge": q4.question_text,
            # "answer.attentive": q5.question_text,
            # "answer.likely_sale": q6.question_text,
            # "answer.next_steps": q7.question_text
        }
    )
)

	Persona	How would you rate the representative's understanding of the customer's needs?	How well did the representative explain the features relevant to the customer's business?	How well did the representative address the questions and concerns of the customer?	How would you rate the representative's knowledge of the products and services discussed?
0	You are an expert business development representative with many years of experience at diverse companies.	Good	Very clearly	Mostly effectively	Good

(
    results
    .select("persona",
            # "understanding", "explain_features", "address_concerns", "product_knowledge",
            "attentive", "likely_sale", "next_steps"
           )
    .print(
        pretty_labels = {
            "agent.persona": "Persona",
            # "answer.understanding": q1.question_text,
            # "answer.explain_features": q2.question_text,
            # "answer.address_concerns": q3.question_text,
            # "answer.product_knowledge": q4.question_text,
            "answer.attentive": q5.question_text,
            "answer.likely_sale": q6.question_text,
            "answer.next_steps": q7.question_text
        },
        format = "rich"
    )
)

	Persona	Did the representative appear attentive to the customer's needs and concerns?	On a scale from 1 to 10, how likely is the customer to use the products and servies discussed?	Did the representative offer relevant next steps for implementation or testing products?
0	You are an expert business development representative with many years of experience at diverse companies.	Mostly attentive	8	Yes, very clear and actionable

(
    results
    .select("helpful_approaches")
    .print(pretty_labels = {"answer.helpful_approaches": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

0

The representative, Alex, demonstrated several helpful aspects in their approach: 1. **Understanding and Empathy**: Alex showed understanding and empathy towards Jamie's concerns, especially regarding the complexity of setting up payment systems and the fear of getting stuck. This helped build trust and made Jamie feel more comfortable discussing their needs. 2. **Clear Communication**: Alex communicated the features of Quick in a clear and straightforward manner, highlighting relevant features like subscriptions, multiple currencies, and fraud detection (Shield) without overwhelming Jamie with technical jargon. 3. **Addressing Concerns**: Alex proactively addressed Jamie's concerns about setup complexity and fraud detection, assuring them of the ease of use and built-in security features, which helped alleviate Jamie's worries. 4. **Providing Solutions**: The representative offered specific solutions to Jamie's needs, such as handling international payments, providing invoicing options, and offering a sandbox account for testing, which aligned well with Jamie's requirements. 5. **Transparency on Costs**: Alex was transparent about the fees associated with Quick, providing a clear breakdown of transaction costs and mentioning potential discounts for high volumes, which is important for building trust and setting clear expectations. 6. **Follow-Up Plan**: By offering to send guides and set up a sandbox account, and suggesting a follow-up meeting, Alex ensured that Jamie would have ongoing support and resources to explore Quick further, demonstrating a commitment to customer success.

(
    results
    .select("inadequate_explanation")
    .print(pretty_labels = {"answer.inadequate_explanation": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

0

Yes, there are a few areas where more detailed explanations could have been beneficial: 1. **Subscription Setup**: While Alex mentioned that Quick can handle subscriptions, there was no detailed explanation of how Jamie could set up different subscription levels. Providing more information or examples on how to configure these levels would have been helpful. 2. **International Payments**: Alex briefly mentioned that Quick handles multiple currencies and automatic conversion, but didn't explain if there are any additional fees or processes involved for international transactions. Clarifying this could help Jamie better understand the implications of serving international users. 3. **Fraud Detection (Shield)**: Although Alex explained that Shield is on by default, it might have been useful to provide a bit more context on how it works, especially since Jamie expressed unfamiliarity with fraud detection. A simple example of how Shield identifies fraud could have added value. 4. **Setup Process**: Jamie expressed concern about the complexity of setting up the system. While Alex reassured Jamie about the availability of guides and support, a brief overview of the initial steps in the setup process could have helped alleviate Jamie's concerns further. 5. **Discounts for High Volume**: Alex mentioned discounts for higher volumes but didn’t specify what qualifies as "a lot of payments" or what the discount rates might be. Providing some threshold numbers or examples could offer Jamie a clearer picture of potential cost savings. 6. **Invoicing Details**: Although Alex confirmed that Quick Invoicing is available, more information on how it integrates with the rest of the system or any customization options could have been beneficial, especially since Jamie mentioned customer requests for invoices.

(
    results
    .select("improvements")
    .print(pretty_labels = {"answer.improvements": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

0

The representative, Alex, did a good job addressing Jamie's concerns and explaining the features of Quick's payment processing system. However, there are a few areas where the explanation could be improved: 1. **Tailored Explanation**: Alex could have asked more specific questions about Jamie's business model and customer base to tailor the explanation of how Quick's features would specifically benefit Jamie's website. This would make the conversation more relevant and engaging. 2. **Technical Jargon**: While Alex did a good job of keeping the language simple, further simplifying technical terms or providing analogies could help Jamie, who is not a developer, better understand the setup process. 3. **Case Studies or Examples**: Providing examples or case studies of similar businesses that successfully implemented Quick could help Jamie visualize the benefits and ease of integration. 4. **Demonstration Offer**: Offering a live demonstration or walkthrough of the setup process could alleviate Jamie's concerns about the complexity of the integration. 5. **Clearer Fee Structure Explanation**: While Alex mentioned the transaction fees, providing a more comprehensive breakdown of potential costs, including any hidden fees or costs associated with international transactions, would give Jamie a clearer picture. 6. **Highlighting Support**: Although Alex mentioned the support team, emphasizing the availability of personalized support, such as onboarding assistance or a dedicated account manager, could further reassure Jamie. 7. **Security Features**: While Alex briefly mentioned the Shield feature, a more detailed explanation of how it works and its benefits could enhance Jamie's understanding of Quick's security measures.

(
    results
    .select("confusing")
    .print(pretty_labels = {"answer.confusing": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

0

The conversation was generally clear and straightforward, but there are a few areas where additional clarity could enhance understanding: 1. **Setup Process**: While Alex reassures Jamie about the ease of setup, it might be beneficial to provide a bit more detail on what the setup process actually entails. Mentioning specific plugins or integration steps could help Jamie feel more confident about the process. 2. **International Payments**: Alex mentions that Quick handles multiple currencies and can automatically handle conversions, but it might be helpful to clarify if there are any additional fees for currency conversion or if there are any specific limitations or requirements for international transactions. 3. **Fraud Detection**: Alex explains that Shield is built-in and uses machine learning, but a brief explanation of how it works or examples of what it detects could provide more reassurance to Jamie, who admits to not knowing much about fraud detection. 4. **Fees and Discounts**: While the transaction fee structure is explained, the mention of discounts for higher volumes is vague. Providing a threshold or example of what constitutes a "higher volume" could help Jamie understand when they might benefit from these discounts. 5. **Invoicing Details**: Alex mentions Quick Invoicing but doesn't elaborate on how it integrates with the existing system or if there are any additional costs associated with its use. More details here could help Jamie assess its value for their needs.

(
    results
    .select("confident")
    .print(pretty_labels = {"answer.confident": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

0

To help the customer feel more confident in the representative's offerings, the following additional information could have been beneficial: 1. **Customer Success Stories or Testimonials:** Sharing examples of similar businesses that successfully implemented Quick and experienced positive outcomes could provide reassurance and credibility. 2. **Detailed Setup Process:** Providing a brief overview of the setup process, perhaps mentioning average setup times or common challenges and how they are addressed, could help alleviate concerns about complexity. 3. **Support Availability:** More specific information about the level and availability of customer support, such as 24/7 support or dedicated account managers, could reassure the customer that help is readily available. 4. **Security Certifications and Compliance:** Mentioning any industry-standard security certifications or compliance with regulations (e.g., PCI DSS) could enhance trust in the platform's security features. 5. **Case Studies on Fraud Detection:** Providing more detail or examples of how the Shield feature has successfully prevented fraud in the past would give more confidence in its effectiveness. 6. **Demo or Walkthrough Offer:** Offering a live demo or a walkthrough session could help the customer visualize how the platform works and how it can be integrated into their existing system. 7. **Flexible Pricing Options:** Mentioning any available flexible pricing models or introductory offers could make the cost aspect more appealing. 8. **Integration with Existing Tools:** Information on how Quick integrates with other tools or platforms that the customer might be using could highlight the ease of adoption.

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

Here we demonstrate how to post this notebook:

from edsl import Notebook

notebook = Notebook(path = "analyze_customer_call.ipynb")

info = notebook.push(description = "Analyzing a customer call", visibility = "public")
info

{'description': 'Analyzing a customer call',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/9e4a79c5-a868-4e78-9b82-23625a1cd5ac',
 'uuid': '9e4a79c5-a868-4e78-9b82-23625a1cd5ac',
 'version': '0.1.43.dev1',
 'visibility': 'public'}

Updating content at the Coop:

notebook = Notebook(path = "analyze_customer_call.ipynb") # resave

notebook.patch(uuid = info["uuid"], value = notebook)

{'status': 'success'}