Analyzing a transcript

This notebook provides example EDSL code for analyzing a transcript using AI. In the steps below we construct a survey of questions about a transcript of a fictional call between a business development representative and a potential customer, and then use a language model to answer the questions in the persona of an business development expert delivering feedback on the call. The code can easily be modified for purposes of analyzing other textual content, images, PDFs, websites, tables and other data types with diverse personas and many popular language models.

EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Constructing questions

We start by importing the tools we will use and constructing questions that we want to ask about the contents of the call. EDSL provides many common question types that we can choose from based on the form of the response that we want to get back from language model: free text, multiple choice, etc.

from edsl import QuestionMultipleChoice, QuestionLinearScale, QuestionFreeText, Survey, Agent, AgentList, Model

q1 = QuestionMultipleChoice(
    question_name = "understanding",
    question_text = "How would you rate the representative's understanding of the customer's needs?",
    question_options = ["Excellent", "Good", "Fair", "Poor"]
)
q2 = QuestionMultipleChoice(
    question_name = "explain_features",
    question_text = "How well did the representative explain the features relevant to the customer's business?",
    question_options = ["Very clearly", "Somewhat clearly", "Unclear", "Not explained"]
)
q3 = QuestionMultipleChoice(
    question_name = "address_concerns",
    question_text = "How well did the representative address the questions and concerns of the customer?",
    question_options = ["Very effectively", "Mostly effectively", "Somewhat effectively", "Not effectively"]
)
q4 = QuestionMultipleChoice(
    question_name = "product_knowledge",
    question_text = "How would you rate the representative's knowledge of the products and services discussed?",
    question_options = ["Excellent", "Good", "Fair", "Poor"]
)
q5 = QuestionMultipleChoice(
    question_name = "attentive",
    question_text = "Did the representative appear attentive to the customer's needs and concerns?",
    question_options = ["Very attentive", "Mostly attentive", "Somewhat attentive", "Not attentive"]
)
q6 = QuestionLinearScale(
    question_name = "likely_sale",
    question_text = "On a scale from 1 to 10, how likely is the customer to use the products and servies discussed?",
    question_options = [1,2,3,4,5,6,7,8,9,10]
)
q7 = QuestionMultipleChoice(
    question_name = "next_steps",
    question_text = "Did the representative offer relevant next steps for implementation or testing products?",
    question_options = ["Yes, very clear and actionable", "Somewhat clear", "Not clear", "No next steps offered"]
)
q8 = QuestionFreeText(
    question_name = "helpful_approaches",
    question_text = "What aspects of the representative’s approach were most helpful to the customer?"
)
q9 = QuestionFreeText(
    question_name = "inadequate_explanation",
    question_text = "Were any features or details not adequately explained?"
)
q10 = QuestionFreeText(
    question_name = "improvements",
    question_text = "How could the representative have improved their explanation of the products?"
)
q11 = QuestionFreeText(
    question_name = "confusing",
    question_text = "Was there anything about the conversation that seemed unclear or confusing?"
)
q12 = QuestionFreeText(
    question_name = "confident",
    question_text = "What additional information could have helped the customer feel more confident in the representative's offerings?"
)

We combine the questions in a survey in order to administer them together:

survey = Survey([q1, q2, q3, q4, q5, q6, q7, q8, q9, q10, q11, q12])

Selecting content for review

Next we import the text of the call that we want to give to the model in answering the questions. Here we use a mock transcript drafted by chatGPT:

transcript = """
Alex (Quick Representative):
Hi, Jamie! Thanks so much for taking the time to chat today. How’s everything going with your website?

Jamie (Customer):
Hi, Alex. It’s going okay, I think. We’re getting some users, but honestly, I’m just trying to figure out the best way to manage payments. It’s all new to me.

Alex:
Got it, no worries! Quick can really help simplify things for you. Our platform allows you to accept payments easily and has a subscription option, which I think you mentioned you were interested in, right?

Jamie:
Yeah, we’re thinking about offering different subscription levels for our software, but I’m not sure how to set it all up. And we have some international users, so that’s been a bit confusing too.

Alex:
Right, right. So, with Quick, you can definitely do subscriptions, and it handles multiple currencies, too. For example, if you have a customer in Europe, Quick can automatically handle the conversion. It’s designed to make that part easy, so you don’t need to worry too much about different currencies.

Jamie:
That sounds helpful. But I’m a bit concerned about the setup. I’m not a developer, so I’m worried this might be too complicated.

Alex:
Oh, yeah, I totally understand! Quick’s setup is meant to be as easy as possible. We have plugins and a lot of guides to walk you through it. I’d say that most people get it up and running without too much trouble. And if you do need help, our support team is available.

Jamie:
Okay, that’s a relief. I’ve heard some stories about people getting stuck for weeks with payment systems. I don’t want that to happen.

Alex:
I get that completely. We try to make it as straightforward as possible. Also, if you’re worried about fraud or security, Quick has a feature called Shield that helps detect fraud. It’s built in, so it’s automatically protecting you from day one.

Jamie:
Oh, that sounds good. I don’t know much about fraud detection. Is it something I’d have to set up?

Alex:
Nope! Shield is on by default, so you don’t have to configure anything special unless you want to. It uses machine learning to detect unusual activity based on data from many transactions, which helps catch fraudulent transactions without much work on your end.

Jamie:
That’s nice to know. What about fees? I’d like to have a clear idea of what this will cost.

Alex:
Absolutely. So, Quick charges 2.9% plus 30 cents per transaction. For subscriptions, there’s no extra fee beyond the normal transaction cost. Also, if you start processing a lot of payments, we do offer some discounts, but those kick in at a higher volume.

Jamie:
Alright, that makes sense. And what about invoicing? A few customers have asked about getting invoices instead of just a credit card charge.

Alex:
Yes, we can do that too. With Quick Invoicing, you can send out one-time or recurring invoices, which is nice if you have larger clients who prefer an invoice. It’s all within the Quick system, so it integrates well.

Jamie:
Good to know. I think this covers most of my questions. It sounds like Quick could be a good fit. I’d just need to make sure I’m able to actually set it up without a ton of back-and-forth.

Alex:
I totally understand, Jamie. I’ll send over some guides and set you up with a sandbox account to test things out. Let’s reconnect in a couple of weeks to go over any questions you might have.

Jamie:
Sounds good, thanks, Alex. Looking forward to trying it out.

Alex:
Great! Thanks for your time, Jamie. I’ll follow up soon. Have a good one!
"""

AI agent personas and instructions

Here we create a persona and instructions for an AI agent that we will give a model to reference in answering questions. We include the transcript of the customer call in the agent instructions. (We could also include it directly in question texts; learn more about using questions with multiple pieces of content with Scenario objects.)

agent = Agent(
        traits = {"persona": "You are an expert business development representative with many years of experience at diverse companies."},
        instruction = """
        You are being asked to review the following transcript of a business development call between a representative of a payments
        processing company and a potential customer and provide critical feedback.
        Transcript:
        """ + transcript
)

Selecting a model

EDSL works with many popular language models that we can select to generate responses to the questions. Here we specify that we want to use OpenAI’s GPT-4o:

model = Model("gpt-4o")

Running the survey

Next we add the agents and model to the survey and run it:

results = survey.by(agent).by(model).run()

▼ Remote Job Log (2024-12-14 10:14:11)

Remote inference activated. Sending job to server...

Your survey is running at the Expected Parrot server...

Job sent to server. (Job uuid=83ea1bff-f820-4ff5-86b7-8260c9f99325).

Job details are available at your Coop account https://www.expectedparrot.com/home/remote-inferencehttps://www.expectedparrot.com/home/remote-inference

View job progress here: https://www.expectedparrot.com/home/remote-job-progress/83ea1bff-f820-4ff5-86b7-8260c9f99325

Job status: queued - last update: 2024-12-14 10:13:51 AM

Job status: queued - last update: 2024-12-14 10:13:54 AM

Job status: running - last update: 2024-12-14 10:13:57 AM

Job status: running - last update: 2024-12-14 10:14:01 AM

Job status: running - last update: 2024-12-14 10:14:04 AM

Job status: running - last update: 2024-12-14 10:14:07 AM

Job completed and Results stored on Coop: https://www.expectedparrot.com/content/906e6465-d33a-402a-99b1-4b5658bbefad

This generates a formatted dataset of results that we can inspect with built-in methods for analyzing responses. Here we print them in some tables together with information about each persona that was used:

(
    results
    .select("persona", "understanding", "explain_features", "address_concerns", "product_knowledge",
            # "attentive", "likely_sale", "next_steps"
           )
    .print(
        pretty_labels = {
            "agent.persona": "Persona",
            "answer.understanding": q1.question_text,
            "answer.explain_features": q2.question_text,
            "answer.address_concerns": q3.question_text,
            "answer.product_knowledge": q4.question_text,
            # "answer.attentive": q5.question_text,
            # "answer.likely_sale": q6.question_text,
            # "answer.next_steps": q7.question_text
        }
    )
)

Persona	How would you rate the representative's understanding of the customer's needs?	How well did the representative explain the features relevant to the customer's business?	How well did the representative address the questions and concerns of the customer?	How would you rate the representative's knowledge of the products and services discussed?
You are an expert business development representative with many years of experience at diverse companies.	Good	Somewhat clearly	Mostly effectively	Good

(
    results
    .select("persona",
            # "understanding", "explain_features", "address_concerns", "product_knowledge",
            "attentive", "likely_sale", "next_steps"
           )
    .print(
        pretty_labels = {
            "agent.persona": "Persona",
            # "answer.understanding": q1.question_text,
            # "answer.explain_features": q2.question_text,
            # "answer.address_concerns": q3.question_text,
            # "answer.product_knowledge": q4.question_text,
            "answer.attentive": q5.question_text,
            "answer.likely_sale": q6.question_text,
            "answer.next_steps": q7.question_text
        },
        format = "rich"
    )
)

Persona	Did the representative appear attentive to the customer's needs and concerns?	On a scale from 1 to 10, how likely is the customer to use the products and servies discussed?	Did the representative offer relevant next steps for implementation or testing products?
You are an expert business development representative with many years of experience at diverse companies.	Mostly attentive	8	Yes, very clear and actionable

(
    results
    .select("helpful_approaches")
    .print(pretty_labels = {"answer.helpful_approaches": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

The representative, Alex, demonstrated several helpful aspects in their approach: 1. **Understanding and Empathy**: Alex started by acknowledging Jamie's situation and concerns, which helped to build rapport and trust. This empathetic approach reassured Jamie that their needs and worries were understood. 2. **Clear Explanation of Features**: Alex provided a straightforward explanation of how Quick’s features, such as handling multiple currencies and offering subscription options, could address Jamie's needs. This helped Jamie see the direct benefits of using Quick. 3. **Addressing Concerns**: Alex anticipated and addressed Jamie's concerns about setup complexity and fraud detection. By explaining that the setup is user-friendly and that fraud protection is automatic, Alex alleviated Jamie’s worries. 4. **Support and Resources**: Alex emphasized the availability of support and resources, including guides and a support team, which reassured Jamie that help would be available if needed. 5. **Transparency on Fees**: Alex provided a clear breakdown of the costs associated with using Quick, which is crucial for building trust and ensuring there are no surprises later on. 6. **Solution for Additional Needs**: Alex responded to Jamie's inquiry about invoicing by introducing Quick Invoicing, showing that the platform could meet additional business requirements. 7. **Follow-Up Plan**: By offering to send guides and set up a sandbox account, and suggesting a follow-up meeting, Alex demonstrated a commitment to ensuring Jamie’s successful onboarding and long-term satisfaction.

(
    results
    .select("inadequate_explanation")
    .print(pretty_labels = {"answer.inadequate_explanation": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

Yes, there were a few areas where the explanation could have been more detailed: 1. **Subscription Setup**: While Alex mentioned that Quick can handle subscriptions, there was no detailed explanation of how the subscription setup process works or what specific features are available for managing different subscription levels. Jamie expressed concern about setting this up, and a more detailed walkthrough or specific examples might have been helpful. 2. **International Payments**: Alex briefly mentioned that Quick handles multiple currencies and automatic conversion, but did not go into detail about how international payments are processed, any potential fees associated with currency conversion, or how Quick ensures compliance with international regulations. 3. **Fraud Detection**: While Alex mentioned that Quick has a feature called Shield for fraud detection, there was no explanation of how effective it is, any specific technologies it uses, or any success stories or statistics that might reassure Jamie about its reliability. 4. **Fees**: Although Alex provided the basic fee structure, there was no mention of any potential hidden fees, setup costs, or how discounts for high-volume processing are structured. More transparency here could help build trust. 5. **Invoicing**: Alex mentioned Quick Invoicing but did not elaborate on how it integrates with the rest of the system, what customization options are available, or how it handles different payment methods.

(
    results
    .select("improvements")
    .print(pretty_labels = {"answer.improvements": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

The representative, Alex, did a good job of addressing Jamie's concerns and providing an overview of Quick's offerings. However, there are a few areas where the explanation could be improved: 1. **Customization and Examples**: Alex could have provided more specific examples or case studies of similar businesses that successfully implemented Quick. This would help Jamie visualize how Quick can be tailored to their specific needs. 2. **Technical Jargon**: While Alex did a decent job of keeping the language accessible, they could have further simplified the explanation of technical aspects like currency conversion and fraud detection, possibly by using analogies or more relatable terms. 3. **Setup Process**: Although Alex mentioned plugins and guides, they could have elaborated on the setup process by outlining the key steps or providing a brief walkthrough of what Jamie might expect. 4. **Emphasizing Support**: Alex mentioned the support team but could have emphasized the availability of live support or onboarding assistance to reassure Jamie about potential setup difficulties. 5. **Addressing Specific Needs**: Alex could have asked more questions to better understand Jamie's specific business model and tailor the explanation of features like subscriptions and invoicing accordingly. 6. **Fee Transparency**: While Alex provided the basic fee structure, they could have offered more detail on any potential hidden costs or additional services that might incur fees, ensuring full transparency.

(
    results
    .select("confusing")
    .print(pretty_labels = {"answer.confusing": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

The conversation was generally clear, but there are a few areas that could benefit from further clarification or elaboration: 1. **Setup Process**: While Alex mentioned that Quick's setup is meant to be easy with plugins and guides, Jamie expressed concern about not being a developer. It might have been helpful for Alex to provide more specific examples of the setup process or offer to connect Jamie with a support specialist for a walkthrough. 2. **International Users and Currency Conversion**: Alex briefly mentioned that Quick handles multiple currencies and automatically manages conversions, but it may have been beneficial to explain how this works in a bit more detail, especially if Jamie has more specific concerns about international transactions. 3. **Fraud Detection**: Jamie expressed a lack of knowledge about fraud detection. While Alex explained that Shield is automatic and uses machine learning, a quick example or a brief explanation of how it benefits Jamie's business could have been more reassuring. 4. **Fees and Discounts**: Alex mentioned the transaction fees and potential discounts for high volumes, but it might have been helpful to specify what "higher volume" means or provide a range to give Jamie a clearer idea of when discounts might apply. 5. **Invoicing Feature**: While Alex confirmed that Quick can handle invoicing, providing a bit more detail on how invoicing is integrated into the system or how easy it is to use might have strengthened Jamie's confidence in the feature.

(
    results
    .select("confident")
    .print(pretty_labels = {"answer.confident": q8.question_text})
)

What aspects of the representative’s approach were most helpful to the customer?

To help Jamie feel more confident in Quick's offerings, Alex could have provided the following additional information: 1. **Case Studies or Testimonials**: Sharing success stories or testimonials from similar businesses that have successfully implemented Quick could provide Jamie with reassurance about the platform's reliability and effectiveness. 2. **Detailed Setup Process**: A step-by-step overview of the setup process, perhaps with a brief demonstration or a video tutorial, could help alleviate Jamie's concerns about the complexity of implementation. 3. **Support Availability**: More specific details about the availability and responsiveness of the support team, such as 24/7 support or dedicated account managers, could reassure Jamie about getting timely help if needed. 4. **Security and Compliance**: While Alex mentioned the fraud detection feature, providing more information about Quick's overall security measures and compliance with industry standards (e.g., PCI DSS) could help build trust. 5. **Comparative Advantage**: Highlighting what sets Quick apart from competitors in terms of features, pricing, or customer service could help Jamie understand why Quick is the best choice for their needs. 6. **Customization Options**: Information on how customizable the platform is to meet specific business needs could be valuable, especially if Jamie has unique requirements. 7. **Long-term Scalability**: Discussing how Quick can scale with Jamie's business as it grows, including any additional features or services that could be beneficial in the future, might be reassuring.

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

Here we demonstrate how to post this notebook:

from edsl import Notebook

notebook = Notebook(path = "analyze_customer_call.ipynb")

info = notebook.push(description = "Analyzing a customer call", visibility = "public")
info

{'description': 'Analyzing a customer call',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/3ed06978-ae82-4185-ba80-d86c4d9e940d',
 'uuid': '3ed06978-ae82-4185-ba80-d86c4d9e940d',
 'version': '0.1.39.dev1',
 'visibility': 'public'}

Updating content at the Coop:

notebook = Notebook(path = "analyze_customer_call.ipynb") # resave

notebook.patch(uuid = info["uuid"], value = notebook)

{'status': 'success'}