Analyzing course evaluations

This notebook provides sample EDSL code for using a language model to analyze course evaluations. The analysis is designed as a survey of questions about a set of evaluations that we prompt an AI agent to answer, using a language model to generate the responses as a dataset.

EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please see instructions on getting started, and tips and tutorials at our documentation page.

Create questions

We start by creating questions about a set of course evaluations for an agent to answer. EDSL comes with a variety of question types that we can choose from based on the form of the response that we want to get back from a model (multiple choice, linear scale, checkbox, free text, numerical, dictionary, etc.). We can use a {{ placeholder }} in the question texts to parameterize them with each evaluation. This allows us to create different “scenarios” of the questions that we can administer at once.

We start by importing some question types and composing questions in the relevant templates (see examples of all types in the docs):

[1]:

from edsl import QuestionList, QuestionMultipleChoice

q_sentiment = QuestionMultipleChoice(
    question_name="sentiment",
    question_text="What is the overall sentiment of this evaluation: {{ scenario.evaluation }}",
    question_options=["Positive", "Neutral", "Negative"],
)

q_themes = QuestionList(
    question_name="themes",
    question_text="Summarize the key points of this evaluation: {{ scenario.evaluation }}",
    max_list_items=3,  # Optional
)

q_improvements = QuestionList(
    question_name="improvements",
    question_text="Identify areas for improvement based on this evaluation: {{ scenario.evaluation }}",
    max_list_items=3,
)

Construct a survey

Next we combine our questions into a survey. This allows us to administer the questions asynchronously (by default), or according to any desired survey logic or rules that we want to add, such as skip/stop rules or giving an agent “memories” of other questions in the survey. Here we create a simple asynchronous survey by passing the list of questions to a Survey object:

[2]:

from edsl import Survey

survey = Survey(questions=[q_sentiment, q_themes, q_improvements])

survey

[2]:

Survey # questions: 3; question_name list: ['sentiment', 'themes', 'improvements'];

	question_options	question_name	question_text	question_type	max_list_items
0	['Positive', 'Neutral', 'Negative']	sentiment	What is the overall sentiment of this evaluation: {{ scenario.evaluation }}	multiple_choice	nan
1	nan	themes	Summarize the key points of this evaluation: {{ scenario.evaluation }}	list	3.000000
2	nan	improvements	Identify areas for improvement based on this evaluation: {{ scenario.evaluation }}	list	3.000000

Add data to the questions

Next we identify the data to be analyzed and create Scenario objects for it. This allows us to efficiently readminister questions with different data. We can create data using EDSL or import it from other sources (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). Here we use some mock evaluations for an Econ 101 course stored as a list of texts:

[3]:

evaluations = [
    "I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings.",
    "This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest.",
    "Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging.",
    "As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times.",
    "I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes.",
    "The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions.",
    "This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies.",
    "I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics.",
    "The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles.",
    "This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material.",
]

[4]:

from edsl import ScenarioList

scenarios = ScenarioList.from_source("list", "evaluation", evaluations)

Design AI agents

Next we design agents with relevant traits and personas for a language model to use in answering the questions. This can be useful if we want to compare responses among different audiences. We do this by passing dictionaries of traits to Agent objects. We can also choose whether to give an agent additional instructions for answering the survey (independent of individual question texts). Please see documentation for more details and example code for creating agents to use with surveys.

Here we create a persona for the professor of the course and pass it some special instructions:

[5]:

from edsl import Agent

persona = "You are a professor reviewing student evaluations for your recent Econ 101 course."
instruction = "Be very specific and constructive in providing feedback and suggestions."

agent = Agent(traits={"persona": persona}, instruction=instruction)

Select language models

EDSL works with many popular language models that we can use to generate responses for our survey. A current list of available models, performance and pricing is available here.

We select models to use with a survey by creating Model objects for them (if no model is specified, gpt-4o is currently used by default):

[6]:

from edsl import Model

model = Model("gemini-1.5-flash", service_name = "google")
model

[6]:

gemini-1.5-flash

	key	value
0	model	gemini-1.5-flash
1	parameters:temperature	0.500000
2	parameters:topP	1
3	parameters:topK	1
4	parameters:maxOutputTokens	2048
5	parameters:stopSequences	[]
6	inference_service	google

Run the survey

Next we add the scenarios and agent to the survey, and then run it with the specified model. This will generate a dataset of Results that we can store and begin analyzing:

[7]:

results = survey.by(scenarios).by(agent).by(model).run()

⌃ Job Status 🦜

Completed (10 completed, 0 failed)

Job Links

Results

Progress Report

Content

Remote Jobs

Remote Cache

Identifiers

Results UUID:

52cea850...14bf

Use Results.pull(uuid) to fetch results.

Job UUID:

a80f46ac...7beb

Use Jobs.pull(uuid) to fetch job.