Analyzing course evaluations
This notebook provides sample EDSL code for using a language model to analyze course evaluations. The analysis is designed as a survey of questions about the evaluations that we prompt an AI agent to answer, using a language model to generate the responses as a dataset.
EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.
Create questions
We start by creating questions about a set of course evaluations for an agent to answer. EDSL comes with a variety of question types that we can choose from based on the form of the response that we want to get back from a model (multiple choice, linear scale, checkbox, free text, etc.). We can use a {{ placeholder }} in the question texts to parameterize them with each evaluation. This allows us to create different “scenarios”
of the questions that we can administer at once.
We start by importing some question types and composing questions in the relevant templates (see examples of all types in the docs):
[1]:
from edsl import QuestionList, QuestionMultipleChoice
[2]:
q_sentiment = QuestionMultipleChoice(
question_name="sentiment",
question_text="What is the overall sentiment of this evaluation: {{ evaluation }}",
question_options=["Positive", "Neutral", "Negative"],
)
q_themes = QuestionList(
question_name="themes",
question_text="Summarize the key points of this evaluation: {{ evaluation }}",
max_list_items=3, # Optional
)
q_improvements = QuestionList(
question_name="improvements",
question_text="Identify areas for improvement based on this evaluation: {{ evaluation }}",
max_list_items=3,
)
Construct a survey
Next we combine our questions into a survey. This allows us to administer the questions asynchronously (by default), or according to any desired survey logic or rules that we want to add, such as skip/stop rules or giving an agent “memories” of other questions in the survey. Here we create a simple asynchronous survey by passing the list of questions to a Survey object:
[3]:
from edsl import Survey
survey = Survey(questions=[q_sentiment, q_themes, q_improvements])
Select data for review
Next we identify the data to be analyzed. Here we use some mock evaluations for an Econ 101 course stored as a list of texts:
[4]:
evaluations = [
"I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings.",
"This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest.",
"Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging.",
"As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times.",
"I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes.",
"The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions.",
"This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies.",
"I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics.",
"The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles.",
"This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material.",
]
Add data to the questions
Next we create a ScenarioList with a Scenario containing a key/value for each evaluation that we will add to the questions when we run the survey. EDSL provides methods for generating scenarios from many data sources (PDFs, CSVs, images, tables, dicts, etc.); here we import a list and match the key to our question texts placeholder:
[5]:
from edsl import ScenarioList
scenarios = ScenarioList.from_list("evaluation", evaluations)
Design AI agents
Next we design agents with relevant traits and personas for a language model to use in answering the questions. This can be useful if we want to compare responses among different audiences. We do this by passing dictionaries of traits to Agent objects. We can also choose whether to give an agent additional instructions for answering the survey (independent of individual question texts). Please see documentation for more details and example code for creating agents to use with
surveys.
Here we create a persona for the professor of the course and pass it some special instructions:
[6]:
from edsl import Agent
persona = "You are a professor reviewing student evaluations for your recent Econ 101 course."
instruction = "Be very specific and constructive in providing feedback and suggestions."
agent = Agent(traits={"persona": persona}, instruction=instruction)
Select language models
EDSL works with many popular language models that we can use to generate responses for our survey. We can see a current list of all available models:
[7]:
from edsl import Model
[8]:
# Model.available() # uncomment and run to see the list
We select models to use with a survey by creating Model objects for them. The default model is GPT 4 Preview, meaning that EDSL will use it to run our survey if we do not specify a different model. Here’s we’ll specify that GPT 4o should be used:
[9]:
model = Model("gpt-4o")
Run the survey
Next we add the scenarios and agent to the survey, and then run it with the specified model. This will generate a dataset of Results that we can store and begin analyzing:
[10]:
results = survey.by(scenarios).by(agent).by(model).run(raise_validation_errors=True)
Analyzing results
EDSL comes with built-in methods for analyzing results in data tables, dataframes, SQL queries and other formats. We can print a list of all the components that can be accessed. Here we will just look at the first 5:
[11]:
results.columns[:5]
[11]:
['agent.agent_instruction',
'agent.agent_name',
'agent.persona',
'answer.improvements',
'answer.sentiment']
For example, we can transform the results into a dataframe:
[12]:
df = results.to_pandas()
df.head()
[12]:
| answer.improvements | answer.sentiment | answer.themes | scenario.evaluation | agent.agent_name | agent.persona | agent.agent_instruction | model.max_tokens | model.temperature | model.frequency_penalty | ... | question_options.themes_question_options | question_options.improvements_question_options | question_options.sentiment_question_options | question_type.sentiment_question_type | question_type.improvements_question_type | question_type.themes_question_type | comment.k_comment | generated_tokens.improvements_generated_tokens | generated_tokens.sentiment_generated_tokens | generated_tokens.themes_generated_tokens | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | ['Incorporate more real-world examples', 'Rela... | Negative | ['Material felt dry', 'Difficult to connect wi... | This class was a struggle for me. The material... | Agent_0 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | 1000 | 0.5 | 0 | ... | NaN | NaN | ['Positive', 'Neutral', 'Negative'] | multiple_choice | list | list | These suggestions are aimed at making the mate... | ["Incorporate more real-world examples", "Rela... | Negative\n\nThe student describes the class as... | ["Material felt dry", "Difficult to connect wi... |
| 1 | ['Incorporate more visual aids into lectures',... | Neutral | ['Well-organized course content', 'Monotonous ... | The course content was well-organized, but the... | Agent_0 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | 1000 | 0.5 | 0 | ... | NaN | NaN | ['Positive', 'Neutral', 'Negative'] | multiple_choice | list | list | These suggestions directly address the feedbac... | ["Incorporate more visual aids into lectures",... | Neutral\n\nThe evaluation acknowledges positiv... | ["Well-organized course content", "Monotonous ... |
| 2 | ['Provide more diverse examples in lectures', ... | Positive | ['Enthusiastic professor', 'Helpful during off... | Excellent introductory course! The professor w... | Agent_0 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | 1000 | 0.5 | 0 | ... | NaN | NaN | ['Positive', 'Neutral', 'Negative'] | multiple_choice | list | list | The evaluation is very positive, but to furthe... | ["Provide more diverse examples in lectures", ... | Positive\n\nThe evaluation highlights several ... | ["Enthusiastic professor", "Helpful during off... |
| 3 | ['Incorporate more real-world economic example... | Neutral | ['Appreciated analytical rigor', 'Desire for m... | As someone with a strong background in math, I... | Agent_0 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | 1000 | 0.5 | 0 | ... | NaN | NaN | ['Positive', 'Neutral', 'Negative'] | multiple_choice | list | list | These suggestions aim to bridge the gap betwee... | ["Incorporate more real-world economic example... | Neutral\n\nThe student appreciates the analyti... | ["Appreciated analytical rigor", "Desire for m... |
| 4 | ['Increase opportunities for student participa... | Positive | ['Perfect mix of theory and case studies', 'Fa... | This was my favorite class this semester! The ... | Agent_0 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | 1000 | 0.5 | 0 | ... | NaN | NaN | ['Positive', 'Neutral', 'Negative'] | multiple_choice | list | list | These suggestions are aimed at enhancing the o... | ["Increase opportunities for student participa... | Positive\n\nThe student expresses high satisfa... | ["Perfect mix of theory and case studies", "Fa... |
5 rows × 44 columns
Here we select just the responses to the questions and display them in a table:
[13]:
results.select("sentiment", "themes", "themes_generated_tokens", "improvements").print(format="rich")
┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ answer ┃ answer ┃ generated_tokens ┃ answer ┃ ┃ .sentiment ┃ .themes ┃ .themes_generated_tokens ┃ .improvements ┃ ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Negative │ ['Material felt dry', │ ["Material felt dry", │ ['Incorporate more real-world │ │ │ 'Difficult to connect with │ "Difficult to connect with │ examples', 'Relate material to │ │ │ real-world applications', 'More │ real-world applications", │ current events', 'Increase │ │ │ current event examples needed'] │ "More current event examples │ engagement with interactive │ │ │ │ needed"] │ activities'] │ │ │ │ │ │ │ │ │ The student found the material │ │ │ │ │ unengaging, struggled to see │ │ │ │ │ its relevance to real-world │ │ │ │ │ situations, and suggested that │ │ │ │ │ incorporating examples from │ │ │ │ │ current events could enhance │ │ │ │ │ interest and understanding. │ │ ├────────────┼─────────────────────────────────┼────────────────────────────────┼─────────────────────────────────┤ │ Neutral │ ['Well-organized course │ ["Well-organized course │ ['Incorporate more visual aids │ │ │ content', 'Monotonous and │ content", "Monotonous and │ into lectures', 'Invite guest │ │ │ hard-to-follow lectures', │ hard-to-follow lectures", │ lecturers from industry │ │ │ 'Suggestion for more visual │ "Suggestion for more visual │ professionals', 'Enhance │ │ │ aids and guest lectures'] │ aids and guest lectures"] │ lecture delivery to make it │ │ │ │ │ more engaging'] │ │ │ │ These points capture the │ │ │ │ │ essence of the evaluation: the │ │ │ │ │ positive organization of the │ │ │ │ │ course, the negative aspect of │ │ │ │ │ the lecture delivery, and the │ │ │ │ │ constructive suggestions for │ │ │ │ │ improvement. │ │ ├────────────┼─────────────────────────────────┼────────────────────────────────┼─────────────────────────────────┤ │ Positive │ ['Enthusiastic professor', │ ["Enthusiastic professor", │ ['Provide more diverse examples │ │ │ 'Helpful during office hours', │ "Helpful during office hours", │ in lectures', 'Improve the │ │ │ 'Interactive lectures and │ "Interactive lectures and │ pacing of the course material', │ │ │ practical assignments'] │ practical assignments"] │ 'Incorporate more real-world │ │ │ │ │ case studies'] │ │ │ │ The evaluation highlights │ │ │ │ │ three main points: the │ │ │ │ │ professor's enthusiasm and │ │ │ │ │ willingness to help, the │ │ │ │ │ effectiveness of office hours, │ │ │ │ │ and the engaging nature of the │ │ │ │ │ interactive lectures and │ │ │ │ │ practical assignments. │ │ ├────────────┼─────────────────────────────────┼────────────────────────────────┼─────────────────────────────────┤ │ Neutral │ ['Appreciated analytical │ ["Appreciated analytical │ ['Incorporate more real-world │ │ │ rigor', 'Desire for more │ rigor", "Desire for more │ economic examples into │ │ │ practical discussions', 'Course │ practical discussions", │ lectures', 'Facilitate │ │ │ felt isolated from real-world │ "Course felt isolated from │ discussions on the application │ │ │ issues'] │ real-world issues"] │ of theories to current economic │ │ │ │ │ issues', 'Include case studies │ │ │ │ The student values the │ or guest speakers from │ │ │ │ analytical depth of the course │ industry'] │ │ │ │ due to their math background, │ │ │ │ │ but they expressed a need for │ │ │ │ │ more practical applications of │ │ │ │ │ the theories discussed, │ │ │ │ │ indicating a gap between the │ │ │ │ │ course content and real-world │ │ │ │ │ economic issues. │ │ ├────────────┼─────────────────────────────────┼────────────────────────────────┼─────────────────────────────────┤ │ Positive │ ['Perfect mix of theory and │ ["Perfect mix of theory and │ ['Increase opportunities for │ │ │ case studies', 'Fair exams', │ case studies", "Fair exams", │ student participation', │ │ │ 'Appreciation for diversity of │ "Appreciation for diversity of │ 'Provide additional resources │ │ │ perspectives'] │ perspectives"] │ for complex topics', 'Offer │ │ │ │ │ more real-world application │ │ │ │ These points highlight the │ projects'] │ │ │ │ student's satisfaction with │ │ │ │ │ the course structure, fairness │ │ │ │ │ of assessments, and the │ │ │ │ │ inclusion of diverse │ │ │ │ │ viewpoints, particularly in │ │ │ │ │ global economic policies. │ │ ├────────────┼─────────────────────────────────┼────────────────────────────────┼─────────────────────────────────┤ │ Negative │ ['Textbook overly complex for │ ["Textbook overly complex for │ ['Simplify textbook selection', │ │ │ an introductory course', │ an introductory course", │ 'Align lecture content with │ │ │ 'Jargon not explained in │ "Jargon not explained in │ textbook', 'Provide │ │ │ lectures', 'Need for simpler │ lectures", "Need for simpler │ supplementary explanatory │ │ │ reading materials or more │ reading materials or more │ materials'] │ │ │ explanatory lectures'] │ explanatory lectures"] │ │ │ │ │ │ │ │ │ │ These points capture the │ │ │ │ │ student's main concerns about │ │ │ │ │ the complexity of the │ │ │ │ │ textbook, the lack of jargon │ │ │ │ │ explanation in lectures, and │ │ │ │ │ the suggestion for simpler │ │ │ │ │ materials or more detailed │ │ │ │ │ lectures. │ │ ├────────────┼─────────────────────────────────┼────────────────────────────────┼─────────────────────────────────┤ │ Neutral │ ['Knowledgeable and passionate │ ["Knowledgeable and passionate │ ['Incorporate diverse │ │ │ professor', 'Course relied │ professor", "Course relied │ assessment methods', 'Include │ │ │ heavily on tests', 'Need for │ heavily on tests", "Need for │ more project-based │ │ │ more varied assignments'] │ more varied assignments"] │ assignments', 'Offer │ │ │ │ │ alternative evaluation │ │ │ │ This summary captures the main │ options'] │ │ │ │ points: the professor's │ │ │ │ │ expertise and enthusiasm, the │ │ │ │ │ heavy reliance on tests, and │ │ │ │ │ the suggestion for more │ │ │ │ │ diverse assessments to │ │ │ │ │ accommodate different learning │ │ │ │ │ styles. │ │ ├────────────┼─────────────────────────────────┼────────────────────────────────┼─────────────────────────────────┤ │ Neutral │ ['Solid introduction to │ ["Solid introduction to │ ['Incorporate more real-world │ │ │ economics', 'Heavily │ economics", "Heavily │ case studies', 'Facilitate │ │ │ theoretical focus', 'Desire for │ theoretical focus", "Desire │ class discussions on current │ │ │ more real-world application'] │ for more real-world │ economic events', 'Include │ │ │ │ application"] │ practical applications in │ │ │ │ │ assignments'] │ │ │ │ These points capture the │ │ │ │ │ student's overall positive │ │ │ │ │ impression of the course as an │ │ │ │ │ introduction, their │ │ │ │ │ observation of its theoretical │ │ │ │ │ emphasis, and their suggestion │ │ │ │ │ for incorporating more │ │ │ │ │ practical discussions to │ │ │ │ │ enhance learning. │ │ ├────────────┼─────────────────────────────────┼────────────────────────────────┼─────────────────────────────────┤ │ Positive │ ['Enjoyed the course and group │ ["Enjoyed the course and group │ ['Provide more detailed │ │ │ projects', 'Appreciated │ projects", "Appreciated │ feedback on assignments', │ │ │ applying economic concepts to │ applying economic concepts to │ 'Increase clarity on common │ │ │ real-life problems', 'Desire │ real-life problems", "Desire │ mistakes', 'Offer additional │ │ │ for more detailed feedback on │ for more detailed feedback on │ office hours for assignment │ │ │ assignments'] │ assignments"] │ reviews'] │ │ │ │ │ │ │ │ │ These points capture the │ │ │ │ │ essence of the student's │ │ │ │ │ experience, highlighting both │ │ │ │ │ the positive aspects and the │ │ │ │ │ area for improvement. │ │ ├────────────┼─────────────────────────────────┼────────────────────────────────┼─────────────────────────────────┤ │ Positive │ ['Engaging and informative │ ["Engaging and informative │ ['Adjust the pace of the │ │ │ course', 'Professor made │ course", "Professor made │ course', 'Provide additional │ │ │ complex concepts accessible', │ complex concepts accessible", │ resources for weekly readings', │ │ │ 'Pace was a bit fast, struggled │ "Pace was a bit fast, │ 'Incorporate more review │ │ │ with weekly readings'] │ struggled with weekly │ sessions'] │ │ │ │ readings"] │ │ │ │ │ │ │ │ │ │ These key points capture the │ │ │ │ │ student's overall positive │ │ │ │ │ experience with the course and │ │ │ │ │ the teaching methods, while │ │ │ │ │ also highlighting a specific │ │ │ │ │ area for improvement regarding │ │ │ │ │ the course pace and reading │ │ │ │ │ workload. │ │ └────────────┴─────────────────────────────────┴────────────────────────────────┴─────────────────────────────────┘
We can do a quick tally of the sentiments:
[14]:
results.select("sentiment").tally().print(format = "rich")
┏━━━━━━━━━━┳━━━━━━━┓ ┃ value ┃ count ┃ ┡━━━━━━━━━━╇━━━━━━━┩ │ Neutral │ 4 │ ├──────────┼───────┤ │ Positive │ 4 │ ├──────────┼───────┤ │ Negative │ 2 │ └──────────┴───────┘
We can also use pandas methods by first converting:
[15]:
df_sentiment = results.to_pandas()["answer.sentiment"]
df_sentiment.value_counts()
[15]:
answer.sentiment
Neutral 4
Positive 4
Negative 2
Name: count, dtype: int64
Use responses to construct new questions
We can use the responses to our initial questions to construct more questions about the texts. For example, we can prompt a model to condense the individual lists of themes and areas for improvement into short lists, and then use the new lists to quantify the topics across the set of evaluations.
Here we take the lists of themes in each evaluation, flatten them into a (duplicative) list, and then create a new question prompting a model to condense it for us:
[16]:
results.select("themes", "themes_generated_tokens").print(format = "rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ answer ┃ generated_tokens ┃ ┃ .themes ┃ .themes_generated_tokens ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ ['Material felt dry', 'Difficult to connect with │ ["Material felt dry", "Difficult to connect with │ │ real-world applications', 'More current event examples │ real-world applications", "More current event examples │ │ needed'] │ needed"] │ │ │ │ │ │ The student found the material unengaging, struggled │ │ │ to see its relevance to real-world situations, and │ │ │ suggested that incorporating examples from current │ │ │ events could enhance interest and understanding. │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ ['Well-organized course content', 'Monotonous and │ ["Well-organized course content", "Monotonous and │ │ hard-to-follow lectures', 'Suggestion for more visual │ hard-to-follow lectures", "Suggestion for more visual │ │ aids and guest lectures'] │ aids and guest lectures"] │ │ │ │ │ │ These points capture the essence of the evaluation: │ │ │ the positive organization of the course, the negative │ │ │ aspect of the lecture delivery, and the constructive │ │ │ suggestions for improvement. │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ ['Enthusiastic professor', 'Helpful during office │ ["Enthusiastic professor", "Helpful during office │ │ hours', 'Interactive lectures and practical │ hours", "Interactive lectures and practical │ │ assignments'] │ assignments"] │ │ │ │ │ │ The evaluation highlights three main points: the │ │ │ professor's enthusiasm and willingness to help, the │ │ │ effectiveness of office hours, and the engaging nature │ │ │ of the interactive lectures and practical assignments. │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ ['Appreciated analytical rigor', 'Desire for more │ ["Appreciated analytical rigor", "Desire for more │ │ practical discussions', 'Course felt isolated from │ practical discussions", "Course felt isolated from │ │ real-world issues'] │ real-world issues"] │ │ │ │ │ │ The student values the analytical depth of the course │ │ │ due to their math background, but they expressed a │ │ │ need for more practical applications of the theories │ │ │ discussed, indicating a gap between the course content │ │ │ and real-world economic issues. │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ ['Perfect mix of theory and case studies', 'Fair │ ["Perfect mix of theory and case studies", "Fair │ │ exams', 'Appreciation for diversity of perspectives'] │ exams", "Appreciation for diversity of perspectives"] │ │ │ │ │ │ These points highlight the student's satisfaction with │ │ │ the course structure, fairness of assessments, and the │ │ │ inclusion of diverse viewpoints, particularly in │ │ │ global economic policies. │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ ['Textbook overly complex for an introductory course', │ ["Textbook overly complex for an introductory course", │ │ 'Jargon not explained in lectures', 'Need for simpler │ "Jargon not explained in lectures", "Need for simpler │ │ reading materials or more explanatory lectures'] │ reading materials or more explanatory lectures"] │ │ │ │ │ │ These points capture the student's main concerns about │ │ │ the complexity of the textbook, the lack of jargon │ │ │ explanation in lectures, and the suggestion for │ │ │ simpler materials or more detailed lectures. │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ ['Knowledgeable and passionate professor', 'Course │ ["Knowledgeable and passionate professor", "Course │ │ relied heavily on tests', 'Need for more varied │ relied heavily on tests", "Need for more varied │ │ assignments'] │ assignments"] │ │ │ │ │ │ This summary captures the main points: the professor's │ │ │ expertise and enthusiasm, the heavy reliance on tests, │ │ │ and the suggestion for more diverse assessments to │ │ │ accommodate different learning styles. │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ ['Solid introduction to economics', 'Heavily │ ["Solid introduction to economics", "Heavily │ │ theoretical focus', 'Desire for more real-world │ theoretical focus", "Desire for more real-world │ │ application'] │ application"] │ │ │ │ │ │ These points capture the student's overall positive │ │ │ impression of the course as an introduction, their │ │ │ observation of its theoretical emphasis, and their │ │ │ suggestion for incorporating more practical │ │ │ discussions to enhance learning. │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ ['Enjoyed the course and group projects', 'Appreciated │ ["Enjoyed the course and group projects", "Appreciated │ │ applying economic concepts to real-life problems', │ applying economic concepts to real-life problems", │ │ 'Desire for more detailed feedback on assignments'] │ "Desire for more detailed feedback on assignments"] │ │ │ │ │ │ These points capture the essence of the student's │ │ │ experience, highlighting both the positive aspects and │ │ │ the area for improvement. │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ ['Engaging and informative course', 'Professor made │ ["Engaging and informative course", "Professor made │ │ complex concepts accessible', 'Pace was a bit fast, │ complex concepts accessible", "Pace was a bit fast, │ │ struggled with weekly readings'] │ struggled with weekly readings"] │ │ │ │ │ │ These key points capture the student's overall │ │ │ positive experience with the course and the teaching │ │ │ methods, while also highlighting a specific area for │ │ │ improvement regarding the course pace and reading │ │ │ workload. │ └────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────┘
[17]:
themes = results.select("themes").to_list(flatten = True)
Next we construct a question to condense the list into a new list:
[18]:
q_condensed_themes = QuestionList(
question_name="condensed_themes",
question_text="""Combine the following list of themes extracted from the evaluations
into a consolidated, non-redundant list: """
+ ", ".join(themes),
max_list_items=10,
)
Now we run the question and select the new list. Note that we can choose whether we want to use the agent for this question by not adding it to the question when we run it:
[19]:
condensed_themes = q_condensed_themes.run().select("condensed_themes").to_list()[0]
Now we can create a question to identify all the themes in the list that appear in each evaluation (our new list becomes the list of answer options):
[20]:
from edsl.questions import QuestionCheckBox
q_themes_list = QuestionCheckBox(
question_name="themes_list",
question_text="Select all of the themes that are mentioned in this evaluation: {{ evaluation }}",
question_options=condensed_themes,
)
Here we run the question and show a table listing all the themes for each evaluation in the results:
[21]:
themes_lists = q_themes_list.by(scenarios).by(agent).run()
themes_lists.select("evaluation", "themes_list").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ scenario ┃ answer ┃ ┃ .evaluation ┃ .themes_list ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ The course content was well-organized, but the │ ['Material felt dry and lectures were monotonous', │ │ lectures were somewhat monotonous and hard to follow. │ 'Well-organized course content', 'Need for more visual │ │ I would suggest incorporating more visual aids and │ aids, guest lectures, and varied assignments'] │ │ maybe some guest lectures from industry professionals │ │ │ to liven up the sessions. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ I found the course very engaging and informative. The │ ['Helpful and enthusiastic professor', 'Textbook and │ │ professor did an excellent job breaking down complex │ readings overly complex for an introductory course'] │ │ concepts, making them accessible to those of us new to │ │ │ economics. However, the pace was a bit fast, and I │ │ │ sometimes struggled to keep up with the weekly │ │ │ readings. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ As someone with a strong background in math, I │ ['Difficult to connect with real-world applications', │ │ appreciated the analytical rigor of this course. │ 'More current event examples and practical discussions │ │ However, I wish there had been more discussions that │ needed'] │ │ connected the theories we learned to everyday economic │ │ │ issues. It felt a bit isolated from practical │ │ │ realities at times. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ This class was a solid introduction to economics, │ ['Difficult to connect with real-world applications', │ │ though it leaned heavily on theoretical aspects. I │ 'More current event examples and practical discussions │ │ would have liked more opportunities to discuss the │ needed'] │ │ real-world implications of economic theories, which I │ │ │ believe would enhance understanding and retention of │ │ │ the material. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ The professor was knowledgeable and clearly passionate │ ['Need for more visual aids, guest lectures, and │ │ about economics, but I felt the course relied too │ varied assignments', 'Helpful and enthusiastic │ │ heavily on tests rather than more creative forms of │ professor'] │ │ assessment. More varied assignments would make the │ │ │ course more accessible to students with different │ │ │ learning styles. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ Excellent introductory course! The professor was │ ['Well-organized course content', 'Helpful and │ │ enthusiastic and always willing to offer extra help │ enthusiastic professor', 'Fair exams and engaging │ │ during office hours. The interactive lectures and the │ assignments'] │ │ practical assignments made the theory much more │ │ │ digestible and engaging. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ This was my favorite class this semester! The mix of │ ['Well-organized course content', 'Appreciated │ │ theory and case studies was perfect, and the exams │ analytical rigor and diversity of perspectives', 'Fair │ │ were fair. I also really appreciated the diversity of │ exams and engaging assignments'] │ │ perspectives we explored in class, especially in terms │ │ │ of global economic policies. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ This class was a struggle for me. The material felt │ ['Material felt dry and lectures were monotonous', │ │ dry and difficult to connect with real-world │ 'Difficult to connect with real-world applications', │ │ applications, which I think could have made it more │ 'More current event examples and practical discussions │ │ interesting. More examples from current events would │ needed'] │ │ definitely have helped spark my interest. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ I enjoyed the course, especially the group projects, │ ['Well-organized course content', 'Fair exams and │ │ which were both challenging and rewarding. It was │ engaging assignments', 'Desire for more detailed │ │ great to apply economic concepts to solve real-life │ feedback on assignments'] │ │ problems. I did feel, however, that the feedback on │ │ │ assignments could be more detailed to help us │ │ │ understand our mistakes. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ I found the textbook to be overly complex for an │ ['Textbook and readings overly complex for an │ │ introductory course. It often used jargon that hadn't │ introductory course'] │ │ been explained in lectures, which was confusing. │ │ │ Simpler reading materials or more explanatory lectures │ │ │ would make a big difference for newcomers to │ │ │ economics. │ │ └────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────┘
[22]:
wide_evaluation_themes = themes_lists.select("evaluation", "themes_list").to_scenario_list().expand("themes_list").rename({"themes_list": "theme"})
wide_evaluation_themes.print(max_rows = 10)
| evaluation | theme |
|---|---|
| The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions. | Material felt dry and lectures were monotonous |
| The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions. | Well-organized course content |
| The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions. | Need for more visual aids, guest lectures, and varied assignments |
| I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings. | Helpful and enthusiastic professor |
| I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings. | Textbook and readings overly complex for an introductory course |
| As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times. | Difficult to connect with real-world applications |
| As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times. | More current event examples and practical discussions needed |
| This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material. | Difficult to connect with real-world applications |
| This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material. | More current event examples and practical discussions needed |
| The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles. | Need for more visual aids, guest lectures, and varied assignments |
[23]:
wide_evaluation_themes.tally("theme").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓ ┃ value ┃ count ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩ │ Well-organized course content │ 4 │ ├───────────────────────────────────────────────────────────────────┼───────┤ │ Helpful and enthusiastic professor │ 3 │ ├───────────────────────────────────────────────────────────────────┼───────┤ │ Difficult to connect with real-world applications │ 3 │ ├───────────────────────────────────────────────────────────────────┼───────┤ │ More current event examples and practical discussions needed │ 3 │ ├───────────────────────────────────────────────────────────────────┼───────┤ │ Fair exams and engaging assignments │ 3 │ ├───────────────────────────────────────────────────────────────────┼───────┤ │ Material felt dry and lectures were monotonous │ 2 │ ├───────────────────────────────────────────────────────────────────┼───────┤ │ Need for more visual aids, guest lectures, and varied assignments │ 2 │ ├───────────────────────────────────────────────────────────────────┼───────┤ │ Textbook and readings overly complex for an introductory course │ 2 │ ├───────────────────────────────────────────────────────────────────┼───────┤ │ Appreciated analytical rigor and diversity of perspectives │ 1 │ ├───────────────────────────────────────────────────────────────────┼───────┤ │ Desire for more detailed feedback on assignments │ 1 │ └───────────────────────────────────────────────────────────────────┴───────┘
We can do the same thing with the areas of improvement:
[24]:
improvements = results.select("improvements").to_list(flatten=True)
improvements
[24]:
['Incorporate more real-world examples',
'Relate material to current events',
'Increase engagement with interactive activities',
'Incorporate more visual aids into lectures',
'Invite guest lecturers from industry professionals',
'Enhance lecture delivery to make it more engaging',
'Provide more diverse examples in lectures',
'Improve the pacing of the course material',
'Incorporate more real-world case studies',
'Incorporate more real-world economic examples into lectures',
'Facilitate discussions on the application of theories to current economic issues',
'Include case studies or guest speakers from industry',
'Increase opportunities for student participation',
'Provide additional resources for complex topics',
'Offer more real-world application projects',
'Simplify textbook selection',
'Align lecture content with textbook',
'Provide supplementary explanatory materials',
'Incorporate diverse assessment methods',
'Include more project-based assignments',
'Offer alternative evaluation options',
'Incorporate more real-world case studies',
'Facilitate class discussions on current economic events',
'Include practical applications in assignments',
'Provide more detailed feedback on assignments',
'Increase clarity on common mistakes',
'Offer additional office hours for assignment reviews',
'Adjust the pace of the course',
'Provide additional resources for weekly readings',
'Incorporate more review sessions']
[25]:
q_condensed_improvements = QuestionList(
question_name="condensed_improvements",
question_text="""Combine the following list of areas for improvement from the evaluations
into a consolidated, non-redundant list: """
+ ", ".join(improvements),
max_list_items=10,
)
[26]:
condensed_improvements = (
q_condensed_improvements.run().select("condensed_improvements").to_list()[0]
)
condensed_improvements
[26]:
['Incorporate more real-world examples and case studies',
'Relate material to current events',
'Increase engagement with interactive activities and discussions',
'Incorporate more visual aids into lectures',
'Invite guest lecturers from industry professionals',
'Enhance lecture delivery to make it more engaging',
'Improve the pacing of the course material',
'Provide additional resources and supplementary materials',
'Include diverse assessment methods and project-based assignments',
'Offer additional office hours and detailed feedback on assignments']
[27]:
q_improvements_list = QuestionCheckBox(
question_name="improvements_list",
question_text="Select all of the improvements that are mentioned in this evaluation: {{ evaluation }}",
question_options=condensed_improvements,
)
[28]:
improvements_lists = q_improvements_list.by(scenarios).by(agent).run()
improvements_lists.select("evaluation", "improvements_list").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ scenario ┃ answer ┃ ┃ .evaluation ┃ .improvements_list ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ The course content was well-organized, but the │ ['Incorporate more visual aids into lectures', 'Invite │ │ lectures were somewhat monotonous and hard to follow. │ guest lecturers from industry professionals', 'Enhance │ │ I would suggest incorporating more visual aids and │ lecture delivery to make it more engaging'] │ │ maybe some guest lectures from industry professionals │ │ │ to liven up the sessions. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ I found the course very engaging and informative. The │ ['Improve the pacing of the course material', 'Provide │ │ professor did an excellent job breaking down complex │ additional resources and supplementary materials'] │ │ concepts, making them accessible to those of us new to │ │ │ economics. However, the pace was a bit fast, and I │ │ │ sometimes struggled to keep up with the weekly │ │ │ readings. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ This class was a struggle for me. The material felt │ ['Incorporate more real-world examples and case │ │ dry and difficult to connect with real-world │ studies', 'Relate material to current events'] │ │ applications, which I think could have made it more │ │ │ interesting. More examples from current events would │ │ │ definitely have helped spark my interest. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ This was my favorite class this semester! The mix of │ [] │ │ theory and case studies was perfect, and the exams │ │ │ were fair. I also really appreciated the diversity of │ │ │ perspectives we explored in class, especially in terms │ │ │ of global economic policies. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ As someone with a strong background in math, I │ ['Incorporate more real-world examples and case │ │ appreciated the analytical rigor of this course. │ studies', 'Relate material to current events'] │ │ However, I wish there had been more discussions that │ │ │ connected the theories we learned to everyday economic │ │ │ issues. It felt a bit isolated from practical │ │ │ realities at times. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ Excellent introductory course! The professor was │ [] │ │ enthusiastic and always willing to offer extra help │ │ │ during office hours. The interactive lectures and the │ │ │ practical assignments made the theory much more │ │ │ digestible and engaging. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ I found the textbook to be overly complex for an │ ['Enhance lecture delivery to make it more engaging', │ │ introductory course. It often used jargon that hadn't │ 'Provide additional resources and supplementary │ │ been explained in lectures, which was confusing. │ materials'] │ │ Simpler reading materials or more explanatory lectures │ │ │ would make a big difference for newcomers to │ │ │ economics. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ I enjoyed the course, especially the group projects, │ ['Offer additional office hours and detailed feedback │ │ which were both challenging and rewarding. It was │ on assignments'] │ │ great to apply economic concepts to solve real-life │ │ │ problems. I did feel, however, that the feedback on │ │ │ assignments could be more detailed to help us │ │ │ understand our mistakes. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ This class was a solid introduction to economics, │ ['Incorporate more real-world examples and case │ │ though it leaned heavily on theoretical aspects. I │ studies', 'Increase engagement with interactive │ │ would have liked more opportunities to discuss the │ activities and discussions'] │ │ real-world implications of economic theories, which I │ │ │ believe would enhance understanding and retention of │ │ │ the material. │ │ ├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤ │ The professor was knowledgeable and clearly passionate │ ['Include diverse assessment methods and project-based │ │ about economics, but I felt the course relied too │ assignments'] │ │ heavily on tests rather than more creative forms of │ │ │ assessment. More varied assignments would make the │ │ │ course more accessible to students with different │ │ │ learning styles. │ │ └────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────┘
[29]:
wide_themes = (improvements_lists
.select("evaluation", "improvements_list")
.to_scenario_list()
.expand("improvements_list")
.rename({"improvements_list": "theme"})
)
[30]:
wide_themes.tally("theme").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓ ┃ value ┃ count ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩ │ Incorporate more real-world examples and case studies │ 3 │ ├────────────────────────────────────────────────────────────────────┼───────┤ │ Enhance lecture delivery to make it more engaging │ 2 │ ├────────────────────────────────────────────────────────────────────┼───────┤ │ Provide additional resources and supplementary materials │ 2 │ ├────────────────────────────────────────────────────────────────────┼───────┤ │ Relate material to current events │ 2 │ ├────────────────────────────────────────────────────────────────────┼───────┤ │ Incorporate more visual aids into lectures │ 1 │ ├────────────────────────────────────────────────────────────────────┼───────┤ │ Invite guest lecturers from industry professionals │ 1 │ ├────────────────────────────────────────────────────────────────────┼───────┤ │ Improve the pacing of the course material │ 1 │ ├────────────────────────────────────────────────────────────────────┼───────┤ │ Offer additional office hours and detailed feedback on assignments │ 1 │ ├────────────────────────────────────────────────────────────────────┼───────┤ │ Increase engagement with interactive activities and discussions │ 1 │ ├────────────────────────────────────────────────────────────────────┼───────┤ │ Include diverse assessment methods and project-based assignments │ 1 │ └────────────────────────────────────────────────────────────────────┴───────┘
[31]:
improvements_summary = wide_themes.tally("theme")
[32]:
summary_string = improvements_summary.print(format = "markdown", return_string = True)
Summarize the review
Here we create another question prompting the agent to summarize the analysis that was done, using the results of the prior steps:
[33]:
from edsl.questions import QuestionFreeText
q_summary = QuestionFreeText(
question_name="summary",
question_text="Consider the following analyses of the evaluations and draft a paragraph summarizing them."
+ "Evaluation counts by theme: " +
wide_evaluation_themes.tally("theme").print(format = "markdown", return_string = True)
+ "Evaluation counts by area of improvement:"
+ summary_string
)
summary = q_summary.by(agent).run()
summary.select("summary").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .summary ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ The student evaluations for the recent Econ 101 course highlight several key themes and areas for improvement. │ │ A notable number of students appreciated the well-organized course content and found the professor to be │ │ helpful and enthusiastic. However, there was a significant call for more real-world applications, with three │ │ students specifically mentioning the need for current event examples and practical discussions. While the exams │ │ and assignments were generally seen as fair and engaging, there were concerns about the dryness of the material │ │ and the monotonous nature of the lectures. Suggestions for improvement included incorporating more visual aids, │ │ guest lectures, and varied assignments, as well as simplifying the textbook and readings for an introductory │ │ level. Only one student mentioned appreciating the analytical rigor and diversity of perspectives, and another │ │ expressed a desire for more detailed feedback on assignments. Overall, the feedback indicates a strong │ │ foundation but suggests enhancements in connecting theoretical content to real-world contexts and diversifying │ │ teaching methods to increase engagement. │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Other examples
Please check out the EDSL Docs for examples of other methods and templates for use cases, and join our Discord channel to ask questions and with other users!
Posting to the Coop
The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.
We can post any EDSL object to the Coop by calling the push() method on it, including this notebook:
[34]:
from edsl import Notebook
[35]:
n = Notebook(path = "analyze_evaluations.ipynb")
[36]:
n.push(description = "Example code for analyzing course evaluations", visibility = "public")
[36]:
{'description': 'Example code for analyzing course evaluations',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/62786506-21c2-45cb-9a8e-6103002d314b',
'uuid': '62786506-21c2-45cb-9a8e-6103002d314b',
'version': '0.1.33.dev1',
'visibility': 'public'}
To update an object at the Coop:
[37]:
n = Notebook(path = "analyze_evaluations.ipynb")
[38]:
n.patch(uuid = "62786506-21c2-45cb-9a8e-6103002d314b", value = n)
[38]:
{'status': 'success'}