Analyzing course evaluations

This notebook provides sample EDSL code for using a language model to analyze course evaluations. The analysis is designed as a survey of questions about a set of evaluations that we prompt an AI agent to answer, using a language model to generate the responses as a dataset.

EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started.

Create questions

We start by creating questions about a set of course evaluations for an agent to answer. EDSL comes with a variety of question types that we can choose from based on the form of the response that we want to get back from a model (multiple choice, linear scale, checkbox, free text, numerical, dictionary, etc.). We can use a {{ placeholder }} in the question texts to parameterize them with each evaluation. This allows us to create different “scenarios” of the questions that we can administer at once.

We start by importing some question types and composing questions in the relevant templates (see examples of all types in the docs):

from edsl import QuestionList, QuestionMultipleChoice

q_sentiment = QuestionMultipleChoice(
    question_name="sentiment",
    question_text="What is the overall sentiment of this evaluation: {{ evaluation }}",
    question_options=["Positive", "Neutral", "Negative"],
)

q_themes = QuestionList(
    question_name="themes",
    question_text="Summarize the key points of this evaluation: {{ evaluation }}",
    max_list_items=3,  # Optional
)

q_improvements = QuestionList(
    question_name="improvements",
    question_text="Identify areas for improvement based on this evaluation: {{ evaluation }}",
    max_list_items=3,
)

Construct a survey

Next we combine our questions into a survey. This allows us to administer the questions asynchronously (by default), or according to any desired survey logic or rules that we want to add, such as skip/stop rules or giving an agent “memories” of other questions in the survey. Here we create a simple asynchronous survey by passing the list of questions to a Survey object:

from edsl import Survey

survey = Survey(questions=[q_sentiment, q_themes, q_improvements])

Select data for review

Next we identify the data to be analyzed. Here we use some mock evaluations for an Econ 101 course stored as a list of texts:

evaluations = [
    "I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings.",
    "This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest.",
    "Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging.",
    "As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times.",
    "I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes.",
    "The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions.",
    "This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies.",
    "I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics.",
    "The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles.",
    "This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material.",
]

Add data to the questions

Next we create a ScenarioList with a Scenario containing a key/value for each evaluation that we will add to the questions when we run the survey. EDSL provides methods for generating scenarios from many data sources (PDFs, CSVs, images, tables, dicts, etc.); here we import a list and match the key to our question texts placeholder:

from edsl import ScenarioList

scenarios = ScenarioList.from_list("evaluation", evaluations)

Design AI agents

Next we design agents with relevant traits and personas for a language model to use in answering the questions. This can be useful if we want to compare responses among different audiences. We do this by passing dictionaries of traits to Agent objects. We can also choose whether to give an agent additional instructions for answering the survey (independent of individual question texts). Please see documentation for more details and example code for creating agents to use with surveys.

Here we create a persona for the professor of the course and pass it some special instructions:

from edsl import Agent

persona = "You are a professor reviewing student evaluations for your recent Econ 101 course."
instruction = "Be very specific and constructive in providing feedback and suggestions."

agent = Agent(traits={"persona": persona}, instruction=instruction)

Select language models

EDSL works with many popular language models that we can use to generate responses for our survey. We can see a current list of all available models:

from edsl import Model

Model.available()

/Users/a16174/edsl/edsl/inference_services/AvailableModelFetcher.py:139: UserWarning: No models found for service ollama
  warnings.warn(f"No models found for service {service_name}")

	Model Name	Service Name
0	Austism/chronos-hermes-13b-v2	deep_infra
1	Gryphe/MythoMax-L2-13b	deep_infra
2	Qwen/Qwen2-72B-Instruct	deep_infra
3	Qwen/Qwen2-7B-Instruct	deep_infra
4	Qwen/Qwen2.5-72B-Instruct	deep_infra
5	Sao10K/L3-70B-Euryale-v2.1	deep_infra
6	Sao10K/L3.1-70B-Euryale-v2.2	deep_infra
7	google/gemma-2-27b-it	deep_infra
8	google/gemma-2-9b-it	deep_infra
9	lizpreciatior/lzlv_70b_fp16_hf	deep_infra
10	meta-llama/Meta-Llama-3-70B-Instruct	deep_infra
11	meta-llama/Meta-Llama-3-8B-Instruct	deep_infra
12	meta-llama/Meta-Llama-3.1-405B-Instruct	deep_infra
13	meta-llama/Meta-Llama-3.1-70B-Instruct	deep_infra
14	meta-llama/Meta-Llama-3.1-8B-Instruct	deep_infra
15	mistralai/Mistral-7B-Instruct-v0.3	deep_infra
16	microsoft/Phi-3-medium-4k-instruct	deep_infra
17	microsoft/WizardLM-2-7B	deep_infra
18	microsoft/WizardLM-2-8x22B	deep_infra
19	mistralai/Mistral-Nemo-Instruct-2407	deep_infra
20	mistralai/Mixtral-8x7B-Instruct-v0.1	deep_infra
21	openbmb/MiniCPM-Llama3-V-2_5	deep_infra
22	openchat/openchat_3.5	deep_infra
23	gemma-7b-it	groq
24	gemma2-9b-it	groq
25	llama-3.1-70b-versatile	groq
26	llama-3.1-8b-instant	groq
27	llama-guard-3-8b	groq
28	llama3-70b-8192	groq
29	llama3-8b-8192	groq
30	llama3-groq-70b-8192-tool-use-preview	groq
31	llama3-groq-8b-8192-tool-use-preview	groq
32	mixtral-8x7b-32768	groq
33	gpt-4o-realtime-preview	openai
34	gpt-4o-realtime-preview-2024-10-01	openai
35	o1-mini-2024-09-12	openai
36	gpt-4-1106-preview	openai
37	gpt-3.5-turbo-16k	openai
38	gpt-4-0125-preview	openai
39	gpt-4-turbo-preview	openai
40	omni-moderation-latest	openai
41	gpt-4o-2024-05-13	openai
42	omni-moderation-2024-09-26	openai
43	chatgpt-4o-latest	openai
44	gpt-4	openai
45	gpt-4-0613	openai
46	gpt-4o	openai
47	gpt-4o-2024-08-06	openai
48	o1-mini	openai
49	gpt-3.5-turbo	openai
50	gpt-3.5-turbo-0125	openai
51	o1-preview	openai
52	o1-preview-2024-09-12	openai
53	gpt-4-turbo	openai
54	gpt-4-turbo-2024-04-09	openai
55	gpt-3.5-turbo-1106	openai
56	gpt-4o-mini-2024-07-18	openai
57	gpt-4o-audio-preview	openai
58	gpt-4o-audio-preview-2024-10-01	openai
59	gpt-4o-mini	openai
60	gpt-4o-realtime-preview-2024-12-17	openai
61	gpt-4o-mini-realtime-preview	openai
62	gpt-4o-mini-realtime-preview-2024-12-17	openai
63	gpt-4o-2024-11-20	openai
64	gpt-4o-audio-preview-2024-12-17	openai
65	gpt-4o-mini-audio-preview	openai
66	gpt-4o-mini-audio-preview-2024-12-17	openai
67	curie:ft-emeritus-2022-12-01-14-49-45	openai
68	curie:ft-emeritus-2022-12-01-16-40-12	openai
69	curie:ft-emeritus-2022-11-30-12-58-24	openai
70	davinci:ft-emeritus-2022-11-30-14-57-33	openai
71	curie:ft-emeritus-2022-12-01-01-51-20	openai
72	curie:ft-emeritus-2022-12-01-01-04-36	openai
73	curie:ft-emeritus-2022-12-01-15-42-25	openai
74	curie:ft-emeritus-2022-12-01-15-29-32	openai
75	curie:ft-emeritus-2022-12-01-15-52-24	openai
76	curie:ft-emeritus-2022-12-01-14-28-00	openai
77	curie:ft-emeritus-2022-12-01-14-16-46	openai
78	meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo	together
79	mistralai/Mixtral-8x22B-Instruct-v0.1	together
80	meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo	together
81	meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo	together
82	Gryphe/MythoMax-L2-13b-Lite	together
83	Salesforce/Llama-Rank-V1	together
84	meta-llama/Meta-Llama-Guard-3-8B	together
85	meta-llama/Meta-Llama-3-70B-Instruct-Turbo	together
86	meta-llama/Meta-Llama-3-70B-Instruct-Lite	together
87	meta-llama/Meta-Llama-3-8B-Instruct-Lite	together
88	meta-llama/Meta-Llama-3-8B-Instruct-Turbo	together
89	meta-llama/Llama-3-70b-chat-hf	together
90	meta-llama/Llama-3-8b-chat-hf	together
91	Qwen/Qwen2-72B-Instruct	together
92	google/gemma-2-27b-it	together
93	google/gemma-2-9b-it	together
94	mistralai/Mistral-7B-Instruct-v0.3	together
95	Qwen/Qwen1.5-110B-Chat	together
96	meta-llama/LlamaGuard-2-8b	together
97	microsoft/WizardLM-2-8x22B	together
98	togethercomputer/StripedHyena-Nous-7B	together
99	databricks/dbrx-instruct	together
100	deepseek-ai/deepseek-llm-67b-chat	together
101	google/gemma-2b-it	together
102	mistralai/Mistral-7B-Instruct-v0.2	together
103	mistralai/Mixtral-8x7B-Instruct-v0.1	together
104	mistralai/Mixtral-8x7B-v0.1	together
105	Qwen/Qwen1.5-72B-Chat	together
106	NousResearch/Nous-Hermes-2-Yi-34B	together
107	Meta-Llama/Llama-Guard-7b	together
108	NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO	together
109	mistralai/Mistral-7B-Instruct-v0.1	together
110	mistralai/Mistral-7B-v0.1	together
111	meta-llama/Llama-2-13b-chat-hf	together
112	meta-llama/Llama-2-7b-chat-hf	together
113	meta-llama/Llama-2-70b-hf	together
114	codellama/CodeLlama-34b-Instruct-hf	together
115	upstage/SOLAR-10.7B-Instruct-v1.0	together
116	togethercomputer/m2-bert-80M-32k-retrieval	together
117	togethercomputer/m2-bert-80M-8k-retrieval	together
118	togethercomputer/m2-bert-80M-2k-retrieval	together
119	WhereIsAI/UAE-Large-V1	together
120	BAAI/bge-large-en-v1.5	together
121	BAAI/bge-base-en-v1.5	together
122	Gryphe/MythoMax-L2-13b	together
123	cursor/Llama-3-8b-hf	together
124	gemini-1.0-pro	google
125	gemini-1.5-flash	google
126	gemini-1.5-pro	google
127	gemini-pro	google
128	azure:gpt-4o	azure
129	azure:gpt-4o-mini	azure
130	amazon.titan-text-express-v1	bedrock
131	amazon.titan-text-lite-v1	bedrock
132	anthropic.claude-3-5-sonnet-20240620-v1:0	bedrock
133	anthropic.claude-3-haiku-20240307-v1:0	bedrock
134	anthropic.claude-3-opus-20240229-v1:0	bedrock
135	anthropic.claude-3-sonnet-20240229-v1:0	bedrock
136	anthropic.claude-instant-v1	bedrock
137	anthropic.claude-v2	bedrock
138	anthropic.claude-v2:1	bedrock
139	cohere.command-light-text-v14	bedrock
140	cohere.command-r-plus-v1:0	bedrock
141	cohere.command-r-v1:0	bedrock
142	cohere.command-text-v14	bedrock
143	meta.llama3-1-405b-instruct-v1:0	bedrock
144	meta.llama3-1-70b-instruct-v1:0	bedrock
145	meta.llama3-1-8b-instruct-v1:0	bedrock
146	meta.llama3-70b-instruct-v1:0	bedrock
147	meta.llama3-8b-instruct-v1:0	bedrock
148	mistral.mistral-7b-instruct-v0:2	bedrock
149	mistral.mistral-large-2402-v1:0	bedrock
150	mistral.mixtral-8x7b-instruct-v0:1	bedrock
151	claude-3-5-sonnet-20240620	anthropic
152	claude-3-opus-20240229	anthropic
153	claude-3-sonnet-20240229	anthropic
154	claude-3-haiku-20240307	anthropic
155	test	test
156	llama-3.1-sonar-huge-128k-online	perplexity
157	llama-3.1-sonar-large-128k-online	perplexity
158	llama-3.1-sonar-small-128k-online	perplexity
159	codestral-2405	mistral
160	mistral-embed	mistral
161	mistral-large-2407	mistral
162	mistral-medium-latest	mistral
163	mistral-small-2409	mistral
164	mistral-small-latest	mistral
165	open-mistral-7b	mistral
166	open-mistral-nemo-2407	mistral
167	open-mixtral-8x22b	mistral
168	open-mixtral-8x7b	mistral
169	pixtral-12b-2409	mistral

We select models to use with a survey by creating Model objects for them. The default model is GPT 4 Preview, meaning that EDSL will use it to run our survey if we do not specify a different model. Here’s we’ll specify that GPT 4o should be used:

model = Model("gemini-1.5-flash")
model

gemini-1.5-flash

	key	value
0	model	gemini-1.5-flash
1	parameters:temperature	0.500000
2	parameters:topP	1
3	parameters:topK	1
4	parameters:maxOutputTokens	2048
5	parameters:stopSequences	[]

Run the survey

Next we add the scenarios and agent to the survey, and then run it with the specified model. This will generate a dataset of Results that we can store and begin analyzing:

results = survey.by(scenarios).by(agent).by(model).run()

▼ Job Status (2025-01-13 16:37:17)

Job UUID	93073e7c-6837-4ef3-87ca-2ebd7fcf2831
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/93073e7c-6837-4ef3-87ca-2ebd7fcf2831
Error Report URL	None
Results UUID	07846ba6-a514-401f-a8ac-bc02f3e81783
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/07846ba6-a514-401f-a8ac-bc02f3e81783

Analyzing results

EDSL comes with built-in methods for analyzing results in data tables, dataframes, SQL queries and other formats. We can print a list of all the components that can be accessed:

results.columns

	0
0	agent.agent_index
1	agent.agent_instruction
2	agent.agent_name
3	agent.persona
4	answer.improvements
5	answer.sentiment
6	answer.themes
7	cache_used.improvements_cache_used
8	cache_used.sentiment_cache_used
9	cache_used.themes_cache_used
10	comment.improvements_comment
11	comment.sentiment_comment
12	comment.themes_comment
13	generated_tokens.improvements_generated_tokens
14	generated_tokens.sentiment_generated_tokens
15	generated_tokens.themes_generated_tokens
16	iteration.iteration
17	model.maxOutputTokens
18	model.model
19	model.model_index
20	model.stopSequences
21	model.temperature
22	model.topK
23	model.topP
24	prompt.improvements_system_prompt
25	prompt.improvements_user_prompt
26	prompt.sentiment_system_prompt
27	prompt.sentiment_user_prompt
28	prompt.themes_system_prompt
29	prompt.themes_user_prompt
30	question_options.improvements_question_options
31	question_options.sentiment_question_options
32	question_options.themes_question_options
33	question_text.improvements_question_text
34	question_text.sentiment_question_text
35	question_text.themes_question_text
36	question_type.improvements_question_type
37	question_type.sentiment_question_type
38	question_type.themes_question_type
39	raw_model_response.improvements_cost
40	raw_model_response.improvements_one_usd_buys
41	raw_model_response.improvements_raw_model_response
42	raw_model_response.sentiment_cost
43	raw_model_response.sentiment_one_usd_buys
44	raw_model_response.sentiment_raw_model_response
45	raw_model_response.themes_cost
46	raw_model_response.themes_one_usd_buys
47	raw_model_response.themes_raw_model_response
48	scenario.evaluation
49	scenario.scenario_index

Here we select just the responses to the questions and display them in a table:

results.select("sentiment", "themes", "themes_generated_tokens", "improvements")

	answer.sentiment	answer.themes	generated_tokens.themes_generated_tokens	answer.improvements
0	Positive	['Engaging and informative course content', 'Effective explanation of complex concepts', 'Fast pace and challenging readings']	["Engaging and informative course content", "Effective explanation of complex concepts", "Fast pace and challenging readings"] The student found the course to be highly engaging and well-taught, but noted the pace was too fast, impacting their ability to keep up with the readings. These three points capture the essence of the feedback.	['Adjust the pace of lectures and incorporate more opportunities for student engagement.', 'Provide more structured guidance on tackling the weekly readings, perhaps by suggesting specific questions to consider.', 'Consider offering supplemental resources, such as online tutorials or practice problems, to reinforce concepts.']
1	Negative	['Student found the material dry and difficult', 'Lack of real-world connection and examples cited', 'Desire for more current event examples to increase engagement']	["Student found the material dry and difficult", "Lack of real-world connection and examples cited", "Desire for more current event examples to increase engagement"] The three points directly address the student's criticisms and are concise. They avoid subjective interpretations and focus on the student's stated concerns.	['Incorporate more real-world examples and case studies', 'Connect course material to current events', 'Develop more engaging teaching methods to improve student comprehension']
2	Positive	['Excellent instruction and enthusiasm', 'Effective interactive lectures and practical assignments', 'Readily available extra help during office hours']	["Excellent instruction and enthusiasm", "Effective interactive lectures and practical assignments", "Readily available extra help during office hours"] The three points chosen directly reflect the student's positive feedback on teaching quality, course structure, and accessibility of the instructor. They are concise and capture the essence of the evaluation.	['Consider incorporating more diverse case studies', 'Explore alternative assessment methods to gauge deeper understanding beyond practical assignments', 'Enhance online resources and accessibility for students with varying learning styles']
3	Neutral	['Appreciated analytical rigor', 'Desired more real-world application discussions', 'Felt disconnect between theory and practice']	["Appreciated analytical rigor", "Desired more real-world application discussions", "Felt disconnect between theory and practice"] The student's feedback highlights both a positive aspect (strong math foundation) and a key area for improvement (linking theory to practical examples). The three points concisely capture the essence of the evaluation.	['Incorporate real-world case studies into lectures', 'Dedicate class time for discussions linking theoretical concepts to current events', 'Assign application-based problem sets or projects']
4	Positive	['Enjoyed the course and group projects', 'Project application of concepts was valuable', 'Feedback on assignments needs improvement']	["Enjoyed the course and group projects", "Project application of concepts was valuable", "Feedback on assignments needs improvement"] The student enjoyed the course material and found the group projects particularly engaging and beneficial for applying economic concepts. However, they clearly identified a need for more detailed feedback on assignments to improve learning. This summarizes the key points concisely and addresses both positive and negative aspects.	['Enhance feedback detail on assignments', 'Consider providing more specific rubric criteria', 'Explore alternative assessment methods to complement group projects']
5	Neutral	['Well-organized course content', 'Monotonous and hard-to-follow lectures', 'Suggestions for improvement: visual aids and guest lecturers']	["Well-organized course content", "Monotonous and hard-to-follow lectures", "Suggestions for improvement: visual aids and guest lecturers"] The summary accurately reflects the student's main points: positive feedback on organization, negative feedback on lecture style, and concrete suggestions for improvement. The three points are concise and capture the essence of the evaluation.	['Incorporate more visual aids into lectures', 'Consider inviting guest lecturers from the industry', 'Review lecture delivery to improve clarity and engagement']
6	Positive	['Enjoyed the balance of theory and case studies', 'Exams perceived as fair', 'Appreciated diverse perspectives on global economic policies']	["Enjoyed the balance of theory and case studies", "Exams perceived as fair", "Appreciated diverse perspectives on global economic policies"] The student's positive feedback highlights three key aspects: the course structure, assessment fairness, and the inclusion of diverse viewpoints. These are valuable points for course improvement and demonstrate student engagement.	['Increase quantitative problem sets', 'Explore alternative assessment methods alongside exams', 'Incorporate more interactive class discussions']
7	Negative	['Textbook complexity', 'Unexplained jargon', 'Need for simpler materials or more explanatory lectures']	["Textbook complexity", "Unexplained jargon", "Need for simpler materials or more explanatory lectures"] The student clearly identifies three interconnected issues: the textbook's difficulty, the use of undefined jargon, and the resulting need for either simpler readings or more thorough lecture explanations. These are the core complaints and represent the most important points to address.	['Simplify textbook selection or supplement with clearer materials', 'Incorporate textbook jargon into lectures with explicit definitions', 'Designate additional time for clarifying complex concepts']
8	Neutral	['Professor possesses strong subject matter expertise and enthusiasm', 'Course over-reliance on testing limits accessibility for diverse learners', 'More varied assignments needed to cater to different learning styles']	["Professor possesses strong subject matter expertise and enthusiasm", "Course over-reliance on testing limits accessibility for diverse learners", "More varied assignments needed to cater to different learning styles"] The summary focuses on the core strengths and weaknesses highlighted in the evaluation, prioritizing conciseness and clarity for efficient review.	['Incorporate more diverse assessment methods (e.g., group projects, presentations, essays)', 'Explore alternative ways to evaluate understanding beyond high-stakes exams', 'Design assignments catering to varied learning styles and preferences']
9	Neutral	["Positive assessment of the course's introductory nature", 'Critique of insufficient real-world application focus', 'Suggestion for increased emphasis on practical examples']	["Positive assessment of the course's introductory nature", "Critique of insufficient real-world application focus", "Suggestion for increased emphasis on practical examples"] The three points accurately reflect the student's main concerns and suggestions, capturing both positive and negative feedback concisely.	['Incorporate more case studies illustrating real-world applications of economic theories', 'Design in-class activities and discussions focusing on current economic events and their theoretical underpinnings', 'Assign projects or papers requiring students to analyze real-world economic issues using the concepts learned in class']

We can do a quick tally of the sentiments:

results.select("sentiment").tally()

	answer.sentiment	count
0	Positive	4
1	Neutral	4
2	Negative	2

We can also transform the results into a dataframe:

df = results.to_pandas()
df.head()

	answer.themes	answer.sentiment	answer.improvements	scenario.scenario_index	scenario.evaluation	agent.agent_index	agent.persona	agent.agent_name	agent.agent_instruction	model.topP	...	question_type.improvements_question_type	comment.themes_comment	comment.improvements_comment	comment.sentiment_comment	generated_tokens.themes_generated_tokens	generated_tokens.improvements_generated_tokens	generated_tokens.sentiment_generated_tokens	cache_used.themes_cache_used	cache_used.sentiment_cache_used	cache_used.improvements_cache_used
0	['Engaging and informative course content', 'E...	Positive	['Adjust the pace of lectures and incorporate ...	0	I found the course very engaging and informati...	0	You are a professor reviewing student evaluati...	Agent_10	Be very specific and constructive in providing...	1	...	list	The student found the course to be highly enga...	The student's feedback highlights two key area...	The student clearly expresses overall positive...	["Engaging and informative course content", "E...	["Adjust the pace of lectures and incorporate ...	Positive\n\nThe student clearly expresses over...	False	False	False
1	['Student found the material dry and difficult...	Negative	['Incorporate more real-world examples and cas...	1	This class was a struggle for me. The material...	0	You are a professor reviewing student evaluati...	Agent_10	Be very specific and constructive in providing...	1	...	list	The three points directly address the student'...	These suggestions directly address the student...	The student explicitly states the class was a ...	["Student found the material dry and difficult...	["Incorporate more real-world examples and cas...	Negative\n\nThe student explicitly states the ...	False	False	False
2	['Excellent instruction and enthusiasm', 'Effe...	Positive	['Consider incorporating more diverse case stu...	2	Excellent introductory course! The professor w...	0	You are a professor reviewing student evaluati...	Agent_10	Be very specific and constructive in providing...	1	...	list	The three points chosen directly reflect the s...	While the feedback is largely positive, the s...	The student uses overwhelmingly positive langu...	["Excellent instruction and enthusiasm", "Effe...	["Consider incorporating more diverse case stu...	Positive\n\nThe student uses overwhelmingly po...	False	False	False
3	['Appreciated analytical rigor', 'Desired more...	Neutral	['Incorporate real-world case studies into lec...	3	As someone with a strong background in math, I...	0	You are a professor reviewing student evaluati...	Agent_10	Be very specific and constructive in providing...	1	...	list	The student's feedback highlights both a posit...	The student's feedback highlights a disconnect...	The evaluation contains both positive and nega...	["Appreciated analytical rigor", "Desired more...	["Incorporate real-world case studies into lec...	Neutral\n\nThe evaluation contains both positi...	False	False	False
4	['Enjoyed the course and group projects', 'Pro...	Positive	['Enhance feedback detail on assignments', 'Co...	4	I enjoyed the course, especially the group pro...	0	You are a professor reviewing student evaluati...	Agent_10	Be very specific and constructive in providing...	1	...	list	The student enjoyed the course material and fo...	The student enjoyed the course and group proje...	The student clearly enjoyed the course and fou...	["Enjoyed the course and group projects", "Pro...	["Enhance feedback detail on assignments", "Co...	Positive\n\nThe student clearly enjoyed the co...	False	False	False

5 rows × 50 columns

We can also use pandas methods by first converting:

df_sentiment = results.to_pandas()["answer.sentiment"]
df_sentiment.value_counts()

answer.sentiment
Positive    4
Neutral     4
Negative    2
Name: count, dtype: int64

Use responses to construct new questions

We can use the responses to our initial questions to construct more questions about the texts. For example, we can prompt a model to condense the individual lists of themes and areas for improvement into short lists, and then use the new lists to quantify the topics across the set of evaluations.

Here we take the lists of themes in each evaluation, flatten them into a (duplicative) list, and then create a new question prompting a model to condense it for us:

results.select("themes", "themes_generated_tokens")

	answer.themes	generated_tokens.themes_generated_tokens
0	['Engaging and informative course content', 'Effective explanation of complex concepts', 'Fast pace and challenging readings']	["Engaging and informative course content", "Effective explanation of complex concepts", "Fast pace and challenging readings"] The student found the course to be highly engaging and well-taught, but noted the pace was too fast, impacting their ability to keep up with the readings. These three points capture the essence of the feedback.
1	['Student found the material dry and difficult', 'Lack of real-world connection and examples cited', 'Desire for more current event examples to increase engagement']	["Student found the material dry and difficult", "Lack of real-world connection and examples cited", "Desire for more current event examples to increase engagement"] The three points directly address the student's criticisms and are concise. They avoid subjective interpretations and focus on the student's stated concerns.
2	['Excellent instruction and enthusiasm', 'Effective interactive lectures and practical assignments', 'Readily available extra help during office hours']	["Excellent instruction and enthusiasm", "Effective interactive lectures and practical assignments", "Readily available extra help during office hours"] The three points chosen directly reflect the student's positive feedback on teaching quality, course structure, and accessibility of the instructor. They are concise and capture the essence of the evaluation.
3	['Appreciated analytical rigor', 'Desired more real-world application discussions', 'Felt disconnect between theory and practice']	["Appreciated analytical rigor", "Desired more real-world application discussions", "Felt disconnect between theory and practice"] The student's feedback highlights both a positive aspect (strong math foundation) and a key area for improvement (linking theory to practical examples). The three points concisely capture the essence of the evaluation.
4	['Enjoyed the course and group projects', 'Project application of concepts was valuable', 'Feedback on assignments needs improvement']	["Enjoyed the course and group projects", "Project application of concepts was valuable", "Feedback on assignments needs improvement"] The student enjoyed the course material and found the group projects particularly engaging and beneficial for applying economic concepts. However, they clearly identified a need for more detailed feedback on assignments to improve learning. This summarizes the key points concisely and addresses both positive and negative aspects.
5	['Well-organized course content', 'Monotonous and hard-to-follow lectures', 'Suggestions for improvement: visual aids and guest lecturers']	["Well-organized course content", "Monotonous and hard-to-follow lectures", "Suggestions for improvement: visual aids and guest lecturers"] The summary accurately reflects the student's main points: positive feedback on organization, negative feedback on lecture style, and concrete suggestions for improvement. The three points are concise and capture the essence of the evaluation.
6	['Enjoyed the balance of theory and case studies', 'Exams perceived as fair', 'Appreciated diverse perspectives on global economic policies']	["Enjoyed the balance of theory and case studies", "Exams perceived as fair", "Appreciated diverse perspectives on global economic policies"] The student's positive feedback highlights three key aspects: the course structure, assessment fairness, and the inclusion of diverse viewpoints. These are valuable points for course improvement and demonstrate student engagement.
7	['Textbook complexity', 'Unexplained jargon', 'Need for simpler materials or more explanatory lectures']	["Textbook complexity", "Unexplained jargon", "Need for simpler materials or more explanatory lectures"] The student clearly identifies three interconnected issues: the textbook's difficulty, the use of undefined jargon, and the resulting need for either simpler readings or more thorough lecture explanations. These are the core complaints and represent the most important points to address.
8	['Professor possesses strong subject matter expertise and enthusiasm', 'Course over-reliance on testing limits accessibility for diverse learners', 'More varied assignments needed to cater to different learning styles']	["Professor possesses strong subject matter expertise and enthusiasm", "Course over-reliance on testing limits accessibility for diverse learners", "More varied assignments needed to cater to different learning styles"] The summary focuses on the core strengths and weaknesses highlighted in the evaluation, prioritizing conciseness and clarity for efficient review.
9	["Positive assessment of the course's introductory nature", 'Critique of insufficient real-world application focus', 'Suggestion for increased emphasis on practical examples']	["Positive assessment of the course's introductory nature", "Critique of insufficient real-world application focus", "Suggestion for increased emphasis on practical examples"] The three points accurately reflect the student's main concerns and suggestions, capturing both positive and negative feedback concisely.

themes = results.select("themes").to_list(flatten = True)

Next we construct a question to condense the list into a new list:

q_condensed_themes = QuestionList(
    question_name="condensed_themes",
    question_text="""Combine the following list of themes extracted from the evaluations
    into a consolidated, non-redundant list: """
    + ", ".join(themes),
    max_list_items=10,
)

Now we run the question and select the new list. Note that we can choose whether we want to use the agent for this question by not adding it to the question when we run it:

condensed_themes = q_condensed_themes.run().select("condensed_themes").to_list()[0]
condensed_themes

▼ Job Status (2025-01-13 16:49:30)

Job UUID	71d589e6-f842-4c72-a3f3-e45634370220
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/71d589e6-f842-4c72-a3f3-e45634370220
Error Report URL	None
Results UUID	2a2c0493-2364-4a50-b15d-32cec16cabc3
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/2a2c0493-2364-4a50-b15d-32cec16cabc3

['Engaging and informative course content',
 'Effective explanation of complex concepts',
 'Need for more real-world examples and applications',
 'Excellent instruction and enthusiasm',
 'Desire for more varied and accessible learning materials',
 'Well-organized course content',
 'Constructive feedback on assignments',
 'Appreciated analytical rigor and diverse perspectives',
 'Suggestions for improvement: visual aids and guest lecturers',
 'Fair exams and valuable project applications']

Now we can create a question to identify all the themes in the list that appear in each evaluation (our new list becomes the list of answer options):

from edsl import QuestionCheckBox

q_themes_list = QuestionCheckBox(
    question_name="themes_list",
    question_text="Select all of the themes that are mentioned in this evaluation: {{ evaluation }}",
    question_options=condensed_themes,
)

Here we run the question and show a table listing all the themes for each evaluation in the results:

themes_lists = q_themes_list.by(scenarios).by(agent).run()
themes_lists.select("evaluation", "themes_list")

▼ Job Status (2025-01-13 16:50:06)

Job UUID	141752af-2db7-4896-9445-83d272105d1d
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/141752af-2db7-4896-9445-83d272105d1d
Error Report URL	None
Results UUID	7a9b833b-b952-431c-b7b0-9ab8321b6a86
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/7a9b833b-b952-431c-b7b0-9ab8321b6a86

	scenario.evaluation	answer.themes_list
0	I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings.	['Engaging and informative course content', 'Effective explanation of complex concepts']
1	This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest.	['Need for more real-world examples and applications', 'Desire for more varied and accessible learning materials']
2	Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging.	['Engaging and informative course content', 'Excellent instruction and enthusiasm']
3	As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times.	['Need for more real-world examples and applications', 'Appreciated analytical rigor and diverse perspectives']
4	I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes.	['Engaging and informative course content', 'Constructive feedback on assignments', 'Fair exams and valuable project applications']
5	The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions.	['Well-organized course content', 'Suggestions for improvement: visual aids and guest lecturers']
6	This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies.	['Engaging and informative course content', 'Appreciated analytical rigor and diverse perspectives', 'Fair exams and valuable project applications']
7	I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics.	['Effective explanation of complex concepts', 'Desire for more varied and accessible learning materials']
8	The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles.	['Excellent instruction and enthusiasm', 'Desire for more varied and accessible learning materials']
9	This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material.	['Need for more real-world examples and applications']

wide_evaluation_themes = themes_lists.select("evaluation", "themes_list").to_scenario_list().expand("themes_list").rename({"themes_list": "theme"})
wide_evaluation_themes

ScenarioList scenarios: 21; keys: ['theme', 'evaluation'];

	evaluation	theme
0	I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings.	Engaging and informative course content
1	I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings.	Effective explanation of complex concepts
2	This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest.	Need for more real-world examples and applications
3	This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest.	Desire for more varied and accessible learning materials
4	Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging.	Engaging and informative course content
5	Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging.	Excellent instruction and enthusiasm
6	As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times.	Need for more real-world examples and applications
7	As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times.	Appreciated analytical rigor and diverse perspectives
8	I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes.	Engaging and informative course content
9	I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes.	Constructive feedback on assignments
10	I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes.	Fair exams and valuable project applications
11	The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions.	Well-organized course content
12	The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions.	Suggestions for improvement: visual aids and guest lecturers
13	This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies.	Engaging and informative course content
14	This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies.	Appreciated analytical rigor and diverse perspectives
15	This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies.	Fair exams and valuable project applications
16	I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics.	Effective explanation of complex concepts
17	I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics.	Desire for more varied and accessible learning materials
18	The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles.	Excellent instruction and enthusiasm
19	The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles.	Desire for more varied and accessible learning materials
20	This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material.	Need for more real-world examples and applications

wide_evaluation_themes.tally("theme")

	theme	count
0	Engaging and informative course content	4
1	Need for more real-world examples and applications	3
2	Desire for more varied and accessible learning materials	3
3	Effective explanation of complex concepts	2
4	Excellent instruction and enthusiasm	2
5	Appreciated analytical rigor and diverse perspectives	2
6	Fair exams and valuable project applications	2
7	Constructive feedback on assignments	1
8	Well-organized course content	1
9	Suggestions for improvement: visual aids and guest lecturers	1

We can do the same thing with the areas of improvement:

improvements = results.select("improvements").to_list(flatten=True)
improvements

	0
0	Adjust the pace of lectures and incorporate more opportunities for student engagement.
1	Provide more structured guidance on tackling the weekly readings, perhaps by suggesting specific questions to consider.
2	Consider offering supplemental resources, such as online tutorials or practice problems, to reinforce concepts.
3	Incorporate more real-world examples and case studies
4	Connect course material to current events
5	Develop more engaging teaching methods to improve student comprehension
6	Consider incorporating more diverse case studies
7	Explore alternative assessment methods to gauge deeper understanding beyond practical assignments
8	Enhance online resources and accessibility for students with varying learning styles
9	Incorporate real-world case studies into lectures
10	Dedicate class time for discussions linking theoretical concepts to current events
11	Assign application-based problem sets or projects
12	Enhance feedback detail on assignments
13	Consider providing more specific rubric criteria
14	Explore alternative assessment methods to complement group projects
15	Incorporate more visual aids into lectures
16	Consider inviting guest lecturers from the industry
17	Review lecture delivery to improve clarity and engagement
18	Increase quantitative problem sets
19	Explore alternative assessment methods alongside exams
20	Incorporate more interactive class discussions
21	Simplify textbook selection or supplement with clearer materials
22	Incorporate textbook jargon into lectures with explicit definitions
23	Designate additional time for clarifying complex concepts
24	Incorporate more diverse assessment methods (e.g., group projects, presentations, essays)
25	Explore alternative ways to evaluate understanding beyond high-stakes exams
26	Design assignments catering to varied learning styles and preferences
27	Incorporate more case studies illustrating real-world applications of economic theories
28	Design in-class activities and discussions focusing on current economic events and their theoretical underpinnings
29	Assign projects or papers requiring students to analyze real-world economic issues using the concepts learned in class

q_condensed_improvements = QuestionList(
    question_name="condensed_improvements",
    question_text="""Combine the following list of areas for improvement from the evaluations
    into a consolidated, non-redundant list: """
    + ", ".join(improvements),
    max_list_items=10,
)

condensed_improvements = (
    q_condensed_improvements.run().select("condensed_improvements").to_list()[0]
)
condensed_improvements

▼ Job Status (2025-01-13 16:50:53)

Job UUID	d636dc50-f043-4419-b5b9-187654d66197
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/d636dc50-f043-4419-b5b9-187654d66197
Error Report URL	None
Results UUID	94a10c71-6c06-4b9a-94c8-e15e5806ff4e
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/94a10c71-6c06-4b9a-94c8-e15e5806ff4e

['Adjust the pace of lectures and enhance engagement through interactive methods',
 'Provide structured guidance and supplemental resources for weekly readings',
 'Incorporate real-world examples, case studies, and current events into the curriculum',
 'Explore diverse and alternative assessment methods beyond traditional exams',
 'Enhance feedback and provide detailed rubric criteria for assignments',
 'Improve clarity and engagement in lecture delivery with visual aids',
 'Develop assignments catering to varied learning styles and preferences',
 'Incorporate industry insights through guest lecturers and practical examples',
 'Designate time for discussions linking theory to real-world applications',
 'Simplify and supplement textbook materials for clarity and comprehension']

q_improvements_list = QuestionCheckBox(
    question_name="improvements_list",
    question_text="Select all of the improvements that are mentioned in this evaluation: {{ evaluation }}",
    question_options=condensed_improvements,
)

improvements_lists = q_improvements_list.by(scenarios).by(agent).run()
improvements_lists.select("evaluation", "improvements_list")

▼ Job Status (2025-01-13 16:51:34)

Job UUID	97580159-903a-4555-986a-1529654b41fa
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/97580159-903a-4555-986a-1529654b41fa
Error Report URL	None
Results UUID	71193b91-5f0c-4c22-9b79-e9ed3d0c97ce
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/71193b91-5f0c-4c22-9b79-e9ed3d0c97ce

	scenario.evaluation	answer.improvements_list
0	I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings.	['Adjust the pace of lectures and enhance engagement through interactive methods', 'Provide structured guidance and supplemental resources for weekly readings']
1	This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest.	['Incorporate real-world examples, case studies, and current events into the curriculum', 'Designate time for discussions linking theory to real-world applications']
2	Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging.	[]
3	As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times.	['Incorporate real-world examples, case studies, and current events into the curriculum', 'Designate time for discussions linking theory to real-world applications']
4	I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes.	['Enhance feedback and provide detailed rubric criteria for assignments']
5	The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions.	['Improve clarity and engagement in lecture delivery with visual aids', 'Incorporate industry insights through guest lecturers and practical examples']
6	This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies.	[]
7	I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics.	['Provide structured guidance and supplemental resources for weekly readings', 'Simplify and supplement textbook materials for clarity and comprehension']
8	The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles.	['Explore diverse and alternative assessment methods beyond traditional exams', 'Develop assignments catering to varied learning styles and preferences']
9	This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material.	['Incorporate real-world examples, case studies, and current events into the curriculum', 'Designate time for discussions linking theory to real-world applications']

wide_themes = (
    improvements_lists
    .select("evaluation", "improvements_list")
    .to_scenario_list()
    .expand("improvements_list")
    .rename({"improvements_list": "theme"})
)

wide_themes.tally("theme")

	theme	count
0	Incorporate real-world examples, case studies, and current events into the curriculum	3
1	Designate time for discussions linking theory to real-world applications	3
2	Provide structured guidance and supplemental resources for weekly readings	2
3	Adjust the pace of lectures and enhance engagement through interactive methods	1
4	Enhance feedback and provide detailed rubric criteria for assignments	1
5	Improve clarity and engagement in lecture delivery with visual aids	1
6	Incorporate industry insights through guest lecturers and practical examples	1
7	Simplify and supplement textbook materials for clarity and comprehension	1
8	Explore diverse and alternative assessment methods beyond traditional exams	1
9	Develop assignments catering to varied learning styles and preferences	1

Other examples

Please check out the EDSL Docs for examples of other methods and templates for use cases, and join our Discord channel to ask questions and with other users!

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

We can post any EDSL object to the Coop by calling the push() method on it, including this notebook:

from edsl import Notebook

n = Notebook(path = "analyze_evaluations.ipynb")

info = n.push(description = "Example code for analyzing course evaluations", visibility = "public")
info

{'description': 'Example code for analyzing course evaluations',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/4e2808a7-2b4e-4603-8ad2-6e1a65989ccb',
 'uuid': '4e2808a7-2b4e-4603-8ad2-6e1a65989ccb',
 'version': '0.1.40.dev1',
 'visibility': 'public'}

To update an object:

n = Notebook(path = "analyze_evaluations.ipynb") # resave

n.patch(uuid = info["uuid"], value = n)

{'status': 'success'}