Evaluating job posts
This notebook provides sample code for conducting a text analysis using EDSL, an open-source library for simulating surveys, experiments and other research with AI agents and large language models.
Using a dataset of job posts as an example, we demonstrate how to:
Import data into EDSL
Create questions about the data
Design an AI agent to answer the questions
Select a language model to generate responses
Analyze results as a formatted dataset
Technical setup
Before running the code below please ensure that you have completed setup:
Create a Coop account and activate remote inference OR store your own API Keys for language models that you want to use.
Our Starter Tutorial also provides examples of EDSL basic components.
Selecting data for review
First we identify some data for review. Data can be created using the EDSL tools or imported from other sources. For purposes of this demo we import a set of job posts:
[1]:
job_posts = [
"Oversee daily operations, manage staff, and ensure customer satisfaction in a fast-paced retail environment.",
"Craft engaging and informative blog posts on health and wellness topics to boost website traffic and engage readers.",
"Analyze sales data using statistical tools to identify trends and provide actionable insights to the marketing team.",
"Prepare gourmet dishes that comply with restaurant standards and delight customers with unique flavor combinations.",
"Design creative visual content for marketing materials, including brochures, banners, and digital ads, using Adobe Creative Suite.",
"Develop, test, and maintain robust software solutions to improve business processes using Python and Java.",
"Craft coffee drinks and manage the coffee station while providing excellent customer service in a busy café.",
"Manage recruitment processes, conduct interviews, and oversee employee benefit programs to ensure a motivated workforce.",
"Assist veterinarians by preparing animals for surgery, administering injections, and providing post-operative care.",
"Design aesthetic and practical outdoor spaces for clients, from residential gardens to public parks.",
"Install and repair residential plumbing systems, including water heaters, pipes, and fixtures to ensure proper functionality.",
"Develop comprehensive marketing strategies that align with company goals, including digital campaigns and branding efforts.",
"Install, maintain, and repair electrical wiring, equipment, and fixtures to ensure safe and effective operation.",
"Provide personalized fitness programs and conduct group fitness classes to help clients achieve their health goals.",
"Diagnose and repair automotive issues, perform routine maintenance, and ensure vehicles meet safety standards.",
"Lead creative campaigns, from concept through execution, coordinating with graphic designers and content creators.",
"Educate students in mathematics using innovative teaching strategies to enhance understanding and interest in the subject.",
"Drive sales through engaging customer interactions, understanding client needs, and providing product solutions.",
"Fold dough into pretzel shapes ensuring each is uniformly twisted and perfectly salted before baking.",
"Address customer inquiries and issues via phone and email, ensuring high levels of satisfaction and timely resolution.",
]
Constructing questions about the data
Next we create some questions about the data. EDSL provides a variety of question types that we can choose from based on the form of the response that we want to get back from the model (multiple choice, free text, checkbox, linear scale, etc.). Learn more about question types.
Note that we use a {{ placeholder }} in each question text in order to parameterize the questions with the individual job posts in the next step:
[4]:
from edsl import (
QuestionList,
QuestionLinearScale,
QuestionMultipleChoice,
QuestionYesNo,
QuestionFreeText,
)
q1 = QuestionList(
question_name="category_list",
question_text="Draft a list of increasingly specific categories for the following job post: {{ job_post }}",
max_list_items=3, # optional
)
q2 = QuestionLinearScale(
question_name="specific_scale",
question_text="How specific is this job post: {{ job_post }}",
question_options=[0, 1, 2, 3, 4, 5],
option_labels={0: "Unclear", 1: "Not at all specific", 5: "Highly specific"},
)
q3 = QuestionMultipleChoice(
question_name="skill_choice",
question_text="What is the skill level required for this job: {{ job_post }}",
question_options=["Entry level", "Intermediate", "Advanced", "Expert"],
)
q4 = QuestionYesNo(
question_name="technical_yn",
question_text="Is this a technical job? Job post: {{ job_post }}",
)
q5 = QuestionFreeText(
question_name="rewrite_text",
question_text="""Draft an improved version of the following job post: {{ job_post }}""",
)
Building a survey
We combine the questions into a survey in order to administer them together:
[5]:
from edsl import Survey
questions = [q1, q2, q3, q4, q5]
survey = Survey(questions)
If we want the agent/model to have information about prior questions in the survey we can add targeted or full memories (learn more about adding survey rules/logic):
[6]:
# Memory of a specific question is presented with another question:
# survey = survey.add_targeted_memory(q2, q1)
# Full memory of all prior questions is presented with each question (token-intensive):
# survey = survey.set_full_memory_mode()
Adding data to the questions
We add the contents of each ticket into each question as an independent “scenario” for review. This allows us to create versions of the questions for each job post and deliver them to the model all at once. EDSL provides many methods for generating scenarios from different data sources (PDFs, CSVs, docs, images, tables, dicts, etc.). Here we import the list from above:
[7]:
from edsl import ScenarioList, Scenario
scenarios = ScenarioList.from_list("job_post", job_posts)
Designing AI agents
A key feature of EDSL is the ability to create personas for AI agents that the language models are prompted to use in generating responses to the questions. This is done by passing a dictionary of traits to Agent objects:
[8]:
from edsl import AgentList, Agent
agent = Agent(traits={"persona":"You are a labor economist."})
Selecting language models
EDSL allows us to select the language models to use in generating results. To see all available models:
[9]:
from edsl import ModelList, Model
# Model.available()
Here we select GPT 4o (if no model is specified, GPT 4 preview is used by default):
[10]:
model = Model("gpt-4o")
Running the survey
We run the survey by adding the scenarios, agent and model with the by() method and then calling the run() method:
[11]:
results = survey.by(scenarios).by(agent).by(model).run()
This generates a dataset of Results that we can analyze with built-in methods for data tables, dataframes, SQL, etc. We can see a list of all the components that can be analyzed:
[12]:
# results.columns
For example, we can filter, sort, select, limit, shuffle, sample and print some components of results in a table:
[13]:
(
results
.filter("specific_scale in [3,4,5]")
.sort_by("skill_choice")
.select(
"model",
"persona",
"job_post",
"category_list",
"specific_scale",
"skill_choice",
"technical_yn",
)
.print(pretty_labels={}, format="rich", max_rows=5)
)
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃ answer ┃ answer ┃ ┃ .model ┃ .persona ┃ .job_post ┃ .category_list ┃ .specific_scale ┃ .skill_choice ┃ .technical_yn ┃ ┡━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ gpt-4o │ You are a labor │ Design creative │ ['Graphic │ 3 │ Advanced │ Yes │ │ │ economist. │ visual content │ Design', │ │ │ │ │ │ │ for marketing │ 'Marketing │ │ │ │ │ │ │ materials, │ Visual Content │ │ │ │ │ │ │ including │ Creation', │ │ │ │ │ │ │ brochures, │ 'Adobe Creative │ │ │ │ │ │ │ banners, and │ Suite │ │ │ │ │ │ │ digital ads, │ Expertise'] │ │ │ │ │ │ │ using Adobe │ │ │ │ │ │ │ │ Creative Suite. │ │ │ │ │ ├────────┼─────────────────┼──────────────────┼─────────────────┼─────────────────┼───────────────┼───────────────┤ │ gpt-4o │ You are a labor │ Install and │ ['Install and │ 4 │ Advanced │ Yes │ │ │ economist. │ repair │ repair plumbing │ │ │ │ │ │ │ residential │ systems', │ │ │ │ │ │ │ plumbing │ 'Install and │ │ │ │ │ │ │ systems, │ repair water │ │ │ │ │ │ │ including water │ heaters', │ │ │ │ │ │ │ heaters, pipes, │ 'Install and │ │ │ │ │ │ │ and fixtures to │ repair pipes │ │ │ │ │ │ │ ensure proper │ and fixtures'] │ │ │ │ │ │ │ functionality. │ │ │ │ │ ├────────┼─────────────────┼──────────────────┼─────────────────┼─────────────────┼───────────────┼───────────────┤ │ gpt-4o │ You are a labor │ Fold dough into │ ['Food │ 5 │ Intermediate │ No │ │ │ economist. │ pretzel shapes │ Preparation', │ │ │ │ │ │ │ ensuring each is │ 'Baking', │ │ │ │ │ │ │ uniformly │ 'Pretzel │ │ │ │ │ │ │ twisted and │ Shaping'] │ │ │ │ │ │ │ perfectly salted │ │ │ │ │ │ │ │ before baking. │ │ │ │ │ ├────────┼─────────────────┼──────────────────┼─────────────────┼─────────────────┼───────────────┼───────────────┤ │ gpt-4o │ You are a labor │ Craft coffee │ ['Service │ 3 │ Intermediate │ No │ │ │ economist. │ drinks and │ Industry', │ │ │ │ │ │ │ manage the │ 'Food and │ │ │ │ │ │ │ coffee station │ Beverage │ │ │ │ │ │ │ while providing │ Service', │ │ │ │ │ │ │ excellent │ 'Barista'] │ │ │ │ │ │ │ customer service │ │ │ │ │ │ │ │ in a busy café. │ │ │ │ │ ├────────┼─────────────────┼──────────────────┼─────────────────┼─────────────────┼───────────────┼───────────────┤ │ gpt-4o │ You are a labor │ Assist │ ['Animal Care', │ 4 │ Intermediate │ Yes │ │ │ economist. │ veterinarians by │ 'Veterinary │ │ │ │ │ │ │ preparing │ Assistance', │ │ │ │ │ │ │ animals for │ 'Surgical │ │ │ │ │ │ │ surgery, │ Veterinary │ │ │ │ │ │ │ administering │ Technician'] │ │ │ │ │ │ │ injections, and │ │ │ │ │ │ │ │ providing │ │ │ │ │ │ │ │ post-operative │ │ │ │ │ │ │ │ care. │ │ │ │ │ └────────┴─────────────────┴──────────────────┴─────────────────┴─────────────────┴───────────────┴───────────────┘
[14]:
results.select("rewrite_text").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .rewrite_text ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ We are seeking a dedicated Customer Service Representative to manage and resolve customer inquiries and issues │ │ through phone and email communications. The ideal candidate will ensure high levels of customer satisfaction by │ │ providing timely and effective solutions. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ We are seeking a talented writer to create captivating and informative blog posts on health and wellness │ │ topics. Your work will play a crucial role in driving website traffic and engaging our readers. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ We are seeking a dynamic and customer-focused professional to drive sales by engaging with clients, │ │ understanding their unique needs, and offering tailored product solutions. If you have a passion for building │ │ relationships and delivering exceptional service, we want to hear from you! │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Join our culinary team and showcase your talent by preparing exquisite gourmet dishes that adhere to our high │ │ restaurant standards. Delight our customers with your creativity and unique flavor combinations, ensuring an │ │ exceptional dining experience with every meal. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ We're seeking a dynamic Marketing Strategist to develop and implement comprehensive marketing strategies that │ │ align with our company's goals. This role involves creating and executing impactful digital campaigns and │ │ enhancing our brand presence. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ We are seeking a talented Graphic Designer to join our team. In this role, you will create visually engaging │ │ marketing materials, including brochures, banners, and digital ads, utilizing Adobe Creative Suite. Your │ │ creativity and attention to detail will help us effectively communicate our brand message and captivate our │ │ audience. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Join our dynamic team as a Software Developer, where you will leverage your expertise in Python and Java to │ │ design, develop, and maintain innovative software solutions that enhance our business operations. Your │ │ contributions will be pivotal in streamlining processes and driving efficiency across the organization. If you │ │ are passionate about creating robust and scalable systems, we want to hear from you! │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Job Opportunity: HR Manager │ │ │ │ We are seeking a dedicated HR Manager to join our team. In this role, you will: │ │ │ │ - Lead and manage the recruitment process to attract top talent. │ │ - Conduct comprehensive interviews to identify the best candidates. │ │ - Oversee and enhance employee benefit programs to ensure a motivated and satisfied workforce. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Job Opportunity: Expert Pretzel Maker │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Job Opportunity: Barista and Coffee Station Manager │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Join our dynamic team as a Fitness Instructor! │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ We are seeking a dynamic and visionary leader to spearhead our creative campaigns from inception to completion. │ │ In this role, you will collaborate closely with graphic designers and content creators to bring innovative │ │ concepts to life. If you have a passion for creativity and a knack for project management, we want to hear from │ │ you. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Join our team as a Landscape Designer and create stunning, functional outdoor spaces for a diverse range of │ │ clients. From crafting serene residential gardens to designing vibrant public parks, your expertise will bring │ │ beauty and practicality to every project. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ **Job Opportunity: Electrical Technician** │ │ │ │ We are seeking a skilled Electrical Technician to join our team. In this role, you will be responsible for │ │ installing, maintaining, and repairing electrical wiring, equipment, and fixtures to ensure their safe and │ │ efficient operation. │ │ │ │ **Key Responsibilities:** │ │ - Install electrical systems and components according to specifications and safety standards. │ │ - Perform routine maintenance and inspections to ensure optimal performance of electrical equipment. │ │ - Diagnose and repair electrical issues to minimize downtime and enhance safety. │ │ - Ensure compliance with all relevant codes and regulations. │ │ │ │ **Qualifications:** │ │ - Proven experience as an Electrical Technician or similar role. │ │ - Strong knowledge of electrical systems, tools, and equipment. │ │ - Ability to read and interpret technical diagrams and blueprints. │ │ - Excellent problem-solving skills and attention to detail. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ We are seeking a passionate and innovative mathematics educator to inspire and engage students through creative │ │ teaching strategies. The ideal candidate will utilize cutting-edge methods to deepen students' understanding │ │ and foster a genuine interest in mathematics. Join our team and make a meaningful impact on students' academic │ │ journeys! │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ **Job Opportunity: Veterinary Assistant** │ │ │ │ Are you passionate about animal care and looking to make a meaningful impact in the veterinary field? We are │ │ seeking a dedicated Veterinary Assistant to join our team. In this role, you will play a crucial part in │ │ supporting our veterinarians and ensuring the well-being of our animal patients. │ │ │ │ **Key Responsibilities:** │ │ │ │ - Prepare animals for surgery, ensuring they are ready for procedures │ │ - Administer injections and medications as directed by veterinarians │ │ - Provide compassionate post-operative care to aid in the recovery of animals │ │ - Assist with routine examinations and treatments │ │ - Maintain a clean and organized work environment │ │ - Communicate effectively with pet owners, addressing their concerns and providing updates on their pets' │ │ progress │ │ │ │ **Qualifications:** │ │ │ │ - Prior experience in a veterinary or animal care setting is preferred │ │ - Strong attention to detail and ability to follow precise instructions │ │ - Excellent communication and interpersonal skills │ │ - Compassionate and empathetic approach to animal care │ │ - Ability to work effectively in a fast-paced environment │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ None │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Job Title: Sales Data Analyst │ │ │ │ Job Description: │ │ │ │ We are seeking a skilled Sales Data Analyst to join our dynamic team. In this role, you will leverage your │ │ expertise in statistical analysis to examine sales data, identify emerging trends, and deliver actionable │ │ insights that will drive our marketing strategies. Your analytical skills and attention to detail will be │ │ crucial in helping our marketing team make informed decisions and optimize performance. │ │ │ │ Key Responsibilities: │ │ │ │ - Utilize advanced statistical tools and methodologies to analyze sales data. │ │ - Identify and interpret sales trends, patterns, and anomalies. │ │ - Provide clear, data-driven insights and recommendations to the marketing team. │ │ - Collaborate with cross-functional teams to support data-driven decision making. │ │ - Develop and maintain comprehensive reports and dashboards. │ │ │ │ Qualifications: │ │ │ │ - Bachelor’s degree in Statistics, Economics, Data Science, or a related field. │ │ - Proven experience in data analysis, preferably in a sales or marketing environment. │ │ - Proficiency with statistical software and data visualization tools (e.g., R, Python, Tableau). │ │ - Strong analytical and problem-solving skills. │ │ - Excellent communication skills, with the ability to convey complex data insights to non-technical │ │ stakeholders. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Position: Retail Operations Manager │ │ │ │ We are seeking a dynamic and experienced Retail Operations Manager to lead and oversee our daily operations, │ │ manage a dedicated team of staff, and ensure exceptional customer satisfaction in our fast-paced retail │ │ environment. The ideal candidate will possess strong leadership skills, a keen eye for detail, and a passion │ │ for delivering outstanding customer service. If you thrive in a bustling retail setting and are committed to │ │ driving operational excellence, we want to hear from you! │ │ │ │ Key Responsibilities: │ │ - Lead and manage daily store operations to ensure efficiency and effectiveness. │ │ - Supervise, mentor, and support staff members, fostering a positive and productive work environment. │ │ - Implement strategies to enhance customer satisfaction and ensure a superior shopping experience. │ │ - Monitor inventory levels, manage stock, and coordinate with suppliers to maintain optimal product │ │ availability. │ │ - Analyze sales data and performance metrics to identify opportunities for improvement and growth. │ │ - Ensure compliance with company policies, procedures, and safety regulations. │ │ - Address and resolve customer inquiries and concerns promptly and professionally. │ │ │ │ Qualifications: │ │ - Proven experience in retail management or a similar role. │ │ - Strong leadership and team management skills. │ │ - Excellent communication and interpersonal abilities. │ │ - Ability to thrive in a fast-paced and dynamic environment. │ │ - Detail-oriented with strong organizational and problem-solving skills. │ │ - Customer-focused mindset with a commitment to delivering exceptional service. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Job Opening: Automotive Technician │ │ │ │ We are seeking a skilled Automotive Technician to join our team. The ideal candidate will be responsible for │ │ diagnosing and repairing automotive issues, performing routine maintenance, and ensuring all vehicles meet │ │ stringent safety standards. If you have a passion for cars and a commitment to excellence, we want to hear from │ │ you! │ │ │ │ Key Responsibilities: │ │ - Diagnose and troubleshoot mechanical and electrical problems in vehicles │ │ - Perform routine maintenance tasks such as oil changes, tire rotations, and brake inspections │ │ - Conduct comprehensive safety inspections to ensure all vehicles meet regulatory and company standards │ │ - Utilize advanced diagnostic tools and equipment to efficiently identify and resolve issues │ │ - Maintain accurate records of all repairs and services performed │ │ - Provide exceptional customer service by clearly explaining technical issues and repair options to clients │ │ │ │ Qualifications: │ │ - Proven experience as an Automotive Technician or similar role │ │ - Strong knowledge of automotive systems and components │ │ - Proficiency with diagnostic tools and repair equipment │ │ - Excellent problem-solving skills and attention to detail │ │ - Ability to work independently and as part of a team │ │ - ASE certification or equivalent is preferred │ │ │ │ What We Offer: │ │ - Competitive salary and benefits package │ │ - Opportunities for professional growth and development │ │ - A supportive and collaborative work environment │ │ - Access to the latest diagnostic tools and technology │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Posting content to the Coop
We can post any objects to the Coop, including this notebook. Objects can be updated or modified at your Coop account, and shared with others or stored privately (default visibility is unlisted):
[15]:
survey.push(description = "Example survey: Job posts analysis", visibility = "public")
[15]:
{'description': 'Example survey: Job posts analysis',
'object_type': 'survey',
'url': 'https://www.expectedparrot.com/content/d53dce63-0a02-4383-88f6-7ae8c5a8534b',
'uuid': 'd53dce63-0a02-4383-88f6-7ae8c5a8534b',
'version': '0.1.33.dev1',
'visibility': 'public'}
[16]:
results.push(description = "Example results: Job posts analysis", visibility = "public")
[16]:
{'description': 'Example results: Job posts analysis',
'object_type': 'results',
'url': 'https://www.expectedparrot.com/content/4730e6db-bd98-4ccc-a663-b4cf84af0313',
'uuid': '4730e6db-bd98-4ccc-a663-b4cf84af0313',
'version': '0.1.33.dev1',
'visibility': 'public'}
[17]:
from edsl import Notebook
n = Notebook(path = "evaluating_job_posts.ipynb")
[18]:
n.push(description = "Example text analysis: evaluating job posts", visibility = "public")
[18]:
{'description': 'Example text analysis: evaluating job posts',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/f0158719-6a94-461f-92e3-8ca045cc1a9d',
'uuid': 'f0158719-6a94-461f-92e3-8ca045cc1a9d',
'version': '0.1.33.dev1',
'visibility': 'public'}
To update an object at the Coop:
[19]:
n = Notebook(path = "evaluating_job_posts.ipynb") # resave the object
n.patch(uuid = "f0158719-6a94-461f-92e3-8ca045cc1a9d", value = n)
[19]:
{'status': 'success'}