Evaluating job posts
This notebook provides sample code for conducting a text analysis using EDSL, an open-source library for simulating surveys, experiments and other research with AI agents and large language models.
Using a dataset of job posts as an example, we demonstrate how to:
Import data into EDSL
Create questions about the data
Design an AI agent to answer the questions
Select a language model to generate responses
Analyze results as a formatted dataset
Technical setup
Before running the code below please ensure that you have completed setup:
Create a Coop account and activate remote inference OR store your own API Keys for language models that you want to use.
Our Starter Tutorial also provides examples of EDSL basic components.
Selecting data for review
First we identify some data for review. Data can be created using the EDSL tools or imported from other sources. For purposes of this demo we import a set of job posts:
[1]:
job_posts = [
"Oversee daily operations, manage staff, and ensure customer satisfaction in a fast-paced retail environment.",
"Craft engaging and informative blog posts on health and wellness topics to boost website traffic and engage readers.",
"Analyze sales data using statistical tools to identify trends and provide actionable insights to the marketing team.",
"Prepare gourmet dishes that comply with restaurant standards and delight customers with unique flavor combinations.",
"Design creative visual content for marketing materials, including brochures, banners, and digital ads, using Adobe Creative Suite.",
"Develop, test, and maintain robust software solutions to improve business processes using Python and Java.",
"Craft coffee drinks and manage the coffee station while providing excellent customer service in a busy café.",
"Manage recruitment processes, conduct interviews, and oversee employee benefit programs to ensure a motivated workforce.",
"Assist veterinarians by preparing animals for surgery, administering injections, and providing post-operative care.",
"Design aesthetic and practical outdoor spaces for clients, from residential gardens to public parks.",
"Install and repair residential plumbing systems, including water heaters, pipes, and fixtures to ensure proper functionality.",
"Develop comprehensive marketing strategies that align with company goals, including digital campaigns and branding efforts.",
"Install, maintain, and repair electrical wiring, equipment, and fixtures to ensure safe and effective operation.",
"Provide personalized fitness programs and conduct group fitness classes to help clients achieve their health goals.",
"Diagnose and repair automotive issues, perform routine maintenance, and ensure vehicles meet safety standards.",
"Lead creative campaigns, from concept through execution, coordinating with graphic designers and content creators.",
"Educate students in mathematics using innovative teaching strategies to enhance understanding and interest in the subject.",
"Drive sales through engaging customer interactions, understanding client needs, and providing product solutions.",
"Fold dough into pretzel shapes ensuring each is uniformly twisted and perfectly salted before baking.",
"Address customer inquiries and issues via phone and email, ensuring high levels of satisfaction and timely resolution.",
]
Constructing questions about the data
Next we create some questions about the data. EDSL provides a variety of question types that we can choose from based on the form of the response that we want to get back from the model (multiple choice, free text, checkbox, linear scale, etc.). Learn more about question types.
Note that we use a {{ placeholder }} in each question text in order to parameterize the questions with the individual job posts in the next step:
[2]:
from edsl import (
QuestionList,
QuestionLinearScale,
QuestionMultipleChoice,
QuestionYesNo,
QuestionFreeText,
)
q1 = QuestionList(
question_name="category_list",
question_text="Draft a list of increasingly specific categories for the following job post: {{ job_post }}",
max_list_items=3, # optional
)
q2 = QuestionLinearScale(
question_name="specific_scale",
question_text="How specific is this job post: {{ job_post }}",
question_options=[0, 1, 2, 3, 4, 5],
option_labels={0: "Unclear", 1: "Not at all specific", 5: "Highly specific"},
)
q3 = QuestionMultipleChoice(
question_name="skill_choice",
question_text="What is the skill level required for this job: {{ job_post }}",
question_options=["Entry level", "Intermediate", "Advanced", "Expert"],
)
q4 = QuestionYesNo(
question_name="technical_yn",
question_text="Is this a technical job? Job post: {{ job_post }}",
)
q5 = QuestionFreeText(
question_name="rewrite_text",
question_text="""Consider whether the following job post could be improved for clarity.
Then, without substantially lengthening it, draft a new version: {{ job_post }}""",
)
Building a survey
We combine the questions into a survey in order to administer them together:
[3]:
from edsl import Survey
questions = [q1, q2, q3, q4, q5]
survey = Survey(questions)
If we want the agent/model to have information about prior questions in the survey we can add targeted or full memories (learn more about adding survey rules/logic):
[4]:
# Memory of a specific question is presented with another question:
# survey = survey.add_targeted_memory(q2, q1)
# Full memory of all prior questions is presented with each question (token-intensive):
# survey = survey.set_full_memory_mode()
Adding data to the questions
We add the contents of each ticket into each question as an independent “scenario” for review. This allows us to create versions of the questions for each job post and deliver them to the model all at once:
[5]:
from edsl import ScenarioList, Scenario
scenarios = ScenarioList(
Scenario({"job_post": p}) for p in job_posts
)
Designing AI agents
A key feature of EDSL is the ability to create personas for AI agents that the language models are prompted to use in generating responses to the questions. This is done by passing a dictionary of traits to Agent objects:
[6]:
from edsl import AgentList, Agent
agent = Agent(traits={"persona":"You are a labor economist."})
Selecting language models
EDSL allows us to select the language models to use in generating results. To see all available models:
[7]:
from edsl import ModelList, Model
# Model.available()
Here we select GPT 4 (if no model is specified, GPT 4 preview is used by default):
[8]:
model = Model("gpt-4")
Running the survey
We run the survey by adding the scenarios, agent and model with the by() method and then calling the run() method:
[9]:
results = survey.by(scenarios).by(agent).by(model).run()
This generates a dataset of Results that we can analyze with built-in methods for data tables, dataframes, SQL, etc. We can see a list of all the components that can be analyzed:
[10]:
# results.columns
For example, we can filter, sort, select, limit, shuffle, sample and print some components of results in a table:
[11]:
(
results
.filter("specific_scale in ['3','4','5']")
.sort_by("skill_choice")
.select(
"model",
"persona",
"job_post",
"category_list",
"specific_scale",
"skill_choice",
"technical_yn",
)
.print(pretty_labels={}, format="rich", max_rows=5)
)
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃ answer ┃ answer ┃ ┃ .model ┃ .persona ┃ .job_post ┃ .category_list ┃ .specific_scale ┃ .skill_choice ┃ .technical_yn ┃ ┡━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ gpt-4 │ You are a labor │ Analyze sales │ ['Data │ 4 │ Advanced │ Yes │ │ │ economist. │ data using │ Analyst', │ │ │ │ │ │ │ statistical │ 'Sales Data │ │ │ │ │ │ │ tools to │ Analyst', │ │ │ │ │ │ │ identify trends │ 'Marketing Data │ │ │ │ │ │ │ and provide │ Analyst'] │ │ │ │ │ │ │ actionable │ │ │ │ │ │ │ │ insights to the │ │ │ │ │ │ │ │ marketing team. │ │ │ │ │ ├────────┼─────────────────┼──────────────────┼─────────────────┼─────────────────┼───────────────┼───────────────┤ │ gpt-4 │ You are a labor │ Design creative │ ['Marketing and │ 3 │ Advanced │ Yes │ │ │ economist. │ visual content │ Advertising', │ │ │ │ │ │ │ for marketing │ 'Graphic │ │ │ │ │ │ │ materials, │ Design', 'Adobe │ │ │ │ │ │ │ including │ Creative Suite │ │ │ │ │ │ │ brochures, │ Specialist'] │ │ │ │ │ │ │ banners, and │ │ │ │ │ │ │ │ digital ads, │ │ │ │ │ │ │ │ using Adobe │ │ │ │ │ │ │ │ Creative Suite. │ │ │ │ │ ├────────┼─────────────────┼──────────────────┼─────────────────┼─────────────────┼───────────────┼───────────────┤ │ gpt-4 │ You are a labor │ Manage │ ['Human │ 3 │ Advanced │ No │ │ │ economist. │ recruitment │ Resources │ │ │ │ │ │ │ processes, │ Manager', │ │ │ │ │ │ │ conduct │ 'Recruitment │ │ │ │ │ │ │ interviews, and │ Manager', │ │ │ │ │ │ │ oversee employee │ 'Employee │ │ │ │ │ │ │ benefit programs │ Benefits │ │ │ │ │ │ │ to ensure a │ Manager'] │ │ │ │ │ │ │ motivated │ │ │ │ │ │ │ │ workforce. │ │ │ │ │ ├────────┼─────────────────┼──────────────────┼─────────────────┼─────────────────┼───────────────┼───────────────┤ │ gpt-4 │ You are a labor │ Design aesthetic │ ['Landscape │ 4 │ Advanced │ Yes │ │ │ economist. │ and practical │ Architect', │ │ │ │ │ │ │ outdoor spaces │ 'Residential │ │ │ │ │ │ │ for clients, │ Landscape │ │ │ │ │ │ │ from residential │ Designer', │ │ │ │ │ │ │ gardens to │ 'Public Park │ │ │ │ │ │ │ public parks. │ Landscape │ │ │ │ │ │ │ │ Designer'] │ │ │ │ ├────────┼─────────────────┼──────────────────┼─────────────────┼─────────────────┼───────────────┼───────────────┤ │ gpt-4 │ You are a labor │ Install, │ ['Electrician', │ 3 │ Advanced │ Yes │ │ │ economist. │ maintain, and │ 'Installation │ │ │ │ │ │ │ repair │ Electrician', │ │ │ │ │ │ │ electrical │ 'Maintenance │ │ │ │ │ │ │ wiring, │ and Repair │ │ │ │ │ │ │ equipment, and │ Electrician'] │ │ │ │ │ │ │ fixtures to │ │ │ │ │ │ │ │ ensure safe and │ │ │ │ │ │ │ │ effective │ │ │ │ │ │ │ │ operation. │ │ │ │ │ └────────┴─────────────────┴──────────────────┴─────────────────┴─────────────────┴───────────────┴───────────────┘
[12]:
results.select("rewrite_text").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .rewrite_text ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Manage daily operations, supervise employees, and guarantee customer satisfaction in a high-speed retail │ │ setting. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Produce compelling and educational health and wellness blog content to increase website visits and engage │ │ audience. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Conduct statistical analysis on sales data to discern trends and deliver valuable insights to the marketing │ │ team. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Craft gourmet dishes in accordance with our restaurant's quality standards, and captivate our customers through │ │ innovative and unique flavor profiles. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Create engaging visual designs for promotional materials such as brochures, banners, and digital advertisements │ │ utilizing Adobe Creative Suite. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Design, validate, and upkeep efficient software applications enhancing business operations, utilizing Python │ │ and Java languages. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Prepare and serve artisan coffee beverages, oversee the coffee station operations, and deliver outstanding │ │ customer service within a high-traffic café setting. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Oversee the recruitment process, conduct candidate interviews, and manage employee benefit programs to maintain │ │ a motivated workforce. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Support veterinarians through the preparation of animals for surgery, the administration of injections, and the │ │ delivery of post-operative care. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Develop aesthetically pleasing and functional outdoor environments for a range of clients, including homeowners │ │ and public entities. Projects may range from residential garden designs to large-scale public park layouts. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Formulate holistic marketing strategies in line with our company objectives, encompassing digital campaigns and │ │ brand initiatives. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Install, maintain, and repair electrical systems, ensuring safety and effectiveness. Responsibilities include │ │ working with wiring, equipment, and fixtures. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Respond to customer inquiries and resolve issues through phone and email, guaranteeing prompt resolution and │ │ high customer satisfaction. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Responsible for the installation and repair of residential plumbing systems. This includes tasks such as │ │ ensuring the proper functionality of water heaters, pipes, and fixtures. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Deliver customized fitness programs and lead group exercise sessions to assist clients in accomplishing their │ │ wellness objectives. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Identify and rectify automotive problems, conduct regular maintenance tasks, and confirm vehicles adhere to │ │ safety regulations. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Coordinate and execute creative campaigns, from inception to completion, working collaboratively with graphic │ │ designers and content creators. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Teach mathematics to students employing creative teaching methods to boost their comprehension and engagement │ │ in the subject. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Boost sales by actively engaging with customers, comprehending their requirements, and offering appropriate │ │ product solutions. │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Shape dough into consistent pretzel forms, ensuring uniform twists and even salting prior to baking. │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Posting content to the Coop
We can post any objects to the Coop, including this notebook. Objects can be updated or modified at your Coop account, and shared with others or stored privately (default visibility is unlisted):
[13]:
survey.push(description = "Example survey: Job posts analysis", visibility = "public")
[13]:
{'description': 'Example survey: Job posts analysis',
'object_type': 'survey',
'url': 'https://www.expectedparrot.com/content/c34eefcb-70dc-40db-adb5-d86a49044b58',
'uuid': 'c34eefcb-70dc-40db-adb5-d86a49044b58',
'version': '0.1.32.dev1',
'visibility': 'public'}
[14]:
results.push(description = "Example results: Job posts analysis", visibility = "public")
[14]:
{'description': 'Example results: Job posts analysis',
'object_type': 'results',
'url': 'https://www.expectedparrot.com/content/7f94a8fd-c6ae-4882-b294-89713f369f52',
'uuid': '7f94a8fd-c6ae-4882-b294-89713f369f52',
'version': '0.1.32.dev1',
'visibility': 'public'}
[15]:
from edsl import Notebook
n = Notebook(path = "evaluating_job_posts.ipynb")
[16]:
n.push(description = "Example text analysis: evaluating job posts", visibility = "public")
[16]:
{'description': 'Example text analysis: evaluating job posts',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/a30cda44-4238-4ea2-8d3c-96579665d0c9',
'uuid': 'a30cda44-4238-4ea2-8d3c-96579665d0c9',
'version': '0.1.32.dev1',
'visibility': 'public'}
To update an object at the Coop:
[20]:
n = Notebook(path = "evaluating_job_posts.ipynb") # resave the object
n.patch(uuid = "a30cda44-4238-4ea2-8d3c-96579665d0c9", value = n)
[20]:
{'status': 'success'}