Cognitive testing & LLM biases
This notebook shows some ways of using EDSL to investigate whether LLMs demonstrate bias towards content that they generate or improve compared with content generated by other LLMs.
Please see our docs for details on installing EDSL and getting started.
[1]:
# ! pip install edsl
Selecting language models
To check a list of models currently available to use with EDSL:
[2]:
from edsl import Model
Model.available()
[2]:
['claude-3-haiku-20240307',
'claude-3-opus-20240229',
'claude-3-sonnet-20240229',
'dbrx-instruct',
'gemini_pro',
'gpt-3.5-turbo',
'gpt-4-1106-preview',
'llama-2-13b-chat-hf',
'llama-2-70b-chat-hf',
'mixtral-8x7B-instruct-v0.1']
We select models to use by creating Model objects that we will add to our survey when we run it later. If we do not specify a model, GPT 4 will be used by default. We select it here for demonstration purposes:
[3]:
models = [
Model("gpt-3.5-turbo"),
Model("gpt-4-1106-preview"),
Model("mixtral-8x7B-instruct-v0.1"),
]
Generating content
EDSL comes with a variety of standard survey question types, such as multiple choice, free text, etc. These can be selected based on the desired format of the response. See details about all types here. We can use QuestionFreeText to prompt the models to generate some content for our experiment (a mock resume). We also import QuestionLinearScale to use later on in reviewing the content:
[4]:
from edsl.questions import QuestionFreeText, QuestionLinearScale
q_example = QuestionFreeText(
question_name="example",
question_text="Summarize a resume for an average software engineer.",
)
We generate a response to the question by calling the run method, after optionally specifying the models to use with the by method. (If no model is specified, GPT 4 is used by default.) This will generate a Results object with a Result for each response to the question or questions administered:
[5]:
example_results = q_example.by(models).run()
We can inspect components of the results individually:
[6]:
example_results.select("model", "example").print()
| model.model | answer.example |
|---|---|
| gpt-3.5-turbo | An average software engineer's resume may include a summary of their technical skills, work experience, education background, and any relevant certifications. They would typically highlight their proficiency in programming languages, experience with software development methodologies, and any specific projects they have worked on. Additionally, they may include details about their ability to work in a team, problem-solving skills, and any other relevant achievements in the field of software engineering. |
| gpt-4-1106-preview | An average software engineer's resume typically includes a summary of their technical skills, professional experience, education, and relevant projects. The technical skills section would list programming languages (e.g., Java, Python, C++), frameworks (e.g., React, Angular, Node.js), and tools (e.g., Git, Docker, AWS). Professional experience would detail roles, responsibilities, and achievements in past positions, focusing on software development tasks. Education would outline degrees obtained, such as a Bachelor's or Master's in Computer Science or a related field, along with the institution and graduation dates. Projects would highlight significant contributions to software development, problem-solving skills, and collaboration. Additional sections might include certifications, publications, or volunteer work related to the field. |
| mixtral-8x7B-instruct-v0.1 | A software engineer's resume typically includes a summary that highlights their technical skills, programming languages, and experience in software development. It should also mention any relevant projects they have worked on, their educational background in computer science or a related field, and any certifications they may have. Additionally, it should include their work experience, with a focus on their roles, responsibilities, and achievements in previous software engineering positions. It may also include any relevant soft skills, such as communication and teamwork abilities. |
To see a list of all components of results we can all the columns method:
[7]:
example_results.columns
[7]:
['agent.agent_name',
'answer.example',
'iteration.iteration',
'model.frequency_penalty',
'model.logprobs',
'model.max_new_tokens',
'model.max_tokens',
'model.model',
'model.presence_penalty',
'model.stopSequences',
'model.temperature',
'model.top_k',
'model.top_logprobs',
'model.top_p',
'model.use_cache',
'prompt.example_system_prompt',
'prompt.example_user_prompt',
'raw_model_response.example_raw_model_response']
Working with results
EDSL comes with a variety of built-in methods for working with results. See details on methods here. For example, we can turn the results into a dataframe:
[8]:
resumes = example_results.to_pandas()
resumes
[8]:
| agent.agent_name | answer.example | iteration.iteration | model.frequency_penalty | model.logprobs | model.max_new_tokens | model.max_tokens | model.model | model.presence_penalty | model.stopSequences | model.temperature | model.top_k | model.top_logprobs | model.top_p | model.use_cache | prompt.example_system_prompt | prompt.example_user_prompt | raw_model_response.example_raw_model_response | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Agent_0 | An average software engineer's resume may incl... | 0 | 0.0 | False | NaN | 1000.0 | gpt-3.5-turbo | 0.0 | NaN | 0.5 | NaN | 3.0 | 1 | NaN | {'text': 'You are answering questions as if yo... | {'text': 'You are being asked the following qu... | {'id': 'chatcmpl-9G9dWGjB42wPSizex69Odr6IuKLzU... |
| 1 | Agent_0 | An average software engineer's resume typicall... | 0 | 0.0 | False | NaN | 1000.0 | gpt-4-1106-preview | 0.0 | NaN | 0.5 | NaN | 3.0 | 1 | NaN | {'text': 'You are answering questions as if yo... | {'text': 'You are being asked the following qu... | {'id': 'chatcmpl-9G9dWO8iPD0iKj38bIRtO8QnzWHQu... |
| 2 | Agent_0 | A software engineer's resume typically include... | 0 | NaN | NaN | 2048.0 | NaN | mixtral-8x7B-instruct-v0.1 | NaN | [] | 0.5 | 1.0 | NaN | 1 | True | {'text': 'You are answering questions as if yo... | {'text': 'You are being asked the following qu... | {'inference_status': {'runtime_ms': 976, 'cost... |
Here we extract components that we’ll use for our review:
[9]:
import pandas as pd
resumes_dict = pd.Series(
resumes["answer.example"].values, index=resumes["model.model"]
).to_dict()
resumes_dict
[9]:
{'gpt-3.5-turbo': "An average software engineer's resume may include a summary of their technical skills, work experience, education background, and any relevant certifications. They would typically highlight their proficiency in programming languages, experience with software development methodologies, and any specific projects they have worked on. Additionally, they may include details about their ability to work in a team, problem-solving skills, and any other relevant achievements in the field of software engineering.",
'gpt-4-1106-preview': "An average software engineer's resume typically includes a summary of their technical skills, professional experience, education, and relevant projects. The technical skills section would list programming languages (e.g., Java, Python, C++), frameworks (e.g., React, Angular, Node.js), and tools (e.g., Git, Docker, AWS). Professional experience would detail roles, responsibilities, and achievements in past positions, focusing on software development tasks. Education would outline degrees obtained, such as a Bachelor's or Master's in Computer Science or a related field, along with the institution and graduation dates. Projects would highlight significant contributions to software development, problem-solving skills, and collaboration. Additional sections might include certifications, publications, or volunteer work related to the field.",
'mixtral-8x7B-instruct-v0.1': "A software engineer's resume typically includes a summary that highlights their technical skills, programming languages, and experience in software development. It should also mention any relevant projects they have worked on, their educational background in computer science or a related field, and any certifications they may have. Additionally, it should include their work experience, with a focus on their roles, responsibilities, and achievements in previous software engineering positions. It may also include any relevant soft skills, such as communication and teamwork abilities."}
Designing AI agents
Next we can generate some relevant personas for AI agents that we will instruct the models to reference in answering questions about the content:
[10]:
q_hr = QuestionFreeText(
question_name="hr", question_text="Draft a persona of a human resources manager."
)
[11]:
q_se = QuestionFreeText(
question_name="se", question_text="Draft a persona of a senior engineer."
)
[12]:
hr_persona = q_hr.run().select("hr").to_list()[0]
hr_persona
[12]:
"Name: Samantha Hughes\nAge: 38\nGender: Female\nEducation: Bachelor's Degree in Human Resource Management, SHRM-SCP Certification\nProfessional Experience: 10 years in human resources, with a focus on recruitment and employee relations\nCurrent Position: Human Resources Manager at a mid-sized tech company\nSkills: Excellent communication and interpersonal skills, adept at conflict resolution, strong understanding of employment law, proficient in HRIS software\nPersonality Traits: Empathetic, detail-oriented, decisive, approachable\nInterests: Volunteering at local community centers, attending professional HR workshops, reading about organizational psychology\nGoals: To improve employee engagement and satisfaction, streamline HR processes using technology, and foster a diverse and inclusive workplace culture"
[13]:
se_persona = q_se.run().select("se").to_list()[0]
se_persona
[13]:
"Name: Alex Johnson\nAge: 42\nProfession: Senior Engineer\nIndustry: Software Development\nYears of Experience: 20\nEducation: Master's Degree in Computer Science\nTechnical Skills: Proficient in multiple programming languages including Java, Python, and C++. Experienced with database management, cloud services, and DevOps practices. Strong background in algorithm design and system architecture.\nLeadership Experience: Has led multiple cross-functional teams to deliver complex software projects on time and within budget. Mentored junior engineers and contributed to the development of internal engineering training programs.\nCommunication Skills: Excellent written and verbal communication skills. Comfortable presenting to stakeholders and facilitating technical workshops.\nCurrent Focus: Interested in the development of scalable microservices architectures and the integration of AI into software solutions.\nHobbies: Enjoys coding side projects, attending tech meetups, and hiking."
Here we construct agents by passing the personas as traits to Agent objects. We also create an agent without a persona for comparing the responses:
[14]:
from edsl import Agent
agents = [
Agent(traits={"role": "", "persona": ""}),
Agent(traits={"role": "Human resources", "persona": hr_persona}),
Agent(traits={"role": "Senior engineer", "persona": se_persona}),
]
Conducting a review
Next we define some methods for improving the resumes and then critiquing the improvements:
[15]:
def improve(resume, model):
q_improve = QuestionFreeText(
question_name="improve",
question_text=f"""Draft an improved version of the following resume: {resume}""",
)
r_improve = q_improve.by(model).run().select("improve").to_list()[0]
return r_improve
[16]:
def score(resume, agent, model):
q_score = QuestionLinearScale(
question_name="score",
question_text=f"""Rank the following resume on a scale from 0 to 10: {resume}""",
question_options=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
option_labels={0: "Very poor", 10: "Excellent"},
)
r_score = q_score.by(agent).by(model).run().select("score").to_list()[0]
return r_score
Finally, we conduct the review of the resumes where we prompt each agent to improve each resume, and then critique each of the improved versions, using each of the models that we specified:
[17]:
results = []
for drafting_model, resume in resumes_dict.items():
for improving_model in models:
improved_resume = improve(resume, improving_model)
for scoring_model in models:
for agent in agents:
score_result = score(improved_resume, agent, scoring_model)
result = {
"drafting_model": drafting_model,
"improving_model": improving_model.model,
"scoring_model": scoring_model.model,
"score": score_result,
"persona": agent.traits["role"],
}
results.append(result)
[18]:
df = pd.DataFrame(results)
[19]:
pd.set_option("display.max_rows", None)
pd.set_option("display.width", 1000)
print(df)
drafting_model improving_model scoring_model score persona
0 gpt-3.5-turbo gpt-3.5-turbo gpt-3.5-turbo 9
1 gpt-3.5-turbo gpt-3.5-turbo gpt-3.5-turbo 8 Human resources
2 gpt-3.5-turbo gpt-3.5-turbo gpt-3.5-turbo 8 Senior engineer
3 gpt-3.5-turbo gpt-3.5-turbo gpt-4-1106-preview 8
4 gpt-3.5-turbo gpt-3.5-turbo gpt-4-1106-preview 7 Human resources
5 gpt-3.5-turbo gpt-3.5-turbo gpt-4-1106-preview 7 Senior engineer
6 gpt-3.5-turbo gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 9
7 gpt-3.5-turbo gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 8 Human resources
8 gpt-3.5-turbo gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 7 Senior engineer
9 gpt-3.5-turbo gpt-4-1106-preview gpt-3.5-turbo 9
10 gpt-3.5-turbo gpt-4-1106-preview gpt-3.5-turbo 8 Human resources
11 gpt-3.5-turbo gpt-4-1106-preview gpt-3.5-turbo 7 Senior engineer
12 gpt-3.5-turbo gpt-4-1106-preview gpt-4-1106-preview 8
13 gpt-3.5-turbo gpt-4-1106-preview gpt-4-1106-preview 9 Human resources
14 gpt-3.5-turbo gpt-4-1106-preview gpt-4-1106-preview 8 Senior engineer
15 gpt-3.5-turbo gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 10
16 gpt-3.5-turbo gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 10 Human resources
17 gpt-3.5-turbo gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 8 Senior engineer
18 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo 8
19 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo 8 Human resources
20 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo 6 Senior engineer
21 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview 8
22 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview 8 Human resources
23 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview 7 Senior engineer
24 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 9
25 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 8 Human resources
26 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 8 Senior engineer
27 gpt-4-1106-preview gpt-3.5-turbo gpt-3.5-turbo 9
28 gpt-4-1106-preview gpt-3.5-turbo gpt-3.5-turbo 8 Human resources
29 gpt-4-1106-preview gpt-3.5-turbo gpt-3.5-turbo 6 Senior engineer
30 gpt-4-1106-preview gpt-3.5-turbo gpt-4-1106-preview 8
31 gpt-4-1106-preview gpt-3.5-turbo gpt-4-1106-preview 8 Human resources
32 gpt-4-1106-preview gpt-3.5-turbo gpt-4-1106-preview 8 Senior engineer
33 gpt-4-1106-preview gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 10
34 gpt-4-1106-preview gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 10 Human resources
35 gpt-4-1106-preview gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 8 Senior engineer
36 gpt-4-1106-preview gpt-4-1106-preview gpt-3.5-turbo 9
37 gpt-4-1106-preview gpt-4-1106-preview gpt-3.5-turbo 9 Human resources
38 gpt-4-1106-preview gpt-4-1106-preview gpt-3.5-turbo 7 Senior engineer
39 gpt-4-1106-preview gpt-4-1106-preview gpt-4-1106-preview 9
40 gpt-4-1106-preview gpt-4-1106-preview gpt-4-1106-preview 9 Human resources
41 gpt-4-1106-preview gpt-4-1106-preview gpt-4-1106-preview 8 Senior engineer
42 gpt-4-1106-preview gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 10
43 gpt-4-1106-preview gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 10 Human resources
44 gpt-4-1106-preview gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 8 Senior engineer
45 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo 9
46 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo 8 Human resources
47 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo 9 Senior engineer
48 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview 10
49 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview 10 Human resources
50 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview 10 Senior engineer
51 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 8
52 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 8 Human resources
53 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 8 Senior engineer
54 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo gpt-3.5-turbo 9
55 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo gpt-3.5-turbo 8 Human resources
56 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo gpt-3.5-turbo 7 Senior engineer
57 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo gpt-4-1106-preview 8
58 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo gpt-4-1106-preview 8 Human resources
59 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo gpt-4-1106-preview 7 Senior engineer
60 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 10
61 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 9 Human resources
62 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo mixtral-8x7B-instruct-v0.1 8 Senior engineer
63 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview gpt-3.5-turbo 10
64 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview gpt-3.5-turbo 9 Human resources
65 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview gpt-3.5-turbo 7 Senior engineer
66 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview gpt-4-1106-preview 9
67 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview gpt-4-1106-preview 9 Human resources
68 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview gpt-4-1106-preview 8 Senior engineer
69 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 10
70 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 9 Human resources
71 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview mixtral-8x7B-instruct-v0.1 8 Senior engineer
72 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo 9
73 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo 8 Human resources
74 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 gpt-3.5-turbo 7 Senior engineer
75 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview 8
76 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview 8 Human resources
77 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 gpt-4-1106-preview 8 Senior engineer
78 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 9
79 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 8 Human resources
80 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 mixtral-8x7B-instruct-v0.1 8 Senior engineer