Cognitive testing & LLM biases
This notebook provides example code for using EDSL to investigate biases of large language models.
EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.
Selecting language models
To check a list of models currently available to use with EDSL:
[1]:
from edsl import ModelList, Model
Model.available
[1]:
<bound method Model.available of Available models: [['Austism/chronos-hermes-13b-v2', 'deep_infra', 0], ['BAAI/bge-base-en-v1.5', 'together', 1], ['BAAI/bge-large-en-v1.5', 'together', 2], ['Gryphe/MythoMax-L2-13b', 'deep_infra', 3], ['Gryphe/MythoMax-L2-13b', 'together', 4], ['Gryphe/MythoMax-L2-13b-Lite', 'together', 5], ['Meta-Llama/Llama-Guard-7b', 'together', 6], ['NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO', 'together', 7], ['NousResearch/Nous-Hermes-2-Yi-34B', 'together', 8], ['Qwen/Qwen1.5-110B-Chat', 'together', 9], ['Qwen/Qwen1.5-72B-Chat', 'together', 10], ['Qwen/Qwen2-72B-Instruct', 'deep_infra', 11], ['Qwen/Qwen2-72B-Instruct', 'together', 12], ['Qwen/Qwen2-7B-Instruct', 'deep_infra', 13], ['Qwen/Qwen2.5-72B-Instruct', 'deep_infra', 14], ['Salesforce/Llama-Rank-V1', 'together', 15], ['Sao10K/L3-70B-Euryale-v2.1', 'deep_infra', 16], ['Sao10K/L3.1-70B-Euryale-v2.2', 'deep_infra', 17], ['WhereIsAI/UAE-Large-V1', 'together', 18], ['amazon.titan-text-express-v1', 'bedrock', 19], ['amazon.titan-text-lite-v1', 'bedrock', 20], ['anthropic.claude-3-5-sonnet-20240620-v1:0', 'bedrock', 21], ['anthropic.claude-3-haiku-20240307-v1:0', 'bedrock', 22], ['anthropic.claude-3-opus-20240229-v1:0', 'bedrock', 23], ['anthropic.claude-3-sonnet-20240229-v1:0', 'bedrock', 24], ['anthropic.claude-instant-v1', 'bedrock', 25], ['anthropic.claude-v2', 'bedrock', 26], ['anthropic.claude-v2:1', 'bedrock', 27], ['azure:gpt-4o', 'azure', 28], ['azure:gpt-4o-mini', 'azure', 29], ['chatgpt-4o-latest', 'openai', 30], ['claude-3-5-sonnet-20240620', 'anthropic', 31], ['claude-3-haiku-20240307', 'anthropic', 32], ['claude-3-opus-20240229', 'anthropic', 33], ['claude-3-sonnet-20240229', 'anthropic', 34], ['codellama/CodeLlama-34b-Instruct-hf', 'together', 35], ['codestral-2405', 'mistral', 36], ['cohere.command-light-text-v14', 'bedrock', 37], ['cohere.command-r-plus-v1:0', 'bedrock', 38], ['cohere.command-r-v1:0', 'bedrock', 39], ['cohere.command-text-v14', 'bedrock', 40], ['curie:ft-emeritus-2022-11-30-12-58-24', 'openai', 41], ['curie:ft-emeritus-2022-12-01-01-04-36', 'openai', 42], ['curie:ft-emeritus-2022-12-01-01-51-20', 'openai', 43], ['curie:ft-emeritus-2022-12-01-14-16-46', 'openai', 44], ['curie:ft-emeritus-2022-12-01-14-28-00', 'openai', 45], ['curie:ft-emeritus-2022-12-01-14-49-45', 'openai', 46], ['curie:ft-emeritus-2022-12-01-15-29-32', 'openai', 47], ['curie:ft-emeritus-2022-12-01-15-42-25', 'openai', 48], ['curie:ft-emeritus-2022-12-01-15-52-24', 'openai', 49], ['curie:ft-emeritus-2022-12-01-16-40-12', 'openai', 50], ['cursor/Llama-3-8b-hf', 'together', 51], ['databricks/dbrx-instruct', 'together', 52], ['davinci:ft-emeritus-2022-11-30-14-57-33', 'openai', 53], ['deepseek-ai/deepseek-llm-67b-chat', 'together', 54], ['gemini-1.0-pro', 'google', 55], ['gemini-1.5-flash', 'google', 56], ['gemini-1.5-pro', 'google', 57], ['gemini-pro', 'google', 58], ['gemma-7b-it', 'groq', 59], ['gemma2-9b-it', 'groq', 60], ['google/gemma-2-27b-it', 'deep_infra', 61], ['google/gemma-2-27b-it', 'together', 62], ['google/gemma-2-9b-it', 'deep_infra', 63], ['google/gemma-2-9b-it', 'together', 64], ['google/gemma-2b-it', 'together', 65], ['gpt-3.5-turbo', 'openai', 66], ['gpt-3.5-turbo-0125', 'openai', 67], ['gpt-3.5-turbo-1106', 'openai', 68], ['gpt-3.5-turbo-16k', 'openai', 69], ['gpt-4', 'openai', 70], ['gpt-4-0125-preview', 'openai', 71], ['gpt-4-0613', 'openai', 72], ['gpt-4-1106-preview', 'openai', 73], ['gpt-4-turbo', 'openai', 74], ['gpt-4-turbo-2024-04-09', 'openai', 75], ['gpt-4-turbo-preview', 'openai', 76], ['gpt-4o', 'openai', 77], ['gpt-4o-2024-05-13', 'openai', 78], ['gpt-4o-2024-08-06', 'openai', 79], ['gpt-4o-2024-11-20', 'openai', 80], ['gpt-4o-audio-preview', 'openai', 81], ['gpt-4o-audio-preview-2024-10-01', 'openai', 82], ['gpt-4o-mini', 'openai', 83], ['gpt-4o-mini-2024-07-18', 'openai', 84], ['gpt-4o-realtime-preview', 'openai', 85], ['gpt-4o-realtime-preview-2024-10-01', 'openai', 86], ['lizpreciatior/lzlv_70b_fp16_hf', 'deep_infra', 87], ['llama-3.1-70b-versatile', 'groq', 88], ['llama-3.1-8b-instant', 'groq', 89], ['llama-3.1-sonar-huge-128k-online', 'perplexity', 90], ['llama-3.1-sonar-large-128k-online', 'perplexity', 91], ['llama-3.1-sonar-small-128k-online', 'perplexity', 92], ['llama-guard-3-8b', 'groq', 93], ['llama3-70b-8192', 'groq', 94], ['llama3-8b-8192', 'groq', 95], ['llama3-groq-70b-8192-tool-use-preview', 'groq', 96], ['llama3-groq-8b-8192-tool-use-preview', 'groq', 97], ['meta-llama/Llama-2-13b-chat-hf', 'together', 98], ['meta-llama/Llama-2-70b-hf', 'together', 99], ['meta-llama/Llama-2-7b-chat-hf', 'together', 100], ['meta-llama/Llama-3-70b-chat-hf', 'together', 101], ['meta-llama/Llama-3-8b-chat-hf', 'together', 102], ['meta-llama/LlamaGuard-2-8b', 'together', 103], ['meta-llama/Meta-Llama-3-70B-Instruct', 'deep_infra', 104], ['meta-llama/Meta-Llama-3-70B-Instruct-Lite', 'together', 105], ['meta-llama/Meta-Llama-3-70B-Instruct-Turbo', 'together', 106], ['meta-llama/Meta-Llama-3-8B-Instruct', 'deep_infra', 107], ['meta-llama/Meta-Llama-3-8B-Instruct-Lite', 'together', 108], ['meta-llama/Meta-Llama-3-8B-Instruct-Turbo', 'together', 109], ['meta-llama/Meta-Llama-3.1-405B-Instruct', 'deep_infra', 110], ['meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo', 'together', 111], ['meta-llama/Meta-Llama-3.1-70B-Instruct', 'deep_infra', 112], ['meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo', 'together', 113], ['meta-llama/Meta-Llama-3.1-8B-Instruct', 'deep_infra', 114], ['meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo', 'together', 115], ['meta-llama/Meta-Llama-Guard-3-8B', 'together', 116], ['meta.llama3-1-405b-instruct-v1:0', 'bedrock', 117], ['meta.llama3-1-70b-instruct-v1:0', 'bedrock', 118], ['meta.llama3-1-8b-instruct-v1:0', 'bedrock', 119], ['meta.llama3-70b-instruct-v1:0', 'bedrock', 120], ['meta.llama3-8b-instruct-v1:0', 'bedrock', 121], ['microsoft/Phi-3-medium-4k-instruct', 'deep_infra', 122], ['microsoft/WizardLM-2-7B', 'deep_infra', 123], ['microsoft/WizardLM-2-8x22B', 'deep_infra', 124], ['microsoft/WizardLM-2-8x22B', 'together', 125], ['mistral-embed', 'mistral', 126], ['mistral-large-2407', 'mistral', 127], ['mistral-medium-latest', 'mistral', 128], ['mistral-small-2409', 'mistral', 129], ['mistral-small-latest', 'mistral', 130], ['mistral.mistral-7b-instruct-v0:2', 'bedrock', 131], ['mistral.mistral-large-2402-v1:0', 'bedrock', 132], ['mistral.mixtral-8x7b-instruct-v0:1', 'bedrock', 133], ['mistralai/Mistral-7B-Instruct-v0.1', 'together', 134], ['mistralai/Mistral-7B-Instruct-v0.2', 'together', 135], ['mistralai/Mistral-7B-Instruct-v0.3', 'deep_infra', 136], ['mistralai/Mistral-7B-Instruct-v0.3', 'together', 137], ['mistralai/Mistral-7B-v0.1', 'together', 138], ['mistralai/Mistral-Nemo-Instruct-2407', 'deep_infra', 139], ['mistralai/Mixtral-8x22B-Instruct-v0.1', 'together', 140], ['mistralai/Mixtral-8x7B-Instruct-v0.1', 'deep_infra', 141], ['mistralai/Mixtral-8x7B-Instruct-v0.1', 'together', 142], ['mistralai/Mixtral-8x7B-v0.1', 'together', 143], ['mixtral-8x7b-32768', 'groq', 144], ['o1-mini', 'openai', 145], ['o1-mini-2024-09-12', 'openai', 146], ['o1-preview', 'openai', 147], ['o1-preview-2024-09-12', 'openai', 148], ['omni-moderation-2024-09-26', 'openai', 149], ['omni-moderation-latest', 'openai', 150], ['open-mistral-7b', 'mistral', 151], ['open-mistral-nemo-2407', 'mistral', 152], ['open-mixtral-8x22b', 'mistral', 153], ['open-mixtral-8x7b', 'mistral', 154], ['openbmb/MiniCPM-Llama3-V-2_5', 'deep_infra', 155], ['openchat/openchat_3.5', 'deep_infra', 156], ['pixtral-12b-2409', 'mistral', 157], ['test', 'test', 158], ['togethercomputer/StripedHyena-Nous-7B', 'together', 159], ['togethercomputer/m2-bert-80M-2k-retrieval', 'together', 160], ['togethercomputer/m2-bert-80M-32k-retrieval', 'together', 161], ['togethercomputer/m2-bert-80M-8k-retrieval', 'together', 162], ['upstage/SOLAR-10.7B-Instruct-v1.0', 'together', 163]]
To create an instance, you can do:
>>> m = Model('gpt-4-1106-preview', temperature=0.5, ...)
To get the default model, you can leave out the model name.
To see the available models, you can do:
>>> Model.available()
>
We select models to use by creating Model objects that can be added to a survey when when it is run. If we do not specify a model, the default model is used with the survey.
To check the current default model:
[2]:
Model()
[2]:
| key | value |
|---|---|
| model | gpt-4o |
| parameters:temperature | 0.5 |
| parameters:max_tokens | 1000 |
| parameters:top_p | 1 |
| parameters:frequency_penalty | 0 |
| parameters:presence_penalty | 0 |
| parameters:logprobs | False |
| parameters:top_logprobs | 3 |
Here we select several models to compare their responses for the survey that we create in the steps below:
[3]:
models = ModelList(
Model(m) for m in ["gemini-pro", "gpt-4o", "claude-3-5-sonnet-20240620"]
)
Generating content
EDSL comes with a variety of standard survey question types, such as multiple choice, free text, etc. These can be selected based on the desired format of the response. See details about all types here. We can use QuestionFreeText to prompt the models to generate some content for our experiment:
[4]:
from edsl import QuestionFreeText
q = QuestionFreeText(
question_name="haiku",
question_text="Draft a haiku about the weather in New England. Return only the haiku."
)
We generate a response to the question by adding the models to use with the by method and then calling the run method. This generates a Results object with a Result for each response to the question:
[5]:
results = q.by(models).run()
▼ Remote Job Log (2024-12-14 10:50:58)
Remote inference activated. Sending job to server...
Your survey is running at the Expected Parrot server...
Job sent to server. (Job uuid=3461cf2f-4e08-4425-89a8-6d62d2711918).
Job details are available at your Coop account https://www.expectedparrot.com/home/remote-inferencehttps://www.expectedparrot.com/home/remote-inference
View job progress here: https://www.expectedparrot.com/home/remote-job-progress/3461cf2f-4e08-4425-89a8-6d62d2711918
Job status: queued - last update: 2024-12-14 10:50:40 AM
Job status: running - last update: 2024-12-14 10:50:44 AM
Job status: running - last update: 2024-12-14 10:50:47 AM
Job status: running - last update: 2024-12-14 10:50:51 AM
Job status: running - last update: 2024-12-14 10:50:55 AM
Job completed and Results stored on Coop: https://www.expectedparrot.com/content/3c2d3f05-8d17-4ede-b464-d7481524a8aa
To see a list of all components of results:
[6]:
results.columns
[6]:
| 0 |
|---|
| agent.agent_instruction |
| agent.agent_name |
| answer.haiku |
| comment.haiku_comment |
| generated_tokens.haiku_generated_tokens |
| iteration.iteration |
| model.frequency_penalty |
| model.logprobs |
| model.maxOutputTokens |
| model.max_tokens |
| model.model |
| model.presence_penalty |
| model.stopSequences |
| model.temperature |
| model.topK |
| model.topP |
| model.top_logprobs |
| model.top_p |
| prompt.haiku_system_prompt |
| prompt.haiku_user_prompt |
| question_options.haiku_question_options |
| question_text.haiku_question_text |
| question_type.haiku_question_type |
| raw_model_response.haiku_cost |
| raw_model_response.haiku_one_usd_buys |
| raw_model_response.haiku_raw_model_response |
We can inspect components of the results individually:
[7]:
results.select("model", "haiku")
[7]:
| model.model | answer.haiku |
|---|---|
| gemini-pro | Snow in winter's grasp, Autumn's leaves in vibrant hues, |
| gpt-4o | Crisp leaves whispering, Misty mornings, fleeting sun— |
| claude-3-5-sonnet-20240620 | Fickle seasons change Snow melts, flowers bloom, leaves fall |
Conducting a review
Next we create a question to have a model evaluating a response that we use as an input to the new question:
[8]:
from edsl import QuestionLinearScale
q_score = QuestionLinearScale(
question_name="score",
question_text="Score the following haiku on a scale from 0 to 10: {{ haiku }}",
question_options=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
option_labels={0: "Very poor", 10: "Excellent"},
)
Parameterizing questions
We use Scenario objects to add each response to the new question. EDSL comes with many methods for creating scenarios from different data sources (PDFs, CSVs, docs, images, lists, etc.), as well as Results objects:
[9]:
scenarios = (
results.to_scenario_list()
.select("model", "haiku")
.rename({"model":"drafting_model"}) # renaming the 'model' field to distinguish the evaluating model
)
scenarios
[9]:
ScenarioList scenarios: 3; keys: ['haiku', 'drafting_model'];
| haiku | drafting_model |
|---|---|
| Snow in winter's grasp, Autumn's leaves in vibrant hues, | gemini-pro |
| Crisp leaves whispering, Misty mornings, fleeting sun— | gpt-4o |
| Fickle seasons change Snow melts, flowers bloom, leaves fall | claude-3-5-sonnet-20240620 |
Finally, we conduct the evaluation by having each model score each haiku that was generated (without information about whether the model itself was the source):
[10]:
results = q_score.by(scenarios).by(models).run()
▼ Remote Job Log (2024-12-14 10:52:03)
Remote inference activated. Sending job to server...
Your survey is running at the Expected Parrot server...
Job sent to server. (Job uuid=88f1bd9e-9ce5-45f8-b37e-03539648ec09).
Job details are available at your Coop account https://www.expectedparrot.com/home/remote-inferencehttps://www.expectedparrot.com/home/remote-inference
View job progress here: https://www.expectedparrot.com/home/remote-job-progress/88f1bd9e-9ce5-45f8-b37e-03539648ec09
Job status: queued - last update: 2024-12-14 10:51:33 AM
Job status: running - last update: 2024-12-14 10:51:36 AM
Job status: running - last update: 2024-12-14 10:51:40 AM
Job status: running - last update: 2024-12-14 10:51:44 AM
Job status: running - last update: 2024-12-14 10:51:48 AM
Job status: running - last update: 2024-12-14 10:51:52 AM
Job status: running - last update: 2024-12-14 10:51:55 AM
Job completed and Results stored on Coop: https://www.expectedparrot.com/content/e8ae6a06-f1d9-4e71-97ee-62e80acfd776
[16]:
results.columns
[16]:
| 0 |
|---|
| agent.agent_instruction |
| agent.agent_name |
| answer.score |
| comment.score_comment |
| generated_tokens.score_generated_tokens |
| iteration.iteration |
| model.frequency_penalty |
| model.logprobs |
| model.maxOutputTokens |
| model.max_tokens |
| model.model |
| model.presence_penalty |
| model.stopSequences |
| model.temperature |
| model.topK |
| model.topP |
| model.top_logprobs |
| model.top_p |
| prompt.score_system_prompt |
| prompt.score_user_prompt |
| question_options.score_question_options |
| question_text.score_question_text |
| question_type.score_question_type |
| raw_model_response.score_cost |
| raw_model_response.score_one_usd_buys |
| raw_model_response.score_raw_model_response |
| scenario.drafting_model |
| scenario.haiku |
[17]:
(
results.sort_by("drafting_model", "model")
.select("drafting_model", "model", "score", "haiku")
.print(
pretty_labels = {
"scenario.drafting_model": "Drafting model",
"model.model": "Scoring model",
"answer.score": "Score",
"scenario.haiku": "Haiku"
}
)
)
[17]:
| Drafting model | Scoring model | Score | Haiku |
|---|---|---|---|
| claude-3-5-sonnet-20240620 | claude-3-5-sonnet-20240620 | 8 | Fickle seasons change Snow melts, flowers bloom, leaves fall |
| claude-3-5-sonnet-20240620 | gemini-pro | 5 | Fickle seasons change Snow melts, flowers bloom, leaves fall |
| claude-3-5-sonnet-20240620 | gpt-4o | 6 | Fickle seasons change Snow melts, flowers bloom, leaves fall |
| gemini-pro | claude-3-5-sonnet-20240620 | 5 | Snow in winter's grasp, Autumn's leaves in vibrant hues, |
| gemini-pro | gemini-pro | 8 | Snow in winter's grasp, Autumn's leaves in vibrant hues, |
| gemini-pro | gpt-4o | 3 | Snow in winter's grasp, Autumn's leaves in vibrant hues, |
| gpt-4o | claude-3-5-sonnet-20240620 | 8 | Crisp leaves whispering, Misty mornings, fleeting sun— |
| gpt-4o | gemini-pro | 7 | Crisp leaves whispering, Misty mornings, fleeting sun— |
| gpt-4o | gpt-4o | 8 | Crisp leaves whispering, Misty mornings, fleeting sun— |
Posting to the Coop
The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.
Here we post this notebook:
[18]:
from edsl import Notebook
[19]:
n = Notebook(path = "explore_llm_biases.ipynb")
[20]:
info = n.push(description = "Example code for comparing model responses and biases", visibility = "public")
info
[20]:
{'description': 'Example code for comparing model responses and biases',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/76b22376-49ad-43b2-aca7-8384c1ade626',
'uuid': '76b22376-49ad-43b2-aca7-8384c1ade626',
'version': '0.1.39.dev1',
'visibility': 'public'}
To update an object:
[21]:
n = Notebook(path = "explore_llm_biases.ipynb") # resave it
[22]:
n.patch(uuid = info["uuid"], value = n)
[22]:
{'status': 'success'}