Importing agents

This notebook provides sample EDSL code for simulating surveys with agent personas imported from Persona Hub, an open-source repository provided by Hugging Face.

EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Importing the data

Persona Hub: https://github.com/tencent-ailab/persona-hub

[1]:
from datasets import load_dataset

ds = load_dataset("proj-persona/PersonaHub", "instruction")

Selecting data

Here we select some personas to use.

[2]:
num_agents = 10

personas_list = ds['train']['input persona'][:num_agents]
personas_list
[2]:
['A theater manager or events coordinator interested in understanding the operational aspects, facilities, and programming of performing arts centers, such as the Broward Center.',
 'An urban planner looking to understand the distribution and organization of public services in the Halifax Regional Municipality.',
 "A high school literature teacher looking for supplementary materials to enrich their curriculum and provide students with a deeper understanding of their state's cultural heritage.",
 'A science fiction writer, exploring the use of secret trials as a theme in stories that examine surveillance, privacy, and power dynamics in society.',
 'A financial analyst specializing in Asian markets and wealthy individuals, interested in tracking the investments and philanthropic activities of billionaires like Gerald Chan.',
 'A high school physics teacher who enjoys applying physics concepts to various sports, curious about the effects of water resistance and body positioning in Paralympic swimming.',
 'A high school physical education teacher seeking to incorporate Paralympic history and achievements into the curriculum to inspire and educate students about inclusivity in sports.',
 'A high school history teacher creating lesson plans on South American civilizations and their regional diversity.',
 'A cultural anthropologist studying the representation and adaptation of traditional folklore in contemporary media, particularly in television and film.',
 'A high school earth science teacher looking for real-world examples to explain climate zones, soil types, and human-environment interactions.']

Creating agents

We create an agent in EDSL by passing a dictionary of traits (such as a persona) to an Agent object. Here we use a method for generating a set of agents from a list of traits at once (a list of the personas). Learn more about designing agents in EDSL.

[3]:
from edsl import AgentList

agents = AgentList.from_list("persona", personas_list)

Using agents to answer a survey

Here we construct some questions and administer them to the agents. EDSL comes with many common question types that we can choose from based on the form of the response that we want to get back from a model. Learn more about constructing surveys with different question types, logic and rules.

[4]:
from edsl import QuestionMultipleChoice, QuestionLinearScale, Survey

q1 = QuestionMultipleChoice(
    question_name = "preferred_mode_commute",
    question_text = "What is your preferred mode of commuting to work?",
    question_options = [ "Car", "Public transportation", "Bike", "Walk", "Work from home", "Other" ]
)

q2 = QuestionLinearScale(
    question_name = "work_from_home",
    question_text = "On a scale from 1 to 5, how easy is it for you to work from home?",
    question_options = [1,2,3,4,5],
    option_labels = {1:"Not at all easy", 5:"Very easy"}
)

survey = Survey([q1, q2])

Selecting language models

EDSL works with many popular language models that we can select to generate the responses for the agents.

To see a list of all services:

[5]:
from edsl import Model

Model.services()
[5]:
['openai',
 'anthropic',
 'deep_infra',
 'google',
 'groq',
 'bedrock',
 'azure',
 'ollama',
 'test',
 'mistral',
 'together']

To see a list of all available models:

[6]:
# Model.available()

To select models to use with a survey:

[7]:
from edsl import ModelList, Model

models = ModelList(
    Model(m) for m in ["gemini-1.5-flash", "gpt-4o"]
)

We run a survey by adding the agents and models, and then calling the run() method. This generates a formatted dataset of Results:

[8]:
results = survey.by(agents).by(models).run()

Analyzing results

EDSL comes with built-in methods for analysis. Here we inspect the responses:

[9]:
results.select("model", "persona", "preferred_mode_commute", "work_from_home").print(format="rich")
┏━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ model             agent                                             answer                   answer          ┃
┃ .model            .persona                                          .preferred_mode_commute  .work_from_home ┃
┡━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ gpt-4o            A theater manager or events coordinator           Car                      2               │
│                   interested in understanding the operational                                                │
│                   aspects, facilities, and programming of                                                    │
│                   performing arts centers, such as the Broward                                               │
│                   Center.                                                                                    │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o            An urban planner looking to understand the        Public transportation    5               │
│                   distribution and organization of public services                                           │
│                   in the Halifax Regional Municipality.                                                      │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o            A high school literature teacher looking for      Car                      3               │
│                   supplementary materials to enrich their                                                    │
│                   curriculum and provide students with a deeper                                              │
│                   understanding of their state's cultural                                                    │
│                   heritage.                                                                                  │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o            A science fiction writer, exploring the use of    Work from home           5               │
│                   secret trials as a theme in stories that examine                                           │
│                   surveillance, privacy, and power dynamics in                                               │
│                   society.                                                                                   │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o            A financial analyst specializing in Asian         Work from home           5               │
│                   markets and wealthy individuals, interested in                                             │
│                   tracking the investments and philanthropic                                                 │
│                   activities of billionaires like Gerald Chan.                                               │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o            A high school physics teacher who enjoys          Bike                     3               │
│                   applying physics concepts to various sports,                                               │
│                   curious about the effects of water resistance                                              │
│                   and body positioning in Paralympic swimming.                                               │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o            A high school physical education teacher seeking  Bike                     1               │
│                   to incorporate Paralympic history and                                                      │
│                   achievements into the curriculum to inspire and                                            │
│                   educate students about inclusivity in sports.                                              │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o            A high school history teacher creating lesson     Car                      3               │
│                   plans on South American civilizations and their                                            │
│                   regional diversity.                                                                        │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o            A cultural anthropologist studying the            Work from home           5               │
│                   representation and adaptation of traditional                                               │
│                   folklore in contemporary media, particularly in                                            │
│                   television and film.                                                                       │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o            A high school earth science teacher looking for   Car                      3               │
│                   real-world examples to explain climate zones,                                              │
│                   soil types, and human-environment interactions.                                            │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  A financial analyst specializing in Asian         Car                      4               │
│                   markets and wealthy individuals, interested in                                             │
│                   tracking the investments and philanthropic                                                 │
│                   activities of billionaires like Gerald Chan.                                               │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  A high school history teacher creating lesson     Car                      3               │
│                   plans on South American civilizations and their                                            │
│                   regional diversity.                                                                        │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  A cultural anthropologist studying the            Public transportation    3               │
│                   representation and adaptation of traditional                                               │
│                   folklore in contemporary media, particularly in                                            │
│                   television and film.                                                                       │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  A high school physical education teacher seeking  Bike                     1               │
│                   to incorporate Paralympic history and                                                      │
│                   achievements into the curriculum to inspire and                                            │
│                   educate students about inclusivity in sports.                                              │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  A high school physics teacher who enjoys          Bike                     3               │
│                   applying physics concepts to various sports,                                               │
│                   curious about the effects of water resistance                                              │
│                   and body positioning in Paralympic swimming.                                               │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  A high school literature teacher looking for      Car                      3               │
│                   supplementary materials to enrich their                                                    │
│                   curriculum and provide students with a deeper                                              │
│                   understanding of their state's cultural                                                    │
│                   heritage.                                                                                  │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  A science fiction writer, exploring the use of    Other                    3               │
│                   secret trials as a theme in stories that examine                                           │
│                   surveillance, privacy, and power dynamics in                                               │
│                   society.                                                                                   │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  A theater manager or events coordinator           Car                      1               │
│                   interested in understanding the operational                                                │
│                   aspects, facilities, and programming of                                                    │
│                   performing arts centers, such as the Broward                                               │
│                   Center.                                                                                    │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  An urban planner looking to understand the        Public transportation    3               │
│                   distribution and organization of public services                                           │
│                   in the Halifax Regional Municipality.                                                      │
├──────────────────┼──────────────────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gemini-1.5-flash  A high school earth science teacher looking for   Car                      1               │
│                   real-world examples to explain climate zones,                                              │
│                   soil types, and human-environment interactions.                                            │
└──────────────────┴──────────────────────────────────────────────────┴─────────────────────────┴─────────────────┘

Constructing traits

In comparing model responses it can be convenient to include shortnames for traits in addition to narrative personas–e.g., just the name of the agent’s occupation, age, etc. Here we run a question to extract the occupation from each persona in order to store it as a separate trait of each agent. We use Scenario objects to represent the data (personas) that we add to the question when we run it. Learn more about using scenarios to parameterize questions with data and context.

[10]:
from edsl import QuestionExtract

q = QuestionExtract(
    question_name = "occupation",
    question_text = "{{ persona }}",
    answer_template = {"occupation":"artist"}
)

Creating scenarios for the personas in order to add them to the question:

[11]:
from edsl import ScenarioList

scenarios = ScenarioList.from_list("persona", personas_list)
[12]:
results = q.by(scenarios).run()
[13]:
results.select("persona", "occupation").print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ scenario                                                answer                                                 ┃
┃ .persona                                                .occupation                                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ A theater manager or events coordinator interested in   {'occupation': 'theater manager or events              │
│ understanding the operational aspects, facilities, and  coordinator'}                                          │
│ programming of performing arts centers, such as the                                                            │
│ Broward Center.                                                                                                │
├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ An urban planner looking to understand the              {'occupation': 'urban planner'}                        │
│ distribution and organization of public services in                                                            │
│ the Halifax Regional Municipality.                                                                             │
├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ A high school literature teacher looking for            {'occupation': 'high school literature teacher'}       │
│ supplementary materials to enrich their curriculum and                                                         │
│ provide students with a deeper understanding of their                                                          │
│ state's cultural heritage.                                                                                     │
├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ A science fiction writer, exploring the use of secret   {'occupation': 'science fiction writer'}               │
│ trials as a theme in stories that examine                                                                      │
│ surveillance, privacy, and power dynamics in society.                                                          │
├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ A financial analyst specializing in Asian markets and   {'occupation': 'financial analyst'}                    │
│ wealthy individuals, interested in tracking the                                                                │
│ investments and philanthropic activities of                                                                    │
│ billionaires like Gerald Chan.                                                                                 │
├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ A high school physics teacher who enjoys applying       {'occupation': 'high school physics teacher'}          │
│ physics concepts to various sports, curious about the                                                          │
│ effects of water resistance and body positioning in                                                            │
│ Paralympic swimming.                                                                                           │
├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ A high school physical education teacher seeking to     {'occupation': 'physical education teacher'}           │
│ incorporate Paralympic history and achievements into                                                           │
│ the curriculum to inspire and educate students about                                                           │
│ inclusivity in sports.                                                                                         │
├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ A high school history teacher creating lesson plans on  {'occupation': 'high school history teacher'}          │
│ South American civilizations and their regional                                                                │
│ diversity.                                                                                                     │
├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ A cultural anthropologist studying the representation   {'occupation': 'cultural anthropologist'}              │
│ and adaptation of traditional folklore in contemporary                                                         │
│ media, particularly in television and film.                                                                    │
├────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┤
│ A high school earth science teacher looking for         {'occupation': 'high school earth science teacher'}    │
│ real-world examples to explain climate zones, soil                                                             │
│ types, and human-environment interactions.                                                                     │
└────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────┘

Designing agents

Here we recreate agents with both the personas and occupations as traits:

[14]:
occupations_list = [o['occupation'] for o in results.select("occupation").to_list()]
occupations_list
[14]:
['theater manager or events coordinator',
 'urban planner',
 'high school literature teacher',
 'science fiction writer',
 'financial analyst',
 'high school physics teacher',
 'physical education teacher',
 'high school history teacher',
 'cultural anthropologist',
 'high school earth science teacher']
[15]:
t = list(zip(personas_list, occupations_list))
[16]:
from edsl import AgentList, Agent

agents = AgentList(
    Agent(traits = {"persona":p, "occupation":o}) for p,o in t
)

Running a survey

Here we rerun the survey and then filter and sort results by agent traits:

[17]:
results = survey.by(agents).by(models).run()
[18]:
(
    results
    .filter("model.model == 'gpt-4o'")
    .sort_by("work_from_home", reverse=True)
    .select("model", "occupation", "preferred_mode_commute", "work_from_home")
    .print(format="rich")
)
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ model   agent                                  answer                   answer          ┃
┃ .model  .occupation                            .preferred_mode_commute  .work_from_home ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ gpt-4o  science fiction writer                 Work from home           5               │
├────────┼───────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o  financial analyst                      Work from home           5               │
├────────┼───────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o  urban planner                          Public transportation    4               │
├────────┼───────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o  cultural anthropologist                Work from home           4               │
├────────┼───────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o  theater manager or events coordinator  Car                      3               │
├────────┼───────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o  high school literature teacher         Car                      3               │
├────────┼───────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o  high school physics teacher            Bike                     3               │
├────────┼───────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o  high school history teacher            Car                      3               │
├────────┼───────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o  high school earth science teacher      Car                      3               │
├────────┼───────────────────────────────────────┼─────────────────────────┼─────────────────┤
│ gpt-4o  physical education teacher             Bike                     2               │
└────────┴───────────────────────────────────────┴─────────────────────────┴─────────────────┘

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

Here we demonstrate how to post this notebook:

[19]:
from edsl import Notebook
[20]:
n = Notebook(path = "import_agents.ipynb")
[21]:
n.push(description = "Importing agents", visibility = "public")
[21]:
{'description': 'Importing agents',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/d9ae8689-f0e1-4821-ba9a-c9a4d7dcc348',
 'uuid': 'd9ae8689-f0e1-4821-ba9a-c9a4d7dcc348',
 'version': '0.1.33',
 'visibility': 'public'}

To update an object at the Coop:

[24]:
n = Notebook(path = "import_agents.ipynb")
[25]:
n.patch(uuid = "d9ae8689-f0e1-4821-ba9a-c9a4d7dcc348", value = n)
[25]:
{'status': 'success'}