Intro to EDSL

This notebook provides example code for base components of EDSL, an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Details on the code below are provided in accompanying slides: How to use EDSL.

Technical setup

Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL.

Documentation

Please also see our documentation page for tips, tutorials and more demo notebooks on using EDSL.

Simple example

We start by selecting a question type and constructing a question in the relevant template:

[1]:

from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
    question_name = "marvel_movies",
    question_text = "Do you enjoy Marvel movies?",
    question_options = ["Yes", "No", "I do not know"]
)

We administer a question by calling the run() method. This generates a dataset of Results including the model’s response to the question:

[2]:

results = q.run()

results.select("marvel_movies").print(format="rich")

┏━━━━━━━━━━━━━━━━┓
┃ answer         ┃
┃ .marvel_movies ┃
┡━━━━━━━━━━━━━━━━┩
│ I do not know  │
└────────────────┘

Designing AI agents

We can create personas for agents to answer the question:

[3]:

from edsl import AgentList, Agent

personas = ["comic book collector", "movie critic"]

a = AgentList(
    Agent(traits = {"persona": p}) for p in personas
)

Selecting language models

We can select language models to generate the responses (in the example above we did not specify a model, so GPT 4 preview was used by default):

[4]:

from edsl import ModelList, Model

models = ["gpt-4o", "claude-3-5-sonnet-20240620"]

m = ModelList(
    Model(m) for m in ["gpt-4o", "claude-3-5-sonnet-20240620"]
)

Generating results

We add agents and models to a question when running it:

[5]:

results = q.by(a).by(m).run()

results.select("model", "persona", "marvel_movies").print(format="rich")

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃ model                      ┃ agent                ┃ answer         ┃
┃ .model                     ┃ .persona             ┃ .marvel_movies ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ gpt-4o                     │ comic book collector │ Yes            │
├────────────────────────────┼──────────────────────┼────────────────┤
│ claude-3-5-sonnet-20240620 │ comic book collector │ Yes            │
├────────────────────────────┼──────────────────────┼────────────────┤
│ gpt-4o                     │ movie critic         │ Yes            │
├────────────────────────────┼──────────────────────┼────────────────┤
│ claude-3-5-sonnet-20240620 │ movie critic         │ Yes            │
└────────────────────────────┴──────────────────────┴────────────────┘

Parameterizing questions

We can use Scenario objects to add data or content to questions:

[6]:

q1 = QuestionMultipleChoice(
    question_name = "politically_motivated",
    question_text = """
    Read the following movie review and determine whether it is politically motivated.
    Movie: {{ title }}
    Review: {{ review }}
    """,
    question_options = ["Yes", "No", "I do not know"]
)

EDSL comes with methods for generating scenarios from many data sources, including PDFs, CSVs, docs, images, tables, lists, dicts:

[7]:

from edsl import Scenario

example_review = {
    "year": 2014,
    "title": "Captain America: The Winter Soldier",
    "review": """
    Part superhero flick, part 70s political thriller.
    It's a bold mix that pays off, delivering a scathing
    critique of surveillance states wrapped in spandex
    and shield-throwing action.
    """
}

s = Scenario.from_dict(example_review)

[8]:

results = q1.by(s).by(a).by(m).run()

(
    results.filter("persona == 'movie critic'")
    .sort_by("model")
    .select("model", "year", "title", "politically_motivated")
    .print(format="rich")
)

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ model                      ┃ scenario ┃ scenario                            ┃ answer                 ┃
┃ .model                     ┃ .year    ┃ .title                              ┃ .politically_motivated ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ 2014     │ Captain America: The Winter Soldier │ No                     │
├────────────────────────────┼──────────┼─────────────────────────────────────┼────────────────────────┤
│ gpt-4o                     │ 2014     │ Captain America: The Winter Soldier │ Yes                    │
└────────────────────────────┴──────────┴─────────────────────────────────────┴────────────────────────┘

Comments

Questions automatically include a “comment” field. This can be useful for understanding the context of a response, or debugging a non-response.

[9]:

(
    results.filter("persona == 'movie critic'")
    .sort_by("model")
    .select("model", "politically_motivated", "politically_motivated_comment")
    .print(format="rich")
)

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ model                      ┃ answer                 ┃ comment                                                   ┃
┃ .model                     ┃ .politically_motivated ┃ .politically_motivated_comment                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ No                     │ This review appears to be a straightforward critique of   │
│                            │                        │ the film's genre-blending and themes, without any overt   │
│                            │                        │ political agenda or bias influencing the assessment.      │
├────────────────────────────┼────────────────────────┼───────────────────────────────────────────────────────────┤
│ gpt-4o                     │ Yes                    │ The review mentions a "scathing critique of surveillance  │
│                            │                        │ states," which indicates that the film's themes and the   │
│                            │                        │ review itself have political undertones.                  │
└────────────────────────────┴────────────────────────┴───────────────────────────────────────────────────────────┘

Combining questions in a survey

We can combine questions in a ``Survey` <https://docs.expectedparrot.com/en/latest/surveys.html>`__ to administer them together. Here we create some variations on the above question to compare responses:

[10]:

from edsl import QuestionYesNo

q2 = QuestionYesNo(
    question_name = "yn",
    question_text = """
    Read the following movie review and determine whether it is politically motivated.
    Movie: {{ title }}
    Review: {{ review }}
    """
)

[11]:

from edsl import QuestionLinearScale

q3 = QuestionLinearScale(
    question_name = "ls",
    question_text = """
    Read the following movie review and indicate whether it is politically motivated.
    Movie: {{ title }}
    Review: {{ review }}
    """,
    question_options = [0,1,2,3,4,5],
    option_labels = {0:"Not at all", 5:"Very much"}
)

[12]:

from edsl import QuestionList

q4 = QuestionList(
    question_name = "favorites",
    question_text = "List your favorite Marvel movies.",
    max_list_items = 3
)

Survey rules & logic

We can add skip/stop and other rules, and “memory” of other questions in a survey:

[13]:

from edsl import Survey

survey = Survey(questions = [q2, q3, q4])

survey = survey.add_stop_rule(q3, "ls < 3")

[14]:

results = survey.by(s).by(a).by(m).run()

[15]:

(
    results.filter("persona == 'comic book collector'")
    .select("model", "persona", "yn", "ls", "favorites")
    .print(pretty_labels = {
        "answer.yn": "Yes/No version",
        "answer.ls": "Linear scale version",
        "answer.favorites": "Favorites"
    }, format="rich")
)

┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ model.model            ┃ agent.persona        ┃ Yes/No version ┃ Linear scale version ┃ Favorites               ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ claude-3-5-sonnet-202… │ comic book collector │ No             │ 2                    │ None                    │
├────────────────────────┼──────────────────────┼────────────────┼──────────────────────┼─────────────────────────┤
│ gpt-4o                 │ comic book collector │ Yes            │ 3                    │ ['The Avengers',        │
│                        │                      │                │                      │ 'Guardians of the       │
│                        │                      │                │                      │ Galaxy', 'Spider-Man:   │
│                        │                      │                │                      │ Into the Spider-Verse'] │
└────────────────────────┴──────────────────────┴────────────────┴──────────────────────┴─────────────────────────┘

Working with results as datasets

EDSL provides built-in methods for analyzing results, e.g., as SQL tables, dataframes:

[16]:

results.sql("select model, persona, yn, ls, favorites from self", shape="wide")

[16]:

	model	persona	yn	ls	favorites
0	claude-3-5-sonnet-20240620	comic book collector	No	2	None
1	claude-3-5-sonnet-20240620	movie critic	No	2	None
2	gpt-4o	comic book collector	Yes	3	['The Avengers', 'Guardians of the Galaxy', 'S...
3	gpt-4o	movie critic	Yes	3	['Iron Man', 'Black Panther', 'Avengers: Endga...

[17]:

results.to_pandas()

[17]:

	answer.ls	answer.yn	answer.favorites	scenario.year	scenario.review	scenario.title	agent.persona	agent.agent_instruction	agent.agent_name	model.temperature	...	question_options.favorites_question_options	question_type.favorites_question_type	question_type.ls_question_type	question_type.yn_question_type	comment.ls_comment	comment.yn_comment	comment.favorites_comment	generated_tokens.yn_generated_tokens	generated_tokens.favorites_generated_tokens	generated_tokens.ls_generated_tokens
0	2	No	NaN	2014	\n Part superhero flick, part 70s political...	Captain America: The Winter Soldier	comic book collector	You are answering questions as if you were a h...	Agent_1	0.5	...	NaN	list	linear_scale	yes_no	As a comic book collector, I don't see this re...	Comment: As a comic book collector, I don't se...	Task was cancelled.	No\n\nComment: As a comic book collector, I do...	NaN	2\n\nAs a comic book collector, I don't see th...
1	2	No	NaN	2014	\n Part superhero flick, part 70s political...	Captain America: The Winter Soldier	movie critic	You are answering questions as if you were a h...	Agent_2	0.5	...	NaN	list	linear_scale	yes_no	While the review mentions political themes lik...	Comment: This review does not appear to be pol...	Task was cancelled.	No\n\nComment: This review does not appear to ...	NaN	2\n\nWhile the review mentions political theme...
2	3	Yes	['The Avengers', 'Guardians of the Galaxy', 'S...	2014	\n Part superhero flick, part 70s political...	Captain America: The Winter Soldier	comic book collector	You are answering questions as if you were a h...	Agent_1	0.5	...	NaN	list	linear_scale	yes_no	The review mentions the movie's critique of su...	The review mentions a "scathing critique of su...	These movies capture the essence of Marvel's s...	Yes\n\nThe review mentions a "scathing critiqu...	["The Avengers", "Guardians of the Galaxy", "S...	3\n\nThe review mentions the movie's critique ...
3	3	Yes	['Iron Man', 'Black Panther', 'Avengers: Endga...	2014	\n Part superhero flick, part 70s political...	Captain America: The Winter Soldier	movie critic	You are answering questions as if you were a h...	Agent_2	0.5	...	NaN	list	linear_scale	yes_no	The review highlights a "scathing critique of ...	The review mentions that the movie delivers "a...	These films stand out for their groundbreaking...	Yes\n\nThe review mentions that the movie deli...	["Iron Man", "Black Panther", "Avengers: Endga...	3\n\nThe review highlights a "scathing critiqu...

4 rows × 48 columns

[18]:

results.to_csv("marvel_movies_survey.csv")

Posting to the Coop

[19]:

from edsl import Notebook

[20]:

n = Notebook(path = "edsl_intro.ipynb")

[21]:

n.push(description = "Example survey: Using EDSL to analyze content", visibility = "public")

[21]:

{'description': 'Example survey: Using EDSL to analyze content',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/b3c45b82-5d3a-4d79-9e00-0ce1f8b1ff65',
 'uuid': 'b3c45b82-5d3a-4d79-9e00-0ce1f8b1ff65',
 'version': '0.1.33.dev1',
 'visibility': 'public'}

To update an object at the Coop:

[23]:

n = Notebook(path = "edsl_intro.ipynb") # resave

[24]:

n.patch(uuid = "b3c45b82-5d3a-4d79-9e00-0ce1f8b1ff65", value = n)

[24]:

{'status': 'success'}