Intro to EDSL

This notebook provides example code for base components of EDSL, an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Details on the code below are provided in accompanying slides: How to use EDSL.

Technical setup

Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL.

Documentation

Please also see our documentation page for tips, tutorials and more demo notebooks on using EDSL.

Simple example

We start by selecting a question type and constructing a question in the relevant template:

from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
    question_name = "marvel_movies",
    question_text = "Do you enjoy Marvel movies?",
    question_options = ["Yes", "No", "I do not know"]
)

We administer a question by calling the run() method. This generates a dataset of Results including the model’s response to the question:

results = q.run()

results.select("marvel_movies")

▼ Job Status (2025-02-07 20:01:02)

Job UUID	c62b796c-f4e0-4028-b256-9b64e6688898
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/c62b796c-f4e0-4028-b256-9b64e6688898
Exceptions Report URL	None
Results UUID	a45fb248-29b1-4339-a2d3-7aee1aaa1da3
Results URL	https://www.expectedparrot.com/content/a45fb248-29b1-4339-a2d3-7aee1aaa1da3

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/a45fb248-29b1-4339-a2d3-7aee1aaa1da3

	answer.marvel_movies
0	I do not know

Designing AI agents

We can create personas for agents to answer the question:

from edsl import AgentList, Agent

personas = ["comic book collector", "movie critic"]

a = AgentList(
    Agent(traits = {"persona": p}) for p in personas
)

Selecting language models

We can select language models to generate the responses (in the example above we did not specify a model, so GPT 4 preview was used by default):

from edsl import ModelList, Model

models = ["gpt-4o", "gemini-1.5-flash"]

m = ModelList(Model(model) for model in models)

Generating results

We add agents and models to a question when running it:

results = q.by(a).by(m).run()

results.select("model", "persona", "marvel_movies")

▼ Job Status (2025-02-07 20:01:43)

Job UUID	4977ba4e-8572-4ce0-9281-56475199e4d4
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/4977ba4e-8572-4ce0-9281-56475199e4d4
Exceptions Report URL	None
Results UUID	872be459-3e97-4bfc-a796-e198c085e8b6
Results URL	https://www.expectedparrot.com/content/872be459-3e97-4bfc-a796-e198c085e8b6

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/872be459-3e97-4bfc-a796-e198c085e8b6

	model.model	agent.persona	answer.marvel_movies
0	gpt-4o	comic book collector	Yes
1	gemini-1.5-flash	comic book collector	Yes
2	gpt-4o	movie critic	Yes
3	gemini-1.5-flash	movie critic	Yes

Parameterizing questions

We can use Scenario objects to add data or content to questions:

q1 = QuestionMultipleChoice(
    question_name = "politically_motivated",
    question_text = """
    Read the following movie review and determine whether it is politically motivated.
    Movie: {{ title }}
    Review: {{ review }}
    """,
    question_options = ["Yes", "No", "I do not know"]
)

EDSL comes with methods for generating scenarios from many data sources, including PDFs, CSVs, docs, images, tables, lists, dicts:

from edsl import Scenario

example_review = {
    "year": 2014,
    "title": "Captain America: The Winter Soldier",
    "review": """
    Part superhero flick, part 70s political thriller.
    It's a bold mix that pays off, delivering a scathing
    critique of surveillance states wrapped in spandex
    and shield-throwing action.
    """
}

s = Scenario.from_dict(example_review)

results = q1.by(s).by(a).by(m).run()

▼ Job Status (2025-02-07 20:01:59)

Job UUID	58a63ef0-269b-4709-8c16-148e59bb01a8
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/58a63ef0-269b-4709-8c16-148e59bb01a8
Exceptions Report URL	None
Results UUID	a1816206-731e-402f-be7f-2e144395adba
Results URL	https://www.expectedparrot.com/content/a1816206-731e-402f-be7f-2e144395adba

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/a1816206-731e-402f-be7f-2e144395adba

(
    results.filter("persona == 'movie critic'")
    .sort_by("model")
    .select("model", "year", "title", "politically_motivated")
)

	model.model	scenario.year	scenario.title	answer.politically_motivated
0	gemini-1.5-flash	2014	Captain America: The Winter Soldier	No
1	gpt-4o	2014	Captain America: The Winter Soldier	Yes

Comments

Questions automatically include a “comment” field. This can be useful for understanding the context of a response, or debugging a non-response.

(
    results.filter("persona == 'movie critic'")
    .sort_by("model")
    .select("model", "politically_motivated", "politically_motivated_comment")
)

	model.model	answer.politically_motivated	comment.politically_motivated_comment
0	gemini-1.5-flash	No	The review focuses on the film's genre blending and its thematic exploration of surveillance states. While the subject matter has political implications, the review itself doesn't explicitly endorse or condemn any specific political ideology or party. It's a critique of a concept, not a political stance.
1	gpt-4o	Yes	The review highlights the film's critique of surveillance states, which is a political theme, suggesting that the review is politically motivated.

Combining questions in a survey

We can combine questions in a ``Survey` <https://docs.expectedparrot.com/en/latest/surveys.html>`__ to administer them together. Here we create some variations on the above question to compare responses:

from edsl import QuestionYesNo

q2 = QuestionYesNo(
    question_name = "yn",
    question_text = """
    Read the following movie review and determine whether it is politically motivated.
    Movie: {{ title }}
    Review: {{ review }}
    """
)

from edsl import QuestionLinearScale

q3 = QuestionLinearScale(
    question_name = "ls",
    question_text = """
    Read the following movie review and indicate whether it is politically motivated.
    Movie: {{ title }}
    Review: {{ review }}
    """,
    question_options = [0,1,2,3,4,5],
    option_labels = {0:"Not at all", 5:"Very much"}
)

from edsl import QuestionList

q4 = QuestionList(
    question_name = "favorites",
    question_text = "List your favorite Marvel movies.",
    max_list_items = 3
)

Survey rules & logic

We can add skip/stop and other rules, and “memory” of other questions in a survey:

from edsl import Survey

survey = Survey(questions = [q2, q3, q4])

survey = survey.add_stop_rule(q3, "ls < 3")

results = survey.by(s).by(a).by(m).run()

▼ Job Status (2025-02-07 20:02:25)

Job UUID	5aca03bf-d1bd-4fb0-984e-c8f53d8a5a08
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/5aca03bf-d1bd-4fb0-984e-c8f53d8a5a08
Exceptions Report URL	None
Results UUID	084c5a89-3bc3-4f7f-b037-34445bfb27a9
Results URL	https://www.expectedparrot.com/content/084c5a89-3bc3-4f7f-b037-34445bfb27a9

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/084c5a89-3bc3-4f7f-b037-34445bfb27a9

(
    results.filter("persona == 'comic book collector'")
    .select("model", "persona", "yn", "ls", "favorites")
    .print(pretty_labels = {
        "answer.yn": "Yes/No version",
        "answer.ls": "Linear scale version",
        "answer.favorites": "Favorites"
    })
)

	model.model	agent.persona	Yes/No version	Linear scale version	Favorites
0	gpt-4o	comic book collector	Yes	4	['Avengers: Endgame', 'Spider-Man: Into the Spider-Verse', 'Guardians of the Galaxy']
1	gemini-1.5-flash	comic book collector	No	1	nan

Working with results as datasets

EDSL provides built-in methods for analyzing results, e.g., as SQL tables, dataframes:

results.sql("select model, persona, yn, ls, favorites from self")

	model	persona	yn	ls	favorites
0	gpt-4o	comic book collector	Yes	4	['Avengers: Endgame', 'Spider-Man: Into the Spider-Verse', 'Guardians of the Galaxy']
1	gemini-1.5-flash	comic book collector	No	1	nan
2	gpt-4o	movie critic	Yes	4	['Avengers: Endgame', 'Black Panther', 'Guardians of the Galaxy']
3	gemini-1.5-flash	movie critic	Yes	1	nan

results.to_pandas()

	answer.ls	answer.yn	answer.favorites	scenario.title	scenario.year	scenario.review	scenario.scenario_index	agent.agent_index	agent.agent_name	agent.agent_instruction	...	comment.favorites_comment	generated_tokens.yn_generated_tokens	generated_tokens.ls_generated_tokens	generated_tokens.favorites_generated_tokens	cache_used.ls_cache_used	cache_used.yn_cache_used	cache_used.favorites_cache_used	cache_keys.yn_cache_key	cache_keys.ls_cache_key	cache_keys.favorites_cache_key
0	4	Yes	['Avengers: Endgame', 'Spider-Man: Into the Sp...	Captain America: The Winter Soldier	2014	\n Part superhero flick, part 70s political...	0	0	Agent_9	You are answering questions as if you were a h...	...	Avengers: Endgame is an epic conclusion to the...	Yes\n\nThe review mentions a "scathing critiqu...	4 \nThe review highlights the movie's critiqu...	["Avengers: Endgame", "Spider-Man: Into the Sp...	True	True	True	28be7af4ca7266aeaaf6dfb1939f2830	f34701a54317094ac2145039c54d2e82	4a65448cf57b2e2529af3a3fe11ad8b0
1	1	No	NaN	Captain America: The Winter Soldier	2014	\n Part superhero flick, part 70s political...	0	0	Agent_10	You are answering questions as if you were a h...	...	Task was cancelled.	No\n\nLook, I'm not saying the movie *doesn't*...	NaN	NaN	NaN	False	NaN	455b0c2040a35ed5ee2b5653a939e8ad	NaN	NaN
2	4	Yes	['Avengers: Endgame', 'Black Panther', 'Guardi...	Captain America: The Winter Soldier	2014	\n Part superhero flick, part 70s political...	0	1	Agent_11	You are answering questions as if you were a h...	...	These films stand out for their compelling sto...	Yes\n\nThe review describes the movie as deliv...	4 \nThe review highlights a critique of surve...	["Avengers: Endgame", "Black Panther", "Guardi...	True	True	True	502a250f2082fec0421839e48967b5d6	b24392669b17dbf2ddcaf21067eafae9	595a2e86d433057894b2609f4e058586
3	1	Yes	NaN	Captain America: The Winter Soldier	2014	\n Part superhero flick, part 70s political...	0	1	Agent_12	You are answering questions as if you were a h...	...	Task was cancelled.	Yes\n\nThe review explicitly mentions a "scath...	NaN	NaN	NaN	False	NaN	6a0c7ae0d01760ce2648dae44cbcfa82	NaN	NaN

4 rows × 62 columns

results.to_csv("marvel_movies_survey.csv")

File written to marvel_movies_survey.csv

Posting to the Coop

from edsl import Notebook

n = Notebook(path = "edsl_intro.ipynb")

info = n.push(description = "Example survey: Using EDSL to analyze content", visibility = "public")
info

{'description': 'Example survey: Using EDSL to analyze content',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/0d416fec-133f-4a93-a87d-5ae398fb9356',
 'uuid': '0d416fec-133f-4a93-a87d-5ae398fb9356',
 'version': '0.1.43.dev1',
 'visibility': 'public'}

To update an object at the Coop:

n = Notebook(path = "edsl_intro.ipynb") # resave

n.patch(uuid = info["uuid"], value = n)

{'status': 'success'}