Intro to EDSL

This notebook provides example code for base components of EDSL, an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Details on the code below are provided in accompanying slides: How to use EDSL.

Technical setup

Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL.

Documentation

Please also see our documentation page for tips, tutorials and more demo notebooks on using EDSL.

Simple example

We start by selecting a question type and constructing a question in the relevant template:

[1]:

from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
    question_name = "marvel_movies",
    question_text = "Do you enjoy Marvel movies?",
    question_options = ["Yes", "No", "I do not know"]
)

We administer a question by calling the run() method. This generates a dataset of Results including the model’s response to the question:

[2]:

results = q.run()

results.select("marvel_movies")

▼ Job Status (2025-03-03 09:34:17)

Job UUID	f9da1b2d-a978-4d2f-ade6-0c6f01d3b882
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/f9da1b2d-a978-4d2f-ade6-0c6f01d3b882
Exceptions Report URL	None
Results UUID	e010863b-febe-4cd2-a580-489800b93a45
Results URL	https://www.expectedparrot.com/content/e010863b-febe-4cd2-a580-489800b93a45

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/e010863b-febe-4cd2-a580-489800b93a45

[2]:

	answer.marvel_movies
0	I do not know

Designing AI agents

We can create personas for agents to answer the question:

[3]:

from edsl import AgentList, Agent

personas = ["comic book collector", "movie critic"]

a = AgentList(
    Agent(traits = {"persona": p}) for p in personas
)

Selecting language models

We can select language models to generate the responses (in the example above we did not specify a model, so GPT 4 preview was used by default):

[4]:

from edsl import ModelList, Model

models = ["gpt-4o", "gemini-1.5-flash"]

m = ModelList(Model(model) for model in models)

Generating results

We add agents and models to a question when running it:

[5]:

results = q.by(a).by(m).run()

results.select("model", "persona", "marvel_movies")

▼ Job Status (2025-03-03 09:34:30)

Job UUID	d79c3a14-cc3b-4a24-ad9b-549c4e04d7cd
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/d79c3a14-cc3b-4a24-ad9b-549c4e04d7cd
Exceptions Report URL	None
Results UUID	c746e1b6-3083-4c74-8131-d4922ef89cee
Results URL	https://www.expectedparrot.com/content/c746e1b6-3083-4c74-8131-d4922ef89cee

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/c746e1b6-3083-4c74-8131-d4922ef89cee

[5]:

	model.model	agent.persona	answer.marvel_movies
0	gpt-4o	comic book collector	Yes
1	gemini-1.5-flash	comic book collector	Yes
2	gpt-4o	movie critic	Yes
3	gemini-1.5-flash	movie critic	Yes

Parameterizing questions

We can use Scenario objects to add data or content to questions:

[6]:

q1 = QuestionMultipleChoice(
    question_name = "politically_motivated",
    question_text = """
    Read the following movie review and determine whether it is politically motivated.
    Movie: {{ scenario.title }}
    Review: {{ scenario.review }}
    """,
    question_options = ["Yes", "No", "I do not know"]
)

EDSL comes with methods for generating scenarios from many data sources, including PDFs, CSVs, docs, images, tables, lists, dicts:

[7]:

from edsl import Scenario

example_review = {
    "year": 2014,
    "title": "Captain America: The Winter Soldier",
    "review": """
    Part superhero flick, part 70s political thriller.
    It's a bold mix that pays off, delivering a scathing
    critique of surveillance states wrapped in spandex
    and shield-throwing action.
    """
}

s = Scenario.from_dict(example_review)

[8]:

results = q1.by(s).by(a).by(m).run()

▼ Job Status (2025-03-03 09:34:39)

Job UUID	ec61cadd-58b3-4e08-8be7-6eace7569750
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/ec61cadd-58b3-4e08-8be7-6eace7569750
Exceptions Report URL	None
Results UUID	b0b2f537-a70b-4ca7-a5d7-a611f9ea7a31
Results URL	https://www.expectedparrot.com/content/b0b2f537-a70b-4ca7-a5d7-a611f9ea7a31

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/b0b2f537-a70b-4ca7-a5d7-a611f9ea7a31

[9]:

(
    results.filter("persona == 'movie critic'")
    .sort_by("model")
    .select("model", "year", "title", "politically_motivated")
)

[9]:

	model.model	scenario.year	scenario.title	answer.politically_motivated
0	gemini-1.5-flash	2014	Captain America: The Winter Soldier	No
1	gpt-4o	2014	Captain America: The Winter Soldier	Yes

Comments

Questions automatically include a “comment” field. This can be useful for understanding the context of a response, or debugging a non-response.

[10]:

(
    results.filter("persona == 'movie critic'")
    .sort_by("model")
    .select("model", "politically_motivated", "politically_motivated_comment")
)

[10]:

	model.model	answer.politically_motivated	comment.politically_motivated_comment
0	gemini-1.5-flash	No	The review focuses on the film's genre blending and its thematic exploration of surveillance states. While the themes touched upon have political undertones, the review itself doesn't explicitly endorse or condemn any specific political ideology or party. It's a critique of a concept, not a political stance.
1	gpt-4o	Yes	The review mentions a "scathing critique of surveillance states," indicating that the movie's themes are politically motivated.

Combining questions in a survey

We can combine questions in a ``Survey` <https://docs.expectedparrot.com/en/latest/surveys.html>`__ to administer them together. Here we create some variations on the above question to compare responses:

[11]:

from edsl import QuestionYesNo

q2 = QuestionYesNo(
    question_name = "yn",
    question_text = """
    Read the following movie review and determine whether it is politically motivated.
    Movie: {{ scenario.title }}
    Review: {{ scenario.review }}
    """
)

[12]:

from edsl import QuestionLinearScale

q3 = QuestionLinearScale(
    question_name = "ls",
    question_text = """
    Read the following movie review and indicate whether it is politically motivated.
    Movie: {{ scenario.title }}
    Review: {{ scenario.review }}
    """,
    question_options = [0,1,2,3,4,5],
    option_labels = {0:"Not at all", 5:"Very much"}
)

[13]:

from edsl import QuestionList

q4 = QuestionList(
    question_name = "favorites",
    question_text = "List your favorite Marvel movies.",
    max_list_items = 3
)

Survey rules & logic

We can add skip/stop and other rules, and “memory” of other questions in a survey:

[14]:

from edsl import Survey

survey = Survey(questions = [q2, q3, q4])

survey = survey.add_stop_rule(q3, "ls < 3")

[15]:

results = survey.by(s).by(a).by(m).run()

▼ Job Status (2025-03-03 09:34:58)

Job UUID	e81d70e2-1430-4f6b-a02c-9769a2b388a0
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/e81d70e2-1430-4f6b-a02c-9769a2b388a0
Exceptions Report URL	None
Results UUID	38cbd8ba-974e-4422-b396-bf0c89e530f4
Results URL	https://www.expectedparrot.com/content/38cbd8ba-974e-4422-b396-bf0c89e530f4

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/38cbd8ba-974e-4422-b396-bf0c89e530f4

[16]:

(
    results.filter("persona == 'comic book collector'")
    .select("model", "persona", "yn", "ls", "favorites")
    .print(pretty_labels = {
        "answer.yn": "Yes/No version",
        "answer.ls": "Linear scale version",
        "answer.favorites": "Favorites"
    })
)

[16]:

	model.model	agent.persona	Yes/No version	Linear scale version	Favorites
0	gpt-4o	comic book collector	Yes	3	['Iron Man', 'The Avengers', 'Guardians of the Galaxy']
1	gemini-1.5-flash	comic book collector	Yes	1	nan

Working with results as datasets

EDSL provides built-in methods for analyzing results, e.g., as SQL tables, dataframes:

[17]:

results.sql("select model, persona, yn, ls, favorites from self")

[17]:

	model	persona	yn	ls	favorites
0	gpt-4o	comic book collector	Yes	3	['Iron Man', 'The Avengers', 'Guardians of the Galaxy']
1	gemini-1.5-flash	comic book collector	Yes	1	nan
2	gpt-4o	movie critic	Yes	3	['Avengers: Endgame', 'Black Panther', 'Guardians of the Galaxy']
3	gemini-1.5-flash	movie critic	Yes	1	nan

[18]:

results.to_pandas()

[18]:

	answer.favorites	answer.ls	answer.yn	scenario.title	scenario.review	scenario.year	agent.agent_name	agent.persona	agent.agent_index	...	comment.yn_comment	generated_tokens.ls_generated_tokens	generated_tokens.yn_generated_tokens	generated_tokens.favorites_generated_tokens	cache_used.yn_cache_used	cache_used.ls_cache_used	cache_used.favorites_cache_used	cache_keys.yn_cache_key	cache_keys.favorites_cache_key	cache_keys.ls_cache_key
0	['Iron Man', 'The Avengers', 'Guardians of the...	3	Yes	Captain America: The Winter Soldier	\n Part superhero flick, part 70s political...	2014	Agent_9	comic book collector	0	...	The review mentions a "scathing critique of su...	3\n\nThe review highlights the film's critique...	Yes\n\nThe review mentions a "scathing critiqu...	["Iron Man", "The Avengers", "Guardians of the...	True	True	True	28be7af4ca7266aeaaf6dfb1939f2830	4a65448cf57b2e2529af3a3fe11ad8b0	f34701a54317094ac2145039c54d2e82
1	NaN	1	Yes	Captain America: The Winter Soldier	\n Part superhero flick, part 70s political...	2014	Agent_10	comic book collector	0	...	Look, I love a good Cap movie as much as the n...	1\n\nThe review explicitly mentions a "scathin...	Yes\n\nLook, I love a good Cap movie as much a...	NaN	True	True	NaN	455b0c2040a35ed5ee2b5653a939e8ad	NaN	f97dfa012a93b0722bd0e173651a1523
2	['Avengers: Endgame', 'Black Panther', 'Guardi...	3	Yes	Captain America: The Winter Soldier	\n Part superhero flick, part 70s political...	2014	Agent_11	movie critic	1	...	The review highlights the film's critique of s...	3 \nThe review highlights a critique of surve...	Yes\n\nThe review highlights the film's critiq...	["Avengers: Endgame", "Black Panther", "Guardi...	True	True	True	502a250f2082fec0421839e48967b5d6	595a2e86d433057894b2609f4e058586	b24392669b17dbf2ddcaf21067eafae9
3	NaN	1	Yes	Captain America: The Winter Soldier	\n Part superhero flick, part 70s political...	2014	Agent_12	movie critic	1	...	The review explicitly mentions a "scathing cri...	1\n\nThe review directly mentions a "scathing ...	Yes\n\nThe review explicitly mentions a "scath...	NaN	True	True	NaN	6a0c7ae0d01760ce2648dae44cbcfa82	NaN	b980819394531b86e6bc4ac8e5e1add3

4 rows × 62 columns

[19]:

results.to_csv("marvel_movies_survey.csv")

File written to marvel_movies_survey.csv

Posting to the Coop

[21]:

from edsl import Notebook

nb = Notebook(path = "edsl_intro.ipynb")

if refresh := False:
    nb.push(
        description = "Example survey: Using EDSL to analyze content",
        alias = "example-edsl-notebook",
        visibility = "public"
    )
else:
    nb.patch('93a3ff75-5a5d-4a69-906d-8626e9bc5321', value = nb)