Starter Tutorial

This tutorial provides step-by-step instructions for getting started using EDSL (Expected Parrot Domain-Specific Language), an open-source Python library for simulating surveys, experiments and other research tasks using AI agents and large language models. EDSL is developed by Expected Parrot and available under the MIT License. The source code is hosted on GitHub.

Goals of this tutorial

We begin with technical setup: instructions for installing the EDSL library and storing API keys to access language models. Then we demonstrate some of the basic features of EDSL, with examples for constructing and running surveys with agents and models, analyzing responses as datasets, and validating results with human respondents. By the end of this tutorial, you will be able to use EDSL to do each of the following:

Construct various types of questions tailored to your research objectives.
Combine questions into surveys and integrate logical rules to control the survey flow.
Design personas for AI agents to simulate responses to your surveys.
Choose and deploy large language models to generate responses for AI agents.
Analyze results as datasets with built-in analytical tools.
Validate LLM answers with human respondents.

Pre-requisites

EDSL is compatible with Python 3.9 - 3.12. Before starting this tutorial, please ensure that you have a Python environment set up on your machine or in a cloud-based environment, such as Google Colab. You can find instructions for installing Python at the Python Software Foundation.

Recommendations

The code examples in this tutorial are designed to be run in a Jupyter notebook or another Python environment, or in a cloud-based environment such as Google Colab.

If you are using Google Colab, please see additional instructions for setting up EDSL in the Colab setup page in the documentation.

We also recommend using a virtual environment when installing and using EDSL in order to avoid conflicts with other Python packages. You can find instructions for setting up a virtual environment at the Python Packaging Authority.

Installation

To begin using EDSL, you first need to install the library. This can either be done locally on your machine or in a cloud-based environment, such as Google Colab. Once you have decided where to install EDSL, you can choose to whether install it from PyPI or GitHub:

From PyPI

Install EDSL directly using pip, which is straightforward and recommended for most users. We also recommend using a virtual environment to manage your Python packages (see Recommendations above). Uncomment and run the following command to install EDSL from PyPI:

[1]:

# pip install edsl

If you have already installed EDSL, you can uncomment and run the following code to check that your version is up to date (compare it to the version at PyPI):

[2]:

# pip show edsl

If your version of EDSL is not up to date, uncomment and run the following code to update it:

[3]:

# pip install --upgrade edsl

From GitHub

You can find the source code for EDSL and contribute to the project at GitHub. Installing from GitHub allows you to get the latest updates to EDSL before they are released to a new version at PyPI. This is recommended if you are using new features or contributing to the project. Uncomment and run the following command to install EDSL from GitHub:

[4]:

# pip install git+https://github.com/expectedparrot/edsl.git@main

Accessing LLMs

The next step is to decide how you want to access language models for running surveys. EDSL works with many popular language models that you can choose from to generate responses to your surveys. These models are hosted by various service providers, such as Anthropic, Azure, Bedrock, Deep Infra, Google, Groq, Mistral, OpenAI, Replicate and Together. In order to run a survey, you need to provide API keys for the service providers of models that you want to use. There are two methods for providing API keys to EDSL:

Use an Expected Parrot API key to access all available models
Provide your own API keys from service providers

Create an account

The easiest way to manage your keys is from your Expected Parrot account. Create an account with an email address and then navigate to your Settings page to view your Expected Parrot API key. It is stored automatically and can be regenerated at any time. You will also see options for activating remote inference and caching; this allows your surveys to be run and your results to be stored remotely at the Expected Parrot server instead of your own machine.

Managing keys

If you want to use your own keys to run surveys, navigate to your Keys page and use the options to add keys and optionally share access to them with other users. You can specify which keys to use at any time, and check the current priority of your keys. Your Expected Parrot API key is used by default.

Please see instructions for alternative methods of storing your own API keys.

Note: If you try to run a survey without storing a required API key, you will be provided a link to activate remote inference and use your Expected Parrot API key.

Credits & tokens

Running surveys with language models requires tokens. If you are using your own API keys, service providers will bill you directly. If you are using your Expected Parrot API key to access models, you will need to purchase credits to cover token costs. Please see the model pricing page for details on available models and their current prices.

Note: Your account comes with 100 free credits. You can purchase more credits at any time at your Credits page.

After installing EDSL and storing API keys you are ready to run some examples!

Example: Running a simple question

EDSL comes with a variety of question types that we can choose from based on the form of the response that we want to get back from a model. To see a list of all question types:

[5]:

from edsl import Question

Question.available()

[5]:

	question_type	question_class	example_question
0	checkbox	QuestionCheckBox	Question('checkbox', question_name = """never_eat""", question_text = """Which of the following foods would you eat if you had to?""", min_selections = 2, max_selections = 5, question_options = ['soggy meatpie', 'rare snails', 'mouldy bread', 'panda milk custard', 'McDonalds'], include_comment = False)
1	dict	QuestionDict	Question('dict', question_name = """example""", question_text = """Please provide a simple recipe for hot chocolate.""", answer_keys = ['title', 'ingredients', 'num_ingredients', 'instructions'], value_types = ['str', 'list[str]', 'int', 'str'], value_descriptions = ['The title of the recipe.', 'A list of ingredients.', 'The number of ingredients.', 'The instructions for making the recipe.'], question_presentation = """Please provide a simple recipe for hot chocolate.""", answering_instructions = """Please respond with a dictionary using the following keys: title, ingredients, num_ingredients, instructions. Here are descriptions of the values to provide: - "title": "The title of the recipe." - "ingredients": "A list of ingredients." - "num_ingredients": "The number of ingredients." - "instructions": "The instructions for making the recipe." The values should be formatted in the following types: - "title": "str" - "ingredients": "list[str]" - "num_ingredients": "int" - "instructions": "str" If you do not have a value for a given key, use "null". After the answer, you can put a comment explaining your response on the next line. """)
2	extract	QuestionExtract	Question('extract', question_name = """extract_name""", question_text = """My name is Moby Dick. I have a PhD in astrology, but I'm actually a truck driver""", answer_template = {'name': 'John Doe', 'profession': 'Carpenter'})
3	free_text	QuestionFreeText	Question('free_text', question_name = """how_are_you""", question_text = """How are you?""")
4	functional	QuestionFunctional	Question('functional', question_name = """sum_and_multiply""", question_text = """Calculate the sum of the list and multiply it by the agent trait multiplier.""")
5	likert_five	QuestionLikertFive	Question('likert_five', question_name = """happy_raining""", question_text = """I'm only happy when it rains.""", question_options = ['Strongly disagree', 'Disagree', 'Neutral', 'Agree', 'Strongly agree'])
6	linear_scale	QuestionLinearScale	Question('linear_scale', question_name = """ice_cream""", question_text = """How much do you like ice cream?""", question_options = [1, 2, 3, 4, 5], option_labels = {1: 'I hate it', 5: 'I love it'})
7	list	QuestionList	Question('list', question_name = """list_of_foods""", question_text = """What are your favorite foods?""", max_list_items = None, min_list_items = None)
8	matrix	QuestionMatrix	Question('matrix', question_name = """child_happiness""", question_text = """How happy would you be with different numbers of children?""", question_items = ['No children', '1 child', '2 children', '3 or more children'], question_options = [1, 2, 3, 4, 5], option_labels = {1: 'Very sad', 3: 'Neutral', 5: 'Extremely happy'})
9	multiple_choice	QuestionMultipleChoice	Question('multiple_choice', question_name = """how_feeling""", question_text = """How are you?""", question_options = ['Good', 'Great', 'OK', 'Bad'], include_comment = False)
10	multiple_choice_with_other	QuestionMultipleChoiceWithOther	Question('multiple_choice_with_other', question_name = """how_feeling_with_other""", question_text = """How are you?""", question_options = ['Good', 'Great', 'OK', 'Bad'], include_comment = False)
11	numerical	QuestionNumerical	Question('numerical', question_name = """age""", question_text = """You are a 45 year old man. How old are you in years?""", min_value = 0, max_value = 86.7, include_comment = False)
12	rank	QuestionRank	Question('rank', question_name = """rank_foods""", question_text = """Rank your favorite foods.""", question_options = ['Pizza', 'Pasta', 'Salad', 'Soup'], num_selections = 2)
13	top_k	QuestionTopK	Question('top_k', question_name = """two_fruits""", question_text = """Which of the following fruits do you prefer?""", min_selections = 2, max_selections = 2, question_options = ['apple', 'banana', 'carrot', 'durian'], use_code = True)
14	yes_no	QuestionYesNo	Question('yes_no', question_name = """is_it_equal""", question_text = """Is 5 + 5 equal to 11?""", question_options = ['No', 'Yes'])

We can see the components of a particular question type by importing the question type class and calling the example method on it:

[6]:

from edsl import (
    # QuestionCheckBox,
    # QuestionExtract,
    # QuestionFreeText,
    # QuestionFunctional,
    # QuestionLikertFive,
    # QuestionLinearScale,
    # QuestionList,
    QuestionMultipleChoice,
    # QuestionNumerical,
    # QuestionRank,
    # QuestionTopK,
    # QuestionYesNo
)

q = QuestionMultipleChoice.example() # substitute any question type class name
q

[6]:

QuestionMultipleChoice

	key	value
0	question_name	how_feeling
1	question_text	How are you?
2	question_options:0	Good
3	question_options:1	Great
4	question_options:2	OK
5	question_options:3	Bad
6	include_comment	False
7	question_type	multiple_choice

Here we create a simple multiple choice question of our own:

[7]:

from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
    question_name = "smallest_prime",
    question_text = "Which is the smallest prime number?",
    question_options = [0, 1, 2, 3]
)

We can administer the question to a language model by calling the run method on it. If you have activated remote inference and stored your Expected Parrot API key (see instructions above), the question will be run remotely at the Expected Parrot server. Results are stored at an unlisted Coop page by default; we can also set the visibility to public or private either when we run it or by updating the object (demonstrated in later examples). We can also view a progress report for the job:

[8]:

results = q.run()

⌃ Job Status 🦜

Completed (1 completed, 0 failed)

Job Links

Results

Progress Report

Content

Remote Jobs

Remote Cache

Identifiers

Results UUID:

1fe04d03...b449

Use Results.pull(uuid) to fetch results.

Job UUID:

0df42d02...c9ad

Use Jobs.pull(uuid) to fetch job.