Prolific studies

More information about the humanize method for generating a shareable web version of your survey for human respondents is available at the Humanize page. This method allows you to collect responses from your own respondents. The responses can then be combined with AI-generated responses (and Prolific responses) for analysis. Please reach out to us if you have any questions or feature requests, or are interested in testing new features for validating AI responses with participants! You can post a message at our Discord channel or send us an email at [email protected] - thank you!

How it works

You can create a Prolific study for your EDSL survey from your workspace (in code) or from your account dashboard (interactively). You can also choose whether to use your own Prolific account key or your Expected Parrot key (by default).

Expected Parrot’s key (default): the study’s cost is paid from your credits balance, and each study can recruit up to your participant limit.
Your own Prolific key: Prolific bills your Prolific account directly, no credits are used for recruitment, and no participant limit applies.

Code-based workflow

Create a survey in a notebook

Create a survey in EDSL:

Construct Questions and pass them to Surveys with desired logic.
Create AI Agents with desired traits.
Specify Language Models to generate responses.
Use the run method to administer a survey with desired agents and models.

This generates a formatted dataset of Results that you can analyze with built-in methods. The survey, agents and results are also automatically added to your Expected Parrot account, where you can access them interactively.

Use humanize to generate a project and web surveys

Use the humanize method to generate a Project for your survey and a shareable web version. You can optionally pass a project_name, survey_description and survey_alias to customize the project details (see example below).

Use Expected Parrot to launch Prolific studies

Create a Coop client object to access your account. Then use Coop methods to create a Prolific study for your survey and gather human responses:

get_prolific_study_settings: Check which account your new studies will run on, whether they are paid in credits, and your participant limit.
create_prolific_study: Create a new study for the project_uuid and specify the study details (name, description, num_participant, estimated_completion_time (minutes), participant_payment_cents), optional filters for targeting specific demographics and Prolific participants (IDs list), and an optional approval_mode (see Submission approval).
get_prolific_study: Retrieve the study details for the project_uuid and study_id.
list_prolific_filters: Show all the available filters for targeting specific demographics and characteristics of participants. Then use the find method to inspect the available filters by filter_id and the create_study_filter method to create a filter for your study with the desired parameters.
update_prolific_study: Update a draft study’s details, filters or approval mode.
preflight_prolific_study: Check a draft study before publishing it, without publishing or spending anything.
publish_prolific_study: Publish the study and make it available for participants.
get_prolific_study_responses: Retrieve the responses from the study.
approve_prolific_study_submission: Approve the responses from a study that uses manual review.
reject_prolific_study_submission: Reject the responses from a study that uses manual review.

If you have also shared the web link with other respondents, you can gather all human responses using the get_project_human_responses method.

Interactive workflow

Create a survey project

Log in to your account. Create a project for a saved survey at your Projects page or create a new survey and select Run survey to view options for running the survey with AI agents and language models, generate a web version of the survey, and launch Prolific studies with human participants.

Create a Prolific study

Select the Run with humans option and fill in the study details:

Study name (Shown to participants)
Description (Shown to participants)
Required number of participants: Specify how many participants you want to recruit. On Expected Parrot’s key, this can’t exceed your participant limit.
Estimated completion time: Provide an estimate of how long it will take participants to complete the survey.
Participant payment amount: Set the payment amount for participants.
Participant allowlist: Optionally, you can specify a list of Prolific IDs of participants who are allowed to take part in the study.
Configure filters: Optionally, you can set filters to target specific demographics or characteristics of participants (e.g., age, gender, ethnicity, languages, location, education, work status, etc.).

Note: The Study URL will be generated automatically. You can open it to view the web version of the survey that will be sent to participants. This is the same link displayed when you select the Web survey option at your project page (or when you use the humanize method in code). You can also share this link with any other respondents on your own to gather their responses.

Click the Create study button to finalize the study creation.

Launch the study

Find the new study at your project dashboard and select the option to launch it. Responses will automatically appear at your project dashboard where you can review and approve them.

Access results

Copy the project UUID and use it to access the participant responses in your EDSL code (see example below).

Example (code-based workflow)

The example below demonstrates how to create a survey with different question types, run it with an AI agent and language model, and then gather human responses. Create and run a survey with AI agents and models: See the Questions, Surveys, Agents and Language Models sections for more details on each of these components.

# Import modules from EDSL
from edsl import (
    QuestionYesNo,
    QuestionNumerical,
    QuestionLinearScale,
    Survey,
    Agent,
    Model,
    Coop
)

# Create a survey with different question types
q1 = QuestionYesNo(
    question_name="drive",
    question_text="Do you drive?"
)

q2 = QuestionNumerical(
    question_name="count",
    question_text="How many vehicles do you currently own or lease?",
)

q3 = QuestionLinearScale(
    question_name="enjoy",
    question_text="On a scale from 1 to 10, how much do you enjoy driving?",
    question_options=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    option_labels={1: "Hate it", 10: "Love it"},
)

 # Create a survey with the questions
survey = Survey(questions=[q1, q2, q3])

# Create an AI agent to respond to the survey
agent = Agent(
    traits={
        "persona":"Middle-aged mom",
        "location":"Massachusetts",
    }
)

# Select a language model to generate the responses
model = Model("gemini-1.5-pro", service_name="google")

# Run the survey with the AI agent and model
llm_results = survey.by(agent).by(model).run()

# Inspect the results
llm_results.select("persona", "location", "drive", "count", "enjoy")

Example output:

agent.persona	Middle-aged mom
agent.location	Massachusetts
answer.drive	Yes
answer.count	2
answer.enjoy	4

Create a project and web version of the survey for human respondents: Use the humanize method to create a project for the survey and a web version that can be shared with respondents (your own and via Prolific):

# Generate a web version of the survey for human respondents
project = survey.humanize(
    project_name = "Vehicle Ownership Survey",  # optional, defaults to the survey name
    survey_description = "A survey on vehicle ownership and driving habits.",  # optional
    survey_alias = "vehicle-ownership-survey"  # optional, used to create a unique URL in addition to Expected Parrot UUID URL
)

# Inspect the project details
project

Example output:

{'project_name': 'Vehicle Ownership Survey',
'uuid': '369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'admin_url': 'https://www.expectedparrot.com/home/projects/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'respondent_url': 'https://www.expectedparrot.com/respond/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc'}

The respondent_url can be shared with any human respondents (it is the same link shared with Prolific participants). Create a study for the project to launch studies on Prolific:

# Create an Expected Parrot client instance
from edsl import Coop
coop = Coop()

project_uuid = project["uuid"]

study = coop.create_prolific_study(
    project_uuid=project_uuid,
    name="Vehicle Ownership Study",
    description="A study on vehicle ownership and driving habits.",
    num_participants=1,
    estimated_completion_time_minutes=1,  # in minutes
    participant_payment_cents=50,  # payment amount in cents
    # approval_mode="manual",  # optional; submissions are approved automatically by default
)

# Inspect the study details
study

Example output:

{'study_id': '684307d08015cf8252ca77cf',
'status': 'UNPUBLISHED',
'admin_url': 'https://www.expectedparrot.com/home/projects/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc/prolific-studies/684307d08015cf8252ca77cf',
'respondent_url': 'https://www.expectedparrot.com/respond/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'name': 'Vehicle Ownership Study',
'description': 'A study on vehicle ownership and driving habits.',
'num_participants': 1,
'estimated_completion_time_minutes': 1,
'participant_payment_cents': 50,
'total_cost_cents': 71,
'device_compatibility': ['desktop', 'mobile', 'tablet'],
'peripheral_requirements': [],
'filters': [],
'approval_mode': 'automatic'}

Submission approval: By default, a submission is approved and the participant is paid as soon as they enter the completion code. To review submissions yourself, pass approval_mode="manual" when you create the study. With manual review, participants are paid when you approve their submissions, or when Prolific automatically approves them 21 days after completion. You can change the approval mode while the study is still a draft. It can’t be changed after the study is published:

coop.update_prolific_study(project_uuid, study["study_id"], approval_mode="manual")

Inspect the study details:

# Get the study details
study_id = study["id"]

coop.get_prolific_study(project_uuid, study_id)

Example output:

{'study_id': '684307d08015cf8252ca77cf',
'status': 'UNPUBLISHED',
'admin_url': 'https://www.expectedparrot.com/home/projects/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc/prolific-studies/684307d08015cf8252ca77cf',
'respondent_url': 'https://www.expectedparrot.com/respond/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'name': 'Vehicle Ownership Study',
'description': 'A study on vehicle ownership and driving habits.',
'num_participants': 1,
'estimated_completion_time_minutes': 1,
'participant_payment_cents': 50,
'total_cost_cents': 71,
'device_compatibility': ['desktop', 'mobile', 'tablet'],
'peripheral_requirements': [],
'filters': []}

Applying filters: You can select and apply filters for targeting specific demographics and characteristics of participants. To see a list of available filters:

filters = coop.list_prolific_filters()

filters

See the notebook above for the full list of available filters. Keys to filter on: ‘range_filter_max’, ‘type’, ‘select_filter_num_options’, ‘select_filter_options’, ‘question’, ‘filter_id’, ‘title’, ‘range_filter_min’ There are two types of filters available: select and range. Select filters will have select_filter_num_options and select_filter_options:

filter_id = "current-country-of-residence"  # Example filter ID

filters.find(filter_id)

(See notebook for output.) Range filters will have range_filter_min and range_filter_max:

filter_id = "age"  # Example filter ID

filters.find(filter_id)

(See notebook for output.) Create a filter by passing the id and desired parameters (note: you can also do this when you create the study). Example range filter:

# Create a filter for the study
age_filter = filters.create_study_filter(
    filter_id="age",  # Example filter ID
    min=40,
    max=60
)

# Inspect the created filter
age_filter

Output:

{'filter_id': 'age', 'selected_range': {'lower': 40, 'upper': 60}}

Example select filter:

# Create a filter for the study
country_filter = filters.create_study_filter(
    filter_id="current-country-of-residence",  # Example filter ID
    values=["United States", "Canada"]
)

# Inspect the created filter
country_filter

Output:

{'filter_id': 'current-country-of-residence', 'selected_values': ['1', '45']}

Update the study with the filter:

# Update the study with the filter
coop.update_prolific_study(
    project_uuid=project_uuid,
    study_id=study_id,
    filters=[
        age_filter,
        country_filter
    ]  # List of filters to apply to the study
)

Example output:

{'study_id': '684307d08015cf8252ca77cf',
'status': 'UNPUBLISHED',
'admin_url': 'https://www.expectedparrot.com/home/projects/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc/prolific-studies/684307d08015cf8252ca77cf',
'respondent_url': 'https://www.expectedparrot.com/respond/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'name': 'Vehicle Ownership Study',
'description': 'A study on vehicle ownership and driving habits.',
'num_participants': 1,
'estimated_completion_time_minutes': 1,
'participant_payment_cents': 50,
'total_cost_cents': 71,
'device_compatibility': ['desktop', 'mobile', 'tablet'],
'peripheral_requirements': [],
'filters': [{'filter_id': 'age', 'min': 40, 'max': 60},
{'filter_id': 'current-country-of-residence',
'values': ['United States', 'Canada']}]}

Publish the study: The above steps have created a draft study. To make it available for participants (note: on Expected Parrot’s key, this consumes credits):

# Publish the study
coop.publish_prolific_study(project_uuid, study_id)

Retrieve the study responses: After the study is published and participants have completed it, you can retrieve the responses, together with the submission ID for each response. For a study that uses manual review, you can use the submission IDs to approve or reject submissions:

# Get the responses from the study
coop.get_prolific_study_responses(project_uuid, study_id)

Approve the responses from the study:

# Approve the responses from the study
submission_id = "1234567890"  # Example submission ID
coop.approve_prolific_study_submission(
    project_uuid=project_uuid,
    study_id=study_id,
    submission_id=submission_id
)

Optionally, you can reject responses if needed:

# Reject the responses from the study
submission_id = "1234567890"  # Example submission ID
coop.reject_prolific_study_submission(
    project_uuid=project_uuid,
    study_id=study_id,
    submission_id=submission_id,
    reason="LOW_EFFORT",
    explanation="I think you may have used AI to complete this submission, as there are no personal thoughts or opinions expressed."
)

Gather all human responses: If you have also shared the web survey link with other respondents, you can gather all human responses from the project:

# Get human responses from the web survey link
human_responses = coop.get_project_human_responses(project_uuid)

# Inspect the human responses
human_responses.select("drive", "count", "enjoy")

Combine the AI-generated results with the human responses:

# Combine results (you can add Results objects for the same survey)
combined_results = llm_results + human_responses

# Inspect the combined results
combined_results.select("persona", "location", "drive", "count", "enjoy")

Costs

Credits for launching Prolific studies on Expected Parrot’s key are deducted from your credits balance. The total cost of a Prolific study is calculated based on the number of participants, the payment amount you set for each participant, and the Prolific platform fee for each response. These costs are displayed at the project page when you create the study, which include the Prolific platform fee and the payment to participants. Studies on your own Prolific key are billed by Prolific to your Prolific account, and no credits are used for recruitment.

Participant limit

Each study on Expected Parrot’s key can recruit up to a set number of participants. Creating a study, raising its number of participants, or publishing it beyond your limit is refused. To check your limit:

from edsl import Coop

coop = Coop()
coop.get_prolific_study_settings()

Example output:

{'key_source': 'ep',
'charged_in_credits': True,
'participant_limit': 100,
'participant_limit_applies': True}

participant_limit applies only when participant_limit_applies is True, i.e. when your new studies run on Expected Parrot’s key. To request a higher limit, email [email protected]. You can view your credits balance at your Credits of your account individual transactions at your Transactions page. Please reach out to us if you have any questions or feature requests! You can post a message at our Discord channel or send us an email at [email protected]

Introduction

Getting Started

Core Concepts

Getting Data

Working with Results

Validating with Humans

No-code Apps

Platform

How-to Guides

Notebook Examples

Developers

How it works

Code-based workflow

Interactive workflow

Example (code-based workflow)

Costs

Participant limit

​How it works

​Code-based workflow

​Interactive workflow

​Example (code-based workflow)

​Costs

​Participant limit

How it works

Code-based workflow

Interactive workflow

Example (code-based workflow)

Costs

Participant limit