Exploring free text responses

This notebook provide sample EDSL code demonstrating ways of modifying instructions for free response questions.

EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Creating free text questions

[1]:

from edsl import QuestionFreeText

q1 = QuestionFreeText(
    question_name="pasttime",
    question_text="What is your favorite pasttime? {{ instruction }}",
)

q2 = QuestionFreeText(
    question_name="vacation",
    question_text="What is your favorite vacation destination? {{ instruction }}",
)

Parameterizing the questions with special instructions

[2]:

from edsl import ScenarioList, Scenario

special_instructions = [
    "Be as specific as possible.",
    "Be concise!",
    "Wax poetic here.",
]

scenarios = ScenarioList.from_list("instruction", special_instructions)

Creating agent personas with specific survey contexts

[3]:

from edsl import AgentList, Agent

personas = [
    "You are middle-aged.",
    "You are a senior citizen.",
    "You are a young adult.",
]

instructions = [
    "You are handwriting answers in a paper survey.",
    "You are typing answers in an online survey.",
    "You are providing answers verbally to a researcher in a live interview.",
]

agents = AgentList(
    Agent(traits={"persona": p}, instruction=i) for p in personas for i in instructions
)

Selecting LLMs

[4]:

from edsl import ModelList, Model

models = ModelList(
    Model(m) for m in ["gpt-4o", "gemini-1.5-flash"]
)

Administering the survey

[5]:

from edsl import Survey

survey = Survey(questions = [q1, q2])

results = survey.by(scenarios).by(agents).by(models).run()

▼ Job Status (2024-12-28 11:24:56)

Job UUID	32aa87b0-bb7c-41e1-8069-0df615ceffb4
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/32aa87b0-bb7c-41e1-8069-0df615ceffb4
Error Report URL	None
Results UUID	28598691-dafd-4909-9dc3-4503d0399612
Results URL	None

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/28598691-dafd-4909-9dc3-4503d0399612

[6]:

(
    results
    .filter("instruction == 'Be concise!' and model.model == 'gpt-4o'")
    .sort_by("model", "persona", "agent_instruction")
    .select("model", "persona", "agent_instruction", "scenario.*", "answer.*")
)

[6]:

	model.model	agent.persona	agent.agent_instruction	scenario.instruction	answer.vacation	answer.pasttime
0	gpt-4o	You are a senior citizen.	You are handwriting answers in a paper survey.	Be concise!	I love the peace and beauty of the countryside.	Gardening.
1	gpt-4o	You are a senior citizen.	You are providing answers verbally to a researcher in a live interview.	Be concise!	I love visiting the serene beaches of Hawaii.	Gardening.
2	gpt-4o	You are a senior citizen.	You are typing answers in an online survey.	Be concise!	The serene beaches of Hawaii.	Gardening.
3	gpt-4o	You are a young adult.	You are handwriting answers in a paper survey.	Be concise!	Japan - for its unique culture and beautiful landscapes.	Playing video games.
4	gpt-4o	You are a young adult.	You are providing answers verbally to a researcher in a live interview.	Be concise!	I love visiting Japan for its mix of tradition and modernity.	I love playing video games.
5	gpt-4o	You are a young adult.	You are typing answers in an online survey.	Be concise!	Japan.	Exploring new music.
6	gpt-4o	You are middle-aged.	You are handwriting answers in a paper survey.	Be concise!	Italy.	Reading historical novels.
7	gpt-4o	You are middle-aged.	You are providing answers verbally to a researcher in a live interview.	Be concise!	I love visiting Italy for its rich history and delicious food.	Reading historical fiction.
8	gpt-4o	You are middle-aged.	You are typing answers in an online survey.	Be concise!	The Mediterranean.	Reading historical fiction.

Posting to Coop

[7]:

from edsl import Notebook

n = Notebook("free_responses.ipynb")

n.push(description = "Free responses example", visibility = "public")

[7]:

{'description': 'Free responses example',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/79161600-3e8a-42a2-9c1e-4b4d7ec51223',
 'uuid': '79161600-3e8a-42a2-9c1e-4b4d7ec51223',
 'version': '0.1.39.dev2',
 'visibility': 'public'}