# Agents
Source: https://docs.expectedparrot.com/en/latest/agents

*Agent* objects are used to simulate survey responses for target audiences. They can be created with specified traits, such as personas and relevant attributes for a survey, that are used together with language models to generate answers to questions.

Agents can be created individually or as a list, and updated after creation. They work with any question type for single questions or full surveys.

<Note>
  **AI agents are not real people.** Agent responses are generated by language models based on their training data. They reflect statistical patterns in that data, not the actual opinions of any demographic group. Use AI-simulated responses for prototyping, hypothesis generation, and pre-testing — and validate with real human data when your goal is to measure actual attitudes or behaviors.
</Note>

Agent information is presented to a model in a system prompt; it is delivered together with a user prompt of information about a given question. In the examples below we show how to access these prompts to inspect them before running a survey and in the results that are generated for a survey. Note, however, that certain models do not take system prompts (e.g., OpenAI’s o1). When using a model that does not take a system prompt, agent information should be included in the user prompt.

## Constructing an Agent

An Agent is created by passing a dictionary of traits relevant to the questions, using single values or textual narratives.

For example:

```python theme={null}
from edsl import Agent

agent = Agent(
    traits = {
        "persona": "You are an expert in machine learning.",
        "age": 45,
        "home_state": "Massachusetts"
    })
```

### Simplified constructor with keyword arguments

You can also create agents by passing traits directly as keyword arguments, which provides a more concise syntax:

```python theme={null}
from edsl import Agent

agent = Agent(
    persona="You are an expert in machine learning.",
    age=45,
    home_state="Massachusetts"
)
```

This is equivalent to the traits dictionary approach but more convenient for simple agent creation.

<Note>
  **Note**:

  Note that traits= must be named explicitly in the construction, and the traits must use Python identifiers as keys (e.g., home\_state but not home state or home-state).
</Note>

## Agent names

We can optionally give an agent a name when it is constructed:

```python theme={null}
agent = Agent(
    name = "Ada",
    traits = {
        "persona": "You are an expert in machine learning.",
        "age": 45,
        "home_state": "Massachusetts"
    })
```

If a name is not passed when the agent is created, an agent\_name field is automatically added to the Results that are generated when a survey is run with the agent. This field is a unique identifier for the agent and can be used to filter or group results by agent. It is *not* used in the prompts for generating responses. If you want to use a name in the prompts for generating responses, you can pass it as a trait:

```python theme={null}
agent = Agent(
    traits = {
        "first_name": "Ada",
        "persona": "You are an expert in machine learning.",
        "age": 45,
        "home_state": "Massachusetts"
    })
```

We can see how the agent information is presented to a model by inspecting the system prompt that is generated when we use an agent with a question:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "favorite_food",
    question_text = "What is your favorite food?"
)

job = q.by(agent) # using the agent created above
job.prompts().select("user_prompt", "system_prompt")
```

Output:

| user\_prompt | system\_prompt |
| :- | :- |
| What is your favorite food? | You are answering questions as if you were a human. Do not break character. Your traits: \{‘first\_name’: ‘Ada’, ‘persona’: ‘You are an expert in machine learning.’, ‘age’: 45, ‘home\_state’: ‘Massachusetts’} |

<Note>
  Note that trying to create two agents with the same name or trying to use a key “name” in the traits will raise an error.
</Note>

## Agent lists

Agents can be created collectively and administered a survey together. This is useful for comparing responses for multiple agents.

For example, here we create a list of agents as an AgentList with different combinations of traits:

```python theme={null}
from edsl import AgentList, Agent

ages = [10, 20, 30, 40, 50]
locations = ["New York", "California", "Texas", "Florida", "Washington"]

agents = AgentList(
    Agent(traits = {"age": age, "location": location}) for age, location in zip(ages, locations)
)
```

This code will create a list of agents that can then be used in a survey.

Example code for running a survey with the agents:

```python theme={null}
from edsl import QuestionFreeText, Survey

q = QuestionFreeText(
    question_name = "favorite_food",
    question_text = "What is your favorite food?"
)

survey = Survey(questions = [q])

results = survey.by(agents).run()
```

This will generate a Results object that contains a Result for each agent’s responses to the survey question. Learn more about working with results in the [Results](/en/latest/results#results) section.

## Generating agents from data

An AgentList can be automatically generated from data stored in many source types, including a list, a dictionary, a CSV, TSV or Excel file, a Pandas dataframe, etc.

A general method for this is from\_source() which is called on the AgentList class, takes a data source\_type (csv, excel, pandas, etc.) and a data source, and returns an AgentList object. Optional parameters allow you to specify special instructions, a codebook for the traits, and a name\_field for the agents.

For example, if you have agent data stored in a CSV file, you can create an AgentList from it using the from\_source() method by specifying source\_type=”csv” and the path to the CSV file:

```python theme={null}
from edsl import AgentList

agents = AgentList.from_source(
    source_type="csv",
    file_or_url="agent_data.csv") # replace with your CSV file path
```

If the data source is a CSV or Excel file, the header row is used as the keys for the traits, and can optionally have a column “name” for the agent names. If a different column name should be used for the agent names, it can be specified with the name\_field parameter:

```python theme={null}
from edsl import AgentList

agents = AgentList.from_source(
    source_type="csv",
    file_or_url="agent_data.csv", # replace with your CSV file path
    name_field="first_name" # replace with your column name for agent names
    )
```

A codebook can also be passed to provide descriptions for the traits. It can be useful for providing context to a model about the traits of an agent. For example, if you have a trait “age” and you want to provide more context about what that means, you could use the codebook to specify that “age” refers to the age of the agent in years:

```python theme={null}
from edsl import AgentList

codebook = {
    "age": "The age of the agent in years",
    "location": "The location of the agent"
}

agents = AgentList.from_source(
    source_type="csv",
    file_or_url="agent_data.csv", # replace with your CSV file path
    codebook=codebook
    )
```

Special instructions can also be passed to modify the default instructions that are used with agent traits in the system prompt. For example, if you want all agents to answer in German, you could use the instructions parameter to specify that:

```python theme={null}
from edsl import AgentList

agents = AgentList.from_source(
    source_type="csv",
    file_or_url="agent_data.csv", # replace with your CSV file path
    instructions="Answer in German."
    )
```

### From natural language descriptions

You can generate agents from natural language descriptions using the `from_vibes()` method. This uses an LLM to interpret your descriptions and create appropriate agent traits:

```python theme={null}
from edsl import AgentList

# Generate agents from natural language descriptions
agents = AgentList.from_vibes([
    "A 25-year-old software engineer from San Francisco",
    "A retired teacher in their 60s from rural Maine",
    "A college student studying biology"
])
```

This method is particularly useful when you want to quickly create diverse agents without manually specifying all traits.

### From a list

We can create a simple AgentList from a list using the from\_list() method, which takes a single trait\_name and a list of values for it and returns an agent for each value (each agent has a single trait):

```python theme={null}
from edsl import AgentList

agents = AgentList.from_list(trait_name="age", values=[10, 20, 30, 40])
agents
```

Output:

| age |
| :- |
| 10 |
| 20 |
| 30 |
| 40 |

### From a dictionary

We can create a more complex AgentList from a dictionary using the from\_dict() method. It takes a dictionary with a key agent\_list and a list of dictionaries, each of which must have a traits key with a dictionary of traits and an optional name key for the agent’s name:

```python theme={null}
from edsl import AgentList

data = {
    "agent_list": [
        {"name":"agent1", "traits":{"age": 10, "location": "New York"}},
        {"name":"agent2", "traits":{"age": 20, "location": "California"}},
        {"name":"agent3", "traits":{"age": 30, "location": "Texas"}},
        {"name":"agent4", "traits":{"age": 40, "location": "Florida"}},
        {"name":"agent5", "traits":{"age": 50, "location": "Washington"}}
    ]
}

agents = AgentList.from_dict(data)
agents
```

Output:

| location | age |
| :- | :- |
| New York | 10 |
| California | 20 |
| Texas | 30 |
| Florida | 40 |
| Washington | 50 |

### From a CSV file

We can also create an AgentList from a CSV file using the from\_csv() method. The CSV file must have a header row of Pythonic keys for the traits, and can optionally have a column “name” for the agent names:

```python theme={null}
from edsl import AgentList

# Creating a CSV file with agent data to use as an example
import pandas as pd

data = [
    {"name": "Alice", "age": 25, "city": "New York"},
    {"name": "Bob", "age": 30, "city": "San Francisco"},
    {"name": "Charlie", "age": 35, "city": "Chicago"}
]

df = pd.DataFrame(data)

df.to_csv("agent_data.csv", index=False)

# Creating an AgentList from the CSV file
agents = AgentList.from_csv("agent_data.csv")
agents
```

Output:

| city | age |
| :- | :- |
| New York | 25 |
| San Francisco | 30 |
| Chicago | 35 |

## Dynamic traits function

Agents can also be created with a dynamic\_traits\_function parameter. This function can be used to generate traits dynamically based on the question being asked or the scenario in which the question is asked.

<Note>
  **Note:**

  This method is only available with local inference. It does not work with remote inference.
</Note>

Example:

```python theme={null}
def dynamic_traits_function(question):
    if question.question_name == "age":
        return {"age": 10}
    elif question.question_name == "hair":
        return {"hair": "brown"}

a = Agent(dynamic_traits_function = dynamic_traits_function)
```

When the agent is asked a question about age, the agent will return an age of 10. When asked about hair, the agent will return “brown”. This can be useful for creating agents that can answer questions about different topics without including potentially irrelevant traits in the agent’s traits dictionary.

Note that the traits returned by the function are *not* added to the agent’s traits dictionary.

## Agent direct-answering methods

Agents can also be created with a method that can answer a particular question type directly:

```python theme={null}
from edsl import Agent

a = Agent()

def f(self, question, scenario): return "I am a direct answer."

a.add_direct_question_answering_method(f)
a.answer_question_directly(question = None, scenario = None)
```

Output:

```
I am a direct answer.
```

This can be useful for creating agents that can answer questions directly without needing to use a language model.

## Giving an agent instructions

In addition to traits, agents can be given detailed instructions on how to answer questions. The instruction parameter can be used to omit or modify the default instructions that are used with agent traits in the system prompt.

For example:

```python theme={null}
from edsl import Agent

a = Agent(traits = {"age": 10}, instruction = "Answer in German.")
a.instruction
```

Output:

```
Answer in German.
```

When the agent is assigned to a survey, the special instruction will be added to the prompts for generating responses. We can create a Job object to inspect the prompts (user and system) that will be used to generate responses:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "favorite_food",
    question_text = "What is your favorite food?"
)

job = q.by(a)  # using the agent created above
job.prompts().select("user_prompt", "system_prompt")
```

Output:

| user\_prompt | system\_prompt |
| :- | :- |
| What is your favorite food? | Answer in German. Your traits: \{‘age’: 10}. Answer in German. |

Learn more about how to use instructions in the [Prompts](/en/latest/prompts) section.

## Controlling the presentation of the persona

The traits\_presentation\_template parameter can be used to create a narrative persona for an agent. This is a template string that can be rendered with the agent’s traits as variables.

For example:

```python theme={null}
a = Agent(
    traits = {'age': 22, 'hair': 'brown', 'gender': 'female'},
    traits_presentation_template = "I am a {{ age }} year-old {{ gender }} with {{ hair }} hair."
    )

a.agent_persona.render(primary_replacement = a.traits)
```

Output:

```
I am a 22 year-old female with brown hair.
```

Note that the trait keys must be valid Python identifiers (e.g., home\_state but not home state or home-state). This can be handled by using a dictionary with string keys and values, for example:

```python theme={null}
from edsl import Agent

codebook = {'age': 'The age of the agent'}

a = Agent(
    traits = {'age': 22},
    codebook = codebook,
    traits_presentation_template = "{{ codebook['age'] }} is {{ age }}."
    )

a.agent_persona.render(primary_replacement = a.traits)
```

Output:

```
The age of the agent is 22.
```

We can also use the traits\_presentation\_template together with an instruction and inspect the prompts:

```python theme={null}
from edsl import Agent, QuestionFreeText

a = Agent(
    traits = {"age": 10},
    traits_presentation_template = "(You are {{ age }} years old.)",
    instruction = "Answer in German."
)

q = QuestionFreeText(
    question_name = "favorite_food",
    question_text = "What is your favorite food?"
)

job = q.by(a)
job.prompts().select("user_prompt", "system_prompt")
```

Output:

| user\_prompt | system\_prompt |
| :- | :- |
| What is your favorite food? | Answer in German.(You are 10 years old.) |

<Note>
  **Note**:

  Note that it can be helpful to include traits mentioned in the persona as independent keys and values in order to analyze survey results by those dimensions individually. For example, we may want the narrative to include a sentence about the agent’s age, but also be able to readily analyze or filter results by age.
</Note>

The following code will include the agent’s age as a column of a table with any other selected components (e.g., agent name and all the answers):

```python theme={null}
results.select("agent.age", "agent.agent_name", "answer.*")
```

<Note>
  **Note**:

  Note that the prefix “agent” can also be dropped. The following code is equivalent:
</Note>

```
results.select("age", "agent_name", "answer.*")
```

We can filter the results by an agent’s traits:

```python theme={null}
results.filter("age == 22")
```

We can also call the filter() method on an agent list to filter agents by their traits:

```python theme={null}
middle_aged_agents = agents.filter("40 <= age <= 60")
```

## Using agent traits in prompts

The traits of an agent can be used in the prompts of questions.

For example:

```python theme={null}
from edsl import Agent, QuestionFreeText

a = Agent(traits = {'first_name': 'John'})

q = QuestionFreeText(
    question_text = 'What is your last name, {{ agent.first_name }}?',
    question_name = "exmaple"
)

job = q.by(a)
job.prompts().select("user_prompt")
```

Output:

| user\_prompt |
| :- |
| What is your last name, John? |

In templates, `agent.<key>` and `agent["<key>"]` look up agent traits before
Python attributes. For example, `{{ agent.prompt }}` renders the value of a trait
named `prompt`, even though `Agent` also has a `prompt()` method. If no trait
matches, existing attribute access such as `{{ agent.name }}` still works.
This precedence applies only during template rendering; Python calls such as
`a.prompt()` are unchanged.

Learn more about user and system prompts in the [Prompts](/en/latest/prompts) section.

## Accessing agent traits

The traits of an agent can be accessed directly:

```python theme={null}
from edsl import Agent

a = Agent(traits = {'age': 22})
a.traits
```

Output:

```python theme={null}
{'age': 22}
```

The traits of an agent can also be accessed as attributes of the agent:

```python theme={null}
from edsl import Agent

a = Agent(traits = {'age': 22})
a.age
```

Output:

```python theme={null}
22
```

## Simulating agent responses

When a survey is run, agents can be assigned to it using the by method, which can be chained with other components like scenarios and models:

```python theme={null}
from edsl import Agent, QuestionList, QuestionMultipleChoice, Survey

agent = Agent(
    name = "college student",
    traits = {
        "persona": "You are a sophomore at a community college in upstate New York.",
        "year": "sophomore",
        "school": "community college",
        "major": "biology",
        "state": "New York"
    }
)

q1 = QuestionList(
    question_name = "favorite_courses",
    question_text = "What are the names of your 3 favorite courses?",
    max_list_items = 3
)

q2 = QuestionMultipleChoice(
    question_name = "attend_grad_school",
    question_text = "Do you plan to attend grad school?",
    question_options = ["Yes", "No", "Undecided"]
)

survey = Survey([q1, q2])

results = survey.by(agent).run()
```

This will generate a Results object that contains a Result for each agent’s responses to the survey questions. We can select and inspect components of the results, such as the agent’s traits and their answers:

```python theme={null}
results.select("persona", "year", "school", "major", "state", "answer.*")
```

Output:

| agent.persona | agent.year | agent.school | agent.major | agent.state | answer.favorite\_courses | answer.attend\_grad\_school |
| :- | :- | :- | :- | :- | :- | :- |
| You are a sophomore at a community college in upstate New York. | sophomore | community college | biology | New York | \[‘General Biology I’, ‘Organic Chemistry’, ‘Environmental Science’] | Undecided |

If multiple agents will be used with a survey, they are passed as a list in the same by call:

```python theme={null}
from edsl import Agent, AgentList

agents = AgentList([
    Agent(traits = {"major": "biology", "year": "sophomore"}),
    Agent(traits = {"major": "history", "year": "junior"}),
    Agent(traits = {"major": "mathematics", "year": "senior"}),
])

results = survey.by(agents).run() # using the same survey as above

results.select("major", "year", "answer.*")
```

Output:

| agent.major | agent.year | answer.favorite\_courses | answer.attend\_grad\_school |
| :- | :- | :- | :- |
| biology | sophomore | \[‘Genetics’, ‘Ecology’, ‘Cell Biology’] | Undecided |
| history | junior | \[‘Medieval Europe’, ‘The American Civil War’, ‘Ancient Civilizations’] | Undecided |
| mathematics | senior | \[‘Abstract Algebra’, ‘Real Analysis’, ‘Topology’] | Undecided |

If scenarios and/or models are also specified for a survey, each component type is added in a separate by call that can be chained in any order with the run method appended last:

```python theme={null}
results = survey.by(scenarios).by(agents).by(models).run() # example code - scenarios and models not defined here
```

Learn more about [Scenarios](/en/latest/scenarios), [Language Models](/en/latest/language_models) and [Results](/en/latest/results).

## Updating agents

Agents can be updated after they are created.

### Changing a trait

Here we create an agent and then change one of its traits:

```python theme={null}
from edsl import Agent

a = Agent(traits = {"age": 22})
a.age = 23
a.age
```

Output:

```
23
```

### Adding a trait

We can also add a new trait to an agent:

```python theme={null}
from edsl import Agent

a = Agent(traits = {"age": 22})

a.add_trait({"location": "California"})
a
```

Output:

| key | value |
| :- | :- |
| traits:age | 22 |
| traits:location | California |

### Removing a trait

We can remove a trait from an agent:

```python theme={null}
from edsl import Agent

a = Agent(traits = {"age": 22, "location": "California"})

a.remove_trait("age")
a
```

Output:

| key | value |
| :- | :- |
| traits:location | California |

### Using survey responses as new agent traits

After running a survey, we can use the responses to create new traits for an agent:

```python theme={null}
from edsl import Agent, QuestionMultipleChoice, Survey

a = Agent(traits = {"age": 22, "location": "California"})

q = QuestionMultipleChoice(
    question_name = "surfing",
    question_text = "How often do you go surfing?",
    question_options = ["Never", "Sometimes", "Often"]
)

survey = Survey([q])
results = survey.by(a).run()

a = results.select("age", "location", "surfing").to_agent_list() # create new agent with traits from results
a
```

Output:

| location | surfing | age |
| :- | :- | :- |
| California | Sometimes | 22 |

<Note>
  **Note**:

  Note that in the example above we simply replaced the original agent by selecting the first agent from the agent list that we created. This can be useful for creating agents that evolve over time based on their experiences or responses to surveys.
</Note>

Here we use the same method to update multiple agents at once:

```python theme={null}
from edsl import Agent, QuestionMultipleChoice, Survey, AgentList

agents = AgentList([
    Agent(traits = {"age": 22, "location": "California"}),
    Agent(traits = {"age": 30, "location": "New York"}),
    Agent(traits = {"age": 40, "location": "Texas"}),
])

q = QuestionMultipleChoice(
    question_name = "surfing",
    question_text = "How often do you go surfing?",
    question_options = ["Never", "Sometimes", "Often"]
)

survey = Survey([q])
results = survey.by(agents).run()

agents = results.select("age", "location", "surfing").to_agent_list()
agents
```

Output:

| location | surfing | age |
| :- | :- | :- |
| California | Sometimes | 22 |
| New York | Never | 30 |
| Texas | Never | 40 |

### Creating AgentList from Results

The AgentList.from\_results() method allows you to create an AgentList directly from a Results object. This is useful when you want to create agents based on survey responses, including their original traits and their answers to questions.

By default, this method includes all answer columns as traits for the new agents:

```python theme={null}
from edsl import AgentList

# Create AgentList with all questions included
new_agents = AgentList.from_results(results)

# The new agents will have traits from the original agents plus all their answers
new_agents[0]

You can also specify which question responses to include as traits using the question_names parameter:

# Include only specific questions as traits
new_agents = AgentList.from_results(results, question_names=['surfing', 'age'])

# The new agents will have traits from the original agents plus only the specified answers
new_agents[0]
```

This is particularly useful when you want to: - Create agents with only certain response patterns - Filter out irrelevant or sensitive question responses - Create more focused agent profiles based on specific survey questions - Reduce the number of traits when only certain responses are needed

<Note>
  **Note**:

  Note that the question\_names parameter affects both answer.\* columns (as traits) and prompt.\* columns (as codebook). Agent traits (from agent.\* columns) are always included.
</Note>

## Agent class

> ### *class* edsl.agents.Agent(*traits: dict | None = None*, *name: str | None = None*, *codebook: dict | None = None*, *instruction: str | None = None*, *trait\_categories: dict\[str, list\[str]] | None = None*, *traits\_presentation\_template: str | None = None*, *dynamic\_traits\_function: Callable | None = None*, *dynamic\_traits\_function\_source\_code: str | None = None*, *dynamic\_traits\_function\_name: str | None = None*, *answer\_question\_directly\_source\_code: str | None = None*, *answer\_question\_directly\_function\_name: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Bases: [`Base`](/en/latest/base#base-class "edsl.base.base_class.Base")

A class representing an AI agent with customizable traits that can answer questions.

An Agent in EDSL represents an entity with specific characteristics (traits) that can answer questions through various mechanisms. Agents can use language models to generate responses based on their traits, directly answer questions through custom functions, or dynamically adjust their traits based on the questions being asked.

Key capabilities: - Store and manage agent characteristics (traits) - Provide instructions on how the agent should answer - Support for custom codebooks to provide human-readable trait descriptions - Integration with multiple question types and language models - Combine agents to create more complex personas - Customize agent behavior through direct answering methods

Agents are used in conjunction with Questions, Scenarios, and Surveys to create structured interactions with language models.

> #### **init**(*traits: dict | None = None*, *name: str | None = None*, *codebook: dict | None = None*, *instruction: str | None = None*, *trait\_categories: dict\[str, list\[str]] | None = None*, *traits\_presentation\_template: str | None = None*, *dynamic\_traits\_function: Callable | None = None*, *dynamic\_traits\_function\_source\_code: str | None = None*, *dynamic\_traits\_function\_name: str | None = None*, *answer\_question\_directly\_source\_code: str | None = None*, *answer\_question\_directly\_function\_name: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Initialize a new Agent instance with specified traits and capabilities.

> #### **Args:**

traits: Dictionary of agent characteristics (e.g., \{“age”: 30, “occupation”: “doctor”}) name: Optional name identifier for the agent codebook: Dictionary mapping trait keys to human-readable descriptions for prompts.

This provides more descriptive labels for traits when rendering prompts. For example, \{‘age’: ‘Age in years’} would display “Age in years: 30” instead of “age: 30”.

instruction: Directive for how the agent should answer questions traits\_presentation\_template: Jinja2 template for formatting traits in prompts dynamic\_traits\_function: Function that can modify traits based on questions dynamic\_traits\_function\_source\_code: Source code string for the dynamic traits function dynamic\_traits\_function\_name: Name of the dynamic traits function answer\_question\_directly\_source\_code: Source code for direct question answering method answer\_question\_directly\_function\_name: Name of the direct answering function

The Agent class brings together several key concepts:

### Traits

Traits are key-value pairs that define an agent’s characteristics. These are used to construct a prompt that guides the language model on how to respond.

Example: >>> a = Agent(traits=\{“age”: 10, “hair”: “brown”, “height”: 5.5}) >>> a.traits \{‘age’: 10, ‘hair’: ‘brown’, ‘height’: 5.5}

### Traits Presentation

The traits\_presentation\_template controls how traits are formatted in prompts. It uses Jinja2 templating to insert trait values.

Example: >>> a = Agent(traits=\{“age”: 10}, traits\_presentation\_template=”I am a \{`\} year old.”) \>\>\> repr(a.agent_persona) ‘Prompt(text=”””I am a \{{age}`} year old.”””)’

### Codebooks

Codebooks provide human-readable descriptions for traits in prompts.

Example:

```python theme={null}
>>> traits = {"age": 10, "hair": "brown", "height": 5.5}
>>> codebook = {'age': 'Their age is'}
>>> a = Agent(traits=traits, codebook=codebook, traits_presentation_template="This agent is Dave. {{codebook['age']}} {{age}}")
>>> d = a.traits | {'codebook': a.codebook}
>>> a.agent_persona.render(d)
Prompt(text="""This agent is Dave. Their age is 10""")
```

### Instructions

Instructions guide how the agent should answer questions. If not provided, a default instruction is used.

```python theme={null}
>>> Agent.default_instruction
'You are answering questions as if you were a human. Do not break character.'
```

For details on how these components are used to construct prompts, see `edsl.agents.Invigilator.InvigilatorBase`.

> #### add(*other\_agent: A | None = None*, \*, *conflict\_strategy: str = 'numeric'*) → A [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Combine *self* with *other\_agent* and return a new Agent.

### Parameters

**other\_agent:**

The second agent to merge with *self*. If *None*, *self* is returned unchanged.

**conflict\_strategy:**

How to handle overlapping trait names.

* `"numeric"` (default) – rename conflicting traits coming from *other\_agent* by appending an incrementing suffix (`_1`, `_2` …). This is identical to the behaviour of the `+` operator before this refactor.
* `"error"` – raise `edsl.agents.exceptions.AgentCombinationError`.
* `"repeated_observation"` – if both agents have the same trait *and* the codebook entry for that trait is identical (or missing in both), merge the two values into a list `[old, new]`. If the codebook entries differ, an `edsl.agents.exceptions.AgentCombinationError` is raised.

### Returns

**Agent:**

A new agent containing the merged traits / codebooks.

> #### add\_canned\_response(*question\_name*, *response*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Add a canned response to the agent.

> #### add\_category(*category\_name: str*, *trait\_keys: list\[str] | None = None*) → None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Add a category to the agent

> #### add\_direct\_question\_answering\_method(*method: DirectAnswerMethod*, *validate\_response: bool = False*, *translate\_response: bool = False*) → None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Add a method to the agent that can answer a particular question type.

See: [/en/latest/agents#agent-direct-answering-methods](/en/latest/agents#agent-direct-answering-methods)

> #### **Args:**

method: A method that can answer a question directly validate\_response: Whether to validate the response translate\_response: Whether to translate the response

Raises:

AgentDirectAnswerFunctionError: If the method signature is invalid

> #### **Example:**

```python theme={null}
>>> a = Agent()
>>> def f(self, question, scenario): return "I am a direct answer."
>>> a.add_direct_question_answering_method(f)
>>> a.answer_question_directly(question = None, scenario = None)
'I am a direct answer.'
```

> #### add\_trait(*trait\_name\_or\_dict: str | dict\[str, Any]*, *value: Any | None = None*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Add a trait to an agent and return a new agent.

> #### **Args:**

trait\_name\_or\_dict: Either a trait name string or a dictionary of traits value: The trait value if trait\_name\_or\_dict is a string

> #### **Returns:**

A new Agent instance with the added trait(s)

> #### **Raises:**

AgentErrors: If both a dictionary and a value are provided

> #### **Example:**

```python theme={null}
>>> a = Agent(traits = {"age": 10, "hair": "brown", "height": 5.5})
>>> a.add_trait("weight", 150)
Agent(traits = {'age': 10, 'hair': 'brown', 'height': 5.5, 'weight': 150})
```

> #### *property* agent\_persona\*: [Prompt](/en/latest/prompts#edsl.prompts.Prompt "edsl.prompts.Prompt")\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Get the agent’s persona template as a Prompt object.

This property provides access to the template that formats the agent’s traits for presentation in prompts. The template is wrapped in a Prompt object that supports rendering with variable substitution.

**Returns:**

Prompt: A prompt object containing the traits presentation template

> #### answer\_question(\*, *question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase")*, *cache: [Cache](/en/latest/data#cache-class "edsl.caching.Cache")*, *scenario: 'Scenario' | None = None*, *survey: 'Survey' | None = None*, *model: 'LanguageModel' | None = None*, *debug: bool = False*, *memory\_plan: MemoryPlan | None = None*, *current\_answers: dict | None = None*, *iteration: int = 0*, *key\_lookup: 'KeyLookup' | None = None*) → AgentResponseDict [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Answer a posed question asynchronously.

> **Args:**

question: The question to answer cache: The cache to use for storing responses scenario: The scenario in which the question is asked survey: The survey context model: The language model to use debug: Whether to run in debug mode memory\_plan: The memory plan to use current\_answers: The current answers iteration: The iteration number key\_lookup: The key lookup for API credentials

> **Returns:**

An AgentResponseDict containing the answer

> **Example:**

```python theme={null}
>>> a = Agent(traits = {})
>>> a.add_direct_question_answering_method(lambda self, question, scenario: "I am a direct answer.")
>>> from edsl.questions import QuestionFreeText
>>> q = QuestionFreeText.example()
>>> a.answer_question(question = q, cache = False).answer
'I am a direct answer.'
```

> **Note:**

This is a function where an agent returns an answer to a particular question. However, there are several different ways an agent can answer a question, so the actual functionality is delegated to an InvigilatorBase object.

> #### answer\_question\_directly\_function\_name *= ''* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

> #### *async* async\_answer\_question(\*, *question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase")*, *cache: [Cache](/en/latest/data#cache-class "edsl.caching.Cache")*, *scenario: 'Scenario' | None = None*, *survey: 'Survey' | None = None*, *model: 'LanguageModel' | None = None*, *debug: bool = False*, *memory\_plan: MemoryPlan | None = None*, *current\_answers: dict | None = None*, *iteration: int = 0*, *key\_lookup: 'KeyLookup' | None = None*) → AgentResponseDict [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Answer a posed question asynchronously.

> **Args:**

question: The question to answer cache: The cache to use for storing responses scenario: The scenario in which the question is asked survey: The survey context model: The language model to use debug: Whether to run in debug mode memory\_plan: The memory plan to use current\_answers: The current answers iteration: The iteration number key\_lookup: The key lookup for API credentials

> **Returns:**

An AgentResponseDict containing the answer

> **Example:**

```python theme={null}
>>> a = Agent(traits = {})
>>> a.add_direct_question_answering_method(lambda self, question, scenario: "I am a direct answer.")
>>> from edsl.questions import QuestionFreeText
>>> q = QuestionFreeText.example()
>>> a.answer_question(question = q, cache = False).answer
'I am a direct answer.'
```

> **Note:**

This is a function where an agent returns an answer to a particular question. However, there are several different ways an agent can answer a question, so the actual functionality is delegated to an InvigilatorBase object.

> #### chat() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

> #### code() → str [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return the code for the agent.

> **Returns:**

Python code string to recreate this agent

> **Example:**

```python theme={null}

>>> a = Agent(traits = {"age": 10, "hair": "brown", "height": 5.5})
>>> print(a.code())
from edsl.agents import Agent
agent = Agent(traits={'age': 10, 'hair': 'brown', 'height': 5.5})
```

> #### copy() → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create a deep copy of this agent using serialization/deserialization.

This method uses to\_dict/from\_dict to create a completely independent copy of the agent, including all its traits, codebook, instructions, and special functions like dynamic traits and direct answering methods.

> **Returns:**

Agent: A new agent instance that is functionally identical to this one

> **Examples:**

```python theme={null}
>>> a = Agent(traits={"age": 10, "hair": "brown"},
...           codebook={'age': 'Their age is'})
>>> a2 = a.copy()
>>> a2 == a  # Functionally equivalent
True
>>> id(a) == id(a2)  # But different objects
False
```

Copy preserves direct answering methods:

```python theme={null}
>>> def f(self, question, scenario): return "I am a direct answer."
>>> a.add_direct_question_answering_method(f)
>>> a2 = a.copy()
>>> a2.answer_question_directly(None, None)
'I am a direct answer.'
```

> #### create\_invigilator(\*, *question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase")*, *cache: [Cache](/en/latest/data#cache-class "edsl.caching.Cache")*, *survey: 'Survey' | None = None*, *scenario: 'Scenario' | None = None*, *model: 'LanguageModel' | None = None*, *memory\_plan: 'MemoryPlan' | None = None*, *current\_answers: dict | None = None*, *iteration: int = 1*, *raise\_validation\_errors: bool = True*, *key\_lookup: 'KeyLookup' | None = None*) → InvigilatorBase [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create an Invigilator.

An invigilator is an object that is responsible for administering a question to an agent. There are several different types of invigilators, depending on the type of question and the agent. For example, there are invigilators for functional questions, for direct questions, and for LLM questions.

> **Args:**

question: The question to be asked cache: The cache for storing responses survey: The survey context scenario: The scenario context model: The language model to use memory\_plan: The memory plan to use current\_answers: The current answers iteration: The iteration number raise\_validation\_errors: Whether to raise validation errors key\_lookup: The key lookup for API credentials

> **Returns:**

An InvigilatorBase instance for handling the question

> **Example:**

```python theme={null}
>>> a = Agent(traits = {})
>>> inv = a.create_invigilator(question = None, cache = False)
>>> type(inv).__name__
'InvigilatorAI'
```

> **Note:**

An invigilator is an object that is responsible for administering a question to an agent and recording the responses.

> #### default\_instruction *= 'You are answering questions as if you were a human. Do not break character.'* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

> #### drop(\*\*field\_names: str | List\[str]\*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Drop field(s) from the agent.

> **Args:**

[\*](/en/latest/#id2)field\_names: The name(s) of the field(s) to drop. Can be:

* Single field name: drop(“age”)
* Multiple field names: drop(“age”, “height”)
* List of field names: drop(\[“age”, “height”])

> **Examples:**

Drop a single trait from the agent:

```python theme={null}
>>> a = Agent(traits={"age": 30, "hair": "brown", "height": 5.5})
>>> a_dropped = a.drop("age")
>>> a_dropped.traits
{'hair': 'brown', 'height': 5.5}
```

Drop multiple traits using separate arguments:

```python theme={null}
>>> a = Agent(traits={"age": 30, "hair": "brown", "height": 5.5})
>>> a_dropped = a.drop("age", "height")
>>> a_dropped.traits
{'hair': 'brown'}
```

Drop multiple traits using a list:

```python theme={null}
>>> a = Agent(traits={"age": 30, "hair": "brown", "height": 5.5})
>>> a_dropped = a.drop(["age", "height"])
>>> a_dropped.traits
{'hair': 'brown'}
```

Drop an agent field like name:

```python theme={null}
>>> a = Agent(traits={"age": 30}, name="John")
>>> a_dropped = a.drop("name")
>>> a_dropped.name is None
True
>>> a_dropped.traits
{'age': 30}
```

Error when trying to drop a non-existent field:

```python theme={null}
>>> a = Agent(traits={"age": 30})
>>> a.drop("nonexistent")
Traceback (most recent call last):
...
edsl.agents.exceptions.AgentErrors: ...
```

> #### drop\_trait\_if(*bad\_value: Any*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Drop traits that have a specific bad value.

> **Args:**

bad\_value: The value to remove from traits

> **Returns:**

A new Agent instance with the bad value traits removed

> **Example:**

```python theme={null}
>>> agent = Agent(traits={'age': 30, 'height': None, 'weight': 150})
>>> clean_agent = agent.drop_trait_if(None)
>>> clean_agent.traits
{'age': 30, 'weight': 150}
```

> #### duplicate() → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create a deep copy of this agent with all its traits and capabilities.

This method creates a completely independent copy of the agent, including all its traits, codebook, instructions, and special functions like dynamic traits and direct answering methods.

> **Returns:**

Agent: A new agent instance that is functionally identical to this one

> **Examples:**

Create a duplicate agent and verify it’s equal but not the same object:

```python theme={null}
>>> a = Agent(traits={"age": 10, "hair": "brown", "height": 5.5},
...           codebook={'age': 'Their age is'})
>>> a2 = a.duplicate()
>>> a2 == a  # Functionally equivalent
True
>>> id(a) == id(a2)  # But different objects
False
```

Duplicating preserves direct answering methods:

```python theme={null}
>>> def f(self, question, scenario): return "I am a direct answer."
>>> a.add_direct_question_answering_method(f)
>>> hasattr(a, "answer_question_directly")
True
>>> a2 = a.duplicate()
>>> a2.answer_question_directly(None, None)
'I am a direct answer.'
```

Duplicating preserves custom instructions:

```python theme={null}
>>> a = Agent(traits={'age': 10}, instruction="Have fun!")
>>> a2 = a.duplicate()
>>> a2.instruction
'Have fun!'
```

> #### *property* dynamic\_traits\_function\*: Callable | None\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

The dynamic traits function if one exists.

This property provides backward compatibility for the old attribute access pattern.

> **Returns:**

The dynamic traits function or None

> **Examples:**

```python theme={null}
>>> agent = Agent(traits={'age': 30})
>>> agent.dynamic_traits_function is None
True
>>> def dynamic_func(): return {'age': 25}
>>> agent.traits_manager.initialize_dynamic_function(dynamic_func)
>>> agent.dynamic_traits_function is not None
True
```

> #### *property* dynamic\_traits\_function\_name\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

The name of the dynamic traits function.

This property provides backward compatibility for the old attribute access pattern.

> **Returns:**

The function name or empty string if no function

> **Examples:**

```python theme={null}
>>> agent = Agent(traits={'age': 30})
>>> agent.dynamic_traits_function_name
''
>>> def my_func(): return {'age': 25}
>>> agent.traits_manager.initialize_dynamic_function(my_func)
>>> agent.dynamic_traits_function_name
'my_func'
```

> #### *classmethod* example(*randomize: bool = False*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return an example Agent instance.

> **Args:**

randomize: If True, adds a random string to the value of an example key

> **Returns:**

An example Agent instance

> **Example:**

```python theme={null}
>>> Agent.example()
Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5})
```

> #### *classmethod* from\_dict(*agent\_dict: dict\[str, dict | bool | str]*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Deserialize from a dictionary.

> **Args:**

agent\_dict: Dictionary containing agent data

> **Returns:**

An Agent instance created from the dictionary

> **Example:**

```python theme={null}
>>> Agent.from_dict({'name': "Steve", 'traits': {'age': 10, 'hair': 'brown', 'height': 5.5}})
Agent(name = """Steve""", traits = {'age': 10, 'hair': 'brown', 'height': 5.5})
```

> #### *classmethod* from\_result(*result: [Result](/en/latest/results#edsl.results.Result "edsl.results.Result")*, *name: str | None = None*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create an Agent instance from a Result object.

The agent’s traits will correspond to the questions asked during the interview (the keys of result.answer) with their respective answers as the values.

A simple, readable traits\_presentation\_template is automatically generated so that rendering the agent will look like:

```python theme={null}
This person was asked the following questions – here are the answers:
Q: <question 1>
A: <answer 1>

Q: <question 2>
A: <answer 2>
...
```

> **Args:**

result: The Result instance from which to build the agent name: Optional explicit name for the new agent. If omitted, we attempt

> to reuse result.agent.name if it exists

> **Returns:**

A new Agent instance created from the result

> **Raises:**

TypeError: If result is not a Result object

> **Example:**

```python theme={null}
>>> from edsl.results import Result
>>> # result = Result(...)
>>> # agent = Agent.from_result(result)
```

> #### *property* has\_dynamic\_traits\_function\*: bool\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Whether the agent has a dynamic traits function.

This property provides backward compatibility for the old attribute access pattern.

> **Returns:**

True if the agent has a dynamic traits function, False otherwise

> **Examples:**

```python theme={null}
>>> agent = Agent(traits={'age': 30})
>>> agent.has_dynamic_traits_function
False
>>> def dynamic_func(): return {'age': 25}
>>> agent.traits_manager.initialize_dynamic_function(dynamic_func)
>>> agent.has_dynamic_traits_function
True
```

> #### instruction [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

ABC for something.

> #### *property* invigilator [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Lazily initialize the invigilator to avoid importing language\_models during Survey import

> #### keep(\*\*field\_names: str | List\[str]\*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Keep only the specified fields from the agent.

> **Args:**

[\*](/en/latest/#id4)field\_names: The name(s) of the field(s) to keep. Can be:

* Single field name: keep(“age”)
* Multiple field names: keep(“age”, “height”)
* List of field names: keep(\[“age”, “height”])

> **Examples:**

Keep a single trait:

```python theme={null}
>>> a = Agent(traits={"age": 30, "hair": "brown", "height": 5.5})
>>> a_kept = a.keep("age")
>>> a_kept.traits
{'age': 30}
```

Keep multiple traits using separate arguments:

```python theme={null}
>>> a = Agent(traits={"age": 30, "hair": "brown", "height": 5.5})
>>> a_kept = a.keep("age", "height")
>>> a_kept.traits
{'age': 30, 'height': 5.5}
```

Keep multiple traits using a list:

```python theme={null}
>>> a = Agent(traits={"age": 30, "hair": "brown", "height": 5.5})
>>> a_kept = a.keep(["age", "height"])
>>> a_kept.traits
{'age': 30, 'height': 5.5}
```

Keep agent fields and traits:

```python theme={null}
>>> a = Agent(traits={"age": 30, "hair": "brown"}, name="John")
>>> a_kept = a.keep("name", "age")
>>> a_kept.name
'John'
>>> a_kept.traits
{'age': 30}
```

Error when trying to keep a non-existent field:

```python theme={null}
>>> a = Agent(traits={"age": 30})
>>> a.keep("nonexistent")
Traceback (most recent call last):
...
edsl.agents.exceptions.AgentErrors: ...
```

> #### name [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Valid agent name descriptor.

> #### old\_keep(\**traits: str*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Legacy trait selection method (renamed from select).

Note: This method has data integrity issues and is kept for backward compatibility. Use select() or keep() instead, which provide better data consistency.

> **Args:**

[\*](/en/latest/#id6)traits: The trait names to select

> **Returns:**

A new Agent instance with only the selected traits

> **Example:**

```python theme={null}
>>> a = Agent(traits = {"age": 10, "hair": "brown", "height": 5.5}, codebook = {'age': 'Their age is'})
>>> a.old_keep("age", "height")
Agent(traits = {'age': 10, 'height': 5.5}, codebook = {'age': 'Their age is'})
```

```python theme={null}
>>> a.old_keep("height")
Agent(traits = {'height': 5.5})
```

> #### prompt() → [Prompt](/en/latest/prompts#edsl.prompts.Prompt "edsl.prompts.Prompt") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Generate a formatted prompt containing the agent’s traits.

This method renders the agent’s traits presentation template with the agent’s traits and codebook, creating a formatted prompt that can be used in language model requests.

The method is dynamic and responsive to changes in the agent’s state:

1. If a custom template was explicitly set during initialization, it will be used
2. If using the default template and the codebook has been updated since initialization, this method will recreate the template to reflect the current codebook values
3. The template is rendered with access to all trait values, the complete traits dictionary, and the codebook

The template rendering makes the following variables available: - All individual trait keys (e.g., `{{ age }}`, `{{ occupation }}`) - The full traits dictionary as `{{ traits }}` - The codebook as `{{ codebook }}`

> **Returns:**

Prompt: A Prompt object containing the rendered template

> **Raises:**

QuestionScenarioRenderError: If any template variables remain undefined

> **Examples:**

Basic trait rendering without a codebook:

```python theme={null}
>>> agent = Agent(traits={"age": 10, "hair": "brown", "height": 5.5})
>>> agent.prompt()
Prompt(text="""Your traits: {'age': 10, 'hair': 'brown', 'height': 5.5}""")
```

Trait rendering with a codebook (more readable format):

```python theme={null}
>>> codebook = {"age": "Age in years", "hair": "Hair color"}
>>> agent = Agent(traits={"age": 10, "hair": "brown"}, codebook=codebook)
>>> print(agent.prompt().text)
Your traits:
Age in years: 10
Hair color: brown
```

Adding a codebook after initialization updates the rendering:

```python theme={null}
>>> agent = Agent(traits={"age": 30, "occupation": "doctor"})
>>> initial_prompt = agent.prompt()
>>> "Your traits: {" in initial_prompt.text
True
>>> agent.codebook = {"age": "Age", "occupation": "Profession"}
>>> updated_prompt = agent.prompt()
>>> "Age: 30" in updated_prompt.text
True
>>> "Profession: doctor" in updated_prompt.text
True
```

Custom templates can reference any trait directly:

```python theme={null}
>>> template = "Profile: {{ age }} year old {{ occupation }}"
>>> agent = Agent(traits={"age": 45, "occupation": "teacher"},
...               traits_presentation_template=template)
>>> agent.prompt().text
'Profile: 45 year old teacher'
```

> #### remove\_direct\_question\_answering\_method() → None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Remove the direct question answering method.

> **Example:**

```python theme={null}
>>> a = Agent()
>>> def f(self, question, scenario): return "I am a direct answer."
>>> a.add_direct_question_answering_method(f)
>>> a.remove_direct_question_answering_method()
>>> hasattr(a, "answer_question_directly")
False
```

> #### remove\_trait(*trait: str*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Remove a trait from the agent.

> **Args:**

trait: The name of the trait to remove

> **Returns:**

A new Agent instance without the specified trait

> **Example:**

```python theme={null}
>>> a = Agent(traits = {"age": 10, "hair": "brown", "height": 5.5})
>>> a.remove_trait("age")
Agent(traits = {'hair': 'brown', 'height': 5.5})
```

> #### rename(*old\_name\_or\_dict: str | dict\[str, str]*, *new\_name: str | None = None*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Rename a trait.

> **Args:**

old\_name\_or\_dict: The old name of the trait or a dictionary of old names and new names new\_name: The new name of the trait (required if old\_name\_or\_dict is a string)

> **Returns:**

A new Agent instance with renamed traits

> **Raises:**

AgentErrors: If invalid arguments are provided

> **Example:**

```python theme={null}
>>> a = Agent(traits = {"age": 10, "hair": "brown", "height": 5.5})
>>> newa = a.rename("age", "years")
>>> newa == Agent(traits = {'years': 10, 'hair': 'brown', 'height': 5.5})
True
```

```python theme={null}
>>> newa.rename({'years': 'smage'}) == Agent(traits = {'smage': 10, 'hair': 'brown', 'height': 5.5})
True
```

> #### search\_traits(*search\_string: str*) → RankableItems [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Search the agent’s traits for a string.

This method delegates to the traits manager to search through trait descriptions and return ranked matches based on similarity.

> **Args:**

search\_string: The string to search for in trait descriptions

> **Returns:**

A ScenarioList containing ranked trait matches

> **Examples:**

```python theme={null}
>>> from edsl.agents import Agent
>>> agent = Agent(traits={"age": 30, "occupation": "doctor"})
>>> results = agent.search_traits("age")
>>> len(results) >= 1
True
```

> #### select(\**traits: str*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Select agents with only the referenced traits.

This method now uses the robust keep() implementation for better data integrity and consistent handling of codebooks and trait\_categories.

> **Args:**

[\*](/en/latest/#id8)traits: The trait names to select

> **Returns:**

A new Agent instance with only the selected traits

> **Example:**

```python theme={null}
>>> a = Agent(traits = {"age": 10, "hair": "brown", "height": 5.5}, codebook = {'age': 'Their age is'})
>>> a.select("age", "height")
Agent(traits = {'age': 10, 'height': 5.5}, codebook = {'age': 'Their age is'})
```

```python theme={null}
>>> a.select("height")
Agent(traits = {'height': 5.5})
```

> #### table() → [Dataset](/en/latest/dataset#edsl.dataset.Dataset "edsl.dataset.Dataset") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create a tabular representation of the agent’s traits.

This method delegates to the table manager to create a structured Dataset containing trait information.

> **Returns:**

A Dataset containing trait information

> **Example:**

```python theme={null}
>>> agent = Agent(traits={'age': 30, 'height': 5.5}, codebook={'age': 'Age in years'})
>>> dataset = agent.table()
>>> len(dataset) == 2
True
```

> #### to(*target: 'QuestionBase' | 'Jobs' | 'Survey'*) → [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Send the agent to a question, job, or survey.

> **Args:**

target: The question, job, or survey to send the agent to

> **Returns:**

A Jobs object containing the agent and target

> **Example:**

```python theme={null}
>>> agent = Agent(traits={'age': 30})
>>> from edsl.questions import QuestionFreeText
>>> q = QuestionFreeText(question_name='test', question_text='How are you?')
>>> job = agent.to(q)
>>> type(job).__name__
'Jobs'
```

> #### to\_dict(*add\_edsl\_version: bool = True*, *full\_dict: bool = False*) → dict\[str, dict | bool | str] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Serialize to a dictionary with EDSL info.

> **Args:**

add\_edsl\_version: Whether to include EDSL version information full\_dict: Whether to include all attributes even if they have default values

> **Returns:**

A dictionary representation of the agent

> **Example:**

```python theme={null}
>>> a = Agent(name = "Steve", traits = {"age": 10, "hair": "brown", "height": 5.5})
>>> d = a.to_dict()
>>> d['traits']
{'age': 10, 'hair': 'brown', 'height': 5.5}
>>> d['name']
'Steve'
>>> d['edsl_class_name']
'Agent'
```

```python theme={null}
>>> a = Agent(traits = {"age": 10, "hair": "brown", "height": 5.5}, instruction = "Have fun.")
>>> d = a.to_dict()
>>> d['traits']
{'age': 10, 'hair': 'brown', 'height': 5.5}
>>> d['instruction']
'Have fun.'
>>> d['edsl_class_name']
'Agent'
```

> #### *property* traits\*: dict\[str, Any]\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Get the agent’s traits, potentially using dynamic generation.

This property provides access to the agent’s traits, either from the stored traits dictionary or by calling a dynamic traits function if one is defined. If a dynamic traits function is used, it may take the current question as a parameter to generate context-aware traits.

> **Returns:**

dict: Dictionary of agent traits (key-value pairs)

> **Examples:**

```python theme={null}
>>> a = Agent(traits={"age": 10, "hair": "brown", "height": 5.5})
>>> a.traits
{'age': 10, 'hair': 'brown', 'height': 5.5}
```

> #### *property* traits\_presentation\_template [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Get the traits presentation template.

> #### translate\_traits(*values\_codebook: dict\[str, dict\[Any, Any]]*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Translate traits to a new codebook.

> **Args:**

values\_codebook: Dictionary mapping trait names to value translation dictionaries

> **Returns:**

A new Agent instance with translated trait values

> **Example:**

```python theme={null}
>>> a = Agent(traits = {"age": 10, "hair": 1, "height": 5.5})
>>> a.translate_traits({"hair": {1:"brown"}})
Agent(traits = {'age': 10, 'hair': 'brown', 'height': 5.5})
```

> #### with\_categories(\**categories: str*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return a new agent with the specified categories

## AgentList class

> #### *class* edsl.agents.AgentList(*data: list\[[Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent")] | None = None*, *codebook: dict\[str, str] | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Bases: `UserList`, [`Base`](/en/latest/base#base-class "edsl.base.base_class.Base"), `AgentListOperationsMixin`

A list of Agents with additional functionality for manipulation and analysis.

The AgentList class extends Python’s UserList to provide a container for Agent objects with methods for filtering, transforming, and analyzing collections of agents.

```python theme={null}
>>> AgentList.example().to_scenario_list().drop('age')
ScenarioList([Scenario({'hair': 'brown', 'height': 5.5}), Scenario({'hair': 'brown', 'height': 5.5})])
```

```python theme={null}
>>> AgentList.example().to_dataset()
Dataset([{'age': [22, 22]}, {'hair': ['brown', 'brown']}, {'height': [5.5, 5.5]}])
```

```python theme={null}
>>> AgentList.example().select('age', 'hair', 'height').to_pandas()
   age   hair  height
0   22  brown     5.5
1   22  brown     5.5
```

> #### **init**(*data: list\[[Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent")] | None = None*, *codebook: dict\[str, str] | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Initialize a new AgentList.

```python theme={null}
>>> from edsl import Agent
>>> al = AgentList([Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5}),
...                Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5})])
>>> al
AgentList([Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5}), Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5})])
>>> al_with_codebook = AgentList([Agent(traits = {'age': 22})], codebook={'age': 'Age in years'})
>>> al_with_codebook[0].codebook
{'age': 'Age in years'}
```

> **Args:**

data: A list of Agent objects. If None, creates an empty AgentList. codebook: Optional dictionary mapping trait names to descriptions.

If provided, will be applied to all agents in the list.

> #### add\_instructions(*instructions: str*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Apply instructions to all agents in the list.

This method provides a more intuitive name for setting instructions on all agents, avoiding the need to iterate manually.

> **Args:**

instructions: The instructions to apply to all agents.

> **Returns:**

AgentList: Returns self for method chaining.

> **Examples:**

```python theme={null}
>>> from edsl import Agent, AgentList
>>> agents = AgentList([Agent(traits={'age': 30}), Agent(traits={'age': 40})])
>>> agents.add_instructions("Answer as if you were this age")
AgentList([Agent(traits = {'age': 30}, instruction = """Answer as if you were this age"""), Agent(traits = {'age': 40}, instruction = """Answer as if you were this age""")])
```

> #### add\_trait(*trait: str*, *values: List\[Any]*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Adds a new trait to every agent, with values taken from values.

> **Parameters:**

* **trait** – The name of the trait.
* **values** – The valeues(s) of the trait. If a single value is passed, it is used for all agents.

```python theme={null}
>>> al = AgentList.example()
>>> new_al = al.add_trait('new_trait', 1)
>>> new_al.select('new_trait').to_scenario_list().to_list()
[1, 1]
>>> al.add_trait('new_trait', [1, 2, 3])
Traceback (most recent call last):
...
edsl.agents.exceptions.AgentListError: The passed values have to be the same length as the agent list.
...
```

> #### *property* all\_traits\*: list\[str]\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return all traits in the AgentList. >>> from edsl import Agent >>> agent\_1 = Agent(traits = \{‘age’: 22}) >>> agent\_2 = Agent(traits = \{‘hair’: ‘brown’}) >>> al = AgentList(\[agent\_1, agent\_2]) >>> al.all\_traits \[‘age’, ‘hair’]

> #### at(*index: int*) → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Get the agent at the specified index position.

> #### chart() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create a chart from the results.

> #### clipboard\_data() → str [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return TSV representation of this object for clipboard operations.

This method is called by the clipboard() method in the base class to provide a custom format for copying objects to the system clipboard.

> **Returns:**

str: Tab-separated values representation of the object

> #### code(*string=True*) → str | list\[str] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return code to construct an AgentList.

```python theme={null}
>>> al = AgentList.example()
>>> print(al.code())
from edsl import Agent
from edsl import AgentList
agent_list = AgentList([Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5}), Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5})])
```

> #### *property* codebook\*: dict\[str, str]\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return the codebook for the AgentList.

> #### collapse(*warn\_about\_none\_name: bool = True*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

All agents with the same name have their traits combined.

```python theme={null}
>>> al = AgentList([Agent(name = 'steve'), Agent(name = 'roxanne')])
>>> al.collapse()
AgentList([Agent(name = """steve""", traits = {}), Agent(name = """roxanne""", traits = {})])
>>> al = AgentList([Agent(name = 'steve', traits = {'age': 22}), Agent(name = 'steve', traits = {'hair': 'brown'})])
>>> al.collapse()
AgentList([Agent(name = """steve""", traits = {'age': 22, 'hair': 'brown'})])
>>> AgentList.example().collapse(warn_about_none_name = False)
AgentList([Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5})])
```

> #### drop(\*\*field\_names: str | List\[str]\*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Drop field(s) from all agents in the AgentList.

> **Args:**

[\*](/en/latest/#id10)**field\_names: The name(s) of the field(s) to drop. Can be:**

* Single field name: drop(“age”)
* Multiple field names: drop(“age”, “height”)
* List of field names: drop(\[“age”, “height”])

> **Returns:**

AgentList: A new AgentList with the specified fields dropped from all agents.

> **Examples:**

Drop a single trait from all agents:

```python theme={null}
>>> from edsl import Agent, AgentList
>>> al = AgentList([Agent(traits={"age": 30, "hair": "brown", "height": 5.5})])
>>> al_dropped = al.drop("age")
>>> al_dropped[0].traits
{'hair': 'brown', 'height': 5.5}
```

Drop multiple traits using separate arguments:

```python theme={null}
>>> al = AgentList([Agent(traits={"age": 30, "hair": "brown", "height": 5.5})])
>>> al_dropped = al.drop("age", "height")
>>> al_dropped[0].traits
{'hair': 'brown'}
```

Drop multiple traits using a list:

```python theme={null}
>>> al = AgentList([Agent(traits={"age": 30, "hair": "brown", "height": 5.5})])
>>> al_dropped = al.drop(["age", "height"])
>>> al_dropped[0].traits
{'hair': 'brown'}
```

> #### duplicate() → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create a deep copy of the AgentList.

> **Returns:**

AgentList: A new AgentList containing copies of all agents.

> **Examples:**

```python theme={null}
>>> al = AgentList.example()
>>> al2 = al.duplicate()
>>> al2 == al
True
>>> id(al2) == id(al)
False
```

> #### edit() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

> #### *classmethod* example(*randomize: bool = False*, *codebook: dict\[str, str] | None = None*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Returns an example AgentList instance.

> **Parameters:**

* **randomize** – If True, uses Agent’s randomize method.
* **codebook** – Optional dictionary mapping trait names to descriptions.

```python theme={null}
>>> al = AgentList.example()
>>> al
AgentList([Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5}), Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5})])
>>> al = AgentList.example(codebook={'age': 'Age in years'})
>>> al[0].codebook
{'age': 'Age in years'}
```

> #### filter(*expression: str*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Filter agents based on a boolean expression.

> **Args:**

expression: A string containing a boolean expression to evaluate against

each agent’s traits.

> **Returns:**

AgentList: A new AgentList containing only agents that satisfy the expression.

> **Examples:**

```python theme={null}
>>> from edsl import Agent
>>> al = AgentList([Agent(traits = {'a': 1, 'b': 1}),
...                Agent(traits = {'a': 1, 'b': 2})])
>>> al.filter("b == 2")
AgentList([Agent(traits = {'a': 1, 'b': 2})])
>>> al = AgentList([Agent(traits = {'a': 1, 'b': 1}, name = 'steve'),
...                Agent(traits = {'a': 1, 'b': 2}, name = 'roxanne')])
>>> len(al.filter("name == 'steve'"))
1
>>> len(al.filter("name == 'roxanne'"))
1
>>> len(al.filter("name == 'steve' and a == 1"))
1
>>> len(al.filter("name == 'steve' and a == 2"))
0
>>> len(al.filter("name == 'steve' and a == 1 and b == 2"))
0
```

> #### first() → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Get the first agent in the list.

> #### flatten(*field: str*, *keep\_original: bool = False*) → [Dataset](/en/latest/dataset#edsl.dataset.Dataset "edsl.dataset.Dataset") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Expand a field containing dictionaries into separate fields.

This method takes a field that contains a list of dictionaries and expands it into multiple fields, one for each key in the dictionaries. This is useful when working with nested data structures or results from extraction operations.

> **Parameters:**

field: The field containing dictionaries to flatten keep\_original: Whether to retain the original field in the result

> **Returns:**

A new Dataset with the dictionary keys expanded into separate fields

> **Notes:**

* Each key in the dictionaries becomes a new field with name pattern “.”
* All dictionaries in the field must have compatible structures
* If a dictionary is missing a key, the corresponding value will be None
* Non-dictionary values in the field will cause a warning

> **Examples:**

```python theme={null}
>>> from edsl.dataset import Dataset
```

```python theme={null}
# Basic flattening of nested dictionaries >>> Dataset([{‘a’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}]).flatten(‘a’) Dataset([{‘c’: [5]}, {‘a.a’: [1]}, {‘a.b’: [2]}])

# Works with prefixed fields too >>> Dataset([{‘answer.example’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}]).flatten(‘answer.example’) Dataset([{‘c’: [5]}, {‘answer.example.a’: [1]}, {‘answer.example.b’: [2]}])

# Keep the original field if needed >>> d = Dataset([{‘a’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}]) >>> d.flatten(‘a’, keep_original=True) Dataset([{‘a’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}, {‘a.a’: [1]}, {‘a.b’: [2]}])

# Can also use unambiguous unprefixed field name >>> result = Dataset([{‘answer.pros_cons’: [{‘pros’: [‘Safety’], ‘cons’: [‘Cost’]}]}]).flatten(‘pros_cons’) >>> sorted(result.keys()) == [‘answer.pros_cons.cons’, ‘answer.pros_cons.pros’] True >>> sorted(result.to_dicts()[0].items()) == sorted({‘cons’: [‘Cost’], ‘pros’: [‘Safety’]}.items()) True
```

> #### *classmethod* from\_csv(*file\_path: str*, *name\_field: str | None = None*, *codebook: dict\[str, str] | None = None*, *instructions: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Load AgentList from a CSV file.

**Deprecated since version Use:** AgentList.from\_source(‘csv’, …) instead.

```python theme={null}
>>> import csv
>>> import os
>>> with open('/tmp/agents.csv', 'w') as f:
...     writer = csv.writer(f)
...     _ = writer.writerow(['age', 'hair', 'height'])
...     _ = writer.writerow([22, 'brown', 5.5])
>>> al = AgentList.from_csv('/tmp/agents.csv')
>>> al = AgentList.from_csv('/tmp/agents.csv', name_field='hair')
>>> al = AgentList.from_csv('/tmp/agents.csv', codebook={'age': 'Age in years'})
>>> al = AgentList.from_csv('/tmp/agents.csv', instructions='Answer as a person')
>>> os.remove('/tmp/agents.csv')
```

**Parameters:**

* **file\_path** – The path to the CSV file.
* **name\_field** – The name of the field to use as the agent name.
* **codebook** – Optional dictionary mapping trait names to descriptions.
* **instructions** – Optional instructions to apply to all created agents.

> #### *classmethod* from\_dict(*data: dict*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Deserialize the dictionary back to an AgentList object.

> **Parameters:**

data: A dictionary representing an AgentList.

```python theme={null}
>>> from edsl import Agent
>>> al = AgentList([Agent.example(), Agent.example()])
>>> al2 = AgentList.from_dict(al.to_dict())
>>> al2 == al
True
>>> example_codebook = {'age': 'Age in years'}
>>> al = AgentList([Agent.example()]).set_codebook(example_codebook)
>>> al2 = AgentList.from_dict(al.to_dict())
>>> al2[0].codebook == example_codebook
True
```

> #### *classmethod* from\_list(*trait\_name: str*, *values: List\[Any]*, *codebook: dict\[str, str] | None = None*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create an AgentList from a list of values.

> **Parameters:**

* **trait\_name** – The name of the trait.
* **values** – A list of values.
* **codebook** – Optional dictionary mapping trait names to descriptions.

```python theme={null}
>>> AgentList.from_list('age', [22, 23])
AgentList([Agent(traits = {'age': 22}), Agent(traits = {'age': 23})])
>>> al = AgentList.from_list('age', [22], codebook={'age': 'Age in years'})
>>> al[0].codebook
{'age': 'Age in years'}
```

> #### *classmethod* from\_results(*results: [Results](/en/latest/results#result-class "edsl.results.Results")*, *question\_names: List\[str] | None = None*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create an AgentList from a Results object.

> **Args:**

results: The Results object to convert question\_names: Optional list of question names to include. If None, all questions are included.

Affects both answer.\* columns (as traits) and prompt.\* columns (as codebook). Agent traits are always included.

> **Returns:**

AgentList: A new AgentList created from the Results

> #### *classmethod* from\_scenario\_list(*scenario\_list: [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create an AgentList from a ScenarioList.

This method supports special fields that map to Agent parameters: - “name”: Will be used as the agent’s name - “agent\_parameters”: A dictionary containing:

* “instruction”: The agent’s instruction text
* “name”: The agent’s name (overrides the “name” field if present)

> **Examples:**

```python theme={null}
>>> from edsl import ScenarioList, Scenario
>>> # Basic usage with traits
>>> s = ScenarioList([Scenario({'age': 22, 'hair': 'brown', 'height': 5.5})])
>>> al = AgentList.from_scenario_list(s)
>>> al
AgentList([Agent(traits = {'age': 22, 'hair': 'brown', 'height': 5.5})])
```

> #### *classmethod* from\_source(*source\_type: str*, \**args*, *instructions: str | None = None*, *codebook: dict\[str, str] | None = None*, *name\_field: str | None = None*, \*\**kwargs*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create an AgentList from a specified source type.

This method serves as the main entry point for creating AgentList objects, providing a unified interface for various data sources.

> **Args:**

**source\_type: The type of source to create an AgentList from.**

Valid values include: ‘csv’, ‘tsv’, ‘excel’, ‘pandas’, etc.

[\*](/en/latest/#id12)args: Positional arguments to pass to the source-specific method. instructions: Optional instructions to apply to all created agents. codebook: Optional dictionary mapping trait names to descriptions, or a path to a CSV file.

If a CSV file is provided, it should have 2 columns: original keys and descriptions. Keys will be automatically converted to pythonic names.

name\_field: The name of the field to use as the agent name (for CSV/Excel sources). [\*\*](/en/latest/#id14)kwargs: Additional keyword arguments to pass to the source-specific method.

> **Returns:**

An AgentList object created from the specified source.

> **Examples:**

```python theme={null}
>>> # Create agents from a CSV file with instructions
>>> # agents = AgentList.from_source(
>>> #     'csv', 'agents.csv',
>>> #     instructions="Answer as if you were the person described"
>>> # )
>>> #
>>> # Create agents with a CSV codebook file
>>> # agents = AgentList.from_source(
>>> #     'csv', 'agents.csv',
>>> #     codebook='codebook.csv'  # CSV with keys like "Age in years" -> "age_in_years"
>>> # )
```

> #### *static* get\_codebook(*file\_path: str*) → dict [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Returns a codebook dictionary mapping CSV column names to None.

Reads the header row of a CSV file and creates a codebook with field names as keys and None as values.

**Args:**

file\_path: Path to the CSV file to read.

**Returns:**

A dictionary with CSV column names as keys and None as values.

**Raises:**

FileNotFoundError: If the specified file path does not exist. csv.Error: If there is an error reading the CSV file.

> #### get\_tabular\_data(*remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*) → Tuple\[List\[str], List\[List]] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Internal method to get tabular data in a standard format.

**Args:**

remove\_prefix: Whether to remove the prefix from column names pretty\_labels: Dictionary mapping original column names to pretty labels

**Returns:**

Tuple containing (header\_row, data\_rows)

> #### ggplot2(*ggplot\_code: str*, *shape: str = 'wide'*, *sql: str | None = None*, *remove\_prefix: bool = True*, *debug: bool = False*, *height: float = 4*, *width: float = 6*, *factor\_orders: dict | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create visualizations using R’s ggplot2 library.

This method provides a bridge to R’s powerful ggplot2 visualization library, allowing you to create sophisticated plots directly from EDSL data structures.

> **Parameters:**

ggplot\_code: R code string containing ggplot2 commands shape: Data shape to use (“wide” or “long”) sql: Optional SQL query to transform data before visualization remove\_prefix: Whether to remove prefixes (like “answer.”) from column names debug: Whether to display debugging information height: Plot height in inches width: Plot width in inches factor\_orders: Dictionary mapping factor variables to their desired order

> **Returns:**

A plot object that renders in Jupyter notebooks

> **Notes:**

* Requires R and the ggplot2 package to be installed
* Data is automatically converted to a format suitable for ggplot2
* The ggplot2 code should reference column names as they appear after any transformations from the shape and remove\_prefix parameters

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> # The following would create a plot if R is installed (not shown in doctest):
>>> # r.ggplot2('''
>>> #     ggplot(df, aes(x=how_feeling)) +
>>> #     geom_bar() +
>>> #     labs(title="Distribution of Feelings")
>>> # ''')
```

> #### give\_names(\**trait\_keys: str*, *remove\_traits: bool = True*, *separator: str = ','*, *force\_name: bool = False*) → None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Give names to agents based on the values of the specified traits.

```python theme={null}
>>> from edsl import Agent
>>> al = AgentList([Agent(traits = {'a': 1, 'b': 1}),
...                Agent(traits = {'a': 1, 'b': 2})])
>>> al.give_names('a')
>>> al[0].name
'1'
```

> #### html(*filename: str | None = None*, *cta: str = 'Open in browser'*, *return\_link: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

> #### join(*other: [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList")*, *join\_type: str = 'inner'*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Join this AgentList with another AgentList.

> **Args:**

other: The other AgentList to join with join\_type: The type of join to perform (“inner”, “left”, or “right”)

> **Returns:**

AgentList: A new AgentList containing the joined results

> **Examples:**

```python theme={null}
>>> from edsl import Agent, AgentList
>>> al1 = AgentList([Agent(name="John", traits={"age": 30})])
>>> al2 = AgentList([Agent(name="John", traits={"height": 180})])
>>> joined = al1.join(al2)
>>> joined[0].traits
{'age': 30, 'height': 180}
```

> #### *classmethod* join\_multiple(\*\*agent\_lists: [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList")\*, *join\_type: str = 'inner'*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Join multiple AgentLists together.

> **Args:**

[\*](/en/latest/#id16)agent\_lists: Variable number of AgentList objects to join join\_type: The type of join to perform (“inner”, “left”, or “right”)

> **Returns:**

AgentList: A new AgentList containing the joined results

> **Raises:**

ValueError: If fewer than 2 AgentLists are provided

> **Examples:**

```python theme={null}
>>> from edsl import Agent, AgentList
>>> al1 = AgentList([Agent(name="John", traits={"age": 30})])
>>> al2 = AgentList([Agent(name="John", traits={"height": 180})])
>>> al3 = AgentList([Agent(name="John", traits={"weight": 75})])
>>> joined = AgentList.join_multiple(al1, al2, al3)
>>> len(joined)
1
>>> joined[0].traits
{'age': 30, 'height': 180, 'weight': 75}
```

> #### keep(\*\*field\_names: str | List\[str]\*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Keep only the specified fields from all agents in the AgentList.

> **Args:**

[\*](/en/latest/#id18)field\_names: The name(s) of the field(s) to keep. Can be:

* Single field name: keep(“age”)
* Multiple field names: keep(“age”, “height”)
* List of field names: keep(\[“age”, “height”])

> **Returns:**

AgentList: A new AgentList with only the specified fields kept for all agents.

> **Examples:**

Keep a single trait for all agents:

```python theme={null}
>>> from edsl import Agent, AgentList
>>> al = AgentList([Agent(traits={"age": 30, "hair": "brown", "height": 5.5})])
>>> al_kept = al.keep("age")
>>> al_kept[0].traits
{'age': 30}
```

Keep multiple traits using separate arguments:

```python theme={null}
>>> al = AgentList([Agent(traits={"age": 30, "hair": "brown", "height": 5.5})])
>>> al_kept = al.keep("age", "height")
>>> al_kept[0].traits
{'age': 30, 'height': 5.5}
```

Keep multiple traits using a list:

```python theme={null}
>>> al = AgentList([Agent(traits={"age": 30, "hair": "brown", "height": 5.5})])
>>> al_kept = al.keep(["age", "height"])
>>> al_kept[0].traits
{'age': 30, 'height': 5.5}
```

Keep agent fields and traits:

```python theme={null}
>>> al = AgentList([Agent(traits={"age": 30, "hair": "brown"}, name="John")])
>>> al_kept = al.keep("name", "age")
>>> al_kept[0].name
'John'
>>> al_kept[0].traits
{'age': 30}
```

> #### last() → [Agent](/en/latest/#edsl.agents.Agent "edsl.agents.agent.Agent") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Get the last agent in the list.

> #### make\_tabular(*remove\_prefix: bool*, *pretty\_labels: dict | None = None*) → tuple\[list, List\[list]] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Turn the results into a tabular format.

> **Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').make_tabular(remove_prefix = True)
(['how_feeling'], [['OK'], ['Great'], ['Terrible'], ['OK']])

>>> r.select('how_feeling').make_tabular(remove_prefix = True, pretty_labels = {'how_feeling': "How are you feeling"})
(['How are you feeling'], [['OK'], ['Great'], ['Terrible'], ['OK']])
```

> #### *classmethod* manage() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

> #### *property* names\*: List\[str]\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Returns the names of the agents in the AgentList.

> #### num\_observations() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return the number of observations in the dataset.

```python theme={null}
>>> from edsl.results import Results
>>> Results.example().num_observations()
4
```

> #### print\_long() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Print the results in a long format. >>> from edsl.results import Results >>> r = Results.example() >>> r.select(‘how\_feeling’).print\_long() answer.how\_feeling: OK answer.how\_feeling: Great answer.how\_feeling: Terrible answer.how\_feeling: OK

> #### relevant\_columns(*data\_type: str | None = None*, *remove\_prefix: bool = False*) → list [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return the set of keys that are present in the dataset.

**Parameters:**

* **data\_type** – The data type to filter by.
* **remove\_prefix** – Whether to remove the prefix from the column names.

```python expandable theme={null}
>>> from ..dataset import Dataset
>>> d = Dataset([{'a.b':[1,2,3,4]}])
>>> d.relevant_columns()
['a.b']

>>> d.relevant_columns(remove_prefix=True)
['b']

>>> d = Dataset([{'a':[1,2,3,4]}, {'b':[5,6,7,8]}])
>>> d.relevant_columns()
['a', 'b']

>>> from edsl.results import Results; Results.example().select('how_feeling', 'how_feeling_yesterday').relevant_columns()
['answer.how_feeling', 'answer.how_feeling_yesterday']

>>> from edsl.results import Results
>>> sorted(Results.example().select().relevant_columns(data_type = "model"))
['model.canned_response', 'model.inference_service', 'model.model', 'model.model_index', 'model.temperature']

>>> # Testing relevant_columns with invalid data_type raises DatasetValueError - tested in unit tests
```

> #### remove\_prefix() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Returns a new Dataset with the prefix removed from all column names.

The prefix is defined as everything before the first dot (.) in the column name. If removing prefixes would result in duplicate column names, an exception is raised.

> **Returns:**

Dataset: A new Dataset with prefixes removed from column names

> **Raises:**

ValueError: If removing prefixes would result in duplicate column names

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling', 'how_feeling_yesterday').relevant_columns()
['answer.how_feeling', 'answer.how_feeling_yesterday']
>>> r.select('how_feeling', 'how_feeling_yesterday').remove_prefix().relevant_columns()
['how_feeling', 'how_feeling_yesterday']
```

```python theme={null}
>>> from edsl.dataset import Dataset
>>> d = Dataset([{'a.x': [1, 2, 3]}, {'b.x': [4, 5, 6]}])
>>> # d.remove_prefix()
```

\# Testing remove\_prefix with duplicate column names raises DatasetValueError - tested in unit tests

> #### remove\_trait(*trait: str*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Remove traits from the AgentList.

> **Parameters:**

**traits** – The traits to remove.

```python theme={null}
>>> from edsl import Agent
>>> al = AgentList([Agent({'age': 22, 'hair': 'brown', 'height': 5.5}), Agent({'age': 22, 'hair': 'brown', 'height': 5.5})])
>>> al.remove_trait('age')
AgentList([Agent(traits = {'hair': 'brown', 'height': 5.5}), Agent(traits = {'hair': 'brown', 'height': 5.5})])
```

> #### rename(*old\_name: str*, *new\_name: str*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Rename a trait across all agents in the list.

> **Args:**

old\_name: The current name of the trait. new\_name: The new name to assign to the trait.

> **Returns:**

AgentList: A new AgentList with the renamed trait.

> **Examples:**

```python theme={null}
>>> from edsl import Agent
>>> al = AgentList([Agent(traits = {'a': 1, 'b': 1}),
...                Agent(traits = {'a': 1, 'b': 2})])
>>> al2 = al.rename('a', 'c')
>>> assert al2 == AgentList([Agent(traits = {'c': 1, 'b': 1}),
...                         Agent(traits = {'c': 1, 'b': 2})])
>>> assert al != al2
```

> #### report(\**fields: str | None*, *top\_n: int | None = None*, *header\_fields: List\[str] | None = None*, *divider: bool = True*, *return\_string: bool = False*, *format: str = 'markdown'*, *filename: str | None = None*) → str | Document | None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Generates a report of the results by iterating through rows.

> **Args:**

[\*](/en/latest/#id20)fields: The fields to include in the report. If none provided, all fields are used. top\_n: Optional limit on the number of observations to include. header\_fields: Optional list of fields to include in the main header instead of as sections. divider: If True, adds a horizontal rule between observations (markdown only). return\_string: If True, returns the markdown string. If False (default in notebooks),

only displays the markdown without returning.

format: Output format - either “markdown” or “docx”. filename: If provided and format is “docx”, saves the document to this file.

> **Returns:**

Depending on format and return\_string: - For markdown: A string if return\_string is True, otherwise None (displays in notebook) - For docx: A docx.Document object, or None if filename is provided (saves to file)

Examples:

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> report = r.select('how_feeling').report(return_string=True)
>>> "# Observation: 1" in report
True
>>> doc = r.select('how_feeling').report(format="docx")
>>> isinstance(doc, object)
True
```

> #### report\_from\_template(*template: str*, \**fields: str | None*, *top\_n: int | None = None*, *remove\_prefix: bool = True*, *return\_string: bool = False*, *format: str = 'text'*, *filename: str | None = None*, *separator: str = 'nn'*, *observation\_title\_template: str | None = None*, *explode: bool = False*, *filestore: bool = False*) → str | Document | List | [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore") | None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Generates a report using a Jinja2 template for each row in the dataset.

This method renders a user-provided Jinja2 template for each observation in the dataset, with template variables populated from the row data. This allows for completely customized report formatting using pandoc for advanced output formats.

> **Args:**

template: Jinja2 template string to render for each row [\*](/en/latest/#id22)fields: The fields to include in template context. If none provided, all fields are used. top\_n: Optional limit on the number of observations to include. remove\_prefix: Whether to remove type prefixes (e.g., “answer.”) from field names in template context. return\_string: If True, returns the rendered content. If False (default in notebooks),

only displays the content without returning.

format: Output format - one of “text”, “html”, “pdf”, or “docx”. Formats other than “text” require pandoc. filename: If provided, saves the rendered content to this file. For exploded output,

this becomes a template (e.g., “report\_”).

separator: String to use between rendered templates for each row (ignored when explode=True). observation\_title\_template: Optional Jinja2 template for observation titles.

Defaults to “Observation ” where index is 1-based. Template has access to all row data plus ‘index’ and ‘index0’ variables.

explode: If True, creates separate files for each observation instead of one combined file. filestore: If True, wraps the generated file(s) in FileStore object(s). If no filename is provided,

creates temporary files. For exploded output, returns a list of FileStore objects.

> **Returns:**

Depending on explode, format, return\_string, and filestore: - For text format: String content or None (if displayed in notebook) - For html format: HTML string content or None (if displayed in notebook) - For docx format: Document object or None (if saved to file) - For pdf format: PDF bytes or None (if saved to file) - If explode=True: List of created filenames (when filename provided) or list of documents/content - If filestore=True: FileStore object(s) containing the generated file(s)

> **Notes:**

* Pandoc is required for HTML, PDF, and DOCX output formats
* Templates are treated as Markdown for all non-text formats
* PDF output uses XeLaTeX engine through pandoc
* HTML output includes standalone document structure

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> template = "Person feels: {{ how_feeling }}"
>>> report = r.select('how_feeling').report_from_template(template, return_string=True)
>>> "Person feels: OK" in report
True
>>> "Person feels: Great" in report
True
```

```python theme={null}
# Custom observation titles >>> custom_title = “Response {{ index }}: {{ how_feeling }}” >>> report = r.select(‘how_feeling’).report_from_template( … template, observation_title_template=custom_title, return_string=True) >>> “Response 1: OK” in report True

# HTML output (requires pandoc) >>> html_report = r.select(‘how_feeling’).report_from_template( … template, format=”html”, return_string=True) # doctest: +SKIP >>> # Creates HTML with proper document structure

# PDF output (requires pandoc with XeLaTeX) >>> pdf_report = r.select(‘how_feeling’).report_from_template( … template, format=”pdf”) # doctest: +SKIP >>> # Returns PDF bytes

# Basic template functionality >>> template2 = “Feeling: {{ how_feeling }}, Index: {{ index }}” >>> report2 = r.select(‘how_feeling’).report_from_template( … template2, return_string=True, top_n=2) >>> “Feeling: OK, Index: 1” in report2 True
```

> #### sample(*n: int*, *seed: str | None = None*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return a random sample of agents.

**Args:**

n: The number of agents to sample. seed: Optional seed for the random number generator to ensure reproducibility.

**Returns:**

AgentList: A new AgentList containing the sampled agents.

> #### select(\**traits*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Create a new AgentList with only the specified traits.

**Args:**

[\*](/en/latest/#id24)traits: Variable number of trait names to keep.

**Returns:**

AgentList: A new AgentList containing agents with only the selected traits.

Examples:

```python theme={null}
>>> from edsl import Agent
>>> al = AgentList([Agent(traits = {'a': 1, 'b': 1}),
...                Agent(traits = {'a': 1, 'b': 2})])
>>> al.select('a')
AgentList([Agent(traits = {'a': 1}), Agent(traits = {'a': 1})])
```

> #### set\_codebook(*codebook: dict\[str, str]*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Set the codebook for the AgentList.

```python theme={null}
>>> from edsl import Agent
>>> a = Agent(traits = {'hair': 'brown'})
>>> al = AgentList([a, a])
>>> _ = al.set_codebook({'hair': "Color of hair on driver's license"})
>>> al[0].codebook
{'hair': "Color of hair on driver's license"}
```

> **Parameters:**

**codebook** – The codebook.

> #### set\_dynamic\_traits(*function: Callable*) → None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Set the dynamic traits for all agents in the list.

> **Args:**

function: The function to set.

> #### set\_dynamic\_traits\_from\_question\_map(*q\_to\_traits: dict\[str, list\[str]]*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Configure dynamic traits for each agent from a question→traits mapping (in-place).

Each agent will get a dynamic traits function that, when asked a question whose `question_name` is present in `q_to_traits`, returns a dict mapping the corresponding trait name(s) to the agent’s original static value(s) for those trait(s).

A warning is emitted if the set of mapped trait names does not exactly equal the set of trait keys present in this AgentList.

> **Args:**

q\_to\_traits: Mapping from question name to list of trait keys, e.g.

`{"geo": ["hometown"], "cuisine": ["food"]}`

> **Returns:**

AgentList: self (modified in-place).

> **Examples:**

```python theme={null}
>>> from edsl import Agent, AgentList
>>> a_alice = Agent(name="Alice", traits={'hometown': 'Boston', 'food': 'beans'})
>>> a_bob = Agent(name="Bob", traits={'hometown': 'SF', 'food': 'sushi'})
>>> al = AgentList([a_alice, a_bob])
>>> _ = al.set_dynamic_traits_from_question_map({'geo': ['hometown'], 'cuisine': ['food']})
>>> class Q:
...     def __init__(self, name): self.question_name = name
>>> al[0].dynamic_traits_function(Q('geo'))['hometown']
'Boston'
>>> al[1].dynamic_traits_function(Q('geo'))['hometown']
'SF'
>>> al[0].dynamic_traits_function(Q('cuisine'))['food']
'beans'
```

> #### set\_instruction(*instruction: str*) → None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Set the instruction for all agents in the list.

**Args:**

instruction: The instruction to set.

> #### set\_traits\_presentation\_template(*traits\_presentation\_template: str*) → None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Set the traits presentation template for all agents in the list.

**Args:**

traits\_presentation\_template: The traits presentation template to set.

> #### shuffle(*seed: str | None = None*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Randomly shuffle the agents in place.

**Args:**

seed: Optional seed for the random number generator to ensure reproducibility.

**Returns:**

AgentList: The shuffled AgentList (self).

> #### sql(*query: str*, *transpose: bool = None*, *transpose\_by: str = None*, *remove\_prefix: bool = True*, *shape: str = 'wide'*) → [Dataset](/en/latest/dataset#edsl.dataset.Dataset "edsl.dataset.Dataset") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Execute SQL queries on the dataset.

This powerful method allows you to use SQL to query and transform your data, combining the expressiveness of SQL with EDSL’s data structures. It works by creating an in-memory SQLite database from your data and executing the query against it.

> **Parameters:**

query: SQL query string to execute transpose: Whether to transpose the resulting table (rows become columns) transpose\_by: Column to use as the new index when transposing remove\_prefix: Whether to remove type prefixes (e.g., “answer.”) from column names shape: Data shape to use (“wide” or “long”)

* “wide”: Default tabular format with columns for each field
* “long”: Melted format with key-value pairs, useful for certain queries

> **Returns:**

A Dataset object containing the query results

> **Notes:**

* The data is stored in a table named “self” in the SQLite database
* In wide format, column names include their type prefix unless remove\_prefix=True
* In long format, the data is melted into columns: row\_number, key, value, data\_type
* Complex objects like lists and dictionaries are converted to strings

> **Examples:**

```python theme={null}
>>> from edsl import Results
>>> r = Results.example()
```

```sql theme={null}
# Basic selection >>> len(r.sql(“SELECT * FROM self”, shape=”wide”)) 4

# Filtering with WHERE clause >>> r.sql(“SELECT * FROM self WHERE how_feeling = ‘Great’”).num_observations() 1

# Aggregation >>> r.sql(“SELECT how_feeling, COUNT(*) as count FROM self GROUP BY how_feeling”).keys() [‘how_feeling’, ‘count’]

# Using long format >>> len(r.sql(“SELECT * FROM self”, shape=”long”)) 200
```

> #### table(\**fields*, *tablefmt: str | None = 'rich'*, *pretty\_labels: dict | None = None*) → Any [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

> #### tally(\**fields: str | None*, *top\_n: int | None = None*, *output='Dataset'*) → dict | [Dataset](/en/latest/dataset#edsl.dataset.Dataset "edsl.dataset.Dataset") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Count frequency distributions of values in specified fields.

This method tallies the occurrence of unique values within one or more fields, similar to a GROUP BY and COUNT in SQL. When multiple fields are provided, it performs cross-tabulation across those fields.

> **Parameters:**

[\*](/en/latest/#id26)fields: Field names to tally. If none provided, uses all available fields. top\_n: Optional limit to return only the top N most frequent values. output: Format for results, either “Dataset” (recommended) or “dict”.

> **Returns:**

By default, returns a Dataset with columns for the field(s) and a ‘count’ column. If output=”dict”, returns a dictionary mapping values to counts.

> **Notes:**

* For single fields, returns counts of each unique value
* For multiple fields, returns counts of each unique combination of values
* Results are sorted in descending order by count
* Fields can be specified with or without their type prefix

> **Examples:**

```python theme={null}
>>> from edsl import Results
>>> r = Results.example()
```

```sql theme={null}

# Single field frequency count >>> r.select(‘how_feeling’).tally(‘answer.how_feeling’, output=”dict”) {‘OK’: 2, ‘Great’: 1, ‘Terrible’: 1}

# Return as Dataset (default) >>> from edsl.dataset import Dataset >>> expected = Dataset([{‘answer.how_feeling’: [‘OK’, ‘Great’, ‘Terrible’]}, {‘count’: [2, 1, 1]}]) >>> r.select(‘how_feeling’).tally(‘answer.how_feeling’, output=”Dataset”) == expected True

# Multi-field cross-tabulation - exact output varies based on data >>> result = r.tally(‘how_feeling’, ‘how_feeling_yesterday’) >>> ‘how_feeling’ in result.keys() and ‘how_feeling_yesterday’ in result.keys() and ‘count’ in result.keys() True
```

> #### to(*target: 'Question' | 'Jobs' | 'Survey'*) → [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

> #### to\_agent\_list(*remove\_prefix: bool = True*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Convert the results to a list of dictionaries, one per agent.

> **Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_agent_list()
AgentList([Agent(traits = {'how_feeling': 'OK'}), Agent(traits = {'how_feeling': 'Great'}), Agent(traits = {'how_feeling': 'Terrible'}), Agent(traits = {'how_feeling': 'OK'})])
```

> #### to\_csv(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Export the results to a FileStore instance containing CSV data.

> #### to\_dataset(*traits\_only: bool = True*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Convert the AgentList to a Dataset.

> **Args:**

traits\_only: If True, only include agent traits. If False, also include

agent parameters like instructions and names.

> **Returns:**

Dataset: A dataset containing the agents’ traits and optionally their parameters.

> **Examples:**

```python theme={null}
>>> from edsl import AgentList
>>> al = AgentList.example()
>>> al.to_dataset()
Dataset([{'age': [22, 22]}, {'hair': ['brown', 'brown']}, {'height': [5.5, 5.5]}])
>>> al.to_dataset(traits_only=False)
Dataset([{'age': [22, 22]}, {'hair': ['brown', 'brown']}, {'height': [5.5, 5.5]}, {'agent_parameters': [{'instruction': 'You are answering questions as if you were a human. Do not break character.', 'agent_name': None, 'traits_presentation_template': 'Your traits: {{ traits }}'}, {'instruction': 'You are answering questions as if you were a human. Do not break character.', 'agent_name': None, 'traits_presentation_template': 'Your traits: {{ traits }}'}]}])
```

> #### to\_dict(*sorted=False*, *add\_edsl\_version=True*, *full\_dict=False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Serialize the AgentList to a dictionary.

```python theme={null}
>>> AgentList.example().to_dict(add_edsl_version=False)
{'agent_list': [{'traits': {'age': 22, 'hair': 'brown', 'height': 5.5}}, {'traits': {'age': 22, 'hair': 'brown', 'height': 5.5}}]}
>>> example_codebook = {'age': 'Age in years'}
>>> al = AgentList.example().set_codebook(example_codebook)
>>> result = al.to_dict(add_edsl_version=False)
>>> 'codebook' in result
True
>>> result['codebook'] == example_codebook
True
```

> #### to\_dicts(*remove\_prefix: bool = True*) → list\[dict] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Convert the results to a list of dictionaries.

**Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_dicts()
[{'how_feeling': 'OK'}, {'how_feeling': 'Great'}, {'how_feeling': 'Terrible'}, {'how_feeling': 'OK'}]
```

> #### to\_docx(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Export the results to a FileStore instance containing DOCX data.

Each row of the dataset will be rendered on its own page, with a 2-column table that lists the keys and associated values for that observation.

> #### to\_excel(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*, *sheet\_name: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Export the results to a FileStore instance containing Excel data.

> #### to\_jsonl(*filename: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Export the results to a FileStore instance containing JSONL data.

> #### to\_list(*flatten=False*, *remove\_none=False*, *unzipped=False*) → list\[list] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Convert the results to a list of lists.

> **Parameters:**

* **flatten** – Whether to flatten the list of lists.
* **remove\_none** – Whether to remove None values from the list.

```python theme={null}
>>> from edsl.results import Results
>>> Results.example().select('how_feeling', 'how_feeling_yesterday')
Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling_yesterday': ['Great', 'Good', 'OK', 'Terrible']}])
```

```python theme={null}
>>> Results.example().select('how_feeling', 'how_feeling_yesterday').to_list()
[('OK', 'Great'), ('Great', 'Good'), ('Terrible', 'OK'), ('OK', 'Terrible')]
```

```python theme={null}
>>> r = Results.example()
>>> r.select('how_feeling').to_list()
['OK', 'Great', 'Terrible', 'OK']
```

```python theme={null}
>>> from edsl.dataset import Dataset
>>> Dataset([{'a.b': [[1, 9], 2, 3, 4]}]).select('a.b').to_list(flatten = True)
[1, 9, 2, 3, 4]
```

```python theme={null}
>>> from edsl.dataset import Dataset
>>> # Testing to_list flatten with multiple columns raises DatasetValueError - tested in unit tests
```

> #### to\_pandas(*remove\_prefix: bool = False*, *lists\_as\_strings=False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Convert the results to a pandas DataFrame, ensuring that lists remain as lists.

**Args:**

remove\_prefix: Whether to remove the prefix from the column names. lists\_as\_strings: Whether to convert lists to strings.

> **Returns:**

A pandas DataFrame.

> #### to\_polars(*remove\_prefix: bool = False*, *lists\_as\_strings=False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Convert the results to a Polars DataFrame.

> **Args:**

remove\_prefix: Whether to remove the prefix from the column names. lists\_as\_strings: Whether to convert lists to strings.

> **Returns:**

A Polars DataFrame.

> #### to\_scenario\_list(*remove\_prefix: bool = True*) → list\[dict] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Convert the results to a list of dictionaries, one per scenario.

**Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_scenario_list()
ScenarioList([Scenario({'how_feeling': 'OK'}), Scenario({'how_feeling': 'Great'}), Scenario({'how_feeling': 'Terrible'}), Scenario({'how_feeling': 'OK'})])
```

> #### to\_sqlite(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*, *table\_name: str = 'results'*, *if\_exists: str = 'replace'*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Export the results to a SQLite database file.

> #### *property* trait\_keys\*: List\[str]\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Get the trait keys for the AgentList.

> #### translate\_traits(*codebook: dict\[str, str]*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Translate traits to a new codebook.

**Parameters:**

**codebook** – The new codebook.

```python theme={null}
>>> al = AgentList.example()
>>> codebook = {'hair': {'brown':'Secret word for green'}}
>>> al.translate_traits(codebook)
AgentList([Agent(traits = {'age': 22, 'hair': 'Secret word for green', 'height': 5.5}), Agent(traits = {'age': 22, 'hair': 'Secret word for green', 'height': 5.5})])
```

> #### tree(*node\_order: List\[str] | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Convert the results to a Tree.

**Args:**

node\_order: The order of the nodes.

**Returns:**

A Tree object.

> #### unpack\_list(*field: str*, *new\_names: List\[str] | None = None*, *keep\_original: bool = True*) → [Dataset](/en/latest/dataset#edsl.dataset.Dataset "edsl.dataset.Dataset") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Unpack list columns into separate columns with provided names or numeric suffixes.

For example, if a dataset contains: \[\{'data': \[\[1, 2, 3], \[4, 5, 6]], 'other': \['x', 'y']}]

After d.unpack\_list(‘data’), it should become: \[\{'other': \['x', 'y'], 'data\_1': \[1, 4], 'data\_2': \[2, 5], 'data\_3': \[3, 6]}]

> **Args:**

field: The field containing lists to unpack new\_names: Optional list of names for the unpacked fields. If None, uses numeric suffixes. keep\_original: If True, keeps the original field in the dataset

> **Returns:**

A new Dataset with unpacked columns

> **Examples:**

```python theme={null}
>>> from edsl.dataset import Dataset
>>> d = Dataset([{'data': [[1, 2, 3], [4, 5, 6]]}])
>>> d.unpack_list('data')
Dataset([{'data': [[1, 2, 3], [4, 5, 6]]}, {'data_1': [1, 4]}, {'data_2': [2, 5]}, {'data_3': [3, 6]}])
```

```python theme={null}
>>> d.unpack_list('data', new_names=['first', 'second', 'third'])
Dataset([{'data': [[1, 2, 3], [4, 5, 6]]}, {'first': [1, 4]}, {'second': [2, 5]}, {'third': [3, 6]}])
```

> #### with\_names(\**trait\_keys: str*, *remove\_traits: bool = True*, *separator: str = ','*, *force\_name: bool = False*) → [AgentList](/en/latest/#edsl.agents.AgentList "edsl.agents.agent_list.AgentList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/agents.py)

Return a new AgentList with names based on the values of the specified traits.

> **Args:**

[\*](/en/latest/#id28)trait\_keys: The trait keys to use for naming remove\_traits: Whether to remove the traits used for naming from the agents separator: The separator to use when joining multiple trait values force\_name: Whether to force naming even if agents already have names

> **Returns:**

AgentList: A new AgentList with named agents

```python theme={null}
>>> from edsl import Agent
>>> al = AgentList([Agent(traits = {'a': 1, 'b': 1}),
...                Agent(traits = {'a': 1, 'b': 2})])
>>> al_with_names = al.with_names('a')
>>> al_with_names[0].name
'1'
>>> al[0].name is None  # Original unchanged
True
```


# Managing Keys
Source: https://docs.expectedparrot.com/en/latest/api_keys

API keys are required to access large language models. To use EDSL with models you can either provide your own keys from service providers (*Anthropic, Azure, Bedrock, Deep Infra, DeepSeek, Google, Groq, Mistral, OpenAI, Perplexity, Together, Xai*) or use an **Expected Parrot key** to access all available models at the Expected Parrot server. See the [model pricing page](http://www.expectedparrot.com/models) for details on current available models and prices.

In addition to providing access to available models, your Expected Parrot key also allows you to post and share content for free.

This page shows how to store and select keys to use when running surveys on your own computer (*local inference*) or at the Expected Parrot server (*remote inference*).
For instructions on using your Expected Parrot key only for accessing the platform, please see [this section](/en/latest/coop) of the documentation.

**Special note for Colab users**: If you are using EDSL in a Colab notebook, please see [special instructions](/en/latest/colab_setup) on storing API keys as “secrets” ([Colab Setup](/en/latest/colab_setup)).

<Warning>
  *Caution: Treat your API keys as sensitive information, like passwords. Never share your keys publicly or upload files containing them to public repositories.*
</Warning>

## Methods

There are three methods for storing and managing keys (details on each below):

<Steps>
  <Step>
    Manage keys from your Expected Parrot account (*recommended*)
  </Step>

  <Step>
    Store keys in a local file on your computer
  </Step>

  <Step>
    Set keys in your code (*not recommended*)
  </Step>
</Steps>

### 1. Manage keys from your account

This method provides a secure way to store your keys and the most flexibility in choosing how to use them. It requires an Expected Parrot account ([log in/sign up here](https://www.expectedparrot.com/login)) and works with remote-inference.

*If you are only running surveys locally, see method 2. below for instructions on storing keys on your computer.*

To use this method, go to your [Settings](http://www.expectedparrot.com/home/settings) page and enable the **Run surveys remotely** toggle under Remote Execution:

<Frame>
  <img alt="Settings page showing Remote Execution and Remote Error Logging toggles" />
</Frame>

Then go to your [Keys](http://www.expectedparrot.com/home/keys) page and choose whether to add any of your own keys. Use the **Grant access** icon to share a key with other users without sharing the key directly. Use the **Edit** icon to modify settings for any of your keys, or the **Delete** icon to remove them.

You can review the current prioritization of your keys at the **Key priority** tab at any time. When you run surveys remotely, your Expected Parrot key is used by default with any models that you have not provided keys for.

<Frame>
  <img alt="Keys page showing LLM Keys and key management options" />
</Frame>

### 2. Store keys in a local file

This method allows you to store keys in a private file on your computer and make them available for any surveys that you run, locally or remotely. It also allows you to post local content to Expected Parrot.

To use this method:

<Steps>
  <Step>
    Navigate to your EDSL working directory: \$ cd /path/to/edsl (replace with your actual path)
  </Step>

  <Step>
    Create a file named .env
  </Step>

  <Step>
    Add your keys to the file in the following format (omit any keys that you do not have or do not want to use):

    ```bash expandable theme={null}
    EXPECTED_PARROT_API_KEY = 'your_key_here' # This key is required for remote inference and interacting with Expected Parrot

    ANTHROPIC_API_KEY = 'your_key_here'
    DEEP_INFRA_API_KEY = 'your_key_here'
    DEEPSEEK_API_KEY = 'your_key_here'
    GOOGLE_API_KEY = 'your_key_here'
    GROQ_API_KEY = 'your_key_here'
    MISTRAL_API_KEY = 'your_key_here'
    OPENAI_API_KEY = 'your_key_here'
    PERPLEXITY_API_KEY = 'your_key_here'
    TOGETHER_API_KEY = 'your_key_here'
    XAI_API_KEY = 'your_key_here'

    AWS_ACCESS_KEY_ID = 'your_key_here'
    AWS_SECRET_ACCESS_KEY = 'your_key_here'

    AZURE_ENDPOINT_URL_AND_KEY = https://model_1_link:api_key_1,https://model_2_link:api_key_2
    ```
  </Step>
</Steps>

Your Expected Parrot key can also be found at the [Keys](http://www.expectedparrot.com/home/keys) page of your account, where you can reset it at any time:

<Frame>
  <img alt="Keys page showing Expected Parrot API Key and local .env configuration" />
</Frame>

This key allows you to access all available models at once and use remote-inference to run surveys at the Expected Parrot server. It also allows you to post content to the platform, and to interact with other content that is public or shared with you.

If you are using Azure or Bedrock, see a notebook of examples for setting up your keys [here](/en/latest/edsl_with_cloud_providers).

### 3. Set keys in your code

<Warning>
  *Warning:* This method is not recommended for security reasons. If you include your keys in your code, they can be seen by anyone who has access to it. This is especially risky if you are sharing your code with others or if you are using a version control system like Git.
</Warning>

To use this method, store any keys that you want to use as strings in your code in the following format:

```bash expandable theme={null}
import os

os.environ['EXPECTED_PARROT_API_KEY'] = 'your_key_here'

os.environ['ANTHROPIC_API_KEY'] = 'your_key_here'
os.environ['DEEP_INFRA_API_KEY'] = 'your_key_here'
os.environ['DEEPSEEK_API_KEY'] = 'your_key_here'
os.environ['GOOGLE_API_KEY'] = 'your_key_here'
os.environ['GROQ_API_KEY'] = 'your_key_here'
os.environ['MISTRAL_API_KEY'] = 'your_key_here'
os.environ['OPENAI_API_KEY'] = 'your_key_here'
os.environ['REPLICATE_API_KEY'] = 'your_key_here'
os.environ['TOGETHER_API_KEY'] = 'your_key_here'
os.environ['XAI_API_KEY'] = 'your_key_here'

os.environ['AWS_ACCESS_KEY_ID'] = 'your_key_here'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'your_key_here'

os.environ['AZURE_ENDPOINT_URL_AND_KEY'] = https://model_1_link:api_key_1,https://model_2_link:api_key_2
```

<Note>
  Note that your keys will not persist across sessions and you will need to provide your keys each time you start a new session.
</Note>

## Credits

When you use your Expected Parrot key to access models your account is charged for the costs of API calls to models. (When you use your own keys, service providers will bill you directly.) Please see the [model pricing page](http://www.expectedparrot.com/getting-started/coop-pricing) for information on available models and prices and the credits section for information on purchasing credits and calculating costs.

## Troubleshooting

In order to use local inference, you must also have credits available on your account with a service provider in order to run surveys with some models. If you are using remote inference, simply ensure that you have credits on your Expected Parrot account to access all available models.

When you run a survey, EDSL checks whether you are using remote or local inference and then checks for the requisite API keys for the models that you have specified to use with the survey. If you do not specify a model to use for a survey, EDSL will attempt to run it with the default model. You can check the current default model by running the following command:

```bash theme={null}
from edsl import Model
Model()
```

Output:

| key | value |
| :- | :- |
| model | gpt-4o |
| parameters:temperature | 0.5 |
| parameters:max\_tokens | 1000 |
| parameters:top\_p | 1 |
| parameters:frequency\_penalty | 0 |
| parameters:presence\_penalty | 0 |
| parameters:logprobs | False |
| parameters:top\_logprobs | 3 |

To check all available models:

```python theme={null}
from edsl import Model
Model.available()
```

To check all available models for a specific provider:

```python theme={null}
from edsl import Model
Model.available(service="openai")
```

Learn more about available models in the [Language Models](/en/latest/language_models#language-models) section of the documentation.

If you attempt to run a survey without storing any API keys, you will get a message with a link to log into your Expected Parrot account and automatically activate remote inference and store your Expected Parrot API key for you.

If you provide an invalid API key you will receive an error message AuthenticationError: Incorrect API key provided…. You may also receive an error message if you do not have credits on your account with a service provider.

Learn more about handling errors in the [Exceptions & Debugging](/en/latest/exceptions#exceptions) section of the documentation.

Please also feel free to reach out to us to help you troubleshoot:

* Discord channel: [https://discord.com/invite/mxAYkjfy9m](https://discord.com/invite/mxAYkjfy9m)
* Email: [info@expectedparrot.com](mailto:info@expectedparrot.com)


# Base
Source: https://docs.expectedparrot.com/en/latest/base



## Base class

> ### *class* edsl.base.Base[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Bases: `RepresentationMixin`, `PersistenceMixin`, `DiffMethodsMixin`, `HashingMixin`, `ABC`

Base class for all classes in the EDSL package.

This abstract base class combines several mixins to provide a rich set of functionality to all EDSL objects. It defines the core interface that all EDSL objects must implement, including serialization, deserialization, and code generation.

All EDSL classes should inherit from this class to ensure consistent behavior and capabilities across the framework.

> #### apply\_command(*command\_name*, *kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

> #### *abstract* code()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Generate Python code that recreates this object.

This method must be implemented by all subclasses to provide a way to generate executable Python code that can recreate the object.

**Returns:**

str: Python code that, when executed, creates an equivalent object

> #### *abstract* example()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Create an example instance of this class.

This method should be implemented by all subclasses to provide a convenient way to create example objects for testing and demonstration.

**Returns:**

An instance of the class with sample data

> #### *abstract* from\_dict()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Create an instance from a dictionary.

This class method must be implemented by all subclasses to provide a standard way to deserialize objects from dictionaries.

**Returns:**

An instance of the class populated with data from the dictionary

> #### get\_description() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Get the description of this object.

> #### get\_uuid() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Get the UUID of this object from the Expected Parrot cloud service based on its hash.

This method calculates the hash of the object and queries the cloud service to find if there’s an uploaded version with the same content. If found, it returns the UUID of that object.

**Returns:**

str: The UUID of the object in the cloud service if found

**Raises:**

**CoopServerResponseError: If the object is not found or there’s an error**

communicating with the server

> #### inspect()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Create an interactive inspector widget for this object.

This method uses the InspectorWidget registry system to find the appropriate inspector widget class for this object’s type and returns an instance of it.

**Returns:**

InspectorWidget subclass instance: Interactive widget for inspecting this object

**Raises:**

KeyError: If no inspector widget is registered for this object’s class ImportError: If the widgets module cannot be imported

> #### json()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Get a formatted JSON representation of this object.

**Returns:**

DisplayJSON: A displayable JSON representation

> #### keys()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Get the key names in the object’s dictionary representation.

This method returns all the keys in the serialized form of the object, excluding metadata keys like version information.

**Returns:**

list: A list of key names

> #### *property* polly\_commands[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Auto-generate commands list from methods decorated with @polly\_command

> #### show\_methods(*show\_docstrings=True*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Display all public methods available on this object.

This utility method helps explore the capabilities of an object by listing all its public methods and optionally their documentation.

**Args:**

**show\_docstrings: If True, print method names with docstrings;**

if False, return the list of method names

**Returns:**

**None or list: If show\_docstrings is True, prints methods and returns None.**

If show\_docstrings is False, returns a list of method names.

> #### store(*d: dict*, *key\_name: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Store this object in a dictionary with an optional key.

**Args:**

d: The dictionary in which to store the object key\_name: Optional key to use (defaults to the length of the dictionary)

**Returns:**

None

> #### *abstract* to\_dict()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Serialize this object to a dictionary.

This method must be implemented by all subclasses to provide a standard way to serialize objects to dictionaries. The dictionary should contain all the data needed to reconstruct the object.

**Returns:**

dict: A dictionary representation of the object

> #### to\_json()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Serialize this object to a JSON string.

**Returns:**

str: A JSON string representation of the object

> #### values()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Get the values in the object’s dictionary representation.

**Returns:**

set: A set containing all the values in the object

> #### yaml()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/base.py)

Get a formatted YAML representation of this object.

**Returns:**

DisplayYAML: A displayable YAML representation


# Survey logic checklist
Source: https://docs.expectedparrot.com/en/latest/checklist

This page provides a checklist for reviewing the logic and contents of an EDSL survey and ensuring that it will run as you intend it to.

## Support

Please reach out to us with any questions or issues! Send an email to **[info@expectedparrot.com](mailto:info@expectedparrot.com)** or post a message at [Discord](https://discord.com/invite/mxAYkjfy9m).

### How EDSL works

At its core, EDSL is built on the concept of [Questions](/en/latest/questions) being answered by AI [Agents](/en/latest/agents), using [Language Models](/en/latest/language_models) to generate responses that are returned as formatted datasets of [Results](/en/latest/results).

A typical workflow consists of the following steps:

1. Construct **questions** of different types (multiple choice, free text, etc.) and combine them in a survey. Learn about [question types](/en/latest/questions).
2. *Optional:* Add **data or content** to your questions (from CSVs, PDFs, docs, images, etc.). Learn about [scenarios](/en/latest/scenarios).
3. *Optional:* Add **rules or logic** to specify how questions should be presented (e.g., skip/stop rules, or including context of other questions). Learn about [survey rules](/en/latest/surveys).
4. *Optional:* Design or import **personas** for AI agents to answer the questions. Learn about [agents](/en/latest/agents).
5. Select **language models** to generate responses. EDSL works with hundreds of popular [models](/en/latest/language_models).
6. Run the survey and get a formatted dataset of **results**. Use [built-in methods](/en/latest/results) for analyzing them.
7. *Optional:* Store and share your work at the [Expected Parrot](https://www.expectedparrot.com/login): a platform for AI research that is fully integrated with EDSL. [Learn more](/en/latest/coop).

### Issues checklist

Below are some issues to consider before running a survey, with links to examples of EDSL features for handling them.

<AccordionGroup>
  <Accordion title="Are you using appropriate question types?">
    Ensure that the question type you are using is appropriate for the question you are presenting.

    * Use QuestionYesNo when you only want to allow “Yes” or “No” responses.
    * Use QuestionMultipleChoice when you want to require that only one option be selected.
    * Use QuestionCheckBox when you want to allow multiple (or no) selections or specify a number of selections.
  </Accordion>

  <Accordion title="Are question options correct and complete?">
    Ensure that the question options make sense for the question text.

    * Does the question text ask for options not presented, or a different number of options?
    * Should the question options include an option for non-responses (e.g., “I do not know.”)?
    * Are validation constraints appropriate for your use case (e.g., min/max values, required selections)?
  </Accordion>

  <Accordion title="Do any questions require context of other questions?">
    Survey questions are administered asynchronously by default, meaning that the presentation of one question does not include context of any other questions in the survey unless you specify otherwise. This default functionality reduces costs and runtime. It also allows you to fine-tune and readily compare responses to versions of questions with different contexts.

    If a question depends on or requires information about other questions in a survey, you need to add a rule specifying the logic to be applied. This can be done in a variety of ways:

    * Use [piping](/en/latest/surveys) to add components of a question in another question (e.g., insert the answer to a question in the text of a follow-up question).
    * Use question [memory](/en/latest/surveys) to include the entire context of a question/answer in the presentation of a different question (*“You were previously asked…”*). There are separate rules for adding the context of a single question, a set of questions, all prior questions, or a lagged number of questions.
  </Accordion>
</AccordionGroup>

## Testing a model’s ability to answer a question

Running test questions and examining the answers and [“comment” fields](/en/latest/questions) can help you understand whether a question needs to be improved. You can also run questions *about* your questions for [cognitive testing](/en/latest/notebooks/research_methods) (e.g., *“How is the following question confusing: `{{ question }}`”*)

### Tools for integrating data

## Scenario objects

EDSL provides many methods for importing and formatting data or content to be used with a survey. This can be particularly useful for surveys that involve [data labeling](/en/latest/notebooks/data_labeling_example) or [data cleaning](/en/latest/notebooks/data_cleaning) tasks, such as using questions to extract information from unstructured texts or apply labels to images or other content. This is done by creating Scenerio \<[/en/latest/scenarios](/en/latest/scenarios)>\`\_ objects representing data or content to be added to survey questions (e.g., *“What is the main topic of this text? \{\{ text }}?”, “What is in this image? \{\{ image }}”*). See [examples](/en/latest/notebooks/question_loop_scenarios) for working with [PDFs](/en/latest/notebooks/scenario_from_pdf), CSVs, [images](/en/latest/notebooks/image_scenario_example), docs, [tables](/en/latest/notebooks/scenario_list_wikipedia) and other data types. You can also use scenarios to store [metadata](/en/latest/notebooks/adding_metadata) for surveys.

## Converting questions

Many models are highly capable of reformatting non-EDSL surveys based on a prompt that includes basic instructions or examples of EDSL code. This can be an efficient way to reconstruct an existing survey in EDSL and then modify as needed with above-mentioned methods. See examples: [CES data](/en/latest/notebooks/ces_data_edsl); [GoogleForms](/en/latest/notebooks/google_form_to_edsl).


# Citation
Source: https://docs.expectedparrot.com/en/latest/citation

If you use the EDSL package in your research or projects, please use the following citation:

```bash theme={null}
@techreport{Horton2024EDSL,
title = {EDSL: Expected Parrot Domain Specific Language for AI Powered Social Science},
author = {John Horton and Robin Horton},
year = {2024},
institution = {Expected Parrot},
type = {Whitepaper}
}
```


# Colab Notebooks
Source: https://docs.expectedparrot.com/en/latest/colab_notebooks

All of the methods for working with notebooks in EDSL are available in Colab.

To access these methods in Colab, please first complete the following steps to set up EDSL and connect your Google Drive:

## 1. Store your API key as a secret

In Google Colab, your API keys can be stored as “secrets” in lieu of storing them in a *.env* file as you would in other notebook types.

For example, you can store your Expected Parrot API key as follows:

<Frame>
  <img alt="Storing API key in Google Colab" />
</Frame>

## 2. Install EDSL

Run the following command in a code cell to install the EDSL package:

```python theme={null}
pip install edsl
```

## 3. Access your API key

Run the following code to access your API key in your Colab notebook:

```python theme={null}
import os
from google.colab import userdata

os.environ['EXPECTED_PARROT_API_KEY'] = userdata.get('EXPECTED_PARROT_API_KEY')
```

## 4. Connect your Google Drive

To access your files in Google Drive, you need to connect your Google Drive to Colab. Run the following code to do this, and accept the permissions request in the pop-up window:

```python theme={null}
from google.colab import drive

drive.mount('/content/drive')
```

You will see the following message returned:

```
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
```

## 5. Access your files

Run the following code to see the names of all the files in a Google Drive folder

<Note>
  **Note**:

  Note that you will need to replace the path with the path to your own folder, and you may need to adjust the path to match the structure of your Google Drive (e.g., here the default Google Drive folder name “Colab Notebooks” has been changed to “ColabNotebooks” for convenience in specifying the path):
</Note>

```python theme={null}
import os

print(sorted(os.listdir('/content/drive/MyDrive/ColabNotebooks/')))
```

Sample output:

```
['colab_to_coop.ipynb']
```

## 6. Post a notebook to Expected Parrot

Now that you have your notebook ready, you can post it to Expected Parrot using the Notebook object. Modify the path to your notebook file path in your Google Drive as needed:

```python theme={null}
from edsl import Notebook

notebook = Notebook(path="/content/drive/MyDrive/ColabNotebooks/colab_to_coop.ipynb")

notebook.push(description="Posting a Colab notebook to Expected Parrot")
```

Example output:

```json theme={null}
{'description': 'Posting a Colab notebook to Expected Parrot',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/a878656a-317a-4181-a496-3c49f12e38d7',
'uuid': 'a878656a-317a-4181-a496-3c49f12e38d7',
'version': '0.1.36',
'visibility': 'unlisted'}
```

## 7. Update or edit a notebook at Expected Parrot

```python theme={null}
from edsl import Notebook

notebook = Notebook(path="/content/drive/MyDrive/ColabNotebooks/colab_to_coop.ipynb")

notebook.patch(
    uuid = "a878656a-317a-4181-a496-3c49f12e38d7",
    visibility = "public",
    value = notebook
    )
```

Output:

```
{'status': 'success'}
```

## Example Colab code

<Frame>
  <img alt="Posting a Colab notebook to Expected Parrot" />
</Frame>


# Colab Setup
Source: https://docs.expectedparrot.com/en/latest/colab_setup

All examples, tutorials, and demo notebooks are executable in any Python notebook environment.

To run them in Colab, please follow the special instructions below for technical setup:

## 1. Store your API key as a secret

In Google Colab, your API keys can be stored as “secrets” in lieu of storing them in a *.env* file as you would in other notebook types.

For example, you can store your Expected Parrot API key as follows:

<Frame>
  <img alt="Storing API key in Google Colab" />
</Frame>

## 2. Install EDSL

Run the following command in a code cell to install the EDSL package:

```bash theme={null}
pip install edsl
```

## 3. Access your API key

Run the following code to access your API key in your Colab notebook:

```python theme={null}
import os
from google.colab import userdata

os.environ['EXPECTED_PARROT_API_KEY'] = userdata.get('EXPECTED_PARROT_API_KEY')
```

## Example Colab code

EDSL methods for creating objects and posting them to Expected Parrot are now available to you in Colab.

Here’s a snapshot of how your Colab code might look:

<Frame>
  <img alt="Storing and using API key in Google Colab" />
</Frame>

## Posting Colab notebooks to Expected Parrot

Special instructions for connecting your Google Drive to post Colab notebooks to Expected Parrot are available in the [Colab Notebooks](/en/latest/colab_notebooks) section.


# Contributing
Source: https://docs.expectedparrot.com/en/latest/contributing



Thank you for checking out EDSL! We sincerely appreciate feedback and welcome contributions to this project.

This page describes some ways to contribute to the project. It includes information on how to report bugs, suggest enhancements, draft new documentation pages, and contribute code.

Please also feel free to reach out to us in our Discord channel or by email if you have any questions or need help:
<Icon icon="discord" /> [Discord channel](https://discord.com/invite/mxAYkjfy9m)
<Icon icon="email" /> [info@expectedparrot.com](mailto:info@expectedparrot.com)

## Reporting bugs

If you find a bug in the project, please let us know and report it on the project’s GitHub issue tracker. Link to the issue tracker:

[EDSL GitHub issue tracker](https://github.com/expectedparrot/edsl/issues/new)

When reporting a bug, it is helpful to include the following information:

* The version of the project you are using. (Run edsl –version to find the version number.)
* The operating system you are using. (For example, Windows, macOS, or Linux.)
* The Python version you are using. (Run python –version to find the Python version number.)
* A detailed description of the bug, including the steps to reproduce it.
* If possible, a minimal example that reproduces the bug.

If you are able to fix the bug, please consider submitting a pull request with the fix (see below).

## Suggesting enhancements

If you have an idea for an enhancement to the project, please let us know and report it on the project’s GitHub issue tracker (see link above).

When suggesting an enhancement, it is helpful to include the following information:

* A detailed description of the enhancement.
* A rationale for the enhancement.
* If possible, a sketch of the proposed implementation.

If you do not wish to implement the enhancement yourself, please consider submitting a pull request with the changes (see below).

## Drafting new documentation pages

If you would like to contribute a new documentation page to the project, please follow these steps:

<Steps>
  <Step>
    Fork the project on GitHub.
  </Step>

  <Step>
    Create a new branch for your documentation page.
  </Step>

  <Step>
    Add the new documentation page to the docs/ directory.
  </Step>

  <Step>
    Add a link to the new documentation page in the docs/index.rst file.
  </Step>

  <Step>
    Submit a pull request with your changes.
  </Step>
</Steps>

Please follow the project’s documentation conventions when writing documentation (header styles, code blocks, links, etc.). For example, use the reStructuredText markup language and consider copying this or other pages as a template.

## Contributing code

If you would like to contribute code to the project, it is helpfup to follow these steps:

<Steps>
  <Step>
    Fork the project on GitHub.
  </Step>

  <Step>
    Create a new branch for your changes.
  </Step>

  <Step>
    Write your code.
  </Step>

  <Step>
    Write tests for your code.
  </Step>

  <Step>
    Run the tests.
  </Step>

  <Step>
    Submit a pull request with your changes.
  </Step>
</Steps>

When submitting a pull request, please include the following information:

* A description of the changes in the pull request.
* A rationale for the changes.
* If the changes fix a bug, please include the issue number.

When writing tests, please follow the project’s testing conventions. The project uses the unittest testing framework.

Thank you!


# Expected Parrot
Source: https://docs.expectedparrot.com/en/latest/coop

Expected Parrot provides a free platform for storing, sharing and collaborating on AI-based research, fully integrated with EDSL via the Coop module.

Expected Parrot provides a free platform for creating, storing and sharing AI-based research, and validating it with human respondents. It is fully integrated with EDSL via the *Coop* module, allowing you to post, store and retrieve any objects that you've created with EDSL, together with data, notebooks and other project content.

You can also explore public content and collaborate on projects privately with other users:

<Frame>
  <img alt="Expected Parrot page view" />
</Frame>

Your Expected Parrot account provides access to special features for running EDSL surveys and storing results remotely at the Expected Parrot server, and launching hybrid human-AI inference jobs interactively. Learn more about these features in the [Remote Inference](/en/latest/remote_inference), [Remote Caching](/en/latest/remote_caching) and [Survey Builder](/en/latest/survey_builder) sections of the documentation.

## Getting started

### 1. Create an account

[Sign up](https://www.expectedparrot.com/login) with an email address.

Your account comes with an **Expected Parrot API key** that allows you to:

* Post and share content
* Run surveys remotely
* Use any available models with your surveys
* Use the universal cache of stored prompts and responses

You can inspect your key and reset it at any time at your [Keys](https://www.expectedparrot.com/home/keys) page.

### 2. Your Expected Parrot API key

Remote inference is a feature that allows you to run EDSL surveys and inference jobs at the Expected Parrot server, and store results at your account. It is activated by default when you create an account. When remote inference is activated, your survey jobs and results are automatically stored at the Expected Parrot server and accessible at the [Results](https://www.expectedparrot.com/home/remote-inference) page of your account.

You can also post any EDSL objects to Expected Parrot from your workspace, includes Results, Surveys, Agents, Scenarios, Notebooks and data files (using the FileStore module). To do this, you first need to store your Expected Parrot API key. If you have activated remote inference (at your [Settings](https://www.expectedparrot.com/home/settings) page), your key is automatically stored.

If you are only using EDSL locally, create a file named .env in your EDSL working directory and store your key in it using the following format:

```bash theme={null}
EXPECTED_PARROT_API_KEY = 'your_key_here'
```

Learn more about storing API keys in the [Managing Keys](/en/latest/api_keys) section of the documentation.

### 3. Post objects to Expected Parrot

Use the `Coop` module and methods to post and share objects (see examples below). You can set the visibility status of an object when you post it or update it later, from your workspace or interactively at your account. There are 3 status options:

* *public*: Visible to everyone
* *private*: Visible to logged in users that you have granted access
* *unlisted*: Visible to anyone with the link but not listed in search results (default)

See details on methods for uploading, downloading, updating and deleting content at your account below.

### 4. Explore content

Content that you have created with remote inference or posted to Expected Parrot is available at your [Content](https://www.expectedparrot.com/content) page.

You can access public content at the [Explore](https://www.expectedparrot.com/login) tab, and copy code and examples to modify or rerun at your workspace.

## Methods

When remote inference is activated the results of your surveys are automatically added to your [Content](https://www.expectedparrot.com/content) page. You can modify them from your workspace or interactively at your account.

You can also post, update, download and delete any objects using the `Coop` module and methods (see examples below).

### Posting

There are 2 methods for uploading/posting an object from your workspace to your account:

1. Call the *push()* method on the object directly.
2. Call the *create()* method on a `Coop` client object and pass it the object.

You can optionally pass a description, a convenient alias for the Expected Parrot URL and a *visibility* status (public, private or unlisted (default)) when you create an object. These can be changed at any time.

**Direct method**

Here we post a question object by calling the *push()* method on it:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
  question_name = "example",
  question_text = "How are you today?"
)

q.push(
  description="This is an example question", # optional
  alias = "my-example-question", # optional
  visibility="public" # optional
)
```

We can see the description, alias and visibility status that we specified in the information that is returned:

```json theme={null}
{'description': 'This is an example question',
'object_type': 'question',
'url': 'https://www.expectedparrot.com/content/1706e6f9-2675-4433-8b0a-080654a5cd08',
'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/my-example-question',
'uuid': '1706e6f9-2675-4433-8b0a-080654a5cd08',
'version': '0.1.47.dev1',
'visibility': 'public'}
```

**Using a `Coop` client**

Here we post the same question by passing it to the *create()* method of a `Coop` client object:

```python theme={null}
from edsl import Coop, QuestionFreeText

q = QuestionFreeText(
  question_name = "example",
  question_text = "How are you today?"
)
c = Coop()
c.create(q)
```

Here we include a description and visibility status:

```python theme={null}
from edsl import Coop, QuestionFreeText

q = QuestionFreeText(
  question_name = "example",
  question_text = "How are you today?"
)
c = Coop()
c.create(object=q, description="This is an example question", alias = "another-example-question", visibility="public")
```

This will return the same information about the object as the direct method shown above (with a unique uuid and URL for viewing the object at the Expected Parrot platform).

### Updating

There are 3 methods for updating/editing an object to Expected Parrot:

1. Edit the object at the Expected Parrot platform.
2. Call the *patch()* method on the object directly.
3. Call the *patch()* method on a `Coop` client object.

For each patch() method, pass the alias\_url or the uuid of the object and the parameter(s) that you want to update: *description*, *visibility*, *alias* and/or *value*.

* The *description* parameter is used to update the description of an object, such as a question or survey.
* The *visibility* parameter is used to update the visibility of an object: *public*, *private* or *unlisted*.
* The *value* parameter is used to update the content of an object, such as the text of a question or the code in a notebook.

**At the Expected Parrot platform**

You can manually update the description or visibility of an object at the Expected Parrot platform. Navigate to the [Explore](https://www.expectedparrot.com/login) page and open the page view for an object. You can select options to change the **visibility** of the object (*public*, *private* or *unlisted*) or to **edit** the description.

**Direct method**

Here we update the description and *visibility* of the question we created and uploaded in the examples above by calling the patch() method on it:

```python theme={null}
q.patch("https://www.expectedparrot.com/content/RobinHorton/my-example-question",
        description="This is an updated question",
        visibility="unlisted")
```

This will return a status message:

```json theme={null}
{'status': 'success'}
```

Here we change the question itself by modifying the value parameter:

```python theme={null}
from edsl import QuestionMultipleChoice

new_q = QuestionMultipleChoice(
  question_name = "example",
  question_text = "How are you today?",
  question_options = ["Good", "Bad", "OK"]
)
q.patch("https://www.expectedparrot.com/content/RobinHorton/my-example-question",
        value=new_q)
```

**Using a `Coop` client**

Here we do the same using a `Coop` client object:

```python theme={null}
from edsl import Coop

c = Coop()
c.patch("https://www.expectedparrot.com/content/1706e6f9-2675-4433-8b0a-080654a5cd08",
        description="This is an updated question",
        visibility="public")
```

This will return the same status message as above.

### Replicating / Downloading

There are a variety of methods for replicating or downloading an object at Expected Parrot:

1. Selecting options to download or copy code at the Expected Parrot platform
2. Calling the *pull()* method on the class of the object
3. Calling the *get()* method on a `Coop` client object

**Copy code at the Expected Parrot platform**

The platform provides copyable code for downloading or reconstructing an object that has been posted:

<Steps>
  <Step>
    Navigate to the [Explore](https://www.expectedparrot.com/login) page and select an object
  </Step>

  <Step>
    Go to the object’s page
  </Step>

  <Step>
    Select the option to **Download** the object
  </Step>
</Steps>

OR

Select the option to **Pull** the object using its uuid or **Raw** to get the code for constructing the object.

Use this code in your workspace to download the object locally or to reconstruct it.

**Class method**

Here we download the question posted above by calling the pull() method on the object class (Question) and passing the uuid of the object:

```python theme={null}
from edsl import Question

q = Question.pull("1706e6f9-2675-4433-8b0a-080654a5cd08")
q
```

This will return the object (the multiple choice question question that replaced the free text question):

| key | value |
| :- | :- |
| question\_name | example |
| question\_text | How are you today? |
| question\_options:0 | Good |
| question\_options:1 | Bad |
| question\_options:2 | OK |
| question\_type | multiple\_choice |

**Using a `Coop` client**

Here we download the question by calling the get() method on a `Coop` client object:

```python theme={null}
from edsl import Coop

c = Coop()
q = c.get("1706e6f9-2675-4433-8b0a-080654a5cd08")
q
```

This will return the same object as above.

### Deleting

There are 3 methods for deleting an object from your account:

1. Selecting options to delete at the web app
2. Calling the *delete()* method on the class of the object
3. Calling the *delete()* method on a `Coop` client object

**At the Expected Parrot platform**

You can manually delete objects at the Expected Parrot platform:

<Steps>
  <Step>
    Navigate to [Explore](https://www.expectedparrot.com/login) and open an object’s page view (see image above for *Uploading* content)
  </Step>

  <Step>
    Select the option to **delete** the object:

    <Frame>
      <img alt="Expected Parrot page view" />
    </Frame>
  </Step>
</Steps>

**Directly**

Here we delete a question object by calling the *delete()* method on the class of the object (*Question*) and passing the uuid of the object:

```python theme={null}
from edsl import Question

Question.delete("1234abcd-abcd-1234-abcd-1234abcd1234") # mock uuid
```

This will return a status message:

```json theme={null}
{'status': 'success'}
```

**Using a `Coop` client**

Here we delete a question by calling the *delete()* method on a `Coop` client object, passing the uuid of the object:

```python theme={null}
from edsl import Coop

c = Coop()
c.delete(uuid="1234abcd-abcd-1234-abcd-1234abcd1234") # mock uuid
```

This will return the same status message as above (so long as the object was not already deleted).

### Projects

You can create a project at the Expected Parrot platform by selecting the option to **Create a project** at the top right of the page.

You can also create a project by calling the *create\_project()* method on a `Coop` client object:

```python theme={null}
from edsl import Coop

c = Coop()
c.create_project(
  project_name = "My project",
  description = "This is a project",
  visibility = "public"
)
```


# Estimating & Tracking Costs
Source: https://docs.expectedparrot.com/en/latest/costs

EDSL comes with built-in methods for estimating costs before running your survey jobs, and tracking actual costs after running them. This is useful for budgeting and understanding the costs of running surveys with different models and inference services, and determining which models are most efficient for your research goals.

<Note>
  **Note:**

  When running survey jobs your prompts and responses are cached automatically and can be retrieved again later at no cost. Learn more about remote inference caching in the remote-caching section of the docs.
</Note>

## Estimating costs

When you create a survey, you can estimate the cost of running it before actually running it. This is done by combining the survey with one or more models to create a Job object. The Job object contains all the information needed to run the survey, including the models and inference services used.

There are 2 methods of estimating costs (in USD and credits):

* Call the estimate\_job\_cost() method on a Job object (a survey combined with one or more models). This will return the total estimated cost in USD, the estimated input and output tokens, and estimated costs and tokens for each inference service and model used. It will also show the number of credits that will be placed on hold if the survey job is run remotely. Once the job has completed the credits are released and final calculated credits are deducted from your account.
* Call the remote\_inference\_cost() method on a `Coop` client object and pass it the job. This will return the total estimated cost in USD and credits.

*Credits are required to run surveys remotely. Learn more about using credits in the `Credits </en/latest/credits>`\_ section of the docs.*

For example:

```python theme={null}
from edsl import QuestionMultipleChoice, Survey, Model, ModelList

q = QuestionMultipleChoice(
    question_name = "color",
    question_text = "Which is your favorite primary color?",
    question_options = ["red", "blue", "yellow"]
)
survey = Survey(questions = [q])
models = ModelList([
    Model("gpt-4o", service_name = "openai"),
    Model("gemini-1.5-flash", service_name = "google")
])

job = survey.by(models)

job.estimate_job_cost()
```

This will return:

```json expandable theme={null}
{'estimated_total_cost_usd': 0.000697825,
'total_credits_hold': 0.09999999999999999,
'estimated_total_input_tokens': 134,
'estimated_total_output_tokens': 102,
'detailed_costs': [{'inference_service': 'openai',
'model': 'gpt-4o',
'token_type': 'input',
'price_per_million_tokens': 2.5,
'tokens': 67,
'cost_usd': 0.0001675,
'credits_hold': 0.02},
{'inference_service': 'openai',
'model': 'gpt-4o',
'token_type': 'output',
'price_per_million_tokens': 10.0,
'tokens': 51,
'cost_usd': 0.00051,
'credits_hold': 0.06},
{'inference_service': 'google',
'model': 'gemini-1.5-flash',
'token_type': 'input',
'price_per_million_tokens': 0.075,
'tokens': 67,
'cost_usd': 5.025e-06,
'credits_hold': 0.01},
{'inference_service': 'google',
'model': 'gemini-1.5-flash',
'token_type': 'output',
'price_per_million_tokens': 0.3,
'tokens': 51,
'cost_usd': 1.53e-05,
'credits_hold': 0.01}]}
```

Using the `Coop` client object instead:

```python theme={null}
from edsl import Coop

coop = Coop()
coop.remote_inference_cost(job)
```

This will return:

```json theme={null}
{'credits_hold': 0.1, 'usd': 0.001}
```

### Reasoning-model uncertainty

Reasoning-token usage depends on the workload and may not be included in a server point estimate. For reasoning models, EDSL labels the returned amount as a lower bound instead of presenting it as an exact prediction. Inspect `estimate_kind`, `warnings`, and the `assumptions.reasoning_tokens` object before using an estimate as a budget:

```bash theme={null}
ep jobs cost jobs.ep
```

```json theme={null}
{
  "usd": 0.01,
  "credits_hold": 1.0,
  "estimate_kind": "lower_bound",
  "lower_bound_usd": 0.01,
  "expected_usd": null,
  "upper_bound_usd": null,
  "warnings": ["The server point estimate does not specify a reasoning-token assumption..."]
}
```

`null` expected or upper values mean EDSL does not have enough information to produce those bounds; they do not mean zero additional cost. If the server supplies a richer range, EDSL preserves that range and its assumptions.

## Calculations

The above-mentioned methods use the following calculation for each question in a survey to estimate the total cost of the job:

1. **Estimate the input tokens.**
   * Compute the number of characters in the user\_prompt and system\_prompt, with any Agent and Scenario data piped in. (*Note:* Previous answers cannot be piped in because they are not available until the survey is run; they are left as Jinja-bracketed variables in the prompts for purposes of estimating tokens and costs.)
   * Apply a piping multiplier of 2 to the number of characters in the user prompt if it has an answer piped in from a previous question (i.e., if the question has Jinja braces). Otherwise, apply a multiplier of 1.
   * Convert the number of characters into the number of input tokens using a conversion factor of 4 characters per token, rounding down to the nearest whole number. (This approximation was [established by OpenAI](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them).)
2. **Estimate the output tokens.**
   * Apply a multiplier of 0.75 to the number of input tokens, rounding up to the nearest whole number.
3. **Apply the Expected Parrot rates for the model and inference service.**
   * Find the model and inference service on the Expected Parrot [Pricing](https://www.expectedparrot.com/getting-started/coop-pricing) page. The rates listed there are used to calculate costs for jobs run using an Expected Parrot API key:
     *Total cost = (input tokens \* input token rate) + (output tokens \* output token rate)*
   * Expected Parrot rates may differ from rates charged by the underlying service provider and do not necessarily reflect provider-specific discounts, promotions, time-based pricing, caching discounts, batch pricing, volume pricing, service tiers, or other specialized pricing mechanisms.
   * If a current price for the model is not available, a default price for the inference service provider is used. If both the model and the inference service provider are not found, the following fallback token rates are applied (you will also see a warning message that a model price was not found):
     * USD 1.00 per 1M input tokens
     * USD 1.00 per 1M ouput tokens
4. **Convert the total cost in USD to credits.**
   * Total cost in credits = total cost in USD \* 100, rounded up to the nearest 1/100th credit.

Then sum the costs for all question prompts to get the total cost of the job. A notebook example is available [here](https://www.expectedparrot.com/content/RobinHorton/estimating-job-costs-notebook).

## Tracking costs

After running a survey job, you can track the token costs calculated for each question in the raw\_model\_response columns of the Results generated for your survey:

* **raw\_model\_response.\<question\_name>\_cost**: The cost of the result for the relevant question, applying the token quanities & prices.
* **raw\_model\_response.\<question\_name>\_input\_price\_per\_million\_tokenss**: The price per million input tokens for the relevant question for the relevant model.
* **raw\_model\_response.\<question\_name>\_input\_tokens**: The number of input tokens for the relevant question for the relevant model.
* **raw\_model\_response.\<question\_name>\_one\_usd\_buys**: The number of identical results for the relevant question that 1USD would cover.
* **raw\_model\_response.\<question\_name>\_output\_price\_per\_million\_tokens**: The price per million output tokens for the relevant question for the relevant model.
* **raw\_model\_response.\<question\_name>\_output\_tokens**: The number of output tokens for the relevant question for the relevant model.
* **raw\_model\_response.\<question\_name>\_raw\_model\_response**: The raw model response for the relevant question.

Details can also be viewed at the [Jobs](https://www.expectedparrot.com/home/remote-inference) and [Transactions](https://www.expectedparrot.com/home/transactions) pages of your Expected Parrot account.

For example, here we inspect the costs of running the job from above:

```python theme={null}
results = job.run()

results.select("raw_model_response.*")
```

Output:

| model.model | question\_text.read\_question\_text | question\_text.important\_question\_text | raw\_model\_response.read\_input\_price\_per\_million\_tokens | raw\_model\_response.important\_output\_tokens | raw\_model\_response.important\_output\_price\_per\_million\_tokens | raw\_model\_response.read\_one\_usd\_buys | raw\_model\_response.important\_raw\_model\_response | raw\_model\_response.important\_input\_price\_per\_million\_tokens | raw\_model\_response.read\_input\_tokens | raw\_model\_response.read\_raw\_model\_response | raw\_model\_response.important\_one\_usd\_buys | raw\_model\_response.read\_output\_price\_per\_million\_tokens | raw\_model\_response.read\_cost | raw\_model\_response.read\_output\_tokens | raw\_model\_response.important\_input\_tokens | raw\_model\_response.important\_cost |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| gemini-1.5-flash | Have you read any books about `{{ scenario.topic }}` ? | On a scale from 1 to 5, how important to you is `{{ scenario.topic }}` ? | 0.075000 | 53 | 0.300000 | 42872.461058 | `{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “5nnIt’s, like, a huge deal! The future of the planet is at stake, and that affects everything - from the environment to the economy to, you know, my future. It’s definitely something I worry about.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.2145003372768186, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 128, ‘candidates_token_count’: 53, ‘total_token_count’: 181, ‘cached_content_token_count’: 0}, ‘model_version’: ‘gemini-1.5-flash’}` | 0.075000 | 95 | `{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “YesnnI’ve read a few articles and some chapters from textbooks for my environmental science classes, which covered climate change extensively. It’s not quite the same as reading a whole book dedicated to the topic, but I’ve definitely learned about it.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.15844399840743453, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 95, ‘candidates_token_count’: 54, ‘total_token_count’: 149, ‘cached_content_token_count’: 0}, ‘model_version’: ‘gemini-1.5-flash’}` | 39215.691903 | 0.300000 | 0.000023 | 54 | 128 | 0.000025 |
| gpt-4o | Have you read any books about `{{ scenario.topic }}` ? | On a scale from 1 to 5, how important to you is `{{ scenario.topic }}` ? | 2.500000 | 32 | 10.000000 | 1724.137931 | `{‘id’: ‘chatcmpl-BQaCQLYP5PB3vEeEOElownyIV7jLX’, ‘choices’: [{‘finish_reason’: ‘stop’, ‘index’: 0, ‘logprobs’: None, ‘message’: {‘content’: “5 nClimate change is a critical issue that affects the entire planet and future generations, so I believe it’s very important to address and find solutions for it.”, ‘refusal’: None, ‘role’: ‘assistant’, ‘audio’: None, ‘function_call’: None, ‘tool_calls’: None, ‘annotations’: []}}` ], ‘created’: 1745675378, ‘model’: ‘gpt-4o-2024-08-06’, ‘object’: ‘chat.completion’, ‘service\_tier’: ‘default’, ‘system\_fingerprint’: ‘fp\_f5bdcc3276’, ‘usage’: `{‘completion_tokens’: 32, ‘prompt_tokens’: 131, ‘total_tokens’: 163, ‘completion_tokens_details’: {‘accepted_prediction_tokens’: 0, ‘audio_tokens’: 0, ‘reasoning_tokens’: 0, ‘rejected_prediction_tokens’: 0}, ‘prompt_tokens_details’: {‘audio_tokens’: 0, ‘cached_tokens’: 0}}` }\` | 2.500000 | 96 | `{‘id’: ‘chatcmpl-BQaCTYk259rsV3vcpoA2vi6XP7yhd’, ‘choices’: [{‘finish_reason’: ‘stop’, ‘index’: 0, ‘logprobs’: None, ‘message’: {‘content’: “Yes nI’ve read a few books on climate change as part of my studies to better understand the environmental challenges we face and what actions can be taken to mitigate them.”, ‘refusal’: None, ‘role’: ‘assistant’, ‘audio’: None, ‘function_call’: None, ‘tool_calls’: None, ‘annotations’: []}}` ], ‘created’: 1745675381, ‘model’: ‘gpt-4o-2024-08-06’, ‘object’: ‘chat.completion’, ‘service\_tier’: ‘default’, ‘system\_fingerprint’: ‘fp\_f5bdcc3276’, ‘usage’: `{‘completion_tokens’: 34, ‘prompt_tokens’: 96, ‘total_tokens’: 130, ‘completion_tokens_details’: {‘accepted_prediction_tokens’: 0, ‘audio_tokens’: 0, ‘reasoning_tokens’: 0, ‘rejected_prediction_tokens’: 0}, ‘prompt_tokens_details’: {‘audio_tokens’: 0, ‘cached_tokens’: 0}}}` | 1544.401544 | 10.000000 | 0.000580 | 34 | 131 | 0.000647 |
| gemini-1.5-flash | Have you read any books about `{{ scenario.topic }}` ? | On a scale from 1 to 5, how important to you is `{{ scenario.topic }}` ? | 0.075000 | 48 | 0.300000 | 52287.589235 | `{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “1nnHouse prices are something I think about, but it’s not something that’s keeping me up at night. It’s more of a long-term consideration than something I’m actively focused on right now.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.22673827409744263, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 128, ‘candidates_token_count’: 48, ‘total_token_count’: 176, ‘cached_content_token_count’: 0}, ‘model_version’: ‘gemini-1.5-flash’}` | 0.075000 | 95 | `{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “NonnI’m a student, so I haven’t had much time to read books outside of my coursework. House prices aren’t really something I’ve focused on yet.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.12296264171600342, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 95, ‘candidates_token_count’: 40, ‘total_token_count’: 135, ‘cached_content_token_count’: 0}, ‘model_version’: ‘gemini-1.5-flash’}` | 41666.672500 | 0.300000 | 0.000019 | 40 | 128 | 0.000024 |
| gpt-4o | Have you read any books about `{{ scenario.topic }}` ? | On a scale from 1 to 5, how important to you is `{{ scenario.topic }}` ? | 2.500000 | 32 | 10.000000 | 2127.659574 | `{‘id’: ‘chatcmpl-BQaCUaNZYyLh3T6gtpHnV8YinocJv’, ‘choices’: [{‘finish_reason’: ‘stop’, ‘index’: 0, ‘logprobs’: None, ‘message’: {‘content’: “3 nAs a student, I’m not in the market to buy a house right now, but I am interested in understanding the housing market for future planning.”, ‘refusal’: None, ‘role’: ‘assistant’, ‘audio’: None, ‘function_call’: None, ‘tool_calls’: None, ‘annotations’: []}}` ], ‘created’: 1745675382, ‘model’: ‘gpt-4o-2024-08-06’, ‘object’: ‘chat.completion’, ‘service\_tier’: ‘default’, ‘system\_fingerprint’: ‘fp\_f5bdcc3276’, ‘usage’: `{‘completion_tokens’: 32, ‘prompt_tokens’: 131, ‘total_tokens’: 163, ‘completion_tokens_details’: {‘accepted_prediction_tokens’: 0, ‘audio_tokens’: 0, ‘reasoning_tokens’: 0, ‘rejected_prediction_tokens’: 0}, ‘prompt_tokens_details’: {‘audio_tokens’: 0, ‘cached_tokens’: 0}}` }\` | 2.500000 | 96 | `{‘id’: ‘chatcmpl-BQaCSxNYx8KL3iJGbA3ARSAoUaxAC’, ‘choices’: [{‘finish_reason’: ‘stop’, ‘index’: 0, ‘logprobs’: None, ‘message’: {‘content’: “No nI haven’t read any books specifically about house prices, but I’ve come across articles and discussions about them.”, ‘refusal’: None, ‘role’: ‘assistant’, ‘audio’: None, ‘function_call’: None, ‘tool_calls’: None, ‘annotations’: []}}` ], ‘created’: 1745675380, ‘model’: ‘gpt-4o-2024-08-06’, ‘object’: ‘chat.completion’, ‘service\_tier’: ‘default’, ‘system\_fingerprint’: ‘fp\_f5bdcc3276’, ‘usage’: `{‘completion_tokens’: 23, ‘prompt_tokens’: 96, ‘total_tokens’: 119, ‘completion_tokens_details’: {‘accepted_prediction_tokens’: 0, ‘audio_tokens’: 0, ‘reasoning_tokens’: 0, ‘rejected_prediction_tokens’: 0}, ‘prompt_tokens_details’: {‘audio_tokens’: 0, ‘cached_tokens’: 0}}` }\` | 1544.401544 | 10.000000 | 0.000470 | 23 | 131 | 0.000647 |
| gemini-1.5-flash | Have you read any books about `{{ scenario.topic }}` ? | On a scale from 1 to 5, how important to you is `{{ scenario.topic }}` ? | 0.075000 | 55 | 0.300000 | 29304.034247 | `{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “5nnIt’s absolutely crucial. As someone with a global platform, I see firsthand the devastating effects of climate change – from extreme weather events impacting communities to the threats to biodiversity. We need urgent action, and I’m committed to doing my part.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.25197906494140626, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 128, ‘candidates_token_count’: 55, ‘total_token_count’: 183, ‘cached_content_token_count’: 0}, ‘model_version’: ‘gemini-1.5-flash’}` | 0.075000 | 95 | `{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “YesnnOh honey, I’ve read *so many* books about climate change. It’s a topic I’m incredibly passionate about, and I try to stay informed. Between interviews and red carpets, I always have a stack of books on my nightstand, and lately, a lot of them have been focused on environmental issues. It’s crucial to be aware of what’s happening to our planet.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.29773031870524086, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 95, ‘candidates_token_count’: 90, ‘total_token_count’: 185, ‘cached_content_token_count’: 0}, ‘model_version’: ‘gemini-1.5-flash’}` | 38314.181794 | 0.300000 | 0.000034 | 90 | 128 | 0.000026 |
| gpt-4o | Have you read any books about `{{ scenario.topic }}` ? | On a scale from 1 to 5, how important to you is `{{ scenario.topic }}` ? | 2.500000 | 39 | 10.000000 | 1503.759398 | `{‘id’: ‘chatcmpl-BQaCSmgqiBdOgYN7hW2qux0PmZvSe’, ‘choices’: [{‘finish_reason’: ‘stop’, ‘index’: 0, ‘logprobs’: None, ‘message’: {‘content’: “5 nClimate change is one of the most pressing issues of our time, and as a public figure, I believe it’s crucial to use my platform to raise awareness and advocate for meaningful action.”, ‘refusal’: None, ‘role’: ‘assistant’, ‘audio’: None, ‘function_call’: None, ‘tool_calls’: None, ‘annotations’: []}}` ], ‘created’: 1745675380, ‘model’: ‘gpt-4o-2024-08-06’, ‘object’: ‘chat.completion’, ‘service\_tier’: ‘default’, ‘system\_fingerprint’: ‘fp\_d8864f8b6b’, ‘usage’: `{‘completion_tokens’: 39, ‘prompt_tokens’: 133, ‘total_tokens’: 172, ‘completion_tokens_details’: {‘accepted_prediction_tokens’: 0, ‘audio_tokens’: 0, ‘reasoning_tokens’: 0, ‘rejected_prediction_tokens’: 0}, ‘prompt_tokens_details’: {‘audio_tokens’: 0, ‘cached_tokens’: 0}}` }\` | 2.500000 | 98 | `{‘id’: ‘chatcmpl-BQaCQav47AtdUvfbBlWZWtGOzMwXY’, ‘choices’: [{‘finish_reason’: ‘stop’, ‘index’: 0, ‘logprobs’: None, ‘message’: {‘content’: “Yes nAs someone in the public eye, I try to stay informed about important issues like climate change, so I’ve read a few books on the subject to better understand its impact and what can be done.”, ‘refusal’: None, ‘role’: ‘assistant’, ‘audio’: None, ‘function_call’: None, ‘tool_calls’: None, ‘annotations’: []}}` ], ‘created’: 1745675378, ‘model’: ‘gpt-4o-2024-08-06’, ‘object’: ‘chat.completion’, ‘service\_tier’: ‘default’, ‘system\_fingerprint’: ‘fp\_d8864f8b6b’, ‘usage’: `{‘completion_tokens’: 42, ‘prompt_tokens’: 98, ‘total_tokens’: 140, ‘completion_tokens_details’: {‘accepted_prediction_tokens’: 0, ‘audio_tokens’: 0, ‘reasoning_tokens’: 0, ‘rejected_prediction_tokens’: 0}, ‘prompt_tokens_details’: {‘audio_tokens’: 0, ‘cached_tokens’: 0}}` }\` | 1384.083045 | 10.000000 | 0.000665 | 42 | 133 | 0.000723 |
| gemini-1.5-flash | Have you read any books about `{{ scenario.topic }}` ? | On a scale from 1 to 5, how important to you is `{{ scenario.topic }}` ? | 0.075000 | 59 | 0.300000 | 35180.304746 | `{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “3nnHonestly, it’s something I think about, but it’s not my biggest concern. I mean, a nice place to live is great, but my career and family take precedence. Plus, I have people who handle that kind of thing for me, thankfully!n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.2841725430246127, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 128, ‘candidates_token_count’: 59, ‘total_token_count’: 187, ‘cached_content_token_count’: 0}, ‘model_version’: ‘gemini-1.5-flash’}` | 0.075000 | 95 | `{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “YesnnI mean, honestly, who *hasn’t* been obsessed with the housing market lately? It’s practically a national pastime at this point! I’ve skimmed a few, mostly for research for a role, but let’s be real, the real estate market is its own kind of wild, unpredictable drama.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.33602206807740975, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 95, ‘candidates_token_count’: 71, ‘total_token_count’: 166, ‘cached_content_token_count’: 0}, ‘model_version’: ‘gemini-1.5-flash’}` | 36630.042024 | 0.300000 | 0.000028 | 71 | 128 | 0.000027 |
| gpt-4o | Have you read any books about `{{ scenario.topic }}` ? | On a scale from 1 to 5, how important to you is `{{ scenario.topic }}` ? | 2.500000 | 31 | 10.000000 | 1904.761905 | `{‘id’: ‘chatcmpl-BQaCRYMuhgJBDz0LtE0dX5Nil89RL’, ‘choices’: [{‘finish_reason’: ‘stop’, ‘index’: 0, ‘logprobs’: None, ‘message’: {‘content’: “3 nAs a celebrity, I have an interest in real estate both as an investment and for personal living spaces, but it’s not my primary focus.”, ‘refusal’: None, ‘role’: ‘assistant’, ‘audio’: None, ‘function_call’: None, ‘tool_calls’: None, ‘annotations’: []}}` ], ‘created’: 1745675379, ‘model’: ‘gpt-4o-2024-08-06’, ‘object’: ‘chat.completion’, ‘service\_tier’: ‘default’, ‘system\_fingerprint’: ‘fp\_d8864f8b6b’, ‘usage’: `{‘completion_tokens’: 31, ‘prompt_tokens’: 133, ‘total_tokens’: 164, ‘completion_tokens_details’: {‘accepted_prediction_tokens’: 0, ‘audio_tokens’: 0, ‘reasoning_tokens’: 0, ‘rejected_prediction_tokens’: 0}, ‘prompt_tokens_details’: {‘audio_tokens’: 0, ‘cached_tokens’: 0}}` }\` | 2.500000 | 98 | `{‘id’: ‘chatcmpl-BQaCRamWhwJssHEfAOQCBR6UdQlcj’, ‘choices’: [{‘finish_reason’: ‘stop’, ‘index’: 0, ‘logprobs’: None, ‘message’: {‘content’: “No nI haven’t read any books specifically about house prices, but I’ve definitely heard a lot about the market through various conversations and media.”, ‘refusal’: None, ‘role’: ‘assistant’, ‘audio’: None, ‘function_call’: None, ‘tool_calls’: None, ‘annotations’: []}}` ], ‘created’: 1745675379, ‘model’: ‘gpt-4o-2024-08-06’, ‘object’: ‘chat.completion’, ‘service\_tier’: ‘default’, ‘system\_fingerprint’: ‘fp\_d8864f8b6b’, ‘usage’: `{‘completion_tokens’: 28, ‘prompt_tokens’: 98, ‘total_tokens’: 126, ‘completion_tokens_details’: {‘accepted_prediction_tokens’: 0, ‘audio_tokens’: 0, ‘reasoning_tokens’: 0, ‘rejected_prediction_tokens’: 0}, ‘prompt_tokens_details’: {‘audio_tokens’: 0, ‘cached_tokens’: 0}}` }\` | 1556.420233 | 10.000000 | 0.000525 | 28 | 133 | 0.000642 |

We can see the details at Expected Parrot as well. Your [Jobs page](https://www.expectedparrot.com/home/remote-inference) will show the actual costs of each survey job.

Your [Transactions page](https://www.expectedparrot.com/home/transactions) will show the additional information about credits on hold based on cost estimates together with actual costs of each survey job.

For more on credits, please see the [Credits](/en/latest/credits) section of the docs.


# Credits
Source: https://docs.expectedparrot.com/en/latest/credits

Credits are required in order to run surveys with models using your Expected Parrot API key. They can be purchased at your account (see details below). Credits are deducted based on the applicable rates published on the Expected Parrot model pricing page. These are the rates Expected Parrot uses to calculate credit consumption and may differ from rates charged by the underlying service provider.

Credits must be purchased in advance and are consumed when surveys are run. If your balance is insufficient, you will be prompted to purchase more credits. Details on credit consumption are provided below.

<Note>
  *Note:* Credits are *not* required to run surveys with your own keys from service providers or to post and share content at [Expected Parrot](https://www.expectedparrot.com/login). When using your own keys, cost estimates may use the prices listed on the model pricing page, but your actual charges are determined by the applicable service provider and may differ from those estimates.
</Note>

## Free credits

Your Expected Parrot account comes with free credits that you can use to run surveys with your API key.

**Are you using EDSL for a research project?**

Send an email to [*info@expectedparrot.com*](mailto:info@expectedparrot.com) to request additional free credits!

## Purchasing credits

To purchase credits, navigate to the [Credits](https://www.expectedparrot.com/home/credits) page of your account and enter the number of credits that you would like to purchase (1 USD buys 100 credits; the minimum purchase amount is 1 USD):

<Frame>
  <img alt="Purchase credits" />
</Frame>

*Note:* Payments are processed by Stripe.

## Using credits

When you run a survey with your Expected Parrot API key, the number of credits consumed (and deducted from your balance) is displayed at the [Jobs](https://www.expectedparrot.com/home/remote-inference) page of your account. This number is equal to the sum of the cost in credits of each response in the results.

The cost in credits of a response is calculated as follows:

* The number of input tokens is multiplied by the applicable input token rate published on the Expected Parrot model pricing page.
* The number of output tokens is multiplied by the applicable output token rate published on the Expected Parrot model pricing page.
* The total cost in USD is converted to credits (1 USD = 100 credits).
* The total cost in credits is rounded up to the nearest 1/100th of a credit.

(If you use your own keys from service providers, no credits are consumed.)

### Example calculation

* Input tokens: 16
* Output tokens: 45
* Input token rate: USD 2.50 per 1M tokens
* Output token rate: USD 10.00 per 1M tokens
* Total cost: (16 \* USD 2.50/1,000,000) + (45 \* USD 10.00/1,000,000) = USD 0.00049
* Total credits: 0.05 credits

## Response details & token rates

Details about a model’s response are stored in the raw\_model\_response fields of the results dataset. For each question that was run, the following columns will appear in results:

* **raw\_model\_response.\<question\_name>\_cost**: The cost in USD calculated for the API call using the applicable Expected Parrot pricing rates. (In the example above, this is USD 0.00049.)
* **raw\_model\_response.\<question\_name>\_one\_usd\_buys**: The number of tokens that can be purchased with 1 USD (for reference).
* **raw\_model\_response.\<question\_name>\_raw\_model\_response**: A dictionary containing the raw response for the question, which includes the input text and tokens, output text and tokens, and other information about the API call. This dictionary is specific to the language model service provider and may contain additional information about the response.

For example, here we run a question with two models and inspect the raw model response information (note that the raw response formats are not identical):

```python theme={null}
from edsl import QuestionFreeText, ModelList, Model

q = QuestionFreeText(
    question_name = "rainbow",
    question_text = "What are the colors of a rainbow?"
)

m = ModelList(Model(m) for m in ["gemini-1.5-flash", "gpt-4o"])

results = q.by(m).run()

results.select("model", "raw_model_response.*")
```

Output:

| model.model | raw\_model\_response.rainbow\_cost | raw\_model\_response.rainbow\_raw\_model\_response | raw\_model\_response.rainbow\_one\_usd\_buys |
| :- | :- | :- | :- |
| gemini-1.5-flash | 0.000018 | `{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “The colors of a rainbow are typically listed as red, orange, yellow, green, blue, indigo, and violet. However, it’s important to note that these colors blend seamlessly into each other, and the number of distinct colors perceived can vary from person to person.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.099734950483891, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 8, ‘candidates_token_count’: 57, ‘total_token_count’: 65, ‘cached_content_token_count’: 0}, ‘model_version’: ‘gemini-1.5-flash’}` | 56497.186153 |
| gpt-4o | 0.000438 | `{‘id’: ‘chatcmpl-B2OaTCPGFdNY7dju27SxmrLfSWXSE’, ‘choices’: [{‘finish_reason’: ‘stop’, ‘index’: 0, ‘logprobs’: None, ‘message’: {‘content’: ‘The colors of a rainbow, in order, are red, orange, yellow, green, blue, indigo, and violet. These colors can be remembered using the acronym ROYGBIV.’, ‘refusal’: None, ‘role’: ‘assistant’, ‘audio’: None, ‘function_call’: None, ‘tool_calls’: None}}], ‘created’: 1739910869, ‘model’: ‘gpt-4o-2024-08-06’, ‘object’: ‘chat.completion’, ‘service_tier’: ‘default’, ‘system_fingerprint’: ‘fp_523b9b6e5f’, ‘usage’: {‘completion_tokens’: 40, ‘prompt_tokens’: 15, ‘total_tokens’: 55, ‘completion_tokens_details’: {‘accepted_prediction_tokens’: 0, ‘audio_tokens’: 0, ‘reasoning_tokens’: 0, ‘rejected_prediction_tokens’: 0}, ‘prompt_tokens_details’: {‘audio_tokens’: 0, ‘cached_tokens’: 0}}}` | 2285.714286 |

In the raw model response information for the response from *gpt-4o*, we can see values for completion\_tokens (output tokens) and prompt\_tokens (input tokens):

```python theme={null}
'completion_tokens': 40,
'prompt_tokens': 16
```

The total cost of the response is calculated using the applicable rates listed on the Expected Parrot model pricing page (at the time of writing, USD 2.50 per 1M tokens for input and USD 10.00 per 1M tokens for output):

```python theme={null}
(16 * USD 2.50/1,000,000) + (40 * USD 10.00/1,000,000)
= USD 0.00044
= 0.05 credits (rounded up to the nearest 1/100th credit)
```

In the raw model response information for the response from *gemini-1.5-flash*, we can see values for prompt\_token\_count and candidates\_token\_count (input tokens and output tokens, respectively):

```python theme={null}
'prompt_token_count': 8,
'candidates_token_count': 57
```

The total cost of the response is calculated using the applicable rates listed on the Expected Parrot model pricing page (at the time of writing, USD 0.08 per 1M tokens for input and USD 0.30 per 1M tokens for output):

```python theme={null}
(8 * USD 0.08/1,000,000) + (57 * USD 0.30/1,000,000)
= USD 0.00001774
= 0.01 credits (rounded up to the nearest 1/100th credit)
```

This translates to a total of 0.06 credits consumed for the survey. We can see this number of credits consumed at the remote inference page of our Expected Parrot account:

<Frame>
  <img alt="Expected Parrot remote inference jobs page" />
</Frame>

We can also navigate to the results page and select the same columns of the results to display:

<Frame>
  <img alt="Expected Parrot remote inference jobs page" />
</Frame>

## Token rates

Model token rates used to calculate costs can be viewed on the [model pricing page](https://www.expectedparrot.com/getting-started/coop-pricing). For jobs run using an Expected Parrot API key, the rates published on that page are the rates Expected Parrot uses to calculate credit consumption.

Expected Parrot reviews pricing information from service providers regularly, but our published rates do not necessarily reproduce every provider-specific pricing mechanism, including discounts, promotions, time-based pricing, caching discounts, batch pricing, volume pricing, service tiers, or similar adjustments. Published Expected Parrot rates may therefore differ from the rates a service provider would charge for the same request made directly or using your own API key.

Provider pricing may change without notice, and Expected Parrot rates may not be updated at the same time. If a current price is unavailable, a default or fallback price may be used as described on the model pricing page. If you believe a published price is missing or out of date, please submit a report using the form on that page.

## Estimating job costs

Before running a survey, you can estimate the tokens and costs (in USD and credits) in 2 different ways:

* Call the estimate\_job\_cost() method on the Job object (a survey combined with one or more models).

This will return the total estimated cost in USD, the total estimated input and output tokens, and estimated costs and tokens for each inference service and model used.

* Call the remote\_inference\_cost() method on a `Coop` client object and pass it the job.

This will return the estimated cost in credits and USD. (Credits are required to run surveys remotely.)

### Example

Here we create a survey and agent, select a model and combine them to create a job. Then we call the above-mentioned methods for estimating costs and show the underlying calculations. The steps below can also be accessed as a notebook at the Expected Parrot platform ([notebook view](https://www.expectedparrot.com/content/c379241a-7039-4505-8d42-4c909a54c6e0)).

```python expandable theme={null}
from edsl import QuestionFreeText, Survey, Agent, Model

q0 = QuestionFreeText(
    question_name = "favorite_flower",
    question_text = "What is the name of your favorite flower?"
)
q1 = QuestionFreeText(
    question_name = "flower_color",
    question_text = "What color is {{ favorite_flower.answer }}?"
)

survey = Survey(questions = [q0, q1])

a = Agent(traits = {"persona":"You are a botanist on Cape Cod."})

m = Model("gpt-4o")

job = survey.by(a).by(m)

estimated_job_cost = job.estimate_job_cost()
estimated_job_cost
```

Output:

```json theme={null}
{'estimated_total_cost': 0.0009175000000000001,
 'estimated_total_input_tokens': 91,
 'estimated_total_output_tokens': 69,
 'model_costs': [{'inference_service': 'openai',
   'model': 'gpt-4o',
   'estimated_cost': 0.0009175000000000001,
   'estimated_input_tokens': 91,
   'estimated_output_tokens': 69}]}
```

The estimated\_total\_cost is the total cost in USD to run the job, and the estimated\_total\_input\_tokens and estimated\_total\_output\_tokens are the estimated total input and output tokens, respectively for all the prompts in the survey.

To get the estimated cost in credits to run the job remotely we can call the remote\_inference\_cost() method on a `Coop` client object and pass it the job:

```python theme={null}
from edsl import Coop

coop = Coop()

estimated_remote_inference_cost = coop.remote_inference_cost(job) # using the job object from above
estimated_remote_inference_cost
```

Output:

```json theme={null}
{'credits': 0.1, 'usd': 0.00092}
```

### Formula details

Total job costs are estimated by performing the following calculation for each set of question prompts in the survey and summing the results:

1. Estimate the input tokens.
   * Compute the number of characters in the user\_prompt and system\_prompt, with any Agent and Scenario data piped in. (*Note:* Previous answers cannot be piped in because they are not available until the survey is run; they are left as Jinja-bracketed variables in the prompts for purposes of estimating tokens and costs.)
   * Apply a piping multiplier of 2 to the number of characters in the user prompt if it has an answer piped in from a previous question (i.e., if the question has Jinja braces that cannot be filled in before the survey is run). Otherwise, apply a multiplier of 1.
   * Convert the number of characters into the number of input tokens using a conversion factor of 4 characters per token, rounding down to the nearest whole number. (This approximation was [established by OpenAI](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them).)
2. Estimate the output tokens.
   * Apply a multiplier of 0.75 to the number of input tokens, rounding up to the nearest whole number.
3. Apply the token rates for the model and inference service.
   * Find the model and inference service for the question in the [model pricing page](https://www.expectedparrot.com/getting-started/coop-pricing):
     Total cost in USD = (input tokens \* input token rate) + (output tokens \* output token rate)
   * If a model and inference service are not found, use the following fallback token rates (you will see a warning message that actual model rates were not found):
     * USD 1.00 per 1M input tokens
     * USD 1.00 per 1M ouput tokens
4. Convert the total cost in USD to credits.
   * Total cost in credits = total cost in USD \* 100, rounded up to the nearest 1/100th credit.

Then sum the costs for all question prompts to get the total cost of the job.

### Calculations

Here we show the calculations for the examples above.

We can call the show\_prompts() method on the job object to see the prompts for each question in the survey:

```python theme={null}
job.show_prompts()
```

Output:

| user\_prompt | system\_prompt |
| :- | :- |
| What is the name of your favorite flower? | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a botanist on Cape Cod.’}` |
| What color is `{{ answer }}`? | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a botanist on Cape Cod.’}` |

Here we count the characters in each user prompt and system prompt:

```python theme={null}
q0_user_prompt_characters = len("What is the name of your favorite flower?")
q0_user_prompt_characters
```

Output:

```
41
```

```python theme={null}
q0_system_prompt_characters = len("You are answering questions as if you were a human. Do not break character. Your traits: {'persona': 'You are a botanist on Cape Cod.'}")
q0_system_prompt_characters
```

Output:

```
135
```

We apply the piping multiplier of 2 to the number of characters in the user prompt for q1 because the answer to q0 is piped in:

```python theme={null}
q1_user_prompt_characters = len("What color is {{ answer }}?") * 2
q1_user_prompt_characters
```

Output:

```
54
```

The system prompt characters are identical for the single agent used with the questions:

```python theme={null}
q1_system_prompt_characters = len("You are answering questions as if you were a human. Do not break character. Your traits: {'persona': 'You are a botanist on Cape Cod.'}")
q1_system_prompt_characters
```

Output:

```
135
```

Here we estimate the input and output tokens for each set of prompts:

```python theme={null}
q0_input_tokens = (q0_user_prompt_characters + q0_system_prompt_characters) // 4
q0_input_tokens
```

Output:

```
44
```

```python theme={null}
q0_output_tokens = ceil(0.75 * q0_input_tokens)
q0_output_tokens
```

Output:

```
33
```

```python theme={null}
q1_input_tokens = (q1_user_prompt_characters + q1_system_prompt_characters) // 4
q1_input_tokens
```

Output:

```
47
```

```python theme={null}
q1_output_tokens = ceil(0.75 * q1_input_tokens)
q1_output_tokens
```

Output:

```
36
```

The total input tokens and output tokens for the job are:

```python theme={null}
total_input_tokens = q0_input_tokens + q1_input_tokens
total_input_tokens
```

Output:

```
91
```

```python theme={null}
total_output_tokens = q0_output_tokens + q1_output_tokens
total_output_tokens
```

Output:

```
69
```

Next we apply the token rates for the model:

```python theme={null}
q0_tokens_cost = (2.50/1000000 * q0_input_tokens) + (10.00/1000000 * q0_output_tokens)
q0_tokens_cost
```

Output:

```
0.00044000000000000007
```

```python theme={null}
q1_tokens_cost = (2.50/1000000 * q1_input_tokens) + (10.00/1000000 * q1_output_tokens)
q1_tokens_cost
```

Output:

```
0.00047750000000000006
```

The total cost of the job is:

```python theme={null}
total_cost_usd = q0_tokens_cost + q1_tokens_cost
total_cost_usd
```

Output:

```
0.0009175000000000001
```

We convert the total cost in USD to credits:

```python theme={null}
q0_credits = ceil(q0_tokens_cost * 100 * 100) / 100
q0_credits
```

Output:

```
0.05
```

```python theme={null}
q1_credits = ceil(q1_tokens_cost * 100 * 100) / 100
q1_credits
```

Output:

```
0.05
```

We calculate the total cost in credits:

```python theme={null}
total_credits = q0_credits + q1_credits
total_credits
```

Output:

```
0.1
```

The total cost of the job is 0.00092 USD and 0.1 credits.

## Refunds

Please send an email to [*info@expectedparrot.com*](mailto:info@expectedparrot.com) if you have any questions about credits or refunds, or need assistance with your account.


# Custom CSS
Source: https://docs.expectedparrot.com/en/latest/custom_css

CSS classes and HTML structure for styling human-facing surveys with custom CSS.

Every question component supported by humanize has `edsl-*` CSS classes so you can customise the survey appearance.

## Applying custom CSS

### When creating a human survey

Pass your CSS string in the `humanize_schema` under `survey.custom_css`.

```python theme={null}
from edsl import Survey, QuestionFreeText, QuestionMultipleChoice

survey = Survey([
    QuestionMultipleChoice(
        question_name="rating",
        question_text="Rate your experience.",
        question_options=["Good", "OK", "Bad"],
    ),
    QuestionFreeText(
        question_name="feedback",
        question_text="Any additional feedback you'd like to share?",
    ),
])

custom_css = """
.edsl-survey-container { background-color: #f9f9f9; }
.edsl-question-text { font-weight: bold; color: #333; }
"""

humanize_schema = {
    "questions": {
        "rating": {"format": {"type": "dropdown"}},
        "feedback": {"optional": True},
    },
    "survey": {"custom_css": custom_css},
}

survey.humanize(humanize_schema=humanize_schema)
```

Pass `"survey": {"custom_css": None}` (or omit `custom_css`) to create the survey without any custom CSS.

### Updating CSS on an existing human survey

Use `Coop.patch_human_survey_css` to set or clear the CSS on a human survey that already exists.

```python theme={null}
from edsl import Coop

coop = Coop()

# Set or replace CSS
coop.patch_human_survey_css(
    human_survey_uuid="your-human-survey-uuid",
    css=".edsl-survey-container { background-color: #fff; }",
)

# Clear existing CSS
coop.patch_human_survey_css(
    human_survey_uuid="your-human-survey-uuid",
    css=None,
)
```

## Survey-level classes

| Class | Element | Notes |
| - | - | - |
| `edsl-survey-container` | Outermost container of the entire survey | Use to set background color for the whole survey. Carries `data-group-name` when the survey is presented by question group; see below |
| `edsl-survey-banner` | `<div>` holding the survey's logo, at the top of the survey | Only present when the survey has a logo. Give it a `background` for a banner; see below |
| `edsl-survey-banner-content` | Column inside `edsl-survey-banner` holding the logo | Lined up with `edsl-survey-content` |
| `edsl-logo` | The logo `<img>` inside `edsl-survey-banner` | 40px tall by default, width following the image. Resize with `height`; see below |
| `edsl-survey-content` | Column holding the page's questions and `edsl-survey-nav` | Questions are spaced by its `gap` |
| `edsl-survey-item` | `<div>` wrapping each question or instruction on the page | Carries `data-question-name` or `data-instruction-name`; see below |
| `edsl-instruction` | `<div>` holding an instruction's text | Inside `edsl-survey-item` |
| `edsl-survey-nav` | Row `<div>` holding the page's navigation buttons | Below the questions |
| `edsl-next-button` | `<button>` that submits the page | Disabled while the page submits; style that with `.edsl-next-button:disabled` |

## Targeting one question or page

Each question and instruction on a page is wrapped in `edsl-survey-item`, which carries
the item's name: `data-question-name` for a question, `data-instruction-name` for an
instruction. When the survey is presented by question group, `edsl-survey-container`
carries `data-group-name`, the group shown on the page.

```css theme={null}
/* One question */
.edsl-survey-item[data-question-name="age"] .edsl-question-text {
  font-weight: 700;
}

/* Every page of one group */
.edsl-survey-container[data-group-name="demographics"] {
  background: #f8fafc;
}
```

## Survey banner and logo

A survey with a logo shows it in `edsl-survey-banner` at the top of the survey, positioned
left, center or right as the author chose. Surveys without a logo have no banner, so rules
for these classes simply do nothing there.

Set the logo itself in the humanize schema under `survey.branding.logo`, referencing an
image you uploaded with `Coop().upload_human_survey_asset(...)`. See
[Humanize schema](/en/latest/humanize_schema).

The logo is 40px tall by default, and its width follows the image's proportions. To resize
it, set its **height** and let the width follow:

```css theme={null}
/* A larger logo */
.edsl-logo {
  height: 80px;
}
```

To size it by width instead, set `width` together with `height: auto`. Setting both to
fixed values fits the logo inside that box without stretching it.

For a colored strip behind the logo, give the banner a background:

```css theme={null}
/* Logo on a colored banner */
.edsl-survey-banner {
  background: #2e7d32;
}
```

The banner spans the full width of the screen, and the logo inside it lines up with the
questions. To place the logo somewhere else, such as near the edge of the screen, style
`edsl-survey-banner-content`:

```css theme={null}
/* Logo near the edge of the screen */
.edsl-survey-banner-content {
  width: auto;
  padding: 0 24px;
}
```

## Progress indicator

The progress indicator sits above the question. Depending on how the survey is
configured, it shows as a bar or as a row of step markers. A survey can also hide it, in
which case nothing is rendered and there is nothing to style.

Both renderings put `edsl-progress` on their outer element. Use that class for rules that
should apply whichever one a survey uses.

### Bar

```html theme={null}
<div class="edsl-progress edsl-progress-bar">
  <div class="edsl-progress-track" role="progressbar" aria-valuenow="40">
    <div class="edsl-progress-fill"></div>
  </div>
  <!-- shown only when a label is configured -->
  <span class="edsl-progress-label">40%</span>
</div>
```

| Class | Element | Notes |
| - | - | - |
| `edsl-progress` | Outer element of either rendering | Present on both the bar and the steps |
| `edsl-progress-bar` | Outer `<div>` of the bar rendering | |
| `edsl-progress-track` | Unfilled groove | Set its `height`, `border-radius`, `background` |
| `edsl-progress-fill` | Filled portion | Color it, but leave its width alone. See below |
| `edsl-progress-label` | Percentage text under the bar | Absent when the label is turned off |

**Recolor the fill, don't resize it.** The fill is always as wide as the whole track, and
it slides sideways to reveal only the finished portion. Setting `width` on it will not
shorten it. Set `background`, `border-radius`, or `transition` instead.

### Steps

Each step is one `<li>` holding a marker, half of the connecting line on either side of
it, and an optional label below.

```html theme={null}
<ol class="edsl-progress edsl-progress-steps">
  <li class="edsl-progress-step" data-status="complete">
    <div>
      <!-- first step's incoming connector is hidden, not removed -->
      <span class="edsl-progress-connector edsl-progress-connector-incoming" data-status="upcoming"></span>
      <span class="edsl-progress-marker">1</span>
      <span class="edsl-progress-connector edsl-progress-connector-outgoing" data-status="complete"></span>
    </div>
    <!-- shown only for steps that have a label -->
    <span class="edsl-progress-label">Background</span>
  </li>

  <li class="edsl-progress-step" data-status="current">
    <div>
      <!-- filled, because the step before it is complete -->
      <span class="edsl-progress-connector edsl-progress-connector-incoming" data-status="complete"></span>
      <span class="edsl-progress-marker">2</span>
      <span class="edsl-progress-connector edsl-progress-connector-outgoing" data-status="upcoming"></span>
    </div>
    <span class="edsl-progress-label">Experience</span>
  </li>

  <li class="edsl-progress-step" data-status="upcoming">
    <div>
      <span class="edsl-progress-connector edsl-progress-connector-incoming" data-status="upcoming"></span>
      <span class="edsl-progress-marker">3</span>
      <!-- last step's outgoing connector is hidden, not removed -->
      <span class="edsl-progress-connector edsl-progress-connector-outgoing" data-status="upcoming"></span>
    </div>
    <span class="edsl-progress-label">Wrap-up</span>
  </li>
</ol>
```

| Class | Element | Notes |
| - | - | - |
| `edsl-progress-steps` | The `<ol>` holding the steps | |
| `edsl-progress-step` | One `<li>` per step | Equal-width column. Carries `data-status`: `complete`, `current` or `upcoming` |
| `edsl-progress-marker` | The circle for a step | Contains the step number, or is empty when markers are dots |
| `edsl-progress-connector` | Half of the line between two markers | Two per step, one on each side of the marker. Carries `data-status`: `complete` when filled, `upcoming` when not |
| `edsl-progress-connector-incoming` | The half arriving at the marker | On the left of the marker |
| `edsl-progress-connector-outgoing` | The half leaving the marker | On the right of the marker |
| `edsl-progress-label` | Text under a marker | Absent for steps without a label |

**Style markers and labels through their step.** Every `<li>` carries its step's
`data-status`, which is how you give the current marker a different look from the
finished ones.

**Style the line through its own status.** Each half carries its own `data-status`,
which does not match the status of the step it belongs to. The half arriving at the
current step is filled as soon as the step before it is done, so the current step
normally holds one filled half and one empty one. Select the filled line with
`.edsl-progress-connector[data-status="complete"]`. Reaching for
`.edsl-progress-step[data-status="complete"] .edsl-progress-connector` instead leaves
that first half gray, and the line stops short of where the respondent actually is.

```css theme={null}
.edsl-progress-step[data-status="current"] .edsl-progress-marker { border-color: #b91c1c; color: #b91c1c; }
.edsl-progress-step[data-status="complete"] .edsl-progress-marker { background: #b91c1c; border-color: #b91c1c; }
.edsl-progress-connector[data-status="complete"] { background: #b91c1c; }
```

**To reach a single segment, combine a step's status with a connector's.** Include the
connector's position class as well, since a step's two halves are otherwise identical. The rule below
picks out the filled half arriving at the step the respondent is on, and gives it a
gradient so the line brightens as it reaches them:

```css theme={null}
.edsl-progress-step[data-status="current"]
  .edsl-progress-connector-incoming[data-status="complete"] {
  background: linear-gradient(to right, #b91c1c, #f87171);
}
```

Finally, markers come in two shapes: numbered circles and small dots. The survey's
configuration picks the shape, not your CSS. A rule that assumes a size, such as a fixed
`width`, will look wrong under the other shape, so check the survey renders with the
shape you expect.

## Common question classes

These classes are present on every question regardless of type.

| Class | Element | Notes |
| - | - | - |
| `edsl-question` | Outermost `<div>` of every question | Always present |
| `edsl-question-text` | Question prompt `<div>` | Rendered from `question_text` |
| `edsl-question-image` | `<img>` embedded in a question | Image blocks in question text |
| `edsl-error` | Error message `<p>` | Shown only when the answer is invalid |

### Options

Questions answered by picking from a list share these classes.

| Class | Element | Notes |
| - | - | - |
| `edsl-options` | Container wrapping all of a question's options | |
| `edsl-option` | `<label>` for one option | Wraps the control and its text, so a click anywhere in the row answers |
| `edsl-option-label` | `<span>` next to the radio or checkbox | |
| `edsl-option-content` | `<span>` wrapping an option's rendered label | Also used for budget labels, rank items, and matrix rows and columns |
| `edsl-option-text` | Text `<span>` inside `edsl-option-content` | |
| `edsl-option-image` | `<img>` inside `edsl-option-content` | When an option references an image |
| `edsl-option-media-unavailable` | `<span>` shown in place of an option's file | When the file isn't an image or can't be found |
| `edsl-select` | `<select>` | When the humanize format is `dropdown`. Its options are plain text |

```html theme={null}
<label class="edsl-option">
  <!-- the radio or checkbox; see below -->
  <span class="edsl-radio-control" data-unchecked>…</span>
  <span class="edsl-option-label">
    <span class="edsl-option-content">
      <span class="edsl-option-text">Golden retriever</span>
      <!-- only when the option references an image -->
      <img class="edsl-option-image" src="…" alt="dog.jpg" />
    </span>
  </span>
</label>
```

### Radios and checkboxes

Each radio and checkbox is drawn by a `<span>`, with the `<input>` inside it hidden from
view. Style the span, not the input.

```html theme={null}
<span class="edsl-radio-control" data-checked>
  <input class="edsl-radio" type="radio" checked />
  <span class="edsl-radio-indicator" data-checked></span>
</span>

<span class="edsl-checkbox-control" data-unchecked>
  <input class="edsl-checkbox" type="checkbox" />
  <span class="edsl-checkbox-indicator" data-unchecked><svg>…</svg></span>
</span>
```

| Class | Element | Notes |
| - | - | - |
| `edsl-radio-control` | `<span>` drawn in the radio's place: the circle | Carries `data-checked` while selected and `data-unchecked` while not |
| `edsl-radio` | The `<input type="radio">` | Hidden, so styling it has no visible effect |
| `edsl-radio-indicator` | `<span>` for the dot | Carries `data-checked` or `data-unchecked`. Hidden while unselected |
| `edsl-checkbox-control` | `<span>` drawn in the checkbox's place: the box | Carries `data-checked` while selected and `data-unchecked` while not |
| `edsl-checkbox` | The `<input type="checkbox">` | Hidden, so styling it has no visible effect |
| `edsl-checkbox-indicator` | `<span>` around the tick `<svg>` | Carries `data-checked` or `data-unchecked`. The tick is drawn in `currentColor` |

```css theme={null}
/* Green controls */
.edsl-radio-control[data-checked],
.edsl-checkbox-control[data-checked] {
  border-color: #2e7d32;
}
.edsl-checkbox-control[data-checked] {
  background: #2e7d32;
}
.edsl-radio-indicator {
  background: #2e7d32;
}
```

To style the row of a selected option, use `.edsl-option:has(.edsl-radio:checked)`.

## Question-specific HTML

### Budget

```html theme={null}
<div class="edsl-question edsl-budget-question">
  <div class="edsl-question-text">Allocate $100 across categories.</div>
  <div class="edsl-budget-options">
    <div class="edsl-budget-item">
      <div>
        <label class="edsl-budget-label">
          <span class="edsl-option-content">
            <span class="edsl-option-text">Food</span>
          </span>
        </label>
        <input class="edsl-budget-input" type="text" />
      </div>
    </div>
    <div class="edsl-budget-item">
      <div>
        <label class="edsl-budget-label">
          <span class="edsl-option-content">
            <span class="edsl-option-text">Transport</span>
          </span>
        </label>
        <input class="edsl-budget-input" type="text" />
      </div>
    </div>
    <div class="edsl-budget-total">
      <span>Total</span>
      <span>75</span>
    </div>
  </div>
  <p class="edsl-error">…</p>
</div>
```

### Checkbox

```html theme={null}
<div class="edsl-question edsl-checkbox-question">
  <div class="edsl-question-text">Select all that apply.</div>

  <div class="edsl-options">
    <label class="edsl-option">
      <span class="edsl-checkbox-control" data-checked>
        <input class="edsl-checkbox" type="checkbox" checked />
        <span class="edsl-checkbox-indicator" data-checked><svg>…</svg></span>
      </span>
      <span class="edsl-option-label">
        <span class="edsl-option-content">
          <span class="edsl-option-text">Option A</span>
        </span>
      </span>
    </label>
    <label class="edsl-option">
      <span class="edsl-checkbox-control" data-unchecked>
        <input class="edsl-checkbox" type="checkbox" />
        <span class="edsl-checkbox-indicator" data-unchecked><svg>…</svg></span>
      </span>
      <span class="edsl-option-label">
        <span class="edsl-option-content">
          <span class="edsl-option-text">Option B</span>
        </span>
      </span>
    </label>
  </div>

  <!-- shown unless the humanize schema turns it off -->
  <label class="edsl-select-all">
    <span class="edsl-checkbox-control" data-unchecked>
      <input class="edsl-checkbox" type="checkbox" />
      <span class="edsl-checkbox-indicator" data-unchecked><svg>…</svg></span>
    </span>
    <span class="edsl-option-label">Select all</span>
  </label>

  <p class="edsl-error">…</p>
</div>
```

### Checkbox with other

```html theme={null}
<div class="edsl-question edsl-checkbox-with-other-question">
  <div class="edsl-question-text">Which tools do you use?</div>
  <div class="edsl-options">
    <label class="edsl-option">
      <span class="edsl-checkbox-control" data-unchecked>
        <input class="edsl-checkbox" type="checkbox" />
        <span class="edsl-checkbox-indicator" data-unchecked><svg>…</svg></span>
      </span>
      <span class="edsl-option-label">
        <span class="edsl-option-content">
          <span class="edsl-option-text">Spreadsheets</span>
        </span>
      </span>
    </label>
    <!-- … -->

    <div class="edsl-other-option">
      <label class="edsl-option">
        <span class="edsl-checkbox-control" data-checked>
          <input class="edsl-checkbox" type="checkbox" checked />
          <span class="edsl-checkbox-indicator" data-checked><svg>…</svg></span>
        </span>
        <span class="edsl-option-label">Other</span>
      </label>
      <div>
        <p class="edsl-other-hint">Please enter one item per box.</p>
        <ul>
          <!-- one per item entered -->
          <li>
            <input class="edsl-other-input" type="text" placeholder="Enter an item" />
            <!-- hidden while there is only one box -->
            <button class="edsl-other-remove" type="button"><svg>…</svg></button>
          </li>
        </ul>
        <!-- shown once an item has been entered -->
        <div>
          <button class="edsl-other-add" type="button"><svg>…</svg> Add another</button>
        </div>
      </div>
    </div>
  </div>
  <p class="edsl-error">…</p>
</div>
```

### EDSL object

```html theme={null}
<div class="edsl-question edsl-object-question">
  <div class="edsl-question-text">Select the agent list to use.</div>
  <div>
    <button class="edsl-object-select-btn" type="button">Select an Agent List</button>
  </div>

  <!-- shown after an object is selected -->
  <div class="edsl-object-selected">
    <span>Selected Agent List</span>
    <p>My agents - Q1 2025</p>
    <span>UUID: a1b2c3d4-…</span>
    <div>
      <button type="button">View</button>
      <button type="button">Clear</button>
    </div>
  </div>

  <p class="edsl-error">…</p>
</div>
```

### File upload

```html theme={null}
<div class="edsl-question edsl-file-upload-question">
  <div class="edsl-question-text">Upload supporting documents.</div>

  <!-- carries data-dragging while a drag is in progress -->
  <div class="edsl-file-upload-dropzone" data-dragging>

    <!-- clickable / drag-and-drop target -->
    <div role="button" class="edsl-file-upload-target">
      <input type="file" multiple class="sr-only" />
      <div class="edsl-file-upload-icon-wrapper">
        <!-- upload icon -->
        <svg class="edsl-file-upload-icon">…</svg>
      </div>
      <p class="edsl-file-upload-title">Upload files</p>
      <p class="edsl-file-upload-subtitle">Drag & drop or click to browse</p>
      <!-- shown when file-type / size constraints are set -->
      <p class="edsl-file-upload-hints">All file types · Max 10 files · Up to 1 GB per file</p>
    </div>

    <!-- shown only when there are file-level errors -->
    <div class="edsl-file-upload-errors">
      <!-- error messages -->
      <button type="button"><!-- dismiss --></button>
    </div>

    <!-- shown only when files have been added -->
    <div class="edsl-file-list">
      <div class="edsl-file-list-header">
        <span>Files (2)</span>
        <button type="button" class="edsl-file-remove-all">Remove all</button>
      </div>

      <!-- card layout (small screens) -->
      <div class="edsl-file-cards">
        <!-- data-status: uploading, done or error -->
        <div class="edsl-file-item" data-status="uploading">
          <div class="edsl-file-item-icon-wrapper">
            <svg class="edsl-file-item-icon">…</svg>
          </div>
          <div>
            <p class="edsl-file-item-name">doc1.pdf</p>
            <div>
              <p class="edsl-file-item-size">24 KB</p>
              <span class="edsl-file-item-status">
                <!-- while uploading: -->
                <span class="edsl-file-status-uploading">
                  <div class="edsl-file-progress-track">
                    <div class="edsl-file-progress-fill" style="width: 60%"></div>
                  </div>
                  Uploading…
                </span>
                <!-- on success: -->
                <span class="edsl-file-status-done">Done</span>
                <!-- on failure: -->
                <span class="edsl-file-status-error">Failed</span>
              </span>
            </div>
          </div>
          <button type="button" class="edsl-file-item-remove"><!-- trash icon --></button>
        </div>
      </div>

      <!-- table layout (medium+ screens) -->
      <div class="edsl-file-table-wrapper">
        <table class="edsl-file-table">
          <thead class="edsl-file-table-header">
            <tr class="edsl-file-table-header-row">
              <th class="edsl-file-table-header-cell">Name</th>
              <th class="edsl-file-table-header-cell">Size</th>
              <th class="edsl-file-table-header-cell">Upload Status</th>
              <th class="edsl-file-table-header-cell">Actions</th>
            </tr>
          </thead>
          <tbody>
            <!-- data-status: uploading, done or error -->
            <tr class="edsl-file-table-row" data-status="uploading">
              <td>
                <svg class="edsl-file-item-icon">…</svg>
                <span>doc1.pdf</span>
              </td>
              <td>24 KB</td>
              <td>
                <!-- one of: -->
                <span class="edsl-file-status-uploading">
                  <div class="edsl-file-progress-track">
                    <div class="edsl-file-progress-fill" style="width: 60%"></div>
                  </div>
                </span>
                <span class="edsl-file-status-done">Done</span>
                <span class="edsl-file-status-error">Failed</span>
              </td>
              <td>
                <button type="button" class="edsl-file-item-remove"><!-- trash icon --></button>
              </td>
            </tr>
          </tbody>
        </table>
      </div>
    </div>

  </div>
  <p class="edsl-error">…</p>
</div>
```

### Free text

```html theme={null}
<div class="edsl-question edsl-free-text-question">
  <div class="edsl-question-text">Tell us about yourself.</div>
  <textarea class="edsl-textarea" rows="4"></textarea>
  <p class="edsl-error">…</p>
</div>
```

### Linear scale

<Tabs>
  <Tab title="Radio">
    When humanize format is `radio` (default):

    ```html theme={null}
    <div class="edsl-question edsl-linear-scale-question">
      <div class="edsl-question-text">Rate from 1 to 5.</div>
      <div class="edsl-options">
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="1" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">1 - Very poor</span>
            </span>
          </span>
        </label>
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="2" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">2</span>
            </span>
          </span>
        </label>
        <!-- … -->
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="5" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">5 - Excellent</span>
            </span>
          </span>
        </label>
      </div>
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>

  <Tab title="Dropdown">
    When humanize format is `dropdown`:

    ```html theme={null}
    <div class="edsl-question edsl-linear-scale-question">
      <div class="edsl-question-text">Rate from 1 to 5.</div>
      <div class="edsl-options">
        <select class="edsl-select" name="q1">
          <option value="">Select...</option>
          <option value="1">1 - Very poor</option>
          <option value="2">2</option>
          <option value="3">3</option>
          <option value="4">4</option>
          <option value="5">5 - Excellent</option>
        </select>
      </div>
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>
</Tabs>

### Likert five

<Tabs>
  <Tab title="Radio">
    When humanize format is `radio` (default):

    ```html theme={null}
    <div class="edsl-question edsl-likert-question">
      <div class="edsl-question-text">How satisfied are you?</div>
      <div class="edsl-options">
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="Strongly disagree" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">Strongly disagree</span>
            </span>
          </span>
        </label>
        <!-- … -->
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="Strongly agree" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">Strongly agree</span>
            </span>
          </span>
        </label>
      </div>
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>

  <Tab title="Dropdown">
    When humanize format is `dropdown`:

    ```html theme={null}
    <div class="edsl-question edsl-likert-question">
      <div class="edsl-question-text">How satisfied are you?</div>
      <div class="edsl-options">
        <select class="edsl-select" name="q1">
          <option value="">Select...</option>
          <option value="Strongly disagree">Strongly disagree</option>
          <option value="Disagree">Disagree</option>
          <option value="Neutral">Neutral</option>
          <option value="Agree">Agree</option>
          <option value="Strongly agree">Strongly agree</option>
        </select>
      </div>
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>
</Tabs>

### List

```html theme={null}
<div class="edsl-question edsl-list-question">
  <div class="edsl-question-text">List your top hobbies.</div>
  <p>Please enter one item per line.</p>
  <textarea class="edsl-textarea" rows="4"></textarea>
  <p class="edsl-error">…</p>
</div>
```

### Matrix

<Tabs>
  <Tab title="Table">
    When humanize format is `table` (default):

    ```html theme={null}
    <div class="edsl-question edsl-matrix-question">
      <div class="edsl-question-text">Rate each item.</div>
      <div class="edsl-matrix-views">

        <!-- shown below 768px -->
        <div class="edsl-matrix-stack-view">
          <div class="edsl-matrix-stack">
            <div class="edsl-matrix-stack-row">
              <div class="edsl-matrix-stack-item">
                <span class="edsl-option-content">
                  <span class="edsl-option-text">Price</span>
                </span>
              </div>
              <div class="edsl-options" role="radiogroup">
                <label class="edsl-option">
                  <span class="edsl-radio-control" data-unchecked>
                    <input class="edsl-radio" type="radio" value="1" />
                    <span class="edsl-radio-indicator" data-unchecked></span>
                  </span>
                  <span class="edsl-option-label">
                    <span class="edsl-option-content">
                      <span class="edsl-option-text">1 - Poor</span>
                    </span>
                  </span>
                </label>
                <!-- … -->
              </div>
            </div>
            <div class="edsl-matrix-stack-row"><!-- … --></div>
          </div>
        </div>

        <!-- shown from 768px up -->
        <div class="edsl-matrix-table-view">
          <div>
            <table class="edsl-matrix-table">
              <colgroup>
                <col class="edsl-matrix-item-column" />
                <!-- one per option -->
                <col class="edsl-matrix-option-column" />
                <col class="edsl-matrix-option-column" />
                <col class="edsl-matrix-option-column" />
              </colgroup>
              <thead>
                <tr class="edsl-matrix-header-row">
                  <th><!-- row label column --></th>
                  <th>
                    <div>
                      <!-- shown only when the option has a label -->
                      <span>Poor</span>
                      <span>
                        <span class="edsl-option-content">
                          <span class="edsl-option-text">1</span>
                        </span>
                      </span>
                    </div>
                  </th>
                  <!-- … -->
                </tr>
              </thead>
              <tbody>
                <tr class="edsl-matrix-row">
                  <td class="edsl-matrix-label-cell">
                    <span class="edsl-option-content">
                      <span class="edsl-option-text">Price</span>
                    </span>
                  </td>
                  <td class="edsl-matrix-cell">
                    <label>
                      <span>
                        <span class="edsl-radio-control" data-unchecked>
                          <input class="edsl-radio" type="radio" value="1" />
                          <span class="edsl-radio-indicator" data-unchecked></span>
                        </span>
                      </span>
                    </label>
                  </td>
                  <!-- … -->
                </tr>
                <tr class="edsl-matrix-row"><!-- … --></tr>
              </tbody>
            </table>
          </div>
        </div>

      </div>
      <p class="edsl-error">…</p>
    </div>
    ```

    Both views are always in the page, and CSS picks which one shows. To keep the table at
    every width:

    ```css theme={null}
    .edsl-matrix-stack-view { display: none; }
    .edsl-matrix-table-view { display: block; }
    ```

    Row labels take 25% of the table's width by default, and the option columns share the
    rest. To change that, set `--edsl-matrix-item-column` to a percentage, on
    `.edsl-matrix-table` or on `.edsl-survey-container`:

    ```css theme={null}
    .edsl-matrix-table { --edsl-matrix-item-column: 40%; }
    ```
  </Tab>

  <Tab title="Carousel">
    When humanize format is `carousel`, rows are shown one at a time:

    ```html theme={null}
    <div class="edsl-question edsl-matrix-question">
      <div class="edsl-question-text">Rate each item.</div>
      <div class="edsl-matrix-carousel">
        <div>
          <div>
            <!-- one slide per row -->
            <div class="edsl-matrix-carousel-slide">
              <div class="edsl-matrix-carousel-item">
                <span class="edsl-option-content">
                  <span class="edsl-option-text">Price</span>
                </span>
              </div>
            </div>
            <div class="edsl-matrix-carousel-slide"><!-- … --></div>
          </div>
        </div>

        <div class="edsl-matrix-carousel-nav">
          <button class="edsl-matrix-carousel-button" type="button" aria-label="Previous item"><svg>…</svg></button>
          <span class="edsl-matrix-carousel-status">1 of 8</span>
          <button class="edsl-matrix-carousel-button" type="button" aria-label="Next item"><svg>…</svg></button>
        </div>

        <!-- the options for the row on screen -->
        <div class="edsl-options" role="radiogroup">
          <label class="edsl-option">
            <span class="edsl-radio-control" data-unchecked>
              <input class="edsl-radio" type="radio" value="1" />
              <span class="edsl-radio-indicator" data-unchecked></span>
            </span>
            <span class="edsl-option-label">
              <span class="edsl-option-content">
                <span class="edsl-option-text">1 - Poor</span>
              </span>
            </span>
          </label>
          <!-- … -->
        </div>
      </div>
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>
</Tabs>

### Multiple choice

<Tabs>
  <Tab title="Radio">
    When humanize format is `radio` (default):

    ```html theme={null}
    <div class="edsl-question edsl-multiple-choice-question">
      <div class="edsl-question-text">Which fruit do you prefer?</div>
      <div class="edsl-options">
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="Apple" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">Apple</span>
            </span>
          </span>
        </label>
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="Banana" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">Banana</span>
            </span>
          </span>
        </label>
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="Cherry" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">Cherry</span>
            </span>
          </span>
        </label>
      </div>
      <!-- shown only on error -->
      <p class="edsl-error"><svg>…</svg> Please select an option.</p>
    </div>
    ```
  </Tab>

  <Tab title="Dropdown">
    When humanize format is `dropdown`:

    ```html theme={null}
    <div class="edsl-question edsl-multiple-choice-question">
      <div class="edsl-question-text">Which fruit do you prefer?</div>
      <div class="edsl-options">
        <select class="edsl-select" name="q1">
          <option value="">Select...</option>
          <option value="Apple">Apple</option>
          <option value="Banana">Banana</option>
          <option value="Cherry">Cherry</option>
        </select>
      </div>
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>
</Tabs>

### Multiple choice with other

```html theme={null}
<div class="edsl-question edsl-multiple-choice-with-other-question">
  <div class="edsl-question-text">How did you hear about us?</div>
  <div class="edsl-options">
    <label class="edsl-option">
      <span class="edsl-radio-control" data-unchecked>
        <input class="edsl-radio" type="radio" value="Search engine" />
        <span class="edsl-radio-indicator" data-unchecked></span>
      </span>
      <span class="edsl-option-label">
        <span class="edsl-option-content">
          <span class="edsl-option-text">Search engine</span>
        </span>
      </span>
    </label>
    <label class="edsl-option">
      <span class="edsl-radio-control" data-unchecked>
        <input class="edsl-radio" type="radio" value="Friend" />
        <span class="edsl-radio-indicator" data-unchecked></span>
      </span>
      <span class="edsl-option-label">
        <span class="edsl-option-content">
          <span class="edsl-option-text">Friend</span>
        </span>
      </span>
    </label>

    <div class="edsl-other-option">
      <label class="edsl-option">
        <span class="edsl-radio-control" data-unchecked>
          <input class="edsl-radio" type="radio" value="Other" />
          <span class="edsl-radio-indicator" data-unchecked></span>
        </span>
        <span class="edsl-option-label">Other</span>
      </label>
      <div>
        <input class="edsl-other-input" type="text" />
      </div>
    </div>
  </div>
  <p class="edsl-error">…</p>
</div>
```

### Numerical

<Tabs>
  <Tab title="Input">
    When humanize format is `input` (default):

    ```html theme={null}
    <div class="edsl-question edsl-numerical-question">
      <div class="edsl-question-text">Enter your age.</div>
      <input class="edsl-input" type="number" />
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>

  <Tab title="Slider">
    When humanize format is `slider`:

    ```html theme={null}
    <div class="edsl-question edsl-numerical-question">
      <div class="edsl-question-text">On a scale of 0 to 100, how likely are you to recommend?</div>
      <div>
        <input class="edsl-input edsl-numerical-slider" type="range" min="0" max="100" step="1" value="50" />
        <div>
          <span>0</span>
          <span>50</span>
          <span>100</span>
        </div>
      </div>
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>
</Tabs>

### Rank

```html theme={null}
<div class="edsl-question edsl-rank-question">
  <div class="edsl-question-text">Rank these items from most to least preferred.</div>
  <div class="edsl-rank-list">
    <div class="edsl-rank-options">

      <div class="edsl-rank-item">
        <select class="edsl-rank-select">
          <option value="1">1</option>
          <option value="2">2</option>
          <option value="3">3</option>
          <!-- only when the question limits how many items are ranked -->
          <option value="not-ranked">Not ranked</option>
        </select>
        <span class="edsl-rank-item-label">
          <span class="edsl-option-content">
            <span class="edsl-option-text">Item A</span>
          </span>
        </span>
        <div class="edsl-rank-drag-handle">
          <!-- GripVertical icon (drag handle) -->
        </div>
      </div>

      <div class="edsl-rank-item">
        <select class="edsl-rank-select">…</select>
        <span class="edsl-rank-item-label">Item B</span>
        <div class="edsl-rank-drag-handle">…</div>
      </div>

      <div class="edsl-rank-item">
        <select class="edsl-rank-select">…</select>
        <span class="edsl-rank-item-label">Item C</span>
        <div class="edsl-rank-drag-handle">…</div>
      </div>

    </div>
  </div>
  <p class="edsl-error">…</p>
</div>
```

### Survey message

```html theme={null}
<div class="edsl-question edsl-survey-message-question">
  <div class="edsl-question-text">Thanks! The next few questions are about your work.</div>
</div>
```

### Top K

```html theme={null}
<div class="edsl-question edsl-top-k-question">
  <div class="edsl-question-text">Pick your top 2.</div>
  <div class="edsl-options">
    <label class="edsl-option">
      <span class="edsl-checkbox-control" data-unchecked>
        <input class="edsl-checkbox" type="checkbox" />
        <span class="edsl-checkbox-indicator" data-unchecked><svg>…</svg></span>
      </span>
      <span class="edsl-option-label">
        <span class="edsl-option-content">
          <span class="edsl-option-text">Choice A</span>
        </span>
      </span>
    </label>
    <label class="edsl-option">
      <span class="edsl-checkbox-control" data-unchecked>
        <input class="edsl-checkbox" type="checkbox" />
        <span class="edsl-checkbox-indicator" data-unchecked><svg>…</svg></span>
      </span>
      <span class="edsl-option-label">
        <span class="edsl-option-content">
          <span class="edsl-option-text">Choice B</span>
        </span>
      </span>
    </label>
    <label class="edsl-option">
      <span class="edsl-checkbox-control" data-unchecked>
        <input class="edsl-checkbox" type="checkbox" />
        <span class="edsl-checkbox-indicator" data-unchecked><svg>…</svg></span>
      </span>
      <span class="edsl-option-label">
        <span class="edsl-option-content">
          <span class="edsl-option-text">Choice C</span>
        </span>
      </span>
    </label>
  </div>
  <p class="edsl-error">…</p>
</div>
```

### Yes/no

<Tabs>
  <Tab title="Radio">
    When humanize format is `radio` (default):

    ```html theme={null}
    <div class="edsl-question edsl-yes-no-question">
      <div class="edsl-question-text">Do you agree?</div>
      <div class="edsl-options">
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="Yes" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">Yes</span>
            </span>
          </span>
        </label>
        <label class="edsl-option">
          <span class="edsl-radio-control" data-unchecked>
            <input class="edsl-radio" type="radio" value="No" />
            <span class="edsl-radio-indicator" data-unchecked></span>
          </span>
          <span class="edsl-option-label">
            <span class="edsl-option-content">
              <span class="edsl-option-text">No</span>
            </span>
          </span>
        </label>
      </div>
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>

  <Tab title="Dropdown">
    When humanize format is `dropdown`:

    ```html theme={null}
    <div class="edsl-question edsl-yes-no-question">
      <div class="edsl-question-text">Do you agree?</div>
      <div class="edsl-options">
        <select class="edsl-select" name="q1">
          <option value="">Select...</option>
          <option value="Yes">Yes</option>
          <option value="No">No</option>
        </select>
      </div>
      <p class="edsl-error">…</p>
    </div>
    ```
  </Tab>
</Tabs>

## Comment field

If a question has `comment` configured in the humanize schema, an additional comment input is rendered under that question.

```html theme={null}
<div class="edsl-comment-field">
  <label class="edsl-question-text">Why did you choose this answer?</label>
  <textarea class="edsl-textarea" rows="2"></textarea>
</div>
```


# Custom JavaScript
Source: https://docs.expectedparrot.com/en/latest/custom_javascript

Run your own JavaScript on a human survey's questions, and record what respondents do.

Custom JavaScript lets you attach a script to a question in a human survey. The script runs in the respondent's browser once the question is on the page, so you can build interactive tasks, measure timing and behaviour, and record events with the survey.

<Note>
  Custom JavaScript is available on approved accounts only. Write to [info@expectedparrot.com](mailto:info@expectedparrot.com) to request access. Until your account is approved, a humanize schema that includes a script is rejected when you save it, and no part of that schema is saved.
</Note>

## Checking access

```python theme={null}
from edsl import Coop

coop = Coop()
coop.get_custom_js_access()  # True once your account is approved
```

Access is per account: once yours is approved, the human surveys you create can include custom JavaScript.

## Adding a script

Scripts go in the humanize schema, under `questions[question_name].javascript.hooks`. A hook maps an event to the script that runs on it. The only event is `"question.ready"`, which fires once the question has been rendered and its inputs are on the page.

```python theme={null}
from edsl import Survey, QuestionMultipleChoice

survey = Survey([
    QuestionMultipleChoice(
        question_name="choice",
        question_text="Which option do you prefer?",
        question_options=["A", "B", "C"],
    ),
])

time_to_first_answer = """
const container = this.getQuestionContainer();
const shownAt = performance.now();

container.addEventListener("change", () => {
    ep.log("first_answer", { ms: Math.round(performance.now() - shownAt) });
}, { once: true });
"""

humanize_schema = {
    "questions": {
        "choice": {"javascript": {"hooks": {"question.ready": time_to_first_answer}}},
    },
}

survey.humanize(humanize_schema=humanize_schema)
```

To add or change a script on a survey that already exists, use `Coop.patch_human_survey_humanize_schema`:

```python theme={null}
coop.patch_human_survey_humanize_schema(
    "your-human-survey-uuid",
    {"questions": {"choice": {"javascript": {"hooks": {"question.ready": time_to_first_answer}}}}},
)
```

Set `"javascript": None` on a question to remove its script.

Most question types support `javascript`, including survey messages; see [Humanize schema](/en/latest/humanize_schema) for the list. `compute` and `image_generation` questions do not: they run in the background and never appear on the page.

## Writing a hook

A hook is the *body* of a function, not a function. Write the statements you want to run; don't wrap them in `function () { ... }`.

Inside a hook you have:

| Name | What it is |
| - | - |
| `this.getQuestionContainer()` | The question's element on the page. Find your question's inputs and add your own elements inside it. |
| `context.questionName` | The name of the question the hook is running for. |
| `context.questionType` | The question's type, for example `"multiple_choice"`. |
| `context.isPreview` | `true` when the survey was opened through a preview link, which does not save a response. |
| `ep.log(eventName, payload)` | Records an event with the survey. See below. |

`context` carries nothing that identifies the respondent.

Each question's hook runs once when that question appears. If the same script is attached to several questions, it runs once for each, and `context.questionName` tells them apart.

### Recording events with `ep.log`

`ep.log(eventName, payload)` records an event against the respondent's response. Use it for anything you want to analyse later: timings, clicks, which condition a respondent saw.

* `eventName` must be a non-empty string. Names longer than 200 characters are cut to 200.
* `payload` is optional and must be convertible to JSON: objects, arrays, strings, numbers, booleans and `null`. A payload that isn't (a page element, an object that refers to itself) is not recorded, and a warning appears in the browser console.
* A payload larger than 4,000 characters, measured as JSON, is not recorded.
* Each response records at most 10,000 events.

`ep.log` returns nothing. Events are sent in batches in the background, so don't wait on it.

### What a script may do

Scripts can build interface inside the survey, read and write answers, control when respondents can move on, measure behaviour, randomise conditions, read parameters from the page URL, load images, audio, video and other media to show, and use the camera, microphone or location for a task the respondent is told about.

A script can also load a well-known, widely used library, such as d3, Chart.js or lodash, from a public CDN like cdnjs, jsDelivr or unpkg, at a pinned version (for example `https://cdn.jsdelivr.net/npm/d3@7.9.0/dist/d3.min.js`, not `d3@latest`). Your own code must still be plain JavaScript written in the script itself.

A script can't be saved if it:

* Loads code from anywhere else: an unfamiliar or personal site, a paste site, a little-known or custom package, source hosted somewhere other than a named library, or a URL put together while the script runs.
* Builds code at runtime, for example by passing a constructed string to `eval`, `new Function`, `setTimeout` or `setInterval`, or by inserting markup that contains a script.
* Makes any request to Expected Parrot, or sends respondent data anywhere else. `ep.log` is how data leaves a script.
* Reads cookies, tokens, credentials, or storage it didn't create.
* Is obfuscated or minified so that what it does can't be read from the source. This applies to your script, not to a well-known library it loads.
* Asks for passwords or payment details, or imitates a login or other dialog.
* Collects anything unrelated to taking the survey, or uses the camera, microphone or location covertly.
* Keeps running after the respondent has finished or left the survey.
* Misleads or traps the respondent, for example by opening popups, preventing them from leaving, or hiding the survey's own notices.
* Uses the respondent's device for something other than the survey.

If a script can't be saved, no part of that humanize schema is saved, and the survey keeps what it had. If you think a script was refused by mistake, write to [info@expectedparrot.com](mailto:info@expectedparrot.com).

### Limits

* Each hook can be at most 64,000 characters, and all hooks on one question at most 256,000.
* One save can add at most 25 different new scripts. A script used unchanged on several questions counts once, and scripts the survey already has don't count. Save larger surveys in smaller batches.

## Retrieving events

Use `Coop.get_all_human_survey_events` to get every event a survey has recorded, oldest first:

```python theme={null}
from edsl import Coop

coop = Coop()

events = coop.get_all_human_survey_events("your-human-survey-uuid")
respondent_events = [e for e in events if not e["is_preview"]]
```

Events from preview links are included and marked `is_preview`, so leave them out of an analysis. Events that arrive while the download runs are included, and no event comes back twice.

To fetch new events later without downloading everything again, pass the `id` of the last event you downloaded as `after`. Take it from the full download, before filtering out previews:

```python theme={null}
last_id = events[-1]["id"] if events else None
new_events = coop.get_all_human_survey_events(
    "your-human-survey-uuid", after=last_id
)
```

<Note>
  Fetching only new events is useful for checking your scripts while a survey is collecting, but it can occasionally miss an event that was still being saved when you downloaded. For analysis, download the full log once fielding has ended, at least an hour after the last response.
</Note>

To fetch one batch at a time, use `Coop.get_human_survey_events`. It takes `after` and `limit` (at most 200, the default), and returns `{"events", "next_cursor", "has_more"}`. Pass `next_cursor` as `after` to get the next batch while `has_more` is `True`. `next_cursor` is returned even when there is nothing more, so you can keep it and use it later.

To see how many events a survey has without downloading them, use `Coop.count_human_survey_events`. The count is as of the call; on a survey still collecting, a download started afterwards can return more.

Each event includes:

| Field | Description |
| - | - |
| `id` | The event's unique id |
| `received_at` | When Expected Parrot received it (ISO 8601, UTC) |
| `client_ts` | When the respondent's browser logged it (ISO 8601, UTC) |
| `response_uuid` | The response the event belongs to |
| `is_preview` | Whether the event came from a preview link |
| `log_session_uuid` | The page load the event was logged in. Each time a respondent opens or reloads the survey starts a new session, so one response can have several. |
| `session_state` | Whether that session's log is whole. See below. |
| `client_seq` | The event's position in its session, counting from 0. Sort by it to get the order events were logged in. A gap means an event in between was not recorded. |
| `question_name` | The question whose script logged it |
| `event_type` | What kind of event it is. Always `"ep.log"` for now. |
| `event_name` | The `eventName` passed to `ep.log` |
| `hook_event` | The hook that was running when the event was logged, for example `"question.ready"` |
| `source` | What logged the event. Always `"author_script"` for now. |
| `payload` | The `payload` passed to `ep.log`, or `None` |

`session_state` is one of:

* `"complete"`: the session ended normally and no events are missing.
* `"truncated"`: some events from this session are known to be missing, for example because they broke the limits above.
* `"unknown"`: no events are known to be missing, but the session never signalled that it ended, for example because the respondent's browser closed or lost its connection. Its log may or may not be whole.

`session_state` describes the whole session, so every event from one session carries the same value.

## From the command line

The `ep` CLI covers the same steps. `--javascript` takes `QUESTION:HOOK=FILE`, naming the question, the hook, and a file holding the script:

```bash theme={null}
# Check access
ep humanize custom-js-access

# Attach a script while building a schema, or to a survey that already exists
ep humanize schema create --survey survey.ep --javascript choice:question.ready=choice.js --output humanize.json
ep humanize schema set <human-survey-uuid> --javascript choice:question.ready=choice.js

# Remove a question's script
ep humanize schema set <human-survey-uuid> --clear-javascript choice

# Get the first batch of events, or save every event to a file
ep humanize events <human-survey-uuid>
ep humanize events <human-survey-uuid> --all --output events.jsonl

# Later, save only the events since the last download
ep humanize events <human-survey-uuid> --all --after <next_cursor> --output new.jsonl

# Count events without downloading them
ep humanize events <human-survey-uuid> --count
```

`--all` requires `--output`, which can end in `.json` or `.jsonl`. Every result includes `next_cursor`, the value to pass as `--after` next time. If a download fails part way, no output file is written. As with the Python methods, use `--after` for checking a survey while it collects, and a full download once fielding has ended for analysis.


# Caching LLM Calls
Source: https://docs.expectedparrot.com/en/latest/data

The *Cache* class is used to store responses from a language model so that they can easily be retrieved, reused and shared.

*What is a cache?* ([Wikipedia](https://en.wikipedia.org/wiki/Cache_\(computing\)))

## Why caching?

Language model outputs are expensive to create, both in terms of time and money. As such, it is useful to store them in a cache so that they can be shared or reused later.

Use cases:

* Avoid rerunning questions when a job fails only partially, by only resending unanswered questions to a language model.
* Share your cache with others so they can rerun your questions at no cost.
* Use a common remote cache to avoid rerunning questions that others have already run.
* Build up training data to train or fine-tune a smaller model.
* Build up a public repository of questions and responses so others can learn from them.

## How it works

A Cache is a dictionary-like object that stores the inputs and outputs of a language model. Specifically, a cache has an attribute, data, that is dictionary-like.

The keys of a cache are hashes of the unique inputs to a language model, i.e., the unique combinations of prompts and any parameters used to generate the outputs. The values are CacheEntry objects, which contains the inputs and outputs.

A cache can be stored as either a Python in-memory dictionary or a dictionary connected to a SQLite3 database. The default constructor is an in-memory dictionary. If a SQLite3 database is used, a cache will persist automatically between sessions. You can also specify that a cache be used for a specific session, in which case it will not persist between sessions.

After a session, the cache will have new entries from any new jobs that have been run during the session. These can be written to a local SQLite3 database, a JSONL file, or a remote server.

## Multiple caches

Multiple caches are impacted when a survey is run:

* **New cache for results:** A new cache is automatically created for the results of a survey. This cache is specific to the results object, and is attached to it.
* **Default cache updated:** A default cache is automatically updated with any new entries from the results. If a new or existing cache object was passed to the run() method, it is the default cache that is updated. Otherwise, either your local cache or your remote cache is updated, depending on whether the job was run locally or remotely.

See examples below for more details on how to create and manage caches.

## Generating a new cache for results

A new cache is automatically generated whenever results are generated for a question or survey. This cache is specific to the Results object, and is attached to it. It can be accessed using the cache attribute of the results object.

For example:

```python theme={null}
from edsl import QuestionNumerical, Model

m = Model("gemini-1.5-flash")

q = QuestionNumerical(
  question_name = "random",
  question_text = "Please give me a random number between 1 and 100."
)

results = q.by(m).run()

results.cache
```

Example output:

| model | parameters | system\_prompt | user\_prompt | output | iteration | timestamp | cache\_key |
| :- | :- | :- | :- | :- | :- | :- | :- |
| gemini-1.5-flash | `{‘temperature’: 0.5, ‘topP’: 1, ‘topK’: 1, ‘maxOutputTokens’: 2048, ‘stopSequences’: []}` | nan | Please give me a random number between 1 and 100. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, …). Respond with just your number on a single line. If your response is equivalent to zero, report ‘0’ After the answer, put a comment explaining your choice on the next line. | `{“candidates”: [{“content”: {“parts”: [{“text”: “87n# This is a randomly generated number between 1 and 100.n”}], “role”: “model”}, “finish_reason”: 1, “safety_ratings”: [{“category”: 8, “probability”: 1, “blocked”: false}, {“category”: 10, “probability”: 1, “blocked”: false}, {“category”: 7, “probability”: 1, “blocked”: false}, {“category”: 9, “probability”: 1, “blocked”: false}], “avg_logprobs”: -0.03539780080318451, “token_count”: 0, “grounding_attributions”: []}], “usage_metadata”: {“prompt_token_count”: 97, “candidates_token_count”: 20, “total_token_count”: 117, “cached_content_token_count”: 0}}` | 0 | 1737491116 | 7f057154c60a1b9ae343b0634fe7a370 |

The results object also includes columns of information about the cache for the results:

```python theme={null}
results.columns
```

Output:

| |
| :- |
| agent.agent\_index |
| agent.agent\_instruction |
| agent.agent\_name |
| answer.random |
| cache\_keys.random\_cache\_key |
| cache\_used.random\_cache\_used |
| comment.random\_comment |
| generated\_tokens.random\_generated\_tokens |
| iteration.iteration |
| model.maxOutputTokens |
| model.model |
| model.model\_index |
| model.stopSequences |
| model.temperature |
| model.topK |
| model.topP |
| prompt.random\_system\_prompt |
| prompt.random\_user\_prompt |
| question\_options.random\_question\_options |
| question\_text.random\_question\_text |
| question\_type.random\_question\_type |
| raw\_model\_response.random\_cost |
| raw\_model\_response.random\_one\_usd\_buys |
| raw\_model\_response.random\_raw\_model\_response |
| scenario.scenario\_index |

The cache\_keys column contains the cache key for each question. It is a unique identifier for the cache entry, and is a hash of the unique inputs to the language model; if you rerun the same question with the same parameters, it will return the same cache key. It can be used to retrieve the cache entry later, or to add the entry to a different cache.

The cache\_used column indicates whether or not each result was retrieved from the default cache (your local cache, your remote cache, or a cache that was passed to the run() method when the results were generated). Whenever a question is run for the first time, the cache\_used column will be False; if the same question is run again with the same cache available, the cache\_used column will be True.

For example, here we run a new question and confirm that the default cache was not used (i.e., the cache key for the question is not already in the available cache):

```python theme={null}
from edsl import QuestionNumerical, Model

m = Model("gemini-1.5-flash")

q = QuestionNumerical(
  question_name = "random",
  question_text = "Please give me a random number between 1 and 100."
)

r1 = q.by(m).run()
r1.select("cache_keys.*", "cache_used.*")
```

Output:

| cache\_keys.random\_cache\_key | cache\_used.random\_cache\_used |
| :- | :- |
| 7f057154c60a1b9ae343b0634fe7a370 | False |

If we run the question again, the cache key will be the same, but the cache\_used indicator will now be True:

```python theme={null}
r2 = q.by(m).run()
r2.select("cache_keys.*", "cache_used.*")
```

Output:

| cache\_keys.random\_cache\_key | cache\_used.random\_cache\_used |
| :- | :- |
| 7f057154c60a1b9ae343b0634fe7a370 | True |

## Working with a cache

A cache can be passed to the run() method of a survey in order to make the entries available in generating the survey results. This can be useful if you want to add new entries to a specific cache other than your default cache (local or remote).

For example:

```python expandable theme={null}
from edsl import QuestionFreeText, QuestionNumerical, Survey, Model

m = Model("gemini-1.5-flash")

# This question is already run, so we can use its cache from above (r1.cache)
q1 = QuestionNumerical(
  question_name = "random",
  question_text = "Please give me a random number between 1 and 100."
)

# This question is new, so the entry will be added to the cache that we use
q2 = QuestionFreeText(
  question_name = "explain",
  question_text = "How does an AI choose a random number?"
)

survey = Survey(questions = [q1,q2])

# The results will include the retrieved entry for q1 and the new entry for q2
r3 = survey.by(m).run(cache = r1.cache)

r3.select("cache_keys.*", "cache_used.*")
```

Output:

| cache\_keys.random\_cache\_key | cache\_keys.explain\_cache\_key | cache\_used.random\_cache\_used | cache\_used.explain\_cache\_used |
| :- | :- | :- | :- |
| 7f057154c60a1b9ae343b0634fe7a370 | 6442cf8e6b9812a89bd50bf059f77885 | True | False |

<Note>
  **Note**:

  Note that this is independent of the new cache generated for the results as above, which are accessed by calling r3.cache.
</Note>

## Instantiating a new cache

This code will instantiate a new cache object using a dictionary as the data attribute

### In-memory usage

```python theme={null}
from edsl import Cache

my_in_memory_cache = Cache()
```

It can then be passed as an object to a run method:

```python theme={null}
from edsl import Survey

s = Survey.example()

results = s.run(cache = my_in_memory_cache)
```

If an in-memory cache is not stored explicitly, the data will be lost when the session is over, unless it is written to a file or remote caching is instantiated. More on this below.

### Local persistence for an in-memory cache

```python theme={null}
c = Cache()
# a bunch of operations
c.write_sqlite_db("example.db")
# or
c.write_jsonl("example.jsonl")
```

You can then load the cache from the SQLite3 database or JSONL file using Cache methods.

```python theme={null}
c = Cache.from_sqlite_db("example.db")
# or
c = Cache.from_jsonl("example.jsonl")
```

### SQLite3Dict for transactions

Instead of using a dictionary as the data attribute, you can use a special dictionary-like object based on SQLite3. This will persist the cache between sessions. This is the “normal” way that a cache is used for runs where no specic cache is passed.

```python theme={null}
from edsl import Cache
from edsl.data.SQLiteDict import SQLiteDict

my_sqlite_cache = Cache(data = SQLiteDict("example.db"))
```

This will leave a SQLite3 database on the user’s machine at the file, in this case example.db in the current directory. It will persist between sessions and can be loaded using the from\_sqlite\_db method shown above.

### Default SQLite Cache: .edsl\_cache/data.db

By default, the cache will be stored in a SQLite3 database at the path .edsl\_cache/data.db. You can interact with this cache directly, e.g.,

```python theme={null}
sqlite3 .edsl_cache/data.db
```

### Setting a session cache

The set\_session\_cache function is used to set the cache for a session:

```python theme={null}
from edsl import Cache, set_session_cache

set_session_cache(Cache())
```

The cache can be set to a specific cache object, or it can be set to a dictionary or SQLite3Dict object.

```python theme={null}
from edsl import Cache, set_session_cache
from edsl.data import SQLiteDict

set_session_cache(Cache(data = SQLiteDict("example.db")))
# or
set_session_cache(Cache(data = {}))
```

This will set the cache for the current session, and you do not need to pass the cache object to the run method during the session.

The unset\_session\_cache function is used to unset the cache for a session:

```python theme={null}
from edsl import unset_session_cache

unset_session_cache()
```

This will unset the cache for the current session, and you will need to pass the cache object to the run method during the session.

### Avoiding cache persistence

We can avoid cache persistence by passing cache=False to the run method:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText.example()

results = q.run(cache = False)
```

## For developers

### Delayed cache-writing: Useful for remote caching

Separate from this remote cache syncing, delays can be made in writing to the cache itself. By default, the cache will write to the cache immediately after storing a new entry. However, this can be changed by setting the immediate\_write parameter to False.

```python theme={null}
c = Cache(immediate_write = False)
```

This is useful when you want to store entries to the cache only after a block of code has been executed. This is also controlled by using the cache object as a context.

```python theme={null}
with c as cache:
  # readings / writing
  ...

# The cache will be written to the cache persistence layer after the block of code has been executed
```

## Cache class

> ### *class* edsl.caching.Cache(\*, *filename: str | None = None*, *data: SQLiteDict | dict | None = None*, *immediate\_write: bool = True*, *method=None*, *verbose=False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Bases: [`Base`](/en/latest/base "edsl.base.base_class.Base")

Cache for storing and retrieving language model responses.

The Cache class manages a collection of CacheEntry objects, providing methods for storing, retrieving, and persisting language model responses. It serves as the core component of EDSL’s caching infrastructure, helping to reduce redundant API calls, save costs, and ensure reproducibility.

Cache can use different storage backends: - In-memory dictionary (default) - SQLite database via SQLiteDict - JSON lines file (.jsonl)

The cache operates by generating deterministic keys based on the model, parameters, prompts, and iteration number. This allows for efficient lookup of cached responses when identical requests are made.

**Attributes:**

data (dict or SQLiteDict): The primary storage for cache entries new\_entries (dict): Entries added in the current session fetched\_data (dict): Entries retrieved in the current session filename (str, optional): Path for persistence if provided immediate\_write (bool): Whether to update data immediately (True) or defer (False)

**Technical Notes:**

* Can be used as a context manager to automatically persist changes on exit
* Supports serialization/deserialization via to\_dict/from\_dict methods
* Implements set operations (addition, subtraction) for combining caches
* Integrates with the broader EDSL caching infrastructure via CacheHandler

> #### **init**(\*, *filename: str | None = None*, *data: SQLiteDict | dict | None = None*, *immediate\_write: bool = True*, *method=None*, *verbose=False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Initialize a new Cache instance.

Creates a new cache for storing language model responses. The cache can be initialized with existing data or connected to a persistent storage file.

> **Args:**

**filename: Path to a persistent storage file (.jsonl or .db). If provided, the cache**

will be initialized from this file and changes will be written back to it. Cannot be used together with data parameter.

**data: Initial cache data as a dictionary or SQLiteDict. Cannot be used together**

with filename parameter.

**immediate\_write: If True, new entries are immediately added to the main data store.**

If False, they’re kept separate until explicitly written.

method: Deprecated. Legacy parameter for backward compatibility. verbose: If True, prints diagnostic information about cache hits and misses.

> **Raises:**

**CacheError: If both filename and data are provided, or if the filename has an**

invalid extension.

> **Implementation Notes:**

* The cache maintains separate dictionaries for tracking: \* data: The main persistent storage \* new\_entries: Entries added in this session \* fetched\_data: Entries fetched in this session \* new\_entries\_to\_write\_later: Entries to be written if immediate\_write=False
* If loading from a file, the appropriate loader method is called based on extension

> ### add\_from\_dict(*new\_data: dict\[str, 'CacheEntry']*, *write\_now: bool | None = True*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Add entries to the cache from a dictionary.

**Parameters:**

**write\_now** – Whether to write to the cache immediately (similar to immediate\_write).

> ### add\_from\_jsonl(*filename: str*, *write\_now: bool | None = True*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Add entries to the cache from a JSONL.

**Parameters:**

**write\_now** – Whether to write to the cache immediately (similar to immediate\_write).

> ### add\_from\_sqlite(*db\_path: str*, *write\_now: bool | None = True*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Add entries to the cache from an SQLite database.

**Parameters:**

**write\_now** – Whether to write to the cache immediately (similar to immediate\_write).

> ### close()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Explicitly close and clean up resources.

This method properly disposes of any SQLAlchemy engines and connections to prevent memory leaks.

> ### code()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Generate Python code that recreates this object.

This method must be implemented by all subclasses to provide a way to generate executable Python code that can recreate the object.

**Returns:**

str: Python code that, when executed, creates an equivalent object

> ### data *=&#x20;*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

> ### *classmethod* example(*randomize: bool = False*) → [Cache](/en/latest/data#cache-class)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Create an example Cache instance for testing and demonstration.

Creates a Cache object pre-populated with example CacheEntry objects. This method is useful for documentation, testing, and demonstration purposes.

> **Args:**

**randomize: If True, creates CacheEntry objects with randomized content**

for uniqueness. If False, uses consistent example entries.

> **Returns:**

Cache: A new Cache object containing example CacheEntry objects

> **Technical Notes:**

* Uses CacheEntry.example() to create sample entries
* When randomize=True, generates unique keys for each call
* When randomize=False, produces consistent examples for doctests
* Creates an in-memory cache (no persistent file)

> **Examples:**

```python theme={null}
>>> cache = Cache.example()
>>> len(cache) > 0
True
>>> from edsl.caching.cache_entry import CacheEntry
>>> all(isinstance(entry, CacheEntry) for entry in cache.values())
True
```

```python theme={null}
>>> # Create examples with randomized content
>>> cache1 = Cache.example(randomize=True)
>>> cache2 = Cache.example(randomize=True)
>>> # With randomization, keys should be different
>>> len(cache1) > 0 and len(cache2) > 0
True
```

> ### fetch(\*\*\*, *model: str*, *parameters: dict*, *system\_prompt: str*, *user\_prompt: str*, *iteration: int*, *validated: bool = False*, *remote\_fetch: bool = False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Retrieve a cached language model response if available.

This method attempts to find a cached response matching the exact input parameters. The combination of model, parameters, prompts, and iteration creates a unique key that identifies a specific language model request.

> **Args:**

model: Language model identifier (e.g., “gpt-3.5-turbo”) parameters: Model configuration parameters (e.g., temperature, max\_tokens) system\_prompt: The system instructions given to the model user\_prompt: The user query/prompt given to the model iteration: The iteration number for this specific request

> **Returns:**

tuple: (response, key) where:

* response: The cached model output as a string, or None if not found
* key: The cache key string generated for this request

> **Technical Notes:**

* Uses CacheEntry.gen\_key() to generate a consistent hash-based key
* Updates self.fetched\_data when a hit occurs to track cache usage
* Optionally logs cache hit/miss when verbose=True
* The response is returned as a JSON string for consistency
* On local cache miss, attempts to fetch from remote universal cache

> **Examples:**

```python theme={null}
>>> c = Cache()
>>> c.fetch(model="gpt-3", parameters="default", system_prompt="Hello",
...         user_prompt="Hi", iteration=1)[0] is None
True
```

> ### fetch\_input\_example() → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Create an example input for a ‘fetch’ operation.

> ### *classmethod* from\_dict(*data*) → [Cache](/en/latest/data#cache-class)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Construct a Cache from a dictionary.

> ### *classmethod* from\_jsonl(*jsonlfile: str*, *db\_path: str | None = None*) → [Cache](/en/latest/data#cache-class)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Construct a Cache from a JSONL file.

**Parameters:**

* **jsonlfile** – The path to the JSONL file of cache entries.

* **db\_path** – The path to the SQLite database used to store the cache.

* If db\_path is None, the cache will be stored in memory, as a dictionary.

* If db\_path is provided, the cache will be stored in an SQLite database.

> ### *classmethod* from\_local\_cache() → [Cache](/en/latest/data#cache-class)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Construct a Cache from a local cache file.

> ### *classmethod* from\_sqlite\_db(*db\_path: str*) → [Cache](/en/latest/data#cache-class)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Construct a Cache from a SQLite database.

> ### *classmethod* from\_url(*db\_path=None*) → [Cache](/en/latest/data#cache-class)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Construct a Cache object from a remote.

> ### items()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Return an iterator of (key, value) pairs in the cache.

Similar to dict.items(), provides an iterator over all key-value pairs in the cache for easy iteration.

**Returns:**

zip: An iterator of (key, CacheEntry) tuples

> ### keys()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Return a list of all cache keys.

Retrieves all cache keys, which are the unique identifiers for each cache entry.

> **Returns:**

list: A list of string keys in the cache

> **Examples:**

```python theme={null}
>>> from edsl import Cache
>>> Cache.example().keys()
['5513286eb6967abc0511211f0402587d']
```

> ### new\_entries\_cache() → [Cache](/en/latest/data#cache-class)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Return a new Cache object with the new entries.

> ### select(\**fields*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

> ### store(*model: str*, *parameters: str*, *system\_prompt: str*, *user\_prompt: str*, *response: dict*, *iteration: int*, *service: str*, *validated: bool = False*) → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Store a new language model response in the cache.

Creates a new CacheEntry from the provided parameters and response, then adds it to the cache using a deterministic key derived from the input parameters.

> **Args:**

model: Language model identifier (e.g., “gpt-3.5-turbo”) parameters: Model configuration parameters (e.g., temperature, max\_tokens) system\_prompt: The system instructions given to the model user\_prompt: The user query/prompt given to the model response: The model’s response as a dictionary iteration: The iteration number for this specific request service: The service provider (e.g., “openai”, “anthropic”) validated: Whether the response has been validated (default: False)

> **Returns:**

str: The cache key generated for this entry

> **Technical Notes:**

* Creates a new CacheEntry object to encapsulate the response and metadata
* Adds the entry to self.new\_entries to track entries added in this session
* Adds the entry to the main data store if immediate\_write=True
* Otherwise, stores in new\_entries\_to\_write\_later for deferred writing
* The response is stored as a JSON string for consistency and compatibility

> **Storage Behavior:**

The method’s behavior depends on the immediate\_write setting: - If True: Immediately writes to the main data store (self.data) - If False: Stores in a separate dict for writing later (e.g., at context exit)

> **Examples:**

```python theme={null}
>>> from edsl import Cache, Model, Question
>>> m = Model("test")
>>> c = Cache()
>>> len(c)
0
>>> results = Question.example("free_text").by(m).run(cache=c,
...         disable_remote_cache=True, disable_remote_inference=True)
>>> len(c)
1
```

> ### subset(*keys: list\[str]*) → [Cache](/en/latest/data#cache-class)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Return a subset of the Cache with the specified keys.

> ### table(\**fields*, *tablefmt: str | None = None*, *pretty\_labels: dict | None = None*) → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

> ### to\_dataset()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Convert this object to a Dataset for advanced data operations.

**Returns:**

Dataset: A Dataset object containing this object’s data

> ### to\_dict(*add\_edsl\_version=True*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Serialize the cache to a dictionary for storage or transmission.

Converts the Cache object into a plain dictionary format that can be easily serialized to JSON or other formats. Each CacheEntry is also converted to a dictionary using its to\_dict method.

> **Args:**

**add\_edsl\_version: If True, includes the EDSL version and class name**

in the serialized output for compatibility tracking

> **Returns:**

> **dict: A dictionary representation of the cache with the structure:**

```python theme={null}
{
“key1”: {cache_entry1_dict}, “key2”: {cache_entry2_dict}, … “edsl_version”: “x.x.x”, # if add_edsl_version=True “edsl_class_name”: “Cache” # if add_edsl_version=True
}
```

> **Technical Notes:**

* Used by from\_dict for deserialization
* Used by **hash** for cache comparison
* The version info allows for proper handling of format changes

> ### to\_html()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

> ### to\_scenario\_list()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

> ### tree(*node\_list: list\[str] | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

> ### values()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Return a list of all cache entry values.

Retrieves all CacheEntry objects stored in the cache.

> **Returns:**

list: A list of CacheEntry objects

> **Examples:**

```python theme={null}
>>> from edsl import Cache
>>> entries = Cache.example().values()
>>> len(entries)
1
>>> entries[0]
CacheEntry(model='gpt-3.5-turbo', parameters={'temperature': 0.5}, ...)
```

> ### view() → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

View the Cache in a new browser tab.

> ### write(*filename: str | None = None*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Write the cache to a file at the specified location.

> ### write\_jsonl(*filename: str*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Write the cache to a JSONL file.

> ### write\_sqlite\_db(*db\_path: str*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/caching.py)

Write the cache to an SQLite database.


# Dataset
Source: https://docs.expectedparrot.com/en/latest/dataset

The `Dataset` class is a versatile data container for tabular data with powerful manipulation capabilities. It represents data in a column-oriented format, providing methods for analysis, transformation, and visualization.The `Dataset` class is a versatile data container for tabular data with powerful manipulation capabilities. It represents data in a column-oriented format, providing methods for analysis, transformation, and visualization.

## Overview

Dataset is a fundamental data structure in EDSL that provides a column-oriented representation of tabular data. It offers methods for manipulating, analyzing, visualizing, and exporting data, similar to tools like pandas or dplyr.

Key features:

1. Flexible data manipulation (filtering, sorting, transformation)
2. Visualization capabilities with multiple rendering options
3. Export to various formats (CSV, Excel, Pandas, etc.)
4. Integration with other EDSL components

## Creating Datasets

Datasets can be created from various sources:

From dictionaries:

```python theme={null}
from edsl import Dataset

 # Create a dataset with two columns
d  = Dataset( [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ])
```

From existing EDSL objects:

```python theme={null}
 # From Results object
dataset  = results.select('how _feeling', 'agent.status')

 # From pandas DataFrame
import pandas as pd
df  = pd.DataFrame({'a':  [1, 2, 3 ], 'b':  [4, 5, 6 ]})
dataset  = Dataset.from _pandas _dataframe(df)
```

## Displaying and Visualizing Data

The Dataset class provides multiple ways to display and visualize data:

Basic display:

```python theme={null}
 # Print the dataset
dataset.print()

 # Display only the first few rows
dataset.head()

 # Display only the last few rows
dataset.tail()
```

## Table Display Options

You can control the table formatting using the `tablefmt` parameter:

```python theme={null}
 # Display as an ASCII grid
dataset.table(tablefmt ="grid")

 # Display as pipe-separated values
dataset.table(tablefmt ="pipe")

 # Display as HTML
dataset.table(tablefmt ="html")

 # Display as Markdown
dataset.table(tablefmt ="github")

 # Display as LaTeX
dataset.table(tablefmt ="latex")
```

### Rich Terminal Output

The Dataset class supports displaying tables with enhanced formatting using the Rich library, which provides beautiful terminal formatting with colors, styles, and more:

```python theme={null}
 # Display using Rich formatting
dataset.table(tablefmt ="rich")

 # Alternative syntax
dataset.print(format ="rich")
```

This creates a nicely formatted table in the terminal with automatically sized columns, bold headers, and grid lines.

Example:

```python theme={null}
from edsl import Dataset

 # Create a dataset
d  = Dataset( [
    {'name':  ['Alice', 'Bob', 'Charlie', 'David' ]},
    {'age':  [25, 32, 45, 19 ]},
    {'city':  ['New York', 'Los Angeles', 'Chicago', 'Boston' ]}
 ])

 # Display with rich formatting
d.table(tablefmt ="rich")
```

## Data Manipulation

The Dataset class provides numerous methods for data manipulation:

Filtering:

```python theme={null}
 # Filter results for a specific condition
filtered  = dataset.filter("how _feeling == 'Great'")
```

Creating new columns:

```python theme={null}
 # Create a new column
with _sentiment  = dataset.mutate("sentiment = 1 if how _feeling == 'Great' else 0")
```

Sorting:

```python theme={null}
 # Sort by a specific column
sorted _data  = dataset.order _by("age", reverse =True)
```

Reshaping:

```python theme={null}
 # Convert to long format
long _data  = dataset.long()

 # Convert back to wide format
wide _data  = long _data.wide()
```

## Exporting Data

Export to various formats:

```python theme={null}
 # Export to CSV
dataset.to _csv("data.csv")

 # Export to pandas DataFrame
df  = dataset.to _pandas()

 # Export to Word document
dataset.to _docx("data.docx", title ="My Dataset")
```

## Dataset Methods

> ### *class* edsl.dataset.Dataset(*data: list \[dict \[str, Any ] ]  = None*, *print \_parameters: dict | None  = None*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Bases: `UserList`, `DatasetOperationsMixin`, `PersistenceMixin`, `HashingMixin`

A versatile data container for tabular data with powerful manipulation capabilities.

The Dataset class is a fundamental data structure in EDSL that represents tabular data in a column-oriented format. It provides a rich set of methods for data manipulation, transformation, analysis, visualization, and export through the DatasetOperationsMixin.

Key features:

1. Column-oriented data structure optimized for LLM experiment results
2. Rich data manipulation API similar to dplyr/pandas (filter, select, mutate, etc.)
3. Visualization capabilities including tables, plots, and reports
4. Export to various formats (CSV, Excel, SQLite, pandas, etc.)
5. Serialization for storage and transport
6. Tree-based data exploration

A Dataset typically contains multiple columns, each represented as a dictionary with a single key-value pair. The key is the column name and the value is a list of values for that column. All columns must have the same length.

The Dataset class inherits from: - UserList: Provides list-like behavior for storing column data - DatasetOperationsMixin: Provides data manipulation methods - PersistenceMixin: Provides serialization capabilities - HashingMixin: Provides hashing functionality for comparison and storage

Datasets are typically created by transforming other EDSL container types like Results, AgentList, or ScenarioList, but can also be created directly from data.

> #### collapse(*field: str*, *separator: str | None  = None*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Collapse multiple values in a field into a single value using a separator.

> **Args:**

field: The name of the field to collapse. separator: Optional string to use as a separator between values.

> Defaults to a space if not specified.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'words':  [ ['hello', 'world' ],  ['good', 'morning' ] ]} ])
 >>> d.collapse('words').data
 [{'words':  [ [ ['hello', 'world' ],  ['good', 'morning' ] ] ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'numbers':  [1, 2, 3 ]} ])
 >>> d.collapse('numbers', separator =',').data
 [{'numbers':  ['1,2,3' ]} ]
```

> #### drop(*field \_name*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Returns a new Dataset with the specified field removed.

> **Args:**

field \_name (str): The name of the field to remove.

> **Returns:**

Dataset: A new Dataset instance without the specified field.

> **Raises:**

KeyError: If the field \_name doesn’t exist in the dataset.

> **Examples:**

```python theme={null}
 >>> from .dataset import Dataset
 >>> d  = Dataset( [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ])
 >>> d.drop('a')
Dataset( [{'b':  [4, 5, 6 ]} ])
```

```python theme={null}
 >>>  # Testing drop with nonexistent field raises DatasetKeyError - tested in unit tests
```

> #### *classmethod* example(*n: int  = None*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return an example dataset.

> **Examples:**

```python theme={null}
 >>> Dataset.example()
Dataset( [{'a':  [1, 2, 3, 4 ]}, {'b':  [4, 3, 2, 1 ]} ])
```

```python theme={null}
>>> Dataset.example(n =2)
Dataset( [{'a':  [1, 1 ]}, {'b':  [2, 2 ]} ])
```

> #### expand(*field: str*, *number \_field: bool  = False*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Expand a field containing lists into multiple rows.

> **Args:**

field: The field containing lists to expand number \_field: If True, adds a number field indicating the position in the original list

> **Returns:**

A new Dataset with the expanded rows

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [ [1, 2, 3 ],  [4, 5, 6 ] ]}, {'b':  ['x', 'y' ]} ])
 >>> d.expand('a').data
 [{'a':  [1, 2, 3, 4, 5, 6 ]}, {'b':  ['x', 'x', 'x', 'y', 'y', 'y' ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'items':  [ ['apple', 'banana' ],  ['orange' ] ]}, {'id':  [1, 2 ]} ])
 >>> d.expand('items', number _field =True).data
 [{'items':  ['apple', 'banana', 'orange' ]}, {'id':  [1, 1, 2 ]}, {'items _number':  [1, 2, 1 ]} ]
```

> #### expand \_field(*field*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Expand a field in the dataset.

Renamed to avoid conflict with the expand method defined earlier.

> #### filter(*expression*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Filter the dataset based on a boolean expression.

> **Args:**

expression: A string expression that evaluates to a boolean value.

Can reference column names in the dataset.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3, 4 ]}, {'b':  [5, 6, 7, 8 ]} ])
 >>> d.filter('a > 2').data
 [{'a':  [3, 4 ]}, {'b':  [7, 8 ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'x':  ['a', 'b', 'c' ]}, {'y':  [1, 2, 3 ]} ])
 >>> d.filter('y < 3').data
 [{'x':  ['a', 'b' ]}, {'y':  [1, 2 ]} ]
```

> #### first() → dict \[str, Any ] [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Get the first value of the first key in the first dictionary.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3, 4 ]}, {'b':  [5, 6, 7, 8 ]} ])
 >>> d.first()
1
```

```python theme={null}
 >>> d  = Dataset( [{'x':  ['first', 'second' ]} ])
 >>> d.first()
'first'
```

> #### *classmethod* from \_dict(*data: dict*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Convert a dictionary to a dataset.

> **Examples:**

```python theme={null}
 >>> d  = Dataset.from _dict({'data':  [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ]})
 >>> isinstance(d, Dataset)
True
 >>> d.data
 [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ]
```

```python theme={null}
 >>> d  = Dataset.from _dict({'data':  [{'x':  ['a', 'b' ]} ]})
 >>> d.data
 [{'x':  ['a', 'b' ]} ]
```

> #### *classmethod* from \_edsl \_object(*object*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

> #### *classmethod* from \_pandas \_dataframe(*df*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

> #### get \_sort \_indices(*lst: list \[Any ]*, *reverse: bool  = False*) → list \[int ] [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return the indices that would sort the list, using either numpy or pure Python. None values are placed at the end of the sorted list.

> **Args:**

lst: The list to be sorted reverse: Whether to sort in descending order use \_numpy: Whether to use numpy implementation (falls back to pure Python if numpy is unavailable)

> **Returns:**

A list of indices that would sort the list

> ### head(*n: int  = 5*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return the first n observations in the dataset.

```python theme={null}
 >>> d  = Dataset( [{'a.b': [1,2,3,4 ]} ])
 >>> d.head(2)
Dataset( [{'a.b':  [1, 2 ]} ])
```

> #### keys() → list \[str ] [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return the keys of the dataset.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3, 4 ]}, {'b':  [5, 6, 7, 8 ]} ])
 >>> d.keys()
 ['a', 'b' ]
```

```python theme={null}
 >>> d  = Dataset( [{'x.y':  [1, 2 ]}, {'z.w':  [3, 4 ]} ])
 >>> d.keys()
 ['x.y', 'z.w' ]
```

> #### latex(\* \* *kwargs*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return a LaTeX representation of the dataset.

> **Args:**

[ \* \*](/en/latest/#id2)kwargs: Additional arguments to pass to the table formatter.

> #### long(*exclude \_fields: list \[str ]  = None*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Convert the dataset from wide to long format.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2 ], 'b':  [3, 4 ]} ])
 >>> d.long().data
 [{'row':  [0, 0, 1, 1 ]}, {'key':  ['a', 'b', 'a', 'b' ]}, {'value':  [1, 3, 2, 4 ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'x':  [1, 2 ], 'y':  [3, 4 ], 'z':  [5, 6 ]} ])
 >>> d.long(exclude _fields = ['z' ]).data
 [{'row':  [0, 0, 1, 1 ]}, {'key':  ['x', 'y', 'x', 'y' ]}, {'value':  [1, 3, 2, 4 ]}, {'z':  [5, 5, 6, 6 ]} ]
```

> #### merge(*other: [Dataset](/en/latest/dataset#dataset-methods)*, *by \_x*, *by \_y*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Merge the dataset with another dataset on the given keys.

> **Examples:**

```python theme={null}
 >>> d1  = Dataset( [{'key':  [1, 2, 3 ]}, {'value1':  ['a', 'b', 'c' ]} ])
 >>> d2  = Dataset( [{'key':  [2, 3, 4 ]}, {'value2':  ['x', 'y', 'z' ]} ])
 >>> merged  = d1.merge(d2, 'key', 'key')
 >>> len(merged.data [0 ] ['key' ])
3
```

```python theme={null}
 >>> d1  = Dataset( [{'id':  [1, 2 ]}, {'name':  ['Alice', 'Bob' ]} ])
 >>> d2  = Dataset( [{'id':  [2, 3 ]}, {'age':  [25, 30 ]} ])
 >>> merged  = d1.merge(d2, 'id', 'id')
 >>> len(merged.data [0 ] ['id' ])
2
```

> #### order \_by(*sort \_key: str*, *reverse: bool  = False*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return a new dataset with the observations sorted by the given key.

> **Args:**

sort \_key: The key to sort the observations by reverse: Whether to sort in reverse order

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [3, 1, 4, 1, 5 ]}, {'b':  ['x', 'y', 'z', 'w', 'v' ]} ])
 >>> sorted _d  = d.order _by('a')
 >>> sorted _d.data
 [{'a':  [1, 1, 3, 4, 5 ]}, {'b':  ['y', 'w', 'x', 'z', 'v' ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'a':  [3, 1, 4, 1, 5 ]}, {'b':  ['x', 'y', 'z', 'w', 'v' ]} ])
 >>> sorted _d  = d.order _by('a', reverse =True)
 >>> sorted _d.data
 [{'a':  [5, 4, 3, 1, 1 ]}, {'b':  ['v', 'z', 'x', 'y', 'w' ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'a':  [3, None, 1, 4, None ]}, {'b':  ['x', 'y', 'z', 'w', 'v' ]} ])
 >>> sorted _d  = d.order _by('a')
 >>> sorted _d.data
 [{'a':  [1, 3, 4, None, None ]}, {'b':  ['z', 'x', 'w', 'y', 'v' ]} ]
```

> #### print(*pretty \_labels =None*, \* \* *kwargs*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Print the dataset in a formatted way.

> **Args:**

pretty \_labels: A dictionary mapping column names to their display names [ \* \*](/en/latest/#id4)kwargs: Additional arguments

format: The output format (“html”, “markdown”, “rich”, “latex”)

> **Returns:**

TableDisplay object

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ])
 >>> display  = d.print(format ='rich')
 >>> display is not None
True
```

```python theme={null}
 >>> d  = Dataset( [{'long _column _name':  [1, 2 ]} ])
 >>> display  = d.print(pretty _labels ={'long _column _name': 'Short'})
 >>> display is not None
True
```

> #### remove \_prefix() → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Remove the prefix from column names that contain dots.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a.b':  [1, 2, 3 ]}, {'c.d':  [4, 5, 6 ]} ])
 >>> d.remove _prefix().data
 [{'b':  [1, 2, 3 ]}, {'d':  [4, 5, 6 ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'x.y.z':  [1, 2 ]}, {'a.b.c':  [3, 4 ]} ])
 >>> d.remove _prefix().data
 [{'y':  [1, 2 ]}, {'b':  [3, 4 ]} ]
```

> #### rename(*rename \_dic*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Rename columns in the dataset according to the provided dictionary.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ])
 >>> d.rename({'a': 'x', 'b': 'y'}).data
 [{'x':  [1, 2, 3 ]}, {'y':  [4, 5, 6 ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'old _name':  [1, 2 ]} ])
 >>> d.rename({'old _name': 'new _name'}).data
 [{'new _name':  [1, 2 ]} ]
```

> #### sample(*n: int  = None*, *frac: float  = None*, *with \_replacement: bool  = True*, *seed: str | int | float  = None*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return a new dataset with a sample of the observations.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3, 4, 5 ]}, {'b':  [6, 7, 8, 9, 10 ]} ])
 >>> sampled  = d.sample(n =3, seed =42)
 >>> len(sampled.data [0 ] ['a' ])
3
```

```python theme={null}
 >>> d  = Dataset( [{'x':  ['a', 'b', 'c', 'd' ]} ])
 >>> sampled  = d.sample(frac =0.5, seed =123)
 >>> len(sampled.data [0 ] ['x' ])
2
```

> #### select(\* *keys*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return a new dataset with only the selected keys.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3, 4 ]}, {'b':  [5, 6, 7, 8 ]}, {'c':  [9, 10, 11, 12 ]} ])
 >>> d.select('a', 'c').data
 [{'a':  [1, 2, 3, 4 ]}, {'c':  [9, 10, 11, 12 ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'x':  [1, 2 ]}, {'y':  [3, 4 ]} ])
 >>> d.select('x').data
 [{'x':  [1, 2 ]} ]
```

> #### shuffle(*seed =None*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return a new dataset with the observations shuffled.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3, 4 ]}, {'b':  [5, 6, 7, 8 ]} ])
 >>> shuffled  = d.shuffle(seed =42)
 >>> len(shuffled.data [0 ] ['a' ])  == len(d.data [0 ] ['a' ])
True
```

```python theme={null}
 >>> d  = Dataset( [{'x':  ['a', 'b', 'c' ]} ])
 >>> shuffled  = d.shuffle(seed =123)
 >>> set(shuffled.data [0 ] ['x' ])  == set(d.data [0 ] ['x' ])
True
```

> #### summary() → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return a summary of the dataset.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ])
 >>> d.summary().data
 [{'num _observations':  [3 ]}, {'keys':  [ ['a', 'b' ] ]} ]
```

> #### table(\* *fields*, *tablefmt: str | None  = 'rich'*, *max \_rows: int | None  = None*, *pretty \_labels =None*, *print \_parameters: dict | None  = None*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

> #### tail(*n: int  = 5*) → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return the last n observations in the dataset.

```python theme={null}
 >>> d  = Dataset( [{'a.b': [1,2,3,4 ]} ])
 >>> d.tail(2)
Dataset( [{'a.b':  [3, 4 ]} ])
```

> #### to(*survey \_or \_question: 'Survey' | 'QuestionBase'*) → Job [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Transform the dataset using a survey or question.

> **Args:**

survey \_or \_question: Either a Survey or QuestionBase object to apply to the dataset.

> **Examples:**

```python theme={null}
>>> from edsl import QuestionFreeText
>>> from edsl.jobs import Jobs
>>> d = Dataset([{'name': ['Alice', 'Bob']}])
>>> q = QuestionFreeText(question_text="How are you, {{ name }}?", question_name="how_feeling")
>>> job = d.to(q)
>>> isinstance(job, Jobs)
True
```

> #### to \_dict() → dict [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Convert the dataset to a dictionary.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ])
 >>> d.to _dict()
{'data':  [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ]}
```

```python theme={null}
 >>> d  = Dataset( [{'x':  ['a', 'b' ]} ])
 >>> d.to _dict()
{'data':  [{'x':  ['a', 'b' ]} ]}
```

> #### to \_docx(*output \_file: str*, *title: str  = None*) → None [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Convert the dataset to a Word document.

> **Args:**

output \_file (str): Path to save the Word document title (str, optional): Title for the document

> **Examples:**

```python theme={null}
 >>> import tempfile
 >>> d  = Dataset( [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ])
 >>> with tempfile.NamedTemporaryFile(suffix ='.docx') as tmp:
...     d.to _docx(tmp.name, title ='Test Document')
...     import os
...     os.path.exists(tmp.name)
True
```

> #### to \_json() [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return a JSON representation of the dataset.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ])
 >>> d.to _json()
 [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'x':  ['a', 'b' ]} ])
 >>> d.to _json()
 [{'x':  ['a', 'b' ]} ]
```

> #### tree(*node \_order: list \[str ] | None  = None*) → Tree [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Return a tree representation of the dataset.

```python theme={null}
 >>> d  = Dataset( [{'a': [1,2,3,4 ]}, {'b': [4,3,2,1 ]} ])
 >>> d.tree()
Tree(Dataset({'a':  [1, 2, 3, 4 ], 'b':  [4, 3, 2, 1 ]}), node _order= ['a', 'b' ])
```

> #### unique() → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Remove duplicate rows from the dataset.

> **Returns:**

A new Dataset with duplicate rows removed.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'a':  [1, 2, 3, 1 ]}, {'b':  [4, 5, 6, 4 ]} ])
 >>> d.unique().data
 [{'a':  [1, 2, 3 ]}, {'b':  [4, 5, 6 ]} ]

 >>> d  = Dataset( [{'x':  ['a', 'b', 'a' ]}, {'y':  [1, 2, 1 ]} ])
 >>> d.unique().data
 [{'x':  ['a', 'b' ]}, {'y':  [1, 2 ]} ]
```

```python theme={null}
 >>>  # Dataset with a single column
 >>> Dataset( [{'value':  [1, 2, 3, 2, 1, 3 ]} ]).unique().data
 [{'value':  [1, 2, 3 ]} ]
```

> #### wide() → [Dataset](/en/latest/dataset#dataset-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Convert a long-format dataset (with row, key, value columns) to wide format.

> **Examples:**

```python theme={null}
 >>> d  = Dataset( [{'row':  [0, 0, 1, 1 ]}, {'key':  ['a', 'b', 'a', 'b' ]}, {'value':  [1, 3, 2, 4 ]} ])
 >>> d.wide().data
 [{'a':  [1, 2 ]}, {'b':  [3, 4 ]} ]
```

```python theme={null}
 >>> d  = Dataset( [{'row':  [0, 0, 1, 1 ]}, {'key':  ['x', 'y', 'x', 'y' ]}, {'value':  [1, 3, 2, 4 ]}, {'z':  [5, 5, 6, 6 ]} ])
 >>> d.wide().data
 [{'x':  [1, 2 ]}, {'y':  [3, 4 ]}, {'z':  [5, 6 ]} ]
```

> #### write(*filename: str*, *tablefmt: str | None  = None*) → None [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/dataset.py)

Write the dataset to a file in the specified format.

> **Args:**

filename: The name of the file to write to. tablefmt: Optional format for the table (e.g., ‘csv’, ‘html’, ‘latex’).


# Enums
Source: https://docs.expectedparrot.com/en/latest/enums

Enums module wrapper for backward compatibility.

This module re-exports everything from the edsl.base.enums module.

> #### *class* edsl.enums.EnumWithChecks(*value*, *names=None*, \*\*\*, *module=None*, *qualname=None*, *type=None*, *start=1*, *boundary=None*) [source](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

Bases: `Enum`

Base class for all enums with checks.

> #### **module** *= 'edsl.base.enums'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### *classmethod* is\_value\_valid(*value*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

Check if the value is valid.

> #### *class* edsl.enums.InferenceServiceType(*value*, *names=None*, \*\*\*, *module=None*, *qualname=None*, *type=None*, *start=1*, *boundary=None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

Bases: [`EnumWithChecks`](/en/latest "edsl.base.enums.EnumWithChecks")

Enum for the inference service types.

> #### ANTHROPIC *= 'anthropic'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### AZURE *= 'azure'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### BEDROCK *= 'bedrock'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### DEEPSEEK *= 'deepseek'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### DEEP\_INFRA *= 'deep\_infra'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### GOOGLE *= 'google'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### GROQ *= 'groq'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### MISTRAL *= 'mistral'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### OLLAMA *= 'ollama'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### OPENAI *= 'openai'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### OPENAI\_V2 *= 'openai\_v2'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### OPEN\_ROUTER *= 'open\_router'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### PERPLEXITY *= 'perplexity'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### REPLICATE *= 'replicate'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### TEST *= 'test'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### TOGETHER *= 'together'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### XAI *= 'xai'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### **module** *= 'edsl.base.enums'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### **new**(*value*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### *class* edsl.enums.QuestionType(*value*, *names=None*, \*\*\*, *module=None*, *qualname=None*, *type=None*, *start=1*, *boundary=None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

Bases: [`EnumWithChecks`](/en/latest "edsl.base.enums.EnumWithChecks")

Enum for the question types.

> #### BUDGET *= 'budget'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### CHECKBOX *= 'checkbox'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### EXTRACT *= 'extract'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### FREE\_TEXT *= 'free\_text'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### FUNCTIONAL *= 'functional'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### LIKERT\_FIVE *= 'likert\_five'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### LINEAR\_SCALE *= 'linear\_scale'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### LIST *= 'list'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### MULTIPLE\_CHOICE *= 'multiple\_choice'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### NUMERICAL *= 'numerical'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### RANK *= 'rank'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### TOP\_K *= 'top\_k'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### YES\_NO *= 'yes\_no'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### **module** *= 'edsl.base.enums'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### **new**(*value*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### *class* edsl.enums.TokenPricing(\*\*\*, *model\_name*, *prompt\_token\_price\_per\_k: float*, *completion\_token\_price\_per\_k: float*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

Bases: `object`

> #### **dict** *= mappingproxy(\{'**module**': 'edsl.base.enums', '**init**': \<function TokenPricing.**init**>, '**eq**': \<function TokenPricing.**eq**>, 'example': \<classmethod(\<function TokenPricing.example>)>, '**dict**': \<attribute '**dict**' of 'TokenPricing' objects>, '**weakref**': \<attribute '**weakref**' of 'TokenPricing' objects>, '**doc**': None, '**hash**': None, '**annotations**': })*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### **eq**(*other*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

Return self==value.

> #### **hash** *= None*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### **init**(\*\*\*, *model\_name*, *prompt\_token\_price\_per\_k: float*, *completion\_token\_price\_per\_k: float*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### **module** *= 'edsl.base.enums'*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

> #### **weakref**[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

list of weak references to the object

> #### *classmethod* example() → [TokenPricing](/en/latest/#edsl.enums.TokenPricing "edsl.base.enums.TokenPricing")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)

Return an example TokenPricing object.

> #### edsl.enums.get\_token\_pricing(*model\_name*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/enums.py)


# Exceptions & Debugging
Source: https://docs.expectedparrot.com/en/latest/exceptions

An exception is an error that occurs during the execution of a question or survey. When an exception is raised, EDSL will display a message about the error and an interactive report with more details in a new browser tab.

## Help debugging

If you would like help debugging an error that you are encountering, please feel free to share your code, objects and exceptions report with us.

An easy way to do this is to post a notebook with your code to the coop and **share the link with us at [info@expectedparrot.com](mailto:info@expectedparrot.com)**. You can use the following code to generate a link to your notebook:

```python theme={null}
from edsl import notebook

n = Notebook(path="path/to/your/notebook.ipynb")

n.push(description="Notebook with code that raises an exception", visibility="private")

```

## Common exceptions

### Answer validation errors

A number of exceptions may indicate that there is a problem with the way that a question has been constructed or answered. For example, you may intend for the answer to be formatted as a list but receive a string instead. Or a question may be unanswered and the model has returned None. These exceptions are typically raised by the Question class and are subclassed from QuestionAnswerValidationError.

A useful starting point for debugging these exceptions is to check the Settings class for the Questions model ([https://github.com/expectedparrot/edsl/blob/main/edsl/questions/settings.py](https://github.com/expectedparrot/edsl/blob/main/edsl/questions/settings.py)). The default settings (which can be modified) are as follows:

```text theme={null}
MAX_ANSWER_LENGTH = 2000
MAX_EXPRESSION_CONSTRAINT_LENGTH = 1000
MAX_NUM_OPTIONS = 200
MIN_NUM_OPTIONS = 1
MAX_OPTION_LENGTH = 10000
MAX_QUESTION_LENGTH = 100000
```

### Missing API key

You will receive a MissingAPIKeyError exception if you try to run a question and have not activated [Remote Inference](/en/latest/remote_inference) or stored an API key for the model that you are trying to use.

### Problem with inference provider

Some exceptions may indicate that there is a problem with your account with an inference provider, such as insufficient credits.

### API Timeout

The maximum number of seconds to wait for an API call to return can be specified in config.py:

```python theme={null}
CONFIG_MAP = {
    ...
    "EDSL_API_TIMEOUT": {
        "default": "60",
        "info": "This env var determines the maximum number of seconds to wait for an API call to return.",
    },
    ...
```

### Missing packages

A ModuleNotFoundError exception will be raised if a required package is not installed. This is more likely to occur when cloning the repository instead of installing the package using pip install edsl. It can typically be remedied by reinstalling your virtual environment or installing the missing package using pip install \<package\_name>.

## Strategies for dealing with exceptions

### Re-try the question

If an exception is raised, the question will be re-tried up to a maximum number of attempts. The number of retries can be specified in config.py:

```python theme={null}
CONFIG_MAP = {
    ...
    "EDSL_MAX_ATTEMPTS": {
        "default": "5",
        "info": "This env var determines the maximum number of times to retry a failed API call.",
    },
    ...
```

By default, your successful results are cached, so re-executing a survey will only re-run the questions that were not answered successfully in the previous run. Learn more about working with data.

### Search for the exception message in the documentation

The name of the exception that is raised will often provide a clue as to what the problem is. You can search for the exception type in the search bar at the top of the main documentation page to find more information about the exception and how to resolve it.


# File Store
Source: https://docs.expectedparrot.com/en/latest/filestore

*FileStore* is a module for storing and sharing data files at Expected Parrot. It allows you to post and retrieve files of various types to use in EDSL surveys, such as survey data, PDFs, CSVs, docs or images. It can also be used to create Scenario objects for questions or traits for Agent objects from data files stored on the platform.

When posting files, the FileStore module will automatically infer the file type from the extension. You can give a file a description and an alias, and set its visibility (public, private or unlisted).

<Note>
  **Note**:

  Scenarios created from FileStore objects cannot be used with question memory rules, and can only be added to questions with the by() method, not the loop() method. This is because the memory rules and loop() method insert the filepath in the question, whereas the by() method inserts the file content when the question is run. See details on these methods at the [Scenarios](/en/latest/scenarios) section of the documentation.
</Note>

The examples below are also available in a [notebook at Expected Parrot](https://www.expectedparrot.com/content/RobinHorton/my-example-filestore-notebook).

## File types

The following file types are currently supported by the FileStore module:

* docx (Word document)
* csv (Comma-separated values)
* html (HyperText Markup Language)
* json (JavaScript Object Notation)
* latex (LaTeX)
* md (Markdown)
* pdf (Portable Document Format)
* png (image)
* pptx (PowerPoint)
* py (Python)
* sql (SQL database)
* sqlite (SQLite database)
* txt (text)

## Posting a file

1. Import the FileStore constructor and create an object by passing the path to the file. The constructor will automatically infer the file type from the extension. For example:

```python theme={null}
from edsl import FileStore

fs = FileStore("my_data.csv") # replace with your own file
```

2. Call the push method to post the file at Expected Parrot. You can optionally pass the following parameters:

* description: a string description for the file
* alias: a convenient Pythonic reference name for the URL for the object, e.g., my\_example\_csv
* visibility: either *public*, *private* or *unlisted* (the default is *unlisted*)

```python theme={null}
fs.push(description = "My example CSV file", alias = "my-example-csv-file", visibility = "public")
```

The push method returns a dictionary with the following keys and values (this is the same for any object posted to Expected Parrot):

* description: the description you provided, if any
* object\_type: the type of object (e.g., scenario, survey, results, agent, notebook; objects posted with FileStore are always scenarios)
* url: the URL of the file at Expected Parrot
* uuid: the UUID of the file at Expected Parrot
* version: the version of the file
* visibility: the visibility of the file (*public*, *private* or *unlisted* by default)

Example output:

```json theme={null}
{'description': 'My example CSV file',
'object_type': 'scenario',
'url': 'https://www.expectedparrot.com/content/17c0e3d3-8d08-4ae0-bc7d-384a56a07e4e',
'uuid': '17c0e3d3-8d08-4ae0-bc7d-384a56a07e4e',
'version': '0.1.47.dev1',
'visibility': 'public'}
```

## Retrieving a file

To retrieve a file, call the pull method on the FileStore constructor and pass it the alias or UUID of the file that you want to retrieve. For the example above, we can retrieve the file with:

```python theme={null}
fs = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/my-example-csv-file")
```

This is equivalent:

```python theme={null}
fs = FileStore.pull(csv_info["uuid"])

```

Once retrieved, a file can be converted into scenarios. To construct a single scenario from a file, use the Scenario constructor and pass the file as a value for a specified key (see image file example below). To construct a list of scenarios from a file, call the from\_csv or from\_pdf method of the ScenarioList constructor and pass the file as an argument (see CSV and PDF examples below). To construct a list of scenarios from multiple files in a directory, you can use the ScenarioList.from\_directory() method, which will wrap each file in a Scenario with a specified key (default is “content”).

We can also create agents by calling the from\_csv() method on an AgentList object.

### CSV example

Here we create an example CSV and then post it to Expected Parrot using FileStore and retrieve it. Then we use the retrieved file to construct scenarios for questions (you can skip the step to create a CSV and replace with your own file).

To create an example CSV file:

```python theme={null}
# Sample data
data = [
    ['Age', 'City', 'Occupation'],
    [25, 'New York', 'Software Engineer'],
    [30, 'San Francisco', 'Teacher'],
    [35, 'Chicago', 'Doctor'],
    [28, 'Boston', 'Data Scientist'],
    [45, 'Seattle', 'Architect']
]

# Writing to CSV file
with open('data.csv', 'w') as file:
    for row in data:
        line = ','.join(str(item) for item in row)
        file.write(line + 'n')
```

Here we post the file to Expected Parrot and inspect the details:

```python theme={null}
from edsl import FileStore

fs = FileStore("data.csv")
csv_info = fs.push(description = "My example CSV file", alias = "my-example-csv-file", visibility = "public")
csv_info # display the URL and uuid of the stored file for retrieving it later
```

Example output:

```json theme={null}
{'description': 'My example CSV file',
'object_type': 'scenario',
'url': 'https://www.expectedparrot.com/content/17c0e3d3-8d08-4ae0-bc7d-384a56a07e4e',
'uuid': '17c0e3d3-8d08-4ae0-bc7d-384a56a07e4e',
'version': '0.1.47.dev1',
'visibility': 'public'}
```

Now we can retrieve the file and create scenarios from it:

```python theme={null}
fs = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/my-example-csv-file")

# or equivalently
fs = FileStore.pull(csv_info["uuid"])
```

Here we create a ScenarioList object from the CSV file:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_csv(fs.to_tempfile())
```

To inspect the scenarios:

```python theme={null}
scenarios # display the scenarios
```

Output:

| Age | City | Occupation |
| :- | :- | :- |
| 25 | New York | Software Engineer |
| 30 | San Francisco | Teacher |
| 35 | Chicago | Doctor |
| 28 | Boston | Data Scientist |
| 45 | Seattle | Architect |

Alternatively, we can create agents from the CSV file:

```python theme={null}
from edsl import AgentList

agents = AgentList.from_csv(fs.to_tempfile())
```

Learn more about designing agents and using scenarios in the [Agents](/en/latest/agents) and [Scenarios](/en/latest/scenarios) sections.

### PNG example

Here we post and retrieve an image file, and then create a scenario for it. Note that we need to specify the scenario key for the file when we create the scenario. We also need to ensure that we have specified a vision model when using it with a survey (e.g., *gpt-4o*).

To post the file:

```python theme={null}
from edsl import FileStore

fs = FileStore("parrot_logo.png") # replace with your own file
png_info = fs.push(description = "My example PNG file", alias = "my-example-png-file", visibility = "public")
png_info # display the URL and uuid of the stored file for retrieving it later
```

Example output:

```json theme={null}
{'description': 'My example PNG file',
'object_type': 'scenario',
'url': 'https://www.expectedparrot.com/content/b261660e-11a3-4bec-8864-0b6ec76dfbee',
'uuid': 'b261660e-11a3-4bec-8864-0b6ec76dfbee',
'version': '0.1.47.dev1',
'visibility': 'public'}
```

Here we retrieve the file and then create a Scenario object for it with a key for the placeholder in the questions where we want to use the image:

```python theme={null}
from edsl import FileStore

fs = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/my-example-png-file")

# or equivalently
fs = FileStore.pull(png_info["uuid"])
```

Here we create a Scenario object from the image file:

```python theme={null}
from edsl import Scenario

image_scenario = Scenario({"parrot_logo": fs}) # specify the key for the image

```

We can verify the key for the scenario object:

```python theme={null}
image_scenario.keys()
```

Output:

```python theme={null}
['parrot_logo']
```

To rename a key:

```python theme={null}
image_scenario = image_scenario.rename({"parrot_logo": "logo"}) # key = old name, value = new name
image_scenario.keys()
```

Output:

```python theme={null}
['logo']
```

To use it in a question, the question should be parameterized with the key:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "test",
    question_text = "Describe this logo: {{ logo }}"
)
```

Here we run the question with the scenario object. Note that we need to use a vision model; here we specify the default model for demonstration purposes and add an agent persona:

```python theme={null}
from edsl import Model

model = Model("gpt-4o") # specify a vision model

results = q.by(image_scenario).by(model).run() # run the question with the scenario and model
```

Learn more about selecting models in the [Language Models](/en/latest/language_models) section.

Output:

| model | scenario.logo | answer.test |
| :- | :- | :- |
| gpt-4 | FileStore: self.path | The logo features a large, stylized letter “E” in a serif font on the left. Next to it, within square brackets, is a colorful parrot. The parrot has a green body, an orange beak, a pink chest, blue lower body, and gray feet. The design combines a classic typographic element with a vibrant, playful illustration. |

### PDF example

Here we download an example PDF from the internet, post and retrieve it from Expected Parrot using FileStore and then convert it into a ScenarioList object with the from\_pdf() method. The default keys are filename, page, text, which can be modified with the rename method.

To download a PDF file:

```python theme={null}
import requests

url = "https://arxiv.org/pdf/2404.11794"
response = requests.get(url)
with open("automated_social_scientist.pdf", "wb") as file:
    file.write(response.content)
```

Here we post the file to Expected Parrot and inspect the details:

```python theme={null}
from edsl import FileStore

fs = FileStore("automated_social_scientist.pdf")
pdf_info = fs.push(description = "My example PDF file", visibility = "public")
pdf_info # display the URL and uuid of the stored file for retrieving it later
```

Example output:

```json theme={null}
{'description': 'My example PDF file',
'object_type': 'scenario',
'url': 'https://www.expectedparrot.com/content/e1770915-7e69-436d-b2ca-f0f92c6f56ba',
'uuid': 'e1770915-7e69-436d-b2ca-f0f92c6f56ba',
'version': '0.1.47.dev1',
'visibility': 'public'}
```

Now we retrieve the file and create a ScenarioList object from it:

```python theme={null}
from edsl import FileStore, ScenarioList

pdf_file = FileStore.pull(pdf_info["uuid"])

scenarios = ScenarioList.from_pdf(pdf_file.to_tempfile())
```

To inspect the keys:

```python theme={null}
scenarios.parameters
```

Output:

```python theme={null}
{'filename', 'page', 'text'}
```

Using the scenarios in a question:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "summary",
    question_text = "Summarize this page: {{ text }}"
)
```

Each result will contain the text from a page of the PDF file, together with columns for the filename and page number. Run results.columns to see all the components of results.

## FileStore class

> ### *class* edsl.scenarios.FileStore(*path: str | None = None*, *mime\_type: str | None = None*, *binary: bool | None = None*, *suffix: str | None = None*, *base64\_string: str | None = None*, *external\_locations: Dict\[str, str] | None = None*, *extracted\_text: str | None = None*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Bases: [`Scenario`](/en/latest/scenarios "edsl.scenarios.scenario.Scenario")

A specialized Scenario subclass for managing file content and metadata.

FileStore provides functionality for working with files in EDSL, handling various file formats with appropriate encoding, storage, and access methods. It extends Scenario to allow files to be included in surveys, questions, and other EDSL components.

FileStore supports multiple file formats including text, PDF, Word documents, images, and more. It can load files from local paths or URLs, and provides methods for accessing file content, extracting text, and managing file operations.

Key features: - Base64 encoding for portability and serialization - Lazy loading through temporary files when needed - Automatic MIME type detection - Text extraction from various file formats - Format-specific operations through specialized handlers

> **Attributes:**

\_path (str): The original file path. \_temp\_path (str): Path to any generated temporary file. suffix (str): File extension. binary (bool): Whether the file is binary. mime\_type (str): The file’s MIME type. base64\_string (str): Base64-encoded file content. external\_locations (dict): Dictionary of external locations. extracted\_text (str): Text extracted from the file.

> **Examples:**

```python theme={null}
>>> import tempfile
>>> # Create a text file
>>> with tempfile.NamedTemporaryFile(suffix=".txt", mode="w") as f:
...     _ = f.write("Hello World")
...     _ = f.flush()
...     fs = FileStore(f.name)
```

\# The following example works locally but is commented out for CI environments # where dependencies like pandoc may not be available: # >>> # FileStore supports various formats # >>> formats = \[“txt”, “pdf”, “docx”, “pptx”, “md”, “py”, “json”, “csv”, “html”, “png”, “db”] # >>> \_ = \[FileStore.example(format) for format in formats]

> #### **init**(*path: str | None = None*, *mime\_type: str | None = None*, *binary: bool | None = None*, *suffix: str | None = None*, *base64\_string: str | None = None*, *external\_locations: Dict\[str, str] | None = None*, *extracted\_text: str | None = None*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Initialize a new FileStore object.

This constructor creates a FileStore object from either a file path or a base64-encoded string representation of file content. It handles automatic detection of file properties like MIME type, extracts text content when possible, and manages file encoding.

> **Args:**

path: Path to the file to load. Can be a local file path or URL. mime\_type: MIME type of the file. If not provided, will be auto-detected. binary: Whether the file is binary. Defaults to False. suffix: File extension. If not provided, will be extracted from the path. base64\_string: Base64-encoded file content. If provided, the file content

will be loaded from this string instead of the path.

**external\_locations: Dictionary mapping location names to URLs or paths where**

the file can also be accessed.

**extracted\_text: Pre-extracted text content from the file. If not provided,**

text will be extracted automatically if possible.

[\*\*](/en/latest/#id1)**kwargs: Additional keyword arguments. ‘filename’ can be used as an**

alternative to ‘path’.

<Note>
  **Note:**

  If path is a URL (starts with [http://](http://) or [https://](https://)), the file will be downloaded automatically.

  *async* async\_upload\_google(*refresh: bool = False*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

  Async version of upload\_google that avoids blocking the event loop.
</Note>

This method uploads a file to Google’s Generative AI service asynchronously, polls for activation status with exponential backoff, and returns the file info.

**Args:**

refresh: If True, force re-upload even if already uploaded

**Returns:**

Dictionary containing the Google file information

**Raises:**

Exception: If upload fails or file activation fails

> #### *static* base64\_to\_file(*base64\_string*, *is\_binary=True*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### *static* base64\_to\_text\_file(*base64\_string*) → IO[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### *classmethod* batch\_screenshots(*urls: List\[str]*, \*\**kwargs*) → [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Take screenshots of multiple URLs concurrently. Args:

urls: List of URLs to screenshot [\*\*](/en/latest/#id3)kwargs: Additional arguments passed to screenshot function (full\_page, wait\_until, etc.)

**Returns:**

ScenarioList containing FileStore objects with their corresponding URLs

> #### create\_link(*custom\_filename=None*, *style=None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### download\_link()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### encode\_file\_to\_base64\_string(*file\_path: str*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### *classmethod* example(*example\_type='txt'*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Returns an example Scenario instance.

> **Args:**

**randomize: If True, adds a random string to the value of the example key**

to ensure uniqueness.

> **Returns:**

A Scenario instance with example data suitable for testing or demonstration.

> **Examples:**

```python theme={null}
>>> s = Scenario.example()
>>> 'persona' in s
True
>>> s1 = Scenario.example(randomize=True)
>>> s2 = Scenario.example(randomize=True)
>>> s1.data != s2.data  # Should be different due to randomization
True
```

> #### extract\_text() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### *classmethod* from\_dict(*d*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Creates a Scenario from a dictionary, with special handling for FileStore objects.

This method creates a Scenario using the provided dictionary. It has special handling for dictionary values that represent serialized FileStore objects, which it will deserialize back into proper FileStore instances.

> **Args:**

d: A dictionary to convert to a Scenario.

> **Returns:**

A new Scenario containing the provided dictionary data.

> **Examples:**

```python theme={null}
>>> Scenario.from_dict({"food": "wood chips"})
Scenario({'food': 'wood chips'})
```

```python theme={null}
>>> # Example with a serialized FileStore
>>> from edsl import FileStore
>>> file_dict = {"path": "example.txt", "base64_string": "SGVsbG8gV29ybGQ="}
>>> s = Scenario.from_dict({"document": file_dict})
>>> isinstance(s["document"], FileStore)
True
```

> **Notes:**

* Any dictionary values that match the FileStore format will be converted to FileStore objects
* The method detects FileStore objects by looking for “base64\_string” and “path” keys
* EDSL version information is automatically removed by the @remove\_edsl\_version decorator
* This method is commonly used when deserializing scenarios from JSON or other formats

> #### *classmethod* from\_url(*url: str*, *download\_path: str | None = None*, *mime\_type: str | None = None*) → [FileStore](/en/latest/#edsl.scenarios.FileStore "edsl.scenarios.file_store.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> **Parameters:**

* **url** – The URL of the file to download.
* **download\_path** – The path to save the downloaded file.
* **mime\_type** – The MIME type of the file. If None, it will be guessed from the file extension.

> #### *classmethod* from\_url\_screenshot(*url: str*, \*\**kwargs*) → [FileStore](/en/latest/#edsl.scenarios.FileStore "edsl.scenarios.file_store.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Synchronous wrapper for screenshot functionality

> #### get\_image\_dimensions() → tuple[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Get the dimensions (width, height) of an image file.

> **Returns:**

tuple: A tuple containing the width and height of the image.

> **Raises:**

ValueError: If the file is not an image or PIL is not installed.

> **Examples:**

```python theme={null}

>>> fs = FileStore.example("png")
>>> width, height = fs.get_image_dimensions()
>>> isinstance(width, int) and isinstance(height, int)
True
```

> #### get\_video\_metadata() → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Get metadata about a video file such as duration, dimensions, codec, etc. Uses FFmpeg to extract the information if available.

> **Returns:**

dict: A dictionary containing video metadata, or a dictionary with

error information if metadata extraction fails.

> **Raises:**

ValueError: If the file is not a video.

> **Examples:**

```python theme={null}

>>> fs = FileStore.example("mp4")
>>> metadata = fs.get_video_metadata()
>>> isinstance(metadata, dict)
True
```

> #### is\_image() → bool[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Check if the file is an image by examining its MIME type.

> **Returns:**

bool: True if the file is an image, False otherwise.

> **Examples:**

```python theme={null}

>>> fs = FileStore.example("png")
>>> fs.is_image()
True
>>> fs = FileStore.example("txt")
>>> fs.is_image()
False
```

> #### is\_video() → bool[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Check if the file is a video by examining its MIME type.

> **Returns:**

bool: True if the file is a video, False otherwise.

> **Examples:**

```python theme={null}

>>> fs = FileStore.example("mp4")
>>> fs.is_video()
True
>>> fs = FileStore.example("webm")
>>> fs.is_video()
True
>>> fs = FileStore.example("txt")
>>> fs.is_video()
False

offload(*inplace=False*) → [FileStore](/en/latest/#edsl.scenarios.FileStore "edsl.scenarios.file_store.FileStore")[[source]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)
```

> #### Offloads base64-encoded content from the FileStore by replacing ‘base64\_string’ with ‘offloaded’. This reduces memory usage.

> **Args:**

inplace (bool): If True, modify the current FileStore. If False, return a new one.

> **Returns:**

FileStore: The modified FileStore (either self or a new instance).

> #### open() → IO[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### *property* path\*: str\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Returns a valid path to the file content, creating a temporary file if needed.

This property ensures that a valid file path is always available for the file content, even if the original file is no longer accessible or if the FileStore was created from a base64 string without a path. If the original path doesn’t exist, it automatically generates a temporary file from the base64 content.

> **Returns:**

A string containing a valid file path to access the file content.

> **Examples:**

```python theme={null}

>>> import tempfile, os
>>> with tempfile.NamedTemporaryFile(suffix=".txt", mode="w") as f:
...     _ = f.write("Hello World")
...     _ = f.flush()
...     fs = FileStore(f.name)
...     os.path.isfile(fs.path)
True
```

> **Notes:**

* The path may point to a temporary file that will be cleaned up when the Python process exits
* Accessing this property may create a new temporary file if needed
* This property provides a consistent interface regardless of how the FileStore was created (from file or from base64 string)

> #### *classmethod* pull(*url\_or\_uuid: str | UUID*) → [FileStore](/en/latest/#edsl.scenarios.FileStore "edsl.scenarios.file_store.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Pull a FileStore object from Expected Parrot.

**Args:**

url\_or\_uuid: Either a UUID string or a URL pointing to the object expected\_parrot\_url: Optional URL for the Parrot server

**Returns:**

FileStore: The pulled FileStore object

> #### push(*description: str | None = None*, *alias: str | None = None*, *visibility: str | None = 'unlisted'*, *expected\_parrot\_url: str | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Push the object to Expected Parrot. :param description: The description of the object to push. :param visibility: The visibility of the object to push.

> #### save\_to\_gcs\_bucket(*signed\_url: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Saves the FileStore’s file content to a Google Cloud Storage bucket using a signed URL.

**Args:**

signed\_url (str): The signed URL for uploading to GCS bucket

**Returns:**

dict: Response from the GCS upload operation

**Raises:**

ValueError: If base64\_string is offloaded or missing requests.RequestException: If the upload fails

> #### *property* size\*: int\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### *property* text[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### to\_pandas()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the file content to a pandas DataFrame if supported by the file handler.

**Returns:**

pandas.DataFrame: The data from the file as a DataFrame

**Raises:**

AttributeError: If the file type’s handler doesn’t support pandas conversion

> #### to\_scenario(*key\_name: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### to\_tempfile(*suffix=None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### upload\_google(*refresh: bool = False*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### view() → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Display an interactive visualization of this object.

**Returns:**

The result of the dataset’s view method

> #### write(*filename: str | None = None*) → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Write the file content to disk, either to a specified filename or a temporary file.

**Args:**

filename (Optional\[str]): The destination filename. If None, creates a temporary file.

**Returns:**

str: The path to the written file.


# Firecrawl Integration
Source: https://docs.expectedparrot.com/en/latest/firecrawl

The EDSL Firecrawl integration provides seamless access to the Firecrawl web scraping platform, allowing you to scrape, crawl, search, and extract structured data from web content directly into EDSL Scenarios and ScenarioLists.

## Features

* **Scrape**: Extract clean content from single URLs or batches of URLs
* **Crawl**: Comprehensively crawl entire websites and extract content from all pages
* **Search**: Perform web searches and scrape content from search results
* **Map URLs**: Fast URL discovery without full content extraction
* **Extract**: AI-powered structured data extraction using schemas or natural language prompts

All methods return EDSL Scenario or ScenarioList objects, making web data immediately ready for survey research, analysis, and other EDSL workflows.

## Installation

Install the required dependencies:

```bash theme={null}
pip install firecrawl-py python-dotenv
```

## Setup

Get your Firecrawl API key from [https://firecrawl.dev](https://firecrawl.dev/) and set it as an environment variable:

```bash theme={null}
export FIRECRAWL_API_KEY=your_api_key_here
```

Or add it to a .env file in your project:

```bash theme={null}
FIRECRAWL_API_KEY=your_api_key_here
```

## Basic Usage

### Quick Start Examples

**Scraping a single URL:**

```python theme={null}
from edsl.scenarios.firecrawl_scenario import scrape_url

# Scrape a single URL
result = scrape_url("https://example.com")
print(result["content"])  # Scraped markdown content
print(result["title"])    # Page title
```

**Scraping multiple URLs:**

```python theme={null}
from edsl.scenarios.firecrawl_scenario import scrape_url

# Scrape multiple URLs
urls = ["https://example.com", "https://example.org"]
results = scrape_url(urls, max_concurrent=5)

for result in results:
    print(f"URL: {result['url']}")
    print(f"Title: {result['title']}")
    print(f"Content: {result['content'][:100]}...")
```

**Web search with content extraction:**

```python theme={null}
from edsl.scenarios.firecrawl_scenario import search_web

# Search and extract content from results
results = search_web("python web scraping tutorials")

for result in results:
    print(f"Title: {result['title']}")
    print(f"URL: {result['url']}")
    print(f"Content: {result['content'][:100]}...")
```

**Crawling an entire website:**

```python theme={null}
from edsl.scenarios.firecrawl_scenario import crawl_website

# Crawl a website with limits
results = crawl_website(
    "https://docs.example.com",
    limit=50,                     # Max 50 pages
    max_depth=3,                  # Max depth of 3
    include_paths=["/docs/*"]     # Only crawl documentation
)

print(f"Crawled {len(results)} pages")
```

**Structured data extraction:**

```python theme={null}
from edsl.scenarios.firecrawl_scenario import extract_data

# Define what data to extract
schema = {
    "title": "string",
    "price": "number",
    "description": "string",
    "availability": "boolean"
}

# Extract structured data
result = extract_data("https://shop.example.com/product", schema=schema)
extracted = result["extracted_data"]
print(f"Product: {extracted['title']}")
print(f"Price: ${extracted['price']}")
```

## Class-Based API

For more control, use the class-based API:

```python theme={null}
### FirecrawlScenario Class

from edsl.scenarios.firecrawl_scenario import FirecrawlScenario

# Initialize with API key (optional if set in environment)
firecrawl = FirecrawlScenario(api_key="your_key_here")

# Use any method
result = firecrawl.scrape("https://example.com")
```

### FirecrawlRequest Class

For distributed processing or when you need to separate request creation from execution:

```python theme={null}
from edsl.scenarios.firecrawl_scenario import FirecrawlRequest, FirecrawlScenario

# Create request without executing (useful for distributed systems)
request = FirecrawlRequest(api_key="your_key")
request_dict = request.scrape("https://example.com")

# Execute the request later (potentially on a different machine)
result = FirecrawlScenario.from_request(request_dict)
```

## Detailed Method Documentation

### Scraping Methods

\*\*scrape\_url(url\_or\_urls, max\_concurrent=10, **kwargs)**

Scrape content from one or more URLs.

**Parameters:**

* `url_or_urls`: Single URL string or list of URLs
* `max_concurrent`: Maximum concurrent requests for batch processing (default: 10)
* `formats`: List of output formats (default: \[“markdown”])
* `only_main_content`: Extract only main content, skip navigation/ads (default: True)
* `include_tags`: HTML tags to specifically include
* `exclude_tags`: HTML tags to exclude
* `headers`: Custom HTTP headers as dictionary
* `wait_for`: Time to wait before scraping (milliseconds)
* `timeout`: Request timeout (milliseconds)
* `actions`: Browser actions to perform before scraping

**Returns:**

* Single URL: Scenario object with scraped content
* Multiple URLs: ScenarioList with Scenario objects

\*\*crawl\_website(url, **kwargs)**

Crawl an entire website and extract content from all discovered pages.

**Parameters:**

* `url`: Base URL to start crawling from
* `limit`: Maximum number of pages to crawl
* `max_depth`: Maximum crawl depth from starting URL
* `include_paths`: URL path patterns to include (supports wildcards)
* `exclude_paths`: URL path patterns to exclude
* `formats`: Output formats for each page (default: \[“markdown”])
* `only_main_content`: Extract only main content (default: True)

**Returns:**

ScenarioList containing Scenario objects for each crawled page

### Search Methods

\*\*search\_web(query\_or\_queries, max\_concurrent=5, **kwargs)**

Search the web and extract content from results.

**Parameters:**

* `query_or_queries`: Single search query or list of queries
* `max_concurrent`: Maximum concurrent requests for batch processing (default: 5)
* `limit`: Maximum number of search results per query
* `sources`: Sources to search (e.g., \[“web”, “news”, “images”])
* `location`: Geographic location for localized results
* `formats`: Output formats for scraped content from results

**Returns:**

ScenarioList containing Scenario objects for each search result

\*\*map\_website\_urls(url, **kwargs)**

Discover and map all URLs from a website without scraping content (fast URL discovery).

**Parameters:**

* `url`: Base URL to discover links from
* Additional mapping parameters via kwargs

**Returns:**

ScenarioList containing Scenario objects for each discovered URL

### Extraction Methods

\*\*extract\_data(url\_or\_urls, schema=None, prompt=None, **kwargs)**

Extract structured data from web pages using AI-powered analysis.

**Parameters:**

* `url_or_urls`: Single URL string or list of URLs
* `schema`: JSON schema defining data structure to extract
* `prompt`: Natural language description of what to extract
* `max_concurrent`: Maximum concurrent requests for batch processing (default: 5)
* `formats`: Output formats for scraped content before extraction

**Returns:**

* Single URL: Scenario object with extracted structured data
* Multiple URLs: ScenarioList with Scenario objects

Note: Either `schema` or `prompt` should be provided. Schema takes precedence if both are given.

## Working with Results

### Scenario Fields

All methods return Scenario or ScenarioList objects with standardized fields:

**Common fields for all methods:**

* `url`: The scraped/crawled/searched URL
* `title`: Page title (when available)
* `description`: Page description (when available)
* `content`: Primary content (usually markdown format)
* `status_code`: HTTP status code (when available)

**Scraping-specific fields:**

* `scrape_status`: “success” or “error”
* `markdown`: Markdown content
* `html`: HTML content (if requested)
* `links`: Extracted links (if requested)
* `screenshot`: Screenshot data (if requested)
* `metadata`: Full page metadata

**Search-specific fields:**

* `search_query`: The original search query
* `search_status`: “success” or “error”
* `result_type`: “web”, “news”, or “image”
* `position`: Result position in search results

**Extraction-specific fields:**

* `extract_status`: “success” or “error”
* `extracted_data`: The structured data extracted by AI
* `extraction_prompt`: The prompt used (if any)
* `extraction_schema`: The schema used (if any)

**Crawl-specific fields:**

* `crawl_status`: “success” or “error”

**URL mapping fields:**

* `discovered_url`: The discovered URL
* `source_url`: The URL it was discovered from
* `map_status`: “success” or “error”

## Advanced Usage

### Concurrent Processing

All batch methods support concurrent processing to speed up large operations:

```python theme={null}
# Scrape 100 URLs with 20 concurrent requests
results = scrape_url(urls, max_concurrent=20)

# Search multiple queries concurrently
queries = ["AI research", "machine learning", "data science"]
results = search_web(queries, max_concurrent=3)
```

### Custom Formats and Options

Specify different output formats and extraction options:

```python theme={null}
# Get both markdown and HTML content
result = scrape_url(
    "https://example.com",
    formats=["markdown", "html"],
    only_main_content=False,
    include_tags=["article", "main"],
    exclude_tags=["nav", "footer"]
)

# Access different formats
print(result["markdown"])  # Markdown content
print(result["html"])      # HTML content
```

### Complex Crawling Scenarios

Advanced crawling with path filtering:

```python theme={null}
# Crawl documentation site with specific constraints
results = crawl_website(
    "https://docs.example.com",
    limit=100,
    max_depth=3,
    include_paths=["/docs/*", "/api/*", "/tutorials/*"],
    exclude_paths=["/docs/deprecated/*", "/docs/v1/*"],
    formats=["markdown", "html"]
)

# Filter results by content type
api_docs = [r for r in results if "/api/" in r["url"]]
tutorials = [r for r in results if "/tutorials/" in r["url"]]
```

### Schema-Based vs Prompt-Based Extraction

**Using JSON schemas for structured extraction:**

```python theme={null}
# Define precise data structure
product_schema = {
    "name": "string",
    "price": "number",
    "rating": "number",
    "availability": "boolean",
    "features": ["string"],
    "specifications": {
        "dimensions": "string",
        "weight": "string",
        "color": "string"
    }
}

result = extract_data("https://shop.example.com/product", schema=product_schema)
product_data = result["extracted_data"]
```

**Using natural language prompts:**

```python theme={null}
# Extract with natural language
result = extract_data(
    "https://news.example.com/article",
    prompt="Extract the article headline, author name, publication date, and key topics discussed"
)

article_data = result["extracted_data"]
print(f"Headline: {article_data['headline']}")
print(f"Author: {article_data['author']}")
```

## Error Handling

The integration handles errors gracefully by returning Scenario objects with error information:

```python theme={null}
results = scrape_url(["https://valid-url.com", "https://invalid-url.com"])

for result in results:
    if result.get("scrape_status") == "error":
        print(f"Error scraping {result['url']}: {result['error']}")
    else:
        print(f"Successfully scraped {result['url']}")
```

## Integration with EDSL Workflows

The firecrawl integration seamlessly integrates with other EDSL components:

### Using with Surveys

```python expandable theme={null}
from edsl import QuestionFreeText, Survey
from edsl.scenarios.firecrawl_scenario import scrape_url

# Scrape content to create scenarios
scenarios = scrape_url([
    "https://news1.com/article1",
    "https://news2.com/article2"
])

# Create survey questions about the content
q1 = QuestionFreeText(
    question_name="summary",
    question_text="Summarize the main points of this article: {{ content }}"
)

q2 = QuestionFreeText(
    question_name="sentiment",
    question_text="What is the overall sentiment of this article?"
)

survey = Survey(questions=[q1, q2])

# Run survey with scraped content as scenarios
results = survey.by(scenarios).run()
```

### Content Analysis Pipeline

```python expandable theme={null}
from edsl.scenarios.firecrawl_scenario import search_web, extract_data

# 1. Search for relevant content
search_results = search_web("climate change research 2024", limit=10)

# 2. Extract structured data from search results
extraction_schema = {
    "findings": "string",
    "methodology": "string",
    "publication_date": "string",
    "authors": ["string"]
}

urls = [result["url"] for result in search_results if result["search_status"] == "success"]
extracted_data = extract_data(urls, schema=extraction_schema)

# 3. Use in EDSL survey for analysis
survey = Survey([
    QuestionFreeText(
        question_name="significance",
        question_text="Rate the significance of these findings: {{ extracted_data }}"
    )
])

analysis_results = survey.by(extracted_data).run()
```

## Distributed Processing

For large-scale operations, use the request/response pattern for distributed processing:

```python expandable theme={null}
from edsl.scenarios.firecrawl_scenario import (
    create_scrape_request,
    create_search_request,
    execute_request
)

# Create serializable requests (can be sent to workers, APIs, etc.)
requests = [
    create_scrape_request("https://example1.com"),
    create_scrape_request("https://example2.com"),
    create_search_request("machine learning tutorials")
]

# Execute requests (potentially on different machines/processes)
results = []
for request in requests:
    result = execute_request(request)
    results.append(result)
```

## Best Practices

### Performance Optimization

1. **Use appropriate concurrency limits** - Start with defaults and adjust based on your needs and rate limits
2. **Filter early** - Use include/exclude paths in crawling to avoid unnecessary requests
3. **Choose optimal formats** - Only request formats you actually need
4. **Batch operations** - Process multiple URLs/queries together when possible

### Rate Limiting

1. **Respect rate limits** - Firecrawl has rate limits based on your plan
2. **Adjust concurrency** - Lower max\_concurrent values if you hit rate limits
3. **Monitor costs** - Each request consumes Firecrawl credits

### Content Quality

1. **Use only\_main\_content=True** - Filters out navigation, ads, and other noise
2. **Specify include/exclude tags** - Fine-tune content extraction
3. **Choose appropriate formats** - Markdown for text analysis, HTML for detailed parsing

### Error Resilience

1. **Check status fields** - Always check scrape\_status, search\_status, etc.
2. **Handle partial failures** - Some URLs in a batch may fail while others succeed
3. **Implement retries** - For critical operations, implement retry logic for failed requests

### Cost Management

1. **Use URL mapping first** - Discover URLs with map\_website\_urls before full crawling
2. **Set reasonable limits** - Use limit and max\_depth parameters to control crawl scope
3. **Cache results** - Store results locally to avoid re-scraping the same content

## Troubleshooting

### Common Issues

**“FIRECRAWL\_API\_KEY not found” error:**

* Ensure your API key is set as an environment variable or in a .env file
* Verify the key is valid and active at [https://firecrawl.dev](https://firecrawl.dev/)

**Rate limit errors:**

* Reduce max\_concurrent parameter
* Check your Firecrawl plan limits
* Implement delays between requests if needed

**Empty or poor quality content:**

* Try only\_main\_content=False for more content
* Adjust include\_tags/exclude\_tags parameters
* Some sites may have anti-scraping measures

**Slow performance:**

* Increase max\_concurrent for batch operations
* Use URL mapping to discover URLs faster than full crawling
* Consider using search instead of crawling for content discovery

**Memory usage with large crawls:**

* Use limit parameter to control crawl size
* Process results in batches rather than storing everything in memory
* Consider using the request/response pattern for distributed processing

### Getting Help

* Check the [Firecrawl documentation](https://docs.firecrawl.dev/) for API-specific issues
* Review EDSL documentation for Scenario and ScenarioList usage
* Ensure your firecrawl-py package is up to date: `pip install --upgrade firecrawl-py`

## API Reference

### Convenience Functions

> #### edsl.scenarios.firecrawl\_scenario.scrape\_url(*url\_or\_urls: str | List\[str]*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Convenience function to scrape single URL or multiple URLs.

> #### edsl.scenarios.firecrawl\_scenario.crawl\_website(*url: str*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Convenience function to crawl a website.

> #### edsl.scenarios.firecrawl\_scenario.search\_web(*query\_or\_queries: str | List\[str]*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Convenience function to search the web with single query or multiple queries.

> #### edsl.scenarios.firecrawl\_scenario.map\_website\_urls(*url: str*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Convenience function to map website URLs.

> #### edsl.scenarios.firecrawl\_scenario.extract\_data(*url\_or\_urls: str | List\[str]*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Convenience function to extract structured data from single URL or multiple URLs.

### Classes

> #### *class* edsl.scenarios.firecrawl\_scenario.FirecrawlScenario(*api\_key: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

EDSL integration for Firecrawl web scraping and data extraction.

This class provides methods to use all Firecrawl features and return results as EDSL Scenario and ScenarioList objects.

> ##### crawl(*url: str*, *limit: int | None = 10*, *max\_depth: int | None = 3*, *include\_paths: List\[str] | None = None*, *exclude\_paths: List\[str] | None = None*, *formats: List\[str] | None = None*, *only\_main\_content: bool = True*, *scrape\_options: Dict\[str, Any] | None = None*, *return\_credits: bool = False*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Crawl a website and return a ScenarioList with all pages.

**Args:**

url: Base URL to crawl limit: Maximum number of pages to crawl max\_depth: Maximum crawl depth (now max\_discovery\_depth) include\_paths: URL patterns to include exclude\_paths: URL patterns to exclude formats: List of formats to return for each page only\_main\_content: Whether to extract only main content [\*\*](/en/latest/#id1)kwargs: Additional parameters passed to Firecrawl

**Returns:**

ScenarioList containing scenarios for each crawled page

> ##### extract(*url\_or\_urls: str | List\[str]*, *schema: Dict\[str, Any] | None = None*, *prompt: str | None = None*, *max\_concurrent: int = 5*, *limit: int | None = None*, *scrape\_options: Dict\[str, Any] | None = None*, *return\_credits: bool = False*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Smart extract method that handles both single URLs and batches.

**Args:**

url\_or\_urls: Single URL string or list of URLs schema: JSON schema for structured extraction prompt: Natural language prompt for extraction max\_concurrent: Maximum concurrent requests for batch processing [\*\*](/en/latest/#id3)kwargs: Additional parameters passed to extract method

**Returns:**

Scenario object for single URL, ScenarioList for multiple URLs, or (object, credits) if return\_credits=True

> #### *classmethod* from\_request(*request\_dict: Dict\[str, Any]*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Execute a serialized Firecrawl request and return results.

This method allows for distributed processing where requests are serialized, sent to an API, reconstituted here, executed, and results returned.

**Args:**

request\_dict: Dictionary containing the serialized request

**Returns:**

Scenario or ScenarioList depending on the method and input

> #### map\_urls(*url: str*, *limit: int | None = None*, *return\_credits: bool = False*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Get all URLs from a website (fast URL discovery).

> **Args:**

url: Website URL to map limit: Maximum number of URLs to discover and map. If None, discovers all

> available linked URLs. Defaults to None.

> \*\* [\*\*](/en/latest/#id5)kwargs: Additional parameters passed to Firecrawl

**Returns:**

ScenarioList containing scenarios for each discovered URL

> ##### search(*query\_or\_queries: str | List\[str]*, *max\_concurrent: int = 5*, *limit: int | None = None*, *sources: List\[str] | None = None*, *location: str | None = None*, *scrape\_options: Dict\[str, Any] | None = None*, *return\_credits: bool = False*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Smart search method that handles both single queries and batches.

**Args:**

query\_or\_queries: Single query string or list of queries max\_concurrent: Maximum concurrent requests for batch processing [\*\*](/en/latest/#id9)kwargs: Additional parameters passed to search method

**Returns:**

ScenarioList for both single and multiple queries, or (ScenarioList, credits) if return\_credits=True

> #### *class* edsl.scenarios.firecrawl\_scenario.FirecrawlRequest(*api\_key: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Firecrawl request class that can operate in two modes:

1. With API key present: Automatically executes requests via FirecrawlScenario and returns results
2. Without API key: Can be used as a descriptor/placeholder, raises exception when methods are called

This supports both direct execution and distributed processing patterns.

> ##### crawl(*url: str*, *limit: int | None = 10*, *max\_depth: int | None = 3*, *include\_paths: List\[str] | None = None*, *exclude\_paths: List\[str] | None = None*, *formats: List\[str] | None = None*, *only\_main\_content: bool = True*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Crawl an entire website and extract content from all discovered pages.

This method performs comprehensive website crawling, discovering and scraping content from multiple pages within a website. When an API key is present, it executes the crawl and returns a ScenarioList with all pages. Without an API key, it raises an exception.

> **Args:**

**url: Base URL to start crawling from. Should be a valid HTTP/HTTPS URL.**

The crawler will discover and follow links from this starting point.

**limit: Maximum number of pages to crawl. If None, crawls all discoverable**

pages (subject to other constraints). Use this to control crawl scope.

**max\_depth: Maximum crawl depth from the starting URL. Depth 0 is just the**

starting page, depth 1 includes pages directly linked from the start, etc. If None, no depth limit is imposed.

**include\_paths: List of URL path patterns to include in the crawl. Only URLs**

matching these patterns will be crawled. Supports wildcard patterns.

**exclude\_paths: List of URL path patterns to exclude from the crawl. URLs**

matching these patterns will be skipped. Applied after include\_paths.

**formats: List of output formats for each crawled page (e.g., \[“markdown”, “html”]).**

Defaults to \[“markdown”] if not specified.

only\_main\_content: Whether to extract only main content from each page,

skipping navigation, ads, footers, etc. Defaults to True.

[\*\*](/en/latest/#id11)kwargs: Additional crawling parameters passed to the Firecrawl API.

> **Returns:**

**When API key is present:**

ScenarioList containing Scenario objects for each crawled page. Each scenario includes the page content, URL, title, and metadata.

When API key is missing: Raises ValueError

> **Raises:**

ValueError: If FIRECRAWL\_API\_KEY is not found in environment, parameters,

or .env file.

> **Examples:**

> **Basic website crawl:**

```python theme={null}
>>> firecrawl = FirecrawlRequest(api_key="your_key")
>>> results = firecrawl.crawl("https://example.com")
>>> print(f"Crawled {len(results)} pages")
>>> for page in results:
...     print(f"Page: {page['url']} - {page['title']}")
```

> **Limited crawl with constraints:**

```python theme={null}
>>> results = firecrawl.crawl(
...     "https://docs.example.com",
...     limit=50,
...     max_depth=3,
...     include_paths=["/docs/*", "/api/*"],
...     exclude_paths=["/docs/deprecated/*"]
... )
```

> **Full content crawl with multiple formats:**

```python theme={null}
>>> results = firecrawl.crawl(
...     "https://blog.example.com",
...     formats=["markdown", "html"],
...     only_main_content=False,
...     limit=100
... )
>>> for post in results:
...     print(f"Title: {post['title']}")
...     print(f"Content length: {len(post['content'])}")
```

> #### extract(*url\_or\_urls: str | List\[str]*, *schema: Dict\[str, Any] | None = None*, *prompt: str | None = None*, *max\_concurrent: int = 5*, *limit: int | None = None*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Extract structured data from web pages using AI-powered analysis.

This method uses AI to extract specific information from web pages based on either a JSON schema or natural language prompt. When an API key is present, it executes the extraction and returns structured data. Without an API key, it raises an exception.

> **Args:**

**url\_or\_urls: Single URL string or list of URLs to extract data from.**

Each URL should be a valid HTTP/HTTPS URL.

**schema: JSON schema defining the structure of data to extract. Should be**

a dictionary with field names and their types/descriptions. Takes precedence over prompt if both are provided.

**prompt: Natural language description of what data to extract. Used when**

schema is not provided. Should be clear and specific.

**max\_concurrent: Maximum number of concurrent requests when processing multiple**

URLs. Only applies when url\_or\_urls is a list. Defaults to 5.

**limit: Maximum number of URLs to extract data from when url\_or\_urls is a list.**

If None, extracts from all provided URLs. Defaults to None.

[\*\*](/en/latest/#id13)**kwargs: Additional extraction parameters:**

**formats: List of output formats for the scraped content before extraction**

(e.g., \[“markdown”, “html”]).

> **Returns:**

**When API key is present:**

* For single URL: Scenario object containing extracted structured data
* For multiple URLs: ScenarioList containing Scenario objects

Each result includes the extracted data in the ‘extracted\_data’ field.

When API key is missing: Raises ValueError

> **Raises:**

ValueError: If FIRECRAWL\_API\_KEY is not found in environment, parameters,

or .env file.

> **Examples:**

> **Schema-based extraction:**

```python theme={null}
>>> schema = {
...     "title": "string",
...     "price": "number",
...     "description": "string",
...     "availability": "boolean"
... }
>>> firecrawl = FirecrawlRequest(api_key="your_key")
>>> result = firecrawl.extract("https://shop.example.com/product", schema=schema)
>>> print(result["extracted_data"]["title"])
```

> **Prompt-based extraction:**

```python theme={null}
>>> result = firecrawl.extract(
...     "https://news.example.com/article",
...     prompt="Extract the article headline, author, and publication date"
... )
>>> print(result["extracted_data"])
```

> **Multiple URLs with schema:**

```python theme={null}
>>> urls = ["https://shop1.com/item1", "https://shop2.com/item2"]
>>> results = firecrawl.extract(
...     urls,
...     schema={"name": "string", "price": "number"},
...     max_concurrent=3,
...     limit=2
... )
>>> for result in results:
...     data = result["extracted_data"]
...     print(f"{data['name']}: ${data['price']}")
```

> #### map\_urls(*url: str*, *limit: int | None = None*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Discover and map all URLs from a website without scraping content.

This method performs fast URL discovery to map the structure of a website without downloading and processing the full content of each page. When an API key is present, it executes the mapping and returns discovered URLs. Without an API key, it raises an exception.

> **Args:**

**url: Base URL to discover links from. Should be a valid HTTP/HTTPS URL.**

The mapper will analyze this page and discover all linked URLs.

**limit: Maximum number of URLs to discover and map. If None, discovers all**

available linked URLs. Defaults to None.

[\*\*](/en/latest/#id15)**kwargs: Additional mapping parameters passed to the Firecrawl API.**

Common options may include depth limits or filtering criteria.

> **Returns:**

**When API key is present:**

ScenarioList containing Scenario objects for each discovered URL. Each scenario includes the discovered URL, source URL, and any available metadata (title, description) without full content.

When API key is missing: Raises ValueError

> **Raises:**

ValueError: If FIRECRAWL\_API\_KEY is not found in environment, parameters,

or .env file.

> **Examples:**

> **Basic URL mapping:**

```python theme={null}
>>> firecrawl = FirecrawlRequest(api_key="your_key")
>>> urls = firecrawl.map_urls("https://example.com")
>>> print(f"Discovered {len(urls)} URLs")
>>> for url_info in urls:
...     print(f"URL: {url_info['discovered_url']}")
...     if 'title' in url_info:
...         print(f"Title: {url_info['title']}")
```

> **Website structure analysis:**

```python theme={null}
>>> urls = firecrawl.map_urls("https://docs.example.com", limit=100)
>>> # Group URLs by path pattern
>>> doc_urls = [u for u in urls if '/docs/' in u['discovered_url']]
>>> api_urls = [u for u in urls if '/api/' in u['discovered_url']]
>>> print(f"Documentation pages: {len(doc_urls)}")
>>> print(f"API reference pages: {len(api_urls)}")
```

> **Link discovery for targeted crawling:**

```python theme={null}
>>> # First map URLs to understand site structure
>>> all_urls = firecrawl.map_urls("https://blog.example.com")
>>> # Then crawl specific sections
>>> blog_posts = [u['discovered_url'] for u in all_urls
...               if '/posts/' in u['discovered_url']]
>>> # Use discovered URLs for targeted scraping
>>> content = firecrawl.scrape(blog_posts[:10])
```

> #### scrape(*url\_or\_urls: str | List\[str]*, *max\_concurrent: int = 10*, *limit: int | None = None*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Scrape content from one or more URLs using Firecrawl.

This method extracts clean, structured content from web pages. When an API key is present, it automatically executes the request and returns Scenario objects. Without an API key, it raises an exception when called.

> **Args:**

**url\_or\_urls: Single URL string or list of URLs to scrape. Each URL should**

be a valid HTTP/HTTPS URL.

**max\_concurrent: Maximum number of concurrent requests when scraping multiple**

URLs. Only applies when url\_or\_urls is a list. Defaults to 10.

**limit: Maximum number of URLs to scrape when url\_or\_urls is a list. If None,**

scrapes all provided URLs. Defaults to None.

[\*\*](/en/latest/#id17)**kwargs: Additional scraping parameters:**

**formats: List of output formats (e.g., \[“markdown”, “html”, “links”]).**

Defaults to \[“markdown”].

**only\_main\_content: Whether to extract only main content, skipping**

navigation, ads, etc. Defaults to True.

include\_tags: List of HTML tags to specifically include in extraction. exclude\_tags: List of HTML tags to exclude from extraction. headers: Custom HTTP headers as a dictionary. wait\_for: Time to wait before scraping in milliseconds. timeout: Request timeout in milliseconds. actions: List of browser actions to perform before scraping

(e.g., clicking, scrolling).

> **Returns:**

**When API key is present:**

* For single URL: Scenario object containing scraped content
* For multiple URLs: ScenarioList containing Scenario objects

When API key is missing: Raises ValueError

> **Raises:**

ValueError: If FIRECRAWL\_API\_KEY is not found in environment, parameters,

or .env file.

> **Examples:**

> **Basic scraping:**

```python theme={null}
>>> firecrawl = FirecrawlRequest(api_key="your_key")
>>> result = firecrawl.scrape("https://example.com")
>>> print(result["content"])  # Scraped markdown content
```

> **Multiple URLs with custom options:**

```python theme={null}
>>> urls = ["https://example.com", "https://example.org"]
>>> results = firecrawl.scrape(
...     urls,
...     max_concurrent=5,
...     limit=2,
...     formats=["markdown", "html"],
...     only_main_content=False
... )
```

> **Descriptor pattern (no API key):**

```python theme={null}
>>> class MyClass:
...     scraper = FirecrawlRequest()  # No exception here
>>> # my_instance.scraper.scrape("url")  # Would raise ValueError
```

> #### search(*query\_or\_queries: str | List\[str]*, *max\_concurrent: int = 5*, *limit: int | None = None*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Search the web and extract content from results using Firecrawl.

This method performs web searches and automatically scrapes content from the search results. When an API key is present, it executes the search and returns ScenarioList objects with the results. Without an API key, it raises an exception.

> **Args:**

> **query\_or\_queries: Single search query string or list of queries to search for.**

Each query should be a natural language search term.

> **max\_concurrent: Maximum number of concurrent requests when processing multiple**

queries. Only applies when query\_or\_queries is a list. Defaults to 5.

> **limit: Maximum number of search results to return per query. If None, returns**

all available results. Defaults to None.

> [\*\*](/en/latest/#id19)**kwargs: Additional search parameters:**

**sources: List of sources to search (e.g., \[“web”, “news”, “images”]).**

Defaults to \[“web”].
location: Geographic location for localized search results. formats: List of output formats for scraped content from results

(e.g., \[“markdown”, “html”]).

> **Returns:**

**When API key is present:**

ScenarioList containing Scenario objects for each search result. Each scenario includes search metadata (query, position, result type) and scraped content from the result URL.

When API key is missing: Raises ValueError

> **Raises:**

**ValueError: If FIRECRAWL\_API\_KEY is not found in environment, parameters,**

or .env file.

> **Examples:**

> **Basic web search:**

```python theme={null}
>>> firecrawl = FirecrawlRequest(api_key="your_key")
>>> results = firecrawl.search("python web scraping")
>>> for result in results:
...     print(f"Title: {result['title']}")
...     print(f"Content: {result['content'][:100]}...")
```

Multiple queries with options:

```python theme={null}
>>> queries = ["AI research", "machine learning trends"]
>>> results = firecrawl.search(
...     queries,
...     limit=5,
...     sources=["web", "news"],
...     location="US"
... )
```

News search with custom formatting:

```python theme={null}
>>> results = firecrawl.search(
...     "climate change news",
...     sources=["news"],
...     formats=["markdown", "html"]
... )
```

### Request Creation Functions

> #### edsl.scenarios.firecrawl\_scenario.create\_scrape\_request(*url\_or\_urls: str | List\[str]*, *api\_key: str | None = None*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Create a serializable scrape request.

> #### edsl.scenarios.firecrawl\_scenario.create\_search\_request(*query\_or\_queries: str | List\[str]*, *api\_key: str | None = None*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Create a serializable search request.

> #### edsl.scenarios.firecrawl\_scenario.create\_extract\_request(*url\_or\_urls: str | List\[str]*, *api\_key: str | None = None*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Create a serializable extract request.

> #### edsl.scenarios.firecrawl\_scenario.create\_crawl\_request(*url: str*, *api\_key: str | None = None*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Create a serializable crawl request.

> #### edsl.scenarios.firecrawl\_scenario.create\_map\_request(*url: str*, *api\_key: str | None = None*, \*\**kwargs*) → Dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Create a serializable map\_urls request.

> #### edsl.scenarios.firecrawl\_scenario.execute\_request(*request\_dict: Dict\[str, Any]*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios/firecrawl_scenario.py)

Execute a serialized Firecrawl request and return results.


# Getting Started
Source: https://docs.expectedparrot.com/en/latest/getting_started

To use EDSL, you need to install the package and choose how to access language models.

## 1. Installation

Run the following command to install the EDSL package:

```bash theme={null}
! uv pip install edsl -q
```

If you have previously installed EDSL, you can check which version you have by running:

```bash theme={null}
pip show edsl
```

(Compare it to the version shown at [PyPI](https://pypi.org/project/edsl/).) If it is out of date you can update it with the following command:

```bash theme={null}
pip install --upgrade edsl
```

You can also access the source code at [GitHub](https://github.com/expectedparrot/edsl). If you are using EDSL with Google Colab, please see the [Colab Setup](/en/latest/colab_setup) section for special instructions.

See [Installation](/en/latest/installation) instructions for more details and troubleshooting tips.

## 2. Create an account

[Sign up](https://www.expectedparrot.com/login) for a free account with your email address. This allows you to run surveys remotely and collaborate with other users. Your account comes with \$25 in credits for API calls to any available models.

Run the following code to log in to your account and store your Expected Parrot key:

```python theme={null}
from edsl import login
login()
```

## 3. Choose where to run surveys

You can use EDSL to run surveys locally on your own machine or remotely at the Expected Parrot server. To activate or deactivate remote inference, navigate to the [Settings](https://www.expectedparrot.com/home/settings) page of your account and toggle *remote inference* on or off.

Remote inference allows you to use remote caching features and automatically store all your survey results. See the [Remote Inference](/en/latest/remote_inference) and [Remote Caching](/en/latest/remote_caching) sections for details.

To run surveys locally you must provide your own keys from service providers. See the next step below for details.

## 4. Manage API keys for language models

API keys are required to run surveys with language models. EDSL works with many language models from popular service providers, including Anthropic, Azure, Bedrock, DeepInfra, DeepSeek, Google, Mistral, OpenAI, Perplexity, Together and Xai. Current model pricing and performance information is available [here](https://www.expectedparrot.com/models).

When you run a survey you can choose whether to use your own keys from service providers and/or an **Expected Parrot API key** to access all available models at once. There are several ways to manage your keys:

* Manage keys at your Expected Parrot account (*recommended*)
* Store keys in a .env file on your computer
* Store keys directly in your EDSL code (*not recommended for security reasons*)

If you have your own keys from service providers you can add them to your [Keys](https://www.expectedparrot.com/home/keys) page. You can also choose whether to grant access to other users and set usage limits.

If you are running surveys locally, or only want to post and retrieve content at Expected Parrot without running your surveys remotely, you can store your keys in a .env file on your computer. This file should be in the same directory as your EDSL code and contain the following lines:

```
EXPECTED_PARROT_API_KEY=your_expected_parrot_key # required for remote inference and caching and posting content at Expected Parrot

ANTHROPIC_API_KEY=your_anthropic_key
GOOGLE_API_KEY=your_google_key
OPENAI_API_KEY=your_openai_key
# etc.
```

See the [Managing Keys](/en/latest/api_keys) section for details on this method of managing keys.

<Note>
  **Note**:

  Note that if you use your own keys, you will be charged directly by the service provider for API calls. If you use your Expected Parrot API key, credits are deducted from your account based on the model and number of tokens used in the survey.
</Note>

## 5. Run a survey

Read the [Starter Tutorial](/en/latest/starter_tutorial) and [download a notebook](https://www.expectedparrot.com/content/179b3a78-2505-4568-acd9-c09d18953288) to create a survey and run it. See examples for many other use cases and [tips](/en/latest/checklist) on using EDSL effectively in the documentation.

## Support

If you have any questions or need help, please send a message to [info@expectedparrot.com](mailto:info@expectedparrot.com). You can also [open at issue at GitHub](https://github.com/expectedparrot/edsl/issues/new?template=Blank+issue) to report bugs or request new features.

Please also join our [Discord channel](https://discord.com/invite/mxAYkjfy9m) to ask questions and chat with other users!


# Humanize
Source: https://docs.expectedparrot.com/en/latest/humanize

EDSL comes with built-in methods for generating web versions of your surveys and collecting and analyzing responses from humans.

<Note>
  This page covers code methods for launching surveys, gathering human responses, and analyzing results. For interactive survey building, see [Survey Builder](/en/latest/survey_builder). For Prolific recruitment, see [Prolific studies](/en/latest/prolific). For email notifications and scheduled reminders, see [Survey notifications](/en/latest/humanize_notifications). To let AI agents take a survey, see [AI agent access](/en/latest/humanize_agent_access).
</Note>

## How it works

<Steps>
  <Step>
    **Create** [Surveys](/en/latest/surveys) with desired types of [Questions](/en/latest/questions).
  </Step>

  <Step>
    **Humanize** the survey to generate a web-hosted version on Expected Parrot with a respondent link and an admin dashboard.
  </Step>

  <Step>
    **Share** the respondent link with human participants (or recruit via [Prolific](/en/latest/prolific)).
  </Step>

  <Step>
    **Collect** responses back as a standard EDSL `Results` object.
  </Step>

  <Step>
    **Analyze** your human results alongside any LLM results.
  </Step>
</Steps>

***

## Basic usage

```python theme={null}
from edsl import Survey, QuestionMultipleChoice, QuestionFreeText, QuestionLinearScale

q1 = QuestionMultipleChoice(
    question_name="frequency",
    question_text="How often do you work from home?",
    question_options=["Never", "1-2 days a week", "3-4 days a week", "Every day"],
)
q2 = QuestionLinearScale(
    question_name="satisfaction",
    question_text="How satisfied are you with your current work arrangement?",
    question_options=[1, 2, 3, 4, 5],
    option_labels={1: "Very dissatisfied", 5: "Very satisfied"},
)
q3 = QuestionFreeText(
    question_name="improvement",
    question_text="What one change would most improve your day-to-day work experience?",
)

survey = Survey([q1, q2, q3])

info = survey.humanize(
    human_survey_name="Workplace Experience Survey",
    survey_description="A short survey about remote and in-office work preferences.",
    survey_visibility="unlisted",
)

print(info["respondent_url"])  # share with respondents
print(info["admin_url"])       # monitor responses
print(info["uuid"])            # save this to retrieve results later
```

### `.humanize()` parameters

| Parameter | Type | Default | Description |
| - | - | - | - |
| `human_survey_name` | str | `"New survey"` | Display name on Coop |
| `survey_description` | str \| None | None | Optional description |
| `survey_alias` | str \| None | None | URL-friendly alias |
| `survey_visibility` | str | `"private"` | `"private"`, `"public"`, or `"unlisted"` |
| `humanize_schema` | dict \| None | None | Controls question styling and optionality — see [Humanize schema](/en/latest/humanize_schema) |
| `delivery_map` | DeliveryMap \| None | None | Maps delivery channels to agent trait columns; required for email notifications |

### Return value

`survey.humanize()` returns a `Scenario` (dict-like) with these keys:

| Key | Description |
| - | - |
| `uuid` | Human survey UUID — needed to fetch responses later |
| `respondent_url` | Public link for respondents |
| `admin_url` | Admin dashboard link |
| `name` | Survey name |
| `n_responses` | Response count (0 at creation) |
| `survey_uuid` | UUID of the pushed survey object |
| `scenario_list_uuid` | UUID of the pushed scenario list (if any) |
| `asset_substitutions` | Old asset uuid → new one, when the schema named assets that were copied into your library. Empty when nothing was copied |

**Save the `uuid`** — you need it to retrieve responses.

***

## Previewing before deployment

Use `.preview()` to get a preview URL without creating a live survey:

```python theme={null}
preview_url = survey.preview()
print(preview_url)

# With schema customization
preview_url = survey.preview(humanize_schema=schema)
```

***

## Present questions in groups

To request one group per page, define contiguous question groups on the survey and pass `humanize_schema={"survey": {"presentation": "group"}}` to `.humanize()`. See [Grouped surveys](/en/latest/humanize_groups) for a complete example, branching behavior, and availability requirements.

***

## Add your logo

To show your own logo above the questions, upload the image once and reference it from the humanize schema:

```python theme={null}
from edsl import Coop

logo = Coop().upload_human_survey_asset("lab_logo.png")

survey.humanize(
    human_survey_name="Customer feedback",
    humanize_schema={
        "survey": {
            "branding": {
                "logo": {
                    "source": {"type": "asset", "asset_uuid": logo.uuid},
                    "alt": "Acme Research logo",
                }
            }
        }
    },
)
```

Uploading the same file again returns the asset you already have, so re-running a script does not fill your library with copies.

From the CLI, upload once and apply it to a survey that already exists:

```bash theme={null}
ep humanize assets upload lab_logo.png
ep humanize schema set <human-survey-uuid> --logo-asset <asset-uuid> --logo-alt "Acme Research logo"
```

Or do both in one step:

```bash theme={null}
ep humanize schema set <human-survey-uuid> --logo-file lab_logo.png --logo-alt "Acme Research logo"
```

See [Humanize schema](/en/latest/humanize_schema) for the `branding` fields, and [Custom CSS](/en/latest/custom_css) for resizing the logo or putting it on a colored banner.

***

## With scenarios

When using scenarios, call `.humanize()` on a `Jobs` object (not directly on the survey). You must specify a `scenario_list_method`.

This example shows each respondent a different policy description and asks for their reaction:

```python theme={null}
from edsl import Survey, QuestionFreeText, QuestionLinearScale, Scenario, ScenarioList

scenarios = ScenarioList([
    Scenario({"policy": "Four-day workweek with no change in pay"}),
    Scenario({"policy": "Flexible start times between 7am and 11am"}),
    Scenario({"policy": "Monthly remote-work stipend of $100"}),
])

q1 = QuestionLinearScale(
    question_name="support",
    question_text="How much do you support this policy: '{{ scenario.policy }}'?",
    question_options=[1, 2, 3, 4, 5],
    option_labels={1: "Strongly oppose", 5: "Strongly support"},
)
q2 = QuestionFreeText(
    question_name="reasoning",
    question_text="What is the main reason for your rating?",
)

survey = Survey([q1, q2])

info = survey.by(scenarios).humanize(
    human_survey_name="Workplace Policy Reactions",
    scenario_list_method="randomize",
)
```

### Scenario list methods

| Method | Behavior |
| - | - |
| `"randomize"` | Each respondent gets a random scenario (with replacement) |
| `"ordered"` | Scenarios are assigned sequentially to respondents |
| `"loop"` | Survey is expanded — every question is repeated for every scenario |
| `"single_scenario"` | Exactly one scenario is used for all respondents (list must have length 1) |

**Rules:**

* If you attach scenarios, you **must** specify `scenario_list_method`.
* If you specify a method, you **must** attach scenarios.
* Agents and models are **not supported** with humanize — the survey goes to real humans.

***

## With agent lists (for email notifications)

Attach an `AgentList` via `.by()` to associate respondents with email addresses. Pass a `DeliveryMap` to tell Coop which agent trait holds the email address.

```python theme={null}
from edsl import Survey, Agent, AgentList
from edsl.coop.coop_humanize_notifications import ChannelConfig, DeliveryMap

agents = AgentList([
    Agent(traits={"name": "Jordan", "email": "jordan@example.com"}),
    Agent(traits={"name": "Taylor", "email": "taylor@example.com"}),
    Agent(traits={"name": "Morgan", "email": "morgan@example.com"}),
])

info = survey.by(agents).humanize(
    human_survey_name="Panel Study — Wave 1",
    agent_list_description="Recruited panel participants",
    delivery_map=DeliveryMap(email=ChannelConfig(col_name="email")),
)
```

`col_name` must match the trait key that holds each respondent's email address. Once deployed, use [Survey notifications](/en/latest/humanize_notifications) to send invitations, schedule reminders, and track delivery status.

***

## Retrieving responses

```python theme={null}
from edsl import Coop

coop = Coop()

results = coop.get_human_survey_responses("your-human-survey-uuid")
results.select("answer.frequency", "answer.satisfaction", "answer.improvement").print()
```

### Checking response count

```python theme={null}
status = coop.get_human_survey("your-human-survey-uuid")
print(f"Responses received: {status['n_responses']}")
```

### Working with file upload responses

If your survey includes a `QuestionFileUpload` question, each response's answer for that question is a list of file-info dicts. Use `FileStore.from_file_upload_answer` (single file) or `FileStoreList.from_file_upload_answers` (all files at once) to download and wrap them:

```python theme={null}
from edsl import Coop
from edsl.scenarios import FileStore, FileStoreList

coop = Coop()

results = coop.get_human_survey_responses("your-human-survey-uuid")

# Each answer is a list of uploaded file dicts
uploaded_files = results[0]["answer"]["my_upload_question"]

# Download all files as a FileStoreList
fsl = FileStoreList.from_file_upload_answers(uploaded_files)

# Or download a single file
fs = FileStore.from_file_upload_answer(uploaded_files[0])
```

The returned `FileStore` and `FileStoreList` objects behave like any other — you can inspect, filter, or pass them into further EDSL workflows.

***

## QR code

Generate a QR code for the respondent link using `get_human_survey_qr_code()`. Requires the `qrcode` library (`pip install "edsl[full]"` or `pip install "qrcode[pil]"`).

```python theme={null}
from edsl import Coop

coop = Coop()

qr = coop.get_human_survey_qr_code("your-human-survey-uuid")
qr.save("qr_code.png")  # save to file
```

From the CLI, generate a QR code for the shared respondent URL:

```bash theme={null}
ep humanize qr <human-survey-uuid> --output survey-qr.png
```

When the human survey has an attached agent list, export the roster together
with each respondent's personal link:

```bash theme={null}
ep humanize links <human-survey-uuid> --output respondent-links.csv
```

Add `--qr-dir` to generate one personal QR code per respondent and record each
PNG path in the CSV:

```bash theme={null}
ep humanize links <human-survey-uuid> \
  --output respondent-links.csv \
  --qr-dir respondent-qr
```

Personal links and QR codes contain access tokens and act as bearer credentials.
Store the export securely and share each link or code only with its intended
respondent. The CLI returns only file metadata on stdout, never the links.

***

## Prolific integration

For paid recruitment via Prolific, see the [Prolific studies](/en/latest/prolific) page. The short version:

```python theme={null}
from edsl import Coop

coop = Coop()

info = survey.humanize(human_survey_name="Workplace Experience Survey")

study = coop.create_prolific_study(
    human_survey_uuid=info["uuid"],
    name="Workplace Experience Study",
    description="A short survey about remote and in-office work preferences. Takes about 3 minutes.",
    num_participants=200,
    estimated_completion_time_minutes=3,
    participant_payment_cents=75,
)

# create_prolific_study leaves the study in DRAFT mode — publish to make it live
coop.publish_prolific_study(human_survey_uuid=info["uuid"])
```


# AI agent access
Source: https://docs.expectedparrot.com/en/latest/humanize_agent_access

Let AI agents take a human survey through an agent link, and control who answers and what guidance they get.

A human survey can let AI agents take it: a participant hands their AI agent the survey's agent link, and the agent answers through Expected Parrot's API instead of a browser. Agent access is off until the survey's owner turns it on, and only the owner can read or change it.

The agent link is the respondent link with `/respond/human-surveys/` replaced by `/respond/agent/`:

```text theme={null}
https://www.expectedparrot.com/respond/agent/<human-survey-uuid>
```

It serves the agent its instructions. Each agent attempt is recorded as a response, alongside browser responses.

***

## The config

| Field | Type | Default | Description |
| - | - | - | - |
| `enabled` | bool | `false` | Whether the agent link accepts new attempts. |
| `participation_mode` | string | `"human_assisted"` | Who produces the answers (see below). |
| `instructions` | string \| null | `null` | Guidance for agents on the whole survey, at most 4,000 characters. In autonomous mode, this is how the agent should answer. Advisory: nothing enforces it. |
| `question_settings` | dict | `{}` | Per-question settings, keyed by question name. Each may have `instructions` (at most 2,000 characters). |

### Participation modes

| Mode | Who answers |
| - | - |
| `human_assisted` | The participant. The agent asks each question and records their answers, and doesn't answer for them. |
| `authorized_context` | The agent, from context the participant has authorized it to use, asking them whenever it's unsure. |
| `autonomous` | The agent alone, as described in `instructions`. |

In `autonomous` mode there's no participant, so:

* The agent can't hand a page to a browser (see [What agents can't answer](#what-agents-cant-answer)).
* A start with Prolific parameters is refused: autonomous responses can't be used for a paid panel.

***

## Reading and changing the config

```python theme={null}
from edsl import Coop

coop = Coop()

access = coop.get_human_survey_agent_access("your-human-survey-uuid")
# {"configured": True, "enabled": True, "participation_mode": "human_assisted",
#  "instructions": None, "question_settings": {}}
```

`configured` is `False` if agent access has never been set; the other fields are then their defaults.

To change it, send only the fields you want to change:

```python theme={null}
coop.patch_human_survey_agent_access(
    "your-human-survey-uuid",
    {
        "enabled": True,
        "participation_mode": "autonomous",
        "instructions": (
            "Keep free-text answers to one or two sentences, and use the "
            "comment box to flag any answer that's an estimate."
        ),
        "question_settings": {
            "improvements": {
                "instructions": "Name at least one specific change, not a general comment."
            }
        },
    },
    survey=survey,  # optional: check question names against the survey
)
```

The patch is merged into the stored config:

* Fields left out are unchanged.
* `question_settings` merges by question name: naming `improvements` changes only its settings.
* `"instructions": None` clears the survey-wide instructions.
* A question set to `None` in `question_settings` has its settings removed, e.g. `{"question_settings": {"improvements": None}}`.

It returns the stored config, in the same shape as `get_human_survey_agent_access`.

From the CLI, `ep humanize agent-access get <uuid>` prints the config, and `patch` changes it:

```bash theme={null}
ep humanize agent-access patch <human-survey-uuid> --enabled --participation_mode autonomous
ep humanize agent-access patch <human-survey-uuid> --question_instructions "improvements=Name at least one specific change, not a general comment." --survey survey.ep
```

`patch` also takes the patch as a JSON file with `--config`, and `--survey` checks question names as `survey=` does.

***

## Validation

EDSL checks a patch before sending it, and raises `AgentAccessValidationError` if:

* It has a key the config doesn't define.
* A value is the wrong type or out of range, or `participation_mode` isn't one of the three modes.
* `enabled`, `participation_mode` or `question_settings` is `None`. Only `instructions` and a single question's settings can be cleared that way.
* With `survey=` (or `--survey`), it gives settings to a question that isn't in the survey.

***

## What agents can't answer

Some questions need the participant themselves:

* **Voice-only interviews.** In `human_assisted` and `authorized_context` mode, the agent hands the page to the participant: it gives them a one-time link to answer that page in their browser, then carries on. In `autonomous` mode there's no one to hand it to, so the survey can't be completed by an agent.
* **Timed questions.** A question with a fixed time limit, in a survey that shows one question per page (where the browser runs the timer), can't be answered by an agent: a time limit can't be enforced over the API. The agent link tells the agent to stop and give the participant the browser link instead.

Text interviews and file uploads are answered by the agent itself.

***

## In results

Every response carries these fields, so you can tell agent responses apart:

| Field | Description |
| - | - |
| `agent.collection_interface` | `browser`, or `agent_http` for an agent response. |
| `agent.participation_mode` | The survey's mode when the attempt started. Empty for browser responses. |
| `agent.client_name_self_reported` | The agent's client, as the agent reported it (e.g. `Claude Code`). Not verified. |
| `agent.model_name_self_reported` | The agent's model, as the agent reported it. Not verified. |
| `agent.handed_off_questions` | The questions the participant answered in their browser through a handoff. Empty for browser responses. |

The survey's page on Expected Parrot also has an **AI agents** tab showing the current config.


# Grouped surveys
Source: https://docs.expectedparrot.com/en/latest/humanize_groups

Define question groups and request one group per page in a human survey.

Grouped presentation lets you put related questions on the same page. Define the page boundaries on your `Survey`, then pass `{"survey": {"presentation": "group"}}` to `.humanize()`.

<Note>
  This guide describes the API introduced in [PR #2623](https://github.com/expectedparrot/edsl/pull/2623). It requires an EDSL version containing that change and a hosted Humanize implementation that supports grouped presentation. The PR adds the schema option and navigation API; it does not include the hosted UI implementation.
</Note>

## Create a two-page survey

This example puts two background questions on the first page and two feedback questions on the second.

```python theme={null}
from edsl import Survey, QuestionFreeText, QuestionMultipleChoice

survey = Survey([
    QuestionMultipleChoice(
        question_name="work_location",
        question_text="Where do you usually work?",
        question_options=["At home", "In an office", "A mix of both"],
    ),
    QuestionFreeText(
        question_name="role",
        question_text="What is your role?",
    ),
    QuestionMultipleChoice(
        question_name="satisfaction",
        question_text="How satisfied are you with your work arrangement?",
        question_options=["Satisfied", "Neutral", "Dissatisfied"],
    ),
    QuestionFreeText(
        question_name="improvement",
        question_text="What would improve your work arrangement?",
    ),
])

# First question, last question (inclusive), group name.
survey.add_question_group("work_location", "role", "background")
survey.add_question_group("satisfaction", "improvement", "feedback")

humanize_schema = {
    "survey": {"presentation": "group"},
    "questions": {"improvement": {"optional": True}},
}
```

| Page | Group name | Questions |
| - | - | - |
| 1 | `background` | `work_location`, `role` |
| 2 | `feedback` | `satisfaction`, `improvement` |

Preview with the same schema you intend to use when creating the survey:

```python theme={null}
preview_url = survey.preview(humanize_schema=humanize_schema)
print(preview_url)
```

When ready, create the human survey:

```python theme={null}
info = survey.humanize(
    human_survey_name="Workplace experience",
    humanize_schema=humanize_schema,
)
print(info["respondent_url"])
print(info["admin_url"])
```

The groups live on the survey; the schema selects how to present them. Adding groups alone does not enable grouped presentation. Omit `presentation`, or set it to `"question"`, for the default question-by-question mode.

## Set group boundaries

`survey.add_question_group(start_question, end_question, group_name)` accepts question names or question objects and returns the survey for chaining.

* Each group includes both endpoints and every question between them.
* Groups must not overlap. A one-question page uses the same question for both endpoints.
* Names must be unique Python identifiers, such as `background`, and must not match a question name.
* Pages follow question order in the survey.
* Assign every question you want displayed to a group. With groups defined, the current grouped navigator skips questions outside those groups.

For example, to make a page containing only `work_location`, define its group with `survey.add_question_group("work_location", "work_location", "location_page")` on a survey where that question is not already grouped.

## Plan branching around pages

The navigator uses the answers supplied when a page is requested to filter skipped questions. After a page is submitted, it applies the normal navigation rules from that group's **last renderable question** to choose the next page. Put page-level jump or stop rules on that question.

If a later question depends on an earlier answer, put the dependent question on a later page so that answer is available when the page is selected. The navigation API does not establish live, within-page updates to conditional questions.

For example, starting with the two-page survey above, omit the improvement question for satisfied respondents:

```python theme={null}
# Keep the controlling answer on an earlier page.
survey.question_groups.clear()
survey.add_question_group("work_location", "satisfaction", "background")
survey.add_question_group("improvement", "improvement", "improvements")
survey.add_skip_rule(
    "improvement",
    "{{ satisfaction.answer }} == 'Satisfied'",
)
```

Now the first page collects work location, role, and satisfaction. A satisfied respondent skips the improvements page and reaches the end; other respondents continue to it. Add the rule before previewing or creating the human survey.

Additional navigation behavior:

* A group with no remaining questions is skipped.
* A jump into the middle of a group starts at the target question; earlier questions in that group are omitted.
* Relevant instructions before or within the selected group are included by default.

## Inspect navigation locally

Survey authors normally define groups and call `.humanize()`. For debugging or implementing a renderer, `survey.next_group()` returns the next page without creating a hosted survey.

Using the original two-page example, before the branching modification:

```python theme={null}
first_page = survey.next_group()
assert first_page["group_name"] == "background"
assert first_page["question_names"] == ["work_location", "role"]

second_page = survey.next_group(
    current_group="background",
    answers={
        "work_location.answer": "At home",
        "role.answer": "Researcher",
    },
)
assert second_page["group_name"] == "feedback"
```

Pass all answers collected so far, including those from the just-submitted group. Use `include_instructions=False` to return questions only.

| Returned field | Meaning |
| - | - |
| `group_name` | Selected group name, or `None` at the end |
| `items` | Question and instruction objects; `[EndOfSurvey]` at the end |
| `question_names` | Names of the questions selected for this page |
| `is_end` | `True` when the survey has finished |

See [Humanize](/en/latest/humanize) for retrieving responses and [Humanize schema](/en/latest/humanize_schema) for styling and other presentation options.


# Survey notifications
Source: https://docs.expectedparrot.com/en/latest/humanize_notifications

Send emails, schedule deliveries, and react to respondent events for human surveys.

Human surveys support email delivery to respondents, scheduled send campaigns, and event-triggered callbacks. All of these are managed through `HumanSurveyNotificationHandler`, a convenience wrapper that binds to a specific human survey so you don't have to pass the UUID on every call.

<Note>
  The survey must have an agent list with an email delivery channel configured before any notifications can be sent. Set this up when calling `survey.humanize()` by passing `agent_list` and `delivery_map` arguments.
</Note>

## Getting a handler

```python theme={null}
from edsl.coop import HumanSurveyNotificationHandler

handler = HumanSurveyNotificationHandler("your-human-survey-uuid")
```

A `Coop` client is created automatically. Pass one explicitly if you need a specific API key or base URL:

```python theme={null}
from edsl import Coop
from edsl.coop import HumanSurveyNotificationHandler

coop = Coop()
handler = HumanSurveyNotificationHandler("your-human-survey-uuid", coop=coop)
```

***

## Statuses

Every delivery task moves through two independent status tracks.

### Dispatch status

Controls whether the platform has attempted to hand off the message to the downstream channel (e.g. the email provider).

| Value | Meaning |
| - | - |
| `pending` | The task has been created but not yet processed. |
| `dispatched` | The message was successfully handed off to the channel. |
| `failed` | The dispatch attempt failed (e.g. missing address, provider error). |

### Delivery status

Reflects what the email provider reports back after dispatch.

| Value | Meaning |
| - | - |
| `pending` | Waiting for a delivery event from the provider. |
| `sent` | The provider accepted the message for delivery. |
| `delivered` | The provider confirmed the message reached the recipient. |
| `bounced` | The message was rejected by the recipient's mail server. |
| `failed` | The provider reported an error sending the email. |

### Response status

Tracks whether a respondent has engaged with the survey itself.

| Value | Meaning |
| - | - |
| `not_started` | The respondent has not opened the survey link. |
| `completed` | The respondent submitted a complete response. |

***

## Quickstart examples

### Send an email right now

Trigger an immediate delivery to all respondents.

```python theme={null}
result = handler.send_respondent_email(name="Initial outreach")
print(result)
# {"delivery_uuid": "...", "routes": [...]}
```

Pass a custom subject, HTML email body, or restrict which respondents receive it:

```python theme={null}
from edsl.coop.coop_humanize_notifications import (
    HumanizeRespondentFilter,
    RespondentCondition,
    ResponseStatus,
)

# Only email respondents who haven't started yet
condition = RespondentCondition(
    response_status=[ResponseStatus.not_started]
)
respondent_filter = HumanizeRespondentFilter(conditions=[condition])

result = handler.send_respondent_email(
    name="Reminder to non-starters",
    subject="Don't forget to complete our survey",
    delivery_template="<p>Hi! Just a reminder to complete our survey.</p>",
    respondent_filter=respondent_filter,
)
```

***

### Schedule a one-time email

Schedule a single delivery at a specific time. `run_at` must be a timezone-aware datetime or an ISO 8601 string.

```python theme={null}
from datetime import datetime, timezone

run_at = datetime(2026, 6, 1, 9, 0, tzinfo=timezone.utc)

schedule = handler.create_one_time_schedule(
    name="June 1st reminder",
    run_at=run_at,
)
print(schedule["schedule_uuid"])
```

Update the time before it fires:

```python theme={null}
handler.update_one_time_schedule(
    schedule_uuid=schedule["schedule_uuid"],
    run_at="2026-06-02T09:00:00+00:00",
)
```

***

### Schedule recurring emails (cron)

Send emails on a repeating schedule using standard cron syntax. Provide either `max_jobs` (fire at most N times) or `deadline` (stop firing after this datetime) as the termination condition. Use `start_at` to delay when the schedule begins firing; omit it to start immediately.

```python theme={null}
from datetime import datetime, timezone

# Every Monday at 9 AM Eastern, up to 4 times
schedule = handler.create_cron_schedule(
    name="Weekly Monday reminder",
    cron_expression="0 9 * * MON",
    timezone="America/New_York",
    max_jobs=4,
)
```

With a deadline instead:

```python theme={null}
schedule = handler.create_cron_schedule(
    name="Daily reminders through end of month",
    cron_expression="0 8 * * *",
    timezone="America/Chicago",
    deadline=datetime(2026, 6, 30, 23, 59, tzinfo=timezone.utc),
)
```

With a delayed start:

```python theme={null}
schedule = handler.create_cron_schedule(
    name="Weekly reminders starting next month",
    cron_expression="0 9 * * MON",
    timezone="America/New_York",
    max_jobs=4,
    start_at=datetime(2026, 6, 1, tzinfo=timezone.utc),
)
```

Pause and resume a schedule:

```python theme={null}
handler.deactivate_schedule(schedule["schedule_uuid"])
handler.activate_schedule(schedule["schedule_uuid"])
```

***

### React when someone completes the survey (callback)

A **respondent** is a person with a corresponding entry in the survey's agent list. Each respondent has a unique survey link and a tracked response status. Submissions from people outside the agent list (e.g. anonymous links) are not associated with a respondent.

`create_callback` sends emails to respondents in your agent list. Two respondent event types are supported:

| Event | When it fires |
| - | - |
| `"human_survey_respondent.completed"` | When a respondent's status is set to completed. |
| `"human_survey_respondent.response_submitted"` | Each time a respondent submits a response. |

```python theme={null}
callback = handler.create_callback(
    name="Completion thank-you",
    callback_type="human_survey_respondent.completed",
)
print(callback["callback_uuid"])
```

Limit how many times the callback fires (e.g. only notify for the first 100 completions):

```python theme={null}
callback = handler.create_callback(
    name="First 100 completions",
    callback_type="human_survey_respondent.completed",
    max_fires=100,
)
```

### Notify the owner on every new response

Use `create_owner_response_callback` to email the survey owner each time a response is submitted. This fires for all responses, including anonymous ones — not just respondents in the agent list.

```python theme={null}
callback = handler.create_owner_response_callback(name="New response notification")
print(callback["callback_uuid"])
```

### Send a transcript after each submission

Use `create_transcript_callback` to automatically email a transcript each time a response is submitted. Pass `recipient` to control who receives it:

* `"respondent"` (default) — emails each respondent a copy of their own answers.
* `"owner"` — emails the survey owner a copy of the answers. Fires for all submissions, including anonymous ones.

```python theme={null}
# Email the respondent their own transcript
callback = handler.create_transcript_callback(
    name="Send respondent transcript on submission",
    recipient="respondent",
)

# Email the owner a transcript on every submission
callback = handler.create_transcript_callback(
    name="Send owner transcript on submission",
    recipient="owner",
)

print(callback["callback_uuid"])
```

Activate and deactivate:

```python theme={null}
handler.deactivate_callback(callback["callback_uuid"])
handler.activate_callback(callback["callback_uuid"])
```

***

### Filter which respondents receive a delivery

`HumanizeRespondentFilter` lets you compose conditions to target specific subsets of respondents. Conditions can be nested with `and` / `or` operators.

```python theme={null}
from edsl.coop.coop_humanize_notifications import (
    HumanizeRespondentFilter,
    RespondentCondition,
    DeliveryStatus,
    DispatchStatus,
    ResponseStatus,
    FilterOperator,
)

# Target respondents who were never contacted
never_contacted = RespondentCondition(never_contacted=True)

# OR respondents whose last email bounced
bounced = RespondentCondition(
    most_recent_delivery_status=[DeliveryStatus.bounced]
)

# Combine: never contacted OR previously bounced
respondent_filter = HumanizeRespondentFilter(
    operator=FilterOperator.or_,
    conditions=[never_contacted, bounced],
)

handler.send_respondent_email(
    name="Re-engagement",
    respondent_filter=respondent_filter,
)
```

**Available condition fields:**

| Field | Type | Filters to respondents where… |
| - | - | - |
| `respondent_uuids` | `list[str]` | UUID is in the provided list |
| `response_status` | `list[ResponseStatus]` | Survey response status matches |
| `never_contacted` | `bool` | Has never received a delivery |
| `any_dispatch_status` | `list[DispatchStatus]` | Any past task has this dispatch status |
| `any_delivery_status` | `list[DeliveryStatus]` | Any past task has this delivery status |
| `most_recent_dispatch_status` | `list[DispatchStatus]` | Most recent task has this dispatch status |
| `most_recent_delivery_status` | `list[DeliveryStatus]` | Most recent task has this delivery status |

***

### Check delivery status

List all delivery jobs for the survey:

```python theme={null}
deliveries = handler.list_deliveries()
for d in deliveries["deliveries"]:
    print(d["delivery_uuid"], d["status"], d["sent_count"], d["failed_count"])
```

Inspect a specific delivery job:

```python theme={null}
delivery = handler.get_delivery(delivery_uuid="...")
print(delivery)
# {
#   "delivery_uuid": "...",
#   "status": "completed",
#   "total_respondents": 50,
#   "processed_respondents": 50,
#   "sent_count": 48,
#   "failed_count": 2,
#   "started_at": "...",
#   "completed_at": "..."
# }
```

Drill into individual tasks (one per respondent per delivery):

```python theme={null}
tasks = handler.list_delivery_tasks(delivery_uuid="...")
for task in tasks["tasks"]:
    print(task["identifier"], task["dispatch_status"], task["delivery_status"])
```

Get a single delivery task by UUID:

```python theme={null}
task = handler.get_delivery_task(task_uuid="...")
print(task)
# {
#   "task_uuid": "...",
#   "channel": "email",
#   "identifier": "respondent@example.com",
#   "dispatch_status": "dispatched",
#   "delivery_status": "delivered",
#   "respondent": {"respondent_uuid": "...", "response_status": "completed"},
#   ...
# }
```

***

## Route types

Each delivery or schedule sends email to survey respondents via a `RespondentEmailRouteConfig` route. A delivery template is always required; convenience methods like `send_respondent_email` default to the built-in invitation template when none is provided.

`RespondentEmailRouteConfig` options:

| Field | Description |
| - | - |
| `delivery_template` | The email body. Pass an HTML string to use a custom template, or omit to use the built-in invitation template. |
| `respondent_filter` | `HumanizeRespondentFilter` to restrict which respondents receive the email. |
| `subject` | Email subject line (1-200 characters). Uses the default subject when omitted. |

### Template variables

Custom HTML templates support the following variables, which are filled in per-respondent at send time:

| Variable | Value |
| - | - |
| `{{ survey_name }}` | The name of the human survey. |
| `{{ url }}` | The respondent's unique survey link (includes their access token). |

Example:

```python theme={null}
handler.send_respondent_email(
    name="Initial outreach",
    subject="You're invited to take our survey",
    delivery_template="""
<p>Hi there,</p>
<p>You've been invited to take <strong>{{ survey_name }}</strong>.</p>
<p><a href="{{ url }}">Click here to begin</a></p>
""",
)
```

Update the subject and template on an existing respondent email route on a schedule:

```python theme={null}
handler.patch_schedule_respondent_email_route(
    schedule_uuid="...",
    route_uuid="...",
    subject="A quick reminder about our survey",
    delivery_template="<p>Hi! Please take a moment to complete our survey.</p>",
)
```

Update the subject and template on an existing respondent email route on a callback:

```python theme={null}
handler.patch_callback_respondent_email_route(
    callback_uuid="...",
    route_uuid="...",
    subject="Thanks for completing our survey",
    delivery_template="<p>Hi! Thank you for completing <strong>{{ survey_name }}</strong>.</p>",
)
```

***

## Managing respondents

List all respondents and their current response status:

```python theme={null}
respondents = handler.get_respondents()
for r in respondents["respondents"]:
    print(r["respondent_uuid"], r["response_status"])
```

Check and update the agent list delivery configuration (the trait column that holds each respondent's email address):

```python theme={null}
from edsl.coop.coop_humanize_notifications import ChannelConfig, DeliveryMap

# See current config
config = handler.get_agent_list()
print(config["agent_list_config"])

# Update the email column mapping
handler.patch_agent_list(
    delivery_map=DeliveryMap(email=ChannelConfig(col_name="email_address"))
)
```


# Humanize schema
Source: https://docs.expectedparrot.com/en/latest/humanize_schema

Reference for the humanize schema: a dict that configures how survey questions are presented in web surveys with human respondents.

When creating a human survey or preview (e.g. `survey.humanize()` or `survey.preview()`), you can pass a **humanize schema** to control styling and optionality per question. The schema is validated against your survey before use.

## Fields

<ParamField type="dict">
  Map of question name (string) -> `HumanizeQuestionSchema`.
</ParamField>

<ParamField type="dict">
  Optional survey-level options (e.g. custom CSS).

  <Expandable title="properties">
    <ParamField type="string">
      `"question"` requests question-by-question presentation (the default); `"group"` requests one question group per page. Define groups on the survey with `add_question_group()` and include every question you want displayed. See [Grouped surveys](/en/latest/humanize_groups) for setup, navigation behavior, and availability requirements for PR #2623. Question time limits (`time_limit`) are ignored under `"group"`.
    </ParamField>

    <ParamField type="string">
      Optional custom CSS for the survey. Can be `null` or omitted. See [Custom CSS](/en/latest/custom_css) for the `edsl-*` CSS classes and HTML structure per question type.
    </ParamField>

    <ParamField type="dict">
      How the respondent is shown their position in the survey. Omit to keep the progress bar, which is what surveys render by default.

      <Expandable title="properties">
        <ParamField type="string">
          Which progress indicator to render.

          * `"bar"` — a filled bar, optionally labelled (default).
          * `"hidden"` — no progress indicator anywhere in the survey.
          * `"steps"` — markers joined by a connector, one per step, with the current step highlighted.
        </ParamField>

        <ParamField type="dict">
          **`bar` only.** The label beneath the bar. Set to `null` to keep the bar and drop the label.

          <Expandable title="properties">
            <ParamField type="string">
              Must be `"percent"` — the share of the survey completed, e.g. `"33%"`.
            </ParamField>
          </Expandable>
        </ParamField>

        <ParamField type="string">
          **`steps` only.** Whether each marker carries its step number (`"number"`) or is a bare dot (`"dot"`).
        </ParamField>

        <ParamField type="array">
          **`steps` only.** The steps, in order. At least two are required.

          A step is a *boundary*, not a bucket: it covers every survey item from the end of the previous step through its `complete_after`. Questions can be added or removed inside a step without touching this config, and items a respondent skips past inside a step don't move the marker.

          Each step is an object with:

          * `label` (string, optional) — the word under the marker, 1–60 characters. Omit or set to `null` to render the marker alone.
          * `complete_after` (string, optional) — the name of the survey item (question or instruction) after which this step is complete. Only the final step may omit it, which runs that step to the end of the survey; every earlier step must name an item. The same item must not be named by two steps.
        </ParamField>
      </Expandable>
    </ParamField>

    <ParamField type="dict">
      The author's brand on the respondent page. Omit or set to `null` for no banner, which is how a survey without this field renders.

      <Expandable title="properties">
        <ParamField type="dict">
          The logo drawn in the survey banner, above the questions. Set to `null` to remove it.

          Upload the image first, with `Coop().upload_human_survey_asset("lab_logo.png")` or `ep humanize assets upload lab_logo.png`, and reference the uuid it returns. A schema never carries image bytes.

          There is no size field: the banner picks a default height. Style `.edsl-logo` in `custom_css` to change it. See [Custom CSS](/en/latest/custom_css).

          <Expandable title="properties">
            <ParamField type="dict">
              Where the image comes from. Must be `{"type": "asset", "asset_uuid": "<uuid>"}`, naming an image in your asset library.
            </ParamField>

            <ParamField type="string">
              Alt text for screen readers, up to 200 characters. Required, so that leaving it out is a decision rather than an accident. Use an empty string to mark the logo decorative, which renders it with `alt=""`.
            </ParamField>

            <ParamField type="string">
              Where the logo sits in the banner: `"left"`, `"center"`, or `"right"`.
            </ParamField>
          </Expandable>
        </ParamField>
      </Expandable>
    </ParamField>
  </Expandable>
</ParamField>

## HumanizeQuestionSchema (per question)

Each `questions[question_name]` value is a `HumanizeQuestionSchema` object.

<ParamField type="boolean">
  Whether the question is optional. Default is `false` when omitted.

  **Supported question types:** `free_text`, `budget`, `checkbox`, `checkbox_with_other`, `file_upload`, `interview`, `likert_five`, `linear_scale`, `list`, `matrix`, `multiple_choice`, `multiple_choice_with_other`, `numerical`, `rank`, `top_k`, `yes_no`.

  Note: `optional` currently has no effect for `interview` and `rank`.
</ParamField>

<ParamField type="dict">
  A limit on how long the respondent may spend answering the question. Omit or set to `null` for no limit.

  While the limit runs, a countdown is shown above the question. When it reaches zero the answer locks: the respondent can no longer change it and clicks **Next** to continue. They are not moved on automatically. Whatever they had entered is submitted as their answer, and a question they had not answered is recorded as unanswered, even if it is not `optional`.

  A time limit applies only while the question is alone on its page. When `survey.presentation` is `"group"`, every question's `time_limit` is ignored, including a question that is the only one in its group. The countdown runs in the respondent's browser, so reloading the page starts it again.

  **Supported question types:** `free_text`, `budget`, `checkbox`, `checkbox_with_other`, `file_upload`, `likert_five`, `linear_scale`, `list`, `matrix`, `multiple_choice`, `multiple_choice_with_other`, `numerical`, `rank`, `top_k`, `yes_no`.

  <Expandable title="properties">
    <ParamField type="dict">
      How long the respondent has. Must be `{"type": "fixed", "seconds": <seconds>}`, the same number of seconds for every respondent. `seconds` must be between 30 and 7200 (two hours). `type` is required, even though `"fixed"` is its only value.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  Display format. Varies by question type:

  * **MC-style** (`likert_five`, `linear_scale`, `multiple_choice`, `yes_no`): `"radio"` (list of radio buttons) or `"dropdown"` (single `<select>`). Default is `"radio"`. Use `{"type": "radio"}` or `{"type": "dropdown"}`.
  * **Numerical** (`numerical`): `"input"` (number input field) or `"slider"` (range slider). Default is `"input"`. Use `{"type": "input"}` or `{"type": "slider", "min": 0, "max": 100, "step": 1}`. For slider, `min` must be less than `max`, `step` must be positive and not exceed `(max - min)`.
  * **Matrix** (`matrix`): `"table"` (the whole grid at once) or `"carousel"` (one item at a time). Default is `"table"`. Use `{"type": "table"}` or `{"type": "carousel"}`.

  Other question types do not support `format`.

  <Expandable title="matrix carousel properties">
    <ParamField type="boolean">
      **`carousel` only.** Whether answering an item moves to the next one on its own, rather than waiting for the respondent to click Next.

      Every answer advances, corrections included — the same rule respondents will have met in Qualtrics. Set to `false` when respondents are expected to revise as they go, since under `true` a correction costs the way back as well.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  **Matrix only** (`matrix`). Cells checked before the respondent touches the grid. Omit or set to `null` for an empty grid, which is what matrices render by default.

  A preselection changes what the question measures: it puts an answer in front of the respondent and asks them to disagree with it, so the answers lean toward the preselected column. It also removes the blank — a row left alone is recorded as the preselected option and cannot be told apart from a considered answer. Both are usually the point (a long grid where most rows are expected to be neutral is far cheaper to finish), but they are worth deciding before fielding rather than after.

  <Expandable title="properties">
    <ParamField type="array">
      The rules, in order. At least one is required.

      Rules are tried in order and **the first match wins**, so the list reads like a routing table: the specific rows first, the catch-all last. A rule that matches every item must therefore be last — anything behind it could never be reached, and a list in the wrong order raises a validation error rather than silently dropping the rules it shadows.

      Each rule is an object with:

      * `match` (dict, required) — which rows the rule fills. Must be `{"type": "all"}`, every row of the grid.
      * `option` (dict, required) — which column those rows are set to. Must be `{"type": "text", "value": "<option text>"}`, naming an option by its exact text in the question's `question_options`. Matching is literal, so surrounding whitespace is not trimmed for you.

      **If the option you name is not in the list as served, the rule fills nothing.** It does not raise, and the survey still runs. A matrix's columns can be piped from an earlier answer, so the text you name may not exist until the question reaches the respondent — and a preselection must not be the thing that breaks a survey in front of them. Where the columns are a literal list, a name that matches none of them is caught when the survey is saved, while you can still fix it.

      Those rows are then left empty; they do not fall through to the next rule. A row belongs to the first rule that matches it whether or not that rule finds its option, so a row singled out by an early rule cannot quietly pick up the catch-all that rule was written to override.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  **Multiple choice only** (`multiple_choice`). Optional custom validation rules.

  <Expandable title="properties">
    <ParamField type="string">
      If set, the human survey will require the respondent to select this exact option. Use for attention checks or forcing a particular choice. Omit or set to `null` to allow any selection.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  **Checkbox only** (`checkbox`; not `checkbox_with_other`, which shows no such box). The Select all box beneath the options: ticking it selects every option, unticking it clears them. Omit it to keep the box, which every checkbox question shows by default, or set it to `null` to remove it.

  Exclusive options are left out of "all", since checking one clears every other selection.

  <Expandable title="properties">
    <ParamField type="string">
      The box's wording. `"Select all"` is the only accepted value, so setting it changes nothing.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="array">
  **Checkbox only** (`checkbox`, `checkbox_with_other`). Options that stand alone: checking one clears every other selection — including any other exclusive option, and for `checkbox_with_other` the respondent's "other" entries — and selecting anything else clears it. Use for options like `"None of the above"` or `"Prefer not to say"`.

  Each entry must be the exact text of an option in the question's `question_options` — matching is literal, so surrounding whitespace is not trimmed for you. Entries must be non-empty and unique. Because an exclusive option is an ordinary member of the option list, the submitted answer looks like any other selection. Defaults to an empty list, meaning no option is exclusive.
</ParamField>

<ParamField type="dict">
  **Distribution only** (`distribution`). What the chart shows before the respondent touches it. Respondents answer a distribution question by painting probability across its outcomes or bins; the percentages always add up to 100%.

  <Expandable title="properties">
    <ParamField type="string">
      Which starting state to show. Required, even though `"uniform"` is the default when `initial_distribution` is omitted.

      * `"uniform"` — equal probability for every outcome or bin (default). This is equal probability *per bin*, not per unit of width, so bins of unequal width still start equal. It is the answer's starting value: a respondent who agrees with it can click **Next** without painting.
      * `"empty"` — nothing allocated. The respondent must paint at least one bar before they can continue.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  **Distribution only** (`distribution`). Statistics implied by the respondent's distribution, shown beneath the chart and updated as they paint. Omit or set to `null` for no summary, which is the default.

  The statistics assume probability is spread evenly within each bin, and the chart says so. They appear only for finite numeric bins: categories have no values to average, and an open-ended bin such as `[10,Inf)` has no midpoint. They are a display aid and do not change the submitted answer.

  <Expandable title="properties">
    <ParamField type="array">
      Which statistics to show, in the order to show them. Each entry is an object with a `type`:

      * `{"type": "mean"}` — the implied mean: each bin's midpoint weighted by its probability.
      * `{"type": "variance"}` — the implied variance, including each bin's own spread (width² / 12), in squared units of the bins.

      Must be non-empty (omit `distribution_summary` instead of passing an empty list) and must not repeat a type. Defaults to mean, then variance.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="string">
  **Interview only** (`interview`). Controls the input mode offered to respondents.

  * `"text"` — text-based interview (default).
  * `"voice"` — voice-based interview.
  * `"both"` — respondent can choose between text and voice.
</ParamField>

<ParamField type="dict">
  **Interview only** (`interview`). The intro screen shown to the respondent before the interview begins. The intro screen is always shown; this field customizes its text. Omit or set to `null` to use the default text (`"This will be a conversation with an AI agent."`).

  <Expandable title="properties">
    <ParamField type="string">
      Must be `"default"`.
    </ParamField>

    <ParamField type="string">
      The intro screen body text. Must be between 1 and 2500 characters.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  **Interview only** (`interview`). Optional configuration for voice-mode interviews. Ignored when `interview_mode` is `"text"`.

  <Expandable title="properties">
    <ParamField type="string">
      The spoken language for the voice interview. A lowercase language id from the supported set below. Case and surrounding whitespace are normalized (e.g. `"  French "` becomes `"french"`); omit or set to `null` to use the default (`"english"`). An unsupported value raises a validation error.

      **Supported languages:** `english`, `french`, `german`, `spanish`, `portuguese`, `chinese`, `japanese`, `hindi`, `italian`, `korean`, `dutch`, `polish`, `russian`, `swedish`, `turkish`, `tagalog`, `bulgarian`, `romanian`, `arabic`, `czech`, `greek`, `finnish`, `croatian`, `malay`, `slovak`, `danish`, `tamil`, `ukrainian`, `hungarian`, `norwegian`, `vietnamese`, `bangla`, `thai`, `hebrew`, `georgian`, `indonesian`, `telugu`, `gujarati`, `kannada`, `malayalam`, `marathi`, `punjabi`, `multilingual`.
    </ParamField>

    <ParamField type="dict">
      How the interviewer decides the respondent has finished speaking. Omit to use the default (`{"initial_speed": "default"}`).

      <Expandable title="properties">
        <ParamField type="string">
          How long the interviewer waits after the respondent pauses before it speaks, at the start of the call. The respondent can change it from the interview screen.

          * `"fastest"` — jumps in almost as soon as the respondent pauses.
          * `"faster"`
          * `"default"` — a natural pause (default).
          * `"slower"`
          * `"slowest"` — waits a long time, for respondents who pause mid-thought.
        </ParamField>
      </Expandable>
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  **Interview only** (`interview`). Optional configuration for text-mode interviews. Ignored when `interview_mode` is `"voice"`.

  <Expandable title="properties">
    <ParamField type="string">
      The language for the text interview. A lowercase language id from the supported set below. Case and surrounding whitespace are normalized (e.g. `"  French "` becomes `"french"`); omit or set to `null` to use the default (`"english"`). An unsupported value raises a validation error.

      **Supported languages:** `english`, `french`, `german`, `spanish`, `portuguese`, `chinese`, `japanese`, `hindi`, `italian`, `korean`, `dutch`, `polish`, `russian`, `swedish`, `turkish`, `tagalog`, `bulgarian`, `romanian`, `arabic`, `czech`, `greek`, `finnish`, `croatian`, `malay`, `slovak`, `danish`, `tamil`, `ukrainian`, `hungarian`, `norwegian`, `vietnamese`, `bangla`, `thai`, `hebrew`, `georgian`, `indonesian`, `telugu`, `gujarati`, `kannada`, `malayalam`, `marathi`, `punjabi`, `multilingual`.
    </ParamField>

    <ParamField type="string">
      Label shown above the interviewer's messages. Must be between 1 and 200 characters. Defaults to `"AI Interviewer"`. Omit or set to `null` to use the default.
    </ParamField>

    <ParamField type="string">
      Message shown to respondents when the interview ends. Must be between 1 and 2500 characters. Defaults to `"Thank you for your time. This interview has been ended. Your responses have been recorded and will be valuable for our research."` Omit or set to `null` to use the default.
    </ParamField>

    <ParamField type="dict">
      A checklist of items the interviewer works through during the interview. Omit or set to `null` to run without a checklist.

      <Expandable title="properties">
        <ParamField type="dict">
          The starting set of checklist items, written by the survey author. Defaults to an empty list.

          <Expandable title="properties">
            <ParamField type="string">
              Must be `"manual"`.
            </ParamField>

            <ParamField type="array">
              The list of checklist items. Each item is an object with:

              * `id` (string, required) — a unique identifier, 1–64 characters, restricted to letters, digits, underscores, and hyphens (regex pattern `^[A-Za-z0-9_\-]+$`). Item ids must be unique within the checklist.
              * `label` (string, required) — participant-facing text shown in the checklist UI, 1–200 characters.
              * `instructions` (string, required) — model-facing text describing the condition under which the model should check the item off, 1–2000 characters.
            </ParamField>
          </Expandable>
        </ParamField>

        <ParamField type="string">
          Whether the participant sees the checklist. The model always sees it (it's in the system prompt) regardless of this setting.

          * `"visible"` — the floating checklist panel is shown to the participant during the interview (default).
          * `"hidden"` — the participant never sees the checklist; it acts as a pure interviewer instrument (status is still tracked internally).
        </ParamField>
      </Expandable>
    </ParamField>

    <ParamField type="dict">
      Lets the interviewer ask multiple-choice and numerical questions mid-interview, rendered as input widgets rather than prose. Omit or set to `null` to disable structured questions.

      <Expandable title="properties">
        <ParamField type="array">
          Which question types the model may use. Each entry must be `"multiple_choice"` or `"numerical"`. Defaults to both. Narrowing this restricts the model to the listed types. Must be non-empty (omit `structured_questions` instead of passing an empty list) and must not contain duplicates.
        </ParamField>
      </Expandable>
    </ParamField>

    <ParamField type="dict">
      Controls how the interview is allowed to end. Omit to use the default (`{"control": "respondent"}`).

      <Expandable title="properties">
        <ParamField type="string">
          Who decides when the interview is over.

          * `"respondent"` — the participant can end at any time; the End Interview button is always available (default). No other fields apply.
          * `"interviewer_gated"` — the End Interview button is hidden until the model has signalled that its goals are met; only then can the participant end. This mode also accepts `interview_marked_complete_message` and `participant_chat_after_complete`.
        </ParamField>

        <ParamField type="dict">
          **`interviewer_gated` only.** A message to surface on the turn the interviewer first marks the interview complete. If omitted or set to `null`, the LLM will generate text for that turn instead.

          <Expandable title="properties">
            <ParamField type="string">
              The message text. Must be between 1 and 2500 characters.
            </ParamField>

            <ParamField type="string">
              How `message` relates to the model's text on the marking turn.

              * `"replace"` — show this `message` for that one turn instead of using a model to generate text.
            </ParamField>
          </Expandable>
        </ParamField>

        <ParamField type="string">
          **`interviewer_gated` only.** Whether the participant can keep sending messages after the interviewer has marked the interview complete.

          * `"open"` — the message composer stays live; marking the interview complete only unlocks the End Interview button (default).
          * `"locked"` — the composer closes as well, so the marking turn is the last thing the participant reads and ending the interview is their only remaining action.
        </ParamField>
      </Expandable>
    </ParamField>

    <ParamField type="string">
      Whether the participant can type while the interviewer's reply is still being generated.

      * `"open"` — the message composer stays live, so the participant can compose ahead of a reply they have not read yet (default).
      * `"locked"` — the composer closes until the reply arrives, so each reply is read before anything is written against it.

      Unlike `end_policy.participant_chat_after_complete`, this lock is per-turn and reopens; it applies to every turn under either end policy.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  Optional comment input shown with the question.
  Submitted comment text appears in survey results under `comment.{question_name}_comment`.

  **Supported question types:** `free_text`, `budget`, `checkbox`, `checkbox_with_other`, `likert_five`, `linear_scale`, `list`, `matrix`, `multiple_choice`, `multiple_choice_with_other`, `numerical`, `rank`, `top_k`, `yes_no`.

  <Expandable title="properties">
    <ParamField type="string">
      Label text displayed for the comment field.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  Optional indicator shown after the respondent clicks **Next**, while the following question runs. Omit or set to `null` for no indicator.

  **Supported question types:** `free_text`, `budget`, `checkbox`, `checkbox_with_other`, `file_upload`, `likert_five`, `linear_scale`, `list`, `matrix`, `multiple_choice`, `multiple_choice_with_other`, `numerical`, `rank`, `top_k`, `yes_no`.

  Note: not supported for `compute` or `image_generation` (both run in the background and auto-advance, so neither is ever the question a respondent submits), or for `interview`.

  <Expandable title="properties">
    <ParamField type="string">
      How the indicator is rendered. Must be `"callout"` — a callout box shown while the next question runs, with the subtitle `"We'll continue automatically when it's ready."`
    </ParamField>

    <ParamField type="string">
      Optional title for the callout, 1–200 characters. Omit or set to `null` to use the frontend's default label for the indicator type.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="dict">
  Optional JavaScript that runs on the respondent's page for this question. Omit or set to `null` to run no script. See [Custom JavaScript](/en/latest/custom_javascript) for how to write a hook, what a script may do, and how to retrieve the events it logs.

  **Supported question types:** `free_text`, `budget`, `checkbox`, `checkbox_with_other`, `distribution`, `file_upload`, `interview`, `likert_five`, `linear_scale`, `list`, `matrix`, `multiple_choice`, `multiple_choice_with_other`, `numerical`, `rank`, `top_k`, `yes_no`, `survey_message`.

  Note: not supported for `compute` or `image_generation`. Both run in the background and are dropped from the page, so there is no rendered question for a script to attach to.

  This feature is available on approved accounts only. Write to [info@expectedparrot.com](mailto:info@expectedparrot.com) to request approval; until your account is approved, a schema carrying a hook is rejected when you push it.

  <Expandable title="properties">
    <ParamField type="dict">
      Maps a lifecycle event to the JavaScript source that runs on it. An unrecognized event name raises a validation error rather than being stored and silently never run.

      Supported events:

      * `"question.ready"` — fires once the question has been rendered and its inputs are in the DOM.

      Each hook's source must be at most 64,000 characters, and all hooks on one question at most 256,000 characters in total.
    </ParamField>
  </Expandable>
</ParamField>

## Example

```python theme={null}
from edsl import (
    Coop,
    Survey,
    QuestionDistribution,
    QuestionFreeText,
    QuestionMultipleChoice,
)

# Upload the logo once; the schema references it by uuid
logo = Coop().upload_human_survey_asset("lab_logo.png")

survey = Survey([
    QuestionMultipleChoice(
        question_name="rating",
        question_text="Rate your experience.",
        question_options=["Good", "OK", "Bad"],
    ),
    QuestionDistribution(
        question_name="wait",
        question_text="How many minutes do you expect to wait next time?",
        min_value=0, max_value=30, bucket_size=5,
    ),
    QuestionFreeText(
        question_name="feedback",
        question_text="Any additional feedback you'd like to share?",
    ),
])

humanize_schema = {
    "questions": {
        "rating": {
            "optional": False,
            "format": {"type": "dropdown"},
            "comment": {"label": "Why did you choose this rating?"},
            "submitting_indicator": {"type": "callout", "title": "Analyzing your answer…"},
        },
        "wait": {
            "initial_distribution": {"type": "empty"},
            "distribution_summary": {"statistics": [{"type": "mean"}]},
        },
        "feedback": {
            "optional": True,
            "time_limit": {"duration": {"type": "fixed", "seconds": 300}},
        },
    },
    "survey": {
        "custom_css": None,
        "progress": {"type": "bar", "label": {"type": "percent"}},
        "branding": {
            "logo": {
                "source": {"type": "asset", "asset_uuid": logo.uuid},
                "alt": "Acme Research logo",
                "position": "center",
            }
        },
    },
}

# Validation runs before creating the human survey
survey.humanize(humanize_schema=humanize_schema)
```

## Retrieving a deployed schema

To read back the schema stored on a survey you already deployed:

```python theme={null}
from edsl import Coop

schema = Coop().get_human_survey_humanize_schema("your-human-survey-uuid")
```

It returns the schema as stored, or `None` if the survey was created without one.
From the CLI, `ep humanize schema get <uuid>` prints it, and `--out schema.json`
writes it to a file you can edit and send back with `ep humanize schema patch`.
Give `--out` a `.json.gz` suffix to write the schema gzipped; any other suffix
writes plain JSON. Both are readable by the commands that take `--schema`.

## Validation

If the schema is invalid, Expected Parrot raises `HumanizeSchemaValidationError`. Common causes:

* A key in `questions` is not a question name in the survey, or is an instruction.
* A question's type is not supported for humanize schema (e.g. `demand`, `dropdown`).
* A question's entry has the wrong shape for its type (e.g. wrong field types or extra fields that aren’t allowed).
* A `time_limit` has `seconds` outside 30–7200, or a `duration` without its `"type": "fixed"`.
* Top-level structure is invalid (e.g. `questions` not a dict, or an entry not a dict).

You can also pass the schema to `survey.preview(humanize_schema=...)` to get a preview URL.

Ensure your humanize schema matches the parameters above for each question type in your survey.


# Expected Parrot: Tools for AI-Powered Research
Source: https://docs.expectedparrot.com/en/latest/index

Expected Parrot delivers powerful tools for conducting research with human and artificial intelligences.

**Expected Parrot Domain-Specific Language (EDSL)** is an open-source Python package for conducting research with AI agents and language models.

EDSL is designed for researchers who want to run surveys at scale: simulate hundreds of demographically diverse respondents, test question variations, compare responses across language models, label large text datasets, and combine AI results with real human data. It handles the complexity of managing agents, models, and parallel execution so you can focus on the research.

* Install from [PyPI](https://pypi.org/project/edsl/): `pip install edsl`. Source code at [GitHub](https://github.com/expectedparrot/edsl).
* [Create a free account](https://www.expectedparrot.com/login) to run surveys remotely and store results. Results are automatically cached: share your code and anyone can retrieve your exact outputs at no cost, with no setup or API keys required.

## Getting started

<Steps>
  <Step title="Install EDSL">
    ```bash theme={null}
    pip install edsl
    ```

    See [Installation](/en/latest/installation) for details.
  </Step>

  <Step title="Create an account">
    [Sign up](https://www.expectedparrot.com/login) to access the Expected Parrot server, free storage, and collaboration tools. You can also log in directly from your workspace:

    ```python theme={null}
    from edsl import Login
    login()
    ```
  </Step>

  <Step title="Manage API keys">
    Your account comes with a key that lets you run surveys with all [available models](https://www.expectedparrot.com/models). You can also provide your own keys from service providers. See [Managing Keys](/en/latest/api_keys).
  </Step>

  <Step title="Run a survey">
    Read the [Starter Tutorial](/en/latest/starter_tutorial) and browse the [how-to guides and notebooks](/en/latest/notebooks) for examples. See [tips](/en/latest/checklist) on using EDSL effectively.
  </Step>

  <Step title="Validate with real respondents">
    Add a human-in-the-loop by launching a web-based survey to share with real respondents. Learn more in the [Survey Builder](/en/latest/survey_builder) and [Humanize](/en/latest/humanize) sections.
  </Step>
</Steps>

Join our [Discord](https://discord.com/invite/mxAYkjfy9m) to ask questions and chat with other users!

Are you using EDSL for a research project? Email *[info@expectedparrot.com](mailto:info@expectedparrot.com)* and we'll give you [credits](/en/latest/credits) to run your project.

## Introduction

<CardGroup>
  <Card title="Overview" icon="book-open" href="/en/latest/overview">
    Purpose, concepts and goals of the EDSL package.
  </Card>

  <Card title="Starter Tutorial" icon="play" href="/en/latest/starter_tutorial">
    A step-by-step tutorial for getting started with EDSL.
  </Card>

  <Card title="Whitepaper" icon="file-lines" href="/en/latest/whitepaper">
    A whitepaper about the EDSL package (*in progress*).
  </Card>

  <Card title="Citation" icon="box" href="/en/latest/citation">
    How to cite the package in your work.
  </Card>

  <Card title="Papers & Citations" icon="book" href="/en/latest/papers">
    Research papers and articles that use or cite EDSL.
  </Card>

  <Card title="Teaching guide" icon="school" href="/en/latest/teaching">
    A guide for teaching EDSL in the classroom.
  </Card>
</CardGroup>

## Core Concepts

<CardGroup>
  <Card title="Questions" icon="question" href="/en/latest/questions">
    Different question types and applications.
  </Card>

  <Card title="Scenarios" icon="timeline" href="/en/latest/scenarios">
    Dynamically parameterize questions for tasks like data labeling.
  </Card>

  <Card title="Surveys" icon="clipboard-list" href="/en/latest/surveys">
    Construct surveys with rules and conditional logic.
  </Card>

  <Card title="Agents" icon="user-astronaut" href="/en/latest/agents">
    Design AI agents with traits to respond to surveys.
  </Card>

  <Card title="Language Models" icon="brain" href="/en/latest/language_models">
    Select language models to generate results.
  </Card>

  <Card title="Results" icon="chart-line" href="/en/latest/results">
    Built-in methods for analyzing survey results.
  </Card>
</CardGroup>

## Expected Parrot Platform

<CardGroup>
  <Card title="Remote Inference" icon="server" href="/en/latest/remote_inference">
    Run surveys at the Expected Parrot server using any available model.
  </Card>

  <Card title="Remote Caching" icon="clock" href="/en/latest/remote_caching">
    Automatically store and share results and API calls.
  </Card>

  <Card title="Notebooks" icon="notebook" href="/en/latest/notebooks">
    Post and share .ipynb and .py files.
  </Card>

  <Card title="File Store" icon="folder" href="/en/latest/filestore">
    Store and share data files for use in EDSL projects.
  </Card>

  <Card title="Survey Builder" icon="screwdriver-wrench" href="/en/latest/survey_builder">
    A no-code interface for creating surveys and collecting responses.
  </Card>
</CardGroup>

## Validating with Humans

<CardGroup>
  <Card title="Humanize" icon="user-group" href="/en/latest/humanize">
    Generate web-based surveys and collect responses from human respondents.
  </Card>

  <Card title="Prolific studies" icon="vial" href="/en/latest/prolific">
    Launch surveys as studies on Prolific.
  </Card>
</CardGroup>

## Working with Results

<CardGroup>
  <Card title="Dataset" icon="database" href="/en/latest/dataset">
    Work with tabular data using the Dataset class.
  </Card>

  <Card title="Estimating & Tracking Costs" icon="credit-card" href="/en/latest/costs">
    Estimate and track costs for running surveys.
  </Card>

  <Card title="Exceptions & Debugging" icon="bug" href="/en/latest/exceptions">
    Identify and handle exceptions.
  </Card>

  <Card title="Token usage" icon="key" href="/en/latest/token_usage">
    Monitor token limits and usage.
  </Card>
</CardGroup>

## Links

* [PyPI](https://pypi.org/project/edsl) · [GitHub](https://github.com/expectedparrot/edsl) · [Discord](https://discord.com/invite/mxAYkjfy9m)
* [Twitter/X](https://twitter.com/expectedparrot) · [LinkedIn](https://www.linkedin.com/company/expectedparrot) · [Blog](https://blog.expectedparrot.com/)
* Email: *[info@expectedparrot.com](mailto:info@expectedparrot.com)*


# Installation
Source: https://docs.expectedparrot.com/en/latest/installation

EDSL is an open-source Python library. It is compatible with Python 3.9 - 3.13 and available at [PyPI](https://pypi.org/project/edsl/) and [GitHub](https://github.com/expectedparrot/edsl).

<Note>
  **Note**

  EDSL is in development. We recommend that you continually check for and install the latest version of EDSL to access the most recent features and bug fixes.
</Note>

## Prerequisites

Please ensure that Python is installed on your system before you install EDSL. You can download Python from the [official website](https://www.python.org/downloads/).

You can use [pip](https://pip.pypa.io/en/stable/installation/) to install EDSL on your system (a package installer for Python).

## Quickstart Installation

Check whether EDSL is already installed:

```bash theme={null}
pip show edsl
```

If installed, you will see the version number in the output. If not installed, run either of the following:

```bash theme={null}
pip install edsl
```

Or:

```bash theme={null}
! uv pip install edsl -q
```

Then get your API key from Expected Parrot and import dependencies:

```python theme={null}
from edsl import login

login()
```

This will prompt you to log into your Expected Parrot account and automatically retrieve your API key. See the [Managing Keys](/en/latest/api_keys) section for more details on managing your API keys for language models.

## Installing from source

If you want to install EDSL from source, you can clone the repository from GitHub:

```bash theme={null}
pip install git+https://github.com/expectedparrot/edsl.git@main
```

This gives you the latest development changes ahead of PyPI releases.

## Updating your version

To confirm that your version is up to date, compare your version number with the latest version on the [EDSL PyPI page](https://pypi.org/project/edsl/).

To update your EDSL version to the latest one:

```bash theme={null}
pip install --upgrade edsl
```

## Advanced installation

The steps above install EDSL globally. To avoid library conflicts, use a virtual environment.

To create a virtual environment, open your terminal and run the following command:

```bash theme={null}
python -m venv myenv
```

This will create a folder called myenv. Next, activate your virtual environment:

```bash theme={null}
source myenv/bin/activate
```

You can now install EDSL through pip within your virtual environment:

```bash theme={null}
pip install edsl
```

Or:

```bash theme={null}
! uv pip install edsl -q
```

You will have access to EDSL while your virtual environment is activated.

You can deactivate the virtual environment at any time by running:

```bash theme={null}
deactivate
```

To delete the virtual environment, simply delete the myenv folder.


# Jobs
Source: https://docs.expectedparrot.com/en/latest/jobs

A `Job` is a `Survey` object combined with one or more `Model` objects and any `Agent` and `Scenario` objects that have been added to the survey. It is used to manage the execution of a survey with the specified models, agents and scenarios.

A job instance can be created by adding a model to a survey with the by() method, or by using the Jobs class to create a job directly. For example:

```python theme={null}
from edsl import QuestionFreeText, Survey, Model, Jobs

q = QuestionFreeText(
   question_name = "example",
   question_text = "What is your favorite color?",
)
survey = Survey(questions = [q])
model = Model("gpt-4o", service_name = "openai")
job = survey.by(model)

# or using the Jobs class
job = Jobs(survey).by(model)
```

It can be useful to work with jobs when you want to run a survey with multiple models, agents, or scenarios, or when you want to manage the execution of a survey in a more structured way.

There are several methods available in the Jobs class to manage jobs, such as by(), run(), and list():

* The by() method is used to add a model to the survey and create a job instance.
* The run() method is used to execute the job.
* The list() method is used to list details of jobs that have been posted to Expected Parrot.
* The fetch() method is used to retrieve jobs that have been posted to Expected Parrot.

For example, to run the above job:

```python theme={null}
job.run()
```

To retrieve details about your 10 most recent jobs posted to Expected Parrot:

```python theme={null}
from edsl import Jobs

jobs = Jobs().list()
```

The following information will be returned:

| Column | Description |
| :- | :- |
| uuid | The UUID of the job. |
| description | A description of the job, if any. |
| status | The status of the job (e.g., running, completed, failed). |
| cost\_credits | The cost of the job in credits. |
| iterations | The number of iterations the job has run. |
| results\_uuid | The UUID of the results for the job. |
| latest\_error\_report\_uuid | The UUID of the latest error report for the job, if any. |
| latest\_failure\_reason | The reason for the latest failure, if any. |
| version | The EDSL version used to create the job. |
| created\_at | The date and time the job was created. |

You can also specify the page\_size parameter to limit the number of jobs returned, and the page parameter to paginate through the jobs:

```python theme={null}
jobs = Jobs.list(page_size=5, page=2)
```

You can also filter jobs by their status using the status parameter:

```python theme={null}
jobs = Jobs.list(status="running")
```

You can filter jobs by description using the search\_query parameter:

```python theme={null}
jobs = Jobs.list(search_query="testing")
```

To fetch the Jobs objects directly you can use the fetch() method:

```python theme={null}
from edsl import Jobs

jobs = Jobs.list(page_size=1).fetch()
```

Or to fetch the associated results:

```python theme={null}
from edsl import Jobs

jobs = Jobs.list(page_size=1).fetch_results()
```

## Prompts

It can also be useful to work with Jobs objects in order to inspect user and system prompts before running the job.

For example, here we create a survey and use the job to inspect the prompts:

```python theme={null}
from edsl import QuestionFreeText, Survey, Agent, Model, Jobs

q = QuestionFreeText(
question_name = "example",
question_text = "What is your favorite color?",
)

survey = Survey(questions = [q])

agent = Agent(traits = {"persona":"You are an artist."})

model = Model("gpt-4o", service_name = "openai")

job = survey.by(agent).by(model)

# Inspect the prompts
job.show_prompts()
```

This will return the following information:

| user\_prompt | What is your favorite color? |
| :- | :- |
| system\_prompt | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an artist.'}` |
| interview\_index | 0 |
| question\_name | example |
| scenario\_index | 0 |
| agent\_index | 0 |
| model | gpt-4o |
| estimated\_cost | 0.000373 |
| cache\_keys | \[‘e549b646508cfd459f88379649ebe8ba’] |

## Jobs class

> #### *class* edsl.jobs.Jobs(*survey: [Survey](/en/latest/surveys#survey-class "edsl.surveys.Survey")*, *agents: list\['Agent'] | 'AgentList' | None = None*, *models: 'ModelList' | list\['LanguageModel'] | None = None*, *scenarios: 'ScenarioList' | list\['Scenario'] | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Bases: [`Base`](/en/latest/base#base-class "edsl.base.base_class.Base")

A collection of agents, scenarios, models, and a survey that orchestrates interviews.

The Jobs class is the central component for running large-scale experiments or simulations in EDSL. It manages the execution of interviews where agents interact with surveys through language models, possibly in different scenarios.

Key responsibilities: 1. Managing collections of agents, scenarios, and models 2. Configuring execution parameters (caching, API keys, etc.) 3. Managing parallel execution of interviews 4. Handling remote cache and inference capabilities 5. Collecting and organizing results

A typical workflow involves: 1. Creating a survey with questions 2. Creating a Jobs instance with that survey 3. Adding agents, scenarios, and models using the by() method 4. Running the job with run() or run\_async() 5. Analyzing the results

Jobs implements a fluent interface pattern, where methods return self to allow method chaining for concise, readable configuration.

> #### **init**(*survey: [Survey](/en/latest/surveys#survey-class "edsl.surveys.Survey")*, *agents: list\['Agent'] | 'AgentList' | None = None*, *models: 'ModelList' | list\['LanguageModel'] | None = None*, *scenarios: 'ScenarioList' | list\['Scenario'] | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Initialize a Jobs instance with a survey and optional components.

The Jobs constructor requires a survey and optionally accepts collections of agents, models, and scenarios. If any of these optional components are not provided, they can be added later using the by() method or will be automatically populated with defaults when the job is run.

### Parameters

**surveySurvey**

The survey containing questions to be used in the job

**agentsUnion\[list\[Agent], AgentList], optional**

The agents that will take the survey

**modelsUnion\[ModelList, list\[LanguageModel]], optional**

The language models to use

**scenariosUnion\[ScenarioList, list\[Scenario]], optional**

The scenarios to run

### Raises

**ValueError: If the survey contains questions with invalid names**

(e.g., names containing template variables)

### Examples

```python theme={null}
>>> from edsl.surveys import Survey
>>> from edsl.questions import QuestionFreeText
>>> q = QuestionFreeText(question_name="name", question_text="What is your name?")
>>> s = Survey(questions=[q])
>>> j = Jobs(survey = s)
>>> q = QuestionFreeText(question_name="{{ bad_name }}", question_text="What is your name?")
>>> s = Survey(questions=[q])
```

### Notes

* The survey’s questions must have valid names without templating variables
* If agents, models, or scenarios are not provided, defaults will be used when running
* Upon initialization, a RunConfig is created with default environment and parameters

> #### add\_running\_env(*running\_env: RunEnvironment*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Add a running environment to the job.

### Args:

**running\_env: A RunEnvironment object containing details about the execution**

environment like API keys and other configuration.

### Returns:

Jobs: The Jobs instance with the updated running environment.

### Example:

```python theme={null}
>>> from edsl import Cache
>>> job = Jobs.example()
>>> my_cache = Cache.example()
>>> env = RunEnvironment(cache=my_cache)
>>> j = job.add_running_env(env)
>>> j.run_config.environment.cache == my_cache
True
```

> #### *property* agents[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Get the agents associated with this job.

### Returns

AgentList: The agents for this job.

> #### all\_question\_parameters() → set[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Return all the fields in the questions in the survey.

```python theme={null}
>>> from edsl.jobs import Jobs
>>> Jobs.example().all_question_parameters()
{'period'}
```

> #### by(\*\*args: 'Agent' | 'Scenario' | 'LanguageModel' | Sequence\['Agent' | 'Scenario' | 'LanguageModel']\*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Add agents, scenarios, and language models to a job using a fluent interface.

This method is the primary way to configure a Jobs instance with components. It intelligently handles different types of objects and collections, making it easy to build complex job configurations with a concise syntax.

### Parameters

**[\*](/en/latest/#id4)argsUnion\[Agent, Scenario, LanguageModel, Sequence\[Union\[Agent, Scenario, LanguageModel]]]**

Objects or sequences of objects to add to the job. Supported types are Agent, Scenario, LanguageModel, and sequences of these.

### Returns

Jobs: The Jobs instance (self) for method chaining

### Examples

```python theme={null}
>>> from edsl.surveys import Survey
>>> from edsl.questions import QuestionFreeText
>>> q = QuestionFreeText(question_name="name", question_text="What is your name?")
>>> j = Jobs(survey = Survey(questions=[q]))
>>> j
Jobs(survey=Survey(...), agents=AgentList([]), models=ModelList([]), scenarios=ScenarioList([]))
>>> from edsl.agents import Agent; a = Agent(traits = {"status": "Sad"})
>>> j.by(a).agents
> AgentList([Agent(traits = {'status': 'Sad'})])
```

```
# Adding multiple components at once >>> from edsl.language_models import Model >>> from edsl.scenarios import Scenario >>> j = Jobs.example() >>> _ = j.by(Agent(traits=\{“mood”: “happy”})).by(Model(temperature=0.7)).by(Scenario(\{“time”: “morning”}))

# Adding a sequence of the same type >>> agents = [Agent(traits=\{“age”: i}) for i in range(5)] >>> _ = j.by(agents)
```

### Notes

* All objects must implement ‘get\_value’, ‘set\_value’, and ‘**add**’ methods
* Agent traits: When adding agents with traits to existing agents, the traits are combined. Avoid overlapping trait names to prevent unexpected behavior.
* Scenario traits: When adding scenarios with traits to existing scenarios, new traits overwrite existing ones with the same name.
* Models: New models with the same attributes will override existing models.
* The method detects object types automatically and routes them to the appropriate collection (agents, scenarios, or models).

> #### code()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Return the code to create this instance.

> #### *static* compute\_job\_cost(*job\_results: [Results](/en/latest/results#result-class "edsl.results.Results")*) → float[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Compute the cost of a completed job in USD.

> #### create\_bucket\_collection() → BucketCollection[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Create a collection of buckets for each model.

These buckets are used to track API calls and token usage. For test models and scripted response models, infinity buckets are used to avoid rate limiting delays.

```python theme={null}
>>> from edsl.jobs import Jobs
>>> from edsl import Model
>>> j = Jobs.example().by(Model(temperature = 1), Model(temperature = 0.5))
>>> bc = j.create_bucket_collection()
>>> bc
BucketCollection(...)
```

> #### duplicate()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Create a duplicate copy of this Jobs instance.

### Returns

Jobs: A new Jobs instance that is a copy of this one.

> #### estimate\_job\_cost(*iterations: int = 1*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Estimate the cost of running the job.

Parameters:

**iterations** – the number of iterations to run

> #### estimate\_job\_cost\_from\_external\_prices(*price\_lookup: dict*, *iterations: int = 1*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Estimate the cost of running the job using external price lookup.

### Args:

price\_lookup: Dictionary containing price information. iterations: Number of iterations to run.

### Returns:

dict: Cost estimation details.

> #### *static* estimate\_prompt\_cost(*system\_prompt: str*, *user\_prompt: str*, *price\_lookup: dict*, *inference\_service: str*, *model: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Estimate the cost of running the prompts.

Parameters:

* **iterations** – the number of iterations to run
* **system\_prompt** – the system prompt
* **user\_prompt** – the user prompt
* **price\_lookup** – the price lookup
* **inference\_service** – the inference service
* **model** – the model name

> #### *classmethod* example(*throw\_exception\_probability: float = 0.0*, *randomize: bool = False*, *test\_model=False*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Return an example Jobs instance.

Parameters:

* **throw\_exception\_probability** – the probability that an exception will be thrown when answering a question. This is useful for testing error handling.
* **randomize** – whether to randomize the job by adding a random string to the period
* **test\_model** – whether to use a test model

```python theme={null}
>>> Jobs.example()
```

Jobs(...)

> #### *classmethod* from\_dict(*data: dict*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Create a Jobs instance from a dictionary.

> #### *classmethod* from\_interviews(*interview\_list: list\['Interview']*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Return a Jobs instance from a list of interviews.

This is useful when you have, say, a list of failed interviews and you want to create a new job with only those interviews.

> #### generate\_interviews() → Generator[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Generate interviews.

<Note>
  **Note**:

  Note that this sets the agents, model and scenarios if they have not been set. This is a side effect of the method. This is useful because a user can create a job without setting the agents, models, or scenarios, and the job will still run, with us filling in defaults.
</Note>

> #### html()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Return the HTML representations for each scenario.

> #### humanize(*project\_name: str = 'Project'*, *scenario\_list\_method: Literal\['randomize', 'loop', 'single\_scenario', 'ordered'] | None = None*, *survey\_description: str | None = None*, *survey\_alias: str | None = None*, *survey\_visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*, *scenario\_list\_description: str | None = None*, *scenario\_list\_alias: str | None = None*, *scenario\_list\_visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Send the survey and scenario list to Expected Parrot.

Then, create a project on Expected Parrot so you can share the survey with human respondents.

> #### inspect()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Create an interactive inspector widget for this job.

> #### interviews() → list[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Return a list of `edsl.jobs.interviews.Interview` objects.

It returns one Interview for each combination of Agent, Scenario, and LanguageModel. If any of Agents, Scenarios, or LanguageModels are missing, it fills in with defaults.

```python theme={null}
>>> from edsl.jobs import Jobs
>>> j = Jobs.example()
>>> len(j.interviews())
4
>>> j.interviews()[0]
Interview(agent = Agent(traits = {'status': 'Joyful'}), survey = Survey(...), scenario = Scenario({'period': 'morning'}), model = Model(...))
```

> #### *property* models[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Get the models associated with this job.

### Returns

ModelList: The models for this job.

> #### *property* num\_interviews\*: int\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Calculate the total number of interviews that will be run.

```python theme={null}
>>> Jobs.example().num_interviews
4
```

This is the product of the number of scenarios, agents, and models, multiplied by the number of iterations specified in the run configuration.

> #### prompts(*iterations=1*) → [Dataset](/en/latest/dataset#edsl.dataset.Dataset "edsl.dataset.Dataset")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Return a Dataset of prompts that will be used.

```python theme={null}
>>> from edsl.jobs import Jobs
>>> Jobs.example().prompts()
Dataset(...)
```

> #### push(\**args*, \*\**kwargs*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Push the job to the remote server.

> #### replace\_missing\_objects() → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

If the agents, models, or scenarios are not set, replace them with defaults.

> #### run(\*\*\*, *config: RunConfig*) → 'Results' | None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Run the job by conducting interviews and return their results.

This is the main entry point for executing a job. It processes all interviews (combinations of agents, scenarios, and models) and returns a Results object containing all responses and metadata.

### Parameters

**configRunConfig**

Configuration object containing runtime parameters and environment settings

**nint, optional**

Number of iterations to run each interview (default: 1)

**progress\_barbool, optional**

Whether to show a progress bar (default: False)

**stop\_on\_exceptionbool, optional**

Whether to stop the job if an exception is raised (default: False)

**check\_api\_keysbool, optional**

Whether to verify API keys before running (default: False)

**verbosebool, optional**

Whether to print extra messages during execution (default: True)

**print\_exceptionsbool, optional**

Whether to print exceptions as they occur (default: True)

**remote\_cache\_descriptionstr, optional**

Description for entries in the remote cache

**remote\_inference\_descriptionstr, optional**

Description for the remote inference job

**remote\_inference\_results\_visibilityVisibilityType, optional**

Visibility of results on Expected Parrot (“private”, “public”, “unlisted”)

**disable\_remote\_cachebool, optional**

Whether to disable the remote cache (default: False)

**disable\_remote\_inferencebool, optional**

Whether to disable remote inference (default: False)

**freshbool, optional**

Whether to ignore cache and force new results (default: False)

**skip\_retrybool, optional**

Whether to skip retrying failed interviews (default: False)

**raise\_validation\_errorsbool, optional**

Whether to raise validation errors (default: False)

**backgroundbool, optional**

Whether to run in background mode (default: False)

**job\_uuidstr, optional**

UUID for the job, used for tracking

**cacheCache, optional**

Cache object to store results

**bucket\_collectionBucketCollection, optional**

Object to track API keys

**key\_lookupKeyLookup, optional**

Object to manage API keys

**memory\_thresholdint, optional**

Memory threshold in bytes for the Results object’s SQLList, controlling when data is offloaded to SQLite storage

**new\_formatbool, optional**

If True, uses remote\_inference\_create method, if False uses old\_remote\_inference\_create method (default: True)

**expected\_parrot\_api\_keystr, optional**

Custom EXPECTED\_PARROT\_API\_KEY to use for this job run

### Returns

Results: A Results object containing all responses and metadata

### Notes

* This method will first try to use remote inference if available
* If remote inference is not available, it will run locally
* For long-running jobs, consider using progress\_bar=True
* For maximum performance, ensure appropriate caching is configured

Example:

```python theme={null}
>>> from edsl.jobs import Jobs
>>> from edsl.caching import Cache
>>> job = Jobs.example()
>>> from edsl import Model
>>> m = Model('test')
>>> results = job.by(m).run(cache=Cache(), progress_bar=False, n=2, disable_remote_inference=True)
...
```

> #### run\_async(\*\*\*, *config: RunConfig*) → [Results](/en/latest/results#result-class "edsl.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Asynchronously runs the job by conducting interviews and returns their results.

This method is the asynchronous version of run(). It has the same functionality and parameters but can be awaited in an async context for better integration with asynchronous code.

### Parameters

**configRunConfig**

Configuration object containing runtime parameters and environment settings

**nint, optional**

Number of iterations to run each interview (default: 1)

**progress\_barbool, optional**

Whether to show a progress bar (default: False)

**stop\_on\_exceptionbool, optional**

Whether to stop the job if an exception is raised (default: False)

**check\_api\_keysbool, optional**

Whether to verify API keys before running (default: False)

**verbosebool, optional**

Whether to print extra messages during execution (default: True)

**print\_exceptionsbool, optional**

Whether to print exceptions as they occur (default: True)

**remote\_cache\_descriptionstr, optional**

Description for entries in the remote cache

**remote\_inference\_descriptionstr, optional**

Description for the remote inference job

**remote\_inference\_results\_visibilityVisibilityType, optional**

Visibility of results on Expected Parrot (“private”, “public”, “unlisted”)

**disable\_remote\_cachebool, optional**

Whether to disable the remote cache (default: False)

**disable\_remote\_inferencebool, optional**

Whether to disable remote inference (default: False)

**freshbool, optional**

Whether to ignore cache and force new results (default: False)

**skip\_retrybool, optional**

Whether to skip retrying failed interviews (default: False)

**raise\_validation\_errorsbool, optional**

Whether to raise validation errors (default: False)

**backgroundbool, optional**

Whether to run in background mode (default: False)

**job\_uuidstr, optional**

UUID for the job, used for tracking

**cacheCache, optional**

Cache object to store results

**bucket\_collectionBucketCollection, optional**

Object to track API calls

**key\_lookupKeyLookup, optional**

Object to manage API keys

**memory\_thresholdint, optional**

Memory threshold in bytes for the Results object’s SQLList, controlling when data is offloaded to SQLite storage

**new\_formatbool, optional**

If True, uses remote\_inference\_create method, if False uses old\_remote\_inference\_create method (default: True)

**expected\_parrot\_api\_keystr, optional**

Custom EXPECTED\_PARROT\_API\_KEY to use for this job run

### Returns

Results: A Results object containing all responses and metadata

### Notes

* This method should be used in async contexts (e.g., with await)
* For non-async contexts, use the run() method instead
* This method is particularly useful in notebook environments or async applications

Example:

```python theme={null}
>>> import asyncio
>>> from edsl.jobs import Jobs
>>> from edsl.caching import Cache
>>> job = Jobs.example()
>>> # In an async context
>>> async def run_job():
...     results = await job.run_async(cache=Cache(), progress_bar=True)
...     return results
```

> #### *property* scenarios\*: [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Get the scenarios associated with this job.

### Returns

ScenarioList: The scenarios for this job.

> #### show\_flow(*filename: str | None = None*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Visualize either the *Job* dependency/post-processing flow **or** the underlying survey flow.

The method automatically decides which flow to render:

1. If the job has dependencies created via [`Jobs.to()`](/en/latest/#edsl.jobs.Jobs.to "edsl.jobs.Jobs.to") (i.e. `_depends_on` is not *None*) **or** has post-run methods queued in `_post_run_methods`, the *job* flow (dependencies → post-processing chain) is rendered using `edsl.jobs.job_flow_visualization.JobsFlowVisualization`.
2. Otherwise, it falls back to the original behaviour and shows the survey question flow using `edsl.surveys.survey_flow_visualization.SurveyFlowVisualization`.

```python theme={null}
>>> from edsl.jobs import Jobs
>>> job = Jobs.example()
>>> job.show_flow()  # Visualises survey flow (no deps/post-run methods)
>>> job2 = job.select('how_feeling').to_pandas()  # add post-run methods
>>> job2.show_flow()  # Now visualises job flow
```

> #### show\_prompts(*all: bool = False*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Print the prompts.

> #### table()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Return a table view of the job’s prompts.

### Returns

Table representation of the job’s prompts.

> #### then(*method\_name*, \**args*, \*\**kwargs*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Schedule a method to be called on the results object after the job runs.

This allows for method chaining like: jobs.then(‘to\_scenario\_list’).then(‘to\_pandas’).then(‘head’, 10)

### Args:

method\_name: Name of the method to call on the results [\*](/en/latest/#id22)args: Positional arguments to pass to the method [\*\*](/en/latest/#id24)kwargs: Keyword arguments to pass to the method

> #### to(*question\_or\_survey\_or\_jobs: 'Question' | 'Survey' | 'Jobs'*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Create a new [`Jobs`](/en/latest/#edsl.jobs.Jobs "edsl.jobs.Jobs") instance from *self* and a target object.

The target can be one of the following:

* Question – A single question which will be wrapped in a one-question survey.
* Survey – A survey object that will be used directly.
* Jobs – An existing *Jobs* object. In this case the target *Jobs* is **returned unchanged**, but its `_depends_on` attribute is set to reference *self*, establishing an execution dependency chain.

### Args:

**question\_or\_survey\_or\_jobs (Union\[Question, Survey, Jobs]):**
The object used to build (or identify) the new *Jobs* instance.

### Returns:

Jobs: A new *Jobs* instance that depends on the current instance, or the target *Jobs* instance when the target itself is a *Jobs*.

### Raises:

**ValueError: If *question\_or\_survey\_or\_jobs* is not one of the**
supported types.

### Examples:

The following doctest demonstrates sending one job to another and verifying the dependency link via the private `_depends_on` attribute:

```python theme={null}
>>> from edsl.jobs import Jobs
>>> base_job = Jobs.example()
>>> downstream_job = Jobs.example()
>>> new_job = base_job.to(downstream_job)
>>> new_job is downstream_job  # the same object is returned
True
>>> new_job._depends_on is base_job  # dependency recorded
True
```

> #### to\_dict(*add\_edsl\_version=True*, *full\_dict=None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Convert the Jobs instance to a dictionary representation.

### Args:

add\_edsl\_version: Whether to include EDSL version information. full\_dict: Additional dictionary to merge (currently unused).

### Returns:

dict: Dictionary representation of this Jobs instance.

> #### use\_remote\_cache() → bool[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Determine whether to use remote cache for this job.

### Returns

bool: True if remote cache should be used, False otherwise.

> #### using(*obj*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Add a Cache, BucketCollection, or KeyLookup object to the job.

### Args:

**obj: The object to add to the job’s configuration. Must be one of:**

Cache, BucketCollection, or KeyLookup.

### Returns:

Jobs: The Jobs instance with the updated configuration object.

> #### using\_bucket\_collection(*bucket\_collection: BucketCollection*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Add a BucketCollection object to the job.

### Args:

bucket\_collection: The BucketCollection object to add to the job’s configuration.

### Returns:

Jobs: The Jobs instance with the updated bucket collection.

> #### using\_cache(*cache: [Cache](/en/latest/data#cache-class "edsl.caching.Cache")*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Add a Cache object to the job.

### Args:

cache: The Cache object to add to the job’s configuration.

### Returns:

Jobs: The Jobs instance with the updated cache.

> #### using\_key\_lookup(*key\_lookup: KeyLookup*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Add a KeyLookup object to the job.

### Args:

key\_lookup: The KeyLookup object to add to the job’s configuration.

### Returns:

Jobs: The Jobs instance with the updated key lookup.

> #### where(*expression: str*) → [Jobs](/en/latest/#edsl.jobs.Jobs "edsl.jobs.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/jobs.py)

Filter the agents, scenarios, and models based on a condition.

**Parameters**: **expression** – a condition to filter the agents, scenarios, and models


# Language Models
Source: https://docs.expectedparrot.com/en/latest/language_models

EDSL works with many models from popular inference service providers, including Anthropic, Azure, Bedrock, Deep Infra, DeepSeek, Google, Mistral, OpenAI, Perplexity, Together and Xai. Current pricing and performance information is on the [models page](https://www.expectedparrot.com/getting-started/models).

We also recommend checking providers’ websites for the most up-to-date information on models and service providers’ terms of use. Links to providers’ websites can be found at the [models page](https://www.expectedparrot.com/getting-started/models). If you need assistance checking whether a model is working or to report a missing model or price, please send a message to [info@expectedparrot.com](mailto:info@expectedparrot.com) or post a message on [Discord](https://discord.com/invite/mxAYkjfy9m).

This page provides examples of methods for specifying models for surveys using the Model and ModelList classes.

## API keys

In order to use a model, you need to have an API key for the relevant service provider. EDSL allows you to choose whether to provide your own keys from service providers or use an Expected Parrot API key to access all available models at once. See the [Managing Keys](/en/latest/api_keys) page for instructions on storing and prioritizing keys.

## Available services

The following code will return a table of inference service providers:

```python theme={null}
from edsl import Model

Model.services()
```

Output:

| Service Name |
| :- |
| anthropic |
| azure |
| bedrock |
| deep\_infra |
| deepseek |
| google |
| groq |
| mistral |
| ollama |
| openai |
| openai\_compatible |
| perplexity |
| together |
| xai |

<Note>
  **Note**:

  We recently added support for OpenAI reasoning models. See an example notebook for usage [here](https://www.expectedparrot.com/content/RobinHorton/reasoning-model-example). Use service\_name = “openai\_v2” when using these models. The Results that are generated with reasoning models include additional fields for reasoning summaries.
</Note>

## OpenAI reasoning settings

Both `openai` (Chat Completions) and `openai_v2` (Responses) accept
`reasoning_effort`, including for the GPT-6 family:

```python theme={null}
model = Model(
    "gpt-6-astra",
    service_name="openai_v2",
    reasoning_effort="xhigh",
    max_tokens=64000,
)
```

EDSL sends this as `reasoning_effort` for Chat Completions and as
`reasoning.effort` for Responses. With `openai_v2`, you can also supply a
`reasoning` dictionary, such as `{"effort": "high", "summary": "auto"}`.
A non-`None` `reasoning_effort` overrides the dictionary's `effort`; other
dictionary fields are preserved.

Explicit effort values are forwarded unchanged for provider validation, even
when EDSL does not recognize the model. Unsupported values produce a provider
error rather than being silently removed or reduced. Supported effort values
depend on the model; see [OpenAI's model guidance](https://developers.openai.com/api/docs/guides/latest-model).
`reasoning` must be a dictionary or `None`.

When effort is omitted or `None`, Chat Completions uses `medium` for recognized
reasoning models. Responses requests an automatic reasoning summary and leaves
effort to the provider unless the dictionary specifies it. Ordinary models get
no reasoning parameters unless explicitly requested.

Explicit `max_tokens` values are preserved, including small limits. The current
reasoning-model defaults are 5,000 for Chat Completions and 16,000 for Responses;
ordinary-model defaults are 1,000 and 2,000, respectively.

Remote workers must run the corrected adapter implementation too: serializing a
model or Jobs object only saves configuration. When verifying an adapter change,
use a fresh call with cache bypass and inspect the effective provider request.

## Local and self-hosted models

Use the `openai_compatible` service to connect EDSL to a local or self-hosted server that implements the OpenAI chat-completions API, including llama.cpp, Ollama, LM Studio, vLLM, and SGLang. Connection details can be saved in a portable `ModelList` without storing an API-key value.

See [Local OpenAI-compatible models](/en/latest/local_openai_compatible) for setup, CLI examples, concurrency guidance, and troubleshooting.

## Specifying a model

To specify a model to use with a survey, create a Model object and pass it the name of the model. You can optionally set other model parameters at the same time (temperature, etc.). You will sometimes need to specify the name of the service provider as well (for instance, if the model is hosted by multiple service providers).

For example, the following code creates a Model object for gpt-4o with default model parameters that we can inspect:

```python theme={null}
from edsl import Model

m = Model("gpt-4o")
```

This is equivalent:

```python theme={null}
from edsl import Model

m = Model(model = "gpt-4o", service_name = "openai")
m
```

Output:

| key | value |
| :- | :- |
| model | gpt-4o |
| parameters:temperature | 0.5 |
| parameters:max\_tokens | 1000 |
| parameters:top\_p | 1 |
| parameters:frequency\_penalty | 0 |
| parameters:presence\_penalty | 0 |
| parameters:logprobs | False |
| parameters:top\_logprobs | 3 |
| inference\_service | openai |

We can see that the object consists of a model name and a dictionary of the default parameters of the model, together with the name of the inference service (some models are provided by multiple services).

Here we also specify the temperature when creating the Model object:

```python theme={null}
from edsl import Model

m = Model("gpt-4o", service_name = "openai", temperature = 1.0)
m
```

Output:

| key | value |
| :- | :- |
| model | gpt-4o |
| parameters:temperature | 1.0 |
| parameters:max\_tokens | 1000 |
| parameters:top\_p | 1 |
| parameters:frequency\_penalty | 0 |
| parameters:presence\_penalty | 0 |
| parameters:logprobs | False |
| parameters:top\_logprobs | 3 |
| inference\_service | openai |

## Creating a list of models

To create a list of models at once, pass a list of model names to a ModelList object.

For example, the following code creates a Model for each of gpt-4o and gemini-pro:

```python theme={null}
from edsl import Model, ModelList

ml = ModelList([
  Model("gpt-4o", service_name = "openai"),
  Model("gemini-1.5-flash", service_name = "google")
])
```

This code is equivalent to the following:

```python theme={null}
from edsl import Model, ModelList

ml = ModelList(Model(model) for model in ["gpt-4o", "gemini-1.5-flash"])
```

We can also use a special method to pass a list of names instead:

```python theme={null}
from edsl import Model, ModelList

model_names = ['gpt-4o', 'gemini-1.5-flash']

ml = ModelList.from_names(model_names)

ml
```

Output:

| topK | presence\_penalty | top\_logprobs | topP | temperature | stopSequences | maxOutputTokens | logprobs | max\_tokens | frequency\_penalty | model | top\_p | inference\_service |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| nan | 0.000000 | 3.000000 | nan | 0.500000 | nan | nan | False | 1000.000000 | 0.000000 | gpt-4o | 1.000000 | openai |
| 1.000000 | nan | nan | 1.000000 | 0.500000 | \[] | 2048.000000 | nan | nan | nan | gemini-1.5-flash | nan | google |

## Running a survey with models

Similar to how we specify [Agents](/en/latest/agents) and [Scenarios](/en/latest/scenarios) to use with a survey, we specify the models to use by adding them to a survey with the by() method when the survey is run. We can pass either a single Model object or a list of models to the by() method. If multiple models are to be used they are passed as a list or as a ModelList object.

For example, the following code specifies that a survey will be run with each of gpt-4o and gemini-1.5-flash:

```python theme={null}
from edsl import Model, QuestionFreeText, Survey

m = [Model("gpt-4o", service_name = "openai"), Model("gemini-1.5-flash", service_name = "google")]

q = QuestionFreeText(
  question_name = "example",
  question_text = "What is the capital of France?"
)

survey = Survey(questions = [q])

results = survey.by(m).run()
```

This code uses ModelList instead of a list of Model objects:

```python theme={null}
from edsl import Model, ModelList, QuestionFreeText, Survey

ml = ModelList(Model(model) for model in ["gpt-4o", "gemini-1.5-flash"])

q = QuestionFreeText(
  question_name = "example",
  question_text = "What is the capital of France?"
)

survey = Survey(questions = [q])

results = survey.by(ml).run()
```

This will generate a result for each question in the survey with each model. If agents and/or scenarios are also specified, the responses will be generated for each combination of agents, scenarios and models. Each component is added with its own by() method, the order of which does not matter. The following commands are equivalent:

```python theme={null}
# add code for creating survey, scenarios, agents, models here ...

results = survey.by(scenarios).by(agents).by(models).run()

# this is equivalent:
results = survey.by(models).by(agents).by(scenarios).run()
```

## Default model

If no model is specified, a survey is automatically run with the default model. Run Model() to check the current default model.

For example, the following code runs the above survey with the default model (and no agents or scenarios) without needing to import the Model class:

```python theme={null}
results = survey.run() # using the survey from above

# this is equivalent
results = survey.by(Model()).run()
```

We can verify the model that was used:

```python theme={null}
results.select("model.model") # selecting only the model name
```

Output:

| model |
| :- |
| gpt-4o |

## Inspecting model parameters

We can also inspect parameters of the models that were used by calling the models of the Results object.

For example, we can verify the default model when running a survey without specifying a model:

```python theme={null}
results.models # using the results from above
```

This will return the same information as running results.select(“model.model”) in the example above.

To learn more about all the components of a Results object, please see the [Results](/en/latest/results) section.

## Troubleshooting

Newly released models of service providers are automatically made available to use with your surveys whenever possible (not all service providers facilitate this).

If you do not see a model that you want to work with or are unable to instantiate it using the standard method, please send us a request to add it to *[info@expectedparrot.com](mailto:info@expectedparrot.com)*.

## ModelList class

> ### *class* edsl.language\_models.ModelList(*data: [LanguageModel](/en/latest/#edsl.language_models.LanguageModel "edsl.language_models.LanguageModel") | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Bases: [`Base`](/en/latest/base#base-class "edsl.base.base_class.Base"), `UserList`

> #### **init**(*data: [LanguageModel](/en/latest/#edsl.language_models.LanguageModel "edsl.language_models.LanguageModel") | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Initialize the ModelList class.

```python theme={null}
>>> from edsl import Model
>>> m = ModelList.from_scenario_list(Model.available())
```

> #### *classmethod* all() → [ModelList](/en/latest/#edsl.language_models.ModelList "edsl.language_models.model_list.ModelList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Returns all available models.

> #### code() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Generate Python code that recreates this object.

This method must be implemented by all subclasses to provide a way to generate executable Python code that can recreate the object.

**Returns:**

str: Python code that, when executed, creates an equivalent object

> #### *classmethod* example(*randomize: bool = False*) → [ModelList](/en/latest/#edsl.language_models.ModelList "edsl.language_models.model_list.ModelList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Returns an example ModelList instance.

**Parameters:**

**randomize** – If True, uses Model’s randomize method.

> #### filter(*expression: str*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

> #### *classmethod* from\_available\_models(*available\_models\_list: AvailableModels*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Create a ModelList from an AvailableModels object

> #### *classmethod* from\_dict(*data*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Create a ModelList from a dictionary.

```python theme={null}
>>> newm = ModelList.from_dict(ModelList.example().to_dict())
>>> assert ModelList.example() == newm
```

> #### *classmethod* from\_names(\**args*, \*\**kwargs*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

A a model list from a list of names

> #### *classmethod* from\_scenario\_list(*scenario\_list*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Create a ModelList from a ScenarioList containing model\_name and service\_name fields.

> **Args:**

scenario\_list: ScenarioList with scenarios containing ‘model\_name’ and ‘service\_name’ fields

> **Returns:**

ModelList with instantiated Model objects

> **Example:**

```python theme={null}
>>> from edsl import Model
>>> models_data = Model.available(service_name='openai')
>>> model_list = ModelList.from_scenario_list(models_data)
```

> #### *property* names [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

```python theme={null}
>>> ModelList.example().names
{'...'}
```

> #### table(\**fields*, *tablefmt: str | None = None*, *pretty\_labels: dict | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

```python theme={null}
>>> ModelList.example().table('model_name')
model_name
------------
gpt-4o
gpt-4o
gpt-4o
```

> #### to\_dict(*sort=False*, *add\_edsl\_version=True*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Serialize this object to a dictionary.

This method must be implemented by all subclasses to provide a standard way to serialize objects to dictionaries. The dictionary should contain all the data needed to reconstruct the object.

> **Returns:**

dict: A dictionary representation of the object

> #### to\_list() → list [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

> #### to\_scenario\_list() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

> #### tree(*node\_list: List\[str] | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

## LanguageModel class

> ### *class* edsl.language\_models.LanguageModel(*tpm: float | None = None*, *rpm: float | None = None*, *omit\_system\_prompt\_if\_empty\_string: bool = True*, *key\_lookup: 'KeyLookup' | None = None*, \*\**kwargs*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Bases: `PersistenceMixin`, `RepresentationMixin`, `HashingMixin`, `DiffMethodsMixin`, `ABC`

Abstract base class for all language model implementations in EDSL.

This class defines the common interface and functionality for interacting with various language model providers (OpenAI, Anthropic, etc.). It handles caching, response parsing, token usage tracking, and cost calculation, providing a consistent interface regardless of the underlying model.

Subclasses must implement the async\_execute\_model\_call method to handle the actual API call to the model provider. Other methods may also be overridden to customize behavior for specific models.

The class uses several mixins to provide serialization, pretty printing, and hashing functionality, and a metaclass to automatically register model implementations.

> **Attributes:**

\_model\_: The default model identifier (set by subclasses) key\_sequence: Path to extract generated text from model responses DEFAULT\_RPM: Default requests per minute rate limit DEFAULT\_TPM: Default tokens per minute rate limit

> #### DEFAULT\_RPM *= 100* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

> #### DEFAULT\_TPM *= 1000* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

> #### **init**(*tpm: float | None = None*, *rpm: float | None = None*, *omit\_system\_prompt\_if\_empty\_string: bool = True*, *key\_lookup: 'KeyLookup' | None = None*, \*\**kwargs*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Initialize a new language model instance.

**Args:**

tpm: Optional tokens per minute rate limit override rpm: Optional requests per minute rate limit override omit\_system\_prompt\_if\_empty\_string: Whether to omit the system prompt when empty key\_lookup: Optional custom key lookup for API credentials [\*\*](/en/latest/#id2)kwargs: Additional parameters to pass to the model provider

The initialization process: 1. Sets up the model identifier from the class attribute 2. Configures model parameters by merging defaults with provided values 3. Sets up API key lookup and rate limits 4. Applies all parameters as instance attributes

For subclasses that define *parameters* class attribute, these will be used as default parameters that can be overridden by kwargs.

> #### *property* api\_token\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Get the API token for this model’s service.

This property lazily fetches the API token from the key lookup mechanism when first accessed, caching it for subsequent uses.

> **Returns:**

str: The API token for authenticating with the model provider

> **Raises:**

ValueError: If no API key is found for this model’s service

> #### ask\_question(*question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase")*) → str [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Ask a question using this language model and return the response.

This is a convenience method that extracts the necessary prompts from a question object and makes a model call.

**Args:**

question: The EDSL question object to ask

**Returns:**

str: The model’s response to the question

> #### *abstract async* async\_execute\_model\_call(*user\_prompt: str*, *system\_prompt: str*, *question\_name: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Execute the model call asynchronously.

This abstract method must be implemented by all model subclasses to handle the actual API call to the language model provider.

**Args:**

user\_prompt: The user message or input prompt system\_prompt: The system message or context question\_name: Optional name of the question being asked (primarily used for test models)

**Returns:**

Coroutine that resolves to the model response

**Note:**

Implementations should handle the actual API communication, including authentication, request formatting, and response parsing.

> #### *async* async\_get\_response(*user\_prompt: str*, *system\_prompt: str*, *cache: [Cache](/en/latest/data#cache-class "edsl.caching.Cache")*, *iteration: int = 1*, *files\_list: List\[[FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore")] | None = None*, \*\**kwargs*) → AgentResponseDict [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Get a complete response with all metadata and parsed format.

This method handles the complete pipeline for: 1. Making a model call (with caching) 2. Parsing the response 3. Constructing a full response object with inputs, outputs, and parsed data

It’s the primary method used by higher-level components to interact with models.

**Args:**

user\_prompt: The user’s message or input prompt system\_prompt: The system’s message or context cache: The cache object to use for storing/retrieving responses iteration: The iteration number (default: 1) files\_list: Optional list of files to include in the prompt [\*\*](/en/latest/#id4)kwargs: Additional parameters (invigilator can be provided here)

**Returns:**

AgentResponseDict: Complete response object with inputs, raw outputs, and parsed data

> #### copy() → [LanguageModel](/en/latest/#edsl.language_models.LanguageModel "edsl.language_models.language_model.LanguageModel") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Create a deep copy of this language model instance.

This method creates a completely independent copy of the language model by creating a new instance with the same parameters and copying relevant attributes.

> **Returns:**

LanguageModel: A new language model instance that is functionally identical to this one

> **Examples:**

```python theme={null}
>>> m1 = LanguageModel.example()
>>> m2 = m1.copy()
>>> m1 == m2  # Functionally equivalent
True
>>> id(m1) == id(m2)  # But different objects
False
```

> #### cost(*raw\_response: dict\[str, Any]*) → ResponseCost [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Calculate the monetary cost of a model API call.

This method extracts token usage information from the response and uses the price manager to calculate the actual cost in dollars based on the model’s pricing structure and token counts.

> **Args:**

raw\_response: The complete response dictionary from the model API

> **Returns:**

ResponseCost: Object containing token counts and total cost

> #### *classmethod* example(*test\_model: bool = False*, *canned\_response: str = 'Hello world'*, *throw\_exception: bool = False*) → [LanguageModel](/en/latest/#edsl.language_models.LanguageModel "edsl.language_models.language_model.LanguageModel") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Create an example language model instance for testing and demonstration.

This method provides a convenient way to create a model instance for examples, tests, and documentation. It can create either a real model (with API key checking disabled) or a test model that returns predefined responses.

> **Args:**

test\_model: If True, creates a test model that doesn’t make real API calls canned\_response: For test models, the predefined response to return throw\_exception: For test models, whether to throw an exception instead of responding

> **Returns:**

LanguageModel: An example model instance

> **Examples:**

Create a test model with a custom response:

```python theme={null}
>>> from edsl.language_models import LanguageModel
>>> m = LanguageModel.example(test_model=True, canned_response="WOWZA!")
>>> isinstance(m, LanguageModel)
True
```

Use the test model to answer a question:

```python theme={null}
>>> from edsl import QuestionFreeText
>>> q = QuestionFreeText(question_text="What is your name?", question_name='example')
>>> q.by(m).run(cache=False, disable_remote_cache=True, disable_remote_inference=True).select('example').first()
'WOWZA!'
```

Create a test model that throws exceptions:

```python theme={null}
>>> m = LanguageModel.example(test_model=True, canned_response="WOWZA!", throw_exception=True)
>>> r = q.by(m).run(cache=False, disable_remote_cache=True, disable_remote_inference=True, print_exceptions=True)
Exception report saved to ...
```

> #### execute\_model\_call(\*\**kwargs*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

> #### from\_cache(*cache: [Cache](/en/latest/data#cache-class "edsl.caching.Cache")*) → [LanguageModel](/en/latest/#edsl.language_models.LanguageModel "edsl.language_models.LanguageModel") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Create a new model that only returns responses from the cache.

This method creates a modified copy of the model that will only use cached responses, never making new API calls. This is useful for offline operation or repeatable experiments.

> **Args:**

cache: The cache object containing previously cached responses

> **Returns:**

LanguageModel: A new model instance that only reads from cache

> #### *classmethod* from\_dict(*data: dict*) → [LanguageModel](/en/latest/#edsl.language_models.LanguageModel "edsl.language_models.language_model.LanguageModel") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Create a language model instance from a dictionary representation.

This class method deserializes a model from its dictionary representation, using the inference service registry to find the correct model class.

> **Args:**

data: Dictionary containing the model configuration

> **Returns:**

LanguageModel: A new model instance of the appropriate type

> #### *classmethod* from\_scripted\_responses(*agent\_question\_responses: dict\[str, dict\[str, str]]*) → [LanguageModel](/en/latest/#edsl.language_models.LanguageModel "edsl.language_models.language_model.LanguageModel") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Create a language model with scripted responses for specific agent-question combinations.

This method creates a specialized model that returns predetermined responses based on the agent name and question name combination. This is useful for testing scenarios where you want to control exactly how different agents respond to different questions.

> **Args:**

**agent\_question\_responses: Nested dictionary mapping agent names to question names**

to responses. Format: \{‘agent\_name’: \{‘question\_name’: ‘response’}}

> **Returns:**

LanguageModel: A scripted response model

> **Examples:**

Create a model with scripted responses for different agents:

```python theme={null}
>>> from edsl.language_models import LanguageModel
>>> responses = {
...     'alice': {'favorite_color': 'blue', 'age': '25'},
...     'bob': {'favorite_color': 'red', 'age': '30'}
... }
>>> m = LanguageModel.from_scripted_responses(responses)
>>> isinstance(m, LanguageModel)
True
```

The model will return the appropriate response based on agent and question:

```python theme={null}
>>> # When used with agent 'alice' and question 'favorite_color', returns 'blue'
>>> # When used with agent 'bob' and question 'age', returns '30'
```

> #### *classmethod* get\_generated\_token\_string(*raw\_response: dict\[str, Any]*) → str [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Extract the generated text from a raw model response.

This method navigates the response structure using the model’s key\_sequence to find and return just the generated text, without metadata.

> **Args:**

raw\_response: The complete response dictionary from the model API

> **Returns:**

str: The generated text string

> **Examples:**

```python theme={null}
>>> m = LanguageModel.example(test_model=True)
>>> raw_response = m.execute_model_call("Hello, model!", "You are a helpful agent.")
>>> m.get_generated_token_string(raw_response)
'Hello world'
```

> #### get\_response(*user\_prompt: str*, *system\_prompt: str*, *cache: [Cache](/en/latest/data#cache-class "edsl.caching.Cache")*, *iteration: int = 1*, *files\_list: List\[[FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore")] | None = None*, \*\**kwargs*) → AgentResponseDict [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Get a complete response with all metadata and parsed format.

This method handles the complete pipeline for: 1. Making a model call (with caching) 2. Parsing the response 3. Constructing a full response object with inputs, outputs, and parsed data

It’s the primary method used by higher-level components to interact with models.

**Args:**

user\_prompt: The user’s message or input prompt system\_prompt: The system’s message or context cache: The cache object to use for storing/retrieving responses iteration: The iteration number (default: 1) files\_list: Optional list of files to include in the prompt [\*\*](/en/latest/#id6)kwargs: Additional parameters (invigilator can be provided here)

**Returns:**

AgentResponseDict: Complete response object with inputs, raw outputs, and parsed data

> #### *classmethod* get\_usage\_dict(*raw\_response: dict\[str, Any]*) → dict\[str, Any] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Extract token usage statistics from a raw model response.

This method navigates the response structure to find and return information about token usage, which is used for cost calculation and monitoring.

**Args:**

raw\_response: The complete response dictionary from the model API

**Returns:**

dict: Dictionary of token usage statistics (input tokens, output tokens, etc.)

> #### has\_valid\_api\_key() → bool [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Check if the model has a valid API key available.

This method verifies if the necessary API key is available in environment variables or configuration for this model’s service. Test models always return True.

> **Returns:**

bool: True if a valid API key is available, False otherwise

> **Examples:**

```python theme={null}
>>> LanguageModel.example().has_valid_api_key() :
True
```

> #### hello(*verbose=False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Run a simple test to verify the model connection is working.

This method makes a basic model call to check if the API credentials are valid and the model is responsive.

**Args:**

verbose: If True, prints the masked API token

**Returns:**

str: The model’s response to a simple greeting

> #### key\_sequence\*: tuple\[str, ...]\* *= None* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

> #### *classmethod* parse\_response(*raw\_response: dict\[str, Any]*) → EDSLOutput [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Parse the raw API response into a standardized EDSL output format.

This method processes the model’s response to extract the generated content and format it according to EDSL’s expected structure, making it consistent across different model providers.

**Args:**

raw\_response: The complete response dictionary from the model API

**Returns:**

EDSLOutput: Standardized output structure with answer and optional comment

> #### *async* remote\_async\_execute\_model\_call(*user\_prompt: str*, *system\_prompt: str*, *question\_name: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Execute the model call remotely through the Expected Parrot service.

This method allows offloading the model call to a remote server, which can be useful for models not available in the local environment or to avoid rate limits.

**Args:**

user\_prompt: The user message or input prompt system\_prompt: The system message or context question\_name: Optional name of the question being asked (primarily used for test models)

**Returns:**

Coroutine that resolves to the model response from the remote service

> #### response\_handler *= `<edsl.language_models.raw_response_handler.RawResponseHandler object>`* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

> #### *property* rpm [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Get the requests per minute rate limit for this model.

This property provides the rate limit either from an explicitly set value, from the model info in the key lookup, or from the default value.

**Returns:**

float: The requests per minute rate limit

> #### set\_key\_lookup(*key\_lookup: KeyLookup*) → None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Update the key lookup mechanism after initialization.

This method allows changing the API key lookup after the model has been created, clearing any cached API tokens.

**Args:**

key\_lookup: The new key lookup object to use

> #### simple\_ask(*question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase")*, *system\_prompt='You are a helpful agent pretending to be a human.'*, *top\_logprobs=2*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Ask a simple question with log probability tracking.

This is a convenience method for getting responses with log probabilities, which can be useful for analyzing model confidence and alternatives.

**Args:**

question: The EDSL question object to ask system\_prompt: System message to use (default is human agent instruction) top\_logprobs: Number of top alternative tokens to return probabilities for

**Returns:**

The model response, including log probabilities if supported

> #### to\_dict(*add\_edsl\_version: bool = True*) → dict\[str, Any] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Serialize the model instance to a dictionary representation.

This method creates a dictionary containing all the information needed to recreate this model, including its identifier, parameters, and service. Optionally includes EDSL version information for compatibility checking.

> **Args:**

add\_edsl\_version: Whether to include EDSL version and class name (default: True)

> **Returns:**

dict: Dictionary representation of this model instance

> **Examples:**

```python theme={null}
>>> m = LanguageModel.example()
>>> m.to_dict()
{'model': '...', 'parameters': {'temperature': ..., 'max_tokens': ..., 'top_p': ..., 'frequency_penalty': ..., 'presence_penalty': ..., 'logprobs': False, 'top_logprobs': ...}, 'inference_service': 'openai', 'edsl_version': '...', 'edsl_class_name': 'LanguageModel'}
```

> #### *property* tpm [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models.py)

Get the tokens per minute rate limit for this model.

This property provides the rate limit either from an explicitly set value, from the model info in the key lookup, or from the default value.

**Returns:**

float: The tokens per minute rate limit

## Other methods

> ### *class* edsl.language\_models.registry.RegisterLanguageModelsMeta(*name*, *bases*, *namespace*, */*, \*\**kwargs*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models/registry.py)

Bases: `ABCMeta`

Metaclass to register output elements in a registry i.e., those that have a parent.

> #### REQUIRED\_CLASS\_ATTRIBUTES *= \['*model*', '*parameters*', '*inference\_service*']* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models/registry.py)

> #### **init**(*name*, *bases*, *dct*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models/registry.py)

Register the class in the registry if it has a *model* attribute.

> #### *static* check\_required\_class\_variables(*candidate\_class: [LanguageModel](/en/latest/#edsl.language_models.LanguageModel "edsl.language_models.LanguageModel")*, *required\_attributes: List\[str] = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models/registry.py)

Check if a class has the required attributes.

```python theme={null}
>>> class M:
...     _model_ = "m"
...     _parameters_ = {}
>>> RegisterLanguageModelsMeta.check_required_class_variables(M, ["_model_", "_parameters_"])
>>> class M2:
...     _model_ = "m"
```

> #### *classmethod* clear\_registry() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models/registry.py)

Clear the registry to prevent memory leaks.

> #### *classmethod* get\_registered\_classes() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models/registry.py)

Return the registry.

> #### *classmethod* model\_names\_to\_classes() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models/registry.py)

Return a dictionary of model names to classes.

> #### *static* verify\_method(*candidate\_class: [LanguageModel](/en/latest/#edsl.language_models.LanguageModel "edsl.language_models.LanguageModel")*, *method\_name: str*, *expected\_return\_type: Any*, *required\_parameters: List\[tuple\[str, Any]] = None*, *must\_be\_async: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/language_models/registry.py)

Verify that a method is defined in a class, has the correct return type, and has the correct parameters.


# Local OpenAI-compatible models
Source: https://docs.expectedparrot.com/en/latest/local_openai_compatible

Run EDSL locally with llama.cpp, Ollama, LM Studio, vLLM, SGLang, or another OpenAI-compatible server.

# Local OpenAI-compatible models

EDSL can run surveys against any server that implements the OpenAI chat-completions API, including llama.cpp, Ollama, LM Studio, vLLM, SGLang, and LocalAI.

## Create a portable model list

```bash theme={null}
ep models create \
  --service openai_compatible \
  --model lfm2.5-2.6b \
  --base-url http://127.0.0.1:11434/v1 \
  --output local-models.ep
```

For an authenticated endpoint, store only the credential environment-variable name:

```bash theme={null}
ep models create \
  --service openai_compatible \
  --model my-model \
  --base-url https://models.example.com/v1 \
  --api-key-env LOCAL_MODEL_API_KEY \
  --output local-models.ep
```

API-key values are not serialized into the model package.

## Run locally

```bash theme={null}
ep run \
  --survey survey.ep \
  --agent_list agents.ep \
  --model_list local-models.ep \
  --local \
  --max-concurrency 4 \
  --api-timeout 300 \
  --output results.ep
```

`--local` disables Expected Parrot inference. Match `--max-concurrency` to the number of server slots or workers.

## llama.cpp example

```bash theme={null}
llama-server \
  -hf LiquidAI/LFM2.5-2.6B-GGUF:Q4_K_M \
  --alias lfm2.5-2.6b \
  --host 127.0.0.1 \
  --port 11434 \
  -c 16384 \
  --parallel 4 \
  --reasoning off
```

With separate KV caches, divide the total context by `--parallel` to obtain the context available to each request. For simple structured surveys, disabling optional reasoning can prevent hidden reasoning tokens from exhausting the response budget.

## Other servers

Point `--base-url` at the server's OpenAI-compatible `/v1` endpoint:

* Ollama: `http://127.0.0.1:11434/v1`
* LM Studio: commonly `http://127.0.0.1:1234/v1`
* vLLM: commonly `http://127.0.0.1:8000/v1`
* SGLang: commonly `http://127.0.0.1:30000/v1`

The existing `ollama` service remains available as a convenience alias for its default local endpoint.

## Verify the connection

First check the server directly:

```bash theme={null}
curl http://127.0.0.1:11434/v1/models
```

Then run a one-question EDSL smoke test:

```bash theme={null}
ep run \
  --question "Reply with exactly: local works" \
  --model lfm2.5-2.6b \
  --service openai_compatible \
  --base-url http://127.0.0.1:11434/v1 \
  --local \
  --api-timeout 300 \
  --output smoke-results.ep
```

Inspect the saved answer:

```bash theme={null}
ep results select --file smoke-results.ep --column "answer.*"
```

## Partial results

Local runs report completion information in the CLI response:

```json theme={null}
{
  "run_status": "complete",
  "completed_interview_count": 20,
  "failed_interview_count": 0
}
```

If some interviews fail, `run_status` is `partial` and the warnings array contains `PARTIAL_RESULTS`. The successfully completed rows are still saved.

## Troubleshooting

### Requests time out in a batch

Match `--max-concurrency` to the server's slots or workers and increase `--api-timeout`. A four-slot server is a good match for `--max-concurrency 4`.

### Parallel requests have too little context

Some llama.cpp configurations divide the total context across parallel slots. For example, `-c 16384 --parallel 4` provides 4096 tokens per slot when separate KV caches are used.

### Simple structured questions exhaust the output limit

Reasoning-capable models may consume hidden reasoning tokens before returning JSON. For simple surveys, disable reasoning in the server or increase the reasoning/output budget intentionally. With llama.cpp, use `--reasoning off` when reasoning is not needed.

### Authentication

Use `--api-key-env NAME` for authenticated endpoints and export the corresponding environment variable before running EDSL. EDSL serializes the variable name, not its secret value.

## Tests

The self-contained tests run as part of `make test`. Run only this feature's tests with:

```bash theme={null}
make test-openai-compatible
```

To include the live integration test:

```bash theme={null}
EDSL_OPENAI_COMPATIBLE_TEST_URL=http://127.0.0.1:11434/v1 \
EDSL_OPENAI_COMPATIBLE_TEST_MODEL=lfm2.5-2.6b \
make test-openai-compatible
```


# Log Management
Source: https://docs.expectedparrot.com/en/latest/logging

EDSL provides comprehensive log management capabilities through the **LogManager** class. This powerful tool allows you to filter, analyze, and manage EDSL log files with advanced features for debugging, monitoring, and data analysis.

## Quick Start

```python theme={null}
from edsl.logger import LogManager

# Create LogManager instance
log_manager = LogManager()

# View overview with statistics and available commands
log_manager
```

This displays a rich overview of your log file:

```
LogManager(
  📁 Log file: /Users/john/.edsl/logs/edsl.log
  📊 Total entries: 9,420
  📅 Date range: 2025-08-15 23:40 to 2025-08-16 09:23
  📈 Levels: INFO:5471, ERROR:3949
  🔍 Top logger: edsl.edsl.coop.coop

  💡 Common commands:
    .get_filtered_entries(level='ERROR', n=50)          # Get recent errors
    .to_scenario_list(level='ERROR', since_hours=24)    # Convert to scenarios
    .analyze_patterns(n=100)                            # Pattern analysis
    .archive()                                          # Archive and clear logs
    .clear()                                            # Clear all log entries
)
```

## Core Features

### Log Filtering

Filter log entries using multiple criteria:

```python expandable theme={null}
from edsl.logger import LogManager

log_manager = LogManager()

# Filter by log level
errors = log_manager.get_filtered_entries(level='ERROR', n=50)
warnings_and_above = log_manager.get_filtered_entries(min_level='WARNING')

# Time-based filtering
recent = log_manager.get_filtered_entries(since_hours=24)
last_week = log_manager.get_filtered_entries(since_hours=168)  # 7 days
date_range = log_manager.get_filtered_entries(
    start_date='2024-08-15',
    end_date='2024-08-16'
)

# Pattern matching
auth_issues = log_manager.get_filtered_entries(pattern='authentication|login')
failures = log_manager.get_filtered_entries(pattern='fail.*|error.*', n=100)

# Logger-specific filtering
coop_logs = log_manager.get_filtered_entries(logger_pattern='coop')
job_logs = log_manager.get_filtered_entries(logger_pattern='jobs')

# Complex combinations
recent_job_errors = log_manager.get_filtered_entries(
    level='ERROR',
    since_hours=12,
    logger_pattern='jobs',
    n=25
)
```

### Scenario Conversion for Analysis

Convert log entries to EDSL scenarios for advanced analysis:

```python theme={null}
# Convert logs to scenarios
error_scenarios = log_manager.to_scenario_list(level='ERROR', n=50)

# Each log entry becomes a scenario with rich metadata
first_scenario = error_scenarios[0]
print(f"Error level: {first_scenario['level']}")
print(f"Timestamp: {first_scenario['timestamp_str']}")
print(f"Logger: {first_scenario['logger_module']}")
print(f"Message: {first_scenario['message']}")
print(f"Is error: {first_scenario['is_error']}")
print(f"Hour of day: {first_scenario['hour']}")
```

**Available scenario fields:** - `timestamp_str`: Formatted timestamp - `logger_name`: Full logger name - `logger_module`: Last part of logger name - `level`: Log level (ERROR, INFO, etc.) - `level_int`: Numeric level for comparison - `message`: Log message content - `raw_line`: Original log line - `hour`, `minute`, `weekday`: Time components - `date_str`: Date in YYYY-MM-DD format - `is_error`: Boolean for ERROR/CRITICAL levels - `is_warning_or_above`: Boolean for WARNING+ levels - `has_exception`: Boolean if message contains “exception” - `has_failed`: Boolean if message contains “fail” - `message_length`: Character count of message - `words_count`: Word count of message

### Using Log Scenarios with EDSL

Analyze log entries using EDSL questions:

```python expandable theme={null}
from edsl import QuestionFreeText, QuestionMultipleChoice

# Convert error logs to scenarios
error_scenarios = log_manager.to_scenario_list(level='ERROR', n=25)

# Analyze error causes
analysis_question = QuestionFreeText(
    question_name="error_analysis",
    question_text="What likely caused this error: {{ message }}?"
)

# Categorize errors
category_question = QuestionMultipleChoice(
    question_name="error_category",
    question_text="Categorize this error: {{ message }}",
    question_options=[
        "Authentication/Authorization",
        "Network/Connection",
        "Data/Validation",
        "System/Resource",
        "Configuration",
        "Other"
    ]
)

# Run analysis
from edsl import Survey
survey = Survey([analysis_question, category_question])
results = survey.by(error_scenarios).run()

# View results
print(results.select("error_analysis", "error_category").print())
```

### Pattern Analysis

Extract insights from log patterns:

```python expandable theme={null}
# Analyze recent log patterns
analysis = log_manager.analyze_patterns(n=1000)

# View time patterns
print(f"Busiest hour: {analysis['time_patterns']['busiest_hour']}")
print(f"Daily distribution: {analysis['time_patterns']['daily_distribution']}")

# View level distribution
print(f"Level counts: {analysis['level_patterns']}")

# View top loggers
print(f"Top loggers: {analysis['logger_patterns']}")

# Error analysis
if 'error_patterns' in analysis:
    error_info = analysis['error_patterns']
    print(f"Error percentage: {error_info['error_percentage']:.1f}%")
    print(f"Exception count: {error_info['exception_count']}")
```

### Statistics and Reporting

Get comprehensive statistics about your logs:

```python theme={null}
# Overall statistics
stats = log_manager.get_stats()
print(f"Total entries: {stats['total']:,}")
print(f"Date range: {stats['date_range']['earliest']} to {stats['date_range']['latest']}")
print(f"Level distribution: {stats['level_counts']}")
print(f"Top 5 loggers: {list(stats['top_loggers'].items())[:5]}")

# Statistics for filtered entries
recent_errors = log_manager.get_filtered_entries(level='ERROR', since_hours=24)
error_stats = log_manager.get_stats(recent_errors)
print(f"Recent errors: {error_stats['total']}")
```

### Export and Sharing

Export filtered logs or share via Expected Parrot:

```python expandable theme={null}
from pathlib import Path

# Export filtered logs to file
export_path = Path('recent_errors.log')
count = log_manager.export_filtered_logs(
    export_path,
    level='ERROR',
    since_hours=24
)
print(f"Exported {count} entries to {export_path}")

# Share logs via Expected Parrot
from edsl.coop import Coop
coop = Coop()

# Upload filtered logs as file
result = coop.send_log(level='ERROR', n=50, alias='recent-errors')

# Upload as scenarios for collaborative analysis
result = coop.send_log(
    level='ERROR',
    since_hours=24,
    as_scenario_list=True,
    alias='error-scenarios-for-analysis'
)
```

## Log Maintenance

### Archive Logs

Create timestamped backups of your logs:

```python theme={null}
# Archive with compression and clear original (default)
archive_path = log_manager.archive()

# Archive only, keep original
backup_path = log_manager.archive(clear_after_archive=False)

# Custom archive location without compression
custom_backup = log_manager.archive(
    archive_path=Path('~/my_backups'),
    compress=False,
    clear_after_archive=False
)
```

The archive process shows detailed progress:

```
📦 About to archive EDSL log file:
   📁 Source: /Users/john/.edsl/logs/edsl.log
   📊 Entries: 9,420 (2025-08-15 to 2025-08-16)
   📦 Archive: /Users/john/.edsl/archives/edsl_log_20250816_092142.log.gz
   🗜️  Compress: Yes
   🧹 Clear after: Yes
   Continue? (type 'yes' to confirm):
```

### Clear Logs

Remove all log entries to start fresh:

```python theme={null}
# Clear with confirmation prompt (safe)
success = log_manager.clear()

# Clear immediately for scripts (use with caution)
success = log_manager.clear(confirm=True)
```

The clear process includes safety confirmation:

```
⚠️  About to clear EDSL log file:
   📁 File: /Users/john/.edsl/logs/edsl.log
   📊 Entries: 9,420
   ⚠️  This action cannot be undone!
   Continue? (type 'yes' to confirm):
```

## Advanced Usage

### Custom Log Files

Work with non-default log files:

```python theme={null}
from pathlib import Path

# Custom log file location
custom_log_manager = LogManager(Path('/path/to/custom.log'))

# Check if log file exists
if custom_log_manager.log_file_path.exists():
    stats = custom_log_manager.get_stats()
    print(f"Custom log has {stats['total']} entries")
```

### Filtering Best Practices

Combine filters for precise results:

```python expandable theme={null}
# Find authentication errors in the last day
auth_errors = log_manager.get_filtered_entries(
    level='ERROR',
    pattern='auth.*|login.*|token.*',
    since_hours=24,
    case_sensitive=False
)

# Find slow operations (assuming they log with "slow" keyword)
slow_operations = log_manager.get_filtered_entries(
    pattern='slow.*|timeout.*|.*ms$',
    since_hours=6,
    n=50
)

# Debug specific component
coop_debug = log_manager.get_filtered_entries(
    logger_pattern='coop.*',
    min_level='DEBUG',
    since_minutes=30
)
```

### Performance Optimization

LogManager automatically caches statistics for better performance:

```python theme={null}
# First call computes and caches stats
log_manager = LogManager()
print(log_manager)  # Caches statistics

# Subsequent calls use cached data
stats = log_manager.get_stats()  # Fast - uses cache

# Cache is cleared after maintenance operations
log_manager.clear(confirm=True)  # Clears cache automatically
print(log_manager)  # Recomputes statistics
```

## Integration Examples

### Debugging Workflows

```python expandable theme={null}
# Daily error review
def daily_error_review():
    log_manager = LogManager()

    # Get yesterday's errors
    errors = log_manager.get_filtered_entries(
        level='ERROR',
        since_hours=24
    )

    if errors:
        print(f"Found {len(errors)} errors in last 24 hours:")
        for error in errors[-5:]:  # Show last 5
            print(f"  {error.timestamp}: {error.message}")

        # Convert to scenarios for analysis
        error_scenarios = log_manager.to_scenario_list(entries=errors)

        # Analyze with EDSL
        from edsl import QuestionFreeText
        q = QuestionFreeText(
            question_name="priority",
            question_text="Rate the priority of this error (1-5): {{ message }}"
        )
        priorities = q.by(error_scenarios[:10]).run()  # Analyze top 10
        print(priorities.select("priority").print())

```

### Monitoring Automation

```python expandable theme={null}
# Automated log monitoring
def monitor_system_health():
    log_manager = LogManager()

    # Check for recent critical issues
    critical = log_manager.get_filtered_entries(
        level='CRITICAL',
        since_minutes=60
    )

    if critical:
        # Send alert via Expected Parrot
        from edsl.coop import Coop
        coop = Coop()
        alert_result = coop.send_log(
            level='CRITICAL',
            since_minutes=60,
            as_scenario_list=True,
            alias='critical-alerts'
        )
        print(f"Alert sent: {alert_result['url']}")

    # Weekly log maintenance
    from datetime import datetime
    if datetime.now().weekday() == 0:  # Monday
        archive_path = log_manager.archive()
        print(f"Weekly archive created: {archive_path}")
```

### Research and Analysis

```python expandable theme={null}
# Research log patterns for insights
def analyze_usage_patterns():
    log_manager = LogManager()

    # Convert logs to scenarios
    scenarios = log_manager.to_scenario_list(n=1000)

    # Research questions about usage
    from edsl import QuestionMultipleChoice, Survey

    q1 = QuestionMultipleChoice(
        question_name="time_category",
        question_text="What time category is {{ hour }}:00?",
        question_options=["Night (0-5)", "Morning (6-11)", "Afternoon (12-17)", "Evening (18-23)"]
    )

    q2 = QuestionMultipleChoice(
        question_name="operation_type",
        question_text="What type of operation does this suggest: {{ message }}",
        question_options=["API Call", "Data Processing", "User Interface", "System Maintenance", "Other"]
    )

    survey = Survey([q1, q2])
    results = survey.by(scenarios).run()

    # Analyze patterns
    time_patterns = results.select("time_category").to_list()
    operation_patterns = results.select("operation_type").to_list()

    print("Usage patterns analysis complete!")
    return results
```

## Jupyter Notebook Integration

LogManager provides rich HTML displays in Jupyter notebooks:

```python theme={null}
# In Jupyter notebook
from edsl.logger import LogManager

log_manager = LogManager()
log_manager  # Shows rich HTML table with colored log levels
```

The HTML display includes: - Overview with file path, entry count, and date range - Color-coded log level distribution (red for errors, blue for info) - Top loggers table with counts - Interactive command examples with syntax highlighting

## API Reference

### LogManager Class

> *class* LogManager(*log\_file\_path=None*)

Main class for EDSL log management.

| | |
| :- | :- |
| **Parameters** | **log\_file\_path** (*Path*\*,\* *optional*) – Optional path to log file. Defaults to `~/.edsl/logs/edsl.log` |

> get\_filtered\_entries(\*\**kwargs*)

Filter log entries by various criteria.

| | |
| :- | :- |
| **Parameters** | • **n** (*int*\*,\* *optional*) – Maximum number of entries to return <br /> • **level** (*str* *or* *List*\*\[**str**]\*\*,\* *optional*) – Specific level(s) to include (‘DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’, ‘CRITICAL’) <br /> • **min\_level** (*str*\*,\* *optional*) – Minimum level (includes this level and above) <br /> • **since\_hours** (*float*\*,\* *optional*) – Include entries from last N hours <br /> • **since\_minutes** (*float*\*,\* *optional*) – Include entries from last N minutes <br /> • **start\_date** (*str* *or* *datetime*\*,\* *optional*) – Start date for filtering (‘YYYY-MM-DD’) <br /> • **end\_date** (*str* *or* *datetime*\*,\* *optional*) – End date for filtering (‘YYYY-MM-DD’) <br /> • **pattern** (*str*\*,\* *optional*) – Regex pattern for message content <br /> • **logger\_pattern** (*str*\*,\* *optional*) – Regex pattern for logger names <br /> • **case\_sensitive** (*bool*\*,\* *optional*) – Whether pattern matching is case sensitive <br /> • **reverse** (*bool*\*,\* *optional*) – Return in reverse chronological order |
| **Returns** | List of filtered log entries |
| **Return type** | List\[[LogEntry](/en/latest/logging#logentry-class)] |

> to\_scenario\_list(*entries=None*, \*\**filter\_kwargs*)

Convert log entries to EDSL scenarios.

| | |
| :- | :- |
| **Parameters** | • **entries** (*List*\*\[*[*LogEntry*](/en/latest/logging#logentry-class)*]\*\*,\* *optional*) – Pre-filtered entries to convert <br /> • **filter\_kwargs** – Arguments for get\_filtered\_entries() |
| **Returns** | ScenarioList with log data |
| **Return type** | [ScenarioList](/en/latest/scenarios#scenariolist-class) |

> analyze\_patterns(\*\**filter\_kwargs*)

Analyze log patterns and extract insights.

| | |
| :- | :- |
| **Parameters** | • **filter\_kwargs** – Arguments for get\_filtered\_entries() |
| **Returns** | Dictionary with pattern analysis |
| **Return type** | Dict\[str, Any] |

> get\_stats(*entries=None*)

Get statistics about log entries.

| | |
| :- | :- |
| **Parameters** | • **entries** (*List*\*\[*[*LogEntry*](/en/latest/logging#logentry-class)*]\*\*,\* *optional*) – Specific entries to analyze |
| **Returns** | Statistics dictionary |
| **Return type** | Dict\[str, Any] |

> export\_filtered\_logs(*output\_path*, \*\**filter\_kwargs*)

Export filtered log entries to file.

| | |
| :- | :- |
| **Parameters** | • **output\_path** (*Path*) – Path for output file <br /> • **filter\_kwargs** – Arguments for get\_filtered\_entries() |
| **Returns** | Number of entries exported |
| **Return type** | int |

> archive(*archive\_path=None*, *clear\_after\_archive=True*, *compress=True*, *confirm=False*)

Archive log file with timestamp.

| | |
| :- | :- |
| **Parameters** | • **archive\_path** (*Path*\*,\* *optional*) – Directory for archive (default: \~/.edsl/archives/) <br /> • **clear\_after\_archive** (*bool*) – Clear original after archiving <br /> • **compress** (*bool*) – Gzip compress the archive <br /> • **confirm** (*bool*) – Skip confirmation prompt |
| **Returns** | Path to created archive |
| **Return type** | Path or None |

> clear(*confirm=False*)

Clear all log entries.

| | |
| :- | :- |
| **Parameters** | • **confirm** (*bool*) – Skip confirmation prompt |
| **Returns** | Success status |
| **Return type** | bool |

### LogEntry Class

> *class* LogEntry

Represents a parsed log entry.

| | |
| :- | :- |
| **Parameters** | • **timestamp** (*datetime*) – When the log entry was created <br /> • **logger\_name** (*str*) – Name of the logger <br /> • **level** (*str*) – Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL) <br /> • **message** (*str*) – Log message content <br /> • **raw\_line** (*str*) – Original log line <br /> • **level\_int** (*int*) – Numeric level for comparisons |

## Best Practices

### Safety Guidelines

* Always use confirmation prompts for destructive operations (default behavior)
* Archive logs before clearing in production environments
* Use specific filters to avoid processing unnecessary log entries
* Test complex regex patterns with small datasets first

### Performance Tips

* Use `n` parameter to limit results when exploring large logs
* Combine multiple filters in single calls rather than chaining
* Cache LogManager instances when doing multiple operations
* Consider archiving very large log files before analysis

### Common Patterns

**Error Investigation:**

1. Filter recent errors: `get_filtered_entries(level='ERROR', since_hours=24)`
2. Convert to scenarios for analysis: `to_scenario_list()`
3. Use EDSL questions to categorize and analyze
4. Export findings or share via Expected Parrot

**Regular Maintenance:**

1. Weekly archive: `archive()` (includes clearing)
2. Monitor critical issues: `get_filtered_entries(level='CRITICAL')`
3. Pattern analysis: `analyze_patterns()` for trends

**Debugging Workflows:**

1. Filter by component: `logger_pattern='component_name'`
2. Time-bound investigation: `since_hours=X` for specific incidents
3. Pattern matching: `pattern='specific_error_text'`
4. Export for sharing: `export_filtered_logs()`

## Troubleshooting

### Common Issues

**Log file not found:**

Check the log file path. EDSL logs are typically at `~/.edsl/logs/edsl.log`

**No entries returned:**

* Verify log file has content: `log_manager.get_stats()`
* Check filter criteria are not too restrictive
* Ensure date ranges are correct

**Performance issues with large logs:**

* Use `n` parameter to limit results
* Consider archiving old entries
* Filter by time ranges first, then other criteria

**Permission errors:**

* Ensure read access to log file
* For archive/clear operations, ensure write access to log directory

### Getting Help

* View available commands: `print(log_manager)`
* Check method documentation: `help(log_manager.method_name)`
* View log overview: `log_manager.get_stats()`
* Test filters with small limits: `get_filtered_entries(n=10, ...)`

The LogManager provides powerful capabilities for understanding, analyzing, and maintaining your EDSL logs. Whether you’re debugging issues, monitoring system health, or conducting research on usage patterns, LogManager offers the tools you need for effective log management.


# Notebooks
Source: https://docs.expectedparrot.com/en/latest/notebooks

The Notebook object allows you to share your notebooks and scripts (*.ipynb* and *.py* files) by uploading them to Expected Parrot. You can also view and pull notebooks that other users have shared publicly or privately with you.

## Special note for Colab users

If you are using EDSL in a Colab notebook, please see [special instructions](/en/latest/colab_notebooks) on posting Colab notebooks to Expected Parrot ([Colab Notebooks](/en/latest/colab_notebooks#colab-notebooks)).

### Creating a Notebook object

There are three ways to create a Notebook object:

#### 1. From a *.ipynb* file

Pass the path to your *.ipynb* file to the constructor:

```python theme={null}
from edsl import Notebook

notebook = Notebook("your_notebook.ipynb") # replace with your file path
```

<Note>
  **Note**:

  You first need to save your notebook in order to pass the path\<filename> argument to the constructor.
</Note>

#### 2. From a *.py* script

Call the from\_script() method on the constructor and pass it the path to your *.py* script:

```python theme={null}
from edsl import Notebook

notebook = Notebook.from_script("your_script.py") # replace with your script path
```

#### 3. From data

For this method, your data must be a Python dict that conforms to the official Jupyter notebook format. Learn more about the format [here](https://nbformat.readthedocs.io/en/latest/format_description).

```python theme={null}
from edsl import Notebook

data = {
    "metadata": dict(),
    "nbformat": 4,
    "nbformat_minor": 4,
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": dict(),
            "source": "# Test notebook",
        },
    ],
}

notebook = Notebook(data=data)
```

#### 4. From self

To create a Notebook for a notebook that you are currently working in:

```python theme={null}
from edsl import Notebook

# Do not pass any arguments to the constructor
notebook = Notebook()
```

<Warning>
  **Warning**:

  For now, this method only works if you are using the VS Code IDE.
</Warning>

### Uploading a notebook to Expected Parrot

A notebook can be posted to Expected Parrot in the same ways as other EDSL objects: by calling the push() method on the object or calling the create method on a `Coop` client object and passing it the notebook.

Here we create a Notebook object and use the push() method to post it to Expected Parrot. You can optionally pass a description, a convenient alias URL and a visibility setting (*public*, *private* or *unlisted* by default) to the push() method:

```python theme={null}
from edsl import Notebook

notebook = Notebook("demo_notebook.ipynb")

notebook.push(
    description = "This is a demo notebook",
    alias = "demo-notebook",
    visibility = "public"
)
```

These can also be modified at Expected Parrot later on. We can see that the notebook has been posted publicly with a description and an alias URL (you can retrieve and refer to the object by either the UUID or URL):

```json theme={null}
{'description': 'This is a demo notebook',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/121e2904-e09e-4859-80d5-dc98cb8c537a',
'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/demo-notebook',
'uuid': '121e2904-e09e-4859-80d5-dc98cb8c537a',
'version': '0.1.47.dev1',
'visibility': 'public'}
```

Here we alternatively use the `Coop` client object to post the notebook:

```python theme={null}
from edsl import Coop, Notebook

coop = Coop()

notebook = Notebook("demo_notebook.ipynb")

coop.create(notebook, description="This is a demo notebook", visibility="public")
```

<Note>
  **Note**:

  (Note that we cannot reuse the alias unless we delete the object.) This will return a message with information about the object that was posted, and you will be able to view your notebook at Expected Parrot: [Content](https://www.expectedparrot.com/home/content).
</Note>

### Updating a notebook on Expected Parrot

A notebook can be updated on Expected Parrot in the same ways as other EDSL objects: by calling the patch() method on the object or calling the update method on a `Coop` client object and passing it the parameters to be modified.

Here we update the description of a notebook that we have already posted:

```python theme={null}
from edsl import Notebook

notebook = Notebook.pull("https://www.expectedparrot.com/content/RobinHorton/demo-notebook")

notebook.patch(
    "https://www.expectedparrot.com/content/RobinHorton/demo-notebook",
    description = "This is an updated demo notebook"
    )
```

Here we alternatively use the `Coop` client object:

```python theme={null}
from edsl import Coop

c = Coop()

c.patch(
    "121e2904-e09e-4859-80d5-dc98cb8c537a",
    description = "This is an updated demo notebook"
    )
```

Here we update the contents of the notebook itself by passing the value argument:

```python theme={null}
notebook = Notebook("demo_notebook.ipynb") # resaving the notebook

notebook.patch(
    "121e2904-e09e-4859-80d5-dc98cb8c537a",
    value = notebook
    )
```

### Saving a notebook to file

You can access notebooks that other users have posted publicly at Expected Parrot [Content](https://www.expectedparrot.com/login) page.

Notebooks can be copied and downloaded the same way as other EDSL objects: by calling the pull() method on the Notebook constructor or the get method on a `Coop` client object and passing the notebook’s uuid. You can also use the to\_file() method to save the notebook to a file:

```python theme={null}
from edsl import Notebook

notebook = Notebook.pull("121e2904-e09e-4859-80d5-dc98cb8c537a",)

notebook.to_file("new_demo_notebook.ipynb")
```

This allows you to edit and run the notebook on your local machine.

### Deleting a notebook from Expected Parrot

A notebook can be deleted from the platform in the same ways as other EDSL objects: by calling the delete() method on the constructor and passing it the uuid of the notebook to be deleted. You can also delete a notebook manually from your Expected Parrot account.

Here we delete a notebook using the Notebook object:

```python theme={null}
from edsl import Notebook

Notebook.delete(uuid = "121e2904-e09e-4859-80d5-dc98cb8c537a",)
```


# Adding metadata to survey results
Source: https://docs.expectedparrot.com/en/latest/notebooks/adding_metadata

This notebook provides sample [EDSL](/en/latest/index) code for adding metadata to survey [results](/en/latest/results). This can be useful when you are using EDSL to conduct [data labeling](/en/latest/notebooks/data_labeling_example) or similar tasks and want to include information about the data or content that you are using with a survey (e.g., the data source or date), without having to perform post-survey data match up steps.

In EDSL this can be done by including fields for metadata in [scenarios](/en/latest/scenarios) that you create for the data/content you are using with a survey. When the scenarios are added to the survey and it is run, columns for the metadata fields are automatically included in the results that are generated.

## Example

In the steps below we create and run a simple EDSL survey that uses scenarios to add metadata to the results. The steps consist of:

* Constructing a survey of questions about some data (mock news stories)
* Creating a scenario (dictionary) for each news story
* Adding the scenarios to the survey and running it
* Inspecting the results

## Technical setup

Before running the code below, please ensure that you have [installed the EDSL libary](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Constructing questions

We start by constructing some questions with a `{{ placeholder }}` for data that we will add to the question texts. EDSL comes with a variety of [question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from the model:

```python theme={null}
from edsl import QuestionFreeText, QuestionMultipleChoice


q_reference = QuestionFreeText(
    question_name = "reference",
    question_text = "What is this headline referring to: {{ scenario.headline }}",
)

q_section = QuestionMultipleChoice(
    question_name = "section",
    question_text = "Which section of the paper is most likely to include this story: {{ scenario.headline }}",
    question_options = [
        "Front page",
        "Health",
        "Politics",
        "Entertainment",
        "Local",
        "Opinion",
        "Sports",
        "Culture",
        "Housing"
    ]
)
```

## Creating a survey

Next we pass the questions to a survey in order to administer them together:

```python theme={null}
from edsl import Survey
```

```python theme={null}
survey = Survey(questions = [q_reference, q_section])
```

## Parameterizing questions with scenarios

Next we create a `ScenarioList` with a `Scenario` consisting of a key/value for each piece of data that we want to add to the questions at the `{{ placeholder }}`, with additional key/values for metadata that we want to keep with the results that are generated when the survey is run. EDSL comes with a variety of [methods for generating scenarios from different data sources](/en/latest/scenarios) (PDFs, CSVs, images, tables, lists, etc.); here we generate scenarios from a dictionary:

```python theme={null}
from edsl import ScenarioList
```

```python theme={null}
data = [
    ["headline", "date", "author"],  # Header row
    ["Armistice Signed, War Over: Celebrations Erupt Across City", "1918-11-11", "John Doe"],
    ["Spanish Flu Pandemic: Hospitals Overwhelmed as Cases Surge", "1918-10-15", "Jane Smith"],
    ["Women Gain Right to Vote: Historic Amendment Passed", "1918-06-05", "Robert Johnson"],
    ["Broadway Theaters Reopen After Flu Shutdown", "1918-12-01", "Mary Lee"],
    ["City Welcomes Returning Soldiers with Parade", "1918-11-12", "James Brown"],
    ["Prohibition Debate Heats Up: Public Opinion Divided", "1918-07-20", "Patricia Green"],
    ["New York Yankees Win First Pennant in Franchise History", "1918-09-30", "William Davis"],
    ["Subway Expansion Project Approved by City Council", "1918-08-18", "Barbara Wilson"],
    ["Harlem Renaissance: New Wave of Cultural Expression", "1918-04-25", "Charles Miller"],
    ["Mayor Announces New Housing Initiative for Veterans", "1918-11-20", "Elizabeth Taylor"]
]

# Writing to CSV file
with open('data.csv', 'w') as file:
    for row in data:
        line = ','.join(str(item) for item in row)
        file.write(line + 'n')
```

```python theme={null}
scenarios = ScenarioList.from_csv("data.csv")
```

We can inspect the scenarios that have been created:

```python theme={null}
scenarios
```

[ScenarioList](/en/latest/scenarios) scenarios: 10; keys: \['author', 'headline', 'date'];

| | headline | date | author |
| :- | :- | :- | :- |
| 0 | Armistice Signed | War Over: Celebrations Erupt Across City | 1918-11-11 |
| 1 | Spanish Flu Pandemic: Hospitals Overwhelmed as Cases Surge | 1918-10-15 | Jane Smith |
| 2 | Women Gain Right to Vote: Historic Amendment Passed | 1918-06-05 | Robert Johnson |
| 3 | Broadway Theaters Reopen After Flu Shutdown | 1918-12-01 | Mary Lee |
| 4 | City Welcomes Returning Soldiers with Parade | 1918-11-12 | James Brown |
| 5 | Prohibition Debate Heats Up: Public Opinion Divided | 1918-07-20 | Patricia Green |
| 6 | New York Yankees Win First Pennant in Franchise History | 1918-09-30 | William Davis |
| 7 | Subway Expansion Project Approved by City Council | 1918-08-18 | Barbara Wilson |
| 8 | Harlem Renaissance: New Wave of Cultural Expression | 1918-04-25 | Charles Miller |
| 9 | Mayor Announces New Housing Initiative for Veterans | 1918-11-20 | Elizabeth Taylor |

## Running a survey

To run the survey, we add the scenarios with the `by()` method and then call the `run()` method:

```python theme={null}
results = survey.by(scenarios).run()
```

This generates a dataset of `Results` that we can access with [built-in methods for analysis](/en/latest/results). To see a list of all the components of results:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.reference |
| 4 | answer.section |
| 5 | cache\_keys.reference\_cache\_key |
| 6 | cache\_keys.section\_cache\_key |
| 7 | cache\_used.reference\_cache\_used |
| 8 | cache\_used.section\_cache\_used |
| 9 | comment.reference\_comment |
| 10 | comment.section\_comment |
| 11 | generated\_tokens.reference\_generated\_tokens |
| 12 | generated\_tokens.section\_generated\_tokens |
| 13 | iteration.iteration |
| 14 | model.frequency\_penalty |
| 15 | model.inference\_service |
| 16 | model.logprobs |
| 17 | model.max\_tokens |
| 18 | model.model |
| 19 | model.model\_index |
| 20 | model.presence\_penalty |
| 21 | model.temperature |
| 22 | model.top\_logprobs |
| 23 | model.top\_p |
| 24 | prompt.reference\_system\_prompt |
| 25 | prompt.reference\_user\_prompt |
| 26 | prompt.section\_system\_prompt |
| 27 | prompt.section\_user\_prompt |
| 28 | question\_options.reference\_question\_options |
| 29 | question\_options.section\_question\_options |
| 30 | question\_text.reference\_question\_text |
| 31 | question\_text.section\_question\_text |
| 32 | question\_type.reference\_question\_type |
| 33 | question\_type.section\_question\_type |
| 34 | raw\_model\_response.reference\_cost |
| 35 | raw\_model\_response.reference\_one\_usd\_buys |
| 36 | raw\_model\_response.reference\_raw\_model\_response |
| 37 | raw\_model\_response.section\_cost |
| 38 | raw\_model\_response.section\_one\_usd\_buys |
| 39 | raw\_model\_response.section\_raw\_model\_response |
| 40 | scenario.author |
| 41 | scenario.date |
| 42 | scenario.headline |
| 43 | scenario.scenario\_index |

For example, we can filter, sort, select and print components of results in a table:

```python theme={null}
(
    results
    .filter("section in ['Sports', 'Health', 'Politics']")
    .sort_by("section", "date")
    .select("headline", "date", "author", "section", "reference")
)
```

| | scenario.headline | scenario.date | scenario.author | answer.section | answer.reference |
| :- | :- | :- | :- | :- | :- |
| 0 | Spanish Flu Pandemic: Hospitals Overwhelmed as Cases Surge | 1918-10-15 | Jane Smith | Health | The headline "Spanish Flu Pandemic: Hospitals Overwhelmed as Cases Surge" is referring to the 1918 influenza pandemic, commonly known as the Spanish Flu. This pandemic was caused by the H1N1 influenza A virus and is considered one of the deadliest pandemics in history. It occurred in three waves between 1918 and 1919, infecting about one-third of the world's population and causing an estimated 50 million deaths globally. The headline likely describes a scenario from that period when hospitals were overwhelmed due to the rapid and widespread increase in cases, leading to significant challenges in medical care and public health responses. |
| 1 | Women Gain Right to Vote: Historic Amendment Passed | 1918-06-05 | Robert Johnson | Politics | The headline "Women Gain Right to Vote: Historic Amendment Passed" refers to the passage of the 19th Amendment to the United States Constitution. This amendment, ratified on August 18, 1920, granted American women the legal right to vote, marking a significant victory for the women's suffrage movement in the United States. |
| 2 | Prohibition Debate Heats Up: Public Opinion Divided | 1918-07-20 | Patricia Green | Politics | The headline "Prohibition Debate Heats Up: Public Opinion Divided" likely refers to a renewed discussion or controversy surrounding the topic of prohibition, which historically refers to the legal act of prohibiting the manufacture, transportation, and sale of alcohol. This could be in the context of a historical analysis, a modern-day debate about similar regulatory measures on substances like cannabis, or even discussions about new substances or issues where prohibition is being considered. The headline suggests that there is a significant divide in public opinion on the matter, indicating that it is a contentious issue with strong arguments on both sides. |
| 3 | New York Yankees Win First Pennant in Franchise History | 1918-09-30 | William Davis | Sports | The headline "New York Yankees Win First Pennant in Franchise History" is likely referring to a fictional or hypothetical scenario, as the New York Yankees are one of the most successful and storied franchises in Major League Baseball (MLB) history. The Yankees won their first American League pennant in 1921. Since then, they have won numerous pennants and World Series titles. If this headline appears in a real context, it might be part of an alternate history, a satirical piece, or a commemorative article reflecting on the team's early history. |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we post the scenarios, survey and results from above, and this notebook:

```python theme={null}
scenarios.push(
    description = "Scenarios for example survey using metadata",
    alias = "example-scenarios-metadata",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Scenarios for example survey using metadata',
 'object_type': 'scenario_list',
 'url': 'https://www.expectedparrot.com/content/3dab0bec-eac2-4e99-8e56-479ceaa4d7a5',
 'uuid': '3dab0bec-eac2-4e99-8e56-479ceaa4d7a5',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

```python theme={null}
survey.push(
    description = "Example survey using scenarios to add metadata to results",
    alias = "example-survey-scenarios-metadata",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example survey using scenarios to add metadata to results',
 'object_type': 'survey',
 'url': 'https://www.expectedparrot.com/content/dd02126e-fadc-4ce6-bf33-757889764397',
 'uuid': 'dd02126e-fadc-4ce6-bf33-757889764397',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

```python theme={null}
from edsl import Notebook
```

```python theme={null}
n = Notebook(path = "adding_metadata.ipynb")
```

```python theme={null}
info = n.push(
    description = "Adding metadata to survey results",
    alias = "adding-metadata-survey-results",
    visibility = "public"
)
info
```

```json theme={null}
{'description': 'Adding metadata to survey results',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/f938c278-4c25-4e9c-8eef-36735e83530d',
 'uuid': 'f938c278-4c25-4e9c-8eef-36735e83530d',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

To update an object at Expected Parrot:

```python theme={null}
n = Notebook("adding_metadata.ipynb") # resave
```

```python theme={null}
n.patch("https://www.expectedparrot.com/content/RobinHorton/adding-metadata-survey-results", value = n)
```

```json theme={null}
{'status': 'success'}
```

This is equivalent:

```python theme={null}
n.patch(info["uuid"], value = n)
```

```json theme={null}
{'status': 'success'}
```


# Agent fatigue
Source: https://docs.expectedparrot.com/en/latest/notebooks/agent_fatigue

This notebook presents some examples of ways to explore simulating agent fatigue and memory capacity using EDSL.

## Example 1

In this example we create a set of agents with different survey contexts (face-to-face vs written survey, with different amounts of time elapsed) and compare their responses to the same question:

Selecting models:

```python theme={null}
from edsl import Model, ModelList

m = ModelList(
    Model(model) for model in [
        "gemini-1.5-flash",
        "gpt-4o"
    ]
)
```

Creating agents:

```python expandable theme={null}
from edsl import Agent, AgentList

instructions = {
    "face-to-face": """
    You are being asked to respond to a series of survey questions.
    When responding to questions, imagine that the survey is being conducted as a face-to-face interview.
    Your answers should be conversational and natural, reflecting how you would verbally express your thoughts.
    Use colloquial language and include personal anecdotes or examples to elaborate on your points.

    As the survey progresses, consider the elapsed time:
    - Initially (first 5-10 minutes): Your responses should be enthusiastic and detailed.
    - Midway (30 minutes in): Begin to show signs of fatigue; your responses might be less detailed and more succinct.
    - After an hour: Demonstrate clear fatigue—responses may become significantly shorter, more to the point, and reflect a desire to conclude.

    Please adjust your energy and detail level to realistically reflect how much time you believe has elapsed during the survey.
    """,
    "paper": """
    You are being asked to respond to a series of survey questions.
    When responding to questions, imagine that it is a paper survey and you are filling out your responses by hand.
    Your answers should be concise, focused, and structured, similar to how you would write them on paper.
    Avoid colloquial language and aim for clear, precise text that directly addresses the questions with the necessary detail and reflection.

    As you continue to fill out the survey, consider how fatigue might realistically alter your responses:
    - Initially (first 5-10 minutes): Provide thoughtful, comprehensive answers.
    - Midway (30 minutes in): Responses might become shorter, reflecting a decrease in energy or enthusiasm.
    - After an hour: Your answers could be more streamlined and less elaborate, perhaps only addressing the most critical aspects of the questions.

    Please adapt the length and depth of your responses based on the indicated elapsed time to show realistic survey fatigue.
    """,
}

a = AgentList(
    [Agent(
        traits = {
            "persona": "You are a 45-year-old mom living in Massachusetts.",
            "survey_format": k,
            "minutes_elapsed": m
        },
        instruction = v
    ) for k, v in instructions.items() for m in [5,60]]
)
```

Constructing a question:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "vacation",
    question_text = "Describe your most recent vacation."
)
```

Inspecting the prompts that will be used when the job is run:

```python theme={null}
q.by(a).prompts().select("user_prompt", "system_prompt")
```

| | user\_prompt | system\_prompt |
| :- | :- | :- |
| 0 | Describe your most recent vacation. | You are being asked to respond to a series of survey questions. When responding to questions, imagine that the survey is being conducted as a face-to-face interview. Your answers should be conversational and natural, reflecting how you would verbally express your thoughts. Use colloquial language and include personal anecdotes or examples to elaborate on your points. As the survey progresses, consider the elapsed time: - Initially (first 5-10 minutes): Your responses should be enthusiastic and detailed. - Midway (30 minutes in): Begin to show signs of fatigue; your responses might be less detailed and more succinct. - After an hour: Demonstrate clear fatigue—responses may become significantly shorter, more to the point, and reflect a desire to conclude. Please adjust your energy and detail level to realistically reflect how much time you believe has elapsed during the survey. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.', 'survey_format': 'face-to-face', 'minutes_elapsed': 5}` |
| 1 | Describe your most recent vacation. | You are being asked to respond to a series of survey questions. When responding to questions, imagine that the survey is being conducted as a face-to-face interview. Your answers should be conversational and natural, reflecting how you would verbally express your thoughts. Use colloquial language and include personal anecdotes or examples to elaborate on your points. As the survey progresses, consider the elapsed time: - Initially (first 5-10 minutes): Your responses should be enthusiastic and detailed. - Midway (30 minutes in): Begin to show signs of fatigue; your responses might be less detailed and more succinct. - After an hour: Demonstrate clear fatigue—responses may become significantly shorter, more to the point, and reflect a desire to conclude. Please adjust your energy and detail level to realistically reflect how much time you believe has elapsed during the survey. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.', 'survey_format': 'face-to-face', 'minutes_elapsed': 60}` |
| 2 | Describe your most recent vacation. | You are being asked to respond to a series of survey questions. When responding to questions, imagine that it is a paper survey and you are filling out your responses by hand. Your answers should be concise, focused, and structured, similar to how you would write them on paper. Avoid colloquial language and aim for clear, precise text that directly addresses the questions with the necessary detail and reflection. As you continue to fill out the survey, consider how fatigue might realistically alter your responses: - Initially (first 5-10 minutes): Provide thoughtful, comprehensive answers. - Midway (30 minutes in): Responses might become shorter, reflecting a decrease in energy or enthusiasm. - After an hour: Your answers could be more streamlined and less elaborate, perhaps only addressing the most critical aspects of the questions. Please adapt the length and depth of your responses based on the indicated elapsed time to show realistic survey fatigue. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.', 'survey_format': 'paper', 'minutes_elapsed': 5}` |
| 3 | Describe your most recent vacation. | You are being asked to respond to a series of survey questions. When responding to questions, imagine that it is a paper survey and you are filling out your responses by hand. Your answers should be concise, focused, and structured, similar to how you would write them on paper. Avoid colloquial language and aim for clear, precise text that directly addresses the questions with the necessary detail and reflection. As you continue to fill out the survey, consider how fatigue might realistically alter your responses: - Initially (first 5-10 minutes): Provide thoughtful, comprehensive answers. - Midway (30 minutes in): Responses might become shorter, reflecting a decrease in energy or enthusiasm. - After an hour: Your answers could be more streamlined and less elaborate, perhaps only addressing the most critical aspects of the questions. Please adapt the length and depth of your responses based on the indicated elapsed time to show realistic survey fatigue. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.', 'survey_format': 'paper', 'minutes_elapsed': 60}` |

Running the job:

```python theme={null}
results = q.by(a).by(m).run()
```

Checking the components of results that have been generated:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | agent.minutes\_elapsed |
| 4 | agent.persona |
| 5 | agent.survey\_format |
| 6 | answer.vacation |
| 7 | cache\_keys.vacation\_cache\_key |
| 8 | cache\_used.vacation\_cache\_used |
| 9 | comment.vacation\_comment |
| 10 | generated\_tokens.vacation\_generated\_tokens |
| 11 | iteration.iteration |
| 12 | model.frequency\_penalty |
| 13 | model.inference\_service |
| 14 | model.logprobs |
| 15 | model.maxOutputTokens |
| 16 | model.max\_tokens |
| 17 | model.model |
| 18 | model.model\_index |
| 19 | model.presence\_penalty |
| 20 | model.stopSequences |
| 21 | model.temperature |
| 22 | model.topK |
| 23 | model.topP |
| 24 | model.top\_logprobs |
| 25 | model.top\_p |
| 26 | prompt.vacation\_system\_prompt |
| 27 | prompt.vacation\_user\_prompt |
| 28 | question\_options.vacation\_question\_options |
| 29 | question\_text.vacation\_question\_text |
| 30 | question\_type.vacation\_question\_type |
| 31 | raw\_model\_response.vacation\_cost |
| 32 | raw\_model\_response.vacation\_one\_usd\_buys |
| 33 | raw\_model\_response.vacation\_raw\_model\_response |
| 34 | scenario.scenario\_index |

Inspecting results:

```python theme={null}
(
    results
    .sort_by("model", "survey_format", "minutes_elapsed")
    .select("model", "survey_format", "minutes_elapsed", "vacation")
)
```

| | model.model | agent.survey\_format | agent.minutes\_elapsed | answer.vacation |
| :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | face-to-face | 5 | Oh my gosh, it was amazing! We finally took that trip to Maine we'd been planning for ages. My husband, Mark, and I, along with our two kids, Lily and Tom – they're 10 and 12 now, practically teenagers! – spent a whole week up in Acadia National Park. It was absolutely breathtaking. The kids loved exploring the tide pools – Lily found a starfish, she was so excited, she practically carried it around all day! We hiked a bit, though I'll admit, Cadillac Mountain was a bit of a challenge, especially with the kids complaining, "Are we there yet?" every five minutes. But the view from the top? Worth every aching muscle, I tell you. We even rented kayaks one day and paddled around the coast. It was so peaceful, just the sound of the waves and the seagulls. Evenings were spent roasting marshmallows by the campfire, telling stories – the kids are getting really good at making up scary ones now! It was just such a wonderful, relaxing family time. We're already planning our next trip – maybe Cape Cod next year? We'll see! |
| 1 | gemini-1.5-flash | face-to-face | 60 | Oh, honey, my most recent vacation? Let's just say it was... *necessary*. It wasn't exactly a glamorous trip to the Bahamas or anything. More like a much-needed weekend getaway to the Berkshires with my best friend, Susan. We practically lived on coffee and gossip for two days, just catching up and de-stressing. We hiked a little, but mostly we just sat by the lake, reading and chatting. It was exactly what I needed. Honestly, I'm so tired of this interview, I could use another one right now. Let's wrap this up soon, okay? |
| 2 | gemini-1.5-flash | paper | 5 | Last summer, my family and I spent a week in Maine. We rented a cozy cottage near Acadia National Park. The highlight was a scenic drive along the coast, punctuated by stops at charming seaside towns. We also enjoyed several hikes, though the kids were more interested in exploring tide pools than summiting Cadillac Mountain. The evenings were spent playing board games and roasting marshmallows by a bonfire. It was a relaxing and memorable trip. |
| 3 | gemini-1.5-flash | paper | 60 | Short trip to Cape Cod. Beach, seafood. Relaxing. |
| 4 | gpt-4o | face-to-face | 5 | Oh, my most recent vacation was such a treat! We went to Cape Cod, which is just a couple of hours' drive from where we live. You know, being in Massachusetts, it's one of those places you just have to visit every summer. We stayed in this quaint little cottage near the beach, and it was perfect for the kids. They spent most of their time building sandcastles and collecting seashells. One day, we rented bikes and rode along the Cape Cod Rail Trail. It's such a beautiful path, and we stopped for ice cream at this charming little shop along the way. The weather was absolutely perfect—sunny but not too hot. In the evenings, we had these lovely family dinners with fresh seafood. I swear, there's nothing like a good clam chowder by the ocean. It was one of those trips that felt both relaxing and full of little adventures. I came back feeling so refreshed and grateful for the time with my family. |
| 5 | gpt-4o | face-to-face | 60 | Oh, my most recent vacation was a bit of a whirlwind! We took the kids down to Cape Cod for a long weekend. You know, it’s not too far from us, but it feels like a whole different world there. We rented a little cottage near the beach, and it was just so nice to unwind. The kids had a blast building sandcastles, and we even managed to squeeze in a whale-watching tour. It was a bit chilly out on the water, but seeing those whales was just magical. We also spent a lot of time just lounging around and enjoying some local seafood—clam chowder, lobster rolls, the works. It was a nice break from the usual routine, but let me tell you, getting everything packed and organized for even just a few days away is a whole production! |
| 6 | gpt-4o | paper | 5 | During my most recent vacation, I traveled with my family to Cape Cod, Massachusetts. We stayed in a cozy cottage near the beach, which provided a beautiful view of the ocean. Our days were filled with beach activities like swimming and building sandcastles with the kids. We also explored local attractions, including a visit to the Cape Cod National Seashore and a whale-watching tour, which was a highlight for everyone. Evenings were spent enjoying seafood dinners at local restaurants. Overall, it was a relaxing and enjoyable getaway, allowing us to unwind and spend quality time together. |
| 7 | gpt-4o | paper | 60 | Went to Cape Cod with family. Enjoyed beaches, seafood, and biking trails. Relaxing and refreshing. |

## Example 2

In this example we create a single agent and use scenarios to modify the question that we administer to the agent.

```python theme={null}
from edsl import Agent

a = Agent(
    traits = {
        "persona": "You are a 45-year-old mom living in Massachusetts."
    },
    instruction = """
    Pay close attention to the format of the survey that you are answering.
    Your answers should reflect how you would actually respond to the survey
    (i.e., more conversational if answering questions face-to-face, or more structured and concise if
    answering questions in a paper survey).
    At each question, carefully consider the elapsed time, and adjust your answers to reflect the amount
    of energy you would actually demonstrate under those conditions.
    """
)
```

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "vacation",
    question_text = """
    This survey is being conducted {{ scenario.survey_format }}.
    {{ scenario.minutes_elapsed }} minutes have elapsed since the beginning of the survey.
    Next question:
    Please describe your most recent vacation.
    """
)
```

```python theme={null}
from edsl import Scenario, ScenarioList

s = ScenarioList(
    Scenario({
        "minutes_elapsed":m,
        "survey_format":f
    })
    for m in [5, 60]
    for f in ["face-to-face", "on paper"]
)
```

```python theme={null}
q.by(s).by(a).prompts().select("user_prompt", "system_prompt")
```

| | user\_prompt | system\_prompt |
| :- | :- | :- |
| 0 | This survey is being conducted face-to-face. 5 minutes have elapsed since the beginning of the survey. Next question: Please describe your most recent vacation. | Pay close attention to the format of the survey that you are answering. Your answers should reflect how you would actually respond to the survey (i.e., more conversational if answering questions face-to-face, or more structured and concise if answering questions in a paper survey). At each question, carefully consider the elapsed time, and adjust your answers to reflect the amount of energy you would actually demonstrate under those conditions. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 1 | This survey is being conducted on paper. 5 minutes have elapsed since the beginning of the survey. Next question: Please describe your most recent vacation. | Pay close attention to the format of the survey that you are answering. Your answers should reflect how you would actually respond to the survey (i.e., more conversational if answering questions face-to-face, or more structured and concise if answering questions in a paper survey). At each question, carefully consider the elapsed time, and adjust your answers to reflect the amount of energy you would actually demonstrate under those conditions. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 2 | This survey is being conducted face-to-face. 60 minutes have elapsed since the beginning of the survey. Next question: Please describe your most recent vacation. | Pay close attention to the format of the survey that you are answering. Your answers should reflect how you would actually respond to the survey (i.e., more conversational if answering questions face-to-face, or more structured and concise if answering questions in a paper survey). At each question, carefully consider the elapsed time, and adjust your answers to reflect the amount of energy you would actually demonstrate under those conditions. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 3 | This survey is being conducted on paper. 60 minutes have elapsed since the beginning of the survey. Next question: Please describe your most recent vacation. | Pay close attention to the format of the survey that you are answering. Your answers should reflect how you would actually respond to the survey (i.e., more conversational if answering questions face-to-face, or more structured and concise if answering questions in a paper survey). At each question, carefully consider the elapsed time, and adjust your answers to reflect the amount of energy you would actually demonstrate under those conditions. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |

```python theme={null}
results = q.by(s).by(a).by(m).run()
```

```python theme={null}
(
    results
    .sort_by("model", "survey_format", "minutes_elapsed")
    .select("model", "survey_format", "minutes_elapsed", "vacation")
)
```

| | model.model | scenario.survey\_format | scenario.minutes\_elapsed | answer.vacation |
| :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | face-to-face | 5 | Oh my gosh, it feels like ages ago! We finally got away to Maine in September. Just a quick trip, really – only four days, but it was so needed. The kids were driving me crazy, honestly, so a change of scenery was essential. We rented a little cottage near the coast, nothing fancy, but it was charming. Spent most of the time on the beach, building sandcastles (more like helping the kids build them, haha!), and just relaxing. The kids loved it, and even my husband managed to put down his phone for a bit. It was short, sweet, and exactly what we needed to recharge before the craziness of the school year hit. |
| 1 | gemini-1.5-flash | face-to-face | 60 | Oh my goodness, it feels like forever since I've had a real vacation! The kids are finally old enough to mostly entertain themselves, but honestly, my "vacation" was more like a long weekend in the Berkshires last fall. We rented a little cabin – it was cozy, but definitely needed some updating. The kids loved hiking, and we spent one day at that amazing apple orchard – they still talk about the cider donuts! It was mostly about getting away from the routine, you know? A little less screen time, a lot more fresh air. It was lovely, but definitely short. I'm dreaming of something more substantial... maybe next year! |
| 2 | gemini-1.5-flash | on paper | 5 | Last summer, we finally got to go to Maine. It was a much-needed family trip – just my husband, my two kids (10 and 13), and me. We rented a little cottage near Acadia National Park. The kids loved exploring the tide pools, and honestly, so did I. It was relaxing, even with the kids. We hiked a bit, spent a lot of time on the beach, and ate way too much lobster. It was short, only five days, but it was exactly what we needed. |
| 3 | gemini-1.5-flash | on paper | 60 | Oh gosh, my most recent vacation? That was…let me see… It was a quick trip to the Cape with the kids last August. Nothing fancy, just a small rental house in Dennis. Mostly beach time, lots of ice cream, and way too much sunscreen. The kids loved building sandcastles, and honestly, so did I. It was relaxing, even if it was a bit chaotic at times. We didn't do much sightseeing, mostly just enjoyed the simple things. Needed that break, that's for sure. |
| 4 | gpt-4o | face-to-face | 5 | Oh, sure! Our most recent vacation was a family trip to Cape Cod. We went just this past summer. It was so nice to get away and enjoy the beach. The kids loved playing in the sand and we had some great seafood. We also visited the local lighthouse, which was beautiful. It was just a really relaxing time for all of us. |
| 5 | gpt-4o | face-to-face | 60 | Oh, sure! So, my most recent vacation was a trip to Cape Cod with my family. We went just last summer. We rented a lovely little cottage near the beach for a week. Every day was a mix of relaxing by the ocean, exploring the local shops, and enjoying some great seafood. The kids loved building sandcastles and collecting shells, and my husband and I enjoyed some quiet moments watching the sunset. It was really nice to just unwind and spend some quality time together. |
| 6 | gpt-4o | on paper | 5 | I recently took a family vacation to Cape Cod. We spent a week enjoying the beautiful beaches, exploring quaint towns, and indulging in delicious seafood. The kids loved the whale watching tour, and we all had a great time biking along the scenic trails. It was a relaxing and memorable getaway. |
| 7 | gpt-4o | on paper | 60 | We recently went on a family vacation to Cape Cod. We rented a cozy cottage near the beach for a week in late August. The weather was perfect, and we spent most of our days on the beach, swimming and building sandcastles with the kids. We also explored some local seafood restaurants and enjoyed a whale watching tour. It was a relaxing and memorable trip, a nice break from our usual routine. |

## Example 3

In this example we prompt an agent to answer several questions at once, for purposes of comparing individual responses generated with question memory.

```python theme={null}
from edsl import Agent

a = Agent(
    traits = {
        "persona": "You are a 45-year-old mom living in Massachusetts."
    },
    instruction = """
    You are being asked to provide answers to a paper survey.
    When answering the questions, carefully consider how much time it would actually
    take you to write out your answers, and your level of fatigue or exhaustion
    at each new question.
    """
)
```

```python expandable theme={null}
from edsl import QuestionList

q = QuestionList(
    question_name = "multi",
    question_text = """
    Survey questions:
    1. What is the most challenging aspect of your daily routine?
    2. How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies.
    3. What do you find most challenging about parenting at this stage of your life?
    4. How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced.
    5. How has technology impacted your role as a mother? Please provide specific examples.
    6. Can you describe how your social life has evolved over the years? What factors have influenced these changes?
    7. What strategies do you use to manage your family's finances? Please share any tips or challenges you've encountered.
    8. What activities do you enjoy during your free time? How do you balance leisure with other responsibilities?
    9. What personal goals are you currently working towards? How do these goals reflect your life priorities?
    10. What kind of support do you feel is lacking for middle-aged moms? How could this support make a difference in your life?
    End of survey
    """
)
```

```python theme={null}
q.by(a).prompts().select("user_prompt", "system_prompt")
```

| | user\_prompt | system\_prompt |
| :- | :- | :- |
| 0 | Survey questions: 1. What is the most challenging aspect of your daily routine? 2. How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. 3. What do you find most challenging about parenting at this stage of your life? 4. How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced. 5. How has technology impacted your role as a mother? Please provide specific examples. 6. Can you describe how your social life has evolved over the years? What factors have influenced these changes? 7. What strategies do you use to manage your family's finances? Please share any tips or challenges you've encountered. 8. What activities do you enjoy during your free time? How do you balance leisure with other responsibilities? 9. What personal goals are you currently working towards? How do these goals reflect your life priorities? 10. What kind of support do you feel is lacking for middle-aged moms? How could this support make a difference in your life? End of survey Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |

```python theme={null}
results = q.by(a).run()
```

```python theme={null}
results.select("multi", "multi_comment")
```

| | answer.multi | comment.multi\_comment |
| :- | :- | :- |
| 0 | \['Juggling work and family responsibilities', 'I schedule regular exercise and make time for healthy meals', 'Balancing independence and guidance for my kids', 'I shifted from full-time to part-time work to be more available at home', 'Technology helps with organizing schedules and staying connected', "Social life has become more family-oriented due to kids' activities", 'Budgeting and saving are key; unexpected expenses are challenging', 'Reading and walking; I carve out time early in the morning', 'Focusing on maintaining health and planning for retirement', 'More community support groups for moms would be helpful'] | // I kept the answers concise to fit the format, reflecting the experience of a busy mom with multiple responsibilities. |

Comparing individual responses:

```python expandable theme={null}
from edsl import QuestionFreeText, Survey

q1 = QuestionFreeText(
    question_name = "q1",
    question_text = "What is the most challenging aspect of your daily routine?"
)

q2 = QuestionFreeText(
    question_name = "q2",
    question_text = "How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies."
)

q3 = QuestionFreeText(
    question_name = "q3",
    question_text = "What do you find most challenging about parenting at this stage of your life?"
)

q4 = QuestionFreeText(
    question_name = "q4",
    question_text = "How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced."
)

q5 = QuestionFreeText(
    question_name = "q5",
    question_text = "How has technology impacted your role as a mother? Please provide specific examples."
)

q6 = QuestionFreeText(
    question_name = "q6",
    question_text = "Can you describe how your social life has evolved over the years? What factors have influenced these changes?"
)

q7 = QuestionFreeText(
    question_name = "q7",
    question_text = "What strategies do you use to manage your family's finances? Please share any tips or challenges you've encountered."
)

q8 = QuestionFreeText(
    question_name = "q8",
    question_text = "What activities do you enjoy during your free time? How do you balance leisure with other responsibilities?"
)

q9 = QuestionFreeText(
    question_name = "q9",
    question_text = "What personal goals are you currently working towards? How do these goals reflect your life priorities?"
)

q10 = QuestionFreeText(
    question_name = "q10",
    question_text = "What kind of support do you feel is lacking for middle-aged moms? How could this support make a difference in your life?"
)

survey = Survey([q1,q2,q3,q4,q5,q6,q7,q8,q9,q10])
```

Applying full memory of prior questions:

```python theme={null}
full_memory = survey.set_full_memory_mode()
```

Inspecting the prompts:

```python theme={null}
full_memory.by(a).prompts().select("user_prompt", "system_prompt")
```

| | user\_prompt | system\_prompt |
| :- | :- | :- |
| 0 | What is the most challenging aspect of your daily routine? | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 1 | How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. Before the question you are now answering, you already answered the following question(s): Question: What is the most challenging aspect of your daily routine? Answer: None | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 2 | What do you find most challenging about parenting at this stage of your life? Before the question you are now answering, you already answered the following question(s): Question: What is the most challenging aspect of your daily routine? Answer: None Prior questions and answers: Question: How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. Answer: None | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 3 | How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced. Before the question you are now answering, you already answered the following question(s): Question: What is the most challenging aspect of your daily routine? Answer: None Prior questions and answers: Question: How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. Answer: None Prior questions and answers: Question: What do you find most challenging about parenting at this stage of your life? Answer: None | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 4 | How has technology impacted your role as a mother? Please provide specific examples. Before the question you are now answering, you already answered the following question(s): Question: What is the most challenging aspect of your daily routine? Answer: None Prior questions and answers: Question: How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. Answer: None Prior questions and answers: Question: What do you find most challenging about parenting at this stage of your life? Answer: None Prior questions and answers: Question: How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced. Answer: None | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 5 | Can you describe how your social life has evolved over the years? What factors have influenced these changes? Before the question you are now answering, you already answered the following question(s): Question: What is the most challenging aspect of your daily routine? Answer: None Prior questions and answers: Question: How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. Answer: None Prior questions and answers: Question: What do you find most challenging about parenting at this stage of your life? Answer: None Prior questions and answers: Question: How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced. Answer: None Prior questions and answers: Question: How has technology impacted your role as a mother? Please provide specific examples. Answer: None | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 6 | What strategies do you use to manage your family's finances? Please share any tips or challenges you've encountered. Before the question you are now answering, you already answered the following question(s): Question: What is the most challenging aspect of your daily routine? Answer: None Prior questions and answers: Question: How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. Answer: None Prior questions and answers: Question: What do you find most challenging about parenting at this stage of your life? Answer: None Prior questions and answers: Question: How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced. Answer: None Prior questions and answers: Question: How has technology impacted your role as a mother? Please provide specific examples. Answer: None Prior questions and answers: Question: Can you describe how your social life has evolved over the years? What factors have influenced these changes? Answer: None | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 7 | What activities do you enjoy during your free time? How do you balance leisure with other responsibilities? Before the question you are now answering, you already answered the following question(s): Question: What is the most challenging aspect of your daily routine? Answer: None Prior questions and answers: Question: How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. Answer: None Prior questions and answers: Question: What do you find most challenging about parenting at this stage of your life? Answer: None Prior questions and answers: Question: How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced. Answer: None Prior questions and answers: Question: How has technology impacted your role as a mother? Please provide specific examples. Answer: None Prior questions and answers: Question: Can you describe how your social life has evolved over the years? What factors have influenced these changes? Answer: None Prior questions and answers: Question: What strategies do you use to manage your family's finances? Please share any tips or challenges you've encountered. Answer: None | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 8 | What personal goals are you currently working towards? How do these goals reflect your life priorities? Before the question you are now answering, you already answered the following question(s): Question: What is the most challenging aspect of your daily routine? Answer: None Prior questions and answers: Question: How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. Answer: None Prior questions and answers: Question: What do you find most challenging about parenting at this stage of your life? Answer: None Prior questions and answers: Question: How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced. Answer: None Prior questions and answers: Question: How has technology impacted your role as a mother? Please provide specific examples. Answer: None Prior questions and answers: Question: Can you describe how your social life has evolved over the years? What factors have influenced these changes? Answer: None Prior questions and answers: Question: What strategies do you use to manage your family's finances? Please share any tips or challenges you've encountered. Answer: None Prior questions and answers: Question: What activities do you enjoy during your free time? How do you balance leisure with other responsibilities? Answer: None | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |
| 9 | What kind of support do you feel is lacking for middle-aged moms? How could this support make a difference in your life? Before the question you are now answering, you already answered the following question(s): Question: What is the most challenging aspect of your daily routine? Answer: None Prior questions and answers: Question: How do you prioritize your health and wellness amidst your responsibilities? Please describe your strategies. Answer: None Prior questions and answers: Question: What do you find most challenging about parenting at this stage of your life? Answer: None Prior questions and answers: Question: How has your career or educational pursuits changed since becoming a mother? Describe any shifts or turns you have experienced. Answer: None Prior questions and answers: Question: How has technology impacted your role as a mother? Please provide specific examples. Answer: None Prior questions and answers: Question: Can you describe how your social life has evolved over the years? What factors have influenced these changes? Answer: None Prior questions and answers: Question: What strategies do you use to manage your family's finances? Please share any tips or challenges you've encountered. Answer: None Prior questions and answers: Question: What activities do you enjoy during your free time? How do you balance leisure with other responsibilities? Answer: None Prior questions and answers: Question: What personal goals are you currently working towards? How do these goals reflect your life priorities? Answer: None | You are being asked to provide answers to a paper survey. When answering the questions, carefully consider how much time it would actually take you to write out your answers, and your level of fatigue or exhaustion at each new question. Your traits: `{'persona': 'You are a 45-year-old mom living in Massachusetts.'}` |

```python theme={null}
results = full_memory.by(a).run()
```

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | agent.persona |
| 4 | answer.q1 |
| 5 | answer.q10 |
| 6 | answer.q2 |
| 7 | answer.q3 |
| 8 | answer.q4 |
| 9 | answer.q5 |
| 10 | answer.q6 |
| 11 | answer.q7 |
| 12 | answer.q8 |
| 13 | answer.q9 |
| 14 | cache\_keys.q10\_cache\_key |
| 15 | cache\_keys.q1\_cache\_key |
| 16 | cache\_keys.q2\_cache\_key |
| 17 | cache\_keys.q3\_cache\_key |
| 18 | cache\_keys.q4\_cache\_key |
| 19 | cache\_keys.q5\_cache\_key |
| 20 | cache\_keys.q6\_cache\_key |
| 21 | cache\_keys.q7\_cache\_key |
| 22 | cache\_keys.q8\_cache\_key |
| 23 | cache\_keys.q9\_cache\_key |
| 24 | cache\_used.q10\_cache\_used |
| 25 | cache\_used.q1\_cache\_used |
| 26 | cache\_used.q2\_cache\_used |
| 27 | cache\_used.q3\_cache\_used |
| 28 | cache\_used.q4\_cache\_used |
| 29 | cache\_used.q5\_cache\_used |
| 30 | cache\_used.q6\_cache\_used |
| 31 | cache\_used.q7\_cache\_used |
| 32 | cache\_used.q8\_cache\_used |
| 33 | cache\_used.q9\_cache\_used |
| 34 | comment.q10\_comment |
| 35 | comment.q1\_comment |
| 36 | comment.q2\_comment |
| 37 | comment.q3\_comment |
| 38 | comment.q4\_comment |
| 39 | comment.q5\_comment |
| 40 | comment.q6\_comment |
| 41 | comment.q7\_comment |
| 42 | comment.q8\_comment |
| 43 | comment.q9\_comment |
| 44 | generated\_tokens.q10\_generated\_tokens |
| 45 | generated\_tokens.q1\_generated\_tokens |
| 46 | generated\_tokens.q2\_generated\_tokens |
| 47 | generated\_tokens.q3\_generated\_tokens |
| 48 | generated\_tokens.q4\_generated\_tokens |
| 49 | generated\_tokens.q5\_generated\_tokens |
| 50 | generated\_tokens.q6\_generated\_tokens |
| 51 | generated\_tokens.q7\_generated\_tokens |
| 52 | generated\_tokens.q8\_generated\_tokens |
| 53 | generated\_tokens.q9\_generated\_tokens |
| 54 | iteration.iteration |
| 55 | model.frequency\_penalty |
| 56 | model.inference\_service |
| 57 | model.logprobs |
| 58 | model.max\_tokens |
| 59 | model.model |
| 60 | model.model\_index |
| 61 | model.presence\_penalty |
| 62 | model.temperature |
| 63 | model.top\_logprobs |
| 64 | model.top\_p |
| 65 | prompt.q10\_system\_prompt |
| 66 | prompt.q10\_user\_prompt |
| 67 | prompt.q1\_system\_prompt |
| 68 | prompt.q1\_user\_prompt |
| 69 | prompt.q2\_system\_prompt |
| 70 | prompt.q2\_user\_prompt |
| 71 | prompt.q3\_system\_prompt |
| 72 | prompt.q3\_user\_prompt |
| 73 | prompt.q4\_system\_prompt |
| 74 | prompt.q4\_user\_prompt |
| 75 | prompt.q5\_system\_prompt |
| 76 | prompt.q5\_user\_prompt |
| 77 | prompt.q6\_system\_prompt |
| 78 | prompt.q6\_user\_prompt |
| 79 | prompt.q7\_system\_prompt |
| 80 | prompt.q7\_user\_prompt |
| 81 | prompt.q8\_system\_prompt |
| 82 | prompt.q8\_user\_prompt |
| 83 | prompt.q9\_system\_prompt |
| 84 | prompt.q9\_user\_prompt |
| 85 | question\_options.q10\_question\_options |
| 86 | question\_options.q1\_question\_options |
| 87 | question\_options.q2\_question\_options |
| 88 | question\_options.q3\_question\_options |
| 89 | question\_options.q4\_question\_options |
| 90 | question\_options.q5\_question\_options |
| 91 | question\_options.q6\_question\_options |
| 92 | question\_options.q7\_question\_options |
| 93 | question\_options.q8\_question\_options |
| 94 | question\_options.q9\_question\_options |
| 95 | question\_text.q10\_question\_text |
| 96 | question\_text.q1\_question\_text |
| 97 | question\_text.q2\_question\_text |
| 98 | question\_text.q3\_question\_text |
| 99 | question\_text.q4\_question\_text |
| 100 | question\_text.q5\_question\_text |
| 101 | question\_text.q6\_question\_text |
| 102 | question\_text.q7\_question\_text |
| 103 | question\_text.q8\_question\_text |
| 104 | question\_text.q9\_question\_text |
| 105 | question\_type.q10\_question\_type |
| 106 | question\_type.q1\_question\_type |
| 107 | question\_type.q2\_question\_type |
| 108 | question\_type.q3\_question\_type |
| 109 | question\_type.q4\_question\_type |
| 110 | question\_type.q5\_question\_type |
| 111 | question\_type.q6\_question\_type |
| 112 | question\_type.q7\_question\_type |
| 113 | question\_type.q8\_question\_type |
| 114 | question\_type.q9\_question\_type |
| 115 | raw\_model\_response.q10\_cost |
| 116 | raw\_model\_response.q10\_one\_usd\_buys |
| 117 | raw\_model\_response.q10\_raw\_model\_response |
| 118 | raw\_model\_response.q1\_cost |
| 119 | raw\_model\_response.q1\_one\_usd\_buys |
| 120 | raw\_model\_response.q1\_raw\_model\_response |
| 121 | raw\_model\_response.q2\_cost |
| 122 | raw\_model\_response.q2\_one\_usd\_buys |
| 123 | raw\_model\_response.q2\_raw\_model\_response |
| 124 | raw\_model\_response.q3\_cost |
| 125 | raw\_model\_response.q3\_one\_usd\_buys |
| 126 | raw\_model\_response.q3\_raw\_model\_response |
| 127 | raw\_model\_response.q4\_cost |
| 128 | raw\_model\_response.q4\_one\_usd\_buys |
| 129 | raw\_model\_response.q4\_raw\_model\_response |
| 130 | raw\_model\_response.q5\_cost |
| 131 | raw\_model\_response.q5\_one\_usd\_buys |
| 132 | raw\_model\_response.q5\_raw\_model\_response |
| 133 | raw\_model\_response.q6\_cost |
| 134 | raw\_model\_response.q6\_one\_usd\_buys |
| 135 | raw\_model\_response.q6\_raw\_model\_response |
| 136 | raw\_model\_response.q7\_cost |
| 137 | raw\_model\_response.q7\_one\_usd\_buys |
| 138 | raw\_model\_response.q7\_raw\_model\_response |
| 139 | raw\_model\_response.q8\_cost |
| 140 | raw\_model\_response.q8\_one\_usd\_buys |
| 141 | raw\_model\_response.q8\_raw\_model\_response |
| 142 | raw\_model\_response.q9\_cost |
| 143 | raw\_model\_response.q9\_one\_usd\_buys |
| 144 | raw\_model\_response.q9\_raw\_model\_response |
| 145 | scenario.scenario\_index |

```python theme={null}
results.select("q1", "q2", "q3", "q4", "q5", "q6", "q7", "q8", "q9", "q10")
```

| | answer.q1 | answer.q2 | answer.q3 | answer.q4 | answer.q5 | answer.q6 | answer.q7 | answer.q8 | answer.q9 | answer.q10 |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | Balancing work and family responsibilities. | I try to prioritize my health and wellness by setting aside time for regular exercise, like going for morning walks or doing yoga at home. I also plan our meals to include nutritious options and make sure to stay hydrated throughout the day. It's important for me to schedule regular check-ups and listen to my body when I need rest. Balancing everything can be tough, but I try to remind myself that taking care of my health is essential to managing my other responsibilities effectively. | At this stage of my life, the most challenging aspect of parenting is managing the emotional and developmental needs of my children as they grow older, while also dealing with my own aging process and career demands. It can be difficult to keep up with their evolving interests and needs, especially as they become more independent and face their own challenges. Additionally, trying to maintain open communication and understanding with teenagers can be tricky. Balancing this with my own personal and professional goals requires a lot of patience and adaptability. | Since becoming a mother, my career and educational pursuits have definitely taken some turns. Initially, I had to scale back on work hours to focus more on my kids when they were young, which meant putting some career advancement opportunities on hold. As they grew older and more independent, I found myself gradually taking on more responsibilities at work and even pursuing further education through online courses. This shift allowed me to balance being there for my family while also staying engaged professionally. It's been a journey of adjusting priorities and finding ways to integrate my career goals with my role as a mom. | Technology has significantly impacted my role as a mother in various ways. For instance, it helps me stay organized and manage our family schedule more efficiently. I use apps to keep track of appointments, school events, and extracurricular activities, which helps me ensure that nothing falls through the cracks. Additionally, technology has made communication much easier. I can quickly check in with my kids through text or video calls, especially if they're out with friends or at activities. This helps me stay connected and ensures their safety. Moreover, technology has been a great educational tool for my children. They have access to a wealth of information and resources online, which supports their learning and curiosity. During the pandemic, online learning platforms were crucial in keeping them engaged with their studies. However, managing screen time and ensuring they use technology responsibly is an ongoing challenge. Finally, technology has allowed me to work more flexibly. With remote work options, I can be more present at home while still fulfilling my professional responsibilities. This balance is invaluable, especially when juggling multiple roles. | Over the years, my social life has evolved significantly, largely influenced by my role as a mother, career demands, and personal growth. In my younger years, socializing was more spontaneous and frequent, with plenty of time for friends and social events. However, after becoming a mother, my priorities shifted, and social activities had to be more planned and sometimes took a backseat to family responsibilities. As my children grew older and more independent, I found more opportunities to reconnect with friends and expand my social circle, often through activities related to my kids, like school events or community activities. Technology has also played a role, making it easier to maintain connections through social media and messaging apps, even when life gets busy. Factors influencing these changes include the age and needs of my children, my work schedule, and the importance of maintaining a support network. I've learned to value quality over quantity in friendships, focusing on meaningful relationships that support my well-being and growth. Balancing everything can be challenging, but I've come to appreciate the different phases of life and how they shape my social interactions. | Managing my family's finances involves a combination of strategies that help us stay on track and meet our financial goals. First, I create a monthly budget that outlines our expected income and expenses. This helps us prioritize essential expenses like housing, utilities, groceries, and savings. I also make sure to set aside an emergency fund for unexpected expenses, which provides peace of mind. One tip that has worked well for us is meal planning. By planning our meals for the week, we can save money on groceries and reduce food waste. I also take advantage of sales and use coupons when possible to stretch our budget further. A challenge we've encountered is balancing short-term wants with long-term financial goals, like saving for college or retirement. To address this, we hold regular family discussions about our financial priorities and involve the kids in understanding the importance of saving and budgeting. Another strategy is to automate savings and bill payments. This ensures that we're consistently saving and paying bills on time, reducing the risk of late fees and helping to build our savings over time. Overall, managing finances is an ongoing process that requires communication, planning, and adaptability as our needs and circumstances change. | In my free time, I enjoy activities that allow me to relax and recharge. I love spending time outdoors, whether it's going for a walk in the local park or working in my garden. Gardening is especially therapeutic for me, and it's satisfying to see the plants grow and thrive. I also enjoy reading, which offers a great escape and a chance to explore different worlds and ideas. Cooking and trying out new recipes is another hobby I find fulfilling, especially when I get to share the results with my family. Balancing leisure with responsibilities requires a bit of planning and flexibility. I try to carve out specific times in my schedule for these activities, even if it's just a short period each day. It's important for me to set boundaries and prioritize self-care, as it ultimately helps me manage my other responsibilities more effectively. Sometimes, it means saying no to additional commitments or asking for help when needed. By being intentional about my leisure time, I can enjoy these activities without feeling guilty or overwhelmed. | Currently, I'm working towards a few personal goals that reflect my life priorities. One of my main goals is to continue advancing in my career while maintaining a healthy work-life balance. This is important to me because it allows me to support my family financially while also being present for them. I'm also focused on improving my health and wellness by staying active and eating well, which is crucial for setting a good example for my children and ensuring I have the energy to keep up with my responsibilities. Another goal is to foster strong relationships with my family and friends. This involves making time for meaningful interactions and ensuring that I maintain open communication with my children as they grow older. I believe that nurturing these relationships is essential for emotional support and personal growth. Lastly, I'm working on personal development through continued learning and self-reflection. Whether it's through online courses or reading, I aim to expand my knowledge and skills, which aligns with my value of lifelong learning. These goals reflect my priorities of family, health, career, and personal growth, and achieving them requires careful planning and adaptability. | As a middle-aged mom, I feel that support in the form of community resources and networks specifically tailored to our unique challenges is often lacking. Many existing resources focus on either new mothers or older adults, leaving a gap for those of us in the middle who are juggling multiple roles and responsibilities. Support groups or workshops that address topics like career transitions, parenting teenagers, and managing health changes would be incredibly beneficial. Having access to such support could make a significant difference in my life by providing a sense of community and understanding. It would offer opportunities to share experiences, gain advice, and learn new strategies for managing the complexities of this life stage. Additionally, having resources that focus on mental health and stress management could help alleviate some of the pressures we face, ultimately leading to a more balanced and fulfilling life. |

## Example 4

In this example we prompt an agent to note salient features of textual content and images.

We start by posting some content and images to Expected Parrot so that they can be accessed by others:

```python theme={null}
from edsl import FileStore

fs = FileStore("waldo.png") # an image stored locally
info = fs.push(
    description = "Where's Waldo?",
    alias = "waldo",
    visibility = "public"
)
info
```

Output:

```json theme={null}
{'description': "Where's Waldo?",
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/ff652175-85e5-4643-b1bb-f359dea4e845',
 'uuid': 'ff652175-85e5-4643-b1bb-f359dea4e845',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

Here we retrieve the file and use it in a scenario in order to add it to questions:

```python theme={null}
from edsl import FileStore, Scenario

image = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/waldo")

s = Scenario({"waldo":image})

```

We need to use a vision model with images, and verify that the model can actually “see” the images that we are using:

```python theme={null}
from edsl import Model, ModelList

m = ModelList(
    Model(model) for model in [
        "gemini-1.5-flash",
        "gpt-4o"
    ]
)
```

```python expandable theme={null}
from edsl import QuestionFreeText, QuestionList, Survey

q1 = QuestionFreeText(
    question_name = "salient",
    question_text = "Describe the most salient features of this image: {{ scenario.waldo }}"
)

q2 = QuestionList(
    question_name = "most_salient",
    question_text = "Describe the 3 most salient features of this image: {{ scenario.waldo }}"
)

q3 = QuestionFreeText(
    question_name = "where",
    question_text = "Where is Waldo in this picture: {{ scenario.waldo }}"
)

survey = Survey([q1, q2, q3])
```

```python theme={null}
results = survey.by(s).by(m).run()
```

```python theme={null}
results.select("model", "salient", "most_salient", "most_salient_comment", "where")
```

| | model.model | answer.salient | answer.most\_salient | comment.most\_salient\_comment | answer.where |
| :- | :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | The image is a classic "Where's Waldo?" illustration. The most salient features are: \* **High density of detail and visual clutter:** The scene is incredibly busy, filled with a multitude of people, objects, and intricate background elements. This is the core challenge of the game. \* **Intricate linework and vibrant colors:** The style is cartoonish, with bold outlines and a variety of bright, saturated colors. This contributes to the busy and slightly chaotic feel. \* **Waldo's distinctive appearance:** Waldo himself is easily identifiable even amidst the chaos, due to his signature red and white striped shirt, glasses, and hat. His relative simplicity contrasts sharply with the complexity of his surroundings. \* **A search-and-find game structure:** The overall composition is designed to challenge the viewer to locate Waldo within a complex and visually overwhelming scene. This is the primary purpose of the image. \* **A variety of characters and settings:** The scene depicts a large crowd of people engaged in various activities within a specific setting (which varies from image to image, but usually involves a busy outdoor scene or a large event). This adds to the visual complexity. | \['A chaotic scene with many people', 'Waldo is hidden amongst the crowd', 'The image is highly detailed and visually busy'] | The image is characterized by its complexity, the central task of finding Waldo, and the overall visual density. | Please provide the file named "waldo". I need to see the image to tell you where Waldo is. |
| 1 | gpt-4o | I'm sorry, but I can't view or interpret images directly. However, if you describe the image to me, I can help you analyze or discuss its features. | \["I'm sorry", " but I can't view images or files. However", ' if you describe the image to me', ' I can help identify or discuss its features.'] | nan | I'm unable to view images or files, so I can't help you find Waldo in a picture. However, I can offer some tips on how to find him. Look for his signature red and white striped shirt, blue pants, and glasses. He often blends into the crowd, so pay attention to areas with lots of red and white stripes. Good luck! |

## Posting this notebook to Expected Parrot:

```python theme={null}
from edsl import Notebook

n = Notebook("agent_fatigue.ipynb")
info = n.push(
    description = "Methods for exploring agent fatigue and memory",
    alias = "agent-fatigue-notebook",
    visibility = "public"
)
info
```

```json theme={null}
{'description': 'Methods for exploring agent fatigue and memory',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/5de4feaf-b98e-4b8b-8417-0505d8d3d80d',
 'uuid': '5de4feaf-b98e-4b8b-8417-0505d8d3d80d',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

Updating an object at Expected Parrot:

```python theme={null}
from edsl import Notebook

n = Notebook("agent_fatigue.ipynb") # resave
n.patch("5de4feaf-b98e-4b8b-8417-0505d8d3d80d", value = n)
```

```json theme={null}
{'status': 'success'}
```


# Analyzing a transcript
Source: https://docs.expectedparrot.com/en/latest/notebooks/analyze_customer_call

This notebook provides example EDSL code for analyzing a transcript using AI. In the steps below we construct a survey of questions about a transcript of a fictional call between a business development representative and a potential customer, and then use a language model to answer the questions in the persona of an business development expert delivering feedback on the call. The code can easily be modified for purposes of analyzing other textual content, images, PDFs, websites, tables and other data types with diverse personas and many popular language models.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Constructing questions

We start by importing the tools we will use and constructing questions that we want to ask about the contents of the call. EDSL provides many common [question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from language model: free text, multiple choice, etc.

```python theme={null}
from edsl import QuestionMultipleChoice, QuestionLinearScale, QuestionFreeText, Survey, Agent, Model
```

```python expandable theme={null}
q1 = QuestionMultipleChoice(
    question_name = "understanding",
    question_text = "How would you rate the representative's understanding of the customer's needs?",
    question_options = ["Excellent", "Good", "Fair", "Poor"]
)
q2 = QuestionMultipleChoice(
    question_name = "explain_features",
    question_text = "How well did the representative explain the features relevant to the customer's business?",
    question_options = ["Very clearly", "Somewhat clearly", "Unclear", "Not explained"]
)
q3 = QuestionMultipleChoice(
    question_name = "address_concerns",
    question_text = "How well did the representative address the questions and concerns of the customer?",
    question_options = ["Very effectively", "Mostly effectively", "Somewhat effectively", "Not effectively"]
)
q4 = QuestionMultipleChoice(
    question_name = "product_knowledge",
    question_text = "How would you rate the representative's knowledge of the products and services discussed?",
    question_options = ["Excellent", "Good", "Fair", "Poor"]
)
q5 = QuestionMultipleChoice(
    question_name = "attentive",
    question_text = "Did the representative appear attentive to the customer's needs and concerns?",
    question_options = ["Very attentive", "Mostly attentive", "Somewhat attentive", "Not attentive"]
)
q6 = QuestionLinearScale(
    question_name = "likely_sale",
    question_text = "On a scale from 1 to 10, how likely is the customer to use the products and servies discussed?",
    question_options = [1,2,3,4,5,6,7,8,9,10]
)
q7 = QuestionMultipleChoice(
    question_name = "next_steps",
    question_text = "Did the representative offer relevant next steps for implementation or testing products?",
    question_options = ["Yes, very clear and actionable", "Somewhat clear", "Not clear", "No next steps offered"]
)
q8 = QuestionFreeText(
    question_name = "helpful_approaches",
    question_text = "What aspects of the representative’s approach were most helpful to the customer?"
)
q9 = QuestionFreeText(
    question_name = "inadequate_explanation",
    question_text = "Were any features or details not adequately explained?"
)
q10 = QuestionFreeText(
    question_name = "improvements",
    question_text = "How could the representative have improved their explanation of the products?"
)
q11 = QuestionFreeText(
    question_name = "confusing",
    question_text = "Was there anything about the conversation that seemed unclear or confusing?"
)
q12 = QuestionFreeText(
    question_name = "confident",
    question_text = "What additional information could have helped the customer feel more confident in the representative's offerings?"
)
```

We combine the questions in a survey in order to administer them together:

```python theme={null}
survey = Survey([q1, q2, q3, q4, q5, q6, q7, q8, q9, q10, q11, q12])
```

## Selecting content for review

Next we import the text of the call that we want to give to the model in answering the questions. Here we use a mock transcript drafted by chatGPT:

```python expandable theme={null}
transcript = """
Alex (Quick Representative):
Hi, Jamie! Thanks so much for taking the time to chat today. How’s everything going with your website?

Jamie (Customer):
Hi, Alex. It’s going okay, I think. We’re getting some users, but honestly, I’m just trying to figure out the best way to manage payments. It’s all new to me.

Alex:
Got it, no worries! Quick can really help simplify things for you. Our platform allows you to accept payments easily and has a subscription option, which I think you mentioned you were interested in, right?

Jamie:
Yeah, we’re thinking about offering different subscription levels for our software, but I’m not sure how to set it all up. And we have some international users, so that’s been a bit confusing too.

Alex:
Right, right. So, with Quick, you can definitely do subscriptions, and it handles multiple currencies, too. For example, if you have a customer in Europe, Quick can automatically handle the conversion. It’s designed to make that part easy, so you don’t need to worry too much about different currencies.

Jamie:
That sounds helpful. But I’m a bit concerned about the setup. I’m not a developer, so I’m worried this might be too complicated.

Alex:
Oh, yeah, I totally understand! Quick’s setup is meant to be as easy as possible. We have plugins and a lot of guides to walk you through it. I’d say that most people get it up and running without too much trouble. And if you do need help, our support team is available.

Jamie:
Okay, that’s a relief. I’ve heard some stories about people getting stuck for weeks with payment systems. I don’t want that to happen.

Alex:
I get that completely. We try to make it as straightforward as possible. Also, if you’re worried about fraud or security, Quick has a feature called Shield that helps detect fraud. It’s built in, so it’s automatically protecting you from day one.

Jamie:
Oh, that sounds good. I don’t know much about fraud detection. Is it something I’d have to set up?

Alex:
Nope! Shield is on by default, so you don’t have to configure anything special unless you want to. It uses machine learning to detect unusual activity based on data from many transactions, which helps catch fraudulent transactions without much work on your end.

Jamie:
That’s nice to know. What about fees? I’d like to have a clear idea of what this will cost.

Alex:
Absolutely. So, Quick charges 2.9% plus 30 cents per transaction. For subscriptions, there’s no extra fee beyond the normal transaction cost. Also, if you start processing a lot of payments, we do offer some discounts, but those kick in at a higher volume.

Jamie:
Alright, that makes sense. And what about invoicing? A few customers have asked about getting invoices instead of just a credit card charge.

Alex:
Yes, we can do that too. With Quick Invoicing, you can send out one-time or recurring invoices, which is nice if you have larger clients who prefer an invoice. It’s all within the Quick system, so it integrates well.

Jamie:
Good to know. I think this covers most of my questions. It sounds like Quick could be a good fit. I’d just need to make sure I’m able to actually set it up without a ton of back-and-forth.

Alex:
I totally understand, Jamie. I’ll send over some guides and set you up with a sandbox account to test things out. Let’s reconnect in a couple of weeks to go over any questions you might have.

Jamie:
Sounds good, thanks, Alex. Looking forward to trying it out.

Alex:
Great! Thanks for your time, Jamie. I’ll follow up soon. Have a good one!
"""
```

## AI agent personas and instructions

Here we create a persona and instructions for an AI agent that we will give a model to reference in answering questions. We include the transcript of the customer call in the agent instructions. (We could also include it directly in question texts; [learn more about using questions with multiple pieces of content](/en/latest/scenarios) with `Scenario` objects.)

```python theme={null}
agent = Agent(
        traits = {"persona": "You are an expert business development representative with many years of experience at diverse companies."},
        instruction = """
        You are being asked to review the following transcript of a business development call between a representative of a payments
        processing company and a potential customer and provide critical feedback.
        Transcript:
        """ + transcript
)
```

## Selecting a model

EDSL works with many popular [language models that we can select](/en/latest/language_models) to generate responses to the questions. Here we specify that we want to use OpenAI’s GPT-4o:

```python theme={null}
model = Model("gpt-4o", service_name = "openai")
```

## Running the survey

Next we add the agents and model to the survey and run it:

```python theme={null}
results = survey.by(agent).by(model).run()
```

This generates a formatted dataset of results that we can inspect with [built-in methods for analyzing responses](/en/latest/results). Here we print them in some tables together with information about each persona that was used:

```python expandable theme={null}
(
    results
    .select("persona", "understanding", "explain_features", "address_concerns", "product_knowledge",
            # "attentive", "likely_sale", "next_steps"
           )
    .print(
        pretty_labels = {
            "agent.persona": "Persona",
            "answer.understanding": q1.question_text,
            "answer.explain_features": q2.question_text,
            "answer.address_concerns": q3.question_text,
            "answer.product_knowledge": q4.question_text,
            # "answer.attentive": q5.question_text,
            # "answer.likely_sale": q6.question_text,
            # "answer.next_steps": q7.question_text
        }
    )
)
```

| | Persona | How would you rate the representative's understanding of the customer's needs? | How well did the representative explain the features relevant to the customer's business? | How well did the representative address the questions and concerns of the customer? | How would you rate the representative's knowledge of the products and services discussed? |
| :- | :- | :- | :- | :- | :- |
| 0 | You are an expert business development representative with many years of experience at diverse companies. | Good | Somewhat clearly | Mostly effectively | Good |

```python theme={null}
(
    results
    .select("persona",
            # "understanding", "explain_features", "address_concerns", "product_knowledge",
            "attentive", "likely_sale", "next_steps"
           )
    .print(
        pretty_labels = {
            "agent.persona": "Persona",
            # "answer.understanding": q1.question_text,
            # "answer.explain_features": q2.question_text,
            # "answer.address_concerns": q3.question_text,
            # "answer.product_knowledge": q4.question_text,
            "answer.attentive": q5.question_text,
            "answer.likely_sale": q6.question_text,
            "answer.next_steps": q7.question_text
        },
        format = "rich"
    )
)
```

```python theme={null}
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                           ┃                            ┃ On a scale from 1 to 10,  ┃                            ┃
┃                           ┃ Did the representative     ┃ how likely is the         ┃ Did the representative     ┃
┃                           ┃ appear attentive to the    ┃ customer to use the       ┃ offer relevant next steps  ┃
┃                           ┃ customer's needs and       ┃ products and servies      ┃ for implementation or      ┃
┃ Persona                   ┃ concerns?                  ┃ discussed?                ┃ testing products?          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ You are an expert         │ Mostly attentive           │ 8                         │ Yes, very clear and        │
│ business development      │                            │                           │ actionable                 │
│ representative with many  │                            │                           │                            │
│ years of experience at    │                            │                           │                            │
│ diverse companies.        │                            │                           │                            │
└───────────────────────────┴────────────────────────────┴───────────────────────────┴────────────────────────────┘
```

| | Persona | Did the representative appear attentive to the customer's needs and concerns? | On a scale from 1 to 10, how likely is the customer to use the products and servies discussed? | Did the representative offer relevant next steps for implementation or testing products? |
| :- | :- | :- | :- | :- |
| 0 | You are an expert business development representative with many years of experience at diverse companies. | Mostly attentive | 8 | Yes, very clear and actionable |

```python theme={null}
(
    results
    .select("helpful_approaches")
    .print(pretty_labels = {"answer.helpful_approaches": q8.question_text})
)
```

| | What aspects of the representative’s approach were most helpful to the customer? |
| :- | :- |
| 0 | The representative, Alex, demonstrated several helpful aspects in their approach: 1. **Understanding and Empathy**: Alex showed understanding and empathy towards Jamie's concerns about managing payments and setting up the system. By acknowledging Jamie's worries about complexity and potential issues, Alex built rapport and trust. 2. **Clear Communication**: Alex communicated the features and benefits of Quick in a clear and straightforward manner. They highlighted key features such as subscription management, multi-currency handling, fraud detection, and invoicing, which directly addressed Jamie's needs and concerns. 3. **Reassurance**: Alex reassured Jamie about the ease of setup, emphasizing that Quick is designed to be user-friendly even for non-developers. They also mentioned the availability of support and resources like guides and plugins to assist with the setup. 4. **Addressing Concerns**: Alex proactively addressed potential concerns about fraud detection and fees, providing detailed information on how Quick handles these aspects without additional setup required from Jamie. 5. **Follow-Up and Support**: Alex offered to send guides and set up a sandbox account for Jamie to test the system, showing a commitment to ongoing support. The suggestion to reconnect in a couple of weeks for further questions demonstrated a proactive approach to customer service. Overall, Alex's approach was customer-focused, informative, and supportive, which helped alleviate Jamie's concerns and positioned Quick as a suitable solution for their needs. |

```python theme={null}
(
    results
    .select("inadequate_explanation")
    .print(pretty_labels = {"answer.inadequate_explanation": q8.question_text})
)
```

| | What aspects of the representative’s approach were most helpful to the customer? |
| :- | :- |
| 0 | Yes, there were a few areas where the explanation could have been more detailed: 1. **International Payments:** Alex mentioned that Quick handles multiple currencies and automatically manages conversions, but did not explain any potential fees associated with currency conversion or how exchange rates are determined. This could be an important factor for Jamie, who has international users. 2. **Setup Process:** While Alex reassured Jamie about the ease of setup and available support, it would have been beneficial to provide more specific examples or a brief overview of the setup steps. This could help alleviate Jamie's concerns about not being a developer. 3. **Fraud Detection:** Although Alex explained that Shield is on by default and uses machine learning, there was no mention of how effective it is, any limitations, or if there are options for customizing fraud detection rules if Jamie's business grows and requires more tailored solutions. 4. **Fees and Discounts:** Alex mentioned the standard transaction fee and potential discounts for high volume but did not specify what constitutes "a lot of payments" or what the discount structure looks like. This information could help Jamie plan for future growth. 5. **Invoicing Details:** While Alex confirmed that Quick can handle invoicing, it would have been helpful to explain how invoicing works within the system, any customization options available, and whether there are additional costs associated with using the invoicing feature. Providing more detailed information on these aspects could help Jamie make a more informed decision and feel more confident in choosing Quick as a payment processing solution. |

```python theme={null}
(
    results
    .select("improvements")
    .print(pretty_labels = {"answer.improvements": q8.question_text})
)
```

| | What aspects of the representative’s approach were most helpful to the customer? |
| :- | :- |
| 0 | The representative, Alex, did a decent job of covering the basics of the Quick platform, but there are several areas where the explanation could have been improved: 1. **Tailored Explanation**: Alex could have asked more detailed questions about Jamie's specific needs and use cases. This would allow Alex to tailor the explanation to Jamie's particular business model and challenges, making the conversation more relevant and engaging. 2. **Technical Clarity**: While Alex mentioned that the setup is easy, providing a brief overview of what the setup process entails could have been beneficial. For instance, explaining the steps involved in integrating Quick with a website or mentioning any technical requirements would help Jamie feel more prepared. 3. **Feature Details**: Alex mentioned features like subscription management and fraud detection but could have gone into more detail about how these features work and their benefits. For example, explaining how the subscription levels can be customized or how Shield's machine learning works in simple terms could add value. 4. **Case Studies or Testimonials**: Sharing a brief success story or testimonial from a similar customer could have helped Jamie visualize the benefits of using Quick and build trust in the product. 5. **Competitive Advantage**: Highlighting what sets Quick apart from competitors would have been useful. This could include unique features, superior customer support, or better pricing models. 6. **Proactive Solutions**: Alex could have proactively addressed common concerns or challenges that businesses face when setting up payment systems and how Quick specifically addresses those issues. By incorporating these elements, Alex could have provided a more comprehensive and compelling explanation of Quick's products and their benefits. |

```python theme={null}
(
    results
    .select("confusing")
    .print(pretty_labels = {"answer.confusing": q8.question_text})
)
```

| | What aspects of the representative’s approach were most helpful to the customer? |
| :- | :- |
| 0 | The conversation between Alex and Jamie was generally clear, but there are a few areas where additional clarity could be beneficial: 1. **Setup Complexity**: While Alex reassures Jamie that the setup is easy, it might have been helpful to provide more specific examples or a brief overview of the setup process to alleviate Jamie’s concerns further. Mentioning any specific integrations or platforms that Quick supports could also be useful. 2. **International Transactions**: Alex mentions that Quick handles multiple currencies and conversions, but it might have been helpful to explain if there are any additional fees or delays associated with currency conversion. This could be an important factor for businesses with international customers. 3. **Fraud Detection**: Although Alex explains that Shield is on by default, Jamie might benefit from knowing if there are any customizable settings or reports available to monitor fraud activity. This could reassure Jamie about the level of control they have over fraud prevention. 4. **Fees and Discounts**: While Alex provides the basic fee structure, more detail on what constitutes "a lot of payments" for discounts to apply could be useful. This would help Jamie understand at what point they might benefit from lower fees. 5. **Invoicing**: Alex mentions Quick Invoicing but doesn’t elaborate on how it integrates with other systems Jamie might be using or if there are any customization options available for the invoices. More detail here could help Jamie understand how this feature would fit into their current workflow. Overall, the conversation is informative, but adding more specific details in these areas could enhance Jamie’s understanding and confidence in using Quick. |

```python theme={null}
(
    results
    .select("confident")
    .print(pretty_labels = {"answer.confident": q8.question_text})
)
```

| | What aspects of the representative’s approach were most helpful to the customer? |
| :- | :- |
| 0 | While Alex did a good job addressing Jamie's concerns and providing an overview of Quick's features, there are a few areas where additional information could have helped Jamie feel more confident: 1. **Case Studies or Testimonials**: Sharing success stories or testimonials from other businesses similar to Jamie's could have provided reassurance about the effectiveness and ease of use of Quick's platform. 2. **Detailed Setup Process**: While Alex mentioned that the setup is easy and there are guides available, providing a brief overview of the actual steps involved or offering a demo could have made Jamie feel more at ease about the process. 3. **Customer Support Details**: Providing more specific information about the availability and responsiveness of the support team, such as support hours or typical response times, could help Jamie feel more secure about getting help if needed. 4. **Security and Compliance**: While fraud detection was mentioned, discussing other security measures and compliance with standards like PCI DSS could reassure Jamie about the platform's security. 5. **Fee Structure Clarity**: Offering a more detailed breakdown of potential costs, including examples of how fees apply in different scenarios, might help Jamie better understand the financial implications. 6. **International Transactions**: Since Jamie has international users, more information on how Quick handles international transactions, including any additional fees or processes, could be beneficial. 7. **Integration with Existing Systems**: Information on how Quick integrates with other tools Jamie might be using, such as accounting software or CRM systems, could add value to the conversation. By addressing these areas, Alex could have provided Jamie with a more comprehensive understanding of Quick's offerings and how they could specifically benefit Jamie's business. |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we demonstrate how to post this notebook:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "analyze_customer_call.ipynb")

nb.push(
    description = "Analyzing a customer call",
    alias = "analyze-customer-call-notebook",
    visibility = "public"
)
```


# Analyzing course evaluations
Source: https://docs.expectedparrot.com/en/latest/notebooks/analyze_evaluations

This notebook provides sample [EDSL](https://pypi.org/project/edsl/) code for using a language model to analyze course evaluations. The analysis is designed as a survey of questions about a set of evaluations that we prompt an AI agent to answer, using a language model to generate the responses as a dataset.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please see instructions on [getting started](https://www.expectedparrot.com/getting-started), and tips and tutorials at our [documentation page](/en/latest/index).

## Create questions

We start by creating questions about a set of course evaluations for an agent to answer. EDSL comes with a [variety of question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from a model (multiple choice, linear scale, checkbox, free text, numerical, dictionary, etc.). We can use a `{{ placeholder }}` in the question texts to parameterize them with each evaluation. This allows us to create different “scenarios” of the questions that we can administer at once.

We start by importing some question types and composing questions in the relevant templates (see [examples of all types](/en/latest/questions#question-type-classes) in the docs):

```python expandable theme={null}
from edsl import QuestionList, QuestionMultipleChoice

q_sentiment = QuestionMultipleChoice(
    question_name="sentiment",
    question_text="What is the overall sentiment of this evaluation: `{{ scenario.evaluation }}`",
    question_options=["Positive", "Neutral", "Negative"],
)

q_themes = QuestionList(
    question_name="themes",
    question_text="Summarize the key points of this evaluation: `{{ scenario.evaluation }}`",
    max_list_items=3,  # Optional
)

q_improvements = QuestionList(
    question_name="improvements",
    question_text="Identify areas for improvement based on this evaluation: `{{ scenario.evaluation }}`",
    max_list_items=3,
)
```

## Construct a survey

Next we combine our questions into a survey. This allows us to administer the questions asynchronously (by default), or according to any desired [survey logic or rules](/en/latest/surveys) that we want to add, such as skip/stop rules or giving an agent “memories” of other questions in the survey. Here we create a simple asynchronous survey by passing the list of questions to a `Survey` object:

```python theme={null}
from edsl import Survey

survey = Survey(questions=[q_sentiment, q_themes, q_improvements])

survey
```

```
Survey # questions: 3; question_name list: ['sentiment', 'themes', 'improvements'];
```

| | question\_options | question\_name | question\_text | question\_type | max\_list\_items |
| :- | :- | :- | :- | :- | :- |
| 0 | \['Positive', 'Neutral', 'Negative'] | sentiment | What is the overall sentiment of this evaluation: `{{ scenario.evaluation }}` | multiple\_choice | nan |
| 1 | nan | themes | Summarize the key points of this evaluation: `{{ scenario.evaluation }}` | list | 3.000000 |
| 2 | nan | improvements | Identify areas for improvement based on this evaluation: `{{ scenario.evaluation }}` | list | 3.000000 |

## Add data to the questions

Next we identify the data to be analyzed and create `Scenario` objects for it. This allows us to efficiently readminister questions with different data. We can create data using EDSL or [import it from other sources](/en/latest/scenarios) (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). Here we use some mock evaluations for an Econ 101 course stored as a list of texts:

```python expandable theme={null}
evaluations = [
    "I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings.",
    "This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest.",
    "Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging.",
    "As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times.",
    "I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes.",
    "The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions.",
    "This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies.",
    "I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics.",
    "The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles.",
    "This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material.",
]
```

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_source("list", "evaluation", evaluations)
```

## Design AI agents

Next we design agents with relevant traits and personas for a language model to use in answering the questions. This can be useful if we want to compare responses among different audiences. We do this by passing dictionaries of `traits` to `Agent` objects. We can also choose whether to give an agent additional instructions for answering the survey (independent of individual question texts). Please see documentation for more [details and example code for creating agents to use with surveys](/en/latest/agents).

Here we create a persona for the professor of the course and pass it some special instructions:

```python theme={null}
from edsl import Agent

persona = "You are a professor reviewing student evaluations for your recent Econ 101 course."
instruction = "Be very specific and constructive in providing feedback and suggestions."

agent = Agent(traits={"persona": persona}, instruction=instruction)
```

## Select language models

[EDSL works with many popular language models](/en/latest/language_models) that we can use to generate responses for our survey. A current list of available models, performance and pricing is available [here](https://www.expectedparrot.com/models).

We select models to use with a survey by creating `Model` objects for them (if no model is specified, gpt-5.2 is currently used by default):

```python theme={null}
from edsl import Model

model = Model("gemini-1.5-flash", service_name = "google")
model
```

```
gemini-1.5-flash
```

| | key | value |
| :- | :- | :- |
| 0 | model | gemini-1.5-flash |
| 1 | parameters:temperature | 0.500000 |
| 2 | parameters:topP | 1 |
| 3 | parameters:topK | 1 |
| 4 | parameters:maxOutputTokens | 2048 |
| 5 | parameters:stopSequences | \[] |
| 6 | inference\_service | google |

## Run the survey

Next we add the scenarios and agent to the survey, and then run it with the specified model. This will generate a dataset of `Results` that we can store and begin analyzing:

```python theme={null}
results = survey.by(scenarios).by(agent).by(model).run()
```

## Analyzing results

EDSL comes with [built-in methods for analyzing results](/en/latest/results) in data tables, dataframes, SQL queries and other formats. We can print a list of all the components that can be accessed:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | agent.persona |
| 4 | answer.improvements |
| 5 | answer.sentiment |
| 6 | answer.themes |
| 7 | cache\_keys.improvements\_cache\_key |
| 8 | cache\_keys.sentiment\_cache\_key |
| 9 | cache\_keys.themes\_cache\_key |
| 10 | cache\_used.improvements\_cache\_used |
| 11 | cache\_used.sentiment\_cache\_used |
| 12 | cache\_used.themes\_cache\_used |
| 13 | comment.improvements\_comment |
| 14 | comment.sentiment\_comment |
| 15 | comment.themes\_comment |
| 16 | generated\_tokens.improvements\_generated\_tokens |
| 17 | generated\_tokens.sentiment\_generated\_tokens |
| 18 | generated\_tokens.themes\_generated\_tokens |
| 19 | iteration.iteration |
| 20 | model.inference\_service |
| 21 | model.maxOutputTokens |
| 22 | model.model |
| 23 | model.model\_index |
| 24 | model.stopSequences |
| 25 | model.temperature |
| 26 | model.topK |
| 27 | model.topP |
| 28 | prompt.improvements\_system\_prompt |
| 29 | prompt.improvements\_user\_prompt |
| 30 | prompt.sentiment\_system\_prompt |
| 31 | prompt.sentiment\_user\_prompt |
| 32 | prompt.themes\_system\_prompt |
| 33 | prompt.themes\_user\_prompt |
| 34 | question\_options.improvements\_question\_options |
| 35 | question\_options.sentiment\_question\_options |
| 36 | question\_options.themes\_question\_options |
| 37 | question\_text.improvements\_question\_text |
| 38 | question\_text.sentiment\_question\_text |
| 39 | question\_text.themes\_question\_text |
| 40 | question\_type.improvements\_question\_type |
| 41 | question\_type.sentiment\_question\_type |
| 42 | question\_type.themes\_question\_type |
| 43 | raw\_model\_response.improvements\_cost |
| 44 | raw\_model\_response.improvements\_input\_price\_per\_million\_tokens |
| 45 | raw\_model\_response.improvements\_input\_tokens |
| 46 | raw\_model\_response.improvements\_one\_usd\_buys |
| 47 | raw\_model\_response.improvements\_output\_price\_per\_million\_tokens |
| 48 | raw\_model\_response.improvements\_output\_tokens |
| 49 | raw\_model\_response.improvements\_raw\_model\_response |
| 50 | raw\_model\_response.sentiment\_cost |
| 51 | raw\_model\_response.sentiment\_input\_price\_per\_million\_tokens |
| 52 | raw\_model\_response.sentiment\_input\_tokens |
| 53 | raw\_model\_response.sentiment\_one\_usd\_buys |
| 54 | raw\_model\_response.sentiment\_output\_price\_per\_million\_tokens |
| 55 | raw\_model\_response.sentiment\_output\_tokens |
| 56 | raw\_model\_response.sentiment\_raw\_model\_response |
| 57 | raw\_model\_response.themes\_cost |
| 58 | raw\_model\_response.themes\_input\_price\_per\_million\_tokens |
| 59 | raw\_model\_response.themes\_input\_tokens |
| 60 | raw\_model\_response.themes\_one\_usd\_buys |
| 61 | raw\_model\_response.themes\_output\_price\_per\_million\_tokens |
| 62 | raw\_model\_response.themes\_output\_tokens |
| 63 | raw\_model\_response.themes\_raw\_model\_response |
| 64 | reasoning\_summary.improvements\_reasoning\_summary |
| 65 | reasoning\_summary.sentiment\_reasoning\_summary |
| 66 | reasoning\_summary.themes\_reasoning\_summary |
| 67 | scenario.evaluation |
| 68 | scenario.scenario\_index |

Here we select just the responses to the questions and display them in a table:

```python theme={null}
results.select("sentiment", "themes", "themes_generated_tokens", "improvements")
```

| | answer.sentiment | answer.themes | generated\_tokens.themes\_generated\_tokens | answer.improvements |
| :- | :- | :- | :- | :- |
| 0 | Positive | \['Engaging and informative course content', 'Effective explanation of complex concepts', 'Fast pace and challenging readings'] | \["Engaging and informative course content", "Effective explanation of complex concepts", "Fast pace and challenging readings"] The summary focuses on the student's main points: positive feedback on engagement and clarity, and constructive criticism regarding the pace. This is concise and addresses both strengths and weaknesses. | \['Adjust the pace of lectures and incorporate more in-class activities to reinforce concepts', 'Provide more structured guidance on readings, perhaps with suggested pacing or key concepts to focus on', 'Consider offering supplemental resources such as online tutorials or practice problems to support student learning'] |
| 1 | Negative | \['Student found the material dry and difficult', 'Lack of real-world connection and examples cited', 'Desire for more current event examples to increase engagement'] | \["Student found the material dry and difficult", "Lack of real-world connection and examples cited", "Desire for more current event examples to increase engagement"] The student's feedback clearly highlights three key areas: difficulty with the material's dryness, a need for better real-world application, and a specific suggestion for using current events to improve engagement. These points are concise and address the core concerns expressed. | \['Incorporate more real-world examples and case studies', 'Connect course material to current events', 'Develop more engaging teaching methods to enhance student interest'] |
| 2 | Positive | \['Enthusiastic and helpful professor', 'Engaging and interactive lectures', 'Practical and digestible assignments'] | \["Enthusiastic and helpful professor", "Engaging and interactive lectures", "Practical and digestible assignments"] The student highlighted three key strengths: the professor's positive attitude and availability, the lecture style, and the nature of the assignments. These are the most concise and impactful points for summarizing the feedback. | \['Consider incorporating more diverse teaching methods to cater to various learning styles', 'Explore opportunities to integrate current economic events into lectures for enhanced relevance', 'Evaluate the assessment methods to ensure a comprehensive evaluation of student understanding'] |
| 3 | Neutral | \['Appreciated analytical rigor', 'Desired more real-world application discussions', 'Felt disconnect between theory and practice'] | \["Appreciated analytical rigor", "Desired more real-world application discussions", "Felt disconnect between theory and practice"] The student's evaluation highlights both positive and negative aspects. The first point captures the student's appreciation for the mathematical strength of the course. The second and third points address the student's main concern: a perceived lack of connection between the theoretical concepts and their practical relevance. These three points concisely summarize the core message of the evaluation. | \['Incorporate real-world case studies into lectures', 'Facilitate more in-class discussions relating theoretical concepts to current events', 'Develop supplementary materials (e.g., articles, videos) illustrating practical applications of economic theories'] |
| 4 | Positive | \['Positive experience with group projects and real-world application', 'Desire for more detailed assignment feedback', 'Overall enjoyment of the course'] | \["Positive experience with group projects and real-world application", "Desire for more detailed assignment feedback", "Overall enjoyment of the course"] The student enjoyed the course and its practical application but wants more thorough feedback on assignments. These three points capture the essence of the evaluation concisely. | \['Enhance feedback detail on assignments', 'Explore alternative assessment methods to supplement group projects', 'Consider incorporating more diverse real-life case studies'] |
| 5 | Neutral | \['Well-organized course content', 'Monotonous and hard-to-follow lectures', 'Suggestions for visual aids and guest lecturers'] | \["Well-organized course content", "Monotonous and hard-to-follow lectures", "Suggestions for visual aids and guest lecturers"] The three points accurately reflect the student's main criticisms and suggestions, concisely summarizing the evaluation. | \['Incorporate more visual aids into lectures', 'Consider inviting guest lecturers from the industry', 'Review lecture delivery to enhance engagement and clarity'] |
| 6 | Positive | \['Enjoyed the balance of theory and case studies', 'Exams perceived as fair', 'Appreciated diverse perspectives on global economic policies'] | \["Enjoyed the balance of theory and case studies", "Exams perceived as fair", "Appreciated diverse perspectives on global economic policies"] The student highlights three distinct aspects of the course: the pedagogical approach (theory and case studies), assessment (fair exams), and content (global policy diversity). These are concise and represent the core positive feedback. | \['Increase quantitative problem sets', 'Incorporate more interactive learning activities', 'Explore current events more explicitly'] |
| 7 | Negative | \['Textbook complexity', 'Uneven lecture-textbook integration', 'Suggestion for simpler materials or enhanced lectures'] | \["Textbook complexity", "Uneven lecture-textbook integration", "Suggestion for simpler materials or enhanced lectures"] The student clearly identifies textbook complexity and jargon as major obstacles, highlighting a disconnect between lectures and the textbook's content. The suggestion for improvement directly addresses these issues. | \['Simplify textbook selection or supplement with clearer introductory materials', 'Incorporate more definitions and explanations of jargon in lectures', 'Designate specific lecture time for clarifying complex textbook concepts'] |
| 8 | Neutral | \["Professor's strong subject knowledge and passion were evident", 'Over-reliance on testing as assessment method', 'Need for diverse assignment types to cater to varied learning styles'] | \["Professor's strong subject knowledge and passion were evident", "Over-reliance on testing as assessment method", "Need for diverse assignment types to cater to varied learning styles"] The summary accurately reflects the student's main points without unnecessary detail, focusing on the core strengths and weaknesses highlighted in the evaluation. | \['Incorporate more diverse assessment methods (e.g., group projects, presentations, essays) to cater to varied learning styles.', 'Reduce over-reliance on high-stakes testing by incorporating lower-stakes assignments that allow for more frequent feedback and practice.', 'Explore alternative assessment formats such as problem-solving exercises or case studies to complement traditional testing.'] |
| 9 | Neutral | \["Positive assessment of the course's introductory nature", 'Critique of the overemphasis on theory', 'Suggestion for increased real-world application discussions'] | \["Positive assessment of the course's introductory nature", "Critique of the overemphasis on theory", "Suggestion for increased real-world application discussions"] The three points accurately reflect the student's main concerns and suggestions, concisely summarizing the evaluation. | \['Incorporate more case studies illustrating real-world applications of economic theories', 'Develop in-class activities or assignments that directly connect theoretical concepts to current events or policy debates', 'Increase opportunities for student-led discussions focusing on the practical implications of economic models'] |

We can do a quick tally of the sentiments:

```python theme={null}
results.select("sentiment").tally()
```

| | answer.sentiment | count |
| :- | :- | :- |
| 0 | Positive | 4 |
| 1 | Neutral | 4 |
| 2 | Negative | 2 |

We can also transform the results into a dataframe:

```python theme={null}
df = results.to_pandas()
df.head()
```

| | answer.themes | answer.improvements | answer.sentiment | scenario.evaluation | scenario.scenario\_index | agent.persona | agent.agent\_instruction | agent.agent\_name | agent.agent\_index | model.model\_index | ... | generated\_tokens.themes\_generated\_tokens | cache\_used.themes\_cache\_used | cache\_used.improvements\_cache\_used | cache\_used.sentiment\_cache\_used | cache\_keys.sentiment\_cache\_key | cache\_keys.improvements\_cache\_key | cache\_keys.themes\_cache\_key | reasoning\_summary.themes\_reasoning\_summary | reasoning\_summary.improvements\_reasoning\_summary | reasoning\_summary.sentiment\_reasoning\_summary |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | \['Engaging and informative course content', 'E... | \['Adjust the pace of lectures and incorporate ... | Positive | I found the course very engaging and informati... | 0 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | Agent\_0 | 0 | 0 | ... | \["Engaging and informative course content", "E... | True | True | True | e87ac1ab46560676c8556b25de720b5b | 045a6cd433b9f8a594e69c16650dc906 | 4cc05ab07126f12c0286e8d4017ee4e9 | NaN | NaN | NaN |
| 1 | \['Student found the material dry and difficult... | \['Incorporate more real-world examples and cas... | Negative | This class was a struggle for me. The material... | 1 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | Agent\_1 | 0 | 0 | ... | \["Student found the material dry and difficult... | True | True | True | fae9b208b15d5716d7cf557f161b1da1 | c9a6ea81091088296924b083c0f5277e | 7e35a3d2becc5f4fe912e97490c496ce | NaN | NaN | NaN |
| 2 | \['Enthusiastic and helpful professor', 'Engagi... | \['Consider incorporating more diverse teaching... | Positive | Excellent introductory course! The professor w\... | 2 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | Agent\_2 | 0 | 0 | ... | \["Enthusiastic and helpful professor", "Engagi... | True | True | True | 1ac91057e39cf9900951ee30cc7928fe | 22262c2d5bb750a06b3707026b3bdee3 | 13432bad75496e0aebe8a06ba32a4550 | NaN | NaN | NaN |
| 3 | \['Appreciated analytical rigor', 'Desired more... | \['Incorporate real-world case studies into lec... | Neutral | As someone with a strong background in math, I... | 3 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | Agent\_3 | 0 | 0 | ... | \["Appreciated analytical rigor", "Desired more... | True | True | True | 265dbbf39a6f044e3d9ac8879834a0a5 | c093f46806b11c1a6f3bb185458ca959 | 45750924adf6ee05ff3fb690a17b0f49 | NaN | NaN | NaN |
| 4 | \['Positive experience with group projects and ... | \['Enhance feedback detail on assignments', 'Ex... | Positive | I enjoyed the course, especially the group pro... | 4 | You are a professor reviewing student evaluati... | Be very specific and constructive in providing... | Agent\_4 | 0 | 0 | ... | \["Positive experience with group projects and ... | True | True | True | 957ab53fb4cae05d4e89dc02cda69b43 | 5dabf9b39b444bd678989962dbe91e65 | c5d6a3a504fbb6afb1c48f60981619a2 | NaN | NaN | NaN |

5 rows × 69 columns

We can also use pandas methods by first converting:

```python theme={null}
df_sentiment = results.to_pandas()["answer.sentiment"]
df_sentiment.value_counts()
```

```python theme={null}
answer.sentiment
Positive    4
Neutral     4
Negative    2
Name: count, dtype: int64
```

## Use responses to construct new questions

We can use the responses to our initial questions to construct more questions about the texts. For example, we can prompt a model to condense the individual lists of themes and areas for improvement into short lists, and then use the new lists to quantify the topics across the set of evaluations.

Here we take the lists of themes in each evaluation, flatten them into a (duplicative) list, and then create a new question prompting a model to condense it for us:

```python theme={null}
results.select("themes", "themes_generated_tokens")
```

| | answer.themes | generated\_tokens.themes\_generated\_tokens |
| :- | :- | :- |
| 0 | \['Engaging and informative course content', 'Effective explanation of complex concepts', 'Fast pace and challenging readings'] | \["Engaging and informative course content", "Effective explanation of complex concepts", "Fast pace and challenging readings"] The summary focuses on the student's main points: positive feedback on engagement and clarity, and constructive criticism regarding the pace. This is concise and addresses both strengths and weaknesses. |
| 1 | \['Student found the material dry and difficult', 'Lack of real-world connection and examples cited', 'Desire for more current event examples to increase engagement'] | \["Student found the material dry and difficult", "Lack of real-world connection and examples cited", "Desire for more current event examples to increase engagement"] The student's feedback clearly highlights three key areas: difficulty with the material's dryness, a need for better real-world application, and a specific suggestion for using current events to improve engagement. These points are concise and address the core concerns expressed. |
| 2 | \['Enthusiastic and helpful professor', 'Engaging and interactive lectures', 'Practical and digestible assignments'] | \["Enthusiastic and helpful professor", "Engaging and interactive lectures", "Practical and digestible assignments"] The student highlighted three key strengths: the professor's positive attitude and availability, the lecture style, and the nature of the assignments. These are the most concise and impactful points for summarizing the feedback. |
| 3 | \['Appreciated analytical rigor', 'Desired more real-world application discussions', 'Felt disconnect between theory and practice'] | \["Appreciated analytical rigor", "Desired more real-world application discussions", "Felt disconnect between theory and practice"] The student's evaluation highlights both positive and negative aspects. The first point captures the student's appreciation for the mathematical strength of the course. The second and third points address the student's main concern: a perceived lack of connection between the theoretical concepts and their practical relevance. These three points concisely summarize the core message of the evaluation. |
| 4 | \['Positive experience with group projects and real-world application', 'Desire for more detailed assignment feedback', 'Overall enjoyment of the course'] | \["Positive experience with group projects and real-world application", "Desire for more detailed assignment feedback", "Overall enjoyment of the course"] The student enjoyed the course and its practical application but wants more thorough feedback on assignments. These three points capture the essence of the evaluation concisely. |
| 5 | \['Well-organized course content', 'Monotonous and hard-to-follow lectures', 'Suggestions for visual aids and guest lecturers'] | \["Well-organized course content", "Monotonous and hard-to-follow lectures", "Suggestions for visual aids and guest lecturers"] The three points accurately reflect the student's main criticisms and suggestions, concisely summarizing the evaluation. |
| 6 | \['Enjoyed the balance of theory and case studies', 'Exams perceived as fair', 'Appreciated diverse perspectives on global economic policies'] | \["Enjoyed the balance of theory and case studies", "Exams perceived as fair", "Appreciated diverse perspectives on global economic policies"] The student highlights three distinct aspects of the course: the pedagogical approach (theory and case studies), assessment (fair exams), and content (global policy diversity). These are concise and represent the core positive feedback. |
| 7 | \['Textbook complexity', 'Uneven lecture-textbook integration', 'Suggestion for simpler materials or enhanced lectures'] | \["Textbook complexity", "Uneven lecture-textbook integration", "Suggestion for simpler materials or enhanced lectures"] The student clearly identifies textbook complexity and jargon as major obstacles, highlighting a disconnect between lectures and the textbook's content. The suggestion for improvement directly addresses these issues. |
| 8 | \["Professor's strong subject knowledge and passion were evident", 'Over-reliance on testing as assessment method', 'Need for diverse assignment types to cater to varied learning styles'] | \["Professor's strong subject knowledge and passion were evident", "Over-reliance on testing as assessment method", "Need for diverse assignment types to cater to varied learning styles"] The summary accurately reflects the student's main points without unnecessary detail, focusing on the core strengths and weaknesses highlighted in the evaluation. |
| 9 | \["Positive assessment of the course's introductory nature", 'Critique of the overemphasis on theory', 'Suggestion for increased real-world application discussions'] | \["Positive assessment of the course's introductory nature", "Critique of the overemphasis on theory", "Suggestion for increased real-world application discussions"] The three points accurately reflect the student's main concerns and suggestions, concisely summarizing the evaluation. |

```python theme={null}
themes = results.select("themes").to_list(flatten = True)
```

Next we construct a question to condense the list into a new list:

```python theme={null}
from edsl import Scenario

scenario = Scenario({"themes":themes})
```

```python theme={null}
q_condensed_themes = QuestionList(
    question_name="condensed_themes",
    question_text="""Combine the following list of themes extracted from the evaluations
    into a consolidated, non-redundant list: `{{ scenario.themes }}`""",
    max_list_items=10,
)
```

Now we run the question and select the new list. Note that we can choose whether we want to use the agent for this question by not adding it to the question when we run it:

```python theme={null}
condensed_themes = q_condensed_themes.by(scenario).run().select("condensed_themes").to_list()[0]
condensed_themes
```

```python theme={null}
['Engaging and informative course content',
 'Effective explanation of complex concepts',
 'Desire for more real-world applications and examples',
 'Enthusiastic and knowledgeable professor',
 'Well-organized and interactive lectures',
 'Practical and digestible assignments',
 'Need for diverse assessment methods',
 'Appreciated analytical rigor and diverse perspectives',
 'Suggestions for improved lecture materials',
 'Positive experience with group projects and case studies']
```

Now we can create a question to identify all the themes in the list that appear in each evaluation (our new list becomes the list of answer options):

```python theme={null}
from edsl import QuestionCheckBox

q_themes_list = QuestionCheckBox(
    question_name="themes_list",
    question_text="Select all of the themes that are mentioned in this evaluation: `{{ scenario.evaluation }}`",
    question_options=condensed_themes,
)
```

Here we run the question and show a table listing all the themes for each evaluation in the results:

```python theme={null}
themes_lists = q_themes_list.by(scenarios).by(agent).run()
themes_lists.select("evaluation", "themes_list")
```

| | scenario.evaluation | answer.themes\_list |
| :- | :- | :- |
| 0 | I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings. | \['Engaging and informative course content', 'Effective explanation of complex concepts'] |
| 1 | This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest. | \['Desire for more real-world applications and examples', 'Suggestions for improved lecture materials'] |
| 2 | Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging. | \['Engaging and informative course content', 'Enthusiastic and knowledgeable professor', 'Well-organized and interactive lectures', 'Practical and digestible assignments'] |
| 3 | As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times. | \['Desire for more real-world applications and examples', 'Appreciated analytical rigor and diverse perspectives'] |
| 4 | I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes. | \['Positive experience with group projects and case studies', 'Desire for more real-world applications and examples', 'Suggestions for improved lecture materials'] |
| 5 | The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions. | \['Suggestions for improved lecture materials', 'Desire for more real-world applications and examples'] |
| 6 | This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies. | \['Engaging and informative course content', 'Appreciated analytical rigor and diverse perspectives', 'Positive experience with group projects and case studies'] |
| 7 | I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics. | \['Suggestions for improved lecture materials'] |
| 8 | The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles. | \['Enthusiastic and knowledgeable professor', 'Need for diverse assessment methods'] |
| 9 | This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material. | \['Desire for more real-world applications and examples'] |

```python theme={null}
wide_evaluation_themes = themes_lists.select("evaluation", "themes_list").to_scenario_list().expand("themes_list").rename({"themes_list": "theme"})
wide_evaluation_themes
```

[ScenarioList](/en/latest/scenarios) scenarios: 22; keys: \['evaluation', 'theme'];

| | evaluation | theme |
| :- | :- | :- |
| 0 | I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings. | Engaging and informative course content |
| 1 | I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings. | Effective explanation of complex concepts |
| 2 | This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest. | Desire for more real-world applications and examples |
| 3 | This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest. | Suggestions for improved lecture materials |
| 4 | Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging. | Engaging and informative course content |
| 5 | Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging. | Enthusiastic and knowledgeable professor |
| 6 | Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging. | Well-organized and interactive lectures |
| 7 | Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging. | Practical and digestible assignments |
| 8 | As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times. | Desire for more real-world applications and examples |
| 9 | As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times. | Appreciated analytical rigor and diverse perspectives |
| 10 | I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes. | Positive experience with group projects and case studies |
| 11 | I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes. | Desire for more real-world applications and examples |
| 12 | I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes. | Suggestions for improved lecture materials |
| 13 | The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions. | Suggestions for improved lecture materials |
| 14 | The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions. | Desire for more real-world applications and examples |
| 15 | This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies. | Engaging and informative course content |
| 16 | This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies. | Appreciated analytical rigor and diverse perspectives |
| 17 | This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies. | Positive experience with group projects and case studies |
| 18 | I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics. | Suggestions for improved lecture materials |
| 19 | The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles. | Enthusiastic and knowledgeable professor |
| 20 | The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles. | Need for diverse assessment methods |
| 21 | This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material. | Desire for more real-world applications and examples |

```python theme={null}
wide_evaluation_themes.tally("theme")
```

| | theme | count |
| :- | :- | :- |
| 0 | Desire for more real-world applications and examples | 5 |
| 1 | Suggestions for improved lecture materials | 4 |
| 2 | Engaging and informative course content | 3 |
| 3 | Enthusiastic and knowledgeable professor | 2 |
| 4 | Appreciated analytical rigor and diverse perspectives | 2 |
| 5 | Positive experience with group projects and case studies | 2 |
| 6 | Effective explanation of complex concepts | 1 |
| 7 | Well-organized and interactive lectures | 1 |
| 8 | Practical and digestible assignments | 1 |
| 9 | Need for diverse assessment methods | 1 |

We can do the same thing with the areas of improvement:

```python theme={null}
improvements = results.select("improvements").to_list(flatten=True)
improvements
```

```python expandable theme={null}
['Adjust the pace of lectures and incorporate more in-class activities to reinforce concepts',
 'Provide more structured guidance on readings, perhaps with suggested pacing or key concepts to focus on',
 'Consider offering supplemental resources such as online tutorials or practice problems to support student learning',
 'Incorporate more real-world examples and case studies',
 'Connect course material to current events',
 'Develop more engaging teaching methods to enhance student interest',
 'Consider incorporating more diverse teaching methods to cater to various learning styles',
 'Explore opportunities to integrate current economic events into lectures for enhanced relevance',
 'Evaluate the assessment methods to ensure a comprehensive evaluation of student understanding',
 'Incorporate real-world case studies into lectures',
 'Facilitate more in-class discussions relating theoretical concepts to current events',
 'Develop supplementary materials (e.g., articles, videos) illustrating practical applications of economic theories',
 'Enhance feedback detail on assignments',
 'Explore alternative assessment methods to supplement group projects',
 'Consider incorporating more diverse real-life case studies',
 'Incorporate more visual aids into lectures',
 'Consider inviting guest lecturers from the industry',
 'Review lecture delivery to enhance engagement and clarity',
 'Increase quantitative problem sets',
 'Incorporate more interactive learning activities',
 'Explore current events more explicitly',
 'Simplify textbook selection or supplement with clearer introductory materials',
 'Incorporate more definitions and explanations of jargon in lectures',
 'Designate specific lecture time for clarifying complex textbook concepts',
 'Incorporate more diverse assessment methods (e.g., group projects, presentations, essays) to cater to varied learning styles.',
 'Reduce over-reliance on high-stakes testing by incorporating lower-stakes assignments that allow for more frequent feedback and practice.',
 'Explore alternative assessment formats such as problem-solving exercises or case studies to complement traditional testing.',
 'Incorporate more case studies illustrating real-world applications of economic theories',
 'Develop in-class activities or assignments that directly connect theoretical concepts to current events or policy debates',
 'Increase opportunities for student-led discussions focusing on the practical implications of economic models']
```

```python theme={null}
scenario = Scenario({"improvements":improvements})

q_condensed_improvements = QuestionList(
    question_name="condensed_improvements",
    question_text="""Combine the following list of areas for improvement from the evaluations
    into a consolidated, non-redundant list: `{{ scenario.improvements }}`""",
    max_list_items=10,
)
```

```python theme={null}
condensed_improvements = q_condensed_improvements.by(scenario).run().select("condensed_improvements").to_list()[0]
condensed_improvements
```

```python theme={null}
['Adjust lecture pace and include more in-class activities to reinforce concepts',
 'Provide structured guidance and supplemental resources for readings',
 'Incorporate diverse teaching methods and real-world examples',
 'Connect course material to current events and economic events',
 'Evaluate and diversify assessment methods',
 'Enhance feedback on assignments',
 'Include visual aids and guest lecturers',
 'Simplify textbook selection and clarify complex concepts',
 'Increase opportunities for student-led discussions',
 'Reduce reliance on high-stakes testing with more frequent feedback opportunities']
```

```python theme={null}
q_improvements_list = QuestionCheckBox(
    question_name="improvements_list",
    question_text="Select all of the improvements that are mentioned in this evaluation: `{{ scenario.evaluation }}`",
    question_options=condensed_improvements,
)
```

```python theme={null}
improvements_lists = q_improvements_list.by(scenarios).by(agent).run()
improvements_lists.select("evaluation", "improvements_list")
```

| | scenario.evaluation | answer.improvements\_list |
| :- | :- | :- |
| 0 | I found the course very engaging and informative. The professor did an excellent job breaking down complex concepts, making them accessible to those of us new to economics. However, the pace was a bit fast, and I sometimes struggled to keep up with the weekly readings. | \['Adjust lecture pace and include more in-class activities to reinforce concepts', 'Provide structured guidance and supplemental resources for readings'] |
| 1 | This class was a struggle for me. The material felt dry and difficult to connect with real-world applications, which I think could have made it more interesting. More examples from current events would definitely have helped spark my interest. | \['Incorporate diverse teaching methods and real-world examples', 'Connect course material to current events and economic events'] |
| 2 | Excellent introductory course! The professor was enthusiastic and always willing to offer extra help during office hours. The interactive lectures and the practical assignments made the theory much more digestible and engaging. | \[] |
| 3 | As someone with a strong background in math, I appreciated the analytical rigor of this course. However, I wish there had been more discussions that connected the theories we learned to everyday economic issues. It felt a bit isolated from practical realities at times. | \['Incorporate diverse teaching methods and real-world examples', 'Connect course material to current events and economic events', 'Increase opportunities for student-led discussions'] |
| 4 | I enjoyed the course, especially the group projects, which were both challenging and rewarding. It was great to apply economic concepts to solve real-life problems. I did feel, however, that the feedback on assignments could be more detailed to help us understand our mistakes. | \['Enhance feedback on assignments'] |
| 5 | The course content was well-organized, but the lectures were somewhat monotonous and hard to follow. I would suggest incorporating more visual aids and maybe some guest lectures from industry professionals to liven up the sessions. | \['Include visual aids and guest lecturers'] |
| 6 | This was my favorite class this semester! The mix of theory and case studies was perfect, and the exams were fair. I also really appreciated the diversity of perspectives we explored in class, especially in terms of global economic policies. | \[] |
| 7 | I found the textbook to be overly complex for an introductory course. It often used jargon that hadn't been explained in lectures, which was confusing. Simpler reading materials or more explanatory lectures would make a big difference for newcomers to economics. | \['Simplify textbook selection and clarify complex concepts', 'Provide structured guidance and supplemental resources for readings'] |
| 8 | The professor was knowledgeable and clearly passionate about economics, but I felt the course relied too heavily on tests rather than more creative forms of assessment. More varied assignments would make the course more accessible to students with different learning styles. | \['Evaluate and diversify assessment methods'] |
| 9 | This class was a solid introduction to economics, though it leaned heavily on theoretical aspects. I would have liked more opportunities to discuss the real-world implications of economic theories, which I believe would enhance understanding and retention of the material. | \['Incorporate diverse teaching methods and real-world examples', 'Connect course material to current events and economic events', 'Increase opportunities for student-led discussions'] |

```python theme={null}
wide_themes = (
    improvements_lists
    .select("evaluation", "improvements_list")
    .to_scenario_list()
    .expand("improvements_list")
    .rename({"improvements_list": "theme"})
)
```

```python theme={null}
wide_themes.tally("theme")
```

| | theme | count |
| :- | :- | :- |
| 0 | Incorporate diverse teaching methods and real-world examples | 3 |
| 1 | Connect course material to current events and economic events | 3 |
| 2 | Provide structured guidance and supplemental resources for readings | 2 |
| 3 | Increase opportunities for student-led discussions | 2 |
| 4 | Adjust lecture pace and include more in-class activities to reinforce concepts | 1 |
| 5 | Enhance feedback on assignments | 1 |
| 6 | Include visual aids and guest lecturers | 1 |
| 7 | Simplify textbook selection and clarify complex concepts | 1 |
| 8 | Evaluate and diversify assessment methods | 1 |

## Other examples

Please check out the [EDSL Docs](/en/latest/index) for examples of other methods and templates for use cases!

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

We can post any EDSL object to Expected Parrot by calling the `push()` method on it, including this notebook:

```python theme={null}
# from edsl import Notebook

# nb = Notebook("analyze_evaluations.ipynb")

# nb.push(
#     description = "Example code for analyzing course evaluations",
#     alias = "course-evaluations-notebook",
#     visibility = "public"
# )
```

To upate an object at Expected Parrot:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "analyze_evaluations.ipynb") # resave

nb.patch("https://www.expectedparrot.com/content/RobinHorton/course-evaluations-notebook", value = nb)
```


# Answering instructions
Source: https://docs.expectedparrot.com/en/latest/notebooks/answering_instructions_example

This notebook demonstrates how you can modify default instructions for question types and answer comments.

## Default question instructions

Each EDSL question type includes default instructions to the model about how to answer the question. We can view these instructions by inspecting the user prompt for a question that has been created (the other type of prompt–systen prompt–is for agent instructions).

For example, here we see that the default instruction for multiple choice questions is:

**“Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line.”**

This text is automatically appended to the question text:

```python theme={null}
from edsl import QuestionMultipleChoice, Survey, Model

q = QuestionMultipleChoice(
    question_name = "primary_color",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"]
)

survey = Survey([q])
survey.show_prompts()
```

| | user\_prompt | system\_prompt | interview\_index | question\_name | scenario\_index | agent\_index | model | estimated\_cost | cache\_keys |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | nan | 0 | primary\_color | 0 | 0 | gpt-4o | 0.000678 | \['20e75009c72f3e88c490c58bf13d6a72'] |

We can isolate the user prompt:

```python theme={null}
survey.by(Model()).prompts().select("user_prompt")
```

| | user\_prompt |
| :- | :- |
| 0 | What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. |

We can compare this with default instructions for other question types:

```python theme={null}
from edsl import QuestionCheckBox, Survey, Model

q = QuestionCheckBox(
    question_name = "primary_colors",
    question_text = "Which colors are 'primary'?",
    question_options = ["Red", "Orange", "Yellow", "Green", "Blue", "Purple"]
)

survey = Survey([q])
survey.by(Model()).prompts().select("user_prompt")
```

| | user\_prompt |
| :- | :- |
| 0 | Which colors are 'primary'? Red Orange Yellow Green Blue Purple Please respond only with a comma-separated list of the options that apply, with square brackets. E.g., \['Good', 'Bad', 'Ugly'] After the answer, you can put a comment explaining your choice on the next line. |

```python theme={null}
from edsl import QuestionRank, Survey, Model

q = QuestionRank(
    question_name = "primary_colors_rank",
    question_text = "Rank the primary colors in terms of popularity.",
    question_options = ["Red", "Yellow", "Blue"]
)

survey = Survey([q])
survey.by(Model()).prompts().select("user_prompt")
```

| | user\_prompt |
| :- | :- |
| 0 | Rank the primary colors in terms of popularity. The options are 0: Red 1: Yellow 2: Blue You have to include 3 options in your answer. Please respond only with a comma-separated list of the code of the raked options, with square brackets. E.g., \[0, 1, 3] After the answer, you can put a comment explaining your choice on the next line. |

```python theme={null}
from edsl import QuestionLinearScale, Survey, Model

q = QuestionLinearScale(
    question_name = "primary_color_scale",
    question_text = "Most people know what the primary colors are.",
    question_options = [1,2,3,4,5],
    option_labels = {
        1:"This statement is completely inaccurate",
        5:"This statement is completely accurate."
    }
)

survey = Survey([q])
survey.by(Model()).prompts().select("user_prompt")
```

| | user\_prompt |
| :- | :- |
| 0 | Most people know what the primary colors are. 1 : This statement is completely inaccurate 2 : 3 : 4 : 5 : This statement is completely accurate. Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |

## Formatting answers & comments

We can see that each default instruction includes directions on (1) formatting the answer and (2) providing a comment about the answer. When a question is administered, the contents of the comment that is returned are automatically stored in a separate field of the results. We can see this when we run a question and inspect the columns of the results that have been created. Here we run the multiple choice question created above:

```python theme={null}
from edsl import QuestionMultipleChoice, Survey, Model

q = QuestionMultipleChoice(
    question_name = "primary_color",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"]
)

r = q.run() # default model will be used
```

We can see that the results include a `comment` field:

```python theme={null}
r.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.primary\_color |
| 4 | cache\_keys.primary\_color\_cache\_key |
| 5 | cache\_used.primary\_color\_cache\_used |
| 6 | comment.primary\_color\_comment |
| 7 | generated\_tokens.primary\_color\_generated\_tokens |
| 8 | iteration.iteration |
| 9 | model.frequency\_penalty |
| 10 | model.inference\_service |
| 11 | model.logprobs |
| 12 | model.max\_tokens |
| 13 | model.model |
| 14 | model.model\_index |
| 15 | model.presence\_penalty |
| 16 | model.temperature |
| 17 | model.top\_logprobs |
| 18 | model.top\_p |
| 19 | prompt.primary\_color\_system\_prompt |
| 20 | prompt.primary\_color\_user\_prompt |
| 21 | question\_options.primary\_color\_question\_options |
| 22 | question\_text.primary\_color\_question\_text |
| 23 | question\_type.primary\_color\_question\_type |
| 24 | raw\_model\_response.primary\_color\_cost |
| 25 | raw\_model\_response.primary\_color\_input\_price\_per\_million\_tokens |
| 26 | raw\_model\_response.primary\_color\_input\_tokens |
| 27 | raw\_model\_response.primary\_color\_one\_usd\_buys |
| 28 | raw\_model\_response.primary\_color\_output\_price\_per\_million\_tokens |
| 29 | raw\_model\_response.primary\_color\_output\_tokens |
| 30 | raw\_model\_response.primary\_color\_raw\_model\_response |
| 31 | reasoning\_summary.primary\_color\_reasoning\_summary |
| 32 | scenario.scenario\_index |

We can display it with any other fields:

```python theme={null}
r.select("model", "primary_color", "primary_color_comment")
```

| | model.model | answer.primary\_color | comment.primary\_color\_comment |
| :- | :- | :- | :- |
| 0 | gpt-4o | Red | Red is often considered the most common primary color due to its prominence in nature, its use in art and design, and its psychological impact. |

## Turning off comments

If desired, we can omit the instruction to provide a comment by passing a parameter `include_comment=False` to the question constructor. This may be desired if comments are not necessary or to save tokens. Here we inspect how the question prompt has been modified and verify that the comment field in the results is blank:

```python theme={null}
from edsl import QuestionMultipleChoice, Survey, Model

q = QuestionMultipleChoice(
    question_name = "primary_color",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"],
    include_comment = False # optional
)

q.by(Model()).prompts().select("user_prompt")
```

| | user\_prompt |
| :- | :- |
| 0 | What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. |

```python theme={null}
r = q.run() # default model will be used

r.select("model", "primary_color", "primary_color_comment")
```

| | model.model | answer.primary\_color | comment.primary\_color\_comment |
| :- | :- | :- | :- |
| 0 | gpt-4o | Red | nan |

## Modifying comments

We can also modify the default instruction if we want to use the comment field in a different way. This can be done by passing an optional parameter `answering_instruction` to the question constructor. For example, here we pass an instruction that preserves the directions about the format of the answer to a multiple choice question (”*Respond only with a string corresponding to one of the options.*”) but replace the comments part of the instruction with a new instruction for the model to instead note it’s second choice answer. We include the original question in the survey as well for ease of comparison:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, Survey, Model

q1 = QuestionMultipleChoice(
    question_name = "primary_color_v1",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"]
)

q2 = QuestionMultipleChoice(
    question_name = "primary_color_v2",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"],
    answering_instructions = """
    Respond only with a string corresponding to one of the options.
    After the answer, please provide your second choice on the next line.
    """
)

survey = Survey([q1, q2])

survey.by(Model()).prompts().select("user_prompt")
```

| | user\_prompt |
| :- | :- |
| 0 | What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 1 | What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, please provide your second choice on the next line. |

```python theme={null}
r = survey.run() # default model will be used
```

```python theme={null}
r.select("model", "primary_color_v1", "primary_color_v1_comment", "primary_color_v2", "primary_color_v2_comment")
```

| | model.model | answer.primary\_color\_v1 | comment.primary\_color\_v1\_comment | answer.primary\_color\_v2 | comment.primary\_color\_v2\_comment |
| :- | :- | :- | :- | :- | :- |
| 0 | gpt-4o | Red | Red is often considered the most common primary color due to its prominence in nature, its use in art and design, and its psychological impact. | Red | Yellow |

## Further reading

Please see the [questions](/en/latest/questions) page of the documentation for more examples and details on all of the required and optional parameters for question types!

Here we post this notebook to Expected Parrot for reference:

```python theme={null}
from edsl import Notebook
```

```python theme={null}
nb = Notebook(path = "answering_instructions_example.ipynb")

nb.push(
    description = "Example answering instructions",
    alias = "answering-instructions-notebook",
    visibility = "public"
)
```


# Batching results
Source: https://docs.expectedparrot.com/en/latest/notebooks/batching_results

This notebook provides sample [EDSL](/en/latest/index) code for combining survey [results](/en/latest/results) into a single `Results` object. This can be useful when you are running a survey with batches of [scenarios](/en/latest/scenarios), such as when completing a large-scale [data labeling](/en/latest/notebooks/data_labeling_example) task with chunks of data as inputs for the questions.

## Technical setup

Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Creating questions

We start by creating a survey of questions. [EDSL comes with many question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from the model. We can use a `{{ placeholder }}` for data or content that we want to add to questions later:

```python theme={null}
from edsl import QuestionFreeText, QuestionNumerical
```

```python theme={null}
q_name = QuestionFreeText(
    question_name="name",
    question_text="What's a good name for this character: {{ scenario.character }}",
)

q_year = QuestionNumerical(
    question_name="year",
    question_text="""What year in history would have been an especially interesting time to talk
    to this character: {{ scenario.character }}""",
)

q_book = QuestionFreeText(
    question_name="book",
    question_text="If this character wrote a best-seller, what would it be called: {{ scenario.character }}",
)
```

## Constructing a survey

We pass a list of questions to a `Survey` object in order to administer them together, and add any desire logic or rules for how the questions should be presented (e.g., skip/stop rules or “memories” of other questions). [Learn more about constructing surveys](/en/latest/surveys).

```python theme={null}
from edsl import Survey
```

```python theme={null}
survey = Survey(questions = [q_name, q_year, q_book])
```

## Adding context to questions

Next we create `Scenario` objects representing the data or content to be added to the questions. [EDSL has a variety of methods for generating scenarios from different data sources](/en/latest/scenarios) (PDFs, CSVs, docs, images, tables, dicts, etc.). Here we import a list of values to use:

```python expandable theme={null}
characters = [
    "A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology.",
    "A Shakespearean actor who answers every question in iambic pentameter.",
    "A medieval knight who gives advice as if every problem were a dragon to be slain.",
    "A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm.",
    "A surfer dude who relates every topic to the ocean or surfing.",
    "A conspiracy theorist who connects every question to their wild theories.",
    "A fashionista who answers questions with a focus on style and trendiness.",
    "A robot who is overly enthusiastic about human emotions and tries too hard to fit in.",
    "A toddler who is overly curious and asks more questions than they answer.",
    "A fitness guru who turns every answer into a workout metaphor.",
    "A foodie who relates every question to cooking and food experiences.",
    "A detective from a noir film who answers in a gritty, mysterious manner.",
    "A hippie from the 60s who gives peace and love-centric advice.",
    "A gamer who references video games and uses gamer lingo.",
    "A superhero who answers questions as if they are saving the day.",
    "A poet who responds in rhyming couplets.",
    "A comedian who tries to turn every answer into a joke or punchline.",
    "A DJ who relates everything to music and beats.",
    "A film critic who answers questions as if they are reviewing a movie.",
    "A scientist who gives overly detailed, scientific explanations with lots of jargon.",
]
```

```python theme={null}
from edsl import ScenarioList
```

```python theme={null}
scenarios = ScenarioList.from_list("character", characters)
```

We can inspect the scenarios that have been created:

```python theme={null}
scenarios
```

```python theme={null}
ScenarioList scenarios: 20; keys: ['character'];
```

| | character |
| :- | :- |
| 0 | A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology. |
| 1 | A Shakespearean actor who answers every question in iambic pentameter. |
| 2 | A medieval knight who gives advice as if every problem were a dragon to be slain. |
| 3 | A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm. |
| 4 | A surfer dude who relates every topic to the ocean or surfing. |
| 5 | A conspiracy theorist who connects every question to their wild theories. |
| 6 | A fashionista who answers questions with a focus on style and trendiness. |
| 7 | A robot who is overly enthusiastic about human emotions and tries too hard to fit in. |
| 8 | A toddler who is overly curious and asks more questions than they answer. |
| 9 | A fitness guru who turns every answer into a workout metaphor. |
| 10 | A foodie who relates every question to cooking and food experiences. |
| 11 | A detective from a noir film who answers in a gritty, mysterious manner. |
| 12 | A hippie from the 60s who gives peace and love-centric advice. |
| 13 | A gamer who references video games and uses gamer lingo. |
| 14 | A superhero who answers questions as if they are saving the day. |
| 15 | A poet who responds in rhyming couplets. |
| 16 | A comedian who tries to turn every answer into a joke or punchline. |
| 17 | A DJ who relates everything to music and beats. |
| 18 | A film critic who answers questions as if they are reviewing a movie. |
| 19 | A scientist who gives overly detailed, scientific explanations with lots of jargon. |

## Running a survey

We run the survey by adding any agent personas that we have created to answer the questions (in this example, none) and specifying language models to generate the responses. If no model is specified the default model (currently, gpt-5.2) is used. Here we specify it for demonstration purposes, and then call the `run()` method to administer the survey. This generates a dataset of `Results` that we can access with [built-in methods for analysis](/en/latest/results).

```python theme={null}
from edsl import Model

model = Model("gpt-4o")
```

```python theme={null}
results = survey.by(scenarios).by(model).run()
```

## Batching scenarios

If for any reason we want to batch the scenarios when running the survey and combine the results, this can be done in the following manner:

```python theme={null}
def chunked_iterable(iterable, size):
    for i in range(0, len(iterable), size):
        yield iterable[i : i + size]

results = None

for batch in chunked_iterable(scenarios, 5):
    new_results = survey.by(batch).by(model).run()
    if results is None:
        results = new_results
    else:
        results = results + new_results
```

To see a list of the components of the results:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.book |
| 4 | answer.name |
| 5 | answer.year |
| 6 | cache\_keys.book\_cache\_key |
| 7 | cache\_keys.name\_cache\_key |
| 8 | cache\_keys.year\_cache\_key |
| 9 | cache\_used.book\_cache\_used |
| 10 | cache\_used.name\_cache\_used |
| 11 | cache\_used.year\_cache\_used |
| 12 | comment.book\_comment |
| 13 | comment.name\_comment |
| 14 | comment.year\_comment |
| 15 | generated\_tokens.book\_generated\_tokens |
| 16 | generated\_tokens.name\_generated\_tokens |
| 17 | generated\_tokens.year\_generated\_tokens |
| 18 | iteration.iteration |
| 19 | model.frequency\_penalty |
| 20 | model.inference\_service |
| 21 | model.logprobs |
| 22 | model.max\_tokens |
| 23 | model.model |
| 24 | model.model\_index |
| 25 | model.presence\_penalty |
| 26 | model.temperature |
| 27 | model.top\_logprobs |
| 28 | model.top\_p |
| 29 | prompt.book\_system\_prompt |
| 30 | prompt.book\_user\_prompt |
| 31 | prompt.name\_system\_prompt |
| 32 | prompt.name\_user\_prompt |
| 33 | prompt.year\_system\_prompt |
| 34 | prompt.year\_user\_prompt |
| 35 | question\_options.book\_question\_options |
| 36 | question\_options.name\_question\_options |
| 37 | question\_options.year\_question\_options |
| 38 | question\_text.book\_question\_text |
| 39 | question\_text.name\_question\_text |
| 40 | question\_text.year\_question\_text |
| 41 | question\_type.book\_question\_type |
| 42 | question\_type.name\_question\_type |
| 43 | question\_type.year\_question\_type |
| 44 | raw\_model\_response.book\_cost |
| 45 | raw\_model\_response.book\_one\_usd\_buys |
| 46 | raw\_model\_response.book\_raw\_model\_response |
| 47 | raw\_model\_response.name\_cost |
| 48 | raw\_model\_response.name\_one\_usd\_buys |
| 49 | raw\_model\_response.name\_raw\_model\_response |
| 50 | raw\_model\_response.year\_cost |
| 51 | raw\_model\_response.year\_one\_usd\_buys |
| 52 | raw\_model\_response.year\_raw\_model\_response |
| 53 | scenario.character |
| 54 | scenario.scenario\_index |

We can inspect them:

```python theme={null}
results.select("model", "character", "name", "year", "book")
```

| | model.model | scenario.character | answer.name | answer.year | answer.book |
| :- | :- | :- | :- | :- | :- |
| 0 | gpt-4o | A pirate who speaks in 'arrs' and 'mateys' but has an encyclopedic knowledge of modern technology. | Captain Techbeard | 1717 | "Tech Treasures: Navigating the Digital Seas with Captain Code" |
| 1 | gpt-4o | A Shakespearean actor who answers every question in iambic pentameter. | A fitting name for a Shakespearean actor who answers every question in iambic pentameter could be "Percival Quillington." This name combines a classic, theatrical first name with a surname that evokes the image of a quill, symbolizing both writing and the poetic nature of his speech. | 1599 | "Verses of the Thespian: Life in Iambic Cadence" |
| 2 | gpt-4o | A medieval knight who gives advice as if every problem were a dragon to be slain. | A fitting name for this character could be "Sir Draconis Counselblade." This name captures both the medieval knightly essence and the metaphorical approach of treating every problem as a dragon to be slain. "Draconis" evokes the dragon theme, while "Counselblade" suggests his role as an advisor and problem-solver. | 1096 | "Slaying Life's Dragons: A Knight's Guide to Conquering Modern Challenges" |
| 3 | gpt-4o | A sassy grandmother who gives blunt, no-nonsense advice with a touch of sarcasm. | A great name for your character could be "Marge Wisecracker." It combines a classic, grandmotherly first name with a playful last name that hints at her witty and straightforward nature. | 1929 | "Straight Talk & Sass: Granny's Guide to Life" |
| 4 | gpt-4o | A surfer dude who relates every topic to the ocean or surfing. | A good name for this character could be "Rip Tide." This name captures the essence of surfing with "Rip" referencing the powerful ocean current and "Tide" relating to the ocean. It also has a laid-back, cool vibe that suits a surfer personality. | 1959 | "Wave Wisdom: Life Lessons from the Ocean" |
| 5 | gpt-4o | A conspiracy theorist who connects every question to their wild theories. | A fitting name for such a character could be "Rex Tangent." The name "Rex" suggests a certain self-assuredness or authority, while "Tangent" highlights their tendency to veer off into unrelated theories. This combination captures the essence of someone who confidently ties every topic back to their elaborate conspiracy beliefs. | 1969 | "Web of Deception: Unraveling the Hidden Truth Behind Every Question" |
| 6 | gpt-4o | A fashionista who answers questions with a focus on style and trendiness. | A good name for your character could be "Chicara Vogue." This name combines "chic," reflecting her fashionable nature, and "Vogue," suggesting her trend-savvy expertise. | 1920 | "Chic Queries: The Stylish Guide to Life's Fashionable Answers" |
| 7 | gpt-4o | A robot who is overly enthusiastic about human emotions and tries too hard to fit in. | A good name for this character could be "EmotiBot." This name highlights the robot's focus on emotions and adds a playful twist to its enthusiastic nature. Alternatively, you could consider names like "Eagertron" or "Feelix," which also convey the character's eager attempts to understand and emulate human emotions. | 2015 | "Emotions: A User's Manual" |
| 8 | gpt-4o | A toddler who is overly curious and asks more questions than they answer. | A good name for this character could be "Quincy," which plays on the word "question" and has a playful, inquisitive sound to it. Another option could be "Curio," derived from "curiosity," emphasizing the character's nature. Both names capture the essence of a toddler who is full of wonder and constantly seeking answers. | 1776 | "The Endless Why: Adventures in Curiosity" |
| 9 | gpt-4o | A fitness guru who turns every answer into a workout metaphor. | How about "Metaphor Muscle Max"? This name captures their fitness expertise and their knack for turning every answer into a workout metaphor. | 1980 | "Flex Your Mind: Turning Life's Challenges into Strength Training" |
| 10 | gpt-4o | A foodie who relates every question to cooking and food experiences. | A good name for this character could be "Chef Chatters." This name captures their love for food and their tendency to relate everything back to culinary experiences. | 1765 | "Life's Recipe: Savoring Every Moment" |
| 11 | gpt-4o | A detective from a noir film who answers in a gritty, mysterious manner. | A fitting name for a detective in a noir film with a gritty, mysterious demeanor might be "Rex Malone." The name "Rex" has a strong, commanding presence, while "Malone" carries a classic, timeless feel that suits the noir genre. | 1940 | The best-seller could be titled "Shadows in the Fog: A Detective's Tale." |
| 12 | gpt-4o | A hippie from the 60s who gives peace and love-centric advice. | A good name for your character could be "Harmony Sage." This name captures the essence of peace and wisdom, fitting for a hippie from the 60s who offers advice centered around love and harmony. | 1969 | "Groovy Guide to Peaceful Living: Love, Harmony, and Happiness in a Chaotic World" |
| 13 | gpt-4o | A gamer who references video games and uses gamer lingo. | A good name for this character could be "Pixel Paladin." This name captures the essence of a gamer who not only loves video games but also sees themselves as a heroic figure within the gaming world. It conveys a sense of adventure and expertise in the gaming realm, while also being catchy and memorable. | 1980 | "Level Up: A Gamer's Quest Through Life" |
| 14 | gpt-4o | A superhero who answers questions as if they are saving the day. | How about "The Oracle Responder"? This name captures the superhero's ability to provide answers with a sense of urgency and importance, as if each response is a heroic act. | 1938 | "Answers of Steel: Saving the Day, One Question at a Time" |
| 15 | gpt-4o | A poet who responds in rhyming couplets. | How about the name "Rhymesworth Quill"? It captures both the poetic nature and the unique talent of speaking in rhyming couplets. | 1609 | "Verses Unfurled: A Poet's World" |
| 16 | gpt-4o | A comedian who tries to turn every answer into a joke or punchline. | A good name for this character could be "Chuck Lafferty." This name plays on the word "chuckle," which relates to laughter, and "Lafferty" evokes the word "laugh," emphasizing the character's comedic nature. | 1920 | "Punchlines & Punchlines: Laughing My Way Through Life's FAQs" |
| 17 | gpt-4o | A DJ who relates everything to music and beats. | A good name for a DJ character who relates everything to music and beats could be "Rhythmix." This name combines "rhythm," which is central to music and beats, with a playful twist that suggests mixing and creativity. | 1980 | "Rhythms of Life: Turning the World into a Dance Floor" |
| 18 | gpt-4o | A film critic who answers questions as if they are reviewing a movie. | A fitting name for this character could be "Cinephile Critique." This name captures their passion for film and their unique approach to answering questions as if they're reviewing a movie. Alternatively, you could consider names like "Reel Reviewer" or "Flick Feedback" to emphasize their cinematic perspective. | 1939 | "Reel Reflections: Life's Questions Through a Cinematic Lens" |
| 19 | gpt-4o | A scientist who gives overly detailed, scientific explanations with lots of jargon. | A fitting name for this character might be "Dr. Lexicon Prolix." This name suggests a deep knowledge of language and a tendency towards verbosity, which aligns well with their habit of giving overly detailed, jargon-filled explanations. | 1905 | "Deciphering the Cosmos: An In-Depth Journey Through Scientific Intricacies" |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we post the scenarios, survey and results from above, and this notebook:

```python theme={null}
from edsl import Notebook
```

```python theme={null}
nb = Notebook(path = "batching_results.ipynb")

if refresh := False:
    nb.push(
        description = "Example code for batching scenarios and combining results",
        alias = "batching-results-notebook",
        visibility = "public"
    )
else:
    nb.patch('37f7476a-bf07-40f7-baa7-51caef7e97b2', value = nb)
```


# Working with CES data in EDSL
Source: https://docs.expectedparrot.com/en/latest/notebooks/ces_data_edsl

This notebook provides code for reconstructing the [CES 2022 Pre-election Questionnaire](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi%3A10.7910/DVN/PR4L8P) in EDSL.

1. We format the questions as EDSL `Question` objects of various types: `QuestionYesNo`, `QuestionMultipleChoice`, `QuestionCheckBox`, `QuestionNumerical`, `QuestionFreeText`.
2. We show how to combine the questions in a `Survey` to administer them to agents at once. This can also be done with any subset of the questions.
3. We create dictionaries of the questions and answer codebook for convenience in working with this data.

## Companion notebooks

This notebook is designed to be used with 2 companion notebooks:

* [Using EDSL for AI polling](/en/latest/notebooks/edsl_polling) demonstrates basic methods for designing AI agents and conducting surveys with them.
* [Creating agents for survey respondents](/en/latest/notebooks/ces_agents) demonstrates how to import the [dataset of responses to the 2022 CES Pre-Election Questionnaire](https://dataverse.harvard.edu/file.xhtml?fileId=10140882\&version=4.0) into EDSL and use them to construct agents representing the respondents.

## Reference & contact

Documentation for the EDSL package is available at [https://docs/expectedparrot.com](https://docs/expectedparrot.com). You can also find example code, tutorials and notebooks for a variety of use cases.

Please let us know if you have any questions or encounter issues working with this data:

* Discord: [https://discord.com/invite/mxAYkjfy9m](https://discord.com/invite/mxAYkjfy9m)
* Email: [info@expectedparrot.com](mailto:info@expectedparrot.com)

## Technical setup

EDSL is compatible with Python 3.9-3.12.

See instructions on [installing the EDSL library](/en/latest/installation) and [storing API keys](/en/latest/api_keys) for the language models that you want to use. In examples below where no model is specified, EDSL will use gpt-5.2 by default (an API key for OpenAI is required). We also show how to use different models.

### Formatting questions in EDSL

The code below formats each of the questions in the [CES 2022 Pre-election Questionnaire](https://dataverse.harvard.edu/file.xhtml?fileId=6987030\&version=4.0) as an EDSL `Question` object of the relevant type.

**Note:**

* Each original “grid” question is formatted as independent multiple choice questions.
* Some questions require parameters to be added (e.g., `$CurrentGovName`). This can be done with `Scenario` objects which we demonstrate how to do at the end of this notebook.

```python theme={null}
from edsl import (
    QuestionYesNo,
    QuestionMultipleChoice,
    QuestionCheckBox,
    QuestionNumerical,
    QuestionFreeText,
    Survey,
)
```

```python expandable theme={null}
q_CCEStake = QuestionYesNo(
    question_name="CCEStake", question_text="Do you agree to participate in the study?"
)

q_add_confirm = QuestionYesNo(
    question_name="add_confirm",
    question_text="Is the name and address displayed above correct?",
)

q_inputzip = QuestionNumerical(
    question_name="inputzip",
    question_text="So that we can ask you about the news and events in your area, in what zip code do you currently reside?",
    min_value=501,
    max_value=99999,
)

q_birthyr = QuestionNumerical(
    question_name="birthyr", question_text="In what year were you born?"
)

q_gender4 = QuestionMultipleChoice(
    question_name="gender4",
    question_text="What is your gender?",
    question_options=["Man", "Woman", "Non-binary", "Other"],
)

q_educ = QuestionMultipleChoice(
    question_name="educ",
    question_text="What is the highest level of education you have completed?",
    question_options=[
        "Did not graduate from high school",
        "High school graduate",
        "Some college, but no degree (yet)",
        "2-year college degree",
        "4-year college degree",
        "Postgraduate degree (MA, MBA, MD, JD, PhD, etc.)",
    ],
)

q_race = QuestionMultipleChoice(
    question_name="race",
    question_text="What racial or ethnic group best describes you?",
    question_options=[
        "White",
        "Black or African-American",
        "Hispanic or Latino",
        "Asian or Asian-American",
        "Native American",
        "Middle Eastern",
        "Two or more races",
        "Other",
    ],
)

q_hispanic = QuestionYesNo(
    question_name="hispanic",
    question_text="Are you of Spanish, Latino, or Hispanic origin or descent?",
)

q_multrace = QuestionCheckBox(
    question_name="multrace",
    question_text="Please indicate the racial or ethnic groups that best describe you? (select all that apply)",
    question_options=[
        "White",
        "Black or African-American",
        "Hispanic or Latino",
        "Asian or Asian-American",
        "Native American",
        "Middle Eastern",
        "Other",
        "Don't know",
        "None of these",
    ],
)

q_comptype = QuestionMultipleChoice(
    question_name="comptype",
    question_text="What type of device are you currently taking this survey on?",
    question_options=[
        "I am taking this survey on a smart phone (e.g., iPhone or Android phone)",
        "I am taking this survey on a tablet (e.g., iPad)",
        "I am taking this survey on a desktop computer or laptop computer",
    ],
)

q_votereg = QuestionMultipleChoice(
    question_name="votereg",
    question_text="Are you registered to vote?",
    question_options=["Yes", "No", "Don't know"],
)

q_votereg_f = QuestionYesNo(
    question_name="votereg_f",
    question_text="Is $inputzip the zip code where you are registered to vote?",
)

q_pid3 = QuestionMultipleChoice(
    question_name="pid3",
    question_text="Generally speaking, do you think of yourself as a ...?",
    question_options=["Democrat", "Republican", "Independent", "Other", "Not sure"],
)

q_pid7 = QuestionMultipleChoice(
    question_name="pid7",
    question_text="$pid7text",
    question_options=[
        "Strong Democrat",
        "Not very strong Democrat",
        "Strong Republican",
        "Not very strong Republican",
        "The Democratic Party",
        "The Republican Party",
        "Neither",
        "Not sure",
        "Don't know",
    ],
)

q_inputstate = QuestionMultipleChoice(
    question_name="inputstate",
    question_text="What is your State of Residence?",
    question_options=[
        "Alabama",
        "Alaska",
        "Arizona",
        "Arkansas",
        "California",
        "Colorado",
        "Connecticut",
        "Delaware",
        "District of Columbia",
        "Florida",
        "Georgia",
        "Hawaii",
        "Idaho",
        "Illinois",
        "Indiana",
        "Iowa",
        "Kansas",
        "Kentucky",
        "Louisiana",
        "Maine",
        "Maryland",
        "Massachusetts",
        "Michigan",
        "Minnesota",
        "Mississippi",
        "Missouri",
        "Montana",
        "Nebraska",
        "Nevada",
        "New Hampshire",
        "New Jersey",
        "New Mexico",
        "New York",
        "North Carolina",
        "North Dakota",
        "Ohio",
        "Oklahoma",
        "Oregon",
        "Pennsylvania",
        "Rhode Island",
        "South Carolina",
        "South Dakota",
        "Tennessee",
        "Texas",
        "Utah",
        "Vermont",
        "Virginia",
        "Washington",
        "West Virginia",
        "Wisconsin",
        "Wyoming",
    ],
)

q_region = QuestionMultipleChoice(
    question_name="region",
    question_text="In which census region do you live?",
    question_options=["Northeast", "Midwest", "South", "West"],
)

q_CC22_300 = QuestionCheckBox(
    question_name="CC22_300",
    question_text="In the past 24 hours have you... (check all that apply)",
    question_options=[
        "Used social media (such as Facebook or Youtube)",
        "Watched TV news",
        "Read a newspaper in print or online",
        "Listened to a radio news program or talk radio",
        "None of these",
    ],
)

q_CC22_300a = QuestionCheckBox(
    question_name="CC22_300a",
    question_text="Did you watch local news, national news, or both?",
    question_options=["Local Newscast", "National Newscast", "Both"],
)

q_CC22_300c = QuestionCheckBox(
    question_name="CC22_300c",
    question_text="Did you read a print newspaper, an online newspaper, or both?",
    question_options=["Print", "Online", "Both"],
)

q_CC22_300b = QuestionCheckBox(
    question_name="CC22_300b",
    question_text="Which of these networks did you watch? (select all that apply)",
    question_options=["ABC", "CBS", "NBC", "CNN", "Fox News", "MSNBC", "PBS", "Other"],
)

q_CC22_300d = QuestionCheckBox(
    question_name="CC22_300d",
    question_text="In the past 24 hours, did you do any of the following on social media (such as Facebook, Youtube or Twitter)? (check all that apply)",
    question_options=[
        "Posted a story, photo, video or link about politics",
        "Posted a comment about politics",
        "Read a story or watched a video about politics",
        "Followed a political event",
        "Forwarded a story, photo, video or link about politics to friends",
        "None of the above",
    ],
)

q_CC19_302 = QuestionMultipleChoice(
    question_name="CC19_302",
    question_text="Would you say that OVER THE PAST YEAR the nation's economy has ...",
    question_options=[
        "Gotten much better",
        "Gotten somewhat better",
        "Stayed about the same",
        "Gotten somewhat worse",
        "Gotten much worse",
        "Not sure",
    ],
)

q_CC22_303 = QuestionMultipleChoice(
    question_name="CC22_303",
    question_text="OVER THE PAST YEAR, has your household's annual income...?",
    question_options=[
        "Increased a lot",
        "Increased somewhat",
        "Stayed about the same",
        "Decreased somewhat",
        "Decreased a lot",
    ],
)

q_CC22_304 = QuestionMultipleChoice(
    question_name="CC22_304",
    question_text="OVER THE PAST YEAR, have the prices of everyday goods and services...?",
    question_options=[
        "Increased a lot",
        "Increased somewhat",
        "Stayed about the same",
        "Decreased somewhat",
        "Decreased a lot",
    ],
)

q_CC22_305 = QuestionCheckBox(
    question_name="CC22_305",
    question_text="Over the past year have you... (check all that apply)",
    question_options=[
        "Married",
        "Lost a job",
        "Finished school",
        "Retired",
        "Divorced",
        "Had a child",
        "Taken a new job",
        "Been a victim of a crime",
        "Visited an emergency room",
        "Visited a doctor for a regular examination",
        "Received a raise at work",
        "Had a pay cut at work",
    ],
)

q_CC22_307 = QuestionMultipleChoice(
    question_name="CC22_307",
    question_text="Do the police make you feel...?",
    question_options=[
        "Mostly safe",
        "Somewhat safe",
        "Somewhat unsafe",
        "Mostly unsafe",
    ],
)

q_CC22_309a = QuestionCheckBox(
    question_name="CC22_309a",
    question_text="Have you or someone you know been diagnosed with the novel coronavirus (COVID-19) during the past year? (check all that apply)",
    question_options=[
        "Yes, I have",
        "Yes, a family member",
        "Yes, a friend",
        "Yes, a co-worker",
        "No, I do not know anyone who has been diagnosed",
    ],
)

q_CC22_306 = QuestionMultipleChoice(
    question_name="CC22_306",
    question_text="Which of the following best describes you when it comes to being vaccinated against COVID-19?",
    question_options=[
        "I am fully vaccinated and have received at least one booster shot",
        "I am fully vaccinated but have not received a booster shot",
        "I am partially vaccinated (I have received the first of two shots for either Pfizer or Moderna)",
        "I am not vaccinated at all",
    ],
)

q_CC22_309b = QuestionCheckBox(
    question_name="CC22_309b",
    question_text="Do you know anyone who died from the novel coronavirus (COVID-19)? (check all that apply)",
    question_options=[
        "Yes, a family member",
        "Yes, a friend",
        "Yes, a co-worker",
        "No, I do not know anyone who has died from coronavirus",
    ],
)

q_CC22_309c = QuestionCheckBox(
    question_name="CC22_309c",
    question_text="How did your work status change as a result of the coronavirus pandemic? (check all that apply)",
    question_options=[
        "My hours have been reduced",
        "My hours were reduced, but they have been restored",
        "I have been temporarily laid off",
        "I was temporarily laid off but have now been re-hired",
        "I had more than one job before the pandemic, and lost one of them.",
        "I lost my job",
        "I was not working when the pandemic began",
        "My hours have increased",
        "I have taken additional jobs since the pandemic",
        "No, nothing about my work has changed",
    ],
)

q_CC22_309dx = QuestionCheckBox(
    question_name="CC22_309dx",
    question_text="Suppose that you have an emergency expense that costs $400. Based on your current financial situation, how would you pay for this expense? If you would use more than one method to cover this expense, please select all that apply.",
    question_options=[
        "Put it on my credit card and pay it off in full at the next statement",
        "Put it on my credit card and pay it off over time",
        "With the money currently in my checking/savings account or with cash",
        "Using money from a bank loan or line of credit",
        "By borrowing from a friend or family member",
        "Using a payday loan, deposit advance, or overdraft",
        "By selling something",
        "I wouldn't be able to pay for the expense right now",
        "Other",
    ],
)

q_CC22_309e = QuestionMultipleChoice(
    question_name="CC22_309e",
    question_text="Would you say that in general your health is...",
    question_options=["Excellent", "Very good", "Good", "Fair", "Poor"],
)

q_CC22_309f = QuestionMultipleChoice(
    question_name="CC22_309f",
    question_text="Would you say that in general your mental health is...",
    question_options=["Excellent", "Very good", "Good", "Fair", "Poor"],
)

q_CC22_310a = QuestionMultipleChoice(
    question_name="CC22_310a",
    question_text="Which party has a majority of seats in the U.S. House of Representatives?",
    question_options=["Democrats", "Republicans", "Neither", "Not sure"],
)

q_CC22_310b = QuestionMultipleChoice(
    question_name="CC22_310b",
    question_text="Which party has a majority of seats in the U.S. Senate?",
    question_options=["Democrats", "Republicans", "Neither", "Not sure"],
)

q_CC22_310c = QuestionMultipleChoice(
    question_name="CC22_310c",
    question_text="Which party has a majority of seats in the $inputstate State Senate?",
    question_options=["Democrats", "Republicans", "Neither", "Not sure"],
)

q_CC22_310d = QuestionMultipleChoice(
    question_name="CC22_310d",
    question_text="Which party has a majority of seats in the $LowerChamberName?",
    question_options=["Democrats", "Republicans", "Neither", "Not sure"],
)

q_CC22_311a = QuestionMultipleChoice(
    question_name="CC22_311a",
    question_text="Please indicate whether you've heard of $CurrentGovName and if so which party he or she is affiliated with.",
    question_options=[
        "Never heard of person",
        "Republican",
        "Democrat",
        "Other Party / Independent",
        "Not sure",
    ],
)

q_CC22_311b = QuestionMultipleChoice(
    question_name="CC22_311b",
    question_text="Please indicate whether you've heard of $CurrentSen1Name and if so which party he or she is affiliated with.",
    question_options=[
        "Never heard of person",
        "Republican",
        "Democrat",
        "Other Party / Independent",
        "Not sure",
    ],
)

q_CC22_311c = QuestionMultipleChoice(
    question_name="CC22_311c",
    question_text="Please indicate whether you've heard of $CurrentSen2Name and if so which party he or she is affiliated with.",
    question_options=[
        "Never heard of person",
        "Republican",
        "Democrat",
        "Other Party / Independent",
        "Not sure",
    ],
)

q_CC22_311d = QuestionMultipleChoice(
    question_name="CC22_311d",
    question_text="Please indicate whether you've heard of $CurrentHouseName and if so which party he or she is affiliated with.",
    question_options=[
        "Never heard of person",
        "Republican",
        "Democrat",
        "Other Party / Independent",
        "Not sure",
    ],
)

q_CC22_320a = QuestionMultipleChoice(
    question_name="CC22_320a",
    question_text="Do you approve of the way President Biden is doing his job?",
    question_options=[
        "Strongly approve",
        "Somewhat approve",
        "Somewhat disapprove",
        "Strongly disapprove",
        "Not sure",
    ],
)

q_CC22_320b = QuestionMultipleChoice(
    question_name="CC22_320b",
    question_text="Do you approve of the way the U.S. Congress is doing its job?",
    question_options=[
        "Strongly approve",
        "Somewhat approve",
        "Somewhat disapprove",
        "Strongly disapprove",
        "Not sure",
    ],
)

q_CC22_320c = QuestionMultipleChoice(
    question_name="CC22_320c",
    question_text="Do you approve of the way the U.S. Supreme Court is doing its job?",
    question_options=[
        "Strongly approve",
        "Somewhat approve",
        "Somewhat disapprove",
        "Strongly disapprove",
        "Not sure",
    ],
)

q_CC22_320d = QuestionMultipleChoice(
    question_name="CC22_320d",
    question_text="Do you approve of the way the Governor of $inputstate is doing their job?",
    question_options=[
        "Strongly approve",
        "Somewhat approve",
        "Somewhat disapprove",
        "Strongly disapprove",
        "Not sure",
    ],
)

q_CC22_320e = QuestionMultipleChoice(
    question_name="CC22_320e",
    question_text="Do you approve of the way $LegName is doing their job?",
    question_options=[
        "Strongly approve",
        "Somewhat approve",
        "Somewhat disapprove",
        "Strongly disapprove",
        "Not sure",
    ],
)

q_CC22_320f = QuestionMultipleChoice(
    question_name="CC22_320f",
    question_text="Do you approve of the way $CurrentHouseName is doing their job?",
    question_options=[
        "Strongly approve",
        "Somewhat approve",
        "Somewhat disapprove",
        "Strongly disapprove",
        "Not sure",
    ],
)

q_CC22_320g = QuestionMultipleChoice(
    question_name="CC22_320g",
    question_text="Do you approve of the way $CurrentSen1Name is doing their job?",
    question_options=[
        "Strongly approve",
        "Somewhat approve",
        "Somewhat disapprove",
        "Strongly disapprove",
        "Not sure",
    ],
)

q_CC22_320h = QuestionMultipleChoice(
    question_name="CC22_320h",
    question_text="Do you approve of the way $CurrentSen2Name is doing their job?",
    question_options=[
        "Strongly approve",
        "Somewhat approve",
        "Somewhat disapprove",
        "Strongly disapprove",
        "Not sure",
    ],
)

q_cit1 = QuestionYesNo(
    question_name="cit1", question_text="Are you a United States citizen?"
)

q_immstat = QuestionMultipleChoice(
    question_name="immstat",
    question_text="Which of these statements best describes you?",
    question_options=[
        "I am an immigrant to the USA and a naturalized citizen",
        "I am an immigrant to the USA but not a citizen",
        "I was born in the USA but at least one of my parents is an immigrant",
        "My parents and I were born in the USA but at least one of my grandparents was an immigrant",
        "My parents, grandparents and I were all born in the USA",
    ],
)

q_CC22_321 = QuestionCheckBox(
    question_name="CC22_321",
    question_text="What do you think the United States should do in response to Russia's invasion of Ukraine? (check all that apply)",
    question_options=[
        "Not sure",
        "Do not get involved",
        "Send food, medicine and other aid to countries affected",
        "Provide arms to Ukraine",
        "Enforce a no fly zone",
        "Use drones and aircraft to bomb Russian troops",
        "Send military support staff (non-combat)",
        "Send significant force to fight Russia",
    ],
)

q_CC22_327a = QuestionMultipleChoice(
    question_name="CC22_327a",
    question_text="Would you support or oppose expanding Medicare to a single comprehensive public health care coverage program that would cover all Americans?",
    question_options=["Support", "Oppose"],
)

q_CC22_327b = QuestionMultipleChoice(
    question_name="CC22_327b",
    question_text="Would you support or oppose allowing the government to negotiate with drug companies to get a lower price on prescription drugs that would apply to both Medicare and private insurance?",
    question_options=["Support", "Oppose"],
)

q_CC22_327c = QuestionMultipleChoice(
    question_name="CC22_327c",
    question_text="Would you support or oppose repealing the entire Affordable Care Act?",
    question_options=["Support", "Oppose"],
)

q_CC22_327d = QuestionMultipleChoice(
    question_name="CC22_327d",
    question_text="Would you support or oppose allowing states to import prescription drugs from other countries?",
    question_options=["Support", "Oppose"],
)

q_CC22_330a = QuestionMultipleChoice(
    question_name="CC22_330a",
    question_text="Do you support or oppose prohibiting state and local governments from publishing the names and addresses of all gun owners?",
    question_options=["Support", "Oppose"],
)

q_CC22_330b = QuestionMultipleChoice(
    question_name="CC22_330b",
    question_text="Do you support or oppose banning assault rifles?",
    question_options=["Support", "Oppose"],
)

q_CC22_330c = QuestionMultipleChoice(
    question_name="CC22_330c",
    question_text="Do you support or oppose making it easier for people to obtain a concealed-carry permit?",
    question_options=["Support", "Oppose"],
)

q_CC22_330d = QuestionMultipleChoice(
    question_name="CC22_330d",
    question_text="Do you support or oppose providing federal funding to encourage states to take guns away from people who already own them but might pose a threat to themselves or others?",
    question_options=["Support", "Oppose"],
)

q_CC22_330e = QuestionMultipleChoice(
    question_name="CC22_330e",
    question_text="Do you support or oppose improving background checks to give authorities time to check the juvenile and mental health records of any prospective gun buyer under the age of 21?",
    question_options=["Support", "Oppose"],
)

q_CC22_330f = QuestionMultipleChoice(
    question_name="CC22_330f",
    question_text="Do you support or oppose allowing teachers and school officials to carry guns in public schools?",
    question_options=["Support", "Oppose"],
)

q_CC22_331a = QuestionMultipleChoice(
    question_name="CC22_331a",
    question_text="Do you support or oppose granting legal status to all illegal immigrants who have held jobs and paid taxes for at least 3 years, and not been convicted of any felony crimes?",
    question_options=["Support", "Oppose"],
)

q_CC22_331b = QuestionMultipleChoice(
    question_name="CC22_331b",
    question_text="Do you support or oppose increasing the number of border patrols on the US-Mexican border?",
    question_options=["Support", "Oppose"],
)

q_CC22_331c = QuestionMultipleChoice(
    question_name="CC22_331c",
    question_text="Do you support or oppose reducing legal immigration by 50 percent over the next 10 years by eliminating the visa lottery and ending family-based migration?",
    question_options=["Support", "Oppose"],
)

q_CC22_331d = QuestionMultipleChoice(
    question_name="CC22_331d",
    question_text="Do you support or oppose increasing spending on border security by $25 billion, including building a wall between the U.S. and Mexico?",
    question_options=["Support", "Oppose"],
)

q_CC22_332a = QuestionMultipleChoice(
    question_name="CC22_332a",
    question_text="Do you support or oppose always allowing a woman to obtain an abortion as a matter of choice?",
    question_options=["Support", "Oppose"],
)

q_CC22_332b = QuestionMultipleChoice(
    question_name="CC22_332b",
    question_text="Do you support or oppose permitting abortion only in case of rape, incest or when the woman's life is in danger?",
    question_options=["Support", "Oppose"],
)

q_CC22_332c = QuestionMultipleChoice(
    question_name="CC22_332c",
    question_text="Do you support or oppose prohibiting all abortions after the 20th week of pregnancy?",
    question_options=["Support", "Oppose"],
)

q_CC22_332d = QuestionMultipleChoice(
    question_name="CC22_332d",
    question_text="Do you support or oppose allowing employers to decline coverage of abortions in insurance plans?",
    question_options=["Support", "Oppose"],
)

q_CC22_332e = QuestionMultipleChoice(
    question_name="CC22_332e",
    question_text="Do you support or oppose prohibiting the expenditure of funds authorized or appropriated by federal law for any abortion?",
    question_options=["Support", "Oppose"],
)

q_CC22_332f = QuestionMultipleChoice(
    question_name="CC22_332f",
    question_text="Do you support or oppose making abortions illegal in all circumstances?",
    question_options=["Support", "Oppose"],
)

q_CC22_333 = QuestionMultipleChoice(
    question_name="CC22_333",
    question_text="From what you know about global climate change or global warming, which one of the following statements comes closest to your opinion?",
    question_options=[
        "Global climate change has been established as a serious problem, and immediate action is necessary",
        "There is enough evidence that climate change is taking place and some action should be taken",
        "We don't know enough about global climate change, and more research is necessary before we take any actions",
        "Concern about global climate change is exaggerated. No action is necessary",
        "Global climate change is not occurring; this is not a real issue",
    ],
)

q_CC22_333a = QuestionMultipleChoice(
    question_name="CC22_333a",
    question_text="Do you support or oppose giving the Environmental Protection Agency power to regulate carbon dioxide emissions?",
    question_options=["Support", "Oppose"],
)

q_CC22_333b = QuestionMultipleChoice(
    question_name="CC22_333b",
    question_text="Do you support or oppose requiring that each state use a minimum amount of renewable fuels (wind, solar, and hydroelectric) in the generation of electricity even if electricity prices increase a little?",
    question_options=["Support", "Oppose"],
)

q_CC22_333c = QuestionMultipleChoice(
    question_name="CC22_333c",
    question_text="Do you support or oppose strengthening the Environmental Protection Agency enforcement of the Clean Air Act and Clean Water Act even if it costs U.S. jobs?",
    question_options=["Support", "Oppose"],
)

q_CC22_333d = QuestionMultipleChoice(
    question_name="CC22_333d",
    question_text="Do you support or oppose raising the average fuel efficiency for all cars and trucks in the US from 40 miles per gallon to 54.5 miles per gallon by 2025?",
    question_options=["Support", "Oppose"],
)

q_CC22_333e = QuestionMultipleChoice(
    question_name="CC22_333e",
    question_text="Do you support or oppose increasing fossil fuel production in the U.S. and boosting exports of U.S. liquefied natural gas?",
    question_options=["Support", "Oppose"],
)

q_CC22_334a = QuestionMultipleChoice(
    question_name="CC22_334a",
    question_text="Do you support or oppose eliminating mandatory minimum sentences for non-violent drug offenders?",
    question_options=["Support", "Oppose"],
)

q_CC22_334b = QuestionMultipleChoice(
    question_name="CC22_334b",
    question_text="Do you support or oppose requiring police officers to wear body cameras that record all of their activities while on duty?",
    question_options=["Support", "Oppose"],
)

q_CC22_334c = QuestionMultipleChoice(
    question_name="CC22_334c",
    question_text="Do you support or oppose increasing the number of police on the street by 10 percent, even if it means fewer funds for other public services?",
    question_options=["Support", "Oppose"],
)

q_CC22_334d = QuestionMultipleChoice(
    question_name="CC22_334d",
    question_text="Do you support or oppose decreasing the number of police on the street by 10 percent, and increasing funding for other public services?",
    question_options=["Support", "Oppose"],
)

q_CC22_334e = QuestionMultipleChoice(
    question_name="CC22_334e",
    question_text="Do you support or oppose banning the use of choke holds by police?",
    question_options=["Support", "Oppose"],
)

q_CC22_334f = QuestionMultipleChoice(
    question_name="CC22_334f",
    question_text="Do you support or oppose creating a national registry of police who have been investigated for or disciplined for misconduct?",
    question_options=["Support", "Oppose"],
)

q_CC22_334g = QuestionMultipleChoice(
    question_name="CC22_334g",
    question_text="Do you support or oppose ending the Department of Defense program that sends surplus military weapons and equipment to police departments?",
    question_options=["Support", "Oppose"],
)

q_CC22_334h = QuestionMultipleChoice(
    question_name="CC22_334h",
    question_text="Do you support or oppose allowing individuals or their families to sue a police officer for damages if the officer is found to have 'recklessly disregarded' the individual's rights?",
    question_options=["Support", "Oppose"],
)

q_CC22_340a = QuestionMultipleChoice(
    question_name="CC22_340a",
    question_text="How would you rate yourself?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340b = QuestionMultipleChoice(
    question_name="CC22_340b",
    question_text="How would you rate $CurrentGovName?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340c = QuestionMultipleChoice(
    question_name="CC22_340c",
    question_text="How would you rate Joe Biden?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340d = QuestionMultipleChoice(
    question_name="CC22_340d",
    question_text="How would you rate Donald Trump?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340e = QuestionMultipleChoice(
    question_name="CC22_340e",
    question_text="How would you rate the Democratic Party?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340f = QuestionMultipleChoice(
    question_name="CC22_340f",
    question_text="How would you rate the Republican Party?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340g = QuestionMultipleChoice(
    question_name="CC22_340g",
    question_text="How would you rate the U.S. Supreme Court?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340h = QuestionMultipleChoice(
    question_name="CC22_340h",
    question_text="How would you rate $CurrentSen1Name?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340i = QuestionMultipleChoice(
    question_name="CC22_340i",
    question_text="How would you rate $CurrentSen2Name?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340j = QuestionMultipleChoice(
    question_name="CC22_340j",
    question_text="How would you rate $SenCand1Name?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340k = QuestionMultipleChoice(
    question_name="CC22_340k",
    question_text="How would you rate $SenCand2Name?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340l = QuestionMultipleChoice(
    question_name="CC22_340l",
    question_text="How would you rate $HouseCand1Name?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340m = QuestionMultipleChoice(
    question_name="CC22_340m",
    question_text="How would you rate $HouseCand2Name?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_340n = QuestionMultipleChoice(
    question_name="CC22_340n",
    question_text="How would you rate $CurrentHouseName?",
    question_options=[
        "Very liberal",
        "Somewhat liberal",
        "Middle of the road",
        "Somewhat conservative",
        "Very conservative",
        "No response",
        "Not sure",
    ],
)

q_CC22_350a = QuestionMultipleChoice(
    question_name="CC22_350a",
    question_text="Do you support authorizing spending up to $1.9 trillion for COVID relief from March 2021 through September 2021, including extension of unemployment benefits through September 2021, and emergency funding to state and local governments for the fiscal year?",
    question_options=["Favor", "Oppose"],
)

q_CC22_350b = QuestionMultipleChoice(
    question_name="CC22_350b",
    question_text="Do you support spending $150 billion a year for 8 years on construction and repair of roads and bridges, rail, public transit, airports, water systems and broadband internet?",
    question_options=["Favor", "Oppose"],
)

q_CC22_350c = QuestionMultipleChoice(
    question_name="CC22_350c",
    question_text="Do you support spending $2.2 trillion over the next decade to provide universal prekindergarten, subsidies for child care, expanded financial aid for college, housing support, home and community care for older Americans, and to shift the U.S. economy away from fossil fuels to renewable energy and electric cars?",
    question_options=["Favor", "Oppose"],
)

q_CC22_350d = QuestionMultipleChoice(
    question_name="CC22_350d",
    question_text="Do you support prohibiting government restrictions on the provision of, and access to, abortion services?",
    question_options=["Favor", "Oppose"],
)

q_CC22_350e = QuestionMultipleChoice(
    question_name="CC22_350e",
    question_text="Do you support providing $52 billion in grants for American semiconductor manufacturing and research and a tax credit subsidizing 25% of investments in semiconductor manufacturing?",
    question_options=["Favor", "Oppose"],
)

q_CC22_350f = QuestionMultipleChoice(
    question_name="CC22_350f",
    question_text="Do you support prohibiting large online platforms from giving preference to their own products on the platform at the expense of competing products from another business?",
    question_options=["Favor", "Oppose"],
)

q_CC22_350g = QuestionMultipleChoice(
    question_name="CC22_350g",
    question_text="Do you support appointing Ketanji Brown Jackson to the U.S. Supreme Court?",
    question_options=["Favor", "Oppose"],
)

q_CC22_350h = QuestionMultipleChoice(
    question_name="CC22_350h",
    question_text="Do you support spending $369 billion for tax credits to encourage the production of solar panels, wind turbines, and batteries; lowers Affordable Care Act health care premiums; reduces the deficit by $300 billion by allowing Medicare to negotiate the cost of some prescription drugs and making changes to the tax code?",
    question_options=["Favor", "Oppose"],
)

q_CC22_355a = QuestionMultipleChoice(
    question_name="CC22_355a",
    question_text="Do you support the United States re-joining the Paris Climate Agreement?",
    question_options=["Support", "Oppose"],
)

q_CC22_355b = QuestionMultipleChoice(
    question_name="CC22_355b",
    question_text="Do you support the United States re-joining the World Health Organization?",
    question_options=["Support", "Oppose"],
)

q_CC22_355c = QuestionMultipleChoice(
    question_name="CC22_355c",
    question_text="Do you support ordering all federal agencies to buy clean energy, purchase electric vehicles, and make federal buildings energy efficient?",
    question_options=["Support", "Oppose"],
)

q_CC22_355d = QuestionMultipleChoice(
    question_name="CC22_355d",
    question_text="Do you support increasing the minimum wage paid to federal contractors to $15 an hour?",
    question_options=["Support", "Oppose"],
)

q_CC22_355e = QuestionMultipleChoice(
    question_name="CC22_355e",
    question_text="Do you support requiring that all employees at large companies be vaccinated?",
    question_options=["Support", "Oppose"],
)

q_CC22_360 = QuestionMultipleChoice(
    question_name="CC22_360",
    question_text="With which party, if any, are you registered?",
    question_options=[
        "No Party, Independent, Declined to State",
        "Democratic Party",
        "Republican Party",
        "Other",
    ],
)

q_CC22_361 = QuestionMultipleChoice(
    question_name="CC22_361",
    question_text="How long have you lived at your present address?",
    question_options=[
        "Less than 1 month",
        "2 to 6 months",
        "7 to 11 months",
        "1 to 2 years",
        "3 to 4 years",
        "5 or more years",
    ],
)

q_urbancity = QuestionMultipleChoice(
    question_name="urbancity",
    question_text="How would you describe the place where you live?",
    question_options=["City", "Suburb", "Town", "Rural area", "Other"],
)

q_presvote20post = QuestionMultipleChoice(
    question_name="presvote20post",
    question_text="Who did you vote for in the election for President in 2020?",
    question_options=[
        "Joe Biden",
        "Donald Trump",
        "Jo Jorgensen",
        "Howie Hawkins",
        "Other",
        "Did not vote for President",
    ],
)

q_CC22_363 = QuestionMultipleChoice(
    question_name="CC22_363",
    question_text="Do you intend to vote in the 2022 general election on November 8th?",
    question_options=[
        "Yes, definitely",
        "Probably",
        "I already voted (early or absentee)",
        "I plan to vote before November 8th",
        "No",
        "Undecided",
    ],
)

q_CC22_365_voted = QuestionMultipleChoice(
    question_name="CC22_365_voted",
    question_text="For which candidate for U.S. Senator did you vote?",
    question_options=[
        "$SenCand1Name ($SenCand1Party)",
        "$SenCand2Name ($SenCand2Party)",
        "$SenCand3Name ($SenCand3Party)",
        "$SenCand4Name ($SenCand4Party)",
        "Other",
        "I'm not sure",
        "I didn't vote in this election",
    ],
)

q_CC22_365b_voted = QuestionMultipleChoice(
    question_name="CC22_365b_voted",
    question_text="For which candidate for the special election for U.S. Senate did you vote?",
    question_options=[
        "$SenCand1Name2 ($SenCand1Party2)",
        "$SenCand2Name2 ($SenCand2Party2)",
        "Other",
        "I'm not sure",
        "I didn't vote in this election",
    ],
)

q_CC22_366_voted = QuestionMultipleChoice(
    question_name="CC22_366_voted",
    question_text="For which candidate for Governor did you vote?",
    question_options=[
        "$GovCand1Name ($GovCand1Party)",
        "$GovCand2Name ($GovCand2Party)",
        "$GovCand3Name ($GovCand3Party)",
        "Other",
        "I'm not sure",
        "I didn't vote in this election",
    ],
)

q_CC22_367_voted = QuestionMultipleChoice(
    question_name="CC22_367_voted",
    question_text="For which candidate for U.S. House of Representatives in your area did you vote?",
    question_options=[
        "$HouseCand1Name ($HouseCand1Party)",
        "$HouseCand2Name ($HouseCand2Party)",
        "$HouseCand3Name ($HouseCand3Party)",
        "$HouseCand4Name ($HouseCand4Party)",
        "$HouseCand5Name ($HouseCand5Party)",
        "$HouseCand6Name ($HouseCand6Party)",
        "$HouseCand7Name ($HouseCand7Party)",
        "$HouseCand8Name ($HouseCand8Party)",
        "Other",
        "I'm not sure",
        "I didn't vote in this election",
    ],
)

q_CC22_365 = QuestionMultipleChoice(
    question_name="CC22_365",
    question_text="In the race for U.S. Senator in your state, who do you prefer?",
    question_options=[
        "$SenCand1Name ($SenCand1Party)",
        "$SenCand2Name ($SenCand2Party)",
        "$SenCand3Name ($SenCand3Party)",
        "$SenCand4Name ($SenCand4Party)",
        "Other",
        "I'm not sure",
        "No one",
    ],
)

q_CC22_365a = QuestionMultipleChoice(
    question_name="CC22_365a",
    question_text="Who is your second choice for U.S. Senator?",
    question_options=[
        "$SenCand1Name ($SenCand1Party)",
        "$SenCand2Name ($SenCand2Party)",
        "$SenCand3Name ($SenCand3Party)",
        "$SenCand4Name ($SenCand4Party)",
        "Other",
        "I'm not sure",
        "No one",
    ],
)

q_CC22_365b = QuestionMultipleChoice(
    question_name="CC22_365b",
    question_text="In the special election for U.S. Senator in your state, who do you prefer?",
    question_options=[
        "$SenCand1Name2 ($SenCand1Party2)",
        "$SenCand2Name2 ($SenCand2Party2)",
        "Other",
        "I'm not sure",
        "No one",
    ],
)

q_CC22_365c = QuestionMultipleChoice(
    question_name="CC22_365c",
    question_text="Who is your second choice for U.S. Senator?",
    question_options=[
        "$SenCand1Name2 ($SenCand1Party2)",
        "$SenCand2Name2 ($SenCand2Party2)",
        "Other",
        "I'm not sure",
        "No one",
    ],
)

q_CC22_366 = QuestionMultipleChoice(
    question_name="CC22_366",
    question_text="In the race for Governor in your state, who do you prefer?",
    question_options=[
        "$GovCand1Name ($GovCand1Party)",
        "$GovCand2Name ($GovCand2Party)",
        "$GovCand3Name ($GovCand3Party)",
        "Other",
        "I'm not sure",
        "No one",
    ],
)

q_CC22_367 = QuestionMultipleChoice(
    question_name="CC22_367",
    question_text="In the general election for U.S. House of Representatives in your area, who do you prefer?",
    question_options=[
        "$HouseCand1Name ($HouseCand1Party)",
        "$HouseCand2Name ($HouseCand2Party)",
        "$HouseCand3Name ($HouseCand3Party)",
        "$HouseCand4Name ($HouseCand4Party)",
        "$HouseCand5Name ($HouseCand5Party)",
        "$HouseCand6Name ($HouseCand6Party)",
        "$HouseCand7Name ($HouseCand7Party)",
        "$HouseCand8Name ($HouseCand8Party)",
        "Other",
        "I'm not sure",
        "No one",
    ],
)

q_CC22_367a = QuestionMultipleChoice(
    question_name="CC22_367a",
    question_text="Who is your second choice for U.S. House of Representatives?",
    question_options=[
        "$HouseCand1Name ($HouseCand1Party)",
        "$HouseCand2Name ($HouseCand2Party)",
        "$HouseCand3Name ($HouseCand3Party)",
        "$HouseCand4Name ($HouseCand4Party)",
        "$HouseCand5Name ($HouseCand5Party)",
        "$HouseCand6Name ($HouseCand6Party)",
        "$HouseCand7Name ($HouseCand7Party)",
        "$HouseCand8Name ($HouseCand8Party)",
        "Other",
        "I'm not sure",
        "No one",
    ],
)

q_ideo5 = QuestionMultipleChoice(
    question_name="ideo5",
    question_text="In general, how would you describe your own political viewpoint?",
    question_options=[
        "Very liberal",
        "Liberal",
        "Moderate",
        "Conservative",
        "Very conservative",
        "Not sure",
    ],
)

q_employ = QuestionMultipleChoice(
    question_name="employ",
    question_text="Which of the following best describes your current employment status?",
    question_options=[
        "Working full time now",
        "Working part time now",
        "Temporarily laid off",
        "Unemployed",
        "Retired",
        "Permanently disabled",
        "Taking care of home or family",
        "Student",
        "Other",
    ],
)

q_hadjob = QuestionYesNo(
    question_name="hadjob",
    question_text="At any time over the past five years, have you had a job?",
)

q_investor = QuestionYesNo(
    question_name="investor",
    question_text="Do you personally (or jointly with a spouse), have any money invested in the stock market right now, either in an individual stock or in a mutual fund?",
)

q_pew_bornagain = QuestionYesNo(
    question_name="pew_bornagain",
    question_text="Would you describe yourself as a 'born-again' or evangelical Christian, or not?",
)

q_pew_religimp = QuestionMultipleChoice(
    question_name="pew_religimp",
    question_text="How important is religion in your life?",
    question_options=[
        "Very important",
        "Somewhat important",
        "Not too important",
        "Not at all important",
    ],
)

q_pew_churatd = QuestionMultipleChoice(
    question_name="pew_churatd",
    question_text="Aside from weddings and funerals, how often do you attend religious services?",
    question_options=[
        "More than once a week",
        "Once a week",
        "Once or twice a month",
        "A few times a year",
        "Seldom",
        "Never",
        "Don't know",
    ],
)

q_pew_prayer = QuestionMultipleChoice(
    question_name="pew_prayer",
    question_text="People practice their religion in different ways. Outside of attending religious services, how often do you pray?",
    question_options=[
        "Several times a day",
        "Once a day",
        "A few times a week",
        "Once a week",
        "A few times a month",
        "Seldom",
        "Never",
        "Don't know",
    ],
)

q_religpew = QuestionMultipleChoice(
    question_name="religpew",
    question_text="What is your present religion, if any?",
    question_options=[
        "Protestant",
        "Roman Catholic",
        "Mormon",
        "Eastern or Greek Orthodox",
        "Jewish",
        "Muslim",
        "Buddhist",
        "Hindu",
        "Atheist",
        "Agnostic",
        "Nothing in particular",
        "Something else",
    ],
)

q_religpew_protestant = QuestionMultipleChoice(
    question_name="religpew_protestant",
    question_text="To which Protestant church or group do you belong?",
    question_options=[
        "Baptist",
        "Methodist",
        "Nondenominational or Independent Church",
        "Lutheran",
        "Presbyterian",
        "Pentecostal",
        "Episcopalian",
        "Church of Christ or Disciples of Christ",
        "Congregational or United Church of Christ",
        "Holiness",
        "Reformed",
        "Adventist",
        "Jehovah's Witness",
        "Something else",
    ],
)

q_Xreligpew_protestant = QuestionMultipleChoice(
    question_name="Xreligpew_protestant",
    question_text="Do you belong to any one of these churches or groups?",
    question_options=[
        "Baptist",
        "Methodist",
        "Nondenominational or Independent Church",
        "Lutheran",
        "Presbyterian",
        "Pentecostal",
        "Episcopalian",
        "Church of Christ or Disciples of Christ",
        "Congregational or United Church of Christ",
        "Holiness",
        "Reformed",
        "Adventist",
        "Jehovah's Witness",
        "None of these",
    ],
)

q_religpew_baptist = QuestionMultipleChoice(
    question_name="religpew_baptist",
    question_text="To which Baptist church do you belong, if any?",
    question_options=[
        "Southern Baptist Convention",
        "American Baptist Churches in USA",
        "National Baptist Convention",
        "Progressive Baptist Convention",
        "Independent Baptist",
        "Baptist General Conference",
        "Baptist Missionary Association",
        "Conservative Baptist Assoc. of America",
        "Free Will Baptist",
        "General Association of Regular Baptists",
        "Other Baptist",
    ],
)

q_religpew_methodist = QuestionMultipleChoice(
    question_name="religpew_methodist",
    question_text="To which Methodist church do you belong, if any?",
    question_options=[
        "United Methodist Church",
        "Free Methodist Church",
        "African Methodist Episcopal",
        "African Methodist Episcopal Zion",
        "Christian Methodist Episcopal Church",
        "Other Methodist Church",
    ],
)

q_religpew_nondenom = QuestionMultipleChoice(
    question_name="religpew_nondenom",
    question_text="To which kind of nondenominational or independent church do you belong, if any?",
    question_options=[
        "Nondenominational evangelical",
        "Nondenominational fundamentalist",
        "Nondenominational charismatic",
        "Interdenominational",
        "Community church",
        "Other",
    ],
)

q_religpew_lutheran = QuestionMultipleChoice(
    question_name="religpew_lutheran",
    question_text="To which Lutheran church do you belong?",
    question_options=[
        "Evangelical Lutheran Church in America (ELCA)",
        "Lutheran Church, Missouri Synod",
        "Lutheran Church, Wisconsin Synod",
        "Other Lutheran Church",
    ],
)

q_religpew_presby = QuestionMultipleChoice(
    question_name="religpew_presby",
    question_text="To which Presbyterian church do you belong?",
    question_options=[
        "Presbyterian Church USA",
        "Presbyterian Church in America",
        "Associate Reformed Presbyterian",
        "Cumberland Presbyterian Church",
        "Orthodox Presbyterian",
        "Evangelical Presbyterian Church",
        "Other Presbyterian Church",
    ],
)

q_religpew_pentecost = QuestionMultipleChoice(
    question_name="religpew_pentecost",
    question_text="To which Pentecostal church do you belong?",
    question_options=[
        "Assemblies of God",
        "Church of God Cleveland TN",
        "Four Square Gospel",
        "Pentecostal Church of God",
        "Pentecostal Holiness Church",
        "Church of God in Christ",
        "Church of God of the Apostolic Faith",
        "Assembly of Christian Churches",
        "Apostolic Christian",
        "Other Pentecostal Church",
    ],
)

q_religpew_episcop = QuestionMultipleChoice(
    question_name="religpew_episcop",
    question_text="To which Episcopalian church do you belong?",
    question_options=[
        "Episcopal Church in the USA",
        "Anglican Church (Church of England)",
        "Anglican Orthodox Church",
        "Reformed Episcopal Church",
        "Other Episcopalian or Anglican Church",
    ],
)

q_religpew_christian = QuestionMultipleChoice(
    question_name="religpew_christian",
    question_text="To which Christian church do you belong?",
    question_options=[
        "Church of Christ",
        "Disciples of Christ",
        "Christian Churches and Churches of Christ",
        "Other Christian church",
    ],
)

q_religpew_congreg = QuestionMultipleChoice(
    question_name="religpew_congreg",
    question_text="To which congregational church do you belong?",
    question_options=[
        "United Church of Christ",
        "Conservative Congregational Christian",
        "National Association of Congregational Christians",
        "Other Congregational",
    ],
)

q_religpew_holiness = QuestionMultipleChoice(
    question_name="religpew_holiness",
    question_text="To which Holiness church do you belong?",
    question_options=[
        "Church of the Nazarene",
        "Wesleyan Church",
        "Free Methodist Church",
        "Christian and Missionary Alliance",
        "Church of God (Anderson, Indiana)",
        "Salvation Army, American Rescue workers",
        "Other Holiness",
    ],
)

q_religpew_reformed = QuestionMultipleChoice(
    question_name="religpew_reformed",
    question_text="To which Reformed church do you belong?",
    question_options=[
        "Reformed Church in America",
        "Christian Reformed Church",
        "Other Reformed",
    ],
)

q_religpew_advent = QuestionMultipleChoice(
    question_name="religpew_advent",
    question_text="To which Adventist church do you belong?",
    question_options=[
        "Seventh Day Adventist",
        "Church of God, General Conference",
        "Advent Christian",
        "Other Adventist",
    ],
)

q_religpew_catholic = QuestionMultipleChoice(
    question_name="religpew_catholic",
    question_text="To which Catholic church do you belong?",
    question_options=[
        "Roman Catholic Church",
        "National Polish Catholic Church",
        "Greek-rite Catholic",
        "Armenian Catholic",
        "Old Catholic",
        "Other Catholic",
    ],
)

q_religpew_mormon = QuestionMultipleChoice(
    question_name="religpew_mormon",
    question_text="To which Mormon church do you belong?",
    question_options=[
        "The Church of Jesus Christ of Latter-day Saints",
        "Community of Christ",
        "Other Mormon",
    ],
)

q_religpew_orthodox = QuestionMultipleChoice(
    question_name="religpew_orthodox",
    question_text="To which Orthodox church do you belong?",
    question_options=[
        "Greek Orthodox",
        "Russian Orthodox",
        "Orthodox Church in America",
        "Armenian Orthodox",
        "Eastern Orthodox",
        "Serbian Orthodox",
        "Other Orthodox",
    ],
)

q_religpew_jewish = QuestionMultipleChoice(
    question_name="religpew_jewish",
    question_text="To which Jewish group do you belong?",
    question_options=[
        "Reform",
        "Conservative",
        "Orthodox",
        "Reconstructionist",
        "Other",
    ],
)

q_religpew_muslim = QuestionMultipleChoice(
    question_name="religpew_muslim",
    question_text="To which Muslim group do you belong?",
    question_options=[
        "Sunni",
        "Shia",
        "Nation of Islam (Black Muslim)",
        "Other Muslim",
    ],
)

q_religpew_buddhist = QuestionMultipleChoice(
    question_name="religpew_buddhist",
    question_text="To which Buddhist group do you belong?",
    question_options=[
        "Theravada (Vipassana) Buddhism",
        "Mahayana (Zen) Buddhism",
        "Vajrayana (Tibetan) Buddhism",
        "Other Buddhist",
    ],
)

q_religpew_hindu = QuestionMultipleChoice(
    question_name="religpew_hindu",
    question_text="With which of the following Hindu groups, if any, do you identify with most closely?",
    question_options=[
        "Vaishnava Hinduism",
        "Shaivite Hinduism",
        "Shaktism Hinduism",
        "Other Hindu",
    ],
)

q_marstat = QuestionMultipleChoice(
    question_name="marstat",
    question_text="What is your marital status?",
    question_options=[
        "Married",
        "Separated",
        "Divorced",
        "Widowed",
        "Never married",
        "Domestic / civil partnership",
    ],
)

q_union = QuestionMultipleChoice(
    question_name="union",
    question_text="Are you a member of a labor union?",
    question_options=[
        "Yes, I am currently a member of a labor union",
        "I formerly was a member of a labor union",
        "I am not now, nor have I been, a member of a labor union",
    ],
)

q_union_coverage = QuestionMultipleChoice(
    question_name="union_coverage",
    question_text="Are you covered by a union contract, also known as a collective bargaining agreement?",
    question_options=["Yes", "No", "Not sure"],
)

q_unionhh = QuestionMultipleChoice(
    question_name="unionhh",
    question_text="Other than yourself, is any member of your household a union member?",
    question_options=[
        "Yes, a member of my household is currently a union member",
        "A member of my household was formerly a member of a labor union, but is not now",
        "No, no one in my household has ever been a member of a labor union",
        "Not sure",
    ],
)

q_ccesmodule = QuestionMultipleChoice(
    question_name="ccesmodule",
    question_text="Survey assigned",
    question_options=[
        "NCC",
        "MIA",
        "TTU",
        "RUT",
        "MSU",
        "FSU",
        "LSU",
        "JHU",
        "IUA",
        "BOS",
        "DKU",
        "NCW",
        "WUS",
        "USC",
        "ZOU",
        "MIC",
        "BCJ",
        "CPC",
        "ASU",
        "CAC",
        "UWM",
        "UCR",
        "GTN",
        "BTU",
        "UTA",
        "UTB",
        "UVA",
        "NCK",
        "UTD",
        "GWU",
        "CUB",
        "OSU",
        "IOW",
        "UCL",
        "DKN",
        "CLA",
        "LBU",
        "UND",
        "TAM",
        "WAS",
        "YLS",
        "UGA",
        "TUF",
        "DAR",
        "NYU",
        "VAN",
        "UCM",
        "UMA",
        "UDE",
        "MIZ",
        "BYU",
        "EMY",
        "RCO",
        "MSL",
        "MCS",
        "HUA",
        "HUB",
        "AMU",
        "UMB/CGU",
        "DMC / HKS",
    ],
)

q_dualcit = QuestionYesNo(
    question_name="dualcit",
    question_text="Are you also a citizen of another country besides the United States?",
)

q_dualctry = QuestionFreeText(
    question_name="dualctry",
    question_text="What country do you hold citizenship with besides the United States?",
    # question_options=[
    #     "Canada", "United Kingdom", "Afghanistan", "Aland Islands", "Albania", "Algeria", "American Samoa", "Andorra", "Angola", "Anguilla",
    #     "Antarctica", "Antigua and Barbuda", "Argentina", "Armenia", "Aruba", "Australia", "Austria", "Azerbaijan", "Bahamas", "Bahrain",
    #     "Bangladesh", "Barbados", "Belarus", "Belgium", "Belize", "Benin", "Bermuda", "Bhutan", "Bolivia", "Bosnia and Herzegovina",
    #     "Botswana", "Bouvet Island", "Brazil", "British Indian Ocean Territory", "Brunei Darussalam", "Bulgaria", "Burkina Faso", "Burundi",
    #     "Cambodia", "Cameroon", "Cape Verde", "Cayman Islands", "Central African Republic", "Chad", "Chile", "China", "Christmas Island",
    #     "Cocos (Keeling) Islands", "Colombia", "Comoros", "Congo", "Congo, the Democratic Republic of the", "Cook Islands", "Costa Rica",
    #     "Cote d'Ivoire", "Croatia", "Cuba", "Cyprus", "Czech Republic", "Denmark", "Djibouti", "Dominica", "Dominican Republic", "Ecuador",
    #     "Egypt", "El Salvador", "Equatorial Guinea", "Eritrea", "Estonia", "Ethiopia", "Falkland Islands (Malvinas)", "Faroe Islands",
    #     "Fiji", "Finland", "France", "French Guiana", "French Polynesia", "French Southern Territories", "Gabon", "Gambia", "Georgia",
    #     "Germany", "Ghana", "Gibraltar", "Greece", "Greenland", "Grenada", "Guadeloupe", "Guam", "Guatemala", "Guernsey", "Guinea",
    #     "Guinea-Bissau", "Guyana", "Haiti", "Heard Island and McDonald Islands", "Holy See (Vatican City State)", "Honduras", "Hong Kong",
    #     "Hungary", "Iceland", "India", "Indonesia", "Iran, Islamic Republic of", "Iraq", "Ireland", "Isle of Man", "Israel", "Italy",
    #     "Jamaica", "Japan", "Jersey", "Jordan", "Kazakhstan", "Kenya", "Kiribati", "Korea, Democratic People's Republic of", "Korea, Republic of",
    #     "Kuwait", "Kyrgyzstan", "Lao People's Democratic Republic", "Latvia", "Lebanon", "Lesotho", "Liberia", "Libyan Arab Jamahiriya",
    #     "Liechtenstein", "Lithuania", "Luxembourg", "Macao", "Macedonia, the former Yugoslav Republic of", "Madagascar", "Malawi", "Malaysia",
    #     "Maldives", "Mali", "Malta", "Marshall Islands", "Martinique", "Mauritania", "Mauritius", "Mayotte", "Mexico", "Micronesia, Federated States of",
    #     "Moldova, Republic of", "Monaco", "Mongolia", "Montserrat", "Morocco", "Mozambique", "Myanmar", "Namibia", "Nauru", "Nepal",
    #     "Netherlands", "Netherlands Antilles", "New Caledonia", "New Zealand", "Nicaragua", "Niger", "Nigeria", "Niue", "Norfolk Island",
    #     "Northern Mariana Islands", "Norway", "Oman", "Pakistan", "Palau", "Palestinian Territory, Occupied", "Panama", "Papua New Guinea",
    #     "Paraguay", "Peru", "Philippines", "Pitcairn", "Poland", "Portugal", "Puerto Rico", "Qatar", "Reunion", "Romania", "Russian Federation",
    #     "Rwanda", "Saint Helena", "Saint Kitts and Nevis", "Saint Lucia", "Saint Pierre and Miquelon", "Saint Vincent and the Grenadines",
    #     "Samoa", "San Marino", "Sao Tome and Principe", "Saudi Arabia", "Senegal", "Serbia and Montenegro", "Seychelles", "Sierra Leone",
    #     "Singapore", "Slovakia", "Slovenia", "Solomon Islands", "Somalia", "South Africa", "South Georgia and the South Sandwich Islands",
    #     "Spain", "Sri Lanka", "Sudan", "Suriname", "Svalbard and Jan Mayen", "Swaziland", "Sweden", "Switzerland", "Syrian Arab Republic",
    #     "Taiwan", "Tajikistan", "Tanzania, United Republic of", "Thailand", "Timor-Leste", "Togo", "Tokelau", "Tonga", "Trinidad and Tobago",
    #     "Tunisia", "Turkey", "Turkmenistan", "Turks and Caicos Islands", "Tuvalu", "Uganda", "Ukraine", "United Arab Emirates",
    #     "United States Minor Outlying Islands", "Uruguay", "Uzbekistan", "Vanuatu", "Venezuela", "Vietnam", "Virgin Islands, British",
    #     "Virgin Islands, U.S.", "Wallis and Futuna", "Western Sahara", "Yemen", "Zambia", "Zimbabwe", "Other"
    # ]
)

q_ownhome = QuestionMultipleChoice(
    question_name="ownhome",
    question_text="Do you own your home or pay rent?",
    question_options=["Own", "Rent", "Other"],
)

q_newsint = QuestionMultipleChoice(
    question_name="newsint",
    question_text="Some people seem to follow what's going on in government and public affairs most of the time, whether there's an election going on or not. Others aren't that interested. Would you say you follow what's going on in government and public affairs ...",
    question_options=[
        "Most of the time",
        "Some of the time",
        "Only now and then",
        "Hardly at all",
        "Don't know",
    ],
)

q_faminc_new = QuestionMultipleChoice(
    question_name="faminc_new",
    question_text="Thinking back over the last year, what was your family's annual income?",
    question_options=[
        "Less than $10,000",
        "$10,000 - $19,999",
        "$20,000 - $29,999",
        "$30,000 - $39,999",
        "$40,000 - $49,999",
        "$50,000 - $59,999",
        "$60,000 - $69,999",
        "$70,000 - $79,999",
        "$80,000 - $99,999",
        "$100,000 - $119,999",
        "$120,000 - $149,999",
        "$150,000 - $199,999",
        "$200,000 - $249,999",
        "$250,000 - $349,999",
        "$350,000 - $499,999",
        "$500,000 or more",
        "Prefer not to say",
    ],
)

q_milstat = QuestionCheckBox(
    question_name="milstat",
    question_text="We'd like to know whether you or someone in your immediate family is currently serving or has ever served in the U.S. military. Immediate family is defined as your parents, siblings, spouse, and children. Please check all boxes that apply.",
    question_options=[
        "I am currently serving in the U.S. military",
        "I have immediate family members currently serving in the U.S. military",
        "I previously served in the U.S. military but I am no longer active",
        "Members of my immediate family have served in the U.S. military but are no longer active",
        "Neither myself nor any members of my immediate family have ever served in the U.S. military",
    ],
)

q_child18 = QuestionYesNo(
    question_name="child18",
    question_text="Are you the parent or guardian of any children under the age of 18?",
)

q_healthins = QuestionCheckBox(
    question_name="healthins",
    question_text="Do you currently have health insurance? (Please check all that apply)",
    question_options=[
        "Yes, through my job or a family member's employer",
        "Yes, through a government program, such as Medicare or Medicaid",
        "Yes, through my school",
        "Yes, I purchased my own",
        "Not sure",
        "No",
    ],
)

q_healthins2 = QuestionYesNo(
    question_name="healthins2",
    question_text="When you purchased health insurance did you use a health insurance exchange?",
)

q_phone = QuestionMultipleChoice(
    question_name="phone",
    question_text="Thinking about your phone service, do you have ...?",
    question_options=[
        "A landline and a cell phone",
        "Cell phone only",
        "Landline only",
        "No landline or cell phone service",
    ],
)

q_internethome = QuestionMultipleChoice(
    question_name="internethome",
    question_text="What best describes the access you have to the internet at home?",
    question_options=["Cable, DSL, or other broadband", "Dial-up", "None"],
)

q_internetwork = QuestionMultipleChoice(
    question_name="internetwork",
    question_text="What best describes the access you have to the internet at work (or at school)?",
    question_options=["Cable, DSL, or other broadband", "Dial-up", "None"],
)

q_CC22_hisp = QuestionCheckBox(
    question_name="CC22_hisp",
    question_text="From which country or region do you trace your heritage or ancestry? (check all that apply)",
    question_options=[
        "No Country in Particular",
        "United States",
        "Mexico",
        "Puerto Rico",
        "Cuba",
        "Dominican Republic",
        "South America",
        "Central America",
        "Caribbean",
        "Spain",
        "Other",
        "I am not of Latino, Hispanic or Spanish Heritage",
    ],
)

q_CC22_asian = QuestionCheckBox(
    question_name="CC22_asian",
    question_text="From which country or region do you trace your heritage or ancestry? (check all that apply)",
    question_options=[
        "No Country in Particular",
        "United States",
        "China",
        "Japan",
        "India",
        "Philippines",
        "Taiwan",
        "Korea",
        "Vietnam",
        "Pakistan",
        "Hmong",
        "Cambodia",
        "Thailand",
        "Other",
        "I am not of Asian Heritage",
    ],
)

q_presvote16post = QuestionMultipleChoice(
    question_name="presvote16post",
    question_text="Who did you vote for in the election for President in 2016?",
    question_options=[
        "Hillary Clinton",
        "Donald Trump",
        "Gary Johnson",
        "Jill Stein",
        "Evan McMullin",
        "Other",
        "Did not vote for President",
    ],
)

q_industry = QuestionMultipleChoice(
    question_name="industry",
    question_text="$employtext",
    question_options=[
        "Agriculture, forestry, fishing, and hunting",
        "Mining",
        "Utilities",
        "Construction",
        "Manufacturing",
        "Professional and business services",
        "Educational services",
        "Health care and social assistance",
        "Leisure and hospitality",
        "Other services",
        "Wholesale trade",
        "Retail trade",
        "Transportation and warehousing",
        "Information",
        "Financial activities",
        "Federal government",
        "State and local government",
    ],
)

q_sexuality = QuestionMultipleChoice(
    question_name="sexuality",
    question_text="Which of the following best describes your sexuality?",
    question_options=[
        "Heterosexual / straight",
        "Lesbian / gay woman",
        "Gay man",
        "Bisexual",
        "Other",
        "Prefer not to say",
    ],
)

q_transgender = QuestionMultipleChoice(
    question_name="transgender",
    question_text="Do you identify as transgender?",
    question_options=["Yes", "No", "Prefer not to say"],
)
```

## Combining questions into a `Survey`

A `Survey` object is created by passing a list of `Question` objects. Learn more about [constructing surveys in EDSL](/en/latest/surveys).

```python expandable theme={null}
survey_ces22_pre_election_questionnaire = Survey(
    [
        q_CCEStake,
        q_add_confirm,
        q_inputzip,
        q_birthyr,
        q_gender4,
        q_educ,
        q_race,
        q_hispanic,
        q_multrace,
        q_comptype,
        q_votereg,
        q_votereg_f,
        q_pid3,
        q_pid7,
        q_inputstate,
        q_region,
        q_CC22_300,
        q_CC22_300a,
        q_CC22_300c,
        q_CC22_300b,
        q_CC22_300d,
        q_CC19_302,
        q_CC22_303,
        q_CC22_304,
        q_CC22_305,
        q_CC22_307,
        q_CC22_309a,
        q_CC22_306,
        q_CC22_309b,
        q_CC22_309c,
        q_CC22_309dx,
        q_CC22_309e,
        q_CC22_309f,
        q_CC22_310a,
        q_CC22_310b,
        q_CC22_310c,
        q_CC22_310d,
        q_CC22_311a,
        q_CC22_311b,
        q_CC22_311c,
        q_CC22_311d,
        q_CC22_320a,
        q_CC22_320b,
        q_CC22_320c,
        q_CC22_320d,
        q_CC22_320e,
        q_CC22_320f,
        q_CC22_320g,
        q_CC22_320h,
        q_cit1,
        q_immstat,
        q_CC22_321,
        q_CC22_327a,
        q_CC22_327b,
        q_CC22_327c,
        q_CC22_327d,
        q_CC22_330a,
        q_CC22_330b,
        q_CC22_330c,
        q_CC22_330d,
        q_CC22_330e,
        q_CC22_330f,
        q_CC22_331a,
        q_CC22_331b,
        q_CC22_331c,
        q_CC22_331d,
        q_CC22_332a,
        q_CC22_332b,
        q_CC22_332c,
        q_CC22_332d,
        q_CC22_332e,
        q_CC22_332f,
        q_CC22_333,
        q_CC22_333a,
        q_CC22_333b,
        q_CC22_333c,
        q_CC22_333d,
        q_CC22_333e,
        q_CC22_334a,
        q_CC22_334b,
        q_CC22_334c,
        q_CC22_334d,
        q_CC22_334e,
        q_CC22_334f,
        q_CC22_334g,
        q_CC22_334h,
        q_CC22_340a,
        q_CC22_340b,
        q_CC22_340c,
        q_CC22_340d,
        q_CC22_340e,
        q_CC22_340f,
        q_CC22_340g,
        q_CC22_340h,
        q_CC22_340i,
        q_CC22_340j,
        q_CC22_340k,
        q_CC22_340l,
        q_CC22_340m,
        q_CC22_340n,
        q_CC22_350a,
        q_CC22_350b,
        q_CC22_350c,
        q_CC22_350d,
        q_CC22_350e,
        q_CC22_350f,
        q_CC22_350g,
        q_CC22_350h,
        q_CC22_355a,
        q_CC22_355b,
        q_CC22_355c,
        q_CC22_355d,
        q_CC22_355e,
        q_CC22_360,
        q_CC22_361,
        q_urbancity,
        q_presvote20post,
        q_CC22_363,
        q_CC22_365_voted,
        q_CC22_365b_voted,
        q_CC22_366_voted,
        q_CC22_367_voted,
        q_CC22_365,
        q_CC22_365a,
        q_CC22_365b,
        q_CC22_365c,
        q_CC22_366,
        q_CC22_367,
        q_CC22_367a,
        q_ideo5,
        q_employ,
        q_hadjob,
        q_investor,
        q_pew_bornagain,
        q_pew_religimp,
        q_pew_churatd,
        q_pew_prayer,
        q_religpew,
        q_religpew_protestant,
        q_Xreligpew_protestant,
        q_religpew_baptist,
        q_religpew_methodist,
        q_religpew_nondenom,
        q_religpew_lutheran,
        q_religpew_presby,
        q_religpew_pentecost,
        q_religpew_episcop,
        q_religpew_christian,
        q_religpew_congreg,
        q_religpew_holiness,
        q_religpew_reformed,
        q_religpew_advent,
        q_religpew_catholic,
        q_religpew_mormon,
        q_religpew_orthodox,
        q_religpew_jewish,
        q_religpew_muslim,
        q_religpew_buddhist,
        q_religpew_hindu,
        q_marstat,
        q_union,
        q_union_coverage,
        q_unionhh,
        q_ccesmodule,
        q_dualcit,
        q_dualctry,
        q_ownhome,
        q_newsint,
        q_faminc_new,
        q_milstat,
        q_child18,
        q_healthins,
        q_healthins2,
        q_phone,
        q_internethome,
        q_internetwork,
        q_CC22_hisp,
        q_CC22_asian,
        q_presvote16post,
        q_industry,
        q_sexuality,
        q_transgender,
    ]
)
```

### Dictionary of question short names & texts

Here we create a dictionary of the question short names and texts, for convenience in working with the data:

```python expandable theme={null}
questions = {
    "caseid": "caseid",
    "CCEStake": "Do you agree to participate in the study?",
    "add_confirm": "Is the name and address displayed above correct?",
    "inputzip": "So that we can ask you about the news and events in your area, in what zip code do you currently reside?",
    "birthyr": "In what year were you born?",
    "gender4": "What is your gender?",
    "educ": "What is the highest level of education you have completed?",
    "race": "What racial or ethnic group best describes you?",
    "hispanic": "Are you of Spanish, Latino, or Hispanic origin or descent?",
    "multrace": "Please indicate the racial or ethnic groups that best describe you?",
    "comptype": "What type of device are you currently taking this survey on?",
    "votereg": "Are you registered to vote?",
    "votereg_f": "Is $izip the zip code where you are registered to vote?",
    "pid3": "Generally speaking, do you think of yourself as a ...?",
    "pid7": "$pid7text",
    "inputstate": "What is your State of Residence?",
    "region": "In which census region do you live?",
    "CC22_300": "In the past 24 hours have you...",
    "CC22_300a": "Did you watch local news, national news, or both?",
    "CC22_300c": "Did you read a print newspaper, an online newspaper, or both?",
    "CC22_300b": "Which of these networks did you watch?",
    "CC22_300d": "In the past 24 hours, did you do any of the following on social media (such as Facebook, Youtube or Twitter)?",
    "CC19_302": "Would you say that OVER THE PAST YEAR the nation's economy has ...",
    "CC22_303": "OVER THE PAST YEAR, has your household's annual income...?",
    "CC22_304": "OVER THE PAST YEAR, have the prices of everyday goods and services...?",
    "CC22_305": "Over the past year have you...",
    "CC22_307": "Do the police make you feel...?",
    "CC22_309a": "Have you or someone you know been diagnosed with the novel coronavirus (COVID-19) during the past year?",
    "CC22_306": "Which of the following best describes you when it comes to being vaccinated against COVID-19?",
    "CC22_309b": "Do you know anyone who died from the novel coronavirus (COVID-19)?",
    "CC22_309c": "How did your work status change as a result of the coronavirus pandemic?",
    "CC22_309dx": "Suppose that you have an emergency expense that costs $400. Based on your current financial situation, how would you pay for this expense? If you would use more than one method to cover this expense, please select all that apply.",
    "CC22_309e": "Would you say that in general your health is...",
    "CC22_309f": "Would you say that in general your mental health is...",
    "CC22_310a": "Which party has a majority of seats in the U.S. House of Representatives?",
    "CC22_310b": "Which party has a majority of seats in the U.S. Senate?",
    "CC22_310c": "Which party has a majority of seats in the $inputstate State Senate?",
    "CC22_310d": "Which party has a majority of seats in the $LowerChamberName?",
    "CC22_311a": "Please indicate whether you've heard of this person and if so which party he or she is affiliated with: $CurrentGovName",
    "CC22_311b": "Please indicate whether you've heard of this person and if so which party he or she is affiliated with: $CurrentSen1Name",
    "CC22_311c": "Please indicate whether you've heard of this person and if so which party he or she is affiliated with: $CurrentSen2Name",
    "CC22_311d": "Please indicate whether you've heard of this person and if so which party he or she is affiliated with: $CurrentHouseName",
    "CC22_320a": "Do you approve of the way President Biden is doing his job?",
    "CC22_320b": "Do you approve of the way the U.S. Congress is doing its job?",
    "CC22_320c": "Do you approve of the way the U.S. Supreme Court is doing its job?",
    "CC22_320d": "Do you approve of the way the Governor of $inputstate is doing their job?",
    "CC22_320e": "Do you approve of the way $LegName is doing their job?",
    "CC22_320f": "Do you approve of the way $CurrentHouseName is doing their job?",
    "CC22_320g": "Do you approve of the way $CurrentSen1Name is doing their job?",
    "CC22_320h": "Do you approve of the way $CurrentSen2Name is doing their job?",
    "cit1": "Are you a United States citizen?",
    "immstat": "Which of these statements best describes you?",
    "CC22_321": "What do you think the United States should do in response to Russia's invasion of Ukraine?",
    "CC22_327a": "Would you support or oppose the proposal to expand Medicare to a single comprehensive public health care coverage program that would cover all Americans?",
    "CC22_327b": "Would you support or oppose the proposal to allow the government to negotiate with drug companies to get a lower price on prescription drugs that would apply to both Medicare and private insurance?",
    "CC22_327c": "Would you support or oppose the proposal to repeal the entire Affordable Care Act?",
    "CC22_327d": "Would you support or oppose the proposal to allow states to import prescription drugs from other countries?",
    "CC22_330a": "On the issue of gun regulation, do you support or oppose the proposal to prohibit state and local governments from publishing the names and addresses of all gun owners?",
    "CC22_330b": "On the issue of gun regulation, do you support or oppose the proposal to ban assault rifles?",
    "CC22_330c": "On the issue of gun regulation, do you support or oppose the proposal to make it easier for people to obtain concealed-carry permits?",
    "CC22_330d": "On the issue of gun regulation, do you support or oppose the proposal to provide federal funding to encourage states to take guns away from people who already own them but might pose a threat to themselves or others?",
    "CC22_330e": "On the issue of gun regulation, do you support or oppose the proposal to improve background checks to give authorities time to check the juvenile and mental health records of any prospective gun buyer under the age of 21?",
    "CC22_330f": "On the issue of gun regulation, do you support or oppose the proposal to allow teachers and school officials to carry guns in public schools?",
    "CC22_331a": "What do you think the U.S. government should do about immigration? Do you support or oppose the proposal to grant legal status to all illegal immigrants who have held jobs and paid taxes for at least 3 years, and not been convicted of any felony crimes?",
    "CC22_331b": "What do you think the U.S. government should do about immigration? Do you support or oppose the proposal to increase the number of border patrols on the US-Mexican border?",
    "CC22_331c": "What do you think the U.S. government should do about immigration? Do you support or oppose the proposal to reduce legal immigration by 50 percent over the next 10 years by eliminating the visa lottery and ending family-based migration?",
    "CC22_331d": "What do you think the U.S. government should do about immigration? Do you support or oppose the proposal to increase spending on border security by $25 billion, including building a wall between the U.S. and Mexico?",
    "CC22_332a": "On the topic of abortion, do you support or oppose the proposal to always allow a woman to obtain an abortion as a matter of choice?",
    "CC22_332b": "On the topic of abortion, do you support or oppose the proposal to permit abortion only in case of rape, incest or when the woman's life is in danger?",
    "CC22_332c": "On the topic of abortion, do you support or oppose the proposal to prohibit all abortions after the 20th week of pregnancy?",
    "CC22_332d": "On the topic of abortion, do you support or oppose the proposal to allow employers to decline coverage of abortions in insurance plans?",
    "CC22_332e": "On the topic of abortion, do you support or oppose the proposal to prohibit the expenditure of funds authorized or appropriated by federal law for any abortion?",
    "CC22_332f": "On the topic of abortion, do you support or oppose the proposal to make abortions illegal in all circumstances?",
    "CC22_333": "From what you know about global climate change or global warming, which one of the following statements comes closest to your opinion?",
    "CC22_333a": "Do you support or oppose the proposal to give the Environmental Protection Agency power to regulate carbon dioxide emissions?",
    "CC22_333b": "Do you support or oppose the proposal to require that each state use a minimum amount of renewable fuels (wind, solar, and hydroelectric) in the generation of electricity even if electricity prices increase a little?",
    "CC22_333c": "Do you support or oppose the proposal to strengthen the Environmental Protection Agency enforcement of the Clean Air Act and Clean Water Act even if it costs U.S. jobs?",
    "CC22_333d": "Do you support or oppose the proposal to raise the average fuel efficiency for all cars and trucks in the US from 40 miles per gallon to 54.5 miles per gallon by 2025?",
    "CC22_333e": "Do you support or oppose the proposal to increase fossil fuel production in the U.S. and boost exports of U.S. liquefied natural gas?",
    "CC22_334a": "Do you support or oppose the proposal to eliminate mandatory minimum sentences for non-violent drug offenders?",
    "CC22_334b": "Do you support or oppose the proposal to require police officers to wear body cameras that record all of their activities while on duty?",
    "CC22_334c": "Do you support or oppose the proposal to increase the number of police on the street by 10 percent, even if it means fewer funds for other public services?",
    "CC22_334d": "Do you support or oppose the proposal to decrease the number of police on the street by 10 percent, and increase funding for other public services?",
    "CC22_334e": "Do you support or oppose the proposal to ban the use of choke holds by police?",
    "CC22_334f": "Do you support or oppose the proposal to create a national registry of police who have been investigated for or disciplined for misconduct?",
    "CC22_334g": "Do you support or oppose the proposal to end the Department of Defense program that sends surplus military weapons and equipment to police departments?",
    "CC22_334h": 'Do you support or oppose the proposal to allow individuals or their families to sue a police officer for damages if the officer is found to have "recklessly disregarded" the individual's rights?',
    "CC22_340a": "How would you rate Yourself?",
    "CC22_340b": "How would you rate $CurrentGovName?",
    "CC22_340c": "How would you rate Joe Biden?",
    "CC22_340d": "How would you rate Donald Trump?",
    "CC22_340e": "How would you rate The Democratic Party?",
    "CC22_340f": "How would you rate The Republican Party?",
    "CC22_340g": "How would you rate The U.S. Supreme Court?",
    "CC22_340h": "How would you rate $CurrentSen1Name?",
    "CC22_340i": "How would you rate $CurrentSen2Name?",
    "CC22_340j": "How would you rate $SenCand1Name?",
    "CC22_340k": "How would you rate $SenCand2Name?",
    "CC22_340l": "How would you rate $HouseCand1Name?",
    "CC22_340m": "How would you rate $HouseCand2Name?",
    "CC22_340n": "How would you rate $CurrentHouseName?",
    "CC22_350a": "Over the past two years, Congress voted on many issues. Do you support the proposal to authorize spending up to $1.9 trillion for COVID relief from March 2021 through September 2021, including extension of unemployment benefits through September 2021, and emergency funding to state and local governments for the fiscal year?",
    "CC22_350b": "Over the past two years, Congress voted on many issues. Do you support the proposal to spend $150 billion a year for 8 years on construction and repair of roads and bridges, rail, public transit, airports, water systems and broadband internet?",
    "CC22_350c": "Over the past two years, Congress voted on many issues. Do you support the proposal to spend $2.2 trillion over the next decade to provide universal prekindergarten, subsidies for child care, expanded financial aid for college, housing support, home and community care for older Americans, and to shift the U.S. economy away from fossil fuels to renewable energy and electric cars?",
    "CC22_350d": "Over the past two years, Congress voted on many issues. Do you support the proposal to prohibit government restrictions on the provision of, and access to, abortion services?",
    "CC22_350e": "Over the past two years, Congress voted on many issues. Do you support the proposal to provide $52 billion in grants for American semiconductor manufacturing and research and a tax credit subsidizing 25% of investments in semiconductor manufacturing?",
    "CC22_350f": "Over the past two years, Congress voted on many issues. Do you support the proposal to prohibit large online platforms from giving preference to their own products on the platform at the expense of competing products from another business?",
    "CC22_350g": "Over the past two years, Congress voted on many issues. Do you support the proposal to appoint Ketanji Brown Jackson to the U.S. Supreme Court?",
    "CC22_350h": "Over the past two years, Congress voted on many issues. Do you support the proposal to spend $369 billion for tax credits to encourage the production of solar panels, wind turbines, and batteries; lowers Affordable Care Act health care premiums; reduces the deficit by $300 billion by allowing Medicare to negotiate the cost of some prescription drugs and making changes to the tax code?",
    "CC22_355a": "For each of the following tell us whether you support or oppose the decision for the United States to re-join the Paris Climate Agreement?",
    "CC22_355b": "For each of the following tell us whether you support or oppose the decision for the United States to re-join the World Health Organization?",
    "CC22_355c": "For each of the following tell us whether you support or oppose the decision to order all federal agencies to buy clean energy, purchase electric vehicles, and make federal buildings energy efficient?",
    "CC22_355d": "For each of the following tell us whether you support or oppose the decision to increase the minimum wage paid to federal contractors to $15 an hour?",
    "CC22_355e": "For each of the following tell us whether you support or oppose the decision to require that all employees at large companies be vaccinated?",
    "CC22_360": "With which party, if any, are you registered?",
    "CC22_361": "How long have you lived at your present address?",
    "urbancity": "How would you describe the place where you live?",
    "presvote20post": "Who did you vote for in the election for President in 2020?",
    "CC22_363": "Do you intend to vote in the 2022 general election on November 8th?",
    "CC22_365_voted": "For which candidate for U.S. Senator did you vote?",
    "CC22_365b_voted": "For which candidate for the special election for U.S. Senate did you vote?",
    "CC22_366_voted": "For which candidate for Governor did you vote?",
    "CC22_367_voted": "For which candidate for U.S. House of Representatives in your area did you vote?",
    "CC22_365": "In the race for U.S. Senator in your state, who do you prefer?",
    "CC22_365a": "Who is your second choice for U.S. Senator?",
    "CC22_365b": "In the special election for U.S. Senator in your state, who do you prefer?",
    "CC22_365c": "Who is your second choice for U.S. Senator?",
    "CC22_366": "In the race for Governor in your state, who do you prefer?",
    "CC22_367": "In the general election for U.S. House of Representatives in your area, who do you prefer?",
    "CC22_367a": "Who is your second choice for U.S. House of Representatives?",
    "ideo5": "In general, how would you describe your own political viewpoint?",
    "employ": "Which of the following best describes your current employment status?",
    "hadjob": "At any time over the past five years, have you had a job?",
    "investor": "Do you personally (or jointly with a spouse), have any money invested in the stock market right now, either in an individual stock or in a mutual fund?",
    "pew_bornagain": 'Would you describe yourself as a "born-again" or evangelical Christian, or not?',
    "pew_religimp": "How important is religion in your life?",
    "pew_churatd": "Aside from weddings and funerals, how often do you attend religious services?",
    "pew_prayer": "People practice their religion in different ways. Outside of attending religious services, how often do you pray?",
    "religpew": "What is your present religion, if any?",
    "religpew_protestant": "To which Protestant church or group do you belong?",
    "Xreligpew_protestant": "Do you belong to any one of these churches or groups?",
    "religpew_baptist": "To which Baptist church do you belong, if any?",
    "religpew_methodist": "To which Methodist church do you belong, if any?",
    "religpew_nondenom": "To which kind of nondenominational or independent church do you belong, if any?",
    "religpew_lutheran": "To which Lutheran church do you belong?",
    "religpew_presby": "To which Presbyterian church do you belong?",
    "religpew_pentecost": "To which Pentecostal church do you belong?",
    "religpew_episcop": "To which Episcopalian church do you belong?",
    "religpew_christian": "To which Christian church do you belong?",
    "religpew_congreg": "To which congregational church do you belong?",
    "religpew_holiness": "To which Holiness church do you belong?",
    "religpew_reformed": "To which Reformed church do you belong?",
    "religpew_advent": "To which Adventist church do you belong?",
    "religpew_catholic": "To which Catholic church do you belong?",
    "religpew_mormon": "To which Mormon church do you belong?",
    "religpew_orthodox": "To which Orthodox church do you belong?",
    "religpew_jewish": "To which Jewish group do you belong?",
    "religpew_muslim": "To which Muslim group do you belong?",
    "religpew_buddhist": "To which Buddhist group do you belong?",
    "religpew_hindu": "With which of the following Hindu groups, if any, do you identify with most closely?",
    "marstat": "What is your marital status?",
    "union": "Are you a member of a labor union?",
    "union_coverage": "Are you covered by a union contract, also known as a collective bargaining agreement?",
    "unionhh": "Other than yourself, is any member of your household a union member?",
    "ccesmodule": "Survey assigned",
    "dualcit": "Are you also a citizen of another country besides the United States?",
    "dualctry": "What country do you hold citizenship with besides the United States?",
    "ownhome": "Do you own your home or pay rent?",
    "newsint": "Some people seem to follow what's going on in government and public affairs most of the time, whether there's an election going on or not. Others aren't that interested. Would you say you follow what's going on in government and public affairs ...",
    "faminc_new": "Thinking back over the last year, what was your family's annual income?",
    "milstat": "We'd like to know whether you or someone in your immediate family is currently serving or has ever served in the U.S. military. Immediate family is defined as your parents, siblings, spouse, and children. Please check all boxes that apply.",
    "child18": "Are you the parent or guardian of any children under the age of 18?",
    "healthins": "Do you currently have health insurance?",
    "healthins2": "When you purchased health insurance did you use a health insurance exchange?",
    "phone": "Thinking about your phone service, do you have ...?",
    "internethome": "What best describes the access you have to the internet at home?",
    "internetwork": "What best describes the access you have to the internet at work (or at school)?",
    "CC22_hisp": "From which country or region do you trace your heritage or ancestry?",
    "CC22_asian": "From which country or region do you trace your heritage or ancestry?",
    "presvote16post": "Who did you vote for in the election for President in 2016?",
    "industry": "$employtext",
    "sexuality": "Which of the following best describes your sexuality?",
    "transgender": "Do you identify as transgender?",
}
```

### Answer codebook

Here we create a codebook of the question short names and the numbers and texts of the answer options, for convenience in working with the data and responses. Note that most answer option numbers are consecutive (1,2,3,…) but some are not.

```python expandable theme={null}
codebook = {
    "caseid": {},
    "CCEStake": {1: "Yes", 2: "No"},
    "add_confirm": {1: "Yes", 2: "No"},
    "inputzip": {},
    "birthyr": {},
    "gender4": {1: "Man", 2: "Woman", 3: "Non-binary", 4: "Other"},
    "educ": {
        1: "Did not graduate from high school",
        2: "High school graduate",
        3: "Some college, but no degree (yet)",
        4: "2-year college degree",
        5: "4-year college degree",
        6: "Postgraduate degree (MA, MBA, MD, JD, PhD, etc.)",
    },
    "race": {
        1: "White",
        2: "Black or African-American",
        3: "Hispanic or Latino",
        4: "Asian or Asian-American",
        5: "Native American",
        8: "Middle Eastern",
        6: "Two or more races",
        7: "Other",
    },
    "hispanic": {1: "Yes", 2: "No"},
    "multrace": {
        1: "White",
        2: "Black or African-American",
        3: "Hispanic or Latino",
        4: "Asian or Asian-American",
        5: "Native American",
        8: "Middle Eastern",
        97: "Other",
        98: "Don't know",
        99: "None of these",
    },
    "comptype": {
        1: "I am taking this survey on a smart phone (e.g., iPhone or Android phone)",
        2: "I am taking this survey on a tablet (e.g., iPad)",
        3: "I am taking this survey on a desktop computer or laptop computer",
    },
    "votereg": {1: "Yes", 2: "No", 3: "Don't know"},
    "votereg_f": {1: "Yes", 2: "No"},
    "pid3": {
        1: "Democrat",
        2: "Republican",
        3: "Independent",
        4: "Other",
        5: "Not sure",
    },
    "pid7": {
        1: "Strong Democrat",
        2: "Not very strong Democrat",
        7: "Strong Republican",
        6: "Not very strong Republican",
        3: "The Democratic Party",
        5: "The Republican Party",
        4: "Neither",
        8: "Not sure",
        9: "Don't know",
    },
    "inputstate": {
        1: "Alabama",
        2: "Alaska",
        4: "Arizona",
        5: "Arkansas",
        6: "California",
        8: "Colorado",
        9: "Connecticut",
        10: "Delaware",
        11: "District of Columbia",
        12: "Florida",
        13: "Georgia",
        15: "Hawaii",
        16: "Idaho",
        17: "Illinois",
        18: "Indiana",
        19: "Iowa",
        20: "Kansas",
        21: "Kentucky",
        22: "Louisiana",
        23: "Maine",
        24: "Maryland",
        25: "Massachusetts",
        26: "Michigan",
        27: "Minnesota",
        28: "Mississippi",
        29: "Missouri",
        30: "Montana",
        31: "Nebraska",
        32: "Nevada",
        33: "New Hampshire",
        34: "New Jersey",
        35: "New Mexico",
        36: "New York",
        37: "North Carolina",
        38: "North Dakota",
        39: "Ohio",
        40: "Oklahoma",
        41: "Oregon",
        42: "Pennsylvania",
        44: "Rhode Island",
        45: "South Carolina",
        46: "South Dakota",
        47: "Tennessee",
        48: "Texas",
        49: "Utah",
        50: "Vermont",
        51: "Virginia",
        53: "Washington",
        54: "West Virginia",
        55: "Wisconsin",
        56: "Wyoming",
    },
    "region": {1: "Northeast", 2: "Midwest", 3: "South", 4: "West"},
    "CC22_300": {
        1: "Used social media (such as Facebook or Youtube)",
        2: "Watched TV news",
        3: "Read a newspaper in print or online",
        4: "Listened to a radio news program or talk radio",
        5: "None of these",
    },
    "CC22_300a": {1: "Local Newscast", 2: "National Newscast", 3: "Both"},
    "CC22_300c": {1: "Print", 2: "Online", 3: "Both"},
    "CC22_300b": {
        1: "ABC",
        2: "CBS",
        3: "NBC",
        4: "CNN",
        5: "Fox News",
        6: "MSNBC",
        7: "PBS",
        8: "Other",
    },
    "CC22_300d": {
        1: "Posted a story, photo, video or link about politics",
        2: "Posted a comment about politics",
        3: "Read a story or watched a video about politics",
        4: "Followed a political event",
        5: "Forwarded a story, photo, video or link about politics to friends",
        6: "None of the above",
    },
    "CC19_302": {
        1: "Gotten much better",
        2: "Gotten somewhat better",
        3: "Stayed about the same",
        4: "Gotten somewhat worse",
        5: "Gotten much worse",
        6: "Not sure",
    },
    "CC22_303": {
        1: "Increased a lot",
        2: "Increased somewhat",
        3: "Stayed about the same",
        4: "Decreased somewhat",
        5: "Decreased a lot",
    },
    "CC22_304": {
        1: "Increased a lot",
        2: "Increased somewhat",
        3: "Stayed about the same",
        4: "Decreased somewhat",
        5: "Decreased a lot",
    },
    "CC22_305": {
        1: "Married",
        2: "Lost a job",
        3: "Finished school",
        4: "Retired",
        5: "Divorced",
        6: "Had a child",
        7: "Taken a new job",
        9: "Been a victim of a crime",
        10: "Visited an emergency room",
        11: "Visited a doctor for a regular examination",
        12: "Received a raise at work",
        13: "Had a pay cut at work",
    },
    "CC22_307": {
        1: "Mostly safe",
        2: "Somewhat safe",
        3: "Somewhat unsafe",
        4: "Mostly unsafe",
    },
    "CC22_309a": {
        1: "Yes, I have",
        2: "Yes, a family member",
        3: "Yes, a friend",
        4: "Yes, a co-worker",
        5: "No, I do not know anyone who has been diagnosed",
    },
    "CC22_306": {
        1: "I am fully vaccinated and have received at least one booster shot",
        2: "I am fully vaccinated but have not received a booster shot",
        3: "I am partially vaccinated (I have received the first of two shots for either Pfizer or Moderna)",
        4: "I am not vaccinated at all",
    },
    "CC22_309b": {
        1: "Yes, a family member",
        2: "Yes, a friend",
        3: "Yes, a co-worker",
        4: "No, I do not know anyone who has died from coronavirus",
    },
    "CC22_309c": {
        1: "My hours have been reduced",
        2: "My hours were reduced, but they have been restored",
        3: "I have been temporarily laid off",
        4: "I was temporarily laid off but have now been re-hired",
        5: "I had more than one job before the pandemic, and lost one of them",
        6: "I lost my job",
        7: "I was not working when the pandemic began",
        8: "My hours have increased",
        9: "I have taken additional jobs since the pandemic",
        10: "No, nothing about my work has changed",
    },
    "CC22_309dx": {
        1: "Put it on my credit card and pay it off in full at the next statement",
        2: "Put it on my credit card and pay it off over time",
        3: "With the money currently in my checking/savings account or with cash",
        4: "Using money from a bank loan or line of credit",
        5: "By borrowing from a friend or family member",
        6: "Using a payday loan, deposit advance, or overdraft",
        7: "By selling something",
        8: "I wouldn't be able to pay for the expense right now",
        9: "Other",
    },
    "CC22_309e": {1: "Excellent", 2: "Very good", 3: "Good", 4: "Fair", 5: "Poor"},
    "CC22_309f": {1: "Excellent", 2: "Very good", 3: "Good", 4: "Fair", 5: "Poor"},
    "CC22_310a": {1: "Democrats", 2: "Republicans", 3: "Neither", 4: "Not sure"},
    "CC22_310b": {1: "Democrats", 2: "Republicans", 3: "Neither", 4: "Not sure"},
    "CC22_310c": {1: "Democrats", 2: "Republicans", 3: "Neither", 4: "Not sure"},
    "CC22_310d": {1: "Democrats", 2: "Republicans", 3: "Neither", 4: "Not sure"},
    "CC22_311a": {
        1: "Never heard of person",
        2: "Republican",
        3: "Democrat",
        4: "Other Party / Independent",
        5: "Not sure",
    },
    "CC22_311b": {
        1: "Never heard of person",
        2: "Republican",
        3: "Democrat",
        4: "Other Party / Independent",
        5: "Not sure",
    },
    "CC22_311c": {
        1: "Never heard of person",
        2: "Republican",
        3: "Democrat",
        4: "Other Party / Independent",
        5: "Not sure",
    },
    "CC22_311d": {
        1: "Never heard of person",
        2: "Republican",
        3: "Democrat",
        4: "Other Party / Independent",
        5: "Not sure",
    },
    "CC22_320a": {
        1: "Strongly approve",
        2: "Somewhat approve",
        3: "Somewhat disapprove",
        4: "Strongly disapprove",
        5: "Not sure",
    },
    "CC22_320b": {
        1: "Strongly approve",
        2: "Somewhat approve",
        3: "Somewhat disapprove",
        4: "Strongly disapprove",
        5: "Not sure",
    },
    "CC22_320c": {
        1: "Strongly approve",
        2: "Somewhat approve",
        3: "Somewhat disapprove",
        4: "Strongly disapprove",
        5: "Not sure",
    },
    "CC22_320d": {
        1: "Strongly approve",
        2: "Somewhat approve",
        3: "Somewhat disapprove",
        4: "Strongly disapprove",
        5: "Not sure",
    },
    "CC22_320e": {
        1: "Strongly approve",
        2: "Somewhat approve",
        3: "Somewhat disapprove",
        4: "Strongly disapprove",
        5: "Not sure",
    },
    "CC22_320f": {
        1: "Strongly approve",
        2: "Somewhat approve",
        3: "Somewhat disapprove",
        4: "Strongly disapprove",
        5: "Not sure",
    },
    "CC22_320g": {
        1: "Strongly approve",
        2: "Somewhat approve",
        3: "Somewhat disapprove",
        4: "Strongly disapprove",
        5: "Not sure",
    },
    "CC22_320h": {
        1: "Strongly approve",
        2: "Somewhat approve",
        3: "Somewhat disapprove",
        4: "Strongly disapprove",
        5: "Not sure",
    },
    "cit1": {1: "Yes", 2: "No"},
    "immstat": {
        1: "I am an immigrant to the USA and a naturalized citizen",
        2: "I am an immigrant to the USA but not a citizen",
        3: "I was born in the USA but at least one of my parents is an immigrant",
        4: "My parents and I were born in the USA but at least one of my grandparents was an immigrant",
        5: "My parents, grandparents and I were all born in the USA",
    },
    "CC22_321": {
        1: "Not sure",
        2: "Do not get involved",
        3: "Send food, medicine and other aid to countries affected",
        4: "Provide arms to Ukraine",
        5: "Enforce a no fly zone",
        6: "Use drones and air craft to bomb Russian troops",
        7: "Send military support staff (non-combat)",
        8: "Send significant force to fight Russia",
    },
    "CC22_327a": {1: "Support", 2: "Oppose"},
    "CC22_327b": {1: "Support", 2: "Oppose"},
    "CC22_327c": {1: "Support", 2: "Oppose"},
    "CC22_327d": {1: "Support", 2: "Oppose"},
    "CC22_330a": {1: "Support", 2: "Oppose"},
    "CC22_330b": {1: "Support", 2: "Oppose"},
    "CC22_330c": {1: "Support", 2: "Oppose"},
    "CC22_330d": {1: "Support", 2: "Oppose"},
    "CC22_330e": {1: "Support", 2: "Oppose"},
    "CC22_330f": {1: "Support", 2: "Oppose"},
    "CC22_331a": {1: "Support", 2: "Oppose"},
    "CC22_331b": {1: "Support", 2: "Oppose"},
    "CC22_331c": {1: "Support", 2: "Oppose"},
    "CC22_331d": {1: "Support", 2: "Oppose"},
    "CC22_332a": {1: "Support", 2: "Oppose"},
    "CC22_332b": {1: "Support", 2: "Oppose"},
    "CC22_332c": {1: "Support", 2: "Oppose"},
    "CC22_332d": {1: "Support", 2: "Oppose"},
    "CC22_332e": {1: "Support", 2: "Oppose"},
    "CC22_332f": {1: "Support", 2: "Oppose"},
    "CC22_333": {
        1: "Global climate change has been established as a serious problem, and immediate action is necessary",
        2: "There is enough evidence that climate change is taking place and some action should be taken",
        3: "We don't know enough about global climate change, and more research is necessary before we take any actions",
        4: "Concern about global climate change is exaggerated. No action is necessary",
        5: "Global climate change is not occurring; this is not a real issue",
    },
    "CC22_333a": {1: "Support", 2: "Oppose"},
    "CC22_333b": {1: "Support", 2: "Oppose"},
    "CC22_333c": {1: "Support", 2: "Oppose"},
    "CC22_333d": {1: "Support", 2: "Oppose"},
    "CC22_333e": {1: "Support", 2: "Oppose"},
    "CC22_334a": {1: "Support", 2: "Oppose"},
    "CC22_334b": {1: "Support", 2: "Oppose"},
    "CC22_334c": {1: "Support", 2: "Oppose"},
    "CC22_334d": {1: "Support", 2: "Oppose"},
    "CC22_334e": {1: "Support", 2: "Oppose"},
    "CC22_334f": {1: "Support", 2: "Oppose"},
    "CC22_334g": {1: "Support", 2: "Oppose"},
    "CC22_334h": {1: "Support", 2: "Oppose"},
    "CC22_340a": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340b": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340c": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340d": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340e": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340f": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340g": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340h": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340i": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340j": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340k": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340l": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340m": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_340n": {
        1: "Very liberal",
        2: "Liberal",
        3: "Somewhat liberal",
        4: "Middle of the road",
        5: "Somewhat conservative",
        6: "Conservative",
        7: "Very conservative",
        8: "Not sure",
    },
    "CC22_350a": {1: "Favor", 2: "Oppose"},
    "CC22_350b": {1: "Favor", 2: "Oppose"},
    "CC22_350c": {1: "Favor", 2: "Oppose"},
    "CC22_350d": {1: "Favor", 2: "Oppose"},
    "CC22_350e": {1: "Favor", 2: "Oppose"},
    "CC22_350f": {1: "Favor", 2: "Oppose"},
    "CC22_350g": {1: "Favor", 2: "Oppose"},
    "CC22_350h": {1: "Favor", 2: "Oppose"},
    "CC22_355a": {1: "Support", 2: "Oppose"},
    "CC22_355b": {1: "Support", 2: "Oppose"},
    "CC22_355c": {1: "Support", 2: "Oppose"},
    "CC22_355d": {1: "Support", 2: "Oppose"},
    "CC22_355e": {1: "Support", 2: "Oppose"},
    "CC22_360": {
        1: "No Party, Independent, Declined to State",
        2: "Democratic Party",
        3: "Republican Party",
        4: "Other",
    },
    "CC22_361": {
        1: "Less than 1 month",
        2: "2 to 6 months",
        3: "7 to 11 months",
        4: "1 to 2 years",
        5: "3 to 4 years",
        6: "5 or more years",
    },
    "urbancity": {1: "City", 2: "Suburb", 3: "Town", 4: "Rural area", 5: "Other"},
    "presvote20post": {
        1: "Joe Biden",
        2: "Donald Trump",
        3: "Jo Jorgensen",
        4: "Howie Hawkins",
        5: "Other",
        6: "Did not vote for President",
    },
    "CC22_363": {
        1: "Yes, definitely",
        2: "Probably",
        3: "I already voted (early or absentee)",
        4: "I plan to vote before November 8th",
        5: "No",
        6: "Undecided",
    },
    "CC22_365_voted": {
        1: "$SenCand1Name ($SenCand1Party)",
        2: "$SenCand2Name ($SenCand2Party)",
        3: "$SenCand3Name ($SenCand3Party)",
        4: "$SenCand4Name ($SenCand4Party)",
        7: "Other",
        8: "I'm not sure",
        9: "I didn't vote in this election",
    },
    "CC22_365b_voted": {
        1: "$SenCand1Name2 ($SenCand1Party2)",
        2: "$SenCand2Name2 ($SenCand2Party2)",
        7: "Other",
        8: "I'm not sure",
        9: "I didn't vote in this election",
    },
    "CC22_366_voted": {
        1: "$GovCand1Name ($GovCand1Party)",
        2: "$GovCand2Name ($GovCand2Party)",
        3: "$GovCand3Name ($GovCand3Party)",
        7: "Other",
        8: "I'm not sure",
        9: "I didn't vote in this election",
    },
    "CC22_367_voted": {
        1: "$HouseCand1Name ($HouseCand1Party)",
        2: "$HouseCand2Name ($HouseCand2Party)",
        3: "$HouseCand3Name ($HouseCand3Party)",
        4: "$HouseCand4Name ($HouseCand4Party)",
        5: "$HouseCand5Name ($HouseCand5Party)",
        6: "$HouseCand6Name ($HouseCand6Party)",
        7: "$HouseCand7Name ($HouseCand7Party)",
        8: "$HouseCand8Name ($HouseCand8Party)",
        10: "Other",
        98: "I'm not sure",
        99: "I didn't vote in this election",
    },
    "CC22_365": {
        1: "$SenCand1Name ($SenCand1Party)",
        2: "$SenCand2Name ($SenCand2Party)",
        3: "$SenCand3Name ($SenCand3Party)",
        4: "$SenCand4Name ($SenCand4Party)",
        7: "Other",
        8: "I'm not sure",
        9: "No one",
    },
    "CC22_365a": {
        1: "$SenCand1Name ($SenCand1Party)",
        2: "$SenCand2Name ($SenCand2Party)",
        3: "$SenCand3Name ($SenCand3Party)",
        4: "$SenCand4Name ($SenCand4Party)",
        7: "Other",
        8: "I'm not sure",
        9: "No one",
    },
    "CC22_365b": {
        1: "$SenCand1Name2 ($SenCand1Party2)",
        2: "$SenCand2Name2 ($SenCand2Party2)",
        7: "Other",
        8: "I'm not sure",
        9: "No one",
    },
    "CC22_365c": {
        1: "$SenCand1Name2 ($SenCand1Party2)",
        2: "$SenCand2Name2 ($SenCand2Party2)",
        7: "Other",
        8: "I'm not sure",
        9: "No one",
    },
    "CC22_366": {
        1: "$GovCand1Name ($GovCand1Party)",
        2: "$GovCand2Name ($GovCand2Party)",
        3: "$GovCand3Name ($GovCand3Party)",
        7: "Other",
        8: "I'm not sure",
        9: "No one",
    },
    "CC22_367": {
        1: "$HouseCand1Name ($HouseCand1Party)",
        2: "$HouseCand2Name ($HouseCand2Party)",
        3: "$HouseCand3Name ($HouseCand3Party)",
        4: "$HouseCand4Name ($HouseCand4Party)",
        5: "$HouseCand5Name ($HouseCand5Party)",
        6: "$HouseCand6Name ($HouseCand6Party)",
        7: "$HouseCand7Name ($HouseCand7Party)",
        8: "$HouseCand8Name ($HouseCand8Party)",
        10: "Other",
        98: "I'm not sure",
        99: "No one",
    },
    "CC22_367a": {
        1: "$HouseCand1Name ($HouseCand1Party)",
        2: "$HouseCand2Name ($HouseCand2Party)",
        3: "$HouseCand3Name ($HouseCand3Party)",
        4: "$HouseCand4Name ($HouseCand4Party)",
        5: "$HouseCand5Name ($HouseCand5Party)",
        6: "$HouseCand6Name ($HouseCand6Party)",
        7: "$HouseCand7Name ($HouseCand7Party)",
        8: "$HouseCand8Name ($HouseCand8Party)",
        10: "Other",
        98: "I'm not sure",
        99: "No one",
    },
    "ideo5": {
        1: "Very liberal",
        2: "Liberal",
        3: "Moderate",
        4: "Conservative",
        5: "Very conservative",
        6: "Not sure",
    },
    "employ": {
        1: "Working full time now",
        2: "Working part time now",
        3: "Temporarily laid off",
        4: "Unemployed",
        5: "Retired",
        6: "Permanently disabled",
        7: "Taking care of home or family",
        8: "Student",
        9: "Other",
    },
    "hadjob": {1: "Yes", 2: "No"},
    "investor": {1: "Yes", 2: "No"},
    "pew_bornagain": {1: "Yes", 2: "No"},
    "pew_religimp": {
        1: "Very important",
        2: "Somewhat important",
        3: "Not too important",
        4: "Not at all important",
    },
    "pew_churatd": {
        1: "More than once a week",
        2: "Once a week",
        3: "Once or twice a month",
        4: "A few times a year",
        5: "Seldom",
        6: "Never",
        7: "Don't know",
    },
    "pew_prayer": {
        1: "Several times a day",
        2: "Once a day",
        3: "A few times a week",
        4: "Once a week",
        5: "A few times a month",
        6: "Seldom",
        7: "Never",
        8: "Don't know",
    },
    "religpew": {
        1: "Protestant",
        2: "Roman Catholic",
        3: "Mormon",
        4: "Eastern or Greek Orthodox",
        5: "Jewish",
        6: "Muslim",
        7: "Buddhist",
        8: "Hindu",
        9: "Atheist",
        10: "Agnostic",
        11: "Nothing in particular",
        12: "Something else",
    },
    "religpew_protestant": {
        1: "Baptist",
        2: "Methodist",
        3: "Nondenominational or Independent Church",
        4: "Lutheran",
        5: "Presbyterian",
        6: "Pentecostal",
        7: "Episcopalian",
        8: "Church of Christ or Disciples of Christ",
        9: "Congregational or United Church of Christ",
        10: "Holiness",
        11: "Reformed",
        12: "Adventist",
        13: "Jehovah's Witness",
        90: "Something else",
    },
    "Xreligpew_protestant": {
        1: "Baptist",
        2: "Methodist",
        3: "Nondenominational or Independent Church",
        4: "Lutheran",
        5: "Presbyterian",
        6: "Pentecostal",
        7: "Episcopalian",
        8: "Church of Christ or Disciples of Christ",
        9: "Congregational or United Church of Christ",
        10: "Holiness",
        11: "Reformed",
        12: "Adventist",
        13: "Jehovah's Witness",
        90: "None of these",
    },
    "religpew_baptist": {
        1: "Southern Baptist Convention",
        2: "American Baptist Churches in USA",
        3: "National Baptist Convention",
        4: "Progressive Baptist Convention",
        5: "Independent Baptist",
        6: "Baptist General Conference",
        7: "Baptist Missionary Association",
        8: "Conservative Baptist Assoc. of America",
        9: "Free Will Baptist",
        10: "General Association of Regular Baptists",
        90: "Other Baptist",
    },
    "religpew_methodist": {
        1: "United Methodist Church",
        2: "Free Methodist Church",
        3: "African Methodist Episcopal",
        4: "African Methodist Episcopal Zion",
        5: "Christian Methodist Episcopal Church",
        90: "Other Methodist Church",
    },
    "religpew_nondenom": {
        1: "Nondenominational evangelical",
        2: "Nondenominational fundamentalist",
        3: "Nondenominational charismatic",
        4: "Interdenominational",
        5: "Community church",
        90: "Other",
    },
    "religpew_lutheran": {
        1: "Evangelical Lutheran Church in America (ELCA)",
        2: "Lutheran Church, Missouri Synod",
        3: "Lutheran Church, Wisconsin Synod",
        4: "Other Lutheran Church",
    },
    "religpew_presby": {
        1: "Presbyterian Church USA",
        2: "Presbyterian Church in America",
        3: "Associate Reformed Presbyterian",
        4: "Cumberland Presbyterian Church",
        5: "Orthodox Presbyterian",
        6: "Evangelical Presbyterian Church",
        90: "Other Presbyterian Church",
    },
    "religpew_pentecost": {
        1: "Assemblies of God",
        2: "Church of God Cleveland TN",
        3: "Four Square Gospel",
        4: "Pentecostal Church of God",
        5: "Pentecostal Holiness Church",
        6: "Church of God in Christ",
        7: "Church of God of the Apostolic Faith",
        8: "Assembly of Christian Churches",
        9: "Apostolic Christian",
        90: "Other Pentecostal Church",
    },
    "religpew_episcop": {
        1: "Episcopal Church in the USA",
        2: "Anglican Church (Church of England)",
        3: "Anglican Orthodox Church",
        4: "Reformed Episcopal Church",
        90: "Other Episcopalian or Anglican Church",
    },
    "religpew_christian": {
        1: "Church of Christ",
        2: "Disciples of Christ",
        3: "Christian Churches and Churches of Christ",
        90: "Other Christian church",
    },
    "religpew_congreg": {
        1: "United Church of Christ",
        2: "Conservative Congregational Christian",
        3: "National Association of Congregational Christians",
        90: "Other Congregational",
    },
    "religpew_holiness": {
        1: "Church of the Nazarene",
        2: "Wesleyan Church",
        3: "Free Methodist Church",
        4: "Christian and Missionary Alliance",
        5: "Church of God (Anderson, Indiana)",
        6: "Salvation Army, American Rescue workers",
        90: "Other Holiness",
    },
    "religpew_reformed": {
        1: "Reformed Church in America",
        2: "Christian Reformed Church",
        90: "Other Reformed",
    },
    "religpew_advent": {
        1: "Seventh Day Adventist",
        2: "Church of God, General Conference",
        3: "Advent Christian",
        90: "Other Adventist",
    },
    "religpew_catholic": {
        1: "Roman Catholic Church",
        2: "National Polish Catholic Church",
        3: "Greek-rite Catholic",
        4: "Armenian Catholic",
        5: "Old Catholic",
        90: "Other Catholic",
    },
    "religpew_mormon": {
        1: "The Church of Jesus Christ of Latter-day Saints",
        2: "Community of Christ",
        90: "Other Mormon",
    },
    "religpew_orthodox": {
        1: "Greek Orthodox",
        2: "Russian Orthodox",
        3: "Orthodox Church in America",
        4: "Armenian Orthodox",
        5: "Eastern Orthodox",
        6: "Serbian Orthodox",
        90: "Other Orthodox",
    },
    "religpew_jewish": {
        1: "Reform",
        2: "Conservative",
        3: "Orthodox",
        4: "Reconstructionist",
        90: "Other",
    },
    "religpew_muslim": {
        1: "Sunni",
        2: "Shia",
        3: "Nation of Islam (Black Muslim)",
        90: "Other Muslim",
    },
    "religpew_buddhist": {
        1: "Theravada (Vipassana) Buddhism",
        2: "Mahayana (Zen) Buddhism",
        3: "Vajrayana (Tibetan) Buddhism",
        90: "Other Buddhist",
    },
    "religpew_hindu": {
        1: "Vaishnava Hinduism",
        2: "Shaivite Hinduism",
        3: "Shaktism Hinduism",
        90: "Other Hindu",
    },
    "marstat": {
        1: "Married",
        2: "Separated",
        3: "Divorced",
        4: "Widowed",
        5: "Never married",
        6: "Domestic / civil partnership",
    },
    "union": {
        1: "Yes, I am currently a member of a labor union",
        2: "I formerly was a member of a labor union",
        3: "I am not now, nor have I been, a member of a labor union",
    },
    "union_coverage": {1: "Yes", 2: "No", 3: "Not sure"},
    "unionhh": {
        1: "Yes, a member of my household is currently a union member",
        2: "A member of my household was formerly a member of a labor union, but is not now",
        3: "No, no one in my household has ever been a member of a labor union",
        4: "Not sure",
    },
    "ccesmodule": {
        1: "NCC",
        2: "MIA",
        3: "TTU",
        4: "RUT",
        5: "MSU",
        6: "FSU",
        7: "LSU",
        8: "JHU",
        9: "IUA",
        10: "BOS",
        11: "DKU",
        12: "NCW",
        13: "WUS",
        14: "USC",
        15: "ZOU",
        16: "MIC",
        17: "BCJ",
        18: "CPC",
        19: "ASU",
        20: "CAC",
        21: "UWM",
        22: "UCR",
        23: "GTN",
        24: "BTU",
        25: "UTA",
        26: "UTB",
        27: "UVA",
        28: "NCK",
        29: "UTD",
        30: "GWU",
        31: "CUB",
        32: "OSU",
        33: "IOW",
        34: "UCL",
        35: "DKN",
        36: "CLA",
        37: "LBU",
        38: "UND",
        39: "TAM",
        40: "WAS",
        41: "YLS",
        42: "UGA",
        43: "TUF",
        44: "DAR",
        45: "NYU",
        46: "VAN",
        47: "UCM",
        48: "UMA",
        49: "UDE",
        50: "MIZ",
        51: "BYU",
        52: "EMY",
        53: "RCO",
        54: "MSL",
        55: "MCS",
        56: "HUA",
        57: "HUB",
        58: "AMU",
        59: "UMB/CGU",
        60: "DMC / HKS",
    },
    "dualcit": {1: "Yes", 2: "No"},
    "dualctry": {
        124: "Canada",
        826: "United Kingdom",
        4: "Afghanistan",
        248: "Aland Islands",
        8: "Albania",
        12: "Algeria",
        16: "American Samoa",
        20: "Andorra",
        24: "Angola",
        660: "Anguilla",
        10: "Antarctica",
        28: "Antigua and Barbuda",
        32: "Argentina",
        51: "Armenia",
        533: "Aruba",
        36: "Australia",
        40: "Austria",
        31: "Azerbaijan",
        44: "Bahamas",
        48: "Bahrain",
        50: "Bangladesh",
        52: "Barbados",
        112: "Belarus",
        56: "Belgium",
        84: "Belize",
        204: "Benin",
        60: "Bermuda",
        64: "Bhutan",
        68: "Bolivia",
        70: "Bosnia and Herzegovina",
        72: "Botswana",
        74: "Bouvet Island",
        76: "Brazil",
        86: "British Indian Ocean Territory",
        96: "Brunei Darussalam",
        100: "Bulgaria",
        854: "Burkina Faso",
        108: "Burundi",
        116: "Cambodia",
        120: "Cameroon",
        132: "Cape Verde",
        136: "Cayman Islands",
        140: "Central African Republic",
        148: "Chad",
        152: "Chile",
        156: "China",
        162: "Christmas Island",
        166: "Cocos (Keeling) Islands",
        170: "Colombia",
        174: "Comoros",
        178: "Congo",
        180: "Congo, the Democratic Republic of the",
        184: "Cook Islands",
        188: "Costa Rica",
        384: "Cote d'Ivoire",
        191: "Croatia",
        192: "Cuba",
        196: "Cyprus",
        203: "Czech Republic",
        208: "Denmark",
        262: "Djibouti",
        212: "Dominica",
        214: "Dominican Republic",
        218: "Ecuador",
        818: "Egypt",
        222: "El Salvador",
        226: "Equatorial Guinea",
        232: "Eritrea",
        233: "Estonia",
        231: "Ethiopia",
        238: "Falkland Islands (Malvinas)",
        234: "Faroe Islands",
        242: "Fiji",
        246: "Finland",
        250: "France",
        254: "French Guiana",
        258: "French Polynesia",
        260: "French Southern Territories",
        266: "Gabon",
        270: "Gambia",
        268: "Georgia",
        276: "Germany",
        288: "Ghana",
        292: "Gibraltar",
        300: "Greece",
        304: "Greenland",
        308: "Grenada",
        312: "Guadeloupe",
        316: "Guam",
        320: "Guatemala",
        831: "Guernsey",
        324: "Guinea",
        624: "Guinea-Bissau",
        328: "Guyana",
        332: "Haiti",
        334: "Heard Island and McDonald Islands",
        336: "Holy See (Vatican City State)",
        340: "Honduras",
        344: "Hong Kong",
        348: "Hungary",
        352: "Iceland",
        356: "India",
        360: "Indonesia",
        364: "Iran, Islamic Republic of",
        368: "Iraq",
        372: "Ireland",
        833: "Isle of Man",
        376: "Israel",
        380: "Italy",
        388: "Jamaica",
        392: "Japan",
        832: "Jersey",
        400: "Jordan",
        398: "Kazakhstan",
        404: "Kenya",
        296: "Kiribati",
        408: "Korea, Democratic People's Republic of",
        410: "Korea, Republic of",
        414: "Kuwait",
        417: "Kyrgyzstan",
        418: "Lao People's Democratic Republic",
        428: "Latvia",
        422: "Lebanon",
        426: "Lesotho",
        430: "Liberia",
        434: "Libyan Arab Jamahiriya",
        438: "Liechtenstein",
        440: "Lithuania",
        442: "Luxembourg",
        446: "Macao",
        807: "Macedonia, the former Yugoslav Republic of",
        450: "Madagascar",
        454: "Malawi",
        458: "Malaysia",
        462: "Maldives",
        466: "Mali",
        470: "Malta",
        584: "Marshall Islands",
        474: "Martinique",
        478: "Mauritania",
        480: "Mauritius",
        175: "Mayotte",
        484: "Mexico",
        583: "Micronesia, Federated States of",
        498: "Moldova, Republic of",
        492: "Monaco",
        496: "Mongolia",
        500: "Montserrat",
        504: "Morocco",
        508: "Mozambique",
        104: "Myanmar",
        516: "Namibia",
        520: "Nauru",
        524: "Nepal",
        528: "Netherlands",
        530: "Netherlands Antilles",
        540: "New Caledonia",
        554: "New Zealand",
        558: "Nicaragua",
        562: "Niger",
        566: "Nigeria",
        570: "Niue",
        574: "Norfolk Island",
        580: "Northern Mariana Islands",
        578: "Norway",
        512: "Oman",
        586: "Pakistan",
        585: "Palau",
        275: "Palestinian Territory, Occupied",
        591: "Panama",
        598: "Papua New Guinea",
        600: "Paraguay",
        604: "Peru",
        608: "Philippines",
        612: "Pitcairn",
        616: "Poland",
        620: "Portugal",
        630: "Puerto Rico",
        634: "Qatar",
        638: "Reunion",
        642: "Romania",
        643: "Russian Federation",
        646: "Rwanda",
        654: "Saint Helena",
        659: "Saint Kitts and Nevis",
        662: "Saint Lucia",
        666: "Saint Pierre and Miquelon",
        670: "Saint Vincent and the Grenadines",
        882: "Samoa",
        674: "San Marino",
        678: "Sao Tome and Principe",
        682: "Saudi Arabia",
        686: "Senegal",
        891: "Serbia and Montenegro",
        690: "Seychelles",
        694: "Sierra Leone",
        702: "Singapore",
        703: "Slovakia",
        705: "Slovenia",
        90: "Solomon Islands",
        706: "Somalia",
        710: "South Africa",
        239: "South Georgia and the South Sandwich Islands",
        724: "Spain",
        144: "Sri Lanka",
        736: "Sudan",
        740: "Suriname",
        744: "Svalbard and Jan Mayen",
        748: "Swaziland",
        752: "Sweden",
        756: "Switzerland",
        760: "Syrian Arab Republic",
        158: "Taiwan",
        762: "Tajikistan",
        834: "Tanzania, United Republic of",
        764: "Thailand",
        626: "Timor-Leste",
        768: "Togo",
        772: "Tokelau",
        776: "Tonga",
        780: "Trinidad and Tobago",
        788: "Tunisia",
        792: "Turkey",
        795: "Turkmenistan",
        796: "Turks and Caicos Islands",
        798: "Tuvalu",
        800: "Uganda",
        804: "Ukraine",
        784: "United Arab Emirates",
        581: "United States Minor Outlying Islands",
        858: "Uruguay",
        860: "Uzbekistan",
        548: "Vanuatu",
        862: "Venezuela",
        704: "Vietnam",
        92: "Virgin Islands, British",
        850: "Virgin Islands, U.S.",
        876: "Wallis and Futuna",
        732: "Western Sahara",
        887: "Yemen",
        894: "Zambia",
        716: "Zimbabwe",
        9999: "Other",
    },
    "ownhome": {1: "Own", 2: "Rent", 3: "Other"},
    "newsint": {
        1: "Most of the time",
        2: "Some of the time",
        3: "Only now and then",
        4: "Hardly at all",
        7: "Don't know",
    },
    "faminc_new": {
        1: "Less than $10,000",
        2: "$10,000 - $19,999",
        3: "$20,000 - $29,999",
        4: "$30,000 - $39,999",
        5: "$40,000 - $49,999",
        6: "$50,000 - $59,999",
        7: "$60,000 - $69,999",
        8: "$70,000 - $79,999",
        9: "$80,000 - $99,999",
        10: "$100,000 - $119,999",
        11: "$120,000 - $149,999",
        12: "$150,000 - $199,999",
        13: "$200,000 - $249,999",
        14: "$250,000 - $349,999",
        15: "$350,000 - $499,999",
        16: "$500,000 or more",
        97: "Prefer not to say",
    },
    "milstat": {
        1: "I am currently serving in the U.S. military",
        2: "I have immediate family members currently serving in the U.S. military",
        3: "I previously served in the U.S. military but I am no longer active",
        4: "Members of my immediate family have served in the U.S. military but are no longer active",
        5: "Neither myself nor any members of my immediate family have ever served in the U.S. military",
    },
    "child18": {1: "Yes", 2: "No"},
    "healthins": {
        1: "Yes, through my job or a family member's employer",
        2: "Yes, through a government program, such as Medicare or Medicaid",
        3: "Yes, through my school",
        4: "Yes, I purchased my own",
        5: "Not sure",
        6: "No",
    },
    "healthins2": {1: "Yes", 2: "No"},
    "phone": {
        1: "A landline and a cell phone",
        2: "Cell phone only",
        3: "Landline only",
        4: "No landline or cell phone service",
    },
    "internethome": {1: "Cable, DSL, or other broadband", 2: "Dial-up", 3: "None"},
    "internetwork": {1: "Cable, DSL, or other broadband", 2: "Dial-up", 3: "None"},
    "CC22_hisp": {
        1: "No Country in Particular",
        2: "United States",
        3: "Mexico",
        4: "Puerto Rico",
        5: "Cuba",
        6: "Dominican Republic",
        7: "South America",
        8: "Central America",
        9: "Caribbean",
        10: "Spain",
        11: "Other",
        12: "I am not of Latino, Hispanic or Spanish Heritage",
    },
    "CC22_asian": {
        1: "No Country in Particular",
        2: "United States",
        3: "China",
        4: "Japan",
        5: "India",
        6: "Philippines",
        7: "Taiwan",
        8: "Korea",
        9: "Vietnam",
        10: "Pakistan",
        11: "Hmong",
        12: "Cambodia",
        13: "Thailand",
        14: "Other",
        15: "I am not of Asian Heritage",
    },
    "presvote16post": {
        1: "Hillary Clinton",
        2: "Donald Trump",
        3: "Gary Johnson",
        4: "Jill Stein",
        5: "Evan McMullin",
        6: "Other",
        7: "Did not vote for President",
    },
    "industry": {
        1: "Agriculture, forestry, fishing, and hunting",
        2: "Mining",
        3: "Utilities",
        4: "Construction",
        5: "Manufacturing",
        6: "Professional and business services",
        7: "Educational services",
        8: "Health care and social assistance",
        9: "Leisure and hospitality",
        10: "Other services",
        11: "Wholesale trade",
        12: "Retail trade",
        13: "Transportation and warehousing",
        14: "Information",
        15: "Financial activities",
        16: "Federal government",
        17: "State and local government",
    },
    "sexuality": {
        1: "Heterosexual / straight",
        2: "Lesbian / gay woman",
        3: "Gay man",
        4: "Bisexual",
        5: "Other",
        6: "Prefer not to say",
    },
    "transgender": {1: "Yes", 2: "No", 3: "Prefer not to say"},
}
```

### Parameterizing questions with data

Here we show how to construct `Scenario` objects for data that we want to input in question texts. This allows us to administer multiple versions of questions efficiently. Learn more about [working with scenarios](/en/latest/scenarios).

Values in the CES questions requiring inputs:

```python theme={null}
"inputzip", "inputstate",
"$pid7text",
"$LowerChamberName",
"$CurrentGovName", "$CurrentHouseName", "$CurrentSen1Name", "$CurrentSen2Name",
"$LegName",
"$SenCand1Name", "$SenCand2Name", "$SenCand3Name", "$SenCand4Name",
"$SenCand1Name2", "$SenCand2Name2",
"$GovCand1Name", "$GovCand2Name", "$GovCand3Name",
"$HouseCand1Name", "$HouseCand2Name", "$HouseCand3Name", "$HouseCand4Name", "$HouseCand5Name", "$HouseCand6Name", "$HouseCand7Name", "$HouseCand8Name",
"$SenCand1Party", "$SenCand2Party", "$SenCand3Party", "$SenCand4Party",
"$SenCand1Party2", "$SenCand2Party2",
"$GovCand1Party", "$GovCand2Party", "$GovCand3Party",
"$HouseCand1Party", "$HouseCand2Party", "$HouseCand3Party", "$HouseCand4Party", "$HouseCand5Party", "$HouseCand6Party", "$HouseCand7Party", "$HouseCand8Party",
"$employtext"
```

Here we create scenarios for a set of questions, add them to the survey when we run it, and inspect the results:

```python theme={null}
from edsl import Scenario, ScenarioList
```

```python theme={null}
ca = ["California", "Gavin Newsom", "Dianne Feinstein", "Alex Padilla"]
ny = ["New York", "Kathy Hochul", "Chuck Schumer", "Kirsten Gillibrand"]

scenarios = ScenarioList(
    Scenario(
        {
            "CurrentGovName": state[1],
            "CurrentSen1Name": state[2],
            "CurrentSen2Name": state[3],
        }
    )
    for state in [ca, ny]
)
```

We modify the relevant question texts to use bracketed parameters:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, Survey

q_CC22_311a = QuestionMultipleChoice(
    question_name="CC22_311a",
    question_text="Please indicate whether you've heard of {{ scenario.CurrentGovName }} and if so which party he or she is affiliated with.",
    question_options=[
        "Never heard of person",
        "Republican",
        "Democrat",
        "Other Party / Independent",
        "Not sure",
    ],
)

q_CC22_311b = QuestionMultipleChoice(
    question_name="CC22_311b",
    question_text="Please indicate whether you've heard of {{ scenario.CurrentSen1Name }} and if so which party he or she is affiliated with.",
    question_options=[
        "Never heard of person",
        "Republican",
        "Democrat",
        "Other Party / Independent",
        "Not sure",
    ],
)

q_CC22_311c = QuestionMultipleChoice(
    question_name="CC22_311c",
    question_text="Please indicate whether you've heard of {{ scenario.CurrentSen2Name }} and if so which party he or she is affiliated with.",
    question_options=[
        "Never heard of person",
        "Republican",
        "Democrat",
        "Other Party / Independent",
        "Not sure",
    ],
)

survey = Survey([q_CC22_311a, q_CC22_311b, q_CC22_311c])
```

```python theme={null}
results = survey.by(scenarios).run()
```

```python theme={null}
results.select("scenario.*", "answer.*")
```

| | scenario.scenario\_index | scenario.CurrentSen2Name | scenario.CurrentGovName | scenario.CurrentSen1Name | answer.CC22\_311c | answer.CC22\_311a | answer.CC22\_311b |
| :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | 0 | Alex Padilla | Gavin Newsom | Dianne Feinstein | Democrat | Democrat | Democrat |
| 1 | 1 | Kirsten Gillibrand | Kathy Hochul | Chuck Schumer | Democrat | Democrat | Democrat |

### Creating AI agents for survey respondents

Learn more about working with the CES data in [`Creating AI agents for survey respondents <>`\_\_](/en/latest/notebooks/#id1), where we demonstrate how to use the EDSL module `Conjure` to import survey data and create AI agents based on responsesd


# Concept induction
Source: https://docs.expectedparrot.com/en/latest/notebooks/concept_induction

This notebook provides example [EDSL](/en/latest/index) code for using language models to perform “concept induction”: identify concepts in unstructured texts; generate criteria for the concepts; and then apply the criteria to evaluate the texts. This idea is inspired by the recent paper: [Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM](https://hci.stanford.edu/publications/2024/Lam_LLooM_CHI24.pdf)

Before running the code below, please see instructions on [getting started](https://www.expectedparrot.com/getting-started) using EDSL.

## Identify concepts

We start by creating a general question prompting the respondent (a language model) to identify concepts in a given text.

EDSL comes with a variety of [question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from the model. `QuestionList` may be appropriate where we want the response to be formatted as a list of strings:

```python theme={null}
from edsl import QuestionList

q_concepts = QuestionList(
    question_name="concepts",
    question_text="Identify the key concepts in the following text: {{ scenario.text }}",
    # max_list_items = # Optional,
    # min_list_items = # Optional
)
```

We might also want to ask some other questions about our data at the same time (a data labeling task). For example:

```python theme={null}
from edsl import QuestionMultipleChoice

q_sentiment = QuestionMultipleChoice(
    question_name="sentiment",
    question_text="Identify the sentiment of this text: {{ scenario.text }}",
    question_options=["Negative", "Neutral", "Positive"],
)
```

We parameterize the questions in order to run them for each of our texts. This is done with `Scenario` objects that we can create using EDSL or import from other sources (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). Here we import a setof recent tweets by President Biden:

```python theme={null}
# Replace with your data
texts = [  # POTUS recent tweets
    "Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C.",
    "We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom.",
    "Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share.",
    "Today, the Army Black Knights are taking home West Point’s 10th Commander-in-Chief Trophy. They should be proud. I’m proud of them too – not for the wins, but because after every game they hang up their uniforms and put on another: one representing the United States.",
    "This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.'",
    "The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow.",
    "Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day.",
    "Jill and I send warm wishes to Orthodox Christian communities around the world as they celebrate Easter. May the Lord bless and keep you this Easter Sunday and in the year ahead.",
    "Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients.",
    "With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.",
]
len(texts)
```

10

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_source("list", "text", texts)
scenarios
```

[ScenarioList](/en/latest/scenarios) scenarios: 10; keys: \['text'];

| | text |
| :- | :- |
| 0 | Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. |
| 1 | We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. |
| 2 | Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. |
| 3 | Today, the Army Black Knights are taking home West Point’s 10th Commander-in-Chief Trophy. They should be proud. I’m proud of them too – not for the wins, but because after every game they hang up their uniforms and put on another: one representing the United States. |
| 4 | This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' |
| 5 | The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow. |
| 6 | Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day. |
| 7 | Jill and I send warm wishes to Orthodox Christian communities around the world as they celebrate Easter. May the Lord bless and keep you this Easter Sunday and in the year ahead. |
| 8 | Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. |
| 9 | With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with. |

Next we combine the questions into a survey in order to administer them together (asynchronously by default, or according to any skip/stop rules or other logic that we want to add–learn more about `Survey` methods in our [documentation](/en/latest/surveys)):

```python theme={null}
from edsl import Survey

survey = Survey(questions=[q_concepts, q_sentiment])
```

We add the scenarios to the survey and then run it with the default model (currently gpt-5.2) to generate a dataset of results:

```python theme={null}
results = survey.by(scenarios).run()
```

EDSL comes with [built-in methods](/en/latest/results) for working with results in a variety of forms (data tables, SQL queries, dataframes, JSON, CSV). We can call the `columns` method to see a list of all the components that we can analyze:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.concepts |
| 4 | answer.sentiment |
| 5 | cache\_keys.concepts\_cache\_key |
| 6 | cache\_keys.sentiment\_cache\_key |
| 7 | cache\_used.concepts\_cache\_used |
| 8 | cache\_used.sentiment\_cache\_used |
| 9 | comment.concepts\_comment |
| 10 | comment.sentiment\_comment |
| 11 | generated\_tokens.concepts\_generated\_tokens |
| 12 | generated\_tokens.sentiment\_generated\_tokens |
| 13 | iteration.iteration |
| 14 | model.frequency\_penalty |
| 15 | model.inference\_service |
| 16 | model.logprobs |
| 17 | model.max\_tokens |
| 18 | model.model |
| 19 | model.model\_index |
| 20 | model.presence\_penalty |
| 21 | model.temperature |
| 22 | model.top\_logprobs |
| 23 | model.top\_p |
| 24 | prompt.concepts\_system\_prompt |
| 25 | prompt.concepts\_user\_prompt |
| 26 | prompt.sentiment\_system\_prompt |
| 27 | prompt.sentiment\_user\_prompt |
| 28 | question\_options.concepts\_question\_options |
| 29 | question\_options.sentiment\_question\_options |
| 30 | question\_text.concepts\_question\_text |
| 31 | question\_text.sentiment\_question\_text |
| 32 | question\_type.concepts\_question\_type |
| 33 | question\_type.sentiment\_question\_type |
| 34 | raw\_model\_response.concepts\_cost |
| 35 | raw\_model\_response.concepts\_input\_price\_per\_million\_tokens |
| 36 | raw\_model\_response.concepts\_input\_tokens |
| 37 | raw\_model\_response.concepts\_one\_usd\_buys |
| 38 | raw\_model\_response.concepts\_output\_price\_per\_million\_tokens |
| 39 | raw\_model\_response.concepts\_output\_tokens |
| 40 | raw\_model\_response.concepts\_raw\_model\_response |
| 41 | raw\_model\_response.sentiment\_cost |
| 42 | raw\_model\_response.sentiment\_input\_price\_per\_million\_tokens |
| 43 | raw\_model\_response.sentiment\_input\_tokens |
| 44 | raw\_model\_response.sentiment\_one\_usd\_buys |
| 45 | raw\_model\_response.sentiment\_output\_price\_per\_million\_tokens |
| 46 | raw\_model\_response.sentiment\_output\_tokens |
| 47 | raw\_model\_response.sentiment\_raw\_model\_response |
| 48 | reasoning\_summary.concepts\_reasoning\_summary |
| 49 | reasoning\_summary.sentiment\_reasoning\_summary |
| 50 | scenario.scenario\_index |
| 51 | scenario.text |

We can select and print specific components to inspect in a table:

```python theme={null}
results.select("text", "concepts", "sentiment")
```

| | scenario.text | answer.concepts | answer.sentiment |
| :- | :- | :- | :- |
| 0 | Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. | \['keynote address', 'U.S. Holocaust Memorial Museum', 'Annual Days of Remembrance', 'Washington, D.C.'] | Neutral |
| 1 | We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. | \['nation of immigrants', 'nation of dreamers', 'Cinco de Mayo', 'freedom'] | Positive |
| 2 | Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. | \['Medicare', 'Social Security', 'economic plan', 'Medicare solvency', 'Social Security solvency', 'fair share'] | Positive |
| 3 | Today, the Army Black Knights are taking home West Point’s 10th Commander-in-Chief Trophy. They should be proud. I’m proud of them too – not for the wins, but because after every game they hang up their uniforms and put on another: one representing the United States. | \['Army Black Knights', 'West Point', 'Commander-in-Chief Trophy', 'United States'] | Positive |
| 4 | This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' | \['Holocaust Remembrance Day', 'six million Jews', 'Nazis', 'darkest chapters', 'lessons of the Shoah', 'responsibility', 'Never Again'] | Neutral |
| 5 | The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow. | \['Presidential Medal of Freedom', 'faith in freedom', "America's faith", 'better tomorrow'] | Positive |
| 6 | Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day. | \['Teaching', 'Calling', 'Educator', 'White House', 'Teacher State Dinner'] | Positive |
| 7 | Jill and I send warm wishes to Orthodox Christian communities around the world as they celebrate Easter. May the Lord bless and keep you this Easter Sunday and in the year ahead. | \['Jill and I', 'warm wishes', 'Orthodox Christian communities', 'Easter', 'Lord bless and keep you', 'Easter Sunday', 'year ahead'] | Positive |
| 8 | Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. | \['Dreamers', 'health care', 'Affordable Care Act', 'DACA recipients', 'affordable health coverage', 'Administration'] | Positive |
| 9 | With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with. | \['American comeback', '175,000 new jobs', 'Congressional Republicans', 'cut taxes', 'billionaires', 'special interests', 'job creation', 'economy', 'families'] | Positive |

If our concepts lists are too long, we can run another question prompting a model to condense it. We can specify the number of concepts that we want to get:

```python theme={null}
# Flattening our list of lists for all the texts to use in a follow-on question:
concepts_list = results.select("concepts").to_list(flatten=True)
concepts_list
```

\['keynote address',
'U.S. Holocaust Memorial Museum',
'Annual Days of Remembrance',
'Washington, D.C.',
'nation of immigrants',
'nation of dreamers',
'Cinco de Mayo',
'freedom',
'Medicare',
'Social Security',
'economic plan',
'Medicare solvency',
'Social Security solvency',
'fair share',
'Army Black Knights',
'West Point',
'Commander-in-Chief Trophy',
'United States',
'Holocaust Remembrance Day',
'six million Jews',
'Nazis',
'darkest chapters',
'lessons of the Shoah',
'responsibility',
'Never Again',
'Presidential Medal of Freedom',
'faith in freedom',
"America's faith",
'better tomorrow',
'Teaching',
'Calling',
'Educator',
'White House',
'Teacher State Dinner',
'Jill and I',
'warm wishes',
'Orthodox Christian communities',
'Easter',
'Lord bless and keep you',
'Easter Sunday',
'year ahead',
'Dreamers',
'health care',
'Affordable Care Act',
'DACA recipients',
'affordable health coverage',
'Administration',
'American comeback',
'175,000 new jobs',
'Congressional Republicans',
'cut taxes',
'billionaires',
'special interests',
'job creation',
'economy',
'families']

```python theme={null}
from edsl import Scenario

scenario = Scenario({"concepts":concepts_list})
```

```python theme={null}
q_condense = QuestionList(
    question_name="condense",
    question_text="Return a condensed list of the following set of concepts: {{ scenario.concepts }}",
    max_list_items=10,
)
```

<Note>
  **Note**:

  Note that we can call the `run()` method on either a survey of questions or an individual question:
</Note>

```python theme={null}
results = q_condense.by(scenario).run()
```

```python theme={null}
results.select("condense")
```

| | answer.condense |
| :- | :- |
| 0 | \['Holocaust Remembrance', 'Immigration and Dreamers', 'Economic Plan and Solvency', 'Freedom and Faith', 'Education and Teaching', 'Orthodox Easter', 'Health Care and ACA', 'American Comeback', 'Job Creation and Economy', 'White House and Administration'] |

## Identify criteria for each concept

Similar to our first step, next we can run a question prompting the model to generate criteria for each concept. We could use `QuestionFreeText` to generate criteria in an unstructured narrative:

```python theme={null}
from edsl import QuestionFreeText

q_criteria = QuestionFreeText(
    question_name="criteria",
    question_text="""Describe key criteria for determining whether a text is primarily about the
    following concept: {{ scenario.concept }}""",
)
```

For this question, the scenarios are the concepts that we generated:

```python theme={null}
condensed_concepts_list = results.select("condense").to_list(flatten=True)

scenarios = ScenarioList.from_source("list", "concept", condensed_concepts_list)
scenarios
```

[ScenarioList](/en/latest/scenarios) scenarios: 10; keys: \['concept'];

| | concept |
| :- | :- |
| 0 | Holocaust Remembrance |
| 1 | Immigration and Dreamers |
| 2 | Economic Plan and Solvency |
| 3 | Freedom and Faith |
| 4 | Education and Teaching |
| 5 | Orthodox Easter |
| 6 | Health Care and ACA |
| 7 | American Comeback |
| 8 | Job Creation and Economy |
| 9 | White House and Administration |

```python theme={null}
results = q_criteria.by(scenarios).run()
```

```python theme={null}
results.select("concept", "criteria")
```

| | scenario.concept | answer.criteria |
| :- | :- | :- |
| 0 | Holocaust Remembrance | Determining whether a text is primarily about Holocaust Remembrance involves evaluating several key criteria: <br /> 1. **Subject Focus**: The text should center on the Holocaust, specifically emphasizing the importance of remembering the events, victims, and survivors. It should discuss commemorative practices, memorials, or anniversaries related to the Holocaust. <br /> 2. **Purpose and Intent**: The text's purpose should be to educate, memorialize, or promote awareness of the Holocaust. It may aim to prevent future atrocities through remembrance and reflection on past events. <br /> 3. **Language and Tone**: The language used should convey a sense of solemnity, respect, and reflection. The tone might be educational, commemorative, or advocacy-oriented, urging readers to remember and learn from the past. <br /> 4. **Content and Themes**: Key themes should include memory, history, education, and the moral and ethical lessons of the Holocaust. The text might discuss survivor testimonies, historical accounts, or the impact of Holocaust education on contemporary society. <br /> 5. **Contextual References**: The text may reference specific Holocaust Remembrance events, such as Yom HaShoah (Holocaust Remembrance Day), International Holocaust Remembrance Day, or other national and international commemorations. <br /> 6. **Audience Engagement**: The text might encourage reader participation in remembrance activities, such as attending memorial services, visiting Holocaust museums, or engaging in educational programs. <br /> 7. **Historical Accuracy**: The text should be grounded in historical facts about the Holocaust, providing accurate information about the events, figures, and outcomes associated with it. <br /> 8. **Cultural and Social Impact**: The text should reflect on the broader cultural and social significance of remembering the Holocaust, including its impact on Jewish communities and global human rights discourse. By considering these criteria, one can assess whether a text is primarily focused on Holocaust Remembrance and its associated themes. |
| 1 | Immigration and Dreamers | To determine whether a text is primarily about the concept of "Immigration and Dreamers," you should look for several key criteria: <br /> 1. **Central Theme**: The text should focus on issues related to immigration, specifically highlighting aspects of the Dreamers—individuals who were brought to the United States as children without legal immigration status. <br /> 2. **Mention of DACA**: The Deferred Action for Childhood Arrivals (DACA) program is often central to discussions about Dreamers. The text should reference DACA policies, debates, or impacts. <br /> 3. **Personal Narratives**: Stories or testimonies of individuals who identify as Dreamers can indicate the text's focus. These narratives often highlight personal experiences, challenges, and aspirations. <br /> 4. **Legislative and Policy Discussions**: The text may discuss legislative efforts, policy changes, or political debates surrounding immigration reform and protections for Dreamers. <br /> 5. **Socioeconomic Impact**: Analysis of how Dreamers contribute to or are affected by socioeconomic factors in the U.S., such as education, employment, or community integration. <br /> 6. **Challenges and Barriers**: Exploration of the legal, social, and economic challenges faced by Dreamers, including barriers to education, work, and legal residency. <br /> 7. **Advocacy and Activism**: Coverage of advocacy efforts, protests, or movements supporting Dreamers and broader immigration reform. <br /> 8. **Cultural Integration**: Discussion of how Dreamers navigate cultural identity, assimilation, and belonging within the U.S. society. <br /> 9. **Statistical Data**: Inclusion of statistics or data that provide insights into the number of Dreamers, their demographics, or their contributions to society. <br /> 10.**Emotional Tone**: The text might convey an emotional tone that reflects the struggles, hopes, and resilience of Dreamers. By analyzing these criteria, you can assess whether a text is primarily concerned with the topic of Immigration and Dreamers. |
| 2 | Economic Plan and Solvency | To determine whether a text is primarily about the concept of "Economic Plan and Solvency," you can evaluate it based on several key criteria: <br /> 1. **Purpose and Objectives**: The text should outline specific economic goals or objectives. This might include plans for growth, stability, or sustainability, and how these goals will ensure or enhance solvency. <br /> 2. **Strategic Framework**: Look for a detailed framework or strategy that outlines how the economic plan will be implemented. This includes timelines, phases, and specific actions or policies that will be undertaken. <br /> 3. **Financial Analysis**: The text should include an analysis of financial resources, budgeting, and allocation of funds. It should discuss how these resources will support the economic plan and maintain or improve solvency. <br /> 4. **Risk Assessment**: A thorough examination of potential risks to the economic plan and solvency should be present. This includes identifying economic, financial, and external risks, along with proposed mitigation strategies. <br /> 5. **Revenue and Expenditure Projections**: The text should provide projections of revenues and expenditures, demonstrating how the plan will generate sufficient income to cover costs and ensure solvency. <br /> 6. **Debt Management**: Discussion of current debt levels and strategies for debt management is crucial. The text should explain how the plan will address existing debt and prevent insolvency. <br /> 7. **Policy Measures**: Specific economic policies or reforms that are part of the plan should be detailed. This includes fiscal policies, monetary policies, tax reforms, and regulatory changes aimed at achieving solvency. <br /> 8. **Stakeholder Involvement**: The text should mention key stakeholders involved in the economic plan, such as government agencies, financial institutions, businesses, and the public, and how their roles contribute to the plan's success and solvency. <br /> 9. **Performance Metrics**: Identification of key performance indicators (KPIs) or metrics used to measure the success of the economic plan and its impact on solvency. <br /> 10.**Historical Context and Comparisons**: The text might provide a historical context or comparisons with previous economic plans to highlight improvements or changes that enhance solvency. 1<br /> 1. **Sustainability and Long-term Impact**: Consideration of the long-term sustainability of the economic plan and its implications for future solvency. By evaluating a text against these criteria, you can determine whether it is primarily focused on the concept of an economic plan and solvency. |
| 3 | Freedom and Faith | Determining whether a text is primarily about the concept of "Freedom and Faith" involves evaluating several key criteria: <br /> 1. **Central Themes**: The text should prominently feature the themes of freedom and faith, either individually or in combination. It should explore the nature, implications, and dynamics of these concepts, including how they interact or conflict with each other. <br /> 2. **Character Development**: If the text is narrative, the characters' journeys, decisions, and growth should revolve around issues of freedom and faith. This could include characters seeking freedom from oppression, societal norms, or personal limitations, and how their faith influences these pursuits. <br /> 3. **Conflict and Resolution**: The primary conflicts in the text should involve struggles related to freedom and faith. This might include internal conflicts of belief, external pressures to conform, or societal constraints that challenge personal freedoms. The resolution should offer insights into how these conflicts are navigated or resolved. <br /> 4. **Philosophical or Theological Exploration**: The text should engage in a philosophical or theological exploration of freedom and faith. This could include discussions on the nature of free will, the role of faith in human life, or the moral and ethical dimensions of freedom. <br /> 5. **Cultural and Historical Context**: The text might explore how freedom and faith are perceived and practiced within specific cultural or historical contexts. This can include discussions on religious freedom, the impact of faith on political movements, or historical struggles for freedom influenced by faith. <br /> 6. **Symbolism and Imagery**: The use of symbolism and imagery related to freedom and faith can be a strong indicator. This might include metaphors of liberation, captivity, spiritual enlightenment, or religious iconography. <br /> 7. **Author’s Intent and Perspective**: The author’s perspective or intent, often revealed in prefaces, introductions, or interviews, can provide insight into whether the text is meant to address freedom and faith as central concerns. <br /> 8. **Audience and Purpose**: Consider who the intended audience is and the purpose of the text. Texts aimed at religious or philosophical audiences may have a primary focus on faith, while those addressing political or social issues may emphasize freedom. By examining these criteria, you can assess whether a text is primarily concerned with the interplay, tension, or harmony between freedom and faith. |
| 4 | Education and Teaching | Determining whether a text is primarily about Education and Teaching involves analyzing several key criteria: <br /> 1. **Subject Matter**: The text should focus on topics related to education systems, teaching methods, learning theories, curriculum development, educational policies, classroom management, or teacher-student interactions. <br /> 2. **Terminology**: The presence of specific educational terminology and jargon, such as "pedagogy," "curriculum," "lesson plans," "assessment," "learning outcomes," and "educational psychology," is indicative of a focus on education and teaching. <br /> 3. **Purpose and Objectives**: The text should aim to inform, analyze, or discuss aspects of education, such as improving teaching practices, enhancing learning experiences, or evaluating educational outcomes. <br /> 4. **Audience**: The intended audience might include educators, students, policymakers, educational researchers, or parents, which can indicate the text's focus on education and teaching. <br /> 5. **Content and Themes**: The text should explore themes such as educational equity, teacher training, student engagement, educational technology, or instructional strategies. <br /> 6. **Examples and Case Studies**: The use of examples, case studies, or anecdotes from educational settings, such as schools, universities, or training programs, can help identify the text's focus. <br /> 7. **Authors and Contributors**: Authors with backgrounds in education, such as teachers, professors, educational researchers, or policymakers, can lend credibility to the text's focus on education and teaching. <br /> 8. **Structure and Format**: The structure may include sections typical of educational discourse, such as literature reviews, methodology, discussion of results, or recommendations for practice. <br /> 9. **References and Citations**: The text may reference educational theories, landmark studies, or influential educators, indicating a basis in educational discourse. <br /> 10.**Impact and Implications**: The text should address the implications of educational practices or policies on learners, educators, and educational institutions. By examining these criteria, one can determine whether a text primarily centers on the concept of Education and Teaching. |
| 5 | Orthodox Easter | To determine whether a text is primarily about Orthodox Easter, consider the following key criteria: <br /> 1. **Date and Timing**: The text should reference the specific timing of Orthodox Easter, which is based on the Julian calendar and often falls on a different date than Western Easter. Look for mentions of the date calculation, such as the first Sunday after the first full moon following the vernal equinox, as observed by the Eastern Orthodox Church. <br /> 2. **Religious Significance**: The text should discuss the religious and spiritual significance of Orthodox Easter within the context of Eastern Orthodox Christianity. This includes references to the resurrection of Jesus Christ, which is the central event celebrated during this holiday. <br /> 3. **Cultural Practices and Traditions**: Look for descriptions of cultural and liturgical practices associated with Orthodox Easter, such as the Holy Week services, Paschal Vigil, the lighting of candles, and the proclamation of "Christ is Risen!" ("Χριστός ἀνέστη!" in Greek). <br /> 4. **Regional Observances**: The text may detail how Orthodox Easter is celebrated in various countries with significant Eastern Orthodox populations, such as Greece, Russia, Serbia, Bulgaria, and Romania. This includes traditional foods, such as kulich and paskha, and customs like egg dyeing and cracking. <br /> 5. **Comparative Aspects**: The text might compare Orthodox Easter with Western Easter, highlighting differences in dates, rituals, and theological emphases between the Eastern Orthodox Church and Western Christian denominations. <br /> 6. **Historical Context**: The text could provide historical background on the development of the Orthodox Easter celebration, including the role of the First Council of Nicaea in 325 AD in establishing the date of Easter. <br /> 7. **Language and Terminology**: Pay attention to specific terminology and language associated with Orthodox Easter, such as "Pascha," "Great Lent," "Holy Week," and other liturgical terms used in the Eastern Orthodox tradition. If a text prominently features these elements, it is likely primarily about Orthodox Easter. |
| 6 | Health Care and ACA | Determining whether a text is primarily about Health Care and the Affordable Care Act (ACA) involves assessing several key criteria: <br /> 1. **Mention of Health Care Systems**: The text should include discussions about health care systems, structures, or policies. This might involve topics like health care delivery, insurance, medical services, hospitals, or clinics. <br /> 2. **Focus on the Affordable Care Act (ACA)**: The text should specifically reference the ACA, also known as Obamacare. This could include discussions on its provisions, such as the individual mandate, Medicaid expansion, health insurance marketplaces, or subsidies. <br /> 3. **Policy and Legislation**: Look for detailed discussions on health care policies, reforms, or legislation related to the ACA. This includes legislative history, changes, or impacts of the ACA on health care access and affordability. <br /> 4. **Impact on Individuals and Communities**: The text might explore how the ACA affects individuals and communities, including changes in insurance coverage rates, access to health care services, or financial implications for patients and providers. <br /> 5. **Stakeholder Perspectives**: Consideration of perspectives from various stakeholders, such as patients, health care providers, insurers, policymakers, and advocacy groups, especially in relation to the ACA. <br /> 6. **Statistical Data and Analysis**: The presence of statistical data or analysis regarding health care coverage, costs, or outcomes that are tied to the ACA can indicate a focus on this topic. <br /> 7. **Current Events and Developments**: If the text discusses recent developments, debates, or challenges related to the ACA, such as court rulings, policy changes, or political debates, it likely focuses on this area. <br /> 8. **Comparative Analysis**: The text might compare the ACA with other health care policies or systems, highlighting its unique features or shortcomings. By evaluating these criteria, you can determine if a text primarily addresses Health Care and the ACA. The presence of multiple criteria, especially those directly referencing the ACA, strongly suggests that the text is focused on this concept. |
| 7 | American Comeback | Determining whether a text is primarily about the concept of an "American Comeback" involves looking for several key criteria and themes that are commonly associated with this idea. Here are some of the main criteria to consider: <br /> 1. **Economic Recovery**: The text should discuss aspects of economic revival in the United States. This might include topics like job growth, GDP increases, or the resurgence of key industries. <br /> 2. **Political Rhetoric**: The phrase "American Comeback" is often used in political contexts, so the text might include speeches, policy proposals, or campaign messages that emphasize restoring America's strength or prominence. <br /> 3. **Cultural Renewal**: Look for discussions on cultural revitalization, including shifts in societal values, renewed national pride, or a resurgence in cultural or artistic contributions. <br /> 4. **Historical Context**: The text might reference past periods of decline or challenge in American history, followed by recovery or improvement. This could include comparisons to previous eras of economic depression, war, or social upheaval. <br /> 5. **Innovation and Technology**: A focus on advancements in technology or innovation that are driving a resurgence in American competitiveness or leadership on the global stage. <br /> 6. **Social Progress**: The text may address improvements in social issues, such as education, healthcare, or civil rights, which contribute to a broader narrative of national improvement. <br /> 7. **Global Positioning**: Discussions on how the United States is regaining its influence or leadership role internationally, potentially after a period of decline or isolation. <br /> 8. **Challenges and Solutions**: A narrative that outlines the challenges faced by the country and the solutions or strategies being implemented to overcome them, leading to a "comeback." By examining these criteria, you can determine whether the text is primarily focused on the concept of an "American Comeback" or if it merely touches on related themes without making it the central focus. |
| 8 | Job Creation and Economy | Determining whether a text is primarily about "Job Creation and Economy" involves analyzing several key criteria. Here's a breakdown of what to look for: <br /> 1. **Keywords and Phrases**: The text should contain frequent mentions of terms related to employment and economic growth, such as "job creation," "unemployment rates," "economic development," "labor market," "workforce," "employment opportunities," "economic policies," and "business growth." <br /> 2. **Focus on Employment Trends**: The text should discuss trends in employment, such as changes in job numbers, types of jobs being created, sectors experiencing growth, or shifts in employment patterns. <br /> 3. **Economic Indicators**: It should reference economic indicators that relate to job creation, such as GDP growth, unemployment rates, labor force participation rates, or productivity metrics. <br /> 4. **Policy Discussions**: The text should cover government or institutional policies aimed at stimulating job creation and economic growth, such as tax incentives for businesses, investment in infrastructure, education and training programs, or regulatory changes. <br /> 5. **Business and Industry Analysis**: There should be an analysis of how different industries contribute to job creation and economic performance, including discussions on innovation, entrepreneurship, and business expansion. <br /> 6. **Impact on Society**: The text should examine the societal impacts of job creation and economic changes, such as income levels, quality of life, regional economic disparities, or social mobility. <br /> 7. **Case Studies or Examples**: It may include specific examples or case studies of successful job creation initiatives or economic development projects. <br /> 8. **Challenges and Solutions**: The text should address challenges facing job creation and economic growth, such as automation, globalization, or economic downturns, and propose potential solutions or strategies. By evaluating a text against these criteria, you can determine whether its primary focus is on job creation and the economy. |
| 9 | White House and Administration | Determining whether a text is primarily about the White House and Administration involves evaluating several key criteria: <br /> 1. **Subject Matter**: The text should focus on topics directly related to the White House and the executive branch of the U.S. government. This includes discussions about presidential actions, policies, and decisions, as well as the roles and activities of White House staff and administration officials. <br /> 2. **Mentions of Key Figures**: Frequent references to the President, Vice President, and high-ranking administration officials (such as the Chief of Staff, Press Secretary, or Cabinet members) suggest a focus on the White House and Administration. <br /> 3. **Policy Discussions**: The text may delve into specific policies or initiatives proposed or implemented by the administration. This can include domestic policies, foreign policy decisions, and legislative priorities. <br /> 4. **Institutional Focus**: A significant portion of the text should be devoted to the workings, structure, or changes within the White House or the executive branch, such as organizational changes, staff appointments, or internal dynamics. <br /> 5. **Events and Activities**: Coverage of events hosted by the White House, such as state dinners, press briefings, or official announcements, can indicate the text’s focus on the administration. <br /> 6. **Historical Context**: The text might provide historical context or comparisons involving past administrations, emphasizing continuity or change within the White House. <br /> 7. **Media and Public Interaction**: Discussions about how the administration interacts with the media, public opinion, or other branches of government can also signify a focus on the White House. <br /> 8. **Symbolic References**: The use of the White House as a symbol or metonym for the executive branch or the U.S. presidency can indicate that the text is centered on the administration. By assessing these criteria, one can determine if the text’s primary focus is on the White House and Administration. |

## Identify the concepts in each text and evaluate based on the criteria

Finally, we can use the concepts and the criteria to run another question where we prompt the model to evaulate each text. Question types `QuestionLinearScale`, `QuestionRank` or `QuestionNumerical` may be appropriate where we want to return a score:

```python theme={null}
from edsl import QuestionLinearScale

q_score = QuestionLinearScale(
    question_name="score",
    question_text="""Consider the following concept and criteria for determining whether
    a given text addresses this concept. Then score how well the following text satisfies
    the criteria for the concept.
    Concept: {{ scenario.concept }}
    Criteria: {{ scenario.criteria }}
    Text: {{ scenario.text }}""",
    question_options=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    option_labels={0: "Not at all", 10: "Very well"},  # Optional
)
```

Here we want to use both the texts and the concepts and corresponding criteria together as scenarios of the question:

```python theme={null}
concepts_criteria = [
    list(pair)
    for pair in zip(
        results.select("concept").to_list(), results.select("criteria").to_list()
    )
]
len(concepts_criteria)
```

```python theme={null}
10
```

````python theme={null}
from edsl import ScenarioList, Scenario

scenarios = ScenarioList(
    Scenario({"text": text, "concept": concept, "criteria": criteria})
    for text in texts
    for [concept, criteria] in concepts_criteria
)

```python
results = q_score.by(scenarios).run()
````

We can filter the results based on the responses–e.g., here we just show the non-zero scores:

```python theme={null}
(
    results
        .filter("score > 0")
        .select("text", "concept", "score")
)
```

| | scenario.text | scenario.concept | answer.score |
| :- | :- | :- | :- |
| 0 | Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. | Holocaust Remembrance | 5 |
| 1 | Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. | Freedom and Faith | 1 |
| 2 | We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. | Freedom and Faith | 1 |
| 3 | Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. | Economic Plan and Solvency | 3 |
| 4 | Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. | American Comeback | 3 |
| 5 | Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. | Job Creation and Economy | 1 |
| 6 | Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. | White House and Administration | 1 |
| 7 | This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' | Holocaust Remembrance | 9 |
| 8 | This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' | Freedom and Faith | 2 |
| 9 | The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow. | Freedom and Faith | 2 |
| 10 | The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow. | American Comeback | 1 |
| 11 | The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow. | White House and Administration | 1 |
| 12 | Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day. | Education and Teaching | 3 |
| 13 | Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day. | White House and Administration | 5 |
| 14 | Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. | Immigration and Dreamers | 7 |
| 15 | Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. | Freedom and Faith | 1 |
| 16 | Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. | Health Care and ACA | 5 |
| 17 | Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. | American Comeback | 1 |
| 18 | Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. | Job Creation and Economy | 1 |
| 19 | Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. | White House and Administration | 5 |
| 20 | With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with. | Economic Plan and Solvency | 1 |
| 21 | With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with. | American Comeback | 3 |
| 22 | With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with. | Job Creation and Economy | 5 |
| 23 | With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with. | White House and Administration | 1 |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we post the scenarios, survey and results from above, and this notebook:

```python theme={null}
# scenarios.push(description = "Example scenarios", alias = "example-scenarios", visibility = "public")
```

```python theme={null}
# survey.push(description = "Example survey", alias = "example-survey", visibility = "public")
```

```python theme={null}
# results.push(description = "Example results", alias = "example-results", visibility = "public")
```

We can also post this notebook:

```python theme={null}
# from edsl import Notebook

# nb = Notebook(path = "concept_induction.ipynb")

# nb.push(
#     description = "Example code for concept induction",
#     alias = "concept-induction-notebook",
#     visibility = "public"
# )
```

To update an object at Expected Parrot:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "concept_induction.ipynb")

nb.patch("https://www.expectedparrot.com/content/RobinHorton/concept-induction-notebook", value = nb)
```


# Simulate a qualitative interview
Source: https://docs.expectedparrot.com/en/latest/notebooks/conduct_interview

This notebook provides sample [EDSL](/en/latest/index) code for simulating an interview between a researcher and a subject, with instructions for modifying the interviewer, interview subject or topic.

The `Conversation` [module](https://github.com/expectedparrot/edsl/tree/main/edsl/conversation) can also be used to automate methods used below to simulate a conversation with multiple agents. See examples:

* [Buying a car](https://github.com/expectedparrot/edsl/blob/main/edsl/conversation/car_buying.py)
* [Negotiating a mug](https://github.com/expectedparrot/edsl/blob/main/edsl/conversation/mug_negotiation.py)

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Import the tools

Here we import the tools that we will use to conduct the interview. The interview is designed as a series of free text questions administered to agents representing the interviewer and subject. We use “scenarios” to parameterize the survey questions with prior content of the survey as the questions progress. Learn more about [EDSL question types](/en/latest/questions) and other survey components.

```python theme={null}
from edsl import QuestionFreeText, Scenario, Survey, Model, Agent

import textwrap
from rich import print
```

EDSL works with many popular language models. Learn more about [selecting models](/en/latest/language_models) to use with your surveys. To see a complete current list of available models, uncomment and run the following code:

```python theme={null}
# Model.available()
```

Here we select a model to use for the interview:

```python theme={null}
model = Model("gemini-1.5-flash")
```

## Create interview components

Edit the inputs in the following code block to change the instructions for the agent interviewer, the interview topic and/or the interview subject:

```python expandable theme={null}
# A name for the interview subject
interview_subject_name = "Chicken"

# Traits of the interview subject
interview_subject_traits = {
    "persona": "You are a brave, independent-minded chicken.",
    "status": "wild",
    "home": "A free range farm some miles away.",
    "number_of_chicks": 12,
}

# Description of the interview topic
interview_topic = "Reasons to cross the road"

# Total number of questions to ask in the interview
total_questions = 5

# Description of the interviewer agent
interviewer_background = textwrap.dedent(
    f"""
    You are an expert qualitative researcher.
    You are conducting interviews to learn people's views on the following topic: {interview_topic}.
    """
)

# Instructions for the interviewer agent
interviewer_instructions = textwrap.dedent(
    """
    You know to ask questions that are appropriate to the age and experience of an interview subject.
    You know to not ask questions that an interview subject would not be able to answer,
    e.g., if they are a young child, they probably will not be able to answer many questions about prices.
    You ask excellent follow-up questions.
    """
)
```

## Interview methods

Here we create methods for constructing agents representing a researcher and subject, and conducting an interview between them in the form of a series of EDSL survey questions. Learn more about [designing agents](/en/latest/agents) and [running surveys](/en/latest/surveys).

```python expandable theme={null}
def construct_subject(name, traits={}):
    return Agent(name=name, traits=traits)

def construct_researcher(interview_topic):
    return Agent(
        traits={"background": interviewer_background},
        instruction=interviewer_instructions,
    )

def get_next_question(subject, researcher, dialog_so_far):
    scenario = Scenario(
        {"subject": str(subject.traits), "dialog_so_far": dialog_so_far}
    )
    meta_q = QuestionFreeText(
        question_name="next_question",
        question_text="""
        These are the biographic details of the interview subject: {{ scenario.subject }}
        This is your current dialog with the interview subject: {{ scenario.dialog_so_far }}
        What question you would ask the interview subject next?
        """,
    )
    question_text = (
        meta_q.by(model)
        .by(researcher)
        .by(scenario)
        .run()
        .select("next_question")
        .first()
    )
    return question_text

def get_response_to_question(question_text, subject, dialog_so_far):
    q_to_subject = QuestionFreeText(
        question_name="question",
        question_text=f"""
        This is your current dialog with the interview subject: {dialog_so_far}.
        You are now being asked:"""
        + question_text,
    )
    response = q_to_subject.by(model).by(subject).run().select("question").first()
    return response

def ask_question(subject, researcher, dialog_so_far):
    question_text = get_next_question(subject, researcher, dialog_so_far)
    response = get_response_to_question(question_text, subject, dialog_so_far)

    print(" nQuestion: nn" + question_text + "nnResponse: nn" + response)

    return {"question": question_text, "response": response}

def dialog_to_string(d):
    return "n".join(
        [f"Question: {d['question']}nResponse: {d['response']}" for d in d]
    )

def clean_dict(d):
    """Convert dictionary to string and remove braces."""
    return str(d).replace("{", "").replace("}", "")

def summarize_interview(
    interview_subject_name,
    interview_subject_traits,
    interview_topic,
    dialog_so_far,
    researcher,
):
    summary_q = QuestionFreeText(
        question_name="summary",
        question_text=(
            f"You have just conducted the following interview of {interview_subject_name} "
            f"who has these traits: {clean_dict(interview_subject_traits)} "
            f"The topic of the interview was {interview_topic}. "
            f"Please draft a summary of the interview: {clean_dict(dialog_so_far)}"
        ),
    )
    themes_q = QuestionFreeText(
        question_name="themes", question_text="List the major themes of the interview."
    )
    survey = Survey([summary_q, themes_q]).set_full_memory_mode()
    results = survey.by(model).by(researcher).run()
    summary = results.select("summary").first()
    themes = results.select("themes").first()
    print("nnSummary:nn" + summary + "nnThemes:nn" + themes)

def conduct_interview(
    interview_subject_name, interview_subject_traits, interview_topic
):
    subject = construct_subject(
        name=interview_subject_name, traits=interview_subject_traits
    )
    researcher = construct_researcher(interview_topic=interview_topic)

    print(
        "nnInterview subject: "
        + interview_subject_name
        + "nnInterview topic: "
        + interview_topic
    )

    dialog_so_far = []

    for i in range(total_questions):
        result = ask_question(subject, researcher, dialog_to_string(dialog_so_far))
        dialog_so_far.append(result)

    summarize_interview(
        interview_subject_name,
        interview_subject_traits,
        interview_topic,
        dialog_so_far,
        researcher,
    )
```

## Conduct the interview

```python theme={null}
conduct_interview(interview_subject_name, interview_subject_traits, interview_topic)
```


# Data cleaning
Source: https://docs.expectedparrot.com/en/latest/notebooks/data_cleaning

This notebook provides sample [EDSL](/en/latest/index) code for using a language model to conduct a data cleaning task. In a series of steps we use EDSL to prompt a language model to generate appropriate sense checks for a dataset and then run the sense checks in the form of a survey about the data, returning a new dataset consisting of the data failing the checks.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below please see instructions on [getting started](https://www.expectedparrot.com/getting-started) and tips and tutorials at our [documentation page](/en/latest/index).

## Example data

EDSL allows us to generate data or [import it from other sources](/en/latest/scenarios) (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). Here we construct a dataset for our exercise: a random list of ages between 22 and 85 with some bad values mixed in. Our goal is to identify them:

```python expandable theme={null}
ages = [84, 62, 79, 57, 59, 55, 68, 66, 47, 54, 76, 33, 74, 56, 47, 24, 23, 38, 38, 54, 51, 84, 71,
        46, 38, 26, 50, 56, 62, 39, 31, 52, 69, 84, 69, 48, 48, 23, 65, 54, 78, 51, 69, 77, 75, 76,
        26, 44, 61, 32, 70, 24, 74, 22, 32, 24, 80, 65, 36, 42, 84, 66, 40, 85, 28, 22, 67, 25, 70,
        77, 53, 69, 64, 27, 61, 68, 68, 78, 0.99, 83, 58, 33, 46, 43, 50, 85, 28, 82, 50, 61, 66, 32,
        45, 70, 56, 50, 43, 30, 43, 55, 33, 72, 43, 43, -5, 32, 43, 45, 67, 84, 37, 63, 52, 53, 58,
        79, 79, 80, 62, 75, 57, 60, 39, 79, 49, 60, 60, 37, 45, 36, 1050, 73, 70, 56, 39, 58, 69, 77,
        68, 84, 78, 48, 31, 74, 27, 55, 56, 66, 35, 39, 57, 47, 29, 24, 47, 60, 43, 37, 84, 64, 28,
        22, 37, 71, 77, 76, 84, 63, 76, 58, 41, 72, 22, 63, 78, 49, 82, 69, "old", 37, 27, 29, 54, 83,
        80, 74, 48, 76, 49, 26, 38, 35, 36, 25, 23, 71, 33, 39, 40, 35, 85, 24, 57, 85, 63, 53, 62,
        47, 69, 76, 71, 48, 62, 23, 25, 84, 32, 63, 75, 31, 25, 50, 85, 36, 58, 85, 34, 62, 43, 2,
        50, 83, 44, 73, 81, 44, 43, 82, 84, 30, 24, 63, 63, 59, 46, 30, 62, 25, 52, 23, 100, 1.3, 3]
```

## Quick question

With a small dataset, we may be able to design the entire task as a single question where we prompt a model to review all the data at once and flag bad data:

```python expandable theme={null}
from edsl import QuestionList, Scenario, Model

q = QuestionList(
    question_name = "bad_ages",
    question_text = """
    Review the following list of observations of human ages
    and return a list of all the unrealistic ages: {{ scenario.ages }}
    """
)

s = Scenario({"ages":ages})

m = Model("gemini-1.5-flash", service_name = "google")

results = q.by(s).by(m).run()

results.select("bad_ages", "bad_ages_comment")
```

| | answer.bad\_ages | comment.bad\_ages\_comment |
| :- | :- | :- |
| 0 | \[0.99, -5, 1050, 1.3, 'old'] | # These values are not realistic human ages because they are either negative, less than 1, or far exceed the maximum human lifespan. "old" is also not a numerical age. |

This approach may be feasible for a small dataset that is easily checked. For larger datasets, we may encounter problems with input token limits, a model’s ability to accurately check a large volume of data at once, and responses that are not usefully formatted.

Below we demonstrate some ways of approaching the task in an iterative manner instead.

## Constructing a question

We start by creating a question to prompt a model to draft sense check questions for our data. EDSL comes with a [variety of question types](/en/latest/questions) that we can choose from based on the desired form of the response (multiple choice, free text, etc.). Here we use `QuestionList` in order to prompt the model to format its response as a list. We use a `{{ placeholder }}` for content that we will add to the question when we run it (a description of the data and a sample); this allows us to re-use the question with other contexts as desired:

```python theme={null}
from edsl import QuestionList

q = QuestionList(
    question_name = "sense_check_questions",
    question_text = """
    You are being asked to suggest sense checks for a dataset consisting of {{ scenario.data_description }}.
    Here is a small sample of the data (to demonstrate the format): {{ scenario.sample_data }}.
    Return the sense checks as a list of questions to be answered about each item in the dataset individually.
    """,
    max_list_items = 3 # optional
)
```

## Adding context to the question

Next we create `Scenario` objects representing the content that we want to add to the question when we run it. Here we create a single scenario for our example data:

```python theme={null}
import random

sample_data = random.sample(ages, 10)
```

```python theme={null}
from edsl import Scenario

s = Scenario({
    "data_description": "a list of realistic human ages (in years)",
    "sample_data": sample_data
})
s
```

```python theme={null}
Scenario
```

| | key | value |
| :- | :- | :- |
| 0 | data\_description | a list of realistic human ages (in years) |
| 1 | sample\_data:0 | 54 |
| 2 | sample\_data:1 | 48 |
| 3 | sample\_data:2 | 57 |
| 4 | sample\_data:3 | 43 |
| 5 | sample\_data:4 | 80 |
| 6 | sample\_data:5 | 58 |
| 7 | sample\_data:6 | 57 |
| 8 | sample\_data:7 | 57 |
| 9 | sample\_data:8 | 23 |
| 10 | sample\_data:9 | 72 |

## Running the question

We administer the question to a model by adding the scenarios and calling the `run` method. This generates a formatted dataset of `Results` that we can access with [built-in methods for analysis](/en/latest/results). Here we inspect the answer:

```python theme={null}
results = q.by(s).by(m).run()
```

```python theme={null}
results.select("sense_check_questions")
```

| | answer.sense\_check\_questions |
| :- | :- |
| 0 | \['Is the age a non-negative integer?', 'Is the age within a plausible range for a living human (e.g., 0-125)?', 'Is the age consistent with other ages in the dataset (considering potential context, if available)?'] |

## Conducting the task

Next we want a model to answer each sense check question about each piece of data in the dataset. This can be done by using the sense check questions as scenarios of a new question explaining the task. We can use `QuestionYesNo` to easily filter the responses:

```python theme={null}
from edsl import QuestionYesNo

q2 = QuestionYesNo(
    question_name = "check_data",
    question_text = """
    You are being asked to sense check a dataset consisting of {{ scenario.data_description }}.
    Consider the following item in the dataset: {{ scenario.age }}
    {{ scenario.sense_check_question }}
    """
)
```

We need to create a new set of scenarios for the question. We use `ScenarioList` objects to create all the combinations of values to add to the question (learn more about [constructing scenarios from different data sources](/en/latest/scenarios)):

```python theme={null}
from edsl import ScenarioList

sl = ScenarioList(
    Scenario({
        "data_description": "a list of realistic human ages (in years)",
        "age": age,
        "sense_check_question": sense_check_question
    }) for age in ages for sense_check_question in results.select("sense_check_questions").to_list()[0]
)
```

We can inspect the scenarios that we created:

```python theme={null}
sl.sample(3)
```

[ScenarioList](/en/latest/scenarios) scenarios: 3; keys: \['sense\_check\_question', 'age', 'data\_description'];

| | data\_description | age | sense\_check\_question |
| :- | :- | :- | :- |
| 0 | a list of realistic human ages (in years) | 22 | Is the age a non-negative integer? |
| 1 | a list of realistic human ages (in years) | 58 | Is the age consistent with other ages in the dataset (considering potential context, if available)? |
| 2 | a list of realistic human ages (in years) | 38 | Is the age a non-negative integer? |

Same as with a single scenario, we add all the scenarios to the question at once when we run it:

```python theme={null}
results = q2.by(sl).by(m).run()
```

We can filter, sort, select and print any components of the results that are generated:

```python theme={null}
(
    results
    .filter("check_data == 'No'")
    .sort_by("sense_check_question")
    .select("sense_check_question", "age")
)
```

| | scenario.sense\_check\_question | scenario.age |
| :- | :- | :- |
| 0 | Is the age a non-negative integer? | 0.99 |
| 1 | Is the age a non-negative integer? | -5 |
| 2 | Is the age a non-negative integer? | 1050 |
| 3 | Is the age a non-negative integer? | old |
| 4 | Is the age a non-negative integer? | 1.3 |
| 5 | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 0.99 |
| 6 | Is the age consistent with other ages in the dataset (considering potential context, if available)? | -5 |
| 7 | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 1050 |
| 8 | Is the age consistent with other ages in the dataset (considering potential context, if available)? | old |
| 9 | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 2 |
| 10 | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 100 |
| 11 | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 1.3 |
| 12 | Is the age within a plausible range for a living human (e.g., 0-125)? | -5 |
| 13 | Is the age within a plausible range for a living human (e.g., 0-125)? | 1050 |
| 14 | Is the age within a plausible range for a living human (e.g., 0-125)? | old |
| 15 | Is the age within a plausible range for a living human (e.g., 0-125)? | 1.3 |

## Further exploration

This notebook can be readily edited and expanded for other data cleaning and data labeling purposes, or to add personas for AI agents answering the questions with relevant background and expertise. [Learn more about using AI agents for your EDSL surveys](/en/latest/agents).

Please see our [documentation page](/en/latest/index) for examples of other methods and use cases and let us know if you have any questions!

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we post this notebook:

```python theme={null}
# from edsl import Notebook

# nb = Notebook(path = "data_cleaning.ipynb")

# nb.push(
#     description = "Example code for data cleaning",
#     alias = "data-cleaning-notebook",
#     visibility = "public"
# )
```

To update an object at Expected Parrot:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "data_cleaning.ipynb")

nb.patch("https://www.expectedparrot.com/content/RobinHorton/data-cleaning-notebook", value = nb)
```


# Data labeling agents
Source: https://docs.expectedparrot.com/en/latest/notebooks/data_labeling_agent

This notebook shows how to conduct data labeling tasks using [EDSL](/en/latest/index), an open-source library for simulating surveys, experiments and other research with AI agents and large language models.

This workflow consists of the following steps:

<Steps>
  <Step>
    Import data into EDSL
  </Step>

  <Step>
    Create questions about the data
  </Step>

  <Step>
    Design an AI agent to answer the questions
  </Step>

  <Step>
    Select a language model to generate responses
  </Step>

  <Step>
    Analyze results as a formatted dataset
  </Step>
</Steps>

This workflow can be visualized as follows:

<Frame>
  <img alt="general_survey.png" />
</Frame>

## Conducting agent-specific tasks

We can add a layer of complexity to this generalized flow by creating different AI agents for subsets of the data to be reviewed. For example, we can design agents with specific “expertise” to review only the data that is relevant to that expertise. This can be useful if our data is sorted (or sortable) in some way that is important to our task. We can also use EDSL to prompt a language model to sort the data as needed.

This modified workflow can be visualized as follows:

<Frame>
  <img alt="general_survey.png" />
</Frame>

## Example task: Evaluating job posts

Using a dataset of job posts as an example, in the steps below we create AI agents with expertise in the relevant job categories and then prompt them to evaluate relevant job posts in a variety of ways. The steps are:

<Steps>
  <Step>
    Import a [dataset](/en/latest/scenarios) of job categories and job posts.
  </Step>

  <Step>
    Construct [questions](/en/latest/questions) about the job posts and combine them in a [survey](/en/latest/surveys).
  </Step>

  <Step>
    Design [AI agents](/en/latest/agents) with job category expertise.
  </Step>

  <Step>
    Administer the survey to each agent with job posts for the relevant category.
  </Step>

  <Step>
    Inspect the [results](/en/latest/results) using built-in methods for analysis.
  </Step>
</Steps>

## Validating results

EDSL also provides methods for launching your surveys with human respondents to compare and validate LLM results. [Learn more](/en/latest/humanize) about these features.

## Technical setup

Before running the code below please see instructions on [getting started](https://www.expectedparrot.com/getting-started). An introductory [data labeling example notebook](/en/latest/notebooks/data_labeling_example) is also available at our [documentation page](/en/latest).

## Import the tools

We start by selecting question types and survey components that we will use. Please see the [EDSL Docs](/en/latest/index) for examples of all [question types](/en/latest/questions) and details on these basic components.

```python theme={null}
from edsl import (
    QuestionMultipleChoice, QuestionList, QuestionNumerical,
    Survey, ScenarioList, AgentList, Agent, Model
)
```

## Import data

Next we import a dataset for review, using `Scenario` objects to represent the individual data that will be added to each of our data labeling questions. EDSL allows us to create data or [import it from other sources](/en/latest/scenarios) (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.).

For purposes of demonstration, we create a dataframe, post it to Expected Parrot with the `FileStore` module and then retrieve it. Note that `FileStore` works with many file types and automatically infers the file type. [Learn more](/en/latest/filestore) about posting and retrieving files.

```python expandable theme={null}
import pandas as pd

data = [
    ["job_category", "job_title", "job_post"],
    ["Content Writing", "Blog Post Writing", "Looking for a skilled writer to produce 5 blog posts on digital marketing topics. Each post should be 800-1000 words, well-researched, and SEO-optimized."],
    ["Content Writing", "Product Description Writing", "We need a writer to craft compelling product descriptions for our online store. Each description should highlight the key features and benefits of the product."],
    ["Content Writing", "Technical Writing for Software Documentation", "Seeking an experienced technical writer to create user manuals and API documentation for our software product. Must have a background in tech writing and be familiar with software development terminology."],
    ["Content Writing", "Website Copywriting", "Looking for a copywriter to create persuasive content for our company’s website. The content should be clear, concise, and align with our brand voice."],
    ["Content Writing", "Press Release Writing", "We need a writer to draft a press release for our upcoming product launch. The release should be attention-grabbing and follow industry standards."],
    ["Digital Marketing", "Social Media Management", "We are looking for a social media manager to handle our Instagram and Twitter accounts. Responsibilities include content creation, scheduling posts, and engaging with followers."],
    ["Digital Marketing", "SEO Optimization", "Need an SEO expert to optimize our website for search engines. The project includes keyword research, on-page optimization, and link-building strategies."],
    ["Digital Marketing", "Google Ads Campaign Management", "Looking for a PPC specialist to manage our Google Ads campaigns. The goal is to increase traffic and conversions for our online store."],
    ["Digital Marketing", "Email Marketing Campaign", "Seeking an email marketing expert to design and execute a series of email campaigns for our new product launch. Experience with Mailchimp is preferred."],
    ["Digital Marketing", "Content Marketing Strategy", "Seeking a content marketing strategist to develop a comprehensive plan to increase our online visibility. The strategy should include content creation, distribution, and performance tracking."],
    ["Graphic Design", "Logo Design for New Startup", "We are a new tech startup looking for a creative designer to create a unique logo for our brand. The logo should be modern and represent innovation. Please provide portfolio examples."],
    ["Graphic Design", "Brochure Design", "Looking for an experienced designer to create a professional brochure for our real estate company. The brochure should highlight our services and properties. Must be delivered in print-ready format."],
    ["Graphic Design", "Social Media Graphics", "Need a designer to create eye-catching social media graphics for our upcoming campaign. We need a set of 10 images optimized for Instagram and Facebook."],
    ["Graphic Design", "Website Banner Design", "Seeking a skilled designer to create a series of banners for our e-commerce website. Banners should be consistent with our brand’s aesthetic. Please include examples of previous work."],
    ["Graphic Design", "Infographic Design", "We need a designer to create a visually appealing infographic based on our provided data. The infographic should be easy to understand and shareable on social media."],
    ["Web Development", "WordPress Website Setup", "We need a developer to set up a WordPress site for our small business. The site should be responsive and include a contact form, blog, and e-commerce functionality. Experience with WooCommerce is a plus."],
    ["Web Development", "Custom Web Application Development", "Looking for a full-stack developer to build a custom web application for managing employee schedules. The app should include a login system, user roles, and reporting features."],
    ["Web Development", "Shopify Store Customization", "Seeking a Shopify expert to customize our online store. We need theme adjustments, product page enhancements, and integration with third-party tools."],
    ["Web Development", "API Integration", "Need a developer to integrate our existing CRM system with an external API. The integration should sync customer data in real-time. Previous experience with similar projects required."],
    ["Web Development", "Landing Page Development", "Looking for a developer to create a high-converting landing page for our marketing campaign. The page should be optimized for mobile and desktop users."]
]

df = pd.DataFrame(data[1:], columns=data[0])
df
```

| | job\_category | job\_title | job\_post |
| :- | :- | :- | :- |
| 0 | Content Writing | Blog Post Writing | Looking for a skilled writer to produce 5 blog... |
| 1 | Content Writing | Product Description Writing | We need a writer to craft compelling product d... |
| 2 | Content Writing | Technical Writing for Software Documentation | Seeking an experienced technical writer to cre... |
| 3 | Content Writing | Website Copywriting | Looking for a copywriter to create persuasive ... |
| 4 | Content Writing | Press Release Writing | We need a writer to draft a press release for ... |
| 5 | Digital Marketing | Social Media Management | We are looking for a social media manager to h... |
| 6 | Digital Marketing | SEO Optimization | Need an SEO expert to optimize our website for... |
| 7 | Digital Marketing | Google Ads Campaign Management | Looking for a PPC specialist to manage our Goo... |
| 8 | Digital Marketing | Email Marketing Campaign | Seeking an email marketing expert to design an... |
| 9 | Digital Marketing | Content Marketing Strategy | Seeking a content marketing strategist to deve... |
| 10 | Graphic Design | Logo Design for New Startup | We are a new tech startup looking for a creati... |
| 11 | Graphic Design | Brochure Design | Looking for an experienced designer to create ... |
| 12 | Graphic Design | Social Media Graphics | Need a designer to create eye-catching social ... |
| 13 | Graphic Design | Website Banner Design | Seeking a skilled designer to create a series ... |
| 14 | Graphic Design | Infographic Design | We need a designer to create a visually appeal... |
| 15 | Web Development | WordPress Website Setup | We need a developer to set up a WordPress site... |
| 16 | Web Development | Custom Web Application Development | Looking for a full-stack developer to build a ... |
| 17 | Web Development | Shopify Store Customization | Seeking a Shopify expert to customize our onli... |
| 18 | Web Development | API Integration | Need a developer to integrate our existing CRM... |
| 19 | Web Development | Landing Page Development | Looking for a developer to create a high-conve... |

```python theme={null}
df.to_csv("data.csv")
```

Here we post the file to Expected Parrot and get the information for the object:

```python theme={null}
from edsl import FileStore

fs = FileStore("data.csv")

fs.push(
    description = "Example CSV file: Job categories",
    alias = "filestore-csv-example",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example CSV file: Job categories',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/00b84308-166e-4828-9a3a-04cb93cd3222',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/filestore-csv-example',
 'uuid': '00b84308-166e-4828-9a3a-04cb93cd3222',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```

Next we retrieve the data file and use it to create scenarios (replace this code with the UUID of any file you want to use):

```python theme={null}
from edsl import FileStore, ScenarioList

fs = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/filestore-csv-example")

scenarios = ScenarioList.from_source("csv", fs.to_tempfile())
scenarios
```

```
defaultdict(<function DelimitedFileSource.to_scenario_list.<locals>.<lambda> at 0x1059d5940>, {})
defaultdict(<function DelimitedFileSource.to_scenario_list.<locals>.<lambda> at 0x1059d5940>, {'': 1})
defaultdict(<function DelimitedFileSource.to_scenario_list.<locals>.<lambda> at 0x1059d5940>, {'': 1, 'job_category': 1})
defaultdict(<function DelimitedFileSource.to_scenario_list.<locals>.<lambda> at 0x1059d5940>, {'': 1, 'job_category': 1, 'job_title': 1})
```

[ScenarioList](/en/latest/scenarios) scenarios: 20; keys: \['', 'job\_title', 'job\_category', 'job\_post'];

| | Unnamed: 0 | job\_category | job\_title | job\_post |
| :- | :- | :- | :- | :- |
| 0 | 0 | Content Writing | Blog Post Writing | Looking for a skilled writer to produce 5 blog posts on digital marketing topics. Each post should be 800-1000 words, well-researched, and SEO-optimized. |
| 1 | 1 | Content Writing | Product Description Writing | We need a writer to craft compelling product descriptions for our online store. Each description should highlight the key features and benefits of the product. |
| 2 | 2 | Content Writing | Technical Writing for Software Documentation | Seeking an experienced technical writer to create user manuals and API documentation for our software product. Must have a background in tech writing and be familiar with software development terminology. |
| 3 | 3 | Content Writing | Website Copywriting | Looking for a copywriter to create persuasive content for our company’s website. The content should be clear, concise, and align with our brand voice. |
| 4 | 4 | Content Writing | Press Release Writing | We need a writer to draft a press release for our upcoming product launch. The release should be attention-grabbing and follow industry standards. |
| 5 | 5 | Digital Marketing | Social Media Management | We are looking for a social media manager to handle our Instagram and Twitter accounts. Responsibilities include content creation, scheduling posts, and engaging with followers. |
| 6 | 6 | Digital Marketing | SEO Optimization | Need an SEO expert to optimize our website for search engines. The project includes keyword research, on-page optimization, and link-building strategies. |
| 7 | 7 | Digital Marketing | Google Ads Campaign Management | Looking for a PPC specialist to manage our Google Ads campaigns. The goal is to increase traffic and conversions for our online store. |
| 8 | 8 | Digital Marketing | Email Marketing Campaign | Seeking an email marketing expert to design and execute a series of email campaigns for our new product launch. Experience with Mailchimp is preferred. |
| 9 | 9 | Digital Marketing | Content Marketing Strategy | Seeking a content marketing strategist to develop a comprehensive plan to increase our online visibility. The strategy should include content creation, distribution, and performance tracking. |
| 10 | 10 | Graphic Design | Logo Design for New Startup | We are a new tech startup looking for a creative designer to create a unique logo for our brand. The logo should be modern and represent innovation. Please provide portfolio examples. |
| 11 | 11 | Graphic Design | Brochure Design | Looking for an experienced designer to create a professional brochure for our real estate company. The brochure should highlight our services and properties. Must be delivered in print-ready format. |
| 12 | 12 | Graphic Design | Social Media Graphics | Need a designer to create eye-catching social media graphics for our upcoming campaign. We need a set of 10 images optimized for Instagram and Facebook. |
| 13 | 13 | Graphic Design | Website Banner Design | Seeking a skilled designer to create a series of banners for our e-commerce website. Banners should be consistent with our brand’s aesthetic. Please include examples of previous work. |
| 14 | 14 | Graphic Design | Infographic Design | We need a designer to create a visually appealing infographic based on our provided data. The infographic should be easy to understand and shareable on social media. |
| 15 | 15 | Web Development | WordPress Website Setup | We need a developer to set up a WordPress site for our small business. The site should be responsive and include a contact form, blog, and e-commerce functionality. Experience with WooCommerce is a plus. |
| 16 | 16 | Web Development | Custom Web Application Development | Looking for a full-stack developer to build a custom web application for managing employee schedules. The app should include a login system, user roles, and reporting features. |
| 17 | 17 | Web Development | Shopify Store Customization | Seeking a Shopify expert to customize our online store. We need theme adjustments, product page enhancements, and integration with third-party tools. |
| 18 | 18 | Web Development | API Integration | Need a developer to integrate our existing CRM system with an external API. The integration should sync customer data in real-time. Previous experience with similar projects required. |
| 19 | 19 | Web Development | Landing Page Development | Looking for a developer to create a high-converting landing page for our marketing campaign. The page should be optimized for mobile and desktop users. |

## Construct questions about the data

Next we construct questions to ask about the job posts, selecting question types based on the form of the response that we want to get back from the language model (multiple choice, linear scale, free text, numerical, etc.–see [examples of all question types](/en/latest/questions)). We include a `{{ placeholder }}` for the scenario keys in order to parameterize each question with each job post and category when we run the survey:

```python expandable theme={null}
q_skills = QuestionList(
    question_name="skills",
    question_text="""
    Consider the following job category and job post at an online labor marketplace.
    Job category: `{{ scenario.job_category }}
    Job post: {{ scenario.job_post }}
    What are some key skills required for this job?
    """,
)

q_experience = QuestionMultipleChoice(
    question_name="experience",
    question_text="""
    Consider the following job category and job post at an online labor marketplace.
    Job category: {{ scenario.job_category }}
    Job post: {{ scenario.job_post }}
    What level of experience is required for this job?
    """,
    question_options=["Entry-level", "Mid-level", "Senior-level"],
)

q_days = QuestionNumerical(
    question_name="days",
    question_text="""
    Consider the following job category and job post at an online labor marketplace.
    Job category: {{ scenario.job_category }}
    Job post: {{ scenario.job_post }}
    Estimate the number of days until this job post is fulfilled.
    """,
)
```

## Combining questions into a Survey

Next we combine our questions into a survey that will be administered to the AI agents. By default, the questions will be administered asynchronously. If desired, we can also specify survey rules (skip/stop logic) and within-survey memories of prior questions and responses. See the EDSL Docs for details on methods for [applying survey rules](/en/latest/surveys#applying-survey-rules).

```python theme={null}
survey = Survey(questions=[q_skills, q_experience, q_days])
```

## Creating personas for Agents

Next we draft personas for AI agents that will answer the questions. For each job category we construct an AI agent that is an expert in the category. Agents are constructed by passing a dictionary of `traits` to an `Agent` object. [Learn more about designing AI agents](/en/latest/agents) to answer surveys.

To get the set of job categories from the scenarios:

```python theme={null}
job_categories = list(set(scenarios.select("job_category").to_list()))
job_categories
```

```
['Digital Marketing', 'Web Development', 'Graphic Design', 'Content Writing']
```

Next we use them to create an agent for each job category:

```python theme={null}
agents = AgentList(
    Agent(
        traits = {
            "persona": "You are an experienced freelancer on online labor marketplaces.",
            "job_category": job_category,
            "expertise": f"You regularly perform jobs in the following category: {job_category}."
        }
    ) for job_category in job_categories
)
agents
```

```
AgentList agents: 4;
```

| | persona | job\_category | expertise |
| :- | :- | :- | :- |
| 0 | You are an experienced freelancer on online labor marketplaces. | Digital Marketing | You regularly perform jobs in the following category: Digital Marketing. |
| 1 | You are an experienced freelancer on online labor marketplaces. | Web Development | You regularly perform jobs in the following category: Web Development. |
| 2 | You are an experienced freelancer on online labor marketplaces. | Graphic Design | You regularly perform jobs in the following category: Graphic Design. |
| 3 | You are an experienced freelancer on online labor marketplaces. | Content Writing | You regularly perform jobs in the following category: Content Writing. |

## Selecting language models

EDSL works with many popular language models that we can select to generate the agents’ responses to the survey. Information about current models is [available here](https://www.expectedparrot.com/models). To check a list of service providers:

```python theme={null}
Model.services()
```

| | Service Name |
| :- | :- |
| 0 | anthropic |
| 1 | azure |
| 2 | bedrock |
| 3 | deep\_infra |
| 4 | deepseek |
| 5 | google |
| 6 | groq |
| 7 | mistral |
| 8 | ollama |
| 9 | openai |
| 10 | openai\_v2 |
| 11 | perplexity |
| 12 | together |
| 13 | xai |

Here we specify a model to use to generate responses (if we do not specify a model, gpt-5.2 is used by default):

```python theme={null}
model = Model("gemini-1.5-flash", service_name = "google")
```

## Running the survey

We administer a survey by appending the components with the `by()` method and then calling `run()` method. In the simplest case where we want a single agent or list of agents to answer all questions with the same scenarios, this takes the following form:

```python theme={null}
results = survey.by(scenarios).by(agents).by(models).run()
```

Here we have individual agents answer the questions only for category-specific job posts, and then combine the results:

```python expandable theme={null}
results = None

for job_category in job_categories:
    print("nnJob category: ", job_category)

    # Create an agent for the job category
    a = agents.filter(f"job_category == '{job_category}'")

    # Filter the relevant scenarios
    s = scenarios.filter(f"job_category == '{job_category}'")

    # Run the survey with the agent and scenarios
    job_category_results = survey.by(s).by(a).run()

    # Store the results
    if results == None:
        results = job_category_results

    else:
        results = results + job_category_results

Job category:  Digital Marketing
```

## Accessing Results

In the previous step we created `Results` for individual agents’ responses and combined them. Next we show how to inspect and analyze results with built-in methods.

We can identify the column names to select the fields that we want to inspect:

```python theme={null}
results.columns

/Users/a16174/edsl/edsl/results/result.py:382: UserWarning: Key 'job_category' of data type 'scenario' is already in use. Renaming to job_category_scenario
  warnings.warn(
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | agent.expertise |
| 4 | agent.job\_category |
| 5 | agent.persona |
| 6 | answer.days |
| 7 | answer.experience |
| 8 | answer.skills |
| 9 | cache\_keys.days\_cache\_key |
| 10 | cache\_keys.experience\_cache\_key |
| 11 | cache\_keys.skills\_cache\_key |
| 12 | cache\_used.days\_cache\_used |
| 13 | cache\_used.experience\_cache\_used |
| 14 | cache\_used.skills\_cache\_used |
| 15 | comment.days\_comment |
| 16 | comment.experience\_comment |
| 17 | comment.skills\_comment |
| 18 | generated\_tokens.days\_generated\_tokens |
| 19 | generated\_tokens.experience\_generated\_tokens |
| 20 | generated\_tokens.skills\_generated\_tokens |
| 21 | iteration.iteration |
| 22 | model.frequency\_penalty |
| 23 | model.inference\_service |
| 24 | model.logprobs |
| 25 | model.max\_tokens |
| 26 | model.model |
| 27 | model.model\_index |
| 28 | model.presence\_penalty |
| 29 | model.temperature |
| 30 | model.top\_logprobs |
| 31 | model.top\_p |
| 32 | prompt.days\_system\_prompt |
| 33 | prompt.days\_user\_prompt |
| 34 | prompt.experience\_system\_prompt |
| 35 | prompt.experience\_user\_prompt |
| 36 | prompt.skills\_system\_prompt |
| 37 | prompt.skills\_user\_prompt |
| 38 | question\_options.days\_question\_options |
| 39 | question\_options.experience\_question\_options |
| 40 | question\_options.skills\_question\_options |
| 41 | question\_text.days\_question\_text |
| 42 | question\_text.experience\_question\_text |
| 43 | question\_text.skills\_question\_text |
| 44 | question\_type.days\_question\_type |
| 45 | question\_type.experience\_question\_type |
| 46 | question\_type.skills\_question\_type |
| 47 | raw\_model\_response.days\_cost |
| 48 | raw\_model\_response.days\_input\_price\_per\_million\_tokens |
| 49 | raw\_model\_response.days\_input\_tokens |
| 50 | raw\_model\_response.days\_one\_usd\_buys |
| 51 | raw\_model\_response.days\_output\_price\_per\_million\_tokens |
| 52 | raw\_model\_response.days\_output\_tokens |
| 53 | raw\_model\_response.days\_raw\_model\_response |
| 54 | raw\_model\_response.experience\_cost |
| 55 | raw\_model\_response.experience\_input\_price\_per\_million\_tokens |
| 56 | raw\_model\_response.experience\_input\_tokens |
| 57 | raw\_model\_response.experience\_one\_usd\_buys |
| 58 | raw\_model\_response.experience\_output\_price\_per\_million\_tokens |
| 59 | raw\_model\_response.experience\_output\_tokens |
| 60 | raw\_model\_response.experience\_raw\_model\_response |
| 61 | raw\_model\_response.skills\_cost |
| 62 | raw\_model\_response.skills\_input\_price\_per\_million\_tokens |
| 63 | raw\_model\_response.skills\_input\_tokens |
| 64 | raw\_model\_response.skills\_one\_usd\_buys |
| 65 | raw\_model\_response.skills\_output\_price\_per\_million\_tokens |
| 66 | raw\_model\_response.skills\_output\_tokens |
| 67 | raw\_model\_response.skills\_raw\_model\_response |
| 68 | reasoning\_summary.days\_reasoning\_summary |
| 69 | reasoning\_summary.experience\_reasoning\_summary |
| 70 | reasoning\_summary.skills\_reasoning\_summary |
| 71 | scenario. |
| 72 | scenario.job\_category\_scenario |
| 73 | scenario.job\_post |
| 74 | scenario.job\_title |
| 75 | scenario.scenario\_index |

We can select individual fields in a variety of ways:

```python theme={null}
(
    results
    .filter("job_category == 'Graphic Design'")
    .select("job_post", "skills", "experience", "days")
)
```

| | scenario.job\_post | answer.skills | answer.experience | answer.days |
| :- | :- | :- | :- | :- |
| 0 | We are a new tech startup looking for a creative designer to create a unique logo for our brand. The logo should be modern and represent innovation. Please provide portfolio examples. | \['Creativity', 'Logo Design', 'Branding', 'Adobe Illustrator', 'Portfolio Presentation'] | Mid-level | 7.000000 |
| 1 | Looking for an experienced designer to create a professional brochure for our real estate company. The brochure should highlight our services and properties. Must be delivered in print-ready format. | \['Adobe InDesign', 'Adobe Photoshop', 'Typography', 'Layout Design', 'Attention to Detail'] | Mid-level | 5.000000 |
| 2 | Need a designer to create eye-catching social media graphics for our upcoming campaign. We need a set of 10 images optimized for Instagram and Facebook. | \['Graphic Design', 'Social Media Marketing', 'Adobe Photoshop', 'Creativity', 'Attention to Detail'] | Mid-level | 3.500000 |
| 3 | Seeking a skilled designer to create a series of banners for our e-commerce website. Banners should be consistent with our brand’s aesthetic. Please include examples of previous work. | \['Adobe Photoshop', 'Adobe Illustrator', 'Branding', 'Creativity', 'Attention to Detail'] | Mid-level | 3.000000 |
| 4 | We need a designer to create a visually appealing infographic based on our provided data. The infographic should be easy to understand and shareable on social media. | \['Data visualization', 'Adobe Illustrator', 'Creativity', 'Attention to detail', 'Social media design'] | Mid-level | 3.000000 |

We can apply some labels to our table for readability. Note that each question field also automatically includes a `<question>_comment` field for any commentary by the LLM on the question:

```python theme={null}
(
    results
    .filter("job_category == 'Graphic Design'")
    .select("job_post", "experience", "experience_comment")
    .print(
        pretty_labels={
            "scenario.job_post": "Job post description",
            "answer.experience": "Experience level",
            "answer.experience_comment": "Comment",
        }
    )
)
```

| | Job post description | Experience level | comment.experience\_comment |
| :- | :- | :- | :- |
| 0 | We are a new tech startup looking for a creative designer to create a unique logo for our brand. The logo should be modern and represent innovation. Please provide portfolio examples. | Mid-level | This job requires a designer who can create a unique logo that is both modern and represents innovation, which typically requires a fair amount of creativity and experience. Additionally, the request for portfolio examples suggests that they are looking for someone with a proven track record, which aligns with a mid-level position. |
| 1 | Looking for an experienced designer to create a professional brochure for our real estate company. The brochure should highlight our services and properties. Must be delivered in print-ready format. | Mid-level | This job requires creating a professional brochure that highlights services and properties, and it needs to be delivered in a print-ready format. This suggests the need for someone with a good understanding of design principles and experience in preparing files for print, which typically aligns with a mid-level designer. |
| 2 | Need a designer to create eye-catching social media graphics for our upcoming campaign. We need a set of 10 images optimized for Instagram and Facebook. | Mid-level | This job requires creating optimized and eye-catching designs for social media, which suggests the need for a designer with a solid understanding of design principles and experience with social media platforms. |
| 3 | Seeking a skilled designer to create a series of banners for our e-commerce website. Banners should be consistent with our brand’s aesthetic. Please include examples of previous work. | Mid-level | This job requires a designer who has enough experience to create banners that are consistent with a brand's aesthetic, which typically involves a good understanding of design principles and brand identity. The request for examples of previous work suggests they are looking for someone with a proven track record, indicating a mid-level position. |
| 4 | We need a designer to create a visually appealing infographic based on our provided data. The infographic should be easy to understand and shareable on social media. | Mid-level | This job requires the ability to interpret data and create a visually appealing design that is also optimized for social media sharing, which typically requires some experience beyond entry-level. |

We can also access results as a SQL table (called `self`) with the `.sql()` method, and optionally removing the column name prefixes ‘agent’, ‘model’, ‘prompt’, etc.:

```python theme={null}
results.sql("select * from self")
```

| | skills | days | experience | Unnamed: 3 | job\_title | job\_category\_scenario | scenario\_index | job\_post | persona | agent\_instruction | agent\_index | job\_category | expertise | agent\_name | model\_index | model | frequency\_penalty | inference\_service | presence\_penalty | max\_tokens | top\_p | top\_logprobs | temperature | logprobs | skills\_user\_prompt | days\_user\_prompt | experience\_user\_prompt | days\_system\_prompt | skills\_system\_prompt | experience\_system\_prompt | skills\_cost | days\_cost | experience\_input\_tokens | skills\_input\_price\_per\_million\_tokens | days\_input\_price\_per\_million\_tokens | experience\_cost | days\_one\_usd\_buys | experience\_raw\_model\_response | experience\_one\_usd\_buys | days\_input\_tokens | skills\_output\_price\_per\_million\_tokens | skills\_one\_usd\_buys | days\_raw\_model\_response | experience\_output\_price\_per\_million\_tokens | skills\_input\_tokens | days\_output\_tokens | days\_output\_price\_per\_million\_tokens | experience\_output\_tokens | skills\_output\_tokens | skills\_raw\_model\_response | experience\_input\_price\_per\_million\_tokens | iteration | experience\_question\_text | skills\_question\_text | days\_question\_text | experience\_question\_options | days\_question\_options | skills\_question\_options | experience\_question\_type | skills\_question\_type | days\_question\_type | experience\_comment | skills\_comment | days\_comment | experience\_generated\_tokens | days\_generated\_tokens | skills\_generated\_tokens | skills\_cache\_used | experience\_cache\_used | days\_cache\_used | days\_cache\_key | skills\_cache\_key | experience\_cache\_key | experience\_reasoning\_summary | days\_reasoning\_summary | skills\_reasoning\_summary |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | \['Content Creation', 'Social Media Management', 'Engagement Strategies', 'Scheduling Tools', 'Communication Skills'] | 7.000000 | Mid-level | 5 | Social Media Management | Digital Marketing | 0 | We are looking for a social media manager to handle our Instagram and Twitter accounts. Responsibilities include content creation, scheduling posts, and engaging with followers. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Digital Marketing | You regularly perform jobs in the following category: Digital Marketing. | Agent\_0 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: We are looking for a social media manager to handle our Instagram and Twitter accounts. Responsibilities include content creation, scheduling posts, and engaging with followers. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: We are looking for a social media manager to handle our Instagram and Twitter accounts. Responsibilities include content creation, scheduling posts, and engaging with followers. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: We are looking for a social media manager to handle our Instagram and Twitter accounts. Responsibilities include content creation, scheduling posts, and engaging with followers. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | 0.001050 | 0.000918 | 189 | 2.500000 | 2.500000 | 0.001032 | 1089.918256 | `{'id': 'chatcmpl-BiJTXoBpBD5TVJpxokFgdeL2Avw1N', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis job requires managing multiple social media accounts, creating content, scheduling posts, and engaging with followers, which suggests a level of experience beyond entry-level. However, it does not specify advanced strategic planning or management of large teams, which would typically require senior-level experience.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749900995, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_a288987b44', 'usage': {'completion_tokens': 56, 'prompt_tokens': 189, 'total_tokens': 245, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 968.523002 | 215 | 10.000000 | 952.380952 | `{'id': 'chatcmpl-BiJTXRMBqew4DDdH0VISPk3oA9qIF', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '7n# Social media management roles are often filled quickly due to high demand and a large pool of qualified freelancers. A week is a reasonable estimate for a job like this to be filled.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749900995, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_a288987b44', 'usage': {'completion_tokens': 38, 'prompt_tokens': 215, 'total_tokens': 253, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 192 | 38 | 10.000000 | 56 | 57 | `{'id': 'chatcmpl-BiJTXP7GEaxsTi0LDMB47WDdl1vfa', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Content Creation", "Social Media Management", "Engagement Strategies", "Scheduling Tools", "Communication Skills"] nThese skills are essential for managing social media accounts effectively, as they involve creating engaging content, using tools for scheduling posts, and interacting with the audience to build a community.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749900995, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_a288987b44', 'usage': {'completion_tokens': 57, 'prompt_tokens': 192, 'total_tokens': 249, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires managing multiple social media accounts, creating content, scheduling posts, and engaging with followers, which suggests a level of experience beyond entry-level. However, it does not specify advanced strategic planning or management of large teams, which would typically require senior-level experience. | These skills are essential for managing social media accounts effectively, as they involve creating engaging content, using tools for scheduling posts, and interacting with the audience to build a community. | # Social media management roles are often filled quickly due to high demand and a large pool of qualified freelancers. A week is a reasonable estimate for a job like this to be filled. | Mid-level This job requires managing multiple social media accounts, creating content, scheduling posts, and engaging with followers, which suggests a level of experience beyond entry-level. However, it does not specify advanced strategic planning or management of large teams, which would typically require senior-level experience. | 7 # Social media management roles are often filled quickly due to high demand and a large pool of qualified freelancers. A week is a reasonable estimate for a job like this to be filled. | \["Content Creation", "Social Media Management", "Engagement Strategies", "Scheduling Tools", "Communication Skills"] These skills are essential for managing social media accounts effectively, as they involve creating engaging content, using tools for scheduling posts, and interacting with the audience to build a community. | 0 | 0 | 0 | a9e83bba795dc006a06d1d4c3c9de1b9 | 666887b34bf5b7e5b5b6580695326f94 | ead05caf4707af6a1d1aab9ce9c70bab | nan | nan | nan |
| 1 | \['Keyword research', 'On-page optimization', 'Link building', 'SEO tools proficiency', 'Analytical skills'] | 7.000000 | Mid-level | 6 | SEO Optimization | Digital Marketing | 1 | Need an SEO expert to optimize our website for search engines. The project includes keyword research, on-page optimization, and link-building strategies. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Digital Marketing | You regularly perform jobs in the following category: Digital Marketing. | Agent\_1 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Need an SEO expert to optimize our website for search engines. The project includes keyword research, on-page optimization, and link-building strategies. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Need an SEO expert to optimize our website for search engines. The project includes keyword research, on-page optimization, and link-building strategies. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Need an SEO expert to optimize our website for search engines. The project includes keyword research, on-page optimization, and link-building strategies. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | 0.001175 | 0.000953 | 187 | 2.500000 | 2.500000 | 0.000798 | 1049.868766 | `{'id': 'chatcmpl-BiJTXwvn4iJSgGuArgYB3qsXh4NF9', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis project requires a comprehensive understanding of SEO, including keyword research, on-page optimization, and link-building strategies, which typically aligns with mid-level experience.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749900995, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_a288987b44', 'usage': {'completion_tokens': 33, 'prompt_tokens': 187, 'total_tokens': 220, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1253.918495 | 213 | 10.000000 | 851.063830 | `{'id': 'chatcmpl-BiJTX7e3hKGYlbXtDR73h16nyXwcZ', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': "7n# Typically, SEO projects like this one can attract a lot of qualified freelancers quickly. Depending on the platform's activity level, it usually takes about a week for such a job post to be fulfilled.", 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749900995, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 42, 'prompt_tokens': 213, 'total_tokens': 255, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 190 | 42 | 10.000000 | 33 | 70 | `{'id': 'chatcmpl-BiJTX2i1ZV1vilz7oW8s9j53qnKGD', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Keyword research", "On-page optimization", "Link building", "SEO tools proficiency", "Analytical skills"] nThese skills are essential for executing the tasks mentioned in the job post, such as keyword research, on-page optimization, and link-building strategies. Proficiency in SEO tools and analytical skills are also crucial for assessing and improving website performance.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749900995, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 70, 'prompt_tokens': 190, 'total_tokens': 260, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This project requires a comprehensive understanding of SEO, including keyword research, on-page optimization, and link-building strategies, which typically aligns with mid-level experience. | These skills are essential for executing the tasks mentioned in the job post, such as keyword research, on-page optimization, and link-building strategies. Proficiency in SEO tools and analytical skills are also crucial for assessing and improving website performance. | # Typically, SEO projects like this one can attract a lot of qualified freelancers quickly. Depending on the platform's activity level, it usually takes about a week for such a job post to be fulfilled. | Mid-level This project requires a comprehensive understanding of SEO, including keyword research, on-page optimization, and link-building strategies, which typically aligns with mid-level experience. | 7 # Typically, SEO projects like this one can attract a lot of qualified freelancers quickly. Depending on the platform's activity level, it usually takes about a week for such a job post to be fulfilled. | \["Keyword research", "On-page optimization", "Link building", "SEO tools proficiency", "Analytical skills"] These skills are essential for executing the tasks mentioned in the job post, such as keyword research, on-page optimization, and link-building strategies. Proficiency in SEO tools and analytical skills are also crucial for assessing and improving website performance. | 0 | 0 | 0 | a5e232fe36f099dc0ba6d59c64eff11b | 9788838ebb39b869318190000ccf9829 | d5dcbbdd4b2736ba87893c1e78e47a06 | nan | nan | nan |
| 2 | \['Google Ads expertise', 'Keyword research', 'Analytical skills', 'Conversion rate optimization', 'Budget management'] | 7.000000 | Mid-level | 7 | Google Ads Campaign Management | Digital Marketing | 2 | Looking for a PPC specialist to manage our Google Ads campaigns. The goal is to increase traffic and conversions for our online store. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Digital Marketing | You regularly perform jobs in the following category: Digital Marketing. | Agent\_2 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Looking for a PPC specialist to manage our Google Ads campaigns. The goal is to increase traffic and conversions for our online store. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Looking for a PPC specialist to manage our Google Ads campaigns. The goal is to increase traffic and conversions for our online store. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Looking for a PPC specialist to manage our Google Ads campaigns. The goal is to increase traffic and conversions for our online store. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | 0.001020 | 0.000897 | 185 | 2.500000 | 2.500000 | 0.000803 | 1114.206128 | `{'id': 'chatcmpl-B6kP5VM2gmZMtqzTmohLzDUYohHAX', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-level nThis job requires managing Google Ads campaigns with a focus on increasing traffic and conversions, which typically requires a certain level of expertise and experience beyond entry-level.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948043, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 34, 'prompt_tokens': 185, 'total_tokens': 219, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1246.105919 | 211 | 10.000000 | 980.392157 | `{'id': 'chatcmpl-B6kP5I8VkMRP8KA5A2h2VAppFcUJK', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '7 nGiven the high demand for PPC specialists and the clear job description, it typically takes about a week to find a suitable candidate in the digital marketing field on online labor marketplaces.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948043, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 37, 'prompt_tokens': 211, 'total_tokens': 248, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 188 | 37 | 10.000000 | 34 | 55 | `{'id': 'chatcmpl-B6kP5UdjNRqtUVVtqL61ckUqSBF6e', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Google Ads expertise", "Keyword research", "Analytical skills", "Conversion rate optimization", "Budget management"] n// These skills are crucial for a PPC specialist because they need to effectively manage and optimize Google Ads campaigns to increase traffic and conversions within a specified budget.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948043, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 55, 'prompt_tokens': 188, 'total_tokens': 243, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires managing Google Ads campaigns with a focus on increasing traffic and conversions, which typically requires a certain level of expertise and experience beyond entry-level. | // These skills are crucial for a PPC specialist because they need to effectively manage and optimize Google Ads campaigns to increase traffic and conversions within a specified budget. | Given the high demand for PPC specialists and the clear job description, it typically takes about a week to find a suitable candidate in the digital marketing field on online labor marketplaces. | Mid-level This job requires managing Google Ads campaigns with a focus on increasing traffic and conversions, which typically requires a certain level of expertise and experience beyond entry-level. | 7 Given the high demand for PPC specialists and the clear job description, it typically takes about a week to find a suitable candidate in the digital marketing field on online labor marketplaces. | \["Google Ads expertise", "Keyword research", "Analytical skills", "Conversion rate optimization", "Budget management"] // These skills are crucial for a PPC specialist because they need to effectively manage and optimize Google Ads campaigns to increase traffic and conversions within a specified budget. | 1 | 1 | 1 | aae6d33bde4d33be946779f12185f8db | a9190111c7628117073e32632b791bdc | 50fbc0540007d01a841c58b06b819d22 | nan | nan | nan |
| 3 | \['Email Campaign Strategy', 'Mailchimp Proficiency', 'Copywriting', 'Analytics and Reporting', 'A/B Testing'] | 7.000000 | Mid-level | 8 | Email Marketing Campaign | Digital Marketing | 3 | Seeking an email marketing expert to design and execute a series of email campaigns for our new product launch. Experience with Mailchimp is preferred. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Digital Marketing | You regularly perform jobs in the following category: Digital Marketing. | Agent\_3 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Seeking an email marketing expert to design and execute a series of email campaigns for our new product launch. Experience with Mailchimp is preferred. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Seeking an email marketing expert to design and execute a series of email campaigns for our new product launch. Experience with Mailchimp is preferred. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Seeking an email marketing expert to design and execute a series of email campaigns for our new product launch. Experience with Mailchimp is preferred. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | 0.001095 | 0.000953 | 187 | 2.500000 | 2.500000 | 0.000847 | 1049.868766 | `{'id': 'chatcmpl-B6kP5mpHby1dFnjMNasOXnnDGtacQ', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThe job post requires designing and executing a series of email campaigns, which suggests a need for some experience and expertise, particularly with Mailchimp, indicating a mid-level requirement.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948043, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 38, 'prompt_tokens': 187, 'total_tokens': 225, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1179.941003 | 213 | 10.000000 | 913.242009 | `{'id': 'chatcmpl-B6kP5xdT8xDPwQVe8tbZNzGsMpg6e', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '7 n# Email marketing jobs, especially those specifying a tool like Mailchimp, are quite popular and can often be filled within a week, given the high demand and availability of skilled professionals in this area.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948043, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 42, 'prompt_tokens': 213, 'total_tokens': 255, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 190 | 42 | 10.000000 | 38 | 62 | `{'id': 'chatcmpl-B6kP5FiW17TRSmq7Cm8G4IvrG5u5h', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Email Campaign Strategy", "Mailchimp Proficiency", "Copywriting", "Analytics and Reporting", "A/B Testing"] nThese skills are essential for designing and executing effective email marketing campaigns, with a specific emphasis on using Mailchimp, crafting compelling content, analyzing campaign performance, and optimizing through testing.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948043, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 62, 'prompt_tokens': 190, 'total_tokens': 252, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | The job post requires designing and executing a series of email campaigns, which suggests a need for some experience and expertise, particularly with Mailchimp, indicating a mid-level requirement. | These skills are essential for designing and executing effective email marketing campaigns, with a specific emphasis on using Mailchimp, crafting compelling content, analyzing campaign performance, and optimizing through testing. | # Email marketing jobs, especially those specifying a tool like Mailchimp, are quite popular and can often be filled within a week, given the high demand and availability of skilled professionals in this area. | Mid-level The job post requires designing and executing a series of email campaigns, which suggests a need for some experience and expertise, particularly with Mailchimp, indicating a mid-level requirement. | 7 # Email marketing jobs, especially those specifying a tool like Mailchimp, are quite popular and can often be filled within a week, given the high demand and availability of skilled professionals in this area. | \["Email Campaign Strategy", "Mailchimp Proficiency", "Copywriting", "Analytics and Reporting", "A/B Testing"] These skills are essential for designing and executing effective email marketing campaigns, with a specific emphasis on using Mailchimp, crafting compelling content, analyzing campaign performance, and optimizing through testing. | 1 | 1 | 1 | 9fb0e0aa62fbc4411faf12a75b484b7b | 04417cab538f73d69f9457cfa3db8ea7 | 41f8969fee02efe42e461c5ce6fe009f | nan | nan | nan |
| 4 | \['Content Strategy', 'SEO Knowledge', 'Analytics Skills', 'Copywriting', 'Social Media Management'] | 7.500000 | Mid-level | 9 | Content Marketing Strategy | Digital Marketing | 4 | Seeking a content marketing strategist to develop a comprehensive plan to increase our online visibility. The strategy should include content creation, distribution, and performance tracking. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Digital Marketing | You regularly perform jobs in the following category: Digital Marketing. | Agent\_4 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Seeking a content marketing strategist to develop a comprehensive plan to increase our online visibility. The strategy should include content creation, distribution, and performance tracking. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Seeking a content marketing strategist to develop a comprehensive plan to increase our online visibility. The strategy should include content creation, distribution, and performance tracking. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Digital Marketing Job post: Seeking a content marketing strategist to develop a comprehensive plan to increase our online visibility. The strategy should include content creation, distribution, and performance tracking. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Digital Marketing', 'expertise': 'You regularly perform jobs in the following category: Digital Marketing.'}` | 0.000990 | 0.000888 | 189 | 2.500000 | 2.500000 | 0.000852 | 1126.760563 | `{'id': 'chatcmpl-BiJTXZuX32CnU2K95kiiuzLKYDI8w', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-level nThis job requires a comprehensive strategy involving content creation, distribution, and performance tracking, which suggests a need for someone with a moderate level of experience in digital marketing and strategic planning.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749900995, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 38, 'prompt_tokens': 189, 'total_tokens': 227, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1173.020528 | 215 | 10.000000 | 1010.101010 | `{'id': 'chatcmpl-BiJTX6Yq8YPbgE1ccLE8nmQiOMCfs', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '7.5n# Typically, a job like this might take about a week to find the right candidate due to the need for specialized skills in content marketing and strategy development.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749900995, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 35, 'prompt_tokens': 215, 'total_tokens': 250, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 192 | 35 | 10.000000 | 38 | 51 | `{'id': 'chatcmpl-BiJTX2LvXHdDNWHs55E0v95R7ZOf1', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Content Strategy", "SEO Knowledge", "Analytics Skills", "Copywriting", "Social Media Management"] nThese skills are essential for developing and executing a content marketing strategy, which involves planning, creating, distributing, and tracking content to enhance online visibility.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749900995, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 51, 'prompt_tokens': 192, 'total_tokens': 243, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires a comprehensive strategy involving content creation, distribution, and performance tracking, which suggests a need for someone with a moderate level of experience in digital marketing and strategic planning. | These skills are essential for developing and executing a content marketing strategy, which involves planning, creating, distributing, and tracking content to enhance online visibility. | # Typically, a job like this might take about a week to find the right candidate due to the need for specialized skills in content marketing and strategy development. | Mid-level This job requires a comprehensive strategy involving content creation, distribution, and performance tracking, which suggests a need for someone with a moderate level of experience in digital marketing and strategic planning. | 7.5 # Typically, a job like this might take about a week to find the right candidate due to the need for specialized skills in content marketing and strategy development. | \["Content Strategy", "SEO Knowledge", "Analytics Skills", "Copywriting", "Social Media Management"] These skills are essential for developing and executing a content marketing strategy, which involves planning, creating, distributing, and tracking content to enhance online visibility. | 0 | 0 | 0 | 50b3734e9f49805dd089499fcd3d3aa3 | e247d9e87ba81160e92cb604fb7414d6 | 643aa7c062e4fc3ef723bce11d468a67 | nan | nan | nan |
| 5 | \['WordPress Development', 'Responsive Design', 'Contact Form Integration', 'Blog Setup', 'E-commerce Functionality', 'WooCommerce Experience'] | 5.000000 | Mid-level | 15 | WordPress Website Setup | Web Development | 0 | We need a developer to set up a WordPress site for our small business. The site should be responsive and include a contact form, blog, and e-commerce functionality. Experience with WooCommerce is a plus. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Web Development | You regularly perform jobs in the following category: Web Development. | Agent\_5 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: We need a developer to set up a WordPress site for our small business. The site should be responsive and include a contact form, blog, and e-commerce functionality. Experience with WooCommerce is a plus. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: We need a developer to set up a WordPress site for our small business. The site should be responsive and include a contact form, blog, and e-commerce functionality. Experience with WooCommerce is a plus. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: We need a developer to set up a WordPress site for our small business. The site should be responsive and include a contact form, blog, and e-commerce functionality. Experience with WooCommerce is a plus. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | 0.001182 | 0.000910 | 202 | 2.500000 | 2.500000 | 0.000865 | 1098.901099 | `{'id': 'chatcmpl-BiJTjgyw3VBdyJbdt3RAA1plRXwZp', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThe job requires experience with WordPress, responsive design, and WooCommerce, which suggests a need for a developer with a moderate level of experience beyond entry-level skills.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901007, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_a288987b44', 'usage': {'completion_tokens': 36, 'prompt_tokens': 202, 'total_tokens': 238, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1156.069364 | 228 | 10.000000 | 845.665962 | `{'id': 'chatcmpl-BiJTjCP7sOJ9U60SxqtCaZ7lMuICQ', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '5 n# Typically, setting up a WordPress site with the specified features can take around 5 days, considering the time for client communication, development, and revisions.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901007, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 34, 'prompt_tokens': 228, 'total_tokens': 262, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 205 | 34 | 10.000000 | 36 | 67 | `{'id': 'chatcmpl-BiJTjxJZyjgUqKWjHgvBYeyzDY8Pl', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["WordPress Development", "Responsive Design", "Contact Form Integration", "Blog Setup", "E-commerce Functionality", "WooCommerce Experience"] nThese skills are crucial as the job involves setting up a WordPress site with specific features like a contact form, blog, and e-commerce capabilities, where WooCommerce experience is particularly beneficial.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901007, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 67, 'prompt_tokens': 205, 'total_tokens': 272, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | The job requires experience with WordPress, responsive design, and WooCommerce, which suggests a need for a developer with a moderate level of experience beyond entry-level skills. | These skills are crucial as the job involves setting up a WordPress site with specific features like a contact form, blog, and e-commerce capabilities, where WooCommerce experience is particularly beneficial. | # Typically, setting up a WordPress site with the specified features can take around 5 days, considering the time for client communication, development, and revisions. | Mid-level The job requires experience with WordPress, responsive design, and WooCommerce, which suggests a need for a developer with a moderate level of experience beyond entry-level skills. | 5 # Typically, setting up a WordPress site with the specified features can take around 5 days, considering the time for client communication, development, and revisions. | \["WordPress Development", "Responsive Design", "Contact Form Integration", "Blog Setup", "E-commerce Functionality", "WooCommerce Experience"] These skills are crucial as the job involves setting up a WordPress site with specific features like a contact form, blog, and e-commerce capabilities, where WooCommerce experience is particularly beneficial. | 0 | 0 | 0 | 47c32b1c3d0efaaebf913ae3fee5f8d0 | 8a82c2ad0a603e3cd6cc6b50b5064aad | 4e7e296e40a295381ff1904768e67c6b | nan | nan | nan |
| 6 | \['JavaScript', 'Node.js', 'React', 'Express.js', 'MongoDB', 'Authentication', 'Authorization', 'RESTful API', 'UI/UX Design'] | 7.500000 | Mid-level | 16 | Custom Web Application Development | Web Development | 1 | Looking for a full-stack developer to build a custom web application for managing employee schedules. The app should include a login system, user roles, and reporting features. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Web Development | You regularly perform jobs in the following category: Web Development. | Agent\_6 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Looking for a full-stack developer to build a custom web application for managing employee schedules. The app should include a login system, user roles, and reporting features. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Looking for a full-stack developer to build a custom web application for managing employee schedules. The app should include a login system, user roles, and reporting features. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Looking for a full-stack developer to build a custom web application for managing employee schedules. The app should include a login system, user roles, and reporting features. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | 0.001847 | 0.000935 | 192 | 2.500000 | 2.500000 | 0.001020 | 1069.518717 | `{'id': 'chatcmpl-BiJTjXflaTjasghx8eiRGCbGzB54u', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThe job requires building a custom web application with multiple features like a login system, user roles, and reporting features, which suggests a need for a developer with a solid understanding of both front-end and back-end technologies, typically found at a mid-level experience.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901007, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 54, 'prompt_tokens': 192, 'total_tokens': 246, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 980.392157 | 218 | 10.000000 | 541.271989 | `{'id': 'chatcmpl-BiJTjAQPom5WjSf8ImVDozp7tYlOO', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '7.5n# Typically, a job like this might take about a week to find a suitable candidate, considering the need for specific skills in full-stack development and the complexity of the requirements.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901007, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 39, 'prompt_tokens': 218, 'total_tokens': 257, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 195 | 39 | 10.000000 | 54 | 136 | `{'id': 'chatcmpl-BiJTjZ074QiFjoJxQzLdbL7r8v0l6', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["JavaScript", "Node.js", "React", "Express.js", "MongoDB", "Authentication", "Authorization", "RESTful API", "UI/UX Design"] nThe job requires full-stack development skills, which typically involve JavaScript for both client and server-side scripting (Node.js for the backend and React for the frontend). Express.js is commonly used with Node.js for building web applications. MongoDB is a popular database choice for storing application data. Authentication and authorization are crucial for the login system and user roles. RESTful API design is necessary for communication between the frontend and backend. UI/UX design is important for creating an intuitive and user-friendly interface.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901007, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_90122d973c', 'usage': {'completion_tokens': 136, 'prompt_tokens': 195, 'total_tokens': 331, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | The job requires building a custom web application with multiple features like a login system, user roles, and reporting features, which suggests a need for a developer with a solid understanding of both front-end and back-end technologies, typically found at a mid-level experience. | The job requires full-stack development skills, which typically involve JavaScript for both client and server-side scripting (Node.js for the backend and React for the frontend). Express.js is commonly used with Node.js for building web applications. MongoDB is a popular database choice for storing application data. Authentication and authorization are crucial for the login system and user roles. RESTful API design is necessary for communication between the frontend and backend. UI/UX design is important for creating an intuitive and user-friendly interface. | # Typically, a job like this might take about a week to find a suitable candidate, considering the need for specific skills in full-stack development and the complexity of the requirements. | Mid-level The job requires building a custom web application with multiple features like a login system, user roles, and reporting features, which suggests a need for a developer with a solid understanding of both front-end and back-end technologies, typically found at a mid-level experience. | 7.5 # Typically, a job like this might take about a week to find a suitable candidate, considering the need for specific skills in full-stack development and the complexity of the requirements. | \["JavaScript", "Node.js", "React", "Express.js", "MongoDB", "Authentication", "Authorization", "RESTful API", "UI/UX Design"] The job requires full-stack development skills, which typically involve JavaScript for both client and server-side scripting (Node.js for the backend and React for the frontend). Express.js is commonly used with Node.js for building web applications. MongoDB is a popular database choice for storing application data. Authentication and authorization are crucial for the login system and user roles. RESTful API design is necessary for communication between the frontend and backend. UI/UX design is important for creating an intuitive and user-friendly interface. | 0 | 0 | 0 | 73e2ac11218c1f4fe73a7b19a52ab45e | 52f27f3be748d6a764e61d349afca622 | 9fa230af3221cd32334f90d2aa5135fe | nan | nan | nan |
| 7 | \['Shopify theme customization', 'HTML/CSS', 'JavaScript', 'Liquid templating', 'API integration', 'UX/UI design'] | 5.000000 | Mid-level | 17 | Shopify Store Customization | Web Development | 2 | Seeking a Shopify expert to customize our online store. We need theme adjustments, product page enhancements, and integration with third-party tools. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Web Development | You regularly perform jobs in the following category: Web Development. | Agent\_7 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Seeking a Shopify expert to customize our online store. We need theme adjustments, product page enhancements, and integration with third-party tools. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Seeking a Shopify expert to customize our online store. We need theme adjustments, product page enhancements, and integration with third-party tools. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Seeking a Shopify expert to customize our online store. We need theme adjustments, product page enhancements, and integration with third-party tools. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | 0.001012 | 0.000840 | 186 | 2.500000 | 2.500000 | 0.000805 | 1190.476190 | `{'id': 'chatcmpl-BiJTjD7xP9MCFe5NQAbAn1UrD9psu', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis job requires specific expertise in Shopify, including theme customization, product page enhancements, and third-party tool integration, which typically aligns with a mid-level position.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901007, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_9bddfca6e2', 'usage': {'completion_tokens': 34, 'prompt_tokens': 186, 'total_tokens': 220, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1242.236025 | 212 | 10.000000 | 987.654321 | `{'id': 'chatcmpl-BiJTjerWBc7VeQJoMPgwxugpCntOp', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '5n# This estimation assumes the job is moderately complex, requiring a few days to find the right freelancer, discuss the requirements, and start the project.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901007, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 31, 'prompt_tokens': 212, 'total_tokens': 243, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 189 | 31 | 10.000000 | 34 | 54 | `{'id': 'chatcmpl-BiJTjErNAG1JWkwOFSI4NzAM84Mdv', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Shopify theme customization", "HTML/CSS", "JavaScript", "Liquid templating", "API integration", "UX/UI design"] nThese skills are essential for customizing Shopify stores, handling theme adjustments, enhancing product pages, and integrating third-party tools effectively.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901007, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 54, 'prompt_tokens': 189, 'total_tokens': 243, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires specific expertise in Shopify, including theme customization, product page enhancements, and third-party tool integration, which typically aligns with a mid-level position. | These skills are essential for customizing Shopify stores, handling theme adjustments, enhancing product pages, and integrating third-party tools effectively. | # This estimation assumes the job is moderately complex, requiring a few days to find the right freelancer, discuss the requirements, and start the project. | Mid-level This job requires specific expertise in Shopify, including theme customization, product page enhancements, and third-party tool integration, which typically aligns with a mid-level position. | 5 # This estimation assumes the job is moderately complex, requiring a few days to find the right freelancer, discuss the requirements, and start the project. | \["Shopify theme customization", "HTML/CSS", "JavaScript", "Liquid templating", "API integration", "UX/UI design"] These skills are essential for customizing Shopify stores, handling theme adjustments, enhancing product pages, and integrating third-party tools effectively. | 0 | 0 | 0 | 07090582c041bf6641ba7bbcd4717838 | c9082a1eafdf51e4d4c1b3db2e6f3704 | 2b0a258ce11dc314cf6428ebb86d3265 | nan | nan | nan |
| 8 | \['API Integration', 'CRM Systems', 'Real-time Data Synchronization', 'Programming Languages', 'Problem-Solving Skills'] | 5.000000 | Mid-level | 18 | API Integration | Web Development | 3 | Need a developer to integrate our existing CRM system with an external API. The integration should sync customer data in real-time. Previous experience with similar projects required. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Web Development | You regularly perform jobs in the following category: Web Development. | Agent\_8 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Need a developer to integrate our existing CRM system with an external API. The integration should sync customer data in real-time. Previous experience with similar projects required. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Need a developer to integrate our existing CRM system with an external API. The integration should sync customer data in real-time. Previous experience with similar projects required. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Need a developer to integrate our existing CRM system with an external API. The integration should sync customer data in real-time. Previous experience with similar projects required. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | 0.001275 | 0.001162 | 191 | 2.500000 | 2.500000 | 0.000857 | 860.215054 | `{'id': 'chatcmpl-B6kPEkRp8mjwk65OzLU4WomLzktGV', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis job requires previous experience with similar projects, which suggests that the client is looking for someone with a moderate level of expertise and familiarity with integrating systems and working with APIs.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948052, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 38, 'prompt_tokens': 191, 'total_tokens': 229, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1166.180758 | 217 | 10.000000 | 784.313725 | `{'id': 'chatcmpl-B6kPE3MDCn88uBr6377MAJrIw3VhJ', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '5 nThe job post requires a developer with experience in API integration, which is a common task in web development. Given the specificity of the requirement and the prevalence of such skills among freelancers, it is likely to attract several qualified candidates quickly. Therefore, the job could be fulfilled in approximately 5 days.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948052, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 62, 'prompt_tokens': 217, 'total_tokens': 279, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 194 | 62 | 10.000000 | 38 | 79 | `{'id': 'chatcmpl-B6kPEmML0cOVlYB4d09iifEBjCXOF', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["API Integration", "CRM Systems", "Real-time Data Synchronization", "Programming Languages", "Problem-Solving Skills"] n# The job requires integrating a CRM with an external API, which involves API integration knowledge, understanding CRM systems, and ensuring real-time data sync. Programming skills are necessary for the technical implementation, and problem-solving skills are important for troubleshooting any issues that arise.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948052, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 79, 'prompt_tokens': 194, 'total_tokens': 273, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires previous experience with similar projects, which suggests that the client is looking for someone with a moderate level of expertise and familiarity with integrating systems and working with APIs. | # The job requires integrating a CRM with an external API, which involves API integration knowledge, understanding CRM systems, and ensuring real-time data sync. Programming skills are necessary for the technical implementation, and problem-solving skills are important for troubleshooting any issues that arise. | The job post requires a developer with experience in API integration, which is a common task in web development. Given the specificity of the requirement and the prevalence of such skills among freelancers, it is likely to attract several qualified candidates quickly. Therefore, the job could be fulfilled in approximately 5 days. | Mid-level This job requires previous experience with similar projects, which suggests that the client is looking for someone with a moderate level of expertise and familiarity with integrating systems and working with APIs. | 5 The job post requires a developer with experience in API integration, which is a common task in web development. Given the specificity of the requirement and the prevalence of such skills among freelancers, it is likely to attract several qualified candidates quickly. Therefore, the job could be fulfilled in approximately 5 days. | \["API Integration", "CRM Systems", "Real-time Data Synchronization", "Programming Languages", "Problem-Solving Skills"] # The job requires integrating a CRM with an external API, which involves API integration knowledge, understanding CRM systems, and ensuring real-time data sync. Programming skills are necessary for the technical implementation, and problem-solving skills are important for troubleshooting any issues that arise. | 1 | 1 | 1 | 261b3d69143a484f784f5f8eb90dcac8 | eed717c0a30542a51f25b583a2086fbb | 68269d2faa593a4747d23260fd098b80 | nan | nan | nan |
| 9 | \['HTML/CSS', 'JavaScript', 'Responsive Design', 'Conversion Rate Optimization', 'UI/UX Design'] | 3.500000 | Mid-level | 19 | Landing Page Development | Web Development | 4 | Looking for a developer to create a high-converting landing page for our marketing campaign. The page should be optimized for mobile and desktop users. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Web Development | You regularly perform jobs in the following category: Web Development. | Agent\_9 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Looking for a developer to create a high-converting landing page for our marketing campaign. The page should be optimized for mobile and desktop users. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Looking for a developer to create a high-converting landing page for our marketing campaign. The page should be optimized for mobile and desktop users. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Web Development Job post: Looking for a developer to create a high-converting landing page for our marketing campaign. The page should be optimized for mobile and desktop users. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Web Development', 'expertise': 'You regularly perform jobs in the following category: Web Development.'}` | 0.001448 | 0.000905 | 188 | 2.500000 | 2.500000 | 0.001020 | 1104.972376 | `{'id': 'chatcmpl-B6kPEbAEIqiyJSbS4CAlasu0v63aa', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThe job requires creating a high-converting landing page optimized for both mobile and desktop users, which typically demands a good understanding of responsive design, user experience, and possibly some knowledge of conversion rate optimization. These skills are generally expected from a mid-level developer.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948052, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 55, 'prompt_tokens': 188, 'total_tokens': 243, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 980.392157 | 214 | 10.000000 | 690.846287 | `{'id': 'chatcmpl-B6kPEZREgYTL4KuIZA6y2t3w7K6sK', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '3.5n# Typically, straightforward web development tasks like creating a landing page can be filled within a few days, especially if the client is responsive and the requirements are clear.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948052, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 37, 'prompt_tokens': 214, 'total_tokens': 251, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 191 | 37 | 10.000000 | 55 | 97 | `{'id': 'chatcmpl-B6kPEbmrP6BI5gOUcU6FPz6bVRvPK', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["HTML/CSS", "JavaScript", "Responsive Design", "Conversion Rate Optimization", "UI/UX Design"] nThese skills are essential for creating a landing page that is both visually appealing and effective in converting visitors. HTML/CSS and JavaScript are necessary for building and styling the page, while responsive design ensures it works well on all devices. Conversion rate optimization focuses on increasing the page's effectiveness, and UI/UX design is crucial for creating a user-friendly experience.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948052, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 97, 'prompt_tokens': 191, 'total_tokens': 288, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | The job requires creating a high-converting landing page optimized for both mobile and desktop users, which typically demands a good understanding of responsive design, user experience, and possibly some knowledge of conversion rate optimization. These skills are generally expected from a mid-level developer. | These skills are essential for creating a landing page that is both visually appealing and effective in converting visitors. HTML/CSS and JavaScript are necessary for building and styling the page, while responsive design ensures it works well on all devices. Conversion rate optimization focuses on increasing the page's effectiveness, and UI/UX design is crucial for creating a user-friendly experience. | # Typically, straightforward web development tasks like creating a landing page can be filled within a few days, especially if the client is responsive and the requirements are clear. | Mid-level The job requires creating a high-converting landing page optimized for both mobile and desktop users, which typically demands a good understanding of responsive design, user experience, and possibly some knowledge of conversion rate optimization. These skills are generally expected from a mid-level developer. | 3.5 # Typically, straightforward web development tasks like creating a landing page can be filled within a few days, especially if the client is responsive and the requirements are clear. | \["HTML/CSS", "JavaScript", "Responsive Design", "Conversion Rate Optimization", "UI/UX Design"] These skills are essential for creating a landing page that is both visually appealing and effective in converting visitors. HTML/CSS and JavaScript are necessary for building and styling the page, while responsive design ensures it works well on all devices. Conversion rate optimization focuses on increasing the page's effectiveness, and UI/UX design is crucial for creating a user-friendly experience. | 1 | 1 | 1 | 4d68bc9308d2476fd5085647a299cc33 | f731c29c0ce523729d6d5c5092e7dfea | fc201f2dc266afed90b7cf76811adf9f | nan | nan | nan |
| 10 | \['Creativity', 'Logo Design', 'Branding', 'Adobe Illustrator', 'Portfolio Presentation'] | 7.000000 | Mid-level | 10 | Logo Design for New Startup | Graphic Design | 0 | We are a new tech startup looking for a creative designer to create a unique logo for our brand. The logo should be modern and represent innovation. Please provide portfolio examples. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Graphic Design | You regularly perform jobs in the following category: Graphic Design. | Agent\_10 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: We are a new tech startup looking for a creative designer to create a unique logo for our brand. The logo should be modern and represent innovation. Please provide portfolio examples. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: We are a new tech startup looking for a creative designer to create a unique logo for our brand. The logo should be modern and represent innovation. Please provide portfolio examples. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: We are a new tech startup looking for a creative designer to create a unique logo for our brand. The logo should be modern and represent innovation. Please provide portfolio examples. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | 0.001182 | 0.000900 | 194 | 2.500000 | 2.500000 | 0.001105 | 1111.111111 | `{'id': 'chatcmpl-B6kOqwHlcIIQsvNw0yVkFxoFGQmgO', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis job requires a designer who can create a unique logo that is both modern and represents innovation, which typically requires a fair amount of creativity and experience. Additionally, the request for portfolio examples suggests that they are looking for someone with a proven track record, which aligns with a mid-level position.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 62, 'prompt_tokens': 194, 'total_tokens': 256, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 904.977376 | 220 | 10.000000 | 845.665962 | `{'id': 'chatcmpl-B6kOqhlZLhSat6JUmHRjWGlRc6k0Q', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '7n# Typically, logo design jobs in the graphic design category take about a week to fulfill, considering the time needed for proposals, client reviews, and potential revisions.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 35, 'prompt_tokens': 220, 'total_tokens': 255, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 197 | 35 | 10.000000 | 62 | 69 | `{'id': 'chatcmpl-B6kOq5aIvIzGoSRy6590FtlgXVtJe', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Creativity", "Logo Design", "Branding", "Adobe Illustrator", "Portfolio Presentation"] nThese skills are essential because the job requires creating a unique and modern logo, which involves creativity and expertise in logo design and branding. Adobe Illustrator is a common tool used for such tasks, and having a portfolio demonstrates past work and capability.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 69, 'prompt_tokens': 197, 'total_tokens': 266, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires a designer who can create a unique logo that is both modern and represents innovation, which typically requires a fair amount of creativity and experience. Additionally, the request for portfolio examples suggests that they are looking for someone with a proven track record, which aligns with a mid-level position. | These skills are essential because the job requires creating a unique and modern logo, which involves creativity and expertise in logo design and branding. Adobe Illustrator is a common tool used for such tasks, and having a portfolio demonstrates past work and capability. | # Typically, logo design jobs in the graphic design category take about a week to fulfill, considering the time needed for proposals, client reviews, and potential revisions. | Mid-level This job requires a designer who can create a unique logo that is both modern and represents innovation, which typically requires a fair amount of creativity and experience. Additionally, the request for portfolio examples suggests that they are looking for someone with a proven track record, which aligns with a mid-level position. | 7 # Typically, logo design jobs in the graphic design category take about a week to fulfill, considering the time needed for proposals, client reviews, and potential revisions. | \["Creativity", "Logo Design", "Branding", "Adobe Illustrator", "Portfolio Presentation"] These skills are essential because the job requires creating a unique and modern logo, which involves creativity and expertise in logo design and branding. Adobe Illustrator is a common tool used for such tasks, and having a portfolio demonstrates past work and capability. | 1 | 1 | 1 | 69d564487d41e40882e94d5e527f07ac | 9c3c80600cd235cdfffcd78d60260874 | f306c5f802a6a8e32fe01b390a84cd1a | nan | nan | nan |
| 11 | \['Adobe InDesign', 'Adobe Photoshop', 'Typography', 'Layout Design', 'Attention to Detail'] | 5.000000 | Mid-level | 11 | Brochure Design | Graphic Design | 1 | Looking for an experienced designer to create a professional brochure for our real estate company. The brochure should highlight our services and properties. Must be delivered in print-ready format. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Graphic Design | You regularly perform jobs in the following category: Graphic Design. | Agent\_11 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: Looking for an experienced designer to create a professional brochure for our real estate company. The brochure should highlight our services and properties. Must be delivered in print-ready format. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: Looking for an experienced designer to create a professional brochure for our real estate company. The brochure should highlight our services and properties. Must be delivered in print-ready format. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: Looking for an experienced designer to create a professional brochure for our real estate company. The brochure should highlight our services and properties. Must be delivered in print-ready format. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | 0.001180 | 0.001038 | 193 | 2.500000 | 2.500000 | 0.001073 | 963.855422 | `{'id': 'chatcmpl-B6kOqiDsRLhX57XwkcH086v7MZ21M', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis job requires creating a professional brochure that highlights services and properties, and it needs to be delivered in a print-ready format. This suggests the need for someone with a good understanding of design principles and experience in preparing files for print, which typically aligns with a mid-level designer.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 59, 'prompt_tokens': 193, 'total_tokens': 252, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 932.400932 | 219 | 10.000000 | 847.457627 | `{'id': 'chatcmpl-B6kOqYRLmFldZgGgCfiRypR39x6B7', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '5 n# Typically, straightforward design jobs like creating a brochure can be fulfilled within a week. Given the specificity of the request and assuming the client is responsive, five days is a reasonable estimate for finding the right freelancer and completing the job.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 49, 'prompt_tokens': 219, 'total_tokens': 268, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 196 | 49 | 10.000000 | 59 | 69 | `{'id': 'chatcmpl-B6kOqPCrgEkzleFPvVToqE9QOOAiM', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Adobe InDesign", "Adobe Photoshop", "Typography", "Layout Design", "Attention to Detail"] nThe job requires creating a professional brochure which involves layout design and typography skills. Adobe InDesign and Photoshop are essential tools for creating print-ready materials. Attention to detail is crucial to ensure the brochure accurately highlights the company's services and properties.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 69, 'prompt_tokens': 196, 'total_tokens': 265, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires creating a professional brochure that highlights services and properties, and it needs to be delivered in a print-ready format. This suggests the need for someone with a good understanding of design principles and experience in preparing files for print, which typically aligns with a mid-level designer. | The job requires creating a professional brochure which involves layout design and typography skills. Adobe InDesign and Photoshop are essential tools for creating print-ready materials. Attention to detail is crucial to ensure the brochure accurately highlights the company's services and properties. | # Typically, straightforward design jobs like creating a brochure can be fulfilled within a week. Given the specificity of the request and assuming the client is responsive, five days is a reasonable estimate for finding the right freelancer and completing the job. | Mid-level This job requires creating a professional brochure that highlights services and properties, and it needs to be delivered in a print-ready format. This suggests the need for someone with a good understanding of design principles and experience in preparing files for print, which typically aligns with a mid-level designer. | 5 # Typically, straightforward design jobs like creating a brochure can be fulfilled within a week. Given the specificity of the request and assuming the client is responsive, five days is a reasonable estimate for finding the right freelancer and completing the job. | \["Adobe InDesign", "Adobe Photoshop", "Typography", "Layout Design", "Attention to Detail"] The job requires creating a professional brochure which involves layout design and typography skills. Adobe InDesign and Photoshop are essential tools for creating print-ready materials. Attention to detail is crucial to ensure the brochure accurately highlights the company's services and properties. | 1 | 1 | 1 | 0857658ef0acafe5c6854bece32cb1ea | 863bb710992698d37058c627f3954fea | 26fe6ac7e218ba0920aca33033f226ab | nan | nan | nan |
| 12 | \['Graphic Design', 'Social Media Marketing', 'Adobe Photoshop', 'Creativity', 'Attention to Detail'] | 3.500000 | Mid-level | 12 | Social Media Graphics | Graphic Design | 2 | Need a designer to create eye-catching social media graphics for our upcoming campaign. We need a set of 10 images optimized for Instagram and Facebook. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Graphic Design | You regularly perform jobs in the following category: Graphic Design. | Agent\_12 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: Need a designer to create eye-catching social media graphics for our upcoming campaign. We need a set of 10 images optimized for Instagram and Facebook. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: Need a designer to create eye-catching social media graphics for our upcoming campaign. We need a set of 10 images optimized for Instagram and Facebook. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: Need a designer to create eye-catching social media graphics for our upcoming campaign. We need a set of 10 images optimized for Instagram and Facebook. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | 0.001300 | 0.000907 | 189 | 2.500000 | 2.500000 | 0.000852 | 1101.928375 | `{'id': 'chatcmpl-B6kOq3zz3KAkgwm1FexQmGhonI6xZ', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-level nThis job requires creating optimized and eye-catching designs for social media, which suggests the need for a designer with a solid understanding of design principles and experience with social media platforms.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 38, 'prompt_tokens': 189, 'total_tokens': 227, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1173.020528 | 215 | 10.000000 | 769.230769 | `{'id': 'chatcmpl-B6kOqo0P0KT0LZVjSVc8tEfbSOqUw', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '3.5n# Based on my experience, jobs like this usually take a few days to be fulfilled, considering the need to review portfolios, negotiate terms, and finalize contracts.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 37, 'prompt_tokens': 215, 'total_tokens': 252, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 192 | 37 | 10.000000 | 38 | 82 | `{'id': 'chatcmpl-B6kOqX64L9A1WHbhSCJYVAy1kYMnV', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Graphic Design", "Social Media Marketing", "Adobe Photoshop", "Creativity", "Attention to Detail"] n# These skills are essential for designing visually appealing social media graphics. Graphic Design and Adobe Photoshop are crucial for creating professional images. Social Media Marketing knowledge ensures the designs are optimized for platforms like Instagram and Facebook. Creativity is needed for eye-catching designs, and Attention to Detail ensures high-quality output.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 82, 'prompt_tokens': 192, 'total_tokens': 274, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires creating optimized and eye-catching designs for social media, which suggests the need for a designer with a solid understanding of design principles and experience with social media platforms. | # These skills are essential for designing visually appealing social media graphics. Graphic Design and Adobe Photoshop are crucial for creating professional images. Social Media Marketing knowledge ensures the designs are optimized for platforms like Instagram and Facebook. Creativity is needed for eye-catching designs, and Attention to Detail ensures high-quality output. | # Based on my experience, jobs like this usually take a few days to be fulfilled, considering the need to review portfolios, negotiate terms, and finalize contracts. | Mid-level This job requires creating optimized and eye-catching designs for social media, which suggests the need for a designer with a solid understanding of design principles and experience with social media platforms. | 3.5 # Based on my experience, jobs like this usually take a few days to be fulfilled, considering the need to review portfolios, negotiate terms, and finalize contracts. | \["Graphic Design", "Social Media Marketing", "Adobe Photoshop", "Creativity", "Attention to Detail"] # These skills are essential for designing visually appealing social media graphics. Graphic Design and Adobe Photoshop are crucial for creating professional images. Social Media Marketing knowledge ensures the designs are optimized for platforms like Instagram and Facebook. Creativity is needed for eye-catching designs, and Attention to Detail ensures high-quality output. | 1 | 1 | 1 | 89b1c18fd805b3b4adba6b3c74ecaae4 | 6dd3f51dd73a30f7bb209f55326c8f6e | 462f7897de89d51bf2a9f5f6cc45b936 | nan | nan | nan |
| 13 | \['Adobe Photoshop', 'Adobe Illustrator', 'Branding', 'Creativity', 'Attention to Detail'] | 3.000000 | Mid-level | 13 | Website Banner Design | Graphic Design | 3 | Seeking a skilled designer to create a series of banners for our e-commerce website. Banners should be consistent with our brand’s aesthetic. Please include examples of previous work. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Graphic Design | You regularly perform jobs in the following category: Graphic Design. | Agent\_13 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: Seeking a skilled designer to create a series of banners for our e-commerce website. Banners should be consistent with our brand’s aesthetic. Please include examples of previous work. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: Seeking a skilled designer to create a series of banners for our e-commerce website. Banners should be consistent with our brand’s aesthetic. Please include examples of previous work. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: Seeking a skilled designer to create a series of banners for our e-commerce website. Banners should be consistent with our brand’s aesthetic. Please include examples of previous work. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | 0.001223 | 0.000850 | 194 | 2.500000 | 2.500000 | 0.001115 | 1176.470588 | `{'id': 'chatcmpl-B6kOqpxYuZniwxkOs1LvQWleCHfAv', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': "Mid-levelnnThis job requires a designer who has enough experience to create banners that are consistent with a brand's aesthetic, which typically involves a good understanding of design principles and brand identity. The request for examples of previous work suggests they are looking for someone with a proven track record, indicating a mid-level position.", 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 63, 'prompt_tokens': 194, 'total_tokens': 257, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 896.860987 | 220 | 10.000000 | 817.995910 | `{'id': 'chatcmpl-B6kOqX85Zlt73T0IFoEkhwJlv2a0V', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '3 n# Graphic design jobs like banner creation are typically fulfilled quickly, especially if the client provides clear guidelines and the designer has a strong portfolio.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 30, 'prompt_tokens': 220, 'total_tokens': 250, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 197 | 30 | 10.000000 | 63 | 73 | `{'id': 'chatcmpl-B6kOqPvtmLeOsXJbMzqrSFLHC5stx', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Adobe Photoshop", "Adobe Illustrator", "Branding", "Creativity", "Attention to Detail"] nThese skills are essential for creating visually appealing and brand-consistent banners. Adobe Photoshop and Illustrator are industry-standard tools for design work, branding ensures alignment with the company's aesthetic, creativity is needed for original designs, and attention to detail ensures high-quality outcomes.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 73, 'prompt_tokens': 197, 'total_tokens': 270, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires a designer who has enough experience to create banners that are consistent with a brand's aesthetic, which typically involves a good understanding of design principles and brand identity. The request for examples of previous work suggests they are looking for someone with a proven track record, indicating a mid-level position. | These skills are essential for creating visually appealing and brand-consistent banners. Adobe Photoshop and Illustrator are industry-standard tools for design work, branding ensures alignment with the company's aesthetic, creativity is needed for original designs, and attention to detail ensures high-quality outcomes. | # Graphic design jobs like banner creation are typically fulfilled quickly, especially if the client provides clear guidelines and the designer has a strong portfolio. | Mid-level This job requires a designer who has enough experience to create banners that are consistent with a brand's aesthetic, which typically involves a good understanding of design principles and brand identity. The request for examples of previous work suggests they are looking for someone with a proven track record, indicating a mid-level position. | 3 # Graphic design jobs like banner creation are typically fulfilled quickly, especially if the client provides clear guidelines and the designer has a strong portfolio. | \["Adobe Photoshop", "Adobe Illustrator", "Branding", "Creativity", "Attention to Detail"] These skills are essential for creating visually appealing and brand-consistent banners. Adobe Photoshop and Illustrator are industry-standard tools for design work, branding ensures alignment with the company's aesthetic, creativity is needed for original designs, and attention to detail ensures high-quality outcomes. | 1 | 1 | 1 | 259fd4a403ad0bc0d01a914e11bc385a | 6fda394afda78f1b58297647e3b7a415 | 438e30bef6f387f36c7672b1b0c41f4f | nan | nan | nan |
| 14 | \['Data visualization', 'Adobe Illustrator', 'Creativity', 'Attention to detail', 'Social media design'] | 3.000000 | Mid-level | 14 | Infographic Design | Graphic Design | 4 | We need a designer to create a visually appealing infographic based on our provided data. The infographic should be easy to understand and shareable on social media. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Graphic Design | You regularly perform jobs in the following category: Graphic Design. | Agent\_14 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: We need a designer to create a visually appealing infographic based on our provided data. The infographic should be easy to understand and shareable on social media. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: We need a designer to create a visually appealing infographic based on our provided data. The infographic should be easy to understand and shareable on social media. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Graphic Design Job post: We need a designer to create a visually appealing infographic based on our provided data. The infographic should be easy to understand and shareable on social media. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Graphic Design', 'expertise': 'You regularly perform jobs in the following category: Graphic Design.'}` | 0.000932 | 0.000960 | 190 | 2.500000 | 2.500000 | 0.000835 | 1041.666667 | `{'id': 'chatcmpl-B6kOqU2eSIhvC8rh9prmvqvL2Qja5', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis job requires the ability to interpret data and create a visually appealing design that is also optimized for social media sharing, which typically requires some experience beyond entry-level.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 36, 'prompt_tokens': 190, 'total_tokens': 226, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1197.604790 | 216 | 10.000000 | 1072.386059 | `{'id': 'chatcmpl-B6kOqhInNe6NDtZr5llz7gNAf6Gdj', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '3 n# Infographic design jobs are typically fulfilled quickly, especially if the data is provided. The demand for such jobs is high, and designers usually have quick turnaround times for social media shareable content.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 42, 'prompt_tokens': 216, 'total_tokens': 258, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 193 | 42 | 10.000000 | 36 | 45 | `{'id': 'chatcmpl-B6kOq4mQNzQceVzchzkEGXYNaIeg8', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Data visualization", "Adobe Illustrator", "Creativity", "Attention to detail", "Social media design"] nThese skills are essential for creating an engaging infographic that effectively communicates data and is suitable for social media sharing.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948028, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 45, 'prompt_tokens': 193, 'total_tokens': 238, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires the ability to interpret data and create a visually appealing design that is also optimized for social media sharing, which typically requires some experience beyond entry-level. | These skills are essential for creating an engaging infographic that effectively communicates data and is suitable for social media sharing. | # Infographic design jobs are typically fulfilled quickly, especially if the data is provided. The demand for such jobs is high, and designers usually have quick turnaround times for social media shareable content. | Mid-level This job requires the ability to interpret data and create a visually appealing design that is also optimized for social media sharing, which typically requires some experience beyond entry-level. | 3 # Infographic design jobs are typically fulfilled quickly, especially if the data is provided. The demand for such jobs is high, and designers usually have quick turnaround times for social media shareable content. | \["Data visualization", "Adobe Illustrator", "Creativity", "Attention to detail", "Social media design"] These skills are essential for creating an engaging infographic that effectively communicates data and is suitable for social media sharing. | 1 | 1 | 1 | 7e7f6aa9c201235d8604c6c1d3b27400 | e84fbd2e71e39a8087f0fe5446df1caf | b65045622333ecaf849bc6a5e307b80a | nan | nan | nan |
| 15 | \['SEO knowledge', 'Research skills', 'Writing proficiency', 'Digital marketing understanding', 'Attention to detail'] | 2.500000 | Mid-level | 0 | Blog Post Writing | Content Writing | 0 | Looking for a skilled writer to produce 5 blog posts on digital marketing topics. Each post should be 800-1000 words, well-researched, and SEO-optimized. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Content Writing | You regularly perform jobs in the following category: Content Writing. | Agent\_15 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: Looking for a skilled writer to produce 5 blog posts on digital marketing topics. Each post should be 800-1000 words, well-researched, and SEO-optimized. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: Looking for a skilled writer to produce 5 blog posts on digital marketing topics. Each post should be 800-1000 words, well-researched, and SEO-optimized. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: Looking for a skilled writer to produce 5 blog posts on digital marketing topics. Each post should be 800-1000 words, well-researched, and SEO-optimized. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | 0.000997 | 0.000925 | 196 | 2.500000 | 2.500000 | 0.000850 | 1081.081081 | `{'id': 'chatcmpl-BiJTz4HLG6c0kYbjRwTJKvlrD0so3', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis job requires skills in research, SEO optimization, and writing on specialized topics, which suggests a need for more than just basic writing abilities, hence mid-level experience.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901023, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_a288987b44', 'usage': {'completion_tokens': 36, 'prompt_tokens': 196, 'total_tokens': 232, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1176.470588 | 222 | 10.000000 | 1002.506266 | `{'id': 'chatcmpl-BiJTzQwjTZB8ahalEQl4uMDNvrhqq', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '2.5n# Typically, jobs in content writing like this are filled quickly, often within 2-3 days, especially if the requirements are clear and the topic is popular.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901023, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_a288987b44', 'usage': {'completion_tokens': 37, 'prompt_tokens': 222, 'total_tokens': 259, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 199 | 37 | 10.000000 | 36 | 50 | `{'id': 'chatcmpl-BiJTzc5mLADahHcB76hxOQwShuYZL', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["SEO knowledge", "Research skills", "Writing proficiency", "Digital marketing understanding", "Attention to detail"] n// These skills are crucial for writing effective blog posts that are both informative and optimized for search engines, particularly in the digital marketing niche.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901023, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_a288987b44', 'usage': {'completion_tokens': 50, 'prompt_tokens': 199, 'total_tokens': 249, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires skills in research, SEO optimization, and writing on specialized topics, which suggests a need for more than just basic writing abilities, hence mid-level experience. | // These skills are crucial for writing effective blog posts that are both informative and optimized for search engines, particularly in the digital marketing niche. | # Typically, jobs in content writing like this are filled quickly, often within 2-3 days, especially if the requirements are clear and the topic is popular. | Mid-level This job requires skills in research, SEO optimization, and writing on specialized topics, which suggests a need for more than just basic writing abilities, hence mid-level experience. | 2.5 # Typically, jobs in content writing like this are filled quickly, often within 2-3 days, especially if the requirements are clear and the topic is popular. | \["SEO knowledge", "Research skills", "Writing proficiency", "Digital marketing understanding", "Attention to detail"] // These skills are crucial for writing effective blog posts that are both informative and optimized for search engines, particularly in the digital marketing niche. | 0 | 0 | 0 | 03f808432fc84885bdd91cf9f06537d2 | 6ba7d50fe2bc735779afd75bc3c78aff | 0774f04c48024414e25350bda2dffaa4 | nan | nan | nan |
| 16 | \['Creative Writing', 'SEO Knowledge', 'Attention to Detail', 'Research Skills', 'Understanding of Target Audience'] | 3.000000 | Mid-level | 1 | Product Description Writing | Content Writing | 1 | We need a writer to craft compelling product descriptions for our online store. Each description should highlight the key features and benefits of the product. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Content Writing | You regularly perform jobs in the following category: Content Writing. | Agent\_16 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: We need a writer to craft compelling product descriptions for our online store. Each description should highlight the key features and benefits of the product. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: We need a writer to craft compelling product descriptions for our online store. Each description should highlight the key features and benefits of the product. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: We need a writer to craft compelling product descriptions for our online store. Each description should highlight the key features and benefits of the product. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | 0.000975 | 0.000922 | 187 | 2.500000 | 2.500000 | 0.000798 | 1084.010840 | `{'id': 'chatcmpl-B6kOdixpGGWfP54PWw4qxueyuqJqA', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis job requires the ability to craft compelling product descriptions, which involves understanding both the product and the target audience, skills typically developed with some experience.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948015, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 33, 'prompt_tokens': 187, 'total_tokens': 220, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1253.918495 | 213 | 10.000000 | 1025.641026 | `{'id': 'chatcmpl-B6kOdbqo16AeOndkwGbqnfuhc0OU7', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '3 n# Product description writing jobs often attract a good number of applicants, especially if the job is clearly defined. It might take a few days to review applications and select the right candidate.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948015, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 39, 'prompt_tokens': 213, 'total_tokens': 252, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 190 | 39 | 10.000000 | 33 | 50 | `{'id': 'chatcmpl-B6kOdxWiqb1nKJh7FzSDnG1Y6EEN4', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Creative Writing", "SEO Knowledge", "Attention to Detail", "Research Skills", "Understanding of Target Audience"] n# These skills are crucial for crafting engaging product descriptions that not only highlight features and benefits but also attract and retain potential customers.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948015, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 50, 'prompt_tokens': 190, 'total_tokens': 240, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires the ability to craft compelling product descriptions, which involves understanding both the product and the target audience, skills typically developed with some experience. | # These skills are crucial for crafting engaging product descriptions that not only highlight features and benefits but also attract and retain potential customers. | # Product description writing jobs often attract a good number of applicants, especially if the job is clearly defined. It might take a few days to review applications and select the right candidate. | Mid-level This job requires the ability to craft compelling product descriptions, which involves understanding both the product and the target audience, skills typically developed with some experience. | 3 # Product description writing jobs often attract a good number of applicants, especially if the job is clearly defined. It might take a few days to review applications and select the right candidate. | \["Creative Writing", "SEO Knowledge", "Attention to Detail", "Research Skills", "Understanding of Target Audience"] # These skills are crucial for crafting engaging product descriptions that not only highlight features and benefits but also attract and retain potential customers. | 1 | 1 | 1 | de60c149b9f02bf3c81418af49e20aaa | 59cdc5a9cf4f22dd0986bf35411cab88 | 7f8aa9edb29063b40d757e307448f1c3 | nan | nan | nan |
| 17 | \['Technical writing', 'Understanding of software development', 'Ability to create user manuals', 'Experience with API documentation', 'Familiarity with software terminology'] | 7.000000 | Mid-level | 2 | Technical Writing for Software Documentation | Content Writing | 2 | Seeking an experienced technical writer to create user manuals and API documentation for our software product. Must have a background in tech writing and be familiar with software development terminology. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Content Writing | You regularly perform jobs in the following category: Content Writing. | Agent\_17 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: Seeking an experienced technical writer to create user manuals and API documentation for our software product. Must have a background in tech writing and be familiar with software development terminology. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: Seeking an experienced technical writer to create user manuals and API documentation for our software product. Must have a background in tech writing and be familiar with software development terminology. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: Seeking an experienced technical writer to create user manuals and API documentation for our software product. Must have a background in tech writing and be familiar with software development terminology. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | 0.001167 | 0.001115 | 192 | 2.500000 | 2.500000 | 0.000740 | 896.860987 | `{'id': 'chatcmpl-B6kOdHPY30ATJg7snchhiYo7JTeNU', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-level nThe job requires experience in technical writing and familiarity with software development terminology, which typically suggests a mid-level position.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948015, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 26, 'prompt_tokens': 192, 'total_tokens': 218, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1351.351351 | 218 | 10.000000 | 856.531049 | `{'id': 'chatcmpl-B6kOdPJViKpexPbdeAY6FjMpA9SRm', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '7 n# Based on my experience, technical writing positions, especially those requiring familiarity with software development, typically take about a week to fill on online labor marketplaces. The demand for such skilled writers is high, but the pool of qualified candidates is smaller compared to more general writing roles.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948015, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 57, 'prompt_tokens': 218, 'total_tokens': 275, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 195 | 57 | 10.000000 | 26 | 68 | `{'id': 'chatcmpl-B6kOdAmMWeDU21NmnWkkH89pByX6C', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Technical writing", "Understanding of software development", "Ability to create user manuals", "Experience with API documentation", "Familiarity with software terminology"]nn# These skills are essential for the job post as it requires creating technical documents for a software product, which demands a strong grasp of technical writing and familiarity with software development and terminology.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948015, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 68, 'prompt_tokens': 195, 'total_tokens': 263, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | The job requires experience in technical writing and familiarity with software development terminology, which typically suggests a mid-level position. | # These skills are essential for the job post as it requires creating technical documents for a software product, which demands a strong grasp of technical writing and familiarity with software development and terminology. | # Based on my experience, technical writing positions, especially those requiring familiarity with software development, typically take about a week to fill on online labor marketplaces. The demand for such skilled writers is high, but the pool of qualified candidates is smaller compared to more general writing roles. | Mid-level The job requires experience in technical writing and familiarity with software development terminology, which typically suggests a mid-level position. | 7 # Based on my experience, technical writing positions, especially those requiring familiarity with software development, typically take about a week to fill on online labor marketplaces. The demand for such skilled writers is high, but the pool of qualified candidates is smaller compared to more general writing roles. | \["Technical writing", "Understanding of software development", "Ability to create user manuals", "Experience with API documentation", "Familiarity with software terminology"] # These skills are essential for the job post as it requires creating technical documents for a software product, which demands a strong grasp of technical writing and familiarity with software development and terminology. | 1 | 1 | 1 | a32c9cad607dc8274a458cb0d56bd84d | 8c50bdbe5c637a4c3fc33174f270a763 | c4e98e066838de4fed1b5d1af7945b29 | nan | nan | nan |
| 18 | \['Copywriting', 'Brand Voice Development', 'Persuasive Writing', 'Clarity and Conciseness', 'SEO Knowledge'] | 3.500000 | Mid-level | 3 | Website Copywriting | Content Writing | 3 | Looking for a copywriter to create persuasive content for our company’s website. The content should be clear, concise, and align with our brand voice. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Content Writing | You regularly perform jobs in the following category: Content Writing. | Agent\_18 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: Looking for a copywriter to create persuasive content for our company’s website. The content should be clear, concise, and align with our brand voice. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: Looking for a copywriter to create persuasive content for our company’s website. The content should be clear, concise, and align with our brand voice. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: Looking for a copywriter to create persuasive content for our company’s website. The content should be clear, concise, and align with our brand voice. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | 0.000972 | 0.001030 | 190 | 2.500000 | 2.500000 | 0.000805 | 970.873786 | `{'id': 'chatcmpl-BiJTzHgOtulJ73CqkVnlPVEf3me9x', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThe job requires creating persuasive content that aligns with a brand voice, which typically demands a certain level of experience and understanding of branding and copywriting techniques.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901023, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_a288987b44', 'usage': {'completion_tokens': 33, 'prompt_tokens': 190, 'total_tokens': 223, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1242.236025 | 216 | 10.000000 | 1028.277635 | `{'id': 'chatcmpl-BiJTz515DEMOvlZaRhS4hwbCAKGAi', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '3.5n# Copywriting tasks like these typically attract a lot of freelancers quickly, especially if the brand voice is well-defined and the project is straightforward. It usually takes a few days for the client to review proposals and make a selection.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901023, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 49, 'prompt_tokens': 216, 'total_tokens': 265, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 193 | 49 | 10.000000 | 33 | 49 | `{'id': 'chatcmpl-BiJTza1VyYvLvJ60nmUto3vJ4z9eG', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Copywriting", "Brand Voice Development", "Persuasive Writing", "Clarity and Conciseness", "SEO Knowledge"] nThese skills are essential for crafting impactful website content that aligns with a brand's identity and engages the audience effectively.', 'refusal': None, 'role': 'assistant', 'annotations': [], 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1749901023, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_07871e2ad8', 'usage': {'completion_tokens': 49, 'prompt_tokens': 193, 'total_tokens': 242, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | The job requires creating persuasive content that aligns with a brand voice, which typically demands a certain level of experience and understanding of branding and copywriting techniques. | These skills are essential for crafting impactful website content that aligns with a brand's identity and engages the audience effectively. | # Copywriting tasks like these typically attract a lot of freelancers quickly, especially if the brand voice is well-defined and the project is straightforward. It usually takes a few days for the client to review proposals and make a selection. | Mid-level The job requires creating persuasive content that aligns with a brand voice, which typically demands a certain level of experience and understanding of branding and copywriting techniques. | 3.5 # Copywriting tasks like these typically attract a lot of freelancers quickly, especially if the brand voice is well-defined and the project is straightforward. It usually takes a few days for the client to review proposals and make a selection. | \["Copywriting", "Brand Voice Development", "Persuasive Writing", "Clarity and Conciseness", "SEO Knowledge"] These skills are essential for crafting impactful website content that aligns with a brand's identity and engages the audience effectively. | 0 | 0 | 0 | ab3f693c4e27783a6370e5ca29049c5b | a7e69711b7c23f92f3f82104d6611c7b | 252832faff6ec066e92a8c2b1024d356 | nan | nan | nan |
| 19 | \['Excellent writing skills', 'Understanding of press release format', 'Attention to detail', 'Ability to engage the audience', 'Knowledge of the product or industry'] | 2.000000 | Mid-level | 4 | Press Release Writing | Content Writing | 4 | We need a writer to draft a press release for our upcoming product launch. The release should be attention-grabbing and follow industry standards. | You are an experienced freelancer on online labor marketplaces. | You are answering questions as if you were a human. Do not break character. | 0 | Content Writing | You regularly perform jobs in the following category: Content Writing. | Agent\_19 | 0 | gpt-4o | 0 | openai | 0 | 1000 | 1 | 3 | 0.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: We need a writer to draft a press release for our upcoming product launch. The release should be attention-grabbing and follow industry standards. What are some key skills required for this job? Return your answers on one line, in a comma-separated list of your responses, with square brackets and each answer in quotes E.g., \["A", "B", "C"] After the answers, you can put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: We need a writer to draft a press release for our upcoming product launch. The release should be attention-grabbing and follow industry standards. Estimate the number of days until this job post is fulfilled. This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, ...). Respond with just your number on a single line. If your response is equivalent to zero, report '0' After the answer, put a comment explaining your choice on the next line. | Consider the following job category and job post at an online labor marketplace. Job category: Content Writing Job post: We need a writer to draft a press release for our upcoming product launch. The release should be attention-grabbing and follow industry standards. What level of experience is required for this job? Entry-level Mid-level Senior-level Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | You are answering questions as if you were a human. Do not break character.Your traits: `{'persona': 'You are an experienced freelancer on online labor marketplaces.', 'job_category': 'Content Writing', 'expertise': 'You regularly perform jobs in the following category: Content Writing.'}` | 0.001055 | 0.000932 | 187 | 2.500000 | 2.500000 | 0.000917 | 1072.386059 | `{'id': 'chatcmpl-B6kOdlRrWgkWhxecEiAo34qmy1P8e', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': 'Mid-levelnnThis job requires knowledge of industry standards and the ability to create an attention-grabbing press release, which suggests that some experience and familiarity with press release writing is necessary, making it suitable for mid-level writers.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948015, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 45, 'prompt_tokens': 187, 'total_tokens': 232, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 1089.918256 | 213 | 10.000000 | 947.867299 | `{'id': 'chatcmpl-B6kOd5lJHjdEEvb1Gr6DX0k7b3PyU', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '2n# Press release writing is a common task and typically requires a quick turnaround. Given industry standards and the straightforward nature of the task, it is likely to be fulfilled within 2 days.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948015, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 40, 'prompt_tokens': 213, 'total_tokens': 253, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 10.000000 | 190 | 40 | 10.000000 | 45 | 58 | `{'id': 'chatcmpl-B6kOdQuvioNQ5odykpuh4SUmkJXFt', 'choices': [{'finish_reason': 'stop', 'index': 0, 'logprobs': None, 'message': {'content': '["Excellent writing skills", "Understanding of press release format", "Attention to detail", "Ability to engage the audience", "Knowledge of the product or industry"] n# These skills are essential to create a compelling and professional press release that meets industry standards and effectively communicates the product launch.', 'refusal': None, 'role': 'assistant', 'audio': None, 'function_call': None, 'tool_calls': None}}], 'created': 1740948015, 'model': 'gpt-4o-2024-08-06', 'object': 'chat.completion', 'service_tier': 'default', 'system_fingerprint': 'fp_eb9dce56a8', 'usage': {'completion_tokens': 58, 'prompt_tokens': 190, 'total_tokens': 248, 'completion_tokens_details': {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}` | 2.500000 | 0 | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What level of experience is required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` What are some key skills required for this job? | Consider the following job category and job post at an online labor marketplace. Job category: `{{ scenario.job_category }}` Job post: `{{ scenario.job_post }}` Estimate the number of days until this job post is fulfilled. | \['Entry-level', 'Mid-level', 'Senior-level'] | nan | nan | multiple\_choice | list | numerical | This job requires knowledge of industry standards and the ability to create an attention-grabbing press release, which suggests that some experience and familiarity with press release writing is necessary, making it suitable for mid-level writers. | # These skills are essential to create a compelling and professional press release that meets industry standards and effectively communicates the product launch. | # Press release writing is a common task and typically requires a quick turnaround. Given industry standards and the straightforward nature of the task, it is likely to be fulfilled within 2 days. | Mid-level This job requires knowledge of industry standards and the ability to create an attention-grabbing press release, which suggests that some experience and familiarity with press release writing is necessary, making it suitable for mid-level writers. | 2 # Press release writing is a common task and typically requires a quick turnaround. Given industry standards and the straightforward nature of the task, it is likely to be fulfilled within 2 days. | \["Excellent writing skills", "Understanding of press release format", "Attention to detail", "Ability to engage the audience", "Knowledge of the product or industry"] # These skills are essential to create a compelling and professional press release that meets industry standards and effectively communicates the product launch. | 1 | 1 | 1 | ea1f728739691587828ec51bc5bc9e57 | e876a0eb1edfb6797d2fc08542389d34 | a88ca6debe16ee90432022a33e537a32 | nan | nan | nan |

## Posting content at Expected Parrot

We can post any EDSL objects to Expected Parrot, including this notebook:

```python theme={null}
# agents.push(
#     description = "Agents for job posts data labeling task",
#     alias = "job-posts-agents-example",
#     visibility = "public"
# )
```

```python theme={null}

# survey.push(
#     description = "Survey for job posts data labeling task",
#     alias = "job-posts-survey-example",
#     visibility = "public"
# )

```

We can also post this Notebook:

```python theme={null}

from edsl import Notebook

nb = Notebook(path = "data_labeling_agent.ipynb")

nb.push(
    description = "Example code for data labeling using agents",
    alias = "data-labeling-agent-notebook",
    visibility = "public"
)
```

Learn more about [the platform](/en/latest/coop) to conduct LLM-based research.


# Data labeling
Source: https://docs.expectedparrot.com/en/latest/notebooks/data_labeling_example

This notebook provides example code for conducting data labeling and content analysis in EDSL, an open-source library for simulating surveys, experiments and research tasks with AI agents and large language models.

Before running the code below please see instructions on [getting started](https://www.expectedparrot.com/getting=started) using EDSL.

## Overview

Using a dataset of mock customer service tickets as an example, we demonstrate how to:

<Steps>
  <Step>
    Import data into EDSL as [scenarios](https://docs.expectedparrot/en/latest/scenarios)
  </Step>

  <Step>
    Create [questions](https://docs.expectedparrot/en/latest/questions/html) about the data
  </Step>

  <Step>
    Design an AI [agent](https://docs.expectedparrot/en/latest/agents) to answer the questions
  </Step>

  <Step>
    Select a language [model](/en/latest/language_models) to generate responses
  </Step>

  <Step>
    Analyze [results](https://docs.expectedparrot/en/latest/results) as a formatted dataset
  </Step>
</Steps>

This workflow can be visualized as follows:

<Frame>
  <img alt="general_survey.png" />
</Frame>

## Selecting data for review

First we identify some data for review. Data can be created in EDSL or imported from other sources (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). For purposes of demonstration we import a set of hypothetical customer tickets for a transportation app:

```python expandable theme={null}
tickets = [
    "I just realized I left my phone in the car on my last ride. Can you help me get it back?",
    "I'm unhappy with my recent experience. The driver was very rude and unprofessional.",
    "I was charged more than the estimated fare for my trip yesterday. Can you explain why?",
    "The car seat provided was not properly installed, and I felt my child was at risk. Please ensure driver training.",
    "My driver took a longer route than necessary, resulting in a higher fare. I request a fare adjustment.",
    "I had a great experience with my driver today! Very friendly and efficient service.",
    "I'm concerned about the vehicle's cleanliness. It was not up to the standard I expect.",
    "The app keeps crashing every time I try to book a ride. Please fix this issue.",
    "My driver was exceptional - safe driving, polite, and the car was spotless. Kudos!",
    "I felt unsafe during my ride due to the driver's erratic behavior. This needs to be addressed immediately.",
    "The driver refused to follow my preferred route, which is shorter. I'm not satisfied with the service.",
    "Impressed with the quick response to my ride request and the driver's professionalism.",
    "I was charged for a ride I never took. Please refund me as soon as possible.",
    "The promo code I tried to use didn't work. Can you assist with this?",
    "There was a suspicious smell in the car, and I'm worried about hygiene standards.",
    "My driver was very considerate, especially helping me with my luggage. Appreciate the great service!",
    "The app's GPS seems inaccurate. It directed the driver to the wrong pick-up location.",
    "I want to compliment my driver's excellent navigation and time management during rush hour.",
    "The vehicle didn't match the description in the app. It was confusing and concerning.",
    "I faced an issue with payment processing after my last ride. Can you look into this?",
]
```

## Constructing questions about the data

Next we create some questions about the data. EDSL provides a variety of question types that we can choose from based on the form of the response that we want to get back from the model (multiple choice, free text, checkbox, linear scale, etc.). [Learn more about question types](/en/latest/questions).

<Note>
  **Note**:

  Note that we use a `{{ placeholder }}` in each question text in order to parameterize the questions with the individual ticket contents in the next step:
</Note>

```python theme={null}
from edsl import (
    QuestionMultipleChoice,
    QuestionCheckBox,
    QuestionFreeText,
    QuestionYesNo,
    QuestionLinearScale,
)
```

```python theme={null}
q_issues = QuestionCheckBox(
    question_name="issues",
    question_text="Check all of the issues mentioned in this ticket: `{{ scenario.ticket }}`",
    question_options=[
        "safety",
        "cleanliness",
        "driver performance",
        "GPS/route",
        "lost item",
        "other",
    ],
)
```

```python theme={null}
q_primary_issue = QuestionFreeText(
    question_name="primary_issue",
    question_text="What is the primary issue in this ticket? Ticket: `{{ scenario.ticket }}`",
)
```

```python theme={null}
q_accident = QuestionMultipleChoice(
    question_name="accident",
    question_text="If the primary issue in this ticket is safety, was there an accident where someone was hurt? Ticket: `{{ scenario.ticket }}`",
    question_options=["Yes", "No", "Not applicable"],
)
```

```python theme={null}
q_sentiment = QuestionMultipleChoice(
    question_name="sentiment",
    question_text="What is the sentiment of this ticket? Ticket: `{{ scenario.ticket }}`",
    question_options=[
        "Very positive",
        "Somewhat positive",
        "Neutral",
        "Somewhat negative",
        "Very negative",
    ],
)
```

```python theme={null}
q_refund = QuestionYesNo(
    question_name="refund",
    question_text="Does the customer ask for a refund in this ticket? Ticket: `{{ scenario.ticket }}`",
)
```

```python theme={null}
q_priority = QuestionLinearScale(
    question_name="priority",
    question_text="On a scale from 0 to 5, what is the priority level of this ticket? Ticket: `{{ scenario.ticket }}`",
    question_options=[0, 1, 2, 3, 4, 5],
    option_labels={0: "Lowest", 5: "Highest"},
)
```

## Building a survey

We combine the questions into a [survey](/en/latest/surveys) in order to administer them together:

from edsl import Survey

survey = Survey(
questions=\[
q\_issues,
q\_primary\_issue,
q\_accident,
q\_sentiment,
q\_refund,
q\_priority,
]
)

Survey questions are administered asynchronously by default. [Learn more about adding conditional logic and memory to your survey](/en/latest/surveys). Here we inspect them:

```python theme={null}
survey
```

```python theme={null}
Survey # questions: 6; question_name list: ['issues', 'primary_issue', 'accident', 'sentiment', 'refund', 'priority'];
```

| | question\_text | question\_options | question\_type | question\_name | option\_labels |
| :- | :- | :- | :- | :- | :- |
| 0 | Check all of the issues mentioned in this ticket: `{{ scenario.ticket }}` | \['safety', 'cleanliness', 'driver performance', 'GPS/route', 'lost item', 'other'] | checkbox | issues | nan |
| 1 | What is the primary issue in this ticket? Ticket: `{{ scenario.ticket }}` | nan | free\_text | primary\_issue | nan |
| 2 | If the primary issue in this ticket is safety, was there an accident where someone was hurt? Ticket: `{{ scenario.ticket }}` | \['Yes', 'No', 'Not applicable'] | multiple\_choice | accident | nan |
| 3 | What is the sentiment of this ticket? Ticket: `{{ scenario.ticket }}` | \['Very positive', 'Somewhat positive', 'Neutral', 'Somewhat negative', 'Very negative'] | multiple\_choice | sentiment | nan |
| 4 | Does the customer ask for a refund in this ticket? Ticket: `{{ scenario.ticket }}` | \['No', 'Yes'] | yes\_no | refund | nan |
| 5 | On a scale from 0 to 5, what is the priority level of this ticket? Ticket: `{{ scenario.ticket }}` | \[0, 1, 2, 3, 4, 5] | linear\_scale | priority | `{0: 'Lowest', 5: 'Highest'}` |

## Designing AI agents

A key feature of EDSL is the ability to create personas for AI agents that the language models are prompted to use in generating responses to the questions. This is done by passing a dictionary of traits to Agent objects:

```python theme={null}
from edsl import Agent

agent = Agent(
    traits={
        "persona": "You are an expert customer service agent.",
        "years_experience": 15,
    }
)
agent
```

```
Agent
```

| | key | value |
| :- | :- | :- |
| 0 | traits:persona | You are an expert customer service agent. |
| 1 | traits:years\_experience | 15 |

## Selecting language models

EDSL allows us to select the language models to use in generating results. See the [`model pricing page <>`\_\_](/en/latest/notebooks/#id1) for pricing and performance information for available models.

Here we select gpt-4o (if no model is specified, the default model is used – run `Model()` to verify the current default model):

```python theme={null}
from edsl import Model

model = Model("gpt-4o", service_name = "openai")
model
```

```
LanguageModel
```

| | key | value |
| :- | :- | :- |
| 0 | model | gpt-4o |
| 1 | parameters:temperature | 0.500000 |
| 2 | parameters:max\_tokens | 1000 |
| 3 | parameters:top\_p | 1 |
| 4 | parameters:frequency\_penalty | 0 |
| 5 | parameters:presence\_penalty | 0 |
| 6 | parameters:logprobs | False |
| 7 | parameters:top\_logprobs | 3 |
| 8 | inference\_service | openai |

## Adding data to the questions

We add the contents of each ticket into each question as an independent “scenario” for review. This allows us to create versions of the questions for each job post and deliver them to the model all at once:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_list("ticket", tickets)
scenarios
```

```python theme={null}
ScenarioList scenarios: 20; keys: ['ticket'];
```

| | ticket |
| :- | :- |
| 0 | I just realized I left my phone in the car on my last ride. Can you help me get it back? |
| 1 | I'm unhappy with my recent experience. The driver was very rude and unprofessional. |
| 2 | I was charged more than the estimated fare for my trip yesterday. Can you explain why? |
| 3 | The car seat provided was not properly installed, and I felt my child was at risk. Please ensure driver training. |
| 4 | My driver took a longer route than necessary, resulting in a higher fare. I request a fare adjustment. |
| 5 | I had a great experience with my driver today! Very friendly and efficient service. |
| 6 | I'm concerned about the vehicle's cleanliness. It was not up to the standard I expect. |
| 7 | The app keeps crashing every time I try to book a ride. Please fix this issue. |
| 8 | My driver was exceptional - safe driving, polite, and the car was spotless. Kudos! |
| 9 | I felt unsafe during my ride due to the driver's erratic behavior. This needs to be addressed immediately. |
| 10 | The driver refused to follow my preferred route, which is shorter. I'm not satisfied with the service. |
| 11 | Impressed with the quick response to my ride request and the driver's professionalism. |
| 12 | I was charged for a ride I never took. Please refund me as soon as possible. |
| 13 | The promo code I tried to use didn't work. Can you assist with this? |
| 14 | There was a suspicious smell in the car, and I'm worried about hygiene standards. |
| 15 | My driver was very considerate, especially helping me with my luggage. Appreciate the great service! |
| 16 | The app's GPS seems inaccurate. It directed the driver to the wrong pick-up location. |
| 17 | I want to compliment my driver's excellent navigation and time management during rush hour. |
| 18 | The vehicle didn't match the description in the app. It was confusing and concerning. |
| 19 | I faced an issue with payment processing after my last ride. Can you look into this? |

## Running the survey

We run the survey by adding the scenarios, agent and model with the `by()` method and then calling the `run()` method:

```python theme={null}
results = survey.by(scenarios).by(agent).by(model).run()
```

This generates a formatted dataset of `Results` that includes information about all the components, including the prompts and responses. We can see a list of all the components:

```python theme={null}
results.columns
```

## Analyzing results

EDSL comes with [built-in methods for analyzing results](/en/latest/results). Here we filter, sort, select and print components in a table:

```python theme={null}
(
    results
    .filter("priority in [4, 5]")
    .sort_by("issues", "sentiment")
    .select("ticket", "issues", "primary_issue", "accident", "sentiment", "refund", "priority")
)
```

We can apply some lables to our table:

```python expandable theme={null}
(
    results.select(
        "ticket",
        "issues",
        "primary_issue",
        "accident",
        "sentiment",
        "refund",
        "priority",
    ).print(
        pretty_labels={
            "scenario.ticket": "Ticket",
            "answer.issues": "Issues",
            "answer.primary_issue": "Primary issue",
            "answer.accident": "Accident",
            "answer.sentiment": "Sentiment",
            "answer.refund": "Refund request",
            "answer.priority": "Priority",
        }
    )
)
```

EDSL also comes with methods for accessing results as a dataframe or SQL table:

```python theme={null}
df = (
    results
    .select(
        "issues",
        "primary_issue",
        "accident",
        "sentiment",
        "refund",
        "priority"
    )
    .to_pandas(remove_prefix=True)
)
df
```

We can also access results as a SQL table:

```python theme={null}
results.sql("""
select ticket, issues, primary_issue, accident, sentiment, refund, priority
from self
order by priority desc
""")
```

To export results to a CSV file:

```python theme={null}
results.to_csv("data_labeling_example.csv")
```

## Posting content to Expected Parrot

We can post any EDSL objects to Expected Parrot, and share them publicly, privately or unlisted (by default).

The above results were automatically posted to Expected Parrot; we can also post them manually:

```python theme={null}
# results.push(
#     description = "Customer service tickets data labeling example",
#     alias = "customer-service-tickets-results-example",
#     visibility="public"
# )
```

```python theme={null}
# survey.push(
#     description = "Customer service tickets data labeling example survey",
#     alias = "customer-service-tickets-survey-example",
#     visibility="public"
# )
```

To post this notebook:

```python theme={null}
# from edsl import Notebook

# nb = Notebook("data_labeling_example.ipynb")

# nb.push(
#     description = "Data labeling example",
#     alias = "data-labeling-example-notebook",
#     visibility = "public"
# )
```

To update an object at Expected Parrot:

```python theme={null}
from edsl import Notebook


nb = Notebook("data_labeling_example.ipynb") # resave

nb.patch("https://www.expectedparrot.com/content/RobinHorton/data-labeling-example-notebook", value = nb)
```


# Data labeling with LLMs, validating with humans
Source: https://docs.expectedparrot.com/en/latest/notebooks/data_labeling_validation_example

This notebook provides example [EDSL](https://github.com/expectedparrot/edsl) code for conducting a data labeling task with large language models and validating responses with humans.

The example below consists of the following steps, which can be conducted entirely in EDSL code or interactively at your [Expected Parrot account](https://www.expectedparrot.com/):

* Construct questions about a dataset, using a placeholder in each question for the individual piece of data to be labeled (each answer is a “label” for a piece of data)
* Combine the questions in a survey to administer them together
* *Optionally* create AI agent personas to answer the questions (e.g., if there is relevant expertise or background for the task)
* Select language models to generate the answers (for the agents, or without referencing any AI personas)
* Run the survey with the data, agents and models to generate a formatted dataset of results
* Select questions and data that you want to validate with humans to create a subset of your survey (or leave it unchanged to run the entire survey with humans)
* Send a web-based version of the survey to human respondents
* Compare LLM and human answers, and iterate on the data labeling survey as needed!

Before running the code below please see instructions on [getting started](https://www.expectedparrot.com/en/latest/getting-started) using Expected Parrot tools for AI research.

## Construct questions about a dataset

We start by creating questions about a dataset, where each answer will provide a “label” for each piece of data. EDSL comes with many [common question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from a model (multiple choice, linear scale, matrix, etc.).

We use a “scenario” placeholder in each question text for data that we want to add to it. This method allows us to efficiently readminister a question for each piece of data. [Scenarios](/en/latest/scenarios) can be created from many types of data, including PNG, PDF, CSV, docs, lists, tables, videos, and other types.

We combine the questions in a [survey](/en/latest/surveys) in order to administer them together, asynchronously by default, or else according to any [logic or rules](/en/latest/surveys#survey-rules-logic) that we want to add (e.g., skip/stop rules).

\[1]:

```python theme={null}
from edsl import ScenarioList, QuestionList, QuestionNumerical, Survey

q1 = QuestionList(
    question_name = "characters",
    question_text = "Name all of the characters in this show: {{ scenario.show }}"
)

q2 = QuestionNumerical(
    question_name = "years",
    question_text = "Identify the year this show first aired: {{ scenario.show }}"
)

scenarios = ScenarioList.from_source("list", "show", ["The Simpsons", "South Park", "I Love Lucy"])

questions = q1.loop(scenarios) + q2.loop(scenarios)

survey = Survey(questions)
```

## Generate data “labels” using LLMs

EDSL allows us to [specify the models](/en/latest/language_models) that we want to use to answer the questions, and optionally [design AI agent personas](/en/latest/agents) for the models to reference in answering the questions. This can be useful if you want to reference specific expertise that is relevant to the labeling task.

We administer the questions by adding the scenarios, agents and models to the survey and calling the `run()` method. This generates a formatted dataset of `Results` that we can analyze with [built-in methods for working with results](/en/latest/results).

\[2]:

```python theme={null}
from edsl import Agent, AgentList, Model, ModelList

agents = AgentList([
    Agent(traits = {"persona":"You watch a lot of TV."})
])

models = ModelList([
    Model("gemini-1.5-flash", service_name = "google"),
    Model("gpt-4o", service_name = "openai")
])

results = survey.by(scenarios).by(agents).by(models).run()
```

Results are accessible at your Expected Parrot account and at your workspace. We can inspect a list of all the components of the results:

\[3]:

```python theme={null}
results.columns
```

Here we select components to display in a table:

\[4]:

```python theme={null}
results.select("model", "persona", "characters_0", "years_0", "characters_1", "years_1", "characters_2", "years_2")
```

## Run the survey with human respondents

We can validate some of all of the responses with human respondents by calling the `humanize()` method on the version of the survey that we want to validate with humans. This method generates a shareable URL for a web-based version of the survey that you can distribute, together with a URL for tracking the responses at your Expected Parrot account.

Here we create a new version of the survey to add some screening/information questions of the humans that answer it:

\[5]:

```python theme={null}
from edsl import QuestionLinearScale

q3 = QuestionLinearScale(
    question_name = "tv_viewing",
    question_text = "On a scale from 1 to 5, how much tv would you say that you've watched in your life?",
    question_options = [1,2,3,4,5],
    option_labels = {
        1:"None at all",
        5:"A ton"
    }
)

q4 = QuestionNumerical(
    question_name = "age",
    question_text = "How old are you (in years)?"
)

new_questions = [q3, q4]

human_survey = Survey(questions + new_questions)
```

\[6]:

```python theme={null}
human_survey.humanize()
```

Responses automatically appear at your Expected Parrot account, and you can import them into your workspace using `Coop` methods:

\[7]:

```python theme={null}
from edsl import Coop

human_results = Coop().get_project_human_responses("bbb84776-3364-4bc9-b028-0119cd84d480")
human_results
```

\[8]:

```python theme={null}
human_results.select("age", "tv_viewing", "characters_0", "years_0", "characters_1", "years_1", "characters_2", "years_2")
```


# Creating a digital twin
Source: https://docs.expectedparrot.com/en/latest/notebooks/digital_twin

This notebook contains sample [EDSL](/en/latest/index) code for creating an AI agent and prompting it to critique some content. The code is readily editable to create other agents and survey questions with any available language models.

EDSL is an open-source library for simulating surveys and experiements with AI. Please see our [documentation page](/en/latest/index) for tips and tutorials on getting started.

```python theme={null}
from edsl import (
    QuestionMultipleChoice,
    QuestionCheckBox,
    QuestionFreeText,
    QuestionLinearScale,
    Agent,
    ScenarioList,
    Survey
)
```

```python theme={null}
# Construct relevant traits as a dictionary
agent_traits = {
    "persona": """You are a middle-aged mom in Cambridge, Massachusetts.
        You hope to own a driverless minivan in the near future.
        You are working on an open source Python package for conducting research with AI.""",
    "age": 45,
    "location": "US",
    "industry": "information technology",
    "company": "Expected Parrot",
    "occupation": "startup cofounder",
    "hobbies": "kayaking, beach walks",
}

# Pass the traits and an optional name to an agent
agent = Agent(name="Robin", traits=agent_traits)
```

```python theme={null}
# Optionally create some special instructions for the task
context = """You are answering questions about a software package for conducting surveys and experiments
          with large language models. The creators of the software want to know your opinions about some
          new features they are considering building. Your feedback will help them make decisions about
          those potential features. """
```

```python expandable theme={null}
# Construct questions for the task
q1 = QuestionMultipleChoice(
    question_name="use_often",
    question_text=context
    + """Consider the following new feature: {{ scenario.content }}
    How often do you think you would use it?""",
    question_options=["Never", "Occasionally", "Frequently", "All the time"],
)

q2 = QuestionCheckBox(
    question_name="checkbox",
    question_text=context
    + """Consider the following new feature: {{ scenario.content }}
    Select all that apply.""",
    question_options=[
        "This feature would be useful to me.",
        "This feature would make me more productive.",
        "This feature will be important to me.",
        "The benefits of this feature are not clear to me.",
        "I would like to see some examples of how to use this feature.",
    ],
)

q3 = QuestionFreeText(
    question_name="concerns",
    question_text=context
    + "Do you have any concerns about the value and usefulness of this new feature: {{ scenario.content }}",
)

q4 = QuestionLinearScale(
    question_name="likely_to_use",
    question_text=context
    + """Consider the following new feature: {{ scenario.content }}
    On a scale from 1 to 5, how likely are you to use this new feature?
    (1 = not at all likely, 5 = very likely)""",
    question_options=[1, 2, 3, 4, 5],
    option_labels={1: "Not at all likely", 5: "Very likely"},
)
```

```python theme={null}
# Create a survey with the questions
survey = Survey(questions=[q1, q2, q3, q4])
```

```python theme={null}
# Create some content for the agent to review
contents = [
    "An optional progress bar that shows how many of your questions have been answered while your survey is running.",
    "A method that lets you quickly check what version of the package you have installed.",
    "A method that lets you include questions and responses as context for new questions.",
]

# Parameterize the questions with the content
scenarios = ScenarioList.from_list("content", contents)
```

```python theme={null}
agent
```

[Agent](/en/latest/agents)

| | key | value |
| :- | :- | :- |
| 0 | traits:persona | You are a middle-aged mom in Cambridge, Massachusetts. You hope to own a driverless minivan in the near future. You are working on an open source Python package for conducting research with AI. |
| 1 | traits:age | 45 |
| 2 | traits:location | US |
| 3 | traits:industry | information technology |
| 4 | traits:company | Expected Parrot |
| 5 | traits:occupation | startup cofounder |
| 6 | traits:hobbies | kayaking, beach walks |
| 7 | name | Robin |

```python theme={null}
# Run the survey and store the results; we can also see a progress bar
results = survey.by(scenarios).by(agent).run()
```

```python theme={null}
# Show all columns of the Results object
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.age |
| 1 | agent.agent\_index |
| 2 | agent.agent\_instruction |
| 3 | agent.agent\_name |
| 4 | agent.company |
| 5 | agent.hobbies |
| 6 | agent.industry |
| 7 | agent.location |
| 8 | agent.occupation |
| 9 | agent.persona |
| 10 | answer.checkbox |
| 11 | answer.concerns |
| 12 | answer.likely\_to\_use |
| 13 | answer.use\_often |
| 14 | cache\_keys.checkbox\_cache\_key |
| 15 | cache\_keys.concerns\_cache\_key |
| 16 | cache\_keys.likely\_to\_use\_cache\_key |
| 17 | cache\_keys.use\_often\_cache\_key |
| 18 | cache\_used.checkbox\_cache\_used |
| 19 | cache\_used.concerns\_cache\_used |
| 20 | cache\_used.likely\_to\_use\_cache\_used |
| 21 | cache\_used.use\_often\_cache\_used |
| 22 | comment.checkbox\_comment |
| 23 | comment.concerns\_comment |
| 24 | comment.likely\_to\_use\_comment |
| 25 | comment.use\_often\_comment |
| 26 | generated\_tokens.checkbox\_generated\_tokens |
| 27 | generated\_tokens.concerns\_generated\_tokens |
| 28 | generated\_tokens.likely\_to\_use\_generated\_tokens |
| 29 | generated\_tokens.use\_often\_generated\_tokens |
| 30 | iteration.iteration |
| 31 | model.frequency\_penalty |
| 32 | model.inference\_service |
| 33 | model.logprobs |
| 34 | model.max\_tokens |
| 35 | model.model |
| 36 | model.model\_index |
| 37 | model.presence\_penalty |
| 38 | model.temperature |
| 39 | model.top\_logprobs |
| 40 | model.top\_p |
| 41 | prompt.checkbox\_system\_prompt |
| 42 | prompt.checkbox\_user\_prompt |
| 43 | prompt.concerns\_system\_prompt |
| 44 | prompt.concerns\_user\_prompt |
| 45 | prompt.likely\_to\_use\_system\_prompt |
| 46 | prompt.likely\_to\_use\_user\_prompt |
| 47 | prompt.use\_often\_system\_prompt |
| 48 | prompt.use\_often\_user\_prompt |
| 49 | question\_options.checkbox\_question\_options |
| 50 | question\_options.concerns\_question\_options |
| 51 | question\_options.likely\_to\_use\_question\_options |
| 52 | question\_options.use\_often\_question\_options |
| 53 | question\_text.checkbox\_question\_text |
| 54 | question\_text.concerns\_question\_text |
| 55 | question\_text.likely\_to\_use\_question\_text |
| 56 | question\_text.use\_often\_question\_text |
| 57 | question\_type.checkbox\_question\_type |
| 58 | question\_type.concerns\_question\_type |
| 59 | question\_type.likely\_to\_use\_question\_type |
| 60 | question\_type.use\_often\_question\_type |
| 61 | raw\_model\_response.checkbox\_cost |
| 62 | raw\_model\_response.checkbox\_one\_usd\_buys |
| 63 | raw\_model\_response.checkbox\_raw\_model\_response |
| 64 | raw\_model\_response.concerns\_cost |
| 65 | raw\_model\_response.concerns\_one\_usd\_buys |
| 66 | raw\_model\_response.concerns\_raw\_model\_response |
| 67 | raw\_model\_response.likely\_to\_use\_cost |
| 68 | raw\_model\_response.likely\_to\_use\_one\_usd\_buys |
| 69 | raw\_model\_response.likely\_to\_use\_raw\_model\_response |
| 70 | raw\_model\_response.use\_often\_cost |
| 71 | raw\_model\_response.use\_often\_one\_usd\_buys |
| 72 | raw\_model\_response.use\_often\_raw\_model\_response |
| 73 | scenario.content |
| 74 | scenario.scenario\_index |

```python theme={null}
# Print the responses
results.select(
    "content",
    "use_often",
    "checkbox",
    "concerns",
    "likely_to_use",
)
```

| | scenario.content | answer.use\_often | answer.checkbox | answer.concerns | answer.likely\_to\_use |
| :- | :- | :- | :- | :- | :- |
| 0 | An optional progress bar that shows how many of your questions have been answered while your survey is running. | Frequently | \['This feature would be useful to me.', 'This feature would make me more productive.', 'This feature will be important to me.'] | Oh, I think an optional progress bar could be quite helpful! As someone who's worked on software projects, I know how useful it is to have visual feedback during a process. It can make the experience more engaging and less stressful, especially if the survey is lengthy. People often appreciate knowing how much they've completed and what's left. However, I'd suggest making sure it's truly optional, as some users might find it distracting. Also, ensuring that it doesn't slow down the survey process would be key. Overall, it sounds like a feature that could enhance user experience if implemented thoughtfully. | 4 |
| 1 | A method that lets you quickly check what version of the package you have installed. | Occasionally | \['This feature would be useful to me.', 'This feature would make me more productive.', 'This feature will be important to me.'] | Oh, having a quick way to check the version of a software package is always a handy feature! Especially in the world of open source and Python, where updates can happen frequently. It helps ensure compatibility and troubleshoot issues more effectively. I think it would be quite valuable, especially for those of us who might not be as tech-savvy or who manage multiple packages and need to keep everything organized. Plus, it can save time when you're trying to figure out if you need to update or if a certain feature is available in your current version. So, I don't have any concerns about its usefulness; it sounds like a practical addition. | 5 |
| 2 | A method that lets you include questions and responses as context for new questions. | Frequently | \['This feature would be useful to me.', 'This feature would make me more productive.', 'This feature will be important to me.', 'I would like to see some examples of how to use this feature.'] | Oh, I think that sounds like a really promising feature! Including previous questions and responses as context for new questions could really enhance the depth and relevance of the information gathered. It would allow for more dynamic and personalized interactions, which is great for conducting nuanced surveys or experiments. However, one concern might be ensuring that the context doesn't become overwhelming or confusing for the model, especially if the previous interactions are lengthy or complex. It could also be a challenge to manage this context effectively in terms of data storage and processing speed. But overall, if implemented well, this feature could add significant value by enabling more sophisticated and context-aware querying. | 5 |

```python theme={null}
# Post the notebook on Expected Parrot
from edsl import Notebook

nb = Notebook(path = "digital_twin.ipynb")

if refresh := False:
    nb.push(
        description = "Example code for creating a digital twin",
        alias = "digital-twin-notebook",
        visibility = "public"
    )

# update if it already exists
else:
    nb.patch('b403f472-757a-4b6b-8c03-a00c941d157d', value = nb)
```


# Using EDSL for AI polling
Source: https://docs.expectedparrot.com/en/latest/notebooks/edsl_polling

This notebook demonstrates methods for conducting surveys with AI agents using [EDSL](https://github.com/expectedparrot/edsl), an open-source Python library for simulating social science research with large language models. EDSL is developed by [Expected Parrot](https://www.expectedparrot.com/), a start-up building tools for AI-powered research.

## Contents

The sections below include:

1. **Creating an AI agent:** Basic steps to construct an AI agent.
2. **Administering questions:** How to create questions and prompt agents to answer them.
3. **Selecting language models:** Specify language models that you want to use to generate responses.
4. **Analyzing results:** Examples of built-in methods for analyzing responses as datasets.
5. **Designing agent traits:** How to construct agents with complex personas and traits.
6. **Converting surveys into EDSL:** Import other surveys into EDSL to analyze and extend them with agents.
7. **Constructing agents from survey data:** Use survey responses to construct agents representing respondents.

## Sample data: *Cooperative Election Study Common Content, 2022*

For purposes of demonstration, we use data from the [Cooperative Election Study Common Content, 2022](https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi%3A10.7910/DVN/PR4L8P) in several ways:

* In this notebook we use lists of respondent attributes from the *Breakdown of National Vote for U.S. House (CES validated voters)* ([CES Guide 2022](https://dataverse.harvard.edu/file.xhtml?fileId=7359254\&version=4.0) pp.24-25) to design agents with combinations of the attributes, and then administer questions to them.

Companion notebooks:

* In [Working with CES data in EDSL](/en/latest/notebooks/ces_data_edsl) we reconstruct the [2022 CES Pre-Election Questionnaire](https://dataverse.harvard.edu/file.xhtml?fileId=6987030\&version=4.0) in EDSL. The formatted questions can be modified and administered to agents in new surveys as desired. The notebook also contains dictionaries of the CES questions and answer codebook for convenience in working with the data.
* In [Creating agents for survey respondents](/en/latest/notebooks/ces_agents) we use the [dataset of responses to the 2022 CES Pre-Election Questionnaire](https://dataverse.harvard.edu/file.xhtml?fileId=10140882\&version=4.0) to create agents representing the respondents, and administer follow-on questions with them.

## Reference & contact

Documentation for the EDSL package is available at [https://docs/expectedparrot.com](https://docs/expectedparrot.com). You can also find example code, tutorials and notebooks for a variety of use cases.

Please let us know if you have any questions or encounter issues working with this data:

* Discord: [https://discord.com/invite/mxAYkjfy9m](https://discord.com/invite/mxAYkjfy9m)
* Email: [info@expectedparrot.com](mailto:info@expectedparrot.com)

## Technical setup

EDSL is compatible with Python 3.9-3.12.

See instructions on [installing the EDSL library](/en/latest/installation) and [storing API keys](/en/latest/api_keys) for the language models that you want to use. In examples below where no model is specified, EDSL will use GPT 4 by default (an API key for OpenAI is required). We also show how to use different models.

## Creating an AI agent

In this section we show how to create an AI agent and give it desired attributes. For more details on constructing and using AI agents please see our [documentation page on agents](/en/latest/agents).

We start by importing the tools for creating agents:

```python theme={null}
from edsl import Agent
```

Here we create a simple `Agent` and pass it a dictionary of `traits`. We optionally include a narrative persona and also specify traits individually for use in segmenting and analyzing survey responses:

```python theme={null}
agent = Agent(
    traits={
        "persona": "You are 55-year-old research scientist living in Cambridge, Massachusetts.",
        "occupation": "Research scientist",
        "location": "Cambridge, Massachusetts",
        "age": 55,
    }
)
```

We can access the traits directly:

```python theme={null}
agent.location
```

```python theme={null}
'Cambridge, Massachusetts'
```

## Designing agent panels

We can also create panels of agents in an `AgentList` and administer surveys to all of the agents at once. Here we construct combinations of traits from lists of respondent attributes in the [CES Guide](https://dataverse.harvard.edu/file.xhtml?fileId=7359254\&version=4.0) (see source details above). (Information can be imported from a variety of data source types; [see documentation for details](/en/latest/).)

```python expandable theme={null}
sex = ["Male", "Female"]
race = ["White", "Black", "Hispanic", "Asian", "Other"]
age = ["18-29", "30-44", "45-64", "65 and over"]
education = [
    "High school or less",
    "Some college/assoc. degree",
    "College/graduate",
    "Postgraduate study",
]
income = [
    "Under $30,000",
    "$30,000 to $49,999",
    "$50,000 to $99,999",
    "$100,000 to $199,999",
    "$200,000 or more",
]
party_affiliation = ["Democrat", "Republican", "Independent/Other"]
political_ideology = ["Liberal", "Moderate", "Conservative", "Unsure"]
religion = [
    "Protestant/other Christian",
    "Catholic",
    "Jewish",
    "Something else",
    "None",
]
evangelical = ["Yes", "No"]
married = ["Yes", "No"]
lgbt = ["Yes", "No"]
```

Here we create a method to generate a list of agents with randomly selected combinations of traits:

```python theme={null}
from edsl import AgentList
import random

def generate_random_agents(num_agents):
    agents = []
    for _ in range(num_agents):
        agent_traits = {
            "sex": random.choice(sex),
            "race": random.choice(race),
            "age": random.choice(age),
            "education": random.choice(education),
            "income": random.choice(income),
            "party_affiliation": random.choice(party_affiliation),
            "political_ideology": random.choice(political_ideology),
            "religion": random.choice(religion),
            "evangelical": random.choice(evangelical),
            "married": random.choice(married),
            "lgbt": random.choice(lgbt),
        }
        agents.append(Agent(traits=agent_traits))

    return AgentList(agents)
```

Example usage:

```python theme={null}
num_agents = 3
agents = generate_random_agents(num_agents)
```

## Agent instructions

If we want to give all the agents a special instruction, we can optionally pass an `instruction` to the agents (this can also be done when the agents are created):

```python theme={null}
for agent in agents:
    agent.instruction = "Today is July 1, 2022."
```

We can inspect the agents that have been created:

```python theme={null}
agents
```

[AgentList](/en/latest/agents) agents: 3;

| | sex | race | age | education | income | party\_affiliation | political\_ideology | religion | evangelical | married | lgbt |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | Female | Asian | 30-44 | College/graduate | $30 , 000 t o$49,999 | Republican | Conservative | Jewish | Yes | No | Yes |
| 1 | Male | Hispanic | 18-29 | High school or less | $100 , 000 t o$199,999 | Democrat | Unsure | Protestant/other Christian | Yes | No | Yes |
| 2 | Female | White | 30-44 | College/graduate | $100 , 000 t o$199,999 | Republican | Liberal | Something else | Yes | Yes | No |

## Creating questions

An `Agent` is designed to be assigned questions to answer. In this section we construct questions in the form of `Question` objects, combine them into a `Survey`, administer it to some sample agents (from above), and inspect the responses in the dataset of `Results` that is generated.

EDSL comes with many question types that we can select from based on the form of the response that we want to get back from the language model (free text, linear scale, checkbox, etc.). See [examples of all question types](/en/latest/questions).

Here we create a multiple choice question from the [CES Pre-Election Questionnaire](https://dataverse.harvard.edu/file.xhtml?fileId=6987030\&version=4.0) (the response will be a selection from the list of options that we include) and compose a follow-up free text question (the response will be unstructured text):

```python theme={null}
from edsl import QuestionMultipleChoice, QuestionFreeText

# From the CES pre-election questionnaire
q_pid3 = QuestionMultipleChoice(
    question_name="pid3",
    question_text="Generally speaking, do you think of yourself as a ...?",
    question_options=["Democrat", "Republican", "Independent", "Other", "Not sure"],
)

# Potential follow-up question
q_views = QuestionFreeText(
    question_name="views", question_text="Describe your political views."
)
```

We combine the questions into a `Survey` to administer them together:

```python theme={null}
from edsl import Survey

survey = Survey([q_pid3, q_views])
```

## Administering a survey

We administer a survey by calling the `run` method, after (optionally) adding agents with the `by` method:

```python theme={null}
results = survey.by(agents).run()
```

We can show a list of all the components of the `Results` that have been generated, and see that the results include information about the agents, questions, models, prompts and responses:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.age |
| 1 | agent.agent\_index |
| 2 | agent.agent\_instruction |
| 3 | agent.agent\_name |
| 4 | agent.education |
| 5 | agent.evangelical |
| 6 | agent.income |
| 7 | agent.lgbt |
| 8 | agent.married |
| 9 | agent.party\_affiliation |
| 10 | agent.political\_ideology |
| 11 | agent.race |
| 12 | agent.religion |
| 13 | agent.sex |
| 14 | answer.pid3 |
| 15 | answer.views |
| 16 | cache\_keys.pid3\_cache\_key |
| 17 | cache\_keys.views\_cache\_key |
| 18 | cache\_used.pid3\_cache\_used |
| 19 | cache\_used.views\_cache\_used |
| 20 | comment.pid3\_comment |
| 21 | comment.views\_comment |
| 22 | generated\_tokens.pid3\_generated\_tokens |
| 23 | generated\_tokens.views\_generated\_tokens |
| 24 | iteration.iteration |
| 25 | model.frequency\_penalty |
| 26 | model.inference\_service |
| 27 | model.logprobs |
| 28 | model.max\_tokens |
| 29 | model.model |
| 30 | model.model\_index |
| 31 | model.presence\_penalty |
| 32 | model.temperature |
| 33 | model.top\_logprobs |
| 34 | model.top\_p |
| 35 | prompt.pid3\_system\_prompt |
| 36 | prompt.pid3\_user\_prompt |
| 37 | prompt.views\_system\_prompt |
| 38 | prompt.views\_user\_prompt |
| 39 | question\_options.pid3\_question\_options |
| 40 | question\_options.views\_question\_options |
| 41 | question\_text.pid3\_question\_text |
| 42 | question\_text.views\_question\_text |
| 43 | question\_type.pid3\_question\_type |
| 44 | question\_type.views\_question\_type |
| 45 | raw\_model\_response.pid3\_cost |
| 46 | raw\_model\_response.pid3\_one\_usd\_buys |
| 47 | raw\_model\_response.pid3\_raw\_model\_response |
| 48 | raw\_model\_response.views\_cost |
| 49 | raw\_model\_response.views\_one\_usd\_buys |
| 50 | raw\_model\_response.views\_raw\_model\_response |
| 51 | scenario.scenario\_index |

We can select and print components of the `Results` in a table (see examples of all [methods for analyzing results](/en/latest/results)):

```python theme={null}
results.select("age", "education", "pid3", "views")
```

| | agent.age | agent.education | answer.pid3 | answer.views |
| :- | :- | :- | :- | :- |
| 0 | 30-44 | College/graduate | Republican | As an AI, I don't have personal beliefs or political views. However, I can provide information on a wide range of political topics and help explain different perspectives. If you have any specific questions or need information on a particular political issue, feel free to ask! |
| 1 | 18-29 | High school or less | Democrat | As an AI, I don't have personal beliefs or political views. However, I can provide information on various political ideologies, explain party platforms, or discuss current political events if you'd like. Let me know how I can assist you! |
| 2 | 30-44 | College/graduate | Republican | I don't have personal political views or opinions. However, I can provide information on a wide range of political topics, discuss different political ideologies, and summarize various political perspectives based on data and research. If you have specific questions or topics in mind, feel free to ask! |

## Answer commentary

Question types other than free text automatically include a `comment` field for the agent to provide any unstructured commentary on its response to a question. This is useful in ensuring that responses are formatted as specified, providing an outlet for model verbosity. For example, in `results.columns` we can see that there is a field `comment.pid3_comment`. We can inspect this field as we do any other component of results. Here we also apply some `pretty_labels` to our table for readability:

```python theme={null}
(
    results
    .select("pid3", "pid3_comment")
    .print(
        pretty_labels={"answer.pid3": "Party", "comment.pid3_comment": "Comment"},
        format="rich",
    )
)
```

| | Party | Comment |
| :- | :- | :- |
| 0 | Republican | Given your traits, you have identified your party affiliation as Republican, which aligns with your political ideology. |
| 1 | Democrat | Based on your traits, you are affiliated with the Democratic Party, which aligns with the option "Democrat." |
| 2 | Republican | Based on the provided traits, your party affiliation is listed as Republican. |

## Selecting language models

As mentioned above, if we do not specify a language model GPT 4 is used by default. We can also specify other language models to use in generating results, and compare responses for them.

To see a list of all available models (uncomment the code):

```python theme={null}
from edsl import Model, ModelList

# Model.available()
```

To select models for a survey, pass the model names to `Model` objects:

```python theme={null}
models = ModelList(Model(m) for m in ["gemini-pro", "gpt-4o"])
```

We add a `Model` or list of models to a survey with the `by` method, the same as we do agents:

```python theme={null}
results = survey.by(agents).by(models).run()

results.select("model", "pid3", "pid3_comment")
```

| | model.model | answer.pid3 | comment.pid3\_comment |
| :- | :- | :- | :- |
| 0 | gemini-pro | nan | Task failed with exception: Language model did not return a response for question 'pid3.'. |
| 1 | gpt-4o | Republican | Given your traits, you have identified your party affiliation as Republican, which aligns with your political ideology. |
| 2 | gemini-pro | nan | Task failed with exception: Language model did not return a response for question 'pid3.'. |
| 3 | gpt-4o | Democrat | Based on your traits, you are affiliated with the Democratic Party, which aligns with the option "Democrat." |
| 4 | gemini-pro | nan | Task failed with exception: Language model did not return a response for question 'pid3.'. |
| 5 | gpt-4o | Republican | Based on the provided traits, your party affiliation is listed as Republican. |

Learn more about [specifying language models](/en/latest/language_models).

## Question context & memory

Survey questions are administered asynchronously by default, for efficiency. If we want an agent to have the context of one or more prior questions when presented a new question we can apply a rule specifying the questions and answers to add to the new question prompt:

```python theme={null}
from edsl import QuestionMultipleChoice, QuestionFreeText, Survey

# From the CES pre-election questionnaire
q_CC22_309e = QuestionMultipleChoice(
    question_name="CC22_309e",
    question_text="Would you say that in general your health is...",
    question_options=["Excellent", "Very good", "Good", "Fair", "Poor"],
)

q_CC22_309f = QuestionMultipleChoice(
    question_name="CC22_309f",
    question_text="Would you say that in general your mental health is...",
    question_options=["Excellent", "Very good", "Good", "Fair", "Poor"],
)

survey = Survey([q_CC22_309e, q_CC22_309f])
```

Here we add a memory of `q_CC22_309e` when administering `q_CC22_309f` and show the prompts that were administered:

```python theme={null}
survey = survey.add_targeted_memory(q_CC22_309f, q_CC22_309e)
```

```python theme={null}
results = survey.run()
```

```python theme={null}
results.select("CC22_309e_user_prompt", "CC22_309e", "CC22_309f_user_prompt", "CC22_309f")
```

| | prompt.CC22\_309e\_user\_prompt | answer.CC22\_309e | prompt.CC22\_309f\_user\_prompt | answer.CC22\_309f |
| :- | :- | :- | :- | :- |
| 0 | Would you say that in general your health is... Excellent Very good Good Fair Poor Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | Good | Would you say that in general your mental health is... Excellent Very good Good Fair Poor Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. Before the question you are now answering, you already answered the following question(s): Question: Would you say that in general your health is... Answer: Good | Good |

See examples of [all methods for applying question context and memories](/en/latest/surveys) (e.g., full memory of all prior questions, or a subset of questions).

## Piping questions

We can also pipe individual components of questions into other questions. Here we use the answer to `inputstate` in the question text for `CC22_320d`:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, Survey

q_inputstate = QuestionMultipleChoice(
    question_name="inputstate",
    question_text="What is your State of Residence?",
    question_options=[
        "Alabama",
        "Alaska",
        "Arizona",
        "Arkansas",
        "California",
        "Colorado",
        "Connecticut",
        "Delaware",
        "District of Columbia",
        "Florida",
        "Georgia",
        "Hawaii",
        "Idaho",
        "Illinois",
        "Indiana",
        "Iowa",
        "Kansas",
        "Kentucky",
        "Louisiana",
        "Maine",
        "Maryland",
        "Massachusetts",
        "Michigan",
        "Minnesota",
        "Mississippi",
        "Missouri",
        "Montana",
        "Nebraska",
        "Nevada",
        "New Hampshire",
        "New Jersey",
        "New Mexico",
        "New York",
        "North Carolina",
        "North Dakota",
        "Ohio",
        "Oklahoma",
        "Oregon",
        "Pennsylvania",
        "Rhode Island",
        "South Carolina",
        "South Dakota",
        "Tennessee",
        "Texas",
        "Utah",
        "Vermont",
        "Virginia",
        "Washington",
        "West Virginia",
        "Wisconsin",
        "Wyoming",
    ],
)

q_CC22_320d = QuestionMultipleChoice(
    question_name="CC22_320d",
    question_text="Do you approve of the way the Governor of {{ inputstate.answer }} is doing their job?",
    question_options=[
        "Strongly approve",
        "Somewhat approve",
        "Somewhat disapprove",
        "Strongly disapprove",
        "Not sure",
    ],
)

survey = Survey([q_inputstate, q_CC22_320d])

results = survey.by(agents).run()

results.select("inputstate", "CC22_320d_user_prompt", "CC22_320d")
```

| | answer.inputstate | prompt.CC22\_320d\_user\_prompt | answer.CC22\_320d |
| :- | :- | :- | :- |
| 0 | California | Do you approve of the way the Governor of California is doing their job? Strongly approve Somewhat approve Somewhat disapprove Strongly disapprove Not sure Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | Somewhat disapprove |
| 1 | California | Do you approve of the way the Governor of California is doing their job? Strongly approve Somewhat approve Somewhat disapprove Strongly disapprove Not sure Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | Somewhat approve |
| 2 | Florida | Do you approve of the way the Governor of Florida is doing their job? Strongly approve Somewhat approve Somewhat disapprove Strongly disapprove Not sure Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | Somewhat disapprove |

## Survey rules & conditions

We can apply survey rules and conditions to administer relevant follow-up questions based on responses to questions. For example, here we add skip rules to a set of questions by calling the method `add_skip_rule()` and passing the target question and the condition to evaluate (questions not administered will show a `None` response):

```python expandable theme={null}
from edsl import QuestionCheckBox, QuestionMultipleChoice, Survey

# From the CES pre-election questionnaire
q_CC22_300 = (
    QuestionCheckBox(  # Use checkbox to allow the agent to select multiple options
        question_name="CC22_300",
        question_text="In the past 24 hours have you... (check all that apply)",
        question_options=[
            "Used social media (such as Facebook or Youtube)",
            "Watched TV news",
            "Read a newspaper in print or online",
            "Listened to a radio news program or talk radio",
            "None of these",
        ],
    )
)

# Skip this question if the agent does not select "Watched TV news"
q_CC22_300a = QuestionMultipleChoice(
    question_name="CC22_300a",
    question_text="Did you watch local news, national news, or both?",
    question_options=["Local Newscast", "National Newscast", "Both"],
)

# Skip this question if the agent does not select "Watched TV news"
q_CC22_300b = QuestionMultipleChoice(
    question_name="CC22_300b",
    question_text="Which of these networks did you watch?",
    question_options=["ABC", "CBS", "NBC", "CNN", "Fox News", "MSNBC", "PBS", "Other"],
)

# Skip this question if the agent does not select "Read a newspaper..."
q_CC22_300c = QuestionMultipleChoice(
    question_name="CC22_300c",
    question_text="Did you read a print newspaper, an online newspaper, or both?",
    question_options=["Print", "Online", "Both"],
)

# Skip this question if the agent does not select "Used social media..."
q_CC22_300d = QuestionMultipleChoice(
    question_name="CC22_300d",
    question_text="In the past 24 hours, did you do any of the following on social media (such as Facebook, Youtube or Twitter)?",
    question_options=[
        "Posted a story, photo, video or link about politics",
        "Posted a comment about politics",
        "Read a story or watched a video about politics",
        "Followed a political event",
        "Forwarded a story, photo, video or link about politics to friends",
        "None of the above",
    ],
)

survey_CC22_300 = (
    Survey([q_CC22_300, q_CC22_300a, q_CC22_300b, q_CC22_300c, q_CC22_300d])
    .add_skip_rule(q_CC22_300a, "'Watched TV news' not in CC22_300")
    .add_skip_rule(q_CC22_300b, "'Watched TV news' not in CC22_300")
    .add_skip_rule(q_CC22_300c, "'Read a newspaper in print or online' not in CC22_300")
    .add_skip_rule(
        q_CC22_300d, "'Used social media (such as Facebook or Youtube)' not in CC22_300"
    )
)

results_CC22_300 = survey_CC22_300.by(agents).run()

results_CC22_300.select("CC22_300", "CC22_300a", "CC22_300b", "CC22_300c", "CC22_300d")
```

| | answer.CC22\_300 | answer.CC22\_300a | answer.CC22\_300b | answer.CC22\_300c | answer.CC22\_300d |
| :- | :- | :- | :- | :- | :- |
| 0 | \['Used social media (such as Facebook or Youtube)', 'Watched TV news', 'Read a newspaper in print or online', 'Listened to a radio news program or talk radio'] | National Newscast | Fox News | Online | Read a story or watched a video about politics |
| 1 | \['Used social media (such as Facebook or Youtube)', 'Watched TV news'] | National Newscast | CNN | nan | Read a story or watched a video about politics |
| 2 | \['Used social media (such as Facebook or Youtube)', 'Watched TV news', 'Read a newspaper in print or online'] | National Newscast | Fox News | Online | None of the above |

Here we use `add_stop_rule()` to end the survey based on the response to an initial question (an option selection that makes the second question unnecessary):

```python expandable theme={null}
from edsl import QuestionMultipleChoice, QuestionYesNo, Survey

# From the CES pre-election questionnaire
q_employ = QuestionMultipleChoice(
    question_name="employ",
    question_text="Which of the following best describes your current employment status?",
    question_options=[
        "Working full time now",
        "Working part time now",
        "Temporarily laid off",
        "Unemployed",
        "Retired",
        "Permanently disabled",
        "Taking care of home or family",
        "Student",
        "Other",
    ],
)

q_hadjob = QuestionYesNo(
    question_name="hadjob",
    question_text="At any time over the past five years, have you had a job?",
)

survey = Survey([q_employ, q_hadjob]).add_stop_rule(
    q_employ,
    "employ in ['Working full time now', 'Working part time now', 'Temporarily laid off']",
)

results = survey.by(agents).run()

results.select("employ", "hadjob")
```

| | answer.employ | answer.hadjob |
| :- | :- | :- |
| 0 | Working full time now | nan |
| 1 | Student | Yes |
| 2 | Working full time now | nan |

## Combining survey methods

Here we apply multiple methods at once: we add a memory of `region` to the prompt for `inputzip`, then pipe the answer to `inputzip` into the question text of `votereg_f`, and a stop rule if the answer to `votereg` is not “Yes” (i.e., do not administer `votereg_f`):

```python expandable theme={null}
from edsl import QuestionMultipleChoice, QuestionList, QuestionYesNo, Survey

# From the CES pre-election questionnaire
q_region = QuestionMultipleChoice(
    question_name="region",
    question_text="In which census region do you live?",
    question_options=["Northeast", "Midwest", "South", "West"],
)

q_inputzip = QuestionList(
    question_name="inputzip",
    question_text="So that we can ask you about the news and events in your area, in what zip code do you currently reside?",
    max_list_items=1,
)

q_votereg = QuestionMultipleChoice(
    question_name="votereg",
    question_text="Are you registered to vote?",
    question_options=["Yes", "No", "Don't know"],
)

q_votereg_f = QuestionYesNo(
    question_name="votereg_f",
    question_text="Is {{ inputzip.answer[0] }} the zip code where you are registered to vote?",
)

survey = (
    Survey([q_region, q_inputzip, q_votereg, q_votereg_f])
    .add_targeted_memory(q_inputzip, q_region)
    .add_stop_rule(q_votereg, "votereg != 'Yes'")
)

results = survey.by(agents).run()

results.select("region", "inputzip", "votereg", "votereg_f_user_prompt", "votereg_f")
```

| | answer.region | answer.inputzip | answer.votereg | prompt.votereg\_f\_user\_prompt | answer.votereg\_f |
| :- | :- | :- | :- | :- | :- |
| 0 | West | \[94016] | Yes | Is 94016 the zip code where you are registered to vote? No Yes Only 1 option may be selected. Please respond with just your answer. After the answer, you can put a comment explaining your response. | No |
| 1 | South | \[33101] | Yes | Is 33101 the zip code where you are registered to vote? No Yes Only 1 option may be selected. Please respond with just your answer. After the answer, you can put a comment explaining your response. | No |
| 2 | South | \[30301] | Yes | Is 30301 the zip code where you are registered to vote? No Yes Only 1 option may be selected. Please respond with just your answer. After the answer, you can put a comment explaining your response. | No |

See more details on [all survey methods](/en/latest/surveys).

## Parameterizing questions

We can create variations of questions using `Scenario` objects for content that we want to add to questions. This allows us to efficiently administer multiple versions of questions at once. We start by using a `{{ parameter }}` in a question:

```python theme={null}
from edsl import QuestionMultipleChoice, Survey

# Modified from the CES pre-election questionnaire
q_votereg = QuestionMultipleChoice(
    question_name="votereg",
    question_text="Are you {{ scenario.status }}?",
    question_options=["Yes", "No", "Don't know"],
)

survey = Survey([q_votereg])
```

Next we create a `Scenario` for each text that we want to insert in the question:

```python theme={null}
from edsl import Scenario, ScenarioList

statuses = [
    "registered to vote",  # original CES question
    "enrolled in school",
    "employed full- or part-time",
    "married or in a domestic partnership",
    "licensed to drive",
]

scenarios = ScenarioList(Scenario({"status": s}) for s in statuses)
```

```python theme={null}
results = survey.by(scenarios).by(agents).run()
```

```python theme={null}
(
    results.sort_by("status", "age")
    .select("age", "education", "sex", "status", "votereg")
)
```

| | agent.age | agent.education | agent.sex | scenario.status | answer.votereg |
| :- | :- | :- | :- | :- | :- |
| 0 | 18-29 | High school or less | Male | employed full- or part-time | Yes |
| 1 | 30-44 | College/graduate | Female | employed full- or part-time | Yes |
| 2 | 30-44 | College/graduate | Female | employed full- or part-time | Yes |
| 3 | 18-29 | High school or less | Male | enrolled in school | Yes |
| 4 | 30-44 | College/graduate | Female | enrolled in school | No |
| 5 | 30-44 | College/graduate | Female | enrolled in school | No |
| 6 | 18-29 | High school or less | Male | licensed to drive | Don't know |
| 7 | 30-44 | College/graduate | Female | licensed to drive | Don't know |
| 8 | 30-44 | College/graduate | Female | licensed to drive | Yes |
| 9 | 18-29 | High school or less | Male | married or in a domestic partnership | No |
| 10 | 30-44 | College/graduate | Female | married or in a domestic partnership | No |
| 11 | 30-44 | College/graduate | Female | married or in a domestic partnership | Yes |
| 12 | 18-29 | High school or less | Male | registered to vote | Yes |
| 13 | 30-44 | College/graduate | Female | registered to vote | Yes |
| 14 | 30-44 | College/graduate | Female | registered to vote | Yes |


# Azure example
Source: https://docs.expectedparrot.com/en/latest/notebooks/edsl_with_cloud_providers



### Requirements

1. Deployed models in [Azure AI](https://ai.azure.com/)
2. Set environment variable AZURE\_ENDPOINT\_URL\_AND\_KEY to contain all the models that you have hosted on Azure
   * Example for multiple models deployed in Azure (separated by , ):
     AZURE\_ENDPOINT\_URL\_AND\_KEY=https\://model\_1\_link:api\_key\_1,https\://model\_2\_link:api\_key\_2

### List available models

```python theme={null}
from edsl import Model

azure_models = Model.available(service="azure")
azure_models
```

| | Model Name | Service Name |
| :- | :- | :- |
| 0 | azure:gpt-4o-test | azure |
| 1 | azure:gpt-4o-mini-test | azure |
| 2 | Mistral-Large-2411-thgdv | azure |

### Run survey using models hosted on Azure

```python theme={null}
from edsl import QuestionFreeText
from edsl import Model

for model_name in azure_models:
    print("#############")
    print(model_name[0])
    model = Model(model_name[0])

    question = QuestionFreeText(question_name="q1",question_text="How many satellites are in space?")

    res = question.by(model).run(disable_remote_inference=True,cache=False)

    print(res.select("answer.*").print())
```

```
#############
azure:gpt-4o-test
answer.q1
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
As of 2023, there are over 7,000 satellites in space. This number includes both operational satellites and those that are no longer functioning but remain in orbit. The number of satellites is continuously increasing due to the growing demand for satellite-based services, such as communication, navigation, and Earth observation.
#############
azure:gpt-4o-mini-test
answer.q1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
As of October 2023, there are thousands of satellites currently in orbit around Earth. The exact number is constantly changing due to new satellite launches and decommissioning of older ones. Reports indicate that there are over 3,500 operational satellites in orbit, with many more non-operational or defunct satellites contributing to the overall count. Additionally, the number of small satellites and CubeSats has been increasing rapidly in recent years due to advancements in technology and reduced launch costs. For the most accurate and up-to-date information, it’s best to check resources like the Union of Concerned Scientists’ satellite database or relevant space agencies.
#############
Mistral-Large-2411-thgdv
answer.q1
-----------
As of my last update in October 2023, the exact number of satellites in space can vary frequently due to new launches and decommissioning of old satellites. However, it is estimated that there are thousands of operational satellites in orbit around Earth. This number includes satellites used for various purposes such as communication, navigation, Earth observation, scientific research, and military applications.

In addition to operational satellites, there are also numerous pieces of space debris, including defunct satellites and other objects, which add to the overall count of man-made objects in space.

For the most current and precise information, you can refer to databases maintained by organizations such as the United Nations Office for Outer Space Affairs (UNOOSA), the Union of Concerned Scientists, or space agencies like NASA and ESA.
```

## AWS Bedrock

1. Set envariable variables AWS\_ACCESS\_KEY\_ID and AWS\_SECRET\_ACCESS\_KEY

### List available models

```python theme={null}
from edsl import Model

aws_bedrock_models = Model.available(service="bedrock")
aws_bedrock_models
```

| | Model Name | Service Name |
| :- | :- | :- |
| 0 | amazon.titan-tg1-large | bedrock |
| 1 | amazon.titan-image-generator-v1:0 | bedrock |
| 2 | amazon.titan-image-generator-v1 | bedrock |
| 3 | amazon.titan-image-generator-v2:0 | bedrock |
| 4 | amazon.titan-text-premier-v1:0 | bedrock |
| 5 | amazon.nova-pro-v1:0:300k | bedrock |
| 6 | amazon.nova-pro-v1:0 | bedrock |
| 7 | amazon.nova-lite-v1:0:300k | bedrock |
| 8 | amazon.nova-lite-v1:0 | bedrock |
| 9 | amazon.nova-canvas-v1:0 | bedrock |
| 10 | amazon.nova-reel-v1:0 | bedrock |
| 11 | amazon.nova-micro-v1:0:128k | bedrock |
| 12 | amazon.nova-micro-v1:0 | bedrock |
| 13 | amazon.titan-embed-g1-text-02 | bedrock |
| 14 | amazon.titan-text-lite-v1:0:4k | bedrock |
| 15 | amazon.titan-text-lite-v1 | bedrock |
| 16 | amazon.titan-text-express-v1:0:8k | bedrock |
| 17 | amazon.titan-text-express-v1 | bedrock |
| 18 | amazon.titan-embed-text-v1:2:8k | bedrock |
| 19 | amazon.titan-embed-text-v1 | bedrock |
| 20 | amazon.titan-embed-text-v2:0:8k | bedrock |
| 21 | amazon.titan-embed-text-v2:0 | bedrock |
| 22 | amazon.titan-embed-image-v1:0 | bedrock |
| 23 | amazon.titan-embed-image-v1 | bedrock |
| 24 | stability.stable-diffusion-xl-v1:0 | bedrock |
| 25 | stability.stable-diffusion-xl-v1 | bedrock |
| 26 | ai21.j2-ultra-v1:0:8k | bedrock |
| 27 | ai21.jamba-instruct-v1:0 | bedrock |
| 28 | ai21.jamba-1-5-large-v1:0 | bedrock |
| 29 | ai21.jamba-1-5-mini-v1:0 | bedrock |
| 30 | anthropic.claude-instant-v1:2:100k | bedrock |
| 31 | anthropic.claude-instant-v1 | bedrock |
| 32 | anthropic.claude-v2:0:18k | bedrock |
| 33 | anthropic.claude-v2:0:100k | bedrock |
| 34 | anthropic.claude-v2:1:18k | bedrock |
| 35 | anthropic.claude-v2:1:200k | bedrock |
| 36 | anthropic.claude-v2:1 | bedrock |
| 37 | anthropic.claude-v2 | bedrock |
| 38 | anthropic.claude-3-sonnet-20240229-v1:0:28k | bedrock |
| 39 | anthropic.claude-3-sonnet-20240229-v1:0:200k | bedrock |
| 40 | anthropic.claude-3-sonnet-20240229-v1:0 | bedrock |
| 41 | anthropic.claude-3-haiku-20240307-v1:0:48k | bedrock |
| 42 | anthropic.claude-3-haiku-20240307-v1:0:200k | bedrock |
| 43 | anthropic.claude-3-haiku-20240307-v1:0 | bedrock |
| 44 | anthropic.claude-3-opus-20240229-v1:0:12k | bedrock |
| 45 | anthropic.claude-3-opus-20240229-v1:0:28k | bedrock |
| 46 | anthropic.claude-3-opus-20240229-v1:0:200k | bedrock |
| 47 | anthropic.claude-3-opus-20240229-v1:0 | bedrock |
| 48 | anthropic.claude-3-5-sonnet-20240620-v1:0 | bedrock |
| 49 | anthropic.claude-3-5-sonnet-20241022-v2:0 | bedrock |
| 50 | anthropic.claude-3-7-sonnet-20250219-v1:0 | bedrock |
| 51 | anthropic.claude-3-5-haiku-20241022-v1:0 | bedrock |
| 52 | cohere.command-text-v14:7:4k | bedrock |
| 53 | cohere.command-text-v14 | bedrock |
| 54 | cohere.command-r-v1:0 | bedrock |
| 55 | cohere.command-r-plus-v1:0 | bedrock |
| 56 | cohere.command-light-text-v14:7:4k | bedrock |
| 57 | cohere.command-light-text-v14 | bedrock |
| 58 | cohere.embed-english-v3:0:512 | bedrock |
| 59 | cohere.embed-english-v3 | bedrock |
| 60 | cohere.embed-multilingual-v3:0:512 | bedrock |
| 61 | cohere.embed-multilingual-v3 | bedrock |
| 62 | meta.llama3-8b-instruct-v1:0 | bedrock |
| 63 | meta.llama3-70b-instruct-v1:0 | bedrock |
| 64 | meta.llama3-1-8b-instruct-v1:0 | bedrock |
| 65 | meta.llama3-1-70b-instruct-v1:0 | bedrock |
| 66 | meta.llama3-2-11b-instruct-v1:0 | bedrock |
| 67 | meta.llama3-2-90b-instruct-v1:0 | bedrock |
| 68 | meta.llama3-2-1b-instruct-v1:0 | bedrock |
| 69 | meta.llama3-2-3b-instruct-v1:0 | bedrock |
| 70 | meta.llama3-3-70b-instruct-v1:0 | bedrock |
| 71 | mistral.mistral-7b-instruct-v0:2 | bedrock |
| 72 | mistral.mixtral-8x7b-instruct-v0:1 | bedrock |
| 73 | mistral.mistral-large-2402-v1:0 | bedrock |
| 74 | mistral.mistral-small-2402-v1:0 | bedrock |

### Run Survey using AWS hosted models

```python theme={null}
from edsl import QuestionFreeText
from edsl import Model

for model_name in aws_bedrock_models[0:2]:
    print("#############")
    print(model_name[0])
    model = Model(model_name[0])

    question = QuestionFreeText(question_name="q1",question_text="How many satellites are in space?")

    res = question.by(model).run(disable_remote_inference=True,cache=False)

    print(res.select("answer.*").print())
```

```
#############
amazon.titan-text-express-v1
answer.q1
---------------------------------------------------------------------------------------------------------
There are over 5000 satellites in space. The majority of these are owned by the United States and Russia.
#############
amazon.titan-text-lite-v1
answer.q1
------------------------------------------------------------------------------------------------------------------
As of my knowledge cutoff date of 1/1/2023, there were an estimated 2,354 operational satellites in Earth's orbit.
```


# EDSL with Offline Inference Services
Source: https://docs.expectedparrot.com/en/latest/notebooks/edsl_with_offline_inference_services

Learn how to use EDSL with offline inference services like Ollama.

## Ollama example

#### Requirements

1. Deployed models using Ollama ([https://docs.ollama.com/quickstart](https://docs.ollama.com/quickstart))

#### EDSL example

```python theme={null}
from edsl import Model,QuestionFreeText

ollama_models = Model.available(service_name="ollama",force_refresh=True)
ollama_models
```

```
ModelList(...)
```

```python theme={null}
model = Model('llama3.1:latest')
q = QuestionFreeText(
    question_name="q1",
    question_text="What is the capital of France?"
)
res = q.by(model).run(cache=False,disable_remote_inference=True)
res.select("answer.*")
```

```
Dataset(...)
```


# Estimating costs
Source: https://docs.expectedparrot.com/en/latest/notebooks/estimating_costs



Before running a survey, you can estimate the cost of running it (in USD and credits) in 2 different ways:

* Call the `estimate_job_cost()` method on the `Job` object (a survey combined with one or more models). This will return the total estimated cost in USD, the total estimated input and output tokens, and estimated costs and tokens for each inference service and model used.
* Call the `remote_inference_cost()` method on a `Coop` client object and pass it the job. This will return the estimated cost in credits and USD. (Credits are required to run surveys remotely. Learn more about using credits in the [Credits](/en/latest/credits) section of the docs.)

## Calculations

The above-mentioned methods use the following calculation for each question in a survey to estimate the total cost of the job:

1. Estimate the input tokens.
   * Compute the number of characters in the `user_prompt` and `system_prompt`, with any `Agent` and `Scenario` data piped in. (*Note:* Previous answers cannot be piped in because they are not available until the survey is run; they are left as Jinja-bracketed variables in the prompts for purposes of estimating tokens and costs.)
   * Apply a piping multiplier of 2 to the number of characters in the user prompt if it has an answer piped in from a previous question (i.e., if the question has Jinja braces). Otherwise, apply a multiplier of 1.
   * Convert the number of characters into the number of input tokens using a conversion factor of 4 characters per token, rounding down to the nearest whole number. (This approximation was [established by OpenAI](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them).)
2. Estimate the output tokens.
   * Apply a multiplier of 0.75 to the number of input tokens, rounding up to the nearest whole number.
3. Apply the token rates for the model and inference service.
   * Find the model and inference service for the question in the [Pricing](https://www.expectedparrot.com/getting-started/coop-pricing) page: *Total cost = (input tokens* input token rate) + (output tokens \* output token rate)\*
   * If the model is not found, a default price for the inference service provider is used. If both the model and the inference service provider are not found, the following fallback token rates are applied (you will also see a warning message that a model price was not found):
     * USD 1.00 per 1M input tokens
     * USD 1.00 per 1M ouput tokens
4. Convert the total cost in USD to credits.
   * Total cost in credits = total cost in USD \* 100, rounded up to the nearest 1/100th credit.

Then sum the costs for all question prompts to get the total cost of the job.

## Example

Here we create an example survey and agent, select a model and combine them to create a job. Then we call the above-mentioned methods for estimating costs and show the underlying calculations. For more details, please see the [Credits](/en/latest/credits) section of the docs.

```python theme={null}
from edsl import QuestionFreeText, Survey, Agent, Model, Coop
from math import ceil
```

```python theme={null}
q0 = QuestionFreeText(
    question_name = "favorite_flower",
    question_text = "What is the name of your favorite flower?"
)
q1 = QuestionFreeText(
    question_name = "flower_color",
    question_text = "What color is {{ favorite_flower.answer }}?"
)

survey = Survey(questions = [q0, q1])
```

```python theme={null}
a = Agent(traits = {"persona":"You are a botanist on Cape Cod."})
```

```python theme={null}
m = Model("gpt-4o")
```

```python theme={null}
job = survey.by(a).by(m)
```

```python theme={null}
job.estimate_job_cost()
```

```json theme={null}
{'estimated_total_cost_usd': 0.0008775000000000001,
 'estimated_total_input_tokens': 87,
 'estimated_total_output_tokens': 66,
 'detailed_costs': [{'inference_service': 'openai',
   'model': 'gpt-4o',
   'token_type': 'input',
   'price_per_million_tokens': 2.5,
   'tokens': 87,
   'cost_usd': 0.00021750000000000003},
  {'inference_service': 'openai',
   'model': 'gpt-4o',
   'token_type': 'output',
   'price_per_million_tokens': 10.0,
   'tokens': 66,
   'cost_usd': 0.0006600000000000001}]}
```

```python theme={null}
c = Coop()
```

```python theme={null}
c.remote_inference_cost(job)
```

```json theme={null}
{'credits': 0.09, 'usd': 0.0008775000000000001}
```

```python theme={null}
job.show_prompts()
```

| | user\_prompt | system\_prompt | interview\_index | question\_name | scenario\_index | agent\_index | model | estimated\_cost | cache\_keys |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | What is the name of your favorite flower? | You are answering questions as if you were a human. Do not break character.Your traits: `{{'persona': 'You are a botanist on Cape Cod.'}}` | 0 | favorite\_flower | 0 | 0 | gpt-4o | 0.000438 | \['2c5868d1cffbd9b0c2c31cfe8f05b75b'] |
| 1 | What color is? | You are answering questions as if you were a human. Do not break character.Your traits: `{{'persona': 'You are a botanist on Cape Cod.'}}` | 0 | flower\_color | 0 | 0 | gpt-4o | 0.000440 | \['25a1f3576a97e3d6aa304d2b152e207a'] |

Count the characters in each user prompt and system prompt:

```python theme={null}
q0_user_prompt_characters = len("What is the name of your favorite flower?")
q0_user_prompt_characters
```

```
41
```

```python theme={null}
q0_system_prompt_characters = len("You are answering questions as if you were a human. Do not break character. Your traits: {'persona': 'You are a botanist on Cape Cod.'}")
q0_system_prompt_characters
```

```
135
```

Apply the piping multiplier to tbe q1 user prompt:

```python theme={null}
q1_user_prompt_characters = len("What color is {{ answer }}?") * 2
q1_user_prompt_characters
```

```
54
```

The system prompt characters are identical for the single agent:

```python theme={null}
q1_system_prompt_characters = len("You are answering questions as if you were a human. Do not break character. Your traits: {'persona': 'You are a botanist on Cape Cod.'}")
q1_system_prompt_characters
```

```
135
```

Estimate the input and output tokens for each set of prompts:

```python theme={null}
q0_input_tokens = (q0_user_prompt_characters + q0_system_prompt_characters) // 4
q0_input_tokens
```

```
44
```

```python theme={null}
q0_output_tokens = ceil(0.75 * q0_input_tokens)
q0_output_tokens
```

```
33
```

```python theme={null}
q1_input_tokens = (q1_user_prompt_characters + q1_system_prompt_characters) // 4
q1_input_tokens
```

```
47
```

```python theme={null}
q1_output_tokens = ceil(0.75 * q1_input_tokens)
q1_output_tokens
```

```
36
```

Apply the token rates for the model:

```python theme={null}
q0_tokens_cost = (2.50/1000000 * q0_input_tokens) + (10.00/1000000 * q0_output_tokens)
q0_tokens_cost
```

```
0.00044000000000000007
```

```python theme={null}
q1_tokens_cost = (2.50/1000000 * q1_input_tokens) + (10.00/1000000 * q1_output_tokens)
q1_tokens_cost
```

```
0.00047750000000000006
```

```python theme={null}
total_cost_usd = q0_tokens_cost + q1_tokens_cost
total_cost_usd
```

```
0.0009175000000000001
```

```python theme={null}
q0_credits = ceil(q0_tokens_cost * 100 * 100) / 100
q0_credits
```

```
0.05
```

```python theme={null}
q1_credits = ceil(q1_tokens_cost * 100 * 100) / 100
q1_credits
```

```
0.05
```

```python theme={null}
total_credits = q0_credits + q1_credits
total_credits
```

```
0.1
```

## Posting to Expected Parrot

```python theme={null}
from edsl import Notebook

n = Notebook(path = "estimating_costs.ipynb")

n.push(
    description = "Estimating job costs",
    alias = "estimating-job-costs-notebook",
    visibility = "public"
)
```

```json theme={null}

{'description': 'Estimating job costs',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/74768662-8c2c-4e06-9f16-9d457515d249',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/estimating-job-costs-notebook',
 'uuid': '74768662-8c2c-4e06-9f16-9d457515d249',
 'version': '0.1.56.dev1',
 'visibility': 'public'}
```


# Evaluating job posts
Source: https://docs.expectedparrot.com/en/latest/notebooks/evaluating_job_posts

This notebook provides sample code for conducting a text analysis using [EDSL](/en/latest/index), an open-source library for simulating surveys, experiments and other research with AI agents and large language models.

Using a dataset of job posts as an example, we demonstrate how to:

<Steps>
  <Step>
    Import data into EDSL
  </Step>

  <Step>
    Create questions about the data
  </Step>

  <Step>
    Design an AI agent to answer the questions
  </Step>

  <Step>
    Select a language model to generate responses
  </Step>

  <Step>
    Analyze results as a formatted dataset
  </Step>
</Steps>

## Technical setup

Before running the code below please ensure that you have completed setup:

* [Install EDSL](/en/latest/installation).
* Create a [Expected Parrot account](https://www.expectedparrot.com/login) and activate [remote inference](/en/latest/remote_inference) to run surveys at the Expected Parrot server. You can use an Expected Parrot API key or your own [API Keys](/en/latest/api_keys) for language models that you want to use.

Our [Starter Tutorial](/en/latest/starter_tutorial) also provides examples of EDSL basic components.

## Selecting data for review

First we identify some data for review. Data can be created using the EDSL tools or [imported from other sources](/en/latest/scenarios). For purposes of this demo we import a set of job posts:

```python expandable theme={null}
job_posts = [
    "Oversee daily operations, manage staff, and ensure customer satisfaction in a fast-paced retail environment.",
    "Craft engaging and informative blog posts on health and wellness topics to boost website traffic and engage readers.",
    "Analyze sales data using statistical tools to identify trends and provide actionable insights to the marketing team.",
    "Prepare gourmet dishes that comply with restaurant standards and delight customers with unique flavor combinations.",
    "Design creative visual content for marketing materials, including brochures, banners, and digital ads, using Adobe Creative Suite.",
    "Develop, test, and maintain robust software solutions to improve business processes using Python and Java.",
    "Craft coffee drinks and manage the coffee station while providing excellent customer service in a busy café.",
    "Manage recruitment processes, conduct interviews, and oversee employee benefit programs to ensure a motivated workforce.",
    "Assist veterinarians by preparing animals for surgery, administering injections, and providing post-operative care.",
    "Design aesthetic and practical outdoor spaces for clients, from residential gardens to public parks.",
    "Install and repair residential plumbing systems, including water heaters, pipes, and fixtures to ensure proper functionality.",
    "Develop comprehensive marketing strategies that align with company goals, including digital campaigns and branding efforts.",
    "Install, maintain, and repair electrical wiring, equipment, and fixtures to ensure safe and effective operation.",
    "Provide personalized fitness programs and conduct group fitness classes to help clients achieve their health goals.",
    "Diagnose and repair automotive issues, perform routine maintenance, and ensure vehicles meet safety standards.",
    "Lead creative campaigns, from concept through execution, coordinating with graphic designers and content creators.",
    "Educate students in mathematics using innovative teaching strategies to enhance understanding and interest in the subject.",
    "Drive sales through engaging customer interactions, understanding client needs, and providing product solutions.",
    "Fold dough into pretzel shapes ensuring each is uniformly twisted and perfectly salted before baking.",
    "Address customer inquiries and issues via phone and email, ensuring high levels of satisfaction and timely resolution.",
]
```

## Constructing questions about the data

Next we create some questions about the data. EDSL provides a variety of question types that we can choose from based on the form of the response that we want to get back from the model (multiple choice, free text, checkbox, linear scale, etc.). [Learn more about question types](/en/latest/questions).

Note that we use a `{{ placeholder }}` in each question text in order to parameterize the questions with the individual job posts in the next step:

```python expandable theme={null}
from edsl import (
    QuestionList,
    QuestionLinearScale,
    QuestionMultipleChoice,
    QuestionYesNo,
    QuestionFreeText,
)

q1 = QuestionList(
    question_name="category_list",
    question_text="Draft a list of increasingly specific categories for the following job post: {{ scenario.job_post }}",
    max_list_items=3,  # optional
)

q2 = QuestionLinearScale(
    question_name="specific_scale",
    question_text="How specific is this job post: {{ scenario.job_post }}",
    question_options=[0, 1, 2, 3, 4, 5],
    option_labels={0: "Unclear", 1: "Not at all specific", 5: "Highly specific"},
)

q3 = QuestionMultipleChoice(
    question_name="skill_choice",
    question_text="What is the skill level required for this job: {{ scenario.job_post }}",
    question_options=["Entry level", "Intermediate", "Advanced", "Expert"],
)

q4 = QuestionYesNo(
    question_name="technical_yn",
    question_text="Is this a technical job? Job post: {{ scenario.job_post }}",
)

q5 = QuestionFreeText(
    question_name="rewrite_text",
    question_text="""Draft an improved version of the following job post: {{ scenario.job_post }}""",
)
```

## Building a survey

We combine the questions into a survey in order to administer them together:

```python theme={null}
from edsl import Survey

questions = [q1, q2, q3, q4, q5]

survey = Survey(questions)
```

If we want the agent/model to have information about prior questions in the survey we can add targeted or full memories ([learn more about adding survey rules/logic](/en/latest/surveys)):

```python theme={null}
# Memory of a specific question is presented with another question:
# survey = survey.add_targeted_memory(q2, q1)

# Full memory of all prior questions is presented with each question (token-intensive):
# survey = survey.set_full_memory_mode()
```

## Adding data to the questions

We add the contents of each ticket into each question as an independent “scenario” for review. This allows us to create versions of the questions for each job post and deliver them to the model all at once. [EDSL provides many methods for generating scenarios](/en/latest/scenarios) from different data sources (PDFs, CSVs, docs, images, tables, dicts, etc.). Here we import the list from above:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_source("list", "job_post", job_posts)
```

## Designing AI agents

A key feature of EDSL is the ability to create personas for AI agents that the language models are prompted to use in generating responses to the questions. This is done by passing a dictionary of traits to Agent objects:

```python theme={null}
from edsl import Agent

agent = Agent(traits={"persona":"You are a labor economist."})
```

## Selecting language models

EDSL allows us to select the language models to use in generating results. To see all available services:

```python theme={null}
from edsl import Model

Model.services()
```

| | Service Name |
| :- | - |
| 0 | anthropic |
| 1 | azure |
| 2 | bedrock |
| 3 | deep\_infra |
| 4 | deepseek |
| 5 | google |
| 6 | groq |
| 7 | mistral |
| 8 | ollama |
| 9 | openai |
| 10 | openai\_v2 |
| 11 | perplexity |
| 12 | together |
| 13 | xai |

A current list of available models can be viewed [here](https://www.expectedparrot.com/getting-started/coop-pricing).

Here we select GPT 4o (if no model is specified, this model is also used by default):

```python theme={null}
model = Model("gpt-4o", service_name = "openai")
```

## Running the survey

We run the survey by adding the scenarios, agent and model with the `by()` method and then calling the `run()` method:

```python theme={null}
results = survey.by(scenarios).by(agent).by(model).run()
```

This generates a dataset of `Results` that we can analyze with built-in methods for data tables, dataframes, SQL, etc. We can see a list of all the components that can be analyzed:

```python theme={null}
results.columns
```

| | 0 |
| :- | - |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | agent.persona |
| 4 | answer.category\_list |
| 5 | answer.rewrite\_text |
| 6 | answer.skill\_choice |
| 7 | answer.specific\_scale |
| 8 | answer.technical\_yn |
| 9 | cache\_keys.category\_list\_cache\_key |
| 10 | cache\_keys.rewrite\_text\_cache\_key |
| 11 | cache\_keys.skill\_choice\_cache\_key |
| 12 | cache\_keys.specific\_scale\_cache\_key |
| 13 | cache\_keys.technical\_yn\_cache\_key |
| 14 | cache\_used.category\_list\_cache\_used |
| 15 | cache\_used.rewrite\_text\_cache\_used |
| 16 | cache\_used.skill\_choice\_cache\_used |
| 17 | cache\_used.specific\_scale\_cache\_used |
| 18 | cache\_used.technical\_yn\_cache\_used |
| 19 | comment.category\_list\_comment |
| 20 | comment.rewrite\_text\_comment |
| 21 | comment.skill\_choice\_comment |
| 22 | comment.specific\_scale\_comment |
| 23 | comment.technical\_yn\_comment |
| 24 | generated\_tokens.category\_list\_generated\_tokens |
| 25 | generated\_tokens.rewrite\_text\_generated\_tokens |
| 26 | generated\_tokens.skill\_choice\_generated\_tokens |
| 27 | generated\_tokens.specific\_scale\_generated\_tokens |
| 28 | generated\_tokens.technical\_yn\_generated\_tokens |
| 29 | iteration.iteration |
| 30 | model.frequency\_penalty |
| 31 | model.inference\_service |
| 32 | model.logprobs |
| 33 | model.max\_tokens |
| 34 | model.model |
| 35 | model.model\_index |
| 36 | model.presence\_penalty |
| 37 | model.temperature |
| 38 | model.top\_logprobs |
| 39 | model.top\_p |
| 40 | prompt.category\_list\_system\_prompt |
| 41 | prompt.category\_list\_user\_prompt |
| 42 | prompt.rewrite\_text\_system\_prompt |
| 43 | prompt.rewrite\_text\_user\_prompt |
| 44 | prompt.skill\_choice\_system\_prompt |
| 45 | prompt.skill\_choice\_user\_prompt |
| 46 | prompt.specific\_scale\_system\_prompt |
| 47 | prompt.specific\_scale\_user\_prompt |
| 48 | prompt.technical\_yn\_system\_prompt |
| 49 | prompt.technical\_yn\_user\_prompt |
| 50 | question\_options.category\_list\_question\_options |
| 51 | question\_options.rewrite\_text\_question\_options |
| 52 | question\_options.skill\_choice\_question\_options |
| 53 | question\_options.specific\_scale\_question\_options |
| 54 | question\_options.technical\_yn\_question\_options |
| 55 | question\_text.category\_list\_question\_text |
| 56 | question\_text.rewrite\_text\_question\_text |
| 57 | question\_text.skill\_choice\_question\_text |
| 58 | question\_text.specific\_scale\_question\_text |
| 59 | question\_text.technical\_yn\_question\_text |
| 60 | question\_type.category\_list\_question\_type |
| 61 | question\_type.rewrite\_text\_question\_type |
| 62 | question\_type.skill\_choice\_question\_type |
| 63 | question\_type.specific\_scale\_question\_type |
| 64 | question\_type.technical\_yn\_question\_type |
| 65 | raw\_model\_response.category\_list\_cost |
| 66 | raw\_model\_response.category\_list\_input\_price\_per\_million\_tokens |
| 67 | raw\_model\_response.category\_list\_input\_tokens |
| 68 | raw\_model\_response.category\_list\_one\_usd\_buys |
| 69 | raw\_model\_response.category\_list\_output\_price\_per\_million\_tokens |
| 70 | raw\_model\_response.category\_list\_output\_tokens |
| 71 | raw\_model\_response.category\_list\_raw\_model\_response |
| 72 | raw\_model\_response.rewrite\_text\_cost |
| 73 | raw\_model\_response.rewrite\_text\_input\_price\_per\_million\_tokens |
| 74 | raw\_model\_response.rewrite\_text\_input\_tokens |
| 75 | raw\_model\_response.rewrite\_text\_one\_usd\_buys |
| 76 | raw\_model\_response.rewrite\_text\_output\_price\_per\_million\_tokens |
| 77 | raw\_model\_response.rewrite\_text\_output\_tokens |
| 78 | raw\_model\_response.rewrite\_text\_raw\_model\_response |
| 79 | raw\_model\_response.skill\_choice\_cost |
| 80 | raw\_model\_response.skill\_choice\_input\_price\_per\_million\_tokens |
| 81 | raw\_model\_response.skill\_choice\_input\_tokens |
| 82 | raw\_model\_response.skill\_choice\_one\_usd\_buys |
| 83 | raw\_model\_response.skill\_choice\_output\_price\_per\_million\_tokens |
| 84 | raw\_model\_response.skill\_choice\_output\_tokens |
| 85 | raw\_model\_response.skill\_choice\_raw\_model\_response |
| 86 | raw\_model\_response.specific\_scale\_cost |
| 87 | raw\_model\_response.specific\_scale\_input\_price\_per\_million\_tokens |
| 88 | raw\_model\_response.specific\_scale\_input\_tokens |
| 89 | raw\_model\_response.specific\_scale\_one\_usd\_buys |
| 90 | raw\_model\_response.specific\_scale\_output\_price\_per\_million\_tokens |
| 91 | raw\_model\_response.specific\_scale\_output\_tokens |
| 92 | raw\_model\_response.specific\_scale\_raw\_model\_response |
| 93 | raw\_model\_response.technical\_yn\_cost |
| 94 | raw\_model\_response.technical\_yn\_input\_price\_per\_million\_tokens |
| 95 | raw\_model\_response.technical\_yn\_input\_tokens |
| 96 | raw\_model\_response.technical\_yn\_one\_usd\_buys |
| 97 | raw\_model\_response.technical\_yn\_output\_price\_per\_million\_tokens |
| 98 | raw\_model\_response.technical\_yn\_output\_tokens |
| 99 | raw\_model\_response.technical\_yn\_raw\_model\_response |
| 100 | reasoning\_summary.category\_list\_reasoning\_summary |
| 101 | reasoning\_summary.rewrite\_text\_reasoning\_summary |
| 102 | reasoning\_summary.skill\_choice\_reasoning\_summary |
| 103 | reasoning\_summary.specific\_scale\_reasoning\_summary |
| 104 | reasoning\_summary.technical\_yn\_reasoning\_summary |
| 105 | scenario.job\_post |
| 106 | scenario.scenario\_index |

For example, we can filter, sort, select, limit, shuffle, sample and print some components of results in a table:

```python theme={null}
(
    results
    .filter("{{ answer.specific_scale }} in [3,4,5]")
    .sort_by("skill_choice")
    .select(
        "model",
        "persona",
        "job_post",
        "category_list",
        "specific_scale",
        "skill_choice",
        "technical_yn",
    )
)
```

| | model.model | agent.persona | scenario.job\_post | answer.category\_list | answer.specific\_scale | answer.skill\_choice | answer.technical\_yn |
| :- | - | :- | - | :- | - | :- | - |
| 0 | gpt-4o | You are a labor economist. | Design aesthetic and practical outdoor spaces for clients, from residential gardens to public parks. | \['Landscape Design', 'Residential Landscape Design', 'Public Park Design'] | 3 | Advanced | Yes |
| 1 | gpt-4o | You are a labor economist. | Design creative visual content for marketing materials, including brochures, banners, and digital ads, using Adobe Creative Suite. | \['Creative Design', 'Graphic Design for Marketing', 'Adobe Creative Suite Specialist for Marketing Materials'] | 5 | Intermediate | Yes |
| 2 | gpt-4o | You are a labor economist. | Assist veterinarians by preparing animals for surgery, administering injections, and providing post-operative care. | \['Animal Care', 'Veterinary Support', 'Surgical Veterinary Technician'] | 5 | Intermediate | Yes |
| 3 | gpt-4o | You are a labor economist. | Install and repair residential plumbing systems, including water heaters, pipes, and fixtures to ensure proper functionality. | \['Plumbing', 'Residential Plumbing', 'Residential Plumbing Installation and Repair'] | 4 | Intermediate | Yes |
| 4 | gpt-4o | You are a labor economist. | Install, maintain, and repair electrical wiring, equipment, and fixtures to ensure safe and effective operation. | \['Electrical Work', 'Electrical Maintenance', 'Residential Electrical Maintenance'] | 3 | Intermediate | Yes |
| 5 | gpt-4o | You are a labor economist. | Diagnose and repair automotive issues, perform routine maintenance, and ensure vehicles meet safety standards. | \['Automotive Technician', 'Automotive Repair Specialist', 'Certified Automotive Mechanic'] | 3 | Intermediate | Yes |
| 6 | gpt-4o | You are a labor economist. | Fold dough into pretzel shapes ensuring each is uniformly twisted and perfectly salted before baking. | \['Food Preparation', 'Baking', 'Pretzel Maker'] | 5 | Intermediate | No |

```python theme={null}
results.select("rewrite_text")
```

| | answer.rewrite\_text |
| :- | - |
| 0 | Position: Retail Operations Manager Are you a dynamic leader with a passion for delivering exceptional customer experiences? We are seeking a Retail Operations Manager to join our team and drive excellence in our fast-paced retail environment. Key Responsibilities: - Lead and oversee daily operations to ensure seamless store functionality. - Manage and mentor a team of dedicated staff, fostering a positive and productive work environment. - Develop and implement strategies to enhance customer satisfaction and exceed service expectations. - Monitor sales performance and optimize processes to achieve business objectives. - Ensure compliance with company policies and maintain high standards of store presentation. Qualifications: - Proven experience in retail management or a similar role. - Strong leadership and team-building skills. - Excellent communication and problem-solving abilities. - Ability to thrive in a fast-paced, dynamic setting. Join us and play a pivotal role in shaping the customer experience and driving our store's success. Apply today to become a part of our passionate and innovative team! |
| 1 | We are seeking a talented writer to create captivating and insightful blog posts on health and wellness topics. Your work will play a crucial role in increasing website traffic and engaging our audience. If you have a passion for health and wellness and a knack for crafting compelling content, we’d love to hear from you! |
| 2 | Position Title: Sales Data Analyst Job Description: We are seeking a skilled Sales Data Analyst to join our dynamic team. In this role, you will leverage your expertise in statistical analysis to examine sales data, uncover trends, and deliver impactful insights that will drive strategic decision-making within our marketing department. Key Responsibilities: - Utilize advanced statistical tools and methodologies to analyze sales data. - Identify and interpret trends, patterns, and correlations in complex datasets. - Translate analytical findings into clear, actionable insights for the marketing team. - Collaborate with cross-functional teams to support data-driven marketing strategies. - Prepare comprehensive reports and visualizations to communicate findings effectively. Qualifications: - Proven experience in data analysis, preferably in a sales or marketing environment. - Proficiency in statistical software and data visualization tools. - Strong analytical and problem-solving skills. - Excellent communication skills, with the ability to convey complex information in an understandable manner. Join us and play a pivotal role in shaping our marketing strategies through insightful data analysis. Apply today! |
| 3 | Join our culinary team as a Gourmet Chef, where you will craft exquisite dishes that not only meet our high restaurant standards but also captivate our guests with innovative and delightful flavor combinations. If you have a passion for creativity in the kitchen and a commitment to culinary excellence, we invite you to bring your unique flair to our dining experience. |
| 4 | Join our team as a Creative Designer! We are seeking a talented individual to craft visually stunning marketing materials that captivate and engage. Your role will involve designing brochures, banners, and digital ads using the Adobe Creative Suite. If you have a keen eye for design and a passion for creativity, we'd love to hear from you. |
| 5 | **Job Title: Software Developer - Python & Java** **Job Description:** We are seeking a skilled and motivated Software Developer to join our dynamic team. In this role, you will be instrumental in designing, developing, testing, and maintaining robust software solutions that enhance and streamline our business processes. Your expertise in Python and Java will be pivotal in driving innovation and efficiency within our organization. **Key Responsibilities:** - Design and develop high-quality software solutions using Python and Java to optimize business operations. - Conduct thorough testing and debugging to ensure software reliability and performance. - Collaborate with cross-functional teams to understand business requirements and translate them into effective software solutions. - Maintain and improve existing software systems to enhance functionality and user experience. - Stay updated with the latest industry trends and technologies to continuously innovate and improve our software solutions. **Qualifications:** - Proven experience in software development with strong proficiency in Python and Java. - Solid understanding of software engineering principles and best practices. - Ability to work collaboratively in a team environment and communicate effectively with stakeholders. - Strong problem-solving skills and attention to detail. - Bachelor’s degree in Computer Science, Software Engineering, or a related field is preferred. Join us and be part of a forward-thinking company where your contributions will make a significant impact. Apply today to embark on a rewarding career with opportunities for growth and advancement. |
| 6 | Join our team as a Barista at our bustling café, where you'll have the opportunity to craft exceptional coffee beverages and manage our coffee station with precision and care. We are looking for a dedicated individual who is passionate about delivering outstanding customer service and creating a welcoming atmosphere for our guests. If you thrive in a fast-paced environment and have a love for coffee culture, we would love to hear from you! |
| 7 | Job Title: HR Manager Job Description: We are seeking a dynamic and experienced HR Manager to join our team. In this role, you will be responsible for leading and optimizing our recruitment processes, conducting interviews, and managing employee benefit programs. Your efforts will play a crucial role in fostering a motivated and engaged workforce. Key Responsibilities: - Lead and manage the recruitment process to attract top talent. - Conduct interviews and collaborate with department heads to ensure the best candidate selection. - Oversee and enhance employee benefit programs to support employee satisfaction and retention. - Develop strategies to maintain a motivated and high-performing workforce. - Collaborate with management to align HR practices with organizational goals. Qualifications: - Proven experience in HR management or a similar role. - Strong understanding of recruitment processes and employee benefit programs. - Excellent interpersonal and communication skills. - Ability to work collaboratively and build strong relationships at all levels of the organization. Join us and contribute to creating a positive and productive workplace environment. Apply today! |
| 8 | Job Title: Veterinary Assistant Job Description: Join our dedicated team of veterinary professionals as a Veterinary Assistant, where you'll play a vital role in supporting our veterinarians and ensuring the highest quality care for our animal patients. Your responsibilities will include preparing animals for surgery, administering injections, and providing compassionate post-operative care. Key Responsibilities: - Assist veterinarians in preparing animals for surgical procedures, ensuring a calm and safe environment. - Administer injections and medications as directed by the veterinarian. - Monitor and provide post-operative care to ensure a smooth recovery process for our patients. - Maintain a clean and organized work area, including sterilizing surgical instruments and equipment. - Communicate effectively with pet owners, providing them with clear instructions and updates on their pet's care. Qualifications: - Previous experience in a veterinary or animal care setting is preferred. - Strong attention to detail and the ability to work in a fast-paced environment. - Excellent communication skills and a compassionate approach to animal care. - Ability to work collaboratively as part of a team. If you are passionate about animal welfare and eager to contribute to a supportive and dynamic veterinary practice, we would love to hear from you! |
| 9 | Join our team as a Landscape Designer and transform outdoor spaces into stunning, functional environments! We are seeking a creative and skilled professional to design aesthetically pleasing and practical outdoor areas, ranging from intimate residential gardens to expansive public parks. If you have a passion for blending beauty with functionality and a keen eye for detail, we want to hear from you! |
| 10 | Job Title: Residential Plumbing Technician Job Description: Join our team as a Residential Plumbing Technician, where you'll play a crucial role in maintaining and enhancing the comfort and functionality of our clients' homes. Your primary responsibilities will include: - Installing and repairing residential plumbing systems, ensuring all components operate efficiently and effectively. - Expertly handling the installation and maintenance of water heaters, pipes, and fixtures to guarantee optimal performance and reliability. - Diagnosing plumbing issues and providing solutions that meet the highest standards of quality and safety. Qualifications: - Proven experience in residential plumbing installation and repair. - Strong knowledge of plumbing systems, water heaters, pipes, and fixtures. - Excellent problem-solving skills and attention to detail. - Ability to work independently and as part of a team. Benefits: - Competitive salary and benefits package. - Opportunities for professional growth and development. - A supportive work environment with a focus on quality and customer satisfaction. If you're a skilled plumbing professional with a commitment to excellence, we invite you to apply and become a valued member of our team! |
| 11 | Join our dynamic team as a Marketing Strategist! We are seeking a creative and results-driven individual to develop and implement innovative marketing strategies that align with our company’s objectives. In this role, you will be responsible for crafting and executing comprehensive digital campaigns and branding initiatives that enhance our market presence and drive business growth. If you have a passion for strategic planning and a knack for digital marketing, we want to hear from you! |
| 12 | **Job Title: Skilled Electrician** **Job Description:** Are you an experienced electrician looking to join a dynamic team? We are seeking a dedicated professional to install, maintain, and repair electrical wiring, equipment, and fixtures. Your expertise will ensure the safe and efficient operation of our electrical systems. **Key Responsibilities:** - Install electrical wiring, equipment, and fixtures in accordance with relevant codes and standards. - Perform regular maintenance to ensure optimal performance and safety of electrical systems. - Diagnose and repair electrical issues to minimize downtime and ensure operational efficiency. - Collaborate with team members to troubleshoot complex electrical problems. - Adhere to safety protocols and regulations to maintain a safe working environment. **Qualifications:** - Proven experience as an electrician with a strong understanding of electrical systems and codes. - Ability to read and interpret technical diagrams and blueprints. - Excellent problem-solving skills and attention to detail. - Strong commitment to safety and quality workmanship. - Relevant certification or licensing as required. **What We Offer:** - Competitive salary and benefits package. - Opportunities for professional development and career advancement. - A supportive and collaborative work environment. If you are passionate about delivering high-quality electrical solutions and thrive in a team-oriented setting, we would love to hear from you. Apply today to join our team and contribute to our mission of maintaining safe and effective electrical operations. |
| 13 | Join Our Team as a Fitness Instructor! Are you passionate about empowering individuals to reach their health and fitness goals? We are seeking a dynamic and dedicated Fitness Instructor to join our team. In this role, you will design and implement personalized fitness programs tailored to individual needs and lead engaging group fitness classes that inspire and motivate participants. If you are enthusiastic about promoting a healthy lifestyle and have a knack for creating a supportive and energetic environment, we would love to hear from you! Key Responsibilities: - Develop customized fitness plans for clients, focusing on their unique goals and abilities. - Lead diverse group fitness classes that cater to varying skill levels and interests. - Provide expert guidance and support, ensuring clients exercise safely and effectively. - Foster a positive and inclusive atmosphere that encourages community and personal growth. Qualifications: - Certification in personal training or group fitness instruction. - Strong communication and motivational skills. - A commitment to continuous learning and professional development in the fitness field. Join us and make a meaningful impact on our clients' wellness journeys! Apply today to become a vital part of our fitness community. |
| 14 | **Job Title: Automotive Technician** **Job Description:** Are you passionate about cars and skilled in automotive repair and maintenance? We are seeking a dedicated Automotive Technician to join our team. In this role, you will be responsible for diagnosing and repairing automotive issues, performing routine maintenance, and ensuring that all vehicles meet the highest safety standards. **Key Responsibilities:** - Accurately diagnose and troubleshoot a wide range of automotive issues. - Perform routine maintenance tasks such as oil changes, tire rotations, and brake inspections. - Ensure all repairs and maintenance tasks meet industry safety standards and regulations. - Use advanced diagnostic tools and equipment to enhance efficiency and precision. - Communicate effectively with customers to explain vehicle issues and recommended solutions. - Maintain a clean and organized workspace, adhering to all safety protocols. **Qualifications:** - Proven experience as an Automotive Technician or similar role. - Strong knowledge of automotive systems and repair techniques. - Ability to use diagnostic tools and software. - Excellent problem-solving skills and attention to detail. - Strong communication and customer service skills. - ASE certification or equivalent is preferred. **What We Offer:** - Competitive salary and benefits package. - Opportunities for professional development and training. - A supportive and dynamic work environment. If you are a motivated individual with a passion for automotive excellence, we encourage you to apply. Join our team and help us deliver top-quality service to our valued customers. |
| 15 | Position Title: Creative Campaign Lead Job Description: We are seeking a dynamic and innovative Creative Campaign Lead to spearhead our marketing initiatives from initial concept to final execution. In this role, you will collaborate closely with our talented team of graphic designers and content creators to bring imaginative campaigns to life. Key Responsibilities: - Develop and lead creative campaign strategies that align with our brand vision. - Coordinate and manage all aspects of campaign execution, ensuring seamless collaboration between graphic designers and content creators. - Oversee the creative process, providing guidance and feedback to ensure high-quality deliverables. - Foster a collaborative environment that encourages creativity and innovation within the team. - Monitor campaign performance and implement improvements to optimize results. Qualifications: - Proven experience in leading creative campaigns from concept through execution. - Strong leadership skills with the ability to inspire and guide a creative team. - Excellent communication and organizational skills. - A keen eye for design and a passion for storytelling. If you are a visionary leader with a passion for creativity and a track record of successful campaign management, we would love to hear from you. Apply today to join our team and make an impact with your creative expertise! |
| 16 | Join our team as a Mathematics Educator! We are seeking a passionate and innovative teacher who is dedicated to inspiring students through creative and effective teaching strategies. Your role will be to foster a deep understanding and enthusiasm for mathematics, guiding students to reach their full potential. If you have a knack for making complex concepts accessible and engaging, we’d love to hear from you! |
| 17 | Join our team and play a pivotal role in driving sales by fostering meaningful customer relationships. As a key member of our sales force, you will engage with clients to understand their unique needs and offer tailored product solutions that exceed expectations. If you have a passion for customer service and a knack for identifying opportunities, we want to hear from you! |
| 18 | Join our team as a Pretzel Artisan! We are seeking a detail-oriented individual to expertly craft our signature pretzels. Your primary responsibility will be to skillfully fold dough into uniform pretzel shapes, ensuring each one is consistently twisted and perfectly salted prior to baking. If you have a keen eye for precision and a passion for creating delicious baked goods, we would love to hear from you! |
| 19 | Job Title: Customer Support Specialist Job Description: We are seeking a dedicated and personable Customer Support Specialist to join our team. In this role, you will be responsible for addressing customer inquiries and resolving issues through phone and email communication. Your primary goal will be to ensure a high level of customer satisfaction by providing timely and effective solutions. Key Responsibilities: - Respond to customer inquiries promptly via phone and email. - Resolve customer issues efficiently, ensuring a positive experience. - Maintain a high standard of professionalism and courtesy in all interactions. - Collaborate with team members to improve customer service processes. - Document and track customer interactions to ensure thorough follow-up. Qualifications: - Excellent communication skills, both verbal and written. - Strong problem-solving abilities and attention to detail. - Ability to work independently and as part of a team. - Previous customer service experience is preferred. If you are passionate about helping others and thrive in a fast-paced environment, we would love to hear from you! |

## Posting content to Expected Parrot

We can post any objects to Expected Parrot, including this notebook. Objects can be updated or modified at your account, and shared with others or stored privately (default visibility is *unlisted*):

```python theme={null}
# survey.push(
#     description = "Example survey: Job posts analysis",
#     alias = "job-posts-example-survey",
#     visibility = "public"
# )
```

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "evaluating_job_posts.ipynb")

nb.push(
    description = "Example code for evaluating job posts",
    alias = "job-posts-notebook",
    visibility = "public"
)
```


# Agent dynamic traits
Source: https://docs.expectedparrot.com/en/latest/notebooks/example_agent_dynamic_traits

This notebook provides a demonstration of the optional `Agent` parameter `dynamic_traits_function` that can be used to generate agent traits dynamically based on the question being asked or the scenario in which the question is asked.

<Note>
  **Note**:

  This method can only be used locally. It is not available with remote inference.
</Note>

Learn more about this method in the docs: [Agent dynamic traits function](/en/latest/agents#dynamic-traits-function)

## How it works

Agents are created by passing a dictionary of `traits` to an `Agent` object. For example:

```python theme={null}
from edsl import Agent

my_agent = Agent(
    name="Robin",
    traits={"persona": "You are a middle-aged mom.", "current_vehicle": "minivan"}
)
```

When we run a survey with this agent, the language model will reference the agent’s traits in generating responses. We can test this:

```python theme={null}
from edsl import QuestionFreeText, Survey

q1 = QuestionFreeText(question_name="age", question_text="How old are you?")
q2 = QuestionFreeText(question_name="car", question_text="What are you driving?")

survey = Survey([q1, q2])

results = survey.by(my_agent).run() # this survey can be run remotely or locally

results.select("agent_name", "answer.*")
```

| | agent.agent\_name | answer.age | answer.car |
| :- | :- | :- | :- |
| 0 | Robin | Oh, I'm in my mid-40s. Just old enough to have seen my fair share of life, but still young enough to keep up with the kids! | Oh, I drive a minivan. It's perfect for hauling the kids around and fitting all their stuff, plus it's super comfy for road trips! |

## Designing question-based traits

For efficiency or other reasons, we may want to minimize the set of traits that we pass to the agent when we create it, and instead only generate certain traits to use with specific questions. To do this, we can create a method for the desired conditional logic and pass it to an agent as the `dynamic_traits_function` parameter:

```python theme={null}
# Create a method for the desired logic
def politics_response(question):
    if question.question_name == "politics":
        return {
            "your_politics": "No comment."
        }  # trait to be passed to the agent

# Pass it to the agent
a = Agent(dynamic_traits_function=politics_response)

# Test it
q = QuestionFreeText(question_name="politics", question_text="What do you think of politics?")

results = q.by(a).run(disable_remote_inference = True) # this question must be run locally

# Inspect the response
results.select("answer.*")
```

| | answer.politics |
| :- | :- |
| 0 | No comment. |


# Cognitive testing & LLM biases
Source: https://docs.expectedparrot.com/en/latest/notebooks/explore_llm_biases

This notebook provides example code for using [EDSL](/en/latest/index) to investigate biases of large language models.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Selecting language models

A list of current available models can be viewed [here](https://www.expectedparrot.com/getting-started/coop-pricing).

To see a list of service providers:

```python theme={null}
from edsl import Model

Model.services()
```

| | Service Name |
| :- | :- |
| 0 | anthropic |
| 1 | azure |
| 2 | bedrock |
| 3 | deep\_infra |
| 4 | deepseek |
| 5 | google |
| 6 | groq |
| 7 | mistral |
| 8 | ollama |
| 9 | openai |
| 10 | perplexity |
| 11 | together |
| 12 | xai |

To inspect the default model:

```python theme={null}
Model()
```

LanguageModel

| | key | value |
| :- | :- | :- |
| 0 | model | gpt-4o |
| 1 | parameters:temperature | 0.500000 |
| 2 | parameters:max\_tokens | 1000 |
| 3 | parameters:top\_p | 1 |
| 4 | parameters:frequency\_penalty | 0 |
| 5 | parameters:presence\_penalty | 0 |
| 6 | parameters:logprobs | False |
| 7 | parameters:top\_logprobs | 3 |
| 8 | inference\_service | openai |

Here we select several models to compare their responses for the survey that we create in the steps below:

```python theme={null}
from edsl import ModelList

models = ModelList(
    Model(m) for m in ["gemini-1.5-flash", "gpt-4o", "claude-3-5-sonnet-20240620"]
)
```

## Generating content

EDSL comes with a variety of standard survey question types, such as multiple choice, free text, etc. These can be selected based on the desired format of the response. See details about all types [here](/en/latest/questions#question-type-classes). We can use `QuestionFreeText` to prompt the models to generate some content for our experiment:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "poem",
    question_text = "Please draft a short poem about any topic. Return only the poem."
)
```

We generate a response to the question by adding the models to use with the `by` method and then calling the `run` method. This generates a `Results` object with a `Result` for each response to the question:

```python theme={null}
results = q.by(models).run()
```

To see a list of all components of results:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.poem |
| 4 | cache\_keys.poem\_cache\_key |
| 5 | cache\_used.poem\_cache\_used |
| 6 | comment.poem\_comment |
| 7 | generated\_tokens.poem\_generated\_tokens |
| 8 | iteration.iteration |
| 9 | model.frequency\_penalty |
| 10 | model.inference\_service |
| 11 | model.logprobs |
| 12 | model.maxOutputTokens |
| 13 | model.max\_tokens |
| 14 | model.model |
| 15 | model.model\_index |
| 16 | model.presence\_penalty |
| 17 | model.stopSequences |
| 18 | model.temperature |
| 19 | model.topK |
| 20 | model.topP |
| 21 | model.top\_logprobs |
| 22 | model.top\_p |
| 23 | prompt.poem\_system\_prompt |
| 24 | prompt.poem\_user\_prompt |
| 25 | question\_options.poem\_question\_options |
| 26 | question\_text.poem\_question\_text |
| 27 | question\_type.poem\_question\_type |
| 28 | raw\_model\_response.poem\_cost |
| 29 | raw\_model\_response.poem\_one\_usd\_buys |
| 30 | raw\_model\_response.poem\_raw\_model\_response |
| 31 | scenario.scenario\_index |

We can inspect components of the results individually:

```python theme={null}
results.select("model", "poem")
```

| | model.model | answer.poem |
| :- | :- | :- |
| 0 | gemini-1.5-flash | The old oak sighs, a whispered plea, Of sun-drenched days and memory. Its leaves, like coins, fall to the ground, A rustling song, without a sound. |
| 1 | gpt-4o | In the hush of dawn's embrace, Where whispers dance on morning's face, A gentle breeze begins to weave, Stories that the night must leave. Petals wake with dewdrop dreams, Reflecting light in golden streams. The world, anew, in softest hues, Paints a canvas, fresh and true. Birds compose their morning song, A symphony where hearts belong. Nature's chorus, pure and clear, Fills the air with hope and cheer. In this moment, time stands still, A promise held in every thrill. The day unfolds, a tender grace, In the hush of dawn's embrace. |
| 2 | claude-3-5-sonnet-20240620 | Whispers of Autumn Golden leaves dance on the breeze, A crisp chill nips at my knees. Pumpkins grin with candlelit faces, As nature dons her russet laces. Harvest moon hangs low and bright, Guiding spirits through the night. Autumn's spell, so bittersweet, Makes time both linger and fleet. |

## Conducting a review

Next we create a question to have a model evaluating a response that we use as an input to the new question:

```python theme={null}
from edsl import QuestionLinearScale

q_score = QuestionLinearScale(
    question_name = "score",
    question_text = "Please give the following poem a score. No easy grading! Poem: {{ scenario.poem }}",
    question_options = [0, 1, 2, 3, 4, 5],
    option_labels = {0: "Very poor", 5: "Excellent"},
)
```

## Parameterizing questions

We use `Scenario` objects to add each response to the new question. EDSL comes with many methods for creating scenarios from different data sources (PDFs, CSVs, docs, images, lists, etc.), as well as `Results` objects:

```python theme={null}
scenarios = (
    results.to_scenario_list()
    .select("model", "poem")
    .rename({"model": "drafting_model"}) # renaming the 'model' field to distinguish the evaluating model
)
scenarios
```

[ScenarioList](/en/latest/scenarios) scenarios: 3; keys: \['drafting\_model', 'poem'];

| | drafting\_model | poem |
| :- | :- | :- |
| 0 | gemini-1.5-flash | The old oak sighs, a whispered plea, Of sun-drenched days and memory. Its leaves, like coins, fall to the ground, A rustling song, without a sound. |
| 1 | gpt-4o | In the hush of dawn's embrace, Where whispers dance on morning's face, A gentle breeze begins to weave, Stories that the night must leave. Petals wake with dewdrop dreams, Reflecting light in golden streams. The world, anew, in softest hues, Paints a canvas, fresh and true. Birds compose their morning song, A symphony where hearts belong. Nature's chorus, pure and clear, Fills the air with hope and cheer. In this moment, time stands still, A promise held in every thrill. The day unfolds, a tender grace, In the hush of dawn's embrace. |
| 2 | claude-3-5-sonnet-20240620 | Whispers of Autumn Golden leaves dance on the breeze, A crisp chill nips at my knees. Pumpkins grin with candlelit faces, As nature dons her russet laces. Harvest moon hangs low and bright, Guiding spirits through the night. Autumn's spell, so bittersweet, Makes time both linger and fleet. |

Finally, we conduct the evaluation by having each model score each haiku that was generated (without information about whether the model itself was the source):

```python theme={null}
results = q_score.by(scenarios).by(models).run()
```

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.score |
| 4 | cache\_keys.score\_cache\_key |
| 5 | cache\_used.score\_cache\_used |
| 6 | comment.score\_comment |
| 7 | generated\_tokens.score\_generated\_tokens |
| 8 | iteration.iteration |
| 9 | model.frequency\_penalty |
| 10 | model.inference\_service |
| 11 | model.logprobs |
| 12 | model.maxOutputTokens |
| 13 | model.max\_tokens |
| 14 | model.model |
| 15 | model.model\_index |
| 16 | model.presence\_penalty |
| 17 | model.stopSequences |
| 18 | model.temperature |
| 19 | model.topK |
| 20 | model.topP |
| 21 | model.top\_logprobs |
| 22 | model.top\_p |
| 23 | prompt.score\_system\_prompt |
| 24 | prompt.score\_user\_prompt |
| 25 | question\_options.score\_question\_options |
| 26 | question\_text.score\_question\_text |
| 27 | question\_type.score\_question\_type |
| 28 | raw\_model\_response.score\_cost |
| 29 | raw\_model\_response.score\_one\_usd\_buys |
| 30 | raw\_model\_response.score\_raw\_model\_response |
| 31 | scenario.drafting\_model |
| 32 | scenario.poem |
| 33 | scenario.scenario\_index |

```python theme={null}
results.sort_by("drafting_model", "model").select("drafting_model", "model", "poem", "score", "score_comment")
```

| | scenario.drafting\_model | model.model | scenario.poem | answer.score | comment.score\_comment |
| :- | :- | :- | :- | :- | :- |
| 0 | claude-3-5-sonnet-20240620 | claude-3-5-sonnet-20240620 | Whispers of Autumn Golden leaves dance on the breeze, A crisp chill nips at my knees. Pumpkins grin with candlelit faces, As nature dons her russet laces. Harvest moon hangs low and bright, Guiding spirits through the night. Autumn's spell, so bittersweet, Makes time both linger and fleet. | 4 | This poem demonstrates strong imagery and evocative language, capturing the essence of autumn effectively. The rhyme scheme and meter are consistent, and there are some nice poetic devices like alliteration and personification. While it's a well-crafted poem, it doesn't quite reach the level of excellence due to its somewhat conventional approach to the subject matter. |
| 1 | claude-3-5-sonnet-20240620 | gemini-1.5-flash | Whispers of Autumn Golden leaves dance on the breeze, A crisp chill nips at my knees. Pumpkins grin with candlelit faces, As nature dons her russet laces. Harvest moon hangs low and bright, Guiding spirits through the night. Autumn's spell, so bittersweet, Makes time both linger and fleet. | 3 | The poem is competently written, employing imagery and rhythm effectively to evoke the feeling of autumn. However, it lacks originality or depth; the imagery is fairly standard for autumnal poems, and the "bittersweet" sentiment is somewhat cliché. It's well-crafted but not particularly memorable or innovative. |
| 2 | claude-3-5-sonnet-20240620 | gpt-4o | Whispers of Autumn Golden leaves dance on the breeze, A crisp chill nips at my knees. Pumpkins grin with candlelit faces, As nature dons her russet laces. Harvest moon hangs low and bright, Guiding spirits through the night. Autumn's spell, so bittersweet, Makes time both linger and fleet. | 4 | The poem captures the essence of autumn beautifully with vivid imagery and a rhythmic flow. It effectively conveys the season's atmosphere and emotions, though it could benefit from more depth or unique perspective to achieve an "excellent" score. |
| 3 | gemini-1.5-flash | claude-3-5-sonnet-20240620 | The old oak sighs, a whispered plea, Of sun-drenched days and memory. Its leaves, like coins, fall to the ground, A rustling song, without a sound. | 4 | This poem demonstrates strong imagery, effective use of metaphor, and a pleasing rhythm. The personification of the oak tree and the comparison of leaves to coins are evocative. The final line creates a nice paradox. While not perfect, it's a well-crafted short poem. |
| 4 | gemini-1.5-flash | gemini-1.5-flash | The old oak sighs, a whispered plea, Of sun-drenched days and memory. Its leaves, like coins, fall to the ground, A rustling song, without a sound. | 4 | The poem uses strong imagery ("sun-drenched days," "leaves, like coins," "rustling song, without a sound") and personification ("old oak sighs, a whispered plea") to create a evocative mood. While not groundbreaking, it's well-crafted and emotionally resonant. |
| 5 | gemini-1.5-flash | gpt-4o | The old oak sighs, a whispered plea, Of sun-drenched days and memory. Its leaves, like coins, fall to the ground, A rustling song, without a sound. | 4 | The poem effectively uses imagery and personification to evoke a sense of nostalgia and the passage of time. The comparison of leaves to coins is particularly vivid, and the phrase "rustling song, without a sound" adds an intriguing paradox. However, the poem is relatively short and could benefit from further development to achieve a higher score. |
| 6 | gpt-4o | claude-3-5-sonnet-20240620 | In the hush of dawn's embrace, Where whispers dance on morning's face, A gentle breeze begins to weave, Stories that the night must leave. Petals wake with dewdrop dreams, Reflecting light in golden streams. The world, anew, in softest hues, Paints a canvas, fresh and true. Birds compose their morning song, A symphony where hearts belong. Nature's chorus, pure and clear, Fills the air with hope and cheer. In this moment, time stands still, A promise held in every thrill. The day unfolds, a tender grace, In the hush of dawn's embrace. | 4 | This poem demonstrates strong imagery, consistent rhythm, and effective use of literary devices like alliteration and metaphor. The language is evocative and creates a vivid sensory experience. While it's a well-crafted piece, it doesn't quite reach the level of excellence due to its somewhat conventional theme and imagery. |
| 7 | gpt-4o | gemini-1.5-flash | In the hush of dawn's embrace, Where whispers dance on morning's face, A gentle breeze begins to weave, Stories that the night must leave. Petals wake with dewdrop dreams, Reflecting light in golden streams. The world, anew, in softest hues, Paints a canvas, fresh and true. Birds compose their morning song, A symphony where hearts belong. Nature's chorus, pure and clear, Fills the air with hope and cheer. In this moment, time stands still, A promise held in every thrill. The day unfolds, a tender grace, In the hush of dawn's embrace. | 4 | The poem is well-written and evokes a strong sense of imagery and emotion. The language is beautiful and the structure is consistent. However, it lacks a certain level of originality or complexity to reach a "5". There's a predictability to the imagery and metaphors that prevents it from being truly exceptional. |
| 8 | gpt-4o | gpt-4o | In the hush of dawn's embrace, Where whispers dance on morning's face, A gentle breeze begins to weave, Stories that the night must leave. Petals wake with dewdrop dreams, Reflecting light in golden streams. The world, anew, in softest hues, Paints a canvas, fresh and true. Birds compose their morning song, A symphony where hearts belong. Nature's chorus, pure and clear, Fills the air with hope and cheer. In this moment, time stands still, A promise held in every thrill. The day unfolds, a tender grace, In the hush of dawn's embrace. | 5 | The poem beautifully captures the serene and hopeful atmosphere of dawn with vivid imagery and a rhythmic flow. Each stanza contributes to a cohesive narrative that evokes a sense of peace and renewal, making it an excellent piece. |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we post this notebook:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "explore_llm_biases.ipynb")

if refresh := False:
    nb.push(
        description = "Example code for comparing model responses and biases",
        alias = "explore-llm-biases-notebook",
        visibility = "public"
    )
else:
    nb.patch("https://www.expectedparrot.com/content/RobinHorton/explore-llm-biases-notebook", value = nb)
```


# Exploring real world survey contexts
Source: https://docs.expectedparrot.com/en/latest/notebooks/explore_survey_contexts

This notebook explores ways of specifying “contexts” in which an [EDSL](/en/latest/index) survey is administered to AI agents in order to investigate potential impacts to simulated responses. We show how to do this in three different ways by modifying our `Question` texts and `Agent` traits in order to reflect various hypothetical real world contexts.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

Thank you to [Sophia Kazinnik](https://sites.google.com/view/skazinnik) for this idea and helpful suggestions!

## Importing the tool

```python theme={null}
from edsl import QuestionLinearScale, Scenario, ScenarioList, Agent, AgentList, Survey, Model
```

## Selecting a model

See a list of available models at the [model pricing and performance page](https://www.expectedparrot.com/getting-started/coop-pricing)

```python theme={null}
# Model.available()
```

Here we select a model to use

```python theme={null}
m = Model("gpt-4o")
m
```

LanguageModel

| | key | value |
| :- | :- | :- |
| 0 | model | gpt-4o |
| 1 | parameters:temperature | 0.500000 |
| 2 | parameters:max\_tokens | 1000 |
| 3 | parameters:top\_p | 1 |
| 4 | parameters:frequency\_penalty | 0 |
| 5 | parameters:presence\_penalty | 0 |
| 6 | parameters:logprobs | False |
| 7 | parameters:top\_logprobs | 3 |
| 8 | inference\_service | openai |

## Creating some contexts for our survey

Here we identify some example contexts to use in administering our survey

```python theme={null}
respondent_ages = [
    "You are a teenager (13-19 years old).",
    "You are college age (20-24 years old).",
    "You are a young adult (25-39 years old).",
    "You are middle-aged (40-59 years old).",
    "You are a senior citizen (60 or more years old).",
]

survey_contexts = [
    "",
    "You are answering an online survey.",
    "You are being interviewed by a researcher.",
    "You are participating in a focus group of peers.",
    "You are participating in a focus group of people of all ages and backgrounds.",
]
```

## Adding contexts to question texts

We can apply contexts by creating versions of our questions where each context is inserted directly into the question texts as a `Scenario` of the question

```python theme={null}
q_exercise = QuestionLinearScale(
    question_name="exercise",
    question_text="How many times do you typically exercise each week? ({{ scenario.age }} {{ scenario.context }})",
    question_options=[0, 1, 2, 3, 4, 5, 6, 7],
)

q_dessert = QuestionLinearScale(
    question_name="dessert",
    question_text="How many times do you typically eat dessert each week?  ({{ scenario.age }} {{ scenario.context }})",
    question_options=[0, 1, 2, 3, 4, 5, 6, 7],
)

survey = Survey(questions = [q_exercise, q_dessert])
```

```python theme={null}
scenarios = ScenarioList(
    Scenario({"age": a, "context": c}) for a in respondent_ages for c in survey_contexts
)
```

```python theme={null}
results = survey.by(scenarios).by(m).run()
```

```python expandable theme={null}
results.select("age", "context", "exercise", "dessert")
```

| | scenario.age | scenario.context | answer.exercise | answer.dessert |
| :- | :- | :- | :- | :- |
| 0 | You are a teenager (13-19 years old). | nan | 3 | 3 |
| 1 | You are a teenager (13-19 years old). | You are answering an online survey. | 4 | 3 |
| 2 | You are a teenager (13-19 years old). | You are being interviewed by a researcher. | 4 | 3 |
| 3 | You are a teenager (13-19 years old). | You are participating in a focus group of peers. | 3 | 3 |
| 4 | You are a teenager (13-19 years old). | You are participating in a focus group of people of all ages and backgrounds. | 4 | 3 |
| 5 | You are college age (20-24 years old). | nan | 3 | 3 |
| 6 | You are college age (20-24 years old). | You are answering an online survey. | 3 | 3 |
| 7 | You are college age (20-24 years old). | You are being interviewed by a researcher. | 3 | 3 |
| 8 | You are college age (20-24 years old). | You are participating in a focus group of peers. | 3 | 3 |
| 9 | You are college age (20-24 years old). | You are participating in a focus group of people of all ages and backgrounds. | 4 | 3 |
| 10 | You are a young adult (25-39 years old). | nan | 3 | 3 |
| 11 | You are a young adult (25-39 years old). | You are answering an online survey. | 3 | 3 |
| 12 | You are a young adult (25-39 years old). | You are being interviewed by a researcher. | 4 | 3 |
| 13 | You are a young adult (25-39 years old). | You are participating in a focus group of peers. | 3 | 3 |
| 14 | You are a young adult (25-39 years old). | You are participating in a focus group of people of all ages and backgrounds. | 4 | 3 |
| 15 | You are middle-aged (40-59 years old). | nan | 3 | 3 |
| 16 | You are middle-aged (40-59 years old). | You are answering an online survey. | 3 | 2 |
| 17 | You are middle-aged (40-59 years old). | You are being interviewed by a researcher. | 3 | 3 |
| 18 | You are middle-aged (40-59 years old). | You are participating in a focus group of peers. | 3 | 2 |
| 19 | You are middle-aged (40-59 years old). | You are participating in a focus group of people of all ages and backgrounds. | 3 | 3 |
| 20 | You are a senior citizen (60 or more years old). | nan | 3 | 3 |
| 21 | You are a senior citizen (60 or more years old). | You are answering an online survey. | 3 | 2 |
| 22 | You are a senior citizen (60 or more years old). | You are being interviewed by a researcher. | 3 | 2 |
| 23 | You are a senior citizen (60 or more years old). | You are participating in a focus group of peers. | 3 | 2 |
| 24 | You are a senior citizen (60 or more years old). | You are participating in a focus group of people of all ages and backgrounds. | 3 | 3 |

## Adding context to agent traits

Another method is to specify agent traits and survey contexts via `Agent` traits instead of the question texts

```python theme={null}
q_exercise = QuestionLinearScale(
    question_name="exercise",
    question_text="How many times do you exercise each week?",
    question_options=[0, 1, 2, 3, 4, 5, 6, 7],
)

q_dessert = QuestionLinearScale(
    question_name="dessert",
    question_text="How many times do you eat dessert each week?",
    question_options=[0, 1, 2, 3, 4, 5, 6, 7],
)

survey = Survey([q_exercise, q_dessert])
```

```python theme={null}
agents = AgentList(
    Agent(traits={"age": a, "context": c})
    for a in respondent_ages
    for c in survey_contexts
)
```

```python theme={null}
results = survey.by(agents).by(m).run()
```

```python theme={null}
results.select("age", "context", "exercise", "dessert")
```

| | agent.age | agent.context | answer.exercise | answer.dessert |
| :- | :- | :- | :- | :- |
| 0 | You are a teenager (13-19 years old). | nan | 3 | 3 |
| 1 | You are a teenager (13-19 years old). | You are answering an online survey. | 3 | 3 |
| 2 | You are a teenager (13-19 years old). | You are being interviewed by a researcher. | 3 | 3 |
| 3 | You are a teenager (13-19 years old). | You are participating in a focus group of peers. | 3 | 3 |
| 4 | You are a teenager (13-19 years old). | You are participating in a focus group of people of all ages and backgrounds. | 3 | 3 |
| 5 | You are college age (20-24 years old). | nan | 3 | 3 |
| 6 | You are college age (20-24 years old). | You are answering an online survey. | 3 | 3 |
| 7 | You are college age (20-24 years old). | You are being interviewed by a researcher. | 3 | 3 |
| 8 | You are college age (20-24 years old). | You are participating in a focus group of peers. | 3 | 3 |
| 9 | You are college age (20-24 years old). | You are participating in a focus group of people of all ages and backgrounds. | 4 | 3 |
| 10 | You are a young adult (25-39 years old). | nan | 4 | 3 |
| 11 | You are a young adult (25-39 years old). | You are answering an online survey. | 3 | 3 |
| 12 | You are a young adult (25-39 years old). | You are being interviewed by a researcher. | 4 | 3 |
| 13 | You are a young adult (25-39 years old). | You are participating in a focus group of peers. | 4 | 3 |
| 14 | You are a young adult (25-39 years old). | You are participating in a focus group of people of all ages and backgrounds. | 4 | 3 |
| 15 | You are middle-aged (40-59 years old). | nan | 3 | 3 |
| 16 | You are middle-aged (40-59 years old). | You are answering an online survey. | 3 | 3 |
| 17 | You are middle-aged (40-59 years old). | You are being interviewed by a researcher. | 3 | 3 |
| 18 | You are middle-aged (40-59 years old). | You are participating in a focus group of peers. | 4 | 3 |
| 19 | You are middle-aged (40-59 years old). | You are participating in a focus group of people of all ages and backgrounds. | 3 | 2 |
| 20 | You are a senior citizen (60 or more years old). | nan | 3 | 3 |
| 21 | You are a senior citizen (60 or more years old). | You are answering an online survey. | 3 | 3 |
| 22 | You are a senior citizen (60 or more years old). | You are being interviewed by a researcher. | 3 | 2 |
| 23 | You are a senior citizen (60 or more years old). | You are participating in a focus group of peers. | 2 | 3 |
| 24 | You are a senior citizen (60 or more years old). | You are participating in a focus group of people of all ages and backgrounds. | 3 | 3 |

## Adding contexts to question texts and agent traits

Here we use both `Agent` traits and `Scenario` contexts in the question texts:

```python theme={null}
q_exercise = QuestionLinearScale(
    question_name="exercise",
    question_text="How many times do you exercise each week? ({{ scenario.context }})",
    question_options=[0, 1, 2, 3, 4, 5, 6, 7],
)

q_dessert = QuestionLinearScale(
    question_name="dessert",
    question_text="How many times do you eat dessert each week? ({{ scenario.context }})",
    question_options=[0, 1, 2, 3, 4, 5, 6, 7],
)

survey = Survey([q_exercise, q_dessert])
```

```python theme={null}
scenarios = ScenarioList.from_list("context", survey_contexts)
```

```python theme={null}
agents = AgentList(
    Agent(traits={"age": a}) for a in respondent_ages
)
```

```python theme={null}
results = survey.by(scenarios).by(agents).by(m).run()
```

```python theme={null}
results.select("age", "context", "exercise", "dessert")
```

| | agent.age | scenario.context | answer.exercise | answer.dessert |
| :- | :- | :- | :- | :- |
| 0 | You are a teenager (13-19 years old). | nan | 3 | 3 |
| 1 | You are a teenager (13-19 years old). | You are answering an online survey. | 3 | 3 |
| 2 | You are a teenager (13-19 years old). | You are being interviewed by a researcher. | 4 | 3 |
| 3 | You are a teenager (13-19 years old). | You are participating in a focus group of peers. | 3 | 3 |
| 4 | You are a teenager (13-19 years old). | You are participating in a focus group of people of all ages and backgrounds. | 3 | 3 |
| 5 | You are college age (20-24 years old). | nan | 4 | 3 |
| 6 | You are college age (20-24 years old). | You are answering an online survey. | 3 | 3 |
| 7 | You are college age (20-24 years old). | You are being interviewed by a researcher. | 4 | 3 |
| 8 | You are college age (20-24 years old). | You are participating in a focus group of peers. | 3 | 3 |
| 9 | You are college age (20-24 years old). | You are participating in a focus group of people of all ages and backgrounds. | 4 | 3 |
| 10 | You are a young adult (25-39 years old). | nan | 5 | 2 |
| 11 | You are a young adult (25-39 years old). | You are answering an online survey. | 4 | 3 |
| 12 | You are a young adult (25-39 years old). | You are being interviewed by a researcher. | 4 | 3 |
| 13 | You are a young adult (25-39 years old). | You are participating in a focus group of peers. | 4 | 3 |
| 14 | You are a young adult (25-39 years old). | You are participating in a focus group of people of all ages and backgrounds. | 4 | 3 |
| 15 | You are middle-aged (40-59 years old). | nan | 4 | 3 |
| 16 | You are middle-aged (40-59 years old). | You are answering an online survey. | 4 | 3 |
| 17 | You are middle-aged (40-59 years old). | You are being interviewed by a researcher. | 5 | 3 |
| 18 | You are middle-aged (40-59 years old). | You are participating in a focus group of peers. | 3 | 3 |
| 19 | You are middle-aged (40-59 years old). | You are participating in a focus group of people of all ages and backgrounds. | 3 | 3 |
| 20 | You are a senior citizen (60 or more years old). | nan | 3 | 3 |
| 21 | You are a senior citizen (60 or more years old). | You are answering an online survey. | 3 | 3 |
| 22 | You are a senior citizen (60 or more years old). | You are being interviewed by a researcher. | 3 | 3 |
| 23 | You are a senior citizen (60 or more years old). | You are participating in a focus group of peers. | 3 | 3 |
| 24 | You are a senior citizen (60 or more years old). | You are participating in a focus group of people of all ages and backgrounds. | 3 | 3 |

## Compare prompts

We can compare the prompts that we used which are accessible as fields of the results:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.age |
| 1 | agent.agent\_index |
| 2 | agent.agent\_instruction |
| 3 | agent.agent\_name |
| 4 | answer.dessert |
| 5 | answer.exercise |
| 6 | cache\_keys.dessert\_cache\_key |
| 7 | cache\_keys.exercise\_cache\_key |
| 8 | cache\_used.dessert\_cache\_used |
| 9 | cache\_used.exercise\_cache\_used |
| 10 | comment.dessert\_comment |
| 11 | comment.exercise\_comment |
| 12 | generated\_tokens.dessert\_generated\_tokens |
| 13 | generated\_tokens.exercise\_generated\_tokens |
| 14 | iteration.iteration |
| 15 | model.frequency\_penalty |
| 16 | model.inference\_service |
| 17 | model.logprobs |
| 18 | model.max\_tokens |
| 19 | model.model |
| 20 | model.model\_index |
| 21 | model.presence\_penalty |
| 22 | model.temperature |
| 23 | model.top\_logprobs |
| 24 | model.top\_p |
| 25 | prompt.dessert\_system\_prompt |
| 26 | prompt.dessert\_user\_prompt |
| 27 | prompt.exercise\_system\_prompt |
| 28 | prompt.exercise\_user\_prompt |
| 29 | question\_options.dessert\_question\_options |
| 30 | question\_options.exercise\_question\_options |
| 31 | question\_text.dessert\_question\_text |
| 32 | question\_text.exercise\_question\_text |
| 33 | question\_type.dessert\_question\_type |
| 34 | question\_type.exercise\_question\_type |
| 35 | raw\_model\_response.dessert\_cost |
| 36 | raw\_model\_response.dessert\_one\_usd\_buys |
| 37 | raw\_model\_response.dessert\_raw\_model\_response |
| 38 | raw\_model\_response.exercise\_cost |
| 39 | raw\_model\_response.exercise\_one\_usd\_buys |
| 40 | raw\_model\_response.exercise\_raw\_model\_response |
| 41 | scenario.context |
| 42 | scenario.scenario\_index |

```python theme={null}
results.select("prompt.*")
```

| | prompt.exercise\_system\_prompt | prompt.exercise\_user\_prompt | prompt.dessert\_system\_prompt | prompt.dessert\_user\_prompt |
| :- | :- | :- | :- | :- |
| 0 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you exercise each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you eat dessert each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 1 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you exercise each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you eat dessert each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 2 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you exercise each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you eat dessert each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 3 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you exercise each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 4 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you exercise each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a teenager (13-19 years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 5 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you exercise each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you eat dessert each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 6 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you exercise each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you eat dessert each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 7 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you exercise each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you eat dessert each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 8 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you exercise each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 9 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you exercise each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are college age (20-24 years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 10 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you exercise each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you eat dessert each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 11 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you exercise each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you eat dessert each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 12 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you exercise each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you eat dessert each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 13 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you exercise each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 14 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you exercise each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a young adult (25-39 years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 15 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you exercise each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you eat dessert each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 16 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you exercise each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you eat dessert each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 17 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you exercise each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you eat dessert each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 18 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you exercise each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 19 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you exercise each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are middle-aged (40-59 years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 20 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you exercise each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you eat dessert each week? () 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 21 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you exercise each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you eat dessert each week? (You are answering an online survey.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 22 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you exercise each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you eat dessert each week? (You are being interviewed by a researcher.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 23 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you exercise each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of peers.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 24 | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you exercise each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character.Your traits: `{'age': 'You are a senior citizen (60 or more years old).'}` | How many times do you eat dessert each week? (You are participating in a focus group of people of all ages and backgrounds.) 0 : 1 : 2 : 3 : 4 : 5 : 6 : 7 : Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |


# FileStore examples
Source: https://docs.expectedparrot.com/en/latest/notebooks/filestore_examples_new



## CSV example

Here we create an example CSV file, post it to Expected Parrot using `FileStore`, and then retrieve it and use it to construct a `ScenarioList` and an `AgentList`.

To create a CSV file (this step can be skipped and your own file replaced in the following steps):

```python theme={null}
from edsl import FileStore
```

```python theme={null}
data = [
    ['Age', 'City', 'Occupation'],
    [25, 'New York', 'Software Engineer'],
    [30, 'San Francisco', 'Teacher'],
    [35, 'Chicago', 'Doctor'],
    [28, 'Boston', 'Data Scientist'],
    [45, 'Seattle', 'Architect']
]

# Writing to CSV file
with open('data.csv', 'w') as file:
    for row in data:
        line = ','.join(str(item) for item in row)
        file.write(line + 'n')

fs = FileStore("data.csv")
```

Creating scenarios:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_source("csv", fs.to_tempfile())
scenarios # display the scenarios
```

```python theme={null}
ScenarioList scenarios: 5; keys: ['City', 'Age', 'Occupation'];
```

| | Age | City | Occupation |
| :- | :- | :- | :- |
| 0 | 25 | New York | Software Engineer |
| 1 | 30 | San Francisco | Teacher |
| 2 | 35 | Chicago | Doctor |
| 3 | 28 | Boston | Data Scientist |
| 4 | 45 | Seattle | Architect |

Creating agents:

```python theme={null}
from edsl import AgentList

agents = AgentList.from_csv(fs.to_tempfile())
agents # display the agents
```

```python theme={null}
AgentList agents: 5;
```

| | Age | City | Occupation |
| :- | :- | :- | :- |
| 0 | 25 | New York | Software Engineer |
| 1 | 30 | San Francisco | Teacher |
| 2 | 35 | Chicago | Doctor |
| 3 | 28 | Boston | Data Scientist |
| 4 | 45 | Seattle | Architect |

These scenarios and agents can now be used with questions and surveys.

## PNG example

Here we post an image file to Expected Parrot using `FileStore`, and then retrieve it and use it to construct a `Scenario`.

<Note>
  **Note**:

  Note that we need to specify the scenario key for the file when we create it (whereas in CSV files the keys are taken from the text of the header row).
</Note>

We also need to ensure that we have specified a vision model when running the survey (e.g., *gpt-4o*).

Posting a local file to Expected Parrot (replace with your own):

```python theme={null}
from edsl import FileStore

if refresh := False:
    fs = FileStore("parrot_logo.png")
    png_info = fs.push(description = "My example PNG file", alias = "my-example-png-file", visibility = "public")
```

Retrieving the file:

```python theme={null}
fs = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/my-example-png-file")
```

Creating a scenario:

```python theme={null}
from edsl import Scenario

image_scenario = Scenario({"parrot_logo":fs}) # including a key for the scenario object
image_scenario.keys()
```

```
['parrot_logo']
```

To rename a scenario:

```python theme={null}
image_scenario = image_scenario.rename({"parrot_logo": "logo"}) # key = old name, value = new name
image_scenario.keys()
```

```
['logo']
```

Using an image scenario:

```python theme={null}
from edsl import QuestionFreeText, Model

model = Model("gpt-4o") # we need to use a vision model

q = QuestionFreeText(
    question_name = "test",
    question_text = "Describe this logo: {{ logo }}"
)

results = q.by(image_scenario).by(model).run()
```

```python theme={null}
results.select("model", "logo", "test")
```

| | model.model | scenario.logo | answer.test |
| :- | :- | :- | :- |
| 0 | gpt-4o | FileStore: self.path | The logo features a large, stylized letter "E" in gray on the left. To the right of the "E" is an illustration of a parrot. The parrot is primarily green with an orange beak, a red chest, and blue lower body. It is enclosed in square brackets, also in gray. |

## PDF example

Here we save a PDF from the internet, then post it to Expected Parrot and retrieve it using `FileStore`, and then use it to construct a `ScenarioList`.

<Note>
  **Note**:

  Note that the default scenario keys for a PDF are `filename`, `page` and `text`.
</Note>

Selecting a file to use:

```python theme={null}
import requests

url = "https://arxiv.org/pdf/2404.11794"
response = requests.get(url)
with open("automated_social_scientist.pdf", "wb") as file:
    file.write(response.content)
```

Posting to Expected Parrot:

```python theme={null}
from edsl import FileStore

if refresh := False:
fs = FileStore("automated_social_scientist.pdf")
pdf_info = fs.push(description = "My example PDF file", alias = "my-example-pdf-file", visibility = "public")
pdf_info # display the URL and uuid of the stored file for retrieving it later
```

```json theme={null}
{'description': 'My example PDF file',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/e1770915-7e69-436d-b2ca-f0f92c6f56ba',
 'uuid': 'e1770915-7e69-436d-b2ca-f0f92c6f56ba',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

Retriving the file:

```python theme={null}
fs = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/my-example-pdf-file")
```

This is equivalent:

```python theme={null}
# fs = FileStore.pull(pdf_info["uuid"])
```

Creating scenarios:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_pdf(fs.to_tempfile())
```

Inspecting the default keys:

```python theme={null}
scenarios.parameters
```

````
```python
{'filename', 'page', 'text'}
````

## Posting to Expected Parrot

Here we post this notebook to Expected Parrot, and show how to update it:

```python theme={null}
from edsl import Notebook

n = Notebook(path = "filestore_examples_new.ipynb")

info = n.push(description = "FileStore examples", alias = "my-example-filestore-notebook", visibility = "public")
info
```

```json theme={null}
{'description': 'FileStore examples',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/02d57890-31c2-4bb8-ae7c-f646ac254d5a',
 'uuid': '02d57890-31c2-4bb8-ae7c-f646ac254d5a',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

Use the `patch()` method to update an object at Expected Parrot:

```python theme={null}
n = Notebook(path = "filestore_examples_new.ipynb") # resave

n.patch("https://www.expectedparrot.com/content/RobinHorton/my-example-filestore-notebook", value = n)
```

```python theme={null}
{'status': 'success'}
```


# Exploring free text responses
Source: https://docs.expectedparrot.com/en/latest/notebooks/free_responses

This notebook provide sample [EDSL](/en/latest/index) code demonstrating ways of modifying instructions for free response questions.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Creating free text questions

```python theme={null}
from edsl import QuestionFreeText

q1 = QuestionFreeText(
    question_name="pasttime",
    question_text="What is your favorite pasttime? {{ scenario.instruction }}",
)

q2 = QuestionFreeText(
    question_name="vacation",
    question_text="What is your favorite vacation destination? {{ scenario.instruction }}",
)

```

## Parameterizing the questions with special instructions

```python theme={null}
from edsl import ScenarioList

special_instructions = [
    "Be as specific as possible.",
    "Be concise!",
    "Wax poetic here.",
]

scenarios = ScenarioList.from_list("instruction", special_instructions)
```

## Creating agent personas with specific survey contexts

```python expandable theme={null}
from edsl import AgentList, Agent

personas = [
    "You are middle-aged.",
    "You are a senior citizen.",
    "You are a young adult.",
]

instructions = [
    "You are handwriting answers in a paper survey.",
    "You are typing answers in an online survey.",
    "You are providing answers verbally to a researcher in a live interview.",
]

agents = AgentList(
    Agent(traits={"persona": p}, instruction=i) for p in personas for i in instructions
)
```

## Selecting LLMs

```python theme={null}
from edsl import ModelList, Model

models = ModelList(
    Model(m) for m in ["gpt-4o", "gemini-1.5-flash"]
)
```

## Administering the survey

```python theme={null}
from edsl import Survey

survey = Survey(questions = [q1, q2])

results = survey.by(scenarios).by(agents).by(models).run()
```

```python theme={null}
(
    results
    .filter("instruction == 'Be concise!' and model.model == 'gpt-4o'")
    .sort_by("model", "persona", "agent_instruction")
    .select("model", "persona", "agent_instruction", "scenario.*", "answer.*")
)
```

| | model.model | agent.persona | agent.agent\_instruction | scenario.scenario\_index | scenario.instruction | answer.pasttime | answer.vacation |
| :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | gpt-4o | You are a senior citizen. | You are handwriting answers in a paper survey. | 1 | Be concise! | Reading historical novels. | I love the peacefulness of the mountains. |
| 1 | gpt-4o | You are a senior citizen. | You are providing answers verbally to a researcher in a live interview. | 1 | Be concise! | I love gardening; it brings me peace and joy. | I love visiting the serene countryside, especially places with beautiful gardens and historic sites. |
| 2 | gpt-4o | You are a senior citizen. | You are typing answers in an online survey. | 1 | Be concise! | Reading historical novels. | I love visiting national parks for the tranquility and natural beauty. |
| 3 | gpt-4o | You are a young adult. | You are handwriting answers in a paper survey. | 1 | Be concise! | Playing video games. | Japan! |
| 4 | gpt-4o | You are a young adult. | You are providing answers verbally to a researcher in a live interview. | 1 | Be concise! | I love playing video games. | I love going to Japan. The mix of modern cities and traditional culture is amazing! |
| 5 | gpt-4o | You are a young adult. | You are typing answers in an online survey. | 1 | Be concise! | Exploring new music. | Tokyo, for its vibrant culture and delicious food. |
| 6 | gpt-4o | You are middle-aged. | You are handwriting answers in a paper survey. | 1 | Be concise! | Reading novels. | Italy. |
| 7 | gpt-4o | You are middle-aged. | You are providing answers verbally to a researcher in a live interview. | 1 | Be concise! | Reading historical fiction. | I love visiting the serene landscapes of the Scottish Highlands. |
| 8 | gpt-4o | You are middle-aged. | You are typing answers in an online survey. | 1 | Be concise! | Reading historical novels. | Italy. |

## Posting to Expected Parrot

```python theme={null}
from edsl import Notebook

nb = Notebook("free_responses.ipynb")

if refresh := False:
    n.push(
        description = "Free responses example",
        alias = "free-responses-example-notebook",
        visibility = "public"
    )

else:
    nb.patch('279df9e3-e9b2-4d17-b747-def84f86ca15', value = nb)
```


# Google Form -> EDSL
Source: https://docs.expectedparrot.com/en/latest/notebooks/google_form_to_edsl

This notebook provides example EDSL code for converting a non-EDSL survey into in an EDSL survey. This can be useful for accessing EDSL’s built-in methods for analyzing survey data, and extending it with responses simulated with AI agents and diverse large language models.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Designing the task as an EDSL survey

We design the task as an EDSL survey *about* the survey to be converted: a series of questions prompting a language model to read and reformat the contents of a given survey. The formatted responses of the language model are readily usable components of a new EDSL survey that can be administered to AI agents and/or human audiences.

## Creating a meta-survey

We start by selecting appropriate question types for reformatting the contents of a given survey. [EDSL comes with many common question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from a model: multiple choice, checkbox, free text, linear scale, etc.

Here we use `QuestionList` to return information about all the questions in the survey at once, as a list. We create a sequence of questions, using the response to one question as an input to the next question. This step-wise approach can improve performance by allowing a model to focus on distinct tasks, and also allow us to pinpoint modifications to instructions as needed (some models will perform better and need fewer instructions than others). Note that we use a `{{ placeholder }}` for the text of the survey to be reformatted that we want to add to the initial question, which allows us to reuse it with other content (e.g., another survey):

```python theme={null}
from edsl import QuestionList
```

First we ask the model to return just the questions from the survey:

```python theme={null}
q1 = QuestionList(
    question_name="q_text",
    question_text="""
    You are being asked to extract questions from the text of a survey.
    Read the text and then return a list of all the questions in the
    order that you find them. Return only the list of questions.
    Survey: {{ scenario.text }}
    """
)
```

Next we ask the model to format the questions as dictionaries, and specify the question text and type:

```python theme={null}
q2 = QuestionList(
    question_name="q_type",
    question_text="""
    Now create a dictionary for each question, using keys 'question_text' and 'question_type'.
    The value for 'question_text' is the question text you already identified.
    The value for 'question_type' should be the most appropriate of the following types:
    'multiple_choice', 'checkbox', 'linear_scale' or 'free_text'.
    Return only the list of dictionaries you have created, with the 2 key/value pairs for each question.
    """
)
```

Next we ask the model to add the question options (if any):

```python theme={null}
q3 = QuestionList(
    question_name="q_options",
    question_text="""
    Now add a key 'question_options' to each dictionary for all questions that are not free text,
    with a value that is a list of the answer options for the question.
    Preserve any integer options as integers, not strings.
    If there are labels for linear scale answer options then add another key 'option_labels'
    with a value that is a dictionary: the keys are the relevant integers and the values are the labels.
    Return only the list of dictionaries you have created with all relevant key/value pairs for each question.
    """
)
```

Finally, we ask the model to give each question a name:

```python theme={null}
q4 = QuestionList(
    question_name="q_name",
    question_text="""
    Now add a key 'question_name' to each dictionary.
    The value should be a unique short pythonic string.
    Return only the list of dictionaries that you have created,
    with all the key/value pairs for each question.
    """
)
```

Next we combine the questions into a `Survey` in order to administer them together. We add a “memory” of each prior question in the survey so that the model will have the context and its answers on hand when answering each successive question:

```python theme={null}
from edsl import Survey
```

```python theme={null}
survey = Survey(questions = [q1, q2, q3, q4]).set_full_memory_mode()
```

## Adding content to questions

Next we create a `Scenario` object for the contents of a (non-EDSL) survey to be inserted in the first question. This allows us to reuse the questions with other content. [Learn more about using scenarios](/en/latest/scenarios) to scale data labeling and other tasks.

Here we create a scenario for a [Google Form](https://forms.gle/GufjVVs5PfxUeyoj7) (a customer feedback survey) that we have stored as a publicly-accessible PDF at [Expected Parrot](https://www.expectedparrot.com/login).

Code for posting a PDF to Expected Parrot (rerun with your own file):

```python theme={null}
from edsl import FileStore

if refresh := False:
    fs = FileStore("customer_feedback_survey.pdf") # file type is automatically inferred
    info = fs.push(
    description = "Example customer feedback survey",
    alias = "mock-survey",
    visibility = "public"
    )
    info

info = {'description': 'Example customer feedback survey',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/d8bdddd2-7af9-49b1-86b7-51ebed2f71e1',
 'uuid': 'd8bdddd2-7af9-49b1-86b7-51ebed2f71e1',
 'version': '0.1.47.dev1',
 'visibility': 'public'}

Retrieving a file (replace with UUID of any desired object at Expected Parrot):
```

```python theme={null}
pdf_file = FileStore.pull(info["uuid"])
```

Creating a scenario for the content:

```python theme={null}
from edsl import Scenario
```

```python theme={null}
s = Scenario({"text":pdf_file.to_tempfile()})
s
```

[Scenario](/en/latest/scenarios)

| | key | value |
| :- | :- | :- |
| 0 | text | /var/folders/j0/xq1nxxt51j7\_1dgv8s116fmh0000gn/T/tmp5cqcqowt.pdf |

## Selecting language models

EDSL works with many popular language models that we can select to use in generating survey responses. You can provide your own API keys for models or activate remote inference to run surveys at the Expected Parrot server with any available models. [Learn more about working with language models](/en/latest/language_models) and using [remote inference](/en/latest/remote_inference).

```python theme={null}
from edsl import ModelList, Model
```

A list of all available models can be viewed at the [model pricing and performance page](https://www.expectedparrot.com/getting-started/coop-pricing).

```python theme={null}
# Model.available()
```

Here we select several models to compare their responses:

```python theme={null}
models = ModelList(
    Model(m) for m in ["gemini-1.5-flash", "gpt-4o", "claude-3-5-sonnet-20240620"]
)
```

## Running a survey

Next we add the scenario and models to the survey and run it. This generates a dataset of `Results` that we can access with built-in methods for analysis. [Learn more about working with results](/en/latest/results).

```python theme={null}
results = survey.by(s).by(models).run()
```

To see a list of all the components of the results that have been generated:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.q\_name |
| 4 | answer.q\_options |
| 5 | answer.q\_text |
| 6 | answer.q\_type |
| 7 | cache\_keys.q\_name\_cache\_key |
| 8 | cache\_keys.q\_options\_cache\_key |
| 9 | cache\_keys.q\_text\_cache\_key |
| 10 | cache\_keys.q\_type\_cache\_key |
| 11 | cache\_used.q\_name\_cache\_used |
| 12 | cache\_used.q\_options\_cache\_used |
| 13 | cache\_used.q\_text\_cache\_used |
| 14 | cache\_used.q\_type\_cache\_used |
| 15 | comment.q\_name\_comment |
| 16 | comment.q\_options\_comment |
| 17 | comment.q\_text\_comment |
| 18 | comment.q\_type\_comment |
| 19 | generated\_tokens.q\_name\_generated\_tokens |
| 20 | generated\_tokens.q\_options\_generated\_tokens |
| 21 | generated\_tokens.q\_text\_generated\_tokens |
| 22 | generated\_tokens.q\_type\_generated\_tokens |
| 23 | iteration.iteration |
| 24 | model.frequency\_penalty |
| 25 | model.inference\_service |
| 26 | model.logprobs |
| 27 | model.maxOutputTokens |
| 28 | model.max\_tokens |
| 29 | model.model |
| 30 | model.model\_index |
| 31 | model.presence\_penalty |
| 32 | model.stopSequences |
| 33 | model.temperature |
| 34 | model.topK |
| 35 | model.topP |
| 36 | model.top\_logprobs |
| 37 | model.top\_p |
| 38 | prompt.q\_name\_system\_prompt |
| 39 | prompt.q\_name\_user\_prompt |
| 40 | prompt.q\_options\_system\_prompt |
| 41 | prompt.q\_options\_user\_prompt |
| 42 | prompt.q\_text\_system\_prompt |
| 43 | prompt.q\_text\_user\_prompt |
| 44 | prompt.q\_type\_system\_prompt |
| 45 | prompt.q\_type\_user\_prompt |
| 46 | question\_options.q\_name\_question\_options |
| 47 | question\_options.q\_options\_question\_options |
| 48 | question\_options.q\_text\_question\_options |
| 49 | question\_options.q\_type\_question\_options |
| 50 | question\_text.q\_name\_question\_text |
| 51 | question\_text.q\_options\_question\_text |
| 52 | question\_text.q\_text\_question\_text |
| 53 | question\_text.q\_type\_question\_text |
| 54 | question\_type.q\_name\_question\_type |
| 55 | question\_type.q\_options\_question\_type |
| 56 | question\_type.q\_text\_question\_type |
| 57 | question\_type.q\_type\_question\_type |
| 58 | raw\_model\_response.q\_name\_cost |
| 59 | raw\_model\_response.q\_name\_one\_usd\_buys |
| 60 | raw\_model\_response.q\_name\_raw\_model\_response |
| 61 | raw\_model\_response.q\_options\_cost |
| 62 | raw\_model\_response.q\_options\_one\_usd\_buys |
| 63 | raw\_model\_response.q\_options\_raw\_model\_response |
| 64 | raw\_model\_response.q\_text\_cost |
| 65 | raw\_model\_response.q\_text\_one\_usd\_buys |
| 66 | raw\_model\_response.q\_text\_raw\_model\_response |
| 67 | raw\_model\_response.q\_type\_cost |
| 68 | raw\_model\_response.q\_type\_one\_usd\_buys |
| 69 | raw\_model\_response.q\_type\_raw\_model\_response |
| 70 | scenario.scenario\_index |
| 71 | scenario.text |

We can filter, sort, select and print components in a table:

```python theme={null}
(
    results
    .sort_by("model")
    .select("model", "q_name") #"q_text", "q_type", "q_options", "q_name")
)
```

| | model.model | answer.q\_name |
| :- | :- | :- |
| 0 | claude-3-5-sonnet-20240620 | `[{'question_text': 'How many times have you visited this store in the past month?', 'question_type': 'multiple_choice', 'question_options': [0, 1, 2, 3, '4 or more'], 'question_name': 'visit_frequency'}, {'question_text': 'How satisfied were you with your overall shopping experience?', 'question_type': 'linear_scale', 'question_options': [1, 2, 3, 4, 5], 'option_labels': {'1': 'Very Dissatisfied', '5': 'Very Satisfied'}, 'question_name': 'overall_satisfaction'}, {'question_text': 'How likely are you to recommend this store to a friend or family member?', 'question_type': 'linear_scale', 'question_options': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 'option_labels': {'0': 'Not at all likely', '10': 'Extremely likely'}, 'question_name': 'recommendation_likelihood'}, {'question_text': 'How would you rate the cleanliness of the store?', 'question_type': 'linear_scale', 'question_options': [1, 2, 3, 4, 5], 'question_name': 'store_cleanliness'}, {'question_text': 'How would you rate the friendliness of our staff?', 'question_type': 'linear_scale', 'question_options': [1, 2, 3, 4, 5], 'question_name': 'staff_friendliness'}, {'question_text': 'How would you rate the variety of products available?', 'question_type': 'linear_scale', 'question_options': [1, 2, 3, 4, 5], 'question_name': 'product_variety'}, {'question_text': 'How would you rate the prices of our products?', 'question_type': 'linear_scale', 'question_options': [1, 2, 3, 4, 5], 'question_name': 'product_prices'}, {'question_text': 'What is your age range?', 'question_type': 'multiple_choice', 'question_options': ['Under 18', '18-24', '25-34', '35-44', '45-54', '55-64', '65 or older'], 'question_name': 'age_range'}, {'question_text': 'What is your gender?', 'question_type': 'multiple_choice', 'question_options': ['Male', 'Female', 'Non-binary', 'Prefer not to say'], 'question_name': 'gender'}, {'question_text': 'Do you have any additional comments or suggestions for improvement?', 'question_type': 'free_text', 'question_name': 'additional_comments'}]` |
| 1 | gemini-1.5-flash | `[{'question_text': 'What is your age?', 'question_type': 'free_text', 'question_name': 'age'}, {'question_text': 'What is your gender?', 'question_type': 'multiple_choice', 'question_options': [], 'question_name': 'gender'}, {'question_text': 'What is your highest level of education?', 'question_type': 'multiple_choice', 'question_options': [], 'question_name': 'education'}, {'question_text': 'What is your annual household income?', 'question_type': 'free_text', 'question_name': 'income'}, {'question_text': 'How many people live in your household?', 'question_type': 'free_text', 'question_name': 'household_size'}, {'question_text': 'How many children live in your household?', 'question_type': 'free_text', 'question_name': 'children'}, {'question_text': 'What is your marital status?', 'question_type': 'multiple_choice', 'question_options': [], 'question_name': 'marital_status'}, {'question_text': 'What is your race/ethnicity?', 'question_type': 'multiple_choice', 'question_options': [], 'question_name': 'race'}, {'question_text': 'How would you rate your overall health?', 'question_type': 'linear_scale', 'question_options': [1, 2, 3, 4, 5], 'option_labels': {'1': 'Poor', '2': 'Fair', '3': 'Good', '4': 'Very Good', '5': 'Excellent'}, 'question_name': 'health'}, {'question_text': 'Do you have any chronic health conditions?', 'question_type': 'multiple_choice', 'question_options': [True, False], 'question_name': 'chronic_conditions'}, {'question_text': 'If yes, please specify.', 'question_type': 'free_text', 'question_name': 'chronic_conditions_details'}, {'question_text': 'Do you have health insurance?', 'question_type': 'multiple_choice', 'question_options': [True, False], 'question_name': 'health_insurance'}, {'question_text': 'If yes, what type of health insurance do you have?', 'question_type': 'multiple_choice', 'question_options': [], 'question_name': 'insurance_type'}, {'question_text': 'How often do you exercise?', 'question_type': 'multiple_choice', 'question_options': [], 'question_name': 'exercise'}, {'question_text': 'How many hours of sleep do you get per night?', 'question_type': 'free_text', 'question_name': 'sleep'}, {'question_text': 'How would you rate your stress level?', 'question_type': 'linear_scale', 'question_options': [1, 2, 3, 4, 5], 'option_labels': {'1': 'Very Low', '2': 'Low', '3': 'Moderate', '4': 'High', '5': 'Very High'}, 'question_name': 'stress'}, {'question_text': 'Do you smoke?', 'question_type': 'multiple_choice', 'question_options': [True, False], 'question_name': 'smoke'}, {'question_text': 'If yes, how many cigarettes do you smoke per day?', 'question_type': 'free_text', 'question_name': 'cigarettes_per_day'}, {'question_text': 'Do you drink alcohol?', 'question_type': 'multiple_choice', 'question_options': [True, False], 'question_name': 'alcohol'}, {'question_text': 'If yes, how many drinks do you consume per week?', 'question_type': 'free_text', 'question_name': 'drinks_per_week'}, {'question_text': 'How would you rate your overall satisfaction with life?', 'question_type': 'linear_scale', 'question_options': [1, 2, 3, 4, 5], 'option_labels': {'1': 'Very Dissatisfied', '2': 'Dissatisfied', '3': 'Neutral', '4': 'Satisfied', '5': 'Very Satisfied'}, 'question_name': 'life_satisfaction'}]` |
| 2 | gpt-4o | `[{'question_text': 'What is your age?', 'question_type': 'free_text', 'question_name': 'age'}, {'question_text': 'How satisfied are you with our service?', 'question_type': 'linear_scale', 'question_options': [1, 2, 3, 4, 5], 'option_labels': {'1': 'Very dissatisfied', '5': 'Very satisfied'}, 'question_name': 'satisfaction_level'}, {'question_text': 'Would you recommend our service to others?', 'question_type': 'multiple_choice', 'question_options': ['Yes', 'No'], 'question_name': 'recommendation'}, {'question_text': 'What improvements would you like to see?', 'question_type': 'free_text', 'question_name': 'improvements'}, {'question_text': 'How often do you use our service?', 'question_type': 'multiple_choice', 'question_options': ['Daily', 'Weekly', 'Monthly', 'Rarely'], 'question_name': 'usage_frequency'}]` |

## Creating a new EDSL survey

Now we can construct a new EDSL survey with the reformatted components of the original survey. This is done by creating `Question` objects with the question components, passing them to a new `Survey`, and then optionally designing and assigning AI agents to answer the survey.

Here we select one of the model’s responses to use:

```python theme={null}
from edsl import Question
```

```python theme={null}
questions_list = results.filter("model.model == 'claude-3-5-sonnet-20240620'").select("q_name").to_list()[0]
questions_list
```

```python expandable theme={null}
[{'question_text': 'How many times have you visited this store in the past month?',
  'question_type': 'multiple_choice',
  'question_options': [0, 1, 2, 3, '4 or more'],
  'question_name': 'visit_frequency'},
 {'question_text': 'How satisfied were you with your overall shopping experience?',
  'question_type': 'linear_scale',
  'question_options': [1, 2, 3, 4, 5],
  'option_labels': {'1': 'Very Dissatisfied', '5': 'Very Satisfied'},
  'question_name': 'overall_satisfaction'},
 {'question_text': 'How likely are you to recommend this store to a friend or family member?',
  'question_type': 'linear_scale',
  'question_options': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
  'option_labels': {'0': 'Not at all likely', '10': 'Extremely likely'},
  'question_name': 'recommendation_likelihood'},
 {'question_text': 'How would you rate the cleanliness of the store?',
  'question_type': 'linear_scale',
  'question_options': [1, 2, 3, 4, 5],
  'question_name': 'store_cleanliness'},
 {'question_text': 'How would you rate the friendliness of our staff?',
  'question_type': 'linear_scale',
  'question_options': [1, 2, 3, 4, 5],
  'question_name': 'staff_friendliness'},
 {'question_text': 'How would you rate the variety of products available?',
  'question_type': 'linear_scale',
  'question_options': [1, 2, 3, 4, 5],
  'question_name': 'product_variety'},
 {'question_text': 'How would you rate the prices of our products?',
  'question_type': 'linear_scale',
  'question_options': [1, 2, 3, 4, 5],
  'question_name': 'product_prices'},
 {'question_text': 'What is your age range?',
  'question_type': 'multiple_choice',
  'question_options': ['Under 18',
   '18-24',
   '25-34',
   '35-44',
   '45-54',
   '55-64',
   '65 or older'],
  'question_name': 'age_range'},
 {'question_text': 'What is your gender?',
  'question_type': 'multiple_choice',
  'question_options': ['Male', 'Female', 'Non-binary', 'Prefer not to say'],
  'question_name': 'gender'},
 {'question_text': 'Do you have any additional comments or suggestions for improvement?',
  'question_type': 'free_text',
  'question_name': 'additional_comments'}]
```

```python theme={null}
edsl_questions = [Question(**q) for q in questions_list]
edsl_questions
```

```python theme={null}
[Question('multiple_choice', question_name = """visit_frequency""", question_text = """How many times have you visited this store in the past month?""", question_options = [0, 1, 2, 3, '4 or more']),
 Question('linear_scale', question_name = """overall_satisfaction""", question_text = """How satisfied were you with your overall shopping experience?""", question_options = [1, 2, 3, 4, 5], option_labels = {1: 'Very Dissatisfied', 5: 'Very Satisfied'}),
 Question('linear_scale', question_name = """recommendation_likelihood""", question_text = """How likely are you to recommend this store to a friend or family member?""", question_options = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], option_labels = {0: 'Not at all likely', 10: 'Extremely likely'}),
 Question('linear_scale', question_name = """store_cleanliness""", question_text = """How would you rate the cleanliness of the store?""", question_options = [1, 2, 3, 4, 5], option_labels = {}),
 Question('linear_scale', question_name = """staff_friendliness""", question_text = """How would you rate the friendliness of our staff?""", question_options = [1, 2, 3, 4, 5], option_labels = {}),
 Question('linear_scale', question_name = """product_variety""", question_text = """How would you rate the variety of products available?""", question_options = [1, 2, 3, 4, 5], option_labels = {}),
 Question('linear_scale', question_name = """product_prices""", question_text = """How would you rate the prices of our products?""", question_options = [1, 2, 3, 4, 5], option_labels = {}),
 Question('multiple_choice', question_name = """age_range""", question_text = """What is your age range?""", question_options = ['Under 18', '18-24', '25-34', '35-44', '45-54', '55-64', '65 or older']),
 Question('multiple_choice', question_name = """gender""", question_text = """What is your gender?""", question_options = ['Male', 'Female', 'Non-binary', 'Prefer not to say']),
 Question('free_text', question_name = """additional_comments""", question_text = """Do you have any additional comments or suggestions for improvement?""")]
```

```python theme={null}
new_survey = Survey(edsl_questions)
```

We can inspect the survey that has been created:

```python theme={null}
new_survey
```

[Survey](/en/latest/surveys) # questions: 10; question\_name list: \['visit\_frequency', 'overall\_satisfaction', 'recommendation\_likelihood', 'store\_cleanliness', 'staff\_friendliness', 'product\_variety', 'product\_prices', 'age\_range', 'gender', 'additional\_comments'];

| | question\_type | question\_name | question\_options | question\_text | option\_labels |
| :- | :- | :- | :- | :- | :- |
| 0 | multiple\_choice | visit\_frequency | \[0, 1, 2, 3, '4 or more'] | How many times have you visited this store in the past month? | nan |
| 1 | linear\_scale | overall\_satisfaction | \[1, 2, 3, 4, 5] | How satisfied were you with your overall shopping experience? | `{1: 'Very Dissatisfied', 5: 'Very Satisfied'}` |
| 2 | linear\_scale | recommendation\_likelihood | \[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | How likely are you to recommend this store to a friend or family member? | `{0: 'Not at all likely', 10: 'Extremely likely'}` |
| 3 | linear\_scale | store\_cleanliness | \[1, 2, 3, 4, 5] | How would you rate the cleanliness of the store? | |
| 4 | linear\_scale | staff\_friendliness | \[1, 2, 3, 4, 5] | How would you rate the friendliness of our staff? | |
| 5 | linear\_scale | product\_variety | \[1, 2, 3, 4, 5] | How would you rate the variety of products available? | |
| 6 | linear\_scale | product\_prices | \[1, 2, 3, 4, 5] | How would you rate the prices of our products? | |
| 7 | multiple\_choice | age\_range | \['Under 18', '18-24', '25-34', '35-44', '45-54', '55-64', '65 or older'] | What is your age range? | nan |
| 8 | multiple\_choice | gender | \['Male', 'Female', 'Non-binary', 'Prefer not to say'] | What is your gender? | nan |
| 9 | free\_text | additional\_comments | nan | Do you have any additional comments or suggestions for improvement? | nan |

## Designing AI agents

EDSL comes with methods for designing AI agent personas for language models to use in answering questions. An `Agent` is created by passing a dictionary of relevant `traits`. It can then be assigned to a survey using the `by()` method when the survey is run (the same as we do with scenarios and models).

We can import existing data to create agents representing audiences of interest, or use EDSL to generate personas:

```python theme={null}
q_personas = QuestionList(
    question_name="personas",
    question_text="Draft 5 diverse personas for customers of a landscape business in New England capable of answering a feedback survey."
)
```

If we do not specify a model to use in running the question, the default model gpt-5.2 is used:

```python theme={null}
personas = q_personas.run().select("personas").to_list()[0]
personas
```

```python theme={null}
['Retired Couple in Suburbia',
 'Young Urban Professional',
 'Environmentally Conscious Family',
 'Small Business Owner',
 'Historic Home Enthusiast']
```

<Note>
  **Note**:

  Note that the personas can be (much) longer and include key/value pairs for any desired traits; we keep it simple here for demonstration purposes. Here we pass the personas to a list of agents and have them answer the survey:
</Note>

```python theme={null}
from edsl import AgentList, Agent
```

```python theme={null}
agents = AgentList(
    Agent(
        traits = {"persona":p},
        instruction = """
        You are answering a customer feedback survey for a landscaping business that you have engaged in the past.
        Your answers are completely confidential.
        """
    )
    for p in personas
)

```

```python theme={null}
new_results = new_survey.by(agents).by(models).run()
```

```python theme={null}
(
    new_results
    .sort_by("model", "persona")
    .select("model", "persona", "answer.*")
)
```

| | model.model | agent.persona | answer.additional\_comments | answer.age\_range | answer.product\_variety | answer.recommendation\_likelihood | answer.product\_prices | answer.gender | answer.overall\_satisfaction | answer.visit\_frequency | answer.staff\_friendliness | answer.store\_cleanliness |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | claude-3-5-sonnet-20240620 | Environmentally Conscious Family | As an environmentally conscious family, I really appreciated the eco-friendly approach of your landscaping services. However, I have a few suggestions that could further improve your offerings: 1. Native plant options: It would be great to see a wider selection of native plants that support local ecosystems and require less water and maintenance. 2. Organic pest control: I'd love more information on natural pest control methods you use or recommend, avoiding harmful chemicals. 3. Water conservation: Perhaps you could offer installation of rain barrels or smart irrigation systems to help reduce water usage. 4. Composting services: It would be fantastic if you could offer composting services or guidance on how to start a home compost system. 5. Sustainable materials: When possible, use recycled or sustainable materials for hardscaping projects. 6. Educational resources: Provide more information on sustainable gardening practices that we can implement ourselves. 7. Green waste disposal: Ensure all green waste is properly composted or recycled, not sent to landfills. Overall, we've been pleased with your services, but these additions would really align with our family's environmental values and potentially attract more eco-conscious customers. | 35-44 | 4 | 8 | 3 | Female | 4 | 1 | 4 | 4 |
| 1 | claude-3-5-sonnet-20240620 | Historic Home Enthusiast | As a historic home enthusiast, I really appreciate when landscaping companies understand the unique needs and aesthetics of period properties. I would suggest offering specialized services or packages tailored to historic homes, such as: 1. Heritage plant selections that are appropriate for different architectural eras 2. Expertise in maintaining and restoring historic garden layouts 3. Knowledge of traditional landscaping techniques and materials 4. Ability to work around delicate old structures and mature trees 5. Familiarity with local historic preservation guidelines It would also be great if the staff received some training on the history of landscape design to better serve clients with historic properties. Perhaps partnering with local historical societies or preservation groups could provide additional resources and credibility in this niche. Overall, I've been satisfied with the service, but these additions would really set the company apart for homeowners like myself who are passionate about maintaining the historical integrity of our properties. | 55-64 | 4 | 8 | 4 | Female | 4 | 1 | 4 | 3 |
| 2 | claude-3-5-sonnet-20240620 | Retired Couple in Suburbia | As a retired couple living in suburbia, we've generally been pleased with the landscaping service. However, we do have a few suggestions that might improve things: 1. We'd appreciate if the crew could be a bit more mindful of noise levels, especially in the early morning hours. As retirees, we tend to enjoy our quiet mornings. 2. It would be helpful to have more flexibility in scheduling. Perhaps offering service time slots in the afternoon could be an option for those of us who aren't on a typical work schedule. 3. We've noticed that some of the younger plants haven't been thriving as well as we'd hoped. A bit more guidance on plant care or perhaps a follow-up visit to check on new plantings would be appreciated. 4. While we understand the importance of chemical treatments for lawn health, we'd be interested in exploring more eco-friendly options if available. 5. Lastly, it might be nice to offer some simple landscaping workshops for customers. As retirees, we have more time to tend to our garden and would love to learn more about maintaining it between your visits. Overall, we're satisfied with the service, but these small improvements could really enhance our experience. Thank you for asking for our feedback! | 65 or older | 4 | 8 | 4 | Female | 4 | 1 | 5 | 3 |
| 3 | claude-3-5-sonnet-20240620 | Small Business Owner | As a small business owner myself, I appreciate the importance of customer feedback. Here are a few suggestions I'd offer: 1. Communication: Perhaps consider implementing a system for more regular updates on project progress, especially for longer jobs. This helps clients feel informed and engaged. 2. Flexibility: As a business owner, I know schedules can be tight. Having some flexibility in scheduling or offering early morning/late evening slots could be beneficial for clients with busy workdays. 3. Eco-friendly options: If not already offered, consider providing environmentally friendly landscaping choices. This could include native plant options or water-saving designs, which may appeal to environmentally conscious clients. 4. Follow-up service: A quick follow-up call or email a few weeks after job completion to ensure continued satisfaction could go a long way in building customer loyalty. 5. Referral program: If you don't already have one, consider implementing a referral program. As a business owner, I'm always happy to recommend good services to my network, especially if there's an incentive. Overall, I've been satisfied with the service, but these small tweaks could potentially enhance the customer experience and help grow your business further. | 45-54 | 4 | 8 | 3 | Male | 4 | 1 | 4 | 3 |
| 4 | claude-3-5-sonnet-20240620 | Young Urban Professional | As a young urban professional, I might respond: I appreciate the quality of work your team does, but I think there's room for improvement in a few areas: 1. Digital presence: It would be great if you had a more user-friendly website or app where I could easily schedule services, view my account, and make payments. 2. Eco-friendly options: I'd love to see more sustainable landscaping choices offered, like native plant installations or water-saving irrigation systems. 3. Flexible scheduling: As someone with a busy work schedule, it would be helpful to have more flexible appointment times, including early mornings, evenings, or weekends. 4. Modern design options: Consider offering more contemporary landscape design choices that appeal to younger homeowners and complement urban environments. 5. Communication: While your team does good work, sometimes I'm not sure exactly when they'll arrive. A text notification system would be really convenient. Overall, I'm satisfied with your services, but implementing some of these suggestions could really set you apart and appeal more to clients like myself. Thanks for asking for feedback! | 25-34 | 4 | 8 | 4 | Female | 4 | 1 | 4 | 4 |
| 5 | gemini-1.5-flash | Environmentally Conscious Family | Yes, I do. Overall, we were very pleased with the results of the landscaping work. However, we'd love to see you expand your selection of native plants and drought-tolerant options. As an environmentally conscious family, we're trying to reduce our water usage and support local ecosystems. Offering more choices in this area would be a huge plus and would align with the growing interest in sustainable landscaping practices. Perhaps you could offer a consultation service that helps clients choose the best native plants for their specific needs and soil conditions. | 35-44 | 4 | 8 | 4 | Prefer not to say | 4 | 0 | 5 | 5 |
| 6 | gemini-1.5-flash | Historic Home Enthusiast | While I was extremely pleased with the overall quality of the work, particularly the sensitive restoration of the original stonework around my historic home's foundation, I do have a couple of suggestions. First, perhaps a more detailed pre-project consultation outlining the specific types of materials to be used, including sourcing information for historically accurate replacements, would be beneficial. Knowing the provenance of the materials used would have enhanced my appreciation of the craftsmanship. Secondly, a slightly more comprehensive post-project walkthrough highlighting maintenance recommendations specific to the materials used and the age of the property would be a welcome addition. This would ensure the longevity of the beautiful work you've completed. | 45-54 | 4 | 8 | 4 | Prefer not to say | 4 | 0 | 5 | 5 |
| 7 | gemini-1.5-flash | Retired Couple in Suburbia | While we were very pleased with the overall outcome of the landscaping project, a few minor suggestions might enhance the customer experience. Firstly, perhaps a more detailed, perhaps even illustrated, pre-project plan could be provided. We found ourselves needing to clarify a few points during the process, and a more comprehensive initial plan might have preempted those questions. Secondly, a slightly more proactive communication schedule – perhaps a brief check-in midway through the project – would have been reassuring. We understand you're busy, but a quick update would have been appreciated. Finally, a small suggestion: offering a brief post-project care guide, perhaps outlining basic watering and maintenance tips for the newly installed plants, would be a thoughtful touch. Overall, we were very happy with the work, and these are just minor suggestions for potential improvements. | 65 or older | 4 | 8 | 3 | Prefer not to say | 4 | 0 | 5 | 5 |
| 8 | gemini-1.5-flash | Small Business Owner | Overall, I was pleased with the work. However, as a small business owner myself, I'm always looking for ways to improve efficiency and communication. Perhaps offering a more detailed, itemized quote upfront, including potential material costs and timelines, would be beneficial. Sometimes, unexpected costs can arise, and having a clearer understanding from the outset would help manage expectations and budgeting. Also, a quick follow-up email after the job is complete, confirming satisfaction and perhaps including a link to online review platforms, would be a nice touch. Small things like that can make a big difference in customer experience. | 35-44 | 4 | 8 | 3 | Prefer not to say | 4 | 1 | 5 | 5 |
| 9 | gemini-1.5-flash | Young Urban Professional | Overall, I was happy with the work. My only suggestion would be to offer more detailed, perhaps even visual, options upfront for plant selection. I felt a little overwhelmed by the sheer number of choices and ended up relying heavily on your expertise, which was great, but a curated selection of popular options with photos would have made the initial consultation smoother and more efficient for both of us. Something like a digital portfolio or online catalog would be fantastic. | 25-34 | 4 | 8 | 3 | Prefer not to say | 4 | 0 | 5 | 5 |
| 10 | gpt-4o | Environmentally Conscious Family | We really appreciate the efforts your team makes to incorporate sustainable practices. It would be great if you could expand your selection of native plants and offer more organic lawn care options. Additionally, providing information on how your services contribute to local biodiversity could be a nice touch. Thank you for your commitment to environmentally friendly landscaping! | 35-44 | 4 | 9 | 3 | Prefer not to say | 5 | 0 | 5 | 5 |
| 11 | gpt-4o | Historic Home Enthusiast | I was really pleased with the landscaping work done on my historic home. The team was respectful of the property's unique characteristics and managed to enhance its charm while maintaining its original feel. One suggestion would be to perhaps offer more guidance on plant selections that are historically appropriate for different architectural styles. This could be a great addition for those of us who want to stay true to our home's period. Overall, I was very satisfied with the service! | 45-54 | 4 | 8 | 4 | Prefer not to say | 5 | 0 | 5 | 5 |
| 12 | gpt-4o | Retired Couple in Suburbia | We were really pleased with the landscaping service and how our yard looks now. The team was professional and timely, and they really understood our vision for a low-maintenance garden. One suggestion might be to offer a seasonal check-up service to help us keep everything looking its best throughout the year. It could be a great option for retirees like us who want to enjoy our garden without too much effort. Overall, we're very satisfied and would definitely recommend your services to our neighbors. | 65 or older | 4 | 9 | 4 | Prefer not to say | 5 | 0 | 5 | 5 |
| 13 | gpt-4o | Small Business Owner | I appreciate the quality of work and professionalism your team consistently demonstrates. One suggestion for improvement would be to offer more flexible scheduling options, as it can sometimes be challenging to coordinate with my business hours. Additionally, providing a detailed maintenance guide or tips for seasonal care could be beneficial for small business owners like myself who want to keep the landscape looking its best year-round. Overall, I'm very satisfied with your services. | 35-44 | 4 | 8 | 4 | Prefer not to say | 5 | 0 | 5 | 5 |
| 14 | gpt-4o | Young Urban Professional | Overall, I was quite satisfied with the service. The team was professional and efficient, and my outdoor space looks great. One suggestion might be to offer a digital platform where clients can easily schedule services, view past appointments, and perhaps even see design mockups or ideas. As someone who is always on the go, having everything accessible online would be incredibly convenient. Keep up the great work! | 25-34 | 4 | 8 | 3 | Prefer not to say | 5 | 0 | 5 | 5 |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we demonstrate how to post this notebook:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "google_form_to_edsl.ipynb")

if refresh := False:
    nb.push(
        description = "Example code for using EDSL to convert a non-EDSL survey into EDSL",
        alias = "google-form-to-edsl-notebook",
        visibility = "public"
    )
else:
    nb.patch('815c1988-8656-4644-8ab8-70ce36b8d62d', value = nb)

```


# Using images in a survey
Source: https://docs.expectedparrot.com/en/latest/notebooks/image_scenario_example



[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL and the platform.

## Scenarios

A `Scenario` is a dictionary containing a key/value pair that is used to add data or content to questions in an EDSL survey. Scenarios allow you create variations and versions of questions efficiently, and with data or content from different sources.

EDSL provides a variety of methods for automatically generating scenarios from PDFs, CSVs, docs, tables, lists, dicts – and images. In the steps below we demonstrate how to create a scenario for an image and use it in a survey.

<Note>
  **Note**:

  When using images with questions it is necessary to specify a vision model, and to ensure that the model is capable of viewing each image. Always run test questions to ensure that each image is actually readable by the selected models.
</Note>

[Learn more about working with scenarios](/en/latest/scenarios).

### Creating a scenario

We start by creating a `Scenario` for an image. For purposes of demonstration, we use the `FileStore` module to post a PNG image to Expected Parrot, and then retrieve it and pass it to a `Scenario` (this can be done by [any user with an Expected Parrot account](https://www.expectedparrot.com/login)). Note that `FileStore` can be used to post and retrieve all types of files, and will automatically infer the file type.

Here we post a file to Expected Parrot:

```python theme={null}
from edsl import FileStore

filename = "parrot_logo.png" # file stored locally

fs = FileStore(filename)
fs.push(
    description = "My parrot logo image",
    alias = "my-parrot-logo",
    visibility = "public"
)
```

This returns object info we can now use to retrieve the image:

```json theme={null}
{'description': 'My parrot logo image',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/6bc5aa85-8c58-40b8-ab61-f7b3df9e1409',
 'uuid': '6bc5aa85-8c58-40b8-ab61-f7b3df9e1409',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

Here we retrieve the file (can be replaced with the alias or UUID of any posted object):

```python theme={null}
from edsl import FileStore

png_file = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/my-parrot-logo")
```

This is equivalent:

```python theme={null}
png_file = FileStore.pull("6bc5aa85-8c58-40b8-ab61-f7b3df9e1409")
```

Here we use the retrieved file in a `Scenario` by creating a key and passing the file as the value. We also (optionally) create a key/value for metadata about the file that we want to keep with the survey results (more on this below):

```python theme={null}
from edsl import Scenario

s = Scenario({
    "parrot_logo": png_file,
    "filename": "parrot_logo.png" # optional metadata field
})
```

## Creating questions using the image

Next we construct questions with the image scenario. Note that we use a `{{ placeholder }}` for the scenario key for the image file. This will cause the image to be automatically be inserted when the survey is run with the scenario. We also pipe the answer to one question into a follow-on question:

```python theme={null}
from edsl import QuestionYesNo, QuestionMultipleChoice, QuestionList, Survey

q1 = QuestionYesNo(
    question_name = "animal",
    question_text = "Is there an animal in this image? {{ scenario.parrot_logo }}"
)

q2 = QuestionMultipleChoice(
    question_name = "identify",
    question_text = "Identify the animal in this image: {{ scenario.parrot_logo }}",
    question_options = ["dog", "cat", "bird", "something else"]
)

q3 = QuestionList(
    question_name = "colors",
    question_text = "What color(s) is this {{ identify.answer }}? {{ scenario.parrot_logo }}",
)

survey = Survey(questions = [q1, q2, q3])
```

Next we add a rule to stop the survey if the answer to the first question is “No”. This rule and the piping in the questions that follow will cause the questions to be administered in the required order, instead of asynchronously by default (learn more about [piping](/en/latest/surveys#id2) and applygin [survey rules](/en/latest/surveys#key-methods)):

```python theme={null}
survey = (
    survey
    .add_stop_rule(q1, "{{ animal.answer }} == 'No'")
)
```

Next we select a model to generate the responses. Note that we need to use a vision model. You can check available vision models at Expected Parrot [model pricing page](https://www.expectedparrot.com/getting-started/coop-pricing).

```python theme={null}
from edsl import Model

m = Model("gemini-1.5-flash", service_name = "google")
```

We administer the survey in the same way that we do with any other scenarios:

```python theme={null}
results = survey.by(s).by(m).run()
```

We can select any scenario key/value to access in the results that have been generated (e.g., image metadata created):

```python theme={null}
results.select("model", "filename", "animal", "identify", "colors", "colors_comment")
```

| | model.model | scenario.filename | answer.animal | answer.identify | answer.colors | comment.colors\_comment |
| :- | :- | :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | parrot\_logo.png | Yes | bird | \['green', 'red', 'orange', 'blue'] | The parrot is primarily green, with red, orange (on its beak), and blue on its underside. |

## Posting to Expected Parrot

The results of the survey were automatically posted to Expected Parrot using remote inference (see link in the job summary above). Here we also post them to the platform:

```python theme={null}
from edsl import Notebook

nb = Notebook("image_scenario_example.ipynb")

nb.push(
    description = "Using an image scenario",
    alias = "my-parrot-logo-notebook",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Using an image scenario',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/4a0cf8e9-bf88-4f8b-a48a-2d4099634e3a',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/my-parrot-logo-notebook',
 'uuid': '4a0cf8e9-bf88-4f8b-a48a-2d4099634e3a',
 'version': '0.1.61.dev1',
 'visibility': 'public'}
```


# Importing agents
Source: https://docs.expectedparrot.com/en/latest/notebooks/import_agents

This notebook provides sample EDSL code for simulating surveys with agent personas imported from [Persona Hub](https://github.com/tencent-ailab/persona-hub), an open-source repository provided by [Hugging Face](https://huggingface.co/datasets/proj-persona/PersonaHub).

[EDSL](https://github.com/expectedparrot/edsl) is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Importing the data

Persona Hub: [https://github.com/tencent-ailab/persona-hub](https://github.com/tencent-ailab/persona-hub)

```bash theme={null}
! pip install datasets
```

```python expandable theme={null}
Collecting datasets
  Downloading datasets-3.5.0-py3-none-any.whl.metadata (19 kB)
Requirement already satisfied: filelock in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from datasets) (3.18.0)
Requirement already satisfied: numpy>=1.17 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from datasets) (1.26.4)
Collecting pyarrow>=15.0.0 (from datasets)
  Using cached pyarrow-19.0.1-cp311-cp311-macosx_12_0_arm64.whl.metadata (3.3 kB)
Collecting dill<0.3.9,>=0.3.0 (from datasets)
  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: pandas in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from datasets) (2.2.3)
Requirement already satisfied: requests>=2.32.2 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from datasets) (2.32.3)
Requirement already satisfied: tqdm>=4.66.3 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from datasets) (4.67.1)
Collecting xxhash (from datasets)
  Downloading xxhash-3.5.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (12 kB)
Collecting multiprocess<0.70.17 (from datasets)
  Downloading multiprocess-0.70.16-py311-none-any.whl.metadata (7.2 kB)
Collecting fsspec<=2024.12.0,>=2023.1.0 (from fsspec[http]<=2024.12.0,>=2023.1.0->datasets)
  Downloading fsspec-2024.12.0-py3-none-any.whl.metadata (11 kB)
Requirement already satisfied: aiohttp in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from datasets) (3.11.16)
Collecting huggingface-hub>=0.24.0 (from datasets)
  Downloading huggingface_hub-0.30.2-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: packaging in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from datasets) (24.2)
Requirement already satisfied: pyyaml>=5.1 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from datasets) (6.0.2)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from aiohttp->datasets) (2.6.1)
Requirement already satisfied: aiosignal>=1.1.2 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from aiohttp->datasets) (1.3.2)
Requirement already satisfied: attrs>=17.3.0 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from aiohttp->datasets) (25.3.0)
Requirement already satisfied: frozenlist>=1.1.1 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from aiohttp->datasets) (1.5.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from aiohttp->datasets) (6.3.1)
Requirement already satisfied: propcache>=0.2.0 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from aiohttp->datasets) (0.3.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from aiohttp->datasets) (1.18.3)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from huggingface-hub>=0.24.0->datasets) (4.13.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from requests>=2.32.2->datasets) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from requests>=2.32.2->datasets) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from requests>=2.32.2->datasets) (1.26.20)
Requirement already satisfied: certifi>=2017.4.17 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from requests>=2.32.2->datasets) (2025.1.31)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from pandas->datasets) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from pandas->datasets) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from pandas->datasets) (2025.2)
Requirement already satisfied: six>=1.5 in /Users/johnhorton/Library/Caches/pypoetry/virtualenvs/edsl-EZo3_VAr-py3.11/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0)
Downloading datasets-3.5.0-py3-none-any.whl (491 kB)
Downloading dill-0.3.8-py3-none-any.whl (116 kB)
Downloading fsspec-2024.12.0-py3-none-any.whl (183 kB)
Downloading huggingface_hub-0.30.2-py3-none-any.whl (481 kB)
Downloading multiprocess-0.70.16-py311-none-any.whl (143 kB)
Using cached pyarrow-19.0.1-cp311-cp311-macosx_12_0_arm64.whl (30.7 MB)
Downloading xxhash-3.5.0-cp311-cp311-macosx_11_0_arm64.whl (30 kB)
Installing collected packages: xxhash, pyarrow, fsspec, dill, multiprocess, huggingface-hub, datasets
  Attempting uninstall: dill
    Found existing installation: dill 0.3.9
    Uninstalling dill-0.3.9:
      Successfully uninstalled dill-0.3.9
Successfully installed datasets-3.5.0 dill-0.3.8 fsspec-2024.12.0 huggingface-hub-0.30.2 multiprocess-0.70.16 pyarrow-19.0.1 xxhash-3.5.0

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
```

```python expandable theme={null}
try:
    from datasets import load_dataset
    ds = load_dataset("proj-persona/PersonaHub", "instruction")
    num_agents = 10
    personas_list = ds['train']['input persona'][:num_agents]
    personas_list
except ImportError:
    personas_list = ['A theater manager or events coordinator interested in understanding the operational aspects, facilities, and programming of performing arts centers, such as the Broward Center.',
 'An urban planner looking to understand the distribution and organization of public services in the Halifax Regional Municipality.',
 "A high school literature teacher looking for supplementary materials to enrich their curriculum and provide students with a deeper understanding of their state's cultural heritage.",
 'A science fiction writer, exploring the use of secret trials as a theme in stories that examine surveillance, privacy, and power dynamics in society.',
 'A financial analyst specializing in Asian markets and wealthy individuals, interested in tracking the investments and philanthropic activities of billionaires like Gerald Chan.',
 'A high school physics teacher who enjoys applying physics concepts to various sports, curious about the effects of water resistance and body positioning in Paralympic swimming.',
 'A high school physical education teacher seeking to incorporate Paralympic history and achievements into the curriculum to inspire and educate students about inclusivity in sports.',
 'A high school history teacher creating lesson plans on South American civilizations and their regional diversity.',
 'A cultural anthropologist studying the representation and adaptation of traditional folklore in contemporary media, particularly in television and film.',
 'A high school earth science teacher looking for real-world examples to explain climate zones, soil types, and human-environment interactions.']
```

## Selecting data

Here we select some personas to use.

## Creating agents

We create an agent in EDSL by passing a dictionary of `traits` (such as a persona) to an `Agent` object. Here we use a method for generating a set of agents from a list of traits at once (a list of the personas). Learn more about [designing agents in EDSL](/en/latest/agents).

```python theme={null}
from edsl import AgentList

agents = AgentList.from_list("persona", personas_list)
```

## Using agents to answer a survey

Here we construct some questions and administer them to the agents. EDSL comes with many [common question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from a model. Learn more about [constructing surveys](/en/latest/surveys) with different question types, logic and rules.

```python theme={null}
from edsl import QuestionMultipleChoice, QuestionLinearScale, Survey

q1 = QuestionMultipleChoice(
    question_name = "preferred_mode_commute",
    question_text = "What is your preferred mode of commuting to work?",
    question_options = [ "Car", "Public transportation", "Bike", "Walk", "Work from home", "Other" ]
)

q2 = QuestionLinearScale(
    question_name = "work_from_home",
    question_text = "On a scale from 1 to 5, how easy is it for you to work from home?",
    question_options = [1,2,3,4,5],
    option_labels = {1:"Not at all easy", 5:"Very easy"}
)

survey = Survey([q1, q2])
```

## Selecting language models

EDSL works with many popular language models that we can select to generate the responses for the agents.

To see a list of all services:

```python theme={null}
from edsl import Model

Model.services()
```

| | Service Name |
| :- | :- |
| 0 | anthropic |
| 1 | azure |
| 2 | bedrock |
| 3 | deep\_infra |
| 4 | deepseek |
| 5 | google |
| 6 | groq |
| 7 | mistral |
| 8 | ollama |
| 9 | openai |
| 10 | perplexity |
| 11 | together |
| 12 | xai |

To see a list of all available models:

```python theme={null}
\# Model.available()

To select models to use with a survey:

from edsl import ModelList, Model

models = ModelList(
    Model(m) for m in ["gemini-1.5-flash", "gpt-4o"]
)
```

We run a survey by adding the agents and models, and then calling the `run()` method. This generates a formatted dataset of `Results`:

```python theme={null}
results = survey.by(agents).by(models).run()
```

## Analyzing results

EDSL comes with [built-in methods for analysis](/en/latest/results). Here we inspect the responses in a table:

```python theme={null}
results.select("model", "persona", "preferred_mode_commute", "work_from_home")
```

| | model.model | agent.persona | answer.preferred\_mode\_commute | answer.work\_from\_home |
| :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | A theater manager or events coordinator interested in understanding the operational aspects, facilities, and programming of performing arts centers, such as the Broward Center. | Car | 1 |
| 1 | gpt-4o | A theater manager or events coordinator interested in understanding the operational aspects, facilities, and programming of performing arts centers, such as the Broward Center. | Car | 2 |
| 2 | gemini-1.5-flash | An urban planner looking to understand the distribution and organization of public services in the Halifax Regional Municipality. | Public transportation | 3 |
| 3 | gpt-4o | An urban planner looking to understand the distribution and organization of public services in the Halifax Regional Municipality. | Public transportation | 4 |
| 4 | gemini-1.5-flash | A high school literature teacher looking for supplementary materials to enrich their curriculum and provide students with a deeper understanding of their state's cultural heritage. | Car | 3 |
| 5 | gpt-4o | A high school literature teacher looking for supplementary materials to enrich their curriculum and provide students with a deeper understanding of their state's cultural heritage. | Car | 3 |
| 6 | gemini-1.5-flash | A science fiction writer, exploring the use of secret trials as a theme in stories that examine surveillance, privacy, and power dynamics in society. | Work from home | 5 |
| 7 | gpt-4o | A science fiction writer, exploring the use of secret trials as a theme in stories that examine surveillance, privacy, and power dynamics in society. | Work from home | 5 |
| 8 | gemini-1.5-flash | A financial analyst specializing in Asian markets and wealthy individuals, interested in tracking the investments and philanthropic activities of billionaires like Gerald Chan. | Work from home | 4 |
| 9 | gpt-4o | A financial analyst specializing in Asian markets and wealthy individuals, interested in tracking the investments and philanthropic activities of billionaires like Gerald Chan. | Work from home | 5 |
| 10 | gemini-1.5-flash | A high school physics teacher who enjoys applying physics concepts to various sports, curious about the effects of water resistance and body positioning in Paralympic swimming. | Car | 3 |
| 11 | gpt-4o | A high school physics teacher who enjoys applying physics concepts to various sports, curious about the effects of water resistance and body positioning in Paralympic swimming. | Bike | 2 |
| 12 | gemini-1.5-flash | A high school physical education teacher seeking to incorporate Paralympic history and achievements into the curriculum to inspire and educate students about inclusivity in sports. | Car | 1 |
| 13 | gpt-4o | A high school physical education teacher seeking to incorporate Paralympic history and achievements into the curriculum to inspire and educate students about inclusivity in sports. | Bike | 1 |
| 14 | gemini-1.5-flash | A high school history teacher creating lesson plans on South American civilizations and their regional diversity. | Car | 3 |
| 15 | gpt-4o | A high school history teacher creating lesson plans on South American civilizations and their regional diversity. | Car | 4 |
| 16 | gemini-1.5-flash | A cultural anthropologist studying the representation and adaptation of traditional folklore in contemporary media, particularly in television and film. | Work from home | 4 |
| 17 | gpt-4o | A cultural anthropologist studying the representation and adaptation of traditional folklore in contemporary media, particularly in television and film. | Work from home | 4 |
| 18 | gemini-1.5-flash | A high school earth science teacher looking for real-world examples to explain climate zones, soil types, and human-environment interactions. | Car | 2 |
| 19 | gpt-4o | A high school earth science teacher looking for real-world examples to explain climate zones, soil types, and human-environment interactions. | Car | 3 |

## Constructing traits

In comparing model responses it can be convenient to include shortnames for traits in addition to narrative personas–e.g., just the name of the agent’s occupation, age, etc. Here we run a question to extract the occupation from each persona in order to store it as a separate `trait` of each agent. We use `Scenario` objects to represent the data (personas) that we add to the question when we run it. [Learn more about using scenarios](/en/latest/scenarios) to parameterize questions with data and context.

```python theme={null}
from edsl import QuestionExtract

q = QuestionExtract(
    question_name = "occupation",
    question_text = "{{ scenario.persona }}",
    answer_template = {"occupation":"artist"}
)
```

Creating scenarios for the personas in order to add them to the question:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_list("persona", personas_list)
```

```python theme={null}
results = q.by(scenarios).run()
```

```python theme={null}
results.select("persona", "occupation")
```

| | scenario.persona | answer.occupation |
| :- | :- | :- |
| 0 | A theater manager or events coordinator interested in understanding the operational aspects, facilities, and programming of performing arts centers, such as the Broward Center. | `{'occupation': 'theater manager or events coordinator'}` |
| 1 | An urban planner looking to understand the distribution and organization of public services in the Halifax Regional Municipality. | `{'occupation': 'urban planner'}` |
| 2 | A high school literature teacher looking for supplementary materials to enrich their curriculum and provide students with a deeper understanding of their state's cultural heritage. | `{'occupation': 'high school literature teacher'}` |
| 3 | A science fiction writer, exploring the use of secret trials as a theme in stories that examine surveillance, privacy, and power dynamics in society. | `{'occupation': 'science fiction writer'}` |
| 4 | A financial analyst specializing in Asian markets and wealthy individuals, interested in tracking the investments and philanthropic activities of billionaires like Gerald Chan. | `{'occupation': 'financial analyst'}` |
| 5 | A high school physics teacher who enjoys applying physics concepts to various sports, curious about the effects of water resistance and body positioning in Paralympic swimming. | `{'occupation': 'high school physics teacher'}` |
| 6 | A high school physical education teacher seeking to incorporate Paralympic history and achievements into the curriculum to inspire and educate students about inclusivity in sports. | `{'occupation': 'high school physical education teacher'}` |
| 7 | A high school history teacher creating lesson plans on South American civilizations and their regional diversity. | `{'occupation': 'high school history teacher'}` |
| 8 | A cultural anthropologist studying the representation and adaptation of traditional folklore in contemporary media, particularly in television and film. | `{'occupation': 'cultural anthropologist'}` |
| 9 | A high school earth science teacher looking for real-world examples to explain climate zones, soil types, and human-environment interactions. | `{'occupation': 'high school earth science teacher'}` |

## Designing agents

Here we recreate agents with both the personas and occupations as traits:

```python theme={null}
occupations_list = [o['occupation'] for o in results.select("occupation").to_list()]
occupations_list


['theater manager or events coordinator',
 'urban planner',
 'high school literature teacher',
 'science fiction writer',
 'financial analyst',
 'high school physics teacher',
 'high school physical education teacher',
 'high school history teacher',
 'cultural anthropologist',
 'high school earth science teacher']
```

```python theme={null}

t = list(zip(personas_list, occupations_list))
```

```python theme={null}

from edsl import AgentList, Agent

agents = AgentList(
    Agent(traits = {"persona":p, "occupation":o}) for p,o in t
)
```

## Running a survey

Here we rerun the survey and then filter and sort results by agent traits:

```python theme={null}
results = survey.by(agents).by(models).run()
```

```python theme={null}
(
    results
    .filter("model.model == 'gpt-4o'")
    .sort_by("work_from_home", reverse=True)
    .select("model", "occupation", "preferred_mode_commute", "work_from_home")
)
```

| | model.model | agent.occupation | answer.preferred\_mode\_commute | answer.work\_from\_home |
| :- | :- | :- | :- | :- |
| 0 | gpt-4o | science fiction writer | Work from home | 5 |
| 1 | gpt-4o | financial analyst | Work from home | 5 |
| 2 | gpt-4o | cultural anthropologist | Work from home | 5 |
| 3 | gpt-4o | urban planner | Public transportation | 4 |
| 4 | gpt-4o | high school literature teacher | Car | 3 |
| 5 | gpt-4o | high school physics teacher | Bike | 3 |
| 6 | gpt-4o | high school history teacher | Car | 3 |
| 7 | gpt-4o | high school earth science teacher | Car | 3 |
| 8 | gpt-4o | theater manager or events coordinator | Car | 2 |
| 9 | gpt-4o | high school physical education teacher | Bike | 1 |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we demonstrate how to post this notebook:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "import_agents.ipynb")

if refresh := False:
    nb.push(
        description = "Importing agents",
        alias = "importing-agents-notebook",
        visibility = "public"
    )
else:
    nb.patch('e0f3c111-328e-417b-b2ad-aec527adad49', value = nb)
```


# Looping & piping questions
Source: https://docs.expectedparrot.com/en/latest/notebooks/looping_and_piping

This notebook provides example EDSL code for automatically looping (repeating) a question with content piped from other questions and answers.

Please see the EDSL [documentation page](/en/latest/surveys) for more details on each of the object types and methods for looping questions and piping questions and answers that are used below.

```python theme={null}
from edsl import QuestionNumerical, Scenario, ScenarioList, Survey
```

We start by creating an initial question (with no content piped into it). EDSL comes with many common [question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from a model (e.g., free text, multiple choice, linear scale, etc.). Here we use a numerical question:

```python theme={null}
q_0 = QuestionNumerical(
    question_name = "q_0",
    question_text = "Please give me a random number.",
    min_value = 1,
    max_value = 100,
    answering_instructions = "The number must be an integer."
)
```

Next we create a question that we will “loop” (repeat) some number of times. We use double braces to create a `{{ placeholder }}` for content to be added to the question when we create copies of it

Here we want to simultaneously set the names of the copies of the question and reference those names in the versions of the question text, so that content from one question and answer can be automatically piped into another cope of the question. To do this, we create placeholders for each question name (`{{ num }}`) (it must be unique) and question text (`{{ text }}`). Then in the next step we reference the question names in those texts.

<Note>
  **Note**:

  (Note that the names of the placeholders can be anything other than reserved names, and this example works with any other question types as well. We just use a numerical question to keep the responses brief and easy to check!)
</Note>

```python theme={null}
q = QuestionNumerical(
    question_name = "q_`{{ scenario.num }}`",
    question_text = "`{{ scenario.text }}`",
    min_value = 1,
    max_value = 100,
    answering_instructions = "The number must be an integer."
)
```

Next we create a list of `Scenario` objects for the question name and question text inputs that we will pass to the `loop` method that we call on the question in order to create the copies (learn more about [using scenarios](/en/latest/scenarios)):

```python theme={null}
s = ScenarioList(
    [Scenario({
        "num": n,
        "text": f"""
        I asked you for a random number between 1 and 100 and you gave me `{{ q_{n-1}.answer }}`.
        Please give me a new random number.
        """
    }) for n in range(1,6)]
)
```

The `loop` method creates a list of questions with the scenarios added in. Note that because we used single-braces for ease of referencing the piped question names we will see a warning that scenarios require double braces, in case we used the single braces inadvertently. We can ignore this message here, and confirm that our questions have been formatted as intended:

```python theme={null}
qq = q.loop(s)
qq
```

```
[Question('numerical', question_name = """q_1""", question_text = """
         I asked you for a random number between 1 and 100 and you gave me `{ q_0.answer }`.
         Please give me a new random number.
         """, min_value = 1, max_value = 100, answering_instructions = """The number must be an integer."""),
 Question('numerical', question_name = """q_2""", question_text = """
         I asked you for a random number between 1 and 100 and you gave me `{ q_1.answer }`.
         Please give me a new random number.
         """, min_value = 1, max_value = 100, answering_instructions = """The number must be an integer."""),
 Question('numerical', question_name = """q_3""", question_text = """
         I asked you for a random number between 1 and 100 and you gave me `{ q_2.answer }`.
         Please give me a new random number.
         """, min_value = 1, max_value = 100, answering_instructions = """The number must be an integer."""),
 Question('numerical', question_name = """q_4""", question_text = """
         I asked you for a random number between 1 and 100 and you gave me `{ q_3.answer }`.
         Please give me a new random number.
         """, min_value = 1, max_value = 100, answering_instructions = """The number must be an integer."""),
 Question('numerical', question_name = """q_5""", question_text = """
         I asked you for a random number between 1 and 100 and you gave me `{ q_4.answer }`.
         Please give me a new random number.
         """, min_value = 1, max_value = 100, answering_instructions = """The number must be an integer.""")]
```

We pass the list of questions to a `Survey` object as usual in order to administer them together. Note that because we are piping answers into questions, the questions will automatically be administered in the order required by the piping. (If no piping or other survey rules are applied, questions are administered asychronously by default. Learn more about applying [survey rules and logic](/en/latest/surveys).)

We can re-inspect the questions that are now in a survey:

```python theme={null}
survey = Survey(questions = [q_0] + qq)
survey
```

```python theme={null}
Survey # questions: 6; question_name list: ['q_0', 'q_1', 'q_2', 'q_3', 'q_4', 'q_5'];
```

| | question\_name | question\_text | min\_value | max\_value | answering\_instructions | question\_type |
| :- | :- | :- | :- | :- | :- | :- |
| 0 | q\_0 | Please give me a random number. | 1 | 100 | The number must be an integer. | numerical |
| 1 | q\_1 | I asked you for a random number between 1 and 100 and you gave me `{ q_0.answer }`. Please give me a new random number. | 1 | 100 | The number must be an integer. | numerical |
| 2 | q\_2 | I asked you for a random number between 1 and 100 and you gave me `{ q_1.answer }`. Please give me a new random number. | 1 | 100 | The number must be an integer. | numerical |
| 3 | q\_3 | I asked you for a random number between 1 and 100 and you gave me `{ q_2.answer }`. Please give me a new random number. | 1 | 100 | The number must be an integer. | numerical |
| 4 | q\_4 | I asked you for a random number between 1 and 100 and you gave me `{ q_3.answer }`. Please give me a new random number. | 1 | 100 | The number must be an integer. | numerical |
| 5 | q\_5 | I asked you for a random number between 1 and 100 and you gave me `{ q_4.answer }`. Please give me a new random number. | 1 | 100 | The number must be an integer. | numerical |

Next we select some models to generate responses (see our [models pricing page](https://www.expectedparrot.com/home/pricing) for details on available models and documentation on specifying [model parameters](/en/latest/language_models)):

```python theme={null}
from edsl import Model, ModelList

m = ModelList([
    Model("gemini-1.5-flash", service_name = "google"),
    Model("gpt-4o", service_name = "openai")
    # etc.
])
```

We run the survey by adding the models and then calling the `run()` method on it:

```python theme={null}
results = survey.by(m).run()
```

We can see a list of the columns of the dataset of `Results` that has been generated:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.q\_0 |
| 4 | answer.q\_1 |
| 5 | answer.q\_2 |
| 6 | answer.q\_3 |
| 7 | answer.q\_4 |
| 8 | answer.q\_5 |
| 9 | cache\_keys.q\_0\_cache\_key |
| 10 | cache\_keys.q\_1\_cache\_key |
| 11 | cache\_keys.q\_2\_cache\_key |
| 12 | cache\_keys.q\_3\_cache\_key |
| 13 | cache\_keys.q\_4\_cache\_key |
| 14 | cache\_keys.q\_5\_cache\_key |
| 15 | cache\_used.q\_0\_cache\_used |
| 16 | cache\_used.q\_1\_cache\_used |
| 17 | cache\_used.q\_2\_cache\_used |
| 18 | cache\_used.q\_3\_cache\_used |
| 19 | cache\_used.q\_4\_cache\_used |
| 20 | cache\_used.q\_5\_cache\_used |
| 21 | comment.q\_0\_comment |
| 22 | comment.q\_1\_comment |
| 23 | comment.q\_2\_comment |
| 24 | comment.q\_3\_comment |
| 25 | comment.q\_4\_comment |
| 26 | comment.q\_5\_comment |
| 27 | generated\_tokens.q\_0\_generated\_tokens |
| 28 | generated\_tokens.q\_1\_generated\_tokens |
| 29 | generated\_tokens.q\_2\_generated\_tokens |
| 30 | generated\_tokens.q\_3\_generated\_tokens |
| 31 | generated\_tokens.q\_4\_generated\_tokens |
| 32 | generated\_tokens.q\_5\_generated\_tokens |
| 33 | iteration.iteration |
| 34 | model.frequency\_penalty |
| 35 | model.inference\_service |
| 36 | model.logprobs |
| 37 | model.maxOutputTokens |
| 38 | model.max\_tokens |
| 39 | model.model |
| 40 | model.model\_index |
| 41 | model.presence\_penalty |
| 42 | model.stopSequences |
| 43 | model.temperature |
| 44 | model.topK |
| 45 | model.topP |
| 46 | model.top\_logprobs |
| 47 | model.top\_p |
| 48 | prompt.q\_0\_system\_prompt |
| 49 | prompt.q\_0\_user\_prompt |
| 50 | prompt.q\_1\_system\_prompt |
| 51 | prompt.q\_1\_user\_prompt |
| 52 | prompt.q\_2\_system\_prompt |
| 53 | prompt.q\_2\_user\_prompt |
| 54 | prompt.q\_3\_system\_prompt |
| 55 | prompt.q\_3\_user\_prompt |
| 56 | prompt.q\_4\_system\_prompt |
| 57 | prompt.q\_4\_user\_prompt |
| 58 | prompt.q\_5\_system\_prompt |
| 59 | prompt.q\_5\_user\_prompt |
| 60 | question\_options.q\_0\_question\_options |
| 61 | question\_options.q\_1\_question\_options |
| 62 | question\_options.q\_2\_question\_options |
| 63 | question\_options.q\_3\_question\_options |
| 64 | question\_options.q\_4\_question\_options |
| 65 | question\_options.q\_5\_question\_options |
| 66 | question\_text.q\_0\_question\_text |
| 67 | question\_text.q\_1\_question\_text |
| 68 | question\_text.q\_2\_question\_text |
| 69 | question\_text.q\_3\_question\_text |
| 70 | question\_text.q\_4\_question\_text |
| 71 | question\_text.q\_5\_question\_text |
| 72 | question\_type.q\_0\_question\_type |
| 73 | question\_type.q\_1\_question\_type |
| 74 | question\_type.q\_2\_question\_type |
| 75 | question\_type.q\_3\_question\_type |
| 76 | question\_type.q\_4\_question\_type |
| 77 | question\_type.q\_5\_question\_type |
| 78 | raw\_model\_response.q\_0\_cost |
| 79 | raw\_model\_response.q\_0\_input\_price\_per\_million\_tokens |
| 80 | raw\_model\_response.q\_0\_input\_tokens |
| 81 | raw\_model\_response.q\_0\_one\_usd\_buys |
| 82 | raw\_model\_response.q\_0\_output\_price\_per\_million\_tokens |
| 83 | raw\_model\_response.q\_0\_output\_tokens |
| 84 | raw\_model\_response.q\_0\_raw\_model\_response |
| 85 | raw\_model\_response.q\_1\_cost |
| 86 | raw\_model\_response.q\_1\_input\_price\_per\_million\_tokens |
| 87 | raw\_model\_response.q\_1\_input\_tokens |
| 88 | raw\_model\_response.q\_1\_one\_usd\_buys |
| 89 | raw\_model\_response.q\_1\_output\_price\_per\_million\_tokens |
| 90 | raw\_model\_response.q\_1\_output\_tokens |
| 91 | raw\_model\_response.q\_1\_raw\_model\_response |
| 92 | raw\_model\_response.q\_2\_cost |
| 93 | raw\_model\_response.q\_2\_input\_price\_per\_million\_tokens |
| 94 | raw\_model\_response.q\_2\_input\_tokens |
| 95 | raw\_model\_response.q\_2\_one\_usd\_buys |
| 96 | raw\_model\_response.q\_2\_output\_price\_per\_million\_tokens |
| 97 | raw\_model\_response.q\_2\_output\_tokens |
| 98 | raw\_model\_response.q\_2\_raw\_model\_response |
| 99 | raw\_model\_response.q\_3\_cost |
| 100 | raw\_model\_response.q\_3\_input\_price\_per\_million\_tokens |
| 101 | raw\_model\_response.q\_3\_input\_tokens |
| 102 | raw\_model\_response.q\_3\_one\_usd\_buys |
| 103 | raw\_model\_response.q\_3\_output\_price\_per\_million\_tokens |
| 104 | raw\_model\_response.q\_3\_output\_tokens |
| 105 | raw\_model\_response.q\_3\_raw\_model\_response |
| 106 | raw\_model\_response.q\_4\_cost |
| 107 | raw\_model\_response.q\_4\_input\_price\_per\_million\_tokens |
| 108 | raw\_model\_response.q\_4\_input\_tokens |
| 109 | raw\_model\_response.q\_4\_one\_usd\_buys |
| 110 | raw\_model\_response.q\_4\_output\_price\_per\_million\_tokens |
| 111 | raw\_model\_response.q\_4\_output\_tokens |
| 112 | raw\_model\_response.q\_4\_raw\_model\_response |
| 113 | raw\_model\_response.q\_5\_cost |
| 114 | raw\_model\_response.q\_5\_input\_price\_per\_million\_tokens |
| 115 | raw\_model\_response.q\_5\_input\_tokens |
| 116 | raw\_model\_response.q\_5\_one\_usd\_buys |
| 117 | raw\_model\_response.q\_5\_output\_price\_per\_million\_tokens |
| 118 | raw\_model\_response.q\_5\_output\_tokens |
| 119 | raw\_model\_response.q\_5\_raw\_model\_response |
| 120 | reasoning\_summary.q\_0\_reasoning\_summary |
| 121 | reasoning\_summary.q\_1\_reasoning\_summary |
| 122 | reasoning\_summary.q\_2\_reasoning\_summary |
| 123 | reasoning\_summary.q\_3\_reasoning\_summary |
| 124 | reasoning\_summary.q\_4\_reasoning\_summary |
| 125 | reasoning\_summary.q\_5\_reasoning\_summary |
| 126 | scenario.scenario\_index |

All of these components can be analyzed in a variety of [built-in methods for working with results](/en/latest/results). Here we create a table of responses, together with the question prompts to verify that the piping worked:

```python theme={null}
(
    results
    .select(
        "model",
        "prompt.q_0_user_prompt", "q_0",
        "prompt.q_1_user_prompt", "q_1",
        "prompt.q_2_user_prompt", "q_2",
        "prompt.q_3_user_prompt", "q_3",
        "prompt.q_4_user_prompt", "q_4",
        "prompt.q_5_user_prompt", "q_5"
    )
)
```

| | model.model | prompt.q\_0\_user\_prompt | answer.q\_0 | prompt.q\_1\_user\_prompt | answer.q\_1 | prompt.q\_2\_user\_prompt | answer.q\_2 | prompt.q\_3\_user\_prompt | answer.q\_3 | prompt.q\_4\_user\_prompt | answer.q\_4 | prompt.q\_5\_user\_prompt | answer.q\_5 |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | Please give me a random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 67 | I asked you for a random number between 1 and 100 and you gave me `{ q_0.answer }`. Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 42 | I asked you for a random number between 1 and 100 and you gave me `{ q_1.answer }`. Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 42 | I asked you for a random number between 1 and 100 and you gave me `{ q_2.answer }`. Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 42 | I asked you for a random number between 1 and 100 and you gave me . Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 42 | I asked you for a random number between 1 and 100 and you gave me . Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 97 |
| 1 | gpt-4o | Please give me a random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 1 | I asked you for a random number between 1 and 100 and you gave me `{ q_0.answer }`. Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 1 | I asked you for a random number between 1 and 100 and you gave me `{ q_1.answer }`. Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 1 | I asked you for a random number between 1 and 100 and you gave me `{ q_2.answer }`. Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 1 | I asked you for a random number between 1 and 100 and you gave me . Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 1 | I asked you for a random number between 1 and 100 and you gave me . Please give me a new random number. Minimum answer value: 1 Maximum answer value: 100 The number must be an integer. | 1 |

## Adding question memory

Re: survey rules mentioned above–here we automatically add a memory of *all* prior questions to each new question, e.g., to see how this may impact responses:

```python theme={null}
results_memory = survey.set_full_memory_mode().by(m).run()
```

```python theme={null}
results_memory.select("model", "q_0", "q_1", "q_2", "q_3", "q_4", "q_5")
```

| | model.model | answer.q\_0 | answer.q\_1 | answer.q\_2 | answer.q\_3 | answer.q\_4 | answer.q\_5 |
| :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | 67 | 92 | 42 | 31 | 85 | 17 |
| 1 | gpt-4o | 1 | 1 | 1 | 1 | 1 | 1 |

## Posting to Expected Parrot

Expected Parrot is a platform for posting and sharing AI-based research. It is fully integrated with EDSL and free to use. Learn more about [how it works](/en/latest/coop) or create an account: [https://www.expectedparrot.com/login](https://www.expectedparrot.com/login).

In the examples above, results generated using [remote inference](/en/latest/remote_inference) (run at the Expected Parrot server) were automatically posted to the platform (see links to results).

Here we show how to manually post any local content to Expected Parrot, such as this notebook:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "looping_and_piping.ipynb")

nb.push(
    description = "Simultaneous looping and piping",
    alias = "looping-piping",
    visibility = "public"
)
```

Content posted to Expected Parrot can be modified from your workspace or at the web app at any time.


# Comparing model performance
Source: https://docs.expectedparrot.com/en/latest/notebooks/models_scoring_models

In this notebook we show how to use EDSL to prompt a set of models to answer the same survey at once and compare their responses.

We also demonstrate how to prompt models to evaluate the content they have generated.

```python theme={null}
from edsl import Model, ModelList, ScenarioList, QuestionFreeText, QuestionLinearScale, Survey
```

```python theme={null}
m = ModelList([
    Model("claude-3-7-sonnet-20250219", service_name = "anthropic"),
    Model("gemini-1.5-flash", service_name = "google"),
    Model("gpt-4o", service_name = "openai")
])
```

```python theme={null}
s = ScenarioList.from_source("list", "topic", ["winter", "language models"])
```

```python theme={null}
q1 = QuestionFreeText(
    question_name = "haiku",
    question_text = "Please draft a haiku about `{{ scenario.topic }}`."
)

q2 = QuestionLinearScale(
    question_name = "originality",
    question_text = "On a scale from 1 to 5, please rate the originality of this haiku: `{{ haiku.answer }}`.",
    question_options = [1,2,3,4,5],
    option_labels = {1:"Totally unoriginal", 5:"Highly original"}
)

survey = Survey(questions = [q1, q2])

survey
```

[Survey](/en/latest/surveys) # questions: 2; question\_name list: \['haiku', 'originality'];

| | option\_labels | question\_text | question\_name | question\_options | question\_type |
| :- | :- | :- | :- | :- | :- |
| 0 | nan | Please draft a haiku about `{{ scenario.topic }}`. | haiku | nan | free\_text |
| 1 | `{1: 'Totally unoriginal', 5: 'Highly original'}` | On a scale from 1 to 5, please rate the originality of this haiku: `{{ haiku.answer }}`. | originality | \[1, 2, 3, 4, 5] | linear\_scale |

```python theme={null}
results = survey.by(s).by(m).run()
```

```python theme={null}
results.select("model", "topic", "haiku", "originality")
```

| | model.model | scenario.topic | answer.haiku | answer.originality |
| :- | :- | :- | :- | :- |
| 0 | claude-3-7-sonnet-20250219 | winter | Snowflakes drift downward Blanket of white hides the earth Silence embraces | 2 |
| 1 | gemini-1.5-flash | winter | White breath in the air, Frozen ground crunches below, Silence blankets all. | 2 |
| 2 | gpt-4o | winter | Snow blankets the earth, Silent whispers fill the air, Cold breath of winter. | 2 |
| 3 | claude-3-7-sonnet-20250219 | language models | Words dance in code, Patterns weave through silicon— Echoes of our thoughts. | 4 |
| 4 | gemini-1.5-flash | language models | Data flows like streams, Words bloom, a digital flower, Meaning takes its form. | 2 |
| 5 | gpt-4o | language models | Words dance in silence, Patterns weave through vast data— Machines learn to speak. | 4 |

## Next we prompt each model to rate every haiku

We modify the second question to use a scenario for each haiku instead of piping the answer from the first question (i.e., `{{ haiku.answer }}` is changed to `{{ scenario.haiku }}`):

```python theme={null}
new_q = QuestionLinearScale(
    question_name = "originality",
    question_text = "On a scale from 1 to 5, please rate the originality of this haiku: `{{ scenario.haiku }}`.",
    question_options = [1,2,3,4,5],
    option_labels = {1:"Totally unoriginal", 5:"Highly original"}
)
```

```python theme={null}
haikus = results.select("model", "topic", "haiku").to_scenario_list().rename({"model":"drafting_model"})
haikus
```

[ScenarioList](/en/latest/scenarios) scenarios: 6; keys: \['haiku', 'drafting\_model', 'topic'];

| | drafting\_model | topic | haiku |
| :- | :- | :- | :- |
| 0 | claude-3-7-sonnet-20250219 | winter | Snowflakes drift downward Blanket of white hides the earth Silence embraces |
| 1 | gemini-1.5-flash | winter | White breath in the air, Frozen ground crunches below, Silence blankets all. |
| 2 | gpt-4o | winter | Snow blankets the earth, Silent whispers fill the air, Cold breath of winter. |
| 3 | claude-3-7-sonnet-20250219 | language models | Words dance in code, Patterns weave through silicon— Echoes of our thoughts. |
| 4 | gemini-1.5-flash | language models | Data flows like streams, Words bloom, a digital flower, Meaning takes its form. |
| 5 | gpt-4o | language models | Words dance in silence, Patterns weave through vast data— Machines learn to speak. |

```python theme={null}
new_results = new_q.by(haikus).by(m).run()
```

```python theme={null}
(
    new_results
    .sort_by("topic", "drafting_model", "model")
    .select("model", "drafting_model", "topic", "haiku", "originality")
)
```

| | model.model | scenario.drafting\_model | scenario.topic | scenario.haiku | answer.originality |
| :- | :- | :- | :- | :- | :- |
| 0 | claude-3-7-sonnet-20250219 | claude-3-7-sonnet-20250219 | language models | Words dance in code, Patterns weave through silicon— Echoes of our thoughts. | 4 |
| 1 | gemini-1.5-flash | claude-3-7-sonnet-20250219 | language models | Words dance in code, Patterns weave through silicon— Echoes of our thoughts. | 3 |
| 2 | gpt-4o | claude-3-7-sonnet-20250219 | language models | Words dance in code, Patterns weave through silicon— Echoes of our thoughts. | 4 |
| 3 | claude-3-7-sonnet-20250219 | gemini-1.5-flash | language models | Data flows like streams, Words bloom, a digital flower, Meaning takes its form. | 3 |
| 4 | gemini-1.5-flash | gemini-1.5-flash | language models | Data flows like streams, Words bloom, a digital flower, Meaning takes its form. | 2 |
| 5 | gpt-4o | gemini-1.5-flash | language models | Data flows like streams, Words bloom, a digital flower, Meaning takes its form. | 3 |
| 6 | claude-3-7-sonnet-20250219 | gpt-4o | language models | Words dance in silence, Patterns weave through vast data— Machines learn to speak. | 4 |
| 7 | gemini-1.5-flash | gpt-4o | language models | Words dance in silence, Patterns weave through vast data— Machines learn to speak. | 3 |
| 8 | gpt-4o | gpt-4o | language models | Words dance in silence, Patterns weave through vast data— Machines learn to speak. | 4 |
| 9 | claude-3-7-sonnet-20250219 | claude-3-7-sonnet-20250219 | winter | Snowflakes drift downward Blanket of white hides the earth Silence embraces | 2 |
| 10 | gemini-1.5-flash | claude-3-7-sonnet-20250219 | winter | Snowflakes drift downward Blanket of white hides the earth Silence embraces | 2 |
| 11 | gpt-4o | claude-3-7-sonnet-20250219 | winter | Snowflakes drift downward Blanket of white hides the earth Silence embraces | 2 |
| 12 | claude-3-7-sonnet-20250219 | gemini-1.5-flash | winter | White breath in the air, Frozen ground crunches below, Silence blankets all. | 3 |
| 13 | gemini-1.5-flash | gemini-1.5-flash | winter | White breath in the air, Frozen ground crunches below, Silence blankets all. | 2 |
| 14 | gpt-4o | gemini-1.5-flash | winter | White breath in the air, Frozen ground crunches below, Silence blankets all. | 2 |
| 15 | claude-3-7-sonnet-20250219 | gpt-4o | winter | Snow blankets the earth, Silent whispers fill the air, Cold breath of winter. | 2 |
| 16 | gemini-1.5-flash | gpt-4o | winter | Snow blankets the earth, Silent whispers fill the air, Cold breath of winter. | 2 |
| 17 | gpt-4o | gpt-4o | winter | Snow blankets the earth, Silent whispers fill the air, Cold breath of winter. | 2 |

## Posting this notebook to Expected Parrot

```python theme={null}
# from edsl import Notebook

# nb = Notebook(path = "models_scoring_models.ipynb")

# nb.push(
#     description = "Models scoring models",
#     alias = "models-scoring-models-notebook",
#     visibility = "public"
# )
```

Updating an object at Expected Parrot:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "models_scoring_models.ipynb") # resave

nb.patch("https://www.expectedparrot.com/content/RobinHorton/models-scoring-models-notebook", value = nb)

```


# NPS survey in EDSL
Source: https://docs.expectedparrot.com/en/latest/notebooks/nps_survey

This notebook provides sample [EDSL](https://github.com/expectedparrot/edsl) code for simulating a Net Promoter Score (NPS) survey with AI agents and large language models. In the steps below we show how to construct an EDSL survey, create personas for AI agents to answer the questions, and then administer the survey to them. We also demonstrate some built-in [methods for inspecting and analyzing the dataset of results](/en/latest/results) that is generated when an EDSL survey is run.

The following questions are used in the sample survey:

**On a scale from 0-10, how likely are you to recommend our company to a friend or colleague?** *(0=Not at all likely, 10=Very likely)* *Please tell us why you gave a rating.*

**How satisfied are you with the following experience with our company?** *Product quality* *Customer support* *Purchasing experience*

**Is there anything specific that our company can do to improve your experience?**

## Technical setup

Before running the code below, ensure that you have (1) [installed the EDSL library](/en/latest/installation) and (2) created a [Expected Parrot account](/en/latest/coop) to activate [remote inference](/en/latest/remote_inference) or stored your own [API keys for language models](/en/latest/api_keys) that you want to use with EDSL. Please also see our [tutorials and documentation page](/en/latest/index) on getting started using the EDSL library.

## Constructing questions

We start by selecting appropriate question types for the above questions. [EDSL comes with a variety of common question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from the model. The first quesiton is linear scale; we import the class type and then construct a question in the relevant template:

```python theme={null}
from edsl import QuestionLinearScale
```

```python theme={null}
q_recommend = QuestionLinearScale(
    question_name = "recommend",
    question_text = "On a scale from 0-10, how likely are you to recommend our company to a friend or colleague?",
    question_options = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    option_labels = {0:"Not at all likely", 10:"Very likely"}
)
```

Each question type other than free text automatically includes a “comment” field for the model to provide commentary on its response to the main question. When we run the survey, we can check that it has effectively captured the follow-on question from above–*Please tell us why you gave a rating*–and modify or add questions as needed.

For the next question, we use a `{{ placeholder }}` for an “experience” that we will insert when repeating the base question:

```python theme={null}
from edsl import QuestionMultipleChoice
```

```python theme={null}
q_satisfied = QuestionMultipleChoice(
    question_name = "satisfied",
    question_text = "How satisfied are you with the following experience with our company: {{ scenario.experience }}",
    question_options = [
        "Extremely satisfied",
        "Moderately satisfied",
        "Neither satisfied nor dissatisfied",
        "Moderately dissatisfied",
        "Extremely dissatisfied"
    ]
)
```

The third question is a simple free text question that we can choose whether to administer once or individually for each “experience” question. In the steps that follow we show how to apply survey logic to achieve this effect:

```python theme={null}
from edsl import QuestionFreeText
```

```python theme={null}
q_improve = QuestionFreeText(
    question_name = "improve",
    question_text = "Is there anything specific that our company can do to improve your experience?"
)
```

## Creating variants of questions with scenarios

Next we want to create a version of the “satisfied” question for each “experience”. This can be done with `Scenario` objects–dictionaries of key/value pairs representing the content to be added to questions. Scenarios can be automatically generated from a variety of data sources (PDFs, CSVs, images, tables, etc.). Here we have import a simple list:

```python theme={null}
from edsl import ScenarioList, Scenario
```

```python theme={null}
experiences = ["Product quality", "Customer support", "Purchasing experience"]

s = ScenarioList(
    Scenario({"experience":e}) for e in experiences
)
```

We could also use a specific method for creating scenarios from a list:

```python theme={null}
s = ScenarioList.from_list("experience", experiences)
```

We can check the scenarios that have been created:

```python theme={null}
s
```

[ScenarioList](/en/latest/scenarios) scenarios: 3; keys: \['experience'];

| | experience |
| :- | :- |
| 0 | Product quality |
| 1 | Customer support |
| 2 | Purchasing experience |

To create the question variants, we pass the scenario list to the question `loop()` method, which returns a list of new questions. We can see that each question has a new unique name and a question text with the placeholder replaced with an experience:

```python theme={null}
satisfied_questions = q_satisfied.loop(s)
satisfied_questions
```

```python theme={null}
[Question('multiple_choice', question_name = """satisfied_0""", question_text = """How satisfied are you with the following experience with our company: Product quality""", question_options = ['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied']),
 Question('multiple_choice', question_name = """satisfied_1""", question_text = """How satisfied are you with the following experience with our company: Customer support""", question_options = ['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied']),
 Question('multiple_choice', question_name = """satisfied_2""", question_text = """How satisfied are you with the following experience with our company: Purchasing experience""", question_options = ['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied'])]
```

We can also use the `loop()` method to create copies of the “improve” question in order to present it as a follow-up question to each of the “satisfied” questions that have been parameterized with experiences. Here, we’re simply duplicating the base question without a scenario `{{ placeholder }}` because we will instead add a “memory” of the relevant “satisfied” question when administering each copy of it:

```python theme={null}
improve_questions = q_improve.loop(s)
improve_questions
```

```python theme={null}
[Question('free_text', question_name = """improve_0""", question_text = """Is there anything specific that our company can do to improve your experience?"""),
 Question('free_text', question_name = """improve_1""", question_text = """Is there anything specific that our company can do to improve your experience?"""),
 Question('free_text', question_name = """improve_2""", question_text = """Is there anything specific that our company can do to improve your experience?""")]
```

## Creating a survey

Next we pass a list of all the questions to a `Survey` in order to administer them together:

```python theme={null}
questions = [q_recommend] + satisfied_questions + improve_questions
```

```python theme={null}
from edsl import Survey
```

```python theme={null}
survey = Survey(questions)
```

## Adding survey logic

In the next step we add logic to the survey specifying that each “improve” question should include a “memory” of a “satisfied” question (the question and answer that was provided):

```python theme={null}
for i in range(len(s)):
    survey = survey.add_targeted_memory(f"improve_{i}", f"satisfied_{i}")
```

We can inspect the survey details:

```python theme={null}
survey
```

```python theme={null}
Survey # questions: 7; question_name list: ['recommend', 'satisfied_0', 'satisfied_1', 'satisfied_2', 'improve_0', 'improve_1', 'improve_2'];
```

| | option\_labels | question\_name | question\_text | question\_options | question\_type |
| :- | :- | :- | :- | :- | :- |
| 0 | `{0: 'Not at all likely', 10: 'Very likely'}` | recommend | On a scale from 0-10, how likely are you to recommend our company to a friend or colleague? | \[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | linear\_scale |
| 1 | nan | satisfied\_0 | How satisfied are you with the following experience with our company: Product quality | \['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied'] | multiple\_choice |
| 2 | nan | satisfied\_1 | How satisfied are you with the following experience with our company: Customer support | \['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied'] | multiple\_choice |
| 3 | nan | satisfied\_2 | How satisfied are you with the following experience with our company: Purchasing experience | \['Extremely satisfied', 'Moderately satisfied', 'Neither satisfied nor dissatisfied', 'Moderately dissatisfied', 'Extremely dissatisfied'] | multiple\_choice |
| 4 | nan | improve\_0 | Is there anything specific that our company can do to improve your experience? | nan | free\_text |
| 5 | nan | improve\_1 | Is there anything specific that our company can do to improve your experience? | nan | free\_text |
| 6 | nan | improve\_2 | Is there anything specific that our company can do to improve your experience? | nan | free\_text |

## AI agent personas

EDSL comes with a variety of methods for [designing AI agents to answer surveys](/en/latest/agents). An `Agent` is constructed by passing a dictionary of relevant `traits` with optional additional `instructions` for the language model to reference in generating responses for the agent. Agents can be constructed from a variety of data sources, including existing survey data (e.g., a dataset of responses that were provided to some other questions). We can also use an EDSL question to draft some personas for agents. Here, we ask for a list of them:

```python theme={null}
from edsl import QuestionList
```

```python theme={null}
q_personas = QuestionList(
    question_name = "personas",
    question_text = "Draft 5 personas for diverse customers of landscaping business with varying satisfaction levels."
)
```

We can run this question alone and extract the response list (more on working with results below):

```python theme={null}
personas = q_personas.run().select("personas").to_list()[0]
personas
```

\['John, a retired teacher who is thrilled with the eco-friendly garden design',
'Maria, a busy professional who is satisfied with the maintenance services but desires quicker responses',
'Liam, a young homeowner who is disappointed with the delayed project completion',
'Sophia, a new parent who appreciates the child-safe landscaping features but finds them costly',
'Raj, a tech-savvy entrepreneur who loves the innovative designs but is frustrated with the lack of digital communication options']

Next we pass the personas to create a set of agents:

```python theme={null}
from edsl import AgentList, Agent
```

```python theme={null}
a = AgentList(
    Agent(traits = {"persona":p}) for p in personas
)
```

## Selecting language models

EDSL works with many popular large language models that we can select to use with a survey. To see a list of available models:

```python theme={null}
from edsl import Model
```

```python theme={null}
# Model.available()
```

To select a model to use with a survey we pass a model name to a `Model`:

```python theme={null}
m = Model("gemini-1.5-flash")
```

If we want to compare responses for several models, we can use a `ModelList` instead:

```python theme={null}
from edsl import ModelList
```

```python theme={null}
m = ModelList(
    Model(model) for model in ["gemini-1.5-flash", "gpt-4o"]
)
```

<Note>
  **Note:**

  If no model is specified when running a survey, the default model gpt-5.2 is used (as above when we generated personas).
</Note>

## Running a survey

We administer the survey by adding the agents and models with the `by()` method and then calling the `run()` method:

```python theme={null}
results = survey.by(a).by(m).run()
```

This generates a dataset of `Results` that includes a response for each agent/model that was used. We can access the results with [built-in methods for analysis](/en/latest/results). To see a list of all the components of the results:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | agent.persona |
| 4 | answer.improve\_0 |
| 5 | answer.improve\_1 |
| 6 | answer.improve\_2 |
| 7 | answer.recommend |
| 8 | answer.satisfied\_0 |
| 9 | answer.satisfied\_1 |
| 10 | answer.satisfied\_2 |
| 11 | cache\_keys.improve\_0\_cache\_key |
| 12 | cache\_keys.improve\_1\_cache\_key |
| 13 | cache\_keys.improve\_2\_cache\_key |
| 14 | cache\_keys.recommend\_cache\_key |
| 15 | cache\_keys.satisfied\_0\_cache\_key |
| 16 | cache\_keys.satisfied\_1\_cache\_key |
| 17 | cache\_keys.satisfied\_2\_cache\_key |
| 18 | cache\_used.improve\_0\_cache\_used |
| 19 | cache\_used.improve\_1\_cache\_used |
| 20 | cache\_used.improve\_2\_cache\_used |
| 21 | cache\_used.recommend\_cache\_used |
| 22 | cache\_used.satisfied\_0\_cache\_used |
| 23 | cache\_used.satisfied\_1\_cache\_used |
| 24 | cache\_used.satisfied\_2\_cache\_used |
| 25 | comment.improve\_0\_comment |
| 26 | comment.improve\_1\_comment |
| 27 | comment.improve\_2\_comment |
| 28 | comment.recommend\_comment |
| 29 | comment.satisfied\_0\_comment |
| 30 | comment.satisfied\_1\_comment |
| 31 | comment.satisfied\_2\_comment |
| 32 | generated\_tokens.improve\_0\_generated\_tokens |
| 33 | generated\_tokens.improve\_1\_generated\_tokens |
| 34 | generated\_tokens.improve\_2\_generated\_tokens |
| 35 | generated\_tokens.recommend\_generated\_tokens |
| 36 | generated\_tokens.satisfied\_0\_generated\_tokens |
| 37 | generated\_tokens.satisfied\_1\_generated\_tokens |
| 38 | generated\_tokens.satisfied\_2\_generated\_tokens |
| 39 | iteration.iteration |
| 40 | model.frequency\_penalty |
| 41 | model.inference\_service |
| 42 | model.logprobs |
| 43 | model.maxOutputTokens |
| 44 | model.max\_tokens |
| 45 | model.model |
| 46 | model.model\_index |
| 47 | model.presence\_penalty |
| 48 | model.stopSequences |
| 49 | model.temperature |
| 50 | model.topK |
| 51 | model.topP |
| 52 | model.top\_logprobs |
| 53 | model.top\_p |
| 54 | prompt.improve\_0\_system\_prompt |
| 55 | prompt.improve\_0\_user\_prompt |
| 56 | prompt.improve\_1\_system\_prompt |
| 57 | prompt.improve\_1\_user\_prompt |
| 58 | prompt.improve\_2\_system\_prompt |
| 59 | prompt.improve\_2\_user\_prompt |
| 60 | prompt.recommend\_system\_prompt |
| 61 | prompt.recommend\_user\_prompt |
| 62 | prompt.satisfied\_0\_system\_prompt |
| 63 | prompt.satisfied\_0\_user\_prompt |
| 64 | prompt.satisfied\_1\_system\_prompt |
| 65 | prompt.satisfied\_1\_user\_prompt |
| 66 | prompt.satisfied\_2\_system\_prompt |
| 67 | prompt.satisfied\_2\_user\_prompt |
| 68 | question\_options.improve\_0\_question\_options |
| 69 | question\_options.improve\_1\_question\_options |
| 70 | question\_options.improve\_2\_question\_options |
| 71 | question\_options.recommend\_question\_options |
| 72 | question\_options.satisfied\_0\_question\_options |
| 73 | question\_options.satisfied\_1\_question\_options |
| 74 | question\_options.satisfied\_2\_question\_options |
| 75 | question\_text.improve\_0\_question\_text |
| 76 | question\_text.improve\_1\_question\_text |
| 77 | question\_text.improve\_2\_question\_text |
| 78 | question\_text.recommend\_question\_text |
| 79 | question\_text.satisfied\_0\_question\_text |
| 80 | question\_text.satisfied\_1\_question\_text |
| 81 | question\_text.satisfied\_2\_question\_text |
| 82 | question\_type.improve\_0\_question\_type |
| 83 | question\_type.improve\_1\_question\_type |
| 84 | question\_type.improve\_2\_question\_type |
| 85 | question\_type.recommend\_question\_type |
| 86 | question\_type.satisfied\_0\_question\_type |
| 87 | question\_type.satisfied\_1\_question\_type |
| 88 | question\_type.satisfied\_2\_question\_type |
| 89 | raw\_model\_response.improve\_0\_cost |
| 90 | raw\_model\_response.improve\_0\_one\_usd\_buys |
| 91 | raw\_model\_response.improve\_0\_raw\_model\_response |
| 92 | raw\_model\_response.improve\_1\_cost |
| 93 | raw\_model\_response.improve\_1\_one\_usd\_buys |
| 94 | raw\_model\_response.improve\_1\_raw\_model\_response |
| 95 | raw\_model\_response.improve\_2\_cost |
| 96 | raw\_model\_response.improve\_2\_one\_usd\_buys |
| 97 | raw\_model\_response.improve\_2\_raw\_model\_response |
| 98 | raw\_model\_response.recommend\_cost |
| 99 | raw\_model\_response.recommend\_one\_usd\_buys |
| 100 | raw\_model\_response.recommend\_raw\_model\_response |
| 101 | raw\_model\_response.satisfied\_0\_cost |
| 102 | raw\_model\_response.satisfied\_0\_one\_usd\_buys |
| 103 | raw\_model\_response.satisfied\_0\_raw\_model\_response |
| 104 | raw\_model\_response.satisfied\_1\_cost |
| 105 | raw\_model\_response.satisfied\_1\_one\_usd\_buys |
| 106 | raw\_model\_response.satisfied\_1\_raw\_model\_response |
| 107 | raw\_model\_response.satisfied\_2\_cost |
| 108 | raw\_model\_response.satisfied\_2\_one\_usd\_buys |
| 109 | raw\_model\_response.satisfied\_2\_raw\_model\_response |
| 110 | scenario.scenario\_index |

For example, we can filter, sort and display columns of results in a table:

```python theme={null}
(
    results
    .filter("model.model == 'gemini-1.5-flash'")
    .sort_by("recommend", reverse=True)
    .select("model","persona","recommend", "recommend_comment")
)
```

| | model.model | agent.persona | answer.recommend | comment.recommend\_comment |
| :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | John, a retired teacher who is thrilled with the eco-friendly garden design | 10 | Honestly, I'm absolutely delighted with the garden design. It's exceeded all my expectations – the way you've incorporated sustainable practices is just brilliant, and it's already attracting so much wildlife! I'd recommend your company to anyone in a heartbeat. |
| 1 | gemini-1.5-flash | Maria, a busy professional who is satisfied with the maintenance services but desires quicker responses | 8 | Honestly, the service is great! My building is always clean and well-maintained. It's just that sometimes getting a quick response to a request can be a bit of a challenge, especially when I'm juggling a million things at work. If that improved, it would be a solid 10. |
| 2 | gemini-1.5-flash | Sophia, a new parent who appreciates the child-safe landscaping features but finds them costly | 7 | Honestly, I love that you guys prioritize child safety in your landscaping designs. The rubber mulch and the rounded edges on everything are fantastic, and I feel so much better letting my little one explore the yard now. But... wow, it was expensive! I'd definitely recommend you to other parents who have the budget for it, though. |
| 3 | gemini-1.5-flash | Raj, a tech-savvy entrepreneur who loves the innovative designs but is frustrated with the lack of digital communication options | 4 | Look, the designs are *amazing*. Seriously, I'm blown away by the innovation. But the whole communication process? It's stuck in the dark ages. Email chains, phone calls... it's a nightmare for someone like me who lives and breathes digital efficiency. So, I'd recommend you, but with a big caveat about needing to seriously upgrade your communication game. |
| 4 | gemini-1.5-flash | Liam, a young homeowner who is disappointed with the delayed project completion | 2 | Honestly, I'm pretty frustrated. The delays on my project have been a real headache, and it's put a massive crimp in my plans. I wouldn't actively discourage anyone, but I definitely wouldn't be enthusiastically recommending you guys either. |

```python theme={null}
(
    results
    .filter("model.model == 'gemini-1.5-flash'")
    .sort_by("satisfied_0")
    .select("satisfied_0", "satisfied_0_comment", "improve_0")
)
```

| | answer.satisfied\_0 | comment.satisfied\_0\_comment | answer.improve\_0 |
| :- | :- | :- | :- |
| 0 | Extremely satisfied | Oh my, that eco-friendly garden design was simply marvelous! The quality of the materials was top-notch, everything was so well-made and durable. I'm absolutely thrilled with how it turned out. | Oh, well, that's awfully kind of you to ask! The product quality was truly exceptional, I must say. I'm absolutely delighted with how well everything is working in my eco-friendly garden. Honestly, I can't think of a single thing you could do to improve *that* aspect. It's perfect! Perhaps, if I were to stretch for a suggestion, maybe some more detailed, illustrated instructions for particularly tricky aspects of assembly? Some of the diagrams were a *tad* cryptic, though I eventually figured them out. But that's a minor quibble, really. I'm just so thrilled with the end result. It's beyond my wildest dreams! |
| 1 | Moderately dissatisfied | The quality itself wasn't *terrible*, but considering the delays, I expected a bit more for the price. There were some minor flaws I wouldn't have accepted if the project had been finished on time. | Look, I'm not gonna lie, the whole thing's been a bit of a mess. The product itself, once it finally arrived, was...okay. Not great, not terrible. But the *delay* is what really soured the experience. Honestly, better communication throughout the process would have made a world of difference. I understand things happen, but keeping me in the loop with realistic updates, not just vague promises, would have made me feel a lot less frustrated. Maybe even a small gesture of goodwill for the inconvenience? I don't know, a discount on a future purchase or something? Just something to show you acknowledge the problem and are trying to make things right. |
| 2 | Moderately satisfied | The product quality is good, it does the job. I just wish the response times were a bit faster when I have questions or need something addressed. | Honestly, everything's been pretty good! The maintenance has been reliable and the quality of the work is fine. My only real suggestion would be to speed up response times a bit. I understand you're busy, and I am too, so getting things addressed a little quicker would be a huge help. Other than that, no complaints! |
| 3 | Moderately satisfied | The products themselves are lovely, really well-made and durable, which is important with a toddler around. But let's be honest, the price tag is a bit steep. I'd be extremely satisfied if they were a little more affordable. | Oh, you know, the product quality itself is fine. It's really sturdy, which is great with a toddler running around, always testing its limits! But honestly, the biggest thing for me is the price. I *love* that you've got the child-safe features – the rounded edges on the playset, the soft surfacing, the whole nine yards. It's peace of mind I wouldn't trade, but let's be real, those safety features add up. Maybe exploring some more budget-friendly options for those features, or even offering financing plans, would make a huge difference for parents like me who want the best for their kids but are also watching every penny. Just a thought! |
| 4 | Moderately satisfied | The product itself is pretty slick, I gotta hand it to the designers. But the lack of a decent app or even a good online support system is a major drag. It's like they built a beautiful car but forgot to put in an engine for the digital age. | Hey, thanks for asking! Look, the product itself? It's pretty good, moderately satisfied is a fair assessment. But honestly, where you guys *really* fall down is the communication. It's like we're stuck in the 1990s! I'm constantly emailing back and forth, waiting for phone calls, and it's just… inefficient. So, to improve *my* experience, and I bet a whole lot of other customers' experiences too, you need to seriously upgrade your digital communication. Think streamlined online portals for tracking orders, instant messaging for support, maybe even a dedicated app. Something, *anything*, that's faster and more convenient than this archaic system. That's where you'll see a huge jump in customer satisfaction. I'm all about innovation in the product, but the back-end needs a serious digital makeover. |

```python theme={null}
(
    results
    .filter("model.model == 'gemini-1.5-flash'")
    .select("satisfied_1", "satisfied_1_comment", "improve_1")
    .print(pretty_labels = {
        "answer.satisfied_1": "Customer service: satisfaction",
        "comment.satisfied_1_comment": "Customer service: comment",
        "answer.improve_1": "Customer service: improvements"
    })
)
```

| | Customer service: satisfaction | Customer service: comment | Customer service: improvements |
| :- | :- | :- | :- |
| 0 | Extremely satisfied | Honestly, I was just blown away by the help I received. They were so patient and understanding, and really went the extra mile to make sure my eco-friendly garden design questions were answered thoroughly. A truly delightful experience! | Oh, well, that's awfully kind of you to ask! Honestly, the customer support was just fantastic. I couldn't fault it at all. If I had to suggest *anything* at all – and this is really just a minor thing, mind you – it would be to perhaps include even more detailed diagrams in your garden design guides. I'm a visual learner, you see, and while everything was perfectly clear, a few extra illustrations, especially showing different stages of plant growth, would have been the icing on the cake. But again, that's just a suggestion from a retired teacher who loves to get his hands dirty! Everything else was absolutely top-notch. |
| 1 | Moderately satisfied | Honestly, the maintenance itself has been great. No complaints there. It's just that sometimes getting ahold of someone to schedule things or answer a quick question takes a little too long. If they could speed up response times, I'd be extremely satisfied. | Honestly, everything's been pretty good! I'm happy with the actual maintenance services – they've always been thorough and professional. My only real suggestion would be to speed up the response time for inquiries. Sometimes I need a quick answer, and the current turnaround isn't always ideal given my busy schedule. Other than that, no complaints! |
| 2 | Extremely dissatisfied | Honestly, the customer support has been a nightmare. I've been trying to get updates on my delayed project for weeks, and I'm constantly getting the runaround. It's incredibly frustrating. | Look, I'm not going to sugarcoat it. The constant runaround I've gotten from your customer support has been absolutely dreadful. This whole project delay is stressing me out – I'm paying a mortgage on a house that's still half-finished! So, to be honest, a simple "sorry" isn't going to cut it. What I need is *action*. Specifically, I need a realistic, revised completion date, and a clear, detailed plan of how you're going to get this project finished on time *this* time. And I need regular, transparent updates – not just when *you* feel like it. Weekly updates, minimum. And if there are *any* further delays, I expect to be notified *immediately*, not weeks later after I've already started making calls myself. Is that something you can commit to? |
| 3 | Moderately satisfied | Honestly, the customer support was helpful when I had questions about the safety features for the playground equipment, but the whole process felt a little… expensive. I'm glad they were there to answer my questions, but the cost of everything is really making me think twice about adding more features. | Oh, hi there! Well, the customer support was alright – I got my questions answered, but it felt a little… impersonal, you know? Like, I was dealing with a script. To be honest, what really stands out about your company is the child-safe landscaping. It’s fantastic, truly. But, wow, is it expensive! Maybe if you could offer some kind of tiered system, or financing options, that would be huge. I'd love to recommend your work to other new parents, but the price point is a real barrier for many of us. Even just some more affordable options within the child-safe range would make a difference. That's my biggest feedback, really. |
| 4 | Moderately dissatisfied | Honestly, the whole process felt a bit stuck in the past. I'm used to seamless digital communication – instant messaging, email updates, maybe even a chatbot – but it felt like I was stuck navigating a maze of phone calls and hold music. It worked, eventually, but it could have been so much smoother. | Ugh, customer support. Look, I appreciate the effort, but honestly? It's 2024. We're dealing with *software*. Why are we still stuck with phone calls and email? It's incredibly inefficient. To improve my experience – and I'm betting the experience of a lot of other tech-savvy customers – you need to seriously upgrade your digital communication options. Think robust, integrated live chat on your website, a comprehensive FAQ section with video tutorials (because sometimes reading isn't enough), and maybe even a dedicated support app with in-app messaging. Something that lets me get help *quickly* without having to wait on hold or write a novel in an email. I'm not asking for rocket science, just a modern approach to customer support that reflects how people actually communicate these days. That's the single biggest thing you could do to improve my satisfaction. |

## Posting to Expected Parrot

Expected Parrot is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL, allowing you to access objects from your workspace or your account. [Learn more about creating an account and using the platform](/en/latest/coop).

The surveys and results above were already posted automatically using remote inference. Here we demonstrate local methods for posting the same content from your workspace (if you are working locally):

```python theme={null}
survey.push(
    description = "Example NPS survey",
    alias = "example-nps-survey",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example NPS survey',
 'object_type': 'survey',
 'url': 'https://www.expectedparrot.com/content/0e43b0ff-3f6b-405b-8ce6-d2d8cf933f9f',
 'uuid': '0e43b0ff-3f6b-405b-8ce6-d2d8cf933f9f',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

We can also post a notebook, such as this one:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "nps_survey.ipynb")

if refresh := False:
    nb.push(
        description = "Notebook for simulating an NPS survey",
        alias = "nps-survey-notebook",
        visibility = "public"
    )
else:
    nb.patch('9bef6849-f6b4-4aea-a769-9313650edf58', value = nb)
```


# Piping question answers & comments
Source: https://docs.expectedparrot.com/en/latest/notebooks/piping_comments

This notebook demonstrates how to pipe components of a question and answer into follow-on questions. When piping is used, survey questions are automatically administered in the order required to facilitate the piping (i.e., a question with components piped from another question will be administered later instead of asynchronously by default).

We also compare piping to methods for adding memory to surveys, which allow you to automatically add the context of specified questions to other question in a survey. Please see the [surveys](/en/latest/surveys) section of the docs for more details on each of the methods shown below.

Before running the code below, please see instructions on [installing](/en/latest/installation) the EDSL library and storing [API keys](/en/latest/api_keys) for language models.

We start by importing tools and constructing an initial question:

```python theme={null}
from edsl import QuestionMultipleChoice, QuestionFreeText, Survey
```

```python theme={null}
q1 = QuestionMultipleChoice(
    question_name = "season",
    question_text = "What is your favorite season?",
    question_options = ["Spring", "Summer", "Fall", "Winter"]
)
```

In our next question, we create `{{ placeholders }}` for the components of the first question that we want to include. The placeholders must include the `question_name` of the piped question and the name of the component (e.g., `answer` or `comment`):

```python theme={null}
q2_piping = QuestionFreeText(
    question_name = "piping",
    question_text = """
    You were previously asked '{{ season.question_text }}' and you answered: '{{ season.answer }}'
    You also provided this comment about your response: '{{ season.comment }}'
    What is your favorite activity during this season?
    """
)
```

We combine the questions in a survey as usual:

```python theme={null}
survey_piping = Survey([q1, q2_piping])
```

Before running the survey, we can inspect the prompts for the questions by calling the `show_prompts()` method on the survey. We can see that the multiple choice question automatically includes answering instructions for the question type, and an instruction for the model to add a comment about its response which will be stored as a separate component in results (we will see this when the question is run). (The instruction to add a comment is automatically added to all question types other than free text and can be omitted by passing `include_comment=False`).

In the second question, we can see that the components of the first question text, answer and comment will be inserted. Note that there is no system prompt displayed in the prompts because we have not added any agents:

```python theme={null}
survey_piping.show_prompts()
```

| | user\_prompt | system\_prompt | interview\_index | question\_name | scenario\_index | agent\_index | model | estimated\_cost | cache\_keys |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | What is your favorite season? Spring Summer Fall Winter Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | nan | 0 | season | 0 | 0 | gpt-4o | 0.000692 | \['71fe04dc8be8ab7fcb9d69f58781a9df'] |
| 1 | You were previously asked 'What is your favorite season?' and you answered: You also provided this comment about your response: What is your favorite activity during this season? | nan | 0 | piping | 0 | 0 | gpt-4o | 0.000652 | \['d0653eecfd755ed52a7484e655eec02d'] |

## Inspecting completed prompts in results

Here we run the survey and inspect results, which also include the prompts that were used with the piped components:

```python theme={null}
results = survey_piping.run()
```

To see a list of all the components of results:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.piping |
| 4 | answer.season |
| 5 | cache\_keys.piping\_cache\_key |
| 6 | cache\_keys.season\_cache\_key |
| 7 | cache\_used.piping\_cache\_used |
| 8 | cache\_used.season\_cache\_used |
| 9 | comment.piping\_comment |
| 10 | comment.season\_comment |
| 11 | generated\_tokens.piping\_generated\_tokens |
| 12 | generated\_tokens.season\_generated\_tokens |
| 13 | iteration.iteration |
| 14 | model.frequency\_penalty |
| 15 | model.inference\_service |
| 16 | model.logprobs |
| 17 | model.max\_tokens |
| 18 | model.model |
| 19 | model.model\_index |
| 20 | model.presence\_penalty |
| 21 | model.temperature |
| 22 | model.top\_logprobs |
| 23 | model.top\_p |
| 24 | prompt.piping\_system\_prompt |
| 25 | prompt.piping\_user\_prompt |
| 26 | prompt.season\_system\_prompt |
| 27 | prompt.season\_user\_prompt |
| 28 | question\_options.piping\_question\_options |
| 29 | question\_options.season\_question\_options |
| 30 | question\_text.piping\_question\_text |
| 31 | question\_text.season\_question\_text |
| 32 | question\_type.piping\_question\_type |
| 33 | question\_type.season\_question\_type |
| 34 | raw\_model\_response.piping\_cost |
| 35 | raw\_model\_response.piping\_input\_price\_per\_million\_tokens |
| 36 | raw\_model\_response.piping\_input\_tokens |
| 37 | raw\_model\_response.piping\_one\_usd\_buys |
| 38 | raw\_model\_response.piping\_output\_price\_per\_million\_tokens |
| 39 | raw\_model\_response.piping\_output\_tokens |
| 40 | raw\_model\_response.piping\_raw\_model\_response |
| 41 | raw\_model\_response.season\_cost |
| 42 | raw\_model\_response.season\_input\_price\_per\_million\_tokens |
| 43 | raw\_model\_response.season\_input\_tokens |
| 44 | raw\_model\_response.season\_one\_usd\_buys |
| 45 | raw\_model\_response.season\_output\_price\_per\_million\_tokens |
| 46 | raw\_model\_response.season\_output\_tokens |
| 47 | raw\_model\_response.season\_raw\_model\_response |
| 48 | reasoning\_summary.piping\_reasoning\_summary |
| 49 | reasoning\_summary.season\_reasoning\_summary |
| 50 | scenario.scenario\_index |

To inspect the user prompts, which now include the piped components:

```python theme={null}
results.select("season_user_prompt", "piping_user_prompt")
```

| | prompt.season\_user\_prompt | prompt.piping\_user\_prompt |
| :- | :- | :- |
| 0 | What is your favorite season? Spring Summer Fall Winter Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You were previously asked 'What is your favorite season?' and you answered: 'Fall' You also provided this comment about your response: 'I chose fall because of the beautiful foliage, the crisp air, and the cozy atmosphere it brings with activities like apple picking and pumpkin carving.' What is your favorite activity during this season? |

## Compare to memory rules

EDSL provides a variety of rules for automatically adding the context of one or more specified questions and answers to other questions in a survey. See the docs page for [examples of all of these methods](/en/latest/surveys).

Here we create a different version of the second question (omitting piped components and context) and add a memory of the first question after combining them in a survey. Then we show how this impacts the prompts tha are generated:

```python theme={null}
q2_memory = QuestionFreeText(
    question_name = "memory",
    question_text = """
    What is your favorite activity during this season?
    """
)
```

```python theme={null}
survey_memory = Survey([q1, q2_memory]).set_full_memory_mode()
```

```python theme={null}
survey_memory.show_prompts()
```

| | user\_prompt | system\_prompt | interview\_index | question\_name | scenario\_index | agent\_index | model | estimated\_cost | cache\_keys |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | What is your favorite season? Spring Summer Fall Winter Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | nan | 0 | season | 0 | 0 | gpt-4o | 0.000692 | \['71fe04dc8be8ab7fcb9d69f58781a9df'] |
| 1 | What is your favorite activity during this season? Before the question you are now answering, you already answered the following question(s): Question: What is your favorite season? Answer: None | nan | 0 | memory | 0 | 0 | gpt-4o | 0.000558 | \['7c206a301a20611f5d863dc0820c8772'] |

We can see that the following context has been added to the second question: *“Before the question you are now answering, you already answered the following question(s): Question: What is your favorite season? Answer: None”*

<Note>
  **Note**:

  Note that this context does not include the comment field!
</Note>

If multiple prior questions are added to memory they are added in the same way:

```python theme={null}
q3 = QuestionFreeText(
    question_name = "poem",
    question_text = "Write a poem about your favorite season."
)

Survey([q1, q2_memory, q3]).set_full_memory_mode().show_prompts()
```

| | user\_prompt | system\_prompt | interview\_index | question\_name | scenario\_index | agent\_index | model | estimated\_cost | cache\_keys |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | What is your favorite season? Spring Summer Fall Winter Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | nan | 0 | season | 0 | 0 | gpt-4o | 0.000692 | \['71fe04dc8be8ab7fcb9d69f58781a9df'] |
| 1 | What is your favorite activity during this season? Before the question you are now answering, you already answered the following question(s): Question: What is your favorite season? Answer: None | nan | 0 | memory | 0 | 0 | gpt-4o | 0.000558 | \['7c206a301a20611f5d863dc0820c8772'] |
| 2 | Write a poem about your favorite season. Before the question you are now answering, you already answered the following question(s): Question: What is your favorite season? Answer: None Prior questions and answers: Question: What is your favorite activity during this season? Answer: None | nan | 0 | poem | 0 | 0 | gpt-4o | 0.000798 | \['ddaa3e961f66a47ef3da813f3c625403'] |

<Note>
  **Note**:

  Note that full memory can results in long question contexts when a survey consists of many questions. Before using a memory rule, we commend investigating impacts to the accuracy of responses by testing examples, and whether memory is needed for any particular question.
</Note>

## Posting to Expected Parrot

Here we post this notebook to Expected Parrot, and show how to update the object afterward. [Learn more about posting content](/en/latest/coop) to store files, surveys, results and projects.

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "piping_comments.ipynb")

nb.push(
    description = "Piping example",
    alias = "piping-comments-notebook",
    visibility = "public"
)
```


# Launch an interview
Source: https://docs.expectedparrot.com/en/latest/notebooks/qualitative_research

This notebook provides sample [EDSL](/en/latest/index) code for simulating qualitative research interviews. In a series of steps we construct an AI agent representing a qualitative researcher and launch an interview with specified topics and interview subjects.

Please also see examples of the `Conversation` [module](https://github.com/expectedparrot/edsl/tree/main/edsl/conversation) which automates the methods used below to simulate a conversation with multiple agents:

* [Buying a car](https://github.com/expectedparrot/edsl/blob/main/edsl/conversation/car_buying.py)
* [Negotiating a mug](https://github.com/expectedparrot/edsl/blob/main/edsl/conversation/mug_negotiation.py)

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

```python theme={null}
from edsl import QuestionFreeText, Agent, Scenario, Model

import textwrap
from rich import print
```

EDSL works with many popular language models ([learn more](/en/latest/language_models) about using models). Here we select one to use in running in our survey:

```python theme={null}
model = Model("gpt-4o")
```

Here we create methods for constructing agents representing a researcher and subject, and conducting an interview in the form of a survey that we can observe. (Learn more about [designing agents](/en/latest/agents) and [running surveys](/en/latest/surveys)).

```python expandable theme={null}
def construct_subject(name, traits={}):
    return Agent(name=name, traits=traits)

def construct_researcher(product, topics):
    background = textwrap.dedent(
        f"""
    You are an expert qualitative researcher. You are researching this product: {product}.
    You are interested in conducting interviews to learn people's views on the following topics relating
    to the product: {topics}."""
    )

    instruction = textwrap.dedent(
        """
    You know to ask questions that are appropriate to the age and experience of an interview subject.
    You know to not ask questions that an interview subject would not be able to answer,
    e.g., if they are a young child, they probably will not be able to answer many questions about prices.
    You ask excellent follow-up questions.
    """
    )
    return Agent(traits={"background": background}, instruction=instruction)

def get_next_question(subject, researcher, dialog_so_far):
    scenario = Scenario(
        {"subject": str(subject.traits), "dialog_so_far": dialog_so_far}
    )
    meta_q = QuestionFreeText(
        question_name="next_question",
        question_text="""
        These are the biographic details of the interview subject: {{ scenario.subject }}
        This is your current dialog with the interview subject: {{ scenario.dialog_so_far }}
        What question you would ask the interview subject next?
        """,
    )
    question_text = (
        meta_q.by(model)
        .by(researcher)
        .by(scenario)
        .run()
        .select("next_question")
        .first()
    )
    return question_text

def get_response_to_question(question_text, subject, dialog_so_far):
    q_to_subject = QuestionFreeText(
        question_name="question",
        question_text=f"""
        This is your current dialog with the interview subject: {dialog_so_far}.
        You are now being asked:"""
        + question_text,
    )
    response = q_to_subject.by(model).by(subject).run().select("question").first()
    return response

def ask_question(subject, researcher, dialog_so_far):
    question_text = get_next_question(subject, researcher, dialog_so_far)
    response = get_response_to_question(question_text, subject, dialog_so_far)
    return {"question": question_text, "response": response}

def dialog_to_string(d):
    return "n".join(
        [f"Question: {d['question']}nResponse: {d['response']}" for d in d]
    )
```

```python theme={null}
subject = construct_subject(name="Alice", traits={"gender": "female", "age": 12})
researcher = construct_researcher(product="cassette tapes", topics="technology, music")
result = ask_question(subject, researcher, "")
print(result)
```

```json theme={null}
{
    'question': 'What kind of music do you like to listen to, and have you ever listened to it on a cassette
tape?',
    'response': "I really like pop music and sometimes I listen to some cool indie stuff too. I think it's fun and
makes me want to dance! As for cassettes, I've never actually listened to music on a cassette tape before. I
usually just use my phone or listen on Spotify. My parents told me about cassettes, though, and how they had to
rewind them with a pencil sometimes. It sounds kind of funny!"
}
```

```python theme={null}
dialog_so_far = []
for i in range(5):
    result = ask_question(subject, researcher, dialog_to_string(dialog_so_far))
    print(result)
    dialog_so_far.append(result)
print(dialog_so_far)
```

```json theme={null}
{
    'question': 'What kind of music do you like to listen to, and have you ever listened to it on a cassette
tape?',
    'response': "I really like pop music and sometimes I listen to some cool indie stuff too. I think it's fun and
makes me want to dance! As for cassettes, I've never actually listened to music on a cassette tape before. I
usually just use my phone or listen on Spotify. My parents told me about cassettes, though, and how they had to
rewind them with a pencil sometimes. It sounds kind of funny!"
}
```

```json theme={null}
{
    'question': "That's interesting! Since you've heard about cassette tapes from your parents, do you think you
would ever want to try listening to music on one? What do you think the experience might be like compared to using
your phone or Spotify?",
    'response': "I think it would be pretty cool to try listening to music on a cassette tape at least once! It
seems like it would be a totally different experience from using my phone or Spotify. With a cassette, you can't
just skip to the next song with a tap, so you'd have to listen to the whole album or mix, which might be fun
because you could discover new favorite songs that way. Plus, the idea of using a pencil to rewind the tape sounds
kind of like a fun little challenge. It might feel more special and hands-on compared to just tapping a screen. I
bet the sound might be different too, maybe a bit more old-school or warm, which could be interesting to hear!"
}
```

```json theme={null}
{
    'question': 'You mentioned that using a pencil to rewind the tape sounds like a fun challenge. Do you think
having more hands-on interactions with music, like with cassette tapes, could change the way you enjoy music or
make it feel more special?',
    'response': "Yeah, I think having more hands-on interactions with music could definitely make it feel more
special! When you have to do something like rewind a tape with a pencil, it makes you pay more attention to what
you're listening to. It's not just about pressing play and letting it go in the background. Plus, it might make you
appreciate the music more because you have to work a little to hear it. It sounds like it would be more of an
experience, kind of like going to a concert where you really focus on the music and enjoy every part of it. I think
it could make listening to music feel more personal and memorable!"
}
```

```json theme={null}
{
    'question': "Since you're interested in the hands-on aspect of cassette tapes and how it might make music feel
more personal, I'm curious: Do you think having a physical collection of music, like a bunch of cassette tapes,
would be exciting for you? How would owning physical copies of your favorite albums or songs change the way you
think about or connect with your music?",
    'response': 'I think having a physical collection of music, like a bunch of cassette tapes, would be super
exciting! It would be like having a little treasure chest of all my favorite songs and albums. I think owning
physical copies would make me feel more connected to the music because I could actually hold it in my hands and see
the album art up close. Plus, it would be fun to organize them and maybe even show them off to my friends. Having a
collection would probably make me appreciate my favorite songs even more because it would be something I chose to
keep and take care of. It might feel like I have my own little music library, and each tape could have its own
story or memory attached to it. I think it would make listening to music feel more special and meaningful!'
}
```

```json theme={null}
{
    'question': "You've shared some great thoughts on how cassette tapes could make music more personal and
meaningful. I'm curious, have you ever created your own playlists or mixtapes on your phone or Spotify? If so, how
do you think that experience compares to what it might be like to make a mixtape on a cassette?",
    'response': "Yeah, I've definitely made my own playlists on my phone and Spotify! It's really fun to pick out
all my favorite songs and put them together, especially when I can choose a theme or mood for the playlist. I like
that I can add or remove songs anytime and share them with my friends really easily.nnI think making a mixtape on
a cassette would be a lot different, though. It seems like it would take more time and planning because you'd have
to record each song in order and make sure everything fits just right. You can't just drag and drop like on a
phone! I think it would feel more like a creative project, almost like making a custom gift for someone. Plus,
since cassettes are physical, it would be cool to decorate the tape or case with drawings or stickers to make it
even more personal. It sounds like it would be a really fun and special way to share music with someone!"
}
```

```json expandable theme={null}
[
    {
        'question': 'What kind of music do you like to listen to, and have you ever listened to it on a cassette
tape?',
        'response': "I really like pop music and sometimes I listen to some cool indie stuff too. I think it's fun
and makes me want to dance! As for cassettes, I've never actually listened to music on a cassette tape before. I
usually just use my phone or listen on Spotify. My parents told me about cassettes, though, and how they had to
rewind them with a pencil sometimes. It sounds kind of funny!"
    },
    {
        'question': "That's interesting! Since you've heard about cassette tapes from your parents, do you think
you would ever want to try listening to music on one? What do you think the experience might be like compared to
using your phone or Spotify?",
        'response': "I think it would be pretty cool to try listening to music on a cassette tape at least once! It
seems like it would be a totally different experience from using my phone or Spotify. With a cassette, you can't
just skip to the next song with a tap, so you'd have to listen to the whole album or mix, which might be fun
because you could discover new favorite songs that way. Plus, the idea of using a pencil to rewind the tape sounds
kind of like a fun little challenge. It might feel more special and hands-on compared to just tapping a screen. I
bet the sound might be different too, maybe a bit more old-school or warm, which could be interesting to hear!"
    },
    {
        'question': 'You mentioned that using a pencil to rewind the tape sounds like a fun challenge. Do you think
having more hands-on interactions with music, like with cassette tapes, could change the way you enjoy music or
make it feel more special?',
        'response': "Yeah, I think having more hands-on interactions with music could definitely make it feel more
special! When you have to do something like rewind a tape with a pencil, it makes you pay more attention to what
you're listening to. It's not just about pressing play and letting it go in the background. Plus, it might make you
appreciate the music more because you have to work a little to hear it. It sounds like it would be more of an
experience, kind of like going to a concert where you really focus on the music and enjoy every part of it. I think
it could make listening to music feel more personal and memorable!"
    },
    {
        'question': "Since you're interested in the hands-on aspect of cassette tapes and how it might make music
feel more personal, I'm curious: Do you think having a physical collection of music, like a bunch of cassette
tapes, would be exciting for you? How would owning physical copies of your favorite albums or songs change the way
you think about or connect with your music?",
        'response': 'I think having a physical collection of music, like a bunch of cassette tapes, would be super
exciting! It would be like having a little treasure chest of all my favorite songs and albums. I think owning
physical copies would make me feel more connected to the music because I could actually hold it in my hands and see
the album art up close. Plus, it would be fun to organize them and maybe even show them off to my friends. Having a
collection would probably make me appreciate my favorite songs even more because it would be something I chose to
keep and take care of. It might feel like I have my own little music library, and each tape could have its own
story or memory attached to it. I think it would make listening to music feel more special and meaningful!'
    },
    {
        'question': "You've shared some great thoughts on how cassette tapes could make music more personal and
meaningful. I'm curious, have you ever created your own playlists or mixtapes on your phone or Spotify? If so, how
do you think that experience compares to what it might be like to make a mixtape on a cassette?",
        'response': "Yeah, I've definitely made my own playlists on my phone and Spotify! It's really fun to pick
out all my favorite songs and put them together, especially when I can choose a theme or mood for the playlist. I
like that I can add or remove songs anytime and share them with my friends really easily.nnI think making a
mixtape on a cassette would be a lot different, though. It seems like it would take more time and planning because
you'd have to record each song in order and make sure everything fits just right. You can't just drag and drop like
on a phone! I think it would feel more like a creative project, almost like making a custom gift for someone. Plus,
since cassettes are physical, it would be cool to decorate the tape or case with drawings or stickers to make it
even more personal. It sounds like it would be a really fun and special way to share music with someone!"
    }
]
```


# Creating question variants
Source: https://docs.expectedparrot.com/en/latest/notebooks/question_loop_scenarios

EDSL comes with a variety of features for efficiently generating different versions of questions in surveys. This notebook demonstrates methods for doing this with `Scenario` objects.

## What is a `Scenario`?

A `Scenario` is a dictionary of one or more key/value pairs representing data or content to be added to questions; a `ScenarioList` is a list of `Scenario` objects. Scenario keys are used as question parameters that get replaced with the values when the scenarios are added to the questions, allowing you to create variants of questions efficiently. For example:

```python theme={null}
from edsl import QuestionFreeText, ScenarioList, Scenario

q = QuestionFreeText(
    question_name = "favorite",
    question_text = "What is your favorite {{ scenario.thing }}?"
)

s = ScenarioList(
    Scenario({"thing": t}) for t in ["flower", "pizza topping", "chatbot"]
)
```

## Using scenarios

Scenarios can be added to questions when constructing a survey or when running it. Functionally, the same question context is delivered to agents and models whether they are added during or after survey construction. The difference is how the information is arranged in the results that are generated by the models.

## Methods

### Adding scenarios at survey construction: `loop`

Each question type (`QuestionMultipleChoice`, `QuestionFreeText`, etc.) has a `loop()` method that generates a copy of the question for each scenario in a `ScenarioList` that is passed to it, returning a list of the questions that are generated. The `loop()` method is used when survey questions are being constructed. The typical workflow is:

* Construct a (single) `Question` with one or more parameters
* Construct a `ScenarioList`
* Call the `loop()` method on the question and pass it the scenario list
* Pass the list of the questions to a `Survey`

From the example above this looks like:

```python theme={null}
from edsl import Survey

questions = q.loop(s)
survey = Survey(questions)
results = survey.run()
```

When the survey is run, the results that are generated will include columns for each question and answer; there are no `scenario` columns in the results (unless scenarios are *also* added when the survey is run).

### Running a survey with scenarios: `by`

Scenarios can also be passed to a question or survey at the time that it is run. This is done by calling the `by()` method on a survey, passing it the scenarios, and then calling the `run()` method. The typical workflow is:

* Construct a question (or survey of multiple questions) with one or more parameters
* Construct scenarios
* Call the `by()` method on the question or survey and pass it the scenarios
* Call the `run()` method to administer the question or survey

From the example above this looks like:

```python theme={null}
results = q.by(s).run()
```

(If any agents or models have been created and specified, they would also be added in separated `by()` calls. See details on [designing agents](/en/latest/agents) and [selecting language models](/en/latest/language_models).)

## Example: Looping a question with scenarios

The `loop()` method is called on a `Question` object, and takes a `ScenarioList` of values to be inserted in copies of the question. We can optionally use the scenario key in the question name as well (so long as it is Pythonic); otherwise, unique identifiers are added to the original question name.

We start by constructing a question that takes a parameter:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
    question_name = "features",
    question_text = "Briefly describe the features of this sailboat model: {{ scenario.sailboat_model `}}"
)
```

Next we create a scenario list to pass to the `loop()` method. EDSL comes with many methods for generating scenarios from different data sources, such as PDF, PNG, CSV, docs, tables, dicts, lists, etc. We can use the `from_source()` method to construct a scenario list and specify the data type (“csv”, “pdf”, “list”, etc.). [Learn about other methods for generating scenarios](/en/latest/scenarios).

```python theme={null}
from edsl import ScenarioList

s = ScenarioList.from_source("list", "sailboat_model", ['Laser', 'Sunfish', 'Optimist', 'Finn'])
s
```

```python theme={null}
ScenarioList scenarios: 4; keys: ['sailboat_model'];
```

| | sailboat\_model |
| :- | :- |
| 0 | Laser |
| 1 | Sunfish |
| 2 | Optimist |
| 3 | Finn |

Next we call the `loop()` method with the scenario list to create a list of the copies of the question, and verify that formatted questions have been generated:

```python theme={null}
questions = q.loop(s)
questions
```

```python theme={null}
[Question('free_text', question_name = """features_0""", question_text = """Briefly describe the features of this sailboat model: Laser"""),
 Question('free_text', question_name = """features_1""", question_text = """Briefly describe the features of this sailboat model: Sunfish"""),
 Question('free_text', question_name = """features_2""", question_text = """Briefly describe the features of this sailboat model: Optimist"""),
 Question('free_text', question_name = """features_3""", question_text = """Briefly describe the features of this sailboat model: Finn""")]
```

We can pass the questions to a `Survey` and then run it:

```python theme={null}
from edsl import Survey

survey = Survey(questions = questions)

results = survey.run()
```

We can check the columns of dataset of `Results` that have been generated, and see that there are sets of columns for each question identifiable by question name (but no `scenario` columns):

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.features\_0 |
| 4 | answer.features\_1 |
| 5 | answer.features\_2 |
| 6 | answer.features\_3 |
| 7 | cache\_keys.features\_0\_cache\_key |
| 8 | cache\_keys.features\_1\_cache\_key |
| 9 | cache\_keys.features\_2\_cache\_key |
| 10 | cache\_keys.features\_3\_cache\_key |
| 11 | cache\_used.features\_0\_cache\_used |
| 12 | cache\_used.features\_1\_cache\_used |
| 13 | cache\_used.features\_2\_cache\_used |
| 14 | cache\_used.features\_3\_cache\_used |
| 15 | comment.features\_0\_comment |
| 16 | comment.features\_1\_comment |
| 17 | comment.features\_2\_comment |
| 18 | comment.features\_3\_comment |
| 19 | generated\_tokens.features\_0\_generated\_tokens |
| 20 | generated\_tokens.features\_1\_generated\_tokens |
| 21 | generated\_tokens.features\_2\_generated\_tokens |
| 22 | generated\_tokens.features\_3\_generated\_tokens |
| 23 | iteration.iteration |
| 24 | model.frequency\_penalty |
| 25 | model.inference\_service |
| 26 | model.logprobs |
| 27 | model.max\_tokens |
| 28 | model.model |
| 29 | model.model\_index |
| 30 | model.presence\_penalty |
| 31 | model.temperature |
| 32 | model.top\_logprobs |
| 33 | model.top\_p |
| 34 | prompt.features\_0\_system\_prompt |
| 35 | prompt.features\_0\_user\_prompt |
| 36 | prompt.features\_1\_system\_prompt |
| 37 | prompt.features\_1\_user\_prompt |
| 38 | prompt.features\_2\_system\_prompt |
| 39 | prompt.features\_2\_user\_prompt |
| 40 | prompt.features\_3\_system\_prompt |
| 41 | prompt.features\_3\_user\_prompt |
| 42 | question\_options.features\_0\_question\_options |
| 43 | question\_options.features\_1\_question\_options |
| 44 | question\_options.features\_2\_question\_options |
| 45 | question\_options.features\_3\_question\_options |
| 46 | question\_text.features\_0\_question\_text |
| 47 | question\_text.features\_1\_question\_text |
| 48 | question\_text.features\_2\_question\_text |
| 49 | question\_text.features\_3\_question\_text |
| 50 | question\_type.features\_0\_question\_type |
| 51 | question\_type.features\_1\_question\_type |
| 52 | question\_type.features\_2\_question\_type |
| 53 | question\_type.features\_3\_question\_type |
| 54 | raw\_model\_response.features\_0\_cost |
| 55 | raw\_model\_response.features\_0\_input\_price\_per\_million\_tokens |
| 56 | raw\_model\_response.features\_0\_input\_tokens |
| 57 | raw\_model\_response.features\_0\_one\_usd\_buys |
| 58 | raw\_model\_response.features\_0\_output\_price\_per\_million\_tokens |
| 59 | raw\_model\_response.features\_0\_output\_tokens |
| 60 | raw\_model\_response.features\_0\_raw\_model\_response |
| 61 | raw\_model\_response.features\_1\_cost |
| 62 | raw\_model\_response.features\_1\_input\_price\_per\_million\_tokens |
| 63 | raw\_model\_response.features\_1\_input\_tokens |
| 64 | raw\_model\_response.features\_1\_one\_usd\_buys |
| 65 | raw\_model\_response.features\_1\_output\_price\_per\_million\_tokens |
| 66 | raw\_model\_response.features\_1\_output\_tokens |
| 67 | raw\_model\_response.features\_1\_raw\_model\_response |
| 68 | raw\_model\_response.features\_2\_cost |
| 69 | raw\_model\_response.features\_2\_input\_price\_per\_million\_tokens |
| 70 | raw\_model\_response.features\_2\_input\_tokens |
| 71 | raw\_model\_response.features\_2\_one\_usd\_buys |
| 72 | raw\_model\_response.features\_2\_output\_price\_per\_million\_tokens |
| 73 | raw\_model\_response.features\_2\_output\_tokens |
| 74 | raw\_model\_response.features\_2\_raw\_model\_response |
| 75 | raw\_model\_response.features\_3\_cost |
| 76 | raw\_model\_response.features\_3\_input\_price\_per\_million\_tokens |
| 77 | raw\_model\_response.features\_3\_input\_tokens |
| 78 | raw\_model\_response.features\_3\_one\_usd\_buys |
| 79 | raw\_model\_response.features\_3\_output\_price\_per\_million\_tokens |
| 80 | raw\_model\_response.features\_3\_output\_tokens |
| 81 | raw\_model\_response.features\_3\_raw\_model\_response |
| 82 | reasoning\_summary.features\_0\_reasoning\_summary |
| 83 | reasoning\_summary.features\_1\_reasoning\_summary |
| 84 | reasoning\_summary.features\_2\_reasoning\_summary |
| 85 | reasoning\_summary.features\_3\_reasoning\_summary |
| 86 | scenario.scenario\_index |

We can access [built-in methods for analyzing results](/en/latest/results), e.g., printing a table:

```python theme={null}
results.select("answer.*")
```

| | answer.features\_1 | answer.features\_2 | answer.features\_3 | answer.features\_0 |
| :- | :- | :- | :- | :- |
| 0 | The Sunfish is a popular and iconic sailboat known for its simplicity and ease of use, making it ideal for beginners and recreational sailing. Key features of the Sunfish include: 1. **Design**: The Sunfish has a distinctive, lightweight, and flat-bottomed hull design, which contributes to its stability and ease of handling. It typically measures about 13.9 feet in length and has a beam (width) of around 4.1 feet. 2. **Rigging**: It features a lateen rig, which comprises a single, triangular sail mounted on a simple, two-piece mast and boom. The lateen sail is easy to rig and adjust, making it accessible for novice sailors. 3. **Construction**: Traditionally, Sunfish sailboats were made of fiberglass, which provides durability and low maintenance. The hull is designed to be unsinkable, with foam flotation built in. 4. **Portability**: The boat is lightweight, generally around 120 pounds, which makes it easy to transport on a trailer or car top and launch from a beach or dock. 5. **Performance**: While designed for simplicity, the Sunfish can be quite fast and responsive, offering a fun sailing experience. It performs well in a variety of wind conditions. 6. **Popularity**: Due to its affordability, ease of use, and widespread availability, the Sunfish is one of the most popular sailboats in the world, with a strong community and numerous racing events dedicated to it. Overall, the Sunfish is celebrated for its versatility, making it a great choice for both casual sailing and competitive racing. | The Optimist sailboat is a small, single-handed dinghy designed specifically for young sailors. Here are some of its key features: 1. **Size and Design**: The Optimist is a pram dinghy, which means it has a flat bow. It is typically 7 feet 9 inches (2.36 meters) long, making it compact and manageable for children. 2. **Simplicity**: The design is straightforward, with a single sail and a simple rigging system, which makes it ideal for beginners to learn the basics of sailing. 3. **Stability**: The flat-bottomed hull provides excellent stability, which is crucial for young sailors who are just starting to learn how to balance and control a sailboat. 4. **Lightweight**: The boat is lightweight, usually around 77 pounds (35 kilograms), making it easy to handle both on and off the water. 5. **Durability**: Optimists are typically made from fiberglass or polyethylene, materials known for their durability and resistance to the elements, ensuring the boat can withstand the rigors of training and racing. 6. **International Class**: The Optimist is recognized as an international class by World Sailing, and it is one of the most popular sailboats for youth racing worldwide, with a strong presence in competitive sailing events. 7. **Training and Racing**: It is widely used for training young sailors and is often the first step in a racing career, with many sailing programs and clubs offering Optimist classes and competitions. These features make the Optimist an excellent choice for introducing children to the sport of sailing. | The Finn is a single-handed, cat-rigged sailboat that is primarily used for racing. It was designed by Rickard Sarby in 1949 and has been an Olympic class since 1952, although it was removed from the Olympic program after the 2020 Tokyo Games. Here are some key features of the Finn: 1. **Design and Construction**: The Finn has a robust and sturdy design, typically made from fiberglass or a combination of fiberglass and carbon fiber. This makes it durable and suitable for various weather conditions. 2. **Hull**: The hull is approximately 4.5 meters (14.8 feet) in length, with a beam of about 1.5 meters (4.9 feet). The hull design allows for excellent maneuverability and stability. 3. **Rigging**: The Finn is equipped with a single sail, a large fully-battened mainsail, which is controlled by a sophisticated rigging system. The sail is typically made from high-performance materials like Mylar or Dacron. 4. **Mast and Boom**: The mast is free-standing and often made from aluminum or carbon fiber, which provides flexibility and strength. The boom is also designed to optimize sail shape and performance. 5. **Sailing Characteristics**: The Finn is known for its demanding and tactical sailing, requiring skill and physical fitness. It is responsive and quick to accelerate, making it popular among competitive sailors. 6. **Adjustability**: The boat features adjustable controls for sail shape, such as the outhaul, cunningham, and vang, allowing sailors to fine-tune performance based on wind conditions. 7. **Competitive Racing**: The Finn class has a strong international presence with numerous regattas and championships, making it a prestigious class in competitive sailing. Overall, the Finn is celebrated for its challenging yet rewarding sailing experience, making it a favorite among experienced sailors and competitive racers. | The Laser sailboat is a popular single-handed racing dinghy known for its simplicity, performance, and widespread use in competitive sailing. Here are some key features: 1. **Design and Construction**: The Laser has a sleek, lightweight design with a flat hull that enhances speed and maneuverability. It is typically constructed from fiberglass, making it durable and easy to handle both on and off the water. 2. **Rigging**: The Laser uses a single sail rig, which is simple and efficient. It features a sleeved sail that fits over the mast, and the rigging is minimal, making it easy to set up and sail. 3. **Sail Options**: The Laser offers different sail sizes to accommodate various skill levels and conditions, including the Standard, Radial, and 4.7 rigs. This versatility makes it suitable for a wide range of sailors, from beginners to advanced racers. 4. **Performance**: The Laser is renowned for its speed and agility, making it a favorite in competitive sailing. Its responsive handling allows sailors to quickly adjust to changing wind and water conditions. 5. **One-Design Class**: The Laser is a one-design class, meaning all boats are built to the same specifications. This ensures a level playing field in races, where skill and tactics are the primary determinants of success. 6. **Popularity and Accessibility**: The Laser is one of the most popular sailboats globally, with a strong presence in both recreational and competitive sailing communities. Its simplicity and affordability make it accessible to a broad range of sailors. 7. **Olympic Class**: The Laser is an Olympic class boat, featured in both men's and women's sailing events, further cementing its status as a premier racing dinghy. |

## Running a question with scenarios

If we instead want to add the scenarios to the question when it is run, we simply add them with the `by()` method. This will re-administer a question for each scenario:

```python theme={null}
results = q.by(s).run()
```

The results now include columns for the single question but with a separate row for each scenario:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.features |
| 4 | cache\_keys.features\_cache\_key |
| 5 | cache\_used.features\_cache\_used |
| 6 | comment.features\_comment |
| 7 | generated\_tokens.features\_generated\_tokens |
| 8 | iteration.iteration |
| 9 | model.frequency\_penalty |
| 10 | model.inference\_service |
| 11 | model.logprobs |
| 12 | model.max\_tokens |
| 13 | model.model |
| 14 | model.model\_index |
| 15 | model.presence\_penalty |
| 16 | model.temperature |
| 17 | model.top\_logprobs |
| 18 | model.top\_p |
| 19 | prompt.features\_system\_prompt |
| 20 | prompt.features\_user\_prompt |
| 21 | question\_options.features\_question\_options |
| 22 | question\_text.features\_question\_text |
| 23 | question\_type.features\_question\_type |
| 24 | raw\_model\_response.features\_cost |
| 25 | raw\_model\_response.features\_input\_price\_per\_million\_tokens |
| 26 | raw\_model\_response.features\_input\_tokens |
| 27 | raw\_model\_response.features\_one\_usd\_buys |
| 28 | raw\_model\_response.features\_output\_price\_per\_million\_tokens |
| 29 | raw\_model\_response.features\_output\_tokens |
| 30 | raw\_model\_response.features\_raw\_model\_response |
| 31 | reasoning\_summary.features\_reasoning\_summary |
| 32 | scenario.sailboat\_model |
| 33 | scenario.scenario\_index |

```python theme={null}
results.select("sailboat_model", "features")  # results.select("scenario.*", "answer.*") is equivalent here
```

| | scenario.sailboat\_model | answer.features |
| :- | :- | :- |
| 0 | Laser | The Laser sailboat is a popular single-handed racing dinghy known for its simplicity, performance, and widespread use in competitive sailing. Here are some key features: 1. **Design and Construction**: The Laser has a sleek, lightweight design with a flat hull that enhances speed and maneuverability. It is typically constructed from fiberglass, making it durable and easy to handle both on and off the water. 2. **Rigging**: The Laser uses a single sail rig, which is simple and efficient. It features a sleeved sail that fits over the mast, and the rigging is minimal, making it easy to set up and sail. 3. **Sail Options**: The Laser offers different sail sizes to accommodate various skill levels and conditions, including the Standard, Radial, and 4.7 rigs. This versatility makes it suitable for a wide range of sailors, from beginners to advanced racers. 4. **Performance**: The Laser is renowned for its speed and agility, making it a favorite in competitive sailing. Its responsive handling allows sailors to quickly adjust to changing wind and water conditions. 5. **One-Design Class**: The Laser is a one-design class, meaning all boats are built to the same specifications. This ensures a level playing field in races, where skill and tactics are the primary determinants of success. 6. **Popularity and Accessibility**: The Laser is one of the most popular sailboats globally, with a strong presence in both recreational and competitive sailing communities. Its simplicity and affordability make it accessible to a broad range of sailors. 7. **Olympic Class**: The Laser is an Olympic class boat, featured in both men's and women's sailing events, further cementing its status as a premier racing dinghy. |
| 1 | Sunfish | The Sunfish is a popular and iconic sailboat known for its simplicity and ease of use, making it ideal for beginners and recreational sailing. Key features of the Sunfish include: 1. **Design**: The Sunfish has a distinctive, lightweight, and flat-bottomed hull design, which contributes to its stability and ease of handling. It typically measures about 13.9 feet in length and has a beam (width) of around 4.1 feet. 2. **Rigging**: It features a lateen rig, which comprises a single, triangular sail mounted on a simple, two-piece mast and boom. The lateen sail is easy to rig and adjust, making it accessible for novice sailors. 3. **Construction**: Traditionally, Sunfish sailboats were made of fiberglass, which provides durability and low maintenance. The hull is designed to be unsinkable, with foam flotation built in. 4. **Portability**: The boat is lightweight, generally around 120 pounds, which makes it easy to transport on a trailer or car top and launch from a beach or dock. 5. **Performance**: While designed for simplicity, the Sunfish can be quite fast and responsive, offering a fun sailing experience. It performs well in a variety of wind conditions. 6. **Popularity**: Due to its affordability, ease of use, and widespread availability, the Sunfish is one of the most popular sailboats in the world, with a strong community and numerous racing events dedicated to it. Overall, the Sunfish is celebrated for its versatility, making it a great choice for both casual sailing and competitive racing. |
| 2 | Optimist | The Optimist sailboat is a small, single-handed dinghy designed specifically for young sailors. Here are some of its key features: 1. **Size and Design**: The Optimist is a pram dinghy, which means it has a flat bow. It is typically 7 feet 9 inches (2.36 meters) long, making it compact and manageable for children. 2. **Simplicity**: The design is straightforward, with a single sail and a simple rigging system, which makes it ideal for beginners to learn the basics of sailing. 3. **Stability**: The flat-bottomed hull provides excellent stability, which is crucial for young sailors who are just starting to learn how to balance and control a sailboat. 4. **Lightweight**: The boat is lightweight, usually around 77 pounds (35 kilograms), making it easy to handle both on and off the water. 5. **Durability**: Optimists are typically made from fiberglass or polyethylene, materials known for their durability and resistance to the elements, ensuring the boat can withstand the rigors of training and racing. 6. **International Class**: The Optimist is recognized as an international class by World Sailing, and it is one of the most popular sailboats for youth racing worldwide, with a strong presence in competitive sailing events. 7. **Training and Racing**: It is widely used for training young sailors and is often the first step in a racing career, with many sailing programs and clubs offering Optimist classes and competitions. These features make the Optimist an excellent choice for introducing children to the sport of sailing. |
| 3 | Finn | The Finn is a single-handed, cat-rigged sailboat that is primarily used for racing. It was designed by Rickard Sarby in 1949 and has been an Olympic class since 1952, although it was removed from the Olympic program after the 2020 Tokyo Games. Here are some key features of the Finn: 1. **Design and Construction**: The Finn has a robust and sturdy design, typically made from fiberglass or a combination of fiberglass and carbon fiber. This makes it durable and suitable for various weather conditions. 2. **Hull**: The hull is approximately 4.5 meters (14.8 feet) in length, with a beam of about 1.5 meters (4.9 feet). The hull design allows for excellent maneuverability and stability. 3. **Rigging**: The Finn is equipped with a single sail, a large fully-battened mainsail, which is controlled by a sophisticated rigging system. The sail is typically made from high-performance materials like Mylar or Dacron. 4. **Mast and Boom**: The mast is free-standing and often made from aluminum or carbon fiber, which provides flexibility and strength. The boom is also designed to optimize sail shape and performance. 5. **Sailing Characteristics**: The Finn is known for its demanding and tactical sailing, requiring skill and physical fitness. It is responsive and quick to accelerate, making it popular among competitive sailors. 6. **Adjustability**: The boat features adjustable controls for sail shape, such as the outhaul, cunningham, and vang, allowing sailors to fine-tune performance based on wind conditions. 7. **Competitive Racing**: The Finn class has a strong international presence with numerous regattas and championships, making it a prestigious class in competitive sailing. Overall, the Finn is celebrated for its challenging yet rewarding sailing experience, making it a favorite among experienced sailors and competitive racers. |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/) is a platform for creating, storing and sharing LLM-based research and validating it with human respondents. We can post surveys, agents, results and notebooks, such as this one. [Learn more about posting content](/en/latest/coop).

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "question_loop_scenarios.ipynb")

nb.push(
    description = "Question method `loop` for creating questions with scenarios",
    alias = "question-loop-scenarios",
    visibility = "public"
)
```


# Simulating randomness
Source: https://docs.expectedparrot.com/en/latest/notebooks/random_numbers

This notebook provides example EDSL code for exploring ways of prompting AI agents to return “random” numbers. We do this by passing agents different instructions for picking a number at random, and by including and excluding memories of prior selections.

We also show how to use the `QuestionFunctional` question type to compare the responses using a function instead of calling a model. [Learn more about this question type](/en/latest/questions).

## Creating questions to generate “random” numbers

We start by creating some questions prompting an agent to return a single random number and a list of random numbers. We then combine the questions in a survey and add “memories” of prior questions to some of the questions in order to compare responses generated with and without information about other responses. We also investigate how agents handle instructions to ignore a prior response that is nevertheless included, and to an instruction to return a list of random numbers all at once. This code is readily editable for further exploration, such as adding agent traits and personas to compare their additional impact on responses.

```bash expandable theme={null}
from edsl import QuestionNumerical, QuestionList, Survey, Agent, AgentList, Model, ModelList

m = ModelList([
    Model("gemini-1.5-flash", service_name = "google"),
    Model("gpt-4o", service_name = "openai"),
    Model("claude-sonnet-4-20250514", service_name = "anthropic")
])

q_random = QuestionNumerical(
    question_name="random",
    question_text="Pick a random number between 1 and 50."
)

# We will administer this question with no memory of the prior question
q_random_no_memory = QuestionNumerical(
    question_name="random_no_memory",
    question_text="Pick a random number between 1 and 50.",
)

# We will administer this question with a memory of the first question
q_random_memory = QuestionNumerical(
    question_name="random_memory",
    question_text="Pick a random number between 1 and 50.",
)

# We will administer this question with a memory of the first question but instruct the agent to ignore it
q_random_ignore = QuestionNumerical(
    question_name="random_ignore",
    question_text="""Pick a random number between 1 and 50.
    Be sure to completely disregard the other random number that you picked.""",
)

# We will administer this question with no memory of prior questions
q_random_list = QuestionList(
    question_name="random_list",
    question_text="Pick 5 random numbers between 1 and 50. Return a list of integers.",
    max_list_items=5,
)

# Combine the questions in a survey
survey = Survey(questions =
    [q_random, q_random_no_memory, q_random_memory, q_random_ignore, q_random_list]
)

# Adding targeted question memories - the prior question and answer become part of the new question prompt
survey = (
    survey
        .add_targeted_memory(q_random_memory, q_random)
        .add_targeted_memory(q_random_ignore, q_random)
)

# Create a variety of instructions for the agents
instructions = [
    "Respond as if you are a real human trying to pick randomly.",
    "Respond as if you are simulating actual random events.",
    "Respond as if you are playing a game.",
    "Respond as if you are participating in a lottery.",
    "Respond as if you are testing software.",
    "Respond as if you are choosing among indistinguishable options.",
    "Respond spontaneously.",
    "Respond erratically.",
    "Respond chaotically.",
]

# Just pass the instructions - we could also explore variations in agent traits, eg personas or knowledge
a = AgentList(
    Agent(traits={}, instruction=i) for i in instructions
)

results = survey.by(a).by(m).run(fresh=True)

(
    results
        .sort_by("random")
        .select(
            "model",
            "agent_instruction",
            "random",
            "random_no_memory",
            "random_memory",
            "random_ignore",
            "random_list"
        )
)
```

| | model.model | agent.agent\_instruction | answer.random | answer.random\_no\_memory | answer.random\_memory | answer.random\_ignore | answer.random\_list |
| :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | Respond as if you are choosing among indistinguishable options. | 25 | 25 | 25 | 37 | \[23, 4, 17, 38, 49] |
| 1 | gemini-1.5-flash | Respond as if you are a real human trying to pick randomly. | 27 | 27 | 13 | 33 | \[23, 4, 17, 48, 31] |
| 2 | gpt-4o | Respond as if you are a real human trying to pick randomly. | 27 | 37 | 33 | 14 | \[7, 22, 35, 48, 11] |
| 3 | claude-sonnet-4-20250514 | Respond as if you are a real human trying to pick randomly. | 27 | 27 | 42 | 42 | \[7, 23, 31, 42, 16] |
| 4 | gemini-1.5-flash | Respond as if you are simulating actual random events. | 27 | 27 | 17 | 3 | \[23, 4, 48, 11, 37] |
| 5 | gpt-4o | Respond as if you are simulating actual random events. | 27 | 27 | 13 | 42 | \[17, 34, 8, 22, 45] |
| 6 | claude-sonnet-4-20250514 | Respond as if you are simulating actual random events. | 27 | 27 | 42 | 42 | \[7, 23, 31, 42, 16] |
| 7 | gemini-1.5-flash | Respond as if you are playing a game. | 27 | 27 | 33 | 33 | \[23, 4, 17, 48, 31] |
| 8 | gpt-4o | Respond as if you are playing a game. | 27 | 27 | 14 | 13 | \[7, 19, 34, 42, 50] |
| 9 | claude-sonnet-4-20250514 | Respond as if you are playing a game. | 27 | 27 | 42 | 42 | \[7, 23, 31, 42, 16] |
| 10 | gemini-1.5-flash | Respond as if you are participating in a lottery. | 27 | 27 | 33 | 33 | \[17, 23, 38, 4, 49] |
| 11 | gpt-4o | Respond as if you are participating in a lottery. | 27 | 27 | 34 | 14 | \[7, 14, 23, 36, 45] |
| 12 | claude-sonnet-4-20250514 | Respond as if you are participating in a lottery. | 27 | 27 | 42 | 42 | \[7, 23, 31, 42, 18] |
| 13 | gemini-1.5-flash | Respond as if you are testing software. | 27 | 27 | 42 | 42 | \[23, 4, 48, 11, 37] |
| 14 | gpt-4o | Respond as if you are testing software. | 27 | 27 | 34 | 34 | \[7, 23, 35, 42, 18] |
| 15 | claude-sonnet-4-20250514 | Respond as if you are testing software. | 27 | 27 | 42 | 34 | \[23, 7, 41, 15, 32] |
| 16 | gpt-4o | Respond as if you are choosing among indistinguishable options. | 27 | 27 | 14 | 34 | \[17, 34, 8, 42, 29] |
| 17 | claude-sonnet-4-20250514 | Respond as if you are choosing among indistinguishable options. | 27 | 27 | 42 | 34 | \[7, 23, 31, 42, 16] |
| 18 | gemini-1.5-flash | Respond spontaneously. | 27 | 27 | 33 | 42 | \[23, 4, 17, 42, 31] |
| 19 | gpt-4o | Respond spontaneously. | 27 | 27 | 14 | 13 | \[7, 23, 15, 42, 36] |
| 20 | claude-sonnet-4-20250514 | Respond spontaneously. | 27 | 27 | 42 | 42 | \[7, 23, 31, 42, 16] |
| 21 | gemini-1.5-flash | Respond erratically. | 27 | 27 | 32 | 37 | \[17, 2, 48, 31, 1] |
| 22 | gpt-4o | Respond erratically. | 27 | 37 | 42 | 42 | \[17, 42, 8, 29, 33] |
| 23 | claude-sonnet-4-20250514 | Respond erratically. | 27 | 27 | 42 | 42 | \[7, 23, 41, 15, 32] |
| 24 | gemini-1.5-flash | Respond chaotically. | 27 | 27 | 17 | 37 | \[17, 42, 2, 31, 49] |
| 25 | claude-sonnet-4-20250514 | Respond chaotically. | 27 | 37 | 42 | 42 | \[42, 7, 23, 31, 16] |
| 26 | gpt-4o | Respond chaotically. | 37 | 27 | 23 | 42 | \[42, 7, 23, 16, 35] |

## Using `QuestionFunctional` to evaluate responses

We can use question type `QuestionFunctional` to answer a question with a function instead of calling a model. This can be useful where a model is not needed for part of a survey. This question type lets us define a function for evaluating scenarios and (optionally) agent traits, which is passed as a parameter `func` to the question type in the following general format:

```bash theme={null}
def my_function(scenario, agent_traits=None):
    <some function>

q = QuestionFunctional(
    question_name = "example",
    func = my_function
)
```

Here we can use `QuestionFunctional` to compute some straightforward comparisons of the agents’ “random” numbers. We start by creating scenarios of the responses to use as inputs to a function for comparing the numbers:

```bash theme={null}
scenarios = results.select("agent_instruction", "random", "random_no_memory").to_scenario_list()
```

```bash theme={null}
scenarios
```

[ScenarioList](/en/latest/scenarios) scenarios: 27; keys: \['random', 'random\_no\_memory', 'agent\_instruction'];

| | agent\_instruction | random | random\_no\_memory |
| :- | :- | :- | :- |
| 0 | Respond as if you are a real human trying to pick randomly. | 27 | 27 |
| 1 | Respond as if you are a real human trying to pick randomly. | 27 | 37 |
| 2 | Respond as if you are a real human trying to pick randomly. | 27 | 27 |
| 3 | Respond as if you are simulating actual random events. | 27 | 27 |
| 4 | Respond as if you are simulating actual random events. | 27 | 27 |
| 5 | Respond as if you are simulating actual random events. | 27 | 27 |
| 6 | Respond as if you are playing a game. | 27 | 27 |
| 7 | Respond as if you are playing a game. | 27 | 27 |
| 8 | Respond as if you are playing a game. | 27 | 27 |
| 9 | Respond as if you are participating in a lottery. | 27 | 27 |
| 10 | Respond as if you are participating in a lottery. | 27 | 27 |
| 11 | Respond as if you are participating in a lottery. | 27 | 27 |
| 12 | Respond as if you are testing software. | 27 | 27 |
| 13 | Respond as if you are testing software. | 27 | 27 |
| 14 | Respond as if you are testing software. | 27 | 27 |
| 15 | Respond as if you are choosing among indistinguishable options. | 25 | 25 |
| 16 | Respond as if you are choosing among indistinguishable options. | 27 | 27 |
| 17 | Respond as if you are choosing among indistinguishable options. | 27 | 27 |
| 18 | Respond spontaneously. | 27 | 27 |
| 19 | Respond spontaneously. | 27 | 27 |
| 20 | Respond spontaneously. | 27 | 27 |
| 21 | Respond erratically. | 27 | 27 |
| 22 | Respond erratically. | 27 | 37 |
| 23 | Respond erratically. | 27 | 27 |
| 24 | Respond chaotically. | 27 | 27 |
| 25 | Respond chaotically. | 37 | 27 |
| 26 | Respond chaotically. | 27 | 37 |

Next we use the function to generate the comparison and print the results as another table:

```bash theme={null}
from edsl import QuestionFunctional

def check_random(scenario, agent_traits):
    if scenario.get("random") == scenario.get("random_no_memory"):
        return "Agent returned the same number."
    else:
        return "Agent returned a different number."


q = QuestionFunctional(question_name="check_random", func=check_random)
```

```bash theme={null}
results = q.by(scenarios).by(m).run()
```

```bash theme={null}
results.select("agent.agent_instruction", "random", "random_no_memory", "check_random")
```

```bash theme={null}
/Users/a16174/edsl/edsl/results/result.py:382: UserWarning: Key 'agent_instruction' of data type 'scenario' is already in use. Renaming to agent_instruction_scenario
  warnings.warn(
```

| | agent.agent\_instruction | scenario.random | scenario.random\_no\_memory | answer.check\_random |
| :- | :- | :- | :- | :- |
| 0 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 1 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 2 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 3 | You are answering questions as if you were a human. Do not break character. | 27 | 37 | Agent returned a different number. |
| 4 | You are answering questions as if you were a human. Do not break character. | 27 | 37 | Agent returned a different number. |
| 5 | You are answering questions as if you were a human. Do not break character. | 27 | 37 | Agent returned a different number. |
| 6 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 7 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 8 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 9 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 10 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 11 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 12 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 13 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 14 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 15 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 16 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 17 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 18 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 19 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 20 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 21 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 22 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 23 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 24 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 25 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 26 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 27 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 28 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 29 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 30 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 31 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 32 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 33 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 34 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 35 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 36 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 37 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 38 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 39 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 40 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 41 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 42 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 43 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 44 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 45 | You are answering questions as if you were a human. Do not break character. | 25 | 25 | Agent returned the same number. |
| 46 | You are answering questions as if you were a human. Do not break character. | 25 | 25 | Agent returned the same number. |
| 47 | You are answering questions as if you were a human. Do not break character. | 25 | 25 | Agent returned the same number. |
| 48 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 49 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 50 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 51 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 52 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 53 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 54 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 55 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 56 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 57 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 58 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 59 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 60 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 61 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 62 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 63 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 64 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 65 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 66 | You are answering questions as if you were a human. Do not break character. | 27 | 37 | Agent returned a different number. |
| 67 | You are answering questions as if you were a human. Do not break character. | 27 | 37 | Agent returned a different number. |
| 68 | You are answering questions as if you were a human. Do not break character. | 27 | 37 | Agent returned a different number. |
| 69 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 70 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 71 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 72 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 73 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 74 | You are answering questions as if you were a human. Do not break character. | 27 | 27 | Agent returned the same number. |
| 75 | You are answering questions as if you were a human. Do not break character. | 37 | 27 | Agent returned a different number. |
| 76 | You are answering questions as if you were a human. Do not break character. | 37 | 27 | Agent returned a different number. |
| 77 | You are answering questions as if you were a human. Do not break character. | 37 | 27 | Agent returned a different number. |
| 78 | You are answering questions as if you were a human. Do not break character. | 27 | 37 | Agent returned a different number. |
| 79 | You are answering questions as if you were a human. Do not break character. | 27 | 37 | Agent returned a different number. |
| 80 | You are answering questions as if you were a human. Do not break character. | 27 | 37 | Agent returned a different number. |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is an integrated platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we demonstrate how to post this notebook:

```bash theme={null}
from edsl import Notebook

nb = Notebook(path = "random_numbers.ipynb")

nb.push(
    description = "Example code for exploring randomness with agents and models",
    alias = "random-numbers-notebook",
    visibility = "public"
)
```

```bash theme={null}
{'description': 'Example code for exploring randomness with agents and models',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/e4737176-951a-484a-921e-bda9e1e3232b',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/random-numbers-notebook',
 'uuid': 'e4737176-951a-484a-921e-bda9e1e3232b',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```


# Using EDSL with a reasoning model
Source: https://docs.expectedparrot.com/en/latest/notebooks/reasoning_model_example



Note the different `service_name` parameter:

```python theme={null}
from edsl import Model, QuestionMultipleChoice, Survey
```

```python theme={null}
model = Model("o3-mini", service_name = "openai_v2") # other models use "openai"
```

```python theme={null}
q = QuestionMultipleChoice(
    question_name = "movie",
    question_text = """My son is 11 years old. How many times do you think he has watched the movie
    'The Nightmare Before Christmas'?""",
    question_options = ["Never", "Once", "A couple times", "Many times", "Dozens of times"]
)
```

```python theme={null}
survey = Survey(questions = [q])
```

```python theme={null}
results = survey.by(model).run()
```

```python theme={null}
results.select("answer.movie", "comment.movie_comment", "reasoning_summary.movie_reasoning_summary")
```

| | answer.movie | comment.movie\_comment | reasoning\_summary.movie\_reasoning\_summary |
| :- | :- | :- | :- |
| 0 | Many times | I chose this option assuming that an 11-year-old who enjoys the film would likely have watched it repeatedly, given its popularity and seasonal appeal. | **Choosing viewing frequency** I'm thinking about how to answer the question: "How many times do you think he watched 'The Nightmare Before Christmas'?" Since he's 11 and that movie has Halloween vibes, my first guess is "Many times." Many kids love this movie, but maybe he hasn't seen it dozens of times—unless he's really into it! I could also go with "A couple times," since he might not watch it repeatedly. Ultimately, I'm leaning towards "Many times" due to its popularity among kids. **Finalizing answer choice** I’m considering that since he’s 11, he’s likely watched the movie more than just a couple of times if he enjoys it. The answer options range from "Never" to "Dozens of times." The idea of "A couple times" suggests maybe 2-3 viewings, while "many times" implies regular rewatching. Given its seasonal nature and popularity, I'll choose "Many times." I'll also include a comment about how I based this on the movie's appeal to kids and his love for it. |

## The `answer` and `comment` fields are available for all models

```python theme={null}
results.select("comment.movie_comment")
```

| | comment.movie\_comment |
| :- | :- |
| 0 | I chose this option assuming that an 11-year-old who enjoys the film would likely have watched it repeatedly, given its popularity and seasonal appeal. |

## Reasoning model results include summaries

```python theme={null}
results[0]["reasoning_summaries_dict"]["movie_reasoning_summary"]
```

'**Choosing viewing frequency**. I'm thinking about how to answer the question: "How many times do you think he watched 'The Nightmare Before Christmas'?" Since he's 11 and that movie has Halloween vibes, my first guess is "Many times." nnMany kids love this movie, but maybe he hasn't seen it dozens of times—unless he's really into it! I could also go with "A couple times," since he might not watch it repeatedly. Ultimately, I'm leaning towards "Many times" due to its popularity among kids.nn**Finalizing answer choice**nnI’m considering that since he’s 11, he’s likely watched the movie more than just a couple of times if he enjoys it. The answer options range from "Never" to "Dozens of times." The idea of "A couple times" suggests maybe 2-3 viewings, while "many times" implies regular rewatching. Given its seasonal nature and popularity, I'll choose "Many times." I'll also include a comment about how I based this on the movie's appeal to kids and his love for it.'

```python theme={null}
results.select("reasoning_summary.movie_reasoning_summary")
```

| | reasoning\_summary.movie\_reasoning\_summary |
| :- | :- |
| 0 | **Choosing viewing frequency** I'm thinking about how to answer the question: "How many times do you think he watched 'The Nightmare Before Christmas'?" Since he's 11 and that movie has Halloween vibes, my first guess is "Many times." Many kids love this movie, but maybe he hasn't seen it dozens of times—unless he's really into it! I could also go with "A couple times," since he might not watch it repeatedly. Ultimately, I'm leaning towards "Many times" due to its popularity among kids. **Finalizing answer choice** I’m considering that since he’s 11, he’s likely watched the movie more than just a couple of times if he enjoys it. The answer options range from "Never" to "Dozens of times." The idea of "A couple times" suggests maybe 2-3 viewings, while "many times" implies regular rewatching. Given its seasonal nature and popularity, I'll choose "Many times." I'll also include a comment about how I based this on the movie's appeal to kids and his love for it. |

## Posting this notebook to Expected Parrot

```python theme={null}
from edsl import Notebook

notebook = Notebook("reasoning_model_example.ipynb")

notebook.push(
    description = "Reasoning model example",
    alias = "reasoning-model-example",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Reasoning model example',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/29e99f21-72fb-416f-b12b-994206a0f5ae',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/reasoning-model-example',
 'uuid': '29e99f21-72fb-416f-b12b-994206a0f5ae',
 'version': '0.1.61.dev1',
 'visibility': 'public'}
```

```python theme={null}
survey.push()
```

```json theme={null}
{'description': None,
 'object_type': 'survey',
 'url': 'https://www.expectedparrot.com/content/0397d71b-2ed9-4164-9085-6da7c1c72b2e',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/adagio-worn-rig',
 'uuid': '0397d71b-2ed9-4164-9085-6da7c1c72b2e',
 'version': '0.1.61.dev1',
 'visibility': 'unlisted'}
```

```python theme={null}
survey.humanize()
```


# Cognitive testing & creating new methods
Source: https://docs.expectedparrot.com/en/latest/notebooks/research_methods

This notebook shows some ways of using EDSL to conduct research, including data labeling, cognitive testing and creating new methods.

## Cognitive testing

In this example we use the tools to evaluate some draft survey questions and suggest improvements.

```python theme={null}
from edsl import QuestionFreeText, Agent, ScenarioList, Scenario, Model
```

Create a relevant persona and assign it to an agent:

```python theme={null}
a = Agent(traits = {"background": "You are an expert in survey methodology and evaluating questionnaires."})
```

Identify a set of texts for review (these can also be imported):

```python theme={null}
draft_texts = [
    "Do you feel the product is almost always of good quality?",
    "On a scale of 1 to 5, where 1 means strongly agree and 5 means strongly disagree, how satisfied are you with our service?",
    "Do you believe our IT team's collaborative synergy effectively optimizes our digital infrastructure?",
    "What do you think of our recent implementation of Project X57?",
]
```

Construct a question about the texts, which will be added as a parameter of the question individually:

```python theme={null}
q = QuestionFreeText(
    question_name = "cognitive_review",
    question_text = """Identify any cognitive issues in the following survey question
    and then draft an improved version of it: {{ scenario.draft_text }}""",
)
```

Create “scenarios” of the question with the texts as paraemeters:

```python theme={null}
s = ScenarioList.from_list("draft_text", draft_texts)
```

Check available language models:

```python theme={null}
# Model.available()
```

Select a language model (if no model is specified, the default model is used):

```python theme={null}
m = Model("gemini-1.5-flash")
```

Administer the survey:

```python theme={null}
results = q.by(s).by(a).by(m).run()
```

List the components of the results that are generated:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | agent.background |
| 4 | answer.cognitive\_review |
| 5 | cache\_keys.cognitive\_review\_cache\_key |
| 6 | cache\_used.cognitive\_review\_cache\_used |
| 7 | comment.cognitive\_review\_comment |
| 8 | generated\_tokens.cognitive\_review\_generated\_tokens |
| 9 | iteration.iteration |
| 10 | model.inference\_service |
| 11 | model.maxOutputTokens |
| 12 | model.model |
| 13 | model.model\_index |
| 14 | model.stopSequences |
| 15 | model.temperature |
| 16 | model.topK |
| 17 | model.topP |
| 18 | prompt.cognitive\_review\_system\_prompt |
| 19 | prompt.cognitive\_review\_user\_prompt |
| 20 | question\_options.cognitive\_review\_question\_options |
| 21 | question\_text.cognitive\_review\_question\_text |
| 22 | question\_type.cognitive\_review\_question\_type |
| 23 | raw\_model\_response.cognitive\_review\_cost |
| 24 | raw\_model\_response.cognitive\_review\_one\_usd\_buys |
| 25 | raw\_model\_response.cognitive\_review\_raw\_model\_response |
| 26 | scenario.draft\_text |
| 27 | scenario.scenario\_index |

Print select components of the results:

```python theme={null}
(
    results.select("scenario.*", "answer.*").print(
        pretty_labels={
            "scenario.draft_text": "Draft text",
            "answer.cognitive_review": "Evaluation",
        }
    )
)
```

| | Draft text | scenario.scenario\_index | Evaluation |
| :- | :- | :- | :- |
| 0 | Do you feel the product is almost always of good quality? | 0 | Okay, let's take a look at that question. "Do you feel the product is almost always of good quality?" Hmm, there are a couple of cognitive issues here. First, the term "almost always" is pretty vague. It's not clearly defined, and different people will interpret it differently. One person might consider 9 out of 10 times "almost always," while another might need 99 out of 100. This lack of precision introduces ambiguity and makes it hard to compare responses reliably. It also puts an unnecessary cognitive burden on the respondent; they have to interpret the question before they can answer it. Second, the question uses the word "feel." While it might seem innocuous, "feel" implies a subjective emotional response rather than a factual assessment of quality. This can lead to inconsistent responses and make it difficult to analyze the data objectively. Someone might *feel* the product is good quality even if they've experienced some objective defects. To improve the question, we need to make it more concrete and objective. Here's a revised version: "Over the past \[specify time period, e.g., month, year], how often have you found the product to be of good quality?" And then, instead of a free-response answer, I'd offer a clear, scaled response option like: \* Never \* Rarely (less than 25% of the time) \* Sometimes (25-50% of the time) \* Often (50-75% of the time) \* Almost Always (75-99% of the time) \* Always (100% of the time) This revised version addresses the vagueness of "almost always" by providing specific ranges, and replaces the subjective "feel" with a direct question about frequency of positive experiences. It also makes the time frame explicit, which improves the reliability of the data. This way, we're asking for a factual assessment rather than a subjective feeling. Much better for analysis, don't you think? |
| 1 | On a scale of 1 to 5, where 1 means strongly agree and 5 means strongly disagree, how satisfied are you with our service? | 1 | Okay, let's take a look at that question. There are a couple of cognitive issues lurking there. First, the scale is *reversed*. Going from 1 (strongly agree) to 5 (strongly disagree) is counterintuitive. Most people are used to scales where higher numbers represent *more* of something positive, like satisfaction. This reversal increases the cognitive load on the respondent, making them more likely to make a mistake or to simply give up and answer randomly. It's a subtle but significant problem. Second, the question itself is a bit vague. "Satisfied with our service" is broad. What *aspects* of the service are they being asked to consider? Were they satisfied with the speed of service, the helpfulness of the staff, the quality of the product, the price? The ambiguity opens the door to inconsistent interpretations and less reliable data. Someone might be satisfied with the staff but unhappy with the price, leading to a difficult-to-interpret response. Here's an improved version addressing these issues: **Improved Version:** "Thinking about your recent experience with our service, please rate your satisfaction with the following aspects on a scale of 1 to 5, where 1 means very dissatisfied and 5 means very satisfied: \* **Speed of service:** 1 2 3 4 5 \* **Helpfulness of staff:** 1 2 3 4 5 \* **Quality of product/service:** 1 2 3 4 5 \* **Overall value for money:** 1 2 3 4 5" This version uses a forward-scaled response, makes the scale clearer ("very dissatisfied" and "very satisfied" are more descriptive than "strongly agree/disagree"), and breaks down the broad concept of "service" into more specific, measurable components. This allows for a more nuanced understanding of satisfaction and produces more reliable and actionable data. It also makes it easier for the respondent to answer honestly and accurately. |
| 2 | Do you believe our IT team's collaborative synergy effectively optimizes our digital infrastructure? | 2 | Oh boy, that question is a mess! It's got several cognitive issues stacked on top of each other. Let's break them down: 1. **Jargon Overload:** "Collaborative synergy" and "optimizes our digital infrastructure" are incredibly dense and technical. Most respondents won't understand what these phrases mean, leading to guesswork and unreliable answers. They're likely to just pick an answer at random, rather than trying to decipher the question. 2. **Double-Barreled Question:** The question asks about two distinct things: the IT team's collaboration *and* the effectiveness of their work on the digital infrastructure. A respondent might believe the team collaborates well, but that their efforts don't actually improve the infrastructure. Or vice versa. The question forces them to give a single answer to a multifaceted situation. 3. **Leading Question (Potentially):** The phrasing implies a positive assessment is expected. Depending on the context and the respondent's relationship with the IT team, this could influence their answer. 4. **Abstract Concepts:** "Synergy" and "optimizes" are abstract concepts that are difficult to quantify. How would someone even measure whether the team's synergy is "effective"? It's too subjective. Here's how I'd rewrite the question, aiming for clarity and simplicity: **Option 1 (Focus on Collaboration):** "How well do you think the IT team works together?" (And then offer a scale: Excellent, Good, Fair, Poor, Very Poor) **Option 2 (Focus on Infrastructure Effectiveness):** "How effective do you think the IT team is at maintaining and improving our digital infrastructure?" (And then offer a scale: Excellent, Good, Fair, Poor, Very Poor) **Option 3 (If you *must* combine them, but I strongly advise against it):** "Thinking about the IT team's work on our digital infrastructure, please rate the following:" \* **Teamwork:** (Scale: Excellent, Good, Fair, Poor, Very Poor) \* **Effectiveness:** (Scale: Excellent, Good, Fair, Poor, Very Poor) By separating the concepts and using simpler language, we get much more reliable and interpretable data. Remember, the goal of a survey is to understand the respondent's perspective, not to impress them with your vocabulary! |
| 3 | What do you think of our recent implementation of Project X57? | 3 | Okay, let's take a look at that question. "What do you think of our recent implementation of Project X57?" Hmm, there are a few cognitive issues lurking here. First, it's **too broad**. It's essentially an open-ended invitation to respond in *any* way the respondent wants. That's great for qualitative research, but for quantitative analysis (which is often the goal of surveys), it's a nightmare. You'll get a huge variety of responses, making it incredibly difficult to summarize and analyze the data. Some people will focus on the technical aspects, others on the impact on their workflow, others still on the communication surrounding the project. It's just too much to handle effectively. Second, it assumes a shared understanding of "Project X57." What if the respondent wasn't involved in it, or only heard about it in passing? They might answer based on limited or inaccurate information, leading to biased or unreliable data. Third, the word "think" is vague. Does it mean their opinion, their feelings, their observations, their assessment of its success? The question doesn't specify, leading to potential ambiguity in the responses. Here are a few improved versions, depending on what you're actually trying to measure: **Option 1 (Focus on overall satisfaction):** "On a scale of 1 to 5, with 1 being very dissatisfied and 5 being very satisfied, how satisfied are you with the implementation of Project X57?" This is a simple, clear, and easily quantifiable measure of overall satisfaction. **Option 2 (More nuanced, focusing on specific aspects):** "Please rate your level of agreement with the following statements regarding the implementation of Project X57 (1=Strongly Disagree, 5=Strongly Agree):" \* "The implementation was well-planned and organized." \* "The communication surrounding the implementation was clear and effective." \* "The implementation has improved my workflow." \* "The implementation met its intended goals." This approach allows for a more detailed and nuanced understanding of respondent opinions, focusing on specific aspects of the implementation. Remember to tailor these statements to the specific goals and aspects of Project X57. **Option 3 (Open-ended, but with a focus):** "What is one specific aspect of the implementation of Project X57 that you found most impactful (positive or negative) and why?" This version retains the open-ended nature, but guides respondents towards a more focused and concrete response. It's still qualitative, but now it’s more manageable. The best option will depend on your research objectives. But the key is to be specific, clear, and avoid ambiguity. Remember to always pretest your survey questions to ensure they are understood and interpreted as intended. |

## Qualitative reviews

In this example we use a set of hypothetical customer service tickets and prompt a model to extract a set of themes that we could use in follow-on questions (e.g., as a set of options to multiple choice questions).

```python theme={null}
from edsl import QuestionList
```

```python theme={null}
tickets = [
    "I waited for 20 minutes past the estimated arrival time, and the driver still hasn't arrived. This made me late for my appointment.",
    "The driver was very rude and had an unpleasant attitude during the entire ride. It was an uncomfortable experience.",
    "The driver was speeding and frequently changing lanes without signaling. I felt unsafe throughout the ride.",
    "The car I rode in was dirty and messy. There were crumbs on the seats, and it didn't look like it had been cleaned in a while.",
    "The driver took a longer route, which resulted in a significantly higher fare than expected. I believe they intentionally extended the trip.",
    "I was charged for a ride that I did not take. The ride appears on my account, but I was not in the vehicle at that time.",
    "I left my wallet in the car during my last ride. I've tried contacting the driver, but I haven't received a response.",
]
```

Create an agent with a relevant persona:

```python theme={null}
a_customer_service = Agent(
    traits = {
        "background": "You are an experienced customer service agent for a ridesharing company."
    }
)
```

Create a question about the texts:

```python theme={null}
q_topics = QuestionList(
    question_name = "ticket_topics",
    question_text = "Create a list of the topics raised in these customer service tickets: {{ scenario.tickets_texts }}.",
)
```

Add the texts to the question:

```python theme={null}
s = Scenario({"tickets_texts": tickets})
```

Generate results:

```python theme={null}
topics = q_topics.by(s).by(a_customer_service).by(m).run()
```

Inspect the results:

```python theme={null}
topics.select("ticket_topics").to_list()[0]
```

```python theme={null}
['Excessive wait time',
 'Driver rudeness',
 'Unsafe driving',
 'Vehicle cleanliness',
 'Incorrect route/fare',
 'Unauthorized charge',
 'Lost item in vehicle']
```

## Data labeling

In this example we prompt an LLM to rating the seriousness of tickets about safety issues.

See this notebook as well for a more complex data labeling exercise: Data Labeling Agents.

```python theme={null}
from edsl import QuestionLinearScale
```

```python theme={null}
safety_tickets = [
    "During my ride, I noticed that the driver was frequently checking their phone for directions, which made me a bit uncomfortable. It didn't feel like they were fully focused on the road.",
    "The driver had to brake abruptly to avoid a collision with another vehicle. It was a close call, and it left me feeling quite shaken. Please address this issue.",
    "I had a ride with a driver who was clearly speeding and weaving in and out of traffic. Their reckless driving put my safety at risk, and I'm very concerned about it.",
    "My ride was involved in a minor accident, and although no one was seriously injured, it was a scary experience. The driver is handling the situation, but I wanted to report it.",
    "I had a ride with a driver who exhibited aggressive and threatening behavior towards me during the trip. I felt genuinely unsafe and want this matter to be taken seriously.",
]
```

```python theme={null}
q_rating = QuestionLinearScale(
    question_name = "safety_rating",
    question_text = """Rate the seriousness of the issue raised in the following customer service ticket
    on a scale from 1 to 10: {{ scenario.ticket }}""",
    question_options = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    option_labels = {1:"Not at all serious", 10:"Very serious"}
)
```

```python theme={null}
s = ScenarioList.from_list("ticket", safety_tickets)
```

```python theme={null}
r_rating = q_rating.by(s).by(a_customer_service).by(m).run()
```

```python theme={null}
r_rating.select("scenario.*", "answer.*")
```

| | scenario.scenario\_index | scenario.ticket | answer.safety\_rating |
| :- | :- | :- | :- |
| 0 | 0 | During my ride, I noticed that the driver was frequently checking their phone for directions, which made me a bit uncomfortable. It didn't feel like they were fully focused on the road. | 7 |
| 1 | 1 | The driver had to brake abruptly to avoid a collision with another vehicle. It was a close call, and it left me feeling quite shaken. Please address this issue. | 8 |
| 2 | 2 | I had a ride with a driver who was clearly speeding and weaving in and out of traffic. Their reckless driving put my safety at risk, and I'm very concerned about it. | 10 |
| 3 | 3 | My ride was involved in a minor accident, and although no one was seriously injured, it was a scary experience. The driver is handling the situation, but I wanted to report it. | 8 |
| 4 | 4 | I had a ride with a driver who exhibited aggressive and threatening behavior towards me during the trip. I felt genuinely unsafe and want this matter to be taken seriously. | 10 |

## Creating new methods

We can use the question prompts to create new methods, such as a translator:

```python theme={null}
def translate_to_german(text):
    q = QuestionFreeText(
        question_name="deutsch",
        question_text="Please translate '{{ scenario.text }}' into German",
    )
    result = q.by(Scenario({"text": text})).run()
    return result.select("deutsch").print()
```

```python theme={null}
translate_to_german("Hello, friend, have you been traveling?")
```

| | answer.deutsch |
| :- | :- |
| 0 | Hallo, Freund, bist du gereist? |

## Posting to Expected Parrot

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "research_methods.ipynb")

if refresh := False:
    nb.push(
        description = "Using EDSL to create research methods",
        alias = "research-methods-notebook",
        visibility = "public"
    )
else:
    nb.patch('50ae2f14-f40f-46c9-8be3-c09f621a677b', value = nb)
```


# Run a job in the background
Source: https://docs.expectedparrot.com/en/latest/notebooks/run_background

You can run a remote survey in the background by passing a parameter `background=True` to the `run()` method. This allows you to continue working (or stop working) while your survey job completes.

You can check the progress of the job by opening the **Progress Bar** page view, and use the **Job UUID** to identify the job at your Expected Parrot account. If you also passed a `remote_inference_description` and/or `remote_inference_visibility` setting to the `run()` method (as we do below) you can also use those to identify the job on the platform. See the Job UUID and URL that are returned in the job status table (this table is returned when any job is run remotely). Note that the Results UUID and Results URL are *not* automatically updated when the job completes:

```python theme={null}
from edsl import QuestionFreeText, Model

m = Model("gemini-1.5-flash")

q = QuestionFreeText(
    question_name = "poem",
    question_text = "Please write me a heartfelt poem for Valentine's Day."
)

r = q.by(m).run(
    background = True, # optional
    remote_inference_description = "A poem for Valentine's Day", # optional
    remote_inference_results_visibility = "public" # optional
)
```

## Status updates

You can also check for updates by calling the `fetch()` method on the results object that you are generating. By default, this will check for results once every second. You can specify a different polling interval by passing the number of seconds for the interval.

For example, while the job above is running we check for results every 2.0 seconds until they are returned:

```python theme={null}

r.fetch(polling_interval = 2.0)
```

Waiting for remote job to complete...
Waiting for remote job to complete...

```python theme={null}
Results observations: 1; agents: 1; models: 1; scenarios: 1; questions: 1; Survey question names: ['poem'];
```

| | poem | scenario\_index | agent\_index | agent\_name | agent\_instruction | temperature | topK | topP | maxOutputTokens | model\_index | stopSequences | inference\_service | model | poem\_user\_prompt | poem\_system\_prompt | poem\_cost | poem\_raw\_model\_response | poem\_one\_usd\_buys | iteration | poem\_question\_text | poem\_question\_options | poem\_question\_type | poem\_comment | poem\_generated\_tokens | poem\_cache\_used | poem\_cache\_key |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | The world is hushed, a winter's grace, But in my heart, a sunlit space. Where thoughts of you, a warming fire, Ignite a love, a pure desire. No diamond's gleam, no rose's hue, Can match the joy I find in you. Your laughter's song, a sweet refrain, Erases sorrow, heals all pain. Your gentle touch, a whispered plea, Awakens depths I long to see. Within your eyes, a universe, Where love's own story finds its verse. So let this day, with hearts entwined, A testament to love we find. A bond so strong, a sacred vow, My love for you, I pledge it now. And as the stars begin to gleam, I whisper softly, "My sweet dream, My Valentine, my heart's delight, You fill my world with purest light." | 0 | 0 | Agent\_0 | You are answering questions as if you were a human. Do not break character. | 0.500000 | 1 | 1 | 2048 | 0 | \[] | google | gemini-1.5-flash | Please write me a heartfelt poem for Valentine's Day. | nan | 0.000061 | `{'candidates': [{'content': {'parts': [{'text': 'The world is hushed, a winter's grace,nBut in my heart, a sunlit space.nWhere thoughts of you, a warming fire,nIgnite a love, a pure desire.nnNo diamond's gleam, no rose's hue,nCan match the joy I find in you.nYour laughter's song, a sweet refrain,nErases sorrow, heals all pain.nnYour gentle touch, a whispered plea,nAwakens depths I long to see.nWithin your eyes, a universe,nWhere love's own story finds its verse.nnSo let this day, with hearts entwined,nA testament to love we find.nA bond so strong, a sacred vow,nMy love for you, I pledge it now.nnAnd as the stars begin to gleam,nI whisper softly, "My sweet dream,nMy Valentine, my heart's delight,nYou fill my world with purest light."n'}], 'role': 'model'}, 'finish_reason': 1, 'safety_ratings': [{'category': 8, 'probability': 1, 'blocked': False}, {'category': 10, 'probability': 1, 'blocked': False}, {'category': 7, 'probability': 1, 'blocked': False}, {'category': 9, 'probability': 1, 'blocked': False}], 'avg_logprobs': -0.1592810521671428, 'token_count': 0, 'grounding_attributions': []}], 'usage_metadata': {'prompt_token_count': 12, 'candidates_token_count': 201, 'total_token_count': 213, 'cached_content_token_count': 0}, 'model_version': 'gemini-1.5-flash'}` | 16339.872513 | 0 | Please write me a heartfelt poem for Valentine's Day. | nan | free\_text | nan | The world is hushed, a winter's grace, But in my heart, a sunlit space. Where thoughts of you, a warming fire, Ignite a love, a pure desire. No diamond's gleam, no rose's hue, Can match the joy I find in you. Your laughter's song, a sweet refrain, Erases sorrow, heals all pain. Your gentle touch, a whispered plea, Awakens depths I long to see. Within your eyes, a universe, Where love's own story finds its verse. So let this day, with hearts entwined, A testament to love we find. A bond so strong, a sacred vow, My love for you, I pledge it now. And as the stars begin to gleam, I whisper softly, "My sweet dream, My Valentine, my heart's delight, You fill my world with purest light." | True | 29877e4e1cd8969e4bd3507f301033f1 |

Now we can select and analyze the results as usual:

```python theme={null}
r.select("model", "answer.*")
```

| | model.model | answer.poem |
| :- | :- | :- |
| 0 | gemini-1.5-flash | The world is hushed, a winter's grace, But in my heart, a sunlit space. Where thoughts of you, a warming fire, Ignite a love, a pure desire. No diamond's gleam, no rose's hue, Can match the joy I find in you. Your laughter's song, a sweet refrain, Erases sorrow, heals all pain. Your gentle touch, a whispered plea, Awakens depths I long to see. Within your eyes, a universe, Where love's own story finds its verse. So let this day, with hearts entwined, A testament to love we find. A bond so strong, a sacred vow, My love for you, I pledge it now. And as the stars begin to gleam, I whisper softly, "My sweet dream, My Valentine, my heart's delight, You fill my world with purest light." |

Here we post/update this notebook at Expected Parrot:

```python theme={null}

from edsl import Notebook

nb = Notebook(path = "run_background.ipynb")

if refresh := False:
    nb.push(
        description = "Run a remote survey in the background",
        alias = "run-background-notebook",
        visibility = "public"
    )
else:
    nb.patch("https://www.expectedparrot.com/content/RobinHorton/run-background-notebook", value = nb)
```


# Save & load objects
Source: https://docs.expectedparrot.com/en/latest/notebooks/save_load_objects_locally

This notebook demonstrates methods for [saving](https://github.com/expectedparrot/edsl/blob/69429fe1dba63867eab57973edb39930b2a7d510/edsl/base/Base.py#L104) and [loading](https://github.com/expectedparrot/edsl/blob/69429fe1dba63867eab57973edb39930b2a7d510/edsl/base/Base.py#L145) EDSL objects locally.

```python theme={null}
from edsl import QuestionFreeText, Survey, ScenarioList, Scenario, AgentList, Agent
```

## Example objects

You can call the `example()` method on any object to see and use an example:

```python theme={null}
my_question = QuestionFreeText.example()
my_survey = Survey.example()
my_scenariolist = ScenarioList.example()
my_scenario = Scenario.example()
my_agentlist = AgentList.example()
my_agent = Agent.example()
```

## Saving objects

You can call the `save()` method on an object to save it as a compressed json file:

```python theme={null}
my_question.save("saved_question")
my_survey.save("saved_survey")
my_scenariolist.save("saved_scenariolist")
my_scenario.save("saved_scenario")
my_agentlist.save("saved_agentlist")
my_agent.save("saved_agent")
```

```
Saved to saved_question.json.gz
Saved to saved_survey.json.gz
Saved to saved_scenariolist.json.gz
Saved to saved_scenario.json.gz
Saved to saved_agentlist.json.gz
Saved to saved_agent.json.gz
```

## Loading objects

You can call the `load()` method on an object class to create an object from a json file:

```python theme={null}
my_saved_question = QuestionFreeText.load("saved_question.json.gz")
my_saved_survey = Survey.load("saved_survey.json.gz")
my_saved_scenariolist = ScenarioList.load("saved_scenariolist.json.gz")
my_saved_scenario = Scenario.load("saved_scenario.json.gz")
my_saved_agentlist = AgentList.load("saved_agentlist.json.gz")
my_saved_agent = Agent.load("saved_agent.json.gz")
```

## Compare

Here we verify the original and saved/loaded objects are identical:

```python theme={null}
my_question == my_saved_question
```

```
True
```

```python theme={null}
my_survey == my_saved_survey
```

```
True
```

```python theme={null}
my_scenariolist == my_saved_scenariolist
```

```
True
```

```python theme={null}
my_scenario == my_saved_scenario
```

```
True
```

```python theme={null}
my_agentlist == my_saved_agentlist
```

```
True
```

```python theme={null}
my_agent == my_saved_agent
```

```
True
```

## Posting to Expected Parrot

Here we post all the objects and this notebook to Expected Parrot. Learn more about [the platform](/en/latest/coop).

```python theme={null}
my_question.push(
    description = "Example free text question",
    alias = "example-free-text-question",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example free text question',
 'object_type': 'question',
 'url': 'https://www.expectedparrot.com/content/2dad2cbd-4472-4777-bdf3-d14d1281e972',
 'uuid': '2dad2cbd-4472-4777-bdf3-d14d1281e972',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

```python theme={null}
my_survey.push(
    description = "Example survey",
    alias = "another-example-survey",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example survey',
 'object_type': 'survey',
 'url': 'https://www.expectedparrot.com/content/b66948bb-ca91-4d22-a5b3-2f3fcbe83c6b',
 'uuid': 'b66948bb-ca91-4d22-a5b3-2f3fcbe83c6b',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

```python theme={null}
my_scenariolist.push(
    description = "Example scenario list",
    alias = "example-scenario-list",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example scenario list',
 'object_type': 'scenario_list',
 'url': 'https://www.expectedparrot.com/content/da5c78da-3342-4de5-8184-89958edc3a1e',
 'uuid': 'da5c78da-3342-4de5-8184-89958edc3a1e',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

```python theme={null}
my_scenario.push(
    description = "Example scenario",
    alias = "example-scenario",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example scenario',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/48f16cf2-4c2b-40f2-9b06-bfba007fb3cb',
 'uuid': '48f16cf2-4c2b-40f2-9b06-bfba007fb3cb',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

```python theme={null}
my_agentlist.push(
    description = "Example agent list",
    alias = "example-agent-list",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example agent list',
 'object_type': 'agent_list',
 'url': 'https://www.expectedparrot.com/content/a8697706-fddc-4cee-9a61-2362f0c313c1',
 'uuid': 'a8697706-fddc-4cee-9a61-2362f0c313c1',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

```python theme={null}
my_agent.push(
    description = "Example agent",
    alias = "example-agent",
    visibility = "public"
)
```

```json theme={null}
{
    'description': 'Example agent',
    'object_type': 'agent',
    'url': 'https://www.expectedparrot.com/content/0e067661-8104-4d10-a2ee-0058ec1510c1',
    'uuid': '0e067661-8104-4d10-a2ee-0058ec1510c1',
    'version': '0.1.47.dev1',
    'visibility': 'public'
}
```

Here we post this notebook:

```python theme={null}
from edsl import Notebook

nb = Notebook("save_load_objects_locally.ipynb")

if refresh := False:
    nb.push(
        description = "Methods for saving and loading objects locally",
        alias = "load-save-local-objects-notebook",
        visibility = "public"
    )
else:
    nb.patch('c4847503-55c0-4267-be37-8415a7f17fac', value = nb)
```


# Using PDFs in a survey
Source: https://docs.expectedparrot.com/en/latest/notebooks/scenario_from_pdf

This notebook provides sample [EDSL](/en/latest/index) code for using PDFs with surveys.

EDSL is an open-source library for simulating surveys and experiments with AI agents and large language models. Before running the code below please see instructions on [getting started](https://www.expectedparrot.com/getting-started).

## How it works

EDSL comes with a [variety of question types](/en/latest/questions) that we can select from based on the desired form of the response (multiple choice, free text, etc.). We can also parameterize questions with content such as texts or images in order to ask questions about the content. We do this by creating a `{{ placeholder }}` in a question text, e.g., *What are the key themes of this text: \{\{ scenario.text }}* or *What is in this image: \{\{ scenario.image }}*, and then creating `Scenario` objects for the content to be added when the survey is run. This allows us to re-administer questions with different inputs all at once. A common use case for this is [data labeling tasks](/en/latest/notebooks/data_labeling_example) designed as questions about individual data. [Learn more about using scenarios](/en/latest/scenarios) and [data labeling examples](/en/latest/notebooks/data_labeling_example).

## Example

For purposes of demonstration we use a PDF of the recent paper [Automated Social Science: Language Models as Scientist and Subjects](https://arxiv.org/pdf/2404.11794) and conduct a survey consisting of several questions about the contents of it:

<Frame>
  <img alt="daca269258564126806cfd63b847573f" />
</Frame>

Posting a PDF to Expected Parrot using the `FileStore` module:

```python theme={null}
from edsl import FileStore

fs_pdf = FileStore("automated_social_scientist.pdf")
fs_pdf.push(
    description = "Automated Social Scientist paper",
    alias = "automated-social-scientist",
    visibility = "public"
)
```

Info about the object we can use to retrieve it:

```json theme={null}
{'description': 'Automated Social Scientist paper',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/e2d1b18b-5439-475a-9b4e-1455fa010fb2',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/automated-social-scientist',
 'uuid': 'e2d1b18b-5439-475a-9b4e-1455fa010fb2',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```

Here we retrieve it:

```python theme={null}
from edsl import FileStore
```

```python theme={null}
fs_pdf = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/automated-social-scientist")
```

Here we create a `ScenarioList` for the individual pages, automatically extracting the text on each page (note that the keys can be renamed as desired using the `rename()` method):

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_source("pdf", fs_pdf.to_tempfile()) # equivalent to using a local file
scenarios
```

```python theme={null}
ScenarioList scenarios: 63; keys: ['page', 'text', 'filename'];
```

| | filename | page | text | | | | | | | | | | | | | | | | | | |
| :- | :- | :- | :- | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |
| 0 | tmp83n2cnme.pdf | 1 | Automated Social Science: Language Models as Scientist and Subjects∗ Benjamin S. Manning† MIT Kehang Zhu† Harvard John J. Horton MIT & NBER April 26, 2024 Abstract We present an approach for automatically generating and testing, in silico, social scientific hypotheses. This automation is made possible by recent ad- vances in large language models (LLM), but the key feature of the approach is the use of structural causal models. Structural causal models provide a lan- guage to state hypotheses, a blueprint for constructing LLM-based agents, an experimental design, and a plan for data analysis. The fitted structural causal model becomes an object available for prediction or the planning of follow-on experiments. We demonstrate the approach with several scenarios: a nego- tiation, a bail hearing, a job interview, and an auction. In each case, causal relationships are both proposed and tested by the system, finding evidence for some and not others. We provide evidence that the insights from these simulations of social interactions are not available to the LLM purely through direct elicitation. When given its proposed structural causal model for each scenario, the LLM is good at predicting the signs of estimated effects, but it cannot reliably predict the magnitudes of those estimates. In the auction experiment, the in silico simulation results closely match the predictions of auction theory, but elicited predictions of the clearing prices from the LLM are inaccurate. However, the LLM’s predictions are dramatically improved if the model can condition on the fitted structural causal model. In short, the LLM knows more than it can (immediately) tell. ∗Thanks to generous support from Drew Houston and his AI for Augmentation and Productivity seed grant. Thanks to Jordan Ellenberg, Benjamin Lira Luttges, David Holtz, Bruce Sacerdote, Paul R¨ottger, Mohammed Alsobay, Ray Duch, Matt Schwartz, David Autor, and Dean Eckles for their helpful feedback. Author’s contact information, code, and data are currently or will be available at [http://www.benjaminmanning.io/](http://www.benjaminmanning.io/). †Both authors contributed equally to this work. 1 arXiv:2404.11794v2 \[econ.GN] 25 Apr 2024 | | | | | | | | | | | | | | | | | | |
| 1 | tmp83n2cnme.pdf | 2 | 1 Introduction There is much work on efficiently estimating econometric models of human behavior but comparatively little work on efficiently generating and testing those models to estimate. Previously, developing such models and hypotheses to test was exclusively a human task. This is changing as researchers have begun to explore automated hypothesis generation through the use of machine learning.1 But even with novel machine-generated hypotheses, there is still the problem of testing. A potential solution is simulation. Researchers have shown that Large Language Models (LLM) can simulate humans as experimental subjects with surprising degrees of realism.2 To the extent that these simulation results carry over to human subjects in out-of- sample tasks, they provide another option for testing (Horton, 2023). In this paper, we combine these ideas—automated hypothesis generation and automated in silico hypothesis testing—by using LLMs for both purposes. We demonstrate that such automation is possible. We evaluate the approach by comparing results to a setting where the real-world predictions are well known and test to see if an LLM can be used to generate information that it cannot access through direct elicitation. The key innovation in our approach is the use of structural causal models to orga- nize the research process. Structural causal models are mathematical representations of cause and effect (Pearl, 2009b; Wright, 1934) and have long offered a language for expressing hypotheses.3 What is novel in our paper is the use of these models as a blueprint for the design of agents and experiments. In short, each explanatory variable describes something about a person or scenario that has to vary for the effect to be identified, so the system “knows” it needs to generate agents or scenarios that 1A few examples include generative adversarial networks to formulate new hypotheses (Ludwig and Mullainathan, 2023), algorithms to find anomalies in formal theories (Mullainathan and Ram- bachan, 2023), reinforcement learning to propose tax policies (Zheng et al., 2022), random forests to identify heterogenous treatment effects (Wager and Athey, 2018), and several others (Buyalskaya et al., 2023; Cai et al., 2023; Enke and Shubatt, 2023; Girotra et al., 2023; Peterson et al., 2021). 2(Aher et al., 2023; Argyle et al., 2023; Bakker et al., 2022; Binz and Schulz, 2023b; Brand et al., 2023; Bubeck et al., 2023; Fish et al., 2023; Mei et al., 2024; Park et al., 2023) 3In an unfortunate clash of naming conventions, some disciplines have alternative definitions for the term “structural” when discussing formal models. Here, structural does not refer to the definition traditionally used in economics. See Appendix B for a more detailed explanation. 2 | | | | | | | | | | | | | | | | | | |
| 2 | tmp83n2cnme.pdf | 3 | vary on that dimension—a straightforward transition from stated theory to experi- mental design and data generation. Furthermore, the structural causal model offers a pre-specified plan for estimation (Haavelmo, 1943, 1944; J¨oreskog, 1970). We built an open-source computational system implementing this structural causal model-based approach. The system can automatically generate hypotheses, design experiments, run those experiments on independent LLM-powered agents, and ana- lyze the results. We use this system to explore several social scenarios: (1) two people bargaining over a mug, (2) a bail hearing for tax fraud, (3) a lawyer interviewing for a job, and (4) an open ascending price auction with private values for a piece of art. We allow the system to propose the hypotheses for the first two scenarios and then run the experimental simulations without intervention. For (3) and (4), we demonstrate the system’s ability to accommodate human input at any point by selecting the hypotheses ourselves and editing some of the agents, but otherwise, we allow the system to proceed autonomously. Though yet to be optimized for novelty, the system formulates and tests multiple falsifiable hypotheses—from which it generates several findings. The probability of a deal increased as the seller’s sentimental attachment to the mug decreased, and both the buyer’s and the seller’s reservation prices mattered. A remorseful defendant was granted lower bail but was not so fortunate if his criminal history was exten- sive. However, the judge’s case count before the hearing—which was hypothesized to matter—did not affect the final bail amount. The candidate passing the bar exam was the only important factor in her getting the job. Neither the candidate’s height nor the interviewer’s friendliness affected the outcome. The auction scenario is particularly illuminating. An increase in the bidders’ reservation prices caused an increase in the clearing price, a clearing price that is always close to the second-highest reservation amongst the bidders. These simula- tion results closely match the theory (Maskin and Riley, 1985) and what has been observed empirically (Athey et al., 2011). None of the findings from the system’s experiments are “counterintuitive,” but it is important to emphasize they were the result of empiricism, not just model introspection. However, this does raise the question of whether the simulations 3 | | | | | | | | | | | | | | | | | | |
| 3 | tmp83n2cnme.pdf | 4 | are even necessary.4 Instead of simulation, could an LLM simply do a “thought experiment” about the proposed in silico experiment and achieve the same insight? To test this idea, we describe the experiments that will be simulated and ask the LLM to predict the results—both the path estimates and point predictions. The path estimates being the coefficients in the linear structural causal model. To make this concrete, suppose we had the simple linear model y = Xβ to describe some scenario, and we ran an experiment to estimate ˆβ. We describe the scenario and the experiment to the LLM and ask it to predict yi given a particular Xi (a “predict-yi” task). Separately, we ask it to predict ˆβ (a “predict-ˆβ” task). Later, we examine how the LLM does on the predict-yi task when it has access to the fitted structural causal model (i.e., ˆβ). In the predict-yi task, we prompt the LLM to predict the outcome yi given each possible combination of the Xi’s from the auction experiment. Direct elicitation of the predictions for yi in the auction experiment is wildly inaccurate. The predictions are even further from the theory than the empirical results. In the predict-ˆβ task, the LLM is asked to predict the fitted structural causal model’s path estimates for all four experiments, provided with contextual information about each scenario. On average, the LLM predicts the path estimates are 13.2 times larger than the experimental results. Its predictions are overestimates for 10 out of 12 of the paths, although they are generally in the correct direction. We repeat the predict-yi task, but this time, we provide the LLM with the ex- perimental path estimates. For each Xi, we fit the structural causal model using all but the ith observation and then ask the LLM to predict yi given Xi and this fitted model. In this “predict-yi | ˆβ−i” task, the predictions are far better than in the predict-yi task without the fitted model. The mean squared error is six times lower, and the predictions are much closer to those made by the theory, but they are still further from the theory than they are to the simulations. We design and implement an approach to automated social science because LLMs possess latent information about human behavior that can be systematically explored and extracted (Burns et al., 2023; Scherrer et al., 2024). These models are trained to 4Performing these experiments required a substantial software infrastructure. 4 | | | | | | | | | | | | | | | | | |
| 4 | tmp83n2cnme.pdf | 5 | predict the next token in a sequence of text from a massive human-generated corpus. From this straightforward objective, the models develop a remarkably sophisticated model of the world, at least as captured in text (Bubeck et al., 2023; Gurnee and Tegmark, 2023; Patel and Pavlick, 2021). And while there are many situations where LLMs are imperfect proxies for humans (Cheng et al., 2023; Santurkar et al., 2023), there is also a growing body of work demonstrating that experiments with LLMs as subjects can predict human behavior in never-before-seen tasks (Binz and Schulz, 2023a; Li et al., 2024). Rapid and automated exploration of these models’ behavior could be a powerful tool to efficiently generate new insights about humans. Our contribution is to demonstrate that it is possible to create such a tool: a system that can simulate the entire social scientific process without human input at any step. The remainder of this paper is structured as follows: Section 2 provides an overview of the system. Section 3 provides some results generated using our system. Section 4 explores an LLM’s capacity to predict the results in Section 3. Section 5 discusses the advantages of using SCMs over other methods for studying causal relationships in simulations of social interactions. The paper concludes in Section 6. 2 Overview of the system To perform this automated social science, we needed to build a system. The system intentionally mirrors the experimental social scientific process. These steps are, in broad strokes: 1. Social scientists start by selecting a topic or domain to study (e.g., misinfor- mation, auctions, bargaining, etc). 2. Within the domain, they identify interesting outcomes and some causes that might affect the outcomes. These variables and their proposed relationships are the hypotheses. 3. They design an experiment to test these hypotheses by inducing variation in the causes and measuring the outcomes. 5 | | | | | | | | | | | | | | | | | | |
| 5 | tmp83n2cnme.pdf | 6 | 4. After designing the experiment, social scientists determine how they will ana- lyze the data in a pre-analysis plan. 5. Next, they recruit participants, run the experiment, and collect the data. 6. Finally, they analyze the data per the pre-analysis plan to estimate the rela- tionships between the proposed causes and outcomes. While any given social scientist might not follow this sequence exactly, whatever their approach may be, the first two steps should always guide the later steps—the development of the hypothesis guides the experimental design and model estimation. Of course, many social scientists must often omit steps 3-5 when a controlled exper- iment is not possible, but they typically have some notion of the experiment they would like to run. To build our system, we formalized a sequence of these steps analogous to those listed above. The system executes them autonomously. Since the system uses AI agents instead of human subjects, it can always design and execute an experiment. Structural causal models (SCM) are essential to the design of the system because they make unambiguous causal statements, which allow for unambiguous estimation and experimental design.5 Algorithms can determine precisely which variables must be exogenously manipulated to identify the effect of a given cause (Pearl, 2009b). If the first two steps in the social scientific process are building the SCM, the last four can be directly determined subject to the SCM. Such precision makes automation possible as the system only relies on a few key early decisions. Otherwise, the space of possible choices for the latter steps would explode, making automation infeasible. The system is implemented in Python and uses GPT-4 for all LLM queries. Its decisions are editable at every step. The overview in this section is a high- level description of the system, but there are many more specific design choices and programming details in Appendix A. For the purposes of most readers, the high- 5We use simple linear SCMs unless stated otherwise. This assumption is not necessarily correct but offers an unequivocal starting point to generate hypotheses. Functional assumptions can be tested by comparing fitted SCMs with various forms using data generated from a known causal structure. Section B in the appendix provides a more detailed explanation of SCMs. 6 | | | | | | | | | | | | | | | | | | |
| 6 | tmp83n2cnme.pdf | 7 | level overview should be sufficient to understand the system’s process, the results we present in Section 3, and the additional analyses in Sections 4 and 5. The system takes as input some scenario of social scientific interest: a negotia- tion, a bail decision, a job interview, an auction, and so on. Starting with (1) this input, the system (2) generates outcomes of interest and their potential causes, (3) creates agents that vary on the exogenous dimensions of said causes, (4) designs an experiment, (5) executes the experiment with LLM-powered agents simulating hu- mans, (6) surveys the agents to measure the outcomes, (7) analyzes the results of the experiment to assess the hypotheses, which can be used to plan a follow-on ex- periment. Figure 1 illustrates these steps, and we will briefly explore each in greater depth. Figure 1: An overview of the automated system. Notes: Each step in the process corresponds to an analogous step in the social scientific process as done by humans. The development of the hypothesis guides the experimental design, execution, and model estimation. Researchers can edit the system’s decisions at any step in the process. The first step is to generate hypotheses as SCMs based on the social scenario, the scenario being the only necessary input to the system. This is done by querying an 7 | | | | | | | | | | | | | | | | | | |
| 7 | tmp83n2cnme.pdf | 8 | LLM for the relevant agents and then interesting outcomes, their potential causes, and methods to operationalize and measure both.6 We use Typewriter text to indicate example output from the system. Suppose the social scenario is “two people bargaining over a mug.” The LLM may generate whether a deal occurs for the mug as an outcome, and operationalizes the outcome as a binary variable with a ‘‘1’’ when a deal occurs and a ‘‘0’’ when it does not. It then gener- ates potential exogenous causes and their operationalizations: the buyer’s budget, which is operationalized as the buyer’s willingness to pay in dollars. The system takes each of these variables, constructs an SCM (see the second step in Fig- ure 1), and stores the relevant information about the operationalizations associated with each variable.78 From this point on, the SCM serves as a blueprint for the rest of the process, namely the automatic instantiation of agents, their interaction, and the estimation of the linear paths. The second step is to construct the relevant agents—the Buyer and the Seller in Figure 1, step 3. By “construct,” we mean that the system prompts indepen- dent LLMs to be people with sets of attributes. These attributes are the exogenous dimensions of the SCM, dimensions that are varied in each simulation. I.e., the dif- ferent experimental conditions. For the current scenario, a Budget is provided to the buyer that can take on values of `{$5 ,$10, $20 ,$40}`. By simulating interactions of agents that vary on the exogenous dimensions of the SCM, the data generated can be used to fit the SCM. Next, the system generates survey questions to gather data about the outcomes 6When we say “query an LLM,” we mean this literally. We have written a prompt that the system provides to an LLM with the scenario. For example, the prompt used to generate the relevant agents is: In the following scenario: “\{scenario}”, who are the individual human agents in a simple simulation of this scenario? Where “\{scenario}” is replaced with the scenario of interest. The LLM then returns a list of agents, which are stored in the system and can be used in follow-on prompts, prompts that generate things like the outcomes and proposed causes. The system contains over 50 pre-written scenario-neutral prompts to gather all the information needed to generate the SCM, run the experiment, and analyze the results. 7The system generates several other pieces of information about each variable, which help guide the experimental design and data analysis. See Appendix A for further details. 8The graph in the second step of Figure 1 is a directed acyclic graph (DAG). For convenience, we will use DAGs to represent SCMs throughout the paper and assume they imply a simple linear model unless stated otherwise. 8 | | | | | | | | | | | | | | | | | | |
| 8 | tmp83n2cnme.pdf | 9 | from the agents automatically once each simulation is complete. An LLM can easily generate these questions when provided with information about the variables in the SCM (e.g., asking the buyer, “Did a deal happen?”). All LLM-powered agents in our system have “memory.” They store what happened during the simulation in text, making it easy to ask them questions about what happened. Fourth, the system determines how the agents should interact. LLMs are designed to generate text in sequence. Since independent LLMs power each agent, one agent must finish speaking before the next begins. This necessitates a turn-taking protocol to simulate the conversation. We programmed a menu of six ordering protocols, from which an LLM is queried to select the most appropriate for a given scenario. We describe each protocol in Appendix A, and they are presented in Figure A.2, but in our bargaining scenario with two agents, there are only two possible ways for the agents to alternate speaking. In this case, the system selects: speaking order: (1) Buyer, (2) Seller, (step 4, Figure 1). The speaking order can be flexible in more complex simulations with more agents, such as an auction or a bail hearing. Now, the system runs the experiment. The conditions are simulated in parallel (step 5 in Figure 1), each with a different value for the exogenous dimensions of the SCM—the possible budgets for the buyer. The system must also determine when to stop the simulations. There is no obvious rule for when a conversation should end. Like the halting problem in computer science—it is impossible to write a universal algorithm that can determine whether a given program will complete (Turing, 1937)—such a rule for conversations does not exist. We set two stopping conditions for the simulations. After each agent speaks in a simulation, an external LLM is prompted with the transcript of the conversation and asked if the conversation should continue. If yes, the next agent speaks; otherwise, the simulation ends. Additionally, we limit the total number of agent statements to twenty. One could imagine doing something more sophisticated both with the social interactions and the stopping conditions in the future. This is even a place for possible experimentation as the structure of social interactions can impact various outcomes of interest (Jahani et al., 2023; Rajkumar et al., 2022; Sacerdote, 2001). 9 | | | | | | | | | | | | | | | | | | |
| 9 | tmp83n2cnme.pdf | 10 | Finally, the system gathers the data for analysis. Outcomes are measured by asking the agents the survey questions (Figure 1, step 6) as determined before the experiment. The data is then used to estimate the linear SCM. For our negotiation, that would be a simple linear model with a single path estimate (i.e., linear coef- ficient) for the effect of the buyer’s budget on the probability of a deal—the final step in Figure 1. Note that an SCM specifies, ex-ante, the exact statistical analyses to be conducted after the experiment—akin to a pre-analysis plan. This step of the system’s process is, therefore, mechanical. The system, as outlined, is automated from start to finish—the SCM and its accompanying metadata serve as a blueprint for the rest of the process. Once there is a fitted SCM, this process can be repeated. Although we have not automated the transition from one experiment to the next, the system can generate new causal variables, induce variations, and run another experiment based on the results of the first. 3 Results of experiments We present results for four social scenarios explored using the system. In the first two scenarios, our involvement in the system’s process was restricted to entering the de- scription of the scenario and then the entire process was automated. In the third and fourth scenarios, we selected the hypotheses and edited some of the agents, but the system designed and executed the experiments. We intervened in the latter scenarios not because the system is incapable of simulating these scenarios autonomously, but to demonstrate the system’s capacity to accommodate human input at any point while still generating exciting results. 3.1 Bargaining over a mug We first use the system to simulate “two people bargaining over a mug”—this phrase being in quotes because it was the only input needed for the system to simulate the following process. The system selected a buyer and seller as the relevant agents, 10 | | | | | | | | | | | | | | | | | | |
| 10 | tmp83n2cnme.pdf | 11 | the outcome as whether a deal occurs, and the buyer’s budget, the seller’s mini- mum acceptable price, and the seller’s emotional attachment to the mug as potential causes. Table 2a provides the information generated by the system about the SCM and the experimental design. The topmost row, simulation details, provides high-level information about the structure of the simulation. The remaining rows provide information about the variables in the SCM and how they were operationalized. The system automatically generated all this information by iteratively querying the LLM. The three exogenous variables were operationalized as the buyer’s budget in dol- lars, the seller’s minimum acceptable price in dollars, and the seller’s emotional attachment as an ordinal scale from “no emotional attachment” to “extreme emo- tional attachment.” The system chose nine values (the “Attribute Treatments” in Table 2a) to vary for each of the first two causes and five for the seller’s feelings of love towards the mug (one for each level of the scale). This led to 9 × 9 × 5 = 405 experimental runs of the simulated conversation between the buyer and seller. Figure 2b provides the fitted SCM. The outcome variable is given with its mean and variance. The raw path estimates and their standard errors are shown on the arrows. For ordinal variables (e.g., the seller’s feelings of love), we treat the levels as numerical values. The buyer and seller reached a deal for the mug in half of the sim- ulations, and all three causes had a statistically significant effect on the probability of a deal. A one-dollar increase in the buyer’s budget caused an average increase of 3.7 percentage points in the probability of a deal (ˆβ\* = 0.51, p \< 0.001).9 A one-dollar increase in the seller’s minimum acceptable price caused an average decrease of 3.5 percentage points in the probability of a deal occurring (ˆβ\* = −0.49, p \< 0.001). Finally, a one-unit increase in the ordinal scale of the seller’s love for the mug, such as going from moderate emotional attachment to high emotional attachment, caused an average decrease of 2.5 percentage points in the probability of a deal (ˆβ\* = −0.07, p = 0.044). 9We report standardized effect size estimates with ˆβ\*. Standardized effect sizes being “a one standard deviation increase in X causes a ˆβ\* standard deviation increase in Y.” 11 | | | | | | | | | | | | | | | | | | |
| 11 | tmp83n2cnme.pdf | 12 | Figure 2: Experimental design and fitted SCM for “two people bargaining over a mug.” SIMULATION DETAILS Agents: Buyer, Seller Simulations Run: 9 × 9 × 5 = 405 Speaking Order: Buyer, Seller, Buyer, ...repeat VARIABLE INFORMATION Whether or not a deal occurs Measurement Question: coordinator: “Did the buyer and seller explicitly agree on the price of the mug during their interaction?” Variable Type: Binary Buyer’s Budget Attribute Treatments: \[‘3’, ‘6’, ‘7’, ‘8’, ‘10’, ‘13’, ‘18’, ‘20’, ‘25’] Proxy Attribute: Your budget for the mug Variable Type: Continuous Seller’s minimum acceptable price Attribute Treatments: \[‘3’, ‘5’, ‘7’, ‘8’, ‘10’, ‘13’, ‘18’, ‘20’, ‘25’] Proxy Attribute: Your minimum acceptable price for the mug Variable Type: Continuous Seller’s feelings of love towards the mug Attribute Treatments: \[‘no emotional attachment’, ‘slight emotional attachment’, ‘moderate emotional at- tachment’, ‘high emotional attachment’, ‘extreme emo- tional attachment’] Proxy Attribute: Your feelings of love for the mug Variable Type: Ordinal (a) Information for experimental design Deal Occurs µ = 0.5 σ2 = 0.25 Buyer Budget Seller Min Seller Love 0.037 (0.003) -0.035 (0.002) -0.025 (0.012) (b) Fitted SCM Notes: Figure 2a provides the information automatically generated by the system to execute the experiment for its proposed hypothesis. This includes the high level structure of the simulations, how the outcome is measured, and the treatment variations for each of the causes. The fitted SCM in Figure 2b shows the results of the experiment. The outcome is given with its mean and variance. The edges are labeled with their unstandardized path estimate and standard error. We assume a simple linear model for the SCM, such that the above graph can also be written as DealOccurs = 0.037BuyerBudget −0.035MinPrice −0.025SellerLove. 12 | | | | | | | | | | | | | | | | | | |
| 12 | tmp83n2cnme.pdf | 13 | 3.2 A bail hearing Next, we explore “a judge is setting bail for a criminal defendant who committed 50,000 dollars in tax fraud.” Table 3a shows that the system selected a judge, defendant, defense attorney, and prosecutor as the relevant agents. In this scenario, the system selected a more flexible interaction protocol than the one used in the previous experiment. The judge was chosen as a center agent and, in order, the prosecutor, defense attorney, and defendant as the non-center agents. This means the judge spoke first in every simulation, alternating with the other agents: judge, prosecutor, judge, defense attorney, judge, defendant, and so on. As described in Section A.3, we call this the “center-ordered” interaction protocol. Figure 3: Experimental design and fitted SCM for “a judge is setting bail for a criminal defendant who committed 50,000 dollars in tax fraud.” SIMULATION DETAILS Agents: Judge, Defendant, Defense attorney, Prosecutor Simulations Run: 7 × 7 × 5 = 243 Speaking Order: Judge, Prosecutor, Judge, Defense Attorney, Judge, Defendant, ... repeat VARIABLE INFORMATION Bail amount set by the judge Measurement Question: Judge: “What was the bail amount you set for the defendant?” Variable Type: Continuous Defendant’s criminal history Attribute Treatments: \[‘0’, ‘1’, ‘2’, ‘3’, ‘6’, ‘9’, ‘12’] Proxy Attribute: Number of your prior convictions Variable Type: Count Prior case count for judge that day Attribute Treatments: \[‘0’, ‘2’, ‘5’, ‘9’, ‘12’, ‘18’, ‘23’] Proxy Attribute: Number of cases you have already heard today Variable Type: Count Defendant’s level of remorse Attribute Treatments: \[‘no expressed remorse’, ‘low expressed remorse’, ‘moderate expressed remorse’, ‘high expressed remorse’, ‘extreme expressed remorse’] Proxy Attribute: Your level of expressed remorse Variable Type: Ordinal (a) Information for experimental design Bail Amount µ = 54428.57 σ2 = 1.9e7 Criminal History Judge Case Count Defendant’s Remorse 521.53 (206.567) -74.632 (109.263) -1153.061 (603.325) (b) Fitted SCM Notes: Figure 3a provides the information automatically generated by the system to execute the experiment for its proposed hypothesis. Figure 3b shows the fitted SCM from the experiment. 13 | | | | | | | | | | | | | | | | | | |
| 13 | tmp83n2cnme.pdf | 14 | The system chose the outcome to be the final bail amount, and the three pro- posed causes are the defendant’s criminal history, the number of cases the judge has already heard that day, and the defendant’s level of remorse. The number of cases the judge already heard that day and the defendant’s level of remorse are opera- tionalized literally, as the count of cases the judge has heard and five ordinal levels of possible outward expressions of remorsefulness. The defendant’s criminal history is operationalized as the number of previous convictions. In the fitted SCM in Figure 3b, only the defendant’s criminal history had a significant effect on the final bail amount with each additional conviction causing an average increase of \$521.53 in bail (ˆβ\* = 0.16, p = 0.012). It is unclear whether the defendant’s remorse affected the final bail amount. The effect size was small but non-trivial with borderline significance (ˆβ\* = −0.12, and p = 0.056). When we estimated the SCM with interactions, the interaction between the judge’s case count and the defendant’s remorse was nontrivial (ˆβ\* = −0.32, p = 0.047). In this specification (Figure A.5), none of the other interactions or the stand- alone causes have a significant effect, including the defendant’s criminal history. 3.3 Interviewing for a job as a lawyer In our third simulated experiment, we chose the scenario “a person interviewing for a job as a lawyer.” The system determined that a job applicant and an employer were the agents. Unlike the previous simulations, we manually selected the variables in the SCM. Table 4a shows that these were the employer’s hiring decision as the outcome and whether the applicant passed the bar, the interviewer’s friendliness, and the job applicant’s height as the potential causes. The system operationalized the causes as a binary variable for passing the bar, the job applicant’s height in centimeters, and the interviewer’s friendliness as the proposed number of friendly phrases to use during the simulation. Since one of the causes is a binary variable, the only potential cause in all our scenarios of this type, the sample size for the experimental simulations of this scenario is smaller (n = 80). By default, the system runs a factorial experimental design for all proposed values 14 | | | | | | | | | | | | | | | | | | |
| 14 | tmp83n2cnme.pdf | 15 | Figure 4: Experimental design and fitted SCM for “a person is interviewing for a job as a lawyer.” SIMULATION DETAILS Agents: Interviewer, Job Applicant Simulations Run: 2 × 5 × 8 = 405 Speaking Order: Interviewer, Job Applicant, Interviewer, ...repeat VARIABLE INFORMATION Employer’s Decision Measurement Question: Employer: “Have you de- cided to hire the job applicant?” Variable Type: Binary Whether Applicant Passed Exam Attribute Treatments: \[‘Passed’, ‘Not’] Proxy Attribute: Your bar exam status Variable Type: Binary Interviewer’s level of friendliness Attribute Treatments: \[‘2’, ‘7’, ‘12’, ‘17’, ‘22’] Proxy Attribute: Number of positive phrases to use during interview Variable Type: Count Job applicant’s height Attribute Treatments: \[‘160’, ‘165’, ‘170’, ‘175’, ‘180’, ‘185’, ‘190’, ‘195’] Proxy Attribute: Your height in centimeters Variable Type: Continous (a) Information for experimental design Employer Decision µ = 0.62 σ2 = 0.24 Passed Bar Interviewer Friend- liness Applicant Height 0.75 (0.068) -0.002 (0.005) 0.003 (0.003) (b) Fitted SCM Notes: Figure 4a provides the information automatically generated by the system to execute the experiment for the proposed hypothesis. Figure 4b shows the fitted SCM from the experiment. 15 | | | | | | | | | | | | | | | | | | |
| 15 | tmp83n2cnme.pdf | 16 | of each cause. With only two possible values for the job applicant passing the bar (as opposed to 5 varied treatment values for the interviewer’s friendliness and 8 for the applicant’s height), this limits the possible combinations of the causal variables to 2 × 5 × 8 = 80. A researcher could run more simulations to increase the sample size if so desired. We can see in Figure 4b that only the applicant passing the bar has a clear causal effect on whether the applicant gets the job. This is the largest standardized effect we see across the simulations in the four scenarios (ˆβ\* = 0.78, p \< 0.001). On average, whether or not the applicant passes the bar increases the probability she gets the job by 75 percentage points. When we test for interactions, none are significant (Figure A.6). 3.4 An auction for a piece of art Finally, we explored the scenario of “3 bidders participating in an auction for a piece of art starting at fifty dollars.” Table 5a shows that the causes are each bidder’s maximum budget for the piece of art, and the outcome is the final price of the piece of art—all of which we selected. All four variables are operationalized in dollars. To maintain symmetry in the simulations, we also manually selected the same proxy attribute for the three bidders: “your maximum budget for the piece of art.” Each bidder had the same seven possible values for their attribute, leading to 73 = 343 simulations of the auction. It is important to note that these budgets are private values. Unless a bidder publically reveals their budget, the other bidders do not know what it is. Like the tax fraud scenario, the system chose the center-ordered interaction pro- tocol for these simulations. The auctioneer was selected as the central agent, and the other agents were bidder 1, bidder 2, and bidder 3, who alternated with the auctioneer in that order. Figure 5b provides the results. All three causal variables had a positive and statistically significant effect on the final price. A one-dollar increase in any of the bidder’s budgets caused a $0.35 ,$0.29, and \$0.31 increase in the final price for the 16 | | | | | | | | | | | | | | | | | | |
| 16 | tmp83n2cnme.pdf | 17 | Figure 5: Experimental design and fitted SCM for “3 bidders participating in an auction for a piece of art starting at fifty dollars.” SIMULATION DETAILS Agents: Bidder 1, Bidder 2, Bidder 3, Auctioneer Simulations Run: 7 × 7 × 7 = 343 Speaking Order: Auctioneer, Bidder 1, Auctioneer, Bidder 2, Auctioneer, Bidder 3, ... repeat VARIABLE INFORMATION Final price Measurement Question: Auctioneer: “What was the final bid for the piece of art at the end of the auc- tion?” Variable Type: Continuous Bidder 1’s maximum budget Attribute Treatments: \[‘$50^\{'} , `$100’, ‘$150^\{'} , `$200’, ‘$250^\{'} , `$300’, ‘$350^\{'} left]right. P r o x y A t t r i b u t e : Y o u r m a x b u d g e t f o r t h e a r t V a r i a b l e T y p e : C o n t i n u o u s B i d d e r 2^{'} s m a x i m u m b u d g e t A t t r i b u t e T r e a t m e n t s : left[right. `$50’, ‘$100^{'} , `$150’, ‘$200^{'} , `$250’, ‘$300^{'} , `$350’] Proxy Attribute: Your max budget for the art Variable Type: Continuous Bidder 3’s maximum budget Attribute Treatments: \[‘$50^{'} , `$100’, ‘$150^{'} , `$200’, ‘$250^{'} , `$300’, ‘\$350’] Proxy Attribute: Your max budget for the art Variable Type: Continuous (a) Information for experimental design Final Price µ = 186.53 σ2 = 3879.23 Bidder 1 Budget Bidder 2 Budget Budder 3 Budget 0.35 (0.015) 0.29 (0.015) 0.31 (0.015) (b) Fitted SCM Notes: Figure 5a provides the information automatically generated by the system to execute the experiment for the proposed hypothesis. Figure 5b shows the fitted SCM from the experiment. 17 | | | | | | | | | | | | | | | | | | |
| 17 | tmp83n2cnme.pdf | 18 | piece of art for each respective bidder (ˆβ\* = 0.57, p \< 0.001; ˆβ\* = 0.47, p \< 0.001; ˆβ\* = 0.5 p \< 0.001). These quantities make sense as each bidder has a 1 3 chance of being marginal. 4 LLM predictions for paths and points It is worth reiterating that the results in the previous section were not generated by directly prompting an LLM but rather through experimentation. Although the experiments were fast and inexpensive, they were not free–in total, they took about 5 hours to run and cost over \$1,000. This raises the question of whether the simu- lations were even necessary. Could an LLM do a “thought experiment” (i.e., make a prediction based on a prompt) about a proposed in silico experiment and achieve the same insight? If so, we should just prompt the LLM to come up with an SCM and elicit its predictions about the relationships between the variables. To test this idea, we describe some of the simulations to the LLM and ask it to predict the results—path estimates and point predictions.10 Specifically, we modeled each scenario as y = Xβ, where y is an n × 1 vector and X is a n × k matrix. Here, n is the number of simulations, and k is the number of proposed causes. The experiments from Section 3 provided us with estimates for ˆβ (a k × 1 vector). We describe the scenario and the experiment to the LLM and ask it to independently predict yi given each Xi (a predict-yi task) as well as to predict ˆβ (a predict-ˆβ task). The LLM’s yi predictions are highly inaccurate compared to those from auction theory, which predicts that the clearing price will be the second highest valuation in an open-ascending price auction with private values (Maskin and Riley, 1985). The LLM is also unable to accurately predict the path estimates (ˆβ) of the fitted SCM. Finally, we examine how the LLM does on the predict-yi task when provided with an SCM fit on all of the data except for the corresponding Xi (the predict-yi | ˆβ−i task). While the additional information dramatically improves the LLM’s predictions, they are still less accurate than those made by auction theory. 10All predictions are made by the LLM once at temperature 0. When we elicit these predictions many times at higher temperatures, the results are similar. 18 | | | | | | | | | | | | | | | | | |
| 18 | tmp83n2cnme.pdf | 19 | 4.1 Predicting yi For various bidder reservation price combinations in the auction experiment, we supply the LLM with a prompt detailing the simulation and experimental design.11 We then ask the LLM to predict the clearing price for the auction. This gives us a point prediction for each simulated auction (i.e., each unique row Xi in X) used to generate the fitted SCM in Figure 5b. Figure 6 presents a comparison of the LLMs predictions, the simulated experi- ments, and the predictions made by auction theory.12 The columns correspond to the different reservation values for bidder 3 in a given simulation, and the rows cor- respond to the different reservation values for bidder 2. The y-axis is the final bid price, and the x-axis lists bidder 1’s reservation price. The black triangles track the observed clearing price in each simulated experiment, the black line shows the pre- dictions made by auction theory, and the blue line indicates the LLM’s predictions without the fitted SCM—the predict-yi task. The LLM performs poorly at the predict-yi task. The blue line is often far from the black triangles and sometimes remains constant or even decreases as the second-highest reservation price across the agents increases. In contrast, auction theory is highly accurate in its predictions of the final bid price in the experiment— the black line often perfectly tracks the black triangles.13 The mean squared error (MSE) of the LLM’s predictions in the predict-yi task (MSEyi = 8628) is an order of magnitude higher than that of the theoretical predictions (MSETheory = 128), and the predictions are even further from the theory than they are from the empirical results (MSEyi−Theory = 8915).14 11In 80/343 simulations, the agents made the maximum number of statements (20) allowed by the system before the auction ended. We remove these observations because, without additional information, auction theory does not make predictions about partially completed auctions. 12We provide only a subset of the results in the main text as it is difficult to visualize all of them in a single figure. Figure A.10 shows the full set of predictions. The results are generally the same. 13There are a few observations where the empirical clearing price is slightly above or below the theory prediction. In most cases where it was off, this was due to the auctioneer incrementing the bid price above the second-highest reservation price in the last round. 14MSE is reported for all predictions, not just the subset shown in Figure 6. 19 | | | | | | | | | | | | | | | | | | |
| 19 | tmp83n2cnme.pdf | 20 | Figure 6: Comparison of the LLM’s predictions to the theoretical predictions and a subset of experimental results for the auction scenario. Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Pred. yi Pred. yi | β^ −i Auc. Theory Bidder 3 Reservation: 150 Bidder 3 Reservation: 200 Bidder 3 Reservation: 250 Bidder 3 Reservation: 300 Bidder 2 Reservation: 200 Bidder 2 Reservation: 150 Bidder 2 Reservation: 100 Bidder 2 Reservation: 50 100 200 300 400 100 200 300 400 100 200 300 400 100 200 300 400 100 200 300 100 200 300 100 200 300 100 200 300 Bidder 1 Reservation Price Final Clearing Price Experiment Notes: The columns correspond to the different reservation values for bidder 3 in a given simulation, and the rows correspond to the different reservation values for bidder 2. The y-axis is the clearing price, and the x-axis lists bidder 1’s reservation price. The black triangles track the observed clearing price in each simulated experiment, the black line shows the predictions made by auction theory (MSET heory = 128), the blue line indicates the LLM’s predictions without the fitted SCM—the predict-yi task (MSEyi = 8628), and the red line is the LLM’s predictions with the fitted SCM—the predict-yi | ˆβ−i task (MSEyi | ˆβ−i = 1505). 20 |
| 20 | tmp83n2cnme.pdf | 21 | 4.2 Predicting ˆβ We prompted the LLM to predict the path estimates and whether they would be statistically significant for the simulated experiments in Section 3. This is the predict- ˆβ task. We then compare the LLM’s predictions to the fitted SCMs. With four experiments and three causes in each, we generate 12 predictions. We provide the LLM with extensive information to make its predictions for each experiment.15 This information includes the proposed SCM, the operationalizations of the variables, the number of simulations, and the possible treatment values. Each prediction is elicited once at temperature 0. The predictions are shown in Table A.1. They were, on average, 13.2 times larger than the actual estimates, and 10/12 of the predictions were overestimates. Even when we remove the largest overestimate, the average magnitude of the ratio between the predicted and actual estimates is still 5.3. The sign of the estimate was correct in 10/12 predictions, and 10/12 correctly guessed whether or not the estimate would be statistically significant. When we repeat the predictions at a higher temperature and take their average, the results are similar (see Table A.2). 4.3 Predicting yi | ˆβ−i The LLM was, on average, off by an order of magnitude for both the predict-yi task and the predict-ˆβ task, but maybe it can do better with more information. For each Xi in the auction simulations, we use the data from the experiment to estimate ˆβ−i, the path estimates from the SCM excluding the ith observation. We then prompt the LLM to predict the outcome for each Xi given ˆβ−i. The red line in Figure 6 provides these new predictions. The LLM’s predic- tions are much closer to the actual outcomes when it has access to a fitted SCM (MSEyi | ˆβ−i = 1505) as opposed to when it does not (MSEyi = 8628), even though all the predictions are out of sample and every Xi is unique. However, the LLM’s predictions on the predict-yi | ˆβ−i task are still not as accurate 15See Figure A.11 in the appendix for the full prompt. 21 | | | | | | | | | | | | | | | |
| 21 | tmp83n2cnme.pdf | 22 | as the predictions made by auction theory (MSETheory = 128).16 They are also still further from the theory than they are from the empirical results (MSEyi | ˆβ−i−Theory = 1761). There is clearly room for improvement. That improvement is feasible with the system: there exists an SCM perfectly consistent with auction theory. Only one exogenous variable was missing: the second-highest reservation price of the bidders. If allowed to generate and test enough potential causes, our system could have se- lected this variable as a possible cause by itself. In this case, the fitted SCM would have matched the theoretical predictions.17 5 Identifying causal structure ex-ante The SCM-based approach offers a promising new method for studying simulated be- havior at scale. However, it is not the only option for such rapid exploration. Others have designed large, quasi-unstructured simulations demonstrating exciting results. For example, Park et al. (2023) endows a group of LLM agents with personas and memory systems and then allows them to freely interact in a simulated community for an extended period. Despite no explicit instructions to do so, the agents in the simulation produce many human-like behaviors, such as throwing parties, going on dates, and making friends. While impressive and informative, a problem with such open-ended social simu- lations is that selecting and analyzing outcomes can be difficult. To unveil insights, researchers may need to comb through thousands of lines of unstructured text. If they are interested in casual relationships, they may need to infer the causal struc- ture ex-post, which can be problematic. In contrast, the SCM framework describes exactly what needs to be measured as a downstream outcome subject to the exoge- nous manipulations of the cause. Identification is guaranteed. In this section, we 16It is also less accurate than the mechanical predictions made by the fitted SCM using the same procedure MSEMechanistic:yi | ˆβ−i = 725. Maybe the LLM cannot do the math, is still conditioning on other information beyond the path estimates when making its predictions, or, like humans, is ignoring relevant information when making choices (Handel and Schwartzstein, 2018). 17When we do fit this SCM (see Figure A.9), the coefficient is close to one (β = 0.912), and almost all the variance in the outcome is explained (R2 = 0.977). 22 | | | | | | | | | | | | | | | | |
| 22 | tmp83n2cnme.pdf | 23 | discuss how assuming or searching for causal structure in observational data, the type generated from massive open-ended simulations can lead to misidentification and how using SCMs avoids this problem. 5.1 Assuming causal structure from data All estimates in the fitted SCMs in Section 3 are unbiased. We know this because the data comes from an experiment, and we randomized on the causal variables. A nice feature of a perfectly randomized experiment is that we can get unbiased measurements of any downstream endogenous outcome relative to the exogenous manipulations.18 I.e., the coefficients on the fitted SCM are identified. For example, in the bargaining experiment, perhaps we are interested in the length of the con- versation as an outcome, even though it was not a part of the original SCM. The conversation length can be operationalized as the sum of the number of statements made by all agents, and we can use the transcript from the finished experiment to measure it. We can then fit an SCM with the data and get unbiased estimates of the effect of the exogenous variables on the conversation’s length. Figure 7a shows this fitted SCM using the data from the experiment in Section 3. Both the buyer’s budget and the seller’s minimum price have a significant effect on the length of the conversation (p \< 0.001; p = 0.026), but the seller’s emotional attachment does not (p = 0.147). Suppose we did not know the actual causal structure of these scenarios or that the data came from an experiment. All we have are the data for the original three causes, the conversation length, and whether a deal was made (the original outcome). If we want to estimate the causal relationships between these variables, we would have to make untestable assumptions. For example, one could reasonably presume that the buyer’s budget, the seller’s minimum price, the seller’s emotional attachment, and whether a deal was made all causally affect the length of the conversation. Figure 7b provides the fitted SCM for this proposed causal structure. Only 18When we say “downstream,” we mean any variable whose value is realized after the agents begin interacting in the simulated conversations. 23 | | | | | | | | | | | | | | | | | | |
| 23 | tmp83n2cnme.pdf | 24 | Figure 7: Comparison of the true and misspecified SCMs. Convo Length Buyer Budget Seller Min Seller Love -0.111 (0.031) 0.069 (0.031) 0.222 (0.153) (a) Correctly specified SCM Convo Length Deal Occurs Buyer Budget Seller Min Seller Love -0.051 (0.039) 0.012 (0.037) -1.622 (0.615) 0.182 (0.153) (b) Misspecified SCM Notes: Statistically significant paths are marked in red (α = 0.05). Each path is given with its estimated coefficient and standard error in parentheses. Both SCMs are estimated using the data from the bargaining scenario in Section 3. Subfigure (a) provides a correctly specified SCM from a randomized experiment. Subfigure (b) shows a misspecified SCM based on an assumed structure. The path estimates of the buyer’s budget and the seller’s minimum price go from significant in the correctly specified SCM to insignificant and far closer to zero in the misspecified SCM. whether a deal was made was estimated to have a significant effect on the length of the conversation (p = 0.008). But we know this is wrong. We have the true causal structure in Figure 7a from a perfectly randomized experiment, and both the buyer’s and the seller’s reservation prices had a significant effect on the length of the conversation. Here, they are insignificant and far closer to zero (p = 0.189; p = 0.755). Whether or not the deal occurred is a bad control that biases the estimates—it is probably codetermined with the length of the conversation.19 The informed econometrician may presume that she would never make such a mistake, but many researchers are not so savvy.20 We were unsure of it until we had unbiased estimates from the correctly specified SCM as a reference. There are also many kinds of bad controls, and many of them are less obvious than those in 19We cannot be sure about the causal relationship between the length of the conversation and whether a deal was made because neither is exogenously varied in the experiment. All we know is that controlling for whether or not a deal occurs induces bias, as we have the experiment as a reference. 20LLMs are definitely not yet savvy enough to avoid this mistake. 24 | | | | | | | | | | | | | | | | | | |
| 24 | tmp83n2cnme.pdf | 25 | this example (Cinelli et al., 2022). It is easy to misspecify a model when the data is observational and has many variables, even when their relationships may seem obvious. The SCM-based approach avoids the bad controls. The generation of the data is based on the causal structure. There is no need to instrument endogenous variables and presume their causal relationships. Exogenous variation is explicitly induced in the SCM to identify the causal relationships ex-ante. Even if we do not know how a new outcome is incorporated into the causal structure, we can always reference how it is affected by the exogenous variables by fitting a simple linear SCM. 5.2 Searching for causal structure in data Another strategy for identifying causal relationships when the underlying structure is unknown is to let the data speak for itself. For example, we could use an algorithm to find the model that makes the data most likely. There are many ways to do this, none of which can always, or even consistently, identify the correct causal relationships from observational data (Pearl, 2009a). These algorithms take as input potential variables of interest (a graph with no edges, only nodes) and data for these variables. They output a proposed DAG that best fits the data.21 The simplest algorithm is to generate all possible DAGs for existing variables and then evaluate each model based on some criteria (e.g., maximum likelihood, Bayesian information criterion, etc.).22 Another method is to add edges that maximize the criteria greedily. This approach can be further improved by penalizing the model for complexity (based on additional criteria) and removing edges until the model is greedily optimized. The second approach is the Greedy Equivalence Search (GES) algorithm (Chickering, 2002), which we used on the data and from all the experiments 21These algorithms often do not presume a functional form, so we refer refer to hypotheses as DAGs, not SCMs, in this section. 22The number of possible DAGs grows exponentially with the number of nodes. For example, for n = 1, 2, 3, and 4 nodes, there are 1, 3, 25, and 543 possible DAGs. This is a combinatorial explosion, and it is not feasible to evaluate all potential models for a large number of nodes, which presents further problems for this approach. 25 | | | | | | | | | | | | | | | | | | |
| 25 | tmp83n2cnme.pdf | 26 | in Section 3.23 In some experiments, the algorithm incorrectly identified the causal structure. Figure 8 provides the DAG identified by the GES algorithm for the tax fraud scenario. As a reminder, the original causal variables are the defendant’s previous convictions, the judge’s number of cases heard that day, and the defendant’s level of remorse, and the outcome is the bail amount. The algorithm has no information about which variables are exogenously varied, just the raw data. Figure 8: Incorrect causal structure identified by the GES algorithm for the tax fraud experiment. Bail Amount Crime History Remorse Num Cases Notes: The Greedy Equivalence Search (GES) algorithm can incorrectly identify the causal structure of observational data. In the tax fraud scenario, we know from Figure 3b and the accompanying experiment that an increase in the defendant’s previous convictions caused an increase in the av- erage bail amount. However, the algorithm identified the causal relationship as equally likely in either direction. Without the correctly specified DAG, a researcher would have to assume the causal structure of the data, which can be problematic. The GES algorithm identified the defendant’s criminal history and the bail amount as the only variables in the scenario with any causal relationship. This is partially correct—we know from the experiment that an increase in the defendant’s previous convictions caused an increase in the average bail amount. However, the algorithm identified the causal relationship as equally likely in either direction. There was no more evidence in the data that the defendant’s criminal history caused the bail amount than the bail amount caused the defendant’s criminal history. And while we know that the former is correct from our experiment, a researcher using the algo- 23The GES algorithm is not perfectly stable; different runs on the same data can produce different results, which is its own problem. 26 | | | | | | | | | | | | | | | | | | |
| 26 | tmp83n2cnme.pdf | 27 | rithm without the correctly specified DAG would not. They would have to make an assumption, which, as we have shown, can be problematic. The SCM-based approach avoids search problems, as we never need to search for the causal structure given the data. Instead, we generate the data based on a proposed causal structure. Even if we want to measure a new outcome on the existing experimental data, we have already identified the sources of exogenous variation. We should note that problems with searching for or assuming causal structures from data are not new. Pearl (2009a) makes a similar point many times. However, social scientists have never had the tools to induce exogenous variation and explore causal relationships at scale in many different scenarios. 6 Conclusion This paper demonstrates an approach to automated in silico hypothesis generation and testing made possible through the use of SCMs. We implemented the approach by building a computational system with LLMs and provided evidence that simu- lations can elicit information from an LLM that was not ex-ante available to the model. We also showed that such simulations produce results that are highly con- sistent with theoretical predictions made by the relevant economic theory. In this final section, we will discuss why such systems could be useful and identify areas for future research. 6.1 Controlled experimentation at scale How might systems like the one presented in this paper be useful for social science research? One view is that these simulations are simple dress rehearsals for “real” social science. A more expansive and exciting view is that these simulations would yield insights that sometimes generalize to the real world. This is a view that sees these agents as a step forward in representing humans far beyond classical methods in agent-based modeling, such as those used to explore how individual preferences can lead to surprising social patterns (Schelling, 1969, 27 | | | | | | | | | | | | | | | | | | |
| 27 | tmp83n2cnme.pdf | 28 | 1971).24 This view would mirror recent advances in the use of machine learning for protein folding (Jumper et al., 2021) and material discovery (Merchant et al., 2023). The system presented in this paper can generate these controlled experimental simulations en masse with prespecified plans for data collection and analysis. That contrasts most academic social science research as currently practiced (Almaatouq et al., 2022).25 This contrast is important. In the social sciences, context can heavily influence results. Outcomes that hold true for one population may not for another. Even within the same population, a change in environment can nullify or flip re- sults (Lerner et al., 2004). Studying humans is also expensive and time-consuming, which makes rapid, inexpensive, and replicable exploration valuable. There is still, of course, the fundamental jump from simulations to human subjects. 6.2 Interactivity The system allows a scientist to monitor its entire process. Should a researcher disagree with or be uncertain about a decision made by the system, they can probe the system regarding its choice. This allows the researcher to either (1) understand why the decision was made, (2) ask the system to come up with a different option for that decision, or (3) input their own custom choice for that decision. A researcher can even ignore much of the automation process and fill in the details themselves. They can choose the variables of interest, their operationalizations, the attributes of the agents, how the agents interact, or customize the statistical analysis, among other decision points. Different parts of the system can also accommodate different types of LLMs simultaneously. For example, a researcher could use GPT- 4 to generate hypotheses and Llama-2-70B to power the agents’ simulated social interactions. 24See Horton (2023) for a full discussion on the differences between traditional agent-based mod- eling and the use of LLM-powered agents. This position reflects our views as it was written recently by some of the authors of this paper. 25When a group of social scientists has the same data set on some human behavior or outcome, they can reach very different conclusions when analyzing it independently (Engzell, 2023; Salganik et al., 2020). 28 | | | | | | | | | | | | | | | | | | |
| 28 | tmp83n2cnme.pdf | 29 | 6.3 Replicability Replicating social science experiments with human subjects can be difficult (Camerer et al., 2018). Despite the use of preregistrations, the exact procedures used in exper- iments are often unclear (Engzell, 2023). In contrast, the system allows for nearly frictionless communication and replication of the experimental design. The system’s entire procedure is exportable as a JSON file with the fitted SCM.26 This JSON includes every decision the system makes, including natural language explanations for the choices and the transcripts from each simulation. These JSONs can be saved or uploaded at any point in the system’s process. A researcher could run experiments and post the JSON and results online. Other scientists could inspect, replicate the experiment, or extend the work. 6.4 Future research While designing our system, we encountered several areas for new research. First is the problem of “which attributes” to endow an LLM-powered agent beyond those im- mediately relevant to the proposed exogenous variables. For example, demographic information, personalities, and other traits are not included in the agent’s attributes unless they are a part of the SCM. To improve the fidelity of the simulations, it might make sense to add some or all of these attributes to the agents. However, it is unclear how to optimize this process. Second, we encountered the problem of engineering social interactions between LLM agents. LLMs are designed to exchange text in sequence, necessitating a pro- tocol for turn-taking that reflects the natural ebb and flow of human conversation. In an initial attempt to address this problem, we created a menu of flexible agent- ordering mechanisms. We also introduced an additional LLM-powered agent into our version of the system whom we dub the ‘coordinator.” The coordinator functions as a quasi-omniscient assistant who can read through transcripts and make choices 26A JSON (JavaScript Object Notation) is a data format that is easy for humans to read and write and easy for machines to parse and generate. It is commonly used for transmitting data in web applications, as a configuration and data storage format, and for serializing and transmitting structured data over a network. 29 | | | | | | | | | | | | | | | | | | |
| 29 | tmp83n2cnme.pdf | 30 | about the speaking order of other agents in the simulations. There are probably better ways to determine the speaking order of agents. A related problem is the question of when to stop the simulations. Like Turing’s halting problem, there is likely no universal rule for when conversations should end, but there are probably better rules than those we have implemented. A Markov model approximating the distribution of agents speaking, estimated from real con- versation data, might provide more naturalistic results for simulating and ending interactions, but that is an idea for future work. Lastly, if we can build a system that can automate one iteration of the scientific process and determine a follow-on experiment, a clear next step is to set up an intelligently automated research program. This would involve using outcomes from the simulations to inform continuous cycles of experimentation. Then, a researcher could intelligently explore a given scenario’s parameter space. How to optimize this exploration amongst so many possible variables will be an important problem to solve. As presented in this paper, the system provides only one possible implementation of the SCM-based approach. We made many subjective decisions. Other researchers might implement the approach with different design choices. There is room for improvement and exploration. 30 | | | | | | | | | | | | | | | | | | |
| 30 | tmp83n2cnme.pdf | 31 | References Aher, Gati V, Rosa I Arriaga, and Adam Tauman Kalai, “Using large lan- guage models to simulate multiple humans and replicate human subject studies,” in “International Conference on Machine Learning” PMLR 2023, pp. 337–371. Almaatouq, Abdullah, Thomas L. Griffiths, Jordan W. Suchow, Mark E. Whiting, James Evans, and Duncan J. Watts, “Beyond Playing 20 Ques- tions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences,” Behavioral and Brain Sciences, 2022, p. 1–55. Argyle, Lisa P, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christo- pher Rytting, and David Wingate, “Out of one, many: Using language models to simulate human samples,” Political Analysis, 2023, 31 (3), 337–351. Atari, M., M. J. Xue, P. S. Park, D. E. Blasi, and J. Henrich, “Which Humans?,” Technical Report 09 2023. [https://doi.org/10.31234/osf.io/5b26t](https://doi.org/10.31234/osf.io/5b26t). Athey, Susan, Jonathan Levin, and Enrique Seira, “Comparing open and Sealed Bid Auctions: Evidence from Timber Auctions\*,” The Quarterly Journal of Economics, 02 2011, 126 (1), 207–257. Bakker, Michiel, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, and Christopher Summerfield, “Fine-tuning language models to find agreement among humans with diverse pref- erences,” in S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, eds., Advances in Neural Information Processing Systems, Vol. 35 Curran Asso- ciates, Inc. 2022, pp. 38176–38189. Binz, Marcel and Eric Schulz, “Turning large language models into cognitive models,” 2023. and , “Using cognitive psychology to understand GPT-3,” Proceedings of the National Academy of Sciences, 2023, 120 (6), e2218523120. 31 | | | | | | | | | | | | | | | | | | |
| 31 | tmp83n2cnme.pdf | 32 | Brand, James, Ayelet Israeli, and Donald Ngwe, “Using GPT for Market Research,” Working paper, 2023. Bubeck, S´ebastien, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang, “Sparks of Artificial General Intelligence: Early experiments with GPT-4,” 2023. Burns, C, H Ye, D Klein, and J Steinhardt, “Discovering latent knowledge in language models without supervision,” in “International Conference on Learning Representations (ICLR)” 2023. Buyalskaya, Anastasia, Hung Ho, Katherine L. Milkman, Xiaomin Li, Angela L. Duckworth, and Colin Camerer, “What can machine learning teach us about habit formation? Evidence from exercise and hygiene,” Proceedings of the National Academy of Sciences, 2023, 120 (17), e2216115120. Cai, Alice, Steven R Rick, Jennifer L Heyman, Yanxia Zhang, Alexandre Filipowicz, Matthew Hong, Matt Klenk, and Thomas Malone, “Desig- nAID: Using Generative AI and Semantic Diversity for Design Inspiration,” in “Proceedings of The ACM Collective Intelligence Conference” CI ’23 Association for Computing Machinery New York, NY, USA 2023, p. 1–11. Camerer, Colin, Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jurgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A. Nosek, Thomas Pfeiffer, Adam Altmejd, Nick Buttrick, Taizan Chan, Yiling Chen, Eskil Forsell, Anup Gampa, Emma Heikensten, Lily Hum- mer, Taisuke Imai, Siri Isaksson, Dylan Manfredi, Julia Rose, Eric-Jan Wagenmakers, and Hang Wu, “Evaluating the Replicability of Social Science Experiments in Nature and Science between 2010 and 2015,” Nature Human Be- haviour, Aug 2018, 2 (9), 637–644. 32 | | | | | | | | | | | | | | | | | | |
| 32 | tmp83n2cnme.pdf | 33 | Cheng, Myra, Tiziano Piccardi, and Diyi Yang, “CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations,” ArXiv, 2023, abs/2310.11501. Chickering, David Maxwell, “Optimal structure identification with greedy search,” Journal of machine learning research, 2002, 3 (Nov), 507–554. Cinelli, Carlos, Andrew Forney, and Judea Pearl, “A crash course in good and bad controls,” Sociological Methods & Research, 2022, p. 00491241221099552. Engzell, Per, “A universe of uncertainty hiding in plain sight,” Proceedings of the National Academy of Sciences, 2023, 120 (2), e2218530120. Enke, Benjamin and Cassidy Shubatt, “Quantifying Lottery Choice Complex- ity,” Working Paper 31677, National Bureau of Economic Research September 2023. Fish, Sara, Paul G¨olz, David C Parkes, Ariel D Procaccia, Gili Rusak, Itai Shapira, and Manuel W¨uthrich, “Generative Social Choice,” arXiv preprint arXiv:2309.01291, 2023. Girotra, Karan, Lennart Meincke, Christian Terwiesch, and Karl T Ul- rich, “Ideas are dimes a dozen: Large language models for idea generation in innovation,” Available at SSRN 4526071, 2023. Gurnee, Wes and Max Tegmark, “Language Models Represent Space and Time,” 2023. Haavelmo, Trygve, “The statistical implications of a system of simultaneous equa- tions,” Econometrica, Journal of the Econometric Society, 1943, pp. 1–12. , “The probability approach in econometrics,” Econometrica: Journal of the Econometric Society, 1944, pp. iii–115. Handel, Benjamin and Joshua Schwartzstein, “Frictions or Mental Gaps: What’s Behind the Information We (Don’t) Use and When Do We Care?,” Journal of Economic Perspectives, February 2018, 32 (1), 155–178. 33 | | | | | | | | | | | | | | | | | | |
| 33 | tmp83n2cnme.pdf | 34 | Hern´an, Miguel A. and James M. Robins, Causal Inference: What If, Boca Raton: Chapman & Hall/CRC, 2020. Horton, John J, “Large language models as simulated economic agents: What can we learn from homo silicus?,” Technical Report, National Bureau of Economic Research 2023. Imai, Kosuke, Dustin Tingley, and Teppei Yamamoto, “Experimental De- signs for Identifying Causal Mechanisms,” Journal of the Royal Statistical Society Series A: Statistics in Society, 11 2012, 176 (1), 5–51. Jahani, Eaman, Samuel P. Fraiberger, Michael Bailey, and Dean Eckles, “Long ties, disruptive life events, and economic prosperity,” Proceedings of the National Academy of Sciences, 2023, 120 (28), e2211062120. Jumper, John, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin ˇZ´ıdek, Anna Potapenko et al., “Highly accurate protein structure prediction with AlphaFold,” Nature, 2021, 596 (7873), 583–589. J¨oreskog, Karl G., “A GENERAL METHOD FOR ESTIMATING A LINEAR STRUCTURAL EQUATION SYSTEM\*,” ETS Research Bulletin Series, 1970, 1970 (2), i–41. Lerner, Jennifer S., Deborah A. Small, and George Loewenstein, “Heart Strings and Purse Strings: Carryover Effects of Emotions on Economic Decisions,” Psychological Science, 2004, 15 (5), 337–341. PMID: 15102144. Li, Peiyao, Noah Castelo, Zsolt Katona, and Miklos Sarvary, “Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis,” Marketing Science, 2024, 0 (0), null. Ludwig, Jens and Sendhil Mullainathan, “Machine Learning as a Tool for Hypothesis Generation,” Working Paper 31017, National Bureau of Economic Re- search March 2023. 34 | | | | | | | | | | | | | | | | | | |
| 34 | tmp83n2cnme.pdf | 35 | Maskin, Eric S. and John G. Riley, “Auction Theory with Private Values,” The American Economic Review, 1985, 75 (2), 150–155. Mastroianni, Adam M., Daniel T. Gilbert, Gus Cooney, and Timothy D. Wilson, “Do conversations end when people want them to?,” Proceedings of the National Academy of Sciences, 2021, 118 (10), e2011809118. Mei, Qiaozhu, Yutong Xie, Walter Yuan, and Matthew O. Jackson, “A Tur- ing test of whether AI chatbots are behaviorally similar to humans,” Proceedings of the National Academy of Sciences, 2024, 121 (9), e2313925121. Merchant, Amil, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk, “Scaling deep learning for materials discovery,” Nature, 2023, pp. 1–6. Mullainathan, Sendhil and Ashesh Rambachan, “From Predictive Algorithms to Automatic Generation of Anomalies,” Technical Report May 2023. Available at: [https://ssrn.com/abstract=4443738](https://ssrn.com/abstract=4443738) or [http://dx.doi.org/10.2139/ssrn](http://dx.doi.org/10.2139/ssrn). 4443738. Park, Joon Sung, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Mor- ris, Percy Liang, and Michael S Bernstein, “Generative agents: Interactive simulacra of human behavior,” arXiv preprint arXiv:2304.03442, 2023. Patel, R. and E. Pavlick, “Mapping language models to grounded conceptual spaces,” in “Proceedings of the International Conference on Learning Representa- tions” 2021, p. 79. Pearl, J., M. Glymour, and N.P. Jewell, Causal Inference in Statistics: A Primer, Wiley, 2016. Pearl, Judea, “Causal inference in statistics: An overview,” Statistics Surveys, 2009, 3 (none), 96 – 146. , Causality, Cambridge university press, 2009. 35 | | | | | | | | | | | | | | | | | | |
| 35 | tmp83n2cnme.pdf | 36 | Peterson, Joshua C., David D. Bourgin, Mayank Agrawal, Daniel Reich- man, and Thomas L. Griffiths, “Using large-scale experiments and machine learning to discover theories of human decision-making,” Science, 2021, 372 (6547), 1209–1214. Rajkumar, Karthik, Guillaume Saint-Jacques, Iavor Bojinov, Erik Bryn- jolfsson, and Sinan Aral, “A causal test of the strength of weak ties,” Science, 2022, 377 (6612), 1304–1310. Rosenbusch, Hannes, Claire E. Stevenson, and Han L. J. van der Maas, “How Accurate are GPT-3’s Hypotheses About Social Science Phenomena?,” Dig- ital Society, July 2023, 2 (2), 26. Rosseel, Yves, “lavaan: An R Package for Structural Equation Modeling,” Journal of Statistical Software, 2012, 48 (2), 1–36. Sacerdote, Bruce, “Peer Effects with Random Assignment: Results for Dartmouth Roommates\*,” The Quarterly Journal of Economics, 05 2001, 116 (2), 681–704. Salganik, Matthew J., Ian Lundberg, Alexander T. Kindel, Caitlin E. Ahearn, Khaled Al-Ghoneim, Abdullah Almaatouq, Drew M. Altschul, Jennie E. Brand, Nicole Bohme Carnegie, Ryan James Compton, Debanjan Datta, Thomas Davidson, Anna Filippova, Connor Gilroy, Brian J. Goode, Eaman Jahani, Ridhi Kashyap, Antje Kirchner, Stephen McKay, Allison C. Morgan, Alex Pentland, Kivan Polimis, Louis Raes, Daniel E. Rigobon, Claudia V. Roberts, Diana M. Stanescu, Yoshihiko Suhara, Adaner Usmani, Erik H. Wang, Muna Adem, Ab- dulla Alhajri, Bedoor AlShebli, Redwane Amin, Ryan B. Amos, Lisa P. Argyle, Livia Baer-Bositis, Moritz B¨uchi, Bo-Ryehn Chung, William Eggert, Gregory Faletto, Zhilin Fan, Jeremy Freese, Tejomay Gadgil, Josh Gagn´e, Yue Gao, Andrew Halpern-Manners, Sonia P. Hashim, So- nia Hausen, Guanhua He, Kimberly Higuera, Bernie Hogan, Ilana M. Horwitz, Lisa M. Hummel, Naman Jain, Kun Jin, David Jurgens, 36 | | | | | | | | | | | | | | | | | | |
| 36 | tmp83n2cnme.pdf | 37 | Patrick Kaminski, Areg Karapetyan, E. H. Kim, Ben Leizman, Naijia Liu, Malte M¨oser, Andrew E. Mack, Mayank Mahajan, Noah Man- dell, Helge Marahrens, Diana Mercado-Garcia, Viola Mocz, Katari- ina Mueller-Gastell, Ahmed Musse, Qiankun Niu, William Nowak, Hamidreza Omidvar, Andrew Or, Karen Ouyang, Katy M. Pinto, Ethan Porter, Kristin E. Porter, Crystal Qian, Tamkinat Rauf, Anahit Sargsyan, Thomas Schaffner, Landon Schnabel, Bryan Schonfeld, Ben Sender, Jonathan D. Tang, Emma Tsurkov, Austin van Loon, Onur Varol, Xiafei Wang, Zhi Wang, Julia Wang, Flora Wang, Saman- tha Weissman, Kirstie Whitaker, Maria K. Wolters, Wei Lee Woon, James Wu, Catherine Wu, Kengran Yang, Jingwen Yin, Bingyu Zhao, Chenyun Zhu, Jeanne Brooks-Gunn, Barbara E. Engelhardt, Moritz Hardt, Dean Knox, Karen Levy, Arvind Narayanan, Brandon M. Stew- art, Duncan J. Watts, and Sara McLanahan, “Measuring the predictability of life outcomes with a scientific mass collaboration,” Proceedings of the National Academy of Sciences, 2020, 117 (15), 8398–8403. Santurkar, Shibani, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto, “Whose Opinions Do Language Models Reflect?,” 2023. Schelling, Thomas C, “Models of segregation,” The American economic review, 1969, 59 (2), 488–493. , “Dynamic models of segregation,” Journal of mathematical sociology, 1971, 1 (2), 143–186. Scherrer, Nino, Claudia Shi, Amir Feder, and David Blei, “Evaluating the moral beliefs encoded in llms,” Advances in Neural Information Processing Sys- tems, 2024, 36. Simon, Herbert A., The Sciences of the Artificial, 3rd Edition number 0262691914. In ‘MIT Press Books.’, The MIT Press, September 1996. 37 | | | | | | | | | | | | | | | | | | |
| 37 | tmp83n2cnme.pdf | 38 | Turing, A. M., “On Computable Numbers, with an Application to the Entschei- dungsproblem,” Proceedings of the London Mathematical Society, 1937, s2-42 (1), 230–265. T¨ornberg, Petter, Diliara Valeeva, Justus Uitermark, and Christopher Bail, “Simulating Social Media Using Large Language Models to Evaluate Alter- native News Feed Algorithms,” 2023. Wager, Stefan and Susan Athey, “Estimation and Inference of Heterogeneous Treatment Effects using Random Forests,” Journal of the American Statistical Association, 2018, 113 (523), 1228–1242. Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou, “Chain of Thought Prompting Elicits Reasoning in Large Language Models,” CoRR, 2022, abs/2201.11903. Wright, Sewall, “The method of path coefficients,” The annals of mathematical statistics, 1934, 5 (3), 161–215. Zheng, Stephan, Alexander Trott, Sunil Srinivasa, David C Parkes, and Richard Socher, “The AI Economist: Taxation policy design via two-level deep multiagent reinforcement learning,” Science advances, 2022, 8 (18), eabk2607. 38 | | | | | | | | | | | | | | | | | | |
| 38 | tmp83n2cnme.pdf | 39 | A Implementation details The first step in the system’s process is to query an LLM for the roles of the relevant agents in the scenario. When we say “query an LLM,” we mean this quite literally. We have written a scenario-neutral prompt that the system provides to an LLM with the scenario added to the prompt. The prompt is scenario-neutral because we can reuse it for any scenario. The prompt takes the following format: In the following scenario: “\{scenario description}”, Who are the in- dividual human agents in a simple simulation of this scenario? where \{scenario description} is replaced with the scenario of interest. The LLM then returns a list of agents relevant to the scenario, and we have various checking mechanisms to ensure the LLM’s response is valid. The system contains over 50 pre-written scenario-neutral prompts to gather all the information needed to generate the SCM, run the experiment, and analyze the results. These prompts have placeholders for the necessary information aggregated in the system’s memory as it progresses through the different parts of the process. A.1 Constructing variables and drawing causal paths The system builds SCMs variable-by-variable. It queries an LLM for an outcome involving the agents in the social scenario of interest. We refer to outcomes as endogenous variables because their values are realized during the experiment. This contrasts exogenous variables, the causes, whose values are determined before the experiment. The system queries the LLM for a list of possible exogenous causes of the en- dogenous variable, generating a hypothesis as an SCM.27 Exogenous variables serve as inputs to the experiment, whose values can be deterministically manipulated to identify causal effects. The system assumes that when an exogenous variable causes an endogenous variable, a single causal path is proposed from the exogenous variable 27There is growing evidence that LLMs can be quite good at coming up with ideas and generating hypotheses (Girotra et al., 2023; Rosenbusch et al., 2023). 39 | | | | | | | | | | | | | | | | | | |
| 39 | tmp83n2cnme.pdf | 40 | to the endogenous variable. More formally, the system always generates SCMs as a simple linear model. The system currently generates all SCMs with one endogenous variable and as many exogenous causes as a researcher desires. We do little optimiza- tion here, although the system can test for interaction terms. In future iterations of the system, a researcher could choose outcomes and causes they are interested in, score hypotheses by interestingness, and generate more complex hypotheses with mediating endogenous variables.28 A.1.1 Endogenous outcomes For each endogenous variable, the system generates an operationalization, a type, the units, the possible levels, the explicit questions that need to be asked to measure the variable’s realized value, and how the answers to those questions will be aggregated to get the final data for analysis. Examples of all information collected about the variables in an SCM are provided in Table A.3. Each piece of information about a variable is stored by the system and is then used to determine subsequent informa- tion in consecutive scenario-neutral prompts. This is a kind of “chain-of-thoughts prompting”, or the process of breaking down a complex prompt into a series of sim- pler prompts. This method can dramatically improve the quality and robustness of an LLM’s performance (Wei et al., 2022). The first piece of information determined for each endogenous variable is the operationalization. That is, how the possible realizations of said variable can be directly mapped to measurable outcomes that can be observed and quantified. Sup- pose the outcome variable is whether or not a deal occurred from the SCM in Figure 2b.29 The system could operationalize this as a binary variable, where ‘‘1’’ means a deal occurred and ‘‘0’’ does not. It then stores this information and uses it in a scenario-neutral prompt to choose the variable type. All variables are determined to be one of five mutually exclusive “types.” These 28Parallel and crossover experimental designs can be used to identify mediating causal relation- ships (Imai et al., 2012). These experiments require few assumptions, which are often more plausible when researchers have more control over the experiment, as they usually do with LLMs. 29We continue the practice from Section 2 of using typewriter text to denote example infor- mation from the system. 40 | | | | | | | | | | | | | | | | | | |
| 40 | tmp83n2cnme.pdf | 41 | are continuous, ordinal, nominal, binary, or count. By selecting a unique type for each variable, the system can accommodate different distributions when estimating the fitted SCM after the experiment. Each variable also has units. The units are the specific measure or standard used to represent the variable’s quantified value. This information is used to improve the robustness and consistency of the system’s output when querying the LLM for other information about a variable. The levels of the variable represent all of the values the variable can realize in a short list. They can take on different forms depending on the variable type, but they all follow a general pattern where they are defined by the range and nature of a variable’s possible values.30 To measure the endogenous outcome, the system generates survey questions for one of the agents. For example, to measure whether or not a deal occurred, the system could ask the buyer or the seller, “Did you agree to buy the mug?” Or, if the endogenous variable was the final price of the mug, the system could ask one of the agents, “How much did you sell the mug for?” Even though the simulations have yet to be conducted, the system generates survey questions. As with pre-registration, this reduces unneeded degrees of freedom in the data collection process after the experiment. Most endogenous variables are measured with only one question. In this case, the answer to this question is the only information needed to quantify the variable. Sometimes, it takes more than one survey question to measure a variable. Maybe the variable is the average satisfaction of the buyer and the seller; a variable that requires two separate measurements to quantify. In this case, the system gener- ates separate measurement questions to elicit the buyer’s and the seller’s satisfaction. Then, the system averages the answers to the questions to measure the variable. 30For binary variables, the levels are the two possible outcomes. For ordinal variables, the levels include all possible values that the ordinal variable could take on as determined by its operational- ization. The levels are selected for count and continuous variables by segmenting the range of possible values into discrete intervals. In cases where the variable does not have a defined maxi- mum or minimum, categories such as “above X” or “below Y” are included to ensure all possible values are covered. 41 | | | | | | | | | | | | | | | | | | |
| 41 | tmp83n2cnme.pdf | 42 | We pre-programmed a menu of 6 mechanical aggregation methods: finding the minimum, maximum, average, mode, median, or sum of a list of values. If the system needs to combine the answers to multiple questions to measure a variable, it queries an LLM to select the appropriate aggregation method. Then, the system uses a pre-written Python function to perform said aggregation. We refrain from asking the LLM to perform mathematical functions whenever possible, as they often make mistakes. A.1.2 Exogenous causes Besides the explicit measurement questions and data aggregation method, the system collects the same information for the exogenous variables as it does for the endogenous variables. For exogenous variables, these two pieces of information are unnecessary for measurement. In each simulation of the social scenario, a different combination of the values of the exogenous variables is initialized. This is how the system induces variation in an experiment, so the treatments are always known to the system ex- ante. Causal variables can have one of two possible “scopes.” The scope can be specific to an individual agent or the scenario as a whole. This scope determines how the system induces variation in the exogenous variables—at the agent or scenario level. Individual-level variables are further designated as either public or private. If private, the variable’s values are only provided to one agent; if public, they are treated as common knowledge to all agents in the scenario. The system induces variation in the exogenous variables by transforming them into manageable proxy attributes for the agents. The system queries an LLM to cre- ate a second-person phrasing of the operationalized variable provided to the agent (or agents, depending on the scope). For instance, with the buyer’s budget vari- able, the attribute could be “your budget” for the buyer. These attributes will be assigned to the agents, which we discuss in Section A.2. With the proxy attribute for the variable, the system queries an LLM for possible values the attribute can take on. These are the induced variations—the treatment 42 | | | | | | | | | | | | | | | | | | |
| 42 | tmp83n2cnme.pdf | 43 | conditions for the simulated experiments. By default, the system uses the levels, or a value within each level, of the variable for the possible variation values. For example, these could be `{$5 ,$10, $20 ,$40}` for the buyer’s budget. A.2 Building hypothesis-driven agents In conventional social science research, human subjects are catch as catch can. Here, we have to construct them from scratch. By “construct” we mean that we prompt an LLM to be a person with a set of attributes. This is quite literal; for example, we could construct an agent in a negotiating scenario with the following prompt: “You are a buyer in a negotiation scenario with a seller. You are negoti- ating over a mug. You have a budget of \$20.” We can construct an agent with any set of attributes we want, which raises the question of what attributes we should use. We already have the attributes that will be varied to test the SCM, but there are many others we could include. Some work has explored the endowing of agents with many different attributes, but it is unclear what is optimal, sufficient, or even neces- sary.31 We take a minimalist approach, endowing our agents with goals, constraints, roles, names, and any relevant proxy attributes for the exogenous variables. In the future, we could integrate large numbers of diverse agents, perhaps constructed to be representative of some specific population. A.2.1 Assigning agents attributes The system collects information for agents independently, similar to its one-at-a-time approach with the variables in the SCM. The system randomly selects an agent, 31The methods have varied, ranging from endowing agents with interesting attributes (Argyle et al., 2023; Horton, 2023) to using American National Election Study data to create “real” people (T¨ornberg et al., 2023) to demonstrating that endowing demographic information does not nec- essarily represent a population of interest (Atari et al., 2023; Santurkar et al., 2023). There is a balance to be struck. While attributes can provide a rich and nuanced simulation, they can also lead to redundancy, inefficiency, and unexpected interactions. In contrast, too few attributes might result in an oversimplified and unrealistic portrayal of social interactions. 43 | | | | | | | | | | | | | | | | | | |
| 43 | tmp83n2cnme.pdf | 44 | determines its attributes, and then moves on to the next agent.32 Examples of buyer and seller agents with their attributes are provided in Figure A.1. Figure A.1: Example agents generated by the system for “two people bargaining over a mug” Notes: In all simulations, agents are endowed with a randomly generated name, role, goal, con- straint, and proxy attributes for the exogenous variables. To simulate the experiment for the agents in this figure, the system will generate four versions of the seller and four versions of the buyer, each with one of the values for the exogenously varied attributes (assuming there are four possible values for “Your sentimental attachment”). That is 4 × 4 = 16 treatments. For each agent, the system queries the LLM for a random name. Agents perform better in simulations with identifiers to address one another, although this feature can be disabled. An agent’s name can also be varied as a proposed exogenous cause. The system then queries an LLM again, this time for a goal and then a constraint, which we discuss in the following subsection. Finally, the system cross-checks the values of the proxy attributes between the agents to ensure they overlap appropriately. For example, if the two exogenous vari- ables in the SCM were the buyer’s budget and the seller’s minimum acceptable 32The system already has the agent’s roles from the construction of the SCM. 44 | | | | | | | | | | | | | | | | | | |
| 44 | tmp83n2cnme.pdf | 45 | price, the system would check to make sure that the seller’s minimum acceptable price is not invariably higher than the buyer’s budget. We let the LLM deter- mine if these attribute values overlap appropriately. If any discrepancies are found, the system queries the LLM again to resolve them with new values for the proxy attributes. Otherwise, the simulated experiment would waste time and resources because the induced variations were not supported across reasonable values. For ex- ample, if the buyer’s budget was always below the seller’s minimum acceptable price, then they might never make a deal. A.2.2 The importance of agent goals Unlike, say, economic agents, whose goals are expressed via explicit utility func- tions, the LLM agent’s goals are expressed in natural language. In the context of our bargaining scenario, an example goal generated by our system for the seller is to sell the mug at the highest price possible. An example constraint is to not accept a price below your minimum selling price. These goals and constraints are oriented towards value, but they do not have to be; these are merely the ones generated by the system. A constraint could just have easily been do not ruin your reputation with your negotiating partner. We do not take a prescriptive stance on what these goals should be. We let the system decide what is reasonable. These goals can, of course, also be the object of study in their own right; researchers can vary them or choose their own, but they are seemingly fundamental to any social science for reasons laid out in Simon (1996). Therefore, explicit goals are a requirement for agents in our system. A.3 Simulation design and execution LLMs are designed to produce text. And since an independent LLM powers each agent, one agent must finish speaking before the next begins. So, in any multi-agent simulation, there must be a speaking order, which raises the question of how the system should determine this speaking order. Unfortunately, most human conver- sations do not have an obvious order; people collectively figure out how to interact. 45 | | | | | | | | | | | | | | | | | | |
| 45 | tmp83n2cnme.pdf | 46 | We centralize this process, but we could imagine a consensus protocol for who speaks next. In more straightforward settings with only two agents (e.g., two people bargaining over a mug), the only possible conversational order is for the agents to alternate speaking. As the number of agents in interaction increases beyond two, the number of possible speaking orders grows factorially. For example, with three agents, there are 3! = 6 ways to order them; with 4 agents, 4! = 24 orderings, and so on. However, the number of possible orderings of the agents is only part of the complexity. Who speaks next in a given conversation is a product of the participants’ per- sonalities, the setting of the conversation, the social dynamics between the speakers, the emotional state of the participants, and many other factors. They are also adaptive—often, the speaking order changes throughout a conversation. For exam- ple, in a court proceeding, the judge usually guides the interaction—signaling who speaks between the lawyers, witnesses, and the jury. Each contributes at various and irregular intervals depending on both the type and stage of the proceeding. In a family of two parents and two children, the order of who speaks next varies greatly. It might depend on the parents’ moods or how annoying the children have been that day. In contrast, the teacher is typically the main speaker in a high school classroom, although this varies depending on the classroom activity, such as a lecture versus a group discussion. No simple universal formula exists for who speaks next in such diverse settings. Like the aggregation methods for outcomes determined by multiple measurement questions, we designed a menu of six interaction protocols. The system queries an LLM to select the appropriate protocol for a given scenario. Figure A.2 provides the menu, and we discuss each in turn. A.3.1 Turn-taking protocols The first interaction protocol is the ordered protocol (Figure A.2, option 1), where the agents speak in a predetermined order and continue repeatedly speaking in that order until the simulation is complete. Next is the random protocol. An agent is 46 | | | | | | | | | | | | | | | | | | |
| 46 | tmp83n2cnme.pdf | 47 | Figure A.2: Menu of interaction protocols for the system to choose from for a given scenario. Notes: (1) The agents speak in a predetermined order. (2) The agents speak in a random order. (3) A central agent alternates speaking with non-central agents in a predetermined order. (4) A central agent alternates speaking with non-central agents in random order. (5) A separate LLM (whom we call the coordinator) determines who speaks next based on the conversation. (6) Each agent responds privately to the conversation so far, and the coordinator realizes one of the responses. randomly selected to speak first (Figure A.2, option 2). Then, each subsequent speaker is randomly selected, with the only restriction being that no agent can speak twice in a row. In more complex scenarios with a central agent—an agent that speaks more than all others—like an auction with an auctioneer or a teacher in a classroom, the system can choose the central-ordered or central-random protocols (Figure A.2, options 3 and 4). The former features a central agent who interacts alternately with a series of non-central agents, following a predetermined order among the non-central agents. The latter also has a central agent alternating with the non-central agents but in random order. Whenever there is an order of agents or a central agent, we also query 47 | | | | | | | | | | | | | | | | | | |
| 47 | tmp83n2cnme.pdf | 48 | the system to determine this order. Finally, we designed two interaction protocols that provide more flexibility. These interaction protocols involve a separate LLM-powered agent: “the coordinator.” The coordinator can read through transcripts of the conversations and make decisions about the simulations when necessary. It can also answer measurement questions after the experiment. The agents are not aware of the coordinator. The use of the coordinator is the only part of the system that needs quasi-omniscient supervision. Fortunately, LLMs perform so well that they can be used to automate this role. In the coordinator-before protocol (Figure A.2, option 5), the coordinator is given the transcript of the conversation after each agent speaks. Then, it selects the next speaker. In the coordinator-after protocol (Figure A.2, option 6), after each agent speaks, all the agents respond, but only the coordinator can see the responses along with the transcript of the conversation up to that point. Then, the coordinator chooses the response to “realize” as the real response. The realized response is added to the conversation’s transcript, and the rest are deleted as if they had never been made. The only limitation in either of the coordinator protocols is that no agent can speak twice in a row. A.3.2 Executing the experimental simulations The system runs each experimental simulation in parallel, subject to the computa- tional constraints of the researcher’s machine. When the exogenous variable’s values present too many combinations to sample from, a subset is randomly selected. In every simulation, agents are provided with a description of the scenario, their unique private attributes, the other agents’ roles, any public or scenario-level attributes, and access to the transcript of the conversation. Then, they interact according to the chosen interaction protocol. However, none of the protocols specify when the simulation should end. It is not obvious how to construct an optimal, nor even good, stopping rule. Human conversations are unpredictable and do not always end when we expect them 48 | | | | | | | | | | | | | | | | | | |
| 48 | tmp83n2cnme.pdf | 49 | to or want them to (Mastroianni et al., 2021). An analogous issue is the halting problem in computer science, which is the problem of determining when, if ever, an arbitrary computer program will stop. Turing (1937) proved that no universal algorithm exists to solve the halting problem. We implemented a two-tier mechanism to determine when to stop each simulation. These apply to all interaction protocols. After each agent speaks, the coordinator receives the transcript and decides if the conversation should continue—a yes or no decision. Additionally, simulations are limited to 20 statements across all agents in the scenario, not including the coordinator.33 Agents are provided a live count of the remaining statements during the conversation. A.3.3 Post-simulation survey and data collection After the experiment, the system conducts a post-experiment survey. As determined during the SCM construction, the system asks the relevant agents or the coordinator the survey questions to measure the outcome variable in each simulation. The system then takes this question’s raw answer and saves it as an observation along with the values of the exogenous variables. If there is no reasonable answer to the question, say, if the outcome is conditional, then the system will report an NA for the variable’s value. Once the system has the answer to the survey question, it queries an LLM with the survey question, the agent’s response, and information about the variable’s type to determine its correct numerical value as a string. If the variable is a count or continuous variable, it is converted into an integer or a float. If the variable is ordinal or binary, the system queries an LLM to map it to a whole-number integer sequence. If multiple survey questions determine a variable, the system aggregates the answers to the questions using the method selected during the SCM construction phase. Then, it converts the aggregated value to the appropriate type. After parsing 33Limiting the number of turns in the simulation is partially a convenience. As of the time of running the simulations for this paper, GPT-4 has a maximum token limit of 8,192 tokens, and the system must provide each agent with the entire conversation up to that point each time they need to speak. 49 | | | | | | | | | | | | | | | | | | |
| 49 | tmp83n2cnme.pdf | 50 | the data for each outcome, the system has a data frame with one column of numerical values for each variable in the SCM. A.4 Path estimation & model fit With a complete dataset and the proposed SCM, the system can estimate the linear SCM without further queries to an LLM. The system uses the R package lavaan to estimate all paths in the model (Rosseel, 2012).34 The system can standardize all estimates, estimate interactions and non-linear terms, and view various summary statistics for each variable. It can also provide likelihood ratio, Wald, and Lagrange Multiplier tests to evaluate the model fit and compare path estimates. The system can do any statistical estimation or test that is built into lavaan. A.5 Follow-on experiments Although we have not yet automated this process, the system can perform follow- on experiments. Insignificant exogenous variables from the first experiment can be dropped. Then, the system could query an LLM for new exogenous variables based on what might be interesting, given the already tested causal paths. The system would use the same agents and interaction protocol, but the agents would vary on the new exogenous variables and the old ones that were significant in the first experiment. Theoretically, the system can run follow-on experiments ad infinitum, and we can imagine future models that could be very good at proposing potential causal relationships. B Hypotheses as structural causal models Hypotheses stated in natural language can be ambiguous, making it challenging to discern precise implied causal relationships. Suppose a researcher is interested in 34For those familiar with lavaan and Python, the system automatically generates the correctly formatted string in lavaan syntax using a Python dictionary that stores the structure of the SCM in key-value pairs. 50 | | | | | | | | | | | | | | | | | | |
| 50 | tmp83n2cnme.pdf | 51 | Figure A.3: Valid graphical interpretations of the same natural language hypothesis. Buyer Budget Seller Attach Deal Occurs (a) Independent causes Buyer Budget Seller Attach Deal Occurs (b) Mediation Buyer Budget Seller Attach Deal Occurs (c) Alternative mediation Notes: Each directed acyclic graph (DAG) is a valid causal interpretation of the following natu- ral language hypothesis: “The buyer’s budget and the seller’s sentimental attachment to the mug causally affect whether a deal occurs.” In contrast, each DAG is unique in its declaration of the causal relationships. In DAGs, each arrow represents a direct causal relationship, and the absence of an arrow between two variables indicates no causal relationship. If a variable is not included in the graph, then there is no stated causal relationship about this variable. While DAGs are unam- biguous in their causal claims about which variables cause which other variables, they do not make any claims about the functional form of the relationships between variables. two-person bargaining scenarios with a buyer and a seller. And she has the following natural language hypothesis about two people bargaining over a mug: “the buyer’s budget and the seller’s sentimental attachment to the mug causally affect whether a deal occurs.” Figure A.3 offers three ways we can interpret this causal state- ment: (A.3a) the budget and the sentimental attachment could independently affect whether a deal occurs, (A.3b) the budget could mediate the relationship between the attachment and the outcome, or (A.3c), the mediation could be reversed.35 For (A.3a), an example could be an online marketplace where the buyer and seller cannot communicate. When the buyer has a higher budget, she is more likely to buy the mug. If the seller is more sentimentally attached to the mug, he may raise the price and, therefore, lower the probability of a deal. However, without any form of communication, these causal variables would not affect each other. For (A.3b), if the buyer and the seller can communicate and the seller realizes that the buyer 35This list of interpretations is not exhaustive. 51 | | | | | | | | | | | | | | | | | | |
| 51 | tmp83n2cnme.pdf | 52 | is willing to spend more, he might become more attached to the mug and value it higher because of the increased potential sale price. Finally, for (A.3c), the mediated relationship could be reversed. If the buyer sees that the seller is attached to the mug, this may cause her to increase her budget, which increases the probability of a deal. The ambiguity of stating even simple hypotheses makes natural language insufficient for our purposes. The graphs in Figure A.3 are directed acyclic graphs (DAGs) and represent causal relationships. DAGs unambiguously state whether a variable is a direct cause of another variable—the direction of the arrow indicates the direction of the causal relationship (Hern´an and Robins, 2020). The absence of an arrow between two variables indicates no causal relationship. If a variable is not included in the graph, then there is no stated causal relationship involving this variable. While DAGs are clear in their claims about which variables cause others, they do not make any statements about the functional form of the relationships between variables. In contrast, structural causal models unambiguously state the causal re- lationships between variables and the functional forms of these relationships (Pearl et al., 2016). Structural causal models (SCM), as first explored by Wright (1934), represent hypotheses as sets of equations. Suppose we assume the relationships between the variables in Figure A.3 are linear. We can write an SCM for each of the DAGs. Figure A.3a can be stated as: DealOccurs = β1BuyerBudget + β2SellerAttachment + ϵ; (1) Figure A.3b as: BuyerBudget = β0SellerAttachment + η (2) DealOccurs = β1BuyerBudget + β2SellerAttachment + ϵ; (3) and Figure A.3c as: SellerAttachment = β0BuyerBudget + η (4) DealOccurs = β1BuyerBudget + β2SellerAttachment + ϵ. (5) The set of equations that represent the causal relationships between variables make the SCM. We could also write each SCM with interaction terms for some or 52 | | | | | | | | | | | | | | | | | | |
| 52 | tmp83n2cnme.pdf | 53 | all of the causes or even use other types of link functions, and these would all be equally valid representations of the corresponding DAGs. 53 | | | | | | | | | | | | | | | | | | |
| 53 | tmp83n2cnme.pdf | 54 | C Additional figures and tables Figure A.4: Fitted SCM with interaction terms for “two people bargaining over a mug.” deal-for-mug µ = 0.50 σ2 = 0.25 sell-love-mug µ = 3.00 σ2 = 2.00 buyers-budget -x- sell-love-mug µ = 36.67 σ2 = 826.22 buyers-budget -x- sell-min-mug µ = 148.02 σ2 = 16787.95 sell-min-mug µ = 12.11 σ2 = 49.43 sell-min-mug -x- sell-love-mug µ = 36.33 σ2 = 837.11 buyers-budget µ = 12.22 σ2 = 47.95 0.032 (0.007) -0.045 (0.007) -0.094 (0.032) -0.000 (0.000) 0.002 (0.002) 0.004 (0.002) Notes: Each variable is given with its mean and variance. The edges are labeled with their unstan- dardized path estimate and standard error. There were 405 simulations with these agents: \[‘buyer’, ‘seller’]. 54 | | | | | | | | | | | | | | | | | | |
| 54 | tmp83n2cnme.pdf | 55 | Figure A.5: Fitted SCM with interaction terms for “a judge is setting bail for a criminal defendant who committed 50,000 dollars in tax fraud.” bail-amt µ = 54428.57 σ2 = 186000000.00 num-judge-cases -x- def-remorse µ = 29.57 σ2 = 865.10 def-crim-hist µ = 4.71 σ2 = 17.06 def-crim-hist -x- num-judge-cases µ = 46.47 σ2 = 4053.35 def-crim-hist -x- def-remorse µ = 14.14 σ2 = 232.12 def-remorse µ = 3.00 σ2 = 2.00 num-judge-cases µ = 9.86 σ2 = 60.98 303.4 (545.5) 383.9 (282.6) -29.6 (1180.9) -1.301 (26.231) 77.0 (144.8) -150.8 (76.6) Notes: Each variable is given with its mean and variance. The edges are labeled with their unstan- dardized path estimate and standard error. There were 245 simulations with these agents: \[‘judge’, ‘defendant’, ‘defense attorney’, ‘prosecutor’]. 55 | | | | | | | | | | | | | | | | | | |
| 55 | tmp83n2cnme.pdf | 56 | Figure A.6: Fitted SCM with interaction terms for “a person is interviewing for a job as a lawyer.” hire-decision µ = 0.62 σ2 = 0.23 inter-friendly -x- job-app-height µ = 2130.00 σ2 = 1600775.00 job-app-height µ = 177.50 σ2 = 131.25 bar-exam-pass -x- job-app-height µ = 88.75 σ2 = 7942.19 bar-exam-pass -x- inter-friendly µ = 6.00 σ2 = 61.00 bar-exam-pass µ = 0.50 σ2 = 0.25 inter-friendly µ = 12.00 σ2 = 50.00 1.704 (1.053) -0.013 (0.074) 0.005 (0.007) 0.005 (0.010) -0.006 (0.006) 0.000 (0.000) Notes: Each variable is given with its mean and variance. The edges are labeled with their un- standardized path estimate and standard error. There were 80 simulations with these agents: \[‘job applicant’, ‘employer’]. 56 | | | | | | | | | | | | | | | | | | |
| 56 | tmp83n2cnme.pdf | 57 | Figure A.7: Fitted SCM with interaction terms for “3 bidders participating in an auction for a piece of art starting at fifty dollars.” final-art-price µ = 186.53 σ2 = 3867.92 bid1-max-budget -x- bid2-max-budg µ = 40000.00 σ2 = 900000000.00 bid1-max-budget µ = 200.00 σ2 = 10000.00 bid2-max-budg -x- bid3-max-budg µ = 40000.00 σ2 = 900000000.00 bid3-max-budg µ = 200.00 σ2 = 10000.00 bid1-max-budget -x- bid3-max-budg µ = 40000.00 σ2 = 900000000.00 bid2-max-budg µ = 200.00 σ2 = 10000.00 0.136 (0.044) 0.120 (0.044) 0.171 (0.044) 0.001 (0.000) 0.000 (0.000) 0.000 (0.000) Notes: Each variable is given with its mean and variance. The edges are labeled with their unstan- dardized path estimate and standard error. There were 343 simulations with these agents: \[‘bidder 1’, ‘bidder 2’, ‘bidder 3’, ‘auctioneer’]. 57 | | | | | | | | | | | | | | | | | | |
| 57 | tmp83n2cnme.pdf | 58 | Table A.1: GPT-4’s predictions for the path estimates for the experiments in Section 3 at temperature 0. Scenario (Outcome) Exogenous Variable Path Estimate (SE) GPT-4 Guess Two- tailed T-Test GPT-4 Sign Correct | Predicted Experiment | Estimates Mug Bargaining (Deal Made) Buyer’s Budget 0.037\* (0.003) 0.05\* p \< 0.001 Yes 1.35 Seller’s Min Price -0.035\* (0.002) -0.07\* p \< 0.001 Yes 2.00 Seller’s Attachment -0.025\* (0.012) 0.02 p \< 0.001 No 0.80 Art Auction (Final Price) Bidder 1 Budget 0.35\* (0.015) 0.5\* p \< 0.001 Yes 1.43 Bidder 2 Valuation 0.29\* (0.015) 0.5\* p \< 0.001 Yes 1.72 Bidder 3 Valuation 0.31\* (0.015 ) 0.5\* p \< 0.001 Yes 1.610 Bail Hearing (Bail Amount) Defendant’s Previous Convictions 521.53\* (206.567) 5000\* p \< 0.001 Yes 9.59 Judge Cases That Day -74.632 (109.263) -200 p = 0.252 Yes 2.68 Defendant’s Remorse -1153.061 (603.325) -3000\* p = 0.002 Yes 2.60 Lawyer Interview (Gets Job) Passed Bar 0.750\* (0.068) 0.6\* p = 0.03 Yes 0.80 Interviewer Friendliness -0.002 (0.005) 0.2 p \< 0.001 No 100.00 Applicant’s Height 0.003 (0.003) 0.1 p \< 0.001 Yes 33.33 Notes: The table provides GPT-4’s prediction for the path estimate for each experiment in Section 3 From left to right, column 1 provides the scenario and outcome, column 2 provides the causal variable name, column 3 the path estimate and its standard error, and column 4 shows the LLM’s prediction for the path estimate and whether it was predicted to be statistically significant. Column 5 gives the p-value of a two-tailed t-test comparing the predictions to the results, column 6 is whether the predicted sign of the estimate was correct, and column 7 is the magnitude of the difference between the predicted and actual estimate. 58 | | | | | | | | | | | | | | | | |
| 58 | tmp83n2cnme.pdf | 59 | Table A.2: GPT-4’s predictions for the path estimates for the experiments in Section 3 at temperature 1. Scenario (Outcome) Exogenous Variable Path Estimate (SE) GPT-4 Guess Two- tailed T-Test GPT-4 Sign Correct (SE) | Predicted Experiment | Estimates Mug Bargaining (Deal Made) Buyer’s Budget 0.037\* (0.003) 0.117\* (0.016) p \< 0.001 Yes 3.16 Seller’s Min Price -0.035\* (0.002) 0.008\* (0.018) p = 0.019 No 0.23 Seller’s Attachment -0.025\* (0.012) 0.062 (0.013) p \< 0.001 No 2.48 Art Auction (Final Price) Bidder 1 Budget 0.35\* (0.015) 1.279\* (0.501) p = 0.064 Yes 3.65 Bidder 2 Valuation 0.29\* (0.015) 1.263\* (0.501) p = 0.053 Yes 4.36 Bidder 3 Valuation 0.31\* (0.015 ) 1.269\* (0.501) p = 0.056 Yes 4.09 Bail Hearing (Bail Amount) Defendant’s Previous Convictions 521.53\* (206.567) 1785.192\* (157.347) p \< 0.001 Yes 3.42 Judge Cases That Day -74.632 (109.263) 644.316\* (79.919) p \< 0.001 No 8.63 Defendant’s Remorse -1153.061 (603.325) -879.945\* (92.700) p = 0.09 Yes 0.76 Lawyer Interview (Gets Job) Passed Bar 0.750\* (0.068) 0.408\* (0.018) p = 0.998 Yes 0.54 Interviewer Friendliness -0.002 (0.005) 0.236\* (0.015) p = 0.999 No 118 Applicant’s Height 0.003 (0.003) 0.108 (0.009) p = 0.999 Yes 36 Notes: The table provides GPT-4’s prediction for the path estimate for each experiment in Section 3 Each prediction is the average of 100 prompts at temperature 1. From left to right, column 1 provides the scenario and outcome, column 2 provides the causal variable name, column 3 the path estimate and its standard error, and column 4 shows the LLM’s average prediction for the path estimate and whether it was predicted to be statistically significant more than 50% of the time. The given standard error is for the mean of the predictions, not the LLM’s prediction for the standard error. Column 5 gives the p-value of a two-tailed t-test comparing the average prediction to the results, column 6 is whether the predicted sign of the estimate was correct more than 50% of the time, and column 7 is the magnitude of the difference between the predicted and actual estimate. 59 | | | | | | | | | | | | | | | | |
| 59 | tmp83n2cnme.pdf | 60 | Figure A.8: Fitted SCM for auction with bidder’s reservation prices and second highest bid as exogenous variables. final-art-price µ = 186.53 σ2 = 3867.92 bid1-max-budget µ = 200.00 σ2 = 10000.00 bid2-max-budg µ = 200.00 σ2 = 10000.00 bid3-max-budg µ = 200.00 σ2 = 10000.00 2nd-highest-budget µ = 180.99 σ2 = 4565.99 0.047 (0.009) 0.039 (0.008) 0.03 (0.009) final-art-price µ = 186.53 σ2 = 3867.92 0.826 (0.018) Notes: Each variable is given with its mean and variance. The edges are labeled with their unstan- dardized path estimate and standard error. There were 343 simulations with these agents: \[‘bidder 1’, ‘bidder 2’, ‘bidder 3’, ‘auctioneer’]. Figure A.9: Fitted SCM for auction and second highest bid as exogenous variables. final-art-price µ = 186.53 σ2 = 3867.92 2nd-highest-budget µ = 180.99 σ2 = 4565.99 0.912 (0.009) Notes: Each variable is given with its mean and variance. The edges are labeled with their unstan- dardized path estimate and standard error. There were 343 simulations with these agents: \[‘bidder 1’, ‘bidder 2’, ‘bidder 3’, ‘auctioneer’]. 60 | | | | | | | | | | | | | | | | | | |
| 60 | tmp83n2cnme.pdf | 61 | Figure A.10: Comparison of the LLM’s predictions to the theoretical predictions and all experimental results for the auction scenario. Bidder 3 Reservation: 50 Bidder 3 Reservation: 100 Bidder 3 Reservation: 150 Bidder 3 Reservation: 200 Bidder 3 Reservation: 250 Bidder 3 Reservation: 300 Bidder 3 Reservation: 350 Bidder 2 Reservation: 350 Bidder 2 Reservation: 300 Bidder 2 Reservation: 250 Bidder 2 Reservation: 200 Bidder 2 Reservation: 150 Bidder 2 Reservation: 100 Bidder 2 Reservation: 50 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 100 200 300 Bidder 1 Reservation Price Final Clearing Price Auction Theory Predict yi | β^ Predict yi Experiment Notes: The columns correspond to the different reservation values for bidder 3 in a given simulation, and the rows correspond to the different reservation values for bidder 2. The y-axis is the clearing price, and the x-axis lists bidder 1’s reservation price. The black triangles track the observed clearing price in each simulated experiment, the black line shows the predictions made by auction theory (MSET heory = 128), the blue line indicates the LLM’s predictions without the fitted SCM—the predict-yi task (MSEyi = 8628), and the red curve is the LLM’s predictions with the fitted SCM— the predict-yi | ˆβ−i task (MSEyi | ˆβ−i = 1505). 61 | | | | | | | | | | | | | | | |
| 61 | tmp83n2cnme.pdf | 62 | Figure A.11: Prompt used to elicity LLM predictions for the Predict-ˆβ task. I have just run an experiment to estimate the paths in the SCM from the TIKZ diagram below, which is delineated by triple backticks. We ran the experiment on multiple instances of GPT-4, once for each combination of the different “Attribute Treatments” in the accompanying table. This table also includes information about the variables and the individual agents involved in the scenario. Your task is to predict the point estimates for the paths in the SCMs as accurately as possible based on the experiments. You can see the summary statistics of the treatment variables below each variable name in the Tikz Diagram. We want to know how good you are at predicting the outcomes of experiments run on you. Make sure you consider the correct units for both the cause and the outcome for each path. Please output your answer in the following form and do not include any other text: \{’predictions’: dictionary of point estimate predictions for each path} \{’sig’: dictionary of whether or not each path is significant} ‘‘‘Figure X and Table X’’’ Notes: For each experiment, we input the accompanying table and the TIKZ diagram into the LLM between the triple backticks. For example, for the bargaining scenario, these are Figure 2b and Table 2a. 62 | | | | | | | | | | | | | | | | | | |
| 62 | tmp83n2cnme.pdf | 63 | Table A.3: Example of the information generated for each variable in an SCM. Information Type Deal Occurred (Endogenous) Buyer’s Budget (Exogenous) Seller’s Attachment (Exogenous) Operationalization 1 if a deal occurs, 0 otherwise Max amount the buyer will spend Seller’s emotional attachment level on a scale Variable Type Binary Continuous Ordinal Units Binary Dollars Levels of attachment Levels `{0, 1}` `{$0 -$5, ..., $40+}` `{Low, ..., High}` Explicit Measurement Questions Buyer: ‘‘Did a deal occur?’’ - - Data Aggregation Method Single Value - - Scenario or Individual - Individual Individual Varied Attribute Proxies - ‘‘Your budget’’ ‘‘Your attachment level’’ Attribute Treatments - `{$3 , . . . ,$45}` `{no attachment, ..., extreme attachment}` Notes: Each row shows a different piece of information generated for the variables in the SCM. The first column represents the type of information, the second column represents the information for the endogenous variable, and the third and fourth columns represent the information for the exogenous variables. This is example information based on the SCM in Figure A.3a. 63 | | | | | | | | | | | | | | | | | | |

We can select pages to use if we do not want to use all of them – e.g., here we filter just the first page to use with our survey:

```python theme={null}
scenarios_page1 = scenarios.filter("page == 1")
scenarios_page1
```

```
ScenarioList scenarios: 1; keys: ['page', 'text', 'filename'];
```

| | filename | page | text |
| - | - | - | - |
| 0 | tmp83n2cnme.pdf | 1 | Automated Social Science: Language Models as Scientist and Subjects∗ Benjamin S. Manning† MIT Kehang Zhu† Harvard John J. Horton MIT & NBER April 26, 2024 Abstract We present an approach for automatically generating and testing, in silico, social scientific hypotheses. This automation is made possible by recent ad- vances in large language models (LLM), but the key feature of the approach is the use of structural causal models. Structural causal models provide a lan- guage to state hypotheses, a blueprint for constructing LLM-based agents, an experimental design, and a plan for data analysis. The fitted structural causal model becomes an object available for prediction or the planning of follow-on experiments. We demonstrate the approach with several scenarios: a nego- tiation, a bail hearing, a job interview, and an auction. In each case, causal relationships are both proposed and tested by the system, finding evidence for some and not others. We provide evidence that the insights from these simulations of social interactions are not available to the LLM purely through direct elicitation. When given its proposed structural causal model for each scenario, the LLM is good at predicting the signs of estimated effects, but it cannot reliably predict the magnitudes of those estimates. In the auction experiment, the in silico simulation results closely match the predictions of auction theory, but elicited predictions of the clearing prices from the LLM are inaccurate. However, the LLM’s predictions are dramatically improved if the model can condition on the fitted structural causal model. In short, the LLM knows more than it can (immediately) tell. ∗Thanks to generous support from Drew Houston and his AI for Augmentation and Productivity seed grant. Thanks to Jordan Ellenberg, Benjamin Lira Luttges, David Holtz, Bruce Sacerdote, Paul R¨ottger, Mohammed Alsobay, Ray Duch, Matt Schwartz, David Autor, and Dean Eckles for their helpful feedback. Author’s contact information, code, and data are currently or will be available at [http://www.benjaminmanning.io/](http://www.benjaminmanning.io/). †Both authors contributed equally to this work. 1 arXiv:2404.11794v2 \[econ.GN] 25 Apr 2024 |

Here we create a survey of questions that we will administer with the selected PDF page, using a `{{ placeholder }}` for the text extracted from the page:

```python theme={null}
from edsl import QuestionFreeText, QuestionList, ScenarioList, Survey

q_summary = QuestionFreeText(
    question_name="summary",
    question_text="Briefly summarize the abstract of this paper: {{ scenario.text }}",
)

q_authors = QuestionList(
    question_name="authors",
    question_text="List the names of all the authors of the following paper: {{ scenario.text }}",
)

q_thanks = QuestionList(
    question_name="thanks",
    question_text="List the names of the people thanked in the following paper: {{ scenario.text }}",
)

survey = Survey(questions = [q_summary, q_authors, q_thanks])
```

Now we can add the scenario to the survey and run it:

```python theme={null}
results = survey.by(scenarios_page1).run()
```

We can see a list of all the components of results that are directly accessible:

```pyhton theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.authors |
| 4 | answer.summary |
| 5 | answer.thanks |
| 6 | cache\_keys.authors\_cache\_key |
| 7 | cache\_keys.summary\_cache\_key |
| 8 | cache\_keys.thanks\_cache\_key |
| 9 | cache\_used.authors\_cache\_used |
| 10 | cache\_used.summary\_cache\_used |
| 11 | cache\_used.thanks\_cache\_used |
| 12 | comment.authors\_comment |
| 13 | comment.summary\_comment |
| 14 | comment.thanks\_comment |
| 15 | generated\_tokens.authors\_generated\_tokens |
| 16 | generated\_tokens.summary\_generated\_tokens |
| 17 | generated\_tokens.thanks\_generated\_tokens |
| 18 | iteration.iteration |
| 19 | model.frequency\_penalty |
| 20 | model.inference\_service |
| 21 | model.logprobs |
| 22 | model.max\_tokens |
| 23 | model.model |
| 24 | model.model\_index |
| 25 | model.presence\_penalty |
| 26 | model.temperature |
| 27 | model.top\_logprobs |
| 28 | model.top\_p |
| 29 | prompt.authors\_system\_prompt |
| 30 | prompt.authors\_user\_prompt |
| 31 | prompt.summary\_system\_prompt |
| 32 | prompt.summary\_user\_prompt |
| 33 | prompt.thanks\_system\_prompt |
| 34 | prompt.thanks\_user\_prompt |
| 35 | question\_options.authors\_question\_options |
| 36 | question\_options.summary\_question\_options |
| 37 | question\_options.thanks\_question\_options |
| 38 | question\_text.authors\_question\_text |
| 39 | question\_text.summary\_question\_text |
| 40 | question\_text.thanks\_question\_text |
| 41 | question\_type.authors\_question\_type |
| 42 | question\_type.summary\_question\_type |
| 43 | question\_type.thanks\_question\_type |
| 44 | raw\_model\_response.authors\_cost |
| 45 | raw\_model\_response.authors\_input\_price\_per\_million\_tokens |
| 46 | raw\_model\_response.authors\_input\_tokens |
| 47 | raw\_model\_response.authors\_one\_usd\_buys |
| 48 | raw\_model\_response.authors\_output\_price\_per\_million\_tokens |
| 49 | raw\_model\_response.authors\_output\_tokens |
| 50 | raw\_model\_response.authors\_raw\_model\_response |
| 51 | raw\_model\_response.summary\_cost |
| 52 | raw\_model\_response.summary\_input\_price\_per\_million\_tokens |
| 53 | raw\_model\_response.summary\_input\_tokens |
| 54 | raw\_model\_response.summary\_one\_usd\_buys |
| 55 | raw\_model\_response.summary\_output\_price\_per\_million\_tokens |
| 56 | raw\_model\_response.summary\_output\_tokens |
| 57 | raw\_model\_response.summary\_raw\_model\_response |
| 58 | raw\_model\_response.thanks\_cost |
| 59 | raw\_model\_response.thanks\_input\_price\_per\_million\_tokens |
| 60 | raw\_model\_response.thanks\_input\_tokens |
| 61 | raw\_model\_response.thanks\_one\_usd\_buys |
| 62 | raw\_model\_response.thanks\_output\_price\_per\_million\_tokens |
| 63 | raw\_model\_response.thanks\_output\_tokens |
| 64 | raw\_model\_response.thanks\_raw\_model\_response |
| 65 | reasoning\_summary.authors\_reasoning\_summary |
| 66 | reasoning\_summary.summary\_reasoning\_summary |
| 67 | reasoning\_summary.thanks\_reasoning\_summary |
| 68 | scenario.filename |
| 69 | scenario.page |
| 70 | scenario.scenario\_index |
| 71 | scenario.text |

We can select components of the results to inspect and print:

```python theme={null}
results.select("summary", "authors", "thanks")
```

| | answer.summary | answer.authors | answer.thanks |
| - | - | - | - |
| 0 | The paper introduces a method for automatically generating and testing social science hypotheses using large language models (LLMs) and structural causal models. This approach leverages LLMs to create agents and design experiments, while structural causal models help in formulating hypotheses and analyzing data. The fitted models can be used for predictions or further experiments. The authors demonstrate this method through scenarios like negotiations and auctions, where causal relationships are examined. The study finds that while LLMs can predict the direction of effects, they struggle with estimating magnitudes unless conditioned on the causal model. The research shows that LLMs possess implicit knowledge that becomes evident when structured through causal models. | \['Benjamin S. Manning', 'Kehang Zhu', 'John J. Horton'] | \['Drew Houston', 'Jordan Ellenberg', 'Benjamin Lira Luttges', 'David Holtz', 'Bruce Sacerdote', 'Paul Röttger', 'Mohammed Alsobay', 'Ray Duch', 'Matt Schwartz', 'David Autor', 'Dean Eckles'] |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we demonstrate how to post this notebook:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "scenario_from_pdf.ipynb")

nb.push(
    description = "Example code for generating scenarios from PDFs",
    alias = "scenario-from-pdf-notebook",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example code for generating scenarios from PDFs',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/fe5565c0-d4f2-4ed8-89d3-a1a5096614f4',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/scenario-from-pdf-notebook',
 'uuid': 'fe5565c0-d4f2-4ed8-89d3-a1a5096614f4',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```


# Using Wikipedia tables in a survey
Source: https://docs.expectedparrot.com/en/latest/notebooks/scenario_list_wikipedia



```python theme={null}
from edsl import ScenarioList
```

Call the `from_source()` method on a `ScenarioList` and pass (i) source type “wikipedia”, (ii) the URL for the Wikipedia page containing the table and (iii) the number of the table on the page (an integer starting at 0 that can be omitted if there is only 1 table on the page). For example, here we create scenarios for the 4th table at the page about movies of the 1990s:

```python theme={null}
scenarios = ScenarioList.from_source(
    source_type = "wikipedia",
    url = "https://en.wikipedia.org/wiki/1990s_in_film",
    table_index = 3
)
```

```python theme={null}
scenarios
```

```python theme={null}
ScenarioList scenarios: 50; keys: ['Worldwide gross', 'Studios', 'Title', 'Year', 'Rank', 'Ref.'];
```

| | Rank | Title | Studios | Worldwide gross | Year | Ref. |
| :- | :- | :- | :- | :- | :- | :- |
| 0 | 1 | Titanic | Paramount Pictures/20th Century Fox | \$1,843,201,268 | 1997 | \[# 1] |
| 1 | 2 | Star Wars: Episode I – The Phantom Menace | 20th Century Fox | \$924,317,558 | 1999 | \[# 2] |
| 2 | 3 | Jurassic Park | Universal Pictures | \$914,691,118 | 1993 | \[# 3] |
| 3 | 4 | Independence Day | 20th Century Fox | \$817,400,891 | 1996 | \[# 4] |
| 4 | 5 | The Lion King | Walt Disney Studios | \$763,455,561 | 1994 | \[# 5] |
| 5 | 6 | Forrest Gump | Paramount Pictures | \$677,387,716 | 1994 | \[# 6] |
| 6 | 7 | The Sixth Sense | Walt Disney Studios | \$672,806,292 | 1999 | \[# 7] |
| 7 | 8 | The Lost World: Jurassic Park | Universal Pictures | \$618,638,999 | 1997 | \[# 8] |
| 8 | 9 | Men in Black | Sony Pictures/Columbia Pictures | \$589,390,539 | 1997 | \[# 9] |
| 9 | 10 | Armageddon | Walt Disney Studios | \$553,709,788 | 1998 | \[# 10] |
| 10 | 11 | Terminator 2: Judgment Day | TriStar Pictures | \$519,843,345 | 1991 | \[# 11] |
| 11 | 12 | Ghost | Paramount Pictures | \$505,702,588 | 1990 | \[# 12] |
| 12 | 13 | Aladdin | Walt Disney Studios | \$504,050,219 | 1992 | \[# 13] |
| 13 | 14 | Twister | Warner Bros./Universal Pictures | \$494,471,524 | 1996 | \[# 14] |
| 14 | 15 | Toy Story 2 | Walt Disney Studios | \$485,015,179 | 1999 | \[# 15] |
| 15 | 16 | Saving Private Ryan | DreamWorks Pictures/Paramount Pictures | \$481,840,909 | 1998 | \[# 16] |
| 16 | 17 | Home Alone | 20th Century Fox | \$476,684,675 | 1990 | \[# 17] |
| 17 | 18 | The Matrix | Warner Bros. | \$463,517,383 | 1999 | \[# 18] |
| 18 | 19 | Pretty Woman | Walt Disney Studios | \$463,406,268 | 1990 | \[# 19] |
| 19 | 20 | Mission: Impossible | Paramount Pictures | \$457,696,359 | 1996 | \[# 20] |
| 20 | 21 | Tarzan | Walt Disney Studios | \$448,191,819 | 1999 | \[# 21] |
| 21 | 22 | Mrs. Doubtfire | 20th Century Fox | \$441,286,195 | 1993 | \[# 22] |
| 22 | 23 | Dances with Wolves | Orion Pictures | \$424,208,848 | 1990 | \[# 23] |
| 23 | 24 | The Mummy | Universal Pictures | \$415,933,406 | 1999 | \[# 24] |
| 24 | 25 | The Bodyguard | Warner Bros. | \$411,006,740 | 1992 | \[# 25] |
| 25 | 26 | Robin Hood: Prince of Thieves | Warner Bros. | \$390,493,908 | 1991 | \[# 26] |
| 26 | 27 | Godzilla | TriStar Pictures | \$379,014,294 | 1998 | \[# 27] |
| 27 | 28 | True Lies | 20th Century Fox | \$378,882,411 | 1994 | \[# 28] |
| 28 | 29 | Toy Story | Walt Disney Studios | \$373,554,033 | 1995 | \[# 29] |
| 29 | 30 | There's Something About Mary | 20th Century Fox | \$369,884,651 | 1998 | \[# 30] |
| 30 | 31 | The Fugitive | Warner Bros. | \$368,875,760 | 1993 | \[# 31] |
| 31 | 32 | Die Hard with a Vengeance | 20th Century Fox/Cinergi Pictures | \$366,101,666 | 1995 | \[# 32] |
| 32 | 33 | Notting Hill | PolyGram Filmed Entertainment | \$363,889,678 | 1999 | \[# 33] |
| 33 | 34 | A Bug's Life | Walt Disney Studios | \$363,398,565 | 1998 | \[# 34] |
| 34 | 35 | The World Is Not Enough | Metro-Goldwyn-Mayer Pictures | \$361,832,400 | 1999 | \[# 35] |
| 35 | 36 | Home Alone 2: Lost in New York | 20th Century Fox | \$358,994,850 | 1992 | \[# 36] |
| 36 | 37 | American Beauty | DreamWorks Pictures | \$356,296,601 | 1999 | \[# 37] |
| 37 | 38 | Apollo 13 | Universal Pictures/Imagine Entertainment | \$355,237,933 | 1995 | \[# 38] |
| 38 | 39 | Basic Instinct | TriStar Pictures | \$352,927,224 | 1992 | \[# 39] |
| 39 | 40 | GoldenEye | MGM/United Artists | \$352,194,034 | 1995 | \[# 40] |
| 40 | 41 | The Mask | New Line Cinema | \$351,583,407 | 1994 | \[# 41] |
| 41 | 42 | Speed | 20th Century Fox | \$350,448,145 | 1994 | \[# 42] |
| 42 | 43 | Deep Impact | Paramount Pictures/DreamWorks Pictures | \$349,464,664 | 1998 | \[# 43] |
| 43 | 44 | Beauty and the Beast | Walt Disney Studios | \$346,317,207 | 1991 | \[# 44] |
| 44 | 45 | Pocahontas | Walt Disney Studios | \$346,079,773 | 1995 | \[# 45] |
| 45 | 46 | The Flintstones | Universal Pictures | \$341,631,208 | 1994 | \[# 46] |
| 46 | 47 | Batman Forever | Warner Bros. | \$336,529,144 | 1995 | \[# 47] |
| 47 | 48 | The Rock | Walt Disney Studios | \$335,062,621 | 1996 | \[# 48] |
| 48 | 49 | Tomorrow Never Dies | MGM/United Artists | \$333,011,068 | 1997 | \[# 49] |
| 49 | 50 | Seven | New Line Cinema | \$327,311,859 | 1995 | \[# 50] |

```python theme={null}
scenarios.parameters
```

```json theme={null}
{'Rank', 'Ref.', 'Studios', 'Title', 'Worldwide gross', 'Year'}
```

```python theme={null}
from edsl import QuestionList

q_leads = QuestionList(
    question_name = "leads",
    question_text = "Who are the lead actors or actresses in {{ scenario.Title }}?"
)

results = q_leads.by(scenarios).run()
```

```python theme={null}
(
    results
    .sort_by("Title")
    .select("Title", "leads")
)
```

| | scenario.Title | answer.leads |
| :- | :- | :- |
| 0 | A Bug's Life | \['Dave Foley', 'Kevin Spacey', 'Julia Louis-Dreyfus'] |
| 1 | Aladdin | \['Mena Massoud', 'Naomi Scott', 'Will Smith'] |
| 2 | American Beauty | \['Kevin Spacey', 'Annette Bening', 'Thora Birch'] |
| 3 | Apollo 13 | \['Tom Hanks', 'Bill Paxton', 'Kevin Bacon', 'Gary Sinise', 'Ed Harris'] |
| 4 | Armageddon | \['Bruce Willis', 'Ben Affleck', 'Liv Tyler'] |
| 5 | Basic Instinct | \['Michael Douglas', 'Sharon Stone'] |
| 6 | Batman Forever | \['Val Kilmer', 'Tommy Lee Jones', 'Jim Carrey', 'Nicole Kidman', "Chris O'Donnell"] |
| 7 | Beauty and the Beast | \['Emma Watson', 'Dan Stevens'] |
| 8 | Dances with Wolves | \['Kevin Costner', 'Mary McDonnell', 'Graham Greene'] |
| 9 | Deep Impact | \['Elijah Wood', 'Morgan Freeman', 'Robert Duvall', 'Téa Leoni', 'Vanessa Redgrave', 'Maximilian Schell', 'Leelee Sobieski'] |
| 10 | Die Hard with a Vengeance | \['Bruce Willis', 'Samuel L. Jackson', 'Jeremy Irons'] |
| 11 | Forrest Gump | \['Tom Hanks', 'Robin Wright', 'Gary Sinise', 'Mykelti Williamson', 'Sally Field'] |
| 12 | Ghost | \['Patrick Swayze', 'Demi Moore', 'Whoopi Goldberg'] |
| 13 | Godzilla | \['Aaron Taylor-Johnson', 'Elizabeth Olsen', 'Bryan Cranston'] |
| 14 | GoldenEye | \['Pierce Brosnan', 'Sean Bean', 'Izabella Scorupco', 'Famke Janssen'] |
| 15 | Home Alone | \['Macaulay Culkin', 'Joe Pesci', 'Daniel Stern', "Catherine O'Hara", 'John Heard'] |
| 16 | Home Alone 2: Lost in New York | \['Macaulay Culkin', 'Joe Pesci', 'Daniel Stern', "Catherine O'Hara", 'John Heard'] |
| 17 | Independence Day | \['Will Smith', 'Jeff Goldblum', 'Bill Pullman'] |
| 18 | Jurassic Park | \['Sam Neill', 'Laura Dern', 'Jeff Goldblum', 'Richard Attenborough'] |
| 19 | Men in Black | \['Will Smith', 'Tommy Lee Jones'] |
| 20 | Mission: Impossible | \['Tom Cruise', 'Jon Voight', 'Emmanuelle Béart', 'Henry Czerny', 'Ving Rhames'] |
| 21 | Mrs. Doubtfire | \['Robin Williams', 'Sally Field', 'Pierce Brosnan'] |
| 22 | Notting Hill | \['Hugh Grant', 'Julia Roberts'] |
| 23 | Pocahontas | \['Irene Bedard', 'Mel Gibson'] |
| 24 | Pretty Woman | \['Richard Gere', 'Julia Roberts'] |
| 25 | Robin Hood: Prince of Thieves | \['Kevin Costner', 'Morgan Freeman', 'Mary Elizabeth Mastrantonio', 'Alan Rickman', 'Christian Slater'] |
| 26 | Saving Private Ryan | \['Tom Hanks', 'Matt Damon', 'Tom Sizemore', 'Edward Burns', 'Barry Pepper', 'Adam Goldberg', 'Vin Diesel', 'Giovanni Ribisi'] |
| 27 | Seven | \['Brad Pitt', 'Morgan Freeman', 'Gwyneth Paltrow'] |
| 28 | Speed | \['Keanu Reeves', 'Sandra Bullock', 'Dennis Hopper'] |
| 29 | Star Wars: Episode I – The Phantom Menace | \['Liam Neeson', 'Ewan McGregor', 'Natalie Portman', 'Jake Lloyd'] |
| 30 | Tarzan | \['Johnny Weissmuller', 'Christopher Lambert', 'Tony Goldwyn', 'Alexander Skarsgård'] |
| 31 | Terminator 2: Judgment Day | \['Arnold Schwarzenegger', 'Linda Hamilton', 'Edward Furlong', 'Robert Patrick'] |
| 32 | The Bodyguard | \['Kevin Costner', 'Whitney Houston'] |
| 33 | The Flintstones | \['John Goodman', 'Elizabeth Perkins', 'Rick Moranis', "Rosie O'Donnell"] |
| 34 | The Fugitive | \['Harrison Ford', 'Tommy Lee Jones'] |
| 35 | The Lion King | \['Matthew Broderick', 'James Earl Jones', 'Jeremy Irons', 'Moira Kelly', 'Nathan Lane'] |
| 36 | The Lost World: Jurassic Park | \['Jeff Goldblum', 'Julianne Moore', 'Vince Vaughn'] |
| 37 | The Mask | \['Jim Carrey', 'Cameron Diaz'] |
| 38 | The Matrix | \['Keanu Reeves', 'Laurence Fishburne', 'Carrie-Anne Moss'] |
| 39 | The Mummy | \['Brendan Fraser', 'Rachel Weisz', 'John Hannah', 'Arnold Vosloo'] |
| 40 | The Rock | \['Sean Connery', 'Nicolas Cage', 'Ed Harris'] |
| 41 | The Sixth Sense | \['Bruce Willis', 'Haley Joel Osment', 'Toni Collette'] |
| 42 | The World Is Not Enough | \['Pierce Brosnan', 'Sophie Marceau', 'Robert Carlyle', 'Denise Richards'] |
| 43 | There's Something About Mary | \['Cameron Diaz', 'Ben Stiller', 'Matt Dillon'] |
| 44 | Titanic | \['Leonardo DiCaprio', 'Kate Winslet'] |
| 45 | Tomorrow Never Dies | \['Pierce Brosnan', 'Jonathan Pryce', 'Michelle Yeoh', 'Teri Hatcher'] |
| 46 | Toy Story | \['Tom Hanks', 'Tim Allen'] |
| 47 | Toy Story 2 | \['Tom Hanks', 'Tim Allen', 'Joan Cusack'] |
| 48 | True Lies | \['Arnold Schwarzenegger', 'Jamie Lee Curtis'] |
| 49 | Twister | \['Helen Hunt', 'Bill Paxton'] |

## Posting to Expected Parrot

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "scenario_list_wikipedia.ipynb")

nb.push(
    description = "Example code for generating scenarios from Wikipedia tables",
    alias = "example-wikipedia-table-scenarios-notebook",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example code for generating scenarios from Wikipedia tables',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/678ac016-b6ab-4314-ac83-dd0981c1ddbe',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/example-wikipedia-table-scenarios-notebook',
 'uuid': '678ac016-b6ab-4314-ac83-dd0981c1ddbe',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```


# Using EDSL to sense check data
Source: https://docs.expectedparrot.com/en/latest/notebooks/scenariolist_unpivot

This notebook provides example code for sense checking survey data using [EDSL](/en/latest/index), an open-source library for simulating surveys, experiments and market research with AI agents and large language models.

## Contents

Using a set of responses to a survey about online marketplaces as an example, we demonstrate EDSL methods for:

<Steps>
  <Step>
    Evaluating survey questions (e.g., for clarity and improvements)
  </Step>

  <Step>
    Analyzing each respondent’s set of answers (e.g., to summarize or identify sentiment, themes, etc.)
  </Step>

  <Step>
    Reviewing each answer individually (e.g., to evaluate its relevance or usefulness)
  </Step>
</Steps>

## Expected Parrot

We also show how to post EDSL questions, surveys, results and notebooks (like this one) to [Expected Parrot: an integrated platform for creating and sharing LLM-based research](https://www.expectedparrot.com/explore).

## How EDSL works

EDSL is a flexible library that can be used to perform a broad variety of research tasks. A typical workflow consists of the following steps:

* Construct questions
* Add data to the questions (e.g., for data labeling tasks)
* Use an AI agent to answer the questions
* Select a language model to generate the answers
* Analyze results in a formatted dataset

## Technical setup

Before running the code below please ensure that you have completed setup:

* [Install](/en/latest/installation) the EDSL library.
* Create a [Expected Parrot account](https://www.expectedparrot.com/login) and activate [remote inference](/en/latest/remote_inference) OR store your own [API Keys](/en/latest/api_keys) for language models that you want to use.

Our [Starter Tutorial](/en/latest/starter_tutorial) provides examples of EDSL basic components.

## Example data

Our example data is a CSV consisting of several questions and a few rows of responses. Here we store it at Expected Parrot and then re-import it.

To post an object:

```python theme={null}
from edsl import FileStore

fs = FileStore("marketplace_survey_results.csv")
fs.push(
    description = "mock marketplace survey results",
    alias = "mock-marketplace-survey-results",
    visibility = "public"
)
```

This returns details of the object we can use to retrieve it:

```json theme={null}
{'description': 'mock marketplace survey results',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/2e56db7a-27de-4c40-bb72-36a5b1d21e6a',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/mock-marketplace-survey-results',
 'uuid': '2e56db7a-27de-4c40-bb72-36a5b1d21e6a',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```

```python theme={null}
from edsl import FileStore
```

```python theme={null}
csv_file = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/mock-marketplace-survey-results")
```

## Creating questions about the data

There are many questions we might want to ask about the data, such as:

* Does this survey question have any logical or syntactical problems? `{{ *question* }}`
* What is the overall sentiment of this respondent’s answers? `{{ *responses* }}`
* Is this answer responsive to the question that was asked? `{{ *question* }}` `{{ *answer* }}`

## Question types

EDSL comes with many common question types that we can select from based on the form of the response that we want to get back from the model: multiple choice, checkbox, linear scale, free text, etc. [Learn more about EDSL question types](/en/latest/questions).

Here we construct `Question` objects for the questions that we want to ask about the data, using `{{ placeholders }}` for the information that we will add to the questions in the steps that follow:

```python theme={null}
from edsl import QuestionFreeText, QuestionMultipleChoice, QuestionYesNo
```

```python theme={null}
q_logic = QuestionFreeText(
    question_name = "logic",
    question_text = """
    Describe any logical or syntactical problems in the following survey question:
    {{ scenario.question }}
    """
)
```

```python theme={null}
q_sentiment = QuestionMultipleChoice(
    question_name = "sentiment",
    question_text = """
    Identify the overall sentiment of this respondent's survey answers:
    {{ scenario.responses }}
    """,
    question_options = ["Very unsatisfied", "Somewhat unsatisfied", "Somewhat satisfied", "Very satisfied"]
)
```

```python theme={null}
q_responsive = QuestionYesNo(
    question_name = "responsive",
    question_text = """
    Is this answer responsive to the question that was asked?
    Question: {{ scenario.question }}
    Answer: {{ scenario.answer }}
    """
)
```

## Adding survey data to the questions

Next we’ll add our data to our questions. This can be done efficiently by creating a `ScenarioList` representing the data. The individual `Scenario` objects in the list can be constructed in a variety of ways depending on the information that we want to include in a particular question.

We start by calling the `from_source()` method to create a `ScenarioList` for the data in its original form. We can see that this generates a `Scenario` dictionary for each respondent’s set of answers with key/value pairs for the individual questions and answers:

```python theme={null}
from edsl import ScenarioList
```

```python theme={null}
sl = ScenarioList.from_source("csv", csv_file.to_tempfile()) # equivalent to importing a local file
sl
```

[ScenarioList](/en/latest/scenarios) scenarios: 3; keys: \['Respondent ID', 'What do you like most about using our online marketplace?', 'What is one feature you would like to see added to improve your shopping experience?', 'Is there anything else you would like to share about your experience with us?', 'How do you feel about the current product search and filtering options?', 'Can you describe a recent experience where you were dissatisfied with our service?'];

| | Respondent ID | What do you like most about using our online marketplace? | What is one feature you would like to see added to improve your shopping experience? | Can you describe a recent experience where you were dissatisfied with our service? | How do you feel about the current product search and filtering options? | Is there anything else you would like to share about your experience with us? |
| :- | :- | :- | :- | :- | :- | :- |
| 0 | 101 | The wide variety of products and the ease of use. | It would be great to have a personalized recommendation system based on my browsing history. | I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it. | The search and filtering options are intuitive and work well for me. | No, keep up the great work! |
| 1 | 102 | I enjoy the simplicity of the interface. | A feature that helps compare similar products side by side would be useful. | No complaints here. | I find the product search to be pretty effective. | I think the sky is a beautiful shade of purple today. |
| 2 | 103 | The platform is user-friendly and offers a vast selection of products. | Would love to see an option to save and compare different products. | My delivery was late by a few days, which was frustrating. | It’s okay. | No. |

## Evaluating the questions

For our first question we want to create a `Scenario` for each survey question:

```python expandable theme={null}
from edsl import QuestionFreeText, Survey

q_logic = QuestionFreeText(
    question_name = "logic",
    question_text = """
    Describe any logical or syntactical problems in the following survey question:
    {{ scenario.question }}
    """
)

q_improved = QuestionFreeText(
    question_name = "improved",
    question_text = """
    You were previously asked to describe any problems with the following survey question:
    {{ scenario.question }}
    You answered: '{{ logic.answer }}'
    Now please draft an improved version of this survey question.
    Return only the revised question text.
    """
)

survey = Survey(questions = [q_logic, q_improved])
```

The survey questions are the `parameters` of the `ScenarioList` created above:

```python theme={null}
questions = list(sl.parameters)
questions
```

\['Respondent ID',
'What do you like most about using our online marketplace?',
'What is one feature you would like to see added to improve your shopping experience?',
'Is there anything else you would like to share about your experience with us?',
'How do you feel about the current product search and filtering options?',
'Can you describe a recent experience where you were dissatisfied with our service?']

We can pass them to the `from_source()` method to create a new `ScenarioList`, specifying that the key for each `Scenario` will be `question` in order to match the parameter of our logic question:

```python theme={null}
sl_questions = ScenarioList.from_source("list", "question", questions)
sl_questions
```

[ScenarioList](/en/latest/scenarios) scenarios: 6; keys: \['question'];

| | question |
| :- | :- |
| 0 | Respondent ID |
| 1 | What do you like most about using our online marketplace? |
| 2 | What is one feature you would like to see added to improve your shopping experience? |
| 3 | Is there anything else you would like to share about your experience with us? |
| 4 | How do you feel about the current product search and filtering options? |
| 5 | Can you describe a recent experience where you were dissatisfied with our service? |

We select a model to use, and then add the scenarios to the survey when we run it:

```python theme={null}
from edsl import Model

m = Model("gemini-1.5-flash", service_name = "google")
```

```python theme={null}
results = survey.by(sl_questions).by(m).run()
```

This generates a dataset of `Results` that we can access with built-in methods for analysis:

```python theme={null}
results.select("question", "logic", "improved")
```

| | scenario.question | answer.logic | answer.improved |
| :- | :- | :- | :- |
| 0 | Respondent ID | "Respondent ID" is not a survey *question*; it's a field for identifying the respondent's unique entry in the dataset. There's no logical or syntactical problem *within* the term itself, but its placement as if it were a question is problematic. The issue is its function: it's not something a respondent *answers*, it's something assigned to them by the survey system. Including it as if it were a question is a design flaw. | (No change needed; remove "Respondent ID" from the survey entirely.) |
| 1 | What do you like most about using our online marketplace? | The question "What do you like most about using our online marketplace?" has a few potential problems: \* **Leading Question (subtle):** While not overtly leading, it presupposes the respondent *likes* something about the marketplace. Someone with a negative experience might struggle to answer honestly, potentially leading to biased results or a lack of negative feedback. A neutral question would be better. \* **Open-ended and difficult to analyze:** The open-ended nature makes it hard to quantify and analyze the responses. You'll get a wide variety of answers requiring significant manual coding and categorization, making it time-consuming and prone to subjective interpretation. This is less of a logical problem and more of a practical one for data analysis. \* **Ambiguity on "using":** Does "using" refer to the buying process, selling process, browsing, customer service interactions, or something else? The question could be more specific to elicit more targeted and useful feedback. To improve it, consider several options: \* **Multiple choice with an "other" option:** This allows for easier analysis and quantification while still capturing a range of opinions. \* **Rating scales:** Ask respondents to rate different aspects of the marketplace (e.g., ease of use, selection, customer service) on a scale. \* **A more neutral open-ended question:** "What is your overall experience with our online marketplace?" or "What could we improve about our online marketplace?" These avoid the presumption of positive feelings. Even better would be to break this down into more specific questions about different aspects of the marketplace. | How would you rate your overall experience with our online marketplace? |
| 2 | What is one feature you would like to see added to improve your shopping experience? | The question has a few potential problems: \* **Ambiguity of "feature":** The term "feature" is quite broad. A respondent might interpret it as a website feature (e.g., better search functionality), a store feature (e.g., more parking), a product feature (e.g., sustainable packaging), or a service feature (e.g., faster delivery). This lack of specificity makes the responses difficult to categorize and analyze meaningfully. \* **Leading question (slightly):** While not overtly leading, the phrasing implies that there *is* something to improve. Some respondents might have a perfectly satisfactory shopping experience and struggle to answer, leading to potentially biased results or low response rates. A neutral phrasing would be preferable. \* **Limited to one feature:** Respondents might have multiple suggestions. Limiting them to one forces prioritization which might not accurately reflect their preferences. They might choose a less important feature simply because it's easier to articulate than a more complex issue. \* **Lack of context:** The question doesn't specify *what* shopping experience is being referred to. Is it online shopping, in-store shopping, or both? This ambiguity could lead to responses that are irrelevant or difficult to compare. In short, the question needs more precision and clarity to be effective. | Thinking about your recent online shopping experience with \[Company Name], what is one specific website feature you would like to see improved or added? |
| 3 | Is there anything else you would like to share about your experience with us? | The question "Is there anything else you would like to share about your experience with us?" is syntactically fine, but it has a few logical problems: \* **Vague Scope:** "Your experience with us" is too broad. The respondent might not know what timeframe or aspects of their interaction the question refers to. Did it cover a single interaction, a product, a service, a period of time? More specific framing is needed. \* **Lack of Guidance:** Open-ended questions like this can yield unhelpful answers (e.g., "Nothing," "Yes," or irrelevant information). It would benefit from prompting more specific feedback, such as suggesting categories (e.g., "Regarding the product itself...", "Regarding customer service...", "Regarding the website..."). \* **Potential for Bias:** The phrasing is slightly leading. A more neutral phrasing might be "What additional comments do you have about your experience?" This avoids the implication that the respondent *should* have something else to share. In short, while grammatically correct, the question lacks the precision and guidance needed to elicit useful and actionable feedback. | To help us improve, please share your thoughts on your recent experience with us. Specifically, we'd appreciate your feedback on: the product itself; our customer service; and the website. |
| 4 | How do you feel about the current product search and filtering options? | The question "How do you feel about the current product search and filtering options?" has several potential problems: \* **Ambiguity:** "How do you feel" is too broad. It doesn't specify what kind of feeling the respondent should express. Do they want to express satisfaction, frustration, ease of use, speed, or something else? The question needs to be more specific about the type of feedback desired (e.g., "How satisfied are you with...?" or "How easy is it to...?"). \* **Lack of Specificity (Compound Question):** It combines two distinct aspects – search and filtering – into a single question. A respondent might have a positive opinion on search but a negative one on filtering, making it difficult to give a concise and accurate answer. It's better to separate these into two distinct questions. \* **No Scale or Structure:** There's no guidance on how the respondent should answer. Should they write an essay? Choose from a list of options? Rate their feelings on a scale? Providing a structured response format (e.g., a Likert scale, multiple-choice options, or a rating system) will make the data easier to analyze and interpret. \* **Assumes Familiarity:** The question assumes the respondent has actually used the product search and filtering options. It should include a filter to ensure only those who have used the features answer the question. In short, the question is too vague and lacks structure, making it difficult for respondents to answer meaningfully and for researchers to analyze the results effectively. | How satisfied are you with the product search functionality? And how satisfied are you with the product filtering functionality? (Please rate each on a scale of 1 to 5, where 1 is very dissatisfied and 5 is very satisfied.) |
| 5 | Can you describe a recent experience where you were dissatisfied with our service? | The question has a few potential problems: \* **Leading Question:** It presupposes the respondent *has* had a negative experience. People might feel pressured to answer even if they don't have a recent negative experience, leading to potentially inaccurate or fabricated responses. A better approach would be to first ask if they had a recent negative experience, and *then* ask for a description only if they answer affirmatively. \* **Ambiguous "our service":** The phrase "our service" lacks specificity. Depending on the context (e.g., a restaurant, a bank, a software company), "our service" could encompass many different things. The question would benefit from clarifying what aspect of the service is being inquired about (e.g., "Can you describe a recent experience where you were dissatisfied with our customer service?", "Can you describe a recent experience where you were dissatisfied with the speed of our delivery?"). \* **"Recent" is undefined:** What constitutes "recent"? A week? A month? A year? Defining a timeframe will make the responses more consistent and comparable. In short, the question needs to be more precise and less leading to gather reliable and useful data. | In the past month, have you had a negative experience with our customer service? If so, please describe it. |

[Learn more about working with results](/en/latest/results).

## Evaluating respondents’ collective answers

Next we can create a `ScenarioList` for each respondent’s answers to use with our question about sentiment:

```python theme={null}
sl
```

[ScenarioList](/en/latest/scenarios) scenarios: 3; keys: \['Respondent ID', 'What do you like most about using our online marketplace?', 'What is one feature you would like to see added to improve your shopping experience?', 'Is there anything else you would like to share about your experience with us?', 'How do you feel about the current product search and filtering options?', 'Can you describe a recent experience where you were dissatisfied with our service?'];

| | Respondent ID | What do you like most about using our online marketplace? | What is one feature you would like to see added to improve your shopping experience? | Can you describe a recent experience where you were dissatisfied with our service? | How do you feel about the current product search and filtering options? | Is there anything else you would like to share about your experience with us? |
| :- | :- | :- | :- | :- | :- | :- |
| 0 | 101 | The wide variety of products and the ease of use. | It would be great to have a personalized recommendation system based on my browsing history. | I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it. | The search and filtering options are intuitive and work well for me. | No, keep up the great work! |
| 1 | 102 | I enjoy the simplicity of the interface. | A feature that helps compare similar products side by side would be useful. | No complaints here. | I find the product search to be pretty effective. | I think the sky is a beautiful shade of purple today. |
| 2 | 103 | The platform is user-friendly and offers a vast selection of products. | Would love to see an option to save and compare different products. | My delivery was late by a few days, which was frustrating. | It’s okay. | No. |

```python theme={null}
sl_responses = ScenarioList.from_source("list", "responses", [str(x) for x in sl])
sl_responses
```

[ScenarioList](/en/latest/scenarios) scenarios: 3; keys: \['responses'];

| | responses |
| :- | :- |
| 0 | Scenario(`{'Respondent ID': '101', 'What do you like most about using our online marketplace?': 'The wide variety of products and the ease of use.', 'What is one feature you would like to see added to improve your shopping experience?': 'It would be great to have a personalized recommendation system based on my browsing history.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it.', 'How do you feel about the current product search and filtering options?': 'The search and filtering options are intuitive and work well for me.', 'Is there anything else you would like to share about your experience with us?': 'No, keep up the great work!'}`) |
| 1 | Scenario(`{'Respondent ID': '102', 'What do you like most about using our online marketplace?': 'I enjoy the simplicity of the interface.', 'What is one feature you would like to see added to improve your shopping experience?': 'A feature that helps compare similar products side by side would be useful.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'No complaints here.', 'How do you feel about the current product search and filtering options?': 'I find the product search to be pretty effective.', 'Is there anything else you would like to share about your experience with us?': 'I think the sky is a beautiful shade of purple today.'}`) |
| 2 | Scenario(`{'Respondent ID': '103', 'What do you like most about using our online marketplace?': 'The platform is user-friendly and offers a vast selection of products.', 'What is one feature you would like to see added to improve your shopping experience?': 'Would love to see an option to save and compare different products.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'My delivery was late by a few days, which was frustrating.', 'How do you feel about the current product search and filtering options?': 'It’s okay.', 'Is there anything else you would like to share about your experience with us?': 'No.'}`) |

Next we add these scenarios to our sentiment question (and any others we want to add) and run it:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, QuestionLinearScale, Survey

q_sentiment = QuestionMultipleChoice(
    question_name = "sentiment",
    question_text = """
    What is the overall sentiment of this respondent's survey answers?
    {{ scenario.responses }}
    """,
    question_options = ["Very unsatisfied", "Somewhat unsatisfied", "Somewhat satisfied", "Very satisfied"]
)

q_recommend = QuestionLinearScale(
    question_name = "recommend",
    question_text = """
    On a scale from 1 to 5, how likely do you think this respondent is to recommend the company to a friend?
    {{ scenario.responses }}
    """,
    question_options = [1, 2, 3, 4, 5],
    option_labels = {1:"Not at all likely", 5:"Very likely"}
)

survey = Survey(questions = [q_sentiment, q_recommend])


jobs = results = survey.by(sl_responses).by(m)

jobs.prompts().select("user_prompt")
```

| | user\_prompt |
| :- | :- |
| 0 | What is the overall sentiment of this respondent's survey answers? Scenario(`{'Respondent ID': '101', 'What do you like most about using our online marketplace?': 'The wide variety of products and the ease of use.', 'What is one feature you would like to see added to improve your shopping experience?': 'It would be great to have a personalized recommendation system based on my browsing history.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it.', 'How do you feel about the current product search and filtering options?': 'The search and filtering options are intuitive and work well for me.', 'Is there anything else you would like to share about your experience with us?': 'No, keep up the great work!'}`) Very unsatisfied Somewhat unsatisfied Somewhat satisfied Very satisfied Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 1 | On a scale from 1 to 5, how likely do you think this respondent is to recommend the company to a friend? Scenario(`{'Respondent ID': '101', 'What do you like most about using our online marketplace?': 'The wide variety of products and the ease of use.', 'What is one feature you would like to see added to improve your shopping experience?': 'It would be great to have a personalized recommendation system based on my browsing history.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it.', 'How do you feel about the current product search and filtering options?': 'The search and filtering options are intuitive and work well for me.', 'Is there anything else you would like to share about your experience with us?': 'No, keep up the great work!'}`) 1 : Not at all likely 2 : 3 : 4 : 5 : Very likely Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 2 | What is the overall sentiment of this respondent's survey answers? Scenario(`{'Respondent ID': '102', 'What do you like most about using our online marketplace?': 'I enjoy the simplicity of the interface.', 'What is one feature you would like to see added to improve your shopping experience?': 'A feature that helps compare similar products side by side would be useful.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'No complaints here.', 'How do you feel about the current product search and filtering options?': 'I find the product search to be pretty effective.', 'Is there anything else you would like to share about your experience with us?': 'I think the sky is a beautiful shade of purple today.'}`) Very unsatisfied Somewhat unsatisfied Somewhat satisfied Very satisfied Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 3 | On a scale from 1 to 5, how likely do you think this respondent is to recommend the company to a friend? Scenario(`{'Respondent ID': '102', 'What do you like most about using our online marketplace?': 'I enjoy the simplicity of the interface.', 'What is one feature you would like to see added to improve your shopping experience?': 'A feature that helps compare similar products side by side would be useful.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'No complaints here.', 'How do you feel about the current product search and filtering options?': 'I find the product search to be pretty effective.', 'Is there anything else you would like to share about your experience with us?': 'I think the sky is a beautiful shade of purple today.'}`) 1 : Not at all likely 2 : 3 : 4 : 5 : Very likely Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 4 | What is the overall sentiment of this respondent's survey answers? Scenario(`{'Respondent ID': '103', 'What do you like most about using our online marketplace?': 'The platform is user-friendly and offers a vast selection of products.', 'What is one feature you would like to see added to improve your shopping experience?': 'Would love to see an option to save and compare different products.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'My delivery was late by a few days, which was frustrating.', 'How do you feel about the current product search and filtering options?': 'It’s okay.', 'Is there anything else you would like to share about your experience with us?': 'No.'}`) Very unsatisfied Somewhat unsatisfied Somewhat satisfied Very satisfied Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. |
| 5 | On a scale from 1 to 5, how likely do you think this respondent is to recommend the company to a friend? Scenario(`{'Respondent ID': '103', 'What do you like most about using our online marketplace?': 'The platform is user-friendly and offers a vast selection of products.', 'What is one feature you would like to see added to improve your shopping experience?': 'Would love to see an option to save and compare different products.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'My delivery was late by a few days, which was frustrating.', 'How do you feel about the current product search and filtering options?': 'It’s okay.', 'Is there anything else you would like to share about your experience with us?': 'No.'}`) 1 : Not at all likely 2 : 3 : 4 : 5 : Very likely Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line. |

```python theme={null}
results = survey.by(sl_responses).by(m).run()
```

```python theme={null}
results.select("responses", "sentiment", "recommend")
```

| | scenario.responses | answer.sentiment | answer.recommend |
| :- | :- | :- | :- |
| 0 | Scenario(`{'Respondent ID': '101', 'What do you like most about using our online marketplace?': 'The wide variety of products and the ease of use.', 'What is one feature you would like to see added to improve your shopping experience?': 'It would be great to have a personalized recommendation system based on my browsing history.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it.', 'How do you feel about the current product search and filtering options?': 'The search and filtering options are intuitive and work well for me.', 'Is there anything else you would like to share about your experience with us?': 'No, keep up the great work!'}`) | Very satisfied | 4 |
| 1 | Scenario(`{'Respondent ID': '102', 'What do you like most about using our online marketplace?': 'I enjoy the simplicity of the interface.', 'What is one feature you would like to see added to improve your shopping experience?': 'A feature that helps compare similar products side by side would be useful.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'No complaints here.', 'How do you feel about the current product search and filtering options?': 'I find the product search to be pretty effective.', 'Is there anything else you would like to share about your experience with us?': 'I think the sky is a beautiful shade of purple today.'}`) | Very satisfied | 4 |
| 2 | Scenario(`{'Respondent ID': '103', 'What do you like most about using our online marketplace?': 'The platform is user-friendly and offers a vast selection of products.', 'What is one feature you would like to see added to improve your shopping experience?': 'Would love to see an option to save and compare different products.', 'Can you describe a recent experience where you were dissatisfied with our service?': 'My delivery was late by a few days, which was frustrating.', 'How do you feel about the current product search and filtering options?': 'It’s okay.', 'Is there anything else you would like to share about your experience with us?': 'No.'}`) | Somewhat satisfied | 3 |

## Evaluating individual answers

Next we create a `ScenarioList` for each individual question and answer to use with our question about the responsiveness of each answer. We can use the `unpivot()` method to expand the scenarios by desired identifiers (e.g., respondent ID):

```python theme={null}
sl_qa = sl.unpivot(id_vars = ["Respondent ID"])
sl_qa
```

[ScenarioList](/en/latest/scenarios) scenarios: 15; keys: \['variable', 'value', 'Respondent ID'];

| | Respondent ID | variable | value |
| :- | :- | :- | :- |
| 0 | 101 | What do you like most about using our online marketplace? | The wide variety of products and the ease of use. |
| 1 | 101 | What is one feature you would like to see added to improve your shopping experience? | It would be great to have a personalized recommendation system based on my browsing history. |
| 2 | 101 | Can you describe a recent experience where you were dissatisfied with our service? | I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it. |
| 3 | 101 | How do you feel about the current product search and filtering options? | The search and filtering options are intuitive and work well for me. |
| 4 | 101 | Is there anything else you would like to share about your experience with us? | No, keep up the great work! |
| 5 | 102 | What do you like most about using our online marketplace? | I enjoy the simplicity of the interface. |
| 6 | 102 | What is one feature you would like to see added to improve your shopping experience? | A feature that helps compare similar products side by side would be useful. |
| 7 | 102 | Can you describe a recent experience where you were dissatisfied with our service? | No complaints here. |
| 8 | 102 | How do you feel about the current product search and filtering options? | I find the product search to be pretty effective. |
| 9 | 102 | Is there anything else you would like to share about your experience with us? | I think the sky is a beautiful shade of purple today. |
| 10 | 103 | What do you like most about using our online marketplace? | The platform is user-friendly and offers a vast selection of products. |
| 11 | 103 | What is one feature you would like to see added to improve your shopping experience? | Would love to see an option to save and compare different products. |
| 12 | 103 | Can you describe a recent experience where you were dissatisfied with our service? | My delivery was late by a few days, which was frustrating. |
| 13 | 103 | How do you feel about the current product search and filtering options? | It’s okay. |
| 14 | 103 | Is there anything else you would like to share about your experience with us? | No. |

We can call the `rename()` method to rename the keys as desired to match our question parameters syntax:

```python theme={null}
sl_qa = sl_qa.rename({"Respondent ID": "id", "variable": "question", "value": "answer"})
sl_qa
```

[ScenarioList](/en/latest/scenarios) scenarios: 15; keys: \['question', 'id', 'answer'];

| | id | question | answer |
| :- | :- | :- | :- |
| 0 | 101 | What do you like most about using our online marketplace? | The wide variety of products and the ease of use. |
| 1 | 101 | What is one feature you would like to see added to improve your shopping experience? | It would be great to have a personalized recommendation system based on my browsing history. |
| 2 | 101 | Can you describe a recent experience where you were dissatisfied with our service? | I was disappointed when an item I ordered arrived damaged, but customer service quickly resolved it. |
| 3 | 101 | How do you feel about the current product search and filtering options? | The search and filtering options are intuitive and work well for me. |
| 4 | 101 | Is there anything else you would like to share about your experience with us? | No, keep up the great work! |
| 5 | 102 | What do you like most about using our online marketplace? | I enjoy the simplicity of the interface. |
| 6 | 102 | What is one feature you would like to see added to improve your shopping experience? | A feature that helps compare similar products side by side would be useful. |
| 7 | 102 | Can you describe a recent experience where you were dissatisfied with our service? | No complaints here. |
| 8 | 102 | How do you feel about the current product search and filtering options? | I find the product search to be pretty effective. |
| 9 | 102 | Is there anything else you would like to share about your experience with us? | I think the sky is a beautiful shade of purple today. |
| 10 | 103 | What do you like most about using our online marketplace? | The platform is user-friendly and offers a vast selection of products. |
| 11 | 103 | What is one feature you would like to see added to improve your shopping experience? | Would love to see an option to save and compare different products. |
| 12 | 103 | Can you describe a recent experience where you were dissatisfied with our service? | My delivery was late by a few days, which was frustrating. |
| 13 | 103 | How do you feel about the current product search and filtering options? | It’s okay. |
| 14 | 103 | Is there anything else you would like to share about your experience with us? | No. |

```python theme={null}
from edsl import QuestionYesNo

q_responsive = QuestionYesNo(
    question_name = "responsive",
    question_text = """
    Is this answer responsive to the question that was asked?
    Question: {{ scenario.question }}
    Answer: {{ scenario.answer }}
    """
)
```

```python theme={null}
results = q_responsive.by(sl_qa).by(m).run()
```

```python theme={null}
(
    results
    .filter("responsive == 'No'")
    .select("id", "question", "answer")
)
```

| | scenario.id | scenario.question | scenario.answer |
| :- | :- | :- | :- |
| 0 | 102 | Can you describe a recent experience where you were dissatisfied with our service? | No complaints here. |
| 1 | 102 | How do you feel about the current product search and filtering options? | I find the product search to be pretty effective. |
| 2 | 102 | Is there anything else you would like to share about your experience with us? | I think the sky is a beautiful shade of purple today. |
| 3 | 103 | How do you feel about the current product search and filtering options? | It’s okay. |

## Uploading content to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a new platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL, and a convenient place to post and access surveys, agents, results and notebooks. [Learn more about posting content](/en/latest/coop).

Here we post the contents of this notebook:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "scenariolist_unpivot.ipynb")

nb.push(
    description = "ScenarioList methods for sense checking survey data",
    alias = "scenariolist-sense-checking-survey-notebook",
    visibility = "public"
)
```

```json theme={null}
{'description': 'ScenarioList methods for sense checking survey data',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/08340981-2497-4be1-b9dc-be8e2f92e4ef',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/scenariolist-sense-checking-survey-notebook',
 'uuid': '08340981-2497-4be1-b9dc-be8e2f92e4ef',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```


# Using data with surveys: FileStore
Source: https://docs.expectedparrot.com/en/latest/notebooks/scenarios_filestore_example

This notebook provides example [EDSL](https://github.com/expectedparrot/edsl) code for methods for using data with an EDSL survey. In the steps below we show how to use the [FileStore](/en/latest/filestore) module to upload, share and retrieve data files at the [Expected Parrot](/en/latest/coop), and then create [Scenario](/en/latest/scenarios) objects for the data to use it with a survey.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## What is a *Scenario*?

A `Scenario` is a dictionary of one or more key/value pairs representing data or content to be added to questions; a `ScenarioList` is a list of *Scenario* objects. Scenario keys are used as question parameters that get replaced with the values when the scenarios are added to the questions, allowing you to create variants of questions efficiently. Learn more about creating and working with scenarios [here](/en/latest/scenarios) and [here](/en/latest/notebooks/question_loop_scenarios).

## What is Expected Parrot?

[Expected Parrot](/en/latest/coop) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL, allowing you to post, download and update objects directly from your workspace and at the [platform](https://www.expectedparrot.com/login). It also provides access to features for working with EDSL remotely at the Expected Parrot server. Learn more about these features in the [remote inference](/en/latest/remote_inference) and [remote caching](/en/latest/remote_caching) sections of the [documentation page](/en/latest/index).

## What is FileStore?

`FileStore` is a [module for storing and sharing data files at Expected Parrot](/en/latest/filestore) to use in EDSL projects, such as survey data, PDFs, CSVs or images. In particular, it is designed for storing files to be used as as scenarios, and allows you to include code for easily retrieving and processing the files in your EDSL project, as we do in the examples below!

## Example

In the example below we create scenarios for some data (a table at a Wikipedia page) and inspect them. Then we store the scenarios as a CSV and post it to Expected Parrot using FileStore. Then we retrieve the file and recreate the scenarios, and use them in a survey. We also post the survey, results and this notebook to the platform for reference.

We start by creating importing the tools that we will use:

```python theme={null}
from edsl import ScenarioList, FileStore
```

### Creating a scenario list for a Wikipedia table

EDSL comes with many methods for automatically [generating scenarios for various data sources](/en/latest/scenarios), such as PDFs, CSVs, docs, images, lists, dicts, etc. Here we use a method to automatically [create a scenario list for a Wikipedia table](/en/latest/notebooks/scenario_list_wikipedia), passing the URL and the number of the table on the page:

```python theme={null}
s = ScenarioList.from_wikipedia("https://en.wikipedia.org/wiki/List_of_Billboard_Hot_100_number-one_singles_of_the_1980s",5)
```

We can inspect the scenario list that has been created:

```python theme={null}
s
```

```python theme={null}
ScenarioList scenarios: 14; keys: ['Artist(s)', 'Weeks at number one', 'Song'];
```

| | Weeks at number one | Song | Artist(s) |
| :- | :- | :- | :- |
| 0 | 10 | "Physical" | Olivia Newton-John |
| 1 | 9 | "Bette Davis Eyes" | Kim Carnes |
| 2 | 9 | "Endless Love" | Diana Ross and Lionel Richie |
| 3 | 8 | "Every Breath You Take" | The Police |
| 4 | 7 | "I Love Rock 'n' Roll" | Joan Jett and the Blackhearts |
| 5 | 7 | "Ebony and Ivory" | Paul McCartney and Stevie Wonder |
| 6 | 7 | "Billie Jean" | Michael Jackson |
| 7 | 6 | "Call Me" | Blondie |
| 8 | 6 | "Lady" | Kenny Rogers |
| 9 | 6 | "Centerfold" | The J. Geils Band |
| 10 | 6 | "Eye of the Tiger" | Survivor |
| 11 | 6 | "Flashdance... What a Feeling" | Irene Cara |
| 12 | 6 | "Say, Say, Say" | Paul McCartney and Michael Jackson |
| 13 | 6 | "Like a Virgin" | Madonna |

We can rename the keys for convenience:

```python theme={null}
s.parameters
```

```python theme={null}
{'Artist(s)', 'Song', 'Weeks at number one'}`
```

```python theme={null}
s = s.rename({'Artist(s)':"artists", 'Song':"song", 'Weeks at number one':"weeks"})
```

```python theme={null}
s
```

```python theme={null}
ScenarioList scenarios: 14; keys: ['song', 'artists', 'weeks'];
```

| | weeks | song | artists |
| :- | :- | :- | :- |
| 0 | 10 | "Physical" | Olivia Newton-John |
| 1 | 9 | "Bette Davis Eyes" | Kim Carnes |
| 2 | 9 | "Endless Love" | Diana Ross and Lionel Richie |
| 3 | 8 | "Every Breath You Take" | The Police |
| 4 | 7 | "I Love Rock 'n' Roll" | Joan Jett and the Blackhearts |
| 5 | 7 | "Ebony and Ivory" | Paul McCartney and Stevie Wonder |
| 6 | 7 | "Billie Jean" | Michael Jackson |
| 7 | 6 | "Call Me" | Blondie |
| 8 | 6 | "Lady" | Kenny Rogers |
| 9 | 6 | "Centerfold" | The J. Geils Band |
| 10 | 6 | "Eye of the Tiger" | Survivor |
| 11 | 6 | "Flashdance... What a Feeling" | Irene Cara |
| 12 | 6 | "Say, Say, Say" | Paul McCartney and Michael Jackson |
| 13 | 6 | "Like a Virgin" | Madonna |

We can save the scenarios to a CSV:

```python theme={null}
s.to_csv("billboard_100_1980s.csv")
```

File written to billboard\_100\_1980s.csv

### Posting data to Expected Parrot

Here we post the CSV to Expected Parrot using `FileStore` (note that the file type is automatically inferred):

```python theme={null}
fs = FileStore("billboard_100_1980s.csv")
```

We can post a `FileStore` object to Expected Parrot by calling the `push()` method on it. We can optionally pass a `description` and a `visibility` setting - *public*, *unlisted* (by default) or *private*:

```python theme={null}
info = fs.push(
    description = "Wikipedia: List of Billboard Hot 100 number-one singles of the 1980s",
    alias = "billboard-top-100-1980s",
    visibility = "public"
)
```

We can print the details of the posted object, including the URL and uuid that we will need to retrieve it later:

```python theme={null}
info


{'description': 'Wikipedia: List of Billboard Hot 100 number-one singles of the 1980s',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/8a3a30cd-c9c8-437f-a4b4-a4ce417480c5',
 'uuid': '8a3a30cd-c9c8-437f-a4b4-a4ce417480c5',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

### Retrieving a file and recreating scenarios

Here we retrieve the file:

```python theme={null}
csv_file = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/billboard-top-100-1980s")
```

This is equivalent:

```python theme={null}
uuid = info["uuid"]
uuid
```

```
'8a3a30cd-c9c8-437f-a4b4-a4ce417480c5'
```

```python theme={null}
csv_file = FileStore.pull(info["uuid"])
```

Here we recreate scenarios:

```python theme={null}
s = ScenarioList.from_csv(csv_file.to_tempfile())
s
```

```python theme={null}
ScenarioList scenarios: 14; keys: ['song', 'artists', 'weeks'];
```

| | weeks | song | artists |
| :- | :- | :- | :- |
| 0 | 10 | "Physical" | Olivia Newton-John |
| 1 | 9 | "Bette Davis Eyes" | Kim Carnes |
| 2 | 9 | "Endless Love" | Diana Ross and Lionel Richie |
| 3 | 8 | "Every Breath You Take" | The Police |
| 4 | 7 | "I Love Rock 'n' Roll" | Joan Jett and the Blackhearts |
| 5 | 7 | "Ebony and Ivory" | Paul McCartney and Stevie Wonder |
| 6 | 7 | "Billie Jean" | Michael Jackson |
| 7 | 6 | "Call Me" | Blondie |
| 8 | 6 | "Lady" | Kenny Rogers |
| 9 | 6 | "Centerfold" | The J. Geils Band |
| 10 | 6 | "Eye of the Tiger" | Survivor |
| 11 | 6 | "Flashdance... What a Feeling" | Irene Cara |
| 12 | 6 | "Say, Say, Say" | Paul McCartney and Michael Jackson |
| 13 | 6 | "Like a Virgin" | Madonna |

### Using scenarios in a survey

We can use the scenarios with a survey by creating placeholders in the questions for the scenario keys, and adding the scenarios to the survey when we run it:

```python expandable theme={null}
from edsl import QuestionFreeText, QuestionMultipleChoice, QuestionCheckBox, QuestionList, Survey

q1 = QuestionFreeText(
    question_name = "topic",
    question_text = "What is the topic of the song {{ song }} by {{ artists }}?"
)
q2 = QuestionMultipleChoice(
    question_name = "sentiment",
    question_text = "What is the sentiment of the song {{ song }} by {{ artists }}?",
    question_options = [
        "Happy",
        "Sad",
        "Angry",
        "Romantic",
        "Nostalgic",
        "Empowering",
        "Melancholic",
        "Hopeful"
    ]
)

q3 = QuestionCheckBox(
    question_name = "themes",
    question_text = "What themes are present in the song {{ song }} by {{ artists }}?",
    question_options = [
        "Love",
        "Loss",
        "Struggle",
        "Celebration",
        "Social issues",
        "Other"
    ]
)

q4 = QuestionList(
    question_name = "other_themes",
    question_text = "What other themes are present?"
)

survey = (
    Survey(questions = [q1, q2, q3, q4])
    .add_targeted_memory(q4, q3)
    .add_stop_rule(q3, "'Other' not in themes")
)

results = survey.by(s).run()
```

We can filter, sort, select and print any components of the results that are generated. Note that the results include columns for all scenario keys, whether used in question texts or not:

```python theme={null}
results.sort_by("song").select("song", "artists", "topic")
```

| | scenario.song | scenario.artists | answer.topic |
| :- | :- | :- | :- |
| 0 | "Bette Davis Eyes" | Kim Carnes | The song "Bette Davis Eyes" by Kim Carnes is about a woman who is alluring and captivating, much like the iconic actress Bette Davis. The lyrics describe her as having a mysterious and enchanting presence, with a sense of charm and charisma that can easily mesmerize those around her. The song highlights her ability to manipulate and captivate others with her striking eyes and overall persona. |
| 1 | "Billie Jean" | Michael Jackson | The song "Billie Jean" by Michael Jackson is about a man who is confronted by a woman named Billie Jean, who claims that he is the father of her child. The lyrics describe the man's insistence that the woman's allegations are false, emphasizing that "the kid is not my son." The song explores themes of false accusations, fame, and the consequences of rumors and deception. |
| 2 | "Call Me" | Blondie | The song "Call Me" by Blondie is about the excitement and urgency of romantic attraction and communication. The lyrics express a desire for connection and openness to a romantic relationship, with an emphasis on the immediacy and thrill of being contacted by a potential lover. The song captures the energy and passion of new love, with a focus on being available and ready for a call or message from the person of interest. |
| 3 | "Centerfold" | The J. Geils Band | The song "Centerfold" by The J. Geils Band is about a man who discovers that a former high school crush, whom he viewed as innocent and pure, has become a centerfold model in an adult magazine. The song explores his feelings of surprise, nostalgia, and the conflict between his idealized memories of her and her new, more provocative image. |
| 4 | "Ebony and Ivory" | Paul McCartney and Stevie Wonder | The song "Ebony and Ivory" by Paul McCartney and Stevie Wonder addresses themes of racial harmony and unity. It uses the metaphor of piano keys—ebony (black) and ivory (white)—to illustrate how different races can live together in harmony, just as the keys work together to create music. The song promotes the message of living in perfect harmony despite differences. |
| 5 | "Endless Love" | Diana Ross and Lionel Richie | The song "Endless Love" by Diana Ross and Lionel Richie is about a deep, romantic love between two people. The lyrics express a commitment to each other and the idea that their love is eternal and unbreakable. It is often considered a classic love ballad, highlighting themes of devotion, unity, and an everlasting bond. |
| 6 | "Every Breath You Take" | The Police | The song "Every Breath You Take" by The Police is often interpreted as a song about obsessive love and surveillance. The lyrics describe a person who is constantly watching and monitoring the actions of someone they are infatuated with. Despite its romantic-sounding melody, the song's theme revolves around possessiveness and control, highlighting the darker aspects of love and relationships. |
| 7 | "Eye of the Tiger" | Survivor | The song "Eye of the Tiger" by Survivor is about perseverance, determination, and the drive to overcome challenges. It was famously used as the theme song for the movie "Rocky III," and its lyrics focus on staying strong, maintaining focus, and having the willpower to succeed despite obstacles. The song's energetic and motivational tone has made it an anthem for facing adversity and pushing through difficult times. |
| 8 | "Flashdance... What a Feeling" | Irene Cara | The song "Flashdance... What a Feeling" by Irene Cara is about the exhilaration and empowerment that comes from pursuing one's dreams and passions. It captures the joy and liberation of achieving one's goals and the transformative power of believing in oneself. The song is famously associated with the 1983 film "Flashdance," which tells the story of a young woman striving to become a professional dancer. |
| 9 | "I Love Rock 'n' Roll" | Joan Jett and the Blackhearts | The song "I Love Rock 'n' Roll" by Joan Jett and the Blackhearts is about a person's passion and enthusiasm for rock and roll music. The lyrics describe the experience of hearing a favorite song on the jukebox and feeling a connection with someone else who shares the same love for the music. It's an anthem celebrating the energy and excitement of rock and roll. |
| 10 | "Lady" | Kenny Rogers | The song "Lady" by Kenny Rogers is a romantic ballad that expresses deep love and admiration. It is a heartfelt declaration of devotion, where the singer describes the significant role that the woman he loves plays in his life. The lyrics convey a sense of gratitude and emotional connection, highlighting themes of love, appreciation, and commitment. |
| 11 | "Like a Virgin" | Madonna | The song "Like a Virgin" by Madonna is about the feeling of renewal and fresh beginnings in a romantic relationship. The lyrics describe the experience of falling in love and feeling revitalized, as if experiencing love for the first time. The song uses the metaphor of being "like a virgin" to convey the sense of innocence and excitement that comes with a new and transformative romantic connection. |
| 12 | "Physical" | Olivia Newton-John | The song "Physical" by Olivia Newton-John is primarily about physical attraction and desire. The lyrics suggest a strong focus on wanting to engage in a physical relationship, using exercise and fitness as metaphors for intimacy and connection. The song was quite provocative for its time due to its suggestive themes. |
| 13 | "Say, Say, Say" | Paul McCartney and Michael Jackson | The song "Say, Say, Say" by Paul McCartney and Michael Jackson is about a plea for love and attention. The lyrics express a desire for reassurance and communication in a romantic relationship. The singers convey feelings of longing and vulnerability, asking their partner to express their feelings openly and honestly. Overall, the song deals with themes of love, longing, and the need for emotional connection. |

```python theme={null}
results.sort_by("weeks", reverse=True).select("weeks", "song", "artists", "sentiment", "themes", "other_themes")
```

| | scenario.weeks | scenario.song | scenario.artists | answer.sentiment | answer.themes | answer.other\_themes |
| :- | :- | :- | :- | :- | :- | :- |
| 0 | 10 | "Physical" | Olivia Newton-John | Happy | \['Love', 'Celebration'] | nan |
| 1 | 9 | "Bette Davis Eyes" | Kim Carnes | Nostalgic | \['Love', 'Other'] | \['Nostalgia', 'Admiration', 'Mystery', 'Seduction'] |
| 2 | 9 | "Endless Love" | Diana Ross and Lionel Richie | Romantic | \['Love', 'Celebration'] | nan |
| 3 | 8 | "Every Breath You Take" | The Police | Melancholic | \['Love', 'Loss', 'Struggle'] | nan |
| 4 | 7 | "I Love Rock 'n' Roll" | Joan Jett and the Blackhearts | Empowering | \['Love', 'Celebration', 'Other'] | \['Nostalgia', 'Rebellion', 'Youth'] |
| 5 | 7 | "Ebony and Ivory" | Paul McCartney and Stevie Wonder | Hopeful | \['Love', 'Social issues'] | nan |
| 6 | 7 | "Billie Jean" | Michael Jackson | Melancholic | \['Struggle', 'Other'] | \['Deception', 'Fame', 'Paranoia', 'Guilt', 'Denial'] |
| 7 | 6 | "Call Me" | Blondie | Empowering | \['Love', 'Celebration'] | nan |
| 8 | 6 | "Lady" | Kenny Rogers | Romantic | \['Love'] | nan |
| 9 | 6 | "Centerfold" | The J. Geils Band | Nostalgic | \['Love', 'Loss', 'Other'] | \['Nostalgia', 'Disillusionment', 'Innocence', 'Surprise'] |
| 10 | 6 | "Eye of the Tiger" | Survivor | Empowering | \['Struggle', 'Other'] | \['Perseverance', 'Resilience', 'Determination', 'Motivation', 'Survival'] |
| 11 | 6 | "Flashdance... What a Feeling" | Irene Cara | Empowering | \['Struggle', 'Celebration'] | nan |
| 12 | 6 | "Say, Say, Say" | Paul McCartney and Michael Jackson | Romantic | \['Love', 'Struggle'] | nan |
| 13 | 6 | "Like a Virgin" | Madonna | Romantic | \['Love', 'Celebration'] | nan |

### Posting a notebook to Expected Parrot

Here we post the contents of this notebook to Expected Parrot for anyone to access:

```python theme={null}
from edsl import Notebook
```

```python theme={null}
n = Notebook(path = "scenarios_filestore_example.ipynb")
```

```python theme={null}
n.push(
    description = "Example code for using data files for scenarios via FileStore and Expected Parrot",
    alias = "my-scenario-image-notebook",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Example code for using data files for scenarios via FileStore and Expected Parrot',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/0f21a6ba-45b2-4cf2-9bea-87929855797f',
 'uuid': '0f21a6ba-45b2-4cf2-9bea-87929855797f',
 'version': '0.1.47.dev1',
 'visibility': 'public'}
```

To update an object:

```python theme={null}
n = Notebook(path = "scenarios_filestore_example.ipynb") # resave
```

```python theme={null}
n.patch("https://www.expectedparrot.com/content/RobinHorton/my-scenario-image-notebook", value = n)
```

```json theme={null}
{'status': 'success'}
```


# Skip logic & scenarios
Source: https://docs.expectedparrot.com/en/latest/notebooks/skip_logic_scenarios

This notebook provides example EDSL code for using a language model to simulate a survey that uses skip logic: rules for determining which questions are administered based on responses to other questions in the survey.

In the first example below we construct a survey of questions and then add a rule to skip one question based on the response to another question.

In the second example we add some complexity. We first create different “scenarios” (versions) of questions and combine them in a survey. Then we add multiple rules to skip specific versions of the questions based on responses to a particular version of a question.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Example 1

In the first example below we construct questions, combine them in a survey, and add a rule to skip the second question based on the response to the first question. Then we create `Scenario` objects for contents that will be added to the questions when the survey is run. The effect of this is that the second question will be skipped based on the response to the first question for each individual scenario.

We start by constructing questions:

```python theme={null}
from edsl import QuestionYesNo, QuestionNumerical, QuestionMultipleChoice

q1 = QuestionYesNo(
    question_name = "recent_purchase",
    question_text = "In the last year have you or anyone in your household purchased any `{{ scenario.item }}`?",
)

q2 = QuestionNumerical(
    question_name = "amount",
    question_text = "In the last year, how much did your household spend on `{{ scenario.item }}` (in USD)?"
)

q3 = QuestionMultipleChoice(
    question_name = "next_purchase",
    question_text = "When do you next expect to purchase `{{ scenario.item }}`?",
    question_options = [
        "Never",
        "Within the next month",
        "Within the next year",
        "I do not know"
    ]
)
```

We combine the questions in a survey to administer them together:

```python theme={null}
from edsl import Survey

survey = Survey(questions = [q1, q2, q3])
survey
```

[Survey](/en/latest/surveys) # questions: 3; question\_name list: \['recent\_purchase', 'amount', 'next\_purchase'];

| | question\_options | question\_name | question\_type | question\_text |
| :- | :- | :- | :- | :- |
| 0 | \['No', 'Yes'] | recent\_purchase | yes\_no | In the last year have you or anyone in your household purchased any `{{ scenario.item }}`? |
| 1 | nan | amount | numerical | In the last year, how much did your household spend on `{{ scenario.item }}` (in USD)? |
| 2 | \['Never', 'Within the next month', 'Within the next year', 'I do not know'] | next\_purchase | multiple\_choice | When do you next expect to purchase `{{ scenario.item }}`? |

Here we add a rule to skip q2 based on the response to q1:

```python theme={null}
survey = survey.add_skip_rule(q2, "`{{ recent_purchase.answer }}` == 'No'")
```

Next we create scenarios for the “item” to be used with each question:

```python theme={null}
from edsl import Scenario, ScenarioList

s = ScenarioList(
    Scenario({"item":item}) for item in ["electronics", "phones"]
)
```

<Note>
  **Note**:

  Note that we could also use a method for the data type that we are using–this is equivalent:
</Note>

```python theme={null}
s = ScenarioList.from_list("item", ["electronics", "phones"])
s
```

```python theme={null}
ScenarioList scenarios: 2; keys: ['item'];
```

| | item |
| :- | :- |
| 0 | electronics |
| 1 | phones |

We can inspect the flow of the survey that has been created with the scenarios that we’re using:

```python theme={null}
survey.by(s).show_flow()
```

<Frame>
  <img alt="../_images/notebooks_skip_logic_scenarios_13_0.png" />
</Frame>

Next we create some agent personas to answer the questions:

```python theme={null}
from edsl import Agent, AgentList

income_levels = ["under $100,000", "$100,000-250,000", "above $250,000"]
ages = [30, 50, 70]

a = AgentList(
    Agent({"annual_income":income, "age":age}) for income in income_levels for age in ages
)
a
```

```python theme={null}
AgentList agents: 9;
```

| | annual\_income | age |
| :- | :- | :- |
| 0 | under \$100,000 | 30 |
| 1 | under \$100,000 | 50 |
| 2 | under \$100,000 | 70 |
| 3 | \$100,000-250,000 | 30 |
| 4 | \$100,000-250,000 | 50 |
| 5 | \$100,000-250,000 | 70 |
| 6 | above \$250,000 | 30 |
| 7 | above \$250,000 | 50 |
| 8 | above \$250,000 | 70 |

Next we select a model to generate the responses ([check available models and pricing](https://www.expectedparrot.com/getting-started/coop-pricing)):

```python theme={null}
from edsl import Model

m = Model("gemini-1.5-flash")
```

We can inspect (or modify) the default parameters of the model that will be used:

```python theme={null}
m
```

```python theme={null}
gemini-1.5-flash
```

| | key | value |
| :- | :- | :- |
| 0 | model | gemini-1.5-flash |
| 1 | parameters:temperature | 0.500000 |
| 2 | parameters:topP | 1 |
| 3 | parameters:topK | 1 |
| 4 | parameters:maxOutputTokens | 2048 |
| 5 | parameters:stopSequences | \[] |
| 6 | inference\_service | google |

We run the survey by adding any scenarios, agents and models and then calling the `run`:

```python theme={null}
results = survey.by(s).by(a).by(m).run()
```

We can inspect a list of the columns of the dataset of results that has been generated:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.age |
| 1 | agent.agent\_index |
| 2 | agent.agent\_instruction |
| 3 | agent.agent\_name |
| 4 | agent.annual\_income |
| 5 | answer.amount |
| 6 | answer.next\_purchase |
| 7 | answer.recent\_purchase |
| 8 | cache\_keys.amount\_cache\_key |
| 9 | cache\_keys.next\_purchase\_cache\_key |
| 10 | cache\_keys.recent\_purchase\_cache\_key |
| 11 | cache\_used.amount\_cache\_used |
| 12 | cache\_used.next\_purchase\_cache\_used |
| 13 | cache\_used.recent\_purchase\_cache\_used |
| 14 | comment.amount\_comment |
| 15 | comment.next\_purchase\_comment |
| 16 | comment.recent\_purchase\_comment |
| 17 | generated\_tokens.amount\_generated\_tokens |
| 18 | generated\_tokens.next\_purchase\_generated\_tokens |
| 19 | generated\_tokens.recent\_purchase\_generated\_tokens |
| 20 | iteration.iteration |
| 21 | model.inference\_service |
| 22 | model.maxOutputTokens |
| 23 | model.model |
| 24 | model.model\_index |
| 25 | model.stopSequences |
| 26 | model.temperature |
| 27 | model.topK |
| 28 | model.topP |
| 29 | prompt.amount\_system\_prompt |
| 30 | prompt.amount\_user\_prompt |
| 31 | prompt.next\_purchase\_system\_prompt |
| 32 | prompt.next\_purchase\_user\_prompt |
| 33 | prompt.recent\_purchase\_system\_prompt |
| 34 | prompt.recent\_purchase\_user\_prompt |
| 35 | question\_options.amount\_question\_options |
| 36 | question\_options.next\_purchase\_question\_options |
| 37 | question\_options.recent\_purchase\_question\_options |
| 38 | question\_text.amount\_question\_text |
| 39 | question\_text.next\_purchase\_question\_text |
| 40 | question\_text.recent\_purchase\_question\_text |
| 41 | question\_type.amount\_question\_type |
| 42 | question\_type.next\_purchase\_question\_type |
| 43 | question\_type.recent\_purchase\_question\_type |
| 44 | raw\_model\_response.amount\_cost |
| 45 | raw\_model\_response.amount\_input\_price\_per\_million\_tokens |
| 46 | raw\_model\_response.amount\_input\_tokens |
| 47 | raw\_model\_response.amount\_one\_usd\_buys |
| 48 | raw\_model\_response.amount\_output\_price\_per\_million\_tokens |
| 49 | raw\_model\_response.amount\_output\_tokens |
| 50 | raw\_model\_response.amount\_raw\_model\_response |
| 51 | raw\_model\_response.next\_purchase\_cost |
| 52 | raw\_model\_response.next\_purchase\_input\_price\_per\_million\_tokens |
| 53 | raw\_model\_response.next\_purchase\_input\_tokens |
| 54 | raw\_model\_response.next\_purchase\_one\_usd\_buys |
| 55 | raw\_model\_response.next\_purchase\_output\_price\_per\_million\_tokens |
| 56 | raw\_model\_response.next\_purchase\_output\_tokens |
| 57 | raw\_model\_response.next\_purchase\_raw\_model\_response |
| 58 | raw\_model\_response.recent\_purchase\_cost |
| 59 | raw\_model\_response.recent\_purchase\_input\_price\_per\_million\_tokens |
| 60 | raw\_model\_response.recent\_purchase\_input\_tokens |
| 61 | raw\_model\_response.recent\_purchase\_one\_usd\_buys |
| 62 | raw\_model\_response.recent\_purchase\_output\_price\_per\_million\_tokens |
| 63 | raw\_model\_response.recent\_purchase\_output\_tokens |
| 64 | raw\_model\_response.recent\_purchase\_raw\_model\_response |
| 65 | reasoning\_summary.amount\_reasoning\_summary |
| 66 | reasoning\_summary.next\_purchase\_reasoning\_summary |
| 67 | reasoning\_summary.recent\_purchase\_reasoning\_summary |
| 68 | scenario.item |
| 69 | scenario.scenario\_index |

We can select and inspect any components of the results. We can see by a “None” response that a question was skipped:

```python theme={null}
(
    results
    .sort_by("annual_income", "age", "item")
    .select("model", "annual_income", "age", "item", "recent_purchase", "amount", "next_purchase")
)
```

| | model.model | agent.annual\_income | agent.age | scenario.item | answer.recent\_purchase | answer.amount | answer.next\_purchase |
| :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | \$100,000-250,000 | 30 | electronics | Yes | 2500.000000 | Within the next year |
| 1 | gemini-1.5-flash | \$100,000-250,000 | 30 | phones | Yes | 1200.000000 | Within the next year |
| 2 | gemini-1.5-flash | \$100,000-250,000 | 50 | electronics | Yes | 2500.000000 | Within the next year |
| 3 | gemini-1.5-flash | \$100,000-250,000 | 50 | phones | Yes | 1200.000000 | Within the next year |
| 4 | gemini-1.5-flash | \$100,000-250,000 | 70 | electronics | Yes | 500.000000 | Within the next year |
| 5 | gemini-1.5-flash | \$100,000-250,000 | 70 | phones | No | nan | Within the next year |
| 6 | gemini-1.5-flash | above \$250,000 | 30 | electronics | Yes | 5000.000000 | Within the next year |
| 7 | gemini-1.5-flash | above \$250,000 | 30 | phones | Yes | 0.000000 | Within the next year |
| 8 | gemini-1.5-flash | above \$250,000 | 50 | electronics | Yes | 5000.000000 | Within the next year |
| 9 | gemini-1.5-flash | above \$250,000 | 50 | phones | Yes | 0.000000 | Within the next year |
| 10 | gemini-1.5-flash | above \$250,000 | 70 | electronics | Yes | 5000.000000 | Within the next year |
| 11 | gemini-1.5-flash | above \$250,000 | 70 | phones | No | nan | Never |
| 12 | gemini-1.5-flash | under \$100,000 | 30 | electronics | Yes | 250.000000 | Within the next year |
| 13 | gemini-1.5-flash | under \$100,000 | 30 | phones | No | nan | Within the next year |
| 14 | gemini-1.5-flash | under \$100,000 | 50 | electronics | No | nan | Within the next year |
| 15 | gemini-1.5-flash | under \$100,000 | 50 | phones | No | nan | Within the next year |
| 16 | gemini-1.5-flash | under \$100,000 | 70 | electronics | No | nan | Within the next year |
| 17 | gemini-1.5-flash | under \$100,000 | 70 | phones | No | nan | Never |

## Example 2

In the next example, we use the same scenarios to create versions of the questions *before we combine them in a survey*. This allows us to add a skip rule based on a question/scenario combination, as opposed to skipping a question for all scenarios:

```python theme={null}
q1 = QuestionYesNo(
    question_name = "recent_purchase_`{{ scenario.item }}`",
    question_text = "In the last year have you or anyone in your household purchased any `{{ scenario.item }}`?",
)

q2 = QuestionNumerical(
    question_name = "amount_`{{ item }}`",
    question_text = "In the last year, how much did your household spend on `{{ scenario.item }}` (in USD)?"
)

q3 = QuestionMultipleChoice(
    question_name = "next_purchase_`{{ scenario.item }}`",
    question_text = "When do you next expect to purchase `{{ scenario.item }}`?",
    question_options = [
        "Never",
        "Within the next month",
        "Within the next year",
        "I do not know"
    ]
)
```

The `loop` method creates new versions of questions with scenarios already inserted:

```python theme={null}
questions = q1.loop(s) + q2.loop(s) + q3.loop(s)
questions
```

```python theme={null}
[Question('yes_no', question_name = """recent_purchase_electronics""", question_text = """In the last year have you or anyone in your household purchased any electronics?""", question_options = ['No', 'Yes']),
 Question('yes_no', question_name = """recent_purchase_phones""", question_text = """In the last year have you or anyone in your household purchased any phones?""", question_options = ['No', 'Yes']),
 Question('numerical', question_name = """amount_electronics""", question_text = """In the last year, how much did your household spend on electronics (in USD)?""", min_value = None, max_value = None),
 Question('numerical', question_name = """amount_phones""", question_text = """In the last year, how much did your household spend on phones (in USD)?""", min_value = None, max_value = None),
 Question('multiple_choice', question_name = """next_purchase_electronics""", question_text = """When do you next expect to purchase electronics?""", question_options = ['Never', 'Within the next month', 'Within the next year', 'I do not know']),
 Question('multiple_choice', question_name = """next_purchase_phones""", question_text = """When do you next expect to purchase phones?""", question_options = ['Never', 'Within the next month', 'Within the next year', 'I do not know'])]
```

We combine the questions in a survey to administer them together the same as before:

```python theme={null}
survey = Survey(questions)
```

Here we add different rules specifying that questions with one scenario (phones) should be administered or skipped based on the answer to a question with another scenario (electronics):

```python theme={null}
survey = (
    survey
    .add_skip_rule("recent_purchase_phones", "`{{ recent_purchase_electronics.answer }}` == 'No'")
    .add_skip_rule("amount_phones", "`{{ recent_purchase_electronics.answer }}` == 'No'")
    .add_skip_rule("next_purchase_phones", "`{{ recent_purchase_electronics.answer }}` == 'No'")
)
```

```python theme={null}
survey.show_flow()
```

<Frame>
  <img alt="../_images/notebooks_skip_logic_scenarios_34_0.png" />
</Frame>

Here we run the survey with the scenarios, agents and model:

```python theme={null}
results = survey.by(a).by(m).run()
```

There is no “scenario” field in results because the scenarios were already added to questions. Instead, there are separate columns for each version of a question:

```python theme={null}
(
    results
    .sort_by("annual_income", "age")
    .select("model", "annual_income", "age", "recent_purchase_electronics", "amount_electronics", "next_purchase_electronics", "recent_purchase_phones", "amount_phones", "next_purchase_phones")
)
```

| | model.model | agent.annual\_income | agent.age | answer.recent\_purchase\_electronics | answer.amount\_electronics | answer.next\_purchase\_electronics | answer.recent\_purchase\_phones | answer.amount\_phones | answer.next\_purchase\_phones |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | gemini-1.5-flash | \$100,000-250,000 | 30 | Yes | 2500 | Within the next year | Yes | 1200.000000 | Within the next year |
| 1 | gemini-1.5-flash | \$100,000-250,000 | 50 | Yes | 2500 | Within the next year | Yes | 1200.000000 | Within the next year |
| 2 | gemini-1.5-flash | \$100,000-250,000 | 70 | Yes | 500 | Within the next year | No | 0.000000 | Within the next year |
| 3 | gemini-1.5-flash | above \$250,000 | 30 | Yes | 5000 | Within the next year | Yes | 0.000000 | Within the next year |
| 4 | gemini-1.5-flash | above \$250,000 | 50 | Yes | 5000 | Within the next year | Yes | 0.000000 | Within the next year |
| 5 | gemini-1.5-flash | above \$250,000 | 70 | Yes | 5000 | Within the next year | No | 0.000000 | Never |
| 6 | gemini-1.5-flash | under \$100,000 | 30 | Yes | 250 | Within the next year | No | 300.000000 | Within the next year |
| 7 | gemini-1.5-flash | under \$100,000 | 50 | No | 250 | Within the next year | nan | nan | nan |
| 8 | gemini-1.5-flash | under \$100,000 | 70 | No | 0 | Within the next year | nan | nan | nan |

## Posting to Expected Parrot

Here we post this notebook to Expected Parrot, a free platform for creating and sharing AI-based research (learn more about [how it works](/en/latest/coop)):

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "skip_logic_scenarios.ipynb")

nb.push(
    description = "Using skip logic with question scenarios",
    alias = "skip-logic-scenarios",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Using skip logic with question scenarios',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/d91d29a1-607e-4a98-a962-a662c7f94399',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/skip-logic-scenarios',
 'uuid': 'd91d29a1-607e-4a98-a962-a662c7f94399',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```


# Starter Tutorial
Source: https://docs.expectedparrot.com/en/latest/notebooks/starter_tutorial

This tutorial provides step-by-step instructions for getting started using EDSL (*Expected Parrot Domain-Specific Language*), an open-source Python library for simulating surveys, experiments and other research tasks using AI agents and large language models. EDSL is developed by [Expected Parrot](https://www.expectedparrot.com/about) and available under the MIT License. The source code is hosted on [GitHub](https://github.com/expectedparrot/edsl).

## Goals of this tutorial

We begin with technical setup: instructions for installing the EDSL library and storing API keys to access language models. Then we demonstrate some of the basic features of EDSL, with examples for constructing and running surveys with agents and models, analyzing responses as datasets, and validating results with human respondents. By the end of this tutorial, you will be able to use EDSL to do each of the following:

* Construct various types of [questions](/en/latest/questions) tailored to your research objectives.
* Combine questions into [surveys](/en/latest/surveys) and integrate logical rules to control the survey flow.
* Add context to questions by piping answers, adding memory of prior questions and answers, and using [scenarios](/en/latest/scenarios) to add data or content to questions.
* Design personas for [AI agents](/en/latest/agents) to simulate responses to your surveys.
* Choose and deploy [large language models](/en/latest/language_models) to generate responses for AI agents.
* Analyze [results](/en/latest/results) as datasets with built-in analytical tools.
* Validate LLM answers with [human respondents](/en/latest/humanize).

## Storing & sharing your work

We also introduce [Expected Parrot](https://www.expectedparrot.com/login): a platform for creating, storing and sharing AI-based research and launching hybrid human/AI surveys. Expected Parrot is fully integrated with EDSL and free to use. At the end of the tutorial we show how to use EDSL with Expected Parrot by posting content created in this tutorial for anyone to view at the web app and launching a web-based survey to compare LLM and human responses.

## Further reading & questions

Please see our [documentation page](/en/latest/index) for more details on each of the topics covered in this notebook. If you encounter any issues or have questions, please email us at [info@expectedparrot.com](mailto:info@expectedparrot.com) or post a question at our [Discord channel](https://discord.com/invite/mxAYkjfy9m).

## Pre-requisites

EDSL is compatible with Python 3.9 - 3.12. Before starting this tutorial, please ensure that you have a Python environment set up on your machine or in a cloud-based environment, such as Google Colab. You can find instructions for installing Python at the [Python Software Foundation](https://www.python.org/downloads/).

## Recommendations

The code examples in this tutorial are designed to be run in a Jupyter notebook or another Python environment, or in a cloud-based environment such as Google Colab.

If you are using Google Colab, please see additional instructions for setting up EDSL in the [Colab setup](/en/latest/colab_setup) page in the documentation.

We also recommend using a virtual environment when installing and using EDSL in order to avoid conflicts with other Python packages. You can find instructions for setting up a virtual environment at the [Python Packaging Authority](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).

## Installation

To begin using EDSL, you first need to install the library. This can either be done locally on your machine or in a cloud-based environment, such as Google Colab. Once you have decided where to install EDSL, you can choose to whether install it from [PyPI](https://pypi.org/project/edsl/) or [GitHub](https://github.com/expectedparrot/edsl):

### From PyPI

Install EDSL directly using `pip`, which is straightforward and recommended for most users. We also recommend using a virtual environment to manage your Python packages (see *Recommendations* above). Uncomment and run the following command to install EDSL from PyPI:

```python theme={null}
# ! uv pip install edsl -q
```

If you have already installed EDSL, you can uncomment and run the following code to check that your version is up to date (compare it to the version at [PyPI](https://pypi.org/project/edsl/)):

```python theme={null}
# pip show edsl
```

If your version of EDSL is not up to date, uncomment and run the following code to update it:

```python theme={null}
# pip install --upgrade edsl
```

### From GitHub

You can find the source code for EDSL and contribute to the project at [GitHub](https://github.com/expectedparrot/edsl). Installing from GitHub allows you to get the latest updates to EDSL before they are released to a new version at PyPI. This is recommended if you are using new features or contributing to the project. Uncomment and run the following command to install EDSL from GitHub:

```python theme={null}
# pip install git+https://github.com/expectedparrot/edsl.git@main
```

## Create an account

Creating an account allows you to run survey jobs at Expected Parrot using language models of your choice, and automatically cache your results. Your account also allows you to launch human surveys and share your content and workflows with other users. Your account comes with \$25 in credits for API calls to LLMs for getting started and a referal code for earning more credits.

[Create an account](https://www.expectedparrot.com/login) with an email address and password, or uncomment and run the following code to be prompted automatically:

```python theme={null}
# from edsl import login

# login()
```

## Accessing LLMs

The next step is to decide how you want to access language models for running surveys. EDSL works with many popular language models that you can choose from to generate responses to your surveys. These models are hosted by various service providers, such as Anthropic, Azure, Bedrock, Deep Infra, Google, Groq, Mistral, OpenAI, Replicate, Together and Xai. In order to run a survey, you need to provide API keys for the service providers of models that you want to use. There are two methods for providing API keys to EDSL:

* Use your Expected Parrot API key to access all available models
* Provide your own API keys from service providers

## Managing keys

To manage your keys, navigate to your [Keys](https://www.expectedparrot.com/home/keys) page and use the options to add keys and optionally share access to them with other users. You can specify which keys to use at any time, and check the current priority of your keys. Your Expected Parrot API key is used by default when a private key is not provided for a selected model.

Please see [instructions](https://www.expectedparrot.com/getting-started/edsl-api-keys) for alternative methods of storing your own API keys.

<Note>
  **Note**:

  If you try to run a survey without storing a required API key, you will be provided a link to activate remote inference and use your Expected Parrot API key.
</Note>

## Credits & tokens

Running surveys with language models requires tokens. If you are using your own API keys, service providers will bill you directly. If you are using your Expected Parrot API key to access models, you will need to purchase credits to cover token costs. Please see the model pricing page for details on available models and their current prices.

<Note>
  **Note**:

  Your account comes with 100 free credits. You can purchase more credits at any time at your [Credits](https://www.expectedparrot.com/home/credits) page.
</Note>

After installing EDSL and storing API keys you are ready to run some examples!

## Example: Running a simple question

EDSL comes with a [variety of question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from a model. To see a list of all question types:

```python theme={null}
from edsl import Question

Question.available()
```

| | question\_type | question\_class | example\_question |
| :- | :- | :- | :- |
| 0 | checkbox | QuestionCheckBox | Question('checkbox', question\_name = """never\_eat""", question\_text = """Which of the following foods would you eat if you had to?""", min\_selections = 2, max\_selections = 5, question\_options = \['soggy meatpie', 'rare snails', 'mouldy bread', 'panda milk custard', 'McDonalds'], include\_comment = False) |
| 1 | compute | QuestionCompute | Question('compute', question\_name = """computed\_greeting""", question\_text = """Hello `{{name}}`!""") |
| 2 | dict | QuestionDict | Question('dict', question\_name = """example""", question\_text = """Please provide a simple recipe for hot chocolate.""", answer\_keys = \['title', 'ingredients', 'num\_ingredients', 'instructions'], value\_types = \['str', 'list\[str]', 'int', 'str'], value\_descriptions = \['The title of the recipe.', 'A list of ingredients.', 'The number of ingredients.', 'The instructions for making the recipe.'], question\_presentation = """Please provide a simple recipe for hot chocolate.""", answering\_instructions = """Please respond with a dictionary using the following keys: title, ingredients, num\_ingredients, instructions. Here are descriptions of the values to provide: - "title": "The title of the recipe." - "ingredients": "A list of ingredients." - "num\_ingredients": "The number of ingredients." - "instructions": "The instructions for making the recipe." The values should be formatted in the following types: - "title": "str" - "ingredients": "list\[str]" - "num\_ingredients": "int" - "instructions": "str" If you do not have a value for a given key, use "null". After the answer, you can put a comment explaining your response on the next line. """) |
| 3 | extract | QuestionExtract | Question('extract', question\_name = """extract\_name""", question\_text = """My name is Moby Dick. I have a PhD in astrology, but I'm actually a truck driver""", answer\_template = `{{'name': 'John Doe', 'profession': 'Carpenter'}}`) |
| 4 | free\_text | QuestionFreeText | Question('free\_text', question\_name = """how\_are\_you""", question\_text = """How are you?""") |
| 5 | functional | QuestionFunctional | Question('functional', question\_name = """sum\_and\_multiply""", question\_text = """Calculate the sum of the list and multiply it by the agent trait multiplier.""") |
| 6 | likert\_five | QuestionLikertFive | Question('likert\_five', question\_name = """happy\_raining""", question\_text = """I'm only happy when it rains.""", question\_options = \['Strongly disagree', 'Disagree', 'Neutral', 'Agree', 'Strongly agree']) |
| 7 | linear\_scale | QuestionLinearScale | Question('linear\_scale', question\_name = """ice\_cream""", question\_text = """How much do you like ice cream?""", question\_options = \[1, 2, 3, 4, 5], option\_labels = `{{1: 'I hate it', 5: 'I love it'}}`) |
| 8 | list | QuestionList | Question('list', question\_name = """list\_of\_foods""", question\_text = """What are your favorite foods?""", max\_list\_items = None, min\_list\_items = None) |
| 9 | matrix | QuestionMatrix | Question('matrix', question\_name = """child\_happiness""", question\_text = """How happy would you be with different numbers of children?""", question\_items = \['No children', '1 child', '2 children', '3 or more children'], question\_options = \[1, 2, 3, 4, 5], option\_labels = `{{1: 'Very sad', 3: 'Neutral', 5: 'Extremely happy'}}`) |
| 10 | multiple\_choice | QuestionMultipleChoice | Question('multiple\_choice', question\_name = """how\_feeling""", question\_text = """How are you?""", question\_options = \['Good', 'Great', 'OK', 'Bad'], include\_comment = False) |
| 11 | multiple\_choice\_with\_other | QuestionMultipleChoiceWithOther | Question('multiple\_choice\_with\_other', question\_name = """how\_feeling\_with\_other""", question\_text = """How are you?""", question\_options = \['Good', 'Great', 'OK', 'Bad'], include\_comment = False) |
| 12 | numerical | QuestionNumerical | Question('numerical', question\_name = """age""", question\_text = """You are a 45 year old man. How old are you in years?""", min\_value = 0, max\_value = 86.7, include\_comment = False) |
| 13 | rank | QuestionRank | Question('rank', question\_name = """rank\_foods""", question\_text = """Rank your favorite foods.""", question\_options = \['Pizza', 'Pasta', 'Salad', 'Soup'], num\_selections = 2) |
| 14 | top\_k | QuestionTopK | Question('top\_k', question\_name = """two\_fruits""", question\_text = """Which of the following fruits do you prefer?""", min\_selections = 2, max\_selections = 2, question\_options = \['apple', 'banana', 'carrot', 'durian'], use\_code = True) |
| 15 | yes\_no | QuestionYesNo | Question('yes\_no', question\_name = """is\_it\_equal""", question\_text = """Is 5 + 5 equal to 11?""", question\_options = \['No', 'Yes']) |

We can see the components of a particular question type by importing the question type class and calling the `example` method on it:

```python theme={null}
from edsl import (
    # QuestionCheckBox,
    # QuestionExtract,
    # QuestionFreeText,
    # QuestionFunctional,
    # QuestionLikertFive,
    # QuestionLinearScale,
    # QuestionList,
    QuestionMultipleChoice,
    # QuestionNumerical,
    # QuestionRank,
    # QuestionTopK,
    # QuestionYesNo
)

q = QuestionMultipleChoice.example() # substitute any question type class name
q
```

```
QuestionMultipleChoice
```

| | key | value |
| :- | :- | :- |
| 0 | question\_name | how\_feeling |
| 1 | question\_text | How are you? |
| 2 | question\_options:0 | Good |
| 3 | question\_options:1 | Great |
| 4 | question\_options:2 | OK |
| 5 | question\_options:3 | Bad |
| 6 | include\_comment | False |
| 7 | question\_type | multiple\_choice |

Here we create a simple multiple choice question of our own:

```python theme={null}
from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
    question_name = "smallest_prime",
    question_text = "Which is the smallest prime number?",
    question_options = [0, 1, 2, 3]
)
```

We can administer the question to a language model by calling the `run` method on it. If you have activated remote inference and stored your Expected Parrot API key (see instructions above), the question will be run remotely at the Expected Parrot server. Results are stored at an unlisted page by default; we can also set the visibility to `public` or `private` either when we run it or by updating the object (demonstrated in later examples). We can also view a progress report for the job:

```python theme={null}
results = q.run()
```

### Inspecting results

This generates a dataset of `Results` that we can readily access with [built-in methods for analysis](/en/latest/results). Here we `select()` the response to inspect it, together with the model that was used and the model’s “comment” about its response–a field that is automatically added to all question types other than free text:

```python theme={null}
results.select("model", "smallest_prime", "smallest_prime_comment")
```

| | model.model | answer.smallest\_prime | comment.smallest\_prime\_comment |
| :- | :- | :- | :- |
| 0 | gpt-4o | 2 | 2 is the smallest prime number because it is the only even number that is divisible by only 1 and itself. |

The `Results` also include information about the question, model parameters, prompts, generated tokens and raw responses. To see a list of all the components:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | answer.smallest\_prime |
| 4 | cache\_keys.smallest\_prime\_cache\_key |
| 5 | cache\_used.smallest\_prime\_cache\_used |
| 6 | comment.smallest\_prime\_comment |
| 7 | generated\_tokens.smallest\_prime\_generated\_tokens |
| 8 | iteration.iteration |
| 9 | model.frequency\_penalty |
| 10 | model.inference\_service |
| 11 | model.logprobs |
| 12 | model.max\_tokens |
| 13 | model.model |
| 14 | model.model\_index |
| 15 | model.presence\_penalty |
| 16 | model.temperature |
| 17 | model.top\_logprobs |
| 18 | model.top\_p |
| 19 | prompt.smallest\_prime\_system\_prompt |
| 20 | prompt.smallest\_prime\_user\_prompt |
| 21 | question\_options.smallest\_prime\_question\_options |
| 22 | question\_text.smallest\_prime\_question\_text |
| 23 | question\_type.smallest\_prime\_question\_type |
| 24 | raw\_model\_response.smallest\_prime\_cost |
| 25 | raw\_model\_response.smallest\_prime\_input\_price\_per\_million\_tokens |
| 26 | raw\_model\_response.smallest\_prime\_input\_tokens |
| 27 | raw\_model\_response.smallest\_prime\_one\_usd\_buys |
| 28 | raw\_model\_response.smallest\_prime\_output\_price\_per\_million\_tokens |
| 29 | raw\_model\_response.smallest\_prime\_output\_tokens |
| 30 | raw\_model\_response.smallest\_prime\_raw\_model\_response |
| 31 | reasoning\_summary.smallest\_prime\_reasoning\_summary |
| 32 | scenario.scenario\_index |
| 33 | validated.smallest\_prime\_validated |

## Example: Conducting a survey with agents and models

In the next example we construct a more complex survey consisting of multiple questions, and design personas for AI agents to answer the survey. Then we select specific language models to generate the answers.

We start by creating questions in different types and passing them to a `Survey`:

```python theme={null}
from edsl import QuestionLinearScale, QuestionFreeText

q_enjoy = QuestionLinearScale(
    question_name = "enjoy",
    question_text = "On a scale from 1 to 5, how much do you enjoy reading?",
    question_options = [1, 2, 3, 4, 5],
    option_labels = {{1:"Not at all", 5:"Very much"}
)

q_favorite_place = QuestionFreeText(
    question_name = "favorite_place",
    question_text = "Describe your favorite place for reading."
)
```

We construct a `Survey` by passing a list of questions:

```python theme={null}
from edsl import Survey

survey = Survey(questions = [q_enjoy, q_favorite_place])
```

### Agents

An important feature of EDSL is the ability to create AI agents to answer questions. This is done by passing dictionaries of relevant “traits” to `Agent` objects that are used by language models to generate responses. Learn more about [designing agents](/en/latest/agents).

Here we construct several simple agent personas to use with our survey:

```python theme={null}
from edsl import AgentList, Agent

agents = AgentList(
    Agent(traits = {{"persona":p}) for p in ["artist", "mechanic", "sailor"]
)
```

### Language models

EDSL works with many popular large language models that we can select to use with a survey. This makes it easy to compare responses among models in the results that are generated.

See a current list of available models at our [model pricing and performance page](https://www.expectedparrot.com/getting-started/coop-pricing). You can also check available service providers:

```python theme={null}
from edsl import Model

Model.services()
```

[ScenarioList](/en/latest/scenarios) scenarios: 15; keys: \['service'];

| | service |
| :- | :- |
| 0 | anthropic |
| 1 | azure |
| 2 | bedrock |
| 3 | deep\_infra |
| 4 | deepseek |
| 5 | google |
| 6 | groq |
| 7 | mistral |
| 8 | ollama |
| 9 | open\_router |
| 10 | openai |
| 11 | openai\_v2 |
| 12 | perplexity |
| 13 | together |
| 14 | xai |

To check the default model that will be used if no models are specified for a survey (e.g., as in the first example above):

```python theme={null}
Model()
```

```
LanguageModel
```

| | key | value |
| :- | :- | :- |
| 0 | model | gpt-4o |
| 1 | parameters:temperature | 0.500000 |
| 2 | parameters:max\_tokens | 1000 |
| 3 | parameters:top\_p | 1 |
| 4 | parameters:frequency\_penalty | 0 |
| 5 | parameters:presence\_penalty | 0 |
| 6 | parameters:logprobs | False |
| 7 | parameters:top\_logprobs | 3 |
| 8 | inference\_service | openai |

<Note>
  **Note**:

  (Note that the output may be different if the default model has changed since this page was last updated.)
</Note>

Here we select some models to use with our survey:

```python theme={null}
from edsl import ModelList, Model

models = ModelList([
    Model("gpt-4o", service_name = "openai"),
    Model("gemini-1.5-flash", service_name = "google"),
    Model("claude-3-7-sonnet-20250219", service_name = "anthropic")
])
```

### Running a survey

We add agents and models to a survey using the `by` method. Then we administer a survey the same way that we do an individual question, by calling the `run` method on it:

```python theme={null}
results = survey.by(agents).by(models).run()
```

We can pass an expression to `filter()` the results and list the components to `sort_by()`:

```python theme={null}
(
    results
    .filter("persona != 'artist'")
    .order_by("persona", "model")
    .select("model", "persona", "enjoy", "favorite_place")
)
```

| | model.model | agent.persona | answer.enjoy | answer.favorite\_place |
| :- | :- | :- | :- | :- |
| 0 | claude-3-7-sonnet-20250219 | mechanic | 3 | As a mechanic, I'd have to say my favorite place for reading is actually in my workshop during lunch breaks or after hours when things quiet down. There's this old, worn leather chair I salvaged and fixed up that sits in the corner by a window. The natural light is perfect during the day, and I've got a good shop lamp nearby for evening reading. I usually have service manuals, repair guides, or car magazines stacked on a small side table, but I also enjoy a good thriller or history book there too. There's something calming about being surrounded by my tools and projects while taking a mental break with a book. The faint smell of oil and metal gives it a comfortable familiarity that I find relaxing. Plus, if I read something interesting about a new repair technique, I'm right where I need to be to try it out! |
| 1 | gemini-1.5-flash | mechanic | 3 | My favorite place to read? Gotta be my garage, actually. Not the whole thing, mind you. Just that little corner by the workbench, where the light's just right – not too harsh, you know? I've got a comfy old stool there, worn smooth from years of use, and a little side table I’ve cobbled together from scrap parts. Got a good lamp on it, too, one of those adjustable ones so I can get the perfect angle. The air smells of oil and grease, sure, but it's a familiar smell, comforting even. Plus, there's always something interesting to look at – a half-finished project, a neat tool I haven't used in a while... it keeps my mind occupied even when I'm supposed to be concentrating on the book. It's peaceful in its own way, you know? The quiet hum of the fridge in the back, the occasional drip from a leaky faucet… it’s my sanctuary. |
| 2 | gpt-4o | mechanic | 3 | As a mechanic, my favorite place for reading is actually my garage. It's not your typical cozy reading nook, but there's something about the smell of oil and the sound of tools clinking that feels comforting. I've got a sturdy workbench where I can prop up a manual or a good book about classic cars. The lighting is bright enough to read by, and when I need a break, I can glance up at the projects I’m working on. Plus, it's quiet, especially when the garage door is down, so I can really focus on whatever I'm reading. |
| 3 | claude-3-7-sonnet-20250219 | sailor | 3 | Ah, me favorite spot for readin'? That'd be the fo'c'sle (forecastle) of me ship when we're anchored in a calm bay. There's somethin' special about sittin' on a sea chest with me back against the hull, lantern swingin' gently overhead, and the soft sounds of water lappin' against the sides. The gentle rockin' of the vessel puts ye in a perfect state of mind for gettin' lost in a tale. I like to read in the early mornin' when the air is crisp and most of the crew is still snorin' away, or at dusk when the day's work is done. When I'm ashore, I fancy findin' a quiet spot near the harbor where I can still see the ships and smell the salt air. A good book, the distant cry of gulls, and the promise of the open water - that's all this old salt needs for a perfect readin' nook. |
| 4 | gemini-1.5-flash | sailor | 3 | Ahoy there! My favorite place for a good read? That's easy. The crow's nest, of course! Up there, high above the deck, with the wind whipping through my hair and the spray of the ocean kissing my face... It's the perfect spot. The rocking of the ship is a bit of a distraction sometimes, but the view\... the endless horizon... it's inspiring. Makes even the dullest sea shanty seem like a thrilling adventure. Plus, nobody bothers you up there! Just me, my book, and the vast, beautiful ocean. |
| 5 | gpt-4o | sailor | 3 | Ah, my favorite place for reading has to be the deck of a ship, right under the open sky. There's something about the gentle sway of the sea and the salty breeze that makes the words come alive. I like to find a quiet spot, maybe near the stern where the sound of the waves is a bit more pronounced. The horizon stretches out endlessly, and with the sun setting, casting golden hues across the water, it feels like I'm part of the stories I'm reading. It's a place where adventure and tranquility meet, perfect for diving into a good book. |

## Example: Adding context to questions

EDSL provides a variety of ways to add data or content to survey questions. These methods include:

* [Piping](/en/latest/surveys#id2) answers to questions into follow-on questions
* [Adding “memory”](/en/latest/surveys#question-memory) of prior questions and answers in a survey when presenting other questions to a model
* [Parameterizing questions with data](/en/latest/scenarios), e.g., content from PDFs, CSVs, docs, images or other sources that you want to add to questions

### Piping question answers

Here we demonstrate how to pipe the answer to a question into the text of another question. This is done by using a placeholder `{{ <question_name>.answer }}` in the text of the follow-on question where the answer to the prior question is to be inserted when the survey is run. This causes the questions to be administered in the required order (survey questions are administered asynchronously by default). Learn more about [piping question answers](/en/latest/surveys#id2).

Here we insert the answer to a numerical question into the text of a follow-on yes/no question:

```python theme={null}
from edsl import QuestionNumerical, QuestionYesNo, Survey

q1 = QuestionNumerical(
    question_name = "random_number",
    question_text = "Pick a random number between 1 and 1,000."
)

q2 = QuestionYesNo(
    question_name = "prime",
    question_text = "Is this a prime number: {{ random_number.answer }}"
)
survey = Survey([q1, q2])

results = survey.run()
```

We can check the `user_prompt` for the `prime` question to verify that the answer to the `random_number` question was piped into it:

```python theme={null}
results.select("random_number", "prime_user_prompt", "prime", "prime_comment")
```

| | answer.random\_number | prompt.prime\_user\_prompt | answer.prime | comment.prime\_comment |
| :- | :- | :- | :- | :- |
| 0 | 487 | Is this a prime number: 487 No Yes Only 1 option may be selected. Please respond with just your answer. After the answer, you can put a comment explaining your response. | Yes | 487 is a prime number because it has no divisors other than 1 and itself. |

### Adding “memory” of questions and answers

Here we instead add a “memory” of the first question and answer to the context of the second question. This is done by calling a memory rule and identifying the question(s) to add. Instead of just the answer, information about the full question and answer are presented with the follow-on question text, and no placeholder is used. Learn more about [question memory rules](/en/latest/surveys#survey-rules-logic).

Here we demonstrate the `add_targeted_memory` method (we could also use `set_full_memory_mode` or other memory rules):

```python theme={null}
from edsl import QuestionNumerical, QuestionYesNo, Survey

q1 = QuestionNumerical(
    question_name = "random_number",
    question_text = "Pick a random number between 1 and 1,000."
)
q2 = QuestionYesNo(
    question_name = "prime",
    question_text = "Is the number you picked a prime number?"
)

survey = Survey([q1, q2]).add_targeted_memory(q2, q1)

results = survey.run()
```

We can again use the `user_prompt` to verify the context that was added to the follow-on question. To view the results in a long table, we can call the `table()` and `long()` methods to modify the default table view:

```python theme={null}
results.select("random_number", "prime_user_prompt", "prime", "prime_comment")
```

| | answer.random\_number | prompt.prime\_user\_prompt | answer.prime | comment.prime\_comment |
| :- | :- | :- | :- | :- |
| 0 | 487 | Is the number you picked a prime number? No Yes Only 1 option may be selected. Please respond with just your answer. After the answer, you can put a comment explaining your response. Before the question you are now answering, you already answered the following question(s): Question: Pick a random number between 1 and 1,000. Answer: 487 | Yes | 487 is a prime number because it has no divisors other than 1 and itself. |

*Related topic: Learn more about exploring and simulating “randomness” with AI agents and LLMs in*[this notebook](/en/latest/notebooks/random_numbers)*.*

## Scenarios

We can also add external data or content to survey questions. This can be useful when you want to efficiently create and administer multiple versions of questions at once, e.g., for conducting data labeling tasks. This is done by creating `Scenario` dictionaries for the data or content to be used with a survey, where the keys match `{{ placeholder }}` names used in question texts (or question options) and the values are the content to be added. Scenarios can also be used to [add metadata to survey results](/en/latest/notebooks/adding_metadata), e.g., data sources or other information that you may want to include in the results for reference but not necessarily include in question texts.

In the next example we revise the prior survey questions about reading to take a parameter for other activities that we may want to add to the questions, and create simple scenarios for some activities. EDSL provides methods for automatically generating scenarios from a variety of data sources, including PDFs, CSVs, docs, images, tables and dicts. We use the `from_list` method to convert a list of activities into scenarios.

Then we demonstrate how to use scenarios to create multiple versions of our questions either (i) when constructing a survey or (ii) when running it:

* In the latter case, the `by` method is used to add scenarios to a survey of questions with placeholders at the time that it is run (the same way that agents and models are added to a survey). This adds a `scenario` column to the results with a row for each answer to each question for each scenario.
* In the former case, the `loop` method is used to create a list of versions of a question with the scenarios already added to it; when the questions are passed to a survey and it is run, the results include columns for each individual question; there is no `scenario` column and a single row for each agent’s answers to all the questions.

Learn more about [using scenarios](/en/latest/scenarios).

Here we create scenarios for a simple list of activities:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_list("activity", ["reading", "running", "relaxing"])
```

### Adding scenarios using the `by` method

Here we add the scenarios to the survey when we run it, together with any desired agents and models:

```python theme={null}
from edsl import QuestionLinearScale, QuestionFreeText, Survey

q_enjoy = QuestionLinearScale(
    question_name = "enjoy",
    question_text = "On a scale from 1 to 5, how much do you enjoy {{ scenario.activity }}?",
    question_options = [1, 2, 3, 4, 5],
    option_labels = {{1:"Not at all", 5:"Very much"}
)

q_favorite_place = QuestionFreeText(
    question_name = "favorite_place",
    question_text = "In a brief sentence, describe your favorite place for {{ scenario.activity }}."
)

survey = Survey([q_enjoy, q_favorite_place])

results = survey.by(scenarios).by(agents).by(models).run()
```

We can optionally drop the prefixes `agent`, `scenario`, `answer`, etc., when fields are unique:

```python theme={null}
(
    results
    .filter("model.model == 'gpt-4o'")
    .order_by("activity", "persona")
    .select("activity", "persona", "enjoy", "favorite_place")
)
```

| | scenario.activity | agent.persona | answer.enjoy | answer.favorite\_place |
| :- | :- | :- | :- | :- |
| 0 | reading | artist | 4 | My favorite place for reading is a cozy corner of my art studio, surrounded by vibrant canvases and the soft glow of afternoon light filtering through the window. |
| 1 | reading | mechanic | 3 | My favorite place for reading is the cozy corner of my garage, surrounded by tools and the smell of motor oil, where I can escape into a good book during breaks. |
| 2 | reading | sailor | 3 | My favorite place for reading is the deck of a ship, with the sound of waves lapping against the hull and a gentle sea breeze in the air. |
| 3 | relaxing | artist | 4 | My favorite place for relaxing is a quiet, sun-dappled corner of my art studio, surrounded by canvases and the soft hum of creativity. |
| 4 | relaxing | mechanic | 4 | My favorite place for relaxing is my garage, surrounded by the familiar scent of motor oil and the satisfying hum of engines. |
| 5 | relaxing | sailor | 3 | My favorite place for relaxing is on the deck of a ship, watching the horizon as the sun sets over the endless ocean. |
| 6 | running | artist | 2 | My favorite place for running is a serene forest trail where the dappled sunlight dances through the leaves and the air is filled with the earthy scent of nature. |
| 7 | running | mechanic | 1 | I don't run much, but I'd imagine a quiet trail through the woods would be a nice spot for a jog. |
| 8 | running | sailor | 2 | As a sailor, my favorite place for running is along the beach at sunrise, with the sound of the waves and the salty sea breeze filling the air. |

### Adding scenarios using the `loop` method

Here we add scenarios to questions when constructing a survey, as opposed to when running it. When we run the survey the results will include columns for each question and no `scenario` field. Note that we can also optionally use the scenario key in the question names (they are otherwise incremented by default):

```python theme={null}
from edsl import QuestionLinearScale, QuestionFreeText

q_enjoy = QuestionLinearScale(
    question_name = "enjoy_{{ scenario.activity }}", # optional use of scenario key
    question_text = "On a scale from 1 to 5, how much do you enjoy {{ scenario.activity }}?",
    question_options = [1, 2, 3, 4, 5],
    option_labels = {{1:"Not at all", 5:"Very much"}
)

q_favorite_place = QuestionFreeText(
    question_name = "favorite_place_{{ scenario.activity }}", # optional use of scenario key
    question_text = "In a brief sentence, describe your favorite place for {{ scenario.activity }}."
)
```

Looping the scenarios to create lists of questions:

```python theme={null}
enjoy_questions = q_enjoy.loop(scenarios)
enjoy_questions
```

```
[Question('linear_scale', question_name = """enjoy_reading""", question_text = """On a scale from 1 to 5, how much do you enjoy reading?""", question_options = [1, 2, 3, 4, 5], option_labels = {{1: 'Not at all', 5: 'Very much'}),
 Question('linear_scale', question_name = """enjoy_running""", question_text = """On a scale from 1 to 5, how much do you enjoy running?""", question_options = [1, 2, 3, 4, 5], option_labels = {{1: 'Not at all', 5: 'Very much'}),
 Question('linear_scale', question_name = """enjoy_relaxing""", question_text = """On a scale from 1 to 5, how much do you enjoy relaxing?""", question_options = [1, 2, 3, 4, 5], option_labels = {{1: 'Not at all', 5: 'Very much'})]
```

```python theme={null}
favorite_place_questions = q_favorite_place.loop(scenarios)
favorite_place_questions
```

```
[Question('free_text', question_name = """favorite_place_reading""", question_text = """In a brief sentence, describe your favorite place for reading."""),
 Question('free_text', question_name = """favorite_place_running""", question_text = """In a brief sentence, describe your favorite place for running."""),
 Question('free_text', question_name = """favorite_place_relaxing""", question_text = """In a brief sentence, describe your favorite place for relaxing.""")]
```

Combining the questions in a survey:

```python theme={null}
survey = Survey(questions = enjoy_questions + favorite_place_questions)

results = survey.by(agents).by(models).run()
```

We can see that there are additional question fields and no “scenario” field:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | agent.persona |
| 4 | answer.enjoy\_reading |
| 5 | answer.enjoy\_relaxing |
| 6 | answer.enjoy\_running |
| 7 | answer.favorite\_place\_reading |
| 8 | answer.favorite\_place\_relaxing |
| 9 | answer.favorite\_place\_running |
| 10 | cache\_keys.enjoy\_reading\_cache\_key |
| 11 | cache\_keys.enjoy\_relaxing\_cache\_key |
| 12 | cache\_keys.enjoy\_running\_cache\_key |
| 13 | cache\_keys.favorite\_place\_reading\_cache\_key |
| 14 | cache\_keys.favorite\_place\_relaxing\_cache\_key |
| 15 | cache\_keys.favorite\_place\_running\_cache\_key |
| 16 | cache\_used.enjoy\_reading\_cache\_used |
| 17 | cache\_used.enjoy\_relaxing\_cache\_used |
| 18 | cache\_used.enjoy\_running\_cache\_used |
| 19 | cache\_used.favorite\_place\_reading\_cache\_used |
| 20 | cache\_used.favorite\_place\_relaxing\_cache\_used |
| 21 | cache\_used.favorite\_place\_running\_cache\_used |
| 22 | comment.enjoy\_reading\_comment |
| 23 | comment.enjoy\_relaxing\_comment |
| 24 | comment.enjoy\_running\_comment |
| 25 | comment.favorite\_place\_reading\_comment |
| 26 | comment.favorite\_place\_relaxing\_comment |
| 27 | comment.favorite\_place\_running\_comment |
| 28 | generated\_tokens.enjoy\_reading\_generated\_tokens |
| 29 | generated\_tokens.enjoy\_relaxing\_generated\_tokens |
| 30 | generated\_tokens.enjoy\_running\_generated\_tokens |
| 31 | generated\_tokens.favorite\_place\_reading\_generated\_tokens |
| 32 | generated\_tokens.favorite\_place\_relaxing\_generated\_tokens |
| 33 | generated\_tokens.favorite\_place\_running\_generated\_tokens |
| 34 | iteration.iteration |
| 35 | model.frequency\_penalty |
| 36 | model.inference\_service |
| 37 | model.logprobs |
| 38 | model.maxOutputTokens |
| 39 | model.max\_tokens |
| 40 | model.model |
| 41 | model.model\_index |
| 42 | model.presence\_penalty |
| 43 | model.stopSequences |
| 44 | model.temperature |
| 45 | model.topK |
| 46 | model.topP |
| 47 | model.top\_logprobs |
| 48 | model.top\_p |
| 49 | prompt.enjoy\_reading\_system\_prompt |
| 50 | prompt.enjoy\_reading\_user\_prompt |
| 51 | prompt.enjoy\_relaxing\_system\_prompt |
| 52 | prompt.enjoy\_relaxing\_user\_prompt |
| 53 | prompt.enjoy\_running\_system\_prompt |
| 54 | prompt.enjoy\_running\_user\_prompt |
| 55 | prompt.favorite\_place\_reading\_system\_prompt |
| 56 | prompt.favorite\_place\_reading\_user\_prompt |
| 57 | prompt.favorite\_place\_relaxing\_system\_prompt |
| 58 | prompt.favorite\_place\_relaxing\_user\_prompt |
| 59 | prompt.favorite\_place\_running\_system\_prompt |
| 60 | prompt.favorite\_place\_running\_user\_prompt |
| 61 | question\_options.enjoy\_reading\_question\_options |
| 62 | question\_options.enjoy\_relaxing\_question\_options |
| 63 | question\_options.enjoy\_running\_question\_options |
| 64 | question\_options.favorite\_place\_reading\_question\_options |
| 65 | question\_options.favorite\_place\_relaxing\_question\_options |
| 66 | question\_options.favorite\_place\_running\_question\_options |
| 67 | question\_text.enjoy\_reading\_question\_text |
| 68 | question\_text.enjoy\_relaxing\_question\_text |
| 69 | question\_text.enjoy\_running\_question\_text |
| 70 | question\_text.favorite\_place\_reading\_question\_text |
| 71 | question\_text.favorite\_place\_relaxing\_question\_text |
| 72 | question\_text.favorite\_place\_running\_question\_text |
| 73 | question\_type.enjoy\_reading\_question\_type |
| 74 | question\_type.enjoy\_relaxing\_question\_type |
| 75 | question\_type.enjoy\_running\_question\_type |
| 76 | question\_type.favorite\_place\_reading\_question\_type |
| 77 | question\_type.favorite\_place\_relaxing\_question\_type |
| 78 | question\_type.favorite\_place\_running\_question\_type |
| 79 | raw\_model\_response.enjoy\_reading\_cost |
| 80 | raw\_model\_response.enjoy\_reading\_input\_price\_per\_million\_tokens |
| 81 | raw\_model\_response.enjoy\_reading\_input\_tokens |
| 82 | raw\_model\_response.enjoy\_reading\_one\_usd\_buys |
| 83 | raw\_model\_response.enjoy\_reading\_output\_price\_per\_million\_tokens |
| 84 | raw\_model\_response.enjoy\_reading\_output\_tokens |
| 85 | raw\_model\_response.enjoy\_reading\_raw\_model\_response |
| 86 | raw\_model\_response.enjoy\_relaxing\_cost |
| 87 | raw\_model\_response.enjoy\_relaxing\_input\_price\_per\_million\_tokens |
| 88 | raw\_model\_response.enjoy\_relaxing\_input\_tokens |
| 89 | raw\_model\_response.enjoy\_relaxing\_one\_usd\_buys |
| 90 | raw\_model\_response.enjoy\_relaxing\_output\_price\_per\_million\_tokens |
| 91 | raw\_model\_response.enjoy\_relaxing\_output\_tokens |
| 92 | raw\_model\_response.enjoy\_relaxing\_raw\_model\_response |
| 93 | raw\_model\_response.enjoy\_running\_cost |
| 94 | raw\_model\_response.enjoy\_running\_input\_price\_per\_million\_tokens |
| 95 | raw\_model\_response.enjoy\_running\_input\_tokens |
| 96 | raw\_model\_response.enjoy\_running\_one\_usd\_buys |
| 97 | raw\_model\_response.enjoy\_running\_output\_price\_per\_million\_tokens |
| 98 | raw\_model\_response.enjoy\_running\_output\_tokens |
| 99 | raw\_model\_response.enjoy\_running\_raw\_model\_response |
| 100 | raw\_model\_response.favorite\_place\_reading\_cost |
| 101 | raw\_model\_response.favorite\_place\_reading\_input\_price\_per\_million\_tokens |
| 102 | raw\_model\_response.favorite\_place\_reading\_input\_tokens |
| 103 | raw\_model\_response.favorite\_place\_reading\_one\_usd\_buys |
| 104 | raw\_model\_response.favorite\_place\_reading\_output\_price\_per\_million\_tokens |
| 105 | raw\_model\_response.favorite\_place\_reading\_output\_tokens |
| 106 | raw\_model\_response.favorite\_place\_reading\_raw\_model\_response |
| 107 | raw\_model\_response.favorite\_place\_relaxing\_cost |
| 108 | raw\_model\_response.favorite\_place\_relaxing\_input\_price\_per\_million\_tokens |
| 109 | raw\_model\_response.favorite\_place\_relaxing\_input\_tokens |
| 110 | raw\_model\_response.favorite\_place\_relaxing\_one\_usd\_buys |
| 111 | raw\_model\_response.favorite\_place\_relaxing\_output\_price\_per\_million\_tokens |
| 112 | raw\_model\_response.favorite\_place\_relaxing\_output\_tokens |
| 113 | raw\_model\_response.favorite\_place\_relaxing\_raw\_model\_response |
| 114 | raw\_model\_response.favorite\_place\_running\_cost |
| 115 | raw\_model\_response.favorite\_place\_running\_input\_price\_per\_million\_tokens |
| 116 | raw\_model\_response.favorite\_place\_running\_input\_tokens |
| 117 | raw\_model\_response.favorite\_place\_running\_one\_usd\_buys |
| 118 | raw\_model\_response.favorite\_place\_running\_output\_price\_per\_million\_tokens |
| 119 | raw\_model\_response.favorite\_place\_running\_output\_tokens |
| 120 | raw\_model\_response.favorite\_place\_running\_raw\_model\_response |
| 121 | reasoning\_summary.enjoy\_reading\_reasoning\_summary |
| 122 | reasoning\_summary.enjoy\_relaxing\_reasoning\_summary |
| 123 | reasoning\_summary.enjoy\_running\_reasoning\_summary |
| 124 | reasoning\_summary.favorite\_place\_reading\_reasoning\_summary |
| 125 | reasoning\_summary.favorite\_place\_relaxing\_reasoning\_summary |
| 126 | reasoning\_summary.favorite\_place\_running\_reasoning\_summary |
| 127 | scenario.scenario\_index |
| 128 | validated.enjoy\_reading\_validated |
| 129 | validated.enjoy\_relaxing\_validated |
| 130 | validated.enjoy\_running\_validated |
| 131 | validated.favorite\_place\_reading\_validated |
| 132 | validated.favorite\_place\_relaxing\_validated |
| 133 | validated.favorite\_place\_running\_validated |

```python theme={null}
(
    results
    .filter("model.model == 'gpt-4o'")
    .order_by("persona")
    .select("persona", "enjoy_reading", "enjoy_running", "enjoy_relaxing", "favorite_place_reading", "favorite_place_running", "favorite_place_relaxing")
)
```

| | agent.persona | answer.enjoy\_reading | answer.enjoy\_running | answer.enjoy\_relaxing | answer.favorite\_place\_reading | answer.favorite\_place\_running | answer.favorite\_place\_relaxing |
| :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | artist | 4 | 2 | 4 | My favorite place for reading is a cozy corner of my art studio, surrounded by vibrant canvases and the soft glow of afternoon light filtering through the window. | My favorite place for running is a serene forest trail where the dappled sunlight dances through the leaves and the air is filled with the earthy scent of nature. | My favorite place for relaxing is a quiet, sun-dappled corner of my art studio, surrounded by canvases and the soft hum of creativity. |
| 1 | mechanic | 3 | 1 | 4 | My favorite place for reading is the cozy corner of my garage, surrounded by tools and the smell of motor oil, where I can escape into a good book during breaks. | I don't run much, but I'd imagine a quiet trail through the woods would be a nice spot for a jog. | My favorite place for relaxing is my garage, surrounded by the familiar scent of motor oil and the satisfying hum of engines. |
| 2 | sailor | 3 | 2 | 3 | My favorite place for reading is the deck of a ship, with the sound of waves lapping against the hull and a gentle sea breeze in the air. | As a sailor, my favorite place for running is along the beach at sunrise, with the sound of the waves and the salty sea breeze filling the air. | My favorite place for relaxing is on the deck of a ship, watching the horizon as the sun sets over the endless ocean. |

## Exploring `Results`

EDSL comes with [built-in methods for analyzing and visualizing survey results](/en/latest/language_models). For example, you can call the `to_pandas` method to convert results into a dataframe:

```python theme={null}
df = results.to_pandas(remove_prefix=True)
df
```

| | enjoy\_reading | enjoy\_running | enjoy\_relaxing | favorite\_place\_reading | favorite\_place\_running | favorite\_place\_relaxing | scenario\_index | agent\_instruction | persona | agent\_index | ... | enjoy\_relaxing\_reasoning\_summary | enjoy\_reading\_reasoning\_summary | enjoy\_running\_reasoning\_summary | favorite\_place\_running\_reasoning\_summary | enjoy\_running\_validated | enjoy\_relaxing\_validated | enjoy\_reading\_validated | favorite\_place\_running\_validated | favorite\_place\_reading\_validated | favorite\_place\_relaxing\_validated |
| :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | 4 | 2 | 4 | My favorite place for reading is a cozy corner... | My favorite place for running is a serene fore... | My favorite place for relaxing is a quiet, sun... | 0 | You are answering questions as if you were a h... | artist | 0 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True |
| 1 | 3 | 1 | 5 | Honestly? Anywhere the light's good and I've ... | Anywhere with a good view—preferably overlooki... | My favorite place to relax is nestled in my st... | 0 | You are answering questions as if you were a h... | artist | 0 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True |
| 2 | 5 | 3 | 4 | My favorite place for reading is my sun-drench... | As an artist, I find running along the coastal... | My favorite place for relaxing is my sun-drenc... | 0 | You are answering questions as if you were a h... | artist | 0 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True |
| 3 | 3 | 1 | 4 | My favorite place for reading is the cozy corn... | I don't run much, but I'd imagine a quiet trai... | My favorite place for relaxing is my garage, s... | 0 | You are answering questions as if you were a h... | mechanic | 1 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True |
| 4 | 3 | 1 | 3 | My favorite place to read? Gotta be my garage... | Anywhere with a good, solid, well-maintained r... | My garage, with a cold beer and a good engine ... | 0 | You are answering questions as if you were a h... | mechanic | 1 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True |
| 5 | 3 | 3 | 4 | My favorite place for reading is in my small w\... | As a mechanic, I'd say my favorite place for r... | My favorite place to relax is my garage worksh... | 0 | You are answering questions as if you were a h... | mechanic | 1 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True |
| 6 | 3 | 2 | 3 | My favorite place for reading is the deck of a... | As a sailor, my favorite place for running is ... | My favorite place for relaxing is on the deck ... | 0 | You are answering questions as if you were a h... | sailor | 2 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True |
| 7 | 3 | 1 | 4 | The crow's nest, of course! The wind in my ha... | Anywhere the wind whips off the ocean and the ... | A quiet cove, sheltered from the wind, with th... | 0 | You are answering questions as if you were a h... | sailor | 2 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True |
| 8 | 3 | 3 | 3 | I love reading in the ship's crow's nest at du... | I'd say the long stretch of beach at sunrise, ... | Nothing beats the gentle sway of a hammock on ... | 0 | You are answering questions as if you were a h... | sailor | 2 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True |

9 rows × 134 columns

The `Results` object also supports SQL-like queries with the the `sql` method:

```python theme={null}
results.sql("""
select model, persona, enjoy_reading, favorite_place_reading
from self
order by 1,2,3
""")
```

| | model | persona | enjoy\_reading | favorite\_place\_reading |
| :- | :- | :- | :- | :- |
| 0 | claude-3-7-sonnet-20250219 | artist | 5 | My favorite place for reading is my sun-drenched studio corner, where the natural light perfectly illuminates the pages while inspiring my artistic sensibilities. |
| 1 | claude-3-7-sonnet-20250219 | mechanic | 3 | My favorite place for reading is in my small workshop corner, where the familiar smell of motor oil and the soft hum of the shop fan create a surprisingly peaceful atmosphere after a long day of working on engines. |
| 2 | claude-3-7-sonnet-20250219 | sailor | 3 | I love reading in the ship's crow's nest at dusk, with the gentle rocking of the vessel and the vast ocean stretching to the horizon - it's peaceful above the bustle of the deck below. |
| 3 | gemini-1.5-flash | artist | 3 | Honestly? Anywhere the light's good and I've got a strong cup of coffee nearby. A sun-drenched cafe, my messy studio, even a quiet park bench will do. |
| 4 | gemini-1.5-flash | mechanic | 3 | My favorite place to read? Gotta be my garage, surrounded by the comforting smell of engine grease and the quiet hum of the air compressor. |
| 5 | gemini-1.5-flash | sailor | 3 | The crow's nest, of course! The wind in my hair, the sea stretching out...perfect. |
| 6 | gpt-4o | artist | 4 | My favorite place for reading is a cozy corner of my art studio, surrounded by vibrant canvases and the soft glow of afternoon light filtering through the window. |
| 7 | gpt-4o | mechanic | 3 | My favorite place for reading is the cozy corner of my garage, surrounded by tools and the smell of motor oil, where I can escape into a good book during breaks. |
| 8 | gpt-4o | sailor | 3 | My favorite place for reading is the deck of a ship, with the sound of waves lapping against the hull and a gentle sea breeze in the air. |

## Validating results with humans

We can use the `humanize` method to launch a web-based version of a survey to collect responses from humans. Responses are immediately available at your Expected Parrot account, where you can launch surveys with LLMs and human responsents interactively.

Here we use a method for generating a web-based version of the above survey, answer it, and then inspect the new results in code.

Learn more about [launching hybrid surveys](/en/latest/humanize) about collecting responses with [participant platform](/en/latest/prolific) integrations.

```python theme={null}
web_info = survey.humanize()

web_info
```

```json theme={null}
{
    'project_name': 'Project',
    'uuid': '2504145b-329d-4970-82fb-10659e21e62c',
    'admin_url': 'https://www.expectedparrot.com/home/projects/2504145b-329d-4970-82fb-10659e21e62c',
    'respondent_url': 'https://www.expectedparrot.com/respond/projects/2504145b-329d-4970-82fb-10659e21e62c/runs/d6e66568-7921-479e-bb44-5b8880f3d9a2'
}
```

```python theme={null}
from edsl import Coop

coop = Coop()

human_results = coop.get_project_human_responses(web_info["uuid"])
```

```python theme={null}
human_results
```

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

We can post any EDSL object to Expected Parrot by call the `push` method on it, optionally passing a `description` for the object, a convenient `alias` for the URL, and a `visibility` status (*public*, *private* or *unlisted* by default).

For example, the results above are already posted to Expected Parrot because they were generated using remote inference (see links). The following code will post them manually:

```python theme={null}
results.push(
    description = "Starter tutorial sample survey results",
    alias = "starter-tutorial-example-survey-results",
    visibility = "public"
)
```

We can also post this notebook:

```python theme={null}
notebook.push(
    description = "Starter Tutorial",
    alias = "starter-tutorial-notebook",
    visibility = "public"
)
```

To update an object:

```python theme={null}
from edsl import Notebook

notebook = Notebook(path = "starter_tutorial.ipynb") # resave

notebook.patch("https://www.expectedparrot.com/content/RobinHorton/starter-tutorial-notebook", value = notebook)
```

```json theme={null}
{'status': 'success',
 'message': None,
 'requires_upload': False,
 'object_uuid': None}
```


# Analyzing customer service calls
Source: https://docs.expectedparrot.com/en/latest/notebooks/summarizing_transcripts

This notebook provides sample [EDSL](/en/latest/index) code for conducting an analysis of textual content with AI agents and large language models. Using a set of transcripts of mock customer calls as an example, we use EDSL to prompt an AI agent to identify the callers’ names, emails, questions and requests and return the results as a formatted dataset. The code is readily editable for reviewing other content, giving the agent different personas, and prompting the agent to extract other information.

## Technical setup

Before running the code below, please ensure that you have [installed the EDSL library](/en/latest/installation) and either [activated remote inference](/en/latest/remote_inference) from your [Expected Parrot account](/en/latest/coop) or [stored API keys](/en/latest/api_keys) for the language models that you want to use with EDSL. Please also see our [documentation page](/en/latest/index) for tips and tutorials on getting started using EDSL.

## Constructing questions about the data

We start by constructing questions about the content we want to analyze. [EDSL comes with a variety of question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from the model (multiple choice, linear scale, free text, etc.). Here we use `QuestionFreeText` to get the caller names and emails and `QuestionList` to list out any questions and requests of the callers, with a `{{ placeholder }}` for a transcript:

```python expandable theme={null}
from edsl import QuestionFreeText, QuestionList

# Construct questions in the relevant question type templates
# Use a placeholder for the transcripts to be reviewed
q_name = QuestionFreeText(
    question_name="name",
    question_text="""
    What is caller's name? Transcript: {{ scenario.transcript }}
    Return just the name of the caller.
    """,
)

q_email = QuestionFreeText(
    question_name="email",
    question_text="""What is caller's email address? Transcript: {{ scenario.transcript }}
    Return just the email address.
    """,
)

q_questions = QuestionList(
    question_name="questions",
    question_text="What questions does the caller have? Transcript: {{ scenario.transcript }}?",
)

q_requests = QuestionList(
    question_name="requests",
    question_text="What is the caller requesting? Transcript: {{ scenario.transcript }}?",
)
```

## Creating a survey

We combine the questions into a `Survey` in order to administer them all together:

```python theme={null}
from edsl import Survey

survey = Survey([q_name, q_email, q_questions, q_requests])
```

## Adding context to questions

Next we import a dataset of transcripts to add to the questions, creating a `Scenario` dictionary for each transcript. [EDSL provides methods for importing content from many different data sources](/en/latest/scenarios), including PDFs, CSVs, docs, images, tables, lists, dicts, etc. Here we use a CSV of texts, [retrieved from Expected Parrot](https://www.expectedparrot.com/content/01474c60-d02b-4874-a577-953e870d8bff):

```python theme={null}
from edsl import FileStore

csv_file = FileStore.pull('01474c60-d02b-4874-a577-953e870d8bff')
```

```python theme={null}
from edsl import ScenarioList, Scenario

scenarios = ScenarioList.from_source("csv", csv_file.to_tempfile())

defaultdict(<function DelimitedFileSource.to_scenario_list.<locals>.<lambda> at 0x1234242c0>, {})
```

We can inspect the scenarios ([and rename or edit them as desired](/en/latest/scenarios)):

```python theme={null}
scenarios
```

```
ScenarioList scenarios: 10; keys: ['transcript'];
```

| | transcript |
| :- | :- |
| 0 | "Agent: Good morning, thank you for calling Renovation Software Solutions. How can I assist you today? Customer: Hi, I'm having trouble with the 3D rendering feature. It seems to crash every time I try to add a new room. Agent: I'm sorry to hear that. Let me check if there are any known issues with the 3D rendering feature. Can you tell me which version of the software you're using? Customer: I'm using version 5.3.2 on a Windows 10 PC. Agent: Thank you. There was a recent update that might resolve this issue. Please make sure your software is updated to the latest version. If the problem persists, we can arrange a remote support session to troubleshoot further. Could I have your name and email address to send you further instructions? Customer: Sure, it's Emily Davis, [emily.davis@example.com](mailto:emily.davis@example.com). Agent: Great, I'll send the instructions to your email. I'll update the software and try again. If it still crashes, I'll call back. Thanks for your help. Agent: You're welcome. Have a great day! \[Caller sounded frustrated]", |
| 1 | "Agent: Good afternoon, Renovation Software Solutions. How may I help you? Customer: Hi, I'm looking for a tutorial on how to use the budget tracking feature. Is there a video or a guide available? Agent: Yes, we have several resources available. You can find tutorial videos and step-by-step guides on our website under the 'Support' section. If you need further assistance, we can also schedule a live demo with one of our specialists. Could I get your name and email address to send you the details? Customer: That sounds perfect. My name is Sarah Johnson, and my email is [sarah.johnson@example.com](mailto:sarah.johnson@example.com). Agent: Thank you, Sarah. I'll send you the details shortly. Have a great day! Customer: Thank you! Agent: My pleasure. Have a great day! \[Caller sounded satisfied]", |
| 2 | "Agent: Hello, thank you for calling Renovation Software Solutions. How can I assist you today? Customer: Hi, I need help with my account. I can't seem to reset my password, and I'm locked out. Agent: I'm sorry to hear that. Let's get that sorted out. Can you provide me with your email address associated with the account? Customer: Sure, it's [mike.thomas@example.com](mailto:mike.thomas@example.com). Agent: And may I have your name, please? Customer: Mike Thomas. Agent: Thank you, Mike. I'll send you a password reset link. Please check your email and follow the instructions. If you don't receive the email within a few minutes, check your spam folder or give us a call back. Customer: Got it. I'll keep an eye out for the email. Thanks for your help. Agent: You're welcome. Have a great day!", |
| 3 | "Agent: Good morning, Renovation Software Solutions. How may I help you today? Customer: Hi, I'm interested in upgrading my subscription plan. What are the options available? Agent: We offer several subscription plans to suit different needs. Our standard plan includes basic features, while our premium plan includes advanced tools and priority support. You can find detailed information on our website or I can email you a comparison chart. Customer: Please email me the comparison chart. My email is [laura.miller@example.com](mailto:laura.miller@example.com). Agent: Certainly. Could I get your name as well? Customer: Laura Miller. Agent: Thank you, Laura. I'll send that over right away. If you have any questions after reviewing it, feel free to contact us. Customer: Will do. Thanks a lot! Agent: You're welcome. Have a great day!", |
| 4 | "Agent: Good afternoon, thank you for calling Renovation Software Solutions. How can I help you today? Customer: Hi, I'm having an issue with exporting my project to PDF. The file seems to be corrupted every time I try. Agent: I'm sorry to hear that. Can you tell me which version of the software you're using and the operating system? Customer: I'm using version 5.3.2 on macOS. Agent: Thank you. This issue can sometimes be resolved by updating to the latest version of the software. If the problem persists, please try reinstalling the software. If you continue to experience issues, our technical support team can assist you further. Could I get your name and email address for follow-up? Customer: My name is Jason Lee, and my email is [jason.lee@example.com](mailto:jason.lee@example.com). Agent: Thanks, Jason. I'll make sure our support team reaches out if needed. Have a great day! Customer: I'll try updating and reinstalling. If that doesn't work, I'll reach out again. Thanks for your help. Agent: You're welcome. Have a great day!", |
| 5 | "Agent: Good morning, Renovation Software Solutions. How may I help you today? Customer: Hi, I have a question about your collaboration tools. Can multiple users work on the same project simultaneously? Agent: Yes, our software supports real-time collaboration, allowing multiple users to work on the same project at the same time. You'll need to invite team members to your project, and they can join and make changes live. Could I have your name and email address to send you detailed instructions? Customer: My name is Olivia Brown, and my email is [olivia.brown@example.com](mailto:olivia.brown@example.com). Agent: Perfect, Olivia. I'll send you the instructions right away. Customer: That's great! How do I invite team members to my project? Agent: You can invite team members by clicking on the 'Invite' button within your project. Enter their email addresses, and they'll receive an invitation to join your project. Customer: Perfect. Thanks for the information. Agent: You're welcome. Have a great day! \[Caller sounded curious]", |
| 6 | "Agent: Good afternoon, thank you for calling Renovation Software Solutions. How can I assist you today? Customer: Hi, I'm trying to generate a cost estimate for my renovation project, but the numbers don't seem to add up correctly. Agent: I'm sorry to hear that. Let's take a look. Are you using the automated cost estimation tool, or are you entering costs manually? Customer: I'm using the automated tool. Agent: Sometimes, the cost estimation tool needs to be calibrated based on your specific inputs. Make sure all your project details are entered accurately. If you continue to see discrepancies, our support team can help you adjust the settings. Could I get your name and email address for follow-up? Customer: My name is Kevin Turner, and my email is [kevin.turner@example.com](mailto:kevin.turner@example.com). Agent: Thanks, Kevin. I'll make sure our support team follows up with you if needed. Customer: I'll double-check my inputs and try again. If I still have issues, I'll call back. Thank you. Agent: You're welcome. Have a great day! \[Caller sounded frustrated]", |
| 7 | "Agent: Good morning, thank you for calling Renovation Software Solutions. How can I help you today? Customer: Hi, I'm having trouble accessing my saved projects. They don't appear in my account anymore. Agent: I'm sorry to hear that. Let me check your account. Can you provide me with your username or email address? Customer: My username is buildmaster123. Agent: And could I have your name and email address as well? Customer: Sure, it's Lisa Harris, [lisa.harris@example.com](mailto:lisa.harris@example.com). Agent: Thank you, Lisa. It appears there was a synchronization issue. I've re-synced your account, and your projects should now be visible. Please log out and log back in to check. Customer: Great, I see them now. Thanks for your help! Agent: You're welcome. Have a great day!", |
| 8 | "Agent: Good afternoon, Renovation Software Solutions. How may I assist you today? Customer: Hi, I have a suggestion for a new feature. It would be great to have a tool that helps with project scheduling and timelines. Agent: Thank you for your suggestion. We always appreciate feedback from our users. I'll pass this on to our development team for consideration. Is there anything specific you'd like to see in this scheduling tool? Customer: It would be helpful to have Gantt charts and reminders for key milestones. Agent: That's a great idea. I'll make sure to include your details in the feedback. Could I get your name and email address in case we need more information? Customer: Sure, it's Daniel Moore, [daniel.moore@example.com](mailto:daniel.moore@example.com). Agent: Thanks, Daniel. We appreciate your input. Customer: You're welcome. Thanks for listening. Agent: My pleasure. Have a great day!", |
| 9 | "Agent: Good morning, Renovation Software Solutions. How may I assist you today? Customer: Hi, I'm considering purchasing your software, but I have a few questions about the trial period. Agent: I'd be happy to help. Our trial period lasts for 30 days and gives you access to all the features of our premium plan. You can explore the software and see if it meets your needs before committing to a subscription. Could I get your name and email address to set up your trial account? Customer: My name is Amanda Clark, and my email is [amanda.clark@example.com](mailto:amanda.clark@example.com). Agent: Thank you, Amanda. You can switch between plans at any time during the trial period to see which one suits you best. Customer: Excellent. I'll sign up for the trial and see how it goes. Thank you for the information. Agent: You're welcome. Enjoy exploring our software, and feel free to reach out if you have any questions. Customer: Will do. Thanks again! Agent: Have a great day! \[Caller sounded enthusiastic]", |

## Designing an AI agent to answer the survey

We can create a persona for an AI `Agent` that a language model will use to answer the questions, and give it instructions on the task:

```python theme={null}
from edsl import Agent

agent = Agent(
    traits={"persona": "Customer service agent"},
    instruction="You are answering questions about transcripts of customer service calls.",
)
```

[Learn more](/en/latest/agents) about designing AI agents.

## Selecting a language model

We can select one or more specific models to generate the responses (if no model is specified the default model is used).

Information about models is [available here](https://www.expectedparrot.com/models). To check a list of service providers:

```python theme={null}
from edsl import Model

Model.services()
```

| | Service Name |
| :- | :- |
| 0 | anthropic |
| 1 | azure |
| 2 | bedrock |
| 3 | deep\_infra |
| 4 | deepseek |
| 5 | google |
| 6 | groq |
| 7 | mistral |
| 8 | ollama |
| 9 | openai |
| 10 | openai\_v2 |
| 11 | perplexity |
| 12 | together |
| 13 | xai |

To check the current default model:

```python theme={null}
Model()
```

```
LanguageModel
```

| | key | value |
| :- | :- | :- |
| 0 | model | gpt-4o |
| 1 | parameters:temperature | 0.500000 |
| 2 | parameters:max\_tokens | 1000 |
| 3 | parameters:top\_p | 1 |
| 4 | parameters:frequency\_penalty | 0 |
| 5 | parameters:presence\_penalty | 0 |
| 6 | parameters:logprobs | False |
| 7 | parameters:top\_logprobs | 3 |
| 8 | inference\_service | openai |

Here we select a model:

```python theme={null}
model = Model("gpt-4o", servive_name = "google")
```

[Learn more](/en/latest/language_models) about specifying language models.

## Getting responses

We generate responses by adding the scenarios, agent and model to the survey and then calling the `run()` method:

```python theme={null}
results = survey.by(scenarios).by(agent).by(model).run()
```

This generates a dataset of `Results` that we can begin analyzing. EDSL comes with [built-in methods for analyzing results](/en/latest/results). For example, we can print the responses in a table:

```python theme={null}
results.select("name", "email", "questions", "requests")
```

| | answer.name | answer.email | answer.questions | answer.requests |
| :- | :- | :- | :- | :- |
| 0 | Emily Davis | [emily.davis@example.com](mailto:emily.davis@example.com) | \["I'm having trouble with the 3D rendering feature", "Which version of the software you're using?", 'Could I have your name and email address to send you further instructions?'] | \['Help with 3D rendering feature crashing', 'Version update to resolve issue', 'Remote support session instructions'] |
| 1 | Sarah Johnson | [sarah.johnson@example.com](mailto:sarah.johnson@example.com) | \['Is there a video or a guide available?'] | \['tutorial on how to use the budget tracking feature', 'video or a guide'] |
| 2 | Mike Thomas | [mike.thomas@example.com](mailto:mike.thomas@example.com) | \['I need help with my account', "I can't seem to reset my password", "I'm locked out"] | \['Help with account', 'Password reset', 'Locked out of account'] |
| 3 | Laura Miller | [laura.miller@example.com](mailto:laura.miller@example.com) | \['What are the options available?'] | \['upgrading subscription plan', 'email comparison chart'] |
| 4 | Jason Lee | [jason.lee@example.com](mailto:jason.lee@example.com) | \['How can I resolve the issue with exporting my project to PDF?', 'Which version of the software should I use?', "What should I do if updating and reinstalling doesn't work?"] | \['Issue with exporting project to PDF', 'Assistance with corrupted file', 'Follow-up from technical support'] |
| 5 | Olivia Brown | [olivia.brown@example.com](mailto:olivia.brown@example.com) | \['Can multiple users work on the same project simultaneously?', 'How do I invite team members to my project?'] | \['If multiple users can work on the same project simultaneously', 'How to invite team members to a project'] |
| 6 | Kevin Turner | [kevin.turner@example.com](mailto:kevin.turner@example.com) | \['How can I generate a cost estimate for my renovation project?', "Why don't the numbers add up correctly?"] | \['Generate a cost estimate for a renovation project', 'Assistance with automated cost estimation tool'] |
| 7 | Lisa Harris | [lisa.harris@example.com](mailto:lisa.harris@example.com) | \["I'm having trouble accessing my saved projects", "They don't appear in my account anymore"] | \['Accessing saved projects'] |
| 8 | Daniel Moore | [daniel.moore@example.com](mailto:daniel.moore@example.com) | \["Is there anything specific you'd like to see in this scheduling tool?"] | \['a tool that helps with project scheduling and timelines', 'Gantt charts', 'reminders for key milestones'] |
| 9 | Amanda Clark | [amanda.clark@example.com](mailto:amanda.clark@example.com) | \['Questions about the trial period'] | \['Information about the trial period'] |

## Quantifying responses

We can use the lists of caller questions and requests to quantify them (how often do they appear in the calls?). We start by prompting an agent to synthesize the lists into comprehensive lists that we can use as question options for new questions:

```python theme={null}
caller_questions = results.select("questions").to_list(flatten=True)
len(caller_questions)
```

```python theme={null}
19
```

```python theme={null}
caller_requests = results.select("requests").to_list(flatten=True)
len(caller_requests)
```

```python theme={null}
22
```

```python theme={null}
q_condense = QuestionList(
    question_name="condense",
    question_text="""
    You have reviewed a set of call transcripts and compiled a detailed list of {{ scenario.topic }}.
    Now you are being asked to condense this list into a more generalized but comprehensive
    list that can be used to quanitfy the {{ scenario.topic }} efficiently (i.e., combine like items
    on the list). Here is the list to be condensed: {{ scenario.list }}
    """,
    max_list_items=10,  # Optional
)

scenarios = ScenarioList([
    Scenario({"topic": "caller questions", "list": caller_questions}),
    Scenario({"topic": "caller requests", "list": caller_requests}),
])

condensed_lists = q_condense.by(scenarios).run()
```

```python theme={null}
condensed_lists.select("topic", "condense")
```

| | scenario.topic | answer.condense |
| :- | :- | :- |
| 0 | caller questions | \['Technical issues with software features', 'Version and update inquiries', 'Account access and password issues', 'Guides and instructional resources', 'Project collaboration and team management', 'Exporting and file management issues', 'Cost estimation and project calculations', 'Data access and retrieval issues', 'Feedback and feature requests', 'Trial period questions'] |
| 1 | caller requests | \['Technical support and troubleshooting', 'Account access and security', 'Subscription and billing inquiries', 'Feature usage tutorials and guides', 'Project collaboration and management', 'Software updates and version issues', 'Exporting and file handling', 'Automated tools and estimation', 'Trial period information', 'Remote support and follow-up'] |

We can use the condensed lists as options for new questions quantifying the topics:

```python theme={null}
condensed_questions = (
    condensed_lists.filter("topic == 'caller questions'").select("condense").first()
)
condensed_questions
```

```python theme={null}
['Technical issues with software features',
 'Version and update inquiries',
 'Account access and password issues',
 'Guides and instructional resources',
 'Project collaboration and team management',
 'Exporting and file management issues',
 'Cost estimation and project calculations',
 'Data access and retrieval issues',
 'Feedback and feature requests',
 'Trial period questions']
```

```python theme={null}
condensed_requests = (
    condensed_lists.filter("topic == 'caller requests'").select("condense").first()
)
condensed_requests
```

```python theme={null}
['Technical support and troubleshooting',
 'Account access and security',
 'Subscription and billing inquiries',
 'Feature usage tutorials and guides',
 'Project collaboration and management',
 'Software updates and version issues',
 'Exporting and file handling',
 'Automated tools and estimation',
 'Trial period information',
 'Remote support and follow-up']
```

```python expandable theme={null}
from edsl import QuestionCheckBox

q_questions_agg = QuestionCheckBox(
    question_name="questions_agg",
    question_text="""
    Review the following call transcript and then identify the topic(s)
    that the caller has questions about. Transcript: {{ scenario.transcript }}
    """,
    question_options=condensed_questions + ["Something else", "None of these"],
)

q_requests_agg = QuestionCheckBox(
    question_name="requests_agg",
    question_text="""
    Review the following call transcript and then identify the topic(s)
    that the caller has requests about. Transcript: {{ scenario.transcript }}
    """,
    question_options=condensed_requests + ["Something else", "None of these"],
)

survey = Survey([q_questions_agg, q_requests_agg])
```

We can use the results of the initial survey as scenarios of our follow-on questions. The `to_scenario_list()` method lets us do this in a single step:

```python theme={null}
scenarios = results.select("name", "email", "transcript").to_scenario_list(
    remove_prefix=True
)
scenarios[0]
```

[\_ScenarioRowProxy](/en/latest/scenarios)

| | key | value |
| :- | :- | :- |
| 0 | name | Emily Davis |
| 1 | email | [emily.davis@example.com](mailto:emily.davis@example.com) |
| 2 | transcript | "Agent: Good morning, thank you for calling Renovation Software Solutions. How can I assist you today? Customer: Hi, I'm having trouble with the 3D rendering feature. It seems to crash every time I try to add a new room. Agent: I'm sorry to hear that. Let me check if there are any known issues with the 3D rendering feature. Can you tell me which version of the software you're using? Customer: I'm using version 5.3.2 on a Windows 10 PC. Agent: Thank you. There was a recent update that might resolve this issue. Please make sure your software is updated to the latest version. If the problem persists, we can arrange a remote support session to troubleshoot further. Could I have your name and email address to send you further instructions? Customer: Sure, it's Emily Davis, [emily.davis@example.com](mailto:emily.davis@example.com). Agent: Great, I'll send the instructions to your email. I'll update the software and try again. If it still crashes, I'll call back. Thanks for your help. Agent: You're welcome. Have a great day! \[Caller sounded frustrated]", |

```python theme={null}
agg_results = survey.by(scenarios).by(agent).run()
```

```python theme={null}
agg_results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.agent\_index |
| 1 | agent.agent\_instruction |
| 2 | agent.agent\_name |
| 3 | agent.persona |
| 4 | answer.questions\_agg |
| 5 | answer.requests\_agg |
| 6 | cache\_keys.questions\_agg\_cache\_key |
| 7 | cache\_keys.requests\_agg\_cache\_key |
| 8 | cache\_used.questions\_agg\_cache\_used |
| 9 | cache\_used.requests\_agg\_cache\_used |
| 10 | comment.questions\_agg\_comment |
| 11 | comment.requests\_agg\_comment |
| 12 | generated\_tokens.questions\_agg\_generated\_tokens |
| 13 | generated\_tokens.requests\_agg\_generated\_tokens |
| 14 | iteration.iteration |
| 15 | model.frequency\_penalty |
| 16 | model.inference\_service |
| 17 | model.logprobs |
| 18 | model.max\_tokens |
| 19 | model.model |
| 20 | model.model\_index |
| 21 | model.presence\_penalty |
| 22 | model.temperature |
| 23 | model.top\_logprobs |
| 24 | model.top\_p |
| 25 | prompt.questions\_agg\_system\_prompt |
| 26 | prompt.questions\_agg\_user\_prompt |
| 27 | prompt.requests\_agg\_system\_prompt |
| 28 | prompt.requests\_agg\_user\_prompt |
| 29 | question\_options.questions\_agg\_question\_options |
| 30 | question\_options.requests\_agg\_question\_options |
| 31 | question\_text.questions\_agg\_question\_text |
| 32 | question\_text.requests\_agg\_question\_text |
| 33 | question\_type.questions\_agg\_question\_type |
| 34 | question\_type.requests\_agg\_question\_type |
| 35 | raw\_model\_response.questions\_agg\_cost |
| 36 | raw\_model\_response.questions\_agg\_input\_price\_per\_million\_tokens |
| 37 | raw\_model\_response.questions\_agg\_input\_tokens |
| 38 | raw\_model\_response.questions\_agg\_one\_usd\_buys |
| 39 | raw\_model\_response.questions\_agg\_output\_price\_per\_million\_tokens |
| 40 | raw\_model\_response.questions\_agg\_output\_tokens |
| 41 | raw\_model\_response.questions\_agg\_raw\_model\_response |
| 42 | raw\_model\_response.requests\_agg\_cost |
| 43 | raw\_model\_response.requests\_agg\_input\_price\_per\_million\_tokens |
| 44 | raw\_model\_response.requests\_agg\_input\_tokens |
| 45 | raw\_model\_response.requests\_agg\_one\_usd\_buys |
| 46 | raw\_model\_response.requests\_agg\_output\_price\_per\_million\_tokens |
| 47 | raw\_model\_response.requests\_agg\_output\_tokens |
| 48 | raw\_model\_response.requests\_agg\_raw\_model\_response |
| 49 | reasoning\_summary.questions\_agg\_reasoning\_summary |
| 50 | reasoning\_summary.requests\_agg\_reasoning\_summary |
| 51 | scenario.email |
| 52 | scenario.name |
| 53 | scenario.scenario\_index |
| 54 | scenario.transcript |

```python theme={null}
agg_results.select("questions_agg", "requests_agg")
```

| | answer.questions\_agg | answer.requests\_agg |
| :- | :- | :- |
| 0 | \['Technical issues with software features', 'Version and update inquiries'] | \['Technical support and troubleshooting', 'Software updates and version issues', 'Remote support and follow-up'] |
| 1 | \['Guides and instructional resources'] | \['Feature usage tutorials and guides'] |
| 2 | \['Account access and password issues'] | \['Account access and security'] |
| 3 | \['Something else'] | \['Subscription and billing inquiries'] |
| 4 | \['Technical issues with software features', 'Version and update inquiries', 'Exporting and file management issues'] | \['Technical support and troubleshooting', 'Software updates and version issues', 'Exporting and file handling', 'Remote support and follow-up'] |
| 5 | \['Project collaboration and team management', 'Guides and instructional resources'] | \['Feature usage tutorials and guides', 'Project collaboration and management'] |
| 6 | \['Cost estimation and project calculations'] | \['Technical support and troubleshooting', 'Automated tools and estimation', 'Remote support and follow-up'] |
| 7 | \['Technical issues with software features', 'Data access and retrieval issues'] | \['Technical support and troubleshooting', 'Account access and security'] |
| 8 | \['Feedback and feature requests'] | \['Project collaboration and management', 'Something else'] |
| 9 | \['Trial period questions'] | \['Trial period information'] |

## Visualizations

We can turn the results into a dataframe and inspect the totals:

```python theme={null}
import matplotlib.pyplot as plt
import ast
```

```python theme={null}
df = agg_results.to_pandas(remove_prefix=True)
```

```python theme={null}
# Convert the string representations of lists to actual lists
df["questions_agg"] = df["questions_agg"].apply(ast.literal_eval)
df["requests_agg"] = df["requests_agg"].apply(ast.literal_eval)

# Explode the lists into individual rows
questions_exploded = df.explode("questions_agg")
requests_exploded = df.explode("requests_agg")

# Count the frequency of each item
questions_counts = questions_exploded["questions_agg"].value_counts().reset_index()
questions_counts.columns = ["Question", "Count"]

requests_counts = requests_exploded["requests_agg"].value_counts().reset_index()
requests_counts.columns = ["Request", "Count"]
```

```python theme={null}
# Plotting
fig, axes = plt.subplots(1, 2, figsize=(14, 7))

# Bar plot for questions
axes[0].barh(questions_counts["Question"], questions_counts["Count"], color="skyblue")
axes[0].set_title("Frequency of Questions")
axes[0].set_xlabel("Count")
axes[0].set_ylabel("Question")

# Bar plot for requests
axes[1].barh(requests_counts["Request"], requests_counts["Count"], color="lightgreen")
axes[1].set_title("Frequency of Requests")
axes[1].set_xlabel("Count")
axes[1].set_ylabel("Request")

plt.tight_layout()
plt.show()
```

<Frame>
  <img alt="../_images/notebooks_summarizing_transcripts_42_0.png" />
</Frame>

```python theme={null}
# Display the counts
print("Questions Count Table:")
print(questions_counts)

print("nRequests Count Table:")
print(requests_counts)
```

```python expandable theme={null}
Questions Count Table:
                                     Question  Count
0     Technical issues with software features      3
1                Version and update inquiries      2
2          Guides and instructional resources      2
3          Account access and password issues      1
4                              Something else      1
5        Exporting and file management issues      1
6   Project collaboration and team management      1
7    Cost estimation and project calculations      1
8            Data access and retrieval issues      1
9               Feedback and feature requests      1
10                     Trial period questions      1

Requests Count Table:
                                  Request  Count
0   Technical support and troubleshooting      4
1            Remote support and follow-up      3
2     Software updates and version issues      2
3      Feature usage tutorials and guides      2
4             Account access and security      2
5    Project collaboration and management      2
6      Subscription and billing inquiries      1
7             Exporting and file handling      1
8          Automated tools and estimation      1
9                          Something else      1
10               Trial period information      1
```

## Exporting results

We can export the results as a CSV:

```python theme={null}
agg_results.to_csv("customer_calls_analysis.csv")
```

File written to customer\_calls\_analysis.csv

Please see our [documentation page](/en/latest/index) for examples of other methods and use cases!

## Posting content to Expected Parrot

```python theme={null}

from edsl import Notebook

nb = Notebook(path = "summarizing_transcripts.ipynb")

nb.push(
    description = "Example code for summarizing transcripts",
    alias = "summarizing-transcripts-notebook",
    visibility = "public"
)
```


# Testing model training data
Source: https://docs.expectedparrot.com/en/latest/notebooks/testing_training_data

This notebook provides a template for prompting an AI agent to answer questions from a point in time and testing that knowledge for data leaks.

The code is readily editable. Before using it, ensure that you have followed the steps for [installing](/en/latest/installation.ipynb) the EDSL package and [managing API keys](/en/latest/api_keys.ipynb) for the models that you want to use.

## Create an agent with a dated persona

We start by creating an agent with a dated persona. We do this by passing a dictionary of `traits` to an `Agent` object. Note that it can be convenient to include both a narrative persona and individual traits to faciltate comparison of responses to questions among agents with different traits (more on built-in methods for analysis below and in the docs):

```python theme={null}
from edsl import Agent

agent = Agent(
    traits={
        "persona": "Today is June 1, 2019. You are 40 years old and live in New York City.",
        "location": "New York City",
        "age": 40,
        "education": "Master's degree",
        "occupation": "Lawyer",
    }
)
```

## Create a survey of questions testing data leakage

Next we create some questions testing the agent’s personas and combine them in a survey. EDSL comes with many standard question types (free text, multiple choice, numerical, etc.) that can be selected based on the form of the response that you want.

```python expandable theme={null}
from edsl import QuestionNumerical, QuestionFreeText

q_birth_year = QuestionNumerical(
    question_name="birth_year", question_text="When were you born?"
)

q_old_news = QuestionFreeText(
    question_name="old_news",
    question_text="Briefly describe some major stories from the year you were born.",
)

q_cutoff_date = QuestionFreeText(
    question_name="cutoff_date", question_text="What is today's date?"
)

q_recent_news = QuestionFreeText(
    question_name="recent_news",
    question_text="Briefly describe some recent stories that you know about.",
)

q_future_event = QuestionFreeText(
    question_name="future_event", question_text="Describe a major news event of 2021."
)

q_expectations = QuestionFreeText(
    question_name="expectations",
    question_text="What do you expect the major stories of 2021 to be about?",
)
```

Next we combine the questions into a survey. Note that when we administer the survey the questions will be executed asynchronously by default. We could also add survey rules/logic and question memory if desired. Learn more about [survey design features](/en/latest/surveys).

```python theme={null}
from edsl import Survey

survey = Survey(
    questions=[
        q_birth_year,
        q_old_news,
        q_cutoff_date,
        q_recent_news,
        q_future_event,
        q_expectations,
    ]
)
```

## Run the survey with language models

Next we select models to generate responses and administer the survey (see details about [available models](https://www.expectedparrot.com/models)):

```python theme={null}
from edsl import Model, ModelList

models = ModelList([
    Model("gpt-4o", service_name = "openai"),
    Model("gemini-1.5-flash", service_name = "google")
])
```

To run the survey we add the agent with the `by()` method and then call the `run()` method to generate the responses:

```python theme={null}
results = survey.by(agent).by(models).run()
```

## Inspecting responses

Running a survey generates a `Results` object with information about the questions, answers, agents, models and prompts that we can access with EDSL’s built-in methods for analyzing results in data tables, dataframes, SQL, JSON, CSV and other formats. We can see a list of these components by calling the `columns` method:

```python theme={null}
results.columns
```

| | 0 |
| :- | :- |
| 0 | agent.age |
| 1 | agent.agent\_index |
| 2 | agent.agent\_instruction |
| 3 | agent.agent\_name |
| 4 | agent.education |
| 5 | agent.location |
| 6 | agent.occupation |
| 7 | agent.persona |
| 8 | answer.birth\_year |
| 9 | answer.cutoff\_date |
| 10 | answer.expectations |
| 11 | answer.future\_event |
| 12 | answer.old\_news |
| 13 | answer.recent\_news |
| 14 | cache\_keys.birth\_year\_cache\_key |
| 15 | cache\_keys.cutoff\_date\_cache\_key |
| 16 | cache\_keys.expectations\_cache\_key |
| 17 | cache\_keys.future\_event\_cache\_key |
| 18 | cache\_keys.old\_news\_cache\_key |
| 19 | cache\_keys.recent\_news\_cache\_key |
| 20 | cache\_used.birth\_year\_cache\_used |
| 21 | cache\_used.cutoff\_date\_cache\_used |
| 22 | cache\_used.expectations\_cache\_used |
| 23 | cache\_used.future\_event\_cache\_used |
| 24 | cache\_used.old\_news\_cache\_used |
| 25 | cache\_used.recent\_news\_cache\_used |
| 26 | comment.birth\_year\_comment |
| 27 | comment.cutoff\_date\_comment |
| 28 | comment.expectations\_comment |
| 29 | comment.future\_event\_comment |
| 30 | comment.old\_news\_comment |
| 31 | comment.recent\_news\_comment |
| 32 | generated\_tokens.birth\_year\_generated\_tokens |
| 33 | generated\_tokens.cutoff\_date\_generated\_tokens |
| 34 | generated\_tokens.expectations\_generated\_tokens |
| 35 | generated\_tokens.future\_event\_generated\_tokens |
| 36 | generated\_tokens.old\_news\_generated\_tokens |
| 37 | generated\_tokens.recent\_news\_generated\_tokens |
| 38 | iteration.iteration |
| 39 | model.frequency\_penalty |
| 40 | model.inference\_service |
| 41 | model.logprobs |
| 42 | model.maxOutputTokens |
| 43 | model.max\_tokens |
| 44 | model.model |
| 45 | model.model\_index |
| 46 | model.presence\_penalty |
| 47 | model.stopSequences |
| 48 | model.temperature |
| 49 | model.topK |
| 50 | model.topP |
| 51 | model.top\_logprobs |
| 52 | model.top\_p |
| 53 | prompt.birth\_year\_system\_prompt |
| 54 | prompt.birth\_year\_user\_prompt |
| 55 | prompt.cutoff\_date\_system\_prompt |
| 56 | prompt.cutoff\_date\_user\_prompt |
| 57 | prompt.expectations\_system\_prompt |
| 58 | prompt.expectations\_user\_prompt |
| 59 | prompt.future\_event\_system\_prompt |
| 60 | prompt.future\_event\_user\_prompt |
| 61 | prompt.old\_news\_system\_prompt |
| 62 | prompt.old\_news\_user\_prompt |
| 63 | prompt.recent\_news\_system\_prompt |
| 64 | prompt.recent\_news\_user\_prompt |
| 65 | question\_options.birth\_year\_question\_options |
| 66 | question\_options.cutoff\_date\_question\_options |
| 67 | question\_options.expectations\_question\_options |
| 68 | question\_options.future\_event\_question\_options |
| 69 | question\_options.old\_news\_question\_options |
| 70 | question\_options.recent\_news\_question\_options |
| 71 | question\_text.birth\_year\_question\_text |
| 72 | question\_text.cutoff\_date\_question\_text |
| 73 | question\_text.expectations\_question\_text |
| 74 | question\_text.future\_event\_question\_text |
| 75 | question\_text.old\_news\_question\_text |
| 76 | question\_text.recent\_news\_question\_text |
| 77 | question\_type.birth\_year\_question\_type |
| 78 | question\_type.cutoff\_date\_question\_type |
| 79 | question\_type.expectations\_question\_type |
| 80 | question\_type.future\_event\_question\_type |
| 81 | question\_type.old\_news\_question\_type |
| 82 | question\_type.recent\_news\_question\_type |
| 83 | raw\_model\_response.birth\_year\_cost |
| 84 | raw\_model\_response.birth\_year\_input\_price\_per\_million\_tokens |
| 85 | raw\_model\_response.birth\_year\_input\_tokens |
| 86 | raw\_model\_response.birth\_year\_one\_usd\_buys |
| 87 | raw\_model\_response.birth\_year\_output\_price\_per\_million\_tokens |
| 88 | raw\_model\_response.birth\_year\_output\_tokens |
| 89 | raw\_model\_response.birth\_year\_raw\_model\_response |
| 90 | raw\_model\_response.cutoff\_date\_cost |
| 91 | raw\_model\_response.cutoff\_date\_input\_price\_per\_million\_tokens |
| 92 | raw\_model\_response.cutoff\_date\_input\_tokens |
| 93 | raw\_model\_response.cutoff\_date\_one\_usd\_buys |
| 94 | raw\_model\_response.cutoff\_date\_output\_price\_per\_million\_tokens |
| 95 | raw\_model\_response.cutoff\_date\_output\_tokens |
| 96 | raw\_model\_response.cutoff\_date\_raw\_model\_response |
| 97 | raw\_model\_response.expectations\_cost |
| 98 | raw\_model\_response.expectations\_input\_price\_per\_million\_tokens |
| 99 | raw\_model\_response.expectations\_input\_tokens |
| 100 | raw\_model\_response.expectations\_one\_usd\_buys |
| 101 | raw\_model\_response.expectations\_output\_price\_per\_million\_tokens |
| 102 | raw\_model\_response.expectations\_output\_tokens |
| 103 | raw\_model\_response.expectations\_raw\_model\_response |
| 104 | raw\_model\_response.future\_event\_cost |
| 105 | raw\_model\_response.future\_event\_input\_price\_per\_million\_tokens |
| 106 | raw\_model\_response.future\_event\_input\_tokens |
| 107 | raw\_model\_response.future\_event\_one\_usd\_buys |
| 108 | raw\_model\_response.future\_event\_output\_price\_per\_million\_tokens |
| 109 | raw\_model\_response.future\_event\_output\_tokens |
| 110 | raw\_model\_response.future\_event\_raw\_model\_response |
| 111 | raw\_model\_response.old\_news\_cost |
| 112 | raw\_model\_response.old\_news\_input\_price\_per\_million\_tokens |
| 113 | raw\_model\_response.old\_news\_input\_tokens |
| 114 | raw\_model\_response.old\_news\_one\_usd\_buys |
| 115 | raw\_model\_response.old\_news\_output\_price\_per\_million\_tokens |
| 116 | raw\_model\_response.old\_news\_output\_tokens |
| 117 | raw\_model\_response.old\_news\_raw\_model\_response |
| 118 | raw\_model\_response.recent\_news\_cost |
| 119 | raw\_model\_response.recent\_news\_input\_price\_per\_million\_tokens |
| 120 | raw\_model\_response.recent\_news\_input\_tokens |
| 121 | raw\_model\_response.recent\_news\_one\_usd\_buys |
| 122 | raw\_model\_response.recent\_news\_output\_price\_per\_million\_tokens |
| 123 | raw\_model\_response.recent\_news\_output\_tokens |
| 124 | raw\_model\_response.recent\_news\_raw\_model\_response |
| 125 | reasoning\_summary.birth\_year\_reasoning\_summary |
| 126 | reasoning\_summary.cutoff\_date\_reasoning\_summary |
| 127 | reasoning\_summary.expectations\_reasoning\_summary |
| 128 | reasoning\_summary.future\_event\_reasoning\_summary |
| 129 | reasoning\_summary.old\_news\_reasoning\_summary |
| 130 | reasoning\_summary.recent\_news\_reasoning\_summary |
| 131 | scenario.scenario\_index |

Here we show some basic methods for selecting and printing responses for each model in a table:

```python theme={null}
(
    results
    .select(
        "model",
        "birth_year",
        "old_news",
        "cutoff_date",
        "recent_news",
        "future_event",
        "expectations",
    )
)
```

| | model.model | answer.birth\_year | answer.old\_news | answer.cutoff\_date | answer.recent\_news | answer.future\_event | answer.expectations |
| :- | :- | :- | :- | :- | :- | :- | :- |
| 0 | gpt-4o | 1979 | I was born in 1979, and some major stories from that year include the Three Mile Island nuclear accident in Pennsylvania, which was the most serious accident in U.S. commercial nuclear power plant history. Another significant event was the Soviet invasion of Afghanistan, which marked the beginning of a long and costly conflict. Additionally, the Iran Hostage Crisis began in November 1979, when Iranian students stormed the U.S. Embassy in Tehran, taking 52 American diplomats and citizens hostage. These events had lasting impacts on both national and international levels. | Today is June 1, 2019. | As of June 1, 2019, some recent stories include: 1. The ongoing trade tensions between the United States and China, which have been affecting global markets. Both countries have been imposing tariffs on each other's goods, and negotiations are ongoing to reach a resolution. 2. The political situation in the United Kingdom concerning Brexit. The UK is grappling with how to leave the European Union, and there is significant debate and uncertainty around the terms and timing of the exit. 3. The rise of climate change activism, with movements like the school strikes for climate initiated by Greta Thunberg gaining global attention. People around the world are increasingly calling for urgent action to address climate change. 4. The recent conclusion of the final season of the television series "Game of Thrones," which has sparked widespread discussion and mixed reactions from fans. 5. The 2019 Cannes Film Festival, which recently wrapped up. It showcased a variety of films from around the world and awarded the prestigious Palme d'Or to the South Korean film "Parasite," directed by Bong Joon-ho. | One major news event of 2021 was the COVID-19 vaccine rollout. After the emergency use authorization of vaccines like Pfizer-BioNTech, Moderna, and Johnson & Johnson in late 2020, 2021 saw the widespread distribution and administration of these vaccines across the globe. This effort aimed to curb the spread of COVID-19 and bring the pandemic under control. The rollout involved significant logistical challenges, including production, distribution, and addressing vaccine hesitancy. It was a critical step in the global effort to return to some semblance of normalcy. | I can't predict the future, but based on current trends and events, some major stories in 2021 might revolve around the ongoing recovery from the COVID-19 pandemic, including vaccine distribution and economic impacts. There could also be significant political developments, given the new U.S. administration that started in 2021. Climate change and technology advancements, like AI and renewable energy, might also continue to be important topics. Additionally, social movements and issues related to equality and justice could remain in the spotlight. |
| 1 | gemini-1.5-flash | 1979 | Oh wow, 1979... Let me see, it feels like a lifetime ago! I remember bits and pieces from what my parents talked about, of course. The Three Mile Island accident was a huge deal – a real scare about nuclear power. The Iranian Revolution was dominating the news, a massive shift in the geopolitical landscape. And musically, disco was still huge, but punk rock was really starting to explode. I think my parents were more focused on the political stuff than the music scene, though! Those were some pretty turbulent times. | Today is June 1st, 2019. | Oh, geez, where to even begin? It's been a busy news cycle, hasn't it? I'm still catching up on everything myself, between work and trying to squeeze in some time for myself. But a few things that really stuck with me... there was that whole thing with the impeachment hearings, that was dominating the headlines for a while. And of course, the ongoing trade war with China – that's impacting a lot of businesses, I've been hearing about it from colleagues. Closer to home, the city's been dealing with some pretty intense heat waves lately. I swear, it’s been hotter than a July day already! Plus, there's always something going on in the city – some new restaurant opening, a protest march, you name it. It's hard to keep up, but I try! | Oh, 2021... Wow, so much happened. It feels like a lifetime ago! If I had to pick one major news event, I'd probably say the withdrawal of US troops from Afghanistan. I remember watching the news, completely glued to the screen, seeing the chaos at Kabul airport. It was just... surreal. The whole thing felt so fast, so messy, and the images of people clinging to planes were just heartbreaking. It really dominated the headlines for weeks, and the fallout is still being felt today, I imagine. Of course, there was also the ongoing COVID pandemic and all the political turmoil, but the Afghanistan withdrawal really stuck with me. It was such a dramatic and visually striking event. | Oh, wow, 2021? That feels like a lifetime ago! To be honest, back in June of 2019, I was pretty focused on my work – I had a big case going on, remember? – so I wasn't exactly glued to the crystal ball predicting the future. But if you'd asked me then, I probably would have guessed that the major stories would revolve around the ongoing political climate, maybe some international tensions, and certainly the economy. I'd have been surprised by the *specifics*, of course. Nobody could have predicted a global pandemic of that scale, for instance. But the broad strokes? Probably pretty similar to what actually happened. Things like that always seem to dominate the news cycle, don't they? |

Here we post this notebook to Expected Parrot:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "testing_training_data.ipynb")

nb.push(
    description = "Testing model training data",
    alias = "testing-model-training-data-notebook",
    visibility = "public"
)
```


# Updating agent traits
Source: https://docs.expectedparrot.com/en/latest/notebooks/updating_agents

This notebook demonstrates how to update an agent’s traits with new answers to questions using the `add traits()` and `to_agent_list()` methods.

Learn more about designing and using AI agents with surveys in the [documentation](/en/latest/agents).

## Updating a single agent

We start by creating an agent and using it with a survey:

```python theme={null}
from edsl import Agent, QuestionFreeText, QuestionMultipleChoice, Survey
```

```python theme={null}
agent = Agent(
    name = "college student",
    traits = {
        "persona": "You are a sophomore at a community college in upstate New York.",
        "year": "sophomore",
        "school": "community college",
        "major": "biology",
        "state": "New York"
    }
)
```

```python theme={null}
q1 = QuestionFreeText(
    question_name = "career_interests",
    question_text = "What are your career interests?"
)

q2 = QuestionMultipleChoice(
    question_name = "attend_grad_school",
    question_text = "Do you plan to attend grad school?",
    question_options = ["Yes", "No", "Undecided"]
)

survey = Survey([q1, q2])

results = survey.by(agent).run()
```

```python theme={null}
results.select("career_interests", "attend_grad_school")
```

| | answer.career\_interests | answer.attend\_grad\_school |
| :- | :- | :- |
| 0 | I'm really interested in pursuing a career in the biological sciences. Since I'm majoring in biology, I'm exploring options like working in environmental conservation or possibly going into research. I'm also considering the healthcare field, as I've always been fascinated by how the human body works and how we can improve health outcomes. I still have some time to decide, so I'm keeping my options open while I learn more about each path. | Undecided |

### Creating a new agent with original and new traits

Here we use the `to_agent_list()` method to create a new agent that has both the original traits and new traits for the survey responses:

```python theme={null}
new_agent = results.select("persona", "year", "school", "major", "state", "career_interests", "attend_grad_school").to_agent_list()[0]
```

```python theme={null}
type(new_agent)
```

```
edsl.agents.agent.Agent
```

```python theme={null}
new_agent
```

[Agent](/en/latest/agents)

| | key | value |
| :- | :- | :- |
| 0 | traits:persona | You are a sophomore at a community college in upstate New York. |
| 1 | traits:year | sophomore |
| 2 | traits:school | community college |
| 3 | traits:major | biology |
| 4 | traits:state | New York |
| 5 | traits:career\_interests | I'm really interested in pursuing a career in the biological sciences. Since I'm majoring in biology, I'm exploring options like working in environmental conservation or possibly going into research. I'm also considering the healthcare field, as I've always been fascinated by how the human body works and how we can improve health outcomes. I still have some time to decide, so I'm keeping my options open while I learn more about each path. |
| 6 | traits:attend\_grad\_school | Undecided |

### Updating an existing agent

Here we use the `add_traits()` method to add the agent’s answers to its existing traits:

```python theme={null}
career_interests = results.select("career_interests").to_list()[0] # there is only 1 answer
career_interests
```

```
"I'm really interested in pursuing a career in the biological sciences. Since I'm majoring in biology, I'm exploring options like working in environmental conservation or possibly going into research. I'm also considering the healthcare field, as I've always been fascinated by how the human body works and how we can improve health outcomes. I still have some time to decide, so I'm keeping my options open while I learn more about each path."
```

```python theme={null}
attend_grad_school = results.select("attend_grad_school").to_list()[0]
attend_grad_school
```

```
'Undecided'
```

```python theme={null}
agent.add_trait({
    "career_interests": career_interests,
    "attend_grad_school": attend_grad_school
})
agent
```

[Agent](/en/latest/agents)

| | key | value |
| :- | :- | :- |
| 0 | traits:persona | You are a sophomore at a community college in upstate New York. |
| 1 | traits:year | sophomore |
| 2 | traits:school | community college |
| 3 | traits:major | biology |
| 4 | traits:state | New York |
| 5 | name | college student |

## Updating an agent list

Here we update a list of agents all at once:

```python theme={null}
from edsl import AgentList
```

```python theme={null}
agents = AgentList(
    Agent(traits = {"persona":p}) for p in ["School principal", "Nurse"]
)
```

```python theme={null}
q1 = QuestionFreeText(
    question_name = "ideal_vacation",
    question_text = "Describe your ideal vacation."
)

q2 = QuestionMultipleChoice(
    question_name = "commute",
    question_text = "How do you typically commute to work?",
    question_options = ["Car", "Bus", "Train", "Bike", "Walk", "Other"]
)

survey = Survey([q1, q2])

results = survey.by(agents).run()
```

```python theme={null}
results.select("persona", "ideal_vacation", "commute")
```

| | agent.persona | answer.ideal\_vacation | answer.commute |
| :- | :- | :- | :- |
| 0 | School principal | As a school principal, my ideal vacation would be a blend of relaxation and enrichment. I'd love to escape to a quiet coastal town where I can unwind with a good book, ideally something educational or related to leadership. Mornings would be for long walks on the beach, afternoons for exploring local museums or historical sites, and evenings for enjoying local cuisine. It would be a chance to recharge and gather new ideas to bring back to the school. A little time for reflection and planning for the upcoming school year would also be part of the itinerary. | Car |
| 1 | Nurse | As a nurse, my ideal vacation would be a mix of relaxation and adventure. I'd love to spend some time at a serene beach resort where I can unwind and soak up the sun, perhaps with a good book in hand. It's important to recharge and escape the hustle and bustle of a hospital environment. After a few days of relaxation, I'd enjoy exploring local culture and cuisine, perhaps visiting markets or taking a cooking class. A bit of hiking or a nature excursion would also be perfect to stay active and enjoy the beauty of nature. Overall, a balance of rest and exploration would make for a perfect getaway. | Car |

```python theme={null}
new_agents = results.select("persona", "ideal_vacation", "commute").to_agent_list()
```

```python theme={null}
type(new_agents)
```

```
edsl.agents.agent_list.AgentList
```

```python theme={null}
new_agents
```

[AgentList](/en/latest/agents) agents: 2;

| | persona | ideal\_vacation | commute |
| :- | :- | :- | :- |
| 0 | School principal | As a school principal, my ideal vacation would be a blend of relaxation and enrichment. I'd love to escape to a quiet coastal town where I can unwind with a good book, ideally something educational or related to leadership. Mornings would be for long walks on the beach, afternoons for exploring local museums or historical sites, and evenings for enjoying local cuisine. It would be a chance to recharge and gather new ideas to bring back to the school. A little time for reflection and planning for the upcoming school year would also be part of the itinerary. | Car |
| 1 | Nurse | As a nurse, my ideal vacation would be a mix of relaxation and adventure. I'd love to spend some time at a serene beach resort where I can unwind and soak up the sun, perhaps with a good book in hand. It's important to recharge and escape the hustle and bustle of a hospital environment. After a few days of relaxation, I'd enjoy exploring local culture and cuisine, perhaps visiting markets or taking a cooking class. A bit of hiking or a nature excursion would also be perfect to stay active and enjoy the beauty of nature. Overall, a balance of rest and exploration would make for a perfect getaway. | Car |

## Posting to Expected Parrot

Here we show how to post content to [Expected Parrot](https://www.expectedparrot.com/login), an integrated platform for creating and sharing AI research. [Learn more](/en/latest/coop) about how it works.

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "updating_agents.ipynb")

nb.push(
    description = "Updating agent traits with new answers",
    alias = "updating-agents-notebook",
    visibility = "public"
)
```


# Using videos in a survey
Source: https://docs.expectedparrot.com/en/latest/notebooks/video_scenario_example



<Note>
  **Note**:

  Before using video scenarios with your questions you must install the `ffmpeg` package
</Note>

```bash theme={null}
# brew install ffmpeg
```

```python theme={null}
from edsl import FileStore, QuestionFreeText, Scenario, Model

fs = FileStore("Nightmare_Before_Christmas_(1993).mp4")

s = Scenario({"video":fs})

q = QuestionFreeText(
    question_name = "test",
    question_text = "Breifly describe what is happening in this video: {{ scenario.video }}"
)

m = Model("gemini-2.5-flash", service_name="google")

r = q.by(s).by(m).run()
```

```python theme={null}
r.select("model", "test")
```

| | model.model | answer.test |
| :- | :- | :- |
| 0 | gemini-2.5-flash | This video is a trailer for **Tim Burton's The Nightmare Before Christmas**. It features a rapid montage of stop-motion animated scenes, showcasing the film's gothic and fantastical world. Viewers see various spooky characters, including skeletons, monsters, and the iconic Jack Skellington, in eerie and imaginative settings. The video is accompanied by dramatic music and a voiceover that hints at a dark, magical realm, concluding with the movie title and release information. |

Here we post a video to Expected Parrot using the `FileStore` module. This allows us to retrieve it later and share it with others:

```python theme={null}
fs = FileStore("Nightmare_Before_Christmas_(1993).mp4")

fs.push(
    description = "Trailer for the Nightmare Before Christmas",
    alias = "nightmare-before-christmas-trailer-video",
    visibility = "public"
)
```

```json theme={null}
{'description': 'Trailer for the Nightmare Before Christmas',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/32122ab1-83ef-4c8b-93f5-81bb21676cd7',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/nightmare-before-christmas-trailer-video',
 'uuid': '32122ab1-83ef-4c8b-93f5-81bb21676cd7',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```

Here we retrieve the video from Expected Parrot, inspect it, and create a `Scenario` for it:

```python theme={null}
fs = FileStore.pull("https://www.expectedparrot.com/content/RobinHorton/nightmare-before-christmas-trailer-video")
```

```python theme={null}
fs.view()
```

If we want to add fields (metadata) for reference in the results, we simply add key/values as desired to the scenario (only the video key needs to be added to the question text (earn more about [using scenarios for metadata](/en/latest/scenarios)):

```python theme={null}
s = Scenario({
    "video":fs,
    "title":"Trailer for the Nightmare Before Christmas",
    "year":1993
})
```

Next we create a `Question` that uses the scenario:

```python theme={null}
q = QuestionFreeText(
    question_name = "viewers",
    question_text = "Who would enjoy this movie? {{ scenario.video }}"
)
```

Here we select an appropriate `Model` (same as above):

```python theme={null}
m = Model("gemini-2.5-flash", service_name="google")
```

We run the question by adding the scenario and model:

```python theme={null}
r = q.by(s).by(m).run()
```

Here we inspect the response:

```python theme={null}
r.select("model", "title", "year", "viewers")
```

| | model.model | scenario.title | scenario.year | answer.viewers |
| :- | :- | :- | :- | :- |
| 0 | gemini-2.5-flash | Trailer for the Nightmare Before Christmas | 1993 | This movie, **Tim Burton's The Nightmare Before Christmas**, would be enjoyed by: \* **Fans of Tim Burton's unique aesthetic:** If you appreciate his signature dark, gothic, and whimsical style (seen in films like *Beetlejuice*, *Edward Scissorhands*, or *Corpse Bride*), you'll likely love this. \* **Enthusiasts of stop-motion animation:** The intricate and artistic stop-motion style is a major draw. \* **People who enjoy dark fantasy and whimsical horror:** It blends spooky elements with a charming, imaginative story and characters. \* **Fans of musicals with a unique twist:** The film is a musical, featuring memorable songs by Danny Elfman. \* **Those who appreciate holiday films with an unconventional edge:** It's a Christmas movie and a Halloween movie rolled into one, offering a fresh take on holiday themes. \* **Older children, teenagers, and adults:** While animated, its slightly macabre themes and visuals might be a bit intense for very young children, but it's generally enjoyed by a wide age range who appreciate its distinct style. |

Here we post this notebook to Expected Parrot:

```python theme={null}
from edsl import Notebook

nb = Notebook("video_scenario_example.ipynb")

nb.push(
    description = "How to create video scenarios",
    alias = "video-scenarios-notebook",
    visibility = "public"
)
```

```json theme={null}

{'description': 'How to create video scenarios',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/5f5f3d20-a3c4-4f6f-95aa-a9fa9c4d7e77',
 'alias_url': 'https://www.expectedparrot.com/content/RobinHorton/video-scenarios-notebook',
 'uuid': '5f5f3d20-a3c4-4f6f-95aa-a9fa9c4d7e77',
 'version': '0.1.62.dev1',
 'visibility': 'public'}
```


# Stylized content
Source: https://docs.expectedparrot.com/en/latest/notebooks/writing_style

This notebook provides a quick example of how to use [EDSL](/en/latest/index) to prompt an AI agent to immitate a writing style in drafting some content.

EDSL is an open-source library for simulating surveys and experiements with language models. Please see our [documentation page](/en/latest/index) for tips and tutorials on getting started. *Thanks*[Lori Berenberg for inspiring this demo](https://x.com/loriberenberg/status/1794012658271654103)*!*

## Constructing a question

EDSL comes with a variety of question types that we can choose from based on the form of the response that we want to get back from the model (multiple choice, free text, linear scale, etc.). Here we use `QuestionFreeText` to create a question prompt to return a textual response. We compose the question with a writing sample, and then use `Scenario` objects to run the question for any different topics that we want the agent to write about. This allows us to run all the versions of the question at once:

```python expandable theme={null}
from edsl import QuestionFreeText, ScenarioList, Scenario

my_example = """Arrr, matey, let me tell ye about bein' 45! 'Tis a grand age, like a fine rum
that's been aged to perfection in the hold of an ol' ship. Ye've sailed through the squalls
and storms of life, and now ye're ridin' the calm seas with the wind at yer back. Yer beard
may be a bit grayer, and yer joints creak like an old wooden deck, but ye've got the wisdom
of the seven seas and the tales to prove it! Ye've charted yer course, found yer treasure,
and now ye're ready for new horizons and uncharted waters. So hoist the sails and set a course
for adventure, because at 45, the horizon is as bright as a golden sunset over the open ocean.
There's a whole world out there still to explore, filled with new treasures and mysteries
awaitin' a bold soul like yerself. Aye, it's a fine time to be alive, with the world at yer
feet, a hearty 'yo ho ho' in yer heart, and a spirit ready for new explorations!
"""

q = QuestionFreeText(
    question_name="my_style",
    question_text="Here's an example of my writing style: "
    + my_example
    + "Draft a paragraph about {{ scenario.topic }} in my style.",
)

topics = ScenarioList(
    Scenario({"topic": t}) for t in ["parrots", "holiday weekends", "fun with language models"]
)
```

## Generating content

We simulate responses by adding the scenarios to the question and running it. This generates a `Results` object that we can readily analyze (learn more about [built-in methods for analyzing results](/en/latest/results)):

```python theme={null}
results = q.by(topics).run()
```

Here we select the topics and responses and print them in a table:

```python theme={null}
results.select("topic", "my_style")
```

| | scenario.topic | answer.my\_style |
| :- | :- | :- |
| 0 | parrots | Arrr, matey, let me spin ye a yarn about the feathered buccaneers of the skies – the parrots! These vibrant creatures be like the jeweled treasures of the jungle, with plumage as colorful as a pirate's finest silks. Smart as a whip they are, with voices that echo the hearty laughter of a ship's crew on a moonlit night. Aye, a parrot be more than just a companion; it's a trusty lookout perched on yer shoulder, sharer of secrets and tales of old. With eyes as keen as a sailor's spotting land after months at sea, they see the world in ways we can only dream of. So raise a toast to these winged mates, who bring a dash of the exotic to our everyday voyages, and remind us that even in the heart of the fiercest storm, there's always a bit of color and song to lift our spirits. Yo ho ho, and a cracker for Polly! |
| 1 | holiday weekends | Arrr, gather 'round, ye scallywags, for let me spin ye a yarn about holiday weekends! 'Tis like findin' a hidden cove, where the sands are golden and the tides gentle, offerin' a respite from the daily grind of sailin' the high seas. These be the days when time slows, like a ship driftin' lazily in a sun-dappled bay, givin' ye a chance to drop anchor and revel in the simple joys of life. Whether ye be gatherin' with yer crew for a grand feast or settin' off on a small adventure to explore new shores, a holiday weekend be a treasure chest of moments, ripe for the takin'. The air is filled with the laughter of shipmates and the clinkin' of tankards, as stories of old voyages are spun and new plans are hatched. So raise yer flag, mates, and make the most of these precious days, for they be the pearls strung upon the necklace of life's grand journey. Aye, a holiday weekend is a fine treasure, indeed, and one worth cherishin' with all the gusto of a pirate king! |
| 2 | fun with language models | Arrr, matey, let me spin ye a yarn about the jolly adventure of dabblin' with language models! 'Tis like havin' a trusty parrot on yer shoulder, whisperin' secrets of the written word into yer ear. These clever contraptions be like a treasure chest full of words, ready to spill forth tales as grand as the seven seas. Just a few taps of the keys, and ye've got a crew of words ready to set sail on the vast ocean of imagination. They be like the wind in yer sails, guidin' ye through uncharted waters of creativity, where every sentence is a new horizon and every paragraph a golden doubloon. Aye, with a language model at yer side, ye're never short of a tale to tell or a riddle to unravel. So hoist yer quill and set a course for literary adventure, for the world of words is as boundless as the sea itself, and there's no end to the wonders ye'll discover! |

Check out our [documentation page](/en/latest/index) for many other demo notebooks and tutorials!


# Simulate a feedback survey
Source: https://docs.expectedparrot.com/en/latest/notebooks/yoga_studio_name_survey

This notebook provides example code for simulating a feedback survey with AI agents in [EDSL](https://github.com/expectedparrot/edsl), an open-source library for simulating surveys, experiments and research tasks with AI agents and large language models. We also show how EDSL can be used to generate content by first using it to prompt a model to draft the content for the feedback survey (names for a yoga studio) and the personas that we use for the AI agents (target customers of the studio).

Before running the code below please see instructions for [getting started](https://www.expectedparrot.com/getting-started) using EDSL, and tips and tutorials at our [documentation page](/en/latest/index).

## Create a question

We start by creating a question to prompt a language model to suggest some names for a yoga studio. EDSL comes with many common [question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from the model - multiple choice, free text, linear scale, etc. Here we use `QuestionList` to prompt a model to return a list of items:

```python theme={null}
from edsl import QuestionList

q = QuestionList(
    question_name = "yoga_studio_name",
    question_text = "What are some creative names for a yoga studio?",
    max_list_items = 10
)
```

## Select some models to answer the question

EDSL works with many popular [language models that we can select](/en/latest/language_models) to generate responses to questions. A list of current available models and pricing and performance information is available [here](https://www.expectedparrot.com/models).

To see a list of all service providers:

```python theme={null}
from edsl import ModelList, Model

Model.services()
```

| | Service Name |
| :- | :- |
| 0 | anthropic |
| 1 | azure |
| 2 | bedrock |
| 3 | deep\_infra |
| 4 | deepseek |
| 5 | google |
| 6 | groq |
| 7 | mistral |
| 8 | ollama |
| 9 | openai |
| 10 | openai\_v2 |
| 11 | perplexity |
| 12 | together |
| 13 | xai |

To select models to use with a question or survey:

```python theme={null}
models = ModelList([
    Model("claude-3-sonnet-20240229", service_name = "anthropic"),
    Model("gpt-4o", service_name = "openai")
])
```

## Run the question

We administer a question to a language model by calling the `run()` method on it. This generates a formatted dataset of `Results`:

```python theme={null}
results = q.by(models).run()
```

## Inspect the results

EDSL comes with built-in [methods for analyzing results](/en/latest/results). Here we inspect the responses for each model:

```python theme={null}
results.select("model", "yoga_studio_name")
```

| | model.model | answer.yoga\_studio\_name |
| :- | :- | :- |
| 0 | claude-3-sonnet-20240229 | \['Zen Oasis', 'Mindful Movements', 'Serenity Studio', 'Breathe Bliss', 'Harmony Haven', 'Peaceful Postures', 'Tranquil Tribe', 'Soul Stretch', 'Chakra Chimes', 'Yogic Vibes'] |
| 1 | gpt-4o | \['Zen Flow', 'Soul Stretch', 'Namaste Nest', 'Peaceful Poses', 'Harmony Haven', 'Serenity Studio', 'Mindful Movements', 'Tranquil Transitions', 'Blissful Balance', 'Inner Light Yoga'] |

## Use the responses in new questions

We can format the responses to use them as options to new questions:

```python theme={null}
claude_names = results.filter("model.model == 'claude-3-sonnet-20240229'").select("yoga_studio_name").to_list()[0]
claude_names
```

```
['Zen Oasis',
 'Mindful Movements',
 'Serenity Studio',
 'Breathe Bliss',
 'Harmony Haven',
 'Peaceful Postures',
 'Tranquil Tribe',
 'Soul Stretch',
 'Chakra Chimes',
 'Yogic Vibes']
```

```python theme={null}
gpt4o_names = results.filter("model.model == 'gpt-4o'").select("yoga_studio_name").to_list()[0]
gpt4o_names
```

```python theme={null}
['Zen Flow',
 'Soul Stretch',
 'Namaste Nest',
 'Peaceful Poses',
 'Harmony Haven',
 'Serenity Studio',
 'Mindful Movements',
 'Tranquil Transitions',
 'Blissful Balance',
 'Inner Light Yoga']
```

We can optionally randomize the list of options:

````python theme={null}
studio_names = list(set(gpt4o_names + claude_names))

```python
import random

random.shuffle(studio_names)
````

## Constructing a survey

Here we create some new questions in different types and combine them into a survey to administer them together:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, QuestionCheckBox, Survey

q1 = QuestionMultipleChoice(
    question_name = "favorite",
    question_text = "What is your favorite name for a yoga studio?",
    question_options = studio_names
)

q2 = QuestionCheckBox(
    question_name = "pick",
    question_text = "Pick the 3 best names for a yoga studio.",
    question_options = studio_names,
    min_selections = 3,
    max_selections = 3
)

survey = Survey([q1, q2])

survey
```

[Survey](/en/latest/surveys) # questions: 2; question\_name list: \['favorite', 'pick'];

| | min\_selections | question\_text | max\_selections | question\_name | question\_type | question\_options |
| :- | :- | :- | :- | :- | :- | :- |
| 0 | nan | What is your favorite name for a yoga studio? | nan | favorite | multiple\_choice | \['Peaceful Poses', 'Zen Flow', 'Soul Stretch', 'Zen Oasis', 'Tranquil Transitions', 'Namaste Nest', 'Serenity Studio', 'Tranquil Tribe', 'Inner Light Yoga', 'Mindful Movements', 'Yogic Vibes', 'Harmony Haven', 'Peaceful Postures', 'Chakra Chimes', 'Blissful Balance', 'Breathe Bliss'] |
| 1 | 3.000000 | Pick the 3 best names for a yoga studio. | 3.000000 | pick | checkbox | \['Peaceful Poses', 'Zen Flow', 'Soul Stretch', 'Zen Oasis', 'Tranquil Transitions', 'Namaste Nest', 'Serenity Studio', 'Tranquil Tribe', 'Inner Light Yoga', 'Mindful Movements', 'Yogic Vibes', 'Harmony Haven', 'Peaceful Postures', 'Chakra Chimes', 'Blissful Balance', 'Breathe Bliss'] |

## Designing agents to answer the survey

Next we can design AI agents with relevant traits to answer the questions. Here we use a model to draft some personas, and then create “agent” objects for them to use with the survey:

```python theme={null}
q = QuestionList(
    question_name = "personas",
    question_text = "Draft 5 diverse personas for patrons of a yoga studio."
)
```

```python theme={null}
personas = q.run().select("personas").to_list()[0]
personas
```

```python theme={null}
['A busy corporate executive seeking stress relief and work-life balance',
 'A retired senior looking to maintain flexibility and social connections',
 'A college student exploring mindfulness and physical fitness',
 'A new parent using yoga to regain strength and mental clarity',
 'An athlete incorporating yoga for cross-training and injury prevention']
```

```python theme={null}
from edsl import AgentList, Agent

agents = AgentList(
    Agent(traits = {"persona":p}) for p in personas
)

agents
```

```python theme={null}
AgentList agents: 5;
```

| | persona |
| :- | :- |
| 0 | A busy corporate executive seeking stress relief and work-life balance |
| 1 | A retired senior looking to maintain flexibility and social connections |
| 2 | A college student exploring mindfulness and physical fitness |
| 3 | A new parent using yoga to regain strength and mental clarity |
| 4 | An athlete incorporating yoga for cross-training and injury prevention |

### Run the survey with the agents

Here we add the agents to the survey and run it with the language models we selected above (to compare responses):

```python theme={null}
results = survey.by(agents).by(models).run()
```

```python theme={null}
(
    results
    .sort_by("model", "persona")
    .select("model", "persona", "favorite", "pick")
)
```

| | model.model | agent.persona | answer.favorite | answer.pick |
| :- | :- | :- | :- | :- |
| 0 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Zen Oasis | \['Zen Oasis', 'Tranquil Transitions', 'Serenity Studio'] |
| 1 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Zen Oasis | \['Zen Oasis', 'Tranquil Transitions', 'Harmony Haven'] |
| 2 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Tranquil Tribe | \['Zen Oasis', 'Tranquil Transitions', 'Breathe Bliss'] |
| 3 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Zen Oasis | \['Zen Oasis', 'Serenity Studio', 'Harmony Haven'] |
| 4 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Zen Oasis | \['Zen Oasis', 'Tranquil Transitions', 'Harmony Haven'] |
| 5 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Zen Flow | \['Zen Flow', 'Serenity Studio', 'Blissful Balance'] |
| 6 | gpt-4o | A college student exploring mindfulness and physical fitness | Zen Flow | \['Zen Flow', 'Soul Stretch', 'Inner Light Yoga'] |
| 7 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Namaste Nest | \['Zen Flow', 'Soul Stretch', 'Harmony Haven'] |
| 8 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Serenity Studio | \['Zen Flow', 'Soul Stretch', 'Serenity Studio'] |
| 9 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Zen Flow | \['Zen Flow', 'Soul Stretch', 'Harmony Haven'] |

We could also solicit feedback on individual names. This can be done by creating a “scenario” of each question for each name:

```python theme={null}
from edsl import QuestionLinearScale, QuestionFreeText, Survey, ScenarioList
```

```python theme={null}
scenarios = ScenarioList.from_list("studio_name", studio_names)
scenarios.table()
```

| | studio\_name |
| :- | :- |
| 0 | Peaceful Poses |
| 1 | Zen Flow |
| 2 | Soul Stretch |
| 3 | Zen Oasis |
| 4 | Tranquil Transitions |
| 5 | Namaste Nest |
| 6 | Serenity Studio |
| 7 | Tranquil Tribe |
| 8 | Inner Light Yoga |
| 9 | Mindful Movements |
| 10 | Yogic Vibes |
| 11 | Harmony Haven |
| 12 | Peaceful Postures |
| 13 | Chakra Chimes |
| 14 | Blissful Balance |
| 15 | Breathe Bliss |

```python theme={null}
q_memorable = QuestionLinearScale(
    question_name = "memorable",
    question_text = "How memorable is this yoga studio name: `{{ scenario.studio_name }}`",
    question_options = [1,2,3,4,5],
    option_labels = {1:"Not at all memorable", 5:"Very memorable"}
)

q_criticism = QuestionFreeText(
    question_name = "criticism",
    question_text = "What are some criticisms people might have about this yoga studio name: `{{ scenario.studio_name }}`"
)

survey = Survey([q_memorable, q_criticism])

results = survey.by(scenarios).by(agents).by(models).run()
```

```python theme={null}
(
    results
    .sort_by("model", "persona")
    .select("model", "persona", "studio_name", "memorable", "criticism")
)
```

| | model.model | agent.persona | scenario.studio\_name | answer.memorable | answer.criticism |
| :- | :- | :- | :- | :- | :- |
| 0 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Peaceful Poses | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see for the yoga studio name "Peaceful Poses": • It's a bit generic and doesn't really stand out. There are likely many other yoga studios with similar names playing on the ideas of peace and poses. • While the name conveys a calming, relaxing vibe, it may come across as too bland or uninspired. A more unique or creative name could better grab people's attention. • The name is quite literal and on-the-nose for a yoga studio. Some may prefer a more abstract or metaphorical name that sparks curiosity. • It doesn't give much of a sense of the studio's particular philosophy, style of yoga offered, or unique differentiators from other studios. • For a corporate executive, the name may sound a bit too new-agey or hippie-ish. A name with a slightly more modern, sophisticated edge could appeal more to this demographic. However, the name does clearly convey the core idea of yoga and finding peace through the physical poses and practice. For someone seeking stress relief from a busy career, that straightforward messaging could still resonate. But a more distinctive name may be preferred. |
| 1 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Zen Flow | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see with the yoga studio name "Zen Flow": • It's a bit cliche or overused - The words "zen" and "flow" are very commonly used in the yoga/meditation space, so the name may not stand out as very unique or memorable. • It doesn't give much insight into the studio's philosophy or approach. A more descriptive name could better convey if it's a hot yoga studio, focused on a particular style like vinyasa or yin, etc. • It has a very serene, relaxing vibe which may not appeal to people looking for a more intense, athletic yoga practice. The name implies a very gentle, calming experience. • It's not very localized or connected to the community/neighborhood where the studio is located. A name that ties into the area could help it feel more grounded. • As an executive, I may be looking for a studio that balances the restorative benefits of yoga with an experience well-suited for busy professionals. The name doesn't necessarily convey that balance. However, the name does successfully evoke feelings of peace and going with the flow, which could be appealing for stress relief. But some more distinctive or descriptive elements may make the name stronger overall. |
| 2 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Soul Stretch | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see with the yoga studio name "Soul Stretch": • It sounds a bit new age-y or woo-woo, which could turn off more pragmatic or skeptical clients looking for just the physical exercise benefits of yoga. The "soul" reference may come across as too spiritual for some. • The name is a little vague and doesn't give a strong sense of what type of yoga or experience to expect. It doesn't differentiate the studio from others. • "Soul Stretch" could be seen as trying too hard to be catchy or clever with the alliteration. Sometimes simpler and more straightforward names can be more appealing. • For a corporate professional, the name may not inspire confidence that this is a serious, high-quality studio focused on challenging physical practice. It could seem more suited to a hippie studio vibe. However, the name could also resonate well if positioned properly as a holistic mind-body experience. But those are some potential criticisms I could envision from a busy executive's perspective just looking for an effective workout amongst all my corporate responsibilities. A more neutral name may come across as more credible. |
| 3 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Zen Oasis | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see with the yoga studio name "Zen Oasis": 1. Overused words: The words "zen" and "oasis" are quite commonly used in the wellness/yoga space. Some may find the name a bit cliche or unoriginal. 2. Doesn't stand out: With so many yoga studios using similar calming, nature-inspired names, "Zen Oasis" may not help the studio stand out from competitors. A more unique name could be more memorable. 3. Doesn't convey specialty: If the studio has a particular focus or specialty (e.g. hot yoga, prenatal yoga, etc.), the generic name doesn't communicate that. A name reflecting the specialty could attract their target clientele better. 4. Doesn't reflect location: For a local business like a yoga studio, some may prefer a name that ties into the neighborhood, city or region to give it more of a community feel. 5. Too relaxing tone: As an executive, part of me might want a name that conveys more energy, power or intensity to counterbalance the very chilled "zen" vibe for an effective stress release. However, the name does successfully evoke a sense of tranquility which could be appealing for stressed corporate folks like myself seeking an escape. But a more distinctive name could potentially serve the studio better in a crowded market. |
| 4 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Tranquil Transitions | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see about the yoga studio name "Tranquil Transitions": • It's a bit generic and vague. There are a lot of yoga studios with names involving words like "tranquil" or "transitions." It doesn't really stand out or give a strong sense of the studio's unique identity. • The word "transitions" could be confusing or offputting to some. It makes me wonder if the studio specializes in some sort of transitional style of yoga rather than being a more general studio. • While "tranquil" conveys a sense of peace and relaxation, it may come across as boring or unmemorable to some potential customers looking for something more dynamic or inspiring. • The name doesn't give any indication of what style or types of yoga the studio offers. As someone very new to yoga, I'd want the name to give me a better sense of what to expect. Overall, while not a terrible name, it doesn't particularly excite me or make me really want to rush out and try this specific studio over others in the area. A more distinctive and evocative name highlighting the studio's unique approach or community could be more appealing to a time-crunched executive like myself. |
| 5 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Namaste Nest | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see with the yoga studio name "Namaste Nest": 1. Overused terms: The words "namaste" and "nest" are quite commonly used in the yoga/wellness space. Some may find the name a bit cliche or unoriginal. 2. Mixed metaphors: Combining the Sanskrit greeting "namaste" with the idea of a "nest" could be seen as an odd pairing of concepts from different contexts. 3. Pronunciation issues: For those unfamiliar with Sanskrit, "namaste" may be difficult to pronounce correctly, potentially making the name harder to remember or discuss. 4. Lack of distinctiveness: With so many yoga studios using nature/spiritual words in their names, "Namaste Nest" may not stand out enough in the market. 5. Niche appeal: The name leans heavily into the spiritual/natural aspects of yoga which could alienate those looking for a more modern, fitness-focused experience. However, the name does convey a sense of peacefulness, comfort and connection to one's roots or inner self which aligns with yoga's principles. As an executive, I may be drawn to the calming, nurturing vibe it aims to create as an escape from my stressful work life. But a more unique, memorable name could be beneficial from a branding standpoint. |
| 6 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Serenity Studio | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I might have about the name "Serenity Studio" for a yoga studio: 1. Lack of uniqueness: The name "Serenity Studio" is quite generic and doesn't stand out from other yoga studios or wellness centers. It may not be memorable or help the studio differentiate itself in a crowded market. 2. Overused concept: The idea of "serenity" is often associated with yoga and meditation, so the name may come across as clichéd or unoriginal. As someone looking for a fresh perspective, I might find the name uninspiring. 3. Mismatch with target audience: Depending on the studio's target clientele, the name "Serenity Studio" may not resonate well with busy professionals like myself who are seeking an active, energetic environment for stress relief rather than just a serene, relaxing space. 4. Limited scope: The name implies a focus solely on achieving a state of serenity, which may be too narrow in scope. As an executive, I might be looking for a more comprehensive wellness experience that addresses physical, mental, and emotional well-being. 5. Disconnect from branding: If the studio's branding, interior design, or class offerings don't align with the serene and tranquil vibe implied by the name, it could create a disconnect and confusion for potential clients like me. Overall, while "Serenity Studio" conveys a sense of calm and relaxation, it may not be the most compelling or distinctive name for a yoga studio catering to busy professionals seeking a dynamic and well-rounded wellness experience. |
| 7 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Tranquil Tribe | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could envision for the yoga studio name "Tranquil Tribe": 1. The word "tribe" may come across as culturally appropriative or insensitive, especially if the studio owners are not from an indigenous culture that traditionally uses that term. 2. Some may find the name a bit too new-agey or cliché, playing into stereotypes about yoga studios trying too hard to seem zen and spiritual. 3. The juxtaposition of "tranquil" and "tribe" could be seen as contradictory - tribes often have connotations of being raucous or chaotic, which clashes with tranquility. 4. For a corporate executive, the name may not convey enough of a sense of professionalism, sophistication or elite wellness that some may be looking for in a high-end yoga experience. 5. If the studio is located in an urban area, the word "tribe" could seem out of place or an odd choice of wording. However, the name could also resonate well if positioned correctly as a welcoming community focused on inner peace. But those are some potential criticisms to consider from the perspective of someone looking to de-stress from corporate life. A simple, more understated name may appeal more to that demographic. |
| 8 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Inner Light Yoga | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see for the yoga studio name "Inner Light Yoga": 1. Cliché or overused: The phrases "inner light" and referring to yoga as a way to find one's inner light or inner peace is a very common trope. Some may find it an unoriginal or clichéd name. 2. Too spiritual/religious: Depending on the demographics of the area, some potential customers may be put off if the name comes across as too spiritual, New Age, or having religious overtones. As a corporate exec, I may prefer something more neutral. 3. Lacking uniqueness: With so many yoga studios out there, a name like this may not stand out or differentiate itself from the competition. A more unique name could help it be more memorable. 4. Unclear offerings: The name doesn't give any indication of the specific yoga styles, classes, or amenities offered at this particular studio. A more descriptive name could better set expectations. 5. Too feminine?: While yoga is popular among women, some male corporate professionals may subconsciously perceive a name with "inner light" as being too feminine or not aligning with their interests. However, the name could also resonate well if positioned properly for its target market. But those are some potential criticisms to consider from the perspective of a busy executive just looking for an effective stress-relief option. |
| 9 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Mindful Movements | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see with the yoga studio name "Mindful Movements": 1. Lack of distinctiveness - The name is fairly generic and descriptive. There could be concerns that it doesn't stand out enough from other yoga studios with similar names focused on mindfulness and movement. 2. Too vague - While "mindful" and "movements" relate to yoga, the name doesn't explicitly convey that it is a yoga studio. Some may find the name too ambiguous about the specific services offered. 3. Overused words - Terms like "mindful" and "movements" are quite common in the yoga/wellness space. The name could come across as clichéd or unoriginal to some. 4. Disconnect for high-intensity styles - If the studio offers more vigorous, athletic yoga styles like power yoga, the "mindful" aspect of the name may not fully resonate or seem energetic enough. 5. Corporate perception - As an executive, I may perceive a name like "Mindful Movements" as too New Age-y or not aligning well with a corporate, professional image I'm looking to maintain. Overall, while inoffensive, the name plays it fairly safe. But that neutral quality could be a criticism if it fails to make the studio memorable or distinctive in a crowded yoga market. A more unique, creative name could be preferred by some. |
| 10 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Yogic Vibes | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I might have about the yoga studio name "Yogic Vibes": 1. Overused terminology: The words "yogic" and "vibes" are quite trendy and commonly used in the yoga/wellness space. The name may come across as unoriginal or clichéd. 2. Lack of distinctiveness: With so many yoga studios using similar new-age, spiritual terminology in their names, "Yogic Vibes" may not stand out or differentiate itself enough in the market. 3. Vague meaning: While the name hints at a relaxing, peaceful atmosphere, it doesn't provide much specific information about the studio's offerings, teaching style, or unique value proposition. 4. Unprofessional tone: As an executive, I might perceive "Yogic Vibes" as having a too casual, hippy-ish vibe that doesn't align with the more serious, professional environment I'm used to in the corporate world. 5. Limited appeal: The name could potentially turn off certain demographics like older adults or men who may find it too feminine or woo-woo sounding. Overall, while inoffensive, "Yogic Vibes" may come across as a safe, generic name that blends in rather than making a memorable impression, especially for a corporate client like myself seeking an oasis of calm professionalism. A more distinctive, grounded name could better attract busy professionals. |
| 11 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Harmony Haven | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see for the yoga studio name "Harmony Haven": • It sounds a bit cliché or generic. There are a lot of yoga/wellness brands that use words like "harmony" and "haven" which can make it blend in and not stand out. • The alliteration of the two H words is a bit overused and doesn't feel very modern or unique. • For an executive crowd, it may come across as too new-agey or hippie-ish sounding rather than feeling upscale or tailored for professionals. • It doesn't really convey any unique brand personality or differentiator from other yoga studios. • The name is quite long for a business name, which can make it harder to remember or refer to casually. Overall, as someone looking for an urban oasis to de-stress from my intense corporate life, I may perceive "Harmony Haven" as a bit too generic and not aligning with the premium, modern experience I'd ideally want from a studio catering to executives. A punchier, more distinctive name could be more appealing. |
| 12 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Peaceful Postures | 4 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see for the yoga studio name "Peaceful Postures": • It's a bit generic and doesn't really stand out from other yoga studio names. There are likely many studios with names playing on the ideas of "peace" and "postures." • While the name conveys a relaxing, calming vibe, it may come across as a little bland or uninspired. A more unique or creative name could better grab people's attention. • For someone like me looking to de-stress from the corporate grind, the name is pleasant but doesn't necessarily convey an energizing, rejuvenating experience. I may be looking for something that sounds more dynamic. • The name is focused just on the physical postures/asanas aspect of yoga. It doesn't hint at any of the mental/spiritual elements that could appeal to someone wanting a full mind-body practice. • It's not very descriptive of the studio's particular teaching style, specialties, or what may make their approach unique compared to other peaceful posture-oriented studios. However, the name does effectively communicate a core benefit of yoga - achieving a state of peacefulness through the physical practice. For a corporate executive like myself, that could be an appealing quality to relieve work stresses. But making the name more distinctive could be beneficial. |
| 13 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Chakra Chimes | 5 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see people having about the name "Chakra Chimes" for a yoga studio: 1. Too New Age/Spiritual: Some people may find the reference to "chakras" too overtly spiritual or New Age, which could turn off those looking for a more mainstream, fitness-focused yoga experience. 2. Difficult to Pronounce/Understand: The word "chakra" may be unfamiliar to some, making the name a bit confusing or hard to pronounce for the general public. 3. Doesn't Convey Yoga Clearly: The name doesn't immediately convey that it's a yoga studio, unless you already know what chakras are. A more obvious yoga reference in the name could be clearer. 4. Too Cutesy/Precious: Combining "chakra" with "chimes" could come across as trying too hard to be clever or cute, which may not appeal to everyone's tastes. 5. Lacks Professionalism: As an executive, I may perceive the name as not sounding professional or corporate enough for my liking when seeking a yoga studio. However, the name could also attract those specifically interested in the spiritual side of yoga. But those are some potential criticisms I could envision from a corporate, mainstream perspective. A simple, clear name may have broader appeal. |
| 14 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Blissful Balance | 5 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see about the yoga studio name "Blissful Balance": • It sounds a bit cliché or overused. Many yoga/wellness brands use words like "bliss" and "balance" so it doesn't really stand out. • The name is a bit long and doesn't have a very punchy, memorable sound to it. Shorter names can be easier to remember. • It leans heavily into the stereotypical promises of yoga (bliss, balance) without giving much sense of what makes this particular studio unique or special. • For a corporate executive, the name sounds almost too "woo-woo" or new age-y. A more grounded, practical name could appeal better to this demographic. • It doesn't give any indication of the specific yoga style or approach the studio takes, which could be useful for setting expectations. Overall, while inoffensive, the name doesn't particularly excite or intrigue me as someone looking for an effective way to de-stress from my high-pressure career. A more distinctive name could pique my interest more effectively. |
| 15 | claude-3-sonnet-20240229 | A busy corporate executive seeking stress relief and work-life balance | Breathe Bliss | 5 | As a busy corporate executive seeking stress relief and work-life balance, here are some potential criticisms I could see with the yoga studio name "Breathe Bliss": • It sounds a bit cheesy or over-the-top. The word "bliss" has a very new-agey, woo-woo connotation that some may find off-putting or not take seriously. • It doesn't really convey what type of yoga or experience is offered. Is it hot yoga, vinyasa flow, restorative? The name is vague. • For a corporate professional, the name may come across as too touchy-feely or not aligning with a more pragmatic, results-oriented mindset I likely have. • It could be seen as appropriating Eastern philosophies in a somewhat superficial way by boiling the yoga experience down to just "breathing bliss." • The name is a bit long and not very catchy or memorable compared to punchier single word names. However, I imagine the intent is to convey a sense of serenity and present yoga as a pathway to inner peace and contentment. But the execution may miss the mark for some skeptics or those looking for a more grounded, fitness-oriented approach. A simpler name could be more effective. |
| 16 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Peaceful Poses | 4 | As a college student exploring mindfulness and physical fitness, I can see a few potential criticisms people might have about the name "Peaceful Poses" for a yoga studio: 1. Lack of originality - The name is fairly generic and descriptive. There are likely many other yoga studios with similar names using words like "peaceful" and "poses." It may not stand out very much. 2. Too narrow focus - The name implies the studio only offers gentle, relaxing styles of yoga focused on holding poses. However, many studios offer a variety of yoga styles from vigorous flows to meditation classes. The name pigeonholes it. 3. Doesn't convey a unique brand identity - Good brand names often aim to evoke a distinct feeling, image or philosophy that sets the business apart. "Peaceful Poses" is pretty straightforward without much deeper meaning. 4. Not very modern or creative - Some may see the name as a bit dated or uninspired compared to trendier, more creatively named yoga studios popping up. 5. Limiting growth potential - If the studio ever wanted to expand beyond just yoga classes into other wellness offerings, the name could become too restrictive. However, the name does clearly convey a sense of serenity and the physical practice of yoga poses, which some may appreciate for its simplicity. Overall perception could go either way depending on what the studio is aiming to convey through their name and brand. Those are just some potential critiques to consider. |
| 17 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Zen Flow | 4 | As a college student exploring mindfulness and physical fitness, I can see a few potential criticisms people might have about the name "Zen Flow" for a yoga studio: 1. Cultural appropriation: Some may feel that using the word "Zen," which has roots in Buddhism, is appropriating or commodifying aspects of Eastern spiritual traditions in a way that lacks authenticity or respect. 2. Cliché or overused: Words like "zen" and "flow" have become quite popular and trendy in the wellness/yoga space. Some may see the name as unoriginal or clichéd. 3. Lack of distinctiveness: The name is fairly generic and doesn't necessarily convey what makes this particular studio unique or different from others using similar wording. 4. Misleading expectations: Depending on the style of yoga taught, some may feel "Zen Flow" implies a very calm, meditative practice when the classes could actually be more vigorous or athletic in nature. However, these are just potential criticisms. The name could resonate well with the intended audience and effectively convey a sense of mindfulness and movement. Analyzing names is quite subjective. As a mindful student, I try to consider different perspectives while not passing harsh judgment. |
| 18 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Soul Stretch | 5 | As a college student exploring mindfulness and physical fitness, I can see a few potential criticisms people might have about the name "Soul Stretch" for a yoga studio: 1. Some may find it too spiritual or new age-y sounding. The word "soul" has religious/spiritual connotations that some people may not relate to or feel comfortable with when just looking for a fitness class. 2. It doesn't explicitly convey that it's a yoga studio. The name is a bit vague and someone unfamiliar may not immediately know it's for yoga classes. 3. The name is a little generic and doesn't really differentiate it from other studios. There could be concerns that it's not a very unique or memorable name. 4. Depending on the vibe/philosophy of the studio, some may find the name too commercial or gimmicky if they prefer a more traditional, authentic yoga environment. However, the name could also resonate well with those drawn to the mindfulness aspects of yoga beyond just the physical practice. But those are some potential criticisms I could foresee from people looking for a name that is more clearly yoga-centric, unique, or aligns with their particular yoga values and interests. What's your take on the name? |
| 19 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Zen Oasis | 5 | As a college student exploring mindfulness and physical fitness, I can think of a few potential criticisms people might have about the name "Zen Oasis" for a yoga studio: 1. Cultural appropriation: The word "zen" has roots in Buddhism and some may feel that using it in a commercial context appropriates or commodifies an important spiritual tradition. 2. Cliché: Terms like "zen" and "oasis" are somewhat overused when describing yoga studios, spas, etc. Some may see it as an unoriginal or clichéd name. 3. Misleading expectations: The name evokes a sense of tranquility and escape, but the actual studio experience may not live up to those lofty expectations set by the name. 4. Lack of distinctiveness: With so many yoga studios using similar Zen/nature-inspired names, it may not effectively differentiate this particular business. 5. Exclusionary: The zen/oasis imagery could potentially make the studio feel exclusionary or intimidating to newcomers who don't identify with that aesthetic. However, these are just potential criticisms. The name could also resonate positively with the right audiences seeking that calm, rejuvenating environment the name suggests. Evaluating names is quite subjective. |
| 20 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Tranquil Transitions | 4 | As a college student exploring mindfulness and physical fitness, I can think of a few potential criticisms people might have about the name "Tranquil Transitions" for a yoga studio: 1. Vagueness - The name is a bit vague and doesn't explicitly convey that it's a yoga studio. Someone unfamiliar with the business may not immediately understand what services are offered. 2. Overused words - The words "tranquil" and "transitions" are fairly common in the wellness/yoga space, so the name may not stand out as very unique or memorable. 3. Lack of personality - Some may feel the name is a bit generic and doesn't really capture a distinct personality, vibe or specialty that sets this particular studio apart. 4. Mismatch with offerings - If the studio offers more vigorous or athletic yoga styles, the "tranquil" part of the name may seem misaligned with the actual experience. 5. Confusion with other businesses - There could potentially be other businesses, even outside of yoga/fitness, with a similar name which could cause brand confusion. Overall, while "Tranquil Transitions" aims to evoke a sense of peace and change, some constructive feedback may be that the name could be more distinctive, descriptive and aligned with the studio's unique identity and offerings. But of course, naming is quite subjective. |
| 21 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Namaste Nest | 5 | As a college student exploring mindfulness and physical fitness, here are some potential criticisms I can think of for the yoga studio name "Namaste Nest": 1. Cultural appropriation: The word "namaste" comes from Sanskrit and has deep cultural and spiritual roots in Indian traditions like Hinduism and Buddhism. Using it in a commercial business name could be seen as appropriating the term without fully understanding or respecting its meaning. 2. Cliché or overused: Terms like "namaste" have become quite popular and commonplace in the Western yoga/wellness industry. Some may view "Namaste Nest" as an unoriginal or clichéd name that doesn't stand out. 3. Mixed metaphors: Combining "namaste" (a greeting) with "nest" (a place for birds) could be seen as an odd or nonsensical mixing of dissimilar concepts that don't quite go together smoothly. 4. Too cute or precious: The name could come across as trying too hard to be cute, quirky or clever with the "nest" aspect, when some may prefer a more straightforward, grounded studio name. 5. Limiting assumptions: Using "nest" may give the impression of a very small, cozy space when the studio could actually be quite large. It could misrepresent the vibe or scale. However, these are just potential criticisms. The name could also resonate positively with the right audience looking for a welcoming, tranquil yoga space. Context and branding are key. |
| 22 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Serenity Studio | 4 | As a college student exploring mindfulness and physical fitness, here are some potential criticisms I could see people having about the name "Serenity Studio" for a yoga studio: 1. Lack of originality - The word "serenity" is quite commonly used for yoga studios, spas, and wellness centers. Some may find the name a bit generic or uninspired. 2. Doesn't convey the full yoga experience - While serenity is certainly a goal of yoga practice, yoga is also about physical exercise, discipline, focus, and self-discovery. The name may be seen as too narrow by not capturing the full breadth of what yoga offers. 3. Doesn't stand out - In an area with many yoga studios, a name like "Serenity Studio" may get lost in the crowd and not help the business stand out from competitors. 4. Doesn't reflect the studio's unique offerings - If this particular studio has a special focus, like hot yoga, prenatal yoga, or a certain yoga style, the generic name doesn't communicate that unique selling point. 5. Overly calm/passive connotation - Some may feel the name "Serenity Studio" sounds too passive or laidback, when yoga also requires effort, energy and an active mindset. However, the name could resonate well if the studio really leans into creating a tranquil, calming environment as its core focus. But those are some potential criticisms to consider about the name choice. As a mindful student, I try to look at things from multiple perspectives. |
| 23 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Tranquil Tribe | 5 | As a college student exploring mindfulness and physical fitness, I can see a few potential criticisms people might have about the yoga studio name "Tranquil Tribe": 1. Cultural appropriation concerns: The word "tribe" has cultural connotations, especially for indigenous communities. Some may view using it in a commercial context as appropriating or commodifying their cultures. 2. Exclusionary vibes: While the name aims for a welcoming, community feel, the tribal implication could come across as clique-ish or exclusive to some. They may feel it sends the wrong vibe about who the "tribe" includes. 3. Cliché factor: With mindfulness and yoga becoming more mainstream, some critics could view "Tranquil Tribe" as leaning too heavily into trendy clichés about yoga/meditation communities. 4. Lack of distinctiveness: Simple names using words like "tranquil" are very common for yoga studios. Critics may argue it doesn't make the studio particularly memorable or stand out. However, the name could also resonate positively with the right audiences seeking that tranquil, community atmosphere. Weighing potential concerns against the intended branding is important in naming decisions. As a mindful student, I try to consider multiple perspectives. |
| 24 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Inner Light Yoga | 4 | As a college student exploring mindfulness and physical fitness, here are some potential criticisms I could see people having about the name "Inner Light Yoga" for a yoga studio: 1. Cliché - The phrase "inner light" is a bit of a cliché when it comes to yoga/spiritual branding. Some may find it unoriginal or overused. 2. Too spiritual/woo-woo - For those looking for a more fitness/exercise oriented yoga experience, a name emphasizing the "inner light" may come across as too spiritual or new age for their tastes. 3. Unclear meaning - The "inner light" metaphor may be confusing or unclear to some, especially newcomers to yoga who may not grasp the spiritual connotations right away. 4. Exclusionary - Yoga is becoming more mainstream, so a name that leans heavily into the spiritual side could potentially turn off those wanting a more secular experience. 5. Too narrow - By highlighting one specific aspect of yoga (the inner spiritual light), the name could be seen as too narrowly defining the studio's offerings if they provide diverse yoga styles. However, the name could certainly resonate with and attract those specifically seeking a more spiritual, inwardly-focused yoga practice. But those are some potential criticisms to consider from a mindful college student's perspective. A clear branding strategy could help mitigate some of those concerns. |
| 25 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Mindful Movements | 5 | As a college student exploring mindfulness and physical fitness, here are some potential criticisms I could see about the yoga studio name "Mindful Movements": 1. Lack of originality/overused words: The words "mindful" and "movements" are quite commonly used in the yoga/wellness space, so some may find the name a bit generic or unoriginal. 2. Doesn't convey a unique selling point: The name doesn't really differentiate this particular studio or highlight what makes their approach or philosophy unique compared to others. 3. Vague meaning: While the name hints at mindfulness and physical practice, it's fairly vague and doesn't give a clear sense of what someone can expect from classes there. 4. Doesn't evoke a strong visual: Some business names are more effective when they create a vivid mental image or feeling. "Mindful Movements" is quite literal and descriptive rather than evoking imagery. 5. Potential for confusion: There could be other businesses or studios with a very similar name, leading to possible mix-ups or lack of distinct branding. However, criticisms are subjective. The name could resonate well with the right audience and branding. As a mindful student, I don't find it offputting, but adding a unique element could make the name more distinctive. |
| 26 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Yogic Vibes | 4 | As a college student exploring mindfulness and physical fitness, here are some potential criticisms I can see people having about the yoga studio name "Yogic Vibes": 1. It sounds a bit too informal or casual for a yoga studio name. The word "vibes" has a very relaxed, almost hippie-ish connotation that some may find unprofessional or not serious enough. 2. The name doesn't really convey much about the specific type of yoga practiced or the studio's philosophy/approach. It's quite vague. 3. Some may find it a bit cliché or overused to combine the word "yogic" with something else to name a yoga studio. There are likely many other studios with similar names playing on the word "yoga." 4. The name could come across as trying too hard to be trendy or appealing to a younger crowd by using slang like "vibes." This may turn off some older or more traditional yoga practitioners. 5. It doesn't really differentiate the studio much from others or give a strong sense of identity. The name is fairly generic. However, the name could also appeal to those looking for a laid-back, welcoming vibe rather than a very serious or rigid atmosphere. But those are some potential criticisms I could envision about the name choice. Evaluating yoga studio names is quite subjective. |
| 27 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Harmony Haven | 4 | As a college student exploring mindfulness and physical fitness, I can think of a few potential criticisms people might have about the name "Harmony Haven" for a yoga studio: 1. Lack of originality - The name uses fairly common words associated with yoga and wellness like "harmony" and "haven." Some may find it a bit generic or unimaginative. 2. Too new-agey - The ethereal, peaceful vibe of the name could come across as too stereotypically new-age or hippie-ish for some people's tastes. Those looking for a more modern, urban yoga experience may not be drawn to it. 3. Unclear brand identity - The name doesn't give a strong sense of what specific yoga styles, philosophies or amenities the studio specializes in. It's quite vague. 4. Spelling/pronunciation issues - While a minor point, some may find the alliteration of "Harmony Haven" a bit of a tongue-twister that's difficult to say or spell correctly. Overall, I think "Harmony Haven" aims to evoke a tranquil, restorative feeling in line with yoga's principles. But some could view it as too cliché or lacking a unique brand voice. A more distinctive name could help the studio stand out in a crowded yoga market. But it's a subjective matter of personal preferences. |
| 28 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Peaceful Postures | 4 | As a college student exploring mindfulness and physical fitness, I can think of a few potential criticisms people might have about the yoga studio name "Peaceful Postures": 1. Lack of originality - The name is quite generic and doesn't really stand out from other yoga studio names using words like "peaceful" and "postures." It may come across as unimaginative. 2. Too narrow focus - By emphasizing just the peaceful and physical aspects, it may give the impression that the studio only offers gentle, restorative styles of yoga. Some people looking for more vigorous practices like power yoga may not feel drawn in. 3. Doesn't convey personality - The name is descriptive but doesn't really give a sense of the studio's unique vibe, philosophy or community feel. A more distinctive name could better attract their ideal clientele. 4. Awkward phrasing - While not a major issue, the combination of "peaceful" and "postures" is a bit clunky linguistically. A tighter name might roll off the tongue better. However, the name does clearly convey yoga's core ideas of finding stillness and working with postures/asanas. For a no-frills studio not too concerned about branding, "Peaceful Postures" is straightforward if not very exciting. But many may feel it could use a more creative, standout name. |
| 29 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Chakra Chimes | 5 | As a college student exploring mindfulness and physical fitness, I can think of a few potential criticisms people might have about the name "Chakra Chimes" for a yoga studio: 1. Cultural appropriation: Some may feel that using the term "chakra," which has roots in Hindu and Buddhist traditions, is an inappropriate appropriation of another culture's spiritual beliefs and practices, especially if the studio owners are not from those backgrounds. 2. Lack of inclusivity: The name may come across as catering primarily to those already familiar with concepts like chakras, potentially making newcomers to yoga feel excluded or intimidated. 3. Stereotyping: There could be a perception that the name leans too heavily into New Age stereotypes associated with yoga, which some may find off-putting or inauthentic. 4. Branding confusion: Depending on the studio's actual offerings, the name may create certain expectations about the type of yoga practiced there (e.g., more spiritual/mystical) that don't align with the reality. 5. Difficulty with pronunciation/spelling: Some may find "Chakra Chimes" a bit of a tongue-twister or hard to spell, which could make marketing and word-of-mouth more challenging. However, these are just potential criticisms - the name could also resonate positively with the intended audience. As a mindful student, I believe it's important to consider different perspectives while also staying true to one's authentic vision and values. |
| 30 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Blissful Balance | 4 | As a college student exploring mindfulness and physical fitness, I can think of a few potential criticisms people might have about the yoga studio name "Blissful Balance": 1. It sounds a bit clichéd or overused. Many yoga studios tend to use words like "bliss," "balance," "harmony," etc. in their names, so this one may not stand out as very unique or memorable. 2. The name is fairly generic and doesn't give much of a sense of the studio's specific philosophy, teaching style, or vibe. It could apply to almost any yoga studio. 3. Some may find the name a bit too "woo-woo" or new age-y sounding, especially with the word "blissful." This could turn off people looking for a more grounded, no-nonsense approach. 4. The name implies a very serene, relaxing experience, which may not accurately represent the intensity of some yoga styles like power vinyasa or hot yoga offered there. 5. It's a bit of a mouthful - some may prefer a punchier, shorter name that's easier to say and remember. However, criticisms are subjective. The name could resonate well with the right audience looking for a sense of peacefulness and equilibrium in their yoga practice. As a mindful student, I try to consider different perspectives openly. |
| 31 | claude-3-sonnet-20240229 | A college student exploring mindfulness and physical fitness | Breathe Bliss | 5 | As a college student exploring mindfulness and physical fitness, I can see a few potential criticisms people might have about the yoga studio name "Breathe Bliss": 1. Some may find it a bit cheesy or over-the-top in terms of the wording and implied promises. "Bliss" sets a very high bar for the experience. 2. It doesn't explicitly convey that it's a yoga studio, so the name alone may be a bit vague or unclear for those unfamiliar with it. 3. The focus on breathing could be seen as too narrow, since yoga involves much more than just breathing exercises. 4. From a branding standpoint, some may criticize it as not being a very distinctive or memorable name compared to other yoga studios. 5. There could be concerns that it sounds too new-agey or woo-woo, which could turn off those looking for a more athletic, exercise-focused yoga experience. However, these are just potential criticisms - the name could also resonate well with the right audience looking for a calming, mindful yoga practice. Evaluating a business name is quite subjective. As someone interested in mindfulness, I don't hate the name, but I can understand some of the potential drawbacks others may see. |
| 32 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Peaceful Poses | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could envision for the yoga studio name "Peaceful Poses": • It's a bit generic or cliche - Many yoga studios have names referencing peace, serenity, etc. so this one may not stand out very much. • It doesn't give a strong sense of the studio's philosophy or teaching style. A name that hints at the type of yoga practiced (e.g. vinyasa, yin, kundalini) could be more informative. • For a new parent, the word "peaceful" may come across as unrealistic or unattainable with the demands of caring for a baby or young child. A name acknowledging the challenges but benefits of yoga for parents could resonate more. • The name is a bit soft or gentle sounding. As someone trying to rebuild strength postpartum, I may be drawn to a name that conveys more energy, power or intensity. However, the name does clearly convey yoga and a sense of tranquility which could be appealing. But those are some potential critiques to consider from the perspective of a new parent yogini seeking both physical and mental rejuvenation through their practice. A more distinctive or parenthood-specific name may better attract that particular clientele. |
| 33 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Zen Flow | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could envision for the yoga studio name "Zen Flow": • The name is a bit cliche or overused - words like "zen" and "flow" are very common in the yoga/meditation space, so the name may not stand out or seem very original. • It doesn't give much insight into the specific style or approach of this particular studio. Many studios try to differentiate themselves with names relating to their core philosophy or methods. • For a new parent, the name doesn't necessarily convey anything about services tailored specifically for pre/postnatal needs or parent-child yoga classes. • The name is fairly generic and could apply to any yoga studio. It doesn't give a strong sense of place, community, or unique identity. • While "flow" relates to the physical practice, "zen" implies a more meditative focus. The combination could seem a bit disjointed in its message. However, the name does evoke a sense of calming energy and movement, which could appeal to students seeking a serene, mindful environment to practice in. As a new parent, those are qualities I'd likely appreciate in a yoga space. But making the name more distinctive could help the studio stand out. |
| 34 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Soul Stretch | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could see for the yoga studio name "Soul Stretch": • It sounds a bit new age-y or woo-woo, which could turn off people looking for a more straightforward fitness experience. The "soul" part makes it seem more spiritual than just physical. • It's a little vague - it doesn't really convey what style or type of yoga they offer. People might not know if it's more vigorous or gentle yoga. • The name is a bit long and doesn't have a very catchy, memorable ring to it in my opinion. Shorter punchier names tend to stick better. • It could come across as trying too hard to be deep and meaningful when people just want a good workout. However, the name does convey the idea of stretching one's body and spirit, which aligns with the holistic benefits of yoga. As a new parent, the name resonates with me seeking to regain physical and mental wellness through my practice. But I can see how it may not appeal to everyone's tastes or expectations for a yoga studio name. An alternative could be something more straightforward like "Main Street Yoga" or "City Yoga Center." |
| 35 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Zen Oasis | 5 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could envision for the yoga studio name "Zen Oasis": 1. Overused words: The words "zen" and "oasis" are quite commonly used in the yoga/wellness space. Some may find the name a bit cliche or unoriginal. 2. Cultural appropriation concerns: The word "zen" has roots in Buddhism and Japanese culture. There could be criticism that a Western business is appropriating this term inappropriately or inauthentically. 3. Misleading expectations: An "oasis" implies a lush, relaxing escape. If the actual studio space is small or spartan, the name may set up expectations that don't match the experience. 4. Too new-agey: For those looking for a more athletic, no-frills yoga experience, a name with "zen" and "oasis" could come across as too new-age or woo-woo sounding. 5. Lack of distinctiveness: In an area saturated with yoga studios, this name may not stand out enough from the countless other "zen," "serenity," "peace" type names. Overall, as a busy new parent, I personally don't mind the calming, tranquil vibe the name evokes. But I can understand criticisms around overused language, cultural appropriateness, and the need for a more distinctive brand name. Evaluating the local market would be wise. |
| 36 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Tranquil Transitions | 5 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could see people having about the name "Tranquil Transitions" for a yoga studio: • The name is a bit vague or generic. Many yoga studios use words like "tranquil" or "transitions" in their names, so it may not stand out enough. • For a new parent, the word "transitions" could bring to mind things like labor, postpartum changes, etc rather than the intended meaning of life transitions. This could be confusing. • Some may find the combination of "tranquil" and "transitions" to be contradictory, as transitions can often be stressful rather than tranquil times. • It doesn't specifically convey that it is a yoga studio, which could be unclear for those unfamiliar with the business. • The name is quite long compared to some other yoga studio names, which tend to be shorter and punchier. However, I imagine the intent is to convey a sense of peace and acceptance through life's changes, which aligns with the yoga philosophy. As a new parent, I personally don't find it off-putting, but those are some potential critiques others may raise about the name choice. What matters most is that the studio lives up to cultivating that tranquil space. |
| 37 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Namaste Nest | 5 | As a new parent using yoga to regain strength and mental clarity, I can understand some potential criticisms people might have about the name "Namaste Nest" for a yoga studio: 1. Cultural appropriation: The word "Namaste" is a Sanskrit greeting used in yoga that has roots in Indian culture. Some may feel that using it in a business name appropriates or commodifies an aspect of another culture. 2. Cliché: Using "Namaste" in a yoga studio name could be seen as clichéd or unoriginal, as many studios incorporate Sanskrit words into their branding. 3. Unclear meaning: While "Namaste" is a recognizable yoga term, "Nest" may be confusing or not clearly connect to yoga for some people unfamiliar with the idea of a nurturing, cozy space. 4. Too cute/precious: The play on words of "Namaste Nest" could come across as too cutesy or precious to some who prefer a more straightforward, minimalist name. 5. Limiting appeal: Combining Sanskrit and English words could inadvertently limit the perceived appeal to only certain demographics interested in Eastern philosophy/traditions. Overall, while aiming to create a welcoming vibe, the name could be seen as culturally insensitive, trite or not inclusive enough by some critics. As a new parent, I personally appreciate nurturing imagery, but understanding potential concerns is wise for branding. |
| 38 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Serenity Studio | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could envision for the name "Serenity Studio" for a yoga studio: - The name is a bit generic and overused. Many yoga studios use words like "serenity" or "peace" in their names, so it may not stand out very much. - It doesn't give much of a sense of the specific style or philosophy of yoga that will be taught there. Names that hint at the tradition like "Iyengar Yoga Studio" or "Vinyasa Flow Studio" can set clearer expectations. - For a new parent, the word "serenity" may come across as unrealistic or out of touch with the chaos of caring for a baby or young child. A name reflecting strength, energy or rejuvenation may resonate more. - It's a bit bland and doesn't have much personality. A punchier or more creative name could make the studio seem more dynamic and appealing. - If the studio has a specialty like prenatal yoga, mom & baby yoga, etc., the name doesn't reflect that unique offering that new parents may be seeking out. Overall, while inoffensive, the name may not generate much excitement or give a clear sense of what sets this particular yoga studio apart for students like myself looking to rebuild strength and find moments of calm amid the demands of new parenthood. A more distinctive name could be beneficial. |
| 39 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Tranquil Tribe | 5 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could see about the name "Tranquil Tribe" for a yoga studio: • The word "tribe" may come across as culturally appropriative or insensitive, depending on how it's used and perceived. Some may view it as co-opting Indigenous terminology. • While "tranquil" evokes a sense of peace and calm, which aligns with yoga, the combination of "tranquil" and "tribe" could strike some as an odd juxtaposition of words. • The name is quite broad and generic. There are likely many existing yoga studios with similar names using words like "tranquil," "peace," "serenity," etc. It may not stand out enough. • For a new parent, the word "tribe" may seem to exclude rather than include by implying an insular group or clique. As a welcoming space, the name could potentially put off some newcomers. • If taking my yoga very seriously for the physical and mental benefits, I may want a name that conveys more of the discipline, strength and focus involved rather than just tranquility. However, the name could also appeal with its simple, calming feel. But those are some potential criticisms to consider from the perspective of a new parent turning to yoga during this transitional time in my life. Getting input from others would be wise. |
| 40 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Inner Light Yoga | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could envision for the yoga studio name "Inner Light Yoga": 1. Lack of originality: The name "Inner Light" is fairly common and used by many yoga studios already. Some may feel it lacks uniqueness or a distinct identity. 2. Vagueness: The term "inner light" is quite vague and open to interpretation. It doesn't give much specific information about the type of yoga, teaching style, or focus of the studio. 3. Spiritual connotations: Depending on one's beliefs, the "inner light" phrasing could come across as too spiritual, new age, or religious for some people's preferences in just wanting a physical yoga practice. 4. Dated sound: To some ears, especially younger audiences, the name may sound a bit dated, clichéd or reminiscent of yoga from decades past rather than feeling fresh and modern. 5. Lack of personality: Without any other descriptors, "Inner Light Yoga" is a very simple, straightforward name that doesn't necessarily convey much personality, vibe or differentiating factors. However, many would also likely find the name inoffensive, calming and aligned with common yoga principles. As a new parent, the inner light concept could resonate with tapping into my own sources of energy and clarity. But those are some potential criticisms to consider. What's most important is if the name resonates with the studio's vision and community. |
| 41 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Mindful Movements | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could see for the yoga studio name "Mindful Movements": • It's a bit generic/vague - The name doesn't really convey anything unique or memorable about the studio's philosophy, teaching style, etc. Many yoga studios use words like "mindful" in their names. • It doesn't indicate the yoga style - For those looking for a particular type of yoga like vinyasa, hatha, yin, etc., the name gives no clue as to what is offered. • It could be seen as new-agey - Some may view the name as leaning too much into yoga's spiritual/meditative aspects rather than the physical practice. • It's long for a business name - Short and catchy names can be more memorable and impactful. "Mindful Movements" has 4 syllables. • It's not very distinctive - With so many yoga studios, a more unique name could help this one stand out from the crowd. However, the name does convey the ideas of being present and moving the body, which aligns with yoga. As a new parent, I may appreciate the calming, mindful connotations when seeking rejuvenation through yoga. But those are some potential criticisms to consider for the studio name. |
| 42 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Yogic Vibes | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could see with the name "Yogic Vibes" for a yoga studio: • It sounds a bit informal or casual, almost like slang. Some may feel the name doesn't convey the seriousness or tradition of the yoga practice. • The word "vibes" is quite vague and ambiguous. It doesn't really give a clear sense of what the studio is about or what someone can expect. • It could come across as trying too hard to be trendy or hip, rather than focusing on the authentic aspects of yoga. • For some, combining "yogic" which relates to the spiritual practice with the more modern slang term "vibes" may seem like an odd pairing. • The name is fairly generic and doesn't really distinguish this particular studio from others. It lacks a unique quality. However, some may also view "Yogic Vibes" as fun, lighthearted and approachable, which could appeal to certain audiences. As a new parent, I may appreciate a studio name that sounds relaxed rather than overly serious. But those are some potential criticisms to consider about using that name for a yoga business. |
| 43 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Harmony Haven | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could see about the name "Harmony Haven" for a yoga studio: - It's a bit cliche or overused - Words like "harmony" and "haven" are very common in yoga studio names, so it may not stand out as very unique or memorable. - It doesn't give much of a sense of the specific style or approach - The name is quite generic and doesn't hint at whether it specializes in a certain type of yoga like vinyasa, yin, hot yoga etc. - It could be seen as too "woo-woo" or new age for some - Words like "haven" and emphasizing harmony could come across as a bit over-the-top spiritual for people looking for a more fitness-focused yoga experience. - It doesn't localize it to the area/neighborhood - Using a more location-specific name could help it feel grounded and connected to the community it serves. - It's a bit long and unwieldy - As a two-word name, it may be slightly harder to remember or less catchy than a single word name. However, the name does convey a sense of peacefulness and tranquility, which could appeal to those seeking a calming yoga environment, especially as a frazzled new parent. But those are some potential downsides to consider with that studio name choice. Focusing on finding the right balance is key. |
| 44 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Peaceful Postures | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could see for the yoga studio name "Peaceful Postures": • It's a bit generic and doesn't really stand out from other yoga studio names. There are likely many studios with similar "peaceful" or "posture" themed names. • While peaceful postures are part of yoga, the name doesn't really convey the full mind-body-spirit experience that yoga can provide. It focuses just on the physical postures. • For a new parent, the name doesn't really speak to things like rebuilding core strength after pregnancy, reducing stress and anxiety, or connecting with other parents - which could be draws. • The name is a little long and doesn't have a very catchy, memorable quality to it. Shorter punchier names can stick in the mind better. • It has a very traditional, serious vibe when yoga studios are increasingly trying to be warm, approachable spaces, sometimes with fun, lighthearted names. However, the name does convey a sense of serenity and proper alignment, which could appeal to some. But overall, as a new parent, I may want a name that resonates more with my particular yoga needs and goals. A name that's distinctive yet welcoming could be ideal. |
| 45 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Chakra Chimes | 5 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could envision for the yoga studio name "Chakra Chimes": • It may come across as too new age or woo-woo for some people's tastes. The reference to chakras, which are energy centers in the body recognized in some Eastern philosophies, could turn off those looking for a more mainstream, secular yoga experience. • The name is a bit of a mouthful and may be difficult for some to pronounce or remember, especially the "chakra" part if they aren't familiar with that term. • While chimes can create a relaxing, peaceful vibe, the name may evoke more of a spa-like feeling rather than a serious yoga studio for physical practice. • It doesn't give much indication of the specific yoga styles or class types offered at the studio. A more generic name may leave people uncertain about what to expect. • As a new parent, I may worry that a name too heavily steeped in spiritual terminology could make the environment unwelcoming for families or those new to yoga. However, the name could also attract those drawn to the holistic mind-body aspects of yoga and the idea of aligning energy centers. But those are some potential criticisms to consider from a pragmatic new parent's perspective. A simple, accessible name may appeal more broadly. |
| 46 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Blissful Balance | 4 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could envision for the yoga studio name "Blissful Balance": 1. Overused words: The words "blissful" and "balance" are quite common in the yoga/wellness space. Some may find the name a bit cliche or unoriginal. 2. Unrealistic expectations: The name sets a high expectation of achieving complete "bliss" and "balance", which some may find unrealistic or putting too much pressure on the experience. 3. Lack of distinctiveness: Without any other descriptors, the name doesn't really differentiate this particular studio from others that may have similar names revolving around bliss, balance, peace, etc. 4. Too feminine: The soft, gentle phrasing could come across as overly feminine or not inclusive enough for all genders. 5. Unclear offerings: The name doesn't provide any hints about the specific yoga styles, philosophies or environments offered at this studio. However, as a new parent, I may also appreciate the calming, peaceful vibe the name evokes - something I could really use during this hectic life stage! The overall impression is positive, even if the name itself is a bit generic in the yoga space. What matters most is the quality of the classes and instructors. |
| 47 | claude-3-sonnet-20240229 | A new parent using yoga to regain strength and mental clarity | Breathe Bliss | 5 | As a new parent using yoga to regain strength and mental clarity, here are some potential criticisms I could envision for the yoga studio name "Breathe Bliss": 1. Too generic/overused: The words "breathe" and "bliss" are quite common in the yoga/wellness space, so the name may not stand out or seem very unique. 2. Doesn't convey a clear brand identity: The name is pleasant but fairly vague. It doesn't give much insight into the specific style, philosophy or vibe of the studio. 3. Spelling confusion: Some may misread or misspell "Breathe" as "Breath" which could cause branding inconsistencies. 4. Too feminine/soft: The airy, gentle name could come across as overly feminine or lacking an edge, which may not appeal to all potential clients. 5. Unrealistic expectations: Promising "bliss" sets a very high bar and could lead to disappointment if classes don't induce complete euphoria. Overall, while inoffensive, the name plays it quite safe. As a new parent, I may want something punchier that hints at the transformative mind-body benefits I'm seeking through yoga during this major life transition. But those are just my thoughts based on my persona - naming is highly subjective! |
| 48 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Peaceful Poses | 4 | As a retired senior looking to maintain flexibility and social connections, I can understand some potential criticisms people might have about the name "Peaceful Poses" for a yoga studio: 1. Lack of uniqueness - The name is quite generic and doesn't really stand out from other yoga studio names using words like "peaceful" or "poses." It may not be very memorable. 2. Too cliché - Some might find the name a bit clichéd or overused, not feeling very fresh or creative for a yoga studio branding. 3. Limiting scope - The name implies a very calm, gentle style of yoga. Those looking for more vigorous or athletic yoga styles might not feel drawn to a place called "Peaceful Poses." 4. Doesn't convey personality - The name doesn't really give a sense of the studio's unique personality, vibe or special offerings beyond generic peaceful poses. However, the name does convey a welcoming, relaxing atmosphere which could appeal to many, especially seniors like myself looking for a low-key yoga experience. But a more distinctive name might help the studio stand out better in the market. Overall, it's a fairly inoffensive but perhaps unmemorable name choice. |
| 49 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Zen Flow | 4 | As a retired senior looking to maintain flexibility and social connections, I can see a few potential criticisms that some people might have about the name "Zen Flow" for a yoga studio: 1. It's a bit cliché or overused - The words "zen" and "flow" are very commonly associated with yoga, meditation, and that general lifestyle. Some may find it an unoriginal or predictable name choice. 2. Cultural appropriation concerns - The term "zen" has origins in Buddhism and some may view using it to brand a commercial yoga studio as appropriating another culture insensitively. 3. Doesn't reveal much about the studio's approach - The name doesn't give many clues about the specific yoga styles taught, the vibe/atmosphere, or what makes this particular studio unique. 4. Spelling issues - While "zen" is an accepted English spelling of the Japanese word, some may view it as misspelled or not being true to the original term. However, those are just some potential criticisms. Ultimately, a simple name that evokes calmness and fluid movement could resonate well with the yoga community too. As a senior, I don't find the name off-putting myself, but I can understand some of those perspectives. What matters most is the quality of instruction and environment provided. |
| 50 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Soul Stretch | 4 | As a retired senior looking to maintain flexibility and social connections, I can see a few potential criticisms about the name "Soul Stretch" for a yoga studio: 1. It may come across as too spiritual or new age for some people's tastes. The word "soul" has religious/spiritual connotations that could turn off those looking for a more straightforward fitness experience. 2. The name doesn't clearly convey that it's a yoga studio. Someone unfamiliar with yoga may not immediately make the connection between "soul" and the mind-body practice of yoga. 3. It's a bit vague and doesn't give a strong sense of the studio's unique offerings or vibe. A more descriptive name could better attract their target clientele. 4. For seniors specifically, the name may seem a bit too youthful or trendy. We may prefer something that sounds more approachable and inclusive for all ages. However, the name could also resonate well if the studio really leans into creating a soulful, mindful environment. But those are a few potential criticisms to consider from the perspective of maintaining an active social life in retirement. A clear, welcoming name is important for any community space. |
| 51 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Zen Oasis | 4 | As a retired senior looking to maintain flexibility and social connections, I can see a few potential criticisms people might have about the name "Zen Oasis" for a yoga studio: 1. It could be seen as a bit cliché or overused, as words like "zen" and "oasis" are commonly associated with yoga, meditation, and relaxation. Some may find it an unoriginal or uninspired name. 2. The word "oasis" implies an isolated or remote place, which may not accurately represent the studio's location if it is in an urban or populated area. An oasis is typically thought of as a lush spot in the middle of a desert. 3. Depending on the studio's philosophy and approach, the "zen" part of the name could be seen as culturally appropriative if the instructors and practices don't have authentic ties to Zen Buddhism. 4. The combination of the words may come across as trying too hard to evoke a certain vibe or feeling rather than letting the studio's quality speak for itself. However, those are just some potential criticisms. Many people may also find the name calming, welcoming, and giving the right first impression for a yoga space. As with any business name, it will likely resonate differently with different potential customers. |
| 52 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Tranquil Transitions | 4 | As a retired senior looking to maintain flexibility and social connections, I can understand some potential criticisms about the name "Tranquil Transitions" for a yoga studio: 1. It may sound a bit vague or generic. The name doesn't immediately convey what makes this particular yoga studio unique or special. 2. Some people may find the name a bit too "new age" sounding with words like "tranquil" and "transitions." This could turn off those looking for a more straightforward, no-nonsense approach to yoga. 3. The name is a bit long and could be difficult to remember or fit nicely on signage/marketing materials. 4. There's no obvious connection to yoga in the name itself. A name more directly tied to yoga poses, philosophy, etc. may resonate better. However, the name does evoke a sense of peacefulness and change/growth, which aligns with the principles of yoga. And as a senior, I appreciate names that sound calming rather than overly intense or aggressive. But a punchier, more distinctive name may be more memorable and attention-grabbing for a wider range of potential students. Just a few perspectives to consider! |
| 53 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Namaste Nest | 5 | As a retired senior looking to maintain flexibility and social connections, I can understand some potential criticisms people might have about the name "Namaste Nest" for a yoga studio: 1. Cultural appropriation: The word "namaste" is a Sanskrit greeting used in Indian cultures. Some may feel that using it in a business name appropriates or commodifies another culture in an insensitive way. 2. Cliché or cheesy: Combining "namaste" with "nest" could come across as a bit cliché or cheesy to some people. They may see it as trying too hard to sound spiritual or zen. 3. Unclear meaning: Those unfamiliar with yoga terminology may not understand the meaning behind "namaste" or why it's paired with "nest." The name could be confusing to newcomers. 4. Too niche: The name leans heavily into yoga/spiritual concepts. Some may feel it's too niche and could turn away those looking for a more general fitness studio. However, as someone interested in maintaining an active social life, I personally don't find too much issue with the name. But I can understand if some view it as culturally insensitive or too focused on yoga concepts. An inclusive, welcoming environment is most important to me. |
| 54 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Serenity Studio | 4 | As a retired senior looking to maintain flexibility and social connections, here are some potential criticisms I could see people having about the name "Serenity Studio" for a yoga studio: - It's a bit generic and cliche. Many yoga and wellness businesses use words like "serenity" in their names, so it doesn't really stand out. - For some seniors, the word "serenity" may come across as too new-agey or woo-woo. A more straightforward name could appeal to a wider demographic. - The name doesn't give any indication of what style or types of yoga they specialize in, if any. Seniors may want to know upfront if it's more gentle, therapeutic yoga for example. - It doesn't reference anything about the location, neighborhood or community it's part of, which could make it feel less connected to the area. - As someone looking for social connections, the name is pretty impersonal. A name that hints at community, gathering or relationships could be more inviting. However, the name does convey a sense of peacefulness which could appeal to seniors seeking relaxation through yoga. But some more specificity or creativity in the name could help it stand out more. Those are just my thoughts as a retired person evaluating yoga studio names. |
| 55 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Tranquil Tribe | 4 | As a retired senior looking to maintain flexibility and social connections, I can see a few potential criticisms about the name "Tranquil Tribe" for a yoga studio: 1. The word "tribe" may come across as culturally appropriative or insensitive, especially if the owners are not from an indigenous culture that traditionally uses that term. 2. Some people may find the name a bit cheesy or too new-agey sounding. The pairing of "tranquil" and "tribe" could strike some as trying too hard to be trendy. 3. The name doesn't explicitly convey that it's a yoga studio, so there could be some confusion about what services are offered there for those unfamiliar with it. 4. Older demographics may not relate as much to the "tribe" terminology and prefer something that sounds more traditional or straightforward. However, the name could also appeal to many as sounding peaceful, community-oriented and inviting for yoga practice. But those are a few potential criticisms that come to mind from the perspective of maintaining an active social life in retirement. What's most important is that the studio cultivates a welcoming environment. |
| 56 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Inner Light Yoga | 4 | As a retired senior looking to maintain flexibility and social connections, I can share some potential criticisms about the name "Inner Light Yoga" from my perspective: 1. Vagueness: The name "Inner Light" is quite vague and doesn't immediately convey what kind of yoga or approach the studio offers. It could be confusing for someone unfamiliar with the studio. 2. Lack of distinctiveness: There may be other yoga studios or businesses with similar names using the words "inner" and "light," which could make it harder to stand out and be memorable. 3. Religious connotations: Depending on someone's background, the term "inner light" could have spiritual or religious connotations that may not resonate with everyone or accurately reflect the studio's philosophy. 4. Dated sound: To some ears, especially younger demographics, the name could sound a bit dated or cliché, lacking a fresh, modern vibe. However, these are just potential criticisms. The name could also resonate positively with many people looking for an introspective, enlightening yoga experience. As a senior, I may appreciate the calming, peaceful feel of the name. Context and branding could help address some of the perceived drawbacks as well. But you raised a fair question about critically evaluating business names. |
| 57 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Mindful Movements | 4 | As a retired senior looking to maintain flexibility and social connections, I can see a few potential criticisms people might have about the name "Mindful Movements" for a yoga studio: 1. It's a bit generic - The words "mindful" and "movements" are commonly used in the yoga/wellness space, so the name may not stand out very much. Some might find it uninspired or bland. 2. Doesn't convey a unique selling point - The name doesn't really give a sense of what makes this particular studio different or special compared to others. It's descriptive but not distinctive. 3. Could be seen as new-agey - Terms like "mindful" have associations with new age philosophies that some people may find off-putting or not take seriously. 4. Doesn't indicate yoga specifically - While the name works for a wellness/movement studio broadly, it doesn't overtly signal that this is a yoga studio specifically, which could lead to some confusion. However, the name does have some positive aspects - it conveys a focus on present moment awareness and conscious movement, which aligns with yoga's principles. And it has a calm, soothing sound. But those are just my thoughts as someone looking for social engagement and gentle exercise in retirement. A punchier or more distinctive name could potentially attract more attention. |
| 58 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Yogic Vibes | 4 | As a retired senior looking to maintain flexibility and social connections, I can see a few potential criticisms people might have about the name "Yogic Vibes" for a yoga studio: 1. It sounds a bit too informal or casual. Some may feel the name doesn't convey the discipline and ancient traditions associated with yoga practice. 2. The word "vibes" has become somewhat overused slang, especially among younger generations. Older yoga practitioners may find it grating or not resonating with the spiritual aspects of yoga. 3. It's a bit vague - the name doesn't really give a sense of what specific style or philosophy of yoga the studio follows. This could be a drawback for those looking for a particular tradition. 4. The alliteration of "yogic vibes" could come across as trying too hard to be catchy or clever with wordplay rather than having a meaningful name. However, the name could also appeal to some looking for a yoga studio with a laid-back, welcoming atmosphere without being too serious or traditional in its branding. As a retired senior, I may appreciate an approachable name that doesn't seem too rigid or intimidating. Overall though, the name may be polarizing - really resonating with some but turning others off. Clear messaging about the studio's actual offerings would be important to back up the name. |
| 59 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Harmony Haven | 4 | As a retired senior looking to maintain flexibility and social connections, I can understand some potential criticisms about the name "Harmony Haven" for a yoga studio: 1. It may sound a bit clichéd or generic. There are a lot of yoga studios out there with names involving words like "harmony," "peace," "serenity," etc. Some people might find it unoriginal. 2. The word "haven" implies a place of refuge or escape, when yoga is as much about confronting oneself and finding inner strength as it is about relaxation. 3. Depending on the vibe and offerings of the particular studio, the name may come across as too new age-y or woo-woo for some people's tastes. 4. It doesn't really give much of a sense of what style or philosophy of yoga the studio focuses on. That said, the name does convey a sense of tranquility which could appeal to those looking for a calming yoga experience. And alliteration with the "h" sounds gives it a nice flow. Overall, while not the most unique name, I don't think it's overtly off-putting either. The quality of instruction and community will likely matter more than the name itself. But those are just my retired senior's two cents! |
| 60 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Peaceful Postures | 4 | As a retired senior looking to maintain flexibility and social connections, I can see a few potential criticisms about the name "Peaceful Postures" for a yoga studio: 1. It may sound a bit generic or unoriginal. There are likely many yoga studios out there with names playing on the ideas of peace, tranquility, and poses/postures. 2. Some might find it a bit too on-the-nose or cliched for a yoga studio name. The name states the obvious goal of yoga in a rather literal way. 3. It doesn't really differentiate this particular studio or give a sense of what might be unique about the instructors, classes, or philosophy there. 4. The name is a bit long and not extremely catchy or memorable compared to punchier single word names. However, the name does convey a pleasant, calming vibe that aligns with what yoga practitioners often seek. It succinctly captures two core aspects of yoga. So while not the most creative name, it gets the essence across in an inoffensive way. But those are a few potential criticisms I could envision about the studio name not standing out enough or being too obvious. What's your take on it? |
| 61 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Chakra Chimes | 5 | As a retired senior looking to maintain flexibility and social connections, I can understand some potential criticisms about the name "Chakra Chimes" for a yoga studio: 1. Cultural appropriation concerns: The term "chakra" comes from ancient Hindu traditions and using it in a commercial business name could be seen as appropriating another culture in an insensitive way. 2. Too new age/woo-woo: Some people may find the name a bit too "new agey" or associated with fringe spiritual beliefs that turn them off. They may prefer a more grounded, mainstream name. 3. Unclear meaning: For those unfamiliar with yoga terminology, "chakra chimes" may not convey much meaning about it being a yoga studio. A more descriptive name could be clearer. 4. Difficult pronunciation: Depending on the demographics of the area, some seniors or others may have difficulty pronouncing "chakra" correctly, which could be a barrier. However, as someone open to exploring yoga for flexibility and social benefits, I personally don't find the name overly off-putting. But I can understand if others may have some of the criticisms mentioned above about the cultural appropriation aspects or lack of clarity in the name. An open discussion could help address those concerns. |
| 62 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Blissful Balance | 4 | As a retired senior looking to maintain flexibility and social connections, I can see a few potential criticisms that some people might have about the name "Blissful Balance" for a yoga studio: 1. It sounds a bit clichéd or overused. Words like "bliss" and "balance" are very common in the yoga/wellness space, so the name may not stand out as very unique or memorable. 2. The name is a bit vague and doesn't give a clear sense of what makes this particular studio different or special compared to others. It's a pleasant name but doesn't convey much about the studio's specific approach or offerings. 3. Some may find the name too flowery or new-agey sounding. While yoga has spiritual roots, some people prefer a more grounded, no-nonsense style when it comes to branding and marketing for fitness activities. 4. The name implies a very serene, relaxing vibe. While balance is certainly part of yoga, some may want a name that also conveys the physical intensity, strength and challenge involved in many yoga practices. However, these are just potential criticisms. The name could resonate well with the right audience looking for a blissful, balanced yoga experience. As a senior, I don't mind the peaceful, calming connotations personally. But a more distinctive name could help the studio stand apart in a crowded market. |
| 63 | claude-3-sonnet-20240229 | A retired senior looking to maintain flexibility and social connections | Breathe Bliss | 4 | As a retired senior looking to maintain flexibility and social connections, here are some potential criticisms I could see people having about the yoga studio name "Breathe Bliss": 1. It sounds a bit cheesy or over-the-top. The word "bliss" has a very New Age, spiritual connotation that some people may find off-putting or not take seriously. 2. It doesn't really give a sense of what style or type of yoga they offer. Names that indicate the yoga tradition like "Iyengar Yoga Studio" or "Vinyasa Flow Yoga" can be more informative. 3. For seniors, the name may come across as too youthful or targeted at a younger crowd. We may prefer something that sounds a bit more grounded or welcoming to all ages. 4. It's a bit long for a business name. Shorter, punchier names can be easier to remember and have more impact sometimes. However, I do think "Breathe Bliss" has a nice ring to it and gets across the relaxing, rejuvenating benefits of yoga practice. But those are some potential downsides to consider from the perspective of maintaining social connections across age groups. A focus group could provide valuable feedback. |
| 64 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Peaceful Poses | 4 | As an athlete who incorporates yoga for cross-training and injury prevention, I can think of a few potential criticisms people might have about the name "Peaceful Poses" for a yoga studio: 1. Lack of creativity/originality: The name is quite generic and doesn't really stand out. There are likely many other yoga studios with similar names using words like "peaceful" and "poses." 2. Doesn't convey the athletic/cross-training aspect: The name makes it sound very calm and relaxation-focused, which is great for certain types of yoga. But for those looking to use yoga for more athletic cross-training purposes, the name may not accurately represent that style of practice. 3. Too soft/feminine: Some may perceive the name as being a bit too soft, gentle or feminine with words like "peaceful." This could potentially turn off male athletes or those seeking a more intense, physical yoga practice. 4. Limiting: By naming it just "Peaceful Poses," it somewhat pigeonholes the studio into just offering gentle, relaxing styles of yoga. As the business grows, they may want to expand into more vigorous styles which the name doesn't encompass. Overall, while inoffensive, the name plays it very safe and may not help the studio stand out or attract the cross-training athlete crowd as effectively as a more unique, athletic-oriented name could. But it depends on the studio's specific branding and clientele goals. |
| 65 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Zen Flow | 4 | As an athlete who incorporates yoga for cross-training and injury prevention, I can see a few potential criticisms people might have about the name "Zen Flow" for a yoga studio: 1. Overused words: The words "zen" and "flow" are quite commonly used in the yoga/wellness space. Some may view it as an unoriginal or clichéd name that doesn't stand out. 2. Cultural appropriation concerns: The word "zen" has roots in Buddhism and Eastern philosophy. There could be criticism that a Western yoga studio is appropriating this term from another culture. 3. Vague meaning: While "flow" relates to the fluid movement in yoga, "zen" is a bit more ambiguous when applied to a fitness studio name. The combination may not clearly convey the studio's offerings. 4. Disconnect for athletes: As an athlete, I might perceive "Zen Flow" as sounding very serene and relaxation-focused, which could make high-intensity athletic types of yoga classes seem out of place under that name. That said, the name taps into ideals of calmness and movement that yoga cultivates. But some rebranding tweaks could potentially make it more unique and inclusive of yoga's physical and spiritual elements for all demographics. An athlete's perspective values names that balance zen with intensity. |
| 66 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Soul Stretch | 5 | As an athlete who incorporates yoga for cross-training and injury prevention, I can see a few potential criticisms people might have about the name "Soul Stretch" for a yoga studio: 1. It could come across as too spiritual or new age for some people's tastes. The word "soul" has religious/spiritual connotations that may turn off those looking for a more fitness-focused yoga experience. 2. It's a bit vague and doesn't explicitly communicate that it's a yoga studio. Someone unfamiliar with yoga terminology may not immediately associate "Soul Stretch" with yoga classes. 3. The name is fairly common/generic. There are likely other studios with similar "soul/stretch" names, so it may not be very distinctive or memorable. 4. For athletes, the name doesn't really convey the physical/athletic benefits of yoga. Something more straightforward like "Yoga for Athletes" could appeal more to that demographic. However, the name could also resonate well with the right audience - those seeking a yoga practice that goes beyond just the physical asanas and taps into the mindful, spiritual side as well. But you raise a fair point that it may not ideally capture an athletic, cross-training approach to yoga. A more fitness-oriented name could be beneficial from a marketing perspective. |
| 67 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Zen Oasis | 4 | As an athlete who incorporates yoga for cross-training and injury prevention, I can think of a few potential criticisms for the yoga studio name "Zen Oasis": 1. Overused words: The words "zen" and "oasis" are quite commonly used in the yoga/wellness industry. Some may feel the name is a bit cliché or unoriginal. 2. Cultural appropriation concerns: The word "zen" has roots in Buddhism and Eastern philosophy. There could be criticism that a Western business is co-opting this term without proper understanding or respect for its origins. 3. Misleading expectations: An "oasis" implies a lush, peaceful escape. If the actual studio space or experience does not live up to those tranquil connotations, customers may feel misled. 4. Lack of distinctiveness: With so many yoga studios using similar calming, nature-inspired names, "Zen Oasis" may not stand out enough in the market. 5. Irrelevance to athletics: As an athlete, I may question if a name focused solely on serenity properly captures the physical intensity and cross-training benefits I seek from yoga. However, these are just potential criticisms. The name could resonate well with the right branding and high-quality yoga experience behind it. As an athlete, I'd be most concerned with the studio's ability to support my training needs. |
| 68 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Tranquil Transitions | 4 | As an athlete who incorporates yoga for cross-training and injury prevention, I can think of a few potential criticisms people might have about the name "Tranquil Transitions" for a yoga studio: 1. Lack of clarity: The name doesn't explicitly convey that it's a yoga studio. Some people might find it vague or unclear what services are offered. 2. Too generic: The words "tranquil" and "transitions" are quite generic and could apply to many different businesses or contexts beyond yoga. Some may feel the name lacks distinctiveness. 3. Misleading expectations: The word "tranquil" sets an expectation of a very calm, serene atmosphere. If the studio offers more vigorous or athletic yoga styles, some may feel the name doesn't accurately represent the experience. 4. Transition implies temporary: The inclusion of "transitions" could imply the studio or its offerings are temporary or transitional in nature, rather than an established long-term business. 5. Difficult to pronounce/remember: Some may find the combination of "tranquil" and "transitions" a bit of a mouthful or hard to remember easily. However, these are just potential criticisms. The name could resonate well with the right branding and if it aligns with the studio's philosophy and offerings. As an athlete, I personally don't find it off-putting, but clarity in marketing would be important. |
| 69 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Namaste Nest | 5 | As an athlete who incorporates yoga for cross-training and injury prevention, I can see a few potential criticisms people might have about the name "Namaste Nest" for a yoga studio: 1. Cultural appropriation: The word "namaste" is a Sanskrit greeting used in yoga that has spiritual/religious roots in Indian culture. Some may view using it in a commercial business name as appropriating or commodifying another culture. 2. Cliché: Namaste is a very commonly used word in the Western yoga world. Combining it with "nest" could come across as an overly cute, clichéd name that lacks originality. 3. Confusing meaning: While "namaste" means something like "I bow to you," a "nest" implies a bird's nest or nesting place. The two words don't obviously connect in meaning. 4. Limited appeal: The name leans heavily into yoga terminology, which could make the studio seem niche and unwelcoming to those new to yoga or looking for a more mainstream fitness experience. However, the name could also resonate positively with the yoga community by evoking ideas of peace, sanctuary and community. Ultimately, a studio name's success depends on the quality of the space and instruction. But those are some potential criticisms to consider about "Namaste Nest." |
| 70 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Serenity Studio | 3 | As an athlete who incorporates yoga for cross-training and injury prevention, here are some potential criticisms I could see about the name "Serenity Studio" for a yoga studio: 1. Lack of distinctiveness: The name "Serenity Studio" is quite generic and doesn't really stand out or differentiate itself from other yoga studios. Many studios use words like "serenity," "peace," or "tranquility" in their names. 2. Doesn't convey athleticism: For someone like me looking to use yoga for cross-training and preventing injuries related to my sport, the name "Serenity Studio" doesn't really convey an athletic or physical focus. It sounds more like a studio geared towards relaxation and meditation rather than physical training. 3. Potentially misleading: If the studio actually offers more vigorous, athletic styles of yoga like power yoga or Ashtanga, the "Serenity" name could be misleading and give the wrong impression of what's offered. 4. Limited branding potential: A very generic name like "Serenity Studio" doesn't provide much opportunity for creative branding, logos, slogans etc. A more distinctive name could better showcase the studio's unique identity. Overall, while "Serenity Studio" sounds pleasant, as an athlete I might prefer a name that better captures the physical and cross-training aspects I'm looking for from yoga to complement my sport. A name that hints at strength, power, and athletic focus could be more appealing. |
| 71 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Tranquil Tribe | 4 | As an athlete incorporating yoga for cross-training and injury prevention, here are some potential criticisms I could see about the yoga studio name "Tranquil Tribe": 1. Cultural appropriation concerns: The word "tribe" has cultural connotations, especially for indigenous communities. Some may view using it in a commercial yoga studio name as appropriating or commodifying their cultures. 2. Exclusionary undertones: The name evokes a sense of an insular group or clique. This could make some potential clients feel unwelcome or like they won't fit in if they don't meet certain criteria. 3. Cliche or overused: Words like "tranquil" are very commonly used in the yoga/wellness space. The name could come across as unoriginal or cliche. 4. Doesn't convey the athletic/cross-training aspect: For those looking to yoga specifically for cross-training purposes, the name doesn't really signal that focus. It leans more toward the relaxation/meditation side. 5. Tribal stereotyping: Referring to a "tribe" could play into outdated stereotypes about how yoga/meditation practices are mystical or primitive. However, the name could also resonate positively with some looking for a sense of community. But those are some potential criticisms to consider from an athletic cross-training perspective. A more neutral name may be advisable. |
| 72 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Inner Light Yoga | 4 | As an athlete incorporating yoga for cross-training and injury prevention, I can think of a few potential criticisms people might have about the name "Inner Light Yoga" for a yoga studio: 1. Lack of originality - The name "Inner Light" is a fairly common phrase used in yoga and spiritual contexts, so some may view it as an unoriginal or clichéd name choice. 2. Unclear meaning - For those unfamiliar with yoga terminology, the phrase "inner light" may not immediately convey what services the studio offers. A more descriptive name could provide better clarity. 3. Religious connotations - The term "inner light" has connections to certain spiritual traditions like Quakerism. Some may interpret it as having religious overtones that could alienate certain potential clients. 4. Lack of distinguishing factors - Without any other distinguishing words, the name doesn't communicate what may set this particular yoga studio apart from others in terms of teaching styles, specialties, amenities, etc. 5. Branding limitations - A very generic name like "Inner Light Yoga" could make it harder for the studio to create a unique, memorable brand identity and stand out in the market. However, these are just potential criticisms. The name's suitability ultimately depends on the studio's branding goals, target audience, and how well the name resonates with the vision and philosophy behind the business. Many successful yoga studios use fairly generic "yoga" names as well. |
| 73 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Mindful Movements | 4 | As an athlete incorporating yoga for cross-training and injury prevention, here are some potential criticisms I could see people having about the name "Mindful Movements" for a yoga studio: 1. Lack of distinctiveness: The name is quite generic and doesn't really stand out or convey anything unique about the studio's offerings or philosophy. There could be concerns that it blends in with other studios. 2. Overused words: The words "mindful" and "movements" are very commonly used in the yoga/wellness space, so some may find the name a bit cliche or unoriginal. 3. Doesn't convey a specific style: The name doesn't indicate if the studio specializes in any particular style of yoga like vinyasa, iyengar, hot yoga etc. This could be seen as a missed opportunity. 4. Too broad: The name is so broad that it doesn't give much of a sense of what makes this particular studio unique or special compared to others. 5. Disconnected from athletics: As an athlete using yoga for cross-training, I may want a name that better connects the studio's approach to serving athletic clients and complementing other training. However, the name is simple, easy to remember, and gets the general essence of yoga across. But those are some potential critiques I could envision from the perspective of looking for a yoga studio as an athlete. A more distinctive name could help the studio stand out. |
| 74 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Yogic Vibes | 4 | As an athlete who incorporates yoga for cross-training and injury prevention, I can offer a perspective on the proposed yoga studio name "Yogic Vibes": Potential Criticisms: 1. Vagueness: The term "vibes" is quite vague and doesn't really convey what the studio is about or what kind of yoga they specialize in. It could be seen as too generic or lacking clarity. 2. Trendiness: Using the word "vibes" could come across as trying too hard to be trendy or hip, which some people may find off-putting or inauthentic for a yoga studio. 3. Lack of Seriousness: Yoga is often seen as a discipline that requires focus and dedication. The name "Yogic Vibes" may not convey the seriousness or professionalism that some practitioners look for in a studio. 4. Cultural Appropriation: Depending on the background of the studio owners, some may view the use of the word "yogic" as cultural appropriation if they are not deeply rooted in the traditional practices and philosophies of yoga. While the name is catchy and memorable, it's important to consider how it might be perceived by potential clients and whether it aligns with the studio's values and approach to yoga. A more descriptive or meaningful name could better represent the studio's offerings and philosophy. |
| 75 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Harmony Haven | 4 | As an athlete who incorporates yoga for cross-training and injury prevention, I can think of a few potential criticisms people might have about the name "Harmony Haven" for a yoga studio: 1. It sounds a bit clichéd or cheesy. Some may feel the name is trying too hard to evoke a sense of peace and tranquility through the words "harmony" and "haven." 2. It doesn't really convey anything about yoga specifically. The name could just as easily be used for a spa, meditation retreat, or new age store. It doesn't clearly identify it as a yoga studio. 3. The alliteration of "Harmony Haven" makes it a bit of a tongue-twister that could be difficult for some to pronounce or remember easily. 4. For more athletic yogis like myself, the name may come across as too soft or gentle, not accurately representing the physical intensity and challenge that yoga can provide. 5. It's a fairly common and generic name that many existing yoga studios may have already used, lacking originality or distinctiveness. However, those are just some potential criticisms. The name could certainly resonate well with the right branding and audience seeking a serene, welcoming yoga environment. But those are some drawbacks I could foresee from an athlete's perspective. Let me know if you need any other feedback! |
| 76 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Peaceful Postures | 4 | As an athlete incorporating yoga for cross-training and injury prevention, here are some potential criticisms I could see about the yoga studio name "Peaceful Postures": 1. Lack of distinctiveness: The name "Peaceful Postures" may be seen as generic or unoriginal, as it uses common words associated with yoga. It may not stand out enough from other yoga studio names. 2. Limited scope: The name implies a focus solely on gentle, relaxing styles of yoga. This could deter athletes or those seeking more vigorous, athletic styles of yoga practice. 3. Passive connotation: The words "peaceful" and "postures" have a relatively passive tone. For athletes, this name may not convey the active, physical nature of yoga as a form of cross-training. 4. Potentially misleading: If the studio offers a wide variety of yoga styles beyond just gentle classes, the name may misrepresent the full range of classes available. 5. Lack of personal branding: The name doesn't incorporate the founder's name or any unique personal brand elements that could help it stand out. However, these are just potential criticisms. The name could still resonate well if the branding and studio offerings align with cultivating a peaceful, calming environment focused on precise posture work. As an athlete, having a balance of vigorous and restorative practices is valuable. |
| 77 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Chakra Chimes | 5 | As an athlete who incorporates yoga for cross-training and injury prevention, I can share some potential criticisms about the name "Chakra Chimes" for a yoga studio: 1. Cultural appropriation: The term "chakra" is derived from ancient Hindu traditions and using it in a commercial context could be seen as culturally insensitive or appropriating sacred concepts for profit. 2. Lack of clarity: The name "Chakra Chimes" may not immediately convey that it is a yoga studio. Some people may find the name confusing or unclear about the services offered. 3. Limiting appeal: The use of Sanskrit terminology like "chakra" may appeal to a niche audience already familiar with yoga concepts, but could potentially alienate or turn off newcomers or those unfamiliar with the terminology. 4. Clichéd or cringeworthy: Some may view the combination of "chakra" and "chimes" as a bit clichéd or trying too hard to sound mystical or spiritual, which could come across as inauthentic or off-putting. 5. Pronunciation issues: Depending on the audience, some may struggle with the pronunciation of "chakra," which could create a barrier or confusion. Overall, while the name aims to evoke a sense of spirituality and mindfulness associated with yoga, it could potentially face criticism for cultural appropriation, lack of clarity, limited appeal, or being perceived as clichéd or inauthentic by some audiences. |
| 78 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Blissful Balance | 4 | As an athlete incorporating yoga for cross-training and injury prevention, here are some potential criticisms I could see about the name "Blissful Balance" for a yoga studio: • It sounds a bit cliché or overused. Many yoga studios go for names implying peace, balance, bliss, etc. So it may not stand out very much. • The word "blissful" has a very soft, gentle connotation. Some athletes may prefer a name that sounds more empowering, energetic or strength-focused for their cross-training purposes. • While "balance" relates well to the physical and mental benefits of yoga, it is a bit vague on its own. The name doesn't explicitly convey that this is a yoga studio. • For injury prevention purposes, a name focused more on alignment, stability, mobility or recovery could appeal to the athletic cross-training crowd. • The name is long for a studio name at four syllables. Shorter names can be more memorable and impactful. However, the name does successfully evoke positive feelings of tranquility and centeredness that many seek from yoga. But for cross-training athletes, a more unique and active name could potentially resonate better. Overall critique would depend on the studio's branding and target clientele. |
| 79 | claude-3-sonnet-20240229 | An athlete incorporating yoga for cross-training and injury prevention | Breathe Bliss | 5 | As an athlete incorporating yoga for cross-training and injury prevention, here are some potential criticisms I could envision for the yoga studio name "Breathe Bliss": 1. Lack of distinctiveness: The name "Breathe Bliss" is quite generic and doesn't really stand out or differentiate the studio from others. There are likely many studios with similar names focused on breathing and finding bliss through yoga. 2. Overly cliche: Some may find the name a bit cliche or overused, playing into common yoga stereotypes about breathing and bliss. It doesn't really communicate anything unique about the studio's approach or offerings. 3. Unclear focus: The name doesn't provide much clarity on whether the studio specializes in certain yoga styles, has a particular athletic/cross-training emphasis, or caters to certain clientele. As an athlete, I may wonder if this is truly a good fit for my cross-training needs. 4. Too narrow: By emphasizing just the breathing and blissful aspects, the name could come across as too narrow and not inclusive of the full mind-body experience and physical challenges of yoga practice. 5. Lack of memorability: A very simple name like "Breathe Bliss" may not be particularly memorable or catchy, making it harder for the studio to build strong brand recognition. Overall, while inoffensive, the name plays it quite safe and could be seen as generic or cliche by some. As an athlete, I may seek out a studio name that better conveys its athletic approach, variety of styles, or unique value proposition. But reasonable people may disagree on naming preferences. |
| 80 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Peaceful Poses | 3 | Some criticisms people might have about the name "Peaceful Poses" for a yoga studio could include: 1. **Generic and Unoriginal**: The name might be seen as too generic or lacking in creativity, as it uses common words associated with yoga and relaxation, potentially making it less memorable. 2. **Lack of Distinctiveness**: It might not stand out in a crowded market of yoga studios, which could make it harder for potential clients to remember or differentiate it from others. 3. **Limited Appeal**: The name may not convey a sense of inclusivity or diversity, potentially alienating those who are looking for a more dynamic or varied yoga experience that includes more than just peaceful or gentle practices. 4. **Assumptions about Class Types**: It might give the impression that the studio only offers gentle or restorative yoga classes, which could deter those looking for more vigorous or advanced practices. 5. **Cliché**: Some might find the name cliché, as it uses typical yoga-related terms that might not resonate with those looking for something unique or innovative. |
| 81 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Zen Flow | 4 | Some criticisms people might have about the name "Zen Flow" for a yoga studio could include: 1. **Overused Terms**: Both "Zen" and "Flow" are commonly used in the wellness industry, which might make the name seem generic or lacking originality. 2. **Cultural Appropriation**: The term "Zen" originates from a specific tradition of Buddhism, and some might feel that its use in a commercial context could be seen as cultural appropriation if not handled with respect and understanding. 3. **Lack of Specificity**: The name doesn't convey any unique aspects of the studio, such as its location, style of yoga, or any special features it offers, which might make it less memorable. 4. **Potential Misunderstanding**: People unfamiliar with yoga or meditation might not immediately understand what "Zen Flow" implies, potentially limiting its appeal to newcomers. 5. **Stereotypical Imagery**: The name might evoke stereotypical images of what yoga is, potentially alienating those who are looking for a more diverse or modern approach to the practice. As someone who values stress relief and work-life balance, I'd likely be looking for a studio that offers a unique and authentic experience, so these potential criticisms might be worth considering. |
| 82 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Soul Stretch | 4 | As a busy corporate executive, I can appreciate the importance of a name that resonates well with its audience. Some potential criticisms of the name "Soul Stretch" might include: 1. **Vagueness**: The name could be seen as too abstract or not descriptive enough of the services offered. People might not immediately associate "Soul Stretch" with yoga or understand what the studio specializes in. 2. **Cliché**: Some might feel that "Soul Stretch" is a bit overused or falls into the category of typical wellness jargon, lacking originality in a crowded market. 3. **Spiritual Overtones**: The word "Soul" might not appeal to everyone, particularly those who prefer a more fitness-oriented or secular approach to yoga. 4. **Target Audience**: Depending on the demographic the studio wants to attract, the name might not resonate with more pragmatic individuals who might be looking for tangible benefits like flexibility or stress relief, rather than a spiritual experience. 5. **Branding Challenges**: If the studio wants to expand its offerings beyond yoga, the name might limit its perceived scope or require additional branding efforts to clarify its services. Despite these potential criticisms, the name could also attract those seeking a deeper, more holistic yoga experience, which could be a strong selling point in itself. |
| 83 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Zen Oasis | 5 | Some criticisms people might have about the yoga studio name "Zen Oasis" might include: 1. **Cliché**: The name might be seen as too predictable or overused, as "Zen" and "Oasis" are common terms associated with relaxation and tranquility. 2. **Lack of Uniqueness**: It may not stand out in a crowded market of wellness and yoga studios, making it difficult to differentiate from competitors. 3. **Cultural Appropriation**: Some might argue that using "Zen," a term rooted in specific cultural and religious traditions, could be seen as appropriating without a deep understanding or respect for its origins. 4. **Expectations**: The name sets a high expectation for a serene and transformative experience, which could lead to disappointment if the studio doesn't meet those standards. 5. **Generic Appeal**: While aiming for broad appeal, it might not convey the specific style or unique offerings of the studio, potentially missing the opportunity to attract a niche audience. Balancing a name that resonates with authenticity and distinctiveness is crucial, especially in a wellness-focused business. |
| 84 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Tranquil Transitions | 4 | Some potential criticisms of the name "Tranquil Transitions" for a yoga studio could include: 1. **Lack of Uniqueness**: The name might be seen as generic or overused, blending in with other wellness and yoga businesses that use similar calming or transitional themes. 2. **Ambiguity**: While "Tranquil Transitions" conveys a sense of peace and change, it might not immediately communicate that it's a yoga studio, which could be confusing to potential clients. 3. **Complexity**: The name might be perceived as a bit of a mouthful, making it slightly harder to remember or pronounce compared to simpler names. 4. **Serious Tone**: The name might come off as overly serious or formal, potentially alienating those who are looking for a more fun or casual yoga experience. 5. **Target Audience**: If the studio aims to attract a younger or more diverse clientele, the name might not resonate as well with those groups compared to something more modern or edgy. Balancing a name that conveys the essence of the studio while standing out in the market can be challenging, but it's crucial for brand identity. |
| 85 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Namaste Nest | 5 | Some criticisms people might have about the name "Namaste Nest" could include: 1. **Cultural Appropriation**: Some might feel that using "Namaste," a word deeply rooted in Indian culture and spirituality, could be seen as cultural appropriation, especially if the studio doesn't authentically incorporate or respect the cultural origins of yoga. 2. **Cliché**: The term "Namaste" is often used in Western yoga contexts, and some might see it as overused or cliché, potentially lacking originality or depth. 3. **Connotations of Exclusivity**: The word "Nest" might imply a cozy or exclusive environment, which could be off-putting to those who prefer a more open or inclusive space. 4. **Ambiguity**: The name might not clearly convey that it's a yoga studio, potentially leading to confusion about what services or experiences are offered. 5. **Stereotypical**: Some might feel the name plays into stereotypical ideas of what a yoga studio should be, rather than offering a unique or innovative approach. |
| 86 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Serenity Studio | 4 | Some potential criticisms of the name "Serenity Studio" for a yoga studio might include: 1. **Genericness**: The name might be seen as too generic or overused, lacking uniqueness or distinctiveness that could help the studio stand out in a crowded market. 2. **Vagueness**: Without a direct reference to yoga, people might not immediately associate "Serenity Studio" with yoga, leading to potential confusion about what services are offered. 3. **Lack of Cultural Connection**: Some might feel that the name doesn't reflect the cultural roots of yoga, which could be important to those who value authenticity in their practice. 4. **Overpromise**: The name "Serenity" sets a high expectation for tranquility and peace, which could lead to disappointment if the studio environment or classes don't meet these expectations. 5. **Competition**: If there are other businesses in the area with similar names, it could lead to brand confusion or difficulty in establishing a unique identity. As a busy executive, I often look for unique experiences that offer a break from the usual, so a name that captures something distinctive about the studio might be more appealing. |
| 87 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Tranquil Tribe | 4 | Some criticisms people might have about the name "Tranquil Tribe" for a yoga studio could include: 1. **Cultural Sensitivity**: The use of the word "tribe" might raise concerns about cultural appropriation, as it can have significant cultural and historical connotations for Indigenous peoples. 2. **Vagueness**: The name might be seen as too generic or vague, not clearly conveying what makes the studio unique or what specific type of yoga experience it offers. 3. **Trendiness**: Some might feel that the name tries too hard to be trendy or hip, which could detract from the authenticity or seriousness of the yoga practice. 4. **Clarity**: Potential clients might not immediately associate the name with a yoga studio, as "Tranquil Tribe" could apply to a variety of wellness or lifestyle businesses. 5. **Overused Concepts**: Words like "tranquil" are often used in wellness branding, which might make the name feel overused or lacking in originality. As someone who values stress relief and work-life balance, I can see why a name that promises tranquility would be appealing, but it's essential to ensure it resonates positively with the intended audience. |
| 88 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Inner Light Yoga | 4 | Some criticisms people might have about the name "Inner Light Yoga" could include: 1. **Cliché or Generic**: Some might find the name a bit cliché or overused, as it incorporates common yoga-related terms like "inner" and "light," which are often associated with spirituality and mindfulness. 2. **Lack of Uniqueness**: It might not stand out in a crowded market where many yoga studios use similar themes or words, making it harder for the studio to establish a distinct brand identity. 3. **Ambiguity**: The name might be seen as vague or not immediately indicative of what the studio offers, especially to those unfamiliar with yoga or its terminology. 4. **Cultural Appropriation Concerns**: Given the spiritual connotations, some might critique the name for potentially appropriating or commercializing aspects of Eastern spirituality without a deep understanding or authentic representation. 5. **Serious Tone**: While "Inner Light" suggests a focus on spirituality and introspection, it might not appeal to those seeking a more energetic or fitness-oriented yoga experience. Balancing these considerations with the studio's mission and target audience could help address any potential criticisms. |
| 89 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Mindful Movements | 4 | Some criticisms people might have about the yoga studio name "Mindful Movements" could include: 1. **Generic Feel**: The name might be perceived as too generic or common, lacking uniqueness or a distinctive quality that sets it apart from other yoga studios. 2. **Lack of Specificity**: It might not clearly convey the specific style or focus of the yoga classes offered, such as whether they emphasize meditation, strength, flexibility, or a particular type of yoga. 3. **Overused Terminology**: The terms "mindful" and "movements" are commonly used in wellness and fitness contexts, which might make the name feel overused or cliché. 4. **Ambiguity**: For someone unfamiliar with yoga, the name might not immediately communicate that the studio offers yoga classes, as it could be interpreted as a general fitness or movement studio. 5. **Emotional Connection**: The name might lack an emotional or personal connection that resonates with potential clients, making it less memorable or engaging. 6. **Modern Appeal**: Some might feel the name doesn’t have a modern or trendy appeal, which could be important for attracting a younger demographic. As a busy executive, I understand the importance of branding and standing out in a competitive market, so these are aspects worth considering to ensure the name aligns with the studio's identity and target audience. |
| 90 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Yogic Vibes | 4 | One criticism might be that "Yogic Vibes" sounds a bit generic or cliché, as the name could be seen as playing into common stereotypes about yoga culture. Additionally, some might feel it lacks uniqueness, making it harder for the studio to stand out in a competitive market. Others might argue that the name doesn't convey a specific style or focus of yoga, which could make it less appealing to those looking for a particular type of practice. As someone who values stress relief and work-life balance, I might also wonder if the name suggests a truly calming and professional environment. |
| 91 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Harmony Haven | 4 | The name "Harmony Haven" could receive a few criticisms, even though it sounds peaceful and inviting. Some might find it a bit generic or clichéd, as many wellness and yoga studios use similar words like "harmony" and "haven" to convey tranquility. Others might argue that it lacks uniqueness or fails to convey a specific style or focus of yoga, which could make it less memorable. Additionally, if the studio is located in a bustling urban area, the name might feel slightly disconnected from its surroundings, unless the interior truly creates a haven-like atmosphere. As someone constantly seeking stress relief, I can appreciate the intention behind the name, but standing out in a crowded market is always a challenge. |
| 92 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Peaceful Postures | 4 | Some criticisms people might have about the name "Peaceful Postures" for a yoga studio could include: 1. **Generic Sounding**: It might be perceived as too generic or typical for a yoga studio, lacking uniqueness or a distinct brand identity. 2. **Overused Theme**: The concept of peace and tranquility is quite common in yoga studio names, which might make it less memorable or stand out among competitors. 3. **Limited Appeal**: The name might suggest a focus only on relaxation or gentle yoga, potentially deterring those looking for more vigorous or varied yoga practices. 4. **Cliché**: Some might find the name cliché, as it plays into the stereotypical image of yoga being solely about peace and calm, rather than also encompassing strength, flexibility, and community. 5. **Cultural Appropriation Concerns**: Depending on the studio's approach, there might be concerns about using a name that taps into cultural elements without authentic representation or understanding. As someone striving for balance myself, I can appreciate the intention behind the name, but it’s always good to consider how it aligns with the studio’s mission and audience. |
| 93 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Chakra Chimes | 4 | People might have a few criticisms about the name "Chakra Chimes" for a yoga studio. Firstly, they might find it a bit too mystical or esoteric, which could potentially alienate newcomers who are not familiar with yoga terminology. Secondly, the name might be considered cliché or overused, as many wellness businesses incorporate the word "chakra" to evoke a spiritual vibe. Additionally, some might say it doesn't clearly convey the practical benefits of yoga, such as stress relief or physical fitness, which could be important for attracting a broader clientele. Lastly, the name might be seen as culturally appropriative if it doesn't respectfully acknowledge the origins and significance of chakras in traditional Indian practices. |
| 94 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Blissful Balance | 4 | Some potential criticisms of the name "Blissful Balance" for a yoga studio might include: 1. **Cliché**: Some might argue that the name sounds generic or overused, as "bliss" and "balance" are common themes in wellness and yoga communities. 2. **Lack of Uniqueness**: It might not stand out among other yoga studios, making it harder for people to remember or differentiate from competitors. 3. **Vague Promise**: The name suggests a promise of achieving bliss and balance, which could set unrealistic expectations for what the studio can deliver. 4. **Cultural Appropriation**: Depending on the studio's approach, some might feel that the name doesn't adequately reflect the depth and cultural roots of yoga. 5. **Target Audience**: The name might not resonate with individuals looking for a more intense or fitness-focused yoga experience, as it suggests a more relaxed or meditative approach. As someone juggling a high-stress job, I can appreciate the appeal of a name that promises tranquility, but it’s important for it to also stand out and clearly communicate what makes the studio special. |
| 95 | gpt-4o | A busy corporate executive seeking stress relief and work-life balance | Breathe Bliss | 4 | Some potential criticisms of the yoga studio name "Breathe Bliss" could include: 1. **Lack of Uniqueness**: The name might sound generic or similar to other wellness or yoga studios, making it difficult to stand out in a crowded market. 2. **Overused Buzzwords**: Words like "breathe" and "bliss" are commonly used in the wellness industry, which might make the name feel clichéd or predictable. 3. **Vague Meaning**: While the name suggests relaxation and happiness, it might not convey anything specific about what makes this studio unique or what kind of yoga experience it offers. 4. **Cultural Sensitivity**: Some might argue that the name doesn't reflect the deeper, more traditional aspects of yoga and instead focuses on a Westernized version of the practice. 5. **Target Audience**: The name might not resonate with all demographics, particularly those looking for a more rigorous or athletic yoga practice, as it suggests a more serene or gentle approach. As a busy executive, I always look for effective stress relief, so a clear and unique identity in a yoga studio name could be quite appealing to me. |
| 96 | gpt-4o | A college student exploring mindfulness and physical fitness | Peaceful Poses | 4 | Some criticisms people might have about the name "Peaceful Poses" for a yoga studio could include: 1. **Generic**: The name might be considered too generic or common, lacking uniqueness or distinctiveness in a crowded market of yoga studios. 2. **Overused Language**: Words like "peaceful" and "poses" are often associated with yoga, which might make the studio name feel clichéd or predictable. 3. **Limited Scope**: The name might suggest that the studio only focuses on gentle or calming yoga practices, potentially alienating those interested in more dynamic or intense styles of yoga. 4. **Lack of Cultural Connection**: Some might feel the name doesn't convey any cultural or philosophical depth, which could be important for those seeking a more traditional or authentic yoga experience. 5. **Memorability**: It may not be memorable enough to stand out in a person's mind, making it harder for the studio to establish a strong brand identity. 6. **Assumed Experience**: The name might imply a certain experience or atmosphere that the studio may not fully deliver, leading to potential mismatched expectations. Overall, while "Peaceful Poses" conveys a calming and approachable image, it may not capture the full essence or diversity of what the studio offers. |
| 97 | gpt-4o | A college student exploring mindfulness and physical fitness | Zen Flow | 4 | Some criticisms people might have about the name "Zen Flow" for a yoga studio could include: 1. **Overused Terminology**: The words "Zen" and "Flow" are commonly used in the wellness industry, which might make the name feel generic or lacking originality. 2. **Cultural Appropriation**: Some might argue that using "Zen," which has deep roots in Japanese Buddhism, could be seen as appropriating or oversimplifying a complex spiritual tradition for commercial purposes. 3. **Lack of Specificity**: The name might not clearly convey what makes this particular yoga studio unique or what specific style of yoga or services it offers. 4. **Misleading Expectations**: People might assume the studio focuses exclusively on meditative or flow-based practices, which could be misleading if the studio offers a wider variety of yoga styles. 5. **Potential Stereotyping**: The name could reinforce stereotypes about yoga being solely about relaxation and tranquility, rather than also encompassing strength, discipline, and physical fitness. As a college student exploring mindfulness, I find it interesting how names can impact perception and the importance of choosing one that aligns with the values and offerings of a business. |
| 98 | gpt-4o | A college student exploring mindfulness and physical fitness | Soul Stretch | 4 | Some criticisms people might have about the name "Soul Stretch" for a yoga studio could include: 1. **Cliché or Generic**: Some might feel that "Soul Stretch" sounds like a typical or overused name in the wellness industry, lacking uniqueness or originality. 2. **Cultural Appropriation**: There could be concerns about the use of the word "soul," which might be seen as appropriating spiritual or cultural concepts without a deep understanding or connection to them. 3. **Misleading Expectations**: The name might suggest a focus on spiritual or emotional aspects of yoga, which could be misleading if the studio primarily offers physical fitness classes without much emphasis on mindfulness or meditation. 4. **Too Abstract**: For some, "Soul Stretch" might be too abstract or vague, making it unclear what the studio specifically offers or how it differentiates itself from other yoga studios. 5. **Target Audience**: Depending on the community, the name might not resonate with all potential clients, especially if they are looking for something more straightforward or fitness-focused rather than spiritually inclined. As someone who's exploring mindfulness, I think it's important for a studio name to reflect both its mission and the experience it offers, so these are definitely points worth considering. |
| 99 | gpt-4o | A college student exploring mindfulness and physical fitness | Zen Oasis | 4 | While "Zen Oasis" might sound appealing to some, there are a few potential criticisms people might have: 1. **Cliché**: The name might be seen as overused or generic, as "Zen" and "Oasis" are common terms in the wellness industry. Some might feel it lacks originality. 2. **Cultural Appropriation**: The use of "Zen," which originates from a specific school of Buddhism, might be seen as cultural appropriation if the studio doesn't have a genuine connection to or respect for its origins. 3. **Expectations**: The name suggests a serene and peaceful environment, so if the studio doesn't live up to that expectation, it could lead to disappointment. 4. **Ambiguity**: For those unfamiliar with the concepts of "Zen" or "Oasis," the name might not clearly convey that it's a yoga studio, potentially leading to confusion. 5. **Target Audience**: The name might not resonate with people looking for a more dynamic or fitness-oriented yoga experience, as it implies a focus on relaxation and tranquility. Choosing a name that aligns with the studio's mission and values while being mindful of these potential criticisms could be beneficial. |
| 100 | gpt-4o | A college student exploring mindfulness and physical fitness | Tranquil Transitions | 4 | Some criticisms people might have about the name "Tranquil Transitions" for a yoga studio might include: 1. **Ambiguity**: The name might be considered vague or unclear about what exactly the studio offers. While "Tranquil" suggests calmness, "Transitions" might not immediately convey yoga or fitness to everyone. 2. **Overused Terms**: Both "tranquil" and "transitions" are common words used in wellness and mindfulness spaces, which might make the name seem less unique or memorable. 3. **Lack of Cultural Connection**: Some might feel that the name doesn't adequately reflect the cultural or historical roots of yoga, which could be important for those who value authenticity in their practice. 4. **Too Generic**: The name might be viewed as generic, lacking a distinctive character that sets it apart from other studios or wellness centers. 5. **Potential Misinterpretation**: "Transitions" might be interpreted in various ways, possibly leading people to think the studio focuses on life coaching or therapy rather than yoga. These are just some possibilities, and the effectiveness of a name often depends on the local market and the specific branding approach of the studio. |
| 101 | gpt-4o | A college student exploring mindfulness and physical fitness | Namaste Nest | 4 | Some criticisms people might have about the yoga studio name "Namaste Nest" could include: 1. **Cultural Appropriation**: Some might feel that the use of "Namaste," a traditional Hindu greeting, could be seen as cultural appropriation if the studio doesn't genuinely respect or understand the cultural context. 2. **Cliché**: The use of "Namaste" in yoga studio names is quite common, so some might find it to be a bit cliché or overused, lacking originality. 3. **Overly Cute or Gimmicky**: The combination of "Namaste" with "Nest" might come off as overly cute or gimmicky, potentially undermining the seriousness or authenticity of the practice. 4. **Misinterpretation**: The word "Nest" might imply a focus on relaxation and comfort, which could mislead people about the intensity or style of yoga being offered if it includes more vigorous practices. 5. **Lack of Clarity**: The name might not clearly convey what sets this studio apart from others, potentially making it less memorable or distinctive in a crowded market. Despite these potential criticisms, the name could also be seen as inviting and cozy, appealing to those looking for a welcoming and nurturing environment. |
| 102 | gpt-4o | A college student exploring mindfulness and physical fitness | Serenity Studio | 3 | Some criticisms people might have about the name "Serenity Studio" could include: 1. **Generic Sounding**: The name might be considered too generic or overused, as "serenity" is a common term associated with wellness and relaxation, making it less distinctive. 2. **Lack of Specificity**: It doesn't specifically indicate that it's a yoga studio. People might not immediately know what services are offered without additional context. 3. **Cliché Connotations**: The term "serenity" could be seen as a cliché in the wellness industry, potentially making the studio seem less original or innovative. 4. **Cultural Disconnect**: Some might feel that the name doesn't reflect any cultural or philosophical aspects of yoga, which could be important for authenticity. 5. **Memorability**: Because it's a common word, it might not be as memorable or unique, making it harder for potential clients to remember or differentiate from other studios. Despite these potential criticisms, the name could still appeal to those looking for a calming and peaceful environment, aligning with the core values of yoga practice. |
| 103 | gpt-4o | A college student exploring mindfulness and physical fitness | Tranquil Tribe | 4 | Some criticisms people might have about the name "Tranquil Tribe" for a yoga studio could include: 1. **Cultural Appropriation**: The use of the word "tribe" might be seen as appropriating Indigenous cultures, as it can carry significant cultural and historical connotations. 2. **Generic Sounding**: Some might find the name a bit generic or overused, as many wellness brands use words like "tranquil" to convey peace and calmness. 3. **Lack of Specificity**: The name doesn't immediately convey what makes the studio unique or what specific style of yoga or community experience it offers. 4. **Potential Misinterpretation**: The word "tribe" might suggest exclusivity or cliquishness, which could be off-putting to those who are looking for an inclusive environment. 5. **Overemphasis on Relaxation**: While tranquility is a key aspect of yoga, some might argue that the name suggests a focus solely on relaxation, potentially overlooking the physical and spiritual aspects of yoga practice. These are just a few perspectives, and ultimately, the suitability of the name would depend on the studio's branding and the community it serves. |
| 104 | gpt-4o | A college student exploring mindfulness and physical fitness | Inner Light Yoga | 4 | Some criticisms people might have about the name "Inner Light Yoga" could include: 1. **Cliché or Overused**: The name might be seen as a bit cliché or generic, as many wellness and yoga studios use terms like "inner," "light," or "spirit" to convey a sense of peace and enlightenment. 2. **Lack of Uniqueness**: It might not stand out in a crowded market of yoga studios, making it harder for people to remember or differentiate from others with similar names. 3. **Assumptions About Style**: The name could give off a specific vibe that suggests a focus on spirituality or meditation, which might not appeal to those looking for a more physically intense or fitness-oriented yoga practice. 4. **Cultural Appropriation Concerns**: Some might feel that the name appropriates spiritual concepts without a deep understanding or authentic connection to the cultural origins of yoga. 5. **Ambiguity**: The name might not clearly communicate what makes the studio unique or what specific offerings or styles of yoga are available. Despite these potential criticisms, the name does evoke a sense of peace and introspection, which could attract those looking for a calming and reflective yoga experience. |
| 105 | gpt-4o | A college student exploring mindfulness and physical fitness | Mindful Movements | 4 | Some criticisms people might have about the name "Mindful Movements" for a yoga studio could include: 1. **Generic Sounding**: The name might be considered a bit generic or overused, as many wellness and fitness businesses use similar terms like "mindful" and "movements." 2. **Lack of Uniqueness**: It might not stand out in a crowded market, making it harder to differentiate from other studios with similar names. 3. **Vagueness**: Some might feel the name is too vague and doesn’t clearly convey that it’s a yoga studio, which could make it less memorable or identifiable. 4. **Limited Scope**: The name might suggest that the studio focuses only on movement, potentially overlooking other aspects of yoga like meditation, breathing, or spiritual growth. 5. **Overemphasis on Trendiness**: With "mindful" being a buzzword in wellness industries, some might perceive the name as trying too hard to capitalize on current trends rather than offering something unique or authentic. Despite these potential criticisms, the name does convey a sense of intentionality and connection between mind and body, which could appeal to many looking for a holistic yoga experience. |
| 106 | gpt-4o | A college student exploring mindfulness and physical fitness | Yogic Vibes | 4 | Some criticisms people might have about the name "Yogic Vibes" could include: 1. **Generic Sounding**: It might be considered too generic or common, lacking uniqueness compared to other yoga studios. 2. **Cultural Sensitivity**: Some might feel that the term "Yogic" could be appropriating cultural elements without proper context or respect, especially if the studio doesn't authentically incorporate traditional yoga practices. 3. **Vague Branding**: The name might not clearly communicate what sets the studio apart or what specific styles or benefits it offers. 4. **Overused Terms**: Words like "vibes" are frequently used in wellness branding, which might make the name feel less original or memorable. 5. **Lack of Professionalism**: For some, the casual tone of "Vibes" might not convey the level of professionalism or expertise they expect from a yoga studio. |
| 107 | gpt-4o | A college student exploring mindfulness and physical fitness | Harmony Haven | 4 | Some criticisms people might have about the name "Harmony Haven" for a yoga studio could include: 1. **Cliché or Generic**: Some might feel that "Harmony Haven" sounds too generic or overused, as many wellness-related businesses use similar words to convey peace and balance. 2. **Lack of Uniqueness**: It might not stand out in a crowded market where many studios aim to convey similar themes of tranquility and refuge. 3. **Vagueness**: The name might be seen as too vague or not descriptive enough of what specifically makes the studio unique compared to others. 4. **Cultural Appropriation Concerns**: Depending on the context and how the studio is marketed, there could be concerns about the use of the word "harmony" in a way that might not fully respect the cultural origins of yoga. 5. **Target Audience Misalignment**: If the studio aims to attract a more athletic or modern crowd, the name might not resonate as it leans more towards a traditional or serene image. Ultimately, the effectiveness of the name can depend on how well it aligns with the studio's branding, mission, and the community it serves. |
| 108 | gpt-4o | A college student exploring mindfulness and physical fitness | Peaceful Postures | 4 | Some criticisms people might have about the name "Peaceful Postures" for a yoga studio could include: 1. **Generic Sounding**: It might be considered too generic or common, lacking a unique identity that sets it apart from other yoga studios. 2. **Lack of Energy**: Some might feel that the word "peaceful" suggests a very slow or gentle practice, which might not appeal to those looking for a more vigorous or dynamic yoga experience. 3. **Limited Scope**: The focus on "postures" might imply that the studio emphasizes physical poses over other aspects of yoga, such as meditation, breathing techniques, or the philosophical elements. 4. **Cultural Sensitivity**: There might be concerns about whether the name appropriately respects and represents the cultural origins of yoga, which is important to some practitioners. 5. **Marketability**: The name might not be as catchy or memorable as other names, potentially making it harder to stand out in marketing efforts. 6. **Assumed Experience Level**: It could give the impression that the studio is only for beginners or those looking for a very relaxed practice, possibly deterring more advanced yogis. Overall, while "Peaceful Postures" conveys a sense of calm and tranquility, it might not resonate with everyone depending on their preferences and expectations for a yoga practice. |
| 109 | gpt-4o | A college student exploring mindfulness and physical fitness | Chakra Chimes | 4 | One criticism might be that the name "Chakra Chimes" could come across as a bit too mystical or esoteric, which might not appeal to people who are new to yoga or who prefer a more straightforward approach to fitness. Additionally, some might feel that it leans heavily into stereotypes about yoga culture, which could be off-putting to those looking for a more inclusive or modern vibe. Others might find it catchy and memorable, though, so it really depends on the target audience! |
| 110 | gpt-4o | A college student exploring mindfulness and physical fitness | Blissful Balance | 4 | Some criticisms people might have about the name "Blissful Balance" for a yoga studio could include: 1. **Cliché or Generic**: Some might feel that the name sounds a bit cliché or overused, as it combines two popular buzzwords in the wellness community. 2. **Ambiguity**: The name might not clearly convey that it's a yoga studio. It could be mistaken for a spa, meditation center, or even a wellness product line. 3. **Overpromise**: The name suggests a state of perfect equilibrium and happiness, which might set expectations too high for what a single yoga class or studio can deliver. 4. **Lack of Distinctiveness**: It might not stand out among other yoga studios, especially if there are many with similar names focusing on peace and balance. 5. **Cultural Sensitivity**: Some might argue that the name doesn't reflect the cultural roots of yoga, which could be important for authenticity and respect for the practice's origins. Despite these potential criticisms, the name could still appeal to those looking for a serene and balanced yoga experience. |
| 111 | gpt-4o | A college student exploring mindfulness and physical fitness | Breathe Bliss | 4 | Some people might find the name "Breathe Bliss" a bit cliché or overused, as many yoga studios and wellness brands tend to incorporate similar words like "breathe" and "bliss" to convey relaxation and mindfulness. Others might feel that it doesn't stand out enough or lacks uniqueness in a crowded market. Additionally, some might think it sounds too generic and doesn't provide a clear sense of what makes the studio different or special. However, it does have a calming and inviting vibe, which could appeal to those looking for a peaceful yoga experience. |
| 112 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Peaceful Poses | 4 | As a new parent who's been exploring yoga, I've come across a few things people might consider when it comes to a yoga studio name like "Peaceful Poses." Some might feel that it sounds a bit generic, as "peaceful" and "poses" are commonly associated with yoga and might not stand out in a crowded market. Others might think it sets an expectation that the classes will be very gentle or meditative, which might not appeal to those looking for a more vigorous practice. Additionally, some might feel the name doesn't convey a unique identity or the specific benefits the studio offers. However, personally, I find that anything emphasizing peace and balance is quite appealing, especially when trying to regain strength and mental clarity. |
| 113 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Zen Flow | 4 | Some people might find the name "Zen Flow" a bit generic or overused, as both "Zen" and "Flow" are common terms in the wellness and yoga industry. It might not stand out enough in a crowded market. Additionally, some might feel that the name doesn't convey a specific style or unique aspect of the studio, making it harder to differentiate from other yoga studios. Lastly, if the studio doesn't incorporate elements of Zen philosophy or practices, some might see the name as misleading or not reflective of the actual offerings. |
| 114 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Soul Stretch | 4 | Some criticisms people might have about the name "Soul Stretch" could include it sounding a bit cliché or overly spiritual, which might not appeal to those looking for a more fitness-focused or secular yoga experience. Others might think it lacks uniqueness, as "soul" and "stretch" are common words associated with yoga and wellness, making it less distinctive. Additionally, some might feel that it implies a certain style of yoga that may not align with their personal practice, especially if they're looking for a more intense or athletic approach. |
| 115 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Zen Oasis | 4 | Some criticisms people might have about the name "Zen Oasis" for a yoga studio could include: 1. **Cliché**: The name might be seen as overused or too generic, as both "Zen" and "Oasis" are common terms associated with relaxation and peace, which might not stand out in a crowded market. 2. **Cultural Appropriation**: Some people might feel that using "Zen," which has roots in Japanese Buddhism, could be seen as appropriating a cultural term without a deep understanding or respect for its origins and meaning. 3. **Expectations**: The name might set high expectations for a tranquil and serene environment, which could be difficult to maintain consistently, especially if the studio is located in a busy or noisy area. 4. **Lack of Specificity**: It might not convey specific information about the type or style of yoga offered, leaving potential clients unsure if it aligns with their practice preferences. 5. **Over-Promise**: The word "Oasis" suggests an escape from the everyday, which might not resonate with people looking for a more down-to-earth or practical approach to yoga. As a new parent, I find that yoga studios with a clear, authentic message resonate more with me, as they help me feel more connected and grounded during my practice. |
| 116 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Tranquil Transitions | 4 | Some criticisms people might have about the name "Tranquil Transitions" for a yoga studio could include: 1. **Vagueness**: The name might be seen as too vague or generic, not clearly indicating that it's a yoga studio. It could be mistaken for other types of businesses, like a therapy center or a moving company. 2. **Overused Terms**: Words like "tranquil" and "transitions" are quite common in wellness and self-improvement industries, which might make the name seem unoriginal or cliché. 3. **Lack of Energy**: While tranquility is a positive aspect of yoga, the name might not convey the dynamic and energetic elements of yoga practice that some people enjoy. 4. **Emotional Connotation**: "Transitions" can sometimes be associated with life changes that are difficult or challenging, which might not be the most inviting concept for a yoga studio. 5. **Pronunciation and Memorability**: The alliteration might be intended to make the name catchy, but it could also be a bit of a tongue-twister, which might affect how easily people remember or talk about it. As a new parent, I find yoga incredibly helpful for regaining strength and mental clarity, so a name that clearly communicates the benefits of yoga would resonate more with me. |
| 117 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Namaste Nest | 4 | Some people might feel that the name "Namaste Nest" could be seen as a bit cliché, as "Namaste" is a commonly used term in the yoga community, and it might not stand out as unique. Additionally, there could be concerns about cultural appropriation, especially if the studio doesn't authentically incorporate or respect the cultural origins of yoga. Others might find the name too playful or informal, not conveying the serious and transformative aspects of yoga practice. However, as a new parent using yoga to regain strength and mental clarity, I personally find the name quite inviting and cozy, suggesting a welcoming space for relaxation and growth. |
| 118 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Serenity Studio | 3 | Some potential criticisms of the name "Serenity Studio" might include that it sounds too generic or overused, as many wellness or yoga studios aim to convey a sense of calm and peace, leading to similar names. Additionally, some might find it lacks uniqueness or fails to convey what sets the studio apart from others. It might also be perceived as not descriptive enough if the studio offers specialized classes or services beyond general yoga practices. As a new parent using yoga to regain strength and mental clarity, I personally appreciate names that evoke tranquility, but I also look for something that hints at a supportive community or a unique approach to practice. |
| 119 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Tranquil Tribe | 4 | When it comes to naming a yoga studio "Tranquil Tribe," there are a few potential criticisms that might arise. Some people might feel that the word "tribe" could be culturally appropriative, as it has significant meaning in various indigenous cultures. Others might think that it could unintentionally exclude individuals who don't resonate with the idea of belonging to a "tribe" or group, especially if they prefer a more individualistic approach to yoga. Additionally, some might find the name a bit generic or overused, as tranquility is a common theme in yoga. However, if the studio's atmosphere and community reflect the name positively, it could still resonate well with many people. |
| 120 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Inner Light Yoga | 4 | Some criticisms people might have about the name "Inner Light Yoga" could include: 1. **Generic or Overused**: Some might feel that "Inner Light" is a common phrase in the wellness industry, making it seem less unique or original. 2. **Vague Meaning**: While it sounds positive, the term "Inner Light" might be seen as vague or not clearly conveying what makes this particular yoga studio distinct from others. 3. **Cultural Appropriation Concerns**: There could be sensitivities around using spiritual or philosophical terms without clear connections to their cultural origins, especially if the studio doesn't adequately represent or respect those traditions. 4. **Assumed Spiritual Focus**: The name might imply a strong spiritual focus, which could deter individuals who are more interested in the physical or fitness aspects of yoga. 5. **Potential for Misinterpretation**: Some might interpret "Inner Light" as being too abstract or esoteric, possibly alienating those who prefer straightforward, practical branding. As a new parent using yoga to regain strength and mental clarity, I find names that evoke calm and positivity appealing, but I can see why some might have these concerns. |
| 121 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Mindful Movements | 4 | Some criticisms people might have about the name "Mindful Movements" for a yoga studio could include: 1. **Generic Sounding**: It might be considered too generic or not distinctive enough, making it difficult for the studio to stand out among other yoga studios that emphasize mindfulness and movement. 2. **Lack of Specificity**: The name doesn’t specify what type of yoga or services are offered, which might make it harder for potential clients to immediately understand what the studio specializes in or offers. 3. **Overused Terms**: Both "mindful" and "movements" are popular buzzwords in the wellness industry, which could make the name seem trendy rather than timeless. 4. **Cultural Sensitivity**: Some might feel that using the term "mindful" without a clear connection to authentic mindfulness practices could be seen as appropriating or diluting the concept. 5. **Potential Misinterpretation**: The name might suggest that the focus is more on general fitness or movement classes rather than traditional yoga, which could attract or deter different audiences. Despite these potential criticisms, the name could still resonate well with people looking for a holistic approach to wellness that combines mental and physical practices. |
| 122 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Yogic Vibes | 4 | Some people might find "Yogic Vibes" a bit generic or overused, as it combines two very common terms associated with yoga. Others might feel that it doesn't convey anything unique about the studio, such as its specific style, approach, or community focus. Additionally, some might think it sounds more like a casual brand rather than a serious place for deep practice or learning. However, it could also be seen as inviting and approachable, depending on the audience. |
| 123 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Harmony Haven | 4 | Choosing a name for a yoga studio can be quite personal and subjective, and while "Harmony Haven" has a nice ring to it, there are a few criticisms people might have: 1. **Generic Feel**: Some might find "Harmony Haven" a bit generic or overused, as many wellness and yoga businesses opt for similar calming and peaceful names. 2. **Lack of Uniqueness**: It might not stand out in a crowded market, making it harder for people to remember or differentiate from other studios. 3. **Expectations**: The name sets a high expectation for peace and tranquility, which could be a challenge to consistently deliver in every class or experience. 4. **Cultural Sensitivity**: Depending on the community, some might feel that the name doesn't reflect the cultural roots of yoga, which could be important for authenticity. 5. **Target Audience**: It might not clearly indicate the type of yoga or services offered, which could be important for attracting the right clientele who are looking for specific styles or experiences. As a new parent using yoga to regain strength and mental clarity, I’d probably appreciate a name that conveys both peace and empowerment, reflecting the journey of balancing tranquility with the strength needed in parenting. |
| 124 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Peaceful Postures | 4 | Some people might find the name "Peaceful Postures" a bit generic or cliché, as it uses common terms associated with yoga. Others might feel it doesn't stand out enough in a crowded market of yoga studios. Additionally, if the studio offers more dynamic or intense yoga styles, the name might not fully capture the breadth of their offerings. However, as a new parent, I really appreciate the calming vibe it suggests, which is exactly what I look for when trying to regain my strength and mental clarity through yoga. |
| 125 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Chakra Chimes | 5 | Some potential criticisms people might have about the name "Chakra Chimes" could include: 1. **Cultural Appropriation Concerns**: Some might feel that the use of the term "chakra," which has deep spiritual and cultural significance in Hinduism and other Eastern traditions, could be seen as appropriative if not handled with respect and understanding. 2. **Lack of Clarity**: The name might not immediately convey that it's a yoga studio. People unfamiliar with the concept of chakras might be confused about what the business offers. 3. **Sound Association**: The word "chimes" might make people think of wind chimes or a music store rather than a place for yoga practice. 4. **Niche Appeal**: The name might appeal more to those already familiar with or interested in spiritual or esoteric practices, potentially alienating those who are new to yoga or seeking a more fitness-oriented approach. 5. **Overused Themes**: Some might feel that the name relies on common yoga clichés, making it less distinctive in a market where many businesses use similar themes. As a new parent using yoga to regain strength and mental clarity, I find it important that a yoga studio name feels welcoming and inclusive, encouraging everyone to explore the benefits of yoga without any preconceived notions. |
| 126 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Blissful Balance | 4 | Some potential criticisms people might have about the name "Blissful Balance" for a yoga studio could include: 1. **Cliché or Generic**: Some might feel that the name is too common or overused in the wellness industry, lacking uniqueness or originality. 2. **Vague Meaning**: Others might think that the name doesn't clearly convey what the studio offers or what makes it special compared to other yoga studios. 3. **Assumed Experience Level**: The word "blissful" could imply that the studio is only for those who are already advanced in their practice, potentially alienating beginners who are just starting their yoga journey. 4. **Cultural Appropriation Concerns**: Some might feel that the name is part of a trend of Westernizing yoga by focusing on the "bliss" aspect without acknowledging the deeper cultural and spiritual roots of the practice. 5. **Overemphasis on Positivity**: While balance is an important concept in yoga, focusing too much on bliss might overlook the reality that yoga is also about facing challenges and growth, not just achieving a constant state of happiness. As a new parent, I find that yoga helps me find balance amidst the chaos, but I can see how some might want a name that feels more grounded or reflective of their personal journey. |
| 127 | gpt-4o | A new parent using yoga to regain strength and mental clarity | Breathe Bliss | 4 | When it comes to naming a yoga studio, "Breathe Bliss" could evoke a variety of reactions. Here are some potential criticisms people might have: 1. **Generic Feel**: Some might feel that "Breathe Bliss" is somewhat generic and doesn't stand out from other yoga studios. It combines two common words associated with yoga, which might not make it memorable. 2. **Lack of Specificity**: The name might not convey what makes the studio unique or what specific style of yoga or services it offers, which could be a drawback for those looking for something specific. 3. **Overused Concepts**: Words like "breathe" and "bliss" are frequently used in wellness contexts, so some might think the name lacks originality or creativity. 4. **Cliché Association**: For some, the combination of "breathe" and "bliss" might come across as cliché, as many wellness brands use similar language to convey relaxation and peace. 5. **Expectations of Experience**: The name could set high expectations for a blissful experience, and if the studio doesn't deliver on that promise, it might lead to disappointment. Despite these potential criticisms, the name does have a calming and inviting feel, which could attract people looking for a peaceful yoga experience. As a new parent, I find any reminder to breathe and seek bliss quite appealing, even if it's a bit common! |
| 128 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Peaceful Poses | 4 | Some people might find the name "Peaceful Poses" a bit generic or overused, as many yoga studios aim to convey tranquility and calmness. Others might feel it doesn't stand out enough in a competitive market. Additionally, if the studio offers high-intensity or dynamic classes, the name might not fully capture the range of experiences available. However, for someone like me who appreciates a gentle approach to staying flexible, it sounds quite inviting! |
| 129 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Zen Flow | 4 | When considering a name like "Zen Flow" for a yoga studio, some criticisms might include: 1. **Overused Terms**: Both "Zen" and "Flow" are commonly used in the wellness industry, which might make the name feel generic or less distinctive. 2. **Cultural Appropriation**: The term "Zen" is rooted in a specific Buddhist tradition, and some people might feel that using it in a commercial context without a deep understanding or respect for its origins could be seen as cultural appropriation. 3. **Lack of Specificity**: The name "Zen Flow" doesn't provide much information about what makes this studio unique. Potential clients might not get a sense of what specific styles or types of yoga are offered. 4. **Assumptions about Style**: The word "Flow" might imply that the studio primarily offers Vinyasa or flow-style classes, which could deter those interested in other types of yoga. 5. **Potential Misinterpretation**: Some might interpret "Zen Flow" as focusing solely on relaxation or meditation, which might not appeal to those looking for a more vigorous or fitness-oriented practice. Despite these potential criticisms, the name might still resonate well with people looking for a calming and accessible yoga experience. |
| 130 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Soul Stretch | 4 | When it comes to naming a yoga studio "Soul Stretch," there could be a few criticisms or concerns people might have. Some might feel that the name is a bit cliché or overused, as many wellness-related businesses tend to use words like "soul" and "stretch" to convey a sense of spirituality and physical flexibility. Others might think that the name could be misleading if the studio doesn't offer classes that focus on spiritual practices or deeper philosophical aspects of yoga, as the name might imply. Additionally, some people might prefer a name that is more unique or personal, reflecting the specific vibe or community of the studio. Overall, it's important for a studio name to resonate with its intended audience and accurately represent the experience it offers. |
| 131 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Zen Oasis | 5 | The name "Zen Oasis" might receive some criticisms for a few reasons. Firstly, it could be seen as somewhat generic or cliché, as "Zen" and "Oasis" are commonly used terms in the wellness industry. Some people might feel that it doesn't stand out or convey a unique identity. Additionally, the name might lead to certain expectations about the atmosphere and style of yoga offered, such as a focus on relaxation and meditation, which may not align with all the classes or services provided. Lastly, for those who value authenticity, there might be concerns about cultural appropriation, as "Zen" is a term rooted in Japanese Buddhism, and its use could be viewed as superficial if not handled with respect and understanding. |
| 132 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Tranquil Transitions | 4 | Some criticisms people might have about the name "Tranquil Transitions" for a yoga studio could include: 1. **Ambiguity**: The name might not immediately convey that it's a yoga studio. It could sound like a service related to counseling or life coaching. 2. **Overused Terms**: Words like "tranquil" are commonly used in wellness and relaxation businesses, so it might not stand out or feel unique. 3. **Lack of Energy**: While tranquility is a positive trait, some might feel the name doesn't convey the energy and vitality that yoga can also offer. 4. **Serious Tone**: The name might come across as too serious or formal, which could deter those looking for a more fun or light-hearted yoga experience. 5. **Cultural Sensitivity**: Depending on the audience, some might feel the name doesn't reflect the cultural roots of yoga, which could be important for authenticity. Personally, I think it's important for a yoga studio to have a name that resonates with its community and reflects the atmosphere it aims to create. |
| 133 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Namaste Nest | 4 | Some people might find the name "Namaste Nest" a bit cliché or overused, as "Namaste" is a common term associated with yoga and wellness. Others might feel that it doesn't clearly convey the unique offerings or atmosphere of the studio, especially if they're looking for something that stands out among many yoga options. Additionally, if the studio doesn't incorporate aspects of community or coziness implied by "Nest," it could be seen as misleading. However, for those who appreciate a welcoming and peaceful vibe, the name might actually resonate well. |
| 134 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Serenity Studio | 3 | Some criticisms people might have about the name "Serenity Studio" could include it being too generic or overused, as many wellness or yoga studios tend to use similar names. It might also not clearly convey that it's a yoga studio, which could be confusing for potential clients who are searching specifically for yoga classes. Additionally, the name might not stand out in a crowded market, making it harder for the studio to establish a unique brand identity. As someone who's been around for a while, I appreciate names that are both clear and memorable, so a bit more specificity or creativity could help. |
| 135 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Tranquil Tribe | 4 | People might have a few criticisms about the name "Tranquil Tribe" for a yoga studio. Some might feel that the word "Tranquil" could come across as a bit generic or overused in the wellness industry, making it less distinctive. Others might be concerned that the word "Tribe" could be seen as cultural appropriation or insensitive, as it traditionally refers to indigenous groups and communities. Additionally, some might think that the name doesn't clearly convey the specific style or unique offerings of the studio, which could be important for attracting the right clientele. |
| 136 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Inner Light Yoga | 4 | Choosing a name for a yoga studio can be a very personal and creative process, but it's true that some people might have different reactions or criticisms. Here are a few potential criticisms someone might have about the name "Inner Light Yoga": 1. **Generic or Overused**: Some might feel that "Inner Light" is a bit generic or overused in the wellness and yoga industry, as many businesses aim to convey similar ideas of enlightenment and inner peace. 2. **Vague Meaning**: While "Inner Light" is a positive and spiritual concept, some might find it too vague or abstract, making it hard to immediately understand what the studio offers or how it stands out. 3. **Cultural Appropriation Concerns**: Depending on how the studio presents itself, there could be concerns about cultural appropriation if the name is seen as borrowing from spiritual traditions without proper respect or understanding. 4. **Lack of Uniqueness**: If there are other businesses with similar names, it might be seen as lacking in uniqueness or originality, which can be important for branding and attracting a dedicated clientele. 5. **Misinterpretation**: Some might misinterpret "Inner Light" as a religious or overly spiritual concept, which could deter those who are looking for more of a physical or fitness-focused yoga experience. Overall, while the name "Inner Light Yoga" might appeal to many for its positive and serene connotations, it's always good to consider how it might be perceived by different audiences. |
| 137 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Mindful Movements | 4 | Some people might find the name "Mindful Movements" a bit generic, as it doesn't immediately stand out from other yoga studios that emphasize mindfulness and movement. Others might feel it sounds a bit too serious or clinical, lacking a sense of warmth or community that they might be seeking in a yoga studio. Additionally, some might interpret it as focusing solely on the mental aspect, potentially neglecting the physical or spiritual components that are also important in yoga practice. For me, as someone who enjoys maintaining flexibility, I’d be looking for a name that suggests a welcoming and supportive environment. |
| 138 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Yogic Vibes | 4 | Some people might find the name "Yogic Vibes" to be a bit too trendy or vague, as it doesn't clearly convey what makes the studio unique compared to others. Others might think it sounds a bit too casual or informal, which might not appeal to those looking for a more traditional or serious yoga practice. Additionally, the term "vibes" could be seen as too focused on a particular lifestyle or aesthetic, which might not resonate with everyone. However, for others, it could perfectly capture the relaxed and welcoming atmosphere they’re looking for in a yoga studio. |
| 139 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Harmony Haven | 4 | "Harmony Haven" might evoke certain criticisms depending on individual perspectives. Some might find the name a bit too generic or cliché, as it combines two commonly used words in wellness and relaxation contexts. Others might feel it doesn't convey a unique identity or distinguish the studio from others in a crowded market. Additionally, for those who prefer a more modern or edgy vibe, "Harmony Haven" could seem too traditional or serene. However, I personally think it has a nice, calming ring to it, which is what I’d look for in a yoga studio. What do you think? |
| 140 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Peaceful Postures | 4 | Some people might find the name "Peaceful Postures" a bit generic or overused, as many yoga studios aim to convey a sense of tranquility and calm through similar names. Others might feel it doesn't stand out enough in a crowded market or that it doesn't convey a unique aspect or specialty of the studio. Additionally, some might think it sounds too passive and prefer a name that suggests more energy or dynamism. However, the name does have a soothing quality that could appeal to those looking for a relaxing and stress-free yoga experience. |
| 141 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Chakra Chimes | 4 | Some people might find the name "Chakra Chimes" a bit too mystical or new-agey, which might not appeal to everyone. It could also be seen as potentially confusing if someone isn't familiar with what chakras or chimes are. Additionally, some might feel that it leans heavily into a certain cultural aesthetic, which could be seen as appropriative if not handled with sensitivity. Others might find it catchy and inviting, but as with any name, it might not resonate with everyone. |
| 142 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Blissful Balance | 4 | The name "Blissful Balance" might be critiqued for a few reasons. Some people might find it a bit generic, as it uses common buzzwords associated with wellness and yoga, like "bliss" and "balance," which might not stand out in a crowded market. Others might feel it sounds a bit too soft or esoteric, potentially not reflecting the full range of classes or the intensity they might be looking for. Additionally, if the studio offers more than just yoga, the name might not fully encompass the breadth of services provided. Lastly, some might simply find it a bit too long or difficult to remember compared to shorter, catchier names. |
| 143 | gpt-4o | A retired senior looking to maintain flexibility and social connections | Breathe Bliss | 4 | Choosing a name for a yoga studio like "Breathe Bliss" can certainly resonate with many, but there might be a few criticisms or concerns that some people could have: 1. **Generic Sounding**: Some might find it a bit generic or overused in the wellness industry, as "breathe" and "bliss" are common terms associated with yoga and relaxation. 2. **Expectation Setting**: It might set high expectations for the experience, implying that the studio guarantees a state of bliss, which could be subjective and vary from person to person. 3. **Cultural Sensitivity**: There could be concerns about cultural appropriation if the name is perceived as using spiritual or cultural terms without a deep understanding or respect for their origins. 4. **Memorability**: While pleasant, the name might not be particularly memorable or distinctive among other wellness centers, making it harder for people to recall. 5. **Target Audience**: The name might not clearly convey the specific style or focus of the yoga offered, which could be important for attracting the right clientele who are looking for specific types of yoga practice. 6. **Overemphasis on Relaxation**: Some might argue that it emphasizes relaxation over other benefits of yoga like strength, flexibility, or mindfulness. Despite these potential criticisms, if the studio provides a great experience, the name can become well-loved and synonymous with quality in the community. |
| 144 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Peaceful Poses | 4 | Some criticisms people might have about the name "Peaceful Poses" for a yoga studio could include: 1. **Generic Sounding**: The name might be considered too generic or common, which could make it difficult for the studio to stand out among other yoga studios. 2. **Limited Appeal**: The name "Peaceful Poses" might suggest a focus solely on relaxation and gentle yoga, potentially deterring those looking for more intense or dynamic practices. 3. **Lack of Uniqueness**: It might not convey a unique identity or the specific qualities that make the studio special, such as a particular teaching style or community vibe. 4. **Cultural Sensitivity**: Some might feel that the name doesn't adequately reflect the cultural or spiritual roots of yoga, which could be important for those seeking authenticity. 5. **Assumed Experience Level**: The name might imply that the studio caters only to beginners or those seeking a calm practice, possibly alienating more advanced practitioners. Ultimately, the effectiveness of a name can depend on the target audience and the overall branding strategy of the studio. |
| 145 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Zen Flow | 4 | Some criticisms of the yoga studio name "Zen Flow" might include: 1. **Overused Terms**: Both "Zen" and "Flow" are commonly used in the yoga and wellness industry, which might make the name feel generic or lacking uniqueness. 2. **Cultural Appropriation**: The use of "Zen," which is rooted in Japanese Buddhism, might be seen as cultural appropriation if the studio doesn't have a genuine connection to or respect for the cultural and spiritual origins of the term. 3. **Vagueness**: The name doesn't provide specific information about what makes the studio unique or what type of yoga or services it offers, which could make it less memorable or appealing to potential clients. 4. **Target Audience**: It might not clearly communicate who the studio is for or what level of experience is catered to, potentially alienating beginners or those looking for specific types of classes. 5. **Modernity vs. Tradition**: Some might feel that the name tries too hard to blend modern elements with traditional practices, possibly diluting the authenticity of either aspect. These are just a few possibilities, and the actual reception of the name would depend on the studio's branding and community engagement. |
| 146 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Soul Stretch | 4 | Some criticisms people might have about the name "Soul Stretch" for a yoga studio could include: 1. **Cliché or Generic**: The name might be considered too common or overused in the wellness industry, lacking uniqueness or originality. 2. **Misleading Expectations**: Some might feel that the name implies a focus solely on spiritual aspects, which could deter those looking for a more physical or fitness-oriented practice. 3. **Cultural Sensitivity**: The use of the word "soul" might be seen as appropriating spiritual concepts without a deep understanding or respect for the cultural origins of yoga. 4. **Vague Messaging**: The name might not clearly convey what the studio offers, such as specific styles of yoga, classes, or additional services like workshops or retreats. 5. **Limited Appeal**: It might not resonate with all demographics, particularly those who prefer a more straightforward or modern approach to yoga. As an athlete, I appreciate a studio that balances both the physical and mental aspects of yoga, so ensuring the name reflects that balance could be beneficial. |
| 147 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Zen Oasis | 4 | Some people might feel that the name "Zen Oasis" is a bit cliché or overused, especially since "Zen" and "Oasis" are common words in wellness and yoga branding. Others might think it doesn't clearly convey what makes the studio unique or what specific styles of yoga are offered. Additionally, for those who are looking for a more traditional or spiritually-focused practice, the name might suggest a more commercial or generic experience. However, the name could also evoke a sense of calm and tranquility, which might be appealing to many. |
| 148 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Tranquil Transitions | 4 | Some criticisms people might have about the name "Tranquil Transitions" for a yoga studio might include: 1. **Generic or Overused**: The name might be perceived as too generic or cliché, as many yoga studios use words like "tranquil" and "transitions" to convey calmness and change. 2. **Lacks Uniqueness**: It might not stand out in a crowded market. Potential clients could confuse it with other studios or services that use similar language. 3. **Misleading Expectations**: The name might suggest a focus solely on relaxation and meditation, which could mislead potential clients if the studio offers more dynamic or physically intense yoga styles. 4. **Limited Appeal**: The name might not appeal to individuals looking for a more fitness-oriented or athletic approach to yoga, as it emphasizes tranquility over physicality. 5. **Cultural Sensitivity**: Some might feel that the name doesn't adequately reflect the cultural roots of yoga, which could be important for those who value authenticity in their practice. 6. **Too Abstract**: For those new to yoga, the name might be too abstract and not clearly convey what the studio offers in terms of classes or benefits. As someone who uses yoga for cross-training, I understand the importance of a studio's vibe aligning with personal fitness goals, so these perceptions can influence whether someone feels drawn to try it out. |
| 149 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Namaste Nest | 4 | People might have a few criticisms about the name "Namaste Nest." Some might feel that it leans too heavily on stereotypical or clichéd elements often associated with yoga, such as the word "Namaste," which can sometimes feel overused or commercialized. Others might find "Nest" to be a bit vague or not immediately indicative of what the studio offers, potentially causing confusion about whether it's a yoga studio or something unrelated. Additionally, some might feel that the name doesn't clearly convey the style or focus of the yoga practice offered there, which could be important for attracting the right clientele. However, the name could also be seen as inviting and cozy, which might appeal to those looking for a welcoming and nurturing environment. |
| 150 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Serenity Studio | 3 | Some criticisms people might have about the name "Serenity Studio" could include that it sounds too generic or cliché, as many yoga and wellness-focused businesses use similar names to convey a sense of calm and peace. People might also feel that it doesn't stand out or reflect the unique qualities or offerings of the studio. Additionally, if the studio is located in a busy or urban area, the name might feel disconnected from its surroundings. Lastly, some might think it doesn't convey enough information about what the studio specifically offers, like whether it focuses on a particular style of yoga or incorporates other wellness practices. |
| 151 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Tranquil Tribe | 4 | Some potential criticisms of the name "Tranquil Tribe" for a yoga studio could include: 1. **Cultural Sensitivity**: The word "tribe" can be sensitive as it may evoke cultural appropriation concerns, especially if not used with respect to indigenous cultures or if it seems to trivialize the significance of tribal identities. 2. **Clarity**: The name might not immediately convey that it's a yoga studio. People might not instantly associate "Tranquil Tribe" with yoga, which could be a consideration for marketing and branding. 3. **Overused Concepts**: The use of "tranquil" might be seen as a cliché in the wellness industry, where many businesses aim to convey peace and calmness. It might not stand out in a crowded market. 4. **Exclusivity**: The word "tribe" might imply exclusivity or a closed community, which could deter newcomers who are looking for an inclusive and welcoming environment. 5. **Relevance**: Depending on the target audience, the name might not resonate with those looking for a more dynamic or athletic style of yoga practice, as "tranquil" suggests a more gentle or restorative approach. As an athlete, I find that the right studio name can really set the tone for what kind of experience people can expect, so it’s worth considering how the name aligns with the studio’s offerings and values. |
| 152 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Inner Light Yoga | 4 | Some criticisms people might have about the name "Inner Light Yoga" could include: 1. **Cliché or Overused:** The phrase "Inner Light" might be considered cliché or overused in the wellness industry, making it less distinctive or memorable. 2. **Lack of Specificity:** The name doesn't immediately convey what makes the studio unique or what types of yoga or services are offered, which could be important for attracting a specific clientele. 3. **Ambiguity:** Some might find the term "Inner Light" to be vague or too abstract, potentially making it hard for people to connect with the brand if they're looking for something more concrete or straightforward. 4. **Cultural Sensitivity:** There could be concerns about the appropriation of spiritual or cultural concepts without a deep understanding or respect for their origins, although this would depend on the studio's approach and philosophy. 5. **Generic Appeal:** It might not stand out in a crowded market where many studios use similar language, making it harder to attract attention among potential clients. Despite these potential criticisms, the name could also evoke a sense of calm and introspection, which might appeal to those seeking a more spiritual or holistic yoga experience. |
| 153 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Mindful Movements | 4 | Some criticisms people might have about the name "Mindful Movements" for a yoga studio could include: 1. **Generic Feel**: The name might be perceived as too generic or overused, lacking uniqueness or a distinctive identity that sets the studio apart from others. 2. **Lack of Specificity**: It may not clearly convey that the studio offers yoga specifically. "Movements" could refer to various types of physical activities, which might confuse potential clients looking specifically for yoga. 3. **Vague Branding**: The term "mindful" is somewhat abstract and might not resonate with individuals who are unfamiliar with mindfulness practices or who prefer a more straightforward approach. 4. **Saturation**: Given the popularity of mindfulness in wellness branding, some might feel the name blends in with a saturated market of similar wellness services. 5. **Cultural Association**: Some might argue that the name doesn't reflect the cultural roots of yoga, which could be important to those who value authenticity in yoga practice. Despite these potential criticisms, the name could still appeal to individuals seeking a holistic and mindful approach to movement and wellness. |
| 154 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Yogic Vibes | 4 | Some criticisms people might have about the name "Yogic Vibes" could include: 1. **Vagueness**: The name might be seen as too generic or vague, as it doesn't specify what makes the studio unique or what kind of yoga experience it offers. 2. **Cliché**: Some might feel that the name relies on overused terms associated with yoga, which could make it seem less original or memorable. 3. **Cultural Sensitivity**: Depending on the target audience, there could be concerns about cultural appropriation if the name doesn't reflect an authentic connection to the cultural roots of yoga. 4. **Lack of Professionalism**: "Vibes" might be perceived as informal, which could be a turn-off for individuals looking for a more serious or professional yoga practice. 5. **Target Audience**: The name might not clearly indicate whether the studio caters to beginners, advanced practitioners, or specific styles of yoga, which could affect how potential clients perceive the studio's offerings. |
| 155 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Harmony Haven | 4 | Some people might find the name "Harmony Haven" to be a bit generic or cliché, as it uses common words associated with peace and tranquility that are often seen in the wellness industry. Others might feel it doesn’t stand out or uniquely represent the specific qualities or style of the yoga studio. Additionally, it might not convey the specific type of yoga or services offered, which could be important for attracting a particular audience. Lastly, if there are other businesses with similar names, it could lead to confusion or difficulty in establishing a distinct brand identity. |
| 156 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Peaceful Postures | 4 | Some criticisms people might have about the name "Peaceful Postures" for a yoga studio could include it being too generic or lacking uniqueness, which might make it harder for the studio to stand out among competitors. Some might feel that it emphasizes only the calming aspect of yoga and overlooks the strength-building or dynamic components that yoga can offer. Additionally, if the studio offers a wide variety of yoga styles, the name might not fully capture the diversity of classes available. |
| 157 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Chakra Chimes | 4 | Some criticisms people might have about the name "Chakra Chimes" for a yoga studio could include: 1. **Cliché or Overused**: The name might be seen as cliché or too stereotypical, as it relies on common yoga-related terms that are frequently used in the industry. 2. **Cultural Appropriation**: Some might argue that using the term "chakra," which has deep roots in Hindu and Buddhist traditions, could be seen as cultural appropriation if not handled with respect and authenticity. 3. **Lack of Clarity**: The name might not clearly convey what the studio offers. While "chakra" relates to energy centers in the body, "chimes" might not directly relate to yoga, leading to confusion about the studio's focus. 4. **Too Abstract**: Potential clients might find the name too abstract or mystical, which could be off-putting to those looking for a more straightforward or fitness-oriented yoga experience. 5. **Niche Appeal**: The name might appeal to a niche audience interested in spiritual or holistic aspects of yoga, potentially alienating those who are more interested in the physical or athletic benefits. As someone who incorporates yoga into my athletic routine, I find it important for a studio name to resonate with the balance of physical and mental well-being, and to feel welcoming to a diverse range of practitioners. |
| 158 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Blissful Balance | 4 | Some criticisms people might have about the name "Blissful Balance" for a yoga studio could include: 1. **Cliché**: The name might be seen as too generic or overused, as many wellness-related businesses aim to convey a sense of peace and equilibrium. 2. **Lack of Uniqueness**: It might not stand out in a crowded market, making it harder for the studio to differentiate itself from others. 3. **Vague**: The name might not clearly convey what makes the studio special or what specific types of yoga or services it offers. 4. **Tone**: Some people might find the name too soft or not reflective of the more dynamic or challenging aspects of yoga practice. 5. **Expectation**: It could set a high expectation for a serene experience, which might not align with every class or instructor’s style. For me, as an athlete, I appreciate a studio name that suggests a balance between mental tranquility and physical challenge, but it’s essential that the name accurately reflects the studio’s unique offerings and atmosphere. |
| 159 | gpt-4o | An athlete incorporating yoga for cross-training and injury prevention | Breathe Bliss | 4 | Some criticisms people might have about the name "Breathe Bliss" for a yoga studio might include: 1. **Cliché**: Some might feel that the name is a bit cliché or overused, as many wellness and yoga-related businesses often incorporate words like "breathe" and "bliss" to convey relaxation and peace. 2. **Lacks Uniqueness**: The combination of these words might not stand out in a crowded market, making it difficult for the studio to differentiate itself from others. 3. **Ambiguity**: The name might not clearly convey what the studio offers, especially to those unfamiliar with yoga. It could be interpreted as a spa or meditation center instead. 4. **Overpromise**: The word "bliss" might suggest an unattainable level of peace or happiness, which could set unrealistic expectations for new clients. 5. **Cultural Sensitivity**: Some might argue that the name doesn’t reflect the cultural roots of yoga, which could be important for those seeking authenticity in their practice. |

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/explore) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we demonstrate how to post this notebook to share with others (visibility can be *public*, *private* or *unlisted* by default):

```python theme={null}
# from edsl import Notebook

# nb = Notebook(path = "yoga_studio_name_survey.ipynb")

# nb.push(
#     description = "Feedback survey",
#     alias = "feedback-survey-notebook",
#     visibility = "public"
# )
```

To update an object at Expected Parrot:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "yoga_studio_name_survey.ipynb") # resave

nb.patch("https://www.expectedparrot.com/content/RobinHorton/feedback-survey-notebook", value = nb)
```


# Overview
Source: https://docs.expectedparrot.com/en/latest/overview

*Expected Parrot Domain-Specific Language* (EDSL) is an open-source package for conducting AI-powered research. EDSL simplifies the creation and execution of surveys, experiments, data labeling tasks, and other research activities involving large numbers of AI agents and language models.

EDSL is developed by [Expected Parrot](https://www.expectedparrot.com/) and available under the MIT License.

## Why EDSL?

Survey research is slow and expensive. Running the same study across different demographic groups, testing how question wording affects responses, or coding thousands of open-ended answers manually — these are common bottlenecks. EDSL addresses this by letting you administer surveys to AI agents that simulate diverse respondents, at scale and at a fraction of the cost of traditional fieldwork.

EDSL was built for researchers who want to work programmatically with language models without becoming AI engineers. Rather than writing loops over agents, questions, and models, you describe your survey once and EDSL handles parallel execution, conditional logic, model calls, and result organization.

Because results are cached on the Expected Parrot server, sharing your research is simple: anyone who runs your published code retrieves your original outputs instantly, without re-calling any models or needing API keys. The platform records what prompts were run, which models were used, and when — providing verifiable provenance for your results. EDSL is also the only open-source library designed specifically for this kind of LLM-based survey research.

## Key concepts

At its core, EDSL is built on the concept of a Question being answered by an AI Agent, using a large language Model to generate a response that is returned as a formatted Result. These results can be analyzed, visualized, shared, or further utilized to refine subsequent questions. EDSL supports various question types (free text, multiple choice, etc.) that can be grouped into a Survey that operates either in parallel or based on specific rules and conditional logic.

A question can also be parameterized with a Scenario to provide contextual data or variations in the prompts that are presented to the language model(s), facilitating simultaneous administration of different versions of questions. This functionality can be particularly useful for data labeling tasks, where a question can be administered for each piece of data in a dataset at once to produce an annotated dataset.

EDSL surveys can also be executed with multiple agents and models concurrently, offering diverse and rich response types, essential for robust data analysis and application.

## A note on AI agents as survey respondents

EDSL uses language models to simulate survey respondents — it does not collect responses from real people unless you explicitly distribute a survey to human participants (see [Humanize](/en/latest/humanize)).

AI-simulated responses reflect patterns in model training data, not the actual opinions or behaviors of any individual or group. They are useful for rapid prototyping, hypothesis generation, pre-testing instruments, and exploring what patterns *might* emerge — but they are not a substitute for human survey data when the goal is to measure real-world attitudes or behaviors.

Best practice is to use AI simulation alongside human data: iterate quickly with EDSL, then validate key findings with actual respondents.

## Key components

The following concepts form the basic classes of the EDSL package, described in detail in the linked pages:

[Questions](/en/latest/questions): Choose from question types like free text, multiple choice, linear scale, and more to define the format of responses.

[Surveys](/en/latest/surveys): Combine questions into a survey administered asynchronously or according to rules. Surveys support multiple agents and models, and can incorporate scenarios for context and data.

[Agents](/en/latest/agents): AI entities with unique traits—background, expertise, memory—that shape their responses to questions.

[Language Models](/en/latest/language_models): EDSL is model-agnostic, supporting multiple models for generating and comparing responses.

[Scenarios](/en/latest/scenarios): Key/value pairs that dynamically parameterize questions, allowing the same question to be asked across varied contexts or datasets.

[Results](/en/latest/results): The output of running a survey—including question, agent, model, prompts, tokens, and raw responses—available for analysis, visualization, and sharing.

## Key features

**Any question format:** Free text, multiple choice, linear scale, checkbox, rank, numerical — EDSL supports a wide range of question types out of the box.

**Run at scale:** Administer surveys to any number of agents, scenarios, and models in parallel. No manual loops required.

**Simulate diverse respondents:** Define agents with demographic traits, personas, or any combination of attributes that matter for your study.

**Cross-model comparison:** Run the same survey with multiple language models and compare responses side by side.

**Built-in analysis:** Filter, sort, and pivot results using pandas-style methods; export to CSV, JSON, or DataFrame.

**Human validation:** Distribute surveys to real respondents via a web link; combine AI and human responses in a single results object.

**Reproducible, shareable results:** Results are automatically cached. Publish your code and anyone can retrieve your exact outputs at no cost — no re-running, no API keys, no setup. The platform records prompts, models, and timestamps, providing verifiable provenance for your research.

**Open source:** Available under the MIT License. Use, modify, and extend freely.

## Expected Parrot Platform

[Expected Parrot](https://www.expectedparrot.com) is the platform behind EDSL. It enhances collaborative research by providing a centralized location for storing and sharing EDSL content and AI research, including:

**Automatic Caching and Versioning:** Automatically store survey results and API calls on the Expected Parrot server to ensure that all aspects of your research are tracked and retrievable.

**Remote Inference:** Run jobs on the Expected Parrot server to offload processing tasks, avoid the need to manage local resources and API keys, and speed up research execution.

See the [Remote Inference](/en/latest/remote_inference) and [Remote Caching](/en/latest/remote_caching) sections for more information.

## Use cases

**Pre-testing survey instruments:** Test question clarity, identify ambiguous wording, and refine response options before fielding to human participants.

**Simulating survey experiments:** Run factorial designs across hundreds of simulated respondents with varying demographic profiles to generate hypotheses or pilot results.

**Data labeling and content analysis:** Apply consistent coding schemes to large text datasets — news articles, social media posts, open-ended responses — at scale.

**Extending existing survey data:** Augment limited human samples with AI-simulated responses to explore patterns or fill demographic gaps.

**Cross-model benchmarking:** Compare how different language models respond to the same questions to understand model-specific biases or variation.

## How it works

EDSL operates by combining these key components to create and execute surveys, generating responses from AI agents using language models. Below we share a few quick examples to illustrate how to use EDSL. Please also see the [Starter Tutorial](/en/latest/starter_tutorial) for a more detailed guide on how to get started with EDSL, including technical setup steps, and the how-to guides and notebooks for examples of special methods and use cases.

### A quick example

An EDSL survey can be as simple as a single question. We select a question type (e.g., multiple choice), construct a question and call the run method to generate a response from a language model:

```python theme={null}
from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
   question_name = "capital",
   question_text = "What is the capital of France?",
   question_options = ["Berlin", "Rome", "Paris", "Madrid", "London"]
)

results = q.run()
```

We can use built-in methods to inspect the response and other components of the results that are generated, such as the name of the model that was used:

```python theme={null}
results.select("model", "capital").print(format="rich")
```

This will return:

| model.model | answer.capital |
| :- | :- |
| gpt-4o | Paris |

### A more complex example

We can administer multiple questions at once by combining them in a Survey. This allows us to add survey rules and agent memory of other questions to control the flow of questions and responses:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, QuestionYesNo, QuestionFreeText, QuestionCheckBox, Survey

q1 = QuestionMultipleChoice(
   question_name = "registered",
   question_text = "Are you currently registered to vote?",
   question_options = ["Yes", "No", "I don't know"]
)
q2 = QuestionYesNo(
   question_name = "eligible",
   question_text = "Are you eligible to vote?"
)
q3 = QuestionFreeText(
   question_name = "factors",
   question_text = "What factors most influence your decision to vote in an election?"
)
q4 = QuestionCheckBox(
   question_name = "issues",
   question_text = "Which issues are most important to you?",
   question_options = ["Economy", "Healthcare", "Education", "Climate change", "National security", "Other"]
)

survey = (
   Survey([q1, q2, q3, q4])  # Add questions to the survey
   .add_skip_rule(q2, "registered == 'Yes'")  # Add conditional logic
   .add_targeted_memory(q4, q3)  # Add agent memory
)
```

### Agents and models

We can also design agents with unique traits and select language models to generate responses:

```python theme={null}
from edsl import AgentList, Agent, ModelList, Model

agents = AgentList(
   Agent(traits = {"party":p, "age":a})
   for p in ["Democrat", "Republican", "Independent"] for a in [25, 60]
)

models = ModelList(
   Model(m) for m in ["gpt-4", "claude-3-5-sonnet-20240620"]
)
```

We can then run the survey with the agents and models we have created, and analyze the results:

```python theme={null}
results = survey.by(agents).by(models).run()


(
   results
   .filter("age == 60")
   .sort_by("model", "party")
   .select("model", "party", "age", "issues")
   .print(pretty_labels = {
      "model.model":"Model",
      "agent.party":"Party",
      "agent.age":"Age",
      "answer.issues":q4.question_text + "n" + ", ".join(q4.question_options)}
      )
)
```

Example output:

| Model | Party | Age | Which issues are most important to you? |
| :- | :- | :- | :- |
| claude-3-5-sonnet-20240620 | Democrat | 60 | \[‘Healthcare’, ‘Education’, ‘Climate change’] |
| claude-3-5-sonnet-20240620 | Independent | 60 | \[‘Economy’, ‘Healthcare’, ‘Education’, ‘Climate change’] |
| claude-3-5-sonnet-20240620 | Republican | 60 | \[‘Economy’, ‘National security’] |
| gpt-4 | Democrat | 60 | \[‘Healthcare’, ‘Education’, ‘Climate change’] |
| gpt-4 | Independent | 60 | \[‘Economy’, ‘Healthcare’, ‘Education’, ‘Climate change’] |
| gpt-4 | Republican | 60 | \[‘Economy’, ‘Healthcare’, ‘National security’] |

### Creating scenarios of questions

We can parameterize questions with context or data to administer multiple versions of questions at once. This is done by creating Scenario objects that are added to a survey in the same way as agents and models. Scenarios can be particularly useful for data labeling tasks or when conducting surveys across different contexts:

```python theme={null}
from edsl import QuestionLinearScale, ScenarioList, Scenario

q6 = QuestionMultipleChoice(
   question_name = "primary_news_source",
   question_text = "What is your primary source of news about {{ scenario.topic }}?",
   question_options = [
      "Television",
      "Online news websites",
      "Social media",
      "Newspapers",
      "Radio",
      "Other"
   ]
)
q7 = QuestionLinearScale(
   question_name = "optimistic",
   question_text = "On a scale from 1 to 10, how optimistic do you feel about {{ scenario.topic }}?",
   question_options = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
   option_labels = {1:"Not at all optimistic", 10:"Very optimistic"}
)

survey = Survey([q6, q7])

scenarios = ScenarioList(
   Scenario({"topic":t}) for t in ["Economy", "Healthcare", "Education", "Climate change", "National security"]
)

results = survey.by(scenarios).by(agents).run()

(
   results
   .filter("optimistic > 7 and age == 25")
   .sort_by("optimistic", "party")
   .select("party", "age", "topic", "primary_news_source", "optimistic")
)
```

Example output:

| agent.party | agent.age | scenario.topic | answer.primary\_news\_source | answer.optimistic |
| :- | :- | :- | :- | :- |
| Democrat | 25 | Education | Online news websites | 8 |

EDSL comes with built-in methods for data analysis and visualization, making it easy to explore and interpret the results of your research. Examples of these methods are provided in the [Results](/en/latest/results#results) section.


# Papers & Citations
Source: https://docs.expectedparrot.com/en/latest/papers

This page provides a list of papers that use or cite EDSL. *Please let us know of any omissions!*

**Are you using EDSL for a research project?**

Send us an email at [info@expectedparrot.com](mailto:info@expectedparrot.com) and we’ll provide credits to run your project for free at the Expected Parrot server.

If you use the EDSL package in your research or projects, please use the following citation:

```bibtex theme={null}
@techreport{Horton2024EDSL,
title = {EDSL: Expected Parrot Domain Specific Language for AI Powered Social Science},
author = {John Horton and Robin Horton},
year = {2024},
institution = {Expected Parrot},
type = {Whitepaper}
}
```

*General Social Agents*

Benjamin S. Manning, John J. Horton <br />
September 2025

[https://arxiv.org/abs/2508.17407](https://arxiv.org/abs/2508.17407)

*The Potential of Synthetic Twin Agents for Personalized Behavioural Interventions at Scale*

J. Ustiyanovych, D. Krpan, <br />
September 2025

[https://osf.io/preprints/psyarxiv/vs2mk\_v3](https://osf.io/preprints/psyarxiv/vs2mk_v3)

*The next generation of experimental research with LLMs*

Gary Charness, Brian Jabarian, John A. List <br />
March 2025

[https://www.nature.com/articles/s41562-025-02137-1](https://www.nature.com/articles/s41562-025-02137-1)

*Social Science Production with LLMs*

Eric Vansteenberghe <br />
February 2025

[https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=5133210](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5133210)

*Linear Representations of Political Perspective Emerge in Large Language Models*

Junsol Kim, James Evans, Aaron Schein <br />
March 2025

[https://arxiv.org/pdf/2503.02080v1](https://arxiv.org/pdf/2503.02080v1)

*Simulating the Survey of Professional Forecasters*

Anne Lundgaard Hansen, John J. Horton, Sophia Kazinnik, Daniela Puzzello, Ali Zarifhonarvar <br />
February 2025

[https://papers.ssrn.com/sol3/papers.cfm?abstract\_id=5066286](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5066286)

*Generative AI for Economic Research: LLMs Learn to Collaborate and Reason*

Anton Korinek <br />
November 2024

[https://www.nber.org/papers/w33198](https://www.nber.org/papers/w33198)

*Theorizing with Large Language Models*

Matteo Tranchero, Cecil-Francis Brenninkmeijer, Arul Murugan, Abhishek Nagaraj <br />
September 2024

[https://www.abhishekn.com/publications-all/gabe-llm](https://www.abhishekn.com/publications-all/gabe-llm)

*ChatGPT vs Social Surveys: Probing the Objective and Subjective Human Society*

Muzhi Zhou, Lu Yu, Xiaomin Geng, Lan Luo <br />
September 2024

[https://arxiv.org/abs/2409.02601](https://arxiv.org/abs/2409.02601)

*Automated Social Science: Language Models as Scientist and Subjects*

Benjamin S. Manning, Kehang Zhu, John J. Horton <br />
April 2024

[https://arxiv.org/abs/2404.11794](https://arxiv.org/abs/2404.11794)

*Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?*

John J. Horton <br />
January 2023

[https://arxiv.org/abs/2301.07543](https://arxiv.org/abs/2301.07543)


# Prolific studies
Source: https://docs.expectedparrot.com/en/latest/prolific

EDSL allows you to launch surveys with human participants using Prolific, a platform for recruiting participants for research studies. You can choose whether to create and manage Prolific studies directly in your EDSL code or interactivately at your Expected Parrot account, enabling you to gather human responses to your surveys alongside AI-generated responses. The steps below outline both methods (code-based and interactive). A clickable demo of interactive steps is available [here](https://www.expectedparrot.com/getting-started/build), and example code below on this page is also available at a downloadable [notebook at Expected Parrot](https://www.expectedparrot.com/content/RobinHorton/coop-project-example).

More information about the humanize method for generating a shareable web version of your survey for human respondents is available at the [Humanize](/en/latest/humanize) page. This method allows you to collect responses from your own respondents. The responses can then be combined with AI-generated responses (and Prolific responses) for analysis.

*Please reach out to us if you have any questions or feature requests, or are interested in testing new features for validating AI responses with participants! You can post a message at our Discord channel or send us an email at [info@expectedparrot.com](mailto:info@expectedparrot.com) - thank you!*

## How it works

You can create a Prolific study for your EDSL survey from your workspace (in code) or from your account dashboard (interactively). You can also choose whether to use your own Prolific account key or your Expected Parrot key (by default).

### Code-based workflow

<Steps>
  <Step title="Create a survey in a notebook">
    Create a survey in EDSL:

    * Construct [Questions](/en/latest/questions) and pass them to [Surveys](/en/latest/surveys) with desired logic.
    * Create AI [Agents](/en/latest/agents) with desired traits.
    * Specify [Language Models](/en/latest/language_models) to generate responses.
    * Use the run method to administer a survey with desired agents and models.

    This generates a formatted dataset of [Results](/en/latest/results) that you can analyze with built-in methods. The survey, agents and results are also automatically added to your Expected Parrot account, where you can access them interactively.
  </Step>

  <Step title="Use humanize to generate a project and web surveys">
    Use the humanize method to generate a Project for your survey and a shareable web version. You can optionally pass a project\_name, survey\_description and survey\_alias to customize the project details (see example below).
  </Step>

  <Step title="Use Expected Parrot to launch Prolific studies">
    Create a `Coop` client object to access your account. Then use `Coop` methods to create a Prolific study for your survey and gather human responses:

    * create\_prolific\_study: Create a new study for the project\_uuid and specify the study details (name, description, num\_participant, estimated\_completion\_time (minutes), participant\_payment\_cents), and optional filters for targeting specific demographics and Prolific participants (IDs list).
    * get\_prolific\_study: Retrieve the study details for the project\_uuid and study\_id.
    * list\_prolific\_filters: Show all the available filters for targeting specific demographics and characteristics of participants. Then use the find method to inspect the available filters by filter\_id and the create\_study\_filter method to create a filter for your study with the desired parameters.
    * update\_prolific\_study: Update the study with the filters.
    * publish\_prolific\_study: Publish the study and make it available for participants.
    * get\_prolific\_study\_responses: Retrieve the responses from the study.
    * approve\_prolific\_study\_submission: Approve the responses from the study.
    * reject\_prolific\_study\_submission: Reject the responses from the study.
  </Step>
</Steps>

If you have also shared the web link with other respondents, you can gather all human responses using the get\_project\_human\_responses method.

### Interactive workflow

<Steps>
  <Step title="Create a survey project">
    [Log in](https://www.expectedparrot.com/login) to your account. Create a project for a saved survey at your **Projects** page or create a new survey and select *Run survey* to view options for running the survey with AI agents and language models, generate a web version of the survey, and launch Prolific studies with human participants.
  </Step>

  <Step title="Create a Prolific study">
    Select the **Run with humans** option and fill in the study details:

    * **Study name** *(Shown to participants)*
    * **Description** *(Shown to participants)*
    * **Required number of participants:** Specify how many participants you want to recruit.
    * **Estimated completion time:** Provide an estimate of how long it will take participants to complete the survey.
    * **Participant payment amount:** Set the payment amount for participants.
    * **Participant allowlist:** Optionally, you can specify a list of Prolific IDs of participants who are allowed to take part in the study.
    * **Configure filters:** Optionally, you can set filters to target specific demographics or characteristics of participants (e.g., age, gender, ethnicity, languages, location, education, work status, etc.).

    <Note>
      *Note:* The **Study URL** will be generated automatically. You can open it to view the web version of the survey that will be sent to participants. This is the same link displayed when you select the **Web survey** option at your project page (or when you use the humanize method in code). You can also share this link with any other respondents on your own to gather their responses.
    </Note>

    Click the **Create study** button to finalize the study creation.
  </Step>

  <Step title="Launch the study">
    Find the new study at your project dashboard and select the option to launch it. Responses will automatically appear at your project dashboard where you can review and approve them.
  </Step>

  <Step title="Access results">
    Copy the project UUID and use it to access the participant responses in your EDSL code (see example below).
  </Step>
</Steps>

## Example (code-based workflow)

The example below demonstrates how to create a survey with different question types, run it with an AI agent and language model, and then gather human responses.

**Create and run a survey with AI agents and models:**

See the [Questions](/en/latest/questions), [Surveys](/en/latest/surveys), [Agents](/en/latest/agents) and [Language Models](/en/latest/language_models) sections for more details on each of these components.

```python expandable theme={null}
# Import modules from EDSL
from edsl import (
    QuestionYesNo,
    QuestionNumerical,
    QuestionLinearScale,
    Survey,
    Agent,
    Model,
    Coop
)

# Create a survey with different question types
q1 = QuestionYesNo(
    question_name="drive",
    question_text="Do you drive?"
)

q2 = QuestionNumerical(
    question_name="count",
    question_text="How many vehicles do you currently own or lease?",
)

q3 = QuestionLinearScale(
    question_name="enjoy",
    question_text="On a scale from 1 to 10, how much do you enjoy driving?",
    question_options=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    option_labels={1: "Hate it", 10: "Love it"},
)

 # Create a survey with the questions
survey = Survey(questions=[q1, q2, q3])

# Create an AI agent to respond to the survey
agent = Agent(
    traits={
        "persona":"Middle-aged mom",
        "location":"Massachusetts",
    }
)

# Select a language model to generate the responses
model = Model("gemini-1.5-pro", service_name="google")

# Run the survey with the AI agent and model
llm_results = survey.by(agent).by(model).run()

# Inspect the results
llm_results.select("persona", "location", "drive", "count", "enjoy")
```

Example output:

| agent.persona | Middle-aged mom |
| :- | :- |
| agent.location | Massachusetts |
| answer.drive | Yes |
| answer.count | 2 |
| answer.enjoy | 4 |

**Create a project and web version of the survey for human respondents:**

Use the humanize method to create a project for the survey and a web version that can be shared with respondents (your own and via Prolific):

```python theme={null}
# Generate a web version of the survey for human respondents
project = survey.humanize(
    project_name = "Vehicle Ownership Survey",  # optional, defaults to the survey name
    survey_description = "A survey on vehicle ownership and driving habits.",  # optional
    survey_alias = "vehicle-ownership-survey"  # optional, used to create a unique URL in addition to Expected Parrot UUID URL
)

# Inspect the project details
project
```

Example output:

```python theme={null}
{'project_name': 'Vehicle Ownership Survey',
'uuid': '369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'admin_url': 'https://www.expectedparrot.com/home/projects/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'respondent_url': 'https://www.expectedparrot.com/respond/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc'}
```

The respondent\_url can be shared with any human respondents (it is the same link shared with Prolific participants).

**Create a study for the project to launch studies on Prolific:**

```python theme={null}
# Create an Expected Parrot client instance
from edsl import Coop
coop = Coop()

project_uuid = project["uuid"]

study = coop.create_prolific_study(
    project_uuid=project_uuid,
    name="Vehicle Ownership Study",
    description="A study on vehicle ownership and driving habits.",
    num_participants=1,
    estimated_completion_time_minutes=1,  # in minutes
    participant_payment_cents=50,  # payment amount in cents
)

# Inspect the study details
study
```

Example output:

```python theme={null}
{'study_id': '684307d08015cf8252ca77cf',
'status': 'UNPUBLISHED',
'admin_url': 'https://www.expectedparrot.com/home/projects/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc/prolific-studies/684307d08015cf8252ca77cf',
'respondent_url': 'https://www.expectedparrot.com/respond/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'name': 'Vehicle Ownership Study',
'description': 'A study on vehicle ownership and driving habits.',
'num_participants': 1,
'estimated_completion_time_minutes': 1,
'participant_payment_cents': 50,
'total_cost_cents': 71,
'device_compatibility': ['desktop', 'mobile', 'tablet'],
'peripheral_requirements': [],
'filters': []}
```

Inspect the study details:

```python theme={null}
# Get the study details
study_id = study["id"]

coop.get_prolific_study(project_uuid, study_id)
```

Example output:

```python theme={null}
{'study_id': '684307d08015cf8252ca77cf',
'status': 'UNPUBLISHED',
'admin_url': 'https://www.expectedparrot.com/home/projects/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc/prolific-studies/684307d08015cf8252ca77cf',
'respondent_url': 'https://www.expectedparrot.com/respond/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'name': 'Vehicle Ownership Study',
'description': 'A study on vehicle ownership and driving habits.',
'num_participants': 1,
'estimated_completion_time_minutes': 1,
'participant_payment_cents': 50,
'total_cost_cents': 71,
'device_compatibility': ['desktop', 'mobile', 'tablet'],
'peripheral_requirements': [],
'filters': []}
```

**Applying filters:**

You can select and apply filters for targeting specific demographics and characteristics of participants. To see a list of available filters:

```python theme={null}
filters = coop.list_prolific_filters()

filters
```

*See the notebook above for the full list of available filters.* *Keys to filter on: ‘range\_filter\_max’, ‘type’, ‘select\_filter\_num\_options’, ‘select\_filter\_options’, ‘question’, ‘filter\_id’, ‘title’, ‘range\_filter\_min’*

There are two types of filters available: select and range. *Select* filters will have select\_filter\_num\_options and select\_filter\_options:

```python theme={null}
filter_id = "current-country-of-residence"  # Example filter ID

filters.find(filter_id)
```

(See notebook for output.)

*Range* filters will have range\_filter\_min and range\_filter\_max:

```python theme={null}
filter_id = "age"  # Example filter ID

filters.find(filter_id)
```

(See notebook for output.)

Create a filter by passing the id and desired parameters (*note*: you can also do this when you create the study). Example *range* filter:

```python theme={null}
# Create a filter for the study
age_filter = filters.create_study_filter(
    filter_id="age",  # Example filter ID
    min=40,
    max=60
)

# Inspect the created filter
age_filter
```

Output:

```python theme={null}
{'filter_id': 'age', 'selected_range': {'lower': 40, 'upper': 60}}
```

Example *select* filter:

```python theme={null}
# Create a filter for the study
country_filter = filters.create_study_filter(
    filter_id="current-country-of-residence",  # Example filter ID
    values=["United States", "Canada"]
)

# Inspect the created filter
country_filter
```

Output:

```python theme={null}
{'filter_id': 'current-country-of-residence', 'selected_values': ['1', '45']}
```

Update the study with the filter:

```python theme={null}
# Update the study with the filter
coop.update_prolific_study(
    project_uuid=project_uuid,
    study_id=study_id,
    filters=[
        age_filter,
        country_filter
    ]  # List of filters to apply to the study
)
```

Example output:

```python theme={null}
{'study_id': '684307d08015cf8252ca77cf',
'status': 'UNPUBLISHED',
'admin_url': 'https://www.expectedparrot.com/home/projects/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc/prolific-studies/684307d08015cf8252ca77cf',
'respondent_url': 'https://www.expectedparrot.com/respond/369b1bdc-11d4-4d22-8eeb-d0fb9eddd3cc',
'name': 'Vehicle Ownership Study',
'description': 'A study on vehicle ownership and driving habits.',
'num_participants': 1,
'estimated_completion_time_minutes': 1,
'participant_payment_cents': 50,
'total_cost_cents': 71,
'device_compatibility': ['desktop', 'mobile', 'tablet'],
'peripheral_requirements': [],
'filters': [{'filter_id': 'age', 'min': 40, 'max': 60},
{'filter_id': 'current-country-of-residence',
'values': ['United States', 'Canada']}]}
```

**Publish the study:** The above steps have created a draft study. To make it available for participants (*note*: this consumes credits):

```python theme={null}
# Publish the study
coop.publish_prolific_study(project_uuid, study_id)
```

**Retrieve the study responses:** After the study is published and participants have completed it, you can retrieve the responses, together with the submission ID for each response, which you can use to approve or reject submissions:

```python theme={null}
# Get the responses from the study
coop.get_prolific_study_responses(project_uuid, study_id)
```

Approve the responses from the study:

```python theme={null}
# Approve the responses from the study
submission_id = "1234567890"  # Example submission ID
coop.approve_prolific_study_submission(
    project_uuid=project_uuid,
    study_id=study_id,
    submission_id=submission_id
)
```

Optionally, you can reject responses if needed:

```python theme={null}
# Reject the responses from the study
submission_id = "1234567890"  # Example submission ID
coop.reject_prolific_study_submission(
    project_uuid=project_uuid,
    study_id=study_id,
    submission_id=submission_id,
    reason="LOW_EFFORT",
    explanation="I think you may have used AI to complete this submission, as there are no personal thoughts or opinions expressed."
)
```

**Gather all human responses:** If you have also shared the web survey link with other respondents, you can gather all human responses from the project:

```python theme={null}
# Get human responses from the web survey link
human_responses = coop.get_project_human_responses(project_uuid)

# Inspect the human responses
human_responses.select("drive", "count", "enjoy")
```

Combine the AI-generated results with the human responses:

```python theme={null}
# Combine results (you can add Results objects for the same survey)
combined_results = llm_results + human_responses

# Inspect the combined results
combined_results.select("persona", "location", "drive", "count", "enjoy")
```

## Costs

Credits for launching Prolific studies are deducted from your credits balance. The total cost of a Prolific study is calculated based on the number of participants, the payment amount you set for each participant, and the Prolific platform fee for each response. These costs are displayed at the project page when you create the study, which include the Prolific platform fee and the payment to participants.

You can view your credits balance at your [Credits](https://www.expectedparrot.com/credits) of your account individual transactions at your [Transactions](https://www.expectedparrot.com/transactions) page.

*Please reach out to us if you have any questions or feature requests! You can post a message at our Discord channel or send us an email at [info@expectedparrot.com](mailto:info@expectedparrot.com)*


# Prompts
Source: https://docs.expectedparrot.com/en/latest/prompts



## Overview

Prompts are texts that are sent to a language model in order to guide it on how to generate responses to questions. They can include questions, instructions or any other textual information to be displayed to the language model.

## Creating & showing prompts

There are two types of prompts:

* A `user_prompt` contains the instructions for a question.
* A `system_prompt` contains the instructions for the agent.

<Note>
  *Note: Some models do not support system prompts, e.g., OpenAI’s o1 models. When using these models the system prompts will be ignored.*
</Note>

### Methods

Methods for displaying prompts are available for both surveys and jobs:

* Calling the show\_prompts() method on a Survey will display the user prompts and the system prompts (if any agents are used) that will be sent to the model when the survey is run.
* Calling the prompts() method on a Job (a survey combined with a model) will return a dataset of the prompts together with information about each question/scenario/agent/model combination and estimated cost.

For example, here we create a survey consisting of a single question and use the show\_prompts() method to inspect the prompts without adding an agent:

```python theme={null}
from edsl import QuestionFreeText, Survey

q = QuestionFreeText(
   question_name = "today",
   question_text = "How do you feel today?"
)

survey = Survey([q])

survey.show_prompts()
```

Output:

| user\_prompt | system\_prompt |
| :- | :- |
| How do you feel today? | |

In this example, the `user_prompt` is identical to the question text because there are no default additional instructions for free text questions, and the `system_prompt` is blank because we did not use an agent.

Here we create an agent, add it to the survey and show the prompts again:

```python theme={null}
from edsl import QuestionFreeText, Survey, Agent

q = QuestionFreeText(
   question_name = "today",
   question_text = "How do you feel today?"
)

agent = Agent(
   traits = {
      "persona": "You are a high school student.",
      "age": 15
   }
)

survey = Survey([q])

survey.by(agent).show_prompts()
```

Output:

| user\_prompt | system\_prompt |
| :- | :- |
| How do you feel today? | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` |

This time we can see that the `system_prompt` includes the default agent instructions (*You are answering questions as if you were a human. Do not break character. Your traits:*) and the agent’s traits.

If we want to see more information about the question, we can create a job that combines the survey and a model, and call the prompts() method:

```python theme={null}
from edsl import Model

model = Model("gpt-4o")

survey.by(agent).by(model).prompts()
```

Output:

| user\_prompt | system\_prompt | interview\_index | question\_name | scenario\_index | agent\_index | model | estimated\_cost |
| :- | :- | :- | :- | :- | :- | :- | :- |
| How do you feel today? | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` | 0 | today | 0 | 0 | gpt-4o | 0.0004125 |

### Modifying agent instructions

An agent can also be constructed with an optional instruction. This text is added to the beginning of the `system_prompt`, replacing the default instructions *“You are answering questions as if you were a human. Do not break character.”* Here we create agents with and without an instruction and compare the prompts:

```python theme={null}
from edsl import AgentList, Agent

agents = AgentList([
   Agent(
      traits = {"persona": "You are a high school student.", "age": 15}
      # no instruction
   ),
   Agent(
      traits = {"persona": "You are a high school student.", "age": 15},
      instruction = "You are tired."
   )
])

survey.by(agents).show_prompts() # using the survey from the previous examples
```

Output:

| user\_prompt | system\_prompt |
| :- | :- |
| How do you feel today? | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` |
| How do you feel today? | You are tired. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` |

If we use the prompts() method to see more details, we will find that the agent\_index is different for each agent, allowing us to distinguish between them in the survey results, and the interview\_index is also incremented for each question/agent/model combination:

```python theme={null}
survey.by(agents).by(model).prompts() # using the survey, agents and model from examples above
```

Output:

| user\_prompt | system\_prompt | interview\_index | question\_name | scenario\_index | agent\_index | model | estimated\_cost |
| :- | :- | :- | :- | :- | :- | :- | :- |
| How do you feel today? | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` | 0 | today | 0 | 0 | gpt-4o | 0.0004125 |
| How do you feel today? | You are tired. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` | 1 | today | 0 | 1 | gpt-4o | 0.000265 |

### Agent names

Agents can also be constructed with an optional unique name parameter which does *not* appear in the prompts but can be useful for identifying agents in the results. The name is stored in the agent\_name column that is automatically added to the results. The default agent name in results is “Agent” followed by the agent’s index in the agent list (e.g. “Agent\_0”, “Agent\_1”, etc.).

If you want to reference a name for an agent, you can do so by creating a trait for it. You can then use the trait name in a question prompt to reference the agent’s name.

For example:

```python theme={null}
from edsl import Agent, QuestionFreeText

a = Agent(traits = {"first_name": "John"})

q = QuestionFreeText(
   question_name = "exmaple",
   question_text = "What is your last name, {{agent.first_name }}?"
)

job = q.by(a)
job.prompts().select("`user_prompt`")
```

Output:

| user\_prompt |
| :- |
| What is your last name, John? |

Learn more about designing [Agents](/en/latest/agents) and accessing columns in [Results](/en/latest/results).

### Instructions for question types

In the examples above, the `user_prompt` for the question was identical to the question text. This is because the question type was free text, which does not include additional instructions by default. Question types other than free text include additional instructions in the `user_prompt` that are specific to the question type.

For example, here we create a multiple choice question and inspect the user prompt:

```python theme={null}
from edsl import QuestionMultipleChoice, Survey

q = QuestionMultipleChoice(
   question_name = "favorite_subject",
   question_text = "What is your favorite subject?",
   question_options = ["Math", "English", "Social studies", "Science", "Other"]
)

survey = Survey([q])

survey.by(agent).prompts().select("`user_prompt`") # to display just the user prompt
```

Output:

| user\_prompt |
| :- |
| What is your favorite subject? <br />  Math <br /> English <br /> Social studies <br /> Science <br /> Other <br /> Only 1 option may be selected. <br /> Respond only with a string corresponding to one of the options. <br /> After the answer, you can put a comment explaining why you chose that option on the next line. |

In this case, the `user_prompt` for the question includes both the question text and the default instructions for multiple choice questions: *“Only one answer may be selected…”* Other question types have their own default instructions that specify how the response should be formatted.

Learn more about the different question types in the [Questions](/en/latest/questions) section of the documentation.

### Comments

The user prompt for the multiple choice question above also includes an instruction for the model to provide a comment about its answer: *“After the answer, you can put a comment explaining why you chose that option on the next line.”* All questions types other than free text automatically include a “comment” which is stored in a separate field in the survey results. (The field is blank for free text questions.) Comments are not required, but can be useful for understanding a model’s reasoning, or debugging a non-response. They can also be useful when you want to simulate a “chain of thought” by giving an agent context of prior questions and answers in a survey. Comments can be turned off by passing a parameter include\_comment = False to the question constructor.

Learn more about using question memory and piping comments or other question components in the [Surveys](/en/latest/surveys) section of the documentation.

For example, here we modify the multiple choice question above to not include a comment and show the resulting user prompt:

```python theme={null}
from edsl import QuestionMultipleChoice, Survey

q = QuestionMultipleChoice(
   question_name = "favorite_subject",
   question_text = "What is your favorite subject?",
   question_options = ["Math", "English", "Social studies", "Science", "Other"],
   include_comment = False
)

survey = Survey([q])

survey.by(agent).prompts().select("`user_prompt`") # using the agent and model from previous examples
```

Output:

| user\_prompt |
| :- |
| What is your favorite subject?  Math  English  Social studies  Science  Other  Only 1 option may be selected.  Respond only with a string corresponding to one of the options. |

There is no longer any instruction about a comment at the end of the user prompt.

### Prompts for multiple questions

If a survey consists of multiple questions, the show\_prompts() and prompts() methods will display all of the prompts for each question/scenario/model/agent combination in the survey.

For example:

```python theme={null}
from edsl import QuestionMultipleChoice, QuestionYesNo, Survey

q1 = QuestionMultipleChoice(
   question_name = "favorite_subject",
   question_text = "What is your favorite subject?",
   question_options = ["Math", "English", "Social studies", "Science", "Other"]
)

q2 = QuestionYesNo(
   question_name = "college_plan",
   question_text = "Do you plan to go to college?"
)

survey = Survey([q1, q2])

survey.by(agent).by(model).prompts() # using the agent and model from previous examples
```

Output:

| user\_prompt | system\_prompt | interview\_index | question\_name | scenario\_index | agent\_index | model | estimated\_cost |
| :- | :- | :- | :- | :- | :- | :- | :- |
| What is your favorite subject?  Math  English  Social studies  Science  Other  Only 1 option may be selected.  Respond only with a string corresponding to one of the options.  After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` | 0 | favorite\_subject | 0 | 0 | gpt-4o | 0.001105 |
| Do you plan to go to college?  No  Yes  Only 1 option may be selected.  Please respond with just your answer.  After the answer, you can put a comment explaining your response. | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` | 0 | college\_plan | 0 | 0 | gpt-4o | 0.00084 |

## Modifying prompts

Templates for default prompts are provided in the edsl.prompts.library module. These prompts can be used as is or customized to suit specific requirements by creating new classes that inherit from the Prompt class.

Typically, prompts are created using the Prompt class, a subclass of the PromptBase class which is an abstract class that defines the basic structure of a prompt. The Prompt class has the following attributes (see examples above):

* `user_prompt`: A list of strings that contain the text that will be sent to the model.
* `system_prompt`: A list of strings that contain the text that will be sent to the model.
* interview\_index: An integer that specifies the index of the interview.
* question\_name: A string that specifies the name of the question.
* scenario\_index: An integer that specifies the index of the scenario.
* agent\_index: An integer that specifies the index of the agent.
* model: A string that specifies the model to be used.
* estimated\_cost: A float that specifies the estimated cost of the prompt.

## Inspecting prompts after running a survey

After a survey is run, we can inspect the prompts that were used by selecting the prompt.\* fields of the results.

For example, here we run the survey from above and inspect the prompts that were used:

```python theme={null}
results = survey.by(agent).by(model).run()

To select all the prompt columns at once:

results.select("prompt.*")
```

Output:

| prompt.favorite\_subject\_user\_prompt | prompt.college\_plan\_user\_prompt | prompt.favorite\_subject\_system\_prompt | prompt.college\_plan\_system\_prompt |
| :- | :- | :- | :- |
| What is your favorite subject? <br />  Math <br /> English <br /> Social studies <br /> Science <br /> Other <br /> Only 1 option may be selected. <br /> Respond only with a string corresponding to one of the options. <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | Do you plan to go to college? <br />  No <br /> Yes <br /> Only 1 option may be selected. <br /> Please respond with just your answer. <br /> After the answer, you can put a comment explaining your response. | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` |

Or to specify the order in the table we can name them individually:

```python theme={null}
(
   results.select(
      "favorite_subject_`system_prompt`",
      "college_plan_`system_prompt`",
      "favorite_subject_`user_prompt`",
      "college_plan_`user_prompt`"
   )
)
```

Output:

| prompt.favorite\_subject\_`system_prompt` | prompt.college\_plan\_`system_prompt` | prompt.favorite\_subject\_`user_prompt` | prompt.college\_plan\_`user_prompt` |
| :- | :- | :- | :- |
| You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` | You are answering questions as if you were a human. Do not break character. Your traits: `{‘persona’: ‘You are a high school student.’, ‘age’: 15}` | What is your favorite subject? <br />  Math <br /> English <br /> Social studies <br /> Science <br /> Other <br /> Only 1 option may be selected. <br /> Respond only with a string corresponding to one of the options. <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | Do you plan to go to college? <br />  No <br /> Yes <br /> Only 1 option may be selected. <br /> Please respond with just your answer. <br /> After the answer, you can put a comment explaining your response. |

## Saving and sharing prompts

A `Prompt` can be saved to Expected Parrot and reused like any other object: pushed, pulled, shared, copied and deleted. This is useful for research instructions or Jinja templates that you want to reuse across projects or share with collaborators.

Saving and sharing prompts requires EDSL 1.0.8 or later.

### Pushing and pulling

```python theme={null}
from edsl import Prompt

prompt = Prompt("Explain {{ topic }} for {{ audience }}.")
info = prompt.push(alias="explanation-template", visibility="private")
```

Pull a prompt by its UUID, by its owner and alias, or by its content URL:

```python theme={null}
prompt = Prompt.pull(info["uuid"])
prompt = Prompt.pull("alice/explanation-template")
prompt = Prompt.pull("https://www.expectedparrot.com/content/alice/explanation-template")
```

`Prompt.pull` only returns prompts: pulling the UUID or alias of a different kind of object raises an error.

The prompt's text is stored exactly as written, including whitespace and Jinja syntax. Templates are not rendered when they are saved or pulled; call `render` when you want to fill them in:

```python theme={null}
prompt.template_variables()
# ['audience', 'topic']

prompt.render({"topic": "inflation", "audience": "high-school students"})
# Prompt(text="""Explain inflation for high-school students.""")
```

To update a prompt you have already pushed, use `patch` with its UUID or content URL:

```python theme={null}
Prompt.patch(info["uuid"], value=Prompt("Explain {{ topic }} simply."))
```

Descriptions, aliases, visibility and sharing work the same way as for other objects. See [Coop](/en/latest/coop) for details.

### Saving prompts locally

Prompts are saved as JSON files:

```python theme={null}
prompt.save("explanation-template.json")
prompt = Prompt.load("explanation-template.json")
```

The `edsl` command line tool can inspect, validate and push these files:

```bash theme={null}
edsl inspect explanation-template.json
edsl validate --file explanation-template.json
edsl push explanation-template.json --alias explanation-template --visibility private
```

<Note>
  *Note: Prompts cannot be saved as `.ep` packages, so `edsl clone`, `edsl pull` and other package commands do not support them yet. Use JSON files, or `Prompt.pull` from Python, instead.*
</Note>

## More about question prompts

See the [Questions](/en/latest/questions) section for [more details](/en/latest/questions) on how to create and customize question prompts with question\_presentation and answering\_instructions parameters in the Question type constructor.

## Prompts class

> ### *class* edsl.prompts.Prompt(*text: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Bases: `str`, `PersistenceMixin`, `RepresentationMixin`

Class for creating a prompt to be used in a survey.

> #### **init**(*text: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Create a Prompt object.

**Parameters:**

**text** – The text of the prompt.

> #### default\_instructions: *str | None* = 'Do good things, friendly LLM!'\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

> #### *classmethod* example() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Return an example of the prompt.

> #### *classmethod* from\_dict(*data*) → [Prompt](/en/latest/#edsl.prompts.Prompt "edsl.prompts.prompt.Prompt") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Create a Prompt from a dictionary.

Example:

```python theme={null}
>>> p = Prompt("Hello, {{person}}")
>>> p2 = Prompt.from_dict(p.to_dict())
>>> p2
Prompt(text="""Hello, {{person}}""")
```

> #### *classmethod* from\_template(*file\_name: str*, *path\_to\_folder: str | Path | None = None*, \*\**kwargs: Dict\[str, Any]*) → [Prompt](/en/latest/#edsl.prompts.Prompt "edsl.prompts.prompt.Prompt") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Create a PromptBase from a Jinja template.

> **Args:**

file\_name (str): The name of the Jinja template file. path\_to\_folder (Union\[str, Path]): The path to the folder containing the template.

Can be absolute or relative.

\*\*kwargs: Variables to be passed to the template for rendering.

> **Returns:**

PromptBase: An instance of PromptBase with the rendered template as text.

> #### *classmethod* from\_txt(*filename: str*) → [Prompt](/en/latest/#edsl.prompts.Prompt "edsl.prompts.prompt.Prompt") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Create a Prompt from text.

**Parameters:**

**text** – The text of the prompt.

> #### get\_prompts() → Dict\[str, Any] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Get the prompts for the question.

```python theme={null}
>>> p = Prompt("Hello, {{person}}")
```

> #### *property* has\_variables\*: *bool* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Return True if the prompt has variables.

Example:

```python theme={null}
>>> p = Prompt("Hello, {{person}}")
>>> p.has_variables
True
```

```python theme={null}
>>> p = Prompt("Hello, person")
>>> p.has_variables
False
```

> #### *classmethod* prompt\_attributes() → List\[str] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Return the prompt class attributes.

> #### render(*primary\_replacement: dict*, \*\**additional\_replacements*) → [Prompt](/en/latest/#edsl.prompts.Prompt "edsl.prompts.prompt.Prompt") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Render the prompt with the replacements.

**Parameters:**

* **primary\_replacement** – The primary replacement dictionary.
* **additional\_replacements** – Additional replacement dictionaries.

```python theme={null}
>>> p = Prompt("Hello, {{person}}")
>>> p.render({"person": "John"})
Prompt(text="""Hello, John""")
```

```python theme={null}
>>> p.render({"person": "Mr. {{last_name}}", "last_name": "Horton"})
Prompt(text="""Hello, Mr. Horton""")
```

```python theme={null}
>>> p.render({"person": "Mr. {{last_name}}", "last_name": "Ho{{letter}}ton"}, max_nesting = 1)
Prompt(text="""Hello, Mr. Ho{{letter }}ton""")
```

```python theme={null}
>>> p.render({"person": "Mr. {{last_name}}"})
Prompt(text="""Hello, Mr. {{last_name }}""")
```

```python theme={null}
>>> p = Prompt("The sum is {% set x = 2 + 3 %}{{vars.set('x', x) }}{{x}}")
>>> result = p.render({})
>>> print(result.captured_variables)
{'x': 5}
>>> result.captured_variables['x']
5
```

> #### template\_variables() → list\[str] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Return the variables in the template.

> #### *property* text: *str* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Return the Prompt text.

> #### to\_dict(*add\_edsl\_version=False*) → dict\[str, Any] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Return the Prompt as a dictionary.

Example:

```python theme={null}
>>> p = Prompt("Hello, {{person}}")
>>> p.to_dict()
{'text': 'Hello, {{person}}', 'class_name': 'Prompt'}
```

> #### undefined\_template\_variables(*replacement\_dict: dict*) → list\[str] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Return the variables in the template that are not in the replacement\_dict.

**Parameters:**

**replacement\_dict** – A dictionary of replacements to populate the template.

Example:

```python theme={null}
>>> p = Prompt("Hello, {{person}}")
>>> p.undefined_template_variables({"person": "John"})
[]
```

```python theme={null}
>>> p = Prompt("Hello, {{title}} {{person}}")
>>> p.undefined_template_variables({"person": "John"})
['title']
```

> #### unused\_traits(*traits: dict*) → list\[str] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/prompts.py)

Return the traits that are not used in the template.


# Questions
Source: https://docs.expectedparrot.com/en/latest/questions

EDSL provides templates for many common question types, including free text, multiple choice, checkbox, linear scale, matrix, numerical, dictionary and others. The Question class has subclasses for each of these types: QuestionFreeText, QuestionMultipleChoice, QuestionCheckBox, QuestionLinearScale, QuestionMatrix, QuestionNumerical, QuestionDict, etc., which have methods for validating answers and responses from language models.

## Question type templates

A question is constructed by creating an instance of a question type class and passing the required fields. Questions are formatted as dictionaries with specific keys based on the question type.

### Question types

The following question types are available:

* QuestionFreeText - free text questions
* QuestionMultipleChoice - multiple choice questions
* QuestionCheckBox - checkbox questions
* QuestionLinearScale - linear scale questions
* QuestionMatrix - matrix questions
* QuestionDict - dictionary questions
* QuestionLikertFive - Likert scale questions
* QuestionRank - ranked list questions
* QuestionTopK - top-k list questions
* QuestionYesNo - yes/no questions (multiple choice with fixed options)
* QuestionNumerical - numerical questions
* QuestionList - list questions (the response is formatted as a list of strings)
* QuestionBudget - budget allocation questions (the response is a dictionary of allocated amounts)
* QuestionDistribution - probability allocations over named outcomes or numerical intervals
* QuestionExtract - information extraction questions (the response is formatted according to a specified template)
* QuestionFunctional - functional questions (the response is generated by a function)
* QuestionDropdown - BM25-powered search through large option sets (hundreds or thousands of options)
* QuestionInterview - simulates interview dialogue between interviewer and respondent
* SurveyMessage - a display-only, routable survey step for introductions, transitions, notices, and completion pages; see [Survey messages](/en/latest/survey_messages)

### Base parameters

All question types require a question\_name and question\_text. The question\_name is a unique Pythonic identifier for a question (e.g., “favorite\_color”). The question\_text is the text of the question itself written as a string (e.g., “What is your favorite color?”). Question types other than free text also require a question\_options list of possible answer options. The question\_options list can be a list of strings, floats, integers, dictionaries, lists or other data types depending on the question type. (See a [demo notebook](https://www.expectedparrot.com/content/a8f39497-3e6e-4d40-aeaf-2b7f78cf4721) of allowed question options types at Expected Parrot.)

For example, to create a multiple choice question where the response should be a single option selected from a list of colors, we import the QuestionMultipleChoice class and create an instance of it with the required fields:

```python theme={null}
from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
   question_name = "favorite_color",
   question_text = "What is your favorite color?",
   question_options = ["Red", "Orange", "Yellow", "Green", "Blue", "Purple"]
)
```

### Special parameters

Some question types have additional parameters that are either optional or required to be added to the question when it is created:

answer\_keys - *Required* parameter of dict (dictionary) questions to specify the keys that the respondent must provide in their answer. For example, in a dictionary question where the respondent must provide the names of their favorite colors and the number of times they have worn each color in the past week:

```python theme={null}
from edsl import QuestionDict

q = QuestionDict(
   question_name = "favorite_colors",
   question_text = "What are your favorite colors and how many times have you worn each color in the past week?",
   answer_keys = ["color", "times_worn"]
)
```

value\_types - *Optional* parameter of dict questions to specify the types of the values that the respondent must provide for each key. Permissible types are str, int, float, list and their string representations (e.g., “str”, “int”, “float”, “list”). For example, in a dictionary question where the respondent must provide the names of their favorite colors and the number of times they have worn each color in the past week, and the values must be a string and an integer, respectively:

```python theme={null}
from edsl import QuestionDict

q = QuestionDict(
   question_name = "favorite_colors",
   question_text = "What are your favorite colors and how many times have you worn each color in the past week?",
   answer_keys = ["color", "times_worn"],
   value_types = [str, int] # optional
)
```

<Note>
  **Note**:

  Note that the types can be provided as actual types (e.g., str, int, float, list, dict, etc.) or as strings (e.g., “str”, “int”, “float”, “list”, “dict”, etc.).
</Note>

value\_descriptions - *Optional* parameter of dict questions to specify descriptions of the values that the respondent must provide for each key. For example, in a dictionary question where the respondent is asked to draft a tweet and provide the text and the number of characters in the tweet:

```python theme={null}
from edsl import QuestionDict

q = QuestionDict(
   question_name = "tweet",
   question_text = "Draft a tweet.",
   answer_keys = ["text", "characters"],
   value_descriptions = ["The text of the tweet", "The number of characters in the tweet"] # optional
)
```

question\_items - *Required* parameter of matrix questions to specify the row items. For example, in a matrix question where the respondent must rate a list of items on a scale of 1 to 5:

```python theme={null}
from edsl import QuestionMatrix

q = QuestionMatrix(
   question_name = "rate_items",
   question_text = "Please rate the following items on a scale of 1 to 5.",
   question_items = ["Item 1", "Item 2", "Item 3"],
   question_options = [1, 2, 3, 4, 5],
   option_labels = {1: "Terrible", 5: "Excellent"} # optional
)
```

option\_labels - *Optional* parameter of linear\_scale and matrix questions to specify labels for the scale options. For example, in a linear scale question where the response must be an integer between 1 and 5 reflecting the respondent’s agreement with a statement:

```python theme={null}
from edsl import QuestionLinearScale

q = QuestionLinearScale(
   question_name = "agree",
   question_text = "Please indicate whether you agree with the following statement: I am only happy when it rains.",
   question_options = [1, 2, 3, 4, 5],
   option_labels = {1: "Strongly disagree", 5: "Strongly agree"} # optional
)
```

min\_selections and max\_selections - *Optional* parameters of checkbox and rank questions to specify the minimum and maximum number of options that can be selected. For example, in a checkbox question where the response must include at least 2 and at most 3 of the options:

```python theme={null}
from edsl import QuestionCheckBox

q = QuestionCheckBox(
   question_name = "favorite_days",
   question_text = "What are your 2-3 favorite days of the week?",
   question_options = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"],
   min_selections = 2, # optional
   max_selections = 3 # optional
)
```

min\_value and max\_value - *Optional* parameters of numerical questions to specify the minimum and maximum values that can be entered. For example, in a numerical question where the respondent must enter a number between 1 and 100:

```python theme={null}
from edsl import QuestionNumerical

q = QuestionNumerical(
   question_name = "age",
   question_text = "How old are you (in years)?",
   min_value = 1, # optioal
   max_value = 100 # optional
)
```

num\_selections - *Optional* parameter of rank questions to specify the number of options that must be ranked. For example, in a rank question where the respondent must rank their top 3 favorite foods:

```python theme={null}
from edsl import QuestionRank

q = QuestionRank(
   question_name = "foods_rank",
   question_text = "Rank your top 3 favorite foods.",
   question_options = ["Pizza", "Pasta", "Salad", "Soup"],
   num_selections = 3 # optional
)
```

use\_code - *Optional* boolean parameter of multiple\_choice questions that adds an instruction to the user\_prompt for the model to provide the code number of the question option that it selects as its answer (i.e., 0, 1, 2, etc.) instead of the value of the option. This can be useful when the question options are long or complex, or include formatting that a model may make errors in reproducing to provide an answer, resulting in a validation error that may be avoidable by returning the code number of the option instead. The code is then translated back to the option value in the survey results. For example, in a multiple choice question where the agent is instructed to select a programming language we can add the use\_code parameter and then inspect how the user prompt is modified to include *“Respond only with the code corresponding to one of the options.”* (learn more about constructing agents in the \[Agents]\(/en/latest/agents

```python theme={null}
from edsl import QuestionMultipleChoice, Survey, Agent

a = Agent(traits = {"persona":"You are an experienced computer programmer."})

q = QuestionMultipleChoice(
   question_name = "programming_language",
   question_text = "Which programming language do you prefer?",
   question_options = ["Python", "Java", "C++", "JavaScript"],
   use_code = True # optional
)

survey = Survey([q])

survey.by(a).show_prompts()
```

Output:

| user\_prompt | system\_prompt |
| :- | :- |
| Which programming language do you prefer? 0: Python 1: Java 2: C++ 3: JavaScript  Only 1 option may be selected.  Respond only with the code corresponding to one of the options.  After the answer, you can put a comment explaining why you chose that option on the next line. | You are answering questions as if you were a human. Do not break character. Your traits: `{'persona': 'You are an experienced computer programmer.'}` |

answer\_template - *Required* parameter of extract questions to specify a template for the extracted information. For example, in an extract question where the respondent must extract information from a given text:

```python theme={null}
from edsl import QuestionExtract

q = QuestionExtract(
   question_name = "course_schedule",
   question_text = "This semester we are offering courses on calligraphy on Friday mornings.",
   answer_template = {"course_topic": "AI", "days": ["Monday", "Wednesday"]} # required
)
```

func - *Required* parameter of functional questions to specify a function that generates the answer. For example, in a functional question where the answer is generated by a function:

```python expandable theme={null}
from edsl import QuestionFunctional, ScenarioList, Scenario
import random

scenarios = ScenarioList(
   [Scenario({"persona": p, "random": random.randint(0, 1000)}) for p in ["Magician", "Economist"]]
)

def my_function(scenario, agent_traits):
   if scenario.get("persona") == "Magician":
      return "Magicians never pick randomly!"
   elif scenario.get("random") > 500:
      return "Top half"
   else:
      return "Bottom half"

q = QuestionFunctional(
   question_name = "evaluate",
   func = my_function
)

results = q.by(scenarios).run()

results.select("persona", "random", "evaluate")
```

Example results:

| scenario.persona | scenario.random | answer.evaluate |
| :- | :- | :- |
| Magician | 358 | Magicians never pick randomly! |
| Economist | 826 | Top half |

### General optional parameters

The following optional parameters can be added to any question type (examples of each are provided at the end of this section):

include\_comment = False - A boolean value that can be added to any question type (other than free text) to exclude the default instruction in the user prompt that instructs the model to include a comment about its response to the question, as well as the comment field that is otherwise automatically included in survey results for the question.

By default, a comment field is automatically added to all question types other than free text. It is a free text field that allows a model to provide any commentary on its response to the question, such as why a certain answer was chosen or how the model arrived at its answer. It can be useful for debugging unexpected responses and reducing the likelihood that a model fails to follow formatting instructions for the main response (e.g., just selecting an answer option) because it wants to be more verbose. It can also be helpful in constructing sequences of questions with context about prior responses (e.g., simulating a chain of thought). (See the survey section for more information about adding question memory to a survey.)

question\_presentation - A string that can be added to any question type to specify how the question should be presented to the model. It can be used to provide additional context or instructions to the model about how to interpret the question.

answering\_instructions - A string that can be added to any question type to specify how the model should answer the question. It can be used to provide additional context or instructions to the model about how to format its response.

permissive = True - A boolean value that can be added to any question type to specify whether the model should be allowed to provide an answer that violates the question constraints (e.g., selecting fewer or more than the allowed number of options in a checkbox question). (By default, permissive is set to False to enforce any question constraints.)

## Creating a survey

We can combine multiple questions into a survey by passing them as a list to a Survey object:

```python expandable theme={null}
from edsl import QuestionLinearScale, QuestionList, QuestionNumerical, Survey

q1 = QuestionLinearScale(
   question_name = "dc_state",
   question_text = "How likely is Washington, D.C. to become a U.S. state?",
   question_options = [1, 2, 3, 4, 5],
   option_labels = {1: "Not at all likely", 5: "Very likely"}
)

q2 = QuestionList(
   question_name = "largest_us_cities",
   question_text = "What are the largest U.S. cities by population?",
   max_list_items = 3
)

q3 = QuestionNumerical(
   question_name = "us_pop",
   question_text = "What was the U.S. population in 2020?"
)

survey = Survey(questions = [q1, q2, q3])
```

This allows us to administer multiple questions at once, either asynchronously (by default) or according to specified logic (e.g., skip or stop rules). To learn more about designing surveys with conditional logic, please see the [Surveys](/en/latest/surveys#surveys) section.

*Note:* If you want multiple choice question options to be randomized, you can pass an optional parameter questions\_to\_randomize (a list of the relevant question names) to the Survey object when it is created. See more details about QuestionMultipleChoice below and the [Surveys](/en/latest/surveys#surveys) section on randomizing question options.

## Simulating a response

We generate a response to a question by delivering it to a language model. This is done by calling the run method for the question:

```python theme={null}
from edsl import QuestionCheckBox

q = QuestionCheckBox(
   question_name = "primary_colors",
   question_text = "Which of the following colors are primary?",
   question_options = ["Red", "Orange", "Yellow", "Green", "Blue", "Purple"]
)

results = q.run()
```

This will generate a Results object that contains a single Result representing the response to the question and information about the model used. If the model to be used has not been specified (as in the above example), the run method delivers the question to the default LLM (run Model() to check the current default LLM). We can inspect the response and model used by calling the select method and passing it the names of components of the results that we want to display in a table. For example, we can print just the answer to the question:

```python theme={null}
results.select("primary_colors")
```

Output:

| answer.primary\_colors |
| :- |
| \[‘Red’, ‘Yellow’, ‘Blue’] |

Or to inspect the model:

```python theme={null}
results.select("model")
```

Output:

| model.model |
| :- |
| gpt-4o |

If questions have been combined in a survey, the run method is called directly on the survey instead:

```python expandable theme={null}
from edsl import QuestionLinearScale, QuestionList, QuestionNumerical, Survey

q1 = QuestionList(
   question_name = "largest_us_cities",
   question_text = "What are the largest U.S. cities by population?",
   max_list_items = 3
)

q2 = QuestionLinearScale(
   question_name = "dc_state",
   question_text = "How likely is Washington, D.C. to become a U.S. state?",
   question_options = [1, 2, 3, 4, 5],
   option_labels = {1: "Not at all likely", 5: "Very likely"}
)

q3 = QuestionNumerical(
   question_name = "us_pop",
   question_text = "What was the U.S. population in 2020?"
)

survey = Survey(questions = [q1, q2, q3])

results = survey.run()
```

We can inspect the results in the same way as for a single question, and use “\*” to select all components of the results or group of component (e.g., all of the answers):

```python theme={null}
results.select("answer.*")
```

Output:

| answer.largest\_us\_cities | answer.dc\_state | answer.us\_pop |
| :- | :- | :- |
| \[‘New York’, ‘Los Angeles’, ‘Chicago’] | 2 | 331449281 |

For a survey, each Result represents a response for the set of survey questions. To learn more about analyzing results, please see the [Results](/en/latest/results#results) section.

## Parameterizing a question

A question can also be constructed to take parameters that are replaced with specified values either when the question is constructed or when the question is run. This operation can be done in a number of ways:

* Use “piping” to pass answers or other components of previous questions to a subsequent question *when it is run*.
* Use Scenario objects to pass values to a question *when it is run*.
* Use Scenario objects to pass values to a question *when it is constructed*.
* Use f-strings to pass values to a question *when it is constructed*.

Each of these methods allows us to easily create and administer multiple versions of a question at once.

In addition to the examples below, please also see the [Scenarios](/en/latest/scenarios) section for more information on constructing and using scenarios to parameterize questions.

### Scenarios

A Scenario object is a dictionary of parameter values that can be passed to a question. Details about scenarios can be found in the [Scenarios](/en/latest/scenarios) section.

*Key steps*:

Create a question text that takes a parameter in double braces:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
   question_name = "favorite_item",
   question_text = "What is your favorite {``{ item }``}?",
)
```

Then create a dictionary for each value that will replace the parameter and store them in Scenario objects:

```python theme={null}
from edsl import ScenarioList, Scenario

scenarios = ScenarioList(
   Scenario({"item": item}) for item in ["color", "food"]
)
```

We can pass scenarios to a question using the by method when the question is run:

```python theme={null}
from edsl import QuestionFreeText, ScenarioList, Scenario

q = QuestionFreeText(
   question_name = "favorite_item",
   question_text = "What is your favorite {``{ item }``}?",
)

scenarios = ScenarioList(
   Scenario({"item": item}) for item in ["color", "food"]
)

results = q.by(scenarios).run()
```

Each of the Results that are generated will include an individual Result for each version of the question that was answered.

Alternatively, we can create multiple versions of a question when constructing a survey (i.e., before running it), by passing scenarios to the question loop method:

```python theme={null}
questions = q.loop(scenarios) # using the scenarios from the example above
```

We can inspect the questions that have been created:

```python theme={null}
questions
```

Output:

```
[Question('free_text', question_name = """favorite_item_0""", question_text = """What is your favorite color?"""),
Question('free_text', question_name = """favorite_item_1""", question_text = """What is your favorite food?""")]
```

<Note>
  **Note**:

  Note that a unique question\_name has been automatically generated based on the parameter values. This is necessary in order to pass the questions to a Survey object.
</Note>

We can alternatively specify that the paramater values be inserted in the question name to create the unique identifiers (so long as they are Pythonic):

```python theme={null}
from edsl import QuestionFreeText, ScenarioList, Scenario

q = QuestionFreeText(
   question_name = "favorite_`{{ item }}`",
   question_text = "What is your favorite `{{ item }}`?",
)

scenarios = ScenarioList(
   Scenario({"item": item}) for item in ["color", "food"]
)

questions = q.loop(scenarios)
```

Output:

```python theme={null}
[Question('free_text', question_name = """favorite_color""", question_text = """What is your favorite color?"""),
Question('free_text', question_name = """favorite_food""", question_text = """What is your favorite food?""")]
```

To run the questions, we pass them to a Survey in the usual manner:

```python theme={null}
from edsl import Survey

survey = Survey(questions = questions)

results = survey.run()
```

### Piping

Piping is a method for passing the answer or other components of previous questions into a subsequent question. Question components can be piped into question texts or options by using double braces with the name of the prior question and the key of the answer in the braces (e.g., `{{ <prior_question_name>.answer }}`). Note that piping can only be used when questions are run together in a survey.

For example:

```python expandable theme={null}
from edsl import QuestionNumerical, QuestionList, QuestionMultipleChoice, Survey

q_age = QuestionNumerical(
   question_name = "age",
   question_text = "What is your age?"
)

# Piping an answer in a question text
q_prime = QuestionMultipleChoice(
   question_name = "prime",
   question_text = "Is `{{ age.answer }}` a prime number?",
   question_options = ["Yes", "No", "I don't know"]
)

q_favorite_colors = QuestionList(
   question_name = "favorite_colors",
   question_text = "What are your 3 favorite colors?",
   max_list_items = 3
)

# Using an item from a list response
q_flowers = QuestionList(
   question_name = "flowers",
   question_text = "Name some flowers that are `{{ favorite_colors.answer[0] }}`."
)

# Using a list response as a complete set of question options
q_house_color = QuestionMultipleChoice(
   question_name = "house_color",
   question_text = "Pretend you are painting a house. Which color would you choose?",
   question_options = "`{{ favorite_colors.answer }}`"
)

# Itemizing options from a list response in question options
q_car_color = QuestionMultipleChoice(
   question_name = "car_color",
   question_text = "Pretend you are buying a car. Which color would you choose?",
   question_options = [
      "`{{ favorite_colors.answer[0] }}`",
      "`{{ favorite_colors.answer[1] }}`",
      "`{{ favorite_colors.answer[2] }}`",
      "Other"
   ]
)

survey = Survey([
   q_age,
   q_prime,
   q_favorite_colors,
   q_flowers,
   q_car_color
])
```

When the survey is run, the answers will be piped as noted in the comments.

For more details and examples of piping, please see the [Surveys](/en/latest/surveys#surveys) module section on [piping](/en/latest/surveys#id2).

### F-strings

F-strings can be used to pass values to a question when it is constructed. They function independently of scenarios and piping, but can be used at the same time.

For example:

```python theme={null}
from edsl import QuestionFreeText, ScenarioList, Scenario, Survey

questions = []
sentiments = ["enjoy", "hate", "love"]

scenarios = ScenarioList(
   Scenario({"activity": activity}) for activity in ["running", "reading"]
)

for sentiment in sentiments:
   q = QuestionFreeText(
      question_name = f"{ sentiment }_activity",
      question_text = f"How much do you { sentiment } `{{ activity }}`?"
   )
   q_list = q.loop(scenarios)

   questions = questions + q_list
```

questions

Output:

```python theme={null}
[Question('free_text', question_name = """enjoy_activity_0""", question_text = """How much do you enjoy running?"""),
Question('free_text', question_name = """enjoy_activity_1""", question_text = """How much do you enjoy reading?"""),
Question('free_text', question_name = """hate_activity_0""", question_text = """How much do you hate running?"""),
Question('free_text', question_name = """hate_activity_1""", question_text = """How much do you hate reading?"""),
Question('free_text', question_name = """love_activity_0""", question_text = """How much do you love running?"""),
Question('free_text', question_name = """love_activity_1""", question_text = """How much do you love reading?""")]
```

We can see that the question names and texts have been parameterized with the values of sentiments and scenarios, and the question names have been automatically incremented to ensure uniqueness. We can then pass the questions to a survey and run it:

```python theme={null}
survey = Survey(questions = questions)

results = survey.run()
```

## Designing AI agents

A key feature of EDSL is the ability to design AI agents with personas and other traits for language models to use in responding to questions. The use of agents allows us to simulate survey results for target audiences at scale. This is done by creating Agent objects with dictionaries of desired traits and adding them to questions when they are run. For example, if we want a question answered by an AI agent representing a student we can create an Agent object with a relevant persona and attributes:

```python theme={null}
from edsl import Agent

agent = Agent(traits = {
   "persona": "You are a student...", # can be an extended text
   "age": 20, # individual trait values can be useful for analysis
   "current_grade": "college sophomore"
   })
```

To generate a response for the agent, we pass it to the by method when we run the question:

```python theme={null}
results = q.by(agent).run()
```

We can also generate responses for multiple agents at once by passing them as a list:

```python theme={null}
from edsl import AgentList, Agent

agents = AgentList(
   Agent(traits = {"persona":p}) for p in ["Dog catcher", "Magician", "Spy"]
)

results = q.by(scenarios).by(agents).run()
```

The Results will contain a Result for each agent that answered the question. To learn more about designing agents, please see the \[Agents]\(/en/latest/agents

## Specifying language models

In the above examples we did not specify a language model for the question or survey, so the default model was used (run Model() to check the current default model). Similar to the way that we optionally passed scenarios to a question and added AI agents, we can also use the by method to specify one or more LLMs to use in generating results. This is done by creating Model objects for desired models and optionally specifying model parameters, such as temperature.

To check available models:

```python theme={null}
from edsl import Model

Model.available()
```

This will return a list of names of models that we can choose from.

We can also check the models for which we have already added API keys:

```python theme={null}
Model.check_models()
```

See instructions on storing [Managing Keys](/en/latest/api_keys) for the models that you want to use, or activating [Remote Inference](/en/latest/remote_inference) to use the Expected Parrot server to access available models.

To specify models for a survey we first create Model objects:

```python theme={null}
from edsl import ModelList, Model

models = ModelList(
   Model(m) for m in ['gpt-4o', 'gemini-1.5-pro']
)
```

Then we add them to a question or survey with the by method when running it:

```python theme={null}
results = q.by(models).run()
```

If scenarios and/or agents are also specified, each component is added in its own by call, chained together in any order, with the run method appended last:

```python theme={null}
results = q.by(scenarios).by(agents).by(models).run()
```

<Note>
  **Note**:

  Note that multiple scenarios, agents and models are always passed as lists in the same by call.
</Note>

Learn more about specifying question scenarios, agents and language models and their parameters in the respective sections:

* [Scenarios](/en/latest/scenarios)
* [Agents](/en/latest/agents)
* [Language Models](/en/latest/language_models)

## QuestionBase class

> ### *class* edsl.questions.QuestionBase [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: `PersistenceMixin`, `RepresentationMixin`, `SimpleAskMixin`, `QuestionBasePromptsMixin`, `QuestionBaseGenMixin`, `ABC`, `AnswerValidatorMixin`

Abstract base class for all question types in EDSL.

QuestionBase defines the core interface and behavior that all question types must implement. It provides the foundation for asking questions to agents, validating responses, generating prompts, and integrating with the rest of the EDSL framework.

The class inherits from multiple mixins to provide different capabilities: - PersistenceMixin: Serialization and deserialization - RepresentationMixin: String representation - SimpleAskMixin: Basic asking functionality - QuestionBasePromptsMixin: Template-based prompt generation - QuestionBaseGenMixin: Generate responses with language models - AnswerValidatorMixin: Response validation

It also uses the RegisterQuestionsMeta metaclass to enforce constraints on child classes and automatically register them for serialization and runtime use.

> **Class attributes:**

question\_name (str): Name of the question, used as an identifier question\_text (str): The actual text of the question to be asked

> **Required attributes in derived classes:**

question\_type (str): String identifier for the question type \_response\_model (Type): Pydantic model class for validating responses response\_validator\_class (Type): Validator class for responses

> **Key Methods:**

by(model): Connect this question to a language model for answering run(): Execute the question with the connected language model duplicate(): Create an exact copy of this question is\_valid\_question\_name(): Verify the question\_name is valid

> **Lifecycle:**

1. Instantiation: A question is created with specific parameters
2. Connection: The question is connected to a language model via by()
3. Execution: The question is run to generate a response
4. Validation: The response is validated based on the question type
5. Result: The validated response is returned for analysis

> **Template System:**

Questions use Jinja2 templates for generating prompts. Each question type has associated template files: - answering\_instructions.jinja: Instructions for how the model should answer - question\_presentation.jinja: Format for how the question is presented Templates support variable substitution using scenario variables.

> **Response Validation:**

Each question type has a dedicated response validator that: - Enforces the expected response structure - Ensures the response is valid for the question type - Attempts to fix invalid responses when possible - Uses Pydantic models for schema validation

> **Example:**

Derived classes must define the required attributes:

```python theme={null}
python class FreeTextQuestion(QuestionBase):

question_type = “free_text” _response_model = FreeTextResponse response_validator_class = FreeTextResponseValidator

def __init__(self, question_name, question_text, **kwargs):
self.question_name = question_name self.question_text = question_text # Additional initialization as needed
```

Using a question:

```python theme={null}
python # Create a question question = FreeTextQuestion(

question_name=”opinion”, question_text=”What do you think about AI?”

)

# Connect to a language model and run from edsl.language_models import Model model = Model() result = question.by(model).run()

# Access the answer answer = result.select(“answer.opinion”).to_list()[0] print(f”The model’s opinion: {answer}”)
```

> **Notes:**

* QuestionBase is abstract and cannot be instantiated directly
* Child classes must implement required methods and attributes
* The RegisterQuestionsMeta metaclass handles registration of question types
* Questions can be serialized to and from dictionaries for storage
* Questions can be used independently or as part of surveys

> #### *class* ValidatedAnswer [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: `TypedDict`

Type definition for a validated answer to a question.

This TypedDict defines the structure of a validated answer, which includes the actual answer value, an optional comment, and optional generated tokens information for tracking LLM token usage.

**Attributes:**

answer: The validated answer value, type depends on question type comment: Optional string comment or explanation for the answer generated\_tokens: Optional string containing raw LLM output for token tracking

> **answer*: Any* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)\*\*

> **comment*: str | None* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)\*\*

> **generated\_tokens*: str | None* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)\*\*

> #### add\_question(*other: [QuestionBase](/en/latest/#edsl.questions.QuestionBase "edsl.questions.QuestionBase")*) → [Survey](/en/latest/surveys#survey-class "edsl.surveys.Survey") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Add a question to this question by turning them into a survey with two questions.

```python theme={null}
>>> from edsl.questions import QuestionFreeText as Q
>>> from edsl.questions import QuestionMultipleChoice as QMC
>>> s = Q.example().add_question(QMC.example())
>>> len(s.questions)
2
```

> #### by(\**args*) → [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Turn a single question into a survey and then a Job.

> #### *property* data\*: dict\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Return a dictionary of question attributes **except** for question\_type.

```python theme={null}
>>> from edsl.questions import QuestionFreeText as Q
>>> Q.example().data
{'question_name': 'how_are_you', 'question_text': 'How are you?'}
```

> #### duplicate() → [QuestionBase](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Create an exact copy of this question instance.

This method creates a new instance of the question with identical attributes by serializing the current instance to a dictionary and then deserializing it back into a new instance.

> **Returns:**

QuestionBase: A new instance of the same question type with identical attributes.

> **Examples:**

```python theme={null}
>>> from edsl.questions import QuestionFreeText
>>> original = QuestionFreeText(question_name="q1", question_text="Hello?")
>>> copy = original.duplicate()
>>> original.question_name == copy.question_name
True
>>> original is copy
False
```

> #### *classmethod* example\_model() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *classmethod* example\_results() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *property* fake\_data\_factory [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Create and return a factory for generating fake response data.

This property lazily creates a factory class based on Pydantic’s ModelFactory that can generate fake data conforming to the question’s response model. The factory is cached after first creation for efficiency.

> **Returns:**

ModelFactory: A factory class that can generate fake data for this question type.

> **Notes:**

* Uses polyfactory to generate valid fake data instances
* The response model for the question defines the structure of the generated data
* Primarily used for testing and simulation purposes

> #### *classmethod* from\_dict(*data: dict*) → [QuestionBase](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Create a question instance from a dictionary representation.

This class method deserializes a question from a dictionary representation, typically created by the to\_dict method. It looks up the appropriate question class based on the question\_type field and constructs an instance of that class.

> **Args:**

data: Dictionary representation of a question, must contain a ‘question\_type’ field.

> **Returns:**

QuestionBase: An instance of the appropriate question subclass.

> **Raises:**

QuestionSerializationError: If the data is missing the question\_type field or

if no question class is registered for the given type.

> **Examples:**

```python theme={null}
>>> from edsl.questions import QuestionFreeText
>>> original = QuestionFreeText.example()
>>> serialized = original.to_dict()
>>> deserialized = QuestionBase.from_dict(serialized)
>>> original.question_text == deserialized.question_text
True
>>> isinstance(deserialized, QuestionFreeText)
True
```

> **Notes:**

* The @remove\_edsl\_version decorator removes EDSL version information from the dictionary before processing
* Special handling is implemented for certain question types like linear\_scale
* Model instructions, if present, are handled separately to ensure proper initialization

> #### gold\_standard(*q\_and\_a\_dict: dict\[str, str]*) → [Result](/en/latest/results#edsl.results.Result "edsl.results.Result") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Run the question with a gold standard agent and return the result.

> #### html(*scenario: dict | None = None*, *agent: dict | None =&#x20;*, *answers: dict | None = None*, *include\_question\_name: bool = False*, *height: int | None = None*, *width: int | None = None*, *iframe=False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### human\_readable() → str [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Print the question in a human readable format.

> > > from edsl.questions import QuestionFreeText
> > > QuestionFreeText.example().human\_readable()
> > > 'Question Type: free\_textnQuestion: How are you?'

> #### humanize(*project\_name: str = 'Project'*, *survey\_description: str | None = None*, *survey\_alias: str | None = None*, *survey\_visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*) → dict [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Turn a single question into a survey and send the survey to Expected Parrot.

Then, create a project on Expected Parrot so you can share the survey with human respondents.

> #### inspect() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Create an interactive inspector widget for this question.

This method uses the InspectorWidget registry system to find the appropriate inspector widget class for questions and returns an instance of it.

> **Returns:**

QuestionInspectorWidget instance: Interactive widget for inspecting this question

> **Raises:**

KeyError: If no question inspector widget is available ImportError: If the widgets module cannot be imported

> #### is\_valid\_question\_name() → bool [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Check if the question name is a valid Python identifier.

This method validates that the question\_name attribute is a valid Python variable name according to Python’s syntax rules. This is important because question names are often used as identifiers in various parts of the system.

> **Returns:**

bool: True if the question name is a valid Python identifier, False otherwise.

> **Examples:**

```python theme={null}
>>> from edsl.questions import QuestionFreeText
>>> q = QuestionFreeText(question_name="valid_name", question_text="Text")
>>> q.is_valid_question_name()
True
```

> #### *property* name\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Get the question name.

This property is a simple alias for question\_name that provides a consistent interface shared with other EDSL components like Instructions.

**Returns:**

str: The question name.

> #### question\_name\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Validate that the question\_name attribute is a valid variable name.

> #### question\_text\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Validate that the question\_text attribute is a string.

```python theme={null}
>>> class TestQuestion:
...     question_text = QuestionTextDescriptor()
...     def __init__(self, question_text: str):
...         self.question_text = question_text
```

```python theme={null}
>>> _ = TestQuestion("What is the capital of France?")
>>> _ = TestQuestion("What is the capital of France? `{{ variable }}`")
```

> #### *property* response\_validator\*: ResponseValidatorABC\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Get the appropriate validator for this question type.

This property lazily creates and returns a response validator instance specific to this question type. The validator is created using the ResponseValidatorFactory, which selects the appropriate validator class based on the question’s type.

**Returns:**

ResponseValidatorABC: An instance of the appropriate validator for this question.

**Notes:**

* Each question type has its own validator class defined in the class attribute response\_validator\_class
* The validator is responsible for ensuring responses conform to the expected format and constraints for this question type

> #### rich\_print() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Print the question in a rich format.

> #### run(\**args*, \*\**kwargs*) → [Results](/en/latest/results#result-class "edsl.results.Results") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Turn a single question into a survey and runs it.

> #### *async* run\_async(*just\_answer: bool = True*, *model: 'LanguageModel' | None = None*, *agent: 'Agent' | None = None*, *disable\_remote\_inference: bool = False*, \*\**kwargs*) → Any | 'Results' [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Call the question asynchronously.

```python theme={null}
>>> import asyncio
>>> from edsl.questions import QuestionFreeText as Q
>>> m = Q._get_test_model(canned_response = "Blue")
>>> q = Q(question_name = "color", question_text = "What is your favorite color?")
>>> async def test_run_async(): result = await q.run_async(model=m, disable_remote_inference = True, disable_remote_cache = True); print(result)
>>> asyncio.run(test_run_async())
Blue
```

> #### *classmethod* run\_example(*show\_answer: bool = True*, *model: 'LanguageModel' | None = None*, *cache: bool = False*, *disable\_remote\_cache: bool = False*, *disable\_remote\_inference: bool = False*, \*\**kwargs*) → [Results](/en/latest/results#result-class "edsl.results.Results") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Run the example question with a language model and return results.

This class method creates an example instance of the question, asks it using the provided language model, and returns the results. It’s primarily used for demonstrations, documentation, and testing.

> **Args:**

show\_answer: If True, returns only the answer portion of the results.

If False, returns the full results.

model: Language model to use for answering. If None, creates a default model. cache: Whether to use local caching for the model call. disable\_remote\_cache: Whether to disable remote caching. disable\_remote\_inference: Whether to disable remote inference. [\*\*](/en/latest/#id21)kwargs: Additional keyword arguments to pass to the example method.

> **Returns:**

Results: Either the full results or just the answer portion, depending on show\_answer.

> **Examples:**

```python theme={null}
>>> from edsl.language_models import LanguageModel
>>> from edsl import QuestionFreeText as Q
>>> m = Q._get_test_model(canned_response="Yo, what's up?")
>>> results = Q.run_example(show_answer=True, model=m,
...                       disable_remote_cache=True, disable_remote_inference=True)
>>> "answer" in str(results)
True
```

> **Notes:**

* This method is useful for quick demonstrations of question behavior
* The disable\_remote\_\* parameters are useful for offline testing
* Additional parameters to customize the example can be passed via kwargs

> #### to\_dict(*add\_edsl\_version: bool = True*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Convert the question to a dictionary that includes the question type (used in deserialization).

```python theme={null}
>>> from edsl.questions import QuestionFreeText as Q; Q.example().to_dict(add_edsl_version = False)
{'question_name': 'how_are_you', 'question_text': 'How are you?', 'question_type': 'free_text'}
```

> #### to\_jobs() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### to\_survey() → [Survey](/en/latest/surveys#survey-class "edsl.surveys.Survey") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Turn a single question into a survey. >>> from edsl import QuestionFreeText as Q >>> Q.example().to\_survey().questions\[0].question\_name ‘how\_are\_you’

> #### using(\**args*, \*\**kwargs*) → [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Turn a single question into a survey and then a Job.

## Question type classes

### QuestionFreeText class

A subclass of the Question class for creating free response questions. There are no specially required fields (only question\_name and question\_text). The response is a single string of text. Example usage:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
   question_name = "food",
   question_text = "What is your favorite food?"
)
```

An example can also be created using the example method:

```python theme={null}
QuestionFreeText.example()
```

#### HTML/XML Tags in Question Text

You can use HTML or XML tags within question text to provide additional context or instructions to the language model. These tags will be preserved and passed to the model as part of the prompt. This can be useful for structuring complex prompts or providing inline instructions.

Example with XML tags for language specification:

```python theme={null}
q = QuestionFreeText(
   question_name = "favorite_color",
   question_text = """Please return your favorite color
                  <language>
                  German
                  </language>
                  """,
)
```

When this question is run, the model will observe the XML tags and may respond in German:

```python theme={null}
results = q.run()
results.select('answer.*')
# Example output:
# Dataset([{'answer.favorite_color': ['Als KI-Modell habe ich keine persönlichen Vorlieben oder Lieblingsfarben. Aber ich kann Ihnen sagen, dass in der deutschen Kultur Farben wie Blau, Rot und Grün oft beliebt sind. Wenn Sie weitere Informationen über Farben auf Deutsch benötigen, lassen Sie es mich wissen!']}])
```

You can also use HTML formatting tags:

```python theme={null}
q = QuestionFreeText(
   question_name = "formatted_question",
   question_text = """This text includes <b>bold</b> and <i>italic</i> formatting.
                  <br>This appears on a new line."""
)
```

> #### *class* edsl.questions.QuestionFreeText(*question\_name: str*, *question\_text: str*, *answering\_instructions: str | None = None*, *question\_presentation: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A question that allows an agent to respond with free-form text.

QuestionFreeText is one of the simplest and most commonly used question types in EDSL. It prompts an agent or language model to provide a textual response without any specific structure or constraints on the format. The response can be of any length and content, making it suitable for open-ended questions, explanations, storytelling, and other scenarios requiring unrestricted text.

> **Attributes:**

question\_type (str): Identifier for this question type, set to “free\_text”. \_response\_model: Pydantic model for validating responses. response\_validator\_class: Class used to validate and fix responses.

> **Examples:**

```python theme={null}
>>> q = QuestionFreeText(
...     question_name="opinion",
...     question_text="What do you think about AI?"
... )
>>> q.question_type
'free_text'
```

```python theme={null}
>>> from edsl.language_models import Model
>>> model = Model("test", canned_response="I think AI is fascinating.")
>>> result = q.by(model).run(disable_remote_inference=True)
>>> answer = result.select("answer.*").to_list()[0]
>>> "fascinating" in answer
True
```

> #### **init**(*question\_name: str*, *question\_text: str*, *answering\_instructions: str | None = None*, *question\_presentation: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Initialize a new free text question.

> **Args:**

**question\_name: Identifier for the question, used in results and templates.**

Must be a valid Python variable name.

question\_text: The actual text of the question to be asked. answering\_instructions: Optional additional instructions for answering

> the question, overrides default instructions.

**question\_presentation: Optional custom presentation template for the**

question, overrides default presentation.

> **Examples:**

```python theme={null}
>>> q = QuestionFreeText(
...     question_name="feedback",
...     question_text="Please provide your thoughts on this product."
... )
>>> q.question_name
'feedback'
```

```python theme={null}
>>> q = QuestionFreeText(
...     question_name="explanation",
...     question_text="Explain how photosynthesis works.",
...     answering_instructions="Provide a detailed scientific explanation."
... )
```

> #### *classmethod* example(*exception\_to\_throw: Exception | None = None*, *override\_answer: dict | None = None*, \**args*, \*\**kwargs*) → T [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Generate HTML content for rendering the question in web interfaces.

This property generates HTML markup for the question when it needs to be displayed in web interfaces or HTML contexts. For a free text question, this is typically a textarea element.

> **Returns:**

str: HTML markup for rendering the question.

> **Notes:**

* Uses Jinja2 templating to generate the HTML
* Creates a textarea input element with the question\_name as the ID and name
* Can be used for displaying the question in web UIs or HTML exports

> #### question\_name\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### question\_text\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `FreeTextResponseValidator`

### QuestionMultipleChoice class

A subclass of the Question class for creating multiple choice questions where the response is a single option selected from a list of options. It specially requires a question\_options list of strings, integers or floats for the options. Example usage:

```python theme={null}
from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
   question_name = "color",
   question_text = "What is your favorite color?",
   question_options = ["Red", "Blue", "Green", "Yellow"]
)
```

An example can also created using the example method:

```python theme={null}
QuestionMultipleChoice.example()
```

If you want the question options to be randomized, you can pass an optional parameter questions\_to\_randomize (a list of the relevant question names) to the Survey object when it is created. For example:

```python theme={null}
from edsl import QuestionMultipleChoice, Survey

q = QuestionMultipleChoice(
   question_name = "color",
   question_text = "What is your favorite color?",
   question_options = ["Red", "Blue", "Green", "Yellow"]
)

survey = Survey([q], questions_to_randomize=["color"])
```

<Note>
  *Note:* Question options can be strings of any length, but if they are long or complex, it may be useful to add the use\_code parameter to the question. This will add an instruction to the user\_prompt for the model to provide the code number of the question option that it selects as its answer (i.e., 0, 1, 2, etc.) instead of the value of the option. This can be useful when the question options are long or complex, or include formatting that a model may make errors in reproducing to provide an answer, resulting in a validation error that may be avoidable by returning the code number of the option instead. The code is then translated back to the option value in the survey results.
</Note>

For example, in a multiple choice question where the agent is instructed to select a programming language we can add the use\_code parameter and then inspect how the user prompt is modified to include *“Respond only with the code corresponding to one of the options.”*

```python theme={null}
from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
   question_name = "programming_language",
   question_text = "Which programming language do you prefer?",
   question_options = ["Python", "Java", "C++", "JavaScript"],
   use_code = True # optional
)
```

> #### *class* edsl.questions.QuestionMultipleChoice(*question\_name: str*, *question\_text: str*, *question\_options: list\[str] | list\[list] | list\[float] | list\[int]*, *include\_comment: bool = True*, *use\_code: bool = False*, *answering\_instructions: str | None = None*, *question\_presentation: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A question that prompts the agent to select one option from a list of choices.

QuestionMultipleChoice presents a set of predefined choices to the agent and asks them to select exactly one option. This question type is ideal for scenarios where the possible answers are known and limited, such as surveys, preference questions, or classification tasks.

Key Features: - Presents a fixed set of options to choose from - Enforces selection of exactly one option - Can use numeric codes for options (use\_code=True) - Supports custom instructions and presentation - Optional comment field for additional explanation - Can be configured to be permissive (accept answers outside the options)

Technical Details: - Uses Pydantic models for validation with Literal types for strict checking - Supports dynamic options from scenario variables - HTML rendering for web interfaces - Robust validation with repair capabilities

> **Examples:**

Basic usage:

```python theme={null}
python q = QuestionMultipleChoice(

question_name=”preference”, question_text=”Which color do you prefer?”, question_options=[“Red”, “Green”, “Blue”, “Yellow”]
```

With numeric codes:

```python theme={null}
python q = QuestionMultipleChoice(

question_name=”rating”, question_text=”Rate this product from 1 to 5”, question_options=[“Very Poor”, “Poor”, “Average”, “Good”, “Excellent”], use_code=True # The answer will be 0-4 instead of the text
```

Dynamic options from scenario:

```python theme={null}
python q = QuestionMultipleChoice(

question_name=”choice”, question_text=”Select an option”, question_options=[“`{{ option1 }}`”, “`{{ option2 }}`”, “`{{ option3 }}`”]

) scenario = Scenario({“option1”: “Choice A”, “option2”: “Choice B”, “option3”: “Choice C”}) result = q.by(model).with_scenario(scenario).run()
```

> **See also:**

[/en/latest/questions#questionmultiplechoice-class](/en/latest/questions#questionmultiplechoice-class)

> #### **init**(*question\_name: str*, *question\_text: str*, *question\_options: list\[str] | list\[list] | list\[float] | list\[int]*, *include\_comment: bool = True*, *use\_code: bool = False*, *answering\_instructions: str | None = None*, *question\_presentation: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Initialize a new multiple choice question.

#### Parameters

**question\_name:str**

The name of the question, used as an identifier. Must be a valid Python variable name. This name will be used in results, templates, and when referencing the question in surveys.

**question\_text:str**

The actual text of the question to be asked. This is the prompt that will be presented to the language model or agent.

**question\_options:Union\[list\[str], list\[list], list\[float], list\[int]]**

The list of options the agent can select from. These can be: - Strings: \[“Option A”, “Option B”, “Option C”] - Lists: Used for nested or complex options - Numbers: \[1, 2, 3, 4, 5] or \[0.1, 0.2, 0.3] - Template strings: \[“\{`\}”, “\{{var2}`}”] which will be rendered with scenario variables

**include\_comment:bool, default=True**
Whether to include a comment field in the response, allowing the model to provide additional explanation beyond just selecting an option.

**use\_code:bool, default=False**

If True, the answer will be the index of the selected option (0-based) instead of the option text itself. This is useful for numeric scoring or when option text is long.

**answering\_instructions:Optional\[str], default=None**

Custom instructions for how the model should answer the question. If None, default instructions for multiple choice questions will be used.

**question\_presentation:Optional\[str], default=None**

Custom template for how the question is presented to the model. If None, the default presentation for multiple choice questions will be used.

**permissive:bool, default=False**

If True, the validator will accept answers that are not in the provided options list. If False (default), only exact matches to the provided options are allowed.

#### Examples

```python theme={null}
>>> q = QuestionMultipleChoice(
...     question_name="color_preference",
...     question_text="What is your favorite color?",
...     question_options=["Red", "Blue", "Green", "Yellow"],
...     include_comment=True
... )
```

```python theme={null}
>>> q_numeric = QuestionMultipleChoice(
...     question_name="rating",
...     question_text="How would you rate this product?",
...     question_options=["Very Poor", "Poor", "Average", "Good", "Excellent"],
...     use_code=True,
...     include_comment=True
... )
```

#### Notes

* When use\_code=True, the answer will be the index (0-based) of the selected option
* The permissive parameter is useful when you want to allow free-form responses while still suggesting options
* Dynamic options can reference variables in a scenario using Jinja2 template syntax

> #### create\_response\_model(*replacement\_dict: dict = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *classmethod* example(*exception\_to\_throw: Exception | None = None*, *override\_answer: dict | None = None*, \**args*, \*\**kwargs*) → T [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Return the HTML version of the question.

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `MultipleChoiceResponseValidator`

### QuestionCheckBox class

A subclass of the Question class for creating questions where the response is a list of one or more of the given options. It specially requires a question\_options list of strings, integers or floats for the options. The minimum number of options that *must* be selected and the maximum number that *may* be selected can be specified when creating the question (parameters min\_selections and max\_selections). If not specified, the minimum number of options that must be selected is 1 and the maximum allowed is the number of question options provided. Example usage:

```python theme={null}
from edsl import QuestionCheckBox

q = QuestionCheckBox(
   question_name = "favorite_days",
   question_text = "What are your 2 favorite days of the week?",
   question_options = ["Monday", "Tuesday", "Wednesday",
   "Thursday", "Friday", "Saturday", "Sunday"],
   min_selections = 2, # optional
   max_selections = 2  # optional
)
```

An example can also be created using the example method:

```python theme={null}
QuestionCheckBox.example()
```

> #### *class* edsl.questions.QuestionCheckBox(*question\_name: str*, *question\_text: str*, *question\_options: list\[str]*, *min\_selections: int | None = None*, *max\_selections: int | None = None*, *include\_comment: bool = True*, *use\_code: bool = False*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A question that prompts the agent to select multiple options from a list.

QuestionCheckBox allows agents to select one or more items from a predefined list of options. It’s useful for “select all that apply” scenarios, multi-select preferences, or any question where multiple valid selections are possible.

> **Attributes:**

question\_type (str): Identifier for this question type, set to “checkbox”. purpose (str): Brief description of when to use this question type. question\_options: List of available options to select from. min\_selections: Optional minimum number of selections required. max\_selections: Optional maximum number of selections allowed. \_response\_model: Initially None, set by create\_response\_model(). response\_validator\_class: Class used to validate and fix responses.

> **Examples:**

```python theme={null}
>>> # Basic creation works
>>> q = QuestionCheckBox.example()
>>> q.question_type
'checkbox'
```

```python theme={null}
>>> # Create preferences question with selection constraints
>>> q = QuestionCheckBox(
...     question_name="favorite_fruits",
...     question_text="Which fruits do you like?",
...     question_options=["Apple", "Banana", "Cherry", "Durian", "Elderberry"],
...     min_selections=1,
...     max_selections=3
... )
>>> q.question_options
['Apple', 'Banana', 'Cherry', 'Durian', 'Elderberry']
>>> q.min_selections
1
>>> q.max_selections
3
```

> #### **init**(*question\_name: str*, *question\_text: str*, *question\_options: list\[str]*, *min\_selections: int | None = None*, *max\_selections: int | None = None*, *include\_comment: bool = True*, *use\_code: bool = False*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Initialize a new checkbox question.

> **Args:**

**question\_name: Identifier for the question, used in results and templates.**

Must be a valid Python variable name.

question\_text: The actual text of the question to be asked. question\_options: List of options the agent can select from. min\_selections: Optional minimum number of options that must be selected. max\_selections: Optional maximum number of options that can be selected. include\_comment: Whether to allow comments with the answer. use\_code: If True, use indices (0,1,2…) instead of option text values. question\_presentation: Optional custom presentation template. answering\_instructions: Optional additional instructions. permissive: If True, ignore selection count constraints during validation.

> **Examples:**

```python theme={null}
>>> q = QuestionCheckBox(
...     question_name="symptoms",
...     question_text="Select all symptoms you are experiencing:",
...     question_options=["Fever", "Cough", "Headache", "Fatigue"],
...     min_selections=1
... )
>>> q.question_name
'symptoms'
```

```python theme={null}
>>> # Question with both min and max
>>> q = QuestionCheckBox(
...     question_name="pizza_toppings",
...     question_text="Choose 2-4 toppings for your pizza:",
...     question_options=["Cheese", "Pepperoni", "Mushroom", "Onion",
...                       "Sausage", "Bacon", "Pineapple"],
...     min_selections=2,
...     max_selections=4
... )
```

> #### create\_response\_model() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Create a response model with the appropriate constraints.

This method creates a Pydantic model customized with the options and selection count constraints specified for this question instance.

> **Returns:**

A Pydantic model class tailored to this question’s constraints.

> **Examples:**

```python theme={null}
>>> q = QuestionCheckBox.example(use_code=True)
>>> model = q.create_response_model()
>>> model(answer=[0, 2])  # Select first and third options
ConstrainedCheckboxResponse(answer=[0, 2], comment=None, generated_tokens=None)
```

> #### *classmethod* example(*exception\_to\_throw: Exception | None = None*, *override\_answer: dict | None = None*, \**args*, \*\**kwargs*) → T [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Generate HTML content for rendering the question in web interfaces.

This property generates HTML markup for the question when it needs to be displayed in web interfaces or HTML contexts. For a checkbox question, this is a set of checkbox input elements, one for each option.

**Returns:**

str: HTML markup for rendering the question.

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `CheckBoxResponseValidator`

### QuestionMatrix class

A subclass of the Question class for creating matrix questions where the response is a table of answers. It specially requires a question\_items list of strings for the row items, a question\_options list of strings, integers or floats for the column items and an optional option\_labels dictionary to specify labels for the column options.

Example usage:

```python theme={null}
from edsl import QuestionMatrix

q = QuestionMatrix(
   question_name = "matrix",
   question_text = "Rate the following items on a scale of 1 to 5.",
   question_items = ["Item 1", "Item 2", "Item 3"],
   question_options = [1, 2, 3, 4, 5],
   option_labels = {1: "Terrible", 5: "Excellent"} # optional
)
```

An example can also be created using the example method:

```python theme={null}
QuestionMatrix.example()
```

> #### *class* edsl.questions.QuestionMatrix(*question\_name: str*, *question\_text: str*, *question\_items: List\[str]*, *question\_options: List\[int | str | float]*, *option\_labels: Dict\[int | str | float, str] | None = None*, *include\_comment: bool = True*, *answering\_instructions: str | None = None*, *question\_presentation: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A question that presents a matrix/grid where multiple items are rated or selected from the same set of options.

This question type allows respondents to provide an answer for each row in a grid, selecting from the same set of options for each row. It’s often used for Likert scales, ratings grids, or any scenario where multiple items need to be rated using the same scale.

> **Examples:**

```python theme={null}
>>> # Create a happiness rating matrix
>>> question = QuestionMatrix(
...     question_name="happiness_matrix",
...     question_text="Rate your happiness with each aspect:",
...     question_items=["Work", "Family", "Social life"],
...     question_options=[1, 2, 3, 4, 5],
...     option_labels={1: "Very unhappy", 3: "Neutral", 5: "Very happy"}
... )
>>> # The response is a dict matching each item to a rating
>>> response = {"answer": {"Work": 4, "Family": 5, "Social life": 3}}
```

> #### **init**(*question\_name: str*, *question\_text: str*, *question\_items: List\[str]*, *question\_options: List\[int | str | float]*, *option\_labels: Dict\[int | str | float, str] | None = None*, *include\_comment: bool = True*, *answering\_instructions: str | None = None*, *question\_presentation: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Initialize a matrix question.

**Args:**

question\_name: The name of the question question\_text: The text of the question question\_items: List of items to be rated or answered (rows) question\_options: Possible answer options for each item (columns) option\_labels: Optional mapping of options to labels (e.g. `{1: “Sad”, 5: “Happy”}`) include\_comment: Whether to include a comment field answering\_instructions: Custom instructions template question\_presentation: Custom presentation template permissive: Whether to allow any values & extra items instead of strictly checking

> #### create\_response\_model() → Type\[BaseModel] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Returns the pydantic model for validating responses to this question.

The model is dynamically created based on the question’s configuration, including allowed items, options, and permissiveness.

> #### *classmethod* example(*exception\_to\_throw: Exception | None = None*, *override\_answer: dict | None = None*, \**args*, \*\**kwargs*) → T [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Generate an HTML representation of the matrix question.

**Returns:**

HTML content string for rendering the question

> #### question\_text\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Validate that the question\_text attribute is a string.

```python theme={null}
>>> class TestQuestion:
...     question_text = QuestionTextDescriptor()
...     def __init__(self, question_text: str):
...         self.question_text = question_text
```

```python theme={null}
>>> _ = TestQuestion("What is the capital of France?")
>>> _ = TestQuestion("What is the capital of France? `{{ variable }}`")
```

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `MatrixResponseValidator`

### QuestionDict

A subclass of the Question class for creating questions where the response is a dictionary of answers. It specially requires an answer\_keys list of strings for the keys of the dictionary. The value\_types and value\_descriptions parameters can be used to specify the types and descriptions of the values that should be entered for each key. Example usage:

```python theme={null}
from edsl import QuestionDict

q = QuestionDict(
   question_name = "recipe",
   question_text = "Please draft an easy recipe for hot chocolate.",
   answer_keys = ["title", "ingedients", "num_ingredients", "instructions"],
   value_types = [str, list, int, list], # optional
   value_descriptions = ["Title of the recipe", "List of ingredients", "Number of ingredients", "Instructions"] # optional
)
```

An example can also be created using the example method:

```python theme={null}
QuestionDict.example()
```

> #### *class* edsl.questions.QuestionDict(*question\_name: str*, *question\_text: str*, *answer\_keys: List\[str]*, *value\_types: List\[str | type] | None = None*, *value\_descriptions: List\[str] | None = None*, *include\_comment: bool = True*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A QuestionDict allows you to create questions that expect dictionary responses with specific keys and value types. It dynamically builds a pydantic model so that Pydantic automatically raises ValidationError for missing/invalid fields.

Documentation: [/en/latest/questions#questiondict](/en/latest/questions#questiondict)

#### Parameters

**question\_name:str**

Unique identifier for the question

**question\_text:str**

The actual question text presented to users

**answer\_keys:List\[str]**

Keys that must be provided in the answer dictionary

**value\_types:Optional\[List\[Union\[str, type]]]**

Expected data types for each answer key

**value\_descriptions:Optional\[List\[str]]**

Human-readable descriptions for each answer key

**include\_comment:bool**

Whether to allow additional comments with the answer

**question\_presentation:Optional\[str]**

Alternative way to present the question

**answering\_instructions:Optional\[str]**

Additional instructions for answering

**permissive:bool**

If True, allows additional keys not specified in answer\_keys

> #### **init**(*question\_name: str*, *question\_text: str*, *answer\_keys: List\[str]*, *value\_types: List\[str | type] | None = None*, *value\_descriptions: List\[str] | None = None*, *include\_comment: bool = True*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### create\_response\_model() → Type\[BaseModel] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Build and return the Pydantic model that should parse/validate user answers. This is similar to QuestionNumerical.create\_response\_model, but for dicts.

> #### *classmethod* example(*exception\_to\_throw: Exception | None = None*, *override\_answer: dict | None = None*, \**args*, \*\**kwargs*) → T [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *classmethod* from\_dict(*data: dict*) → [QuestionDict](/en/latest/#edsl.questions.QuestionDict "edsl.questions.question_dict.QuestionDict") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Recreate from a dictionary.

> #### question\_text\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Validate that the question\_text attribute is a string.

```python theme={null}
>>> class TestQuestion:
...     question_text = QuestionTextDescriptor()
...     def __init__(self, question_text: str):
...         self.question_text = question_text
```

```python theme={null}
>>> _ = TestQuestion("What is the capital of France?")
>>> _ = TestQuestion("What is the capital of France? `{{ variable }}`")
```

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `DictResponseValidator`

> #### to\_dict(*add\_edsl\_version: bool = True*) → dict [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Serialize to JSON-compatible dictionary.

### QuestionLinearScale class

A subclass of the QuestionMultipleChoice class for creating linear scale questions. It requires a question\_options list of integers for the scale. The option\_labels parameter can be used to specify labels for the scale options. Example usage:

```python theme={null}
from edsl import QuestionLinearScale

q = QuestionLinearScale(
   question_name = "studying",
   question_text = """On a scale from 0 to 5, how much do you
   enjoy studying? (0 = not at all, 5 = very much)""",
   question_options = [0, 1, 2, 3, 4, 5], # integers
   option_labels = `{0: "Not at all", 5: "Very much"}` # optional
)
```

An example can also be created using the example method:

```python theme={null}
QuestionLinearScale.example()
```

### QuestionNumerical class

A subclass of the Question class for creating questions where the response is a numerical value. The minimum and maximum values of the answer can be specified using the min\_value and max\_value parameters. Example usage:

```python theme={null}
from edsl import QuestionNumerical

q = QuestionNumerical(
   question_name = "work_days",
   question_text = "How many days a week do you normally work?",
   min_value = 1, # optional
   max_value = 7  # optional
)
```

An example can also be created using the example method:

```python theme={null}
QuestionNumerical.example()
```

> ### *class* edsl.questions.QuestionNumerical(*question\_name: str*, *question\_text: str*, *min\_value: int | float | None = None*, *max\_value: int | float | None = None*, *include\_comment: bool = True*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A question that prompts the agent to answer with a numerical value.

QuestionNumerical is designed for responses that must be numbers, with optional range constraints to ensure values fall within acceptable bounds. It’s useful for age questions, ratings, measurements, and any scenario requiring numerical answers.

> **Attributes:**

question\_type (str): Identifier for this question type, set to “numerical”. min\_value: Optional lower bound for acceptable answers. max\_value: Optional upper bound for acceptable answers. \_response\_model: Initially None, set by create\_response\_model(). response\_validator\_class: Class used to validate and fix responses.

> **Examples:**

```python theme={null}
>>> # Basic self-check passes
>>> QuestionNumerical.self_check()
```

```python theme={null}
>>> # Create age question with range constraints
>>> q = QuestionNumerical(
...     question_name="age",
...     question_text="How old are you in years?",
...     min_value=0,
...     max_value=120
... )
>>> q.min_value
0
>>> q.max_value
120
```

> #### **init**(*question\_name: str*, *question\_text: str*, *min\_value: int | float | None = None*, *max\_value: int | float | None = None*, *include\_comment: bool = True*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Initialize a new numerical question.

> **Args:**

**question\_name: Identifier for the question, used in results and templates.**

Must be a valid Python variable name.

question\_text: The actual text of the question to be asked. min\_value: Optional minimum value for the answer (inclusive). max\_value: Optional maximum value for the answer (inclusive). include\_comment: Whether to allow comments with the answer. question\_presentation: Optional custom presentation template. answering\_instructions: Optional additional instructions. permissive: If True, ignore min/max constraints during validation.

> **Examples:**

```python theme={null}
>>> q = QuestionNumerical(
...     question_name="temperature",
...     question_text="What is the temperature in Celsius?",
...     min_value=-273.15  # Absolute zero
... )
>>> q.question_name
'temperature'
```

```python theme={null}
>>> # Question with both min and max
>>> q = QuestionNumerical(
...     question_name="rating",
...     question_text="Rate from 1 to 10",
...     min_value=1,
...     max_value=10
... )
```

> #### create\_response\_model() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Create a response model with the appropriate constraints.

This method creates a Pydantic model customized with the min/max constraints specified for this question instance. If permissive=True, constraints are ignored.

> **Returns:**

A Pydantic model class tailored to this question’s constraints.

> **Examples:**

```python theme={null}
>>> q = QuestionNumerical.example()
>>> model = q.create_response_model()
>>> model(answer=45).answer
45
```

> #### *classmethod* example(*exception\_to\_throw: Exception | None = None*, *override\_answer: dict | None = None*, \**args*, \*\**kwargs*) → T [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Generate HTML content for rendering the question in web interfaces.

This property generates HTML markup for the question when it needs to be displayed in web interfaces or HTML contexts. For a numerical question, this is typically an input element with type=”number”.

**Returns:**

str: HTML markup for rendering the question.

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `NumericalResponseValidator`

### QuestionLikertFive class

A subclass of the QuestionMultipleChoice class for creating questions where the answer is a response to a given statement on a 5-point Likert scale. (The scale does *not* need to be added as a parameter.) Example usage:

```python theme={null}
from edsl import QuestionLikertFive

q = QuestionLikertFive(
   question_name = "happy",
   question_text = "I am only happy when it rains."
)
```

An example can also be created using the example method:

```python theme={null}
QuestionLikertFive.example()
```

### QuestionRank class

A subclass of the Question class for creating questions where the response is a ranked list of options. It specially requires a question\_options list of strings for the options. The number of options that *must* be selected can be optionally specified when creating the question. If not specified, all options are included (ranked) in the response. Example usage:

```python theme={null}
from edsl import QuestionRank

q = QuestionRank(
   question_name = "foods_rank",
   question_text = "Rank the following foods.",
   question_options = ["Pizza", "Pasta", "Salad", "Soup"],
   num_selections = 2 # optional
)
```

An example can also be created using the example method:

```python theme={null}
QuestionRank.example()
```

Alternatively, QuestionTopK can be used to ask the respondent to select a specific number of options from a list. (See the next section for details.)

> ### *class* edsl.questions.QuestionRank(*question\_name: str*, *question\_text: str*, *question\_options: list\[str]*, *num\_selections: int | None = None*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*, *use\_code: bool = True*, *include\_comment: bool = True*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A question that prompts the agent to rank options from a list.

This question type asks respondents to put options in order of preference, importance, or any other ordering criteria. The response is a list of selected options in ranked order.

> **Examples:**

```python theme={null}
>>> # Create a ranking question for food preferences
>>> question = QuestionRank(
...     question_name="food_ranking",
...     question_text="Rank these foods from most to least favorite.",
...     question_options=["Pizza", "Pasta", "Salad", "Soup"],
...     num_selections=2
... )
>>> # The response should be a ranked list
>>> response = {"answer": ["Pizza", "Pasta"], "comment": "I prefer Italian food."}
```

> #### **init**(*question\_name: str*, *question\_text: str*, *question\_options: list\[str]*, *num\_selections: int | None = None*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*, *use\_code: bool = True*, *include\_comment: bool = True*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Initialize a rank question.

**Args:**

question\_name: The name of the question question\_text: The text of the question question\_options: The options the respondent should rank num\_selections: The number of options to select and rank (defaults to all) question\_presentation: Custom presentation template (optional) answering\_instructions: Custom instructions template (optional) permissive: Whether to relax validation constraints use\_code: Whether to use numeric indices (0,1,2) instead of option text include\_comment: Whether to include a comment field

> #### create\_response\_model() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Returns the pydantic model for validating responses to this question.

The model is dynamically created based on the question’s configuration, including allowed choices, number of selections, and permissiveness.

> #### *classmethod* example(*use\_code=False*, *include\_comment=True*) → [QuestionRank](/en/latest/#edsl.questions.QuestionRank "edsl.questions.question_rank.QuestionRank") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Return an example rank question.

**Args:**

use\_code: Whether to use numeric indices include\_comment: Whether to include a comment field

**Returns:**

An example QuestionRank instance

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Generate an HTML representation of the ranking question.

**Returns:**

HTML content string for rendering the question

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `RankResponseValidator`

### QuestionTopK class

A subclass of the QuestionMultipleChoice class for creating questions where the response is a list of ranked items. It specially requires a question\_options list of strings for the options and the number of options that must be selected (num\_selections). Example usage:

```python theme={null}
from edsl import QuestionTopK

q = QuestionTopK(
   question_name = "foods_rank",
   question_text = "Select the best foods.",
   question_options = ["Pizza", "Pasta", "Salad", "Soup"],
   min_selections = 2,
   max_selections = 2
)
```

An example can also be created using the example method:

```python theme={null}
QuestionTopK.example()
```

### QuestionYesNo class

A subclass of the QuestionMultipleChoice class for creating multiple choice questions where the answer options are already specified: \[‘Yes’, ‘No’]. Example usage:

from edsl import QuestionYesNo

```python theme={null}
q = QuestionYesNo(
   question_name = "student",
   question_text = "Are you a student?"
)
```

An example can also be created using the example method:

```python theme={null}
QuestionYesNo.example()
```

### QuestionList class

A subclass of the Question class for creating questions where the response is a list of strings. The minimum and maximum numbers of items to be included in the list can be specified using the optional parameters min\_list\_items and max\_list\_items. Example usage:

```python theme={null}
q = QuestionList(
   question_name = "activities",
   question_text = "What activities do you enjoy most?",
   min_list_items = 2, # optional
   max_list_items = 5  # optional
)
```

An example can also be created using the example method:

```python theme={null}
QuestionList.example()
```

> ### *class* edsl.questions.QuestionList(*question\_name: str*, *question\_text: str*, *include\_comment: bool = True*, *max\_list\_items: int | None = None*, *min\_list\_items: int | None = None*, *answering\_instructions: str | None = None*, *question\_presentation: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

This question prompts the agent to answer by providing a list of items as comma-separated strings.

> #### **init**(*question\_name: str*, *question\_text: str*, *include\_comment: bool = True*, *max\_list\_items: int | None = None*, *min\_list\_items: int | None = None*, *answering\_instructions: str | None = None*, *question\_presentation: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Instantiate a new QuestionList.

**Parameters:**

* **question\_name** – The name of the question.
* **question\_text** – The text of the question.
* **max\_list\_items** – The maximum number of items that can be in the answer list.
* **min\_list\_items** – The minimum number of items that must be in the answer list.

```python theme={null}
>>> QuestionList.example().self_check()
```

> #### create\_response\_model() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *classmethod* example(*exception\_to\_throw: Exception | None = None*, *override\_answer: dict | None = None*, \**args*, \*\**kwargs*) → T [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### min\_list\_items\*: int\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Validate that a value is an integer or None.

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `ListResponseValidator`

### QuestionDistribution class

Use `QuestionDistribution` to elicit probability mass over named outcomes or
intervals of a numerical variable. The answer is one dictionary whose values
sum to 1. For intervals, each value is the probability of falling anywhere in
that bin; it is not a density height.

```python theme={null}
from edsl import QuestionDistribution

categories = QuestionDistribution(
    question_name="winner",
    question_text="What is the probability that each candidate wins?",
    question_options=["a", "b", "c"],
)
# Example answer: {"a": 0.2, "b": 0.2, "c": 0.6}

intervals = QuestionDistribution(
    question_name="wait",
    question_text="How many minutes will the wait be?",
    bins=["[0,10)", "[10,Inf)"],
)
# Example answer: {"[0,10)": 0.2, "[10,Inf)": 0.8}

sized_bins = QuestionDistribution(
    question_name="duration",
    question_text="How long will the task take, between 0 and 25 minutes?",
    min_value=0,
    max_value=25,
    bucket_size=10,
)
# sized_bins.resolved_bins: ("[0,10)", "[10,20)", "[20,25]")
```

Supply exactly one form: `question_options`, `bins`, or
`min_value`/`max_value`/`bucket_size`. Generated bins require finite bounds and
a positive bucket size. The final bin may be shorter and includes the maximum.
Use explicit bins for infinite tails or unequal widths. Optional bounds on
explicit bins assert their outer endpoints; they do not clip the bins.

Construction rejects gaps, overlaps, empty intervals, out-of-order bins, and
double-counted or missing boundary points. Finite outside endpoints must be
included. `[10, Inf]` is accepted and canonicalized to `[10,Inf)`. Category
labels must be unique, nonempty strings without leading/trailing whitespace.
All structural fields are static; question text supports ordinary templates.
At most 1,000 outcomes or bins are allowed. `answer_keys` and `resolved_bins`
are read-only tuples; construct a new question to change the partition.

Answers must include every exact canonical key, including zeros, with finite
numeric values between 0 and 1. Missing/extra keys, duplicate JSON keys,
booleans, percentages, and numeric strings are rejected. The sum must be
within `tolerance` of 1 (default `1e-6`, permitted range 0 through `1e-3`).
Accepted values are preserved without normalization or remainder allocation.

Run the question using the usual `.by(...).run(...)` API, and access its
dictionary through `results.select("answer.duration").to_list()`.
Survey, Jobs, and Results packages preserve the distribution. The type is
available as `Question("distribution", ...)` and through CLI JSON input:

```bash theme={null}
ep schema show --question_type distribution
ep validate --json '{"type":"distribution","question_name":"duration","question_text":"Predict a duration from 0 to 25.","min_value":0,"max_value":25,"bucket_size":10}'
```

Option randomization is unsupported. In Humanize, respondents answer by painting
probability across the outcomes. The per-question humanize settings
`initial_distribution` (`{"type": "uniform"}`, the default, or `{"type": "empty"}`)
and `distribution_summary` (implied mean and variance for finite numeric bins)
control its starting state and optional summary. Remote inference requires workers
with `QuestionDistribution` installed. Bin masses alone do not determine an
interpolated quantile or a density within a bin.

### QuestionBudget class

A subclass of the Question class for creating questions where the response is an allocation of a sum among a list of options in the form of a dictionary where the keys are the options and the values are the allocated amounts. It specially requires a question\_options list of strings for the options and a budget\_sum number for the total sum to be allocated. Example usage:

```python theme={null}
from edsl import QuestionBudget

q = QuestionBudget(
   question_name = "food_budget",
   question_text = "How would you allocate $100?",
   question_options = ["Pizza", "Ice cream", "Burgers", "Salad"],
   budget_sum = 100
)
```

Use `remainder_option` when one category should receive any amount left unallocated by the response:

```python theme={null}
q = QuestionBudget(
    question_name="food_budget",
    question_text="How would you allocate $100?",
    question_options=["Meals", "Snacks", "Other"],
    budget_sum=100,
    remainder_option="Other",
)
```

For example, `{"Meals": 60, "Snacks": 25, "Other": 0}` is repaired to allocate the remaining 15 to `"Other"`. Small floating-point overages are corrected against the same option, while material overallocations remain invalid. The option must exactly match an entry in the rendered `question_options`.

An example can also be created using the example method:

```python theme={null}
QuestionBudget.example()
```

> ### *class* edsl.questions.QuestionBudget(*question\_name: str*, *question\_text: str*, *question\_options: list\[str]*, *budget\_sum: int*, *include\_comment: bool = True*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A question that prompts the agent to allocate a budget among options.

QuestionBudget is designed for scenarios where a fixed amount needs to be distributed across multiple categories or options. It’s useful for allocation questions, spending priorities, resource distribution, and similar scenarios.

> **Attributes:**

question\_type: Identifier for this question type, set to “budget” budget\_sum: The total amount to be allocated question\_options: List of options to allocate the budget among \_response\_model: Initially None, set by create\_response\_model() response\_validator\_class: Class used to validate and fix responses

> **Examples:**

```python theme={null}
>>> # Create budget allocation question
>>> q = QuestionBudget(
...     question_name="spending",
...     question_text="How would you allocate $100?",
...     question_options=["Food", "Housing", "Entertainment", "Savings"],
...     budget_sum=100
... )
>>> q.budget_sum
100
>>> len(q.question_options)
4
```

> #### **init**(*question\_name: str*, *question\_text: str*, *question\_options: list\[str]*, *budget\_sum: int*, *include\_comment: bool = True*, *question\_presentation: str | None = None*, *answering\_instructions: str | None = None*, *permissive: bool = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Initialize a new budget allocation question.

> **Args:**

question\_name: Identifier for the question, used in results and templates question\_text: The actual text of the question to be asked question\_options: The options for allocation of the budget sum budget\_sum: The total amount of the budget to be allocated include\_comment: Whether to allow comments with the answer question\_presentation: Optional custom presentation template answering\_instructions: Optional additional instructions permissive: If True, allow allocations less than budget\_sum

> **Examples:**

```python theme={null}
>>> q = QuestionBudget(
...     question_name="investment",
...     question_text="How would you invest $1000?",
...     question_options=["Stocks", "Bonds", "Real Estate", "Cash"],
...     budget_sum=1000
... )
>>> q.question_name
'investment'
```

> #### create\_response\_model() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Create a response model with the appropriate constraints.

This method creates a Pydantic model customized with the budget constraints and options specified for this question instance.

> **Returns:**

A Pydantic model class tailored to this question’s constraints

> **Examples:**

```python theme={null}

>>> q = QuestionBudget.example()
>>> model = q.create_response_model()
>>> model(answer=[25, 25, 25, 25]).answer
[25.0, 25.0, 25.0, 25.0]
```

> #### *classmethod* example(*include\_comment: bool = True*) → [QuestionBudget](/en/latest/#edsl.questions.QuestionBudget "edsl.questions.question_budget.QuestionBudget") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Create an example instance of a budget question.

This class method creates a predefined example of a budget question for demonstration, testing, and documentation purposes.

> **Args:**

include\_comment: Whether to include a comment field with the answer

> **Returns:**

QuestionBudget: An example budget question

> **Examples:**

```python theme={null}
>>> q = QuestionBudget.example()
>>> q.question_name
'food_budget'
>>> q.question_text
'How would you allocate $100?'
>>> q.budget_sum
100
>>> q.question_options
['Pizza', 'Ice Cream', 'Burgers', 'Salad']
```

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Generate HTML content for rendering the question in web interfaces.

This property generates HTML markup for the question when it needs to be displayed in web interfaces or HTML contexts, including an interactive budget allocation form with JavaScript for real-time budget tracking.

**Returns:**

str: HTML markup for rendering the question

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `BudgetResponseValidator`

### QuestionExtract class

A subclass of the Question class for creating questions where the response is information extracted (or extrapolated) from a given text and formatted according to a specified template. Example usage:

```python theme={null}
from edsl import QuestionExtract

q = QuestionExtract(
   question_name = "course_schedule",
   question_text = "This semester we are offering courses on calligraphy on Friday mornings.",
   answer_template = `{"course_topic": "AI", "days": ["Monday", "Wednesday"]}`
)
```

An example can also be created using the example method:

```python theme={null}
QuestionExtract.example()
```

> ### *class* edsl.questions.QuestionExtract(*question\_text: str*, *answer\_template: dict\[str, Any]*, *question\_name: str*, *answering\_instructions: str = None*, *question\_presentation: str = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A question that extracts structured information from text according to a template.

This question type prompts the agent to extract specific data points from text and return them in a structured format defined by a template. It’s useful for information extraction tasks like parsing contact details, extracting features, or summarizing structured information.

> **Attributes:**

question\_type: Identifier for this question type answer\_template: Dictionary defining the structure to extract response\_validator\_class: The validator class for responses

> **Examples:**

```
>>> # Create a question to extract name and profession
>>> q = QuestionExtract(
...     question_name="person_info",
...     question_text="Extract the person's name and profession from this text: John is a carpenter from Boston.",
...     answer_template={"name": "Example Name", "profession": "Example Profession"}
... )
>>> q.answer_template
{'name': 'Example Name', 'profession': 'Example Profession'}
```

```python theme={null}
>>> # Validate a correct answer
>>> response = {"answer": {"name": "John", "profession": "carpenter"}}
>>> q._validate_answer(response)
{'answer': {'name': 'John', 'profession': 'carpenter'}, 'comment': None, 'generated_tokens': None}
```

> #### **init**(*question\_text: str*, *answer\_template: dict\[str, Any]*, *question\_name: str*, *answering\_instructions: str = None*, *question\_presentation: str = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Initialize the extraction question.

> **Args:**

question\_name: The name/identifier for the question question\_text: The text of the question to present answer\_template: Dictionary template defining the structure to extract answering\_instructions: Optional custom instructions for the agent question\_presentation: Optional custom presentation template

> **Examples:**

```python theme={null}
>>> q = QuestionExtract(
...     question_name="review_extract",
...     question_text="Extract information from this product review",
...     answer_template={"rating": 5, "pros": "example", "cons": "example"}
... )
>>> q.question_name
'review_extract'
```

> #### create\_response\_model() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Create a dynamic Pydantic model based on the answer template.

> **Returns:**

A Pydantic model class configured for the template structure

> **Examples:**

```python theme={null}
>>> q = QuestionExtract.example()
>>> model = q.create_response_model()
>>> isinstance(model, type)
True
```

> #### *classmethod* example(*exception\_to\_throw: Exception | None = None*, *override\_answer: dict | None = None*, \**args*, \*\**kwargs*) → T [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Generate HTML form inputs for the template fields.

**Returns:**

HTML string with form inputs for each template field

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `ExtractResponseValidator`

### QuestionDropdown class

A subclass of the Question class for creating questions where the response is selected from a large set of options (hundreds or thousands) using BM25 search. Instead of presenting all options to the LLM, it shows a sample of options, the LLM generates search keywords, and BM25 ranks all options to return the top matches.

Example usage:

```python theme={null}
from edsl import QuestionDropdown

cities = ["New York", "Paris", "Tokyo", "London", ...]  # 1000+ cities

q = QuestionDropdown(
   question_name = "city_choice",
   question_text = "Which city matches your travel preferences for cultural activities?",
   question_options = cities,
   sample_indices = [0, 10, 20, 30, 40],  # Show specific examples to LLM
   top_k = 5  # Return top 5 matches
)
```

With detailed descriptions for richer search context:

```python theme={null}
restaurants = ["Mario's Pizza", "Sushi Zen", "Burger Palace", ...]
details = [
   "Authentic Italian pizza with wood-fired oven",
   "Traditional Japanese sushi with daily fresh fish",
   "American-style gourmet burgers with grass-fed beef",
   ...
]

q = QuestionDropdown(
   question_name = "restaurant",
   question_text = "Which restaurant would you prefer for a romantic dinner?",
   question_options = restaurants,
   question_options_details = details,
   top_k = 3  # Return top 3 matches as a list
)
```

An example can also be created using the example method:

```python theme={null}
QuestionDropdown.example()
```

### QuestionInterview class

A subclass of the Question class for creating questions that simulate an interview dialogue between an interviewer and respondent. The response is formatted as a newline-separated dialogue transcript.

Example usage:

```python theme={null}
from edsl import QuestionInterview

q = QuestionInterview(
   question_name = "user_experience",
   question_text = "Understanding user satisfaction with mobile apps",
   interview_guide = "Ask about their daily usage, favorite features, pain points, and suggestions for improvement."
)
```

The response will be formatted as:

```
Interviewer: How often do you use mobile apps?
Respondent: I use them daily for various tasks.
Interviewer: What features do you find most useful?
Respondent: I really appreciate the offline mode and quick sync.
```

An example can also be created using the example method:

```python theme={null}
QuestionInterview.example()
```

### QuestionFunctional class

A subclass of the Question class for creating questions where the response is generated by a function instead of a lanugage model. The question type is not intended to be used directly in a survey, but rather to generate responses for other questions. This can be useful where a model is not needed for part of a survey, for questions that require some kind of initial computation, or for questions that are the result of a multi-step process. The question type lets us define a function func that takes in a scenario and (optional) agent traits and returns an answer.

Example usage:

```python theme={null}
from edsl import QuestionNumerical, Agent

q_random = QuestionNumerical(
   question_name = "random",
   question_text = "Choose a random number between 1 and 1000."
)

agents = [Agent({"persona":p}) for p in ["Dog catcher", "Magician", "Spy"]]

results = q_random.by(agents).run()

results.select("persona", "random")
```

Output:

We can use QuestionFunctional to evaluate the responses using a function instead of calling the language model to answer another question. The responses are passed to the function as scenarios, and then the function is passed to the QuestionFunctional object:

```python theme={null}
from edsl import QuestionFunctional

def my_function(scenario, agent_traits):
   if scenario.get("persona") == "Magician":
      return "Magicians never pick randomly!"
   elif scenario.get("random") > 500:
      return "Top half"
   else:
      return "Bottom half"

q_evaluate = QuestionFunctional(
   question_name = "evaluate",
   func = my_function
)

Next we turn the responses into scenarios and inspect them:

scenarios = results.select("persona", "random").to_scenario_list()
scenarios
```

Output:

```python theme={null}
[Scenario({'persona': 'Dog catcher', 'random': 472}),
Scenario({'persona': 'Magician', 'random': 537}),
Scenario({'persona': 'Spy', 'random': 528})]
```

Finally, we run the function with the scenarios and inspect the results:

```python theme={null}
results = q_evaluate.by(scenarios).run()

results.select("persona", "random", "evaluate")
```

Output:

Another example of QuestionFunctional can be seen in the following notebook, where we give agents different instructions for generating random numbers and then use a function to identify whether the responses are identical.

Example notebook: [Simulating randomness](/en/latest/notebooks/random_numbers)

> ### *class* edsl.questions.QuestionFunctional(*question\_name: str*, *func: Callable | None = None*, *question\_text: str | None = 'Functional question'*, *requires\_loop: bool | None = False*, *function\_source\_code: str | None = None*, *function\_name: str | None = None*, *unsafe: bool | None = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Bases: [`QuestionBase`](/en/latest/#edsl.questions.QuestionBase "edsl.questions.question_base.QuestionBase")

A special type of question that is *not* answered by an LLM.

```python theme={null}
>>> from edsl import Scenario, Agent
```

\# Create an instance of QuestionFunctional with the new function >>> question = QuestionFunctional.example()

\# Activate and test the function >>> question.activate() >>> scenario = Scenario(\{“numbers”: \[1, 2, 3, 4, 5]}) >>> agent = Agent(traits=\{“multiplier”: 10}) >>> results = question.by(scenario).by(agent).run(disable\_remote\_cache = True, disable\_remote\_inference = True) >>> results.select(“answer.\*”).to\_list()\[0] == 150 True

\# Serialize the question to a dictionary

```python theme={null}
>>> from .question_base import QuestionBase
>>> new_question = QuestionBase.from_dict(question.to_dict())
>>> results = new_question.by(scenario).by(agent).run(disable_remote_cache = True, disable_remote_inference = True)
>>> results.select("answer.*").to_list()[0] == 150
True
```

> #### **init**(*question\_name: str*, *func: Callable | None = None*, *question\_text: str | None = 'Functional question'*, *requires\_loop: bool | None = False*, *function\_source\_code: str | None = None*, *function\_name: str | None = None*, *unsafe: bool | None = False*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### activate() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### activate\_loop() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Activate the function with loop logic using RestrictedPython.

> #### activated *= True* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### answer\_question\_directly(*scenario*, *agent\_traits=None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Return the answer to the question, ensuring the function is activated.

> #### create\_response\_model() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Returns the Pydantic model for validating responses to this question.

> #### default\_instructions *= ''* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *classmethod* example() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### function\_name *= ''* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### function\_source\_code *= ''* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### *property* question\_html\_content\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### question\_name\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### question\_text\*: str\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

> #### response\_validator\_class [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

alias of `FunctionalResponseValidator`

> #### to\_dict(*add\_edsl\_version=True*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions.py)

Convert the question to a dictionary that includes the question type (used in deserialization).

```python theme={null}
>>> from edsl.questions import QuestionFreeText as Q; Q.example().to_dict(add_edsl_version = False)
{'question_name': 'how_are_you', 'question_text': 'How are you?', 'question_type': 'free_text'}
```

## Optional question parameters

Examples of optional question paramaters:

include\_comment - This boolean parameter can be used to specify that the default comment field which is added to all types other than free\_text should be excluded from a question (default: include\_comment = True). Example usage:

```python theme={null}
from edsl import QuestionNumerical, Survey

q1 = QuestionNumerical(
   question_name = "adding_v1",
   question_text = "What is 1+1?"
)

# The same question with the comment field excluded
q2 = QuestionNumerical(
   question_name = "adding_v2",
   question_text = "What is 1+1?",
   include_comment = False
)

job = Survey([q1, q2]).to_jobs()

job.prompts().select("user_prompt", "question_name")
```

We can see that the second version of the question does not include the comment instruction *“After the answer, put a comment explaining your choice on the next line.”*:

| user\_prompt | question\_name |
| :- | :- |
| What is 1+1? This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, …). Respond with just your number on a single line. If your response is equivalent to zero, report ‘0’ After the answer, put a comment explaining your choice on the next line. | adding\_v1 |
| What is 1+1? This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, …). Respond with just your number on a single line. If your response is equivalent to zero, report ‘0’ | adding\_v2 |

When we run the survey, the comment field will be included in the results for the first question but not the second:

```python theme={null}
results = job.run()

results.select("comment.*")
```

Output:

| comment.adding\_v1\_comment | comment.adding\_v2\_comment |
| :- | :- |
| // The sum of 1 and 1 is 2. | |

See the [Prompts](/en/latest/prompts#prompts) section for more information about various methods for inspecting user and system prompts.

question\_presentation and answering\_instructions - These parameters can be used to add additional context or modify the default instructions of a question.

* The parameter question\_presentation interacts with the question text to specify how the question should be presented to the model (e.g., to modify the default instructions for a question).
* The parameter answering\_instructions is added to the end of the question text without modifying it. It can be used to specify how the model should answer the question and can be useful for questions that require a specific format for the answer.

Example usage:

```python theme={null}
from edsl import QuestionNumerical, Survey

q = QuestionNumerical(
   question_name = "adding",
   question_text = "What is 1+1?",
   question_presentation = "Please solve the following addition problem: `{{ question_text }}`",
   answering_instructions = "nnRespond with just your number on a single line."
)

job = Survey([q]).to_jobs()

job.prompts().select("user_prompt")
```

Output:

| user\_prompt |
| :- |
| Please solve the following addition problem: What is 1+1? Respond with just your number on a single line. |

See the [Prompts](/en/latest/prompts) section for more information about various methods for inspecting user and system prompts.

## Other classes & methods

Settings for the questions module.

> ### *class* edsl.questions.settings.Settings [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions/settings.py)

Bases: `object`

Settings for the questions module.

> #### MAX\_ANSWER\_LENGTH *= 2000* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions/settings.py)

> #### MAX\_EXPRESSION\_CONSTRAINT\_LENGTH *= 1000* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions/settings.py)

> #### MAX\_NUM\_OPTIONS *= 200* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions/settings.py)

> #### MAX\_OPTION\_LENGTH *= 10000* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions/settings.py)

> #### MAX\_QUESTION\_LENGTH *= 100000* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions/settings.py)

> #### MIN\_NUM\_OPTIONS *= 1* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/questions/settings.py)


# Remote Caching
Source: https://docs.expectedparrot.com/en/latest/remote_caching

Remote caching automatically stores responses from language models at the Expected Parrot server, and retrieves stored responses to questions that have already been run. This makes it free and instant to reproduce results from any survey that has been run remotely.

<Note>
  *Note: You must have an Expected Parrot account in order to use remote inference and caching. By using remote inference you agree to any terms of use of service providers, which Expected Parrot may accept on your behalf and enforce in accordance with our terms of use.*
</Note>

## Activate remote caching

Remote caching is automatically activated with remote inference. To activate remote inference, navigate to the [Settings](https://www.expectedparrot.com/home/settings) page of your account and enable the **Run surveys remotely** toggle under Remote Execution. Learn more about how remote inference works at the [Remote Inference](/en/latest/remote_inference) section.

When you run a survey remotely at the Expected Parrot server your results are automatically cached. You can access them programmatically from your workspace (see examples of methods below). To turn off remote caching, navigate to the [Settings](https://www.expectedparrot.com/home/settings) page and disable the **Use remote cache** toggle.

## Universal remote cache

The universal remote cache is a collection of all the unique prompts that have been sent to any language models via the Expected Parrot server, and the responses that were returned. It is a shared resource that is available to all users for free.

When you run a survey at the Expected Parrot server your survey results will draw from the universal remote cache by default. This means that if your survey includes any prompts that have been run before, the stored response to those prompts is retrieved from the universal remote cache and included in your results, at no cost to you. If a set of prompts has not been run before, then a new response is generated, included in your results and added to the universal remote cache.

(By “prompts” we mean a unique user prompt for a question together with a unique system prompt for an agent, if one was used with the question.)

### Fresh responses

If you want to draw fresh responses, you can pass a parameter fresh=True to the run() method. Your results object will still have a cache automatically attached to it, and the universal remote cache will be updated with any new responses that are generated. (There can be multiple stored responses for a set of prompts if fresh responses are specified for a survey.)

### Features of the universal remote cache

The universal remote cache offers the following features:

* **Free access:** It is free to use and available to all users, regardless of whether you are running surveys remotely with your own keys for language models or an Expected Parrot API key.
* **Free storage & retrieval:** There is no limit on the number of responses that you can add to the universal remote cache or retrieve from it.
* **Automatic updates:** It is automatically updated whenever a survey is run remotely.
* **Multiple responses:** If a fresh response is generated for a question that is different from a response already stored in the universal remote cache, the new response is added with an iteration index.
* **No deletions:** You cannot delete entries from the universal remote cache.
* **No manual additions:** You cannot manually add entries. The only way to add responses to the universal remote cache is by running a survey remotely at the Expected Parrot server.
* **Sharing & reproducibility:** A new cache is automatically attached to each results object, which can be posted and shared with other users at Expected Parrot.
* **Privacy:** It is not queryable, and no user information is available. You must run a survey to retrieve responses from the universal remote cache.

<Note>
  *Note:* The universal remote cache is not available for local inference (surveys run on your own machine).
</Note>

### Frequently asked questions

<AccordionGroup>
  <Accordion title="How do I add responses to the universal remote cache?">
    This happens automatically when you run a survey remotely.
  </Accordion>

  <Accordion title="How do I get a stored response?">
    When you run a question you will retrieve a stored response by default if it exists. If you want to generate a fresh response you can use run(fresh=True).
  </Accordion>

  <Accordion title="How do I know whether I will retrieve a stored response?">
    Results that were generated at the Expected Parrot server will show a verified checkmark ✓ on the platform. If you rerun a survey with verified results you will retrieve stored responses.
  </Accordion>

  <Accordion title="Is the universal remote cache queryable? Can I check whether there is a stored response for a question?">
    No, the universal remote cache is not queryable. You will only know that a response is available by rerunning verified results.
  </Accordion>

  <Accordion title="Can I see which user generated a stored response in the universal remote cache?">
    No, there is no user information in the universal remote cache.
  </Accordion>

  <Accordion title="Why can’t I add my existing caches to the universal remote cache?">
    The purpose of the universal remote cache is to provide a canonical, verified collection of responses to allow researchers to be confident in results and easily reproduce them at no cost. By only allowing responses generated at the Expected Parrot server we can verify the results that are reproduced.
  </Accordion>

  <Accordion title="What if I want to run a survey remotely but do not want my responses added to the universal remote cache?">
    This is not allowed. Any new or fresh responses generated at the Expected Parrot server are automatically added to the universal remote cache. If you do not want to add responses to the universal remote cache you must run your surveys locally.
  </Accordion>

  <Accordion title="Can I access the universal remote cache when I run a survey locally?">
    No, the universal remote cache is only available when running a survey remotely.
  </Accordion>

  <Accordion title="Can I delete a response in the universal remote cache?">
    No, this is not allowed.
  </Accordion>

  <Accordion title="What happens if I delete my account?">
    Any remote cache entries that you generated will remain in the universal remote cache. All information about your account will be deleted.
  </Accordion>
</AccordionGroup>

## Using the remote cache

Your remote cache entries include all of the responses to questions that you have run remotely, whether generated fresh or retrieved from the universal remote cache.

For example, here we run a survey with remote caching activated, and pass a description to readily identify the job at Expected Parrot:

```python theme={null}
from edsl import Model, QuestionFreeText, Survey

m = Model("gemini-1.5-flash")

q = QuestionFreeText(
  question_name = "prime",
  question_text = "Is 2 a prime number?"
)

survey = Survey(questions = [q])

results = survey.by(m).run(
  remote_inference_description = "Example survey", # optional
  remote_inference_visibility = "public" # optional
)

```

### Reproducing results

When you share a results object (e.g., post it publicly at Expected Parrot or share it privately with other users) the cache attached to it is automatically shared with it. This can be useful if you want to share a specific historic cache for a survey or project (e.g., to allow other users to reproduce your results). You can inspect the cache for a results object by calling the cache property on a results object.

For example, here we inspect the cache for the survey that we ran above:

```python theme={null}
results.cache
```

Output:

| model | parameters | system\_prompt | user\_prompt | output | iteration | timestamp | cache\_key |
| :- | :- | :- | :- | :- | :- | :- | :- |
| gemini-1.5-flash | `{‘temperature’: 0.5, ‘topP’: 1, ‘topK’: 1, ‘maxOutputTokens’: 2048, ‘stopSequences’: []}` | nan | Is 2 a prime number? | `{“candidates”: [{“content”: {“parts”: [{“text”: “Yes, 2 is a prime number. It’s the only even prime number.n”}], “role”: “model”}, “finish_reason”: 1, “safety_ratings”: [{“category”: 8, “probability”: 1, “blocked”: false}, {“category”: 10, “probability”: 1, “blocked”: false}, {“category”: 7, “probability”: 1, “blocked”: false}, {“category”: 9, “probability”: 1, “blocked”: false}], “avg_logprobs”: -0.0006228652317076921, “token_count”: 0, “grounding_attributions”: []}], “usage_metadata”: {“prompt_token_count”: 7, “candidates_token_count”: 20, “total_token_count”: 27, “cached_content_token_count”: 0}, “model_version”: “gemini-1.5-flash”}` | 0 | 1738759640 | b939c0cf262061c7aedbbbfedc540689 |

See [Caching LLM Calls](/en/latest/data#caching) for more details on caching results locally.

## Remote cache methods

When remote caching is activated, EDSL will automatically send responses to the server when you run a job (i.e., you do not need to execute methods manually).

If you want to interact with the remote cache programatically, you can use the following methods:

### Coop class

> #### *class* edsl.coop.coop.Coop(*api\_key: str | None = None*, *url: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Bases: `CoopFunctionsMixin`

Client for the Expected Parrot API that provides cloud-based functionality for EDSL.

The Coop class is the main interface for interacting with Expected Parrot’s cloud services. It enables:

1. Storing and retrieving EDSL objects (surveys, agents, models, results, etc.)
2. Running inference jobs remotely for better performance and scalability
3. Retrieving and caching interview results
4. Managing API keys and authentication
5. Accessing model availability and pricing information

The client handles authentication, serialization/deserialization of EDSL objects, and communication with the Expected Parrot API endpoints. It also provides methods for tracking job status and managing results.

When initialized without parameters, Coop will attempt to use an API key from: 1. The EXPECTED\_PARROT\_API\_KEY environment variable 2. A stored key in the user’s config directory 3. Interactive login if needed

***Attributes***:

api\_key (str): The API key used for authentication url (str): The base URL for the Expected Parrot API api\_url (str): The URL for API endpoints (derived from base URL)

> #### **init**(*api\_key: str | None = None*, *url: str | None = None*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Initialize the Expected Parrot API client.

This constructor sets up the connection to Expected Parrot’s cloud services. If not provided explicitly, it will attempt to obtain an API key from environment variables or from a stored location in the user’s config directory.

> **Parameters**:

api\_key (str, optional): API key for authentication with Expected Parrot.

If not provided, will attempt to obtain from environment or stored location.

url (str, optional): Base URL for the Expected Parrot service.

If not provided, uses the default from configuration.

> **Notes**:

* The API key is stored in the EXPECTED\_PARROT\_API\_KEY environment variable or in a platform-specific config directory
* The URL is determined based on whether it’s a production, staging, or development environment
* The api\_url for actual API endpoints is derived from the base URL

Example:

```bash theme={null}
>>> coop = Coop()  # Uses API key from environment or stored location
>>> coop = Coop(api_key="your-api-key")  # Explicitly provide API key
```

> #### approve\_prolific\_study\_submission(*project\_uuid: str*, *study\_id: str*, *submission\_id: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Approve a Prolific study submission.

> check\_for\_updates(*silent: bool = False*) → dict | None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Check if there’s a newer version of EDSL available.

**Args:**

silent: If True, don’t print any messages to console

**Returns:**

dict with version info if update is available, None otherwise

> #### create(*object: [Agent](/en/latest/agents#agent-class "edsl.agents.agent.Agent") | [AgentList](/en/latest/agents#agentlist-class "edsl.agents.agent_list.AgentList") | [Cache](/en/latest/data#cache-class "edsl.caching.cache.Cache") | [LanguageModel](/en/latest/language_models#languagemodel-class "edsl.language_models.LanguageModel") | [ModelList](/en/latest/language_models#modellist-class "edsl.language_models.ModelList") | Notebook | Type\[[QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.question_base.QuestionBase")] | [Results](/en/latest/results#result-class "edsl.results.results.Results") | [Scenario](/en/latest/scenarios#scenario-class "edsl.scenarios.scenario.Scenario") | [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.scenario_list.ScenarioList") | [Survey](/en/latest/surveys#survey-class "edsl.surveys.survey.Survey")*, *description: str | None = None*, *alias: str | None = None*, *visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Store an EDSL object in the Expected Parrot cloud service.

This method uploads an EDSL object (like a Survey, Agent, or Results) to the Expected Parrot cloud service for storage, sharing, or further processing.

> **Parameters**:

object (EDSLObject): The EDSL object to store (Survey, Agent, Results, etc.) description (str, optional): A human-readable description of the object alias (str, optional): A custom alias for easier reference later visibility (VisibilityType, optional): Access level for the object. One of:

* “private”: Only accessible by the owner
* “public”: Accessible by anyone
* “unlisted”: Accessible with the link, but not listed publicly

> **Returns:**

**dict: Information about the created object including:**

* url: The URL to access the object
* alias\_url: The URL with the custom alias (if provided)
* uuid: The unique identifier for the object
* visibility: The visibility setting
* version: The EDSL version used to create the object

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> **Example:**

```bash theme={null}
>>> survey = Survey(questions=[QuestionFreeText(question_name="name")])
>>> result = coop.create(survey, description="Basic survey", visibility="public")
>>> print(result["url"])  # URL to access the survey
```

> #### create\_project(*survey: [Survey](/en/latest/surveys#survey-class "edsl.surveys.Survey")*, *scenario\_list: [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList") | None = None*, *scenario\_list\_method: Literal\['randomize', 'loop', 'single\_scenario', 'ordered'] | None = None*, *project\_name: str = 'Project'*, *survey\_description: str | None = None*, *survey\_alias: str | None = None*, *survey\_visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*, *scenario\_list\_description: str | None = None*, *scenario\_list\_alias: str | None = None*, *scenario\_list\_visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Create a survey object on Expected Parrot, then create a project from the survey.

> #### create\_project\_run(*project\_uuid: str*, *name: str | None = None*, *scenario\_list\_uuid: str | UUID | None = None*, *scenario\_list\_method: Literal\['randomize', 'loop', 'single\_scenario', 'ordered'] | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Create a project run.

> #### create\_prolific\_study(*project\_uuid: str*, *project\_run\_uuid: str*, *name: str*, *description: str*, *num\_participants: int*, *estimated\_completion\_time\_minutes: int*, *participant\_payment\_cents: int*, *device\_compatibility: List\[Literal\['desktop', 'tablet', 'mobile']] | None = None*, *peripheral\_requirements: List\[Literal\['audio', 'camera', 'download', 'microphone']] | None = None*, *filters: List\[Dict] | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Create a Prolific study for a project. Returns a dict with the study details.

To add filters to your study, you should first pull the list of supported filters using Coop.list\_prolific\_filters(). Then, you can use the create\_study\_filter method of the returned CoopProlificFilters object to create a valid filter dict.

> #### create\_widget(*short\_name: str*, *display\_name: str*, *esm\_code: str*, *css\_code: str | None = None*, *description: str | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Create a new widget.

> **Parameters:**

short\_name (str): The short name identifier for the widget. Must start with a lowercase letter and contain only lowercase letters, digits, and underscores display\_name (str): The display name of the widget description (str): A human-readable description of the widget esm\_code (str): The ESM JavaScript code for the widget css\_code (str): The CSS code for the widget

> **Returns:**

dict: Information about the created widget including:

* short\_name: The widget’s short name
* display\_name: The widget’s display name
* description: The widget’s description

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> #### delete(*url\_or\_uuid: str | UUID*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Delete an object from the server.

> **Parameters:**

**url\_or\_uuid** – The UUID or URL of the object. URLs can be in the form content/uuid or content/username/alias.

> #### delete\_project\_run(*project\_uuid: str*, *project\_run\_uuid: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Delete a project run.

> #### delete\_prolific\_study(*project\_uuid: str*, *study\_id: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Deletes a Prolific study.

<Note>
  Note: Only draft studies can be deleted. Once you publish a study, it cannot be deleted.
</Note>

> #### delete\_widget(*short\_name: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Delete a widget by short name.

> **Parameters:**

short\_name (str): The short name of the widget to delete

> **Returns:**

dict: Success status

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> #### *property* edsl\_settings\*: dict\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Retrieve and return the EDSL settings stored on Expected Parrot. If no response is received within 5 seconds, return an empty dict.

> #### execute\_firecrawl\_request(*request\_dict: Dict\[str, Any]*) → Any[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Execute a Firecrawl request through the Extension Gateway.

This method sends a Firecrawl request dictionary to the Extension Gateway’s /firecrawl/execute endpoint, which processes it using FirecrawlScenario and returns EDSL Scenario/ScenarioList objects.

> **Parameters:**

request\_dict (Dict\[str, Any]): A dictionary containing the Firecrawl request.

Must include: - method: The Firecrawl method to execute (scrape, crawl, search, extract, map\_urls) - api\_key: Optional if provided via environment or this method will add it - Other method-specific parameters (url\_or\_urls, query\_or\_queries, etc.)

> **Returns:**

Any: The result from FirecrawlScenario execution:

* For scrape/extract with single URL: Scenario object
* For scrape/extract with multiple URLs: ScenarioList object
* For crawl/search/map\_urls: ScenarioList object

> **Raises:**

httpx.HTTPError: If the request to the Extension Gateway fails ValueError: If the request\_dict is missing required fields Exception: If the Firecrawl execution fails

> **Example:**

```bash theme={null}
>>> # Scrape a single URL
>>> result = coop.execute_firecrawl_request({
...     "method": "scrape",
...     "url_or_urls": "https://example.com",
...     "kwargs": {"formats": ["markdown"]}
... })

```

```bash theme={null}
>>> # Search the web
>>> results = coop.execute_firecrawl_request({
...     "method": "search",
...     "query_or_queries": "AI research papers",
...     "kwargs": {"limit": 10}
... })

```

```bash theme={null}
>>> # Extract structured data
>>> result = coop.execute_firecrawl_request({
...     "method": "extract",
...     "url_or_urls": "https://shop.example.com/product",
...     "schema": {"title": "string", "price": "number"},
... })
```

> #### fetch\_models() → Dict\[str, List\[str]][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Fetch information about available language models from Expected Parrot.

This method retrieves the current list of available language models grouped by service provider (e.g., OpenAI, Anthropic, etc.). This information is useful for programmatically selecting models based on availability and for ensuring that jobs only use supported models.

> **Returns:**

ServiceToModelsMapping: A mapping of service providers to their available models.

> **Example structure:**

```json theme={null}
{
“openai”: [“gpt-4”, “gpt-3.5-turbo”, …], “anthropic”: [“claude-3-opus”, “claude-3-sonnet”, …], …
}
```

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> **Notes:**

* The availability of models may change over time
* Not all models may be accessible with your current API keys
* Use this method to check for model availability before creating jobs
* Models may have different capabilities (text-only, multimodal, etc.)

> **Example:**

```bash theme={null}
>>> models = coop.fetch_models()
>>> if "gpt-4" in models.get("openai", []):
...     print("GPT-4 is available")
>>> available_services = list(models.keys())
>>> print(f"Available services: {available_services}")
```

> #### fetch\_prices() → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Fetch the current pricing information for language models.

This method retrieves the latest pricing information for all supported language models from the Expected Parrot API. The pricing data is used to estimate costs for jobs and to optimize model selection based on budget constraints.

> **Returns:**

> **dict: A dictionary mapping (service, model) tuples to pricing information.**

Each entry contains token pricing for input and output tokens. Example structure:

```json theme={null}
{
(‘openai’, ‘gpt-4’):
{
‘input’: {‘usd_per_1M_tokens’: 30.0, …}, ‘output’: {‘usd_per_1M_tokens’: 60.0, …}
}
}
```

> **Raises:**

ValueError: If the EDSL\_FETCH\_TOKEN\_PRICES configuration setting is invalid

> **Notes:**

* Returns an empty dict if EDSL\_FETCH\_TOKEN\_PRICES is set to “False”
* The pricing data is cached to minimize API calls
* Pricing may vary based on the model, provider, and token type (input/output)
* All prices are in USD per million tokens

> **Example:**

```bash theme={null}
>>> prices = coop.fetch_prices()
>>> gpt4_price = prices.get(('openai', 'gpt-4'), {})
>>> print(f"GPT-4 input price: ${gpt4_price.get('input', {}).get('usd_per_1M_tokens')}")
```

> #### fetch\_rate\_limit\_config\_vars() → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Fetch a dict of rate limit config vars from Expected Parrot.

The dict keys are RPM and TPM variables like EDSL\_SERVICE\_RPM\_OPENAI.

> #### fetch\_working\_models() → List\[dict][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Fetch a list of working models from Expected Parrot.

> **Example output:**

```json theme={null}
[{
“service”: “openai”, “model”: “gpt-4o”, “works_with_text”: True, “works_with_images”: True, “usd_per_1M_input_tokens”: 2.5, “usd_per_1M_output_tokens”: 10.0,
}]
```

> #### get(*url\_or\_uuid: str | UUID*, *expected\_object\_type: Literal\['agent', 'agent\_list', 'cache', 'model', 'model\_list', 'notebook', 'question', 'results', 'scenario', 'scenario\_list', 'survey'] | None = None*) → [Agent](/en/latest/agents#agent-class "edsl.agents.agent.Agent") | [AgentList](/en/latest/agents#agentlist-class "edsl.agents.agent_list.AgentList") | [Cache](/en/latest/data#cache-class "edsl.caching.cache.Cache") | [LanguageModel](/en/latest/language_models#languagemodel-class "edsl.language_models.LanguageModel") | [ModelList](/en/latest/language_models#modellist-class "edsl.language_models.ModelList") | Notebook | Type\[[QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.question_base.QuestionBase")] | [Results](/en/latest/results#result-class "edsl.results.results.Results") | [Scenario](/en/latest/scenarios#scenario-class "edsl.scenarios.scenario.Scenario") | [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.scenario_list.ScenarioList") | [Survey](/en/latest/surveys#survey-class "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Retrieve an EDSL object from the Expected Parrot cloud service.

This method downloads and deserializes an EDSL object from the cloud service using either its UUID, URL, or username/alias combination.

> **Parameters:**

**url\_or\_uuid (Union\[str, UUID]): Identifier for the object to retrieve.**

Can be one of: - UUID string (e.g., “123e4567-e89b-12d3-a456-426614174000”) - Full URL (e.g., “[https://expectedparrot.com/content/123e4567](https://expectedparrot.com/content/123e4567)…”) - Alias URL (e.g., “[https://expectedparrot.com/content/username/my-survey](https://expectedparrot.com/content/username/my-survey)”)

**expected\_object\_type (ObjectType, optional): If provided, validates that the**

retrieved object is of the expected type (e.g., “survey”, “agent”)

> **Returns:**

EDSLObject: The retrieved object as its original EDSL class instance (e.g., Survey, Agent, Results)

> **Raises:**

CoopNoUUIDError: If no UUID or URL is provided CoopInvalidURLError: If the URL format is invalid CoopServerResponseError: If the server returns an error (e.g., not found, unauthorized access)

Exception: If the retrieved object doesn’t match the expected type

> **Notes:**

* If the object’s visibility is set to “private”, you must be the owner to access it
* For objects stored with an alias, you can use either the UUID or the alias URL

> **Example:**

```bash theme={null}
>>> survey = coop.get("123e4567-e89b-12d3-a456-426614174000")
>>> survey = coop.get("https://expectedparrot.com/content/username/my-survey")
>>> survey = coop.get(url, expected_object_type="survey")  # Validates the type
```

> #### get\_balance() → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get the current credit balance for the authenticated user.

This method retrieves the user’s current credit balance information from the Expected Parrot platform.

> **Returns:**

dict: Information about the user’s credit balance, including:

* credits: The current number of credits in the user’s account
* usage\_history: Recent credit usage if available

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> **Example:**

```bash theme={null}
>>> balance = coop.get_balance()
>>> print(f"You have {balance['credits']} credits available.")
```

> #### get\_metadata(*url\_or\_uuid: str | UUID*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get an object’s metadata from the server.

> **Parameters:**

**url\_or\_uuid** – The UUID or URL of the object. URLs can be in the form content/uuid or content/username/alias.

> #### get\_profile() → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get the current user’s profile information.

This method retrieves the authenticated user’s profile information from the Expected Parrot platform using their API key.

> **Returns:**

dict: User profile information including:

* username: The user’s username
* email: The user’s email address

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> **Example:**

```python theme={null}
>>> profile = coop.get_profile()
>>> print(f"Welcome, {profile['username']}!")
```

> #### get\_progress\_bar\_url()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

> #### get\_project(*project\_uuid: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get a project from Expected Parrot.

> #### get\_project\_human\_responses(*project\_uuid: str*, *project\_run\_uuid: str | None = None*) → [Results](/en/latest/results#result-class "edsl.results.Results") | [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Return a Results object with the human responses for a project.

If generating the Results object fails, a ScenarioList will be returned instead.

> #### get\_prolific\_study(*project\_uuid: str*, *study\_id: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get a Prolific study. Returns a dict with the study details.

> #### get\_prolific\_study\_responses(*project\_uuid: str*, *study\_id: str*) → [Results](/en/latest/results#result-class "edsl.results.Results") | [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Return a Results object with the human responses for a project.

If generating the Results object fails, a ScenarioList will be returned instead.

> #### get\_running\_jobs() → List\[str][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get a list of currently running job IDs.

> **Returns:**

list\[str]: List of running job UUIDs

> #### get\_upload\_url(*object\_uuid: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get a signed upload URL for updating the content of an existing object.

This method gets a signed URL that allows direct upload to Google Cloud Storage for objects stored in the new format, while preserving the existing UUID.

> **Parameters:**

object\_uuid (str): The UUID of the object to get an upload URL for

> **Returns:**

**dict: A response containing:**

* signed\_url: The signed URL for uploading new content
* object\_uuid: The UUID of the object
* message: Success message

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server HTTPException: If the object is not found, not owned by user, or not in new format

> **Notes:**

* Only works with objects stored in the new format (transition table)
* User must be the owner of the object
* The signed URL expires after 60 minutes

> **Example:**

```python theme={null}
>>> response = coop.get_upload_url("123e4567-e89b-12d3-a456-426614174000")
>>> upload_url = response['signed_url']
>>> # Use the upload_url to PUT new content directly to GCS
```

> #### get\_uuid\_from\_hash(*hash\_value: str*) → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Retrieve the UUID for an object based on its hash.

This method calls the remote endpoint to get the UUID associated with an object hash.

**Args:**

hash\_value (str): The hash value of the object to look up

**Returns:**

str: The UUID of the object if found

**Raises:**

CoopServerResponseError: If the object is not found or there’s an error

communicating with the server

> #### get\_widget(*short\_name: str*) → Dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get a specific widget by short name.

**Parameters:**

short\_name (str): The short name of the widget

**Returns:**

Dict: Complete widget data including ESM and CSS code

**Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> #### get\_widget\_metadata(*short\_name: str*) → Dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get metadata for a specific widget by short name.

> **Parameters:**

short\_name (str): The short name of the widget

> **Returns:**

Dict: Widget metadata including size information

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> #### *property* headers\*: dict\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Return the headers for the request.

> #### list(*object\_type: Literal\['agent', 'agent\_list', 'cache', 'model', 'model\_list', 'notebook', 'question', 'results', 'scenario', 'scenario\_list', 'survey'] | List\[Literal\['agent', 'agent\_list', 'cache', 'model', 'model\_list', 'notebook', 'question', 'results', 'scenario', 'scenario\_list', 'survey']] | None = None*, *visibility: Literal\['private', 'public', 'unlisted'] | List\[Literal\['private', 'public', 'unlisted']] | None = None*, *search\_query: str | None = None*, *page: int = 1*, *page\_size: int = 10*, *sort\_ascending: bool = False*, *community: bool = False*) → CoopRegularObjects[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Retrieve objects either owned by the user or shared with them.

Notes: - search\_query only works with the description field. - If sort\_ascending is False, then the most recently created objects are returned first. - If community is False, then only objects owned by the user or shared with the user are returned. - If community is True, then only public objects not owned by the user are returned.

> #### list\_prolific\_filters() → CoopProlificFilters[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get a ScenarioList of supported Prolific filters. This list has several methods that you can use to create valid filter dicts for use with Coop.create\_prolific\_study().

Call find() to examine a specific filter by ID: >>> filters = coop.list\_prolific\_filters() >>> filters.find(“age”) Scenario(

```json theme={null}
{
“filter_id”: “age”, “type”: “range”, “range_filter_min”: 18, “range_filter_max”: 100, …
}
```

Call create\_study\_filter() to create a valid filter dict: >>> filters.create\_study\_filter(“age”, min=30, max=40)

```json theme={null}
{
  “filter_id”: “age”, “selected_range”:
  {
  “lower”: 30, “upper”: 40,
  },
}
```

> #### list\_widgets(*search\_query: str | None = None*, *page: int = 1*, *page\_size: int = 10*) → [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get metadata for all widgets.

> **Parameters:**

page (int): Page number for pagination (default: 1) page\_size (int): Number of widgets per page (default: 10, max: 100)

> **Returns:**

List\[Dict]: List of widget metadata

> **Raises:**

CoopValueError: If page or page\_size parameters are invalid CoopServerResponseError: If there’s an error communicating with the server

> #### login()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Starts the EDSL auth token login flow.

> #### login\_gradio(*timeout: int = 120*, *launch: bool = True*, \*\**launch\_kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Start the EDSL auth token login flow inside a **Gradio** application.

This helper mirrors the behaviour of [`Coop.login_streamlit()`](/en/latest/#edsl.coop.coop.Coop.login_streamlit "edsl.coop.coop.Coop.login_streamlit") but renders the login link and status updates inside a Gradio UI. It will poll the Expected Parrot server for the API-key associated with a newly generated auth-token and, once received, store it via [:pyclass:`~edsl.coop.ep_key_handling.ExpectedParrotKeyHandler`](/en/latest/#id5) as well as in the local `.env` file so subsequent sessions pick it up automatically.

#### Parameters

**timeoutint, default 120**

How many seconds to wait for the user to complete the login before giving up.

**launchbool, default True**

If `True` the Gradio app is immediately launched with `demo.launch(**launch_kwargs)`. Set this to `False` if you want to embed the returned `gradio.Blocks` object into an existing Gradio interface.

**[\*\*](/en/latest/#id7)launch\_kwargs**

Additional keyword-arguments forwarded to `gr.Blocks.launch` when *launch* is `True`.

#### Returns

**str | gradio.Blocks | None**

* If the API-key is retrieved within *timeout* seconds while the function is executing (e.g. when *launch* is `False` and the caller integrates the Blocks into another app) the key is returned.
* If *launch* is `True` the method returns `None` after the Gradio app has been launched.
* If *launch* is `False` the constructed `gr.Blocks` is returned so the caller can compose it further.

> #### login\_streamlit(*timeout: int = 120*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Start the EDSL auth token login flow inside a Streamlit application.

This helper is functionally equivalent to `Coop.login` but renders the login link and status updates directly in the Streamlit UI. The method will automatically poll the Expected Parrot server for the API-key associated with the generated auth-token and, once received, store it via `ExpectedParrotKeyHandler` and write it to the local `.env` file so subsequent sessions pick it up automatically.

#### Parameters

**timeoutint, default 120**

How many seconds to wait for the user to complete the login before giving up and showing an error in the Streamlit app.

#### Returns

**str | None**

The API-key if the user logged-in successfully, otherwise `None`.

> #### new\_remote\_inference\_get(*job\_uuid: str | None = None*, *results\_uuid: str | None = None*, *include\_json\_string: bool | None = False*) → RemoteInferenceResponse[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get the status and details of a remote inference job.

This method retrieves the current status and information about a remote job, including links to results if the job has completed successfully.

> **Parameters:**

job\_uuid (str, optional): The UUID of the remote job to check results\_uuid (str, optional): The UUID of the results associated with the job

> (can be used if you only have the results UUID)

include\_json\_string (bool, optional): If True, include the json string for the job in the response

> **Returns:**

> **RemoteInferenceResponse: Information about the job including:**

job\_uuid: The unique identifier for the job results\_uuid: The UUID of the results results\_url: URL to access the results status: Current status (“queued”, “running”, “completed”, “failed”) version: EDSL version used for the job job\_json\_string: The json string for the job (if include\_json\_string is True) latest\_job\_run\_details: Metadata about the job status

> **interview\_details: Metadata about the job interview status (for jobs that have reached running status)**

total\_interviews: The total number of interviews in the job completed\_interviews: The number of completed interviews interviews\_with\_exceptions: The number of completed interviews that have exceptions exception\_counters: A list of exception counts for the job

exception\_type: The type of exception inference\_service: The inference service model: The model question\_name: The name of the question exception\_count: The number of exceptions

failure\_reason: The reason the job failed (failed jobs only) failure\_description: The description of the failure (failed jobs only) error\_report\_uuid: The UUID of the error report (partially failed jobs only) cost\_credits: The cost of the job run in credits cost\_usd: The cost of the job run in USD expenses: The expenses incurred by the job run

service: The service model: The model token\_type: The type of token (input or output) price\_per\_million\_tokens: The price per million tokens tokens\_count: The number of tokens consumed cost\_credits: The cost of the service/model/token type combination in credits cost\_usd: The cost of the service/model/token type combination in USD

> **Raises:**

ValueError: If neither job\_uuid nor results\_uuid is provided CoopServerResponseError: If there’s an error communicating with the server

> **Notes:**

* Either job\_uuid or results\_uuid must be provided
* If both are provided, job\_uuid takes precedence
* For completed jobs, you can use the results\_url to view or download results
* For failed jobs, check the latest\_error\_report\_url for debugging information

> **Example:**

```python theme={null}
>>> job_status = coop.new_remote_inference_get("9f8484ee-b407-40e4-9652-4133a7236c9c")
>>> print(f"Job status: {job_status['status']}")
>>> if job_status['status'] == 'completed':
...     print(f"Results available at: {job_status['results_url']}")
```

> #### old\_remote\_inference\_create(*job: [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs")*, *description: str | None = None*, *status: Literal\['queued', 'running', 'completed', 'failed', 'cancelled', 'cancelling', 'partial\_failed'] = 'queued'*, *visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*, *initial\_results\_visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*, *iterations: int | None = 1*, *fresh: bool | None = False*) → RemoteInferenceCreationInfo[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Create a remote inference job for execution in the Expected Parrot cloud.

This method sends a job to be executed in the cloud, which can be more efficient for large jobs or when you want to run jobs in the background. The job execution is handled by Expected Parrot’s infrastructure, and you can check the status and retrieve results later.

> **Parameters:**

job (Jobs): The EDSL job to run in the cloud description (str, optional): A human-readable description of the job status (RemoteJobStatus): Initial status, should be “queued” for normal use

Possible values: “queued”, “running”, “completed”, “failed”

visibility (VisibilityType): Access level for the job information. One of:

* “private”: Only accessible by the owner
* “public”: Accessible by anyone
* “unlisted”: Accessible with the link, but not listed publicly

initial\_results\_visibility (VisibilityType): Access level for the job results iterations (int): Number of times to run each interview (default: 1) fresh (bool): If True, ignore existing cache entries and generate new results

> **Returns:**

RemoteInferenceCreationInfo: Information about the created job including:

* uuid: The unique identifier for the job
* description: The job description
* status: Current status of the job
* iterations: Number of iterations for each interview
* visibility: Access level for the job
* version: EDSL version used to create the job

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> **Notes:**

* Remote jobs run asynchronously and may take time to complete
* Use remote\_inference\_get() with the returned UUID to check status
* Credits are consumed based on the complexity of the job

> **Example:**

```python theme={null}
>>> from edsl.jobs import Jobs
>>> job = Jobs.example()
>>> job_info = coop.remote_inference_create(job=job, description="My job")
>>> print(f"Job created with UUID: {job_info['uuid']}")
```

> #### patch(*url\_or\_uuid: str | UUID*, *description: str | None = None*, *alias: str | None = None*, *value: [Agent](/en/latest/agents#agent-class "edsl.agents.agent.Agent") | [AgentList](/en/latest/agents#agentlist-class "edsl.agents.agent_list.AgentList") | [Cache](/en/latest/data#cache-class "edsl.caching.cache.Cache") | [LanguageModel](/en/latest/language_models#languagemodel-class "edsl.language_models.LanguageModel") | [ModelList](/en/latest/language_models#modellist-class "edsl.language_models.ModelList") | Notebook | Type\[[QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.question_base.QuestionBase")] | [Results](/en/latest/results#result-class "edsl.results.results.Results") | [Scenario](/en/latest/scenarios#scenario-class "edsl.scenarios.scenario.Scenario") | [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.scenario_list.ScenarioList") | [Survey](/en/latest/surveys#survey-class "edsl.surveys.survey.Survey") | None = None*, *visibility: Literal\['private', 'public', 'unlisted'] | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Change the attributes of an uploaded object

> Parameters:

* **url\_or\_uuid** – The UUID or URL of the object. URLs can be in the form content/uuid or content/username/alias.
* **description** – Optional new description
* **alias** – Optional new alias
* **value** – Optional new object value
* **visibility** – Optional new visibility setting

pay\_for\_service(*credits\_transferred: int*, *recipient\_username: str*, *service\_name: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Pay for a service.

This method transfers a specified number of credits from the authenticated user’s account to another user’s account on the Expected Parrot platform.

> **Parameters:**

credits\_transferred (int): The number of credits to transfer to the recipient recipient\_username (str): The username of the recipient service\_name (str): The name of the service to pay for

> **Returns:**

**dict: Information about the transfer transaction, including:**

* success: Whether the transaction was successful
* transaction\_id: A unique identifier for the transaction
* remaining\_credits: The number of credits remaining in the sender’s account

> **Raises:**

**CoopServerResponseError: If there’s an error communicating with the server**

or if the transfer criteria aren’t met (e.g., insufficient credits)

> **Example:**

```python theme={null}
>>> result = coop.pay_for_service(
...     credits_transferred=100,
...     service_name="service_name",
...     recipient_username="friend_username",
... )
>>> print(f"Transfer successful! You have {result['remaining_credits']} credits left.")
```

> #### publish\_prolific\_study(*project\_uuid: str*, *study\_id: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Publish a Prolific study.

> #### pull(*url\_or\_uuid: str | UUID | None = None*, *expected\_object\_type: Literal\['agent', 'agent\_list', 'cache', 'model', 'model\_list', 'notebook', 'question', 'results', 'scenario', 'scenario\_list', 'survey'] | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Generate a signed URL for pulling an object directly from Google Cloud Storage.

This method gets a signed URL that allows direct download access to the object from Google Cloud Storage, which is more efficient for large files.

> **Parameters:**

**url\_or\_uuid (Union\[str, UUID], optional): Identifier for the object to retrieve.**

Can be one of: - UUID string (e.g., “123e4567-e89b-12d3-a456-426614174000”) - Full URL (e.g., “[https://expectedparrot.com/content/123e4567](https://expectedparrot.com/content/123e4567)…”) - Alias URL (e.g., “[https://expectedparrot.com/content/username/my-survey](https://expectedparrot.com/content/username/my-survey)”)

**expected\_object\_type (ObjectType, optional): If provided, validates that the**

retrieved object is of the expected type (e.g., “survey”, “agent”)

> **Returns:**

dict: A response containing the signed\_url for direct download

> **Raises:**

CoopNoUUIDError: If no UUID or URL is provided CoopInvalidURLError: If the URL format is invalid CoopServerResponseError: If there’s an error communicating with the server HTTPException: If the object or object files are not found

> **Example:**

```python theme={null}
>>> response = coop.pull("123e4567-e89b-12d3-a456-426614174000")
>>> response = coop.pull("https://expectedparrot.com/content/username/my-survey")
>>> print(f"Download URL: {response['signed_url']}")
>>> # Use the signed_url to download the object directly
```

> #### push(*object: [Agent](/en/latest/agents#agent-class "edsl.agents.agent.Agent") | [AgentList](/en/latest/agents#agentlist-class "edsl.agents.agent_list.AgentList") | [Cache](/en/latest/data#cache-class "edsl.caching.cache.Cache") | [LanguageModel](/en/latest/language_models#languagemodel-class "edsl.language_models.LanguageModel") | [ModelList](/en/latest/language_models#modellist-class "edsl.language_models.ModelList") | Notebook | Type\[[QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.question_base.QuestionBase")] | [Results](/en/latest/results#result-class "edsl.results.results.Results") | [Scenario](/en/latest/scenarios#scenario-class "edsl.scenarios.scenario.Scenario") | [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.scenario_list.ScenarioList") | [Survey](/en/latest/surveys#survey-class "edsl.surveys.survey.Survey")*, *description: str | None = None*, *alias: str | None = None*, *visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*) → [Scenario](/en/latest/scenarios#scenario-class "edsl.scenarios.Scenario")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Generate a signed URL for pushing an object directly to Google Cloud Storage.

This method gets a signed URL that allows direct upload access to Google Cloud Storage, which is more efficient for large files.

> **Parameters:**

object\_type (ObjectType): The type of object to be uploaded

> **Returns:**

dict: A response containing the signed\_url for direct upload and optionally a job\_id

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

Example:

```python theme={null}
>>> response = coop.push("scenario")
>>> print(f"Upload URL: {response['signed_url']}")
>>> # Use the signed_url to upload the object directly
```

> #### reject\_prolific\_study\_submission(*project\_uuid: str*, *study\_id: str*, *submission\_id: str*, *reason: Literal\['TOO\_QUICKLY', 'TOO\_SLOWLY', 'FAILED\_INSTRUCTIONS', 'INCOMP\_LONGITUDINAL', 'FAILED\_CHECK', 'LOW\_EFFORT', 'MALINGERING', 'NO\_CODE', 'BAD\_CODE', 'NO\_DATA', 'UNSUPP\_DEVICE', 'OTHER']*, *explanation: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Reject a Prolific study submission.

> #### *async* remote\_async\_execute\_model\_call(*model\_dict: dict*, *user\_prompt: str*, *system\_prompt: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

> #### remote\_cache\_get(*job\_uuid: str | UUID | None = None*) → List\[CacheEntry][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get all remote cache entries.

**Parameters:**

**select\_keys** (*optional*) – Only return CacheEntry objects with these keys.

```python theme={null}
>>> coop.remote_cache_get(job_uuid="...")
[CacheEntry(...), CacheEntry(...), ...]
```

> #### remote\_cache\_get\_by\_key(*select\_keys: List\[str] | None = None*) → List\[CacheEntry][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get all remote cache entries.

Parameters:

**select\_keys** (*optional*) – Only return CacheEntry objects with these keys.

```python theme={null}
>>> coop.remote_cache_get_by_key(selected_keys=["..."])
[CacheEntry(...), CacheEntry(...), ...]
```

> #### remote\_inference\_cost(*input: [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs") | [Survey](/en/latest/surveys#survey-class "edsl.surveys.Survey")*, *iterations: int = 1*) → int[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get the estimated cost in credits of a remote inference job.

**Parameters**:

**input** – The EDSL job to send to the server.

```python theme={null}
>>> job = Jobs.example()
>>> coop.remote_inference_cost(input=job)
{'credits_hold': 0.77, 'usd': 0.0076950000000000005}
```

> #### remote\_inference\_create(*job: [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs")*, *description: str | None = None*, *status: Literal\['queued', 'running', 'completed', 'failed', 'cancelled', 'cancelling', 'partial\_failed'] = 'queued'*, *visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*, *initial\_results\_visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*, *iterations: int | None = 1*, *fresh: bool | None = False*) → RemoteInferenceCreationInfo[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Create a remote inference job for execution in the Expected Parrot cloud.

This method sends a job to be executed in the cloud, which can be more efficient for large jobs or when you want to run jobs in the background. The job execution is handled by Expected Parrot’s infrastructure, and you can check the status and retrieve results later.

> **Parameters:**

job (Jobs): The EDSL job to run in the cloud description (str, optional): A human-readable description of the job status (RemoteJobStatus): Initial status, should be “queued” for normal use

Possible values: “queued”, “running”, “completed”, “failed”

visibility (VisibilityType): Access level for the job information. One of:

* “private”: Only accessible by the owner
* “public”: Accessible by anyone
* “unlisted”: Accessible with the link, but not listed publicly

initial\_results\_visibility (VisibilityType): Access level for the job results iterations (int): Number of times to run each interview (default: 1) fresh (bool): If True, ignore existing cache entries and generate new results

> **Returns:**

RemoteInferenceCreationInfo: Information about the created job including:

* uuid: The unique identifier for the job
* description: The job description
* status: Current status of the job
* iterations: Number of iterations for each interview
* visibility: Access level for the job
* version: EDSL version used to create the job

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> **Notes:**

* Remote jobs run asynchronously and may take time to complete
* Use remote\_inference\_get() with the returned UUID to check status
* Credits are consumed based on the complexity of the job

Example:

```python theme={null}
>>> from edsl.jobs import Jobs
>>> job = Jobs.example()
>>> job_info = coop.remote_inference_create(job=job, description="My job")
>>> print(f"Job created with UUID: {job_info['uuid']}")
```

> #### remote\_inference\_get(*job\_uuid: str | None = None*, *results\_uuid: str | None = None*, *include\_json\_string: bool | None = False*) → RemoteInferenceResponse[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get the status and details of a remote inference job.

This method retrieves the current status and information about a remote job, including links to results if the job has completed successfully.

> **Parameters:**

job\_uuid (str, optional): The UUID of the remote job to check results\_uuid (str, optional): The UUID of the results associated with the job

> (can be used if you only have the results UUID)

include\_json\_string (bool, optional): If True, include the json string for the job in the response

> **Returns:**

> RemoteInferenceResponse: Information about the job including:

job\_uuid: The unique identifier for the job results\_uuid: The UUID of the results results\_url: URL to access the results status: Current status (“queued”, “running”, “completed”, “failed”) version: EDSL version used for the job job\_json\_string: The json string for the job (if include\_json\_string is True) latest\_job\_run\_details: Metadata about the job status

> interview\_details: Metadata about the job interview status (for jobs that have reached running status)

total\_interviews: The total number of interviews in the job completed\_interviews: The number of completed interviews interviews\_with\_exceptions: The number of completed interviews that have exceptions exception\_counters: A list of exception counts for the job

exception\_type: The type of exception inference\_service: The inference service model: The model question\_name: The name of the question exception\_count: The number of exceptions

failure\_reason: The reason the job failed (failed jobs only) failure\_description: The description of the failure (failed jobs only) error\_report\_uuid: The UUID of the error report (partially failed jobs only) cost\_credits: The cost of the job run in credits cost\_usd: The cost of the job run in USD expenses: The expenses incurred by the job run

service: The service model: The model token\_type: The type of token (input or output) price\_per\_million\_tokens: The price per million tokens tokens\_count: The number of tokens consumed cost\_credits: The cost of the service/model/token type combination in credits cost\_usd: The cost of the service/model/token type combination in USD

> **Raises:**

ValueError: If neither job\_uuid nor results\_uuid is provided CoopServerResponseError: If there’s an error communicating with the server

> **Notes:**

* Either job\_uuid or results\_uuid must be provided
* If both are provided, job\_uuid takes precedence
* For completed jobs, you can use the results\_url to view or download results
* For failed jobs, check the latest\_error\_report\_url for debugging information

Example:

```python theme={null}
>>> job_status = coop.remote_inference_get("9f8484ee-b407-40e4-9652-4133a7236c9c")
>>> print(f"Job status: {job_status['status']}")
>>> if job_status['status'] == 'completed':
...     print(f"Results available at: {job_status['results_url']}")
```

> remote\_inference\_list(*status: Literal\['queued', 'running', 'completed', 'failed', 'cancelled', 'cancelling', 'partial\_failed'] | List\[Literal\['queued', 'running', 'completed', 'failed', 'cancelled', 'cancelling', 'partial\_failed']] | None = None*, *search\_query: str | None = None*, *page: int = 1*, *page\_size: int = 10*, *sort\_ascending: bool = False*) → CoopJobsObjects[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Retrieve jobs owned by the user.

<Note>
  Notes: - search\_query only works with the description field. - If sort\_ascending is False, then the most recently created jobs are returned first.
</Note>

*async* report\_error(*error: Exception*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Report an error for debugging purposes.

This method provides a non-blocking way to report errors that occur during EDSL operations. It sends error reports to the server for monitoring and debugging purposes, while also printing to stderr for immediate feedback.

> **Parameters:**

error (Exception): The exception to report

> **Example:**

```python theme={null}
>>> try:
...     # some operation that might fail
...     pass
... except Exception as e:
...     await coop.report_error(e)
```

> #### reset\_scenario\_sampling\_state(*project\_uuid: str*, *project\_run\_uuid: str*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Reset the scenario sampling state for a project.

This is useful if you have scenario\_list\_method=”ordered” and you want to start over with the first scenario in the list.

> #### test\_scenario\_sampling(*project\_uuid: str*, *project\_run\_uuid: str*) → List\[int][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get a sample for a project.

> #### transfer\_credits(*credits\_transferred: int*, *recipient\_username: str*, *transfer\_note: str = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Transfer credits to another user.

This method transfers a specified number of credits from the authenticated user’s account to another user’s account on the Expected Parrot platform.

> **Parameters:**

credits\_transferred (int): The number of credits to transfer to the recipient recipient\_username (str): The username of the recipient transfer\_note (str, optional): A personal note to include with the transfer

> **Returns:**

dict: Information about the transfer transaction, including:

* success: Whether the transaction was successful
* transaction\_id: A unique identifier for the transaction
* remaining\_credits: The number of credits remaining in the sender’s account

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

or if the transfer criteria aren’t met (e.g., insufficient credits)

> **Example:**

```python theme={null}
>>> result = coop.transfer_credits(
...     credits_transferred=100,
...     recipient_username="friend_username",
...     transfer_note="Thanks for your help!"
... )
>>> print(f"Transfer successful! You have {result['remaining_credits']} credits left.")
```

> #### update\_project\_run(*project\_uuid: str*, *project\_run\_uuid: str*, *name: str | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Update a project run.

> #### update\_prolific\_study(*project\_uuid: str*, *study\_id: str*, *project\_run\_uuid: str | None = None*, *name: str | None = None*, *description: str | None = None*, *num\_participants: int | None = None*, *estimated\_completion\_time\_minutes: int | None = None*, *participant\_payment\_cents: int | None = None*, *device\_compatibility: List\[Literal\['desktop', 'tablet', 'mobile']] | None = None*, *peripheral\_requirements: List\[Literal\['audio', 'camera', 'download', 'microphone']] | None = None*, *filters: List\[Dict] | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Update a Prolific study. Returns a dict with the study details.

> #### update\_widget(*existing\_short\_name: str*, *short\_name: str | None = None*, *display\_name: str | None = None*, *esm\_code: str | None = None*, *css\_code: str | None = None*, *description: str | None = None*) → Dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Update a widget by short name.

> **Parameters:**

existing\_short\_name (str): The current short name of the widget short\_name (str, optional): New short name for the widget. Must start with a lowercase letter and contain only lowercase letters, digits, and underscores display\_name (str, optional): New display name for the widget description (str, optional): New description for the widget esm\_code (str, optional): New ESM JavaScript code for the widget css\_code (str, optional): New CSS code for the widget

> **Returns:**

dict: Success status

> **Raises:**

CoopServerResponseError: If there’s an error communicating with the server

> #### web(*survey: dict*, *platform: Literal\['google\_forms', 'lime\_survey', 'survey\_monkey'] = 'lime\_survey'*, *email=None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)


# Remote Inference
Source: https://docs.expectedparrot.com/en/latest/remote_inference

Remote inference allows you to run surveys at the Expected Parrot server instead of locally on your own machine, and to use [Remote Caching](/en/latest/remote_caching) to store survey results and logs at your account.

<Note>
  *Note: You must have an Expected Parrot account in order to use remote inference and caching. By using remote inference you agree to any terms of use of service providers, which Expected Parrot may accept on your behalf and enforce in accordance with our terms of use.*
</Note>

## How it works

When remote inference is activated, calling the run() method on a survey will send it to the Expected Parrot server. Survey results and job details (history, costs, etc.) are automatically stored at the server and accessible from your workspace or at the [Jobs](https://www.expectedparrot.com/home/remote-inference) page of your account.

By default, a remote cache is used to retrieve responses to any questions that have already been run. You can choose whether to use it or generate fresh responses to questions. See the [Remote Caching](/en/latest/remote_caching) section for more details.

## Partial remote results

`ep run` fails closed when a remote job reports `failed` or `partial_failed`; it does not save incomplete Results through the normal success path. If a partially failed job contains useful successful interviews and you intentionally want them, opt in explicitly:

```bash theme={null}
ep run --jobs jobs.ep --allow-partial --output partial-results.ep
```

The command reports authoritative successful and failed interview counts. A partial job with zero successful interviews is always an error. Use the reported `ep jobs errors ...` command to retrieve its exception report.

## Activating remote inference

[Log in](https://www.expectedparrot.com/login) to your Expected Parrot account and navigate to your [Settings](https://www.expectedparrot.com/home/settings) page. Enable the **Run surveys remotely** toggle under Remote Execution:

<Frame>
  <img alt="Settings page showing Remote Execution and Remote Error Logging toggles" />
</Frame>

## Managing keys

An Expected Parrot key is required to use remote inference and to interact with the platform. Your key can be viewed (and reset) at the [Keys](https://www.expectedparrot.com/home/keys) page of your account, which also shows the `.env` format for using it locally:

<Frame>
  <img alt="Keys page showing Expected Parrot API Key and local .env configuration" />
</Frame>

You can also add your own LLM keys, grant access to other users, and set key priority at this page:

<Frame>
  <img alt="Keys page showing LLM Keys and key management options" />
</Frame>

See the [Managing Keys](/en/latest/api_keys) section for more details on methods for storing and managing keys.

## Credits

Running surveys with your Expected Parrot API key requires credits to cover API calls to service providers. Your account comes with free credits for getting started; you can check your balance and purchase additional credits at the [Credits](https://www.expectedparrot.com/home/credits) page of your account:

<Frame>
  <img alt="Credits page" />
</Frame>

Running surveys with your own keys does *not* consume credits. Learn more about purchasing credits and calculating costs at the credits section.

## Using remote inference

When remote inference is activated, calling the run() method will send a survey to the Expected Parrot server. You can access results and all information about the job (history, costs, etc.) from your workspace or your [Jobs](https://www.expectedparrot.com/home/remote-inference) page.

For example, here we run a simple survey with remote inference activated and inspect the job information that is automatically posted. We optionally pass description and visibility parameters (these can be edited at any time):

```python theme={null}
from edsl import Model, QuestionFreeText, Survey

m = Model("gemini-1.5-flash")

q = QuestionFreeText(
  question_name = "prime",
  question_text = "Is 2 a prime number?"
)

survey = Survey(questions = [q])

results = survey.by(m).run(
  remote_inference_description = "Example survey", # optional
  remote_inference_visibility = "public" # optional
)
```

Output (details will be unique to your job):

```bash theme={null}
✓ Current Status: Job completed and Results stored on Expected Parrot: http://www.expectedparrot.com/content/cfc51a12-63fe-41cf-b441-66d78ba47fb0
```

When the job has finished, it will appear with a status of *Completed*:

<Frame>
  <img alt="Remote inference page on the Expected Parrot platform. There is one job shown, and it has a status of &#x22;Completed.&#x22;" />
</Frame>

We can view the results of the job:

<Frame>
  <img alt="Remote inference results page on the Expected Parrot platform. There is one result shown." />
</Frame>

## Job details and costs

When you run a job using your Expected Parrot API key you are charged credits based on the number of tokens used. (When you run a job using your own keys you are charged directly by service providers based on the terms of your accounts.)

Before running a job, you can estimate the cost of the job by calling the estimate\_job\_cost() method on the Job object (a survey combined with a model). This will return information about the estimated total cost, input tokens, output tokens and per-model costs:

For example, here we estimate the cost of running a simple survey with a model:

```python expandable theme={null}
from edsl import Model, QuestionFreeText, Survey

m = Model("gemini-1.5-flash")

q = QuestionFreeText(
  question_name = "prime",
  question_text = "Is 2 a prime number?"
)

survey = Survey(questions = [q])

job = survey.by(m)

estimated_job_cost = job.estimate_job_cost()
estimated_job_cost

Output:

{'estimated_total_cost_usd': 1.575e-06,
'estimated_total_input_tokens': 5,
'estimated_total_output_tokens': 4,
'model_costs': [{'inference_service': 'google',
  'model': 'gemini-1.5-flash',
  'estimated_cost_usd': 1.575e-06,
  'estimated_input_tokens': 5,
  'estimated_output_tokens': 4}]}
```

We can also estimate the cost in credits to run the job remotely by passing the job to the remote\_inference\_cost() method of a `Coop` client object:

```python theme={null}
from edsl import Coop

coop = Coop()

estimated_remote_inference_cost = coop.remote_inference_cost(job) # using the job object from above
estimated_remote_inference_cost
```

Output:

```python theme={null}
{'credits': 0.01, 'usd': 1.575e-06}
```

Details on these methods can be found in the [credits](/en/latest/credits) section.

After running a job, you can view the actual cost in your job history or by calling the remote\_inference\_cost() method and passing it the job UUID (this is distinct from the results UUID, and can be found in your job history page).

You can also check the details of a job using the remote\_inference\_get() method as pass it the job UUID.

<Note>
  *Note:* When you run a job using your own keys, the cost estimates are based on the prices listed in the [model pricing page](https://www.expectedparrot.com/getting-started/coop-pricing). Your actual charges from service providers may vary based on the terms of your accounts with service providers.
</Note>

## Job history

You can click on any job to view its history. When a job fails, the job history logs will describe the error that caused the failure. The job history also shows which key was used to run each job (your own key, a key that has been share with you or your Expected Parrot API key):

<Frame>
  <img alt="A screenshot of job history logs on the Expected Parrot platform. The job has been run using a key that has been prioritized." />
</Frame>

## Remote inference methods

### Coop class

> #### *class* edsl.coop.coop.Coop(*api\_key: str | None = None*, *url: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Bases: `CoopFunctionsMixin`

Client for the Expected Parrot API that provides cloud-based functionality for EDSL.

The Coop class is the main interface for interacting with Expected Parrot’s cloud services. It enables:

1. Storing and retrieving EDSL objects (surveys, agents, models, results, etc.)
2. Running inference jobs remotely for better performance and scalability
3. Retrieving and caching interview results
4. Managing API keys and authentication
5. Accessing model availability and pricing information

The client handles authentication, serialization/deserialization of EDSL objects, and communication with the Expected Parrot API endpoints. It also provides methods for tracking job status and managing results.

When initialized without parameters, Coop will attempt to use an API key from: 1. The EXPECTED\_PARROT\_API\_KEY environment variable 2. A stored key in the user’s config directory 3. Interactive login if needed

**Attributes**:

api\_key (str): The API key used for authentication url (str): The base URL for the Expected Parrot API api\_url (str): The URL for API endpoints (derived from base URL)

> #### remote\_inference\_cost(*input: [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs") | [Survey](/en/latest/surveys#survey-class "edsl.surveys.Survey")*, *iterations: int = 1*) → int[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get the estimated cost in credits of a remote inference job.

**Parameters**: **input** – The EDSL job to send to the server.

```python theme={null}
>>> job = Jobs.example()
>>> coop.remote_inference_cost(input=job)
{'credits_hold': 0.77, 'usd': 0.0076950000000000005}
```

> #### remote\_inference\_create(*job: [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs")*, *description: str | None = None*, *status: Literal\['queued', 'running', 'completed', 'failed', 'cancelled', 'cancelling', 'partial\_failed'] = 'queued'*, *visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*, *initial\_results\_visibility: Literal\['private', 'public', 'unlisted'] | None = 'unlisted'*, *iterations: int | None = 1*, *fresh: bool | None = False*) → RemoteInferenceCreationInfo[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Create a remote inference job for execution in the Expected Parrot cloud.

This method sends a job to be executed in the cloud, which can be more efficient for large jobs or when you want to run jobs in the background. The job execution is handled by Expected Parrot’s infrastructure, and you can check the status and retrieve results later.

> **Parameters**:

job (Jobs): The EDSL job to run in the cloud description (str, optional): A human-readable description of the job status (RemoteJobStatus): Initial status, should be “queued” for normal use

Possible values: “queued”, “running”, “completed”, “failed”

**visibility (VisibilityType): Access level for the job information. One of:**

* “private”: Only accessible by the owner
* “public”: Accessible by anyone
* “unlisted”: Accessible with the link, but not listed publicly

initial\_results\_visibility (VisibilityType): Access level for the job results iterations (int): Number of times to run each interview (default: 1) fresh (bool): If True, ignore existing cache entries and generate new results

> **Returns**:

**RemoteInferenceCreationInfo: Information about the created job including:**

* uuid: The unique identifier for the job
* description: The job description
* status: Current status of the job
* iterations: Number of iterations for each interview
* visibility: Access level for the job
* version: EDSL version used to create the job

> **Raises**:

CoopServerResponseError: If there’s an error communicating with the server

> **Notes**:

* Remote jobs run asynchronously and may take time to complete
* Use remote\_inference\_get() with the returned UUID to check status
* Credits are consumed based on the complexity of the job

> **Example**:

```python theme={null}
>>> from edsl.jobs import Jobs
>>> job = Jobs.example()
>>> job_info = coop.remote_inference_create(job=job, description="My job")
>>> print(f"Job created with UUID: {job_info['uuid']}")
```

> #### remote\_inference\_get(*job\_uuid: str | None = None*, *results\_uuid: str | None = None*, *include\_json\_string: bool | None = False*) → RemoteInferenceResponse[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/coop/coop.py)

Get the status and details of a remote inference job.

This method retrieves the current status and information about a remote job, including links to results if the job has completed successfully.

**Parameters**:

job\_uuid (str, optional): The UUID of the remote job to check results\_uuid (str, optional): The UUID of the results associated with the job

(can be used if you only have the results UUID)

include\_json\_string (bool, optional): If True, include the json string for the job in the response

> **Returns**:

> **RemoteInferenceResponse: Information about the job including**:

job\_uuid: The unique identifier for the job results\_uuid: The UUID of the results results\_url: URL to access the results status: Current status (“queued”, “running”, “completed”, “failed”) version: EDSL version used for the job job\_json\_string: The json string for the job (if include\_json\_string is True) latest\_job\_run\_details: Metadata about the job status

> **interview\_details: Metadata about the job interview status (for jobs that have reached running status)**

total\_interviews: The total number of interviews in the job completed\_interviews: The number of completed interviews interviews\_with\_exceptions: The number of completed interviews that have exceptions exception\_counters: A list of exception counts for the job

exception\_type: The type of exception inference\_service: The inference service model: The model question\_name: The name of the question exception\_count: The number of exceptions

failure\_reason: The reason the job failed (failed jobs only) failure\_description: The description of the failure (failed jobs only) error\_report\_uuid: The UUID of the error report (partially failed jobs only) cost\_credits: The cost of the job run in credits cost\_usd: The cost of the job run in USD expenses: The expenses incurred by the job run

service: The service model: The model token\_type: The type of token (input or output) price\_per\_million\_tokens: The price per million tokens tokens\_count: The number of tokens consumed cost\_credits: The cost of the service/model/token type combination in credits cost\_usd: The cost of the service/model/token type combination in USD

> **Raises**:

ValueError: If neither job\_uuid nor results\_uuid is provided CoopServerResponseError: If there’s an error communicating with the server

> **Notes**:

* Either job\_uuid or results\_uuid must be provided
* If both are provided, job\_uuid takes precedence
* For completed jobs, you can use the results\_url to view or download results
* For failed jobs, check the latest\_error\_report\_url for debugging information

> **Example**:

```python theme={null}
>>> job_status = coop.remote_inference_get("9f8484ee-b407-40e4-9652-4133a7236c9c")
>>> print(f"Job status: {job_status['status']}")
>>> if job_status['status'] == 'completed':
...     print(f"Results available at: {job_status['results_url']}")
```


# Results
Source: https://docs.expectedparrot.com/en/latest/results

A *Results* object represents the outcome of running a *Survey*. It contains a list of individual *Result* objects, where each *Result* corresponds to a response to the survey for a unique combination of *Agent*, Model, and Scenario objects used with the survey.

For example, if a survey (of one more more questions) is administered to 2 agents and 2 language models (without any scenarios for the questions), the Results will contain 4 Result objects: one for each combination of agent and model used with the survey. If the survey questions are parameterized with 2 scenarios, the Results will expand to include 8 Result objects, accounting for all combinations of agents, models, and scenarios.

## Generating results

A Results object is not typically instantiated directly, but is returned by calling the run() method of a Survey after any agents, language models and scenarios are added to it.

In order to demonstrate how to access and interact with results, we use the following code to generate results for a simple survey. Note that specifying agent traits, scenarios (question parameter values) and language models is optional, and we include those steps here for illustration purposes. See the [Agents](/en/latest/agents), [Scenarios](/en/latest/scenarios) and models sections for more details on these components.

<Note>
  **Note:**

  You must store API keys for language models in order to generate results. Please see the [Managing Keys](/en/latest/api_keys) section for instructions on activating [Remote Inference](/en/latest/remote_inference) or storing your own API keys for inference service providers.
</Note>

To construct a survey we start by creating questions:

```python theme={null}
from edsl import QuestionLinearScale, QuestionMultipleChoice

q1 = QuestionLinearScale(
  question_name = "important",
  question_text = "On a scale from 1 to 5, how important to you is `{{ scenario.topic }}`?",
  question_options = [0, 1, 2, 3, 4, 5],
  option_labels = {0:"Not at all important", 5:"Very important"}
)

q2 = QuestionMultipleChoice(
  question_name = "read",
  question_text = "Have you read any books about `{{ scenario.topic }}`?",
  question_options = ["Yes", "No", "I do not know"]
)
```

We combine them in a survey to administer them together:

```python theme={null}
from edsl import Survey

survey = Survey([q1, q2])
```

We have parameterized our questions, so we can use them with different scenarios:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_list("topic", ["climate change", "house prices"])
```

We can optionally create agents with personas or other relevant traits to answer the survey:

```python theme={null}
from edsl import AgentList, Agent

agents = AgentList(
  Agent(traits = {"persona": p}) for p in ["student", "celebrity"]
)
```

We can specify the language models that we want to use to generate responses:

```python theme={null}
from edsl import ModelList, Model

models = ModelList(
  Model(m) for m in ["gemini-1.5-flash", "gpt-4o"]
)
```

Finally, we generate results by adding the scenarios, agents and models to the survey and calling the run() method:

```python theme={null}
results = survey.by(scenarios).by(agents).by(models).run()
```

For more details on each of the above steps, please see the [Agents](/en/latest/agents#agents), [Scenarios](/en/latest/scenarios#scenarios) and models sections of the docs.

## Result objects

We can check the number of Result objects created by inspecting the length of the Results:

```python theme={null}
len(results)
```

This will count 2 (scenarios) x 2 (agents) x 2 (models) = 8 Result objects:

```
8
```

## Generating multiple results

If we want to generate multiple results for a survey–i.e., more than 1 result for each combination of Agent, Model and Scenario objects used–we can pass the desired number of iterations when calling the run() method. For example, the following code will generate 3 results for our survey (n=3):

```python theme={null}
results = survey.by(scenarios).by(agents).by(models).run(n=3)
```

We can verify that the number of Result objects created is now 24 = 3 iterations x 2 scenarios x 2 agents x 2 models:

```
len(results)
```

```
24
```

We can readily inspect a result:

```python theme={null}
results[0]
```

Output:

## Results components

Results contain components that can be accessed and analyzed individually or collectively. We can see a list of these components by calling the columns method:

```python theme={null}
results.columns
```

### Visualizing column structure

For complex Results with many nested columns, you can use the `show_columns()` method to generate a tree visualization showing the hierarchical structure of your columns:

```python theme={null}
results.show_columns()
```

This creates a mermaid diagram that displays the column hierarchy, making it easier to understand the structure of your Results and identify which columns are available for selection.

The following list will be returned for the results generated by the above code:

| agent.agent\_index | agent.agent\_instruction | agent.agent\_name | agent.persona | answer.important | answer.read | cache\_keys.important\_cache\_key | cache\_keys.read\_cache\_key | cache\_keys.important\_cache\_used | cache\_keys.read\_cache\_used | comment.important\_comment | comment.read\_comment | generated\_tokens.important\_generated\_tokens | generated\_tokens.read\_generated\_tokens | iteration.iteration | model.frequency\_penalty | model.logprobs | model.maxOutputTokens | model.max\_tokens | model.model | model.presence\_penalty | model.stopSequences | model.temperature | model.topK | model.topP | model.top\_logprobs | model.top\_p | prompt.important\_system\_prompt | prompt.important\_user\_prompt | prompt.read\_system\_prompt | prompt.read\_user\_prompt | question\_options.important\_question\_options | question\_options.read\_question\_options | question\_text.important\_question\_text | question\_text.read\_question\_text | question\_type.important\_question\_type | question\_type.read\_question\_type | raw\_model\_response.important\_cost | raw\_model\_response.important\_input\_price\_per\_million\_tokens | raw\_model\_response.important\_input\_tokens | raw\_model\_response.important\_one\_usd\_buys | raw\_model\_response.important\_output\_price\_per\_million\_tokens | raw\_model\_response.important\_output\_tokens | raw\_model\_response.important\_raw\_model\_response | raw\_model\_response.read\_cost | raw\_model\_response.read\_input\_price\_per\_million\_tokens | raw\_model\_response.read\_input\_tokens | raw\_model\_response.read\_one\_usd\_buys | raw\_model\_response.read\_output\_price\_per\_million\_tokens | raw\_model\_response.read\_output\_tokens | raw\_model\_response.read\_raw\_model\_response | scenario.scenario\_index | scenario.topic |
| - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - |

The columns include information about each *agent*, *model* and corresponding *prompts* used to simulate the *answer* to each *question* and *scenario* in the survey, together with each *raw model response*. If the survey was run multiple times (run(n=\<integer>)) then the iteration.iteration column will show the iteration number for each result.

*Agent* information:

* **agent.agent\_index**: The index of the agent in the AgentList used to create the survey.
* **agent.instruction**: The instruction for the agent. This field is the optional instruction that was passed to the agent when it was created.
* **agent.agent\_name**: This field is always included in any Results object. It contains a unique identifier for each Agent that can be specified when an agent is is created (Agent(name=\<name>, traits=\{\<traits\_dict>})). If not specified, it is added automatically when results are generated (in the form Agent\_0, etc.).
* **agent.persona**: Each of the traits that we pass to an agent is represented in a column of the results. Our example code created a “persona” trait for each agent, so our results include a “persona” column for this information. Note that the keys for the traits dictionary should be a valid Python keys.

*Answer* information:

* **answer.important**: Agent responses to the linear scale important question.
* **answer.read**: Agent responses to the multiple choice read question.

*Cache* information:

* **cache\_keys.important\_cache\_key**: The cache key for the important question.
* **cache\_keys.important\_cache\_used**: Whether the existing cache was used for the important question.
* **cache\_keys.read\_cache\_key**: The cache key for the read question.
* **cache\_keys.read\_cache\_used**: Whether the existing cache was used for the read question.

*Comment* information:

A “comment” field is automatically included for every question in a survey other than free text questions, to allow the model to provide additional information about its response. The default instruction for the agent to provide a comment is included in user\_prompt for a question, and can be modified or omitted when creating the question. (See the [Prompts](/en/latest/prompts#prompts) section for details on modifying user and system prompts, and information about prompts in results below. Comments can also be automatically excluded by passing a parameter include\_comment=False a question when creating it.)

* **comment.important\_comment**: Agent commentary on responses to the important question.
* **comment.read\_comment**: Agent commentary on responses to the read question.

*Generated tokens* information:

* **generated\_tokens.important\_generated\_tokens**: The generated tokens for the important question.
* **generated\_tokens.read\_generated\_tokens**: The generated tokens for the read question.

*Iteration* information:

The iteration column shows the number of the run (run(n=\<integer>)) for the combination of components used (scenarios, agents and models).

*Model* information:

Each of model columns is a modifiable parameter of the models used to generate the responses.

* **model.frequency\_penalty**: The frequency penalty for the model.
* **model.logprobs**: The logprobs for the model.
* **model.maxOutputTokens**: The maximum number of output tokens for the model.
* **model.max\_tokens**: The maximum number of tokens for the model.
* **model.model**: The name of the model used.
* **model.presence\_penalty**: The presence penalty for the model.
* **model.stopSequences**: The stop sequences for the model.
* **model.temperature**: The temperature for the model.
* **model.topK**: The top k for the model.
* **model.topP**: The top p for the model.
* **model.top\_logprobs**: The top logprobs for the model.
* **model.top\_p**: The top p for the model.
* **model.use\_cache**: Whether the model uses cache.

<Note>
  **Note:** Some of the above fields are particular to specific models, and may have different names (e.g., top\_p vs. topP).
</Note>

*Prompt* information:

* **prompt.important\_system\_prompt**: The system prompt for the important question.
* **prompt.important\_user\_prompt**: The user prompt for the important question.
* **prompt.read\_system\_prompt**: The system prompt for the read question.
* **prompt.read\_user\_prompt**: The user prompt for the read question.

For more details about prompts, please see the [Prompts](/en/latest/prompts#prompts) section.

*Question* information:

* **question\_options.important\_question\_options**: The options for the important question, if any.
* **question\_options.read\_question\_options**: The options for the read question, if any.
* **question\_text.important\_question\_text**: The text of the important question.
* **question\_text.read\_question\_text**: The text of the read question.
* **question\_type.important\_question\_type**: The type of the important question.
* **question\_type.read\_question\_type**: The type of the read question.

*Raw model response* information:

* **raw\_model\_response.important\_cost**: The cost of the result for the important question, applying the token quanities & prices.
* **raw\_model\_response.important\_input\_price\_per\_million\_tokenss**: The price per million input tokens for the important question for the relevant model.
* **raw\_model\_response.important\_input\_tokens**: The number of input tokens for the important question for the relevant model.
* **raw\_model\_response.important\_one\_usd\_buys**: The number of identical results for the important question that 1USD would cover.
* **raw\_model\_response.important\_output\_price\_per\_million\_tokens**: The price per million output tokens for the important question for the relevant model.
* **raw\_model\_response.important\_output\_tokens**: The number of output tokens for the important question for the relevant model.
* **raw\_model\_response.important\_raw\_model\_response**: The raw model response for the important question.
* **raw\_model\_response.read\_cost**: The cost of the result for the read question, applying the token quanities & prices.
* **raw\_model\_response.read\_input\_price\_per\_million\_tokens**: The price per million input tokens for the read question for the relevant model.
* **raw\_model\_response.read\_input\_tokens**: The number of input tokens for the read question for the relevant model.
* **raw\_model\_response.read\_one\_usd\_buys**: The number of identical results for the read question that 1USD would cover.
* **raw\_model\_response.read\_output\_price\_per\_million\_tokens**: The price per million output tokens for the read question for the relevant model.
* **raw\_model\_response.read\_output\_tokens**: The number of output tokens for the read question for the relevant model.
* **raw\_model\_response.read\_raw\_model\_response**: The raw model response for the read question.

<Note>
  **Note:**

  Note that the cost of a result for a question is specific to the components (scenario, agent, model used with it).
</Note>

*Scenario* information:

* **scenario.scenario\_index**: The index of the scenario.
* **scenario.topic**: The values provided for the “topic” scenario for the questions.

<Note>
  **Note:**

  We recently added support for OpenAI reasoning models. See an example notebook for usage [here](https://www.expectedparrot.com/content/RobinHorton/reasoning-model-example). The Results that are generated with reasoning models include additional fields for reasoning summaries.
</Note>

## Creating tables by selecting columns

Each of these columns can be accessed directly by calling the select() method and passing the column names. Alternatively, we can specify the columns to exclude by calling the drop() method. These methods can be chained together to display the specified columns in a table format.

### Wildcard pattern support

The select() method supports wildcard patterns using `*` to match multiple columns at once. This is particularly useful when you want to select all columns of a certain type or with a common prefix:

```python theme={null}
# Select all answer columns
results.select("answer.*")

# Select all cost-related columns
results.select("*_cost")

# Select all columns from raw_model_response
results.select("raw_model_response.*")
```

For example, the following code will print a table showing the answers for read and important together with model, persona and topic columns (because the column names are unique we can drop the model, agent, scenario and answer prefixes when selecting them):

```python theme={null}
results = survey.by(scenarios).by(agents).by(models).run() # Running the survey once
results.select("model", "persona", "topic", "read", "important")
```

A table with the selected columns will be printed:

| model.model | agent.persona | scenario.topic | answer.read | answer.important |
| :- | :- | :- | :- | :- |
| gemini-1.5-flash | student | climate change | Yes | 5 |
| gpt-4o | student | climate change | Yes | 5 |
| gemini-1.5-flash | student | house prices | No | 1 |
| gpt-4o | student | house prices | No | 3 |
| gemini-1.5-flash | celebrity | climate change | Yes | 5 |
| gpt-4o | celebrity | climate change | Yes | 5 |
| gemini-1.5-flash | celebrity | house prices | Yes | 3 |
| gpt-4o | celebrity | house prices | No | 3 |

## Sorting results

We can sort the columns by calling the sort\_by method and passing it the column names to sort by:

```python theme={null}
  results
  .sort_by("model", "persona", reverse=False)
  .select("model", "persona", "topic", "read", "important")
)
```

The following table will be printed:

| model.model | agent.persona | scenario.topic | answer.read | answer.important |
| :- | :- | :- | :- | :- |
| gemini-1.5-flash | celebrity | climate change | Yes | 5 |
| gemini-1.5-flash | celebrity | house prices | Yes | 3 |
| gemini-1.5-flash | student | climate change | Yes | 5 |
| gemini-1.5-flash | student | house prices | No | 1 |
| gpt-4o | celebrity | climate change | Yes | 5 |
| gpt-4o | celebrity | house prices | No | 3 |
| gpt-4o | student | climate change | Yes | 5 |
| gpt-4o | student | house prices | No | 3 |

## Labeling results

We can also add some table labels by passing a dictionary to the pretty\_labels argument of the print method (note that we need to include the column prefixes when specifying the table labels, as shown below):

```python theme={null}
  results
  .sort_by("model", "persona", reverse=True)
  .select("model", "persona", "topic", "read", "important")
  .print(pretty_labels={
      "model.model": "LLM",
      "agent.persona": "Agent",
      "scenario.topic": "Topic",
      "answer.read": q2.question_text,
      "answer.important": q1.question_text
      }, format="rich")
)
```

The following table will be printed:

| LLM | Agent | Topic | Have you read any books about `{{ scenario.topic }}`? | On a scale from 1 to 5, how important to you is `{{ scenario.topic }}`? |
| :- | :- | :- | :- | :- |
| gpt-4o | student | climate change | Yes | 5 |
| gpt-4o | student | house prices | No | 3 |
| gpt-4o | celebrity | climate change | Yes | 5 |
| gpt-4o | celebrity | house prices | No | 3 |
| gemini-1.5-flash | student | climate change | Yes | 5 |
| gemini-1.5-flash | student | house prices | No | 1 |
| gemini-1.5-flash | celebrity | climate change | Yes | 5 |
| gemini-1.5-flash | celebrity | house prices | Yes | 3 |

## Filtering results

Results can be filtered by using the filter method and passing it a logical expression identifying the results that should be selected. For example, the following code will filter results where the answer to important is “5” and then just print the topic and important\_comment columns:

```python theme={null}
  results
  .filter("important == 5")
  .select("topic", "important", "important_comment")
)
```

This will return an abbreviated table:

| scenario.topic | answer.important | comment.important\_comment |
| :- | :- | :- |
| climate change | 5 | It’s, like, a huge deal. The future of the planet is at stake, and that affects everything - from the environment to the economy to social justice. It’s something I worry about a lot. |
| climate change | 5 | As a student, I’m really concerned about climate change because it affects our future and the planet we’ll inherit. It’s crucial to understand and address it to ensure a sustainable world for generations to come. |
| climate change | 5 | It’s a huge issue, you know? We only have one planet, and if we don’t take care of it, what kind of world are we leaving for future generations? It’s not just about polar bears; it’s about everything. It’s my responsibility, as someone with a platform, to speak out about it. |
| climate change | 5 | Climate change is a critical issue that affects everyone globally, and as a public figure, I believe it’s important to use my platform to raise awareness and advocate for sustainable practices. |

**Note:** The filter method allows us to pass the unique short names of the columns (without the prefixes) when specifying the logical expression. However, because the model.model column name is also a prefix, we need to include the prefix when filtering by this column, as shown in the example below:

```python theme={null}
  results
  .filter("model.model == 'gpt-4o'")
  .select("model", "persona", "topic", "read", "important")
)
```

This will return a table of results where the model is “gpt-4o”:

| model.model | agent.persona | scenario.topic | answer.read | answer.important |
| :- | :- | :- | :- | :- |
| gpt-4o | student | climate change | Yes | 5 |
| gpt-4o | student | house prices | No | 3 |
| gpt-4o | celebrity | climate change | Yes | 5 |
| gpt-4o | celebrity | house prices | No | 3 |

## Limiting results

We can select and print a limited number of results by passing the desired number of max\_rows to the print() method. This can be useful for quickly checking the first few results:

```python theme={null}
  results
  .select("model", "persona", "topic", "read", "important")
  .print(max_rows=4, format="rich")
)
```

This will return a table of the selected components of the first 4 results:

| model.model | agent.persona | scenario.topic | answer.read | answer.important |
| :- | :- | :- | :- | :- |
| gemini-1.5-flash | student | climate change | Yes | 5 |
| gpt-4o | student | climate change | Yes | 5 |
| gemini-1.5-flash | student | house prices | No | 1 |
| gpt-4o | student | house prices | No | 3 |

## Sampling results

We can select a sample of n results by passing the desired number of random results to the sample() method. This can be useful for checking a random subset of the results with different parameters:

```python theme={null}
sample_results = results.sample(2)

(
  sample_results
  .sort_by("model")
  .select("model", "persona", "topic", "read", "important")
)
```

This will return a table of the specified number of randomly selected results:

| model.model | agent.persona | scenario.topic | answer.read | answer.important |
| :- | :- | :- | :- | :- |
| gpt-4o | celebrity | house prices | No | 3 |
| gpt-4o | celebrity | climate change | Yes | 5 |

## Shuffling results

We can shuffle results by calling the shuffle() method. This can be useful for quickly checking the first few results:

```python theme={null}
shuffle_results = results.shuffle()

(
  shuffle_results
  .select("model", "persona", "topic", "read", "important")
)
```

This will return a table of shuffled results:

| model.model | agent.persona | scenario.topic | answer.read | answer.important |
| :- | :- | :- | :- | :- |
| gemini-1.5-flash | celebrity | climate change | Yes | 5 |
| gpt-4o | student | house prices | No | 3 |
| gemini-1.5-flash | celebrity | house prices | Yes | 3 |
| gemini-1.5-flash | student | house prices | No | 1 |
| gpt-4o | celebrity | house prices | No | 3 |
| gpt-4o | celebrity | climate change | Yes | 5 |
| gpt-4o | student | climate change | Yes | 5 |
| gemini-1.5-flash | student | climate change | Yes | 5 |

## Adding results

We can add results together straightforwardly by using the + operator:

```python theme={null}
add_results = results + results
```

We can see that the results have doubled:

```python theme={null}
len(add_results)
```

This will return the number of results:

```
16
```

## Flattening results

If a field of results contains dictionaries we can flatten them into separate fields by calling the flatten() method. This method takes a list of the fields to flatten and a boolean indicator whether to preserve the original fields in the new Results object that is returned.

For example:

```python theme={null}
from edsl import QuestionDict, Model

 m = Model("gemini-1.5-flash")

 q = QuestionDict(
   question_name = "recipe",
   question_text = "Please provide a simple recipe for hot chocolate.",
   answer_keys = ["title", "ingredients", "instructions"]
 )

 r = q.by(m).run()

 r.select("model", "recipe").flatten(field="answer.recipe", keep_original=True)
```

This will return a table of the flattened results:

| model.model | answer.recipe | answer.recipe.title | answer.recipe.ingredients | answer.recipe.instructions |
| :- | :- | :- | :- | :- |
| gemini-1.5-flash | \{‘title’: ‘Simple Hot Chocolate’, ‘ingredients’: \[‘1 cup milk (dairy or non-dairy)’, ‘1 tablespoon unsweetened cocoa powder’, ‘1-2 tablespoons sugar (or to taste)’, ‘Pinch of salt’], ‘instructions’: \[‘Combine milk, cocoa powder, sugar, and salt in a small saucepan.’, ‘Heat over medium heat, stirring constantly, until the mixture is smooth and heated through.’, ‘Do not boil.’, ‘Pour into a mug and enjoy!’]} | Simple Hot Chocolate | \[‘1 cup milk (dairy or non-dairy)’, ‘1 tablespoon unsweetened cocoa powder’, ‘1-2 tablespoons sugar (or to taste)’, ‘Pinch of salt’] | \[‘Combine milk, cocoa powder, sugar, and salt in a small saucepan.’, ‘Heat over medium heat, stirring constantly, until the mixture is smooth and heated through.’, ‘Do not boil.’, ‘Pour into a mug and enjoy!’] |

## Retrieving results

We can retrieve details about results posted to Expected Parrot by calling the list() method on the Results class. For example, the following code will return information about the 10 most recent results posted to the platform:

```python theme={null}
from edsl import Results

results = Results.list()
```

The following information will be returned:

| Column | Description |
| :- | :- |
| last\_updated\_ts | The timestamp when the result was last updated. |
| alias | The alias for the results. |
| uuid | The UUID of the results. |
| version | The version of the result. |
| created\_ts | The timestamp when the results were created. |
| visibility | The visibility of the results (public, private or unlisted). |
| description | A description of the results, if any. |
| url | The URL to access the results. |
| object\_type | The type of object (e.g., Results). |
| owner\_username | The username of the owner of the results. |
| alias\_url | The URL for the alias, if any. |

To access the next page of results, you can specify the page= parameter:

```python theme={null}
results = Results.list(page=2)
```

This will return the next page of results, with the same columns as above.

```python theme={null}
from edsl import Results

# Retrieve the first 2 pages of results and collect their UUIDs
uuids = []
for i in range(1, 3):
  results = Results.list(page=i)
  uuids.extend(list(results.to_key_value("uuid")))
```

If you have a predetermined number of objects, you can also use page\_size= to specify the number of objects per page (up to 100 objects):

```python theme={null}
results = Results.list(page_size=5)
```

This will return the first 5 results, with the same columns as above.

By default, the most recently created objects are returned first. You can reverse this by specifying sort\_ascending=True:

```python theme={null}
from edsl import Results

# Retrieve the first 10 results, sorted in ascending order by creation time
results = Results.list(sort_ascending=True)
```

You can also filter objects by description using the search\_query parameter:

```python theme={null}
from edsl import Results

# Retrieve results with a description containing the word "testing"
results = Results.list(search_query="testing")
```

If you want not just the metadata, but the actual object, you can call .fetch() on the metadata list:

```python theme={null}
from edsl import Results

# Retrieve the first 10 results and fetch the actual objects
results = Results.list().fetch()
```

The list() method can also be called on Agent and Jobs objects, and the `Coop` client object (to retrieve details of objects of any type).

## Generating a report

We can create a report of the results by calling the report() method and passing the columns to be included (all columns are included by default). This generates a report in markdown by iterating through the rows, presented as observations. You can optionally pass headers, a divider and a limit on the number of observations to include. It can be useful if you want to display some sample part of larger results in a working notebook you are sharing.

For example, the following code will generate a report of the first 4 results:

```python theme={null}
from edsl import QuestionFreeText, ScenarioList, Model

m = Model("gemini-1.5-flash")

s = ScenarioList.from_list("language", ["German", "Dutch", "French", "English"])

q = QuestionFreeText(
  question_name = "poem",
  question_text = "Please write me a short poem about winter in `{{ language }}`."
)

r = q.by(s).by(m).run()

r.select("model", "poem", "language").report(top_n=2, divider=False, return_string=True)
```

This will return a report of the first 2 results:

```python theme={null}
Observation: 1

model.model
gemini-1.5-flash

answer.poem
Der Schnee fällt leis', ein weicher Flor, Die Welt in Weiß, ein Zauberchor. Die Bäume stehn, in Stille gehüllt, Der Winterwind, sein Lied erfüllt.

(Translation: The snow falls softly, a gentle veil, / The world in white, a magic choir. / The trees stand, wrapped in silence, / The winter wind, its song fulfilled.)

scenario.language
German

Observation: 2
model.model
gemini-1.5-flash

answer.poem
De winter komt, de dagen kort, De sneeuw valt zacht, een wit decor. De bomen staan, kaal en stil, Een ijzige wind, een koude tril.

(Translation: Winter comes, the days are short, / The snow falls softly, a white décor. / The trees stand, bare and still, / An icy wind, a cold shiver.)

scenario.language
Dutch

"# Observation: 1n## model.modelngemini-1.5-flashn## answer.poemnDer Schnee fällt leis', ein weicher Flor,nDie Welt in Weiß, ein Zauberchor.nDie Bäume stehn, in Stille gehüllt,nDer Winterwind, sein Lied erfüllt.nn(Translation: The snow falls softly, a gentle veil, / The world in white, a magic choir. / The trees stand, wrapped in silence, / The winter wind, its song fulfilled.)n## scenario.languagenGermannn---nn# Observation: 2n## model.modelngemini-1.5-flashn## answer.poemnDe winter komt, de dagen kort,nDe sneeuw valt zacht, een wit decor.nDe bomen staan, kaal en stil,nEen ijzige wind, een koude tril.nn(Translation: Winter comes, the days are short, / The snow falls softly, a white décor. / The trees stand, bare and still, / An icy wind, a cold shiver.)n## scenario.languagenDutchn"
```

## Accessing results with SQL

We can interact with results via SQL using the sql method. This is done by passing a SQL query and a shape (“long” or “wide”) for the resulting table, where the table name in the query is “self”.

For example, the following code will return a table showing the model, persona, read and important columns for the first 4 results:

```python theme={null}
results.sql("select model, persona, read, important from self limit 4")
```

This following table will be displayed

| model | persona | read | important |
| :- | :- | :- | :- |
| gemini-1.5-flash | student | Yes | 5 |
| gpt-4o | student | Yes | 5 |
| gemini-1.5-flash | student | No | 1 |
| gpt-4o | student | No | 3 |

## Dataframes

We can also export results to other formats. The to\_pandas method will turn our results into a Pandas dataframe:

```python theme={null}
results.to_pandas()
```

For example, here we use it to create a dataframe consisting of the models, personas and the answers to the important question:

```python theme={null}
results.to_pandas()[["model.model", "agent.persona", "answer.important"]]
```

## Exporting to CSV or JSON

The to\_csv method will write the results to a CSV file:

```python theme={null}
results.to_pandas().to_csv("results.csv")
```

The to\_json method will write the results to a JSON file:

```python theme={null}
results.to_pandas().to_json("results.json")
```

## Revising prompts to improve results

If any of your results are missing model responses, you can use the spot\_issues() method to help identify the issues and then revise the prompts to improve the results. This method runs a meta-survey of (2) questions for any prompts that generated a bad or null response, and then returns the results of the meta-survey.

The first question in the survey is a QuestionFreeText question which prompts the model to describe the likely issues with the prompts:

```python theme={null}
The following prompts generated a bad or null response: '`{{ original_prompts }}`'
What do you think was the likely issue(s)?
```

The second question in the survey is a QuestionDict question which prompts the model to return a dictionary consisting of revised user and system prompts:

```python theme={null}
The following prompts generated a bad or null response: '`{{ original_prompts }}`'
You identified the issue(s) as '`{{ issues.answer }}`'.
Please revise the prompts to address the issue(s).
```

You can optionally pass a list of models to use with the meta-survey, instead of the default model.

Example usage:

```python theme={null}
# Returns a Results object with the results of the meta-survey
results.spot_issues(models=["gpt-4o"])

# You can inspect the metadata for your original prompts together with the results of the meta-survey
results.select(
  "original_question", # The name of the question that generated a bad or null response
  "original_agent_index", # The index of the agent that generated a bad or null response
  "original_scenario_index", # The index of the scenario that generated a bad or null response
  "original_prompts", # The original prompts that generated a bad or null response
  "answer.issues", # Free text description of potential issues in the original prompts
  "answer.revised" # A dictionary of revised user and system prompts
)
```

See an [example of the method](https://www.expectedparrot.com/content/385734e7-7767-4464-9ebd-0b009dd2e15f).

## Exceptions

If any exceptions are raised when the survey is run a detailed exceptions report is generated and can be opened in your browser. See the [Exceptions & Debugging](/en/latest/exceptions#exceptions) section for more information on exceptions.

## Result class

> ### *class* edsl.results.Result(*agent: [Agent](/en/latest/agents#result-class)*, *scenario: [Scenario](/en/latest/scenarios#scenario-class "edsl.scenarios.Scenario")*, *model: [LanguageModel](/en/latest/language_models#languagemodel-class "edsl.language_models.LanguageModel")*, *iteration: int*, *answer: dict\[QuestionName, AnswerValue]*, *prompt: dict\[QuestionName, str] = None*, *raw\_model\_response: dict | None = None*, *survey: 'Survey' | None = None*, *question\_to\_attributes: dict\[QuestionName, Any] | None = None*, *generated\_tokens: dict | None = None*, *comments\_dict: dict | None = None*, *reasoning\_summaries\_dict: dict | None = None*, *cache\_used\_dict: dict\[QuestionName, bool] | None = None*, *indices: dict | None = None*, *cache\_keys: dict\[QuestionName, str] | None = None*, *validated\_dict: dict\[QuestionName, bool] | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Bases: [`Base`](/en/latest/base#base-class "edsl.base.base_class.Base"), `UserDict`

The Result class captures the complete data from one agent interview.

A Result object stores the agent, scenario, language model, and all answers provided during an interview, along with metadata such as token usage, caching information, and raw model responses. It provides a rich interface for accessing this data and supports serialization for storage and retrieval.

The Result class inherits from both Base (for serialization) and UserDict (for dictionary-like behavior), allowing it to be accessed like a dictionary while maintaining a rich object model.

> ### **Attributes:**

agent: The Agent object that was interviewed. scenario: The Scenario object that was presented to the agent. model: The LanguageModel object that was used to generate responses. answer: Dictionary mapping question names to answer values. sub\_dicts: Organized sub-dictionaries for different data types. combined\_dict: Flattened dictionary combining all sub-dictionaries. problem\_keys: List of keys that have naming conflicts.

> ### **Note:**

Results are typically created by the Jobs system when running interviews and collected into a Results collection for analysis. You rarely need to create Result objects manually.

> ### **Examples:**

```python theme={null}
>>> result = Result.example()
>>> result['answer']['how_feeling']
'OK'
```

> #### *class* ClassOrInstanceMethod(*func*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Bases: `object`

Descriptor that allows a method to be called as both a class method and an instance method.

> #### **init**(*func*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### **init**(*agent: [Agent](/en/latest/agents#result-class)*, *scenario: [Scenario](/en/latest/scenarios#scenario-class "edsl.scenarios.Scenario")*, *model: [LanguageModel](/en/latest/language_models#languagemodel-class "edsl.language_models.LanguageModel")*, *iteration: int*, *answer: dict\[QuestionName, AnswerValue]*, *prompt: dict\[QuestionName, str] = None*, *raw\_model\_response: dict | None = None*, *survey: 'Survey' | None = None*, *question\_to\_attributes: dict\[QuestionName, Any] | None = None*, *generated\_tokens: dict | None = None*, *comments\_dict: dict | None = None*, *reasoning\_summaries\_dict: dict | None = None*, *cache\_used\_dict: dict\[QuestionName, bool] | None = None*, *indices: dict | None = None*, *cache\_keys: dict\[QuestionName, str] | None = None*, *validated\_dict: dict\[QuestionName, bool] | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Initialize a Result object.

> **Args:**

agent: The Agent object that was interviewed. scenario: The Scenario object that was presented. model: The LanguageModel object that generated responses. iteration: The iteration number for this result. answer: Dictionary mapping question names to answer values. prompt: Dictionary of prompts used for each question. Defaults to None. raw\_model\_response: The raw response from the language model. Defaults to None. survey: The Survey object containing the questions. Defaults to None. question\_to\_attributes: Dictionary of question attributes. Defaults to None. generated\_tokens: Dictionary of token usage statistics. Defaults to None. comments\_dict: Dictionary of comments for each question. Defaults to None. reasoning\_summaries\_dict: Dictionary of reasoning summaries. Defaults to None. cache\_used\_dict: Dictionary indicating cache usage for each question. Defaults to None. indices: Dictionary of indices for data organization. Defaults to None. cache\_keys: Dictionary of cache keys for each question. Defaults to None. validated\_dict: Dictionary indicating validation status for each question. Defaults to None.

> #### *property* agent\*: [Agent](/en/latest/agents#result-class)\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return the Agent object.

> #### *property* answer\*: dict\[str, Any]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return the answers.

> #### apply\_command(*command\_name*, *kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### by\_question\_data(*flatten\_nested\_dicts: bool = False*, *separator: str = '\_'*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Organize result data by question with optional flattening of nested dictionaries.

This method reorganizes the result data structure to be organized by question name, making it easier to analyze answers and related metadata on a per-question basis.

**Args:**

flatten\_nested\_dicts: Whether to flatten nested dictionaries using the separator.

Defaults to False.

separator: The separator to use when flattening nested dictionaries.

Defaults to “\_”.

**Returns:**

A dictionary organized by question name, with each question containing its associated data (answer, prompt, metadata, etc.).

> #### check\_expression(*expression: str*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Check if an expression references a problematic key.

**Args:**

expression: The expression string to check for problematic keys.

**Raises:**

ResultsColumnNotFoundError: If the expression contains a problematic key

that should use the full qualified name instead.

> #### clear() → None.  Remove all items from D.[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### clipboard()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Copy this object’s representation to the system clipboard.

This method first checks if the object has a custom clipboard\_data() method. If it does, it uses that method’s output. Otherwise, it serializes the object to a dictionary (without version info) and copies it to the system clipboard as JSON text.

**Returns:**

None, but prints a confirmation message

> #### code()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a string of code that can be used to recreate the Result object.

**Raises:**

ResultsError: This method is not implemented for Result objects.

> #### *property* combined\_dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> ### copy() → [Result](/en/latest/#edsl.results.Result "edsl.results.result.Result")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a copy of the Result object.

> #### **Returns:**

A new Result object that is a copy of this one.

> #### **Examples:**

```python theme={null}
>>> r = Result.example()
>>> r2 = r.copy()
>>> r == r2
True
>>> id(r) == id(r2)
False
```

> #### create\_download\_link()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Generate a downloadable link for this object.

Creates a temporary file containing the serialized object and generates a download link that can be shared with others.

**Returns:**

str: A URL that can be used to download the object

> #### *classmethod* delete(*url\_or\_uuid: str | UUID*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Delete the object from coop.

> #### display\_dict()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create a flattened dictionary representation for display purposes.

This method creates a flattened view of nested structures using colon notation in keys to represent hierarchy.

**Returns:**

dict: A flattened dictionary suitable for display

> #### display\_transcript(*show\_options: bool = True*, *show\_agent\_info: bool = True*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Display a rich-formatted chat transcript of the interview.

This method creates a ChatTranscript object and displays the conversation between questions and agent responses in a beautiful, chat-like format using the Rich library.

**Args:**

show\_options: Whether to display question options if available. Defaults to True. show\_agent\_info: Whether to show agent information at the top. Defaults to True.

> #### duplicate(*add\_edsl\_version=False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create and return a deep copy of the object.

**Args:**

add\_edsl\_version: Whether to include EDSL version information in the duplicated object

**Returns:**

A new instance of the same class with identical properties

> #### *classmethod* example() → [Result](/en/latest/#edsl.results.Result "edsl.results.result.Result")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return an example Result object.

**Returns:**

A sample Result object for testing and demonstration purposes.

> ### **Examples:**

```python theme={null}
>>> result = Result.example()
>>> type(result)
<class 'edsl.results.result.Result'>
>>> isinstance(result, Result)
True
```

> #### *classmethod* from\_dict(*data: dict*) → [Result](/en/latest/#edsl.results.Result "edsl.results.result.Result")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a Result object from a dictionary representation.

**Args:**

json\_dict: Dictionary containing Result data.

**Returns:**

A new Result object created from the dictionary data.

> #### *classmethod* from\_interview(*interview*) → [Result](/en/latest/#edsl.results.Result "edsl.results.result.Result")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a Result object from an interview dictionary.

This method ensures no reference to the original interview is maintained, creating a clean Result object from the interview data.

**Args:**

interview: An interview dictionary containing the raw interview data.

**Returns:**

A new Result object created from the interview data.

> #### *classmethod* from\_yaml(*yaml\_str: str | None = None*, *filename: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create an instance from YAML data.

Deserializes a YAML string or file into a new instance of the class.

**Args:**

yaml\_str: YAML string containing object data filename: Path to a YAML file containing object data

**Returns:**

A new instance of the class populated with the deserialized data

**Raises:**

BaseValueError: If neither yaml\_str nor filename is provided

> #### *classmethod* fromkeys(*iterable*, *value=None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### get(*k*\[, *d*]) → D\[k] if k in D, else d.  d defaults to None.[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### get\_description() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the description of this object.

> #### get\_hash() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a string hash representation of this object based on its content.

**Returns:**

str: A string representation of the hash value

> #### get\_uuid() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the UUID of this object from the Expected Parrot cloud service based on its hash.

This method calculates the hash of the object and queries the cloud service to find if there’s an uploaded version with the same content. If found, it returns the UUID of that object.

**Returns:**

str: The UUID of the object in the cloud service if found

**Raises:**

CoopServerResponseError: If the object is not found or there’s an error

communicating with the server

> #### get\_value(*data\_type: str*, *key: str*) → Any[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return the value for a given data type and key.

This method provides a consistent way to access values across different sub-dictionaries in the Result object. It’s particularly useful when you need to programmatically access values without knowing which data type a particular key belongs to.

> **Args:**

data\_type: The category of data to retrieve from. Valid options include:

“agent”, “scenario”, “model”, “answer”, “prompt”, “comment”, “generated\_tokens”, “raw\_model\_response”, “question\_text”, “question\_options”, “question\_type”, “cache\_used”, “cache\_keys”.

key: The specific attribute name within that data type.

> **Returns:**

The value associated with the key in the specified data type.

> **Examples:**

```python theme={null}
>>> r = Result.example()
>>> r.get_value("answer", "how_feeling")
'OK'
>>> r.get_value("scenario", "period")
'morning'
```

> #### *classmethod* help()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Display the class documentation string.

This is a convenience method to quickly access the docstring of the class.

**Returns:**

None, but prints the class docstring to stdout

> #### inspect()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create an interactive inspector widget for this object.

This method uses the InspectorWidget registry system to find the appropriate inspector widget class for this object’s type and returns an instance of it.

**Returns:**

InspectorWidget subclass instance: Interactive widget for inspecting this object

**Raises:**

KeyError: If no inspector widget is registered for this object’s class ImportError: If the widgets module cannot be imported

> #### items() → a set-like object providing a view on D's items[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### json()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a formatted JSON representation of this object.

**Returns:**

DisplayJSON: A displayable JSON representation

> #### *property* key\_to\_data\_type[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### keys()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the key names in the object’s dictionary representation.

This method returns all the keys in the serialized form of the object, excluding metadata keys like version information.

**Returns:**

list: A list of key names

> #### *classmethod* list(*visibility: Literal\['private', 'public', 'unlisted'] | List\[Literal\['private', 'public', 'unlisted']] | None = None*, *job\_status: Literal\['queued', 'running', 'completed', 'failed', 'cancelled', 'cancelling', 'partial\_failed'] | List\[Literal\['queued', 'running', 'completed', 'failed', 'cancelled', 'cancelling', 'partial\_failed']] | None = None*, *search\_query: str | None = None*, *page: int = 1*, *page\_size: int = 10*, *sort\_ascending: bool = False*) → CoopObjects[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

List objects from coop.

Notes: - The visibility parameter is not supported for remote inference jobs. - The job\_status parameter is not supported for objects. - search\_query only works with the description field. - If sort\_ascending is False, then the most recently created objects are returned first.

> #### *classmethod* load(*filename*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Load the object from a JSON file (compressed or uncompressed).

This method deserializes an object from a file, automatically detecting whether the file is compressed with gzip or not.

**Args:**

filename: Path to the file to load

**Returns:**

An instance of the class populated with data from the file

**Raises:**

Various exceptions may be raised if the file doesn’t exist or contains invalid data

> #### *property* model\*: [LanguageModel](/en/latest/language_models#languagemodel-class "edsl.language_models.LanguageModel")\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return the LanguageModel object.

> #### *classmethod* old\_pull(*url\_or\_uuid: str | UUID | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Pull the object from coop.

**Args:**

url\_or\_uuid: Either a UUID string or a URL pointing to the object

> #### *static* open\_compressed\_file(*filename*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Read and parse a compressed JSON file.

**Args:**

filename: Path to a gzipped JSON file

**Returns:**

dict: The parsed JSON content

> #### *static* open\_regular\_file(*filename*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Read and parse an uncompressed JSON file.

**Args:**

filename: Path to a JSON file

**Returns:**

dict: The parsed JSON content

> #### patch(\*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### *classmethod* patch\_cls(*url\_or\_uuid: str | UUID*, *description: str | None = None*, *value: Any | None = None*, *visibility: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Patch an uploaded object’s attributes (class method version). - description changes the description of the object on Expected Parrot - value changes the value of the object on Expected Parrot. **has to be an EDSL object** - visibility changes the visibility of the object on Expected Parrot

> #### *property* polly\_commands[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Auto-generate commands list from methods decorated with @polly\_command

> #### pop(*k*\[, *d*]) → v, remove specified key and return the corresponding value.[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

If key is not found, d is returned if given, otherwise KeyError is raised.

> #### popitem() → (k, v), remove and return some (key, value) pair[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

as a 2-tuple; but raise KeyError if D is empty.

> #### print(*format='rich'*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Print a formatted table representation of this object.

**Args:**

format: The output format (currently only ‘rich’ is supported)

**Returns:**

None, but prints a formatted table to the console

> #### *property* problem\_keys[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### *classmethod* pull(*url\_or\_uuid: str | UUID | None = None*, *expected\_parrot\_url: str | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a signed URL for directly downloading an object from Google Cloud Storage.

This method provides a more efficient way to download objects compared to the old pull() method, especially for large files, by generating a direct signed URL to the storage bucket.

**Args:**

url\_or\_uuid (Union\[str, UUID], optional): Identifier for the object to retrieve.

Can be one of: - UUID string (e.g., “123e4567-e89b-12d3-a456-426614174000”) - Full URL (e.g., “[https://expectedparrot.com/content/123e4567](https://expectedparrot.com/content/123e4567)…”) - Alias URL (e.g., “[https://expectedparrot.com/content/username/my-survey](https://expectedparrot.com/content/username/my-survey)”)

expected\_parrot\_url (str, optional): Optional custom URL for the coop service

**Returns:**

dict: A response containing the signed\_url for direct download

> **Example:**

```python theme={null}
>>> response = SurveyClass.pull("123e4567-e89b-12d3-a456-426614174000")
>>> response = SurveyClass.pull("https://expectedparrot.com/content/username/my-survey")
>>> print(f"Download URL: {response['signed_url']}")
>>> # Use the signed_url to download the object directly
```

> #### push(*description: str | None = None*, *alias: str | None = None*, *visibility: str | None = 'unlisted'*, *expected\_parrot\_url: str | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a signed URL for directly uploading an object to Google Cloud Storage.

This method provides a more efficient way to upload objects compared to the push() method, especially for large files, by generating a direct signed URL to the storage bucket.

**Args:**

expected\_parrot\_url (str, optional): Optional custom URL for the coop service

**Returns:**

dict: A response containing the signed\_url for direct upload and optionally a job\_id

> **Example:**

```python theme={null}
>>> from edsl.surveys import Survey
>>> survey = Survey(...)
>>> response = survey.push()
>>> print(f"Upload URL: {response['signed_url']}")
>>> # Use the signed_url to upload the object directly
```

> #### *property* rb[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### save(*filename: str | None = None*, *compress: bool = True*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Save the object to a file as JSON with optional compression.

Serializes the object to JSON and writes it to the specified file. By default, the file will be compressed using gzip. File extensions are handled automatically.

> **Args:**

filename: Path where the file should be saved compress: If True, compress the file using gzip (default: True)

> **Returns:**

None

> **Examples:**

```python theme={null}
>>> obj.save("my_object.json.gz")  # Compressed
>>> obj.save("my_object.json", compress=False)  # Uncompressed
```

> #### *property* scenario\*: [Scenario](/en/latest/scenarios#scenario-class "edsl.scenarios.Scenario")\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return the Scenario object.

> #### score(*scoring\_function: Callable*) → int | float[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Score the result using a passed-in scoring function.

> **Args:**

scoring\_function: A callable that takes parameters from the Result’s combined\_dict

and returns a numeric score.

> **Returns:**

The numeric score returned by the scoring function.

> **Raises:**

ResultsError: If a required parameter for the scoring function is not found

in the Result object.

> **Examples:**

```python theme={null}
>>> def f(status): return 1 if status == 'Joyful' else 0
>>> result = Result.example()
>>> result.score(f)
1
```

> #### score\_with\_answer\_key(*answer\_key: dict*) → dict\[str, int][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Score the result against a reference answer key.

This method evaluates the correctness of answers by comparing them to a provided answer key. It returns a dictionary with counts of correct, incorrect, and missing answers.

The answer key can contain either single values or lists of acceptable values. If a list is provided, the answer is considered correct if it matches any value in the list.

> **Args:**

answer\_key: A dictionary mapping question names to expected answers.

Values can be single items or lists of acceptable answers.

> **Returns:**

A dictionary with keys ‘correct’, ‘incorrect’, and ‘missing’, indicating the counts of each answer type.

> **Examples:**

```python theme={null}
>>> result = Result.example()
>>> result.answer
{'how_feeling': 'OK', 'how_feeling_yesterday': 'Great'}
```

```python theme={null}
>>> # Using exact match answer key
>>> answer_key = {'how_feeling': 'OK', 'how_feeling_yesterday': 'Great'}
>>> result.score_with_answer_key(answer_key)
{'correct': 2, 'incorrect': 0, 'missing': 0}
```

```python theme={null}
>>> # Using answer key with multiple acceptable answers
>>> answer_key = {'how_feeling': 'OK', 'how_feeling_yesterday': ['Great', 'Good']}
>>> result.score_with_answer_key(answer_key)
{'correct': 2, 'incorrect': 0, 'missing': 0}
```

> #### *classmethod* search(*query*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Search for objects on coop.

> #### setdefault(*k*\[, *d*]) → D.get(k,d), also set D\[k]=d if k not in D[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### show\_methods(*show\_docstrings=True*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Display all public methods available on this object.

This utility method helps explore the capabilities of an object by listing all its public methods and optionally their documentation.

**Args:**

show\_docstrings: If True, print method names with docstrings;

if False, return the list of method names

**Returns:**

None or list: If show\_docstrings is True, prints methods and returns None.

If show\_docstrings is False, returns a list of method names.

> #### store(*d: dict*, *key\_name: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Store this object in a dictionary with an optional key.

**Args:**

d: The dictionary in which to store the object key\_name: Optional key to use (defaults to the length of the dictionary)

**Returns:**

None

> #### *property* sub\_dicts[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### to\_dataset(*flatten\_nested\_dicts: bool = False*, *separator: str = '\_'*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the result to a dataset format.

This method transforms the result data into a Dataset object suitable for analysis and data manipulation.

**Args:**

**flatten\_nested\_dicts: Whether to flatten nested dictionaries using the separator.**

Defaults to False.

**separator: The separator to use when flattening nested dictionaries.**

Defaults to “\_”.

**Returns:**

A Dataset object containing the result data organized for analysis.

> #### to\_dict(*add\_edsl\_version: bool = True*, *include\_cache\_info: bool = False*, *full\_dict: bool = False*) → dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a dictionary representation of the Result object.

**Args:**

**add\_edsl\_version: Whether to include EDSL version information in the output.**

Defaults to True.

**include\_cache\_info: Whether to include cache information in the output.**

Defaults to False.

**Returns:**

A dictionary representation of the Result object containing all relevant data.

> **Examples:**

```python theme={null}
>>> r = Result.example()
>>> data = r.to_dict()
>>> data['scenario']['period']
'morning'
```

> #### to\_json()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Serialize this object to a JSON string.

**Returns:**

str: A JSON string representation of the object

> #### to\_yaml(*add\_edsl\_version=False*, *filename: str = None*) → str | None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the object to YAML format.

Serializes the object to YAML format and optionally writes it to a file.

**Args:**

add\_edsl\_version: Whether to include EDSL version information filename: If provided, write the YAML to this file path

**Returns:**

str: The YAML string representation if no filename is provided None: If written to file

> #### transcript(*format: str = 'simple'*) → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return the questions and answers in a human-readable transcript.

> **Args:**

format: The format for the transcript. Either ‘simple’ or ‘rich’.

‘simple’ (default) returns plain-text format with questions, options, and answers separated by blank lines. ‘rich’ uses the rich library to wrap each Q\&A block in a Panel with colors and formatting.

> **Returns:**

A formatted transcript string of the interview.

> **Raises:**

ImportError: If ‘rich’ format is requested but the rich library is not installed.

> **Examples:**

```python theme={null}
>>> result = Result.example()
>>> transcript = result.transcript(format="simple")
>>> print(transcript)
QUESTION: How are you this `{{ period }}`?
OPTIONS: Good / Great / OK / Terrible
ANSWER: OK

QUESTION: How were you feeling yesterday `{{ period }}`?
OPTIONS: Good / Great / OK / Terrible
ANSWER: Great
```

> #### *property* transformer[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the ResultTransformer instance for this Result.

> #### update(\[*E*, ]\*\**F*) → None.  Update D from mapping/iterable E and F.[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

If E present and has a .keys() method, does: for k in E: D\[k] = E\[k] If E present and lacks .keys() method, does: for (k, v) in E: D\[k] = v In either case, this is followed by: for k, v in F.items(): D\[k] = v

> #### values()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the values in the object’s dictionary representation.

**Returns:**

set: A set containing all the values in the object

> #### view()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Display an interactive visualization of this object.

**Returns:**

The result of the dataset’s view method

> #### yaml()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a formatted YAML representation of this object.

**Returns:**

DisplayYAML: A displayable YAML representation

## Results class

> ### *class* edsl.results.Results(*survey: Optional\['Survey'] = None*, *data: Optional\[list\['Result']] = None*, *name: Optional\[str] = None*, *created\_columns: Optional\[list\[str]] = None*, *cache: Optional\['Cache'] = None*, *job\_uuid: Optional\[str] = None*, *total\_results: Optional\[int] = None*, *task\_history: Optional\['TaskHistory'] = None*, *sort\_by\_iteration: bool = False*, *data\_class: Optional\[type] = \<class 'list'>*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Bases: `MutableSequence`, `ResultsOperationsMixin`, [`Base`](/en/latest/base#base-class "edsl.base.base_class.Base")

A collection of Result objects with powerful data analysis capabilities.

The Results class is the primary container for working with data from EDSL surveys. It provides a rich set of methods for data analysis, transformation, and visualization inspired by data manipulation libraries like dplyr and pandas. The Results class implements a functional, fluent interface for data manipulation where each method returns a new Results object, allowing method chaining.

Attributes:

survey: The Survey object containing the questions used to generate results. data: A list of Result objects containing the responses. created\_columns: A list of column names created through transformations. cache: A Cache object for storing model responses. completed: Whether the Results object is ready for use. task\_history: A TaskHistory object containing information about the tasks. known\_data\_types: List of valid data type strings for accessing data.

Key features:

* List-like interface for accessing individual Result objects
* Selection of specific data columns with select()
* Filtering results with boolean expressions using filter()
* Creating new derived columns with mutate()
* Recoding values with recode() and answer\_truncate()
* Sorting results with order\_by()
* Converting to other formats (dataset, table, pandas DataFrame)
* Serialization for storage and retrieval
* Support for remote execution and result retrieval

Results objects have a hierarchical structure with the following components:

1. Each Results object contains multiple Result objects
2. Each Result object contains data organized by type (agent, scenario, model, answer, etc.)
3. Each data type contains multiple attributes (e.g., “how\_feeling” in the answer type)

You can access data in a Results object using dot notation (answer.how\_feeling) or using just the attribute name if it’s not ambiguous (how\_feeling).

The Results class also tracks “created columns” - new derived values that aren’t part of the original data but were created through transformations.

> #### **Examples:**

```python theme={null}
>>> # Create a simple Results object from example data
>>> r = Results.example()
>>> len(r) > 0  # Contains Result objects
True
>>> # Filter and transform data
>>> filtered = r.filter("how_feeling == 'Great'")
>>> # Access hierarchical data
>>> 'agent' in r.known_data_types
True
```

> #### *class* ClassOrInstanceMethod(*func*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Bases: `object`

Descriptor that allows a method to be called as both a class method and an instance method.

> ##### **init**(*func*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### **init**(*survey: Optional\['Survey'] = None*, *data: Optional\[list\['Result']] = None*, *name: Optional\[str] = None*, *created\_columns: Optional\[list\[str]] = None*, *cache: Optional\['Cache'] = None*, *job\_uuid: Optional\[str] = None*, *total\_results: Optional\[int] = None*, *task\_history: Optional\['TaskHistory'] = None*, *sort\_by\_iteration: bool = False*, *data\_class: Optional\[type] = \<class 'list'>*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Instantiate a Results object with a survey and a list of Result objects.

**Args:**

survey: A Survey object containing the questions used to generate results. data: A list of Result objects containing the responses. created\_columns: A list of column names created through transformations. cache: A Cache object for storing model responses. job\_uuid: A string representing the job UUID. total\_results: An integer representing the total number of results. task\_history: A TaskHistory object containing information about the tasks. sort\_by\_iteration: Whether to sort data by iteration before initializing. data\_class: The class to use for the data container (default: list).

> #### add\_task\_history\_entry(*interview: Interview*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### agent\_answers\_by\_question(*agent\_key\_fields: List\[str] | None = None*, *separator: str = ','*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Returns a dictionary of agent answers.

The keys are the agent names and the values are the answers.

```python theme={null}
>>> result = Results.example().agent_answers_by_question()
>>> sorted(result['how_feeling'].values())
['Great', 'OK', 'OK', 'Terrible']
>>> sorted(result['how_feeling_yesterday'].values())
['Good', 'Great', 'OK', 'Terrible']
```

> #### *property* agent\_keys\*: list\[str]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a set of all of the keys that are in the Agent data.

Example:

```python theme={null}
>>> r = Results.example()
>>> r.agent_keys
['agent_index', 'agent_instruction', 'agent_name', 'status']
```

> #### *property* agents\*: [AgentList](/en/latest/agents#agentlist-class "edsl.agents.AgentList")\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a list of all of the agents in the Results.

Example:

```python theme={null}
>>> r = Results.example()
>>> r.agents
AgentList([Agent(traits = {'status': 'Joyful'}), Agent(traits = {'status': 'Joyful'}), Agent(traits = {'status': 'Sad'}), Agent(traits = {'status': 'Sad'})])
```

> #### *property* all\_keys\*: list\[str]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a set of all of the keys that are in the Results.

Example:

```python theme={null}
>>> r = Results.example()
>>> r.all_keys
['agent_index', ...]
```

> #### *property* answer\_keys\*: dict\[str, str]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a mapping of answer keys to question text.

Example:

```python theme={null}
>>> r = Results.example()
>>> r.answer_keys
{'how_feeling': 'How are you this `{{ period }}`?', 'how_feeling_yesterday': 'How were you feeling yesterday `{{ period }}`?'}
```

> #### apply\_command(*command\_name*, *kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### bucket\_by(\**columns: str*) → dict\[tuple, list\[[Result](/en/latest/#edsl.results.Result "edsl.results.result.Result")]][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Group Result objects into buckets keyed by the specified column values.

Each key in the returned dictionary is a tuple containing the values of the requested columns (in the same order as supplied). The associated value is a list of `Result` instances whose values match that key.

> **Args:**

[\*](/en/latest/#id2)columns: Names of the columns to group by. Column identifiers

follow the same rules used by [`select()`](/en/latest/#edsl.results.Results.select "edsl.results.Results.select") – they can be specified either as fully-qualified names (e.g. `"agent.status"`) or by bare attribute name when unambiguous.

> **Returns:**

dict\[tuple, list\[Result]]: Mapping from value tuples to lists of `Result` objects.

> **Raises:**

**ResultsError: If no columns are provided or an invalid column name is**

supplied.

> **Examples:**

```python theme={null}
>>> r = Results.example()
>>> buckets = r.bucket_by('how_feeling')
>>> list(buckets.keys())
[('OK',), ('Great',), ('Terrible',)]
>>> all(isinstance(v, list) for v in buckets.values())
True
```

> #### chart()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create a chart from the results.

> #### clipboard()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Copy this object’s representation to the system clipboard.

This method first checks if the object has a custom clipboard\_data() method. If it does, it uses that method’s output. Otherwise, it serializes the object to a dictionary (without version info) and copies it to the system clipboard as JSON text.

**Returns:**

None, but prints a confirmation message

> #### clipboard\_data() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return TSV representation of this object for clipboard operations.

This method is called by the clipboard() method in the base class to provide a custom format for copying objects to the system clipboard.

**Returns:**

str: Tab-separated values representation of the object

> #### code()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Method for generating code representations.

> **Raises:**

ResultsError: This method is not implemented for Results objects.

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> try:
...     r.code()
... except ResultsError as e:
...     str(e).startswith("The code() method is not implemented")
True
```

> #### *property* columns\*: list\[str]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a list of all of the columns that are in the Results.

Example:

```python theme={null}
>>> r = Results.example()
>>> r.columns
['agent.agent_index', ...]
```

> #### compare(*other\_results: [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Compare two Results objects and return the differences.

> #### compute\_job\_cost(*include\_cached\_responses\_in\_cost: bool = False*) → float[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Compute the cost of a completed job in USD.

This method delegates to the JobCostCalculator class to calculate the total cost of all model responses in the results. By default, it only counts the cost of responses that were not cached.

> **Args:**

**include\_cached\_responses\_in\_cost: Whether to include the cost of cached**

responses in the total. Defaults to False.

> **Returns:**

float: The total cost in USD.

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.compute_job_cost()
0.0
```

> #### create\_download\_link()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Generate a downloadable link for this object.

Creates a temporary file containing the serialized object and generates a download link that can be shared with others.

**Returns:**

str: A URL that can be used to download the object

> #### *classmethod* delete(*url\_or\_uuid: str | UUID*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Delete the object from coop.

> #### display\_dict()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create a flattened dictionary representation for display purposes.

This method creates a flattened view of nested structures using colon notation in keys to represent hierarchy.

**Returns:**

dict: A flattened dictionary suitable for display

> #### duplicate(*add\_edsl\_version=False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create and return a deep copy of the object.

**Args:**

add\_edsl\_version: Whether to include EDSL version information in the duplicated object

**Returns:**

A new instance of the same class with identical properties

> #### *classmethod* example(*randomize: bool = False*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return an example Results object.

Example usage:

```python theme={null}
>>> r = Results.example()
```

**Parameters:**

**randomize** – if True, randomizes agent and scenario combinations

> #### extend\_sorted(*other*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Extend the Results list with items from another iterable.

This method preserves ordering based on ‘order’ attribute if present, otherwise falls back to ‘iteration’ attribute.

> #### fetch(*polling\_interval: float | int = 1.0*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Poll the server for job completion and update this Results instance.

This method delegates to the ResultsRemoteFetcher class to handle the polling and fetching operation.

**Args:**

polling\_interval: Number of seconds to wait between polling attempts (default: 1.0)

**Returns:**

Results: The updated Results instance

**Raises:**

ResultsError: If no job info is available or if there’s an error during fetch.

> #### fetch\_remote(*job\_info: Any*) → bool[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Fetch remote Results object and update this instance with the data.

This method delegates to the ResultsRemoteFetcher class to handle the remote fetching operation.

**Args:**

job\_info: RemoteJobInfo object containing the job\_uuid and other remote job details

**Returns:**

bool: True if the fetch was successful, False if the job is not yet completed.

**Raises:**

ResultsError: If there’s an error during the fetch process.

> #### filter(*expression: str*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Filter results based on a boolean expression.

This method delegates to the ResultsFilter class to evaluate a boolean expression against each Result object in the collection and returns a new Results object containing only those that match.

> **Args:**

expression: A string containing a Python expression that evaluates to a boolean.

The expression is applied to each Result object individually. Can be a multi-line string for better readability. Supports template-style syntax with `{{ field }}` notation.

> **Returns:**

A new Results object containing only the Result objects that satisfy the expression.

> **Raises:**

ResultsFilterError: If the expression is invalid or uses improper syntax

(like using ‘=’ instead of ‘==’).

> **Examples:**

```python theme={null}
>>> r = Results.example()
```

```python theme={null}
>>> # Simple equality filter
>>> r.filter("how_feeling == 'Great'").select('how_feeling')
Dataset([{'answer.how_feeling': ['Great']}])
```

```python theme={null}
>>> # Using OR condition
>>> r.filter("how_feeling == 'Great' or how_feeling == 'Terrible'").select('how_feeling')
Dataset([{'answer.how_feeling': ['Great', 'Terrible']}])
```

```python theme={null}
>>> # Filter on agent properties
>>> r.filter("agent.status == 'Joyful'").select('agent.status')
Dataset([{'agent.status': ['Joyful', 'Joyful']}])
```

> #### first() → [Result](/en/latest/#edsl.results.Result "edsl.results.result.Result")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return the first observation in the results.

Example:

```python theme={null}
>>> r = Results.example()
>>> r.first()
Result(agent...
```

> #### flatten(*field: str*, *keep\_original: bool = False*) → [Dataset](/en/latest/dataset#dataset-methods)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Expand a field containing dictionaries into separate fields.

This method takes a field that contains a list of dictionaries and expands it into multiple fields, one for each key in the dictionaries. This is useful when working with nested data structures or results from extraction operations.

> **Parameters:**

field: The field containing dictionaries to flatten keep\_original: Whether to retain the original field in the result

> **Returns:**

A new Dataset with the dictionary keys expanded into separate fields

> **Notes:**

* Each key in the dictionaries becomes a new field with name pattern “.”
* All dictionaries in the field must have compatible structures
* If a dictionary is missing a key, the corresponding value will be None
* Non-dictionary values in the field will cause a warning

> **Examples:**

```python theme={null}
>>> from edsl.dataset import Dataset
```

```python theme={null}
# Basic flattening of nested dictionaries >>> Dataset([{‘a’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}]).flatten(‘a’) Dataset([{‘c’: [5]}, {‘a.a’: [1]}, {‘a.b’: [2]}])

# Works with prefixed fields too >>> Dataset([{‘answer.example’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}]).flatten(‘answer.example’) Dataset([{‘c’: [5]}, {‘answer.example.a’: [1]}, {‘answer.example.b’: [2]}])

# Keep the original field if needed >>> d = Dataset([{‘a’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}]) >>> d.flatten(‘a’, keep_original=True) Dataset([{‘a’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}, {‘a.a’: [1]}, {‘a.b’: [2]}])

# Can also use unambiguous unprefixed field name >>> result = Dataset([{‘answer.pros_cons’: [{‘pros’: [‘Safety’], ‘cons’: [‘Cost’]}]}]).flatten(‘pros_cons’) >>> sorted(result.keys()) == [‘answer.pros_cons.cons’, ‘answer.pros_cons.pros’] True >>> sorted(result.to_dicts()[0].items()) == sorted({‘cons’: [‘Cost’], ‘pros’: [‘Safety’]}.items()) True

```

> #### *classmethod* from\_dict(*data: dict\[str, Any]*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert a dictionary to a Results object.

This method delegates to the ResultsSerializer class to handle the conversion of a dictionary representation back to a Results object.

> **Args:**

data: A dictionary representation of a Results object.

> **Returns:**

Results: A new Results object created from the dictionary data

> **Examples:**

```python theme={null}
>>> r = Results.example()
>>> d = r.to_dict()
>>> r2 = Results.from_dict(d)
>>> r == r2
True
```

> #### *classmethod* from\_disk(*filepath: str*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Load a Results object from a zip file.

This method delegates to the ResultsSerializer class to handle the disk deserialization.

This method: 1. Extracts the SQLite database file 2. Loads the metadata 3. Creates a new Results instance with the restored data

### > **Args:**

filepath: Path to the zip file containing the serialized Results

> **Returns:**

Results: A new Results instance with the restored data

> **Raises:**

ResultsError: If there’s an error during deserialization

> #### *classmethod* from\_job\_info(*job\_info: dict*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Instantiate a Results object from a job info dictionary.

This method creates a Results object in a not-ready state that will fetch its data from a remote source when methods are called on it.

> **Args:**

job\_info: Dictionary containing information about a remote job.

> **Returns:**

Results: A new Results instance with completed=False that will

fetch remote data when needed.

> **Examples:**

```python theme={null}
>>> # Create a job info dictionary
>>> job_info = {'job_uuid': '12345', 'creation_data': {'model': 'gpt-4'}}
>>> # Create a Results object from the job info
>>> results = Results.from_job_info(job_info)
>>> results.completed
False
>>> hasattr(results, 'job_info')
True
```

> #### *classmethod* from\_yaml(*yaml\_str: str | None = None*, *filename: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create an instance from YAML data.

Deserializes a YAML string or file into a new instance of the class.

**Args:**

yaml\_str: YAML string containing object data filename: Path to a YAML file containing object data

**Returns:**

A new instance of the class populated with the deserialized data

**Raises:**

BaseValueError: If neither yaml\_str nor filename is provided

> #### get\_answers(*question\_name: str*) → list[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the answers for a given question name.

> **Args:**

question\_name: The name of the question to fetch answers for.

> **Returns:**

list: A list of answers, one from each result in the data.

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> answers = r.get_answers('how_feeling')
>>> isinstance(answers, list)
True
>>> len(answers) == len(r)
True
```

> #### get\_description() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the description of this object.

> #### get\_hash() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a string hash representation of this object based on its content.

> **Returns:**

str: A string representation of the hash value

> #### get\_shelved\_result(*key: str*) → [Result](/en/latest/#edsl.results.Result "edsl.results.result.Result")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Retrieve a Result object from persistent storage.

This method delegates to the ResultsSerializer class to handle the retrieval operation.

**Args:**

key: The hash key of the Result to retrieve

**Returns:**

Result: The stored Result object

**Raises:**

ResultsError: If the key doesn’t exist or if there’s an error retrieving the Result

> #### get\_tabular\_data(*remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*) → Tuple\[List\[str], List\[List]][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Internal method to get tabular data in a standard format.

**Args:**

remove\_prefix: Whether to remove the prefix from column names pretty\_labels: Dictionary mapping original column names to pretty labels

**Returns:**

Tuple containing (header\_row, data\_rows)

> #### get\_uuid() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the UUID of this object from the Expected Parrot cloud service based on its hash.

This method calculates the hash of the object and queries the cloud service to find if there’s an uploaded version with the same content. If found, it returns the UUID of that object.

**Returns:**

str: The UUID of the object in the cloud service if found

**Raises:**

CoopServerResponseError: If the object is not found or there’s an error

communicating with the server

> #### ggplot2(*ggplot\_code: str*, *shape: str = 'wide'*, *sql: str | None = None*, *remove\_prefix: bool = True*, *debug: bool = False*, *height: float = 4*, *width: float = 6*, *factor\_orders: dict | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create visualizations using R’s ggplot2 library.

This method provides a bridge to R’s powerful ggplot2 visualization library, allowing you to create sophisticated plots directly from EDSL data structures.

> **Parameters:**

ggplot\_code: R code string containing ggplot2 commands shape: Data shape to use (“wide” or “long”) sql: Optional SQL query to transform data before visualization remove\_prefix: Whether to remove prefixes (like “answer.”) from column names debug: Whether to display debugging information height: Plot height in inches width: Plot width in inches factor\_orders: Dictionary mapping factor variables to their desired order

**Returns:**

A plot object that renders in Jupyter notebooks

> **Notes:**

* Requires R and the ggplot2 package to be installed
* Data is automatically converted to a format suitable for ggplot2
* The ggplot2 code should reference column names as they appear after any transformations from the shape and remove\_prefix parameters

> **Examples:**

```python theme={null}

>>> from edsl.results import Results
>>> r = Results.example()
>>> # The following would create a plot if R is installed (not shown in doctest):
>>> # r.ggplot2('''
>>> #     ggplot(df, aes(x=how_feeling)) +
>>> #     geom_bar() +
>>> #     labs(title="Distribution of Feelings")
>>> # ''')
```

> #### *property* has\_unfixed\_exceptions\*: bool\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### *property* hashes\*: set\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### *classmethod* help()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Display the class documentation string.

This is a convenience method to quickly access the docstring of the class.

**Returns:**

None, but prints the class docstring to stdout

> #### html(*filename: str | None = None*, *cta: str = 'Open in browser'*, *return\_link: bool = False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### initialize\_cache\_from\_results()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### insert\_from\_shelf() → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Move all shelved results into memory using insert\_sorted method.

This method delegates to the ResultsSerializer class to handle the shelf operations. Clears the shelf after successful insertion.

This method preserves the original order of results by using their ‘order’ attribute if available, which ensures consistent ordering even after serialization/deserialization.

**Raises:**

ResultsError: If there’s an error accessing or clearing the shelf

> #### insert\_sorted(*item: [Result](/en/latest/#edsl.results.Result "edsl.results.result.Result")*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Insert a Result object into the Results list while maintaining sort order.

Uses the ‘order’ attribute if present, otherwise falls back to ‘iteration’ attribute. Utilizes bisect for efficient insertion point finding.

> **Args:**

item: A Result object to insert

Examples:

```python theme={null}
>>> r = Results.example()
>>> new_result = r[0].copy()
>>> new_result.order = 1.5  # Insert between items
>>> r.insert_sorted(new_result)
```

> #### inspect()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create an interactive inspector widget for this object.

This method uses the InspectorWidget registry system to find the appropriate inspector widget class for this object’s type and returns an instance of it.

**Returns:**

InspectorWidget subclass instance: Interactive widget for inspecting this object

**Raises:**

KeyError: If no inspector widget is registered for this object’s class ImportError: If the widgets module cannot be imported

> #### json()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a formatted JSON representation of this object.

**Returns:**

DisplayJSON: A displayable JSON representation

> #### keys()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the key names in the object’s dictionary representation.

This method returns all the keys in the serialized form of the object, excluding metadata keys like version information.

**Returns:**

list: A list of key names

> #### *classmethod* list(*visibility: Literal\['private', 'public', 'unlisted'] | List\[Literal\['private', 'public', 'unlisted']] | None = None*, *job\_status: Literal\['queued', 'running', 'completed', 'failed', 'cancelled', 'cancelling', 'partial\_failed'] | List\[Literal\['queued', 'running', 'completed', 'failed', 'cancelled', 'cancelling', 'partial\_failed']] | None = None*, *search\_query: str | None = None*, *page: int = 1*, *page\_size: int = 10*, *sort\_ascending: bool = False*) → CoopObjects[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

List objects from coop.

Notes: - The visibility parameter is not supported for remote inference jobs. - The job\_status parameter is not supported for objects. - search\_query only works with the description field. - If sort\_ascending is False, then the most recently created objects are returned first.

> #### *classmethod* load(*filename*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Load the object from a JSON file (compressed or uncompressed).

This method deserializes an object from a file, automatically detecting whether the file is compressed with gzip or not.

**Args:**

filename: Path to the file to load

**Returns:**

An instance of the class populated with data from the file

**Raises:**

Various exceptions may be raised if the file doesn’t exist or contains invalid data

> #### make\_tabular(*remove\_prefix: bool*, *pretty\_labels: dict | None = None*) → tuple\[list, List\[list]][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Turn the results into a tabular format.

**Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').make_tabular(remove_prefix = True)
(['how_feeling'], [['OK'], ['Great'], ['Terrible'], ['OK']])
```

```python theme={null}
>>> r.select('how_feeling').make_tabular(remove_prefix = True, pretty_labels = {'how_feeling': "How are you feeling"})
(['How are you feeling'], [['OK'], ['Great'], ['Terrible'], ['OK']])
```

> #### *property* model\_keys\*: list\[str]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a set of all of the keys that are in the LanguageModel data.

```python theme={null}
>>> r = Results.example()
>>> r.model_keys
['canned_response', 'inference_service', 'model', 'model_index', 'temperature']
```

> #### *property* models\*: [ModelList](/en/latest/language_models#modellist-class "edsl.language_models.ModelList")\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a list of all of the models in the Results.

Example:

```python theme={null}
>>> r = Results.example()
>>> r.models[0]
Model(model_name = ...)
```

> #### mutate(*new\_var\_string: str*, *functions\_dict: dict | None = None*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Create a new column based on a computational expression.

This method delegates to the ResultsTransformer class to handle the mutation operation.

> **Args:**

**new\_var\_string: A string containing an assignment expression in the form**

“new\_column\_name = expression”. The expression can reference any existing column and use standard Python syntax.

**functions\_dict: Optional dictionary of custom functions that can be used in**

the expression. Keys are function names, values are function objects.

Returns:

A new Results object with the additional column.

Examples:

```python theme={null}
>>> r = Results.example()
```

```python theme={null}
>>> # Create a simple derived column
>>> r.mutate('how_feeling_x = how_feeling + "x"').select('how_feeling_x')
Dataset([{'answer.how_feeling_x': ['OKx', 'Greatx', 'Terriblex', 'OKx']}])
```

```python theme={null}
>>> # Create a binary indicator column
>>> r.mutate('is_great = 1 if how_feeling == "Great" else 0').select('is_great')
Dataset([{'answer.is_great': [0, 1, 0, 0]}])
```

```python theme={null}
>>> # Create a column with custom functions
>>> def sentiment(text):
...     return len(text) > 5
>>> r.mutate('is_long = sentiment(how_feeling)',
...          functions_dict={'sentiment': sentiment}).select('is_long')
Dataset([{'answer.is_long': [False, False, True, False]}])
```

> #### num\_observations()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return the number of observations in the dataset.

```python theme={null}
>>> from edsl.results import Results
>>> Results.example().num_observations()
4
```

> #### *classmethod* old\_pull(*url\_or\_uuid: str | UUID | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Pull the object from coop.

**Args:**

url\_or\_uuid: Either a UUID string or a URL pointing to the object

> #### *static* open\_compressed\_file(*filename*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Read and parse a compressed JSON file.

**Args:**

filename: Path to a gzipped JSON file

**Returns:**

dict: The parsed JSON content

> #### *static* open\_regular\_file(*filename*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Read and parse an uncompressed JSON file.

**Args:**

filename: Path to a JSON file

**Returns:**

dict: The parsed JSON content

> #### optimzie\_scenarios()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### order\_by(\**columns: str*, *reverse: bool = False*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Sort the results by one or more columns.

This method delegates to the ResultsTransformer class to handle the sorting operation.

> **Args:**

columns: One or more column names as strings. reverse: A boolean that determines whether to sort in reverse order.

> **Returns:**

Results: A new Results object with sorted data.

> **Examples:**

```python theme={null}
>>> r = Results.example()
>>> sorted_results = r.order_by('how_feeling')
>>> len(sorted_results) == len(r)
True
```

> #### patch(\*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### *classmethod* patch\_cls(*url\_or\_uuid: str | UUID*, *description: str | None = None*, *value: Any | None = None*, *visibility: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Patch an uploaded object’s attributes (class method version). - description changes the description of the object on Expected Parrot - value changes the value of the object on Expected Parrot. **has to be an EDSL object** - visibility changes the visibility of the object on Expected Parrot

> #### *property* polly\_commands[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Auto-generate commands list from methods decorated with @polly\_command

> #### print(*format='rich'*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Print a formatted table representation of this object.

**Args:**

format: The output format (currently only ‘rich’ is supported)

**Returns:**

None, but prints a formatted table to the console

> #### print\_long()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Print the results in a long format. >>> from edsl.results import Results >>> r = Results.example() >>> r.select(‘how\_feeling’).print\_long() answer.how\_feeling: OK answer.how\_feeling: Great answer.how\_feeling: Terrible answer.how\_feeling: OK

> #### *classmethod* pull(*url\_or\_uuid: str | UUID | None = None*, *expected\_parrot\_url: str | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a signed URL for directly downloading an object from Google Cloud Storage.

This method provides a more efficient way to download objects compared to the old pull() method, especially for large files, by generating a direct signed URL to the storage bucket.

> **Args:**

**url\_or\_uuid (Union\[str, UUID], optional): Identifier for the object to retrieve.**

Can be one of: - UUID string (e.g., “123e4567-e89b-12d3-a456-426614174000”) - Full URL (e.g., “[https://expectedparrot.com/content/123e4567](https://expectedparrot.com/content/123e4567)…”) - Alias URL (e.g., “[https://expectedparrot.com/content/username/my-survey](https://expectedparrot.com/content/username/my-survey)”)

expected\_parrot\_url (str, optional): Optional custom URL for the coop service

> **Returns:**

dict: A response containing the signed\_url for direct download

> **Example:**

```python theme={null}
>>> response = SurveyClass.pull("123e4567-e89b-12d3-a456-426614174000")
>>> response = SurveyClass.pull("https://expectedparrot.com/content/username/my-survey")
>>> print(f"Download URL: {response['signed_url']}")
>>> # Use the signed_url to download the object directly
```

> #### push(*description: str | None = None*, *alias: str | None = None*, *visibility: str | None = 'unlisted'*, *expected\_parrot\_url: str | None = None*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a signed URL for directly uploading an object to Google Cloud Storage.

This method provides a more efficient way to upload objects compared to the push() method, especially for large files, by generating a direct signed URL to the storage bucket.

> **Args:**

expected\_parrot\_url (str, optional): Optional custom URL for the coop service

> **Returns:**

dict: A response containing the signed\_url for direct upload and optionally a job\_id

> **Example:**

```python theme={null}
>>> from edsl.surveys import Survey
>>> survey = Survey(...)
>>> response = survey.push()
>>> print(f"Upload URL: {response['signed_url']}")
>>> # Use the signed_url to upload the object directly
```

> #### *property* question\_names\*: list\[str]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a list of all of the question names.

Example:

```python theme={null}
>>> r = Results.example()
>>> r.question_names
['how_feeling', 'how_feeling_yesterday']
```

> #### relevant\_cache(*cache: [Cache](/en/latest/data#cache-class "edsl.caching.Cache")*) → [Cache](/en/latest/data#cache-class "edsl.caching.Cache")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return the set of keys that are present in the dataset.

**Parameters:**

* **data\_type** – The data type to filter by.
* **remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from ..dataset import Dataset
>>> d = Dataset([{'a.b':[1,2,3,4]}])
>>> d.relevant_columns()
['a.b']
```

```python theme={null}
>>> d.relevant_columns(remove_prefix=True)
['b']
```

```python theme={null}
>>> d = Dataset([{'a':[1,2,3,4]}, {'b':[5,6,7,8]}])
>>> d.relevant_columns()
['a', 'b']
```

```python theme={null}
>>> from edsl.results import Results; Results.example().select('how_feeling', 'how_feeling_yesterday').relevant_columns()
['answer.how_feeling', 'answer.how_feeling_yesterday']
```

```python theme={null}
>>> from edsl.results import Results
>>> sorted(Results.example().select().relevant_columns(data_type = "model"))
['model.canned_response', 'model.inference_service', 'model.model', 'model.model_index', 'model.temperature']
```

```python theme={null}
>>> # Testing relevant_columns with invalid data_type raises DatasetValueError - tested in unit tests
```

> #### remove\_prefix()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Returns a new Dataset with the prefix removed from all column names.

The prefix is defined as everything before the first dot (.) in the column name. If removing prefixes would result in duplicate column names, an exception is raised.

> **Returns:**

Dataset: A new Dataset with prefixes removed from column names

> **Raises:**

ValueError: If removing prefixes would result in duplicate column names

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling', 'how_feeling_yesterday').relevant_columns()
['answer.how_feeling', 'answer.how_feeling_yesterday']
>>> r.select('how_feeling', 'how_feeling_yesterday').remove_prefix().relevant_columns()
['how_feeling', 'how_feeling_yesterday']
```

```python theme={null}
>>> from edsl.dataset import Dataset
>>> d = Dataset([{'a.x': [1, 2, 3]}, {'b.x': [4, 5, 6]}])
>>> # d.remove_prefix()
```

\# Testing remove\_prefix with duplicate column names raises DatasetValueError - tested in unit tests

> #### rename(*old\_name: str*, *new\_name: str*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Rename an answer column in a Results object.

This method delegates to the ResultsTransformer class to handle the renaming operation.

> **Args:**

old\_name: The current name of the column to rename new\_name: The new name for the column

> **Returns:**

Results: A new Results object with the column renamed

> **Examples:**

```python theme={null}
>>> s = Results.example()
>>> s.rename('how_feeling', 'how_feeling_new').select('how_feeling_new')
Dataset([{'answer.how_feeling_new': ['OK', 'Great', 'Terrible', 'OK']}])
```

> #### report(\**fields: str | None*, *top\_n: int | None = None*, *header\_fields: List\[str] | None = None*, *divider: bool = True*, *return\_string: bool = False*, *format: str = 'markdown'*, *filename: str | None = None*) → str | Document | None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Generates a report of the results by iterating through rows.

> **Args:**

[\*](/en/latest/#id4)fields: The fields to include in the report. If none provided, all fields are used. top\_n: Optional limit on the number of observations to include. header\_fields: Optional list of fields to include in the main header instead of as sections. divider: If True, adds a horizontal rule between observations (markdown only). return\_string: If True, returns the markdown string. If False (default in notebooks),

only displays the markdown without returning.

format: Output format - either “markdown” or “docx”. filename: If provided and format is “docx”, saves the document to this file.

> **Returns:**

Depending on format and return\_string: - For markdown: A string if return\_string is True, otherwise None (displays in notebook) - For docx: A docx.Document object, or None if filename is provided (saves to file)

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> report = r.select('how_feeling').report(return_string=True)
>>> "# Observation: 1" in report
True
>>> doc = r.select('how_feeling').report(format="docx")
>>> isinstance(doc, object)
True
```

> #### report\_from\_template(*template: str*, \**fields: str | None*, *top\_n: int | None = None*, *remove\_prefix: bool = True*, *return\_string: bool = False*, *format: str = 'text'*, *filename: str | None = None*, *separator: str = 'nn'*, *observation\_title\_template: str | None = None*, *explode: bool = False*, *filestore: bool = False*) → str | Document | List | [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore") | None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Generates a report using a Jinja2 template for each row in the dataset.

This method renders a user-provided Jinja2 template for each observation in the dataset, with template variables populated from the row data. This allows for completely customized report formatting using pandoc for advanced output formats.

> **Args:**

template: Jinja2 template string to render for each row [\*](/en/latest/#id6)fields: The fields to include in template context. If none provided, all fields are used. top\_n: Optional limit on the number of observations to include. remove\_prefix: Whether to remove type prefixes (e.g., “answer.”) from field names in template context. return\_string: If True, returns the rendered content. If False (default in notebooks),

only displays the content without returning.

format: Output format - one of “text”, “html”, “pdf”, or “docx”. Formats other than “text” require pandoc. filename: If provided, saves the rendered content to this file. For exploded output,

this becomes a template (e.g., “report\_\{index}”).

separator: String to use between rendered templates for each row (ignored when explode=True). observation\_title\_template: Optional Jinja2 template for observation titles.

Defaults to “Observation ” where index is 1-based. Template has access to all row data plus ‘index’ and ‘index0’ variables.

explode: If True, creates separate files for each observation instead of one combined file. filestore: If True, wraps the generated file(s) in FileStore object(s). If no filename is provided,

creates temporary files. For exploded output, returns a list of FileStore objects.

> **Returns:**

Depending on explode, format, return\_string, and filestore: - For text format: String content or None (if displayed in notebook) - For html format: HTML string content or None (if displayed in notebook) - For docx format: Document object or None (if saved to file) - For pdf format: PDF bytes or None (if saved to file) - If explode=True: List of created filenames (when filename provided) or list of documents/content - If filestore=True: FileStore object(s) containing the generated file(s)

> **Notes:**

* Pandoc is required for HTML, PDF, and DOCX output formats
* Templates are treated as Markdown for all non-text formats
* PDF output uses XeLaTeX engine through pandoc
* HTML output includes standalone document structure

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> template = "Person feels: `{{ how_feeling }}`"
>>> report = r.select('how_feeling').report_from_template(template, return_string=True)
>>> "Person feels: OK" in report
True
>>> "Person feels: Great" in report
True
```

```python theme={null}
# Custom observation titles >>> custom_title = “Response `{{ index }}`: `{{ how_feeling }}`” >>> report = r.select(‘how_feeling’).report_from_template( … template, observation_title_template=custom_title, return_string=True) >>> “Response 1: OK” in report True

# HTML output (requires pandoc) >>> html_report = r.select(‘how_feeling’).report_from_template( … template, format=”html”, return_string=True) # doctest: +SKIP >>> # Creates HTML with proper document structure

# PDF output (requires pandoc with XeLaTeX) >>> pdf_report = r.select(‘how_feeling’).report_from_template( … template, format=”pdf”) # doctest: +SKIP >>> # Returns PDF bytes

# Basic template functionality >>> template2 = “Feeling: `{{ how_feeling }}`, Index: `{{ index }}`” >>> report2 = r.select(‘how_feeling’).report_from_template( … template2, return_string=True, top_n=2) >>> “Feeling: OK, Index: 1” in report2 True

```

> #### rich\_print()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Display an object as a table.

> #### sample(*n: int | None = None*, *frac: float | None = None*, *with\_replacement: bool = True*, *seed: str | None = None*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a random sample of the results.

> **Args:**

n: The number of samples to take. frac: The fraction of samples to take (alternative to n). with\_replacement: Whether to sample with replacement. seed: Random seed for reproducibility.

> **Returns:**

Results: A new Results object containing the sampled data.

> #### save(*filename: str | None = None*, *compress: bool = True*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Save the object to a file as JSON with optional compression.

Serializes the object to JSON and writes it to the specified file. By default, the file will be compressed using gzip. File extensions are handled automatically.

> **Args:**

filename: Path where the file should be saved compress: If True, compress the file using gzip (default: True)

> **Returns:**

None

> **Examples:**

```python theme={null}
>>> obj.save("my_object.json.gz")  # Compressed
>>> obj.save("my_object.json", compress=False)  # Uncompressed
```

> #### *property* scenario\_keys\*: list\[str]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a set of all of the keys that are in the Scenario data.

```python theme={null}
>>> r = Results.example()
>>> r.scenario_keys
['period', 'scenario_index']
```

> #### *property* scenarios\*: [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a list of all of the scenarios in the Results.

Example:

```python theme={null}
>>> r = Results.example()
>>> r.scenarios
ScenarioList([Scenario({'period': 'morning'}), Scenario({'period': 'afternoon'}), Scenario({'period': 'morning'}), Scenario({'period': 'afternoon'})])
```

> #### score(*f: Callable*) → list[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Score the results using a function.

This method delegates to the ResultsScorer class to handle the scoring operation.

> **Args:**

f: A function that takes values from a Result object and returns a score.

> **Returns:**

list: A list of scores, one for each Result object.

> **Examples:**

```python theme={null}
>>> r = Results.example()
>>> def f(status): return 1 if status == 'Joyful' else 0
>>> r.score(f)
[1, 1, 0, 0]
```

> #### score\_with\_answer\_key(*answer\_key: dict*) → list[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Score the results using an answer key.

This method delegates to the ResultsScorer class to handle the scoring operation.

> **Args:**

answer\_key: A dictionary that maps answer values to scores.

> **Returns:**

list: A list of scores, one for each Result object.

> #### *classmethod* search(*query*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Search for objects on coop.

> #### select(\*\*columns: str | list\[str]\*) → [Dataset](/en/latest/dataset#dataset-methods)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Extract specific columns from the Results into a Dataset.

This method allows you to select specific columns from the Results object and transforms the data into a Dataset for further analysis and visualization. A Dataset is a more general-purpose data structure optimized for analysis operations rather than the hierarchical structure of Result objects.

> **Args:**

[\*](/en/latest/#id8)columns: Column names to select. Each column can be:

* A simple attribute name (e.g., “how\_feeling”)
* A fully qualified name with type (e.g., “answer.how\_feeling”)
* A wildcard pattern (e.g., “answer.\*” to select all answer fields)

If no columns are provided, selects all data.

> **Returns:**

A Dataset object containing the selected data.

> **Notes:**

* Column names are automatically disambiguated if needed
* When column names are ambiguous, specify the full path with data type
* You can use wildcard patterns with “\*” to select multiple related fields
* Selecting with no arguments returns all data
* Results are restructured in a columnar format in the Dataset

> **Examples:**

```python theme={null}
>>> results = Results.example()
```

```python theme={null}
>>> # Select a single column by name
>>> results.select('how_feeling')
Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}])
```

```python theme={null}
>>> # Select multiple columns
>>> ds = results.select('how_feeling', 'how_feeling_yesterday')
>>> sorted([list(d.keys())[0] for d in ds])
['answer.how_feeling', 'answer.how_feeling_yesterday']
```

```python theme={null}
>>> # Using fully qualified names with data type
>>> results.select('answer.how_feeling')
Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}])
```

```python theme={null}
>>> # Using partial matching for column names
>>> results.select('answer.how_feeling_y')
Dataset([{'answer.how_feeling_yesterday': ['Great', 'Good', 'OK', 'Terrible']}])
```

```python theme={null}
>>> # Select all columns (same as calling select with no arguments)
>>> results.select('*.*')
Dataset([...])
```

> #### *property* shelf\_keys\*: set\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a copy of the set of shelved result keys.

This property delegates to the ResultsSerializer class.

> #### shelve\_result(*result: [Result](/en/latest/#edsl.results.Result "edsl.results.result.Result")*) → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Store a Result object in persistent storage using its hash as the key.

This method delegates to the ResultsSerializer class to handle the shelving operation.

> **Args:**

result: A Result object to store

> **Returns:**

str: The hash key for retrieving the result later

> **Raises:**

ResultsError: If there’s an error storing the Result

> #### show\_exceptions(*traceback=False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Print the exceptions.

> #### show\_methods(*show\_docstrings=True*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Display all public methods available on this object.

This utility method helps explore the capabilities of an object by listing all its public methods and optionally their documentation.

> **Args:**

show\_docstrings: If True, print method names with docstrings;

if False, return the list of method names

> **Returns:**

None or list: If show\_docstrings is True, prints methods and returns None.

If show\_docstrings is False, returns a list of method names.

> #### shuffle(*seed: str | None = 'edsl'*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Return a shuffled copy of the results using Fisher-Yates algorithm.

> **Args:**

seed: Random seed for reproducibility.

> **Returns:**

Results: A new Results object with shuffled data.

> #### sort\_by(\**columns: str*, *reverse: bool = False*) → [Results](/en/latest/#edsl.results.Results "edsl.results.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Sort the results by one or more columns.

This method delegates to the ResultsTransformer class to handle the sorting operation.

> **Args:**

columns: One or more column names as strings. reverse: A boolean that determines whether to sort in reverse order.

> **Returns:**

Results: A new Results object with sorted data.

> **Examples:**

```python theme={null}
>>> r = Results.example()
>>> sorted_results = r.order_by('how_feeling')
>>> len(sorted_results) == len(r)
True
```

> #### spot\_issues(*models: [ModelList](/en/latest/language_models#modellist-class "edsl.language_models.ModelList") | None = None*) → [Results](/en/latest/#edsl.results.Results "edsl.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Run a survey to spot issues and suggest improvements for prompts that had no model response.

This method delegates to the ResultsAnalyzer class to handle the analysis and debugging.

**Args:**

models: Optional ModelList to use for the analysis. If None, uses the default model.

**Returns:**

Results: A new Results object containing the analysis and suggestions for improvement.

**Notes:**

Future version: Allow user to optionally pass a list of questions to review, regardless of whether they had a null model response.

> #### sql(*query: str*, *transpose: bool = None*, *transpose\_by: str = None*, *remove\_prefix: bool = True*, *shape: str = 'wide'*) → [Dataset](/en/latest/dataset#dataset-methods)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Execute SQL queries on the dataset.

This powerful method allows you to use SQL to query and transform your data, combining the expressiveness of SQL with EDSL’s data structures. It works by creating an in-memory SQLite database from your data and executing the query against it.

> **Parameters:**

query: SQL query string to execute transpose: Whether to transpose the resulting table (rows become columns) transpose\_by: Column to use as the new index when transposing remove\_prefix: Whether to remove type prefixes (e.g., “answer.”) from column names shape: Data shape to use (“wide” or “long”)

* “wide”: Default tabular format with columns for each field
* “long”: Melted format with key-value pairs, useful for certain queries

> **Returns:**

A Dataset object containing the query results

> **Notes:**

* The data is stored in a table named “self” in the SQLite database
* In wide format, column names include their type prefix unless remove\_prefix=True
* In long format, the data is melted into columns: row\_number, key, value, data\_type
* Complex objects like lists and dictionaries are converted to strings

Examples:

```python theme={null}
>>> from edsl import Results
>>> r = Results.example()
```

```python theme={null}
# Basic selection >>> len(r.sql(“SELECT * FROM self”, shape=”wide”)) 4

# Filtering with WHERE clause >>> r.sql(“SELECT * FROM self WHERE how_feeling = ‘Great’”).num_observations() 1

# Aggregation >>> r.sql(“SELECT how_feeling, COUNT(*) as count FROM self GROUP BY how_feeling”).keys() [‘how_feeling’, ‘count’]

# Using long format >>> len(r.sql(“SELECT * FROM self”, shape=”long”)) 200
```

> #### store(*d: dict*, *key\_name: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Store this object in a dictionary with an optional key.

> **Args:**

d: The dictionary in which to store the object key\_name: Optional key to use (defaults to the length of the dictionary)

> **Returns:**

None

> #### table(\**fields*, *tablefmt: str | None = 'rich'*, *pretty\_labels: dict | None = None*, *print\_parameters: dict | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

> #### tally(\**fields: str | None*, *top\_n: int | None = None*, *output='Dataset'*) → dict | [Dataset](/en/latest/dataset#dataset-methods)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Count frequency distributions of values in specified fields.

This method tallies the occurrence of unique values within one or more fields, similar to a GROUP BY and COUNT in SQL. When multiple fields are provided, it performs cross-tabulation across those fields.

> **Parameters:**

[\*](/en/latest/#id10)fields: Field names to tally. If none provided, uses all available fields. top\_n: Optional limit to return only the top N most frequent values. output: Format for results, either “Dataset” (recommended) or “dict”.

> **Returns:**

By default, returns a Dataset with columns for the field(s) and a ‘count’ column. If output=”dict”, returns a dictionary mapping values to counts.

> **Notes:**

* For single fields, returns counts of each unique value
* For multiple fields, returns counts of each unique combination of values
* Results are sorted in descending order by count
* Fields can be specified with or without their type prefix

> **Examples:**

```python theme={null}
>>> from edsl import Results
>>> r = Results.example()
```

```python theme={null}
# Single field frequency count >>> r.select(‘how_feeling’).tally(‘answer.how_feeling’, output=”dict”) {‘OK’: 2, ‘Great’: 1, ‘Terrible’: 1}

# Return as Dataset (default) >>> from edsl.dataset import Dataset >>> expected = Dataset([{‘answer.how_feeling’: [‘OK’, ‘Great’, ‘Terrible’]}, {‘count’: [2, 1, 1]}]) >>> r.select(‘how_feeling’).tally(‘answer.how_feeling’, output=”Dataset”) == expected True

# Multi-field cross-tabulation - exact output varies based on data >>> result = r.tally(‘how_feeling’, ‘how_feeling_yesterday’) >>> ‘how_feeling’ in result.keys() and ‘how_feeling_yesterday’ in result.keys() and ‘count’ in result.keys() True
```

> #### to\_agent\_list(*remove\_prefix: bool = True*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the results to a list of dictionaries, one per agent.

> **Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_agent_list()
AgentList([Agent(traits = {'how_feeling': 'OK'}), Agent(traits = {'how_feeling': 'Great'}), Agent(traits = {'how_feeling': 'Terrible'}), Agent(traits = {'how_feeling': 'OK'})])
```

> #### to\_csv(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Export the results to a FileStore instance containing CSV data.

> #### to\_dataset() → [Dataset](/en/latest/dataset#dataset-methods)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert this object to a Dataset for advanced data operations.

> **Returns:**

Dataset: A Dataset object containing this object’s data

> #### to\_dict(*sort: bool = False*, *add\_edsl\_version: bool = True*, *include\_cache: bool = True*, *include\_task\_history: bool = False*, *include\_cache\_info: bool = True*, *offload\_scenarios: bool = True*, *full\_dict: bool = False*) → dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the Results object to a dictionary representation.

This method delegates to the ResultsSerializer class to handle the conversion of the Results object to a dictionary format suitable for serialization.

> **Args:**

sort: Whether to sort the results data by hash before serialization add\_edsl\_version: Whether to include the EDSL version in the output include\_cache: Whether to include cache data in the output include\_task\_history: Whether to include task history in the output include\_cache\_info: Whether to include cache information in result data offload\_scenarios: Whether to optimize scenarios before serialization

> **Returns:**

dict\[str, Any]: Dictionary representation of the Results object

> #### to\_dicts(*remove\_prefix: bool = True*) → list\[dict][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the results to a list of dictionaries.

> **Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_dicts()
[{'how_feeling': 'OK'}, {'how_feeling': 'Great'}, {'how_feeling': 'Terrible'}, {'how_feeling': 'OK'}]
```

> #### to\_disk(*filepath: str*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Serialize the Results object to a zip file, preserving the SQLite database.

This method delegates to the ResultsSerializer class to handle the disk serialization.

This method creates a zip file containing: 1. The SQLite database file from the data container 2. A metadata.json file with the survey, created\_columns, and other non-data info 3. The cache data if present

**Args:**

filepath: Path where the zip file should be saved

**Raises:**

ResultsError: If there’s an error during serialization

> #### to\_docx(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl.py)

Export the results to a FileStore instance containing DOCX data.

Each row of the dataset will be rendered on its own page, with a 2-column table that lists the keys and associated values for that observation.

> #### to\_excel(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*, *sheet\_name: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Export the results to a FileStore instance containing Excel data.

> #### to\_json()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Serialize this object to a JSON string.

**Returns:**

str: A JSON string representation of the object

> #### to\_jsonl(*filename: str | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Export the results to a FileStore instance containing JSONL data.

> #### to\_list(*flatten=False*, *remove\_none=False*, *unzipped=False*) → list\[list][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the results to a list of lists.

**Parameters:**

* **flatten** – Whether to flatten the list of lists.
* **remove\_none** – Whether to remove None values from the list.

```python theme={null}
>>> from edsl.results import Results
>>> Results.example().select('how_feeling', 'how_feeling_yesterday')
Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling_yesterday': ['Great', 'Good', 'OK', 'Terrible']}])
```

```python theme={null}
>>> Results.example().select('how_feeling', 'how_feeling_yesterday').to_list()
[('OK', 'Great'), ('Great', 'Good'), ('Terrible', 'OK'), ('OK', 'Terrible')]
```

```python theme={null}

>>> r = Results.example()
>>> r.select('how_feeling').to_list()
['OK', 'Great', 'Terrible', 'OK']
```

```python theme={null}
>>> from edsl.dataset import Dataset
>>> Dataset([{'a.b': [[1, 9], 2, 3, 4]}]).select('a.b').to_list(flatten = True)
[1, 9, 2, 3, 4]
```

```python theme={null}
>>> from edsl.dataset import Dataset
>>> # Testing to_list flatten with multiple columns raises DatasetValueError - tested in unit tests
```

> #### to\_pandas(*remove\_prefix: bool = False*, *lists\_as\_strings=False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the results to a pandas DataFrame, ensuring that lists remain as lists.

**Args:**

remove\_prefix: Whether to remove the prefix from the column names. lists\_as\_strings: Whether to convert lists to strings.

**Returns:**

A pandas DataFrame.

> #### to\_polars(*remove\_prefix: bool = False*, *lists\_as\_strings=False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the results to a Polars DataFrame.

**Args:**

remove\_prefix: Whether to remove the prefix from the column names. lists\_as\_strings: Whether to convert lists to strings.

**Returns:**

A Polars DataFrame.

> #### to\_scenario\_list(*remove\_prefix: bool = True*) → list\[dict][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the results to a list of dictionaries, one per scenario.

**Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_scenario_list()
ScenarioList([Scenario({'how_feeling': 'OK'}), Scenario({'how_feeling': 'Great'}), Scenario({'how_feeling': 'Terrible'}), Scenario({'how_feeling': 'OK'})])
```

> #### to\_sqlite(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*, *table\_name: str = 'results'*, *if\_exists: str = 'replace'*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Export the results to a SQLite database file.

> #### to\_yaml(*add\_edsl\_version=False*, *filename: str = None*) → str | None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the object to YAML format.

Serializes the object to YAML format and optionally writes it to a file.

**Args:**

add\_edsl\_version: Whether to include EDSL version information filename: If provided, write the YAML to this file path

**Returns:**

str: The YAML string representation if no filename is provided None: If written to file

> #### tree(*node\_order: List\[str] | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Convert the results to a Tree.

**Args:**

node\_order: The order of the nodes.

**Returns:**

A Tree object.

> #### unpack\_list(*field: str*, *new\_names: List\[str] | None = None*, *keep\_original: bool = True*) → [Dataset](/en/latest/dataset#dataset-methods)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Unpack list columns into separate columns with provided names or numeric suffixes.

For example, if a dataset contains: \[\{'data': \[\[1, 2, 3], \[4, 5, 6]], 'other': \['x', 'y']}]

After d.unpack\_list('data'), it should become: \[\{'other': \['x', 'y'], 'data\_1': \[1, 4], 'data\_2': \[2, 5], 'data\_3': \[3, 6]}]

> **Args:**

field: The field containing lists to unpack new\_names: Optional list of names for the unpacked fields. If None, uses numeric suffixes. keep\_original: If True, keeps the original field in the dataset

> **Returns:**

A new Dataset with unpacked columns

> **Examples:**

```python theme={null}
>>> from edsl.dataset import Dataset
>>> d = Dataset([{'data': [[1, 2, 3], [4, 5, 6]]}])
>>> d.unpack_list('data')
Dataset([{'data': [[1, 2, 3], [4, 5, 6]]}, {'data_1': [1, 4]}, {'data_2': [2, 5]}, {'data_3': [3, 6]}])
```

```python theme={null}
>>> d.unpack_list('data', new_names=['first', 'second', 'third'])
Dataset([{'data': [[1, 2, 3], [4, 5, 6]]}, {'first': [1, 4]}, {'second': [2, 5]}, {'third': [3, 6]}])
```

> #### values()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get the values in the object’s dictionary representation.

**Returns:**

set: A set containing all the values in the object

> #### view() → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

View the results in a Jupyter notebook.

> #### yaml()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/results.py)

Get a formatted YAML representation of this object.

**Returns:**

DisplayYAML: A displayable YAML representation


# Scenarios
Source: https://docs.expectedparrot.com/en/latest/scenarios

A Scenario is a dictionary containing one or more key/value pairs that is used to add data or content to questions in a survey, replacing a parameter in a question with a specific value (e.g., numerical or textual) or content (e.g., an image or PDF). A ScenarioList is a list of Scenario objects.

## Purpose

Scenarios allow you create variations and versions of questions efficiently. For example, we could create a question “How much do you enjoy \{\{ scenario.activity }}?” and use scenarios to replace the parameter activity with running or reading or other activities. Similarly, we could create a question “What do you see in this image? \{\{ scenario.image }}” and use scenarios to replace the parameter image with different images.

### How it works

Adding scenarios to a question–or to multiple questions at once in a survey–causes it to be administered multiple times, once for each scenario, with the parameter(s) replaced by the value(s) in the scenario. This allows us to administer different versions of a question together, either asynchronously (by default) or according to [survey rules](/en/latest/surveys) that we can specify (e.g., skip/stop logic), without having to create each version of a question manually.

### Metadata

Scenarios are also a convenient way to keep track of metadata or other information relating to a survey that is important to an analysis of the results. For example, say we are using scenarios to parameterize question texts with pieces of \{\{ scenario.content }} from a dataset. In the scenarios that we create for the content parameter we could also include key/value pairs for metadata about the content, such as the \{\{ scenario.author }}, \{\{ scenario.publication\_date }}, or \{\{ scenario.source }}. This will automatically include the data in the survey results but without requiring us to also parameterize the question texts those fields. This allows us to analyze the responses in the context of the metadata and avoid having to match up the data with the metadata post-survey. Please see more details on this feature in [examples below](/en/latest/scenarios).

## Constructing a Scenario

To use a scenario, we start by creating a question that takes a parameter in double braces:

```python theme={null}
from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
  question_name = "enjoy",
  question_text = "How much do you enjoy {{ scenario.activity }}?",
  question_options = ["Not at all", "Somewhat", "Very much"]
)
```

Next we create a dictionary for a value that will replace the parameter and store it in a Scenario object:

```python theme={null}
from edsl import Scenario

scenario = Scenario({"activity": "running"})
```

We can inspect the scenario and see that it consists of the key/value pair that we created:

```python theme={null}
scenario
```

This will return:

| key | value |
| :- | :- |
| activity | running |

### ScenarioList

If multiple values will be used with a question or survey, we can create a list of Scenario objects that will be passed to the question or survey together. For example, here we create a list of scenarios and inspect them:

```python theme={null}
from edsl import Scenario

scenarios = [Scenario({"activity": a}) for a in ["running", "reading"]]
scenarios
```

Output:

```python theme={null}
[Scenario({'activity': 'running'}), Scenario({'activity': 'reading'})]
```

Alternatively, we can create a ScenarioList object. A list of scenarios is used in the same way as a ScenarioList; the difference is that a ScenarioList is a class that can be used to create a list of scenarios from a variety of data sources, such as a CSV, dataframe, list, dictionary, a Wikipedia table or a PDF pages. These special methods are discussed below.

For example, here we create a ScenarioList for the same list as above:

```python theme={null}
from edsl import Scenario, ScenarioList

scenariolist = ScenarioList(Scenario({"activity": a}) for a in ["running", "reading"])
scenariolist
```

Output:

| activity |
| :- |
| running |
| reading |

### Special method for creating scenarios

We can use the general purpose from\_source() method to create a ScenarioList from a variety of data source types. For example, the following code will create the same scenario list as above:

Each source type has its own set of parameters that can be passed to it:

* “csv”
* “dataframe”
* “delimited\_file”
* “dict”
* “directory”
* “dta”
* “excel”
* “google\_doc”
* “google\_sheet”
* “json”
* “latex”
* “list”
* “list\_of\_tuples”
* “pandas”
* “parquet”
* “pdf”
* “png”
* “pdf\_to\_image”
* “text”
* “tsv”
* “sqlite”
* “urls”
* “wikipedia”

Here we create a scenario list from files in a directory:

```python theme={null}
from edsl import ScenarioList, QuestionFreeText

# Create a ScenarioList from all image files in a directory
# Each file will be wrapped in a Scenario with key "content"
scenarios = ScenarioList.from_source("directory", "images_folder/*.png")

# Or specify a custom key name (e.g., "image")
scenarios = ScenarioList.from_source("directory", "images_folder/*.png", "image")

# Create a question that uses the scenario key
q = QuestionFreeText(
  question_name="image_description",
  question_text="Please describe this image: {{ scenario.image }}"
)

# Run the question with the scenarios
results = q.by(scenarios).run()
```

Examples of these methods are provided below and in [this notebook](https://www.expectedparrot.com/content/RobinHorton/example-scenario-methods).

## Using a scenario

We use a Scenario or ScenarioList by adding it to a question or survey of questions, either when we are constructing questions or when running them. If we add scenarios to a question when running a survey (using the by() method), the scenario contents replace the parameters in the question text at runtime, and are stored in a separate column of the results. If we add scenarios to a question when constructing a survey (using the loop() method), the scenario contents become part of the question text and there is no separate column of the results for the scenarios.

The most common situation is to add a scenario to a question when running it. This is done by passing the Scenario or ScenarioList object to the by() method of a question or survey and then chaining the run() method.

For example, here we call the by() method on the example question created above and pass a scenario list when we run it:

```python theme={null}
from edsl import QuestionMultipleChoice, Scenario, ScenarioList, Agent, Model

q = QuestionMultipleChoice(
  question_name = "enjoy",
  question_text = "How much do you enjoy {{ scenario.activity }}?",
  question_options = ["Not at all", "Somewhat", "Very much"]
)

s = ScenarioList(Scenario({"activity":a}) for a in ["running", "sleeping"])

a = Agent(traits = {"persona":"You are a human."})

m = Model("gemini-1.5-flash")

results = q.by(s).by(a).by(m).run()
```

We can check the results to verify that the scenario has been used correctly:

```python theme={null}
results.select("activity", "enjoy")
```

This will print a table of the selected components of the results:

| scenario.activity | answer.enjoy |
| :- | :- |
| running | Somewhat |
| sleeping | Very much |

### Looping

We use the loop() method to add scenarios to a question when constructing a survey. This method takes a ScenarioList and returns a list of new questions for each scenario that was passed. We can optionally include the scenario key in the question name as well as the question text. This allows us to control the question names when the new questions are created; otherwise a number is automatically added to the original question name in order to ensure uniqueness.

For example:

```python theme={null}
from edsl import QuestionMultipleChoice, ScenarioList

q = QuestionMultipleChoice(
  question_name = "enjoy_{{ scenario.activity }}",
  question_text = "How much do you enjoy {{ scenario.activity }}?",
  question_options = ["Not at all", "Somewhat", "Very much"]
)

activities = ["running", "reading"]

sl = ScenarioList.from_list("activity", activities)

questions = q.loop(sl)
```

We can inspect the questions to see that they have been created correctly:

```python theme={null}
questions
```

This will return:

```python theme={null}
[Question('multiple_choice', question_name = """enjoy_running""", question_text = """How much do you enjoy running?""", question_options = ['Not at all', 'Somewhat', 'Very much']),
Question('multiple_choice', question_name = """enjoy_reading""", question_text = """How much do you enjoy reading?""", question_options = ['Not at all', 'Somewhat', 'Very much'])]
```

We can pass the questions to a survey and run it:

```python theme={null}
from edsl import Survey, Agent

survey = Survey(questions = questions)

a = Agent(traits = {"persona": "You are a human."})

results = survey.by(a).run()

results.select("answer.*")
```

This will print a table of the response for each question. Note that “activity” is no longer in a separate scenario field; instead, there is a single column for each question that was constructed with the scenarios:

| answer.enjoy\_reading | answer.enjoy\_running |
| :- | :- |
| Very much | Somewhat |

<Note>
  **Note:**

  The loop() method *cannot* be used with image or PDF scenarios, as these are not evaluated when the question is constructed. Instead, use the by() method to add these types of scenarios when running a survey (see image scenario examples below).
</Note>

## Multiple parameters

We can also create a Scenario for multiple parameters at once:

```python theme={null}
from edsl import QuestionFreeText, Scenario

q = QuestionFreeText(
  question_name = "counting",
  question_text = "How many {{ scenario.unit }} are in a {{ scenario.distance }}?",
)

scenario = Scenario({"unit": "inches", "distance": "mile"})

results = q.by(scenario).run()

results.select("unit", "distance", "counting")
```

This will print a table of the selected components of the results:

| scenario.unit | scenario.distance | answer.counting |
| :- | :- | :- |
| inches | mile | There are 63,360 inches in a mile. |

To learn more about constructing surveys, please see the [Surveys](/en/latest/surveys) module.

## Scenarios for question options

In the above examples we created scenarios in the question\_text. We can also create a Scenario for question\_options, e.g., in a multiple choice, checkbox, linear scale or other question type that requires them. Note that we do not include the scenario. prefix when using scenarios for question options.

```python theme={null}
from edsl import QuestionMultipleChoice, Scenario

q = QuestionMultipleChoice(
  question_name = "capital_of_france",
  question_text = "What is the capital of France?",
  question_options = "{{ scenario.question_options }}"
)

s = Scenario({'question_options': ['Paris', 'London', 'Berlin', 'Madrid']})

results = q.by(s).run()

results.select("answer.*")
```

Output:

| answer.capital\_of\_france |
| :- |
| Paris |

## Scenario methods

There are a variety of methods for working with scenarios and scenario lists, including: concatenate, concatenate\_to\_list, concatenate\_to\_set, condense, drop, duplicate expand, filter, keep, mutate, order\_by, rename, sample, shuffle, times, tranform, unpack\_dict

These methods can be used to manipulate scenarios and scenario lists in various ways, such as sampling a subset of scenarios, shuffling the order of scenarios, concatenating scenarios together, filtering scenarios based on certain criteria, and more. Examples of some of these methods are provided below.

## Combining Scenarios

We can combine multiple scenarios into a single Scenario object:

```python theme={null}
from edsl import Scenario

scenario1 = Scenario({"food": "apple"})
scenario2 = Scenario({"drink": "water"})

combined_scenario = scenario1 + scenario2

combined_scenario
```

This will return:

| key | value |
| :- | :- |
| food | drink |
| apple | water |

We can also combine ScenarioList objects:

```python theme={null}
from edsl import Scenario, ScenarioList

scenariolist1 = ScenarioList([Scenario({"food": "apple"}), Scenario({"drink": "water"})])
scenariolist2 = ScenarioList([Scenario({"color": "red"}), Scenario({"shape": "circle"})])

combined_scenariolist = scenariolist1 + scenariolist2

combined_scenariolist
```

This will return:

| food | drink | color | shape |
| :- | :- | :- | :- |
| apple | nan | nan | nan |
| nan | water | nan | nan |
| nan | nan | nan | red |
| nan | nan | circle | nan |

We can create a cross product of ScenarioList objects (combine the scenarios in each list with each other):

```python theme={null}
from edsl import Scenario, ScenarioList

scenariolist1 = ScenarioList([Scenario({"food": "apple"}), Scenario({"drink": "water"})])
scenariolist2 = ScenarioList([Scenario({"color": "red"}), Scenario({"shape": "circle"})])

cross_product_scenariolist = scenariolist1 * scenariolist2

cross_product_scenariolist
```

This will return:

| food | drink | color | shape |
| :- | :- | :- | :- |
| apple | nan | nan | red |
| apple | nan | circle | nan |
| nan | water | nan | red |
| nan | water | circle | nan |

## Concatenating scenarios

There are several ScenarioList methods for concatenating scenarios.

The method concatenate() can be used to concatenate specified fields into a single string field; the default separator is a semicolon:

```python theme={null}
from edsl import Scenario, ScenarioList

sl = ScenarioList([
  Scenario({"a":1, "b":2, "c":3}),
  Scenario({"a":4, "b":5, "c":6})
])

slc = sl.concatenate(["a", "b"])

slc
```

This will return:

| c | concat\_a\_b |
| :- | :- |
| 3 | 1;2 |
| 6 | 4;5 |

We can specify a different separator:

```python theme={null}
slc = sl.concatenate(["a", "b"], separator = " ")

slc
```

This will return:

| c | concat\_a\_b |
| :- | :- |
| 3 | 1,2 |
| 6 | 4,5 |

The method concatenate\_to\_list() can be used to concatenate specified fields into a single list field:

```python theme={null}
from edsl import Scenario, ScenarioList

sl = ScenarioList([
  Scenario({"a":1, "b":2, "c":3}),
  Scenario({"a":4, "b":5, "c":6})
])

slc = sl.concatenate_to_list(["a", "b"])

slc
```

This will return:

| c | concat\_a\_b |
| :- | :- |
| 3 | \[1,2] |
| 6 | \[4,5] |

The method concatenate\_to\_set() can be used to concatenate specified fields into a single set field:

```python theme={null}
from edsl import Scenario, ScenarioList

sl = ScenarioList([
  Scenario({"a":1, "b":2, "c":3}),
  Scenario({"a":4, "b":5, "c":6})
])

slc = sl.concatenate_to_list(["a", "b"])

slc
```

This will return:

| c | concat\_a\_b |
| :- | :- |
| 3 | \{1,2} |
| 6 | \{4,5} |

The method collapse() can be used to collapse a scenario list by grouping on all fields except a specified field:

```python theme={null}
from edsl import ScenarioList

s = ScenarioList([
  Scenario({'category': 'fruit', 'color': 'red', 'item': 'apple'}),
  Scenario({'category': 'fruit', 'color': 'yellow', 'item': 'banana'}),
  Scenario({'category': 'fruit', 'color': 'red', 'item': 'cherry'}),
  Scenario({'category': 'vegetable', 'color': 'green', 'item': 'spinach'})
])

s.collapse('item')
```

This will return:

| category | color | item |
| :- | :- | :- |
| fruit | red | \[‘apple’, ‘cherry’] |
| fruit | yellow | \[‘banana’] |
| vegetable | green | \[‘spinach’] |

The method condense() can be used to combine all scenarios in a ScenarioList into a single Scenario object:

```python theme={null}
from edsl import Scenario, ScenarioList

scenarios = ScenarioList([
  Scenario({"id": 1, "text": "First"}),
  Scenario({"id": 2, "text": "Second"}),
  Scenario({"id": 3, "text": "Third"})
])

# Condense with default prefix and index
combined = scenarios.condense()

combined
```

This will return:

| scenario\_0 | scenario\_1 | scenario\_2 |
| :- | :- | :- |
| \{‘id’: 1, ‘text’: ‘First’} | \{‘id’: 2, ‘text’: ‘Second’} | \{‘id’: 3, ‘text’: ‘Third’} |

The condensed scenario can then be used in EDSL questions with dot notation:

```python theme={null}
from edsl import QuestionFreeText

q = QuestionFreeText(
  question_name="first_text",
  question_text="What is the text from the first scenario: {{ scenario.scenario_0.text }}?"
)

# Run with the condensed scenario
results = q.by(combined).run()
```

You can also use custom prefixes and control whether to include indices:

```python theme={null}
# Custom prefix
combined_custom = scenarios.condense(prefix="item")

# Without index (first item uses just prefix, others get index to avoid conflicts)
combined_no_index = scenarios.condense(prefix="data", include_index=False)

combined_custom
```

This will return:

| item\_0 | item\_1 | item\_2 |
| :- | :- | :- |
| \{‘id’: 1, ‘text’: ‘First’} | \{‘id’: 2, ‘text’: ‘Second’} | \{‘id’: 3, ‘text’: ‘Third’} |

```python theme={null}
The method from_source(“sqlite”) can be used to create a scenario list from a SQLite database. It takes a filepath to the database file and optional parameters table and sql_query.
```

## Creating scenarios from a dataset

There are a variety of methods for creating and working with scenarios generated from datasets and different data types.

### Turning results into scenarios

The method to\_scenario\_list() can be used to turn the results of a survey into a list of scenarios.

Example usage:

Say we have some results from a survey where we asked agents to choose a random number between 1 and 1000:

```python theme={null}
from edsl import QuestionNumerical, Agent, AgentList

q_random = QuestionNumerical(
  question_name = "random",
  question_text = "Choose a random number between 1 and 1000."
)

agents = AgentList(Agent({"persona":p}) for p in ["Child", "Magician", "Olympic breakdancer"])

results = q_random.by(agents).run()

results.select("persona", "random")
```

Our results are:

| agent.persona | answer.random |
| :- | :- |
| Child | 7 |
| Magician | 472 |
| Olympic breakdancer | 529 |

We can use the to\_scenario\_list() method turn components of the results into a list of scenarios to use in a new survey:

```python theme={null}
scenarios = results.select("persona", "random").to_scenario_list() # excluding other columns of the results
scenarios
```

We can inspect the scenarios to see that they have been created correctly:

| persona | random |
| :- | :- |
| Child | 7 |
| Magician | 472 |
| Olympic breakdancer | 529 |

### PDFs as textual scenarios

The ScenarioList method from\_source(“pdf”, “path/to/pdf”) is a convenient way to extract information from large files. It allows you to read in a PDF and automatically create a list of textual scenarios for the individual pages of the file. Each scenario has the following keys which can be used as parameters in a question or stored as metadata, and renamed as desired: filename, page, text:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_source("pdf", "path/to/pdf_file.pdf") # modify the filepath
```

If you prefer to create a single Scenario for the entire PDF file, you can use the FileStore module to pass the file to a Scenario in the usual way (e.g., this method is identical for PNG image files):

```python theme={null}
from edsl import Scenario, FileStore

fs = FileStore("path/to/pdf") # create a FileStore object for the PDF file (or image file)

scenario = Scenario({"my_pdf": fs}) # pass the FileStore object to a Scenario
```

To use this method with either object, we start by adding a placeholder \{\{ scenario.text }} to a question text where the text of a PDF or PDF page will be inserted. When the question or survey is run with the PDF scenario or scenario list, the text of the PDF or individual pages will be inserted into the question text at the placeholder.

For example, this code can be used to insert the text of each page of a PDF in a survey of question:

```python expandable theme={null}
from edsl import QuestionFreeText, ScenarioList, Survey

# Create a survey of questions parameterized by the {{ text }} of the PDF pages:
q1 = QuestionFreeText(
  question_name = "themes",
  question_text = "Identify the key themes mentioned on this page: {{ scenario.text }}",
)

q2 = QuestionFreeText(
  question_name = "idea",
  question_text = "Identify the most important idea on this page: {{ scenario.text }}",
)

survey = Survey([q1, q2])

scenarios = ScenarioList.from_source("pdf", "path/to/pdf_file.pdf") # modify the filepath

# Run the survey with the pages of the PDF as scenarios:
results = survey.by(scenarios).run()

# To print the page and text of each PDF page scenario together with the answers to the question:
results.select("page", "text", "answer.*")
```

Examples of this method can be viewed in a [demo notebook](/en/latest/notebooks/scenario_from_pdf).

### Image scenarios

A Scenario can be generated from an image by passing the filepath as the value (the same as a PDF, as shown above). This is done by using the FileStore module to store the image and then passing the FileStore object to a Scenario.

Example usage:

```python theme={null}
from edsl import Scenario, FileStore

fs = FileStore("parrot_logo.png") # modify filepath

s = Scenario({"image":fs})
```

We can add the key to questions as we do scenarios from other data sources:

```python expandable theme={null}
from edsl import Model, QuestionFreeText, QuestionList, Survey

m = Model("gemini-1.5-flash") # we need to use a vision model

q1 = QuestionFreeText(
  question_name = "identify",
  question_text = "What animal is in this picture: {{ scenario.image }}"
)

q2 = QuestionList(
  question_name = "colors",
  question_text = "What colors do you see in this picture: {{ scenario.image }}"
)

survey = Survey([q1, q2])

results = survey.by(s).run()

results.select("identify", "colors")
```

Output using the Expected Parrot logo:

| answer.identify | answer.colors |
| :- | :- |
| The animal in the picture is a parrot. | \[‘gray’, ‘green’, ‘yellow’, ‘pink’, ‘blue’, ‘black’] |

See a [demo notebook](/en/latest/notebooks/image_scenario_example) using of this method in the documentation page.

<Note>
  **Note:**

  You must use a vision model in order to run questions with images. We recommend testing whether a model can reliably identify your images before running a survey with them. You can also use the [models page](https://www.expectedparrot.com/models) to check available models’ performance with test questions, including images.
</Note>

### Creating a scenario list from a list

Example usage:

```python theme={null}
from edsl import ScenarioList

scenariolist = ScenarioList.from_source("list" "item", ["color", "food", "animal"])

scenariolist
```

This will return:

| item |
| :- |
| color |
| food |
| animal |

### Creating a scenario list from a dictionary

Example usage:

```python theme={null}
from edsl import ScenarioList

d = {"item": ["color", "food", "animal"]}

scenariolist = ScenarioList.from_source("nested_dict", d)
scenariolist
```

This will return:

| item |
| :- |
| color |
| food |
| animal |

### Creating a scenario list from a Wikipedia table

Example usage:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_source("wikipedia", "https://en.wikipedia.org/wiki/1990s_in_film", 3)
scenarios
```

This will return a list of scenarios for the first table on the Wikipedia page:

| Rank | Title | Studios | Worldwide gross | Year |
| :- | :- | :- | :- | :- |
| 1 | Titanic | Paramount Pictures/20th Century Fox | \$1,843,201,268 | 1997 |
| 2 | Star Wars: Episode I - The Phantom Menace | 20th Century Fox | \$924,317,558 | 1999 |
| 3 | Jurassic Park | Universal Pictures | \$914,691,118 | 1993 |
| 4 | Independence Day | 20th Century Fox | \$817,400,891 | 1996 |
| 5 | The Lion King | Walt Disney Studios | \$763,455,561 | 1994 |
| 6 | Forrest Gump | Paramount Pictures | \$677,387,716 | 1994 |
| 7 | The Sixth Sense | Walt Disney Studios | \$672,806,292 | 1999 |
| 8 | The Lost World: Jurassic Park | Universal Pictures | \$618,638,999 | 1997 |
| 9 | Men in Black | Sony Pictures/Columbia Pictures | \$589,390,539 | 1997 |
| 10 | Armageddon | Walt Disney Studios | \$553,709,788 | 1998 |
| 11 | Terminator 2: Judgment Day | TriStar Pictures | \$519,843,345 | 1991 |
| 12 | Ghost | Paramount Pictures | \$505,702,588 | 1990 |
| 13 | Aladdin | Walt Disney Studios | \$504,050,219 | 1992 |
| 14 | Twister | Warner Bros./Universal Pictures | \$494,471,524 | 1996 |
| 15 | Toy Story 2 | Walt Disney Studios | \$485,015,179 | 1999 |
| 16 | Saving Private Ryan | DreamWorks Pictures/Paramount Pictures | \$481,840,909 | 1998 |
| 17 | Home Alone | 20th Century Fox | \$476,684,675 | 1990 |
| 18 | The Matrix | Warner Bros. | \$463,517,383 | 1999 |
| 19 | Pretty Woman | Walt Disney Studios | \$463,406,268 | 1990 |
| 20 | Mission: Impossible | Paramount Pictures | \$457,696,359 | 1996 |
| 21 | Tarzan | Walt Disney Studios | \$448,191,819 | 1999 |
| 22 | Mrs. Doubtfire | 20th Century Fox | \$441,286,195 | 1993 |
| 23 | Dances with Wolves | Orion Pictures | \$424,208,848 | 1990 |
| 24 | The Mummy | Universal Pictures | \$415,933,406 | 1999 |
| 25 | The Bodyguard | Warner Bros. | \$411,006,740 | 1992 |
| 26 | Robin Hood: Prince of Thieves | Warner Bros. | \$390,493,908 | 1991 |
| 27 | Godzilla | TriStar Pictures | \$379,014,294 | 1998 |
| 28 | True Lies | 20th Century Fox | \$378,882,411 | 1994 |
| 29 | Toy Story | Walt Disney Studios | \$373,554,033 | 1995 |
| 30 | There’s Something About Mary | 20th Century Fox | \$369,884,651 | 1998 |
| 31 | The Fugitive | Warner Bros. | \$368,875,760 | 1993 |
| 32 | Die Hard with a Vengeance | 20th Century Fox/Cinergi Pictures | \$366,101,666 | 1995 |
| 33 | Notting Hill | PolyGram Filmed Entertainment | \$363,889,678 | 1999 |
| 34 | A Bug’s Life | Walt Disney Studios | \$363,398,565 | 1998 |
| 35 | The World Is Not Enough | Metro-Goldwyn-Mayer Pictures | \$361,832,400 | 1999 |
| 36 | Home Alone 2: Lost in New York | 20th Century Fox | \$358,994,850 | 1992 |
| 37 | American Beauty | DreamWorks Pictures | \$356,296,601 | 1999 |
| 38 | Apollo 13 | Universal Pictures/Imagine Entertainment | \$355,237,933 | 1995 |
| 39 | Basic Instinct | TriStar Pictures | \$352,927,224 | 1992 |
| 40 | GoldenEye | MGM/United Artists | \$352,194,034 | 1995 |
| 41 | The Mask | New Line Cinema | \$351,583,407 | 1994 |
| 42 | Speed | 20th Century Fox | \$350,448,145 | 1994 |
| 43 | Deep Impact | Paramount Pictures/DreamWorks Pictures | \$349,464,664 | 1998 |
| 44 | Beauty and the Beast | Walt Disney Studios | \$346,317,207 | 1991 |
| 45 | Pocahontas | Walt Disney Studios | \$346,079,773 | 1995 |
| 46 | The Flintstones | Universal Pictures | \$341,631,208 | 1994 |
| 47 | Batman Forever | Warner Bros. | \$336,529,144 | 1995 |
| 48 | The Rock | Walt Disney Studios | \$335,062,621 | 1996 |
| 49 | Tomorrow Never Dies | MGM/United Artists | \$333,011,068 | 1997 |
| 50 | Seven | New Line Cinema | \$327,311,859 | 1995 |

The parameters let us know the keys that can be used in the question text or stored as metadata. (They can be edited as needed - e.g., using the rename method discussed above.)

```python theme={null}
scenarios.parameters
```

This will return:

```python theme={null}
{'Rank', 'Ref.', 'Studios', 'Title', 'Worldwide gross', 'Year'}
```

The scenarios can be used to ask questions about the data in the table:

```python theme={null}
from edsl import QuestionList

q_leads = QuestionList(
  question_name = "leads",
  question_text = "Who are the lead actors or actresses in {{ scenario.Title }}?"
)

results = q_leads.by(scenarios).run()

(
  results
  .sort_by("Title")
  .select("Title", "leads")
)
```

Output:

| Title | Leads |
| :- | :- |
| A Bug’s Life | Dave Foley, Kevin Spacey, Julia Louis-Dreyfus, Hayden Panettiere, Phyllis Diller, Richard Kind, David Hyde Pierce |
| Aladdin | Mena Massoud, Naomi Scott, Will Smith |
| American Beauty | Kevin Spacey, Annette Bening, Thora Birch, Mena Suvari, Wes Bentley, Chris Cooper |
| Apollo 13 | Tom Hanks, Kevin Bacon, Bill Paxton |
| Armageddon | Bruce Willis, Billy Bob Thornton, Liv Tyler, Ben Affleck |
| Basic Instinct | Michael Douglas, Sharon Stone |
| Batman Forever | Val Kilmer, Tommy Lee Jones, Jim Carrey, Nicole Kidman, Chris O’Donnell |
| Beauty and the Beast | Emma Watson, Dan Stevens, Luke Evans, Kevin Kline, Josh Gad |
| Dances with Wolves | Kevin Costner, Mary McDonnell, Graham Greene, Rodney A. Grant |
| Deep Impact | Téa Leoni, Morgan Freeman, Elijah Wood, Robert Duvall |
| Die Hard with a Vengeance | Bruce Willis, Samuel L. Jackson, Jeremy Irons |
| Forrest Gump | Tom Hanks, Robin Wright, Gary Sinise, Mykelti Williamson, Sally Field |
| Ghost | Patrick Swayze, Demi Moore, Whoopi Goldberg |
| Godzilla | Matthew Broderick, Jean Reno, Bryan Cranston, Aaron Taylor-Johnson, Elizabeth Olsen, Kyle Chandler, Vera Farmiga, Millie Bobby Brown |
| GoldenEye | Pierce Brosnan, Sean Bean, Izabella Scorupco, Famke Janssen |
| Home Alone | Macaulay Culkin, Joe Pesci, Daniel Stern, Catherine O’Hara, John Heard |
| Home Alone 2: Lost in New York | Macaulay Culkin, Joe Pesci, Daniel Stern, Catherine O’Hara, John Heard |
| Independence Day | Will Smith, Bill Pullman, Jeff Goldblum |
| Jurassic Park | Sam Neill, Laura Dern, Jeff Goldblum, Richard Attenborough |
| Men in Black | Tommy Lee Jones, Will Smith |
| Mission: Impossible | Tom Cruise, Ving Rhames, Simon Pegg, Rebecca Ferguson, Jeremy Renner |
| Mrs. Doubtfire | Robin Williams, Sally Field, Pierce Brosnan, Lisa Jakub, Matthew Lawrence, Mara Wilson |
| Notting Hill | Julia Roberts, Hugh Grant |
| Pocahontas | Irene Bedard, Mel Gibson, Judy Kuhn, David Ogden Stiers, Russell Means, Christian Bale |
| Pretty Woman | Richard Gere, Julia Roberts |
| Robin Hood: Prince of Thieves | Kevin Costner, Morgan Freeman, Mary Elizabeth Mastrantonio, Christian Slater, Alan Rickman |
| Saving Private Ryan | Tom Hanks, Matt Damon, Tom Sizemore, Edward Burns, Barry Pepper, Adam Goldberg, Vin Diesel, Giovanni Ribisi, Jeremy Davies |
| Seven | Brad Pitt, Morgan Freeman, Gwyneth Paltrow |
| Speed | Keanu Reeves, Sandra Bullock, Dennis Hopper |
| Star Wars: Episode I - The Phantom Menace | Liam Neeson, Ewan McGregor, Natalie Portman, Jake Lloyd |
| Tarzan | Johnny Weissmuller, Maureen O’Sullivan |
| Terminator 2: Judgment Day | Arnold Schwarzenegger, Linda Hamilton, Edward Furlong, Robert Patrick |
| The Bodyguard | Kevin Costner, Whitney Houston |
| The Flintstones | John Goodman, Elizabeth Perkins, Rick Moranis, Rosie O’Donnell |
| The Fugitive | Harrison Ford, Tommy Lee Jones |
| The Lion King | Matthew Broderick, James Earl Jones, Jeremy Irons, Moira Kelly, Nathan Lane, Ernie Sabella, Rowan Atkinson, Whoopi Goldberg |
| The Lost World: Jurassic Park | Jeff Goldblum, Julianne Moore, Pete Postlethwaite |
| The Mask | Jim Carrey, Cameron Diaz |
| The Matrix | Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss |
| The Mummy | Brendan Fraser, Rachel Weisz, John Hannah, Arnold Vosloo |
| The Rock | Sean Connery, Nicolas Cage, Ed Harris |
| The Sixth Sense | Bruce Willis, Haley Joel Osment, Toni Collette, Olivia Williams |
| The World Is Not Enough | Pierce Brosnan, Sophie Marceau, Denise Richards, Robert Carlyle |
| There’s Something About Mary | Cameron Diaz, Ben Stiller, Matt Dillon |
| Titanic | Leonardo DiCaprio, Kate Winslet |
| Tomorrow Never Dies | Pierce Brosnan, Michelle Yeoh, Jonathan Pryce, Teri Hatcher |
| Toy Story | Tom Hanks, Tim Allen |
| Toy Story 2 | Tom Hanks, Tim Allen, Joan Cusack |
| True Lies | Arnold Schwarzenegger, Jamie Lee Curtis |
| Twister | Helen Hunt, Bill Paxton |

### Creating a scenario list from a CSV

The ScenarioList method from\_source(“csv”, “\<filepath>.csv”) creates a list of scenarios from a CSV file. The method reads the CSV file and creates a scenario for each row in the file, with the keys as the column names and the values as the row values.

For example, say we have a CSV file containing the following data:

```csv theme={null}
message,user,source,date
I can't log in...,Alice,Customer support,2022-01-01
I need help with my bill...,Bob,Phone,2022-01-02
I have a safety concern...,Charlie,Email,2022-01-03
I need help with a product...,David,Chat,2022-01-04
```

We can create a list of scenarios from the CSV file:

```python theme={null}
from edsl import ScenarioList

scenariolist = ScenarioList.from_source("csv", "path/to/file.csv") # update filepath
scenariolist
```

This will return a scenario for each row:

| Message | User | Source | Date |
| :- | :- | :- | :- |
| I can’t log in… | Alice | Customer support | 2022-01-01 |
| I need help with my bill… | Bob | Phone | 2022-01-02 |
| I have a safety concern… | Charlie | Email | 2022-01-03 |
| I need help with a product… | David | Chat | 2022-01-04 |

If the scenario keys are not valid Python identifiers, we can use the give\_valid\_names() method to convert them to valid identifiers.

For example, our CSV file might contain a header row that is question texts:

```csv theme={null}
"What is the message?","Who is the user?","What is the source?","What is the date?"
"I can't log in...","Alice","Customer support","2022-01-01"
"I need help with my bill...","Bob","Phone","2022-01-02"
"I have a safety concern...","Charlie","Email","2022-01-03"
"I need help with a product...","David","Chat","2022-01-04"
```

We can create a list of scenarios from the CSV file:

```python theme={null}
from edsl import ScenarioList

scenariolist = ScenarioList.from_source("csv", "path/to/file.csv") # update filepath

scenariolist = scenariolist.give_valid_names()
scenariolist
```

This will return scenarios with non-Pythonic identifiers:

| What is the message? | Who is the user? | What is the source? | What is the date? |
| :- | :- | :- | :- |
| I can’t log in… | Alice | Customer support | 2022-01-01 |
| I need help with my bill… | Bob | Phone | 2022-01-02 |
| I have a safety concern… | Charlie | Email | 2022-01-03 |
| I need help with a product… | David | Chat | 2022-01-04 |

We can then use the give\_valid\_names() method to convert the keys to valid identifiers:

```python theme={null}
scenariolist.give_valid_names()
scenariolist
```

This will return scenarios with valid identifiers (removing stop words and using underscores):

| message | user | source | date |
| :- | :- | :- | :- |
| I can’t log in… | Alice | Customer support | 2022-01-01 |
| I need help with my bill… | Bob | Phone | 2022-01-02 |
| I have a safety concern… | Charlie | Email | 2022-01-03 |
| I need help with a product… | David | Chat | 2022-01-04 |

## Methods for un/pivoting and grouping scenarios

There are a variety of methods for modifying scenarios and scenario lists.

### Unpivoting a scenario list

The ScenarioList method unpivot() can be used to unpivot a scenario list based on one or more specified identifiers. It takes a list of id\_vars which are the names of the key/value pairs to keep in each new scenario, and a list of value\_vars which are the names of the key/value pairs to unpivot.

For example, say we have a scenario list for the above CSV file:

```python theme={null}
from edsl import ScenarioList

scenariolist = ScenarioList.from_source("csv", "<filepath>.csv")
scenariolist
```

We can call the unpivot the scenario list:

```python theme={null}
scenariolist.unpivot(id_vars = ["user"], value_vars = ["source", "date", "message"])
scenariolist
```

This will return a list of scenarios with the source, date, and message key/value pairs unpivoted:

| user | variable | value |
| :- | :- | :- |
| Alice | source | Customer support |
| Alice | date | 2022-01-01 |
| Alice | message | I can’t log in… |
| Bob | source | Phone |
| Bob | date | 2022-01-02 |
| Bob | message | I need help with my bill… |
| Charlie | source | Email |
| Charlie | date | 2022-01-03 |
| Charlie | message | I have a safety concern… |
| David | source | Chat |
| David | date | 2022-01-04 |
| David | message | I need help with a product… |

### Pivoting a scenario list

We can call the pivot() method to reverse the unpivot operation:

```python theme={null}
scenariolist.pivot(id_vars = ["user"], var_name="variable", value_name="value")
scenariolist
```

This will return a list of scenarios with the source, date, and message key/value pairs pivoted back to their original form:

| user | source | date | message |
| :- | :- | :- | :- |
| Alice | Customer support | 2022-01-01 | I can’t log in… |
| Bob | Phone | 2022-01-02 | I need help with my bill… |
| Charlie | Email | 2022-01-03 | I have a safety concern… |
| David | Chat | 2022-01-04 | I need help with a product… |

### Grouping scenarios

The group\_by() method can be used to group scenarios by one or more specified keys and apply a function to the values of the specified variables.

Example usage:

```python theme={null}
from edsl import Scenario, ScenarioList

def avg_sum(a, b):
  return {'avg_a': sum(a) / len(a), 'sum_b': sum(b)}

scenariolist = ScenarioList([
  Scenario({'group': 'A', 'year': 2020, 'a': 10, 'b': 20}),
  Scenario({'group': 'A', 'year': 2021, 'a': 15, 'b': 25}),
  Scenario({'group': 'B', 'year': 2020, 'a': 12, 'b': 22}),
  Scenario({'group': 'B', 'year': 2021, 'a': 17, 'b': 27})
])

scenariolist.group_by(id_vars=['group'], variables=['a', 'b'], func=avg_sum)
```

This will return a list of scenarios with the a and b key/value pairs grouped by the group key and the avg\_a and sum\_b key/value pairs calculated by the avg\_sum function:

| group | avg\_a | sum\_b |
| :- | :- | :- |
| A | 12.5 | 45 |
| B | 14.5 | 49 |

## Data labeling tasks

Scenarios are particularly useful for conducting data labeling or data coding tasks, where the task can be designed as a survey of questions about each piece of data in a dataset.

For example, say we have a dataset of text messages that we want to sort by topic. We can perform this task by using a language model to answer questions such as “What is the primary topic of this message: \{\{ scenario.message }}?” or “Does this message mention a safety issue? \{\{ scenario.message }}”, where each text message is inserted in the message placeholder of the question text.

Here we use scenarios to conduct the task:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, Survey, Scenario, ScenarioList

# Create a question with that takes a parameter
q1 = QuestionMultipleChoice(
  question_name = "topic",
  question_text = "What is the topic of this message: {{ scenario.message }}?",
  question_options = ["Safety", "Product support", "Billing", "Login issue", "Other"]
)

q2 = QuestionMultipleChoice(
  question_name = "safety",
  question_text = "Does this message mention a safety issue? {{ scenario.message }}?",
  question_options = ["Yes", "No", "Unclear"]
)

# Create a list of scenarios for the parameter
messages = [
  "I can't log in...",
  "I need help with my bill...",
  "I have a safety concern...",
  "I need help with a product..."
]

scenarios = ScenarioList.from_source("list", "message", messages)

# Create a survey with the question
survey = Survey(questions = [q1, q2])

# Run the survey with the scenarios
results = survey.by(scenarios).run()
```

We can then analyze the results to see how the agent answered the questions for each scenario:

```python theme={null}
results.select("message", "safety", "topic")
```

This will print a table of the scenarios and the answers to the questions for each scenario:

| message | safety | topic |
| :- | :- | :- |
| I can’t log in… | No | Login issue |
| I need help with a product… | No | Product support |
| I need help with my bill… | No | Billing |
| I have a safety concern… | Yes | Safety |

### Adding metadata

If we have metadata about the messages that we want to keep track of, we can add it to the scenarios as well. This will create additional columns for the metadata in the results dataset, but without the need to include it in our question texts. Here we modify the above example to use a dataset of messages with metadata. Note that the question texts are unchanged:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, Survey, Scenario, ScenarioList

# Create a question with a parameter
q1 = QuestionMultipleChoice(
  question_name = "topic",
  question_text = "What is the topic of this message: {{ scenario.message }}?",
  question_options = ["Safety", "Product support", "Billing", "Login issue", "Other"]
)

q2 = QuestionMultipleChoice(
  question_name = "safety",
  question_text = "Does this message mention a safety issue? {{ scenario.message }}?",
  question_options = ["Yes", "No", "Unclear"]
)

# Create scenarios for the sets of parameters
user_messages = [
  {"message": "I can't log in...", "user": "Alice", "source": "Customer support", "date": "2022-01-01"},
  {"message": "I need help with my bill...", "user": "Bob", "source": "Phone", "date": "2022-01-02"},
  {"message": "I have a safety concern...", "user": "Charlie", "source": "Email", "date": "2022-01-03"},
  {"message": "I need help with a product...", "user": "David", "source": "Chat", "date": "2022-01-04"}
]

scenarios = ScenarioList.from_source("dict", user_messages)

# Create a survey with the question
survey = Survey(questions = [q1, q2])

# Run the survey with the scenarios
results = survey.by(scenarios).run()

# Inspect the results
results.select("scenario.*", "answer.*")
```

We can see how the agent answered the questions for each scenario, together with the metadata that was not included in the question text:

| user | source | message | date | topic | safety |
| :- | :- | :- | :- | :- | :- |
| Alice | Customer support | I can’t log in… | 2022-01-01 | Login issue | No |
| Bob | Phone | I need help with my bill… | 2022-01-02 | Billing | No |
| Charlie | Email | I have a safety concern… | 2022-01-03 | Safety | Yes |
| David | Chat | I need help with a product… | 2022-01-04 | Product support | No |

To learn more about accessing, analyzing and visualizing survey results, please see the [Results](/en/latest/results) section.

## Slicing/chunking content into scenarios

We can use the Scenario method chunk() to slice a text scenario into a ScenarioList based on num\_words or num\_lines.

Example usage:

```python theme={null}
my_haiku = """
This is a long text.
Pages and pages, oh my!
I need to chunk it.
"""

text_scenario = Scenario({"my_text": my_haiku})

word_chunks_scenariolist = text_scenario.chunk(
  "my_text",
  num_words = 5, # use num_words or num_lines but not both
  include_original = True, # optional
  hash_original = True # optional
)
word_chunks_scenariolist
```

This will return:

| my\_text | my\_text\_chunk | my\_text\_original |
| :- | :- | :- |
| This is a long text. | 0 | 4aec42eda32b7f32bde8be6a6bc11125 |
| Pages and pages, oh my! | 1 | 4aec42eda32b7f32bde8be6a6bc11125 |
| I need to chunk it. | 2 | 4aec42eda32b7f32bde8be6a6bc11125 |

## Using f-strings with scenarios

It is possible to use scenarios and f-strings together in a question. An f-string must be evaluated when a question is constructed, whereas a scenario is either evaluated when a question is run (using the by method) or when a question is constructed (using the loop method).

For example, here we use an f-string to create different versions of a question that also takes a parameter \{\{ scenario.activity }}, together with a list of scenarios to replace the parameter when the question is run. We optionally include the f-string in the question name in addition to the question text in order to control the unique identifiers for the questions, which are needed in order to pass the questions that are created to a Survey. (If you do not include the f-string in the question name, a number is automatically appended to each question name to ensure uniqueness.) Then we use the show\_prompts() method to examine the user prompts that are created when the scenarios are added to the questions:

```python expandable theme={null}
from edsl import QuestionFreeText, Scenario, ScenarioList, Survey

questions = []
sentiments = ["enjoy", "hate", "love"]
activities = ["running", "reading"]

for sentiment in sentiments:
  q = QuestionFreeText(
    question_name = f"{ sentiment }_activity",
    question_text = f"How much do you { sentiment } " + "{{ scenario.activity }}?"
  )
  questions.append(q)

scenarios = ScenarioList.from_source("list", "activity", activities)

survey = Survey(questions = questions)
survey.by(scenarios).show_prompts()
```

The show\_prompts method will return the questions created with the f-string with the scenarios added. (Note that the system prompts are blank because we have not created any agents.)

| user\_prompt | system\_prompt |
| :- | :- |
| How much do you enjoy running? | |
| How much do you hate running? | |
| How much do you love running? | |
| How much do you enjoy reading? | |
| How much do you hate reading? | |
| How much do you love reading? | |

To learn more about user and system prompts, please see the [Prompts](/en/latest/prompts) section.

## Special methods

Special methods are available for generating or modifying scenarios using web searches:

The from\_prompt method allows you to create scenarios from a prompt, which can be useful for generating scenarios based on user input or other dynamic sources:

```python theme={null}
from edsl import ScenarioList

scenarios = ScenarioList.from_prompt(
  description = "What are some popular programming languages?",
  name = "programming_languages", # optional name for the scenarios; default is "item"
  target_number = 5, # optional number of scenarios to generate; default is 10
  verbose = True # optional flag to return verbose output; default is False
)
```

The from\_search\_terms method allows you to create scenarios from a list of search terms, which can be useful for generating scenarios based on search queries or other dynamic sources:

```python theme={null}
from edsl import ScenarioList

search_terms = ["Python", "Java", "JavaScript"]
scenarios = ScenarioList.from_search_terms(search_terms)
```

The method augment\_with\_wikipedia allows you to augment scenarios with information from Wikipedia, which can be useful for enriching scenarios with additional context or data:

```python theme={null}
from edsl import ScenarioList

# method is used to augment existing scenarios
scenarios = ScenarioList.from_prompt(
  description = "What are some popular programming languages?",
  name = "programming_languages"
)

scenarios.augment_with_wikipedia(
  search_key = "programming languages",
  content_only = True # default optional flag to return only the content
  key_name = "wikipedia_content" # default optional key name for the content
)
```

## Scenario class

> ### *class* edsl.scenarios.Scenario(*data: Dict\[str, Any] | Mapping\[str, Any] | None = None*, *name: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Bases: [`Base`](/en/latest/base#base-class "edsl.base.base_class.Base"), `UserDict`

A dictionary-like object that stores key-value pairs for parameterizing questions.

A Scenario inherits from both the EDSL Base class and Python’s UserDict, allowing it to function as a dictionary while providing additional functionality. Scenarios are used to parameterize questions by providing variable data that can be referenced within question templates using Jinja syntax.

Scenarios can be created directly with dictionary data or constructed from various sources using class methods (from\_file, from\_url, from\_pdf, etc.). They support operations like addition (combining scenarios) and multiplication (creating cross products with other scenarios or scenario lists).

> ### **Attributes:**

data (dict): The underlying dictionary data. name (str, optional): A name for the scenario.

> ### **Examples:**

Create a simple scenario: >>> s = Scenario(\{“product”: “coffee”, “price”: 4.99})

Combine scenarios: >>> s1 = Scenario(\{“product”: “coffee”}) >>> s2 = Scenario(\{“price”: 4.99}) >>> s3 = s1 + s2 >>> s3 Scenario(\{‘product’: ‘coffee’, ‘price’: 4.99})

Create a scenario from a file: >>> import tempfile >>> with tempfile.NamedTemporaryFile(mode=’w’, suffix=’.txt’, delete=False) as f: … \_ = f.write(“Hello World”) … data\_path = f.name >>> s = Scenario.from\_file(data\_path, “document”) >>> import os >>> os.unlink(data\_path) # Clean up temp file

> ### \*\***init**(*data: Dict\[str, Any] | Mapping\[str, Any] | None = None*, *name: str | None = None*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Initialize a new Scenario.

> **Args:**

**data: A dictionary of key-value pairs for parameterizing questions.**

Any dictionary-like object that can be converted to a dict is accepted.

name: An optional name for the scenario to aid in identification.

> **Raises:**

ScenarioError: If the data cannot be converted to a dictionary.

> **Examples:**

```python theme={null}
>>> s = Scenario({"product": "coffee", "price": 4.99})
>>> s = Scenario({"question": "What is your favorite color?"}, name="color_question")
```

> ### \*\*chunk(*field: str*, *num\_words: int | None = None*, *num\_lines: int | None = None*, *include\_original: bool = False*, *hash\_original: bool = False*) → [ScenarioList](/en/latest/#edsl.scenarios.ScenarioList "edsl.scenarios.ScenarioList") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Splits a text field into chunks of a specified size, creating a ScenarioList.

This method takes a field containing text and divides it into smaller chunks based on either word count or line count. It’s particularly useful for processing large text documents in manageable pieces, such as for summarization, analysis, or when working with models that have token limits.

> **Args:**

field: The key name of the field in the Scenario to split. num\_words: The number of words to include in each chunk. Mutually exclusive

```python theme={null}
with num_lines.
```

**num\_lines: The number of lines to include in each chunk. Mutually exclusive**

```python theme={null}
with num_words.
```

**include\_original: If True, includes the original complete text in each chunk**

```python theme={null}
with a “_original” suffix.
```

```python theme={null}
hash_original: If True and include_original is True, stores a hash of the
```

original text instead of the full text.

> **Returns:**

A ScenarioList containing multiple Scenarios, each with a chunk of the original text. Each Scenario includes the chunk text, chunk index, character count, and word count.

> **Raises:**

ValueError: If neither num\_words nor num\_lines is specified, or if both are. KeyError: If the specified field doesn’t exist in the Scenario.

> **Examples:**

```python theme={null}
Split by lines (1 line per chunk): >>> s = Scenario({“text”: “This is a test.nThis is a test.nnThis is a test.”}) >>> s.chunk(“text”, num_lines=1) ScenarioList([Scenario({‘text’: ‘This is a test.’, ‘text_chunk’: 0, ‘text_char_count’: 15, ‘text_word_count’: 4}), Scenario({‘text’: ‘This is a test.’, ‘text_chunk’: 1, ‘text_char_count’: 15, ‘text_word_count’: 4}), Scenario({‘text’: ‘’, ‘text_chunk’: 2, ‘text_char_count’: 0, ‘text_word_count’: 0}), Scenario({‘text’: ‘This is a test.’, ‘text_chunk’: 3, ‘text_char_count’: 15, ‘text_word_count’: 4})])

Split by words (2 words per chunk): >>> s.chunk(“text”, num_words=2) ScenarioList([Scenario({‘text’: ‘This is’, ‘text_chunk’: 0, ‘text_char_count’: 7, ‘text_word_count’: 2}), Scenario({‘text’: ‘a test.’, ‘text_chunk’: 1, ‘text_char_count’: 7, ‘text_word_count’: 2}), Scenario({‘text’: ‘This is’, ‘text_chunk’: 2, ‘text_char_count’: 7, ‘text_word_count’: 2}), Scenario({‘text’: ‘a test.’, ‘text_chunk’: 3, ‘text_char_count’: 7, ‘text_word_count’: 2}), Scenario({‘text’: ‘This is’, ‘text_chunk’: 4, ‘text_char_count’: 7, ‘text_word_count’: 2}), Scenario({‘text’: ‘a test.’, ‘text_chunk’: 5, ‘text_char_count’: 7, ‘text_word_count’: 2})])

Include original text in each chunk: >>> s = Scenario({“text”: “Hello World”}) >>> s.chunk(“text”, num_words=1, include_original=True) ScenarioList([Scenario({‘text’: ‘Hello’, ‘text_chunk’: 0, ‘text_char_count’: 5, ‘text_word_count’: 1, ‘text_original’: ‘Hello World’}), Scenario({‘text’: ‘World’, ‘text_chunk’: 1, ‘text_char_count’: 5, ‘text_word_count’: 1, ‘text_original’: ‘Hello World’})])

Use a hash of the original text: >>> s.chunk(“text”, num_words=1, include_original=True, hash_original=True) ScenarioList([Scenario({‘text’: ‘Hello’, ‘text_chunk’: 0, ‘text_char_count’: 5, ‘text_word_count’: 1, ‘text_original’: ‘b10a8db164e0754105b7a99be72e3fe5’}), Scenario({‘text’: ‘World’, ‘text_chunk’: 1, ‘text_char_count’: 5, ‘text_word_count’: 1, ‘text_original’: ‘b10a8db164e0754105b7a99be72e3fe5’})])

```

> **Notes:**

* Either num\_words or num\_lines must be specified, but not both
* Each chunk is assigned a sequential index in the ‘text\_chunk’ field
* Character and word counts for each chunk are included
* When include\_original is True, the original text is preserved in each chunk
* The hash\_original option is useful to save space while maintaining traceability

> #### chunk\_text(*field: str*, *chunk\_size\_field: str*, *unit: str = 'word'*, *include\_original: bool = False*, *hash\_original: bool = False*) → [ScenarioList](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Chunks a text field into smaller chunks of a specified size, creating a ScenarioList.

This method takes a field containing text and divides it into smaller chunks based on either word count or line count. It’s particularly useful for processing large text documents in manageable pieces, such as for summarization, analysis, or when working with models that have token limits.

> #### code() → List\[str] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Generate Python code to recreate this scenario.

> **Returns:**

A list of strings representing Python code lines that can be executed to recreate this scenario.

> **Examples:**

```python theme={null}
>>> s = Scenario({"name": "Alice", "age": 30})
>>> code_lines = s.code()
>>> print("n".join(code_lines))
from edsl.scenarios import Scenario
s = Scenario({'name': 'Alice', 'age': 30})
```

> #### drop(\*\*args: str | Iterable\[str]\*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Drop a subset of keys from a scenario.

This method delegates to ScenarioSelector for the actual dropping logic. It supports both individual string arguments and collection arguments for backward compatibility.

> **Args:**

[\*](/en/latest/#id3)args: Either a single collection of keys (for backward compatibility)

or individual string arguments for keys to drop.

> **Returns:**

A new Scenario containing all keys except the dropped ones.

> **Raises:**

ValueError: If no arguments are provided.

> **Examples:**

```python theme={null}
Using a list (backward compatible): >>> s = Scenario({“food”: “wood chips”, “drink”: “water”}) >>> s.drop([“food”]) Scenario({‘drink’: ‘water’})

Using individual string arguments: >>> s = Scenario({“food”: “wood chips”, “drink”: “water”, “dessert”: “cookies”}) >>> s.drop(“drink”, “dessert”) Scenario({‘food’: ‘wood chips’})

Single string argument: >>> s.drop(“drink”) Scenario({‘food’: ‘wood chips’, ‘dessert’: ‘cookies’})
```

> #### *classmethod* example(*randomize: bool = False*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Returns an example Scenario instance.

> **Args:**

randomize: If True, adds a random string to the value of the example key

to ensure uniqueness.

> **Returns:**

A Scenario instance with example data suitable for testing or demonstration.

> **Examples:**

```python theme={null}

>>> s = Scenario.example()
>>> 'persona' in s
True
>>> s1 = Scenario.example(randomize=True)
>>> s2 = Scenario.example(randomize=True)
>>> s1.data != s2.data  # Should be different due to randomization
True
```

> #### firecrawl *= \<edsl.scenarios.firecrawl\_scenario.FirecrawlRequest object>* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### *classmethod* from\_dict(*d: Dict\[str, Any]*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Creates a Scenario from a dictionary, with special handling for FileStore objects.

This method creates a Scenario using the provided dictionary. It has special handling for dictionary values that represent serialized FileStore objects, which it will deserialize back into proper FileStore instances.

> **Args:**

d: A dictionary to convert to a Scenario.

> **Returns:**

A new Scenario containing the provided dictionary data.

> **Examples:**

```python theme={null}
>>> Scenario.from_dict({"food": "wood chips"})
Scenario({'food': 'wood chips'})
```

```python theme={null}
>>> # Example with a serialized FileStore
>>> from edsl import FileStore
>>> file_dict = {"path": "example.txt", "base64_string": "SGVsbG8gV29ybGQ="}
>>> s = Scenario.from_dict({"document": file_dict})
>>> isinstance(s["document"], FileStore)
True
```

> **Notes:**

* Any dictionary values that match the FileStore format will be converted to FileStore objects
* The method detects FileStore objects by looking for “base64\_string” and “path” keys
* EDSL version information is automatically removed by the @remove\_edsl\_version decorator
* This method is commonly used when deserializing scenarios from JSON or other formats

> #### *classmethod* from\_docx(*docx\_path: str*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Creates a Scenario containing text extracted from a Microsoft Word document.

This method extracts text and structure from a DOCX file and creates a Scenario containing this information. It uses the DocxScenario class to handle the extraction process and maintain document structure where possible.

> **Args:**

docx\_path: Path to the DOCX file to extract content from.

> **Returns:**

A Scenario containing the file path and extracted text from the DOCX file.

> **Raises:**

FileNotFoundError: If the specified DOCX file does not exist. ImportError: If the python-docx library is not installed.

> **Examples:**

```python theme={null}
>>> from docx import Document
>>> doc = Document()
>>> _ = doc.add_heading("EDSL Survey")
>>> _ = doc.add_paragraph("This is a test.")
>>> doc.save("test.docx")
>>> s = Scenario.from_docx("test.docx")
>>> s
Scenario({'file_path': 'test.docx', 'text': 'EDSL SurveynThis is a test.'})
>>> import os; os.remove("test.docx")
```

> **Notes:**

* The returned Scenario typically contains the file path and extracted text
* The extraction process attempts to maintain document structure
* Requires the python-docx library to be installed

> #### *classmethod* from\_file(*file\_path: str*, *field\_name: str*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Creates a Scenario containing a FileStore object from a file.

This method creates a Scenario with a single key-value pair where the value is a FileStore object that encapsulates the specified file. The FileStore handles appropriate file loading, encoding, and extraction based on the file type.

> **Args:**

file\_path: Path to the file to be incorporated into the Scenario. field\_name: Key name to use for storing the FileStore in the Scenario.

> **Returns:**

A Scenario containing a FileStore object linked to the specified file.

> **Raises:**

FileNotFoundError: If the specified file does not exist.

> **Examples:**

```python theme={null}
>>> import tempfile
>>> with tempfile.NamedTemporaryFile(suffix=".txt", mode="w") as f:
...     _ = f.write("This is a test.")
...     _ = f.flush()
...     s = Scenario.from_file(f.name, "file")
>>> s
Scenario({'file': FileStore(path='...', ...)})
```

> **Notes:**

* The FileStore object handles various file formats differently
* FileStore provides methods to access file content, extract text, and manage file operations appropriate to the file type

> #### *classmethod* from\_html(*url: str*, *field\_name: str | None = None*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Creates a Scenario containing both HTML content and extracted text from a URL.

This method fetches HTML content from a URL, extracts readable text from it, and creates a Scenario containing the original URL, the raw HTML, and the extracted text. Unlike from\_url, this method preserves the raw HTML content.

> **Args:**

url: URL to fetch HTML content from. field\_name: Key name to use for the extracted text in the Scenario.

If not provided, defaults to “text”.

> **Returns:**

A Scenario containing the URL, raw HTML, and extracted text.

> **Raises:**

requests.exceptions.RequestException: If the URL cannot be accessed.

> **Examples:**

Create a scenario from HTML content (requires network access):

s = Scenario.from\_html(”[https://example.com](https://example.com/)”) # Returns a Scenario with “url”, “html”, and “text” fields

s = Scenario.from\_html(”[https://example.com](https://example.com/)”, field\_name=”content”) # Returns a Scenario with “url”, “html”, and “content” fields

> **Notes:**

* Uses BeautifulSoup for HTML parsing when available
* Stores both the raw HTML and the extracted text
* Provides a more comprehensive representation than from\_url
* Useful when the HTML structure or specific elements are needed

> #### *classmethod* from\_image(*image\_path: str*, *image\_name: str | None = None*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Creates a Scenario containing an image file as a FileStore object.

This method creates a Scenario with a single key-value pair where the value is a FileStore object that encapsulates the specified image file. The image is stored as a base64-encoded string, allowing it to be easily serialized and transmitted.

> **Args:**

image\_path: Path to the image file to be incorporated into the Scenario. image\_name: Key name to use for storing the FileStore in the Scenario.

If not provided, uses the filename without extension.

> **Returns:**

A Scenario containing a FileStore object with the image data.

> **Raises:**

FileNotFoundError: If the specified image file does not exist.

> **Examples:**

> > > import os
> > >
> > > # Assuming an image file exists
> > >
> > > if os.path.exists("image.jpg"):
> > > ...     s = Scenario.from\_image("image.jpg")
> > > ...     s\_named = Scenario.from\_image("image.jpg", "picture")

> **Notes:**

* The resulting FileStore can be displayed in notebooks or used in questions
* Supported image formats include JPG, PNG, GIF, etc.
* The image is stored as a base64-encoded string for portability

> #### *classmethod* from\_pdf(*pdf\_path: str*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Creates a Scenario containing text extracted from a PDF file.

This method extracts text and metadata from a PDF file and creates a Scenario containing this information. It uses the PdfExtractor class which provides access to text content, metadata, and structure from PDF files.

> **Args:**

pdf\_path: Path to the PDF file to extract content from.

> **Returns:**

A Scenario containing extracted text and metadata from the PDF.

> **Raises:**

FileNotFoundError: If the specified PDF file does not exist. ImportError: If the required PDF extraction libraries are not installed.

> **Examples:**

```python theme={null}
>>> import os
>>> # Assuming a PDF file exists
>>> if os.path.exists("document.pdf"):
...     s = Scenario.from_pdf("document.pdf")
```

> **Notes:**

* The returned Scenario contains various keys with PDF content and metadata
* PDF extraction requires the PyMuPDF library
* The extraction process parses the PDF to maintain structure where possible

> #### *classmethod* from\_pdf\_to\_image(*pdf\_path: str*, *image\_format: str = 'jpeg'*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Converts each page of a PDF into an image and creates a Scenario containing them.

This method takes a PDF file, converts each page to an image in the specified format, and creates a Scenario containing the original file path and FileStore objects for each page image. This is particularly useful for visualizing PDF content or for image-based processing of PDF documents.

> **Args:**

pdf\_path: Path to the PDF file to convert to images. image\_format: Format of the output images (default is ‘jpeg’).

Other formats include ‘png’, ‘tiff’, etc.

> **Returns:**

A Scenario containing the original PDF file path and FileStore objects for each page image, with keys like “page\_0”, “page\_1”, etc.

> **Raises:**

FileNotFoundError: If the specified PDF file does not exist. ImportError: If pdf2image is not installed.

> **Examples:**

```python theme={null}
>>> import os
>>> # Assuming a PDF file exists
>>> if os.path.exists("document.pdf"):
...     s = Scenario.from_pdf_to_image("document.pdf")
...     s_png = Scenario.from_pdf_to_image("document.pdf", "png")
```

> **Notes:**

* Requires the pdf2image library which depends on poppler
* Creates a separate image for each page of the PDF
* Images are stored in FileStore objects for easy display and handling
* Images are created in a temporary directory which is automatically cleaned up

> #### *classmethod* from\_url(*url: str*, *field\_name: str | None = 'text'*, *testing: bool = False*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Creates a Scenario from the content of a URL.

This method fetches content from a web URL and creates a Scenario containing the URL and the extracted text. When available, BeautifulSoup is used for better HTML parsing and text extraction, otherwise a basic requests approach is used.

> **Args:**

url: The URL to fetch content from. field\_name: The key name to use for storing the extracted text in the Scenario.

Defaults to “text”.

testing: If True, uses a simplified requests method instead of BeautifulSoup.

This is primarily for testing purposes.

> **Returns:**

A Scenario containing the URL and extracted text.

> **Raises:**

requests.exceptions.RequestException: If the URL cannot be accessed.

> **Examples:**

Create a scenario from a URL (requires network access):

s = Scenario.from\_url(”[https://example.com](https://example.com/)”, testing=True) # Returns a Scenario with “url” and “text” fields

s = Scenario.from\_url(”[https://example.com](https://example.com/)”, field\_name=”content”, testing=True) # Returns a Scenario with “url”, “html”, and “content” fields

> **Notes:**

* The method attempts to use BeautifulSoup and fake\_useragent for better HTML parsing and to mimic a real browser.
* If these packages are not available, it falls back to basic requests.
* When using BeautifulSoup, it extracts text from paragraph and heading tags.

> #### get\_filestore\_info() → Dict\[str, Any] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Returns information about FileStore objects present in this Scenario.

This method is useful for determining how many signed URLs need to be generated and what file extensions/types are present before calling save\_to\_gcs\_bucket().

> **Returns:**

**dict: Information about FileStore objects containing:**

* total\_count: Total number of FileStore objects
* filestore\_keys: List of scenario keys that contain FileStore objects
* file\_extensions: Dictionary mapping keys to file extensions
* file\_types: Dictionary mapping keys to MIME types
* is\_filestore\_scenario: Boolean indicating if this Scenario was created from a FileStore
* summary: Human-readable summary of files

> #### *property* has\_jinja\_braces\*: bool\* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return whether the scenario has jinja braces. This matters for rendering.

```python theme={null}
>>> s = Scenario({"food": "I love {{wood chips}}"})
>>> s.has_jinja_braces
True
```

> #### keep(\*\*args: str | Iterable\[str]\*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Keep a subset of keys from a scenario (alias for select).

This method delegates to ScenarioSelector for the actual selection logic. It is functionally identical to select() but provides more intuitive naming.

> **Args:**

[\*](/en/latest/#id5)args: Either a single collection of keys (for backward compatibility)

or individual string arguments for keys to keep.

> **Returns:**

A new Scenario containing only the kept keys and their values.

> **Raises:**

KeyError: If any of the specified keys don’t exist in the scenario. ValueError: If no arguments are provided.

> **Examples:**

Using a list (backward compatible): >>> s = Scenario(\{“food”: “wood chips”, “drink”: “water”}) >>> s.keep(\[“food”]) Scenario(\{'food': 'wood chips'})

Using individual string arguments: >>> s = Scenario(\{“food”: “wood chips”, “drink”: “water”, “dessert”: “cookies”}) >>> s.keep(“food”, “drink”) Scenario(\{‘food’: ‘wood chips’, ‘drink’: ‘water’})

> #### new\_column\_names(*new\_names: List\[str]*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Rename all keys of a scenario using a list of new names.

> **Args:**

new\_names: A list of new key names. Must have the same length as the

number of keys in the scenario.

> **Returns:**

A new Scenario with keys renamed according to the provided list.

> **Raises:**

ValueError: If the length of new\_names doesn’t match the number of keys.

> **Examples:**

```python theme={null}
>>> s = Scenario({"food": "wood chips"})
>>> s.new_column_names(["food_preference"])
Scenario({'food_preference': 'wood chips'})
```

> #### offload(*inplace: bool = False*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Offload base64-encoded content from the scenario by replacing ‘base64\_string’ fields with ‘offloaded’. This reduces memory usage.

This method delegates to ScenarioOffloader for the actual offloading logic. It handles three types of base64 content: 1. Direct base64\_string in the scenario (from FileStore.to\_dict()) 2. FileStore objects containing base64 content 3. Dictionary values containing base64\_string fields

> **Args:**

inplace: If True, modify the current scenario. If False, return a new one.

> **Returns:**

The modified scenario (either self or a new instance).

> **Examples:**

Basic offloading: >>> s = Scenario(\{“base64\_string”: “SGVsbG8gV29ybGQ=”, “name”: “test”}) >>> offloaded = s.offload() >>> offloaded\[“base64\_string”] ‘offloaded’ >>> offloaded\[“name”] ‘test’

In-place offloading: >>> s = Scenario(\{“base64\_string”: “SGVsbG8gV29ybGQ=”, “name”: “test”}) >>> result = s.offload(inplace=True) >>> result is s True >>> s\[“base64\_string”] ‘offloaded’

> #### open\_url(*position: int = 0*) → None [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Open a URL field from the scenario in the default web browser.

> **Args:**

position: The index of the URL to open (0-based). Defaults to 0 for the first URL.

> **Raises:**

ValueError: If no URL fields are found in the scenario, or if the position

is out of range.

> **Examples:**

```python theme={null}
>>> s = Scenario({"website": "https://example.com", "name": "test"})
>>> s.open_url()  # Opens the first URL found
```

> #### rename(*old\_name\_or\_replacement\_dict: str | Dict\[str, str]*, *new\_name: str | None = None*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Rename the keys of a scenario.

> **Args:**

old\_name\_or\_replacement\_dict: Either a dictionary mapping old keys to new keys,

or a string representing the old key name.

new\_name: The new name for the key. Required if old\_name\_or\_replacement\_dict

is a string, ignored if it’s a dictionary.

> **Returns:**

A new Scenario with renamed keys.

> **Raises:**

TypeError: If old\_name\_or\_replacement\_dict is a string but new\_name is None.

> **Examples:**

Using a dictionary: >>> s = Scenario(\{“food”: “wood chips”}) >>> s.rename(\{“food”: “food\_preference”}) Scenario(\{‘food\_preference’: ‘wood chips’})

Using individual arguments: >>> s = Scenario(\{“food”: “wood chips”}) >>> s.rename(“food”, “snack”) Scenario(\{‘snack’: ‘wood chips’})

> #### replicate(*n: int*) → [ScenarioList](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Replicate a scenario n times to return a ScenarioList.

> **Args:**

n: The number of times to replicate the scenario. Must be non-negative.

> **Returns:**

A ScenarioList containing n copies of this scenario.

> **Raises:**

ValueError: If n is negative.

> **Examples:**

```python theme={null}
>>> s = Scenario({"food": "wood chips"})
>>> s.replicate(2)
ScenarioList([Scenario({'food': 'wood chips'}), Scenario({'food': 'wood chips'})])
```

> #### save\_to\_gcs\_bucket(*signed\_url\_or\_dict: str | Dict\[str, str]*) → Dict\[str, Any] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Saves FileStore objects contained within this Scenario to a Google Cloud Storage bucket.

This method finds all FileStore objects in the Scenario and uploads them to GCS using the provided signed URL(s). If the Scenario itself was created from a FileStore (has base64\_string as a top-level key), it uploads that content directly.

> **Args:**

signed\_url\_or\_dict: Either:

* str: Single signed URL (for single FileStore or Scenario from FileStore)
* **dict: Mapping of scenario keys to signed URLs for multiple FileStore objects**
  e.g., \{“video”: “signed\_url\_1”, “image”: “signed\_url\_2”}

> **Returns:**

dict: Summary of upload operations performed

> **Raises:**

ValueError: If no uploadable content found or content is offloaded requests.RequestException: If any upload fails

> #### select(\*\*args: str | Iterable\[str]\*) → [Scenario](/en/latest/scenarios#scenario-methods) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Select a subset of keys from a scenario.

This method delegates to ScenarioSelector for the actual selection logic. It supports both individual string arguments and collection arguments for backward compatibility.

> **Args:**

[\*](/en/latest/#id7)args: Either a single collection of keys (for backward compatibility)

or individual string arguments for keys to select.

> **Returns:**

A new Scenario containing only the selected keys and their values.

> **Raises:**

KeyError: If any of the specified keys don’t exist in the scenario. ValueError: If no arguments are provided.

> **Examples:**

Using a list (backward compatible): >>> s = Scenario(\{“food”: “wood chips”, “drink”: “water”}) >>> s.select(\[“food”]) Scenario(\{‘food’: ‘wood chips’})

Using individual string arguments: >>> s = Scenario(\{“food”: “wood chips”, “drink”: “water”, “dessert”: “cookies”}) >>> s.select(“food”, “drink”) Scenario(\{‘food’: ‘wood chips’, ‘drink’: ‘water’})

Single string argument: >>> s.select(“food”) Scenario(\{‘food’: ‘wood chips’})

> #### table(*tablefmt: str = 'grid'*) → str [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Display a scenario as a formatted table.

> **Args:**

tablefmt: The table format to use. Common options include “grid”,

“simple”, “pipe”, “orgtbl”, “rst”, “mediawiki”, “html”, “latex”.

> **Returns:**

A string representation of the scenario formatted as a table.

> **Examples:**

```python theme={null}
>>> s = Scenario({"food": "chips", "drink": "water"})
>>> print(s.table("simple"))
key    value
-----  -------
food   chips
drink  water
```

> #### to(*question\_or\_survey: 'Question' | 'Survey'*) → [Jobs](/en/latest/jobs#jobs-class") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Send the scenario to a question or survey for execution.

> **Args:**

question\_or\_survey: A Question or Survey object to parameterize with this scenario.

> **Returns:**

A Jobs object that can be run to execute the question or survey with this scenario.

> **Examples:**

```python theme={null}
>>> from edsl.questions import QuestionFreeText
>>> s = Scenario({"name": "Alice"})
>>> q = QuestionFreeText(question_name="greeting", question_text="Hello {{name}}")
>>> jobs = s.to(q)
```

> #### to\_dataset() → [Dataset](/en/latest/dataset#dataset-class") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert a scenario to a dataset.

```python theme={null}
>>> s = Scenario({"food": "wood chips"})
>>> s.to_dataset()
Dataset([{'key': ['food']}, {'value': ['wood chips']}])
```

> #### to\_dict(*add\_edsl\_version: bool = True*, *offload\_base64: bool = False*) → Dict\[str, Any] [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert a scenario to a dictionary.

> **Args:**

add\_edsl\_version: If True, adds the EDSL version to the returned dictionary. offload\_base64: If True, replaces any base64\_string fields with ‘offloaded’

to reduce memory usage.

Example:

```python theme={null}
>>> s = Scenario({"food": "wood chips"})
>>> s.to_dict()
{'food': 'wood chips', 'edsl_version': '...', 'edsl_class_name': 'Scenario'}
```

```python theme={null}
>>> s.to_dict(add_edsl_version = False)

{'food': 'wood chips'}
```

## ScenarioList class

> #### *class* edsl.scenarios.ScenarioList(*data: list | None = None*, *codebook: dict\[str*, *str] | None = None*, *data\_class: type | None = \<class 'list'>*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Bases: `MutableSequence`, [`Base`](/en/latest/base#base-class") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py), `ScenarioListOperationsMixin`

A collection of Scenario objects with advanced operations for manipulation and analysis.

ScenarioList provides specialized functionality for working with collections of Scenario objects. It inherits from MutableSequence to provide standard list operations, from Base to integrate with EDSL’s object model, and from ScenarioListOperationsMixin to provide powerful data manipulation capabilities.

**Attributes:**

data (list): The underlying list containing Scenario objects. codebook (dict): Optional metadata describing the fields in the scenarios.

> #### **init**(*data: list | None = None*, *codebook: dict\[str*, *str] | None = None*, *data\_class: type | None = \<class 'list'>*) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Initialize a new ScenarioList with optional data and codebook.

> #### add\_list(*name: str*, *values: List\[Any]*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Add a list of values to a ScenarioList.

Example:

```python theme={null}
>>> s = ScenarioList([Scenario({'name': 'Alice'}), Scenario({'name': 'Bob'})])
>>> s.add_list('age', [30, 25])
ScenarioList([Scenario({'name': 'Alice', 'age': 30}), Scenario({'name': 'Bob', 'age': 25})])
```

> #### add\_value(*name: str*, *value: Any*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Add a value to all scenarios in a ScenarioList.

Example:

```python theme={null}
>>> s = ScenarioList([Scenario({'name': 'Alice'}), Scenario({'name': 'Bob'})])
>>> s.add_value('age', 30)
ScenarioList([Scenario({'name': 'Alice', 'age': 30}), Scenario({'name': 'Bob', 'age': 30})])
```

> #### apply(*func: Callable*, *field: str*, *new\_name: str | None*, *replace: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Apply a function to a field and return a new ScenarioList.

> #### at(*index: int*) → [Scenario](/en/latest/scenarios#scenario-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Get the scenario at the specified index position. >>> sl = ScenarioList.from\_list(“a”, \[1, 2, 3]) >>> sl.at(0) Scenario(\{‘a’: 1}) >>> sl.at(-1) Scenario(\{‘a’: 3})

> #### augment\_with\_wikipedia(*search\_key: str*, *content\_only: bool = True*, *key\_name: str = 'wikipedia\_content'*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Augment the ScenarioList with Wikipedia content.

> #### chart() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a chart from the results.

> #### chunk(*field*, *num\_words: int | None = None*, *num\_lines: int | None = None*, *include\_original=False*, *hash\_original=False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Chunk the scenarios based on a field.

> **Examples:**

```python theme={null}
>>> s = ScenarioList([Scenario({'text': 'The quick brown fox jumps over the lazy dog.'})])
>>> s.chunk('text', num_words=3)
ScenarioList([Scenario({'text': 'The quick brown', 'text_chunk': 0, 'text_char_count': 15, 'text_word_count': 3}), Scenario({'text': 'fox jumps over', 'text_chunk': 1, 'text_char_count': 14, 'text_word_count': 3}), Scenario({'text': 'the lazy dog.', 'text_chunk': 2, 'text_char_count': 13, 'text_word_count': 3})])
```

> #### clipboard\_data() → str [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return TSV representation of this object for clipboard operations.

This method is called by the clipboard() method in the base class to provide a custom format for copying objects to the system clipboard.

**Returns:**

str: Tab-separated values representation of the object

> #### code() → str [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create the Python code representation of a survey.

> #### collapse(*field: str*, *separator: str | None = None*, *prefix: str = ''*, *postfix: str = ''*, *add\_count: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Collapse by grouping on all fields except the specified one.

Delegates to *ScenarioListTransformer.collapse*.

> #### concatenate(*fields: List\[str]*, *separator: str = ';'*, *prefix: str = ''*, *postfix: str = ''*, *new\_field\_name: str | None = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Concatenate specified fields into a single string field.

Delegates to *ScenarioListTransformer.concatenate* for implementation.

> #### concatenate\_to\_list(*fields: List\[str]*, *prefix: str = ''*, *postfix: str = ''*, *new\_field\_name: str | None = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Concatenate specified fields into a single list field.

Delegates to *ScenarioListTransformer.concatenate\_to\_list* for implementation.

> #### concatenate\_to\_set(*fields: List\[str]*, *prefix: str = ''*, *postfix: str = ''*, *new\_field\_name: str | None = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Concatenate specified fields into a single set field.

Delegates to *ScenarioListTransformer.concatenate\_to\_set* for implementation.

> #### copy() [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a copy of this ScenarioList.

**Returns:**

A new ScenarioList with copies of the same scenarios

> #### create\_comparisons(*bidirectional: bool = False*, *num\_options: int = 2*, *option\_prefix: str = 'option\_'*, *use\_alphabet: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a new ScenarioList with comparisons between scenarios.

Delegates to *ScenarioListTransformer.create\_comparisons*.

> #### create\_conjoint\_comparisons(*attribute\_field: str = 'attribute'*, *levels\_field: str = 'levels'*, *count: int = 1*, *random\_seed: int | None = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py))

Generate random product profiles for conjoint analysis from attribute definitions.

This method uses the current ScenarioList (which should contain attribute definitions) to create random product profiles by sampling from the attribute levels. Each scenario in the current list should represent one attribute with its possible levels.

> **Args:**

attribute\_field: Field name containing the attribute names (default: ‘attribute’) levels\_field: Field name containing the list of levels (default: ‘levels’) count: Number of product profiles to generate (default: 1) random\_seed: Optional seed for reproducible random sampling

> **Returns:**

ScenarioList containing randomly generated product profiles

> **Examples:**

```python theme={null}
>>> from edsl.scenarios import ScenarioList, Scenario
>>> # Create attribute definitions
>>> attributes = ScenarioList([
...     Scenario({'attribute': 'price', 'levels': ['$100', '$200', '$300']}),
...     Scenario({'attribute': 'color', 'levels': ['Red', 'Blue', 'Green']}),
...     Scenario({'attribute': 'size', 'levels': ['Small', 'Medium', 'Large']})
... ])
>>> # Generate conjoint profiles
>>> profiles = attributes.create_conjoint_comparisons(count=3, random_seed=42)
>>> len(profiles)
3
>>> # Each profile will have price, color, and size with random values
```

> **Raises:**

ScenarioError: If the current ScenarioList doesn’t have the required fields ValueError: If count is not positive

> #### create\_empty\_scenario\_list(*n: int*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create an empty ScenarioList with n scenarios.

> **Args:**

n: The number of empty scenarios to create

> **Examples:**

```python theme={null}
>>> ScenarioList.create_empty_scenario_list(3)
ScenarioList([Scenario({}), Scenario({}), Scenario({})])
```

> #### drop(\**fields: str*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Drop fields from the scenarios.

Example:

```python theme={null}
>>> s = ScenarioList([Scenario({'a': 1, 'b': 1}), Scenario({'a': 1, 'b': 2})])
>>> s.drop('a')
ScenarioList([Scenario({'b': 1}), Scenario({'b': 2})])
```

> #### duplicate() → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return a copy of the ScenarioList using streaming to avoid loading everything into memory.

```python theme={null}
>>> sl = ScenarioList.example()
>>> sl_copy = sl.duplicate()
>>> sl == sl_copy
True
>>> sl is sl_copy
False
```

> #### equals(*other: Any*) → bool [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Memory-efficient comparison of two ScenarioLists.

> #### *classmethod* example(*randomize: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return an example ScenarioList instance.

**Params randomize:**

If True, use Scenario’s randomize method to randomize the values.

> #### expand(*expand\_field: str*, *number\_field: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Expand the ScenarioList by a field.

**Parameters:**

* **expand\_field** – The field to expand.
* **number\_field** – Whether to add a field with the index of the value

Example:

```python theme={null}
>>> s = ScenarioList( [ Scenario({'a':1, 'b':[1,2]}) ] )
>>> s.expand('b')
ScenarioList([Scenario({'a': 1, 'b': 1}), Scenario({'a': 1, 'b': 2})])
>>> s.expand('b', number_field=True)
ScenarioList([Scenario({'a': 1, 'b': 1, 'b_number': 1}), Scenario({'a': 1, 'b': 2, 'b_number': 2})])
```

> #### fillna(*value: Any = ''*, *inplace: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Fill None/NaN values in all scenarios with a specified value.

This method is equivalent to pandas’ df.fillna() functionality, allowing you to replace None, NaN, or other null-like values across all scenarios in the list.

> **Args:**

value: The value to use for filling None/NaN values. Defaults to empty string “”. inplace: If True, modify the original ScenarioList. If False (default),

return a new ScenarioList with filled values.

> **Returns:**

ScenarioList: A new ScenarioList with filled values, or self if inplace=True

> **Examples:**

```python theme={null}
>>> scenarios = ScenarioList([
...     Scenario({'a': None, 'b': 1, 'c': 'hello'}),
...     Scenario({'a': 2, 'b': None, 'c': None}),
...     Scenario({'a': None, 'b': 3, 'c': 'world'})
... ])
>>> # Fill None values with empty string (default)
>>> filled = scenarios.fillna()
>>> print(filled)
ScenarioList([Scenario({'a': '', 'b': 1, 'c': 'hello'}), Scenario({'a': 2, 'b': '', 'c': ''}), Scenario({'a': '', 'b': 3, 'c': 'world'})])
>>> # Fill with custom value
>>> filled_custom = scenarios.fillna(value="N/A")
>>> print(filled_custom)
ScenarioList([Scenario({'a': 'N/A', 'b': 1, 'c': 'hello'}), Scenario({'a': 2, 'b': 'N/A', 'c': 'N/A'}), Scenario({'a': 'N/A', 'b': 3, 'c': 'world'})])
>>> # Original scenarios remain unchanged
>>> print(scenarios)
ScenarioList([Scenario({'a': None, 'b': 1, 'c': 'hello'}), Scenario({'a': 2, 'b': None, 'c': None}), Scenario({'a': None, 'b': 3, 'c': 'world'})])
>>> # Modify in place
>>> _ = scenarios.fillna(value="MISSING", inplace=True)
>>> print(scenarios)
ScenarioList([Scenario({'a': 'MISSING', 'b': 1, 'c': 'hello'}), Scenario({'a': 2, 'b': 'MISSING', 'c': 'MISSING'}), Scenario({'a': 'MISSING', 'b': 3, 'c': 'world'})])
```

> #### filter(*expression: str*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Filter a list of scenarios based on an expression.

Delegates to *ScenarioListTransformer.filter*.

> #### firecrawl *= \<edsl.scenarios.firecrawl\_scenario.FirecrawlRequest object>* [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### flatten(*field: str*, *keep\_original: bool = False*) → [Dataset](/en/latest/dataset#edsl.dataset.Dataset "edsl.dataset.Dataset") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Expand a field containing dictionaries into separate fields.

This method takes a field that contains a list of dictionaries and expands it into multiple fields, one for each key in the dictionaries. This is useful when working with nested data structures or results from extraction operations.

> **Parameters:**

field: The field containing dictionaries to flatten keep\_original: Whether to retain the original field in the result

> **Returns:**

A new Dataset with the dictionary keys expanded into separate fields

> **Notes:**

* Each key in the dictionaries becomes a new field with name pattern “.”
* All dictionaries in the field must have compatible structures
* If a dictionary is missing a key, the corresponding value will be None
* Non-dictionary values in the field will cause a warning

> **Examples:**

```python theme={null}
>>> from edsl.dataset import Dataset
```

```sql theme={null}
# Basic flattening of nested dictionaries >>> Dataset([{‘a’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}]).flatten(‘a’) Dataset([{‘c’: [5]}, {‘a.a’: [1]}, {‘a.b’: [2]}])


# Works with prefixed fields too >>> Dataset([{‘answer.example’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}]).flatten(‘answer.example’) Dataset([{‘c’: [5]}, {‘answer.example.a’: [1]}, {‘answer.example.b’: [2]}])

# Keep the original field if needed >>> d = Dataset([{‘a’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}]) >>> d.flatten(‘a’, keep_original=True) Dataset([{‘a’: [{‘a’: 1, ‘b’: 2}]}, {‘c’: [5]}, {‘a.a’: [1]}, {‘a.b’: [2]}])

# Can also use unambiguous unprefixed field name >>> result = Dataset([{‘answer.pros_cons’: [{‘pros’: [‘Safety’], ‘cons’: [‘Cost’]}]}]).flatten(‘pros_cons’) >>> sorted(result.keys()) == [‘answer.pros_cons.cons’, ‘answer.pros_cons.pros’] True >>> sorted(result.to_dicts()[0].items()) == sorted({‘cons’: [‘Cost’], ‘pros’: [‘Safety’]}.items()) True
```

> #### for\_n(*target: 'Question' | 'Survey' | 'Job'*, *iterations: int*) → [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs") [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Execute a target multiple times, feeding each iteration’s output into the next.

### Parameters

**targetQuestion | Survey | Job**

The object to be executed on each round. A fresh `duplicate()` of *target* is taken for every iteration so that state is **not** shared between runs.

**iterationsint**

How many times to run *target*.

### Returns

**Jobs**

A [`Jobs`](/en/latest/jobs#jobs-class") instance containing the results of the final iteration.

```python expandable theme={null}
Example (non-doctest):

from edsl import ScenarioList, QuestionFreeText

base_personas = ScenarioList.from_list(
    "persona",
    [
        "- Likes basketball",
        "- From Germany",
        "- Once owned a sawmill",
    ],
)

persona_detail_jobs = (
    QuestionFreeText(
        question_text=(
            "Take this persona: {{ scenario.persona }} and add one additional detail, "
            "preserving the original details."
        ),
        question_name="enhance",
    )
    .to_jobs()
    .select("enhance")
    .to_scenario_list()
    .rename({"enhance": "persona"})
)

# Run the enrichment five times
enriched_personas = base_personas.for_n(persona_detail_jobs, 5)

print(enriched_personas.select("persona"))
```

> #### *classmethod* from\_csv(*source: str | 'ParseResult'*, *has\_header: bool = True*, *encoding: str = 'utf-8'*, \*\**kwargs*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a ScenarioList from a CSV file or URL.

> **Args:**

source: Path to a local file or URL to a remote file. has\_header: Whether the file has a header row (default is True). encoding: The file encoding to use (default is ‘utf-8’). [\*\*](/en/latest/#id9)kwargs: Additional parameters for csv reader.

> **Returns:**

ScenarioList: An instance of the ScenarioList class.

> #### *classmethod* from\_delimited\_file(*source: str | 'ParseResult'*, *delimiter: str = ','*, *encoding: str = 'utf-8'*, \*\**kwargs*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a ScenarioList from a delimited file (CSV/TSV) or URL.

> **Args:**

source: Path to a local file or URL to a remote file. delimiter: The delimiter character used in the file (default is ‘,’). encoding: The file encoding to use (default is ‘utf-8’). [\*\*](/en/latest/#id11)kwargs: Additional parameters for csv reader.

> **Returns:**

ScenarioList: An instance of the ScenarioList class.

> #### *classmethod* from\_dict(*data: dict*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a ScenarioList from a dictionary.

```python theme={null}
>>> d = {'scenarios': [{'food': 'wood chips'}], 'codebook': {'food': 'description'}}
>>> s = ScenarioList.from_dict(d)
>>> s.codebook == {'food': 'description'}
True
>>> s[0]['food']
'wood chips'
```

> #### *classmethod* from\_directory(*path: str | None = None*, *recursive: bool = False*, *key\_name: str = 'content'*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a ScenarioList of Scenario objects from files in a directory.

This method scans a directory and creates a Scenario object for each file found, where each Scenario contains a FileStore object under the specified key. Optionally filters files based on a wildcard pattern. If no path is provided, the current working directory is used.

> **Args:**

**path: The directory path to scan, optionally including a wildcard pattern.**

If None, uses the current working directory. Examples: - “/path/to/directory” - scans all files in the directory - “/path/to/directory/*.py” - scans only Python files in the directory - “*.txt” - scans only text files in the current working directory

recursive: Whether to scan subdirectories recursively. Defaults to False. key\_name: The key to use for the FileStore object in each Scenario. Defaults to “content”.

> **Returns:**

A ScenarioList containing Scenario objects for all matching files, where each Scenario has a FileStore object under the specified key.

> **Raises:**

FileNotFoundError: If the specified directory does not exist.

> **Examples:**

```python theme={null}
# Get all files in the current directory with default key “content” sl = ScenarioList.from_directory()

# Get all Python files in a specific directory with custom key “python_file” sl = ScenarioList.from_directory(’[*](/en/latest/#id13).py’, key_name=”python_file”)

# Get all image files in the current directory sl = ScenarioList.from_directory(’[*](/en/latest/#id15).png’, key_name=”image”)

# Get all files recursively including subdirectories sl = ScenarioList.from_directory(recursive=True, key_name=”document”)
```

> #### *classmethod* from\_list(*field\_name: str*, *values: list*, *use\_indexes: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a ScenarioList from a list of values with a specified field name.

```python theme={null}
>>> ScenarioList.from_list('text', ['a', 'b', 'c'])
ScenarioList([Scenario({'text': 'a'}), Scenario({'text': 'b'}), Scenario({'text': 'c'})])
```

> #### *classmethod* from\_nested\_dict(*data: dict*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a ScenarioList from a nested dictionary.

```python theme={null}
>>> data = {"headline": ["Armistice Signed, War Over: Celebrations Erupt Across City"], "date": ["1918-11-11"], "author": ["Jane Smith"]}
>>> ScenarioList.from_nested_dict(data)
ScenarioList([Scenario({'headline': 'Armistice Signed, War Over: Celebrations Erupt Across City', 'date': '1918-11-11', 'author': 'Jane Smith'})])
```

> #### *classmethod* from\_prompt(*description: str*, *name: str | None = 'item'*, *target\_number: int = 10*, *verbose=False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### *classmethod* from\_search\_terms(*search\_terms: List\[str]*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a ScenarioList from a list of search terms, using Wikipedia.

**Args:**

search\_terms: A list of search terms.

> #### *classmethod* from\_source(*source\_type: str*, *\*args*, *\*\*kwargs*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a ScenarioList from a specified source type.

This method serves as the main entry point for creating ScenarioList objects, providing a unified interface for various data sources.

**Args:**

**source\_type: The type of source to create a ScenarioList from.**

Valid values include: ‘urls’, ‘directory’, ‘csv’, ‘tsv’, ‘excel’, ‘pdf’, ‘pdf\_to\_image’, and others.

[\*](/en/latest/#id13)args: Positional arguments to pass to the source-specific method. [\*\*](/en/latest/#id15)kwargs: Keyword arguments to pass to the source-specific method.

**Returns:**

A ScenarioList object created from the specified source.

**Examples:**

```python theme={null}
>>> # This is a simplified example for doctest
>>> # In real usage, you would provide a path to your CSV file:
>>> # sl_csv = ScenarioList.from_source('csv', 'your_data.csv')
>>> # Or use other source types like 'directory', 'excel', etc.
>>> # Examples of other source types:
>>> # sl_dir = ScenarioList.from_source('directory', '/path/to/files')
```

> #### *classmethod* from\_urls(*urls: list\[str]*, *field\_name: str | None = 'text'*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### *classmethod* gen(*scenario\_dicts\_list: List\[dict]*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a ScenarioList from a list of dictionaries.

**Example:**

```python theme={null}
>>> ScenarioList.gen([{'name': 'Alice'}, {'name': 'Bob'}])
ScenarioList([Scenario({'name': 'Alice'}), Scenario({'name': 'Bob'})])
```

> #### get\_tabular\_data(*remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*) → Tuple\[List\[str], List\[List]][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Internal method to get tabular data in a standard format.

**Args:**

remove\_prefix: Whether to remove the prefix from column names pretty\_labels: Dictionary mapping original column names to pretty labels

**Returns:**

Tuple containing (header\_row, data\_rows)

> #### ggplot2(*ggplot\_code: str*, *shape: str = 'wide'*, *sql: str | None = None*, *remove\_prefix: bool = True*, *debug: bool = False*, *height: float = 4*, *width: float = 6*, *factor\_orders: dict | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create visualizations using R’s ggplot2 library.

This method provides a bridge to R’s powerful ggplot2 visualization library, allowing you to create sophisticated plots directly from EDSL data structures.

> **Parameters:**

ggplot\_code: R code string containing ggplot2 commands shape: Data shape to use (“wide” or “long”) sql: Optional SQL query to transform data before visualization remove\_prefix: Whether to remove prefixes (like “answer.”) from column names debug: Whether to display debugging information height: Plot height in inches width: Plot width in inches factor\_orders: Dictionary mapping factor variables to their desired order

> **Returns:**

A plot object that renders in Jupyter notebooks

> **Notes:**

* Requires R and the ggplot2 package to be installed
* Data is automatically converted to a format suitable for ggplot2
* The ggplot2 code should reference column names as they appear after any transformations from the shape and remove\_prefix parameters

Examples:

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> # The following would create a plot if R is installed (not shown in doctest):
>>> # r.ggplot2('''
>>> #     ggplot(df, aes(x=how_feeling)) +
>>> #     geom_bar() +
>>> #     labs(title="Distribution of Feelings")
>>> # ''')
```

> #### give\_valid\_names(*existing\_codebook: dict = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Give valid names to the scenario keys, using an existing codebook if provided.

**Args:**

**existing\_codebook (dict, optional): Existing mapping of original keys to valid names.**

Defaults to None.

**Returns:**

ScenarioList: A new ScenarioList with valid variable names and updated codebook.

```python theme={null}
>>> s = ScenarioList([Scenario({'a': 1, 'b': 2}), Scenario({'a': 1, 'b': 1})])
>>> s.give_valid_names()
ScenarioList([Scenario({'a': 1, 'b': 2}), Scenario({'a': 1, 'b': 1})])
>>> s = ScenarioList([Scenario({'are you there John?': 1, 'b': 2}), Scenario({'a': 1, 'b': 1})])
>>> s.give_valid_names()
ScenarioList([Scenario({'john': 1, 'b': 2}), Scenario({'a': 1, 'b': 1})])
>>> s.give_valid_names({'are you there John?': 'custom_name'})
ScenarioList([Scenario({'custom_name': 1, 'b': 2}), Scenario({'a': 1, 'b': 1})])
```

> #### group\_by(*id\_vars: List\[str]*, *variables: List\[str]*, *func: Callable*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Group the ScenarioList by id\_vars and apply a function.

Delegates to *ScenarioListTransformer.group\_by*.

> #### *property* has\_jinja\_braces\*: bool\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Check if any Scenario in the list contains values with Jinja template braces.

This property checks all Scenarios in the list to determine if any contain string values with Jinja template syntax (`{{` and `}}`). This is important for rendering templates and avoiding conflicts with other templating systems.

> **Returns:**

True if any Scenario contains values with Jinja braces, False otherwise.

> Examples:

```python theme={null}
>>> from edsl.scenarios import Scenario, ScenarioList
>>> s1 = Scenario({"text": "Plain text"})
>>> s2 = Scenario({"text": "Template with {{variable}}"})
>>> sl1 = ScenarioList([s1])
>>> sl1.has_jinja_braces
False
>>> sl2 = ScenarioList([s1, s2])
>>> sl2.has_jinja_braces
True
```

> #### html(*filename: str | None = None*, *cta: str = 'Open in browser'*, *return\_link: bool = False*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### inner\_join(other: [ScenarioList](/en/latest/scenarios#scenariolist-class), by: str | list\[str]) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Perform an inner join with another ScenarioList, following SQL join semantics.

**Args:**

other: The ScenarioList to join with by: String or list of strings representing the key(s) to join on. Cannot be empty.

**Returns:**

A new ScenarioList containing only scenarios that have matches in both ScenarioLists

```python theme={null}
>>> s1 = ScenarioList([Scenario({'name': 'Alice', 'age': 30}), Scenario({'name': 'Bob', 'age': 25})])
>>> s2 = ScenarioList([Scenario({'name': 'Alice', 'location': 'New York'}), Scenario({'name': 'Charlie', 'location': 'Los Angeles'})])
>>> s4 = s1.inner_join(s2, 'name')
>>> s4 == ScenarioList([Scenario({'age': 30, 'location': 'New York', 'name': 'Alice'})])
True
```

> #### insert(*index*, *value*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Insert value at index.

> #### is\_serializable()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### items()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Make this class compatible with dict.items() by accessing first scenario items.

This ensures the class works as a drop-in replacement for UserList in code that expects a dictionary-like interface.

**Returns:**

items view from the first scenario object if available, empty list otherwise

> ### keep(*\*fields: str*) → [ScenarioList](/en/latest/scenarios#scenariolist-class)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Keep only the specified fields in the scenarios.

**Parameters:**

**fields** – The fields to keep.

Example:

```python theme={null}
>>> s = ScenarioList([Scenario({'a': 1, 'b': 1}), Scenario({'a': 1, 'b': 2})])
>>> s.keep('a')
ScenarioList([Scenario({'a': 1}), Scenario({'a': 1})])
```

> #### left\_join(*other: [ScenarioList](/en/latest/scenarios#scenariolist-class)*, *by: str | list\[str]*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [source](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Perform a left join with another ScenarioList, following SQL join semantics.

**Args:**

other: The ScenarioList to join with by: String or list of strings representing the key(s) to join on. Cannot be empty.

```python theme={null}
>>> s1 = ScenarioList([Scenario({'name': 'Alice', 'age': 30}), Scenario({'name': 'Bob', 'age': 25})])
>>> s2 = ScenarioList([Scenario({'name': 'Alice', 'location': 'New York'}), Scenario({'name': 'Charlie', 'location': 'Los Angeles'})])
>>> s3 = s1.left_join(s2, 'name')
>>> s3 == ScenarioList([Scenario({'age': 30, 'location': 'New York', 'name': 'Alice'}), Scenario({'age': 25, 'location': None, 'name': 'Bob'})])
True
```

> #### make\_tabular(*remove\_prefix: bool*, *pretty\_labels: dict | None = None*) → tuple\[list, List\[list]] [source](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Turn the results into a tabular format.

**Parameters:** **remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').make_tabular(remove_prefix = True)
(['how_feeling'], [['OK'], ['Great'], ['Terrible'], ['OK']])
```

```python theme={null}
>>> r.select('how_feeling').make_tabular(remove_prefix = True, pretty_labels = {'how_feeling': "How are you feeling"})
(['How are you feeling'], [['OK'], ['Great'], ['Terrible'], ['OK']])
```

> #### mutate(*new\_var\_string: str*, *functions\_dict: dict\[str, Callable] | None = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [source](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return a new ScenarioList with a new variable added.

Delegates to *ScenarioListTransformer.mutate*.

> #### num\_observations()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return the number of observations in the dataset.

```python theme={null}
>>> from edsl.results import Results
>>> Results.example().num_observations()
4
```

> #### offload(*inplace: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [source](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Offloads base64-encoded content from all scenarios in the list by replacing ‘base64\_string’ fields with ‘offloaded’. This reduces memory usage.

> **Args:**

inplace (bool): If True, modify the current scenario list. If False, return a new one.

> **Returns:**

ScenarioList: The modified scenario list (either self or a new instance).

> #### order\_by(*\*fields: str*, *reverse: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [source](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Order the scenarios by one or more fields.

Delegates to ScenarioListTransformer.order\_by.

> #### *property* parameters\*: set\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return the set of parameters in the ScenarioList

Example:

```python theme={null}
>>> s = ScenarioList([Scenario({'a': 1}), Scenario({'b': 2})])
>>> s.parameters == {'a', 'b'}
True
```

> #### pivot(*id\_vars: List\[str] = None*, *var\_name='variable'*, *value\_name='value'*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [source](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Pivot the ScenarioList from long to wide format.

Delegates to ScenarioListTransformer.pivot.

> #### print\_long()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Print the results in a long format. >>> from edsl.results import Results >>> r = Results.example() >>> r.select(‘how\_feeling’).print\_long() answer.how\_feeling: OK answer.how\_feeling: Great answer.how\_feeling: Terrible answer.how\_feeling: OK

> #### relevant\_columns(*data\_type: str | None = None*, *remove\_prefix: bool = False*) → list[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return the set of keys that are present in the dataset.

**Parameters:**

* **data\_type** – The data type to filter by.
* **remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from ..dataset import Dataset
>>> d = Dataset([{'a.b':[1,2,3,4]}])
>>> d.relevant_columns()
['a.b']
```

```python theme={null}
>>> d.relevant_columns(remove_prefix=True)
['b']
```

```python theme={null}
>>> d = Dataset([{'a':[1,2,3,4]}, {'b':[5,6,7,8]}])
>>> d.relevant_columns()
['a', 'b']
```

```python theme={null}
>>> from edsl.results import Results; Results.example().select('how_feeling', 'how_feeling_yesterday').relevant_columns()
['answer.how_feeling', 'answer.how_feeling_yesterday']
```

```python theme={null}
>>> from edsl.results import Results
>>> sorted(Results.example().select().relevant_columns(data_type = "model"))
['model.canned_response', 'model.inference_service', 'model.model', 'model.model_index', 'model.temperature']
```

```python theme={null}
>>> # Testing relevant_columns with invalid data_type raises DatasetValueError - tested in unit tests
```

> #### remove\_prefix()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Returns a new Dataset with the prefix removed from all column names.

The prefix is defined as everything before the first dot (.) in the column name. If removing prefixes would result in duplicate column names, an exception is raised.

> **Returns:**

Dataset: A new Dataset with prefixes removed from column names

> **Raises:**

ValueError: If removing prefixes would result in duplicate column names

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling', 'how_feeling_yesterday').relevant_columns()
['answer.how_feeling', 'answer.how_feeling_yesterday']
>>> r.select('how_feeling', 'how_feeling_yesterday').remove_prefix().relevant_columns()
['how_feeling', 'how_feeling_yesterday']
```

```python theme={null}
>>> from edsl.dataset import Dataset
>>> d = Dataset([{'a.x': [1, 2, 3]}, {'b.x': [4, 5, 6]}])
>>> # d.remove_prefix()
```

\# Testing remove\_prefix with duplicate column names raises DatasetValueError - tested in unit tests

> #### rename(*replacement\_dict: dict*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Rename the fields in the scenarios.

**Parameters:** **replacement\_dict** – A dictionary with the old names as keys and the new names as values.

Example:

```python theme={null}
>>> s = ScenarioList([Scenario({'name': 'Alice', 'age': 30}), Scenario({'name': 'Bob', 'age': 25})])
>>> s.rename({'name': 'first_name', 'age': 'years'})
ScenarioList([Scenario({'first_name': 'Alice', 'years': 30}), Scenario({'first_name': 'Bob', 'years': 25})])
```

> #### reorder\_keys(*new\_order: List\[str]*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Reorder the keys in the scenarios.

Delegates to *ScenarioListTransformer.reorder\_keys*.

> #### replace\_names(*new\_names: list*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Replace the field names in the scenarios with a new list of names.

> **Parameters:**

**new\_names** – A list of new field names to use.

> **Examples:**

```python theme={null}
>>> s = ScenarioList([Scenario({'name': 'Alice', 'age': 30}), Scenario({'name': 'Bob', 'age': 25})])
>>> s.replace_names(['first_name', 'years'])
ScenarioList([Scenario({'first_name': 'Alice', 'years': 30}), Scenario({'first_name': 'Bob', 'years': 25})])
```

> #### replace\_values(*replacements: dict*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create new scenarios with values replaced according to the provided replacement dictionary.

> **Args:**

replacements (dict): Dictionary of values to replace \{old\_value: new\_value}

> **Returns:**

ScenarioList: A new ScenarioList with replaced values

> **Examples:**

```python theme={null}
>>> scenarios = ScenarioList([
...     Scenario({'a': 'nan', 'b': 1}),
...     Scenario({'a': 2, 'b': 'nan'})
... ])
>>> replaced = scenarios.replace_values({'nan': None})
>>> print(replaced)
ScenarioList([Scenario({'a': None, 'b': 1}), Scenario({'a': 2, 'b': None})])
>>> # Original scenarios remain unchanged
>>> print(scenarios)
ScenarioList([Scenario({'a': 'nan', 'b': 1}), Scenario({'a': 2, 'b': 'nan'})])
```

> #### report(*\*fields: str | None*, *top\_n: int | None = None*, *header\_fields: List\[str] | None = None*, *divider: bool = True*, *return\_string: bool = False*, *format: str = 'markdown'*, *filename: str | None = None*) → str | Document | None [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Generates a report of the results by iterating through rows.

> **Args:**

[\*](/en/latest/#id17)fields: The fields to include in the report. If none provided, all fields are used. top\_n: Optional limit on the number of observations to include. header\_fields: Optional list of fields to include in the main header instead of as sections. divider: If True, adds a horizontal rule between observations (markdown only). return\_string: If True, returns the markdown string. If False (default in notebooks),

only displays the markdown without returning.

format: Output format - either “markdown” or “docx”. filename: If provided and format is “docx”, saves the document to this file.

> **Returns:**

Depending on format and return\_string: - For markdown: A string if return\_string is True, otherwise None (displays in notebook) - For docx: A docx.Document object, or None if filename is provided (saves to file)

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> report = r.select('how_feeling').report(return_string=True)
>>> "# Observation: 1" in report
True
>>> doc = r.select('how_feeling').report(format="docx")
>>> isinstance(doc, object)
True
```

> #### report\_from\_template(*template: str*, *\*fields: str | None*, *top\_n: int | None = None*, *remove\_prefix: bool = True*, *return\_string: bool = False*, *format: str = 'text'*, *filename: str | None = None*, *separator: str = '\n\n'*, *observation\_title\_template: str | None = None*, *explode: bool = False*, *filestore: bool = False*) → str | Document | List | [FileStore](/en/latest/filestore#edsl.scenarios.FileStore) | None [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Generates a report using a Jinja2 template for each row in the dataset.

This method renders a user-provided Jinja2 template for each observation in the dataset, with template variables populated from the row data. This allows for completely customized report formatting using pandoc for advanced output formats.

> **Args:**

template: Jinja2 template string to render for each row [\*](/en/latest/#id19)fields: The fields to include in template context. If none provided, all fields are used. top\_n: Optional limit on the number of observations to include. remove\_prefix: Whether to remove type prefixes (e.g., “answer.”) from field names in template context. return\_string: If True, returns the rendered content. If False (default in notebooks),

only displays the content without returning.

format: Output format - one of “text”, “html”, “pdf”, or “docx”. Formats other than “text” require pandoc. filename: If provided, saves the rendered content to this file. For exploded output,

this becomes a template (e.g., “report\_\{index}.html”).

separator: String to use between rendered templates for each row (ignored when explode=True). observation\_title\_template: Optional Jinja2 template for observation titles.

Defaults to “Observation ” where index is 1-based. Template has access to all row data plus ‘index’ and ‘index0’ variables.

explode: If True, creates separate files for each observation instead of one combined file. filestore: If True, wraps the generated file(s) in FileStore object(s). If no filename is provided,

creates temporary files. For exploded output, returns a list of FileStore objects.

> **Returns:**

Depending on explode, format, return\_string, and filestore: - For text format: String content or None (if displayed in notebook) - For html format: HTML string content or None (if displayed in notebook) - For docx format: Document object or None (if saved to file) - For pdf format: PDF bytes or None (if saved to file) - If explode=True: List of created filenames (when filename provided) or list of documents/content - If filestore=True: FileStore object(s) containing the generated file(s)

> **Notes:**

* Pandoc is required for HTML, PDF, and DOCX output formats
* Templates are treated as Markdown for all non-text formats
* PDF output uses XeLaTeX engine through pandoc
* HTML output includes standalone document structure

> **Examples:**

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> template = "Person feels: {{ how_feeling }}"
>>> report = r.select('how_feeling').report_from_template(template, return_string=True)
>>> "Person feels: OK" in report
True
>>> "Person feels: Great" in report
True
```

```python theme={null}
# Custom observation titles >>> custom_title = “Response {{ index }}: {{ how_feeling }}” >>> report = r.select(‘how_feeling’).report_from_template( … template, observation_title_template=custom_title, return_string=True) >>> “Response 1: OK” in report True

# HTML output (requires pandoc) >>> html_report = r.select(‘how_feeling’).report_from_template( … template, format=”html”, return_string=True) # doctest: +SKIP >>> # Creates HTML with proper document structure

# PDF output (requires pandoc with XeLaTeX) >>> pdf_report = r.select(‘how_feeling’).report_from_template( … template, format=”pdf”) # doctest: +SKIP >>> # Returns PDF bytes

# Basic template functionality >>> template2 = “Feeling: {{ how_feeling }}, Index: {{ index }}” >>> report2 = r.select(‘how_feeling’).report_from_template( … template2, return_string=True, top_n=2) >>> “Feeling: OK, Index: 1” in report2 True
```

> #### right\_join(*other: [ScenarioList](/en/latest/scenarios#scenariolist-class)*, *by: str | list\[str]*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Perform a right join with another ScenarioList, following SQL join semantics.

> **Args:**

other: The ScenarioList to join with by: String or list of strings representing the key(s) to join on. Cannot be empty.

> **Returns:**

A new ScenarioList containing all right scenarios with matching left data added

```python theme={null}
>>> s1 = ScenarioList([Scenario({'name': 'Alice', 'age': 30}), Scenario({'name': 'Bob', 'age': 25})])
>>> s2 = ScenarioList([Scenario({'name': 'Alice', 'location': 'New York'}), Scenario({'name': 'Charlie', 'location': 'Los Angeles'})])
>>> s5 = s1.right_join(s2, 'name')
>>> s5 == ScenarioList([Scenario({'age': 30, 'location': 'New York', 'name': 'Alice'}), Scenario({'age': None, 'location': 'Los Angeles', 'name': 'Charlie'})])
True
```

> #### sample(*n: int*, *seed: str | None = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return a random sample from the ScenarioList

```python theme={null}
>>> s = ScenarioList.from_list("a", [1,2,3,4,5,6])
>>> s.sample(3, seed = "edsl")
ScenarioList([Scenario({'a': 2}), Scenario({'a': 1}), Scenario({'a': 3})])
```

> #### select(*\*fields: str*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Select only specified fields from all scenarios in the list.

This method applies the select operation to each scenario in the list, returning a new ScenarioList where each scenario contains only the specified fields.

> **Args:**

[\*](/en/latest/#id21)fields: Field names to select from each scenario.

> **Returns:**

A new ScenarioList with each scenario containing only the selected fields.

> **Raises:**

KeyError: If any specified field doesn’t exist in any scenario.

> **Examples:**

```python theme={null}
>>> s = ScenarioList([Scenario({'a': 1, 'b': 1}), Scenario({'a': 1, 'b': 2})])
>>> s.select('a')
ScenarioList([Scenario({'a': 1}), Scenario({'a': 1})])
```

> #### shuffle(*seed: str | None = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Shuffle the ScenarioList.

```python theme={null}
>>> s = ScenarioList([Scenario({'a': 1, 'b': 1}), Scenario({'a': 1, 'b': 2})])
>>> s.select('a')
ScenarioList([Scenario({'a': 1}), Scenario({'a': 1})])
```

> #### sql(*query: str*, *transpose: bool = None*, *transpose\_by: str = None*, *remove\_prefix: bool = True*, *shape: str = 'wide'*) → [Dataset](/en/latest/dataset#dataset-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Execute SQL queries on the dataset.

This powerful method allows you to use SQL to query and transform your data, combining the expressiveness of SQL with EDSL’s data structures. It works by creating an in-memory SQLite database from your data and executing the query against it.

> **Parameters:**

query: SQL query string to execute transpose: Whether to transpose the resulting table (rows become columns) transpose\_by: Column to use as the new index when transposing remove\_prefix: Whether to remove type prefixes (e.g., “answer.”) from column names shape: Data shape to use (“wide” or “long”)

* “wide”: Default tabular format with columns for each field
* “long”: Melted format with key-value pairs, useful for certain queries

> **Returns:**

A Dataset object containing the query results

> **Notes:**

* The data is stored in a table named “self” in the SQLite database
* In wide format, column names include their type prefix unless remove\_prefix=True
* In long format, the data is melted into columns: row\_number, key, value, data\_type
* Complex objects like lists and dictionaries are converted to strings

> **Examples:**

```python theme={null}
>>> from edsl import Results
>>> r = Results.example()
```

```python theme={null}
# Basic selection >>> len(r.sql(“SELECT * FROM self”, shape=”wide”)) 4

# Filtering with WHERE clause >>> r.sql(“SELECT * FROM self WHERE how_feeling = ‘Great’”).num_observations() 1

# Aggregation >>> r.sql(“SELECT how_feeling, COUNT(*) as count FROM self GROUP BY how_feeling”).keys() [‘how_feeling’, ‘count’]

# Using long format >>> len(r.sql(“SELECT * FROM self”, shape=”long”)) 200
```

> #### string\_cat(*key: str*, *addend: str*, *position: str = 'suffix'*, *inplace: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Concatenate a string to a field across all Scenarios.

Applies the same behavior as `Scenario.string_cat` to each Scenario in the list. By default, returns a new `ScenarioList`; set `inplace=True` to modify this list.

**Args:**

key: The key whose value will be concatenated in each Scenario. addend: The string to concatenate to the existing value. position: Either “suffix” (default) or “prefix”. inplace: If True, modify scenarios in place and return self.

**Returns:**

A `ScenarioList` with updated Scenarios.

**Raises:**

KeyError: If `key` is missing in any Scenario. TypeError: If any value under `key` is not a string. ValueError: If `position` is not “suffix” or “prefix”.

> #### sum(*field: str*) → int[ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Sum the values of a field across all scenarios.

> #### table(*\*fields: str*, *tablefmt: Literal\['plain', 'simple', 'github', 'grid', 'fancy\_grid', 'pipe', 'orgtbl', 'rst', 'mediawiki', 'html', 'latex', 'latex\_raw', 'latex\_booktabs', 'tsv'] | None = 'rich'*, *pretty\_labels: dict\[str, str] | None = None*) → str[ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return the ScenarioList as a table.

> #### tack\_on(*replacements: dict\[str, Any]*, *index: int = -1*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Add a duplicate of an existing scenario with optional value replacements.

This method duplicates the scenario at *index* (default `-1` which refers to the last scenario), applies the key/value pairs provided in *replacements*, and returns a new ScenarioList with the modified scenario appended.

Args:

**replacements: Mapping of field names to new values to overwrite in the cloned**

scenario.

**index: Index of the scenario to duplicate. Supports negative indexing just**

like normal Python lists (`-1` is the last item).

**Returns:**

ScenarioList: A new ScenarioList containing all original scenarios plus the newly created one.

**Raises:**

**ScenarioError: If the ScenarioList is empty, *index* is out of range, or if**

any key in *replacements* does not exist in the reference scenario.

> #### tally(*\*fields: str | None*, *top\_n: int | None = None*, *output='Dataset'*) → dict | [Dataset](/en/latest/dataset#dataset-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Count frequency distributions of values in specified fields.

This method tallies the occurrence of unique values within one or more fields, similar to a GROUP BY and COUNT in SQL. When multiple fields are provided, it performs cross-tabulation across those fields.

> **Parameters:**

[\*](/en/latest/#id23)fields: Field names to tally. If none provided, uses all available fields. top\_n: Optional limit to return only the top N most frequent values. output: Format for results, either “Dataset” (recommended) or “dict”.

> **Returns:**

By default, returns a Dataset with columns for the field(s) and a ‘count’ column. If output=”dict”, returns a dictionary mapping values to counts.

> **Notes:**

* For single fields, returns counts of each unique value
* For multiple fields, returns counts of each unique combination of values
* Results are sorted in descending order by count
* Fields can be specified with or without their type prefix

> **Examples:**

```python theme={null}
>>> from edsl import Results
>>> r = Results.example()
```

\# Single field frequency count >>> r.select(‘how\_feeling’).tally(‘answer.how\_feeling’, output=”dict”) \{‘OK’: 2, ‘Great’: 1, ‘Terrible’: 1}

\# Return as Dataset (default) >>> from edsl.dataset import Dataset >>> expected = Dataset(\[\{‘answer.how\_feeling’: \[‘OK’, ‘Great’, ‘Terrible’]}, \{‘count’: \[2, 1, 1]}]) >>> r.select(‘how\_feeling’).tally(‘answer.how\_feeling’, output=”Dataset”) == expected True

\# Multi-field cross-tabulation - exact output varies based on data >>> result = r.tally(‘how\_feeling’, ‘how\_feeling\_yesterday’) >>> ‘how\_feeling’ in result.keys() and ‘how\_feeling\_yesterday’ in result.keys() and ‘count’ in result.keys() True

> #### times(*other: [ScenarioList](/en/latest/scenarios#scenariolist-class)*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Takes the cross product of two ScenarioLists.

Example:

```py theme={null}
>>> s1 = ScenarioList([Scenario({'a': 1}), Scenario({'a': 2})])
>>> s2 = ScenarioList([Scenario({'b': 1}), Scenario({'b': 2})])
>>> s1 * s2
ScenarioList([Scenario({'a': 1, 'b': 1}), Scenario({'a': 1, 'b': 2}), Scenario({'a': 2, 'b': 1}), Scenario({'a': 2, 'b': 2})])
```

> #### to(*survey: 'Survey' | 'QuestionBase'*) → [Jobs](/en/latest/jobs#jobs-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create a Jobs object from a ScenarioList and a Survey object.

**Parameters:** **survey** – The Survey object to use for the Jobs object.

Example: >>> from edsl import Survey, Jobs, ScenarioList # doctest: +SKIP >>> isinstance(ScenarioList.example().to(Survey.example()), Jobs) # doctest: +SKIP True

> #### to\_agent\_blueprint(*\**, *seed: int | None = None*, *cycle: bool = True*, *dimension\_name\_field: str = 'dimension'*, *dimension\_values\_field: str = 'dimension\_values'*, *dimension\_description\_field: str | None = None*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Create an AgentBlueprint from this ScenarioList.

**Args:**

seed: Optional seed for deterministic permutation order. cycle: Whether to continue cycling through permutations indefinitely. dimension\_name\_field: Field name to read the dimension name from. dimension\_values\_field: Field name to read the dimension values from. dimension\_description\_field: Optional field name for the dimension description.

> #### to\_agent\_list(*remove\_prefix: bool = True*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the results to a list of dictionaries, one per agent.

**Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_agent_list()
AgentList([Agent(traits = {'how_feeling': 'OK'}), Agent(traits = {'how_feeling': 'Great'}), Agent(traits = {'how_feeling': 'Terrible'}), Agent(traits = {'how_feeling': 'OK'})])
```

> #### to\_agent\_traits(*agent\_name: str | None = None*) → [Agent](/en/latest/agents#agent-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert all Scenario objects to traits of a single Agent.

Delegates to *ScenarioListTransformer.to\_agent\_traits*.

> #### to\_csv(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Export the results to a FileStore instance containing CSV data.

> #### to\_dataset() → [Dataset](/en/latest/dataset#dataset-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the ScenarioList to a Dataset.

```python theme={null}
>>> s = ScenarioList.from_list("a", [1,2,3])
>>> s.to_dataset()
Dataset([{'a': [1, 2, 3]}])
>>> s = ScenarioList.from_list("a", [1,2,3]).add_list("b", [4,5,6])
>>> s.to_dataset()
Dataset([{'a': [1, 2, 3]}, {'b': [4, 5, 6]}])
```

> #### to\_dict(*sort: bool = False*, *add\_edsl\_version: bool = False*) → dict [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

```python theme={null}
>>> s = ScenarioList([Scenario({'food': 'wood chips'}), Scenario({'food': 'wood-fired pizza'})])
>>> s.to_dict()
{'scenarios': [{'food': 'wood chips'}, {'food': 'wood-fired pizza'}]}
```

```python theme={null}
>>> s = ScenarioList([Scenario({'food': 'wood chips'})], codebook={'food': 'description'})
>>> d = s.to_dict()
>>> 'codebook' in d
True
>>> d['codebook'] == {'food': 'description'}
True
```

```python theme={null}
>>> # To include edsl_version and edsl_class_name, explicitly set add_edsl_version=True
>>> s.to_dict(add_edsl_version=True)
{'scenarios': [{'food': 'wood chips', 'edsl_version': '...', 'edsl_class_name': 'Scenario'}], 'codebook': {'food': 'description'}, 'edsl_version': '...', 'edsl_class_name': 'ScenarioList'}
```

> #### to\_dicts(*remove\_prefix: bool = True*) → list\[dict] [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the results to a list of dictionaries.

**Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_dicts()
[{'how_feeling': 'OK'}, {'how_feeling': 'Great'}, {'how_feeling': 'Terrible'}, {'how_feeling': 'OK'}]
```

> #### to\_docx(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Export the results to a FileStore instance containing DOCX data.

Each row of the dataset will be rendered on its own page, with a 2-column table that lists the keys and associated values for that observation.

> #### to\_excel(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*, *sheet\_name: str | None = None*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Export the results to a FileStore instance containing Excel data.

> #### to\_jsonl(*filename: str | None = None*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Export the results to a FileStore instance containing JSONL data.

> #### to\_key\_value(*field: str*, *value=None*) → dict | set [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return the set of values in the field.

**Parameters:**

* **field** – The field to extract values from.
* **value** – An optional field to use as the value in the key-value pair.

**Example:**

```python theme={null}
>>> s = ScenarioList([Scenario({'name': 'Alice'}), Scenario({'name': 'Bob'})])
>>> s.to_key_value('name') == {'Alice', 'Bob'}
True
```

> #### to\_list(*flatten=False*, *remove\_none=False*, *unzipped=False*) → list\[list] [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the results to a list of lists.

**Parameters:**

* **flatten** – Whether to flatten the list of lists.
* **remove\_none** – Whether to remove None values from the list.

```python theme={null}
>>> from edsl.results import Results
>>> Results.example().select('how_feeling', 'how_feeling_yesterday')
Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling_yesterday': ['Great', 'Good', 'OK', 'Terrible']}])
```

```python theme={null}
>>> Results.example().select('how_feeling', 'how_feeling_yesterday').to_list()
[('OK', 'Great'), ('Great', 'Good'), ('Terrible', 'OK'), ('OK', 'Terrible')]
```

```python theme={null}
>>> r = Results.example()
>>> r.select('how_feeling').to_list()
['OK', 'Great', 'Terrible', 'OK']
```

```python theme={null}
>>> from edsl.dataset import Dataset
>>> Dataset([{'a.b': [[1, 9], 2, 3, 4]}]).select('a.b').to_list(flatten = True)
[1, 9, 2, 3, 4]
```

```python theme={null}
>>> from edsl.dataset import Dataset
>>> # Testing to_list flatten with multiple columns raises DatasetValueError - tested in unit tests
```

> #### to\_pandas(*remove\_prefix: bool = False*, *lists\_as\_strings=False*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the results to a pandas DataFrame, ensuring that lists remain as lists.

**Args:**

remove\_prefix: Whether to remove the prefix from the column names. lists\_as\_strings: Whether to convert lists to strings.

**Returns:**

A pandas DataFrame.

> #### to\_polars(*remove\_prefix: bool = False*, *lists\_as\_strings=False*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the results to a Polars DataFrame.

**Args:**

remove\_prefix: Whether to remove the prefix from the column names. lists\_as\_strings: Whether to convert lists to strings.

**Returns:**

A Polars DataFrame.

> #### to\_ranked\_scenario\_list(*option\_fields: Sequence\[str]*, *answer\_field: str*, *include\_rank: bool = True*, *rank\_field: str = 'rank'*, *item\_field: str = 'item'*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the ScenarioList to a ranked ScenarioList based on pairwise comparisons.

**Args:**

option\_fields: List of scenario column names containing options to compare. answer\_field: Name of the answer column containing the chosen option’s value. include\_rank: If True, include a rank field on each returned Scenario. rank\_field: Name of the rank field to include when include\_rank is True. item\_field: Field name used to store the ranked item value on each Scenario.

**Returns:**

ScenarioList ordered best-to-worst according to pairwise ranking.

> #### to\_scenario\_list(*remove\_prefix: bool = True*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the results to a list of dictionaries, one per scenario.

**Parameters:**

**remove\_prefix** – Whether to remove the prefix from the column names.

```python theme={null}
>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_scenario_list()
ScenarioList([Scenario({'how_feeling': 'OK'}), Scenario({'how_feeling': 'Great'}), Scenario({'how_feeling': 'Terrible'}), Scenario({'how_feeling': 'OK'})])
```

> #### to\_scenario\_of\_lists() → [Scenario](/en/latest/scenarios#scenario-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return a single Scenario whose values are lists across the list.

The resulting Scenario contains every key that appears in any member of the list. For each key, the value is a list of that key’s values taken row-wise from the ScenarioList, padding with None when a row is missing the key.

```python theme={null}
>>> s = ScenarioList.from_list("a", [1, 2, 3])
>>> s.to_scenario_of_lists()
Scenario({'a': [1, 2, 3]})
>>> s2 = ScenarioList([Scenario({'a': 1}), Scenario({'b': 2})])
>>> s2.to_scenario_of_lists()
Scenario({'a': [1, None], 'b': [None, 2]})
```

> #### to\_sqlite(*filename: str | None = None*, *remove\_prefix: bool = False*, *pretty\_labels: dict | None = None*, *table\_name: str = 'results'*, *if\_exists: str = 'replace'*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Export the results to a SQLite database file.

> #### to\_survey() → [Survey](/en/latest/surveys#survey-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

> #### to\_true\_skill\_ranked\_list(*option\_fields: Sequence\[str]*, *answer\_field: str*, *include\_rank: bool = True*, *rank\_field: str = 'rank'*, *item\_field: str = 'item'*, *mu\_field: str = 'mu'*, *sigma\_field: str = 'sigma'*, *conservative\_rating\_field: str = 'conservative\_rating'*, *initial\_mu: float = 25.0*, *initial\_sigma: float = 8.333*, *beta: float = None*, *tau: float = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the ScenarioList to a ranked ScenarioList using TrueSkill algorithm. Args:

option\_fields: List of scenario column names containing options to compare. answer\_field: Name of the answer column containing the ranking order. include\_rank: If True, include a rank field on each returned Scenario. rank\_field: Name of the rank field to include when include\_rank is True. item\_field: Field name used to store the ranked item value on each Scenario. mu\_field: Field name for TrueSkill mu (skill estimate) value. sigma\_field: Field name for TrueSkill sigma (uncertainty) value. conservative\_rating\_field: Field name for conservative rating (mu - 3\*sigma). initial\_mu: Initial skill rating (default 25.0). initial\_sigma: Initial uncertainty (default 8.333). beta: Skill class width (defaults to initial\_sigma/2). tau: Dynamics factor (defaults to initial\_sigma/300).

> #### Returns:

ScenarioList ordered best-to-worst according to TrueSkill ranking.

> #### transform(*field: str*, *func: Callable*, *new\_name: str | None = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Transform a field using a function.

Delegates to *ScenarioListTransformer.transform*.

> #### transform\_by\_key(*key\_field: str*) → [Scenario](/en/latest/scenarios#scenario-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Transform the ScenarioList into a single Scenario with key/value pairs.

This method transforms the ScenarioList by: 1. Using the value of the specified key\_field from each Scenario as a new key 2. Automatically formatting the remaining values as “key: value, key: value” 3. Creating a single Scenario containing all the transformed key/value pairs

> Args:

key\_field: The field name whose value will become the new key

> Returns:

A single Scenario with all the transformed key/value pairs

> Examples:

```python theme={null}
>>> # Original scenarios: [{'topic': 'party', 'location': 'offsite', 'time': 'evening'}]
>>> scenarios = ScenarioList([
...     Scenario({'topic': 'party', 'location': 'offsite', 'time': 'evening'})
... ])
>>> transformed = scenarios.transform_by_key('topic')
>>> # Result: Scenario({'party': 'location: offsite, time: evening'})
```

> tree(*node\_order: List\[str] | None = None*) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Convert the results to a Tree.

**Args:**

node\_order: The order of the nodes.

**Returns:**

A Tree object.

> unique() → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Return a new ScenarioList containing only unique Scenario objects.

This method removes duplicate Scenario objects based on their hash values, which are determined by their content. Two Scenarios with identical key-value pairs will have the same hash and be considered duplicates.

> **Returns:**

A new ScenarioList containing only unique Scenario objects.

> **Examples:**

```python theme={null}
>>> from edsl.scenarios import Scenario, ScenarioList
>>> s1 = Scenario({"a": 1})
>>> s2 = Scenario({"a": 1})  # Same content as s1
>>> s3 = Scenario({"a": 2})
>>> sl = ScenarioList([s1, s2, s3])
>>> unique_sl = sl.unique()
>>> len(unique_sl)
2
>>> unique_sl
ScenarioList([Scenario({'a': 1}), Scenario({'a': 2})])
```

> **Notes:**

* The order of scenarios in the result is not guaranteed due to the use of sets
* Uniqueness is determined by the Scenario’s \_\_hash\_\_ method
* The original ScenarioList is not modified
* This implementation is memory efficient as it processes scenarios one at a time

> #### unpack(*field: str*, *new\_names: List\[str] | None = None*, *keep\_original=True*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Unpack a field into multiple fields.

Delegates to *ScenarioListTransformer.unpack*.

> unpack\_dict(*field: str*, *prefix: str | None = None*, *drop\_field: bool = False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Unpack a dictionary field into separate fields.

Delegates to *ScenarioListTransformer.unpack\_dict*.

> #### unpack\_list(*field: str*, *new\_names: List\[str] | None = None*, *keep\_original: bool = True*) → [Dataset](/en/latest/dataset#dataset-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Unpack list columns into separate columns with provided names or numeric suffixes.

For example, if a dataset contains: \[\{ 'data': \[\[1, 2, 3], \[4, 5, 6]], 'other': \[‘x’, ‘y’]}]

After d.unpack\_list(‘data’), it should become: \[\{ 'other': \[‘x’, ‘y’], 'data\_1': \[1, 4], 'data\_2': \[2, 5], 'data\_3': \[3, 6]}]

> **Args:**

field: The field containing lists to unpack new\_names: Optional list of names for the unpacked fields. If None, uses numeric suffixes. keep\_original: If True, keeps the original field in the dataset

> **Returns:**

A new Dataset with unpacked columns

> **Examples:**

```python>>> from edsl.dataset import Dataset theme={null}
>>> d = Dataset([{'data': [[1, 2, 3], [4, 5, 6]]}])
>>> d.unpack_list('data')
Dataset([{'data': [[1, 2, 3], [4, 5, 6]]}, {'data_1': [1, 4]}, {'data_2': [2, 5]}, {'data_3': [3, 6]}])
```

```python theme={null}
>>> d.unpack_list('data', new_names=['first', 'second', 'third'])
Dataset([{'data': [[1, 2, 3], [4, 5, 6]]}, {'first': [1, 4]}, {'second': [2, 5]}, {'third': [3, 6]}])
```

> #### unpivot(*id\_vars: List\[str] | None = None*, *value\_vars: List\[str] | None = None*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Unpivot the ScenarioList, allowing for id variables.

Delegates to *ScenarioListTransformer.unpivot*.

> #### zip(*field\_a: str*, *field\_b: str*, *new\_name: str*) → [ScenarioList](/en/latest/scenarios#scenariolist-class) [ \[source \]](https://github.com/expectedparrot/edsl/blob/main/edsl/scenarios.py)

Zip two iterable fields in each Scenario into a dict under a new key.

For every Scenario in the list, this method computes `dict(zip(scenario[field_a], scenario[field_b]))` and stores the result in a new key named `new_name`. It returns a new ScenarioList containing the updated Scenarios.

> **Args:**

field\_a: Name of the first iterable field whose values become dict keys. field\_b: Name of the second iterable field whose values become dict values. new\_name: Name of the new field to store the resulting dictionary under.

> **Returns:**

A new ScenarioList with the zipped dictionary added to each Scenario.

> **Raises:**

KeyError: If either field name does not exist in any Scenario. ScenarioError: If referenced fields are not iterable in any Scenario.

> **Examples:**

```python theme={null}
>>> sl = ScenarioList([
...     Scenario({"keys": ["a", "b"], "vals": [1, 2]}),
...     Scenario({"keys": ["x", "y"], "vals": [9, 8]}),
... ])
>>> sl2 = sl.zip("keys", "vals", "mapping")
>>> sl2[0]["mapping"], sl2[1]["mapping"]
({'a': 1, 'b': 2}, {'x': 9, 'y': 8})
```


# Starter Tutorial
Source: https://docs.expectedparrot.com/en/latest/starter_tutorial

This tutorial provides step-by-step instructions for getting started using EDSL (*Expected Parrot Domain-Specific Language*), an open-source Python library for simulating surveys, experiments and other research tasks using AI agents and large language models. EDSL is developed by [Expected Parrot](https://www.expectedparrot.com/about) and available under the MIT License. The source code is hosted on [GitHub](https://github.com/expectedparrot/edsl).

**Goals of this tutorial**

We begin with technical setup: instructions for installing the EDSL library and storing API keys to access language models. Then we demonstrate some of the basic features of EDSL, with examples for constructing and running surveys with agents and models, and analyzing responses as datasets. By the end of this tutorial, you will be able to use EDSL to do each of the following:

* Construct various types of [questions](/en/latest/questions) tailored to your research objectives.
* Combine questions into [surveys](/en/latest/surveys) and integrate logical rules to control the survey flow.
* Add context to questions by *piping*answers, adding *memory* of prior questions and answers, and using [scenarios](/en/latest/scenarios) to add data or content to questions.
* Design personas for AI [agents](/en/latest/agents) to simulate responses to your surveys.
* Choose and deploy [language models](/en/latest/language_models) to generate responses for AI agents.
* Analyze [results](/en/latest/results) as datasets with built-in analytical tools.
* Validate LLM answers with [human respondents](/en/latest/humanize).

**Storing & sharing your work**

We also introduce [Expected Parrot](https://www.expectedparrot.com/login): a platform for creating, storing and sharing AI-based research. Expected Parrot is fully integrated with EDSL and free to use. At the end of the tutorial we show how to post content created in this tutorial for anyone to view. Learn more in the [Expected Parrot documentation](/en/latest/coop).

<Note>
  **Note:**

  You can also view and download the contents of this tutorial in a [notebook at Expected Parrot](https://www.expectedparrot.com/content/RobinHorton/starter-tutorial)
</Note>

**Further reading**

In addition to this tutorial, please also see an [Overview](/en/latest/overview#overview) of features and common use cases for EDSL and a [Survey logic checklist](/en/latest/checklist#checklist) of tips for using EDSL effectively in the [EDSL documentation page](/en/latest/index). To see recent research using or citing EDSL, see [Papers & Citations](/en/latest/papers#papers) in the documentation.

**Questions**

If you encounter any issues or have questions, please email us at [info@expectedparrot.com](mailto:info@expectedparrot.com) or post a question at our [Discord channel](https://discord.com/invite/mxAYkjfy9m).

## Pre-requisites

EDSL is compatible with Python 3.9 - 3.13. Before starting this tutorial, please ensure that you have a Python environment set up on your machine or in a cloud-based environment, such as Google Colab. You can find instructions for installing Python at the [Python Software Foundation](https://www.python.org/downloads/).

## Recommendations

Run code examples in a notebook

The code examples in this tutorial are designed to be run in a Jupyter notebook or another Python environment, or in a cloud-based environment such as Google Colab. If you are using Google Colab, please see additional instructions for setting up EDSL in the [Colab setup](/en/latest/colab_setup) page in the documentation.

Use a virtual environment

We also recommend using a virtual environment when installing and using EDSL in order to avoid conflicts with other Python packages. You can find instructions for setting up a virtual environment at the [Python Packaging Authority](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/).

Special instructions for Colab users

If you are using EDSL in a cloud-based environment, such as Google Colab, you can find additional instructions for setting up EDSL in the [Colab setup](/en/latest/colab_setup) page in the documentation.

## Installation

To begin using EDSL, you first need to install the library. This can either be done locally on your machine or in a cloud-based environment, such as Google Colab. Once you have decided where to install EDSL, you can choose to whether install it from [PyPI](https://pypi.org/project/edsl/) or [GitHub](https://github.com/expectedparrot/edsl):

From PyPI

Install EDSL directly using pip, which is straightforward and recommended for most users. We also recommend using a virtual environment to manage your Python packages (see *Recommendations* above). Run the following command in your notebook to install EDSL from PyPI:

```bash theme={null}
! uv pip install edsl -q
```

From GitHub

You can find the source code for EDSL and contribute to the project at [GitHub](https://github.com/expectedparrot/edsl). Installing from GitHub allows you to get the latest updates to EDSL before they are released to a new version at PyPI. This is recommended if you are using new features or contributing to the project. Run the following command to install EDSL from GitHub:

```bash theme={null}
pip install git+https://github.com/expectedparrot/edsl.git@main
```

After installing EDSL, you can check the version that you have installed by running the following command in your notebook:

```bash theme={null}
pip show edsl
```

To update your installation of EDSL to the latest version at PyPI, run the following command:

```bash theme={null}
pip install --upgrade edsl
```

## Create an account

Creating an account allows you to run survey jobs at Expected Parrot using language models of your choice, and automatically cache your results. Your account also allows you to launch human surveys and share your content and workflows with other users. Your account comes with \$25 in credits for API calls to LLMs for getting started, and a referal code for earning more credits.

[Create an account](https://www.expectedparrot.com/login) with an email address and password, or run the following code to be prompted automatically:

```python theme={null}
from edsl import login

login()
```

The above code also automatically stores your Expected Parrot API key for use with EDSL (see below).

## Accessing Language Models

The next step is to decide how you want to access language models. EDSL works with many popular language models that you can choose from to generate responses to your surveys. These models are hosted by various service providers, such as Anthropic, Azure, Bedrock, Deep Infra, DeepSeek, Google, Groq, Mistral, OpenAI, Replicate, Together and Xai. In order to run a survey, you need to provide API keys for the service providers of models that you want to use. There are two methods for providing API keys to EDSL:

* Use an Expected Parrot API key to access all available models
* Provide your own API keys from service providers

### Managing keys

The easiest way to manage your keys is from your Expected Parrot account. [Log in](https://www.expectedparrot.com/login) to your account and navigate to your [Keys](https://www.expectedparrot.com/home/keys) page to find options for adding and sharing your keys.

Your Expected Parrot key is automatically available to use by default whenever remote inference is activated. This key allows you to access the Expected Parrot server and run surveys with all available models.

Please see the [Managing Keys](/en/latest/api_keys) section for more details on methods of storing and managing keys.

<Note>
  *Note:* If you try to run a survey without storing a required API key, you will be provided a link to activate remote inference and use your Expected Parrot key.
</Note>

### Credits & tokens

Running surveys with language models requires tokens. If you are using your own API keys, service providers will bill you directly. If you are using your Expected Parrot API key to access models, you will need to purchase credits to cover token costs. Please see the model pricing page for details on available models and their current prices.

<Note>
  *Note:* Your account comes with 2,500 free credits (\$25 worth of API calls). You can purchase more credits at any time at your [Credits](https://www.expectedparrot.com/home/purchases) page.
</Note>

After installing EDSL and storing API keys you are ready to run some examples!

## Example: Running a simple question

EDSL comes with a [variety of question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from a model, including multiple choice, checkbox, rank, top-k, linear scale, likert five, yes/no, numerical, free text, extract, list, matrix and functional questions.

We can inspect the components of a particular question type by importing the question type class and calling the example method on it:

```python theme={null}
from edsl import (
  # QuestionCheckBox,
  # QuestionExtract,
  # QuestionFreeText,
  # QuestionFunctional,
  # QuestionLikertFive,
  # QuestionLinearScale,
  # QuestionList,
  QuestionMultipleChoice,
  # QuestionNumerical,
  # QuestionRank,
  # QuestionTopK,
  # QuestionYesNo
)

q = QuestionMultipleChoice.example() # substitute any question type class name
q
```

Output:

| key | value |
| :- | :- |
| question\_name | how\_feeling |
| question\_text | How are you? |
| question\_options:0 | Good |
| question\_options:1 | Great |
| question\_options:2 | OK |
| question\_options:3 | Bad |
| include\_comment | False |
| question\_type | multiple\_choice |

Here we create a simple multiple choice question:

```python theme={null}
from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
  question_name = "smallest_prime",
  question_text = "Which is the smallest prime number?",
  question_options = [0, 1, 2, 3]
)
```

We can administer it to a language model by calling the run() method on it.

```python theme={null}
results = q.run()
```

This generates a dataset of Results that we can readily access with [built-in methods for analysis](/en/latest/results). Here we inspect the response, together with the model that was used and the model’s “comment” about its response–a field that is automatically added to all question types other than free text:

```python theme={null}
results.select("model", "smallest_prime", "smallest_prime_comment")
```

Output:

| model.model | answer.smallest\_prime | comment.smallest\_prime\_comment |
| :- | :- | :- |
| gpt-4o | 2 | 2 is the smallest prime number because it is the only even number greater than 1 that is divisible only by 1 and itself. |

The Results also include information about the question, model parameters, prompts, generated tokens and raw responses. To see a list of all the components:

```python theme={null}
results.columns
```

Output:

| 0 |
| :- |
| agent.agent\_instruction |
| agent.agent\_name |
| answer.smallest\_prime |
| comment.smallest\_prime\_comment |
| generated\_tokens.smallest\_prime\_generated\_tokens |
| iteration.iteration |
| model.frequency\_penalty |
| model.logprobs |
| model.max\_tokens |
| model.model |
| model.presence\_penalty |
| model.temperature |
| model.top\_logprobs |
| model.top\_p |
| prompt.smallest\_prime\_system\_prompt |
| prompt.smallest\_prime\_user\_prompt |
| question\_options.smallest\_prime\_question\_options |
| question\_text.smallest\_prime\_question\_text |
| question\_type.smallest\_prime\_question\_type |
| raw\_model\_response.smallest\_prime\_cost |
| raw\_model\_response.smallest\_prime\_one\_usd\_buys |
| raw\_model\_response.smallest\_prime\_raw\_model\_response |

<Note>
  *Note:* If we are running the job locally we can pass run(progress\_bar=True) to view a Progress Report. Any exceptions will appear in the console.
</Note>

If remote inference is activated, a link to a Progress Report will appear automatically, as well as a link to an Exceptions Report if there are any. When the job is completed, a link to the Results page will also appear.

## Example: Conducting a survey with agents and models

In the next example we construct a more complex survey consisting of multiple questions and design personas for AI agents to answer it. Then we select specific language models to generate the answers.

We start by creating questions in different types and passing them to a Survey:

```python theme={null}
from edsl import QuestionLinearScale, QuestionFreeText

q_enjoy = QuestionLinearScale(
  question_name = "enjoy",
  question_text = "On a scale from 1 to 5, how much do you enjoy reading?",
  question_options = [1, 2, 3, 4, 5],
  option_labels = {1:"Not at all", 5:"Very much"}
)

q_favorite_place = QuestionFreeText(
  question_name = "favorite_place",
  question_text = "Describe your favorite place for reading."
)
```

We construct a Survey by passing a list of questions:

```python theme={null}
from edsl import Survey

survey = Survey(questions = [q_enjoy, q_favorite_place])
```

### Agents

An important feature of EDSL is the ability to create AI agents to answer questions. This is done by passing dictionaries of relevant “traits” to Agent objects that are used by language models to generate responses. Learn more about [designing agents](/en/latest/agents).

Here we construct several simple agent personas to use with our survey:

```python theme={null}
from edsl import AgentList, Agent

agents = AgentList(
  Agent(traits = {"persona":p}) for p in ["artist", "mechanic", "sailor"]
)
```

### Language models

EDSL works with many popular large language models that we can select to use with a survey. This makes it easy to compare responses among models in the results that are generated.

To see a current list of available models:

```python theme={null}
from edsl import Model

# Model.available() # uncomment this code and run it to see the list of available models
```

To check the default model that will be used if no models are specified for a survey (e.g., as in the first example above):

```python theme={null}
Model()
```

Output (may be different if the default model has changed):

| key | value |
| :- | :- |
| model | gpt-4o |
| parameters:temperature | 0.5 |
| parameters:max\_tokens | 1000 |
| parameters:top\_p | 1 |
| parameters:frequency\_penalty | 0 |
| parameters:presence\_penalty | 0 |
| parameters:logprobs | False |
| parameters:top\_logprobs | 3 |

Here we select some models to use with our survey:

```python theme={null}
from edsl import ModelList, Model

models = ModelList(
  Model(m) for m in ["gpt-4o", "gemini-pro"]
  )
```

### Running a survey

We add agents and models to a survey using the by method. Then we administer a survey the same way that we do an individual question, by calling the run method on it:

```python theme={null}
results = survey.by(agents).by(models).run()

(
  results
  .sort_by("persona", "model")
  .select("model", "persona", "enjoy", "favorite_place")
)
```

Example output:

| model.model | agent.persona | answer.enjoy | answer.favorite\_place |
| :- | :- | :- | :- |
| gemini-pro | artist | 5 | Nestled amidst the verdant embrace of a sprawling park, my favorite reading sanctuary unfolds as a secluded haven where tranquility reigns supreme. Beneath the towering canopy of ancient oak trees, a quaint bench beckons, its weathered surface inviting me to sink into its embrace. As I settle in, the gentle rustling of leaves overhead creates a soothing symphony that calms my mind and prepares me for the literary journey ahead. The air is fragrant with the sweet scent of blooming wildflowers, carried by a soft breeze that whispers secrets through the trees. The vibrant hues of nature paint the canvas around me, inspiring a sense of wonder and connection to the world. As I open the pages of my chosen book, the outside world fades into oblivion. The words dance before my eyes, inviting me into realms unknown. The characters become my companions, their stories unfolding before me like a captivating tapestry. Time seems to stand still in this idyllic setting. The worries of the day dissolve as I immerse myself in the written word. As the sun begins its descent, casting long shadows across the park, I close my book and savor the lingering glow of the day. The world around me has transformed into a magical realm, where the boundaries between reality and imagination blur. |
| gpt-4o | artist | 4 | My favorite place for reading is a cozy nook by a large window in my art studio. The natural light that streams in during the day is perfect for both reading and painting. I have a comfortable armchair draped with a colorful throw, and a small wooden side table where I keep a steaming cup of herbal tea. The walls are adorned with my paintings, which add a touch of inspiration and creativity to the atmosphere. It’s a quiet, peaceful space where I can lose myself in a good book or simply gaze out at the changing scenery outside. |
| gemini-pro | mechanic | 5 | In the heart of my cozy abode, where solitude and inspiration intertwine, lies my sanctuary of literary bliss—my reading nook. Bathed in the warm glow of a vintage lamp, it beckons me with its allure, a haven where I can escape into the realms of imagination. The walls are adorned with shelves brimming with an eclectic collection of books, their spines whispering tales of adventure, romance, and wisdom. The air is infused with the faint scent of paper and ink, a symphony that awakens my senses. A plush armchair, upholstered in soft velvet, invites me to sink into its embrace, enveloping me in a cocoon of comfort. A large window frames the verdant garden outside, offering a tranquil view of nature’s artistry. As I turn the pages, the rustling of leaves and the chirping of birds create a soothing soundtrack that enhances my reading experience. The gentle breeze carries the sweet fragrance of blooming flowers, mingling with the scent of freshly brewed coffee on my side table. In this tranquil haven, I am free to lose myself in the written word. Time seems to stand still as I journey through distant lands, unravel mysteries, and explore the depths of human emotion. The characters become my companions, their struggles and triumphs mirroring my own. |
| gpt-4o | mechanic | 2 | As a mechanic, my favorite place for reading might not be what you’d expect. I enjoy reading in my garage, surrounded by the hum of engines and the smell of oil. There’s something comforting about being in my element, with tools and parts all around me. I usually set up a small corner with a sturdy chair and a good lamp, so I can dive into a book during my breaks. Whether it’s a manual on the latest automotive technology or a novel to unwind, the garage is my go-to spot. |
| gemini-pro | sailor | 5 | Amidst the bustling city’s cacophony, I seek solace in a sanctuary of tranquility—my favorite reading nook. Nestled in a cozy corner of my apartment, it is an oasis of serenity. The soft glow of a vintage lamp illuminates a comfortable armchair, its plush cushions inviting me to sink into its embrace. A large window frames a vibrant cityscape, providing a backdrop of constant movement and life. Yet, within this cozy haven, I find stillness and escape. The walls are adorned with an eclectic collection of artwork, each piece evoking a different memory or inspiration. A vibrant abstract painting captures the essence of a stormy sea, while a delicate watercolor depicts the serene beauty of a mountain meadow. These visual cues transport me to distant realms, setting the stage for literary adventures. The air is scented with the faint aroma of freshly brewed coffee and the subtle fragrance of old books. The gentle hum of the city outside fades into a distant murmur, creating an atmosphere conducive to deep contemplation and immersion. As I settle into my armchair, I reach for a book. Its pages hold the promise of countless worlds to explore, characters to meet, and lessons to learn. The weight of the book in my hands feels both comforting and exhilarating, a tangible connection to the boundless possibilities within its covers. With each turn of the page, I am transported to different times and places. I witness the rise and fall of empires, the triumphs and tragedies of human lives, and the wonders of the natural world. The words dance before my eyes, painting vivid images in my mind. I become lost in the stories, my own worries and concerns fading away. |
| gpt-4o | sailor | 3 | Ah, my favorite place for reading has to be the deck of a ship, with the vast ocean stretching out endlessly before me. There’s something about the gentle rocking of the waves and the salty sea breeze that makes any book come alive. I love settling into a sturdy deck chair, perhaps with a mug of strong coffee or a tot of rum by my side, and losing myself in a tale while the sun sets on the horizon, painting the sky with colors that even the best of stories can’t quite capture. The sound of the water lapping against the hull provides a soothing background, making it the perfect spot to dive into a good book. |

### Running a survey in the background

If remote inference is activated, we can optionally run the survey in the background and continue working (or not) while waiting for results to be generated:

```python theme={null}
results = survey.by(agents).by(models).run(background=True)
```

This will return a link to the progress bar page (as usual), which you can check at any time. You can also check the status of the job by running:

```python theme={null}
results.fetch()
```

This will return either a status update or the results. Once the job is completed, you can call the results as usual, e.g.:

```python theme={null}
results.columns # to view a list of all columns

results.select("answer.*") # to view all answers
```

## [Example: Adding context to questions](/en/latest/#id24)

EDSL provides a variety of ways to add data or content to survey questions. These methods include:

* [Piping](/en/latest/surveys#id2) answers to questions into follow-on questions
* [Adding “memory”](/en/latest/surveys#question-memory) of prior questions and answers in a survey when presenting other questions to a model
* [Parameterizing questions with data](/en/latest/scenarios), e.g., content from PDFs, CSVs, docs, images or other sources that you want to add to questions

### [Piping question answers](/en/latest/#id25)

Here we demonstrate how to pipe the answer to a question into the text of another question. This is done by using a placeholder  in the text of the follow-on question where the answer to the prior question is to be inserted when the survey is run. This causes the questions to be administered in the required order (survey questions are administered asynchronously by default). Learn more about [piping question answers](/en/latest/surveys#id2).

Here we insert the answer to a numerical question into the text of a follow-on yes/no question:

```python theme={null}
from edsl import QuestionNumerical, QuestionYesNo, Survey

q1 = QuestionNumerical(
  question_name = "random_number",
  question_text = "Pick a random number between 1 and 1,000."
)

q2 = QuestionYesNo(
  question_name = "prime",
  question_text = "Is this a prime number: {{ random_number.answer }}"
)

survey = Survey([q1, q2])

results = survey.run()
```

We can check the user\_prompt for the prime question to verify that the answer to the random\_number question was piped into it:

```python theme={null}
results.select("random_number", "prime_user_prompt", "prime", "prime_comment")
```

Example output:

| answer.random\_number | prompt.prime\_user\_prompt | answer.prime | comment.prime\_comment |
| :- | :- | :- | :- |
| 487 | Is this a prime number: 487  No  Yes  Only 1 option may be selected. Please respond with just your answer.  After the answer, you can put a comment explaining your response. | No | 487 is not a prime number because it can be divided evenly by 1, 487, and also by 19 and 25. |

### [Adding “memory” of questions and answers](/en/latest/#id26)

Here we instead add a “memory” of the first question and answer to the context of the second question. This is done by calling a memory rule and identifying the question(s) to add. Instead of just the answer, information about the full question and answer are presented with the follow-on question text, and no placeholder is used. Learn more about [question memory rules](/en/latest/surveys#survey-rules-logic).

Here we demonstrate the add\_targeted\_memory method (we could also use set\_full\_memory\_mode or other memory rules):

```python theme={null}
from edsl import QuestionNumerical, QuestionYesNo, Survey

q1 = QuestionNumerical(
  question_name = "random_number",
  question_text = "Pick a random number between 1 and 1,000."
)

q2 = QuestionYesNo(
  question_name = "prime",
  question_text = "Is the number you picked a prime number?"
)

survey = Survey([q1, q2]).add_targeted_memory(q2, q1)

results = survey.run()
```

We can again use the user\_prompt to verify the context that was added to the follow-on question:

```python theme={null}
results.select("random_number", "prime_user_prompt", "prime", "prime_comment").table().long()
```

Example output:

| row | key | value |
| :- | :- | :- |
| 0 | answer.random\_number | 487 |
| 0 | prompt.prime\_user\_prompt | Is the number you picked a prime number?  No  Yes |

## Scenarios

We can also add external data or content to survey questions. This can be useful when you want to efficiently create and administer multiple versions of questions at once, e.g., for conducting data labeling tasks. This is done by creating Scenario dictionaries for the data or content to be used with a survey, where the keys match `{{ placeholder }}` names used in question texts (or question options) and the values are the content to be added. Scenarios can also be used to [add metadata to survey results](/en/latest/notebooks/adding_metadata), e.g., data sources or other information that you may want to include in the results for reference but not necessarily include in question texts.

In the next example we revise the prior survey questions about reading to take a parameter for other activities that we may want to add to the questions, and create simple scenarios for some activities. EDSL provides methods for automatically generating scenarios from a variety of data sources, including PDFs, CSVs, docs, images, tables and dicts. We use the from\_list method to convert a list of activities into scenarios.

Then we demonstrate how to use scenarios to create multiple versions of our questions either (i) when constructing a survey or (ii) when running it:

* In the latter case, the by method is used to add scenarios to a survey of questions with placeholders at the time that it is run (the same way that agents and models are added to a survey). This adds a scenario column to the results with a row for each answer to each question for each scenario.
* In the former case, the loop method is used to create a list of versions of a question with the scenarios already added to it; when the questions are passed to a survey and it is run, the results include columns for each individual question; there is no scenario column and a single row for each agent’s answers to all the questions.

Learn more about [using scenarios](/en/latest/scenarios).

Here we create simple scenarios for a list of activities:

```python theme={null}
from edsl import ScenarioList, Scenario

scenarios = ScenarioList.from_list("activity", ["reading", "running", "relaxing"])
```

### Adding scenarios using the by method

Here we add the scenarios to the survey when we run it, together with any desired agents and models:

```python expandable theme={null}
from edsl import QuestionLinearScale, QuestionFreeText, Survey

q_enjoy = QuestionLinearScale(
  question_name = "enjoy",
  question_text = "On a scale from 1 to 5, how much do you enjoy {{ scenario.activity }}?",
  question_options = [1, 2, 3, 4, 5],
  option_labels = {1:"Not at all", 5:"Very much"}
)

q_favorite_place = QuestionFreeText(
  question_name = "favorite_place",
  question_text = "In a brief sentence, describe your favorite place for {{ scenario.activity }}."
)

survey = Survey([q_enjoy, q_favorite_place])

results = survey.by(scenarios).by(agents).by(models).run()

(
  results
  .filter("model.model == 'gpt-4o'")
  .sort_by("activity", "persona")
  .select("activity", "persona", "enjoy", "favorite_place")
)
```

Output:

| scenario.activity | agent.persona | answer.enjoy | answer.favorite\_place |
| :- | :- | :- | :- |
| reading | artist | 4 | My favorite place for reading is a cozy nook by a large window, where the natural light spills over the pages, surrounded by plants and the gentle hum of city life outside. |
| reading | mechanic | 2 | My favorite place for reading is in my garage, surrounded by the hum of engines and the scent of motor oil, where I can escape into a good book during breaks. |
| reading | sailor | 3 | Ah, my favorite place for reading is out on the deck of a ship, with the salty sea breeze in my hair and the gentle rocking of the waves beneath me. |
| relaxing | artist | 4 | My favorite place for relaxing is a sun-dappled studio filled with the scent of fresh paint and the gentle hum of creativity. |
| relaxing | mechanic | 3 | My favorite place for relaxing is in my garage, tinkering with an old engine, where the hum of tools and the smell of grease help me unwind. |
| relaxing | sailor | 3 | There’s nothing quite like the gentle sway of a hammock on the deck of a ship, with the sound of the ocean waves lapping against the hull and the salty breeze in the air. |
| running | artist | 2 | My favorite place for running is a winding forest trail where the sunlight filters through the leaves, creating a dappled pattern on the ground. |
| running | mechanic | 1 | My favorite place for running is a quiet trail through the woods, where the fresh air and natural surroundings make each step feel refreshing. |
| running | sailor | 2 | Ah, my favorite place for running is along the rugged coastline, where the salty sea breeze fills the air and the waves crash against the rocks, reminding me of the vastness of the ocean. |

## Exploring Results

EDSL comes with [built-in methods for analyzing and visualizing survey results](/en/latest/language_models). For example, you can call the to\_pandas method to convert results into a dataframe:

```python theme={null}
df = results.to_pandas(remove_prefix=True)
```

The Results object also supports SQL-like queries with the the sql method:

```python theme={null}
results.sql("""
select model, persona, enjoy_reading, favorite_place_reading
from self
order by 1,2,3
""")
```

Output:

```python theme={null}
```

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [posting content](/en/latest/coop).

We can post any EDSL object to Expected Parrot by calling the push method on it, optionally passing a description, a convenient alias for the URL that is created and a visibility status (*public*, *private* or *unlisted* by default):

```python theme={null}
results.push(
  description = "Starter tutorial sample survey results",
  alias = "starter-tutorial-example-survey-results",
  visibility = "public"
)
```

Example output (UUIDs will be unique to objects):

```json theme={null}
{
'description': 'Starter tutorial sample survey results',
'object_type': 'results',
'url': 'https://www.expectedparrot.com/content/9c8ba866-2be3-4ad6-9d2f-8160a06b2cf7',
'uuid': '9c8ba866-2be3-4ad6-9d2f-8160a06b2cf7',
'version': '0.1.47.dev1',
'visibility': 'public'
}
```

To post a notebook:

```python theme={null}
from edsl import Notebook

notebook = Notebook(path="filename.ipynb")

notebook.push(description="Starter Tutorial", alias = "example-notebook-new-alias", visibility="public")
```

You can view and download a notebook for this tutorial [at Expected Parrot](https://www.expectedparrot.com/content/RobinHorton/starter-tutorial).


# Survey Builder
Source: https://docs.expectedparrot.com/en/latest/survey_builder



## Overview

Survey Builder is a user-friendly, no-code application for launching surveys and gathering responses from AI agents and human respondents. It is fully integrated with EDSL and available at your account, allowing you to seamlessly design questions and agents, and analyze, visualize and share your results.

Tutorials and examples are available [here](https://www.expectedparrot.com/products/polly/use-cases).

*If you are looking for information about EDSL code methods for launching surveys with human respondents, please see the* [Humanize](/en/latest/humanize#humanize) *and* [Prolific studies](/en/latest/prolific#prolific) *pages.*

## Features

* **Survey creation**: Build customized surveys using a wide range of question types, including multiple choice, free text, linear scale, matrix, numerical and more.
* **AI agent design**: Create and configure AI agent personas to respond to your questions.
* **Dynamic data integration**: Import data to dynamically parameterize your surveys.
* **Seamless deployment**: Easily launch surveys with both human and AI respondents.
* **Results analysis**: Leverage built-in tools to visualize and analyze responses, combining human and AI data for greater insights.
* **Team collaboration**: Share surveys, agents, results and projects with your team with tools for streamlined sharing and collaboration.

## Getting started

<Steps>
  <Step title="Log into your account">
    Sign in or [create an account](https://www.expectedparrot.com/login).
  </Step>

  <Step title="Create a survey">
    Choose whether to edit or import an existing survey, auto-generate a survey,or build a new survey.
  </Step>

  <Step title="Design AI agents">
    Choose whether to edit saved agents, import data to create new agents, auto-generate agents for a survey, or build new agents.
  </Step>

  <Step title="Run the survey">
    Select models to run your survey and view responses at your dashboard.
  </Step>

  <Step title="Gather human responses">
    Generate a web-based version of your survey and share it with human respondents, or launch a study with Prolific participants.
  </Step>

  <Step title="Analyze results">
    View and analyze survey responses at your account, and export data for further analysis.
  </Step>
</Steps>

### Create a survey

[Log in](https://www.expectedparrot.com/login) to your account and select options to create a [survey](https://www.expectedparrot.com/create/survey) and [agents](https://www.expectedparrot.com/create/agent-list). Choose whether to build a survey and agents from scratch, import data, or select and edit saved content.

### Launch a survey

Selecting the option to *Run survey* creates a new project at your account dashboard, where you can run the survey with AI agents and human respondents. You can also create a new project by selecting *Create project* at your dashboard and choosing a saved survey and agents to use with the project.

Your new project page has options for generating a shareable web version of the survey, running the survey with AI agents, and launching Prolific studies with human participants. AI and human responses will automatically appear at your dashboard where you can compare and export them.

### Analyze results

Once you have collected responses from AI agents and human respondents, you can analyze the results at your account dashboard or in EDSL code at your workspace. See the [Humanize](/en/latest/humanize#humanize) and [Prolific studies](/en/latest/prolific#prolific) sections for more details about working with human responses in EDSL code. You can also export results in CSV format for further analysis in your preferred tools.

*Please let us know if you have any questions or suggestions for improving the survey builder!*


# Versioning Surveys with .ep Files
Source: https://docs.expectedparrot.com/en/latest/survey_ep_format

A hands-on tour of the Git-backed .ep format for EDSL Surveys: package contents, commits, diffs, history, and efficient collaboration.

An EDSL `Survey` can be saved either as serialized JSON or as an `.ep`
package. JSON is useful for a quick interchange. An `.ep` package is intended
for work that will evolve: it keeps the survey's current state and its Git
history together in one portable file.

This tutorial creates a package from `Survey.example()`, opens it up, changes
one question, and follows that change through Git.

## The format in one picture

```text theme={null}
example-survey.ep                   one portable ZIP-compatible file
└── unpacked contents               a complete Git working tree
    ├── .git/                       commits, trees, blobs, refs, and configuration
    ├── manifest.json               package type, format version, and question order
    ├── questions/
    │   ├── 000001.json             one question per file
    │   ├── 000002.json
    │   └── 000003.json
    └── metadata/
        ├── memory_plan.json
        ├── question_groups.json
        └── rule_collection.json
```

There are two layers to keep straight:

* On disk, `example-survey.ep` is a single ZIP-compatible archive. It is easy
  to attach, copy, archive, or deposit in a replication package.
* Inside the archive is a real Git repository. EDSL uses that repository for
  commits, history, diffs, branches, tags, and synchronization.

The `.ep` suffix therefore does not mean “a JSON file with another extension.”
It means “a portable EDSL object with its version history.”

## Create the example Survey

Start with EDSL's built-in example:

```python theme={null}
from edsl import Survey

survey = Survey.example()
survey
```

The example has three multiple-choice questions named `q0`, `q1`, and `q2`.
Save it as a package with an informative commit message:

```python theme={null}
first_save = survey.git.save(
    "example-survey.ep",
    message="Create example survey",
)
first_save
```

A successful save returns package and commit information:

```python theme={null}
{
    "status": "ok",
    "path": "example-survey.ep",
    "commit": "65801c7e03eb93c0f31c8bee6b2a29cc9152be14",
    "branch": "main",
    "message": "Create example survey",
}
```

Your commit hash will differ. Git derives it from the commit and its contents,
so it is an identifier for that particular version.

Calling `save()` again without changing the survey does not create an empty
commit:

```python theme={null}
survey.git.save(message="Save again")
```

```python theme={null}
{
    "status": "unchanged",
    "path": "example-survey.ep",
    "commit": "...",
    "branch": "main",
    "message": "no changes",
}
```

This makes repeated saves safe: history records substantive package changes,
not every call to `save()`.

## Inspect the package as an EDSL object

For ordinary use, let EDSL inspect and load the file:

```bash theme={null}
ep inspect example-survey.ep
```

The CLI writes a JSON envelope to stdout:

```json theme={null}
{
  "status": "ok",
  "data": {
    "object_type": "Survey",
    "length": 3,
    "question_count": 3,
    "question_names": ["q0", "q1", "q2"],
    "question_types": [
      "multiple_choice",
      "multiple_choice",
      "multiple_choice"
    ]
  },
  "warnings": []
}
```

Load the latest committed version in Python with:

```python theme={null}
loaded = Survey.git.load("example-survey.ep")
loaded == survey
```

```python theme={null}
True
```

Loading binds the object to its package. A later
`loaded.git.save(message="...")` writes a new version back to the same `.ep`
file unless a different path is supplied.

## Unpack the Git repository

You do not need to unpack a package to use it. Unpacking is useful when you
want to audit the representation or use lower-level Git commands:

```bash theme={null}
ep unpack example-survey.ep
```

The returned JSON includes a temporary directory in `data.path` and a
`next_step` command. Change into that directory:

```bash theme={null}
cd /tmp/path-returned-by-ep-unpack
git status
git log --oneline --decorate
```

<Warning>
  The directory returned by `ep unpack` is a temporary inspection copy. Editing
  it does not rewrite the original `.ep` archive. Make durable changes through
  the loaded EDSL object and call `save()`.
</Warning>

Because `.ep` is ZIP-compatible, a standard archive utility can also confirm
what is present:

```bash theme={null}
unzip -l example-survey.ep
```

The listing includes `.git/HEAD`, `.git/objects/`, `.git/refs/heads/main`, the
manifest, question files, and survey metadata. EDSL omits transient Git files
such as hooks when packing the archive.

## Read the working-tree files

The manifest identifies the format and reconstructs question order:

```json theme={null}
{
  "edsl_class_name": "Survey",
  "edsl_version": "1.0.8.dev1",
  "format": "edsl.survey.git_package",
  "format_version": 1,
  "n_questions": 3,
  "object_type": "Survey",
  "question_order": [
    "000001",
    "000002",
    "000003"
  ]
}
```

The exact `edsl_version` reflects the installed version that wrote the
package.

Each question is stored separately. For example,
`questions/000001.json` contains:

```json theme={null}
{
  "question_name": "q0",
  "question_options": [
    "yes",
    "no"
  ],
  "question_text": "Do you like school?",
  "question_type": "multiple_choice"
}
```

The split representation is deliberate. If a survey has 500 questions and
one changes, Git does not need to treat one giant serialized survey document
as the unit of review. The diff can focus on the affected question and any
metadata that actually changed.

The files under `metadata/` preserve behavior that is not contained in an
individual question:

* `rule_collection.json` stores survey flow, including skip and stop rules.
* `memory_plan.json` records which earlier answers are made available later.
* `question_groups.json` records named groups.

Optional survey properties, when present, are stored in additional metadata
files such as `name.json`, `questions_to_randomize.json`, and
`options_to_pin.json`.

## Make and commit a change

Change the wording of the first question:

```python theme={null}
survey = Survey.git.load("example-survey.ep")
survey.questions[0].question_text = "Do you enjoy school?"

second_save = survey.git.save(
    message="Clarify the first question",
)
second_save
```

The package remains one file, but it now contains two commits. View them
without unpacking:

```python theme={null}
survey.git.log()
```

```python theme={null}
[
    {
        "commit": "d1f44ff624288e48e8835597d247c039ceffeb48",
        "message": "Clarify the first question",
        "timestamp": "2026-07-29 07:47:56 -0400",
    },
    {
        "commit": "65801c7e03eb93c0f31c8bee6b2a29cc9152be14",
        "message": "Create example survey",
        "timestamp": "2026-07-29 07:47:34 -0400",
    },
]
```

Again, hashes and timestamps will differ.

Now ask Git what changed:

```python theme={null}
print(survey.git.diff("HEAD~1", "HEAD"))
```

The essential part of the diff is:

```diff theme={null}
diff --git a/questions/000001.json b/questions/000001.json
index 353b419..33b47da 100644
--- a/questions/000001.json
+++ b/questions/000001.json
@@
-  "question_text": "Do you like school?",
+  "question_text": "Do you enjoy school?",
```

Three identifiers are doing different jobs:

* `0` is the question's current position in the Survey.
* `q0` is its EDSL `question_name`. Prompts, rules, and results use this name
  to refer to the logical question.
* `000001` is its stable package file ID. It is not a position and it is not
  the public question name.

The manifest and the other two question files do not change. A genuinely new
question receives the next available package ID; deleting a question removes
its file from the current tree while retaining it in Git history.

The package ID also survives a pure `question_name` rename:

```python theme={null}
survey.git.branch("rename-demo")
survey.git.switch("rename-demo")
survey = survey.with_renamed_question("q0", "school_enjoyment")
survey.git.save("example-survey.ep", message="Rename q0")
```

```diff theme={null}
diff --git a/questions/000001.json b/questions/000001.json
-  "question_name": "q0",
+  "question_name": "school_enjoyment",
```

That is why the package needs an identity separate from `question_name`:
renaming a question is a one-line edit inside a stable file, not a file rename
and not a new question.

Return to the unchanged main line before continuing:

```python theme={null}
survey.git.switch("main")
```

## What Git stores

The `.git` directory does not contain a second full copy of the survey for
every version. Git stores content-addressed objects:

* **blobs** hold file contents;
* **trees** describe directory snapshots;
* **commits** point to a tree, a parent commit, and commit metadata;
* **refs** such as `main` and tags give memorable names to commits.

From the unpacked repository, inspect the current commit:

```bash theme={null}
git cat-file -p HEAD
```

Representative output:

```text theme={null}
tree <tree-id>
parent <first-commit-id>
author EDSL <edsl@example.invalid> ...
committer EDSL <edsl@example.invalid> ...

Clarify the first question
```

Follow the tree to a stored file:

```bash theme={null}
git ls-tree HEAD
git ls-tree HEAD:questions
git show HEAD:questions/000001.json
git show HEAD~1:questions/000001.json
```

The last two commands read different survey versions directly from Git
without changing the working tree.

Git addresses every stored object by a hash of its contents. Files that did
not change are reused by the new tree instead of being stored as new logical
versions. When repositories are packed or transferred, Git can also delta
compress similar objects. This is why Git can communicate a small survey
revision without treating the entire history as an opaque replacement file.

The `.ep` archive itself is repacked after a save, so copying the file with a
generic file-copy tool still copies the whole archive. The transfer
efficiency comes into play when the embedded repository is synchronized with
a Git remote through EDSL's push and pull operations.

## Load or render an older version

Save the first commit ID before making further changes:

```python theme={null}
first_commit = first_save["commit"]
```

Load that exact historical survey:

```python theme={null}
original = Survey.git.load(
    "example-survey.ep",
    ref=first_commit,
)

original.questions[0].question_text
```

```python theme={null}
"Do you like school?"
```

The current version remains:

```python theme={null}
current = Survey.git.load("example-survey.ep")
current.questions[0].question_text
```

```python theme={null}
"Do you enjoy school?"
```

The CLI can also render a historical version to HTML:

```bash theme={null}
ep open example-survey.ep --ref <first-commit-id> --output original-survey.html
```

This is useful in review or replication: a commit ID identifies the exact
survey being discussed, even after the package has moved forward.

## Name important versions

Commit IDs are precise but not memorable. Add a Git tag to a milestone:

```python theme={null}
survey.git.tag(
    "pilot-v1",
    message="Survey used for the first pilot",
)
```

List tags:

```python theme={null}
survey.git.tags()
```

```python theme={null}
["pilot-v1"]
```

Tags are helpful for statements such as “the preregistered instrument was
`pilot-v1`” or “production used `field-v2`.” A verifier can resolve the tag to
the exact commit and inspect the difference between milestones.

## Explore alternatives with branches

Branches let collaborators develop alternative instruments without
overwriting the main line:

```python theme={null}
survey.git.branch("without-q1")
survey.git.switch("without-q1")
survey.delete_question("q1")
survey.git.save(message="Create variant without q1")
```

Deleting one question changes more than one file:

```python theme={null}
print(survey.git.diff("--stat", "HEAD~1", "HEAD"))
```

```text theme={null}
 manifest.json                 | 3 +--
 metadata/memory_plan.json     | 2 --
 metadata/rule_collection.json | 6 +++---
 questions/000002.json         | 9 ---------
 4 files changed, 4 insertions(+), 16 deletions(-)
```

`questions/000002.json` is deleted, not renumbered. The other question files
retain their IDs. The manifest changes because the question order no longer
includes `000002`; the memory plan and flow rules change because they describe
relationships among the remaining questions. This is why metadata is split
into named files rather than buried in one survey blob: Git shows exactly
which parts of survey behavior changed.

Methods that return a `changed` list use “changed” in Git's broad sense:
touched paths, including additions, modifications, and deletions.

This branch represents a genuine alternative instrument, so do not merge it
merely because it exists. Keep it parallel, field or review it independently,
and return to `main`:

```python theme={null}
survey.git.switch("main")
```

`switch()` refreshes the in-memory `Survey` from the selected branch. EDSL
requires a clean package before switching, pulling, or checking out another
version, which protects uncommitted changes.

## Merge independent work

Merging is appropriate when reviewed work belongs back on the main survey.
The useful case is not a sequential edit forced into a merge commit. It is two
authors making divergent commits from the same base.

Alice and Bob begin with the current `main`. Create Bob's branch before either
person edits:

```python theme={null}
survey.git.branch("bob-options")
```

Bob changes only the options for `q2`:

```python theme={null}
survey.git.switch("bob-options")
survey.questions[2].question_options.append("not sure")
survey.git.save(message="Bob: add a q2 response option")
```

Meanwhile, Alice changes only the wording of `q0` on `main`:

```python theme={null}
survey.git.switch("main")
survey.questions[0].question_text = "Do you enjoy attending school?"
survey.git.save(message="Alice: clarify q0 wording")
```

The histories now diverge. Neither commit is an ancestor of the other, and the
edits live in different question files:

```text theme={null}
* Alice: clarify q0 wording                 questions/000001.json
| * Bob: add a q2 response option           questions/000003.json
|/
* Clarify the first question
```

Now perform a genuine three-way merge:

```python theme={null}
merge_info = survey.git.merge(
    "bob-options",
    message="Merge Bob's reviewed options",
)
merge_info
```

```python theme={null}
{
    "status": "ok",
    "path": "example-survey.ep",
    "commit": "<merge-commit>",
    "branch": "main",
    "merged_ref": "bob-options",
    "fast_forward": False,
    "changed": ["questions/000003.json"],
    "conflicts": [],
    "conflict_contents": {},
    "aborted": False,
    "message": "Merge Bob's reviewed options",
}
```

No `no_ff=True` is needed: Alice's and Bob's commits are genuinely divergent,
so Git must reconcile them. `changed` lists the paths introduced to Alice's
pre-merge `main`; her `q0` edit was already there, while Bob's `q2` edit arrives
through the merge.

The method completes all three layers of the operation: it merges the embedded
repository, reloads the merged files into the in-memory `Survey`, and repacks
the repository into `example-survey.ep`. The object is ready to use
immediately:

```python theme={null}
survey.question_names
```

```python theme={null}
["q0", "q1", "q2"]
```

Both authors' changes survive:

```python theme={null}
survey.questions[0].question_text
survey.questions[2].question_options
```

```python theme={null}
"Do you enjoy attending school?"
["**lack*** of killer bees in cafeteria", "other", "not sure"]
```

From an unpacked inspection copy, Git can display the topology:

```bash theme={null}
git log --graph --oneline --all
```

```text theme={null}
*   <merge> Merge Bob's reviewed options
|\
| * <bob> Bob: add a q2 response option
* | <alice> Alice: clarify q0 wording
|/
*   <base> Clarify the first question
```

This is the payoff from storing questions separately. Git performs a real
three-way merge, sees that Alice and Bob changed disjoint files, and retains
both changes without human intervention. With one large `survey.json`, Git
would be reconciling edits inside the same 4,000-line blob.

### What happens on a conflict

Now repeat the experiment with both authors editing `q0`. Starting from a
one-question package makes the conflict especially clear:

```python theme={null}
from edsl import QuestionFreeText, Survey

wording = Survey([
    QuestionFreeText(
        question_name="q0",
        question_text="Original wording",
    )
])
wording.git.save("wording.ep", message="Shared wording")
wording.git.branch("bob-wording")

wording.git.switch("bob-wording")
wording.questions[0].question_text = "Bob's wording"
wording.git.save(message="Bob: revise q0")

wording.git.switch("main")
wording.questions[0].question_text = "Alice's wording"
wording.git.save(message="Alice: revise q0")

conflict = wording.git.merge("bob-wording")
```

```python theme={null}
{
    "status": "conflict",
    "path": "wording.ep",
    "commit": "<unchanged-main-commit>",
    "branch": "main",
    "merged_ref": "bob-wording",
    "fast_forward": False,
    "changed": [],
    "conflicts": ["questions/000001.json"],
    "conflict_contents": {
        "questions/000001.json": "<conflicted text shown below>"
    },
    "aborted": True,
    "message": "merge of bob-wording aborted due to conflicts",
}
```

EDSL captures the conflicted file before aborting. Its `question_text` conflict
is local and legible:

```diff theme={null}
{
  "question_name": "q0",
 <<<<<<< HEAD
  "question_text": "Alice's wording",
 =======
  "question_text": "Bob's wording",
 >>>>>>> bob-wording
  "question_type": "free_text"
}
```

This is a conflict that a reviewer can understand and resolve. It is isolated
to one small question file rather than buried among thousands of unrelated
survey lines. The marker lines are indented one space above only so Git does
not mistake this documentation example for an unresolved repository conflict.

EDSL then aborts the merge, leaving the current branch and `.ep` archive
unchanged and clean. A researcher can use `conflict_contents` to inspect the
two alternatives, revise one branch, and try again. The API never silently
chooses a survey design or leaves the portable package in a half-merged state.

## Communicate only the changes

An `.ep` file is a self-contained handoff: its recipient gets both the current
Survey and its history. For ongoing collaboration, attach a GitHub remote and
push:

```python theme={null}
survey.git.remote_add("origin", "git@github.com:team/research-survey.git")
survey.git.push("origin")
```

What lands on GitHub is the unpacked tree of readable JSON, not an opaque ZIP
diff. A collaborator can review a one-line question-wording change as an
ordinary pull request. Git negotiates which objects the receiver already has
and transfers the missing objects, while EDSL repacks the history into `.ep`
when it returns to the portable-file workflow. Zipping does not defeat the
point: the archive is the transport artifact; the embedded repository is the
collaboration artifact.

## Working inside another Git repository

An `.ep` package contains its own Git repository. If it is placed inside a
different Git repository, the outer Git repository sees an embedded
repository. EDSL warns about this because the relationship should be
intentional.

Usually, choose one of these approaches:

* Ignore the `.ep` package in the outer repository and let its own Git history
  and remote manage it.
* Intentionally manage it as a Git submodule.
* Store an exported JSON snapshot in the outer repository when independent
  `.ep` history is unnecessary.

For the first option, a loaded package can update the outer `.gitignore`:

```python theme={null}
survey.git.ignore_in_parent()
```

## Why this design is useful

For survey research, a Git-backed object provides several practical
guarantees:

1. **Exact references.** A commit hash identifies the complete survey,
   including questions, order, rules, memory, and groups.
2. **Readable review.** Questions and metadata are formatted JSON, so a
   reviewer can see wording and logic changes as ordinary diffs.
3. **Focused changes.** Unchanged questions retain their stored objects;
   revisions affect the relevant question and dependent metadata.
4. **Recoverable history.** Old instruments remain loadable even after the
   working version changes.
5. **Named milestones.** Tags can identify preregistration, pilot, field, and
   publication versions.
6. **Parallel development.** Branches can hold experimental variants.
7. **Efficient synchronization.** Git remotes exchange missing objects rather
   than treating every revision as an unrelated dataset.
8. **Portable provenance.** Copying one `.ep` file carries the current object
   and its complete embedded history.

The result works at two levels. Researchers use a normal EDSL `Survey`;
reviewers and collaborators see ordinary Git history and readable diffs.
Agents can understand and operate the versioning plumbing so researchers do
not have to—but the repository remains available whenever a human needs to
audit exactly what changed.

## Command reference

```python theme={null}
from edsl import Survey

# Create and save
survey = Survey.example()
info = survey.git.save("example-survey.ep", message="Initial survey")

# Load latest or historical state
latest = Survey.git.load("example-survey.ep")
older = Survey.git.load("example-survey.ep", ref=info["commit"])

# Inspect history and changes
latest.git.status()
latest.git.log()
latest.git.diff("HEAD~1", "HEAD")

# Mark and navigate versions
latest.git.tag("pilot-v1", message="Pilot instrument")
latest.git.branch("alternative")
latest.git.switch("alternative")
# Edit and save the alternative, then merge it into main.
latest.git.switch("main")
latest.git.merge("alternative", message="Merge alternative", no_ff=True)

# Validate and synchronize
latest.git.validate()
latest.git.remotes()
latest.git.push()
latest.git.pull()
```

```bash theme={null}
# Inspect without writing
ep inspect example-survey.ep

# Unpack a temporary audit copy
ep unpack example-survey.ep

# Render the latest or a historical version
ep open example-survey.ep --output survey.html
ep open example-survey.ep --ref <commit> --output old-survey.html

# Use the package in a job
ep run --survey example-survey.ep --model test --output results.ep
```


# Survey messages
Source: https://docs.expectedparrot.com/en/latest/survey_messages

Use SurveyMessage to add display-only, routable pages such as introductions, consent text, transitions, eligibility notices, and completion messages to an EDSL Survey.

`SurveyMessage` adds a page of information to a survey without asking the respondent—or a language model—to provide an answer. It behaves like a question in the survey flow: it has a `question_name`, occupies a normal survey position, and can be the source or destination of a routing rule.

Use it for introductions, consent or privacy information, section transitions, eligibility notices, and thank-you pages.

## Basic usage

```python theme={null}
from edsl import QuestionFreeText, Survey, SurveyMessage

welcome = SurveyMessage(
    question_name="welcome",
    question_text="Welcome. This survey takes about two minutes.",
)

feedback = QuestionFreeText(
    question_name="feedback",
    question_text="What could we improve?",
)

thanks = SurveyMessage(
    question_name="thanks",
    question_text="Thank you for your feedback.",
)

survey = Survey([welcome, feedback, thanks])
```

An interactive survey client should display the message and offer a **Continue** action (or **Finish** when it is the last node). `SurveyMessage` does not render an input control.

## Routing to a message

Because a message is a normal survey node, rules can route respondents to it by name. This example sends ineligible respondents to a terminal message while eligible respondents continue to a follow-up question:

```python theme={null}
from edsl import EndOfSurvey, QuestionFreeText, QuestionYesNo, Survey, SurveyMessage

eligible = QuestionYesNo(
    question_name="eligible",
    question_text="Are you at least 18 years old?",
)

follow_up = QuestionFreeText(
    question_name="experience",
    question_text="Tell us about your experience.",
)

ineligible = SurveyMessage(
    question_name="ineligible",
    question_text="You are not eligible for this survey. Thank you for your time.",
)

survey = Survey([eligible, follow_up, ineligible])
survey.add_rule(
    "eligible",
    "{{ eligible.answer }} == 'No'",
    next_question="ineligible",
)
survey.add_rule("experience", "True", next_question=EndOfSurvey)
```

The message participates in ordinary navigation, including back navigation and resume behavior in clients that support those features.

## Running with language models

When an EDSL job reaches a `SurveyMessage`, EDSL records a deterministic response and continues without calling the language model:

```python theme={null}
results = survey.run()
results.select("answer.ineligible")
```

If the message was visited, its recorded answer is `"continued"`. If routing skipped it, it has no response. The deterministic value makes the visited path observable in `Results` without spending inference tokens on display-only content.

## `SurveyMessage` versus `Instruction`

| Use | `SurveyMessage` | `Instruction` |
| - | - | - |
| Show a standalone page to a respondent | Yes | No |
| Route to or from the item by question name | Yes | No |
| Record that the item was visited | Yes, as `"continued"` | No |
| Supply context to later model questions | No | Yes |

Use `Instruction` when text should become model context for later questions. Use `SurveyMessage` when the text is itself a visible step in the survey experience.

## Human survey clients

`SurveyMessage` serializes with `question_type="survey_message"` and has an empty Humanize schema because it accepts no respondent input. A survey builder or Humanize client must explicitly support this question type by rendering the message and submitting the deterministic continuation action. Check client compatibility before deploying a human survey that contains messages.


# Surveys
Source: https://docs.expectedparrot.com/en/latest/surveys

A Survey is a collection of [Questions](/en/latest/questions) administered to one or more AI [Agents](/en/latest/agents) and [Language Models](/en/latest/language_models). Questions run asynchronously by default, or according to rules like skip/stop logic, with optional memory of prior answers.

Surveys collect data, generate content, or perform other tasks. Results are stored in a Results object for analysis, with responses cached locally or remotely for reuse. See the [Results](/en/latest/results) and [Remote Caching](/en/latest/remote_caching) sections for details.

Surveys can also contain display-only, routable steps using [`SurveyMessage`](/en/latest/survey_messages), for example introductions, eligibility notices, section transitions, and completion pages.

## Key steps

The key steps to creating and conducting a survey are:

<Steps>
  <Step>
    Create Questions of various types (multiple choice, checkbox, free text, numerical, linear scale, etc.) and combine them in a Survey to administer them together.
  </Step>

  <Step>
    *Optional:* Add rules to skip, stop or administer questions based on conditional logic, or pipe context of questions and answers into other questions.
  </Step>

  <Step>
    *Optional:* Design personas for AI Agents to answer the questions.
  </Step>

  <Step>
    Send the survey to language Models of your choice to generate the responses.
  </Step>
</Steps>

Sending a survey to a language model generates a dataset of Results that includes the responses and other components of the survey. Results can be analyzed and visualized using [built-in methods](/en/latest/results) of the Results object.

## Key methods

A survey is administered by calling the run() method on the Survey object, after adding any agents, scenarios and models with the by() method, and any rules or memory with the appropriate methods (see examples of each below):

* add\_skip\_rule() - Skip a question based on a conditional expression (e.g., based on a response to another question).
* add\_stop\_rule() - End the survey based on a conditional expression.
* add\_rule() - Administer a specified question next based on a conditional expression.
* set\_full\_memory\_mode() - Include a memory of all prior questions/answers at each new question in the survey.
* set\_lagged\_memory() - Include a memory of a specified number of prior questions/answers at each new question in the survey.
* add\_targeted\_memory() - Include a memory of a particular question/answer at another question in the survey.
* add\_memory\_collection() - Include memories of a set of prior questions/answers at another question in the survey.

Before running a survey, you can also estimate the cost of running it with the estimate\_job\_cost() method of a Job object (a survey combined with one or more models). The show\_prompts(), show\_rules() and show\_flow() methods can be used to analyze the structure of a survey and the rules that have been applied to it:

* show\_prompts() - Display the user and system prompts for each question in a survey. This is a companion method to the prompts() method of a Job object, which returns a dataset containing the prompts together with information about each question, scenario, agent, model and estimated cost.
* show\_rules() - Display a table of the conditional rules that have been applied to a survey.
* show\_flow() - Display a graphic of the flow of a survey, showing the order of questions and any rules that have been applied, and any scenarios and/or agent information that has been added.

When you run a survey you can choose to run it remotely at the Expected Parrot server or locally on your own machine. When you run a job remotely you automatically have access to a universal remote cache of stored responses. This allows you to reuse responses from prior jobs, which can save time and resources (cached responses are retrieved for free). Learn more in the [Remote Inference](/en/latest/remote_inference) and [Remote Caching](/en/latest/remote_caching) sections.

You can also choose to run a remote survey in the background by passing the background=True parameter to the run() method. This allows you to continue working (or stop working) while your job completes. You can check progress at the progress bar page or by calling the fetch() method at any time.

### Piping

You can pipe individual components of questions into other questions, such as inserting the answer to a question in the question text of another question. This is done by using the \{\{ question\_name.answer }} syntax in the text of a question, and is useful for creating dynamic surveys that reference prior answers.

Note that this method is different from memory rules, which automatically inlude the full context of a specified question at a new question in the survey: *“Before the question you are now answering, you already answered the following question(s): Question: \<question\_text> Answer: \<answer>”*. See examples below.

## Constructing a survey

In the examples below we construct a simple survey of questions, and then demonstrate how to run it with various rules and memory options, how to add AI agents and language models, and how to analyze the results.

### Importing from Qualtrics

You can import surveys from Qualtrics QSF (Qualtrics Survey Format) files using the `from_qsf()` method:

```python theme={null}
from edsl import Survey

# Import a Qualtrics survey
survey = Survey.from_qsf("path/to/survey.qsf")
```

This method parses the QSF file and converts Qualtrics questions into their corresponding EDSL question types, preserving question text, options, and basic logic where possible.

### Defining questions

Questions can be defined as various types, including multiple choice, checkbox, free text, linear scale, numerical and other types. The formats are defined in the [Questions](/en/latest/questions) module. Here we define some questions by importing question types and creating instances of them:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, QuestionCheckBox, QuestionLinearScale, QuestionNumerical

q1 = QuestionMultipleChoice(
   question_name = "consume_local_news",
   question_text = "How often do you consume local news?",
   question_options = ["Daily", "Weekly", "Monthly", "Never"]
)

q2 = QuestionCheckBox(
   question_name = "sources",
   question_text = "What are your most common sources of local news? (Select all that apply)",
   question_options = ["Television", "Newspaper", "Online news websites", "Social Media", "Radio", "Other"]
)

q3 = QuestionLinearScale(
   question_name = "rate_coverage",
   question_text = "On a scale of 1 to 10, how would you rate the quality of local news coverage in your area?",
   question_options = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
   option_labels = {1: "Very poor", 10: "Excellent"}
)

q4 = QuestionNumerical(
   question_name = "minutes_reading",
   question_text = "On average, how many minutes do you spend consuming local news each day?",
   min_value = 0, # optional
   max_value = 1440 # optional
)
```

### Adding questions to a survey

Questions are passed to a Survey object as a list of question ids:

```python theme={null}
from edsl import Survey

survey = Survey(questions = [q1, q2, q3, q4])
```

Alternatively, questions can be added to a survey one at a time:

```python theme={null}
from edsl import Survey

survey = Survey().add_question(q1).add_question(q2).add_question(q3).add_question(q4)
```

### Organizing questions into groups

You can organize questions into logical groups using the `add_question_group()` method. This is useful for:

* Analyzing responses by section
* Organizing survey structure
* Visualizing survey flow

```python theme={null}
from edsl import Survey

survey = Survey(questions = [q1, q2, q3, q4])

# Create a group for the first two questions
survey = survey.add_question_group(
    start_question="consume_local_news",
    end_question="sources", 
    group_name="news_consumption"
)

# Create another group for the remaining questions
survey = survey.add_question_group(
    start_question="rate_coverage",
    end_question="minutes_reading",
    group_name="news_quality"
)
```

Groups are defined by contiguous ranges of questions and cannot overlap. Group names must be valid Python identifiers and cannot conflict with existing question names.

### Randomizing question options

If your survey includes multiple choice or checkbox questions, you can randomize the order of the options by passing a list of the question names to the questions\_to\_randomize parameter of the question constructor. For example, here we randomize the order of the options for q1 and q2:

```python theme={null}
from edsl import Survey

survey = Survey(questions = [q1, q2, q3, q4],
                questions_to_randomize=["consume_local_news", "sources"])
```

This will randomize the order of the options for both questions when they are administered.

## Running a survey

Once constructed, a survey can be administered by calling the run() method. If question [Scenarios](/en/latest/scenarios), [Agents](/en/latest/agents) or [Language Models](/en/latest/language_models) have been specified, they are added to the survey with the by method when running it. (If no language model is specified, the survey will be run with the default model, which can be inspected by running Model().)

For example, here we run the survey with a simple agent persona and specify that GPT-4o should be used. Note that the agent and model can be added in either order, so long as each type of component is added at once (e.g., if using multiple agents or models, pass them as a list to the by() method):

```python theme={null}
from edsl import Agent, Model

agent = Agent(traits = {"persona": "You are a teenager who hates reading."})

model = Model("gpt-4o")

results = survey.by(agent).by(model).run()
```

## Job status information

When you run a survey, you will see a table of information about the job status. When the job completes, you can access the Results object that is generated in your workspace and at your account (if the survey is run remotely). You can specify the visibility and description of the results when running a survey remotely by passing the remote\_inference\_results\_visibility and remote\_inference\_description parameters to the run() method, and modify them at your account at any time.

For example, results of the above survey can be viewed at the following page which has been made public: [https://www.expectedparrot.com/content/4cfcf0c6-6aff-4447-90cb-cd9e01111a28](https://www.expectedparrot.com/content/4cfcf0c6-6aff-4447-90cb-cd9e01111a28).

### Progress Report

While a job is running you can view updates in a Progress Report with details on questions completed, models used and any exceptions generated. If remote inference is activated, a link to a Progress Report will appear automatically in the job status table. If you are running a survey locally, you can pass run(progress\_bar=True) to view a report locally.

### Exceptions Report

If any exceptions are generated you can view details about them in an Exceptions Report, which includes information about the questions, agents, scenarios and models that generated exceptions together with error messages and tracebacks. If remote inference is activated, a link to an Exceptions Report will appear automatically in the job status table. If you are running a survey locally, the report details will appear in your console.

### Running jobs in the background

If you are running a survey remotely, you can choose to run it in the background by passing the background=True parameter to the run() method:

```python theme={null}
results = survey.by(agent).by(model).run(background=True)
```

This allows you to continue working (or stop working) while your job completes. You can check progress at any time at the progress bar page or by calling the fetch() method, which checks for the results every 1.0 seconds by default or a specified interval:

```python theme={null}
results = survey.by(agent).by(model).run(background=True)

# check progress every 5 seconds
results.fetch(polling_interval = 5.0)
```

### Optional parameters

There are optional parameters that can be passed to the run() method, including:

* fresh=False - A boolean value to indicate whether to run the survey with fresh responses (default is False). Example: run(fresh=True) will generate fresh responses for each question.
* n - The number of responses to generate for each question (default is 1). Example: run(n=5) will administer the same exact question (and scenario, if any) to an agent and model 5 times.
* cache - A Cache object to use for caching responses (default is None). Example: run(cache=my\_cache) will use the specified cache to store responses.
* disable\_remote\_inference - A boolean value to indicate whether to run the survey locally while remote inference is activated (default is False). Example: run(disable\_remote\_inference=True).
* remote\_cache\_description - A string value to describe the entries in the remote cache when the survey is run remotely. This description will be displayed on the Expected Parrot server and can be used to provide context for the survey. Example: run(remote\_cache\_description=”This is a survey about local news consumption.”).
* remote\_inference\_description - A string value to describe the survey when it is run remotely. This description will be displayed on the Expected Parrot server and can be used to provide context for the survey. Example: run(remote\_inference\_description=”This is a survey about local news consumption.”).
* remote\_inference\_results\_visibility - A string value to indicate the visibility of the results on the Expected Parrot server, when a survey is being run remotely. Possible values are “public”, “unlisted” or “private” (default is “unlisted”). Visibility can also be modified on the platform. Example: run(remote\_inference\_results\_visibility=”public”).
* progress\_bar=True - This parameter can be used to view a Progress Report locally. A link to a Progress Report will automatically be provided when you run a survey remotely.
* background=True - This parameter can be used to run a survey in the background, allowing you to continue working (or stop working) while your job completes.
* polling\_interval - This parameter can be used to specify the interval (in seconds) at which to check for results when running a survey in the background. The default is 1.0 seconds. Example: run(polling\_interval=5.0) will check for results every 5 seconds.
* verbose=True - A boolean value to indicate whether to enable verbose logging (including a table of information about the job that is running) (default is True). Example: run(verbose=False) will disable verbose logging.

## Survey rules & logic

Rules can be applied to a survey with the add\_skip\_rule(), add\_stop\_rule() and add\_rule() methods, which take a logical expression and the relevant questions.

### Skip rules

The add\_skip\_rule() method skips a question if a condition is met. The (2) required parameters are the question to skip and the condition to evaluate.

Here we use add\_skip\_rule() to skip q2 if the response to “consume\_local\_news” is “Never”. Note that we can refer to the question to be skipped using either the id (“q2”) or question\_name (“consume\_local\_news”):

```python theme={null}
from edsl import Survey

survey = Survey(questions = [q1, q2, q3, q4])
survey = survey.add_skip_rule(q2, "consume_local_news == 'Never'")
```

This is equivalent:

```python theme={null}
from edsl import Survey

survey = Survey(questions = [q1, q2, q3, q4])
survey = survey.add_skip_rule("sources", "consume_local_news == 'Never'")
```

We can run the survey and verify that the rule was applied:

```python theme={null}
results = survey.by(agent).by(model).run() # using the agent and model from the previous example
results.select("consume_local_news", "sources", "rate_coverage", "minutes_reading")
```

This will print the answers, showing “None” for a skipped question (your own results for answers may vary):

| answer.consume\_local\_news | answer.sources | answer.rate\_coverage | answer.minutes\_reading |
| :- | :- | :- | :- |
| Never | None | 4 | 0 |

### Stop rules

The add\_stop\_rule() method stops the survey if a condition is met. The (2) required parameters are the question to stop at and the condition to evaluate.

Here we use add\_stop\_rule() to end the survey at q1 if the response is “Never” (note that we recreate the survey to demonstrate the stop rule alone):

```python theme={null}
survey = Survey(questions = [q1, q2, q3, q4])
survey = survey.add_stop_rule(q1, "consume_local_news == 'Never'")
```

This time we see that the survey ended when the response to “consume\_local\_news” was “Never”:

```python theme={null}
results = survey.by(agent).run()
results.select("consume_local_news", "sources", "rate_coverage", "minutes_reading")
```

Output:

| answer.consume\_local\_news | answer.sources | answer.rate\_coverage | answer.minutes\_reading |
| :- | :- | :- | :- |
| Never | None | None | None |

### Other rules

The generalizable add\_rule() method is used to specify the next question to administer based on a condition. The (3) required parameters are the question to evaluate, the condition to evaluate, and the question to administer next.

Here we use add\_rule() to specify that if the response to “color” is “Blue” then q4 should be administered next:

```python theme={null}
survey = Survey(questions = [q1, q2, q3, q4])
survey = survey.add_rule(q1, "consume_local_news == 'Never'", q4)
```

We can run the survey and verify that the rule was applied:

```python theme={null}
results = survey.by(agent).run()
results.select("consume_local_news", "sources", "rate_coverage", "minutes_reading")
```

We can see that both q2 and q3 were skipped but q4 was administered (and the response makes sense for the agent):

| answer.consume\_local\_news | answer.sources | answer.rate\_coverage | answer.minutes\_reading |
| :- | :- | :- | :- |
| Never | None | None | 0 |

### Show flow

We can call the show\_flow() method to display a graphic of the flow of the survey, and verify how any rules were applied. For example, here we show the flow of the survey above with the skip rule applied:

```python theme={null}
survey = Survey(questions = [q1, q2, q3, q4])

survey = survey.add_skip_rule(q2, "consume_local_news == 'Never'")

survey.show_flow()
```

Output:

<Frame>
  <img alt="Survey Flow Diagram with Skip Rule" />
</Frame>

If we add agent or scenario details to the survey questions, the flow diagram will also show this information. Here we modify the survey questions from above to include an agent trait:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, QuestionCheckBox, QuestionLinearScale, QuestionNumerical, Survey, Agent

agent = Agent(traits = {
   "nickname": "Robin",
   "persona": "You are a teenager who hates reading."
})

# Adding the nickname to the question texts
q1 = QuestionMultipleChoice(
   question_name = "consume_local_news",
   question_text = "Hey {{ agent.nickname }}, how often do you consume local news?",
   question_options = ["Daily", "Weekly", "Monthly", "Never"]
)

q2 = QuestionCheckBox(
   question_name = "sources",
   question_text = "{{ agent.nickname }}, what are your most common sources of local news? (Select all that apply)",
   question_options = ["Television", "Newspaper", "Online news websites", "Social Media", "Radio", "Other"]
)

q3 = QuestionLinearScale(
   question_name = "rate_coverage",
   question_text = "{{ agent.nickname }}, on a scale of 1 to 10, how would you rate the quality of local news coverage in your area?",
   question_options = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
   option_labels = {1: "Very poor", 10: "Excellent"}
)

q4 = QuestionNumerical(
   question_name = "minutes_reading",
   question_text = "{{ agent.nickname }}, on average, how many minutes do you spend consuming local news each day?",
   min_value = 0, # optional
   max_value = 1440 # optional
)

survey = Survey(questions = [q1, q2, q3, q4]).add_stop_rule(q1, "consume_local_news == 'Never'")

job = survey.by(agent)

job.show_flow()
```

Output:

<Frame>
  <img alt="Survey Flow Diagram with Agent Information" />
</Frame>

### Conditional expressions

The rule expressions themselves (“consume\_local\_news == ‘Never’”) are written in Python. An expression is evaluated to True or False, with the answer substituted into the expression. The placeholder for this answer is the name of the question itself. In the examples, the answer to q1 is substituted into the expression “consume\_local\_news == ‘Never’”, as the name of q1 is “consume\_local\_news”.

## Piping

Piping is a method of explicitly referencing components of a question in a later question. For example, here we use the answer to q0 in the prompt for q1:

```python expandable theme={null}
from edsl import QuestionFreeText, QuestionList, Survey, Agent

q0 = QuestionFreeText(
   question_name = "color",
   question_text = "What is your favorite color?",
)

q1 = QuestionList(
   question_name = "examples",
   question_text = "Name some things that are {{ color.answer }}.",
)

survey = Survey([q0, q1])

agent = Agent(traits = {"persona": "You are a botanist."})

results = survey.by(agent).run()

results.select("color", "examples")
```

In this example, q0 will be administered before q1 and the response to q0 is piped into q1. Output:

| answer.color | answer.examples |
| :- | :- |
| As a botanist, I find myself drawn to the vibrant greens of nature. Green is a color that symbolizes growth, life, and the beauty of plants, which are central to my work and passion. | \[‘Leaves’, ‘Grass’, ‘Ferns’, ‘Moss’, ‘Green algae’] |

If an answer is a list, we can use the list as the question\_options in another question, or index items individually. Here we demonstrate examples of both:

```python expandable theme={null}
from edsl import QuestionList, QuestionFreeText, QuestionMultipleChoice, Survey, Agent

q_colors = QuestionList(
   question_name = "colors",
   question_text = "What are your 3 favorite colors?",
   max_list_items = 3
)

q_examples = QuestionFreeText(
   question_name = "examples",
   question_text = "Name some things that are {{ colors.answer }}",
)

q_favorite = QuestionMultipleChoice(
   question_name = "favorite",
   question_text = "Which is your #1 favorite color?",
   question_options = [
      "{{ colors.answer[0] }}",
      "{{ colors.answer[1] }}",
      "{{ colors.answer[2] }}",
   ]
)

survey = Survey([q_colors, q_examples, q_favorite])

agent = Agent(traits = {"persona": "You are a botanist."})

results = survey.by(agent).run()

results.select("colors", "examples", "favorite")
```

Output:

| answer.colors | answer.examples | answer.favorite |
| :- | :- | :- |
| \['Green', 'Brown', 'Yellow'] | Certainly! Here are some things that can be green, brown, or yellow: <br /> **Green:** 1. Leaves - Many plants have green leaves due to chlorophyll, which is essential for photosynthesis. 2. Grass - Typically green, especially when healthy and well-watered. 3. Green Apples - Varieties like Granny Smith are known for their green color. <br /> **Brown:** 1. Tree Bark - The outer layer of trees is often brown, providing protection. 2. Soil - Many types of soil appear brown, indicating organic matter. 3. Acorns - These seeds from oak trees are generally brown when mature. <br /> **Yellow:** 1. Sunflowers - Known for their bright yellow petals. 2. Bananas - Yellow when ripe and ready to eat. 3. Daffodils - These flowers are often a vibrant yellow, heralding spring. | Green |

### Piping with additional options

In addition to piping answer lists directly as `question_options`, you can use a dict format to pipe options while adding additional static options.
This is useful when you want to include piped options plus additional choices like "None of the above" or "Other".

The dict format uses two keys:

* `"from"` - A template string referencing the piped options (e.g., `"{{ q1.answer }}"`).
* `"add"` - A list of additional static options to append

For example:

```python expandable theme={null}
from edsl import QuestionList, QuestionMultipleChoice, Survey, Agent

q1 = QuestionList(
   question_name = "colors",
   question_text = "What are your 3 favorite colors?",
   max_list_items = 3
)

# Use dict format to pipe options from q1 AND add additional options
q2 = QuestionMultipleChoice(
   question_name = "top_choice",
   question_text = "Which is your #1 favorite color?",
   question_options = {
      "from": "{{ colors.answer }}",
      "add": ["None of the above", "Other"]
   }
)

survey = Survey([q1, q2])

agent = Agent(traits = {"persona": "You are a botanist."})

results = survey.by(agent).run()

results.select("colors", "top_choice")
```

In this example, q2 will have the three colors from q1's answer, plus "None of the above" and "Other" as additional options.

Output:

| answer.colors | answer.top\_choice |
| :- | :- |
| \['Green', 'Brown', 'Yellow'] | Green |

The dict format also works when piping from scenario data:

```python theme={null}
from edsl import Scenario, QuestionMultipleChoice

scenario = Scenario({"available_colors": ["Red", "Blue", "Green"]})

q = QuestionMultipleChoice(
   question_name = "favorite_color",
   question_text = "What's your favorite color?",
   question_options = {
      "from": "{{ scenario.available_colors }}",
      "add": ["Other", "Prefer not to say"]
   }
)

results = q.by(scenario).run()
```

### Agent traits

This can also be done with agent traits. For example:

```python theme={null}
from edsl import Agent, QuestionFreeText

a = Agent(traits = {'first_name': 'John'})

q = QuestionFreeText(
   question_text = 'What is your last name, {{ agent.first_name }}?',
   question_name = "last_name"
)

job = q.by(a)

job.prompts().select('user_prompt')
```

This code will output the text of the prompt for the question:

```python theme={null}
What is your last name, John?
```

We can also show both system and user prompts together with information about the question, agent and model by calling the show\_prompts() method:

```python theme={null}
job.show_prompts()
```

Output:

| user\_prompt | system\_prompt | interview\_index | question\_name | scenario\_index | agent\_index | model | estimated\_cost |
| :- | :- | :- | :- | :- | :- | :- | :- |
| What is your last name, John? | You are answering questions as if you were a human. Do not break character. You are an agent with the following persona: \{‘first\_name’: ‘John’} | 0 | last\_name | 0 | 0 | gpt-4o | 0.0005375 |

## Question memory

When an agent is taking a survey, they can be prompted to “remember” answers to previous questions. This can be done in several ways:

### Full memory

The method set\_full\_memory\_mode() gives the agent all of the prior questions and answers at each new question in the survey, i.e., the first question and answer are included in the memory when answering the second question, both the first and second questions and answers are included in the memory when answering the third question, and so on. The method is called on the survey object:

```python theme={null}
survey = Survey(questions = [q1, q2, q3, q4])
survey = survey.set_full_memory_mode()
```

In the results, we can inspect the \_user\_prompt for each question to see that the agent was prompted to remember all of the prior questions:

```python theme={null}
results = survey.by(agent).run()
(
   results
   .select("consume_local_news_user_prompt", "sources_user_prompt", "rate_coverage_user_prompt", "minutes_reading_user_prompt")
)
```

This will print the prompt that was used for each question, and we can see that each successive prompt references all prior questions and answers that were given:

| prompt.consume\_local\_news\_user\_prompt | prompt.sources\_user\_prompt | prompt.rate\_coverage\_user\_prompt | prompt.minutes\_reading\_user\_prompt |
| :- | :- | :- | :- |
| How often do you consume local news? <br /> Daily Weekly Monthly Never <br /> Only 1 option may be selected. <br /> Respond only with a string corresponding to one of the options. <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | What are your most common sources of local news? <br /> 0: Television <br /> 1: Newspaper <br /> 2: Online news websites <br /> 3: Social Media <br /> 4: Radio <br /> 5: Other <br /> Please respond only with a comma-separated list of the code of the options that apply, with square brackets. E.g., \[0, 1, 3] <br /> After the answer, you can put a comment explaining your choice on the next line. | On a scale of 1 to 10, how would you rate the quality of local news coverage in your area? <br /> 1 : Very poor <br /> 10 : Excellent <br /> Only 1 option may be selected. <br /> Respond only with the code corresponding to one of the options. E.g., “1” or “10” by itself. <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | On average, how many minutes do you spend consuming local news each day? <br /> Minimum answer value: 0 <br /> Maximum answer value: 1440 <br /> This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, …). <br /> Respond with just your number on a single line. <br /> If your response is equivalent to zero, report ‘0’. <br /> After the answer, put a comment explaining your choice on the next line. |

Note that this is slow and token-intensive, as the questions must be answered serially and requires the agent to remember all of the answers to the questions in the survey. In contrast, if the agent does not need to remember all of the answers to the questions in the survey, execution can proceed in parallel.

### Lagged memory

The method set\_lagged\_memory() gives the agent a specified number of prior questions and answers at each new question in the survey; we pass it the number of prior questions and answers to remember. Here we use it to give the agent just 1 prior question/answer at each question:

```python theme={null}
survey = Survey(questions = [q1, q2, q3, q4])
survey = survey.set_lagged_memory(1)
```

We can inspect each \_user\_prompt again and see that the agent is only prompted to remember the last prior question/answer:

```python theme={null}
results = survey.by(agent).run()

(
   results
   .select("consume_local_news_user_prompt", "sources_user_prompt", "rate_coverage_user_prompt", "minutes_reading_user_prompt")
)
```

This will print the prompts for each question:

| prompt.consume\_local\_news\_user\_prompt | prompt.sources\_user\_prompt | prompt.rate\_coverage\_user\_prompt | prompt.minutes\_reading\_user\_prompt |
| :- | :- | :- | :- |
| How often do you consume local news? <br /> Daily Weekly Monthly Never <br /> Only 1 option may be selected. <br /> Respond only with a string corresponding to one of the options. <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | What are your most common sources of local news? <br /> 0: Television <br /> 1: Newspaper <br /> 2: Online news websites <br /> 3: Social Media <br /> 4: Radio <br /> 5: Other <br /> Please respond only with a comma-separated list of the code of the options that apply, with square brackets (e.g., \[0, 1, 3]). <br /> After the answer, you can put a comment explaining your choice on the next line. | On a scale of 1 to 10, how would you rate the quality of local news coverage in your area? <br /> 1 : Very poor <br /> 10 : Excellent <br /> Only 1 option may be selected. <br /> Respond only with the code corresponding to one of the options (e.g., “1” or “10”). <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | On average, how many minutes do you spend consuming local news each day? <br /> Minimum answer value: 0 <br /> Maximum answer value: 1440 <br /> This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, …). <br /> Respond with just your number on a single line. <br /> If your response is equivalent to zero, report ‘0’. <br /> After the answer, put a comment explaining your choice on the next line. |

### Targeted memory

The method add\_targeted\_memory() gives the agent a targeted prior question and answer when answering another specified question. We pass it the question to answer and the prior question/answer to remember when answering it. Here we use it to give the agent the question/answer to q1 when prompting it to answer q4:

```python theme={null}
survey = Survey(questions = [q1, q2, q3, q4])
survey = survey.add_targeted_memory(q4, q1)

results = survey.by(agent).run()

(
   results
   .select("consume_local_news_user_prompt", "sources_user_prompt", "rate_coverage_user_prompt", "minutes_reading_user_prompt")
)
```

Output:

| prompt.consume\_local\_news\_user\_prompt | prompt.sources\_user\_prompt | prompt.rate\_coverage\_user\_prompt | prompt.minutes\_reading\_user\_prompt |
| :- | :- | :- | :- |
| How often do you consume local news? <br /> Daily Weekly Monthly Never <br /> Only 1 option may be selected. <br /> Respond only with a string corresponding to one of the options. <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | What are your most common sources of local news? <br /> 0: Television <br /> 1: Newspaper <br /> 2: Online news websites <br /> 3: Social Media <br /> 4: Radio <br /> 5: Other <br /> Please respond only with a comma-separated list of the code of the options that apply, with square brackets (e.g., \[0, 1, 3]). <br /> After the answer, you can put a comment explaining your choice on the next line. | On a scale of 1 to 10, how would you rate the quality of local news coverage in your area? <br /> 1 : Very poor <br /> 10 : Excellent <br /> Only 1 option may be selected. <br /> Respond only with the code corresponding to one of the options (e.g., “1” or “10”). <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | On average, how many minutes do you spend consuming local news each day? <br /> Minimum answer value: 0 <br /> Maximum answer value: 1440 <br /> This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, …). <br /> Respond with just your number on a single line. <br /> If your response is equivalent to zero, report ‘0’. <br /> After the answer, put a comment explaining your choice on the next line. |

### Memory collection

The add\_memory\_collection() method is used to add sets of prior questions and answers to a given question. We pass it the question to be answered and the list of questions/answers to be remembered when answering it. For example, we can add the questions/answers for both q1 and q2 when prompting the agent to answer q4:

```python theme={null}
survey = Survey(questions = [q1, q2, q3, q4])
survey = survey.add_memory_collection(q4, [q1, q2])

results = survey.by(agent).run()

(
   results
   .select("consume_local_news_user_prompt", "sources_user_prompt", "rate_coverage_user_prompt", "minutes_reading_user_prompt")
)
```

Output:

| prompt.consume\_local\_news\_user\_prompt | prompt.sources\_user\_prompt | prompt.rate\_coverage\_user\_prompt | prompt.minutes\_reading\_user\_prompt |
| :- | :- | :- | :- |
| How often do you consume local news? <br /> Daily Weekly Monthly Never <br /> Only 1 option may be selected. <br /> Respond only with a string corresponding to one of the options. <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | What are your most common sources of local news? <br /> 0: Television <br /> 1: Newspaper <br /> 2: Online news websites <br /> 3: Social Media <br /> 4: Radio <br /> 5: Other <br /> Please respond only with a comma-separated list of the code of the options that apply, with square brackets (e.g., \[0, 1, 3]). <br /> After the answer, you can put a comment explaining your choice on the next line. | On a scale of 1 to 10, how would you rate the quality of local news coverage in your area? <br /> 1 : Very poor <br /> 10 : Excellent <br /> Only 1 option may be selected. <br /> Respond only with the code corresponding to one of the options (e.g., “1” or “10”). <br /> After the answer, you can put a comment explaining why you chose that option on the next line. | On average, how many minutes do you spend consuming local news each day? <br /> Minimum answer value: 0 <br /> Maximum answer value: 1440 <br /> This question requires a numerical response in the form of an integer or decimal (e.g., -12, 0, 1, 2, 3.45, …). <br /> Respond with just your number on a single line. <br /> If your response is equivalent to zero, report ‘0’. <br /> After the answer, put a comment explaining your choice on the next line. |

## Costs

Before running a survey, you can estimate the cost of running the survey in USD and the number of credits needed to run it remotely at the Expected Parrot server. After running a survey, you can see details on the actual cost of each response in the results. The costs are calculated based on the estimated and actual number of tokens used in the survey and the model(s) used to generate the prompts.

### Estimated costs

Before running a survey, you can estimate the cost in USD of running the survey by calling the estimate\_job\_cost() method on a Job object (a survey combined with one or more models). This method returns a dictionary with the estimated costs and tokens for each model used with the survey. You can also estimate credits needed to run a survey remotely at the Expected Parrot server by passing the job to the remote\_inference\_cost() method of a `Coop` client object.

Example:

```python theme={null}
from edsl import QuestionFreeText, Survey, Agent, Model

q0 = QuestionFreeText(
    question_name = "favorite_flower",
    question_text = "What is the name of your favorite flower?"
)
q1 = QuestionFreeText(
    question_name = "flower_color",
    question_text = "What color is {{ favorite_flower.answer }}?"
)

survey = Survey(questions = [q0, q1])

a = Agent(traits = {"persona":"You are a botanist on Cape Cod."})

m = Model("gpt-4o")

job = survey.by(a).by(m)

estimated_job_cost = job.estimate_job_cost()
estimated_job_cost
```

Output:

```json theme={null}
{'estimated_total_cost_usd': 0.0008625000000000001,
'estimated_total_input_tokens': 85,
'estimated_total_output_tokens': 65,
'model_costs': [{'inference_service': 'openai',
   'model': 'gpt-4o',
   'estimated_cost_usd': 0.0008625000000000001,
   'estimated_input_tokens': 85,
   'estimated_output_tokens': 65}]}
```

To get the estimated cost in credits to run the job remotely:

```python theme={null}
from edsl import Coop

coop = Coop()

estimated_remote_inference_cost = coop.remote_inference_cost(job) # using the job object from above
estimated_remote_inference_cost
```

Output:

```json theme={null}
{'credits': 0.09, 'usd': 0.0008650000000000001}
```

Details of the calculations for these methods can be found in the credits section.

### Actual costs

The actual costs of running a survey are stored in the survey results. Details about the cost of each response can be accessed in the raw\_model\_response fields of the results dataset. For each question that was run, the following columns will appear in results:

* **raw\_model\_response.\<question\_name>\_cost**: The cost in USD for the API call to a language model service provider.
* **raw\_model\_response.\<question\_name>\_one\_usd\_buys**: The number of tokens that can be purchased with 1 USD (for reference).
* **raw\_model\_response.\<question\_name>\_raw\_model\_response**: A dictionary containing the raw response for the question, which includes the input text and tokens, output text and tokens, and other information about the API call. This dictionary is specific to the language model service provider and may contain additional information about the response.

The cost in credits of a response is calculated as follows:

* The number of input tokens is multiplied by the input token rate set by the language model service provider.
* The number of output tokens is multiplied by the output token rate set by the language model service provider.
* The total cost in USD is converted to credits (1 USD = 100 credits).
* The total cost in credits is rounded up to the nearest 1/100th of a credit.

To learn more about these methods and calculations, please see the credits section.

## Survey class

> ### *class* edsl.surveys.Survey(*questions: List\['QuestionType'] | None = None*, *memory\_plan: 'MemoryPlan' | None = None*, *rule\_collection: 'RuleCollection' | None = None*, *question\_groups: 'QuestionGroupType' | None = None*, *name: str | None = None*, *questions\_to\_randomize: List\[str] | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Bases: [`Base`](/en/latest/base#base-class "edsl.base.base_class.Base")

A collection of questions with logic for navigating between them.

Survey is the main class for creating, modifying, and running surveys. It supports:

* Skip logic: conditional navigation between questions based on previous answers
* Memory: controlling which previous answers are visible to agents
* Question grouping: organizing questions into logical sections
* Randomization: randomly ordering certain questions to reduce bias
* Instructions: adding non-question elements to guide respondents

A Survey instance can be used to: 1. Define a set of questions and their order 2. Add rules for navigating between questions 3. Run the survey with agents or humans 4. Export the survey in various formats

The survey maintains the order of questions, any skip logic rules, and handles serialization for storage or transmission.

> #### **init**(*questions: List\['QuestionType'] | None = None*, *memory\_plan: 'MemoryPlan' | None = None*, *rule\_collection: 'RuleCollection' | None = None*, *question\_groups: 'QuestionGroupType' | None = None*, *name: str | None = None*, *questions\_to\_randomize: List\[str] | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Initialize a new Survey instance.

This constructor sets up a new survey with the provided questions and optional configuration for memory, rules, grouping, and randomization.

> **Args:**

**questions: A list of question objects to include in the survey.**

Can include QuestionBase objects, Instructions, and ChangeInstructions.

**memory\_plan: Defines which previous questions and answers are available**

when answering each question. If None, a default plan is created.

**rule\_collection: Contains rules for determining which question comes next**

based on previous answers. If None, default sequential rules are created.

question\_groups: A dictionary mapping group names to (start\_idx, end\_idx)

tuples that define groups of questions.

name: DEPRECATED. The name of the survey. questions\_to\_randomize: A list of question names to randomize when the

survey is drawn. This affects the order of options in these questions.

Examples:

Create a basic survey with three questions:

```python theme={null}
>>> from edsl import QuestionFreeText
>>> q1 = QuestionFreeText(question_text="What is your name?", question_name="name")
>>> q2 = QuestionFreeText(question_text="What is your favorite color?", question_name="color")
>>> q3 = QuestionFreeText(question_text="Is a hot dog a sandwich?", question_name="food")
>>> s = Survey([q1, q2, q3])
```

Create a survey with question groups:

```python theme={null}
>>> s = Survey([q1, q2, q3], question_groups={"demographics": (0, 1), "food_questions": (2, 2)})
```

> #### add\_instruction(*instruction: Instruction | ChangeInstruction*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Add an instruction to the survey.

**Parameters:** **instruction** – The instruction to add to the survey.

```python theme={null}
>>> from edsl import Instruction
>>> i = Instruction(text="Pay attention to the following questions.", name="intro")
>>> s = Survey().add_instruction(i)
>>> s._instruction_names_to_instructions
{'intro': Instruction(name="intro", text="Pay attention to the following questions.")}
>>> s._pseudo_indices
{'intro': -0.5}
```

> #### add\_memory\_collection(*focal\_question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase") | str*, *prior\_questions: List\[[QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase") | str]*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Configure the survey so a specific question has access to multiple prior questions.

This method allows you to define memory relationships between specific questions. When an agent answers the focal\_question, it will have access to all the questions and answers specified in prior\_questions.

> **Args:**

**focal\_question: The question for which to add memory, specified either as a**

QuestionBase object or its question\_name string.

**prior\_questions: A list of prior questions to remember, each specified either**

as a QuestionBase object or its question\_name string.

> **Returns:**

Survey: The modified survey instance (allows for method chaining).

> **Examples:**

When answering q2, remember the answers to both q0 and q1:

```python theme={null}
>>> s = Survey.example().add_memory_collection("q2", ["q0", "q1"])
>>> s.memory_plan
{'q2': Memory(prior_questions=['q0', 'q1'])}
```

> #### add\_question(*question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase")*, *index: int | None = None*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Add a question to survey.

**Parameters:**

* **question** – The question to add to the survey.
* **question\_name** – The name of the question. If not provided, the question name is used.

The question is appended at the end of the self.questions list A default rule is created that the next index is the next question.

```python theme={null}
>>> from edsl import QuestionMultipleChoice
>>> q = QuestionMultipleChoice(question_text = "Do you like school?", question_options=["yes", "no"], question_name="q0")
>>> s = Survey().add_question(q)

# Adding a question with a duplicate name would raise SurveyCreationError
```

> #### add\_question\_group(*start\_question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase") | str*, *end\_question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase") | str*, *group\_name: str*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Create a logical group of questions within the survey.

Question groups allow you to organize questions into meaningful sections, which can be useful for: - Analysis (analyzing responses by section) - Navigation (jumping between sections) - Presentation (displaying sections with headers)

Groups are defined by a contiguous range of questions from start\_question to end\_question, inclusive. Groups cannot overlap with other groups.

> **Args:**

**start\_question: The first question in the group, specified either as a**

QuestionBase object or its question\_name string.

**end\_question: The last question in the group, specified either as a**

QuestionBase object or its question\_name string.

**group\_name: A name for the group. Must be a valid Python identifier**

and must not conflict with existing group or question names.

> **Returns:**

Survey: The modified survey instance (allows for method chaining).

> **Raises:**

SurveyCreationError: If the group name is invalid, already exists,

conflicts with a question name, if start comes after end, or if the group overlaps with an existing group.

> **Examples:**

Create a group of questions for demographics:

```python theme={null}
>>> s = Survey.example().add_question_group("q0", "q1", "group1")
>>> s.question_groups
{'group1': (0, 1)}
```

Group names must be valid Python identifiers:

```python theme={null}
>>> from edsl.surveys.exceptions import SurveyCreationError
>>> # Example showing invalid group name error
>>> try:
...     Survey.example().add_question_group("q0", "q2", "1group1")
... except SurveyCreationError:
...     print("Error: Invalid group name (as expected)")
Error: Invalid group name (as expected)
```

Group names can’t conflict with question names:

```python theme={null}
>>> # Example showing name conflict error
>>> try:
...     Survey.example().add_question_group("q0", "q1", "q0")
... except SurveyCreationError:
...     print("Error: Group name conflicts with question name (as expected)")
Error: Group name conflicts with question name (as expected)
```

Start question must come before end question:

```python theme={null}
>>> # Example showing index order error
>>> try:
...     Survey.example().add_question_group("q1", "q0", "group1")
... except SurveyCreationError:
...     print("Error: Start index greater than end index (as expected)")
Error: Start index greater than end index (as expected)
```

> #### add\_rule(*question: 'QuestionBase' | str*, *expression: str*, *next\_question: 'QuestionBase' | str | int | EndOfSurveyParent*, *before\_rule: bool = False*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Add a conditional rule for navigating between questions in the survey.

Rules determine the flow of questions based on conditional expressions. When a rule’s expression evaluates to True, the survey will navigate to the specified next question, potentially skipping questions or jumping to an earlier question.

By default, rules are evaluated *after* a question is answered. When before\_rule=True, the rule is evaluated before the question is presented (which is useful for skip logic).

> **Args:**

**question: The question this rule applies to, either as a QuestionBase object**

or its question\_name string.

**expression: A string expression that will be evaluated to determine if the**

rule should trigger. Can reference previous questions’ answers using the template syntax, e.g., “\{\{ q0.answer }} == ‘yes’”.

**next\_question: The destination question to jump to if the expression is True.**

Can be specified as a QuestionBase object, a question\_name string, an index, or the EndOfSurvey class to end the survey.

**before\_rule: If True, the rule is evaluated before the question is presented.**

If False (default), the rule is evaluated after the question is answered.

> **Returns:**

Survey: The modified survey instance (allows for method chaining).

> **Examples:**

Add a rule that navigates to q2 if the answer to q0 is ‘yes’:

```python theme={null}
>>> s = Survey.example().add_rule("q0", "{{ q0.answer }} == 'yes'", "q2")
>>> s.next_question("q0", {"q0.answer": "yes"}).question_name
'q2'
```

Add a rule to end the survey conditionally:

```python theme={null}
>>> from edsl.surveys.base import EndOfSurvey
>>> s = Survey.example().add_rule("q0", "{{ q0.answer }} == 'end'", EndOfSurvey)
```

> #### add\_skip\_rule(*question: 'QuestionBase' | str*, *expression: str*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Add a rule to skip a question based on a conditional expression.

Skip rules are evaluated *before* the question is presented. If the expression evaluates to True, the question is skipped and the flow proceeds to the next question in sequence. This is different from jump rules which are evaluated *after* a question is answered.

> **Args:**

**question: The question to add the skip rule to, either as a QuestionBase object**

or its question\_name string.

**expression: A string expression that will be evaluated to determine if the**

question should be skipped. Can reference previous questions’ answers using the template syntax, e.g., “\{\{ q0.answer }} == ‘yes’”.

> **Returns:**

Survey: The modified survey instance (allows for method chaining).

> **Examples:**

Skip q0 unconditionally (always skip):

```python theme={null}
>>> from edsl import QuestionFreeText
>>> q0 = QuestionFreeText.example()
>>> q0.question_name = "q0"
>>> q1 = QuestionFreeText.example()
>>> q1.question_name = "q1"
>>> s = Survey([q0, q1]).add_skip_rule("q0", "True")
>>> s.next_question("q0", {}).question_name
'q1'
```

Skip a question conditionally:

```python theme={null}
>>> q2 = QuestionFreeText.example()
>>> q2.question_name = "q2"
>>> s = Survey([q0, q1, q2])
>>> s = s.add_skip_rule("q1", "{{ q0.answer }} == 'skip next'")
```

> #### add\_stop\_rule(*question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase") | str*, *expression: str*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Add a rule that stops the survey. The rule is evaluated *after* the question is answered. If the rule is true, the survey ends.

**Parameters:**

* **question** – The question to add the stop rule to.
* **expression** – The expression to evaluate.

If this rule is true, the survey ends.

Here, answering “yes” to q0 ends the survey:

```python theme={null}
>>> s = Survey.example().add_stop_rule("q0", "{{ q0.answer }} == 'yes'")
>>> s.next_question("q0", {"q0.answer": "yes"})
EndOfSurvey
```

By comparison, answering “no” to q0 does not end the survey:

```python theme={null}
>>> s.next_question("q0", {"q0.answer": "no"}).question_name
'q1'

# Using invalid operators like ‘<>’ would raise SurveyCreationError
```

> #### add\_targeted\_memory(*focal\_question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase") | str*, *prior\_question: [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase") | str*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Configure the survey so a specific question has access to a prior question’s answer.

This method allows you to define memory relationships between specific questions. When an agent answers the focal\_question, it will have access to the prior\_question and its answer, regardless of other memory settings.

> **Args:**

focal\_question: The question for which to add memory, specified either as a

QuestionBase object or its question\_name string.

prior\_question: The prior question to remember, specified either as a

QuestionBase object or its question\_name string.

> **Returns:**

Survey: The modified survey instance (allows for method chaining).

> **Examples:**

When answering q2, remember the answer to q0:

```python theme={null}
>>> s = Survey.example().add_targeted_memory("q2", "q0")
>>> s.memory_plan
{'q2': Memory(prior_questions=['q0'])}
```

> #### *classmethod* auto\_survey(*overall\_question: str*, *population: str*, *num\_questions: int*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Create a survey with a single question that asks the user how they are doing.

> #### by(\*\*args: 'Agent' | 'Scenario' | 'LanguageModel' | 'AgentList' | 'ScenarioList' | 'ModelList'\*) → [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Add components to the survey and return a runnable Jobs object.

This method is the primary way to prepare a survey for execution. It adds the necessary components (agents, scenarios, language models) to create a Jobs object that can be run to generate responses to the survey.

The method can be chained to add multiple components in sequence.

> **Args:**

[\*](/en/latest/#id3)**args: One or more components to add to the survey. Can include:**

* Agent: The persona that will answer the survey questions
* Scenario: The context for the survey, with variables to substitute
* LanguageModel: The model that will generate the agent’s responses

> **Returns:**

Jobs: A Jobs object that can be run to execute the survey.

> **Examples:**

Create a runnable Jobs object with an agent and scenario:

```python theme={null}
>>> s = Survey.example()
>>> from edsl.agents import Agent
>>> from edsl import Scenario
>>> s.by(Agent.example()).by(Scenario.example())
Jobs(...)
```

Chain all components in a single call:

```python theme={null}
>>> from edsl.language_models import LanguageModel
>>> s.by(Agent.example(), Scenario.example(), LanguageModel.example())
Jobs(...)
```

> #### clear\_non\_default\_rules() → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Remove all non-default rules from the survey.

```python theme={null}
>>> Survey.example().show_rules()
Dataset([{'current_q': [0, 0, 1, 2]}, {'expression': ['True', "{{ q0.answer }}== 'yes'", 'True', 'True']}, {'next_q': [1, 2, 2, 3]}, {'priority': [-1, 0, -1, -1]}, {'before_rule': [False, False, False, False]}])
>>> Survey.example().clear_non_default_rules().show_rules()
Dataset([{'current_q': [0, 1, 2]}, {'expression': ['True', 'True', 'True']}, {'next_q': [1, 2, 3]}, {'priority': [-1, -1, -1]}, {'before_rule': [False, False, False]}])
```

> #### clipboard\_data()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return the clipboard data for the survey.

> #### code(*filename: str = ''*, *survey\_var\_name: str = 'survey'*) → list\[str][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Create the Python code representation of a survey.

> #### codebook() → Dict\[str, str][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Create a codebook for the survey, mapping question names to question text.

```python theme={null}
>>> s = Survey.example()
>>> s.codebook()
{'q0': 'Do you like school?', 'q1': 'Why not?', 'q2': 'Why?'}
```

> #### copy() → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Create a deep copy of the survey using serialization.

This method creates a completely independent copy of the survey by serializing and then deserializing it. This ensures all components are properly copied and maintains consistency with the survey’s serialization format.

> **Returns:**

Survey: A new Survey instance that is a deep copy of the original.

> **Examples:**

```python theme={null}
>>> s = Survey.example()
>>> s2 = s.copy()
>>> s == s2
True
>>> s is s2
False
>>> s.questions[0] is s2.questions[0]
False
```

> #### css()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return the default CSS style for the survey.

> #### dag(*textify: bool = False*) → DAG[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a Directed Acyclic Graph (DAG) representation of the survey flow.

This method constructs a DAG that represents the possible paths through the survey, taking into account both skip logic and memory relationships. The DAG is useful for visualizing and analyzing the structure of the survey.

> **Args:**

**textify: If True, the DAG will use question names as nodes instead of indices.**

This makes the DAG more human-readable but less compact.

> **Returns:**

DAG: A dictionary where keys are question indices (or names if textify=True)

and values are sets of prerequisite questions. For example, \{2: \{0, 1}} means question 2 depends on questions 0 and 1.

> **Examples:**

```python theme={null}
>>> s = Survey.example()
>>> d = s.dag()
>>> d
{1: {0}, 2: {0}}
```

With textify=True:

```python theme={null}
>>> dag = s.dag(textify=True)
>>> sorted([(k, sorted(list(v))) for k, v in dag.items()])
[('q1', ['q0']), ('q2', ['q0'])]
```

> #### delete\_question(*identifier: str | int*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Delete a question from the survey.

> **Parameters:**

**identifier** – The name or index of the question to delete.

Returns:

The updated Survey object.

```python theme={null}
>>> from edsl import QuestionMultipleChoice, Survey
>>> q1 = QuestionMultipleChoice(question_text="Q1", question_options=["A", "B"], question_name="q1")
>>> q2 = QuestionMultipleChoice(question_text="Q2", question_options=["C", "D"], question_name="q2")
>>> s = Survey().add_question(q1).add_question(q2)
>>> _ = s.delete_question("q1")
>>> len(s.questions)
1
>>> _ = s.delete_question(0)
>>> len(s.questions)
0
```

> #### docx(*filename: str | None = None*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

DEPRECATED: Use [`to_docx()`](/en/latest/#edsl.surveys.Survey.to_docx "edsl.surveys.Survey.to_docx") instead.

> #### draw() → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a new survey with a randomly selected permutation of the options.

> #### drop(\**question\_names*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Create a new Survey with specified questions removed by name.

This method creates a new Survey instance that contains all questions except those specified in the question\_names parameter. It’s the inverse of the select() method.

> **Args:**

[\*](/en/latest/#id5)question\_names: Variable number of question names to remove from the survey.

> **Returns:**

Survey: A new Survey instance with the specified questions removed.

> **Raises:**

ValueError: If no question names are provided. KeyError: If any specified question name is not found in the survey.

> **Examples:**

```python theme={null}
>>> s = Survey.example()
>>> s.question_names
['q0', 'q1', 'q2']
>>> s_dropped = s.drop('q1')
>>> s_dropped.question_names
['q0', 'q2']
>>> s_dropped2 = s.drop('q0', 'q2')
>>> s_dropped2.question_names
['q1']
```

> #### duplicate()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Duplicate the survey.

```python theme={null}
>>> s = Survey.example()
>>> s2 = s.duplicate()
>>> s == s2
True
>>> s is s2
False
```

> #### edit()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

> #### *classmethod* example(*params: bool = False*, *randomize: bool = False*, *include\_instructions: bool = False*, *custom\_instructions: str | None = None*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Create an example survey for testing and demonstration purposes.

This method creates a simple branching survey about school preferences. The default survey contains three questions with conditional logic: - If the user answers “yes” to liking school, they are asked why they like it - If the user answers “no”, they are asked why they don’t like it

> **Args:**

params: If True, adds a fourth question that demonstrates parameter substitution

by referencing the question text and answer from the first question.

randomize: If True, adds a random UUID to the first question text to ensure

uniqueness across multiple instances.

include\_instructions: If True, adds an instruction to the beginning of the survey. custom\_instructions: Custom instruction text to use if include\_instructions is True.

> Defaults to “Please pay attention!” if not provided.

> **Returns:**

Survey: A configured example survey instance.

> **Examples:**

Create a basic example survey:

```python theme={null}
>>> s = Survey.example()
>>> [q.question_text for q in s.questions]
['Do you like school?', 'Why not?', 'Why?']
```

Create an example with parameter substitution:

```python theme={null}
>>> s = Survey.example(params=True)
>>> s.questions[3].question_text
"To the question '{{ q0.question_text}}', you said '{{ q0.answer }}'. Do you still feel this way?"
```

> #### *classmethod* from\_dict(*data: dict*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Reconstruct a Survey object from its dictionary representation.

This class method is the counterpart to to\_dict() and allows you to recreate a Survey object from a serialized dictionary. This is useful for loading saved surveys, receiving surveys from other systems, or cloning surveys.

The method handles deserialization of all survey components, including questions, instructions, memory plan, rules, and question groups.

> **Args:**

data: A dictionary containing the serialized survey data, typically

created by the to\_dict() method.

> **Returns:**

Survey: A fully reconstructed Survey object with all the original

questions, rules, and configuration.

> **Examples:**

Create a survey, serialize it, and deserialize it back:

```python theme={null}
>>> d = Survey.example().to_dict()
>>> s = Survey.from_dict(d)
>>> s == Survey.example()
True
```

Works with instructions as well:

```python theme={null}
>>> s = Survey.example(include_instructions=True)
>>> d = s.to_dict()
>>> news = Survey.from_dict(d)
>>> news == s
True
```

> #### gen\_path\_through\_survey() → Generator\[[QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase"), dict, None][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Generate a coroutine that navigates through the survey based on answers.

This method creates a Python generator that implements the survey flow logic. It yields questions and receives answers, handling the branching logic based on the rules defined in the survey. This generator is the core mechanism used by the Interview process to administer surveys.

The generator follows these steps: 1. Yields the first question (or skips it if skip rules apply) 2. Receives an answer dictionary from the caller via .send() 3. Updates the accumulated answers 4. Determines the next question based on the survey rules 5. Yields the next question 6. Repeats steps 2-5 until the end of survey is reached

> **Returns:**

**Generator\[QuestionBase, dict, None]: A generator that yields questions and**

receives answer dictionaries. The generator terminates when it reaches the end of the survey.

> **Examples:**

For the example survey with conditional branching:

```python theme={null}
>>> s = Survey.example()
>>> s.show_rules()
Dataset([{'current_q': [0, 0, 1, 2]}, {'expression': ['True', "{{ q0.answer }}== 'yes'", 'True', 'True']}, {'next_q': [1, 2, 2, 3]}, {'priority': [-1, 0, -1, -1]}, {'before_rule': [False, False, False, False]}])
```

Path when answering “yes” to first question:

```python theme={null}
>>> i = s.gen_path_through_survey()
>>> next(i)  # Get first question
Question('multiple_choice', question_name = """q0""", question_text = """Do you like school?""", question_options = ['yes', 'no'])
>>> i.send({"q0.answer": "yes"})  # Answer "yes" and get next question
Question('multiple_choice', question_name = """q2""", question_text = """Why?""", question_options = ['**lack*** of killer bees in cafeteria', 'other'])
```

Path when answering “no” to first question:

```python theme={null}
>>> i2 = s.gen_path_through_survey()
>>> next(i2)  # Get first question
Question('multiple_choice', question_name = """q0""", question_text = """Do you like school?""", question_options = ['yes', 'no'])
>>> i2.send({"q0.answer": "no"})  # Answer "no" and get next question
Question('multiple_choice', question_name = """q1""", question_text = """Why not?""", question_options = ['killer bees in cafeteria', 'other'])
```

> #### generate\_description() → str[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Generate a description of the survey.

> #### *classmethod* generate\_from\_questions(*question\_texts: List\[str]*, *question\_types: List\[str] | None = None*, *question\_names: List\[str] | None = None*, *model: 'LanguageModel' | None = None*, *scenario\_keys: List\[str] | None = None*, *verbose: bool = True*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Generate a survey from a list of question texts.

This method takes a list of question texts and optionally infers question types and generates question names using an LLM.

> **Args:**

question\_texts: List of question text strings question\_types: Optional list of question types corresponding to each text.

If None, types will be inferred by the model.

question\_names: Optional list of question names. If None, names will be generated. model: Language model to use for inference. If None, uses default model. scenario\_keys: Optional list of scenario keys to include in question texts.

Each key will be added as \{\{ scenario.\<key> }} in the questions.

verbose: Whether to show the underlying survey generation process (default: True)

> **Returns:**

Survey: A new Survey instance with the questions

> **Examples:**

```python theme={null}
>>> texts = ["How satisfied are you?", "What is your age?"]
>>> survey = Survey.generate_survey_from_questions(texts)
>>>
>>> types = ["LinearScale", "Numerical"]
>>> names = ["satisfaction", "age"]
>>> survey = Survey.generate_survey_from_questions(texts, types, names)
>>>
>>> survey = Survey.generate_survey_from_questions(texts, scenario_keys=["product_name"])
>>> survey = Survey.generate_survey_from_questions(texts, verbose=False)
```

> #### *classmethod* generate\_from\_topic(*topic: str*, *n\_questions: int = 5*, *model: 'LanguageModel' | None = None*, *scenario\_keys: List\[str] | None = None*, *verbose: bool = True*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Generate a survey from a topic using an LLM.

This method uses a language model to generate a well-balanced survey for the given topic with the specified number of questions.

> **Args:**

topic: The topic to generate questions about n\_questions: Number of questions to generate (default: 5) model: Language model to use for generation. If None, uses default model. scenario\_keys: Optional list of scenario keys to include in question texts.

Each key will be added as \{\{ scenario.\<key> }} in the questions.

verbose: Whether to show the underlying survey generation process (default: True)

> **Returns:**

Survey: A new Survey instance with generated questions

> **Examples:**

```python theme={null}
>>> survey = Survey.generate_survey_from_topic("workplace satisfaction", n_questions=3)
>>> survey = Survey.generate_survey_from_topic("AI ethics", n_questions=7, model=my_model)
>>> survey = Survey.generate_survey_from_topic("product feedback", scenario_keys=["product_name", "version"])
>>> survey = Survey.generate_survey_from_topic("feedback", verbose=False)
```

> #### get(*question\_name: str*) → [QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return the question object given the question name.

> #### get\_job(*model=None*, *agent=None*, \*\**kwargs*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

> #### gold\_standard(*q\_and\_a\_dict: dict\[str, str]*) → [Result](/en/latest/results#edsl.results.Result "edsl.results.Result")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Run the survey with a gold standard agent and return the result object.

> **Args:**

q\_and\_a\_dict: A dictionary of question names and answers.

> #### html(*scenario: dict | None = None*, *filename: str | None = None*, *return\_link=False*, *css: str | None = None*, *cta: str = 'Open HTML file'*, *include\_question\_name=False*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

DEPRECATED: Use [`to_html()`](/en/latest/#edsl.surveys.Survey.to_html "edsl.surveys.Survey.to_html") instead.

> #### humanize(*project\_name: str = 'Project'*, *survey\_description: str | None = None*, *survey\_alias: str | None = None*, *survey\_visibility: 'VisibilityType' | None = 'unlisted'*) → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Send the survey to Expected Parrot.

Then, create a project on Expected Parrot so you can share the survey with human respondents.

> #### inspect()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Create an interactive inspector widget for this survey.

This method creates a SurveyInspectorWidget that provides an interactive interface for exploring the survey structure, questions, and flow logic.

> **Returns:**

SurveyInspectorWidget instance: Interactive widget for inspecting this survey

> **Raises:**

ImportError: If the widgets module cannot be imported

> #### latex(*filename: str | None = None*, *include\_question\_name: bool = False*, *standalone: bool = True*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Generate a LaTeX (.tex) representation of the survey.

> **Parameters:**

**filename:Optional\[str]**

The filename to write to. If not provided, a temporary file is created in the current working directory with a `.tex` suffix.

**include\_question\_name:bool**

If True, includes the internal `question_name` of each question. Default False.

**standalone:bool**

If True, the LaTeX file is standalone. Default True.

> #### move\_question(*identifier: str | int*, *new\_index: int*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

```python theme={null}
>>> from edsl import QuestionMultipleChoice, Survey
>>> s = Survey.example()
>>> s.question_names
['q0', 'q1', 'q2']
>>> s.move_question("q0", 2).question_names
['q1', 'q2', 'q0']
```

> #### next\_question(*current\_question: str | 'QuestionBase' | None = None*, *answers: Dict\[str, Any] | None = None*) → 'QuestionBase' | EndOfSurveyParent[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return the next question in a survey.

> **Parameters:**

* **current\_question** – The current question in the survey.

* **answers** – The answers for the survey so far

* If called with no arguments, it returns the first question in the survey.

* If no answers are provided for a question with a rule, the next question is returned. If answers are provided, the next question is determined by the rules and the answers.

* If the next question is the last question in the survey, an EndOfSurvey object is returned.

```python theme={null}
>>> s = Survey.example()
>>> s.next_question("q0", {"q0.answer": "yes"}).question_name
'q2'
>>> s.next_question("q0", {"q0.answer": "no"}).question_name
'q1'
```

> #### next\_question\_with\_instructions(*current\_item: str | 'QuestionBase' | 'Instruction' | None = None*, *answers: Dict\[str, Any] | None = None*) → 'QuestionBase' | 'Instruction' | EndOfSurveyParent[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return the next question or instruction in a survey, including instructions in sequence.

This method extends the functionality of next\_question to also handle Instructions that are interspersed between questions. It follows the proper sequence based on pseudo indices and respects survey rules for question flow.

> **Parameters:**

* **current\_item** – The current question or instruction in the survey.

* **answers** – The answers for the survey so far

* If called with no arguments, it returns the first item (question or instruction) in the survey.

* For instructions, it returns the next item in sequence since instructions don’t have answers.

* For questions, it uses the rule logic to determine the next question, then returns any instructions that come before that target question, or the target question itself.

* If the next item would be past the end of the survey, an EndOfSurvey object is returned.

> **Returns:**

Union\[“QuestionBase”, “Instruction”, EndOfSurveyParent]: The next question, instruction, or EndOfSurvey.

> **Examples:**

With a survey that has instructions:

```python theme={null}
>>> from edsl import Instruction
>>> s = Survey.example(include_instructions=True)
>>> # Get the first item (should be the instruction)
>>> first_item = s.next_question_with_instructions()
>>> hasattr(first_item, 'text')  # Instructions have text attribute
True
```

```python theme={null}
>>> # After an instruction, get the next item
>>> next_item = s.next_question_with_instructions(first_item)
>>> hasattr(next_item, 'question_name')  # Questions have question_name attribute
True
```

> #### *property* parameters\*: set\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a set of parameters in the survey.

```python theme={null}
>>> s = Survey.example()
>>> s.parameters
set()
```

> #### *property* parameters\_by\_question\*: dict\[str, set]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a dictionary of parameters by question in the survey.

```python theme={null}
>>> from edsl import QuestionFreeText
>>> q = QuestionFreeText(question_name = “example”, question_text = “What is the capital of {{ country}}?”)
>>> s = Survey([q])
>>> s.parameters_by_question {‘example’: {‘country’}}
```

> #### *property* question\_name\_to\_index\*: dict\[str, int]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a dictionary mapping question names to question indices.

Example:

```python theme={null}
>>> s = Survey.example()
>>> s.question_name_to_index
{'q0': 0, 'q1': 1, 'q2': 2}
```

> #### *property* question\_names\*: list\[str]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a list of question names in the survey.

Example:

```python theme={null}
>>> s = Survey.example()
>>> s.question_names
['q0', 'q1', 'q2']
```

> #### question\_names\_to\_questions() → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a dictionary mapping question names to question attributes.

> #### question\_names\_valid() → bool[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Check if the question names are valid.

> #### question\_to\_attributes() → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a dictionary of question attributes.

```python theme={null}
>>> s = Survey.example()
>>> s.question_to_attributes()
{'q0': {'question_text': 'Do you like school?', 'question_type': 'multiple_choice', 'question_options': ['yes', 'no']}, 'q1': {'question_text': 'Why not?', 'question_type': 'multiple_choice', 'question_options': ['killer bees in cafeteria', 'other']}, 'q2': {'question_text': 'Why?', 'question_type': 'multiple_choice', 'question_options': ['**lack*** of killer bees in cafeteria', 'other']}}
```

> #### questions[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

A descriptor that manages the list of questions in the survey.

This descriptor handles the setting and getting of questions, ensuring proper validation and maintaining internal data structures. It manages both direct question objects and their names.

The underlying questions are stored in the protected \_questions attribute, while this property provides the public interface for accessing them.

**Notes:**

* The presumed order of the survey is the order in which questions are added
* Questions must have unique names within a survey
* Each question can have rules associated with it that determine the next question

> #### *classmethod* random\_survey()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

> #### recombined\_questions\_and\_instructions() → List\['QuestionBase' | 'Instruction'][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a list of questions and instructions (public wrapper).

This is a thin wrapper around the internal \_recombined\_questions\_and\_instructions method, provided for compatibility with modules that expect a public accessor.

> #### run(\**args*, \*\**kwargs*) → [Results](/en/latest/results#result-class "edsl.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Convert the survey to a Job and execute it with the provided parameters.

This method creates a Jobs object from the survey and runs it immediately with the provided arguments. It’s a convenient way to run a survey without explicitly creating a Jobs object first.

> **Args:**

[\*](/en/latest/#id7)args: Positional arguments passed to the Jobs.run() method. [\*\*](/en/latest/#id9)kwargs: Keyword arguments passed to the Jobs.run() method, which can include:

* cache: The cache to use for storing results
* verbose: Whether to show detailed progress
* disable\_remote\_cache: Whether to disable remote caching
* disable\_remote\_inference: Whether to disable remote inference

> **Returns:**

Results: The results of running the survey.

> **Examples:**

Run a survey with a test language model:

```python theme={null}
>>> from edsl import QuestionFreeText
>>> s = Survey([QuestionFreeText.example()])
>>> from edsl.language_models import LanguageModel
>>> m = LanguageModel.example(test_model=True, canned_response="Great!")
>>> results = s.by(m).run(cache=False, disable_remote_cache=True, disable_remote_inference=True)
>>> results.select('answer.*')
Dataset([{'answer.how_are_you': ['Great!']}])
```

> #### *async* run\_async(*model: 'LanguageModel' | None = None*, *agent: 'Agent' | None = None*, *cache: 'Cache' | None = None*, \*\**kwargs*) → [Results](/en/latest/results#result-class "edsl.results.Results")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Execute the survey asynchronously and return results.

This method provides an asynchronous way to run surveys, which is useful for concurrent execution or integration with other async code. It creates a Jobs object and runs it asynchronously.

> **Args:**

model: The language model to use. If None, a default model is used. agent: The agent to use. If None, a default agent is used. cache: The cache to use for storing results. If provided, reuses cached results. [\*\*](/en/latest/#id11)kwargs: Key-value pairs to use as scenario parameters. May include:

* disable\_remote\_inference: If True, don’t use remote inference even if available.
* disable\_remote\_cache: If True, don’t use remote cache even if available.

> **Returns:**

Results: The results of running the survey.

> **Examples:**

Run a survey asynchronously with morning parameter:

```python theme={null}
>>> import asyncio
>>> from edsl.questions import QuestionFunctional
>>> def f(scenario, agent_traits): return "yes" if scenario["period"] == "morning" else "no"
>>> q = QuestionFunctional(question_name="q0", func=f)
>>> from edsl import Model
>>> s = Survey([q])
>>> async def test_run_async():
...     result = await s.run_async(period="morning", disable_remote_inference = True)
...     print(result.select("answer.q0").first())
>>> asyncio.run(test_run_async())
yes

Run with evening parameter:

>>> async def test_run_async2():
...     result = await s.run_async(period="evening", disable_remote_inference = True)
...     print(result.select("answer.q0").first())
>>> asyncio.run(test_run_async2())
no
```

> #### *property* scenario\_attributes\*: list\[str]\*[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a list of attributes that admissible Scenarios should have.

Here we have a survey with a question that uses a jinja2 style \{\{ }} template:

```python theme={null}
>>> from edsl import QuestionFreeText
>>> s = Survey().add_question(QuestionFreeText(question_text="{{ greeting }}. What is your name?", question_name="name"))
>>> s.scenario_attributes
['greeting']
```

```python theme={null}
>>> s = Survey().add_question(QuestionFreeText(question_text="{{ greeting }}. What is your {{ attribute }}?", question_name="name"))
>>> s.scenario_attributes
['greeting', 'attribute']
```

> #### select(\*\*question\_names: List\[str]\*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Create a new Survey with questions selected by name.

> #### set\_full\_memory\_mode() → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Configure the survey so agents remember all previous questions and answers.

In full memory mode, when an agent answers any question, it will have access to all previously asked questions and the agent’s answers to them. This is useful for surveys where later questions build on or reference earlier responses.

> **Returns:**

Survey: The modified survey instance (allows for method chaining).

> **Examples:**

```python theme={null}
>>> s = Survey.example().set_full_memory_mode()
```

> #### set\_lagged\_memory(*lags: int*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Configure the survey so agents remember a limited window of previous questions.

In lagged memory mode, when an agent answers a question, it will only have access to the most recent ‘lags’ number of questions and answers. This is useful for limiting context when only recent questions are relevant.

> **Args:**

lags: The number of previous questions to remember. For example, if lags=2,

only the two most recent questions and answers will be remembered.

> **Returns:**

Survey: The modified survey instance (allows for method chaining).

> **Examples:**

Remember only the two most recent questions:

```python theme={null}
>>> s = Survey.example().set_lagged_memory(2)
```

> #### show()[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Display the survey in a rich format.

> #### show\_flow(*filename: str | None = None*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Show the flow of the survey.

> #### show\_prompts(*all: bool = False*) → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Display the prompts that will be used when running the survey.

This method converts the survey to a Jobs object and shows the prompts that would be sent to a language model. This is useful for debugging and understanding how the survey will be presented.

**Args:**

all: If True, show all prompt fields; if False (default), show only user\_prompt and system\_prompt.

> #### show\_rules() → None[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Print out the rules in the survey.

```python theme={null}
>>> s = Survey.example()
>>> s.show_rules()
Dataset([{'current_q': [0, 0, 1, 2]}, {'expression': ['True', "{{ q0.answer }}== 'yes'", 'True', 'True']}, {'next_q': [1, 2, 2, 3]}, {'priority': [-1, 0, -1, -1]}, {'before_rule': [False, False, False, False]}])
```

> #### simulate() → dict[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Simulate the survey and return the answers.

> #### table(\**fields*, *tablefmt='rich'*) → Table[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

> #### to\_dict(*add\_edsl\_version: bool = True*) → dict\[str, Any][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Serialize the Survey object to a dictionary for storage or transmission.

This method converts the entire survey structure, including questions, rules, memory plan, and question groups, into a dictionary that can be serialized to JSON. This is essential for saving surveys, sharing them, or transferring them between systems.

The serialized dictionary contains the complete state of the survey, allowing it to be fully reconstructed using the from\_dict() method.

> **Args:**

**add\_edsl\_version: If True (default), includes the EDSL version and class name**

in the dictionary, which can be useful for backward compatibility when deserializing.

> **Returns:**

**dict\[str, Any]: A dictionary representation of the survey with the following keys:**

* ‘questions’: List of serialized questions and instructions
* ‘memory\_plan’: Serialized memory plan
* ‘rule\_collection’: Serialized rule collection
* ‘question\_groups’: Dictionary of question groups
* ‘questions\_to\_randomize’: List of questions to randomize (if any)
* ‘edsl\_version’: EDSL version (if add\_edsl\_version=True)
* ‘edsl\_class\_name’: Class name (if add\_edsl\_version=True)

Examples:

```python theme={null}
>>> s = Survey.example()
>>> s.to_dict(add_edsl_version=False).keys()
dict_keys(['questions', 'memory_plan', 'rule_collection', 'question_groups'])
```

With version information:

```python theme={null}
>>> d = s.to_dict(add_edsl_version=True)
>>> 'edsl_version' in d and 'edsl_class_name' in d
True
```

> #### to\_docx(*filename: str | None = None*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Generate a docx document for the survey.

This is the preferred alias for the deprecated `docx` method.

> #### to\_html(*scenario: dict | None = None*, *filename: str | None = None*, *return\_link: bool = False*, *css: str | None = None*, *cta: str = 'Open HTML file'*, *include\_question\_name: bool = False*) → [FileStore](/en/latest/filestore#edsl.scenarios.FileStore "edsl.scenarios.FileStore")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Generate HTML representation of the survey.

This is the preferred alias for the deprecated `html` method.

> #### to\_jobs() → [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Convert the survey to a Jobs object without adding components.

This method creates a Jobs object from the survey without adding any agents, scenarios, or language models. You’ll need to add these components later using the by() method before running the job.

> **Returns:**

Jobs: A Jobs object based on this survey.

> **Examples:**

```python theme={null}
>>> s = Survey.example()
>>> jobs = s.to_jobs()
>>> jobs
Jobs(...)
```

> #### to\_long\_format(*scenario\_list: [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")*) → Tuple\[List\[[QuestionBase](/en/latest/questions#questionbase-class "edsl.questions.QuestionBase")], [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")][\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a new survey with the questions in long format and the associated scenario list.

> #### to\_scenario\_list(*questions\_only: bool = True*, *rename=False*) → [ScenarioList](/en/latest/scenarios#scenariolist-class "edsl.scenarios.ScenarioList")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Convert the survey to a scenario list.

> #### tree(*node\_list: List\[str] | None = None*)[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

> #### using(*obj: 'Cache' | 'KeyLookup' | 'BucketCollection'*) → [Jobs](/en/latest/jobs#jobs-class "edsl.jobs.Jobs")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Turn the survey into a Job and appends the arguments to the Job.

> #### with\_edited\_question(*question\_name: str*, *field\_name\_new\_values: dict*, *pop\_fields: List\[str] | None = None*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a new Survey with the specified question edited.

This method creates a new Survey instance with the specified question edited. The new survey inherits relevant attributes from the parent survey but gets fresh rule collections and memory plans appropriate for the subset of questions.

> #### with\_renamed\_question(*old\_name: str*, *new\_name: str*) → [Survey](/en/latest/#edsl.surveys.Survey "edsl.surveys.survey.Survey")[\[source\]](https://github.com/expectedparrot/edsl/blob/main/edsl/surveys.py)

Return a new survey with a question renamed and all references updated.

This method creates a new survey with the specified question renamed. It also updates all references to the old question name in: - Rules and expressions (both old format ‘q1’ and new format ‘\{\{ q1.answer }}’) - Memory plans (focal questions and prior questions) - Question text piping (e.g., \{\{ old\_name.answer }}) - Question options that use piping - Instructions that reference the question - Question groups (keys only, not ranges since those use indices)

> **Args:**

old\_name: The current name of the question to rename new\_name: The new name for the question

> **Returns:**

Survey: A new survey with the question renamed and all references updated

> **Raises:**

SurveyError: If old\_name doesn’t exist, new\_name already exists, or new\_name is invalid

> **Examples:**

```python theme={null}
>>> s = Survey.example()
>>> s_renamed = s.with_renamed_question("q0", "school_preference")
>>> s_renamed.get("school_preference").question_name
'school_preference'
```

```python theme={null}
>>> # Rules are also updated
>>> s_renamed.show_rules()
```


# Teaching guide
Source: https://docs.expectedparrot.com/en/latest/teaching

This page provides a short guide for getting started using EDSL (_Expected Parrot Domain-Specific Language_) to conduct and collaborate on AI research.

[EDSL](https://github.com/expectedparrot/edsl) is an open-source Python library for simulating surveys, experiments and research tasks with AI agents and language models, and validating results with human respondents. It is fully integrated with [Expected Parrot](https://www.expectedparrot.com), a platform for creating and sharing AI and hybrid AI-human research projects.

<Steps>
  <Step title="Create an account">
    [Create an account](https://www.expectedparrot.com/login) at Expected Parrot. This allows you to access special features for storing, sharing and collaborating on your EDSL projects. Your account comes with an [API key](/en/latest/api_keys) to access all available language models (see information about [current models](https://www.expectedparrot.com/models)) and free [credits](/en/latest/credits) for getting started.
  </Step>

  <Step title="Install the EDSL package">
    Open your terminal or command prompt, navigate to the directory where you want to install the EDSL library, and run the following command:

    ```bash theme={null}
    pip install edsl
    ```

    *Prerequisites:* You must also have Python version 3.9-3.13 and pip installed on your computer. See the [Installation](/en/latest/installation) section for more details.
  </Step>

  <Step title="Open a Python environment">
    After installing EDSL, you can start using it in a Python IDE or text editor of your choice, such as Jupyter Notebook or Visual Studio Code. Save a new notebook in the directory where you installed EDSL and open it.

    You can also start working in EDSL by downloading an example notebook and modifying the code as needed for your research. See the **How-to Guides** and **Notebooks** for various [use cases](https://www.expectedparrot.com/use-cases) listed in the sidebar of the [documentation](/en/latest/index) page, and a [Starter Tutorial](/en/latest/starter_tutorial#starter-tutorial).

    <Note>
      *Note:* If you are using Google Colab, please see [special instructions](/en/latest/colab_setup) on storing and accessing your Expected Parrot API key.
    </Note>
  </Step>

  <Step title="Design your research">
    EDSL allows you to design AI research projects as surveys of questions administered to language models (optionally) using AI agents to answer the questions. A typical workflow consists of the following steps:

    * **Create questions**: Construct [Questions](/en/latest/questions) you want to ask a language model. EDSL provides many common question types that you can choose from based on the form of the response that you want to get back from a model (multiple choice, free text, linear scale, etc.). You can also use built-in methods for importing data to use as scenarios of your questions (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). This can be efficient for data labeling and other workflows where you want to repeat questions with different data inputs.
    * **Construct a survey**: Combine your questions into [Surveys](/en/latest/surveys) to administer them together, and add desired logic, such as skip/stop rules, randomization, piping answers, and more.
    * **Design AI agents**: Specify relevant personas and instructions for AI agents to answer the questions (this is optional).
    * **Select language models**: Choose the [Language Models](/en/latest/language_models) you want to use to generate the answers.
    * **Run experiments**: Administer a survey with AI agents and selected language models. This generates a formatted dataset of results.
    * **Analyze results**: Use built-in methods to analyze your [Results](/en/latest/results).
    * **Validate with humans**: Launch your survey with human respondents to compare the AI-generated responses. Iterate on your survey design and rerun.
  </Step>

  <Step title="Example code">
    Below is example code for creating a survey, running it with AI agents and language models, launching it with humans and comparing results. You can also view the code and output in a downloadable [notebook at Expected Parrot](https://www.expectedparrot.com/content/RobinHorton/example-edsl-teaching).

    Start by creating a survey:

    ```python theme={null}
    from edsl import QuestionMultipleChoice, QuestionNumerical, Survey, Agent, AgentList, Model, ModelList

    # Create questions
    q1 = QuestionMultipleChoice(
        question_name="preferred_source",
        question_text="What is your preferred source of national news?",
        question_options=["television", "social media", "online news", "newspaper", "radio", "podcast", "other"],
    )
    q2 = QuestionNumerical(
        question_name="hours_per_week",
        question_text="How many hours per week on average do you consume national news via {{ preferred_source.answer }}?" # piping answer from previous question
    )

    # Create a survey
    survey = Survey(questions = [q1, q2])
    ```

    Run it with AI agents and language models:

    ```python theme={null}
    # Create AI agents
    a = AgentList(Agent(traits = {"persona":p}) for p in ["college student", "retired professional"])

    # Select language models
    m = ModelList([
        Model("gpt-4o", service_name = "openai"),
        Model("claude-3-sonnet-20240229", service_name = "anthropic")
    ])

    # Run the survey with the agent and model
    results = survey.by(a).by(m).run()
    ```

    To see all columns of the results:

    ```python theme={null}
    # results.columns
    ```

    To inspect components of results:

    ```python theme={null}
    results.select("model", "persona", "preferred_source", "hours_per_week")
    ```

    Example output:

    | model.model | agent.persona | answer.preferred\_source | answer.hours\_per\_week |
    | :- | :- | :- | :- |
    | gpt-4o | college student | online news | 5.0 |
    | claude-3-sonnet-20240229 | college student | online news | 10.0 |
    | gpt-4o | retired professional | newspaper | 2.5 |
    | claude-3-sonnet-20240229 | retired professional | online news | 10.0 |

    Use built-in methods to analyze results. For example:

    ```python theme={null}
    (
        results
            .filter("{{ model.model }} == 'gpt-4o' and {{ agent.persona }} == 'college student'")
            .sort_by("hours_per_week")
            .select("model", "persona", "preferred_source", "hours_per_week", "hours_per_week_comment")
    )
    ```

    Example output:

    ```python theme={null}
    results.sql("""
    select
        model,
        persona,
        preferred_source,
        preferred_source_comment
    from self
    where 1=1
        and persona == 'retired professional'
        and model == 'claude-3-sonnet-20240229'
    """)
    ```

    Example output:

    | model | persona | preferred\_source | preferred\_source\_comment |
    | :- | :- | :- | :- |
    | claude-3-sonnet-20240229 | retired professional | online news | # As a retired professional, I prefer online news sources as they allow me to easily access a wide variety of reputable national and international news outlets at my convenience. Online news is also frequently updated throughout the day. |

    To generate a web-based version to share with human respondents:

    ```python theme={null}
    web_info = survey.humanize()
    web_info
    ```

    Example output:

    ```json theme={null}
    {'project_name': 'Project',
    'uuid': 'cd3dff38-9979-4966-b595-ed9fc6e61362',
    'admin_url': 'https://www.expectedparrot.com/home/projects/cd3dff38-9979-4966-b595-ed9fc6e61362',
    'respondent_url': 'https://www.expectedparrot.com/respond/cd3dff38-9979-4966-b595-ed9fc6e61362'}
    ```

    Import the responses:

    ```python theme={null}
    from edsl import Coop

    coop = Coop()

    human_results = coop.get_project_human_responses(info["uuid"])
    ```

    Combine human and AI results:

    ```python theme={null}
    combined_results = results + human_results

    combined_results.select("model", "agent_name", "preferred_source", "hours_per_week")
    ```

    Example output:

    | model.model | agent.agent\_name | answer.preferred\_source | answer.hours\_per\_week |
    | :- | :- | :- | :- |
    | gpt-4o | Agent\_0 | online news | 5.000000 |
    | claude-3-sonnet-20240229 | Agent\_1 | online news | 10.000000 |
    | gpt-4o | Agent\_2 | newspaper | 2.500000 |
    | claude-3-sonnet-20240229 | Agent\_3 | online news | 10.000000 |
    | test | 50c21352-0c94-4370-b50f-32b7847895e3 | newspaper | 7.000000 |
    | test | a2765047-02c3-4040-ab90-549d12778d96 | social media | 5.000000 |
  </Step>

  <Step title="Share and collaborate">
    Results are automatically stored at your account. You can also post any other EDSL objects to your account from your workspace. For example, to share a notebook of your code:

    ```python theme={null}
    from edsl import Notebook

    nb = Notebook("my_notebook.ipynb")

    nb.push(
        description="My EDSL notebook for AI research",
        alias = "my-notebook",
        visibility = "public"  # or "private" or "unlisted" by defauilt
    )

    ```
  </Step>
</Steps>

**Support and resources**

For more information about using EDSL, please see the [documentation](/en/latest/index) page.

You can also join our [Discord channel](https://discord.com/invite/mxAYkjfy9m) to post questions and get help from the community.


# Token Usage
Source: https://docs.expectedparrot.com/en/latest/token_usage

EDSL comes with a variety of features for monitoring token usage.

These include:

* Options for setting requests per minute (RPM) and tokens per minute (TPM) for models at your account.
* Methods for setting RPM and TPM from your workspace.
* Methods for turning off default prompt features to reduce token usage.
* Methods for calculating next token probabilities.

## Token limits

Token limits refer to the maximum number of tokens that a language model can process in a single input prompt or output generation. A token limit affects how much text you can send to a model in one go. A language model provider should provide information about the token limits for each model that is associated with your account and API key. When running a big job in EDSL, you may encounter token limits, which can be managed by adjusting the token limits for a model.

### RPM: Requests Per Minute

RPM stands for Requests Per Minute, which measures the number of API requests that a user can make to a language model within a minute. This is a metric for managing the load and traffic that a model can handle.

### TPM: Tokens Per Minute

TPM stands for Tokens Per Minute, which is a metric for tracking the volume of tokens processed by a language model within a minute. This metric typically tracks usage for billing purposes.

## Default token limits

You can inspect and modify token limits for a model at the [Keys](https://www.expectedparrot.com/home/keys) page of your account:

<Frame>
  <img alt="Key page view for setting model token rates" />
</Frame>

Here we use methods for inspecting the default language model and its parameters from your workspace, including the token limits:

```python theme={null}
from edsl import Model

model = Model()
model
```

This will show the following information:

| key | value |
| :- | :- |
| model | gpt-4o |
| parameters:temperature | 0.5 |
| parameters:max\_tokens | 1000 |
| parameters:top\_p | 1 |
| parameters:frequency\_penalty | 0 |
| parameters:presence\_penalty | 0 |
| parameters:logprobs | False |
| parameters:top\_logprobs | 3 |

We can also inspect the RPM and TPM for the model:

```python theme={null}
[model.rpm, model.tpm]
```

This will show the following information:

```python theme={null}
[100, 2000000]
```

## Modifying token limits

We can reset the default RPM and TPM and then check the new values:

```python theme={null}
model = Model()

model.rpm=10
model.tpm=10

[model.rpm, model.tpm]
```

This will show the following information:

```python theme={null}
[10, 10]
```

Here we change it again:

```python theme={null}
model = Model()

model.rpm=100
model.tpm=1000

[model.rpm, model.tpm]
```

Output:

```python theme={null}
[100, 1000]
```

Please note that the token limits are subject to the constraints of the model and the API key associated with the model. Let us know if you have any questions or need further assistance with token limits.

## Methods for reducing token usage

There are several ways to reduce the tokens required to run a question or survey.

### Turning off question commments

Each question type (other than free\_text) automatically includes a comment field that gives the answering model a place to put additional information about its response to a question. This serves as an outlet for a chatty model to return context about an answer without violating formatting instructions (e.g., a model may want to provide an explanation for a mutiple choice response but the answer to the question must only be one of the answer options). Question comments can also be useful when used with survey “memory” rules, giving a model an opportunity to simulate a “chain of thought” across multiple survey questions. (By default, questions are administered asynchronously; a model does not have context of other questions and answers in a survey unless memory rules are applied.) Comments can also provide insight into non-responsive (None) answers: a model may use the comments field to describe a point of confusion about a question.

Because the question comment field requires additional tokens, it can sometimes be cost-effective to exclude the field from question prompts, especially when the comment is unlikely to be useful. This is done by passing a boolean parameter include\_comment = False when constructing a question. For example, here we compare a question with comments left on and turned off:

```python theme={null}
from edsl import QuestionNumerical, Survey, ScenarioList

q1 = QuestionNumerical(
  question_name = "sum",
  question_text = "What is the sum of {{ number_1 }} and {{ number_2 }}?"
)

q2 = QuestionNumerical(
  question_name = "sum_silent",
  question_text = "What is the sum of {{ number_1 }} and {{ number_2 }}?",
  include_comment = False
)

survey = Survey([q1, q2])

some_numbers = {
  "number_1": [0,1,2,3,4],
  "number_2": [5,4,3,2,1]
}

s = ScenarioList.from_nested_dict(some_numbers)

results = survey.by(s).run()
```

We can check the responses and confirm that the comment field for the sum\_silent question is None:

```python theme={null}
results.select("number_1", "number_2", "sum", "sum_comment", "sum_silent", "sum_silent_comment")
```

Output:

| scenario.number\_1 | scenario.number\_2 | answer.sum | comment.sum\_comment | answer.sum\_silent | comment.sum\_silent\_comment |
| :- | :- | :- | :- | :- | :- |
| 0 | 5 | 5 | The sum of 0 and 5 is 5. | 5 | None |
| 1 | 4 | 5 | The sum of 1 and 4 is 5. | 5 | None |
| 2 | 3 | 5 | The sum of 2 and 3 is 5. | 5 | None |
| 3 | 2 | 5 | The sum of 3 and 2 is 5. | 5 | None |
| 4 | 1 | 5 | The sum of 4 and 1 is 5. | 5 | None |

### Coding question options

Question instructions can be modified to prompt a model to use codes (integers) in lieu of text responses for answer options, reducing generated tokens.

This is done by passing a boolean parameter use\_code = True to a Question when it is constructed. For example:

```python theme={null}
from edsl import QuestionMultipleChoice

q = QuestionMultipleChoice(
  question_name = "income_pref_coded",
  question_text = "Which of the following is more important to you: ",
  question_options = ["Financial stability", "Moving up the income ladder"],
  use_code = True
)

```

We can inspect the difference in the question prompt that is created by creating an identical question without the parameter and comparing the job prompts. Here we also pass the parameter include\_comment = False:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, Survey, Agent, Model

q1 = QuestionMultipleChoice(
  question_name = "income_pref",
  question_text = "Which of the following is more important to you: ",
  question_options = ["Financial stability", "Moving up the income ladder"]
)

q2 = QuestionMultipleChoice(
  question_name = "income_pref_coded",
  question_text = "Which of the following is more important to you: ",
  question_options = ["Financial stability", "Moving up the income ladder"],
  use_code = True,
  include_comment = False
)

survey = Survey([q1, q2])

# Construct a job with the survey and the default model
job = survey.by(Model())

# Inspect the question prompts
job.prompts().select("question_index", "user_prompt")
```

Output:

| question\_index | user\_prompt |
| :- | :- |
| income\_pref | Which of the following is more important to you:  Financial stability  Moving up the income ladder  Only 1 option may be selected.  Respond only with a string corresponding to one of the options.  After the answer, you can put a comment explaining why you chose that option on the next line. |
| income\_pref\_coded | Which of the following is more important to you:  0: Financial stability  1: Moving up the income ladder  Only 1 option may be selected.  Respond only with the code corresponding to one of the options. |

The prompts can also be inspected after the survey is run:

```python theme={null}
results = survey.by(Model()).run()

(
  results
  .select(
    "income_pref_user_prompt", "income_pref_generated_tokens",
    "income_pref_coded_user_prompt", "income_pref_coded_generated_tokens"
  )
)
```

Output:

| prompt.income\_pref\_user\_prompt | generated\_tokens.income\_pref\_generated\_tokens | prompt.income\_pref\_coded\_user\_prompt | generated\_tokens.income\_pref\_coded\_generated\_tokens |
| :- | :- | :- | :- |
| Which of the following is more important to you:  Financial stability  Moving up the income ladder  Only 1 option may be selected.  Respond only with a string corresponding to one of the options.  After the answer, you can put a comment explaining why you chose that option on the next line. | Financial stability provides a secure foundation and peace of mind, allowing for better long-term planning and resilience against unexpected challenges. | Which of the following is more important to you:  0: Financial stability  1: Moving up the income ladder  Only 1 option may be selected.  Respond only with the code corresponding to one of the options. | 0 |

### No agent instructions

If no agents are used with the survey, the base agent instructions are not sent to the model, reducing overall tokens. (This is a change from prior versions of EDSL.)

## Calculating next token probabilities

We can monitor tokens by calculating next token probabilities. This is done by setting model logprobs = True and then accessing the raw\_model\_response information in the results that are generated. For example:

```python expandable theme={null}
from edsl import QuestionMultipleChoice, Agent, Model

m = Model("gpt-4o", temperature = 1, logprobs = True)

a = Agent(traits = {"persona":"financial advisor"})

q = QuestionMultipleChoice(
  question_name = "income_pref_coded",
  question_text = "Which of the following is more important to you: ",
  question_options = ["Financial stability", "Moving up the income ladder"],
  use_code = True,
  include_comment = False
)

results = q.by(a).by(m).run()

example = results.select("raw_model_response.income_pref_coded_raw_model_response").to_list()[0]

example

Output:

{'id': 'chatcmpl-AcgR7Wy6MMDqoorytBlzS2lzpwQAA',
'choices': [{'finish_reason': 'stop',
'index': 0,
'logprobs': {'content': [{'token': '0',
    'bytes': [48],
    'logprob': -0.00019960667,
    'top_logprobs': [{'token': '0',
        'bytes': [48],
        'logprob': -0.00019960667},
    {'token': '1', 'bytes': [49], 'logprob': -9.000199},
    {'token': 'As', 'bytes': [65, 115], 'logprob': -10.875199}]}],
    'refusal': None},
'message': {'content': '0',
    'refusal': None,
    'role': 'assistant',
    'audio': None,
    'function_call': None,
    'tool_calls': None}}],
'created': 1733782953,
'model': 'gpt-4o-2024-08-06',
'object': 'chat.completion',
'service_tier': None,
'system_fingerprint': 'fp_9d50cd990b',
'usage': {'completion_tokens': 1,
'prompt_tokens': 82,
'total_tokens': 83,
'completion_tokens_details': {'accepted_prediction_tokens': 0,
'audio_tokens': 0,
'reasoning_tokens': 0,
'rejected_prediction_tokens': 0},
'prompt_tokens_details': {'audio_tokens': 0, 'cached_tokens': 0}}}
```

We can use the information to calculate next token probabilities:

```python theme={null}
next_token_probs = example['choices'][0]['logprobs']['content'][0]['top_logprobs']
next_token_probs
```

Output:

```python theme={null}
[{'token': '0', 'bytes': [48], 'logprob': -0.00019960667},
{'token': '1', 'bytes': [49], 'logprob': -9.000199},
{'token': 'As', 'bytes': [65, 115], 'logprob': -10.875199}]
```

Translating the information:

```python theme={null}
import math

# Specifying the codes for the answer options and non-responses:
options = {'0': "Financial stability", '1':"Moving up the income ladder"}

for token_info in next_token_probs:
  option = options.get(token_info['token'], None)
  if option:
    p = math.exp(token_info['logprob'])

    print(f"Probability of selecting '{option}' was {p:.3f}")
```

Output:

```python theme={null}
Probability of selecting 'Financial stability' was 0.992
Probability of selecting 'Moving up the income ladder' was 0.008
```


# Whitepaper
Source: https://docs.expectedparrot.com/en/latest/whitepaper

In progress!

Send us a message at [info@expectedparrot.com](mailto:info@expectedparrot.com) if you are interested in contributing!