Experimental conditions

An experimental condition is a version of the material or situation you want to compare: a different message, price, policy, or product description. In EDSL, scenarios hold those versions, while a shared question defines how each version will be evaluated. Keeping the question and response options the same makes the answers easier to compare. This page builds a small message test: does an instruction that names the file format receive different clarity ratings from a shorter instruction? Four agent profiles evaluate both messages. You will learn how to combine scenarios and agents, run all their combinations, and keep the condition and profile information alongside each answer.

Define what varies

The example represents two kinds of variation separately. The messages ScenarioList supplies what is being evaluated: a short message and a more explicit message. The AgentList supplies the perspective used to evaluate it: new or experienced users, each acting as an individual contributor or team lead.

from edsl import Agent, AgentList, QuestionMultipleChoice, Scenario, ScenarioList

agents = AgentList([
    Agent(traits={"experience": experience, "role": role},
          instruction="Answer using only the supplied profile and message.")
    for experience in ["new user", "experienced user"]
    for role in ["individual contributor", "team lead"]
])
messages = ScenarioList([
    Scenario({"condition": "short", "message": "Export your report."}),
    Scenario({"condition": "explicit", "message": "Download your report as a CSV file."}),
])
question = QuestionMultipleChoice(
    question_name="clarity",
    question_text="How clear is this instruction: {{ scenario.message }}?",
    question_options=["Clear", "Unclear", "Unsure"],
)
job = question.by(agents).by(messages)
# Four profiles crossed with two messages gives eight interviews per model.

Execute the design

Chaining .by(agents).by(messages) gives every profile both messages, in separate interviews. Four profiles times two messages produces eight interviews per model and iteration, each answering the same clarity question. This is a crossed design: EDSL runs every combination of the supplied lists. With model access configured, the following code uses the default model. You can replace Model() with a model you select. First view the answers together with the variables that produced them:

from edsl import Model

results = job.by(Model()).run()
table = results.select("scenario.condition", "agent.experience", "agent.role", "answer.clarity")
table

The resulting table has eight rows. scenario.condition identifies the message, agent.experience and agent.role identify the profile, and answer.clarity contains its rating. To count the ratings for each message in pandas:

frame = table.to_pandas()
counts = frame.groupby(["scenario.condition", "answer.clarity"]).size()
counts

These counts summarize the four configured profiles for each message. Keep the profile columns when you want to see whether a difference is concentrated among new users or team leads, rather than looking only at the totals. The responses describe these model-generated evaluations; the profiles are not recruited human participants.

Choose a different assignment pattern

Crossing is useful when every profile should evaluate every condition. Other designs call for a different relationship between the inputs:

What you want to compare	EDSL approach
Conditions without specified respondent profiles	Use `question.by(messages)` and let the selected model answer without the custom agents.
One scenario for each corresponding agent	Use `job.zip_assign(over=("agents", "scenarios"))` with equally sized lists; see Job assignments.
Only particular agent–scenario combinations	Use `job.assign(...)` for explicit assignments or `job.include_when(...)` for a condition on their fields.
Several message versions within one interview	Use `question.loop(messages)` to create a separate question for each message, then put those questions in a Survey. Add memory if later answers should see earlier ones.
Human participants assigned to scenarios	Publish with `survey.by(messages).humanize(...)` and choose a scenario assignment method; see Human studies.

For additional factors such as price and message wording, construct scenarios containing each combination you want to test. To compare the same design across models, attach a ModelList as shown in Comparing models. Each added profile, condition, or model increases the number of interviews, which you can inspect and estimate costs for before running.

Introduction

Setup and tutorials

EDSL objects

How-to guides

Research examples

Human research

Execution and costs

Saving and sharing

Research and teaching

Expected Parrot platform

Contributing

Define what varies

Execute the design

Choose a different assignment pattern

​Define what varies

​Execute the design

​Choose a different assignment pattern

Define what varies

Execute the design

Choose a different assignment pattern