# Agents Source: https://docs.expectedparrot.com/en/latest/agents Define and vary simulated respondent traits and instructions. An EDSL `Agent` describes the perspective from which a model should answer: for example, a first-time user, an experienced researcher, or a reviewer applying a codebook. Its traits supply background information and its instructions say how to respond. The [Model](/en/latest/language_models) generates the answers; the Agent supplies the role or profile. Use agents to compare responses across defined perspectives or to give a consistent reviewing instruction across questions. You can omit a custom agent when the task only needs a question and source material. An EDSL Agent is distinct from a coding assistant or the [Research Agent](/en/latest/research_agent). ## Define and vary profiles The example combines two experience levels with two roles to create four profiles. Each evaluates both a short message and a more explicit message, answering the same clarity question: ```python theme={null} from edsl import Agent, AgentList, QuestionMultipleChoice, Scenario, ScenarioList agents = AgentList([ Agent(traits={"experience": experience, "role": role}, instruction="Answer using only the supplied profile and message.") for experience in ["new user", "experienced user"] for role in ["individual contributor", "team lead"] ]) messages = ScenarioList([ Scenario({"condition": "short", "message": "Export your report."}), Scenario({"condition": "explicit", "message": "Download your report as a CSV file."}), ]) question = QuestionMultipleChoice( question_name="clarity", question_text="How clear is this instruction: {{ scenario.message }}?", question_options=["Clear", "Unclear", "Unsure"], ) job = question.by(agents).by(messages) # Four profiles crossed with two messages gives eight interviews per model. ``` Four profiles times two messages gives eight interviews per model and iteration. The results retain `agent.experience` and `agent.role`, so you can compare message ratings within each profile. [Experimental conditions](/en/latest/guides/experimental_conditions) shows how to run and analyze this design. These are responses generated from the specified profiles, rather than observations collected from people with those characteristics. ## Traits, instructions, and names `traits` contains profile data. `instruction` describes how the agent should respond. An optional `name` identifies the agent in results; it does not automatically become a demographic trait. Use [Prompts](/en/latest/prompts) to see what is actually sent to the model. For data imported from a CSV or DataFrame, see [Importing data](/en/latest/guides/importing_data). Inspect field names, missing values, and category definitions before using those rows as profiles. ## Advanced: dynamic traits A dynamic traits function supplies traits based on the question being asked. This is useful when different questions should receive different parts of a profile, rather than sending the whole profile every time. The example below keeps a reviewer role and adds the current question's name: ```python theme={null} from edsl import Agent def traits_for_question(question): return {"role": "reviewer", "current_question": question.question_name} agent = Agent(dynamic_traits_function=traits_for_question) ``` Inspect prompts for every affected question and preserve the function with the study. ## Common methods | Call | Arguments and result | | - | - | | `Agent(traits=..., instruction=..., name=...)` | Create a profile from a trait dictionary, response instructions, and an optional identifying name. | | `AgentList([agent, ...])` | Collect profiles to apply to a study. | | `agent.add_trait("role", "reviewer")` | Return a new Agent with the added or replaced trait. | | `agent.remove_trait("role")` | Return a new Agent without that trait. | | `agents.select("role", "experience")` | Return an AgentList containing only the named traits. | | `agents.filter("experience == 'new user'")` | Return an AgentList with profiles matching the expression. | Using the profiles above, select new users before adding them to a job: ```python theme={null} new_users = agents.filter("experience == 'new user'") new_users.select("experience", "role") # In a script, append .print() to display the selected profiles. ``` ## Revise profiles and choose a subset Trait transformations return new Agents, so assign the result: ```python theme={null} from edsl import Agent, AgentList profile = Agent(traits={"role": "reviewer", "experience": "new"}) experienced = profile.update_trait("experience", "experienced") profiles = AgentList([profile, experienced]).with_names("experience", remove_traits=False) profiles.set_instruction("Answer from the perspective of the described person.") profiles.to_scenario_list() ``` `update_trait()` requires the trait to exist; `add_trait()` can introduce a new trait. `agent.rename({"role": "occupation"})` changes trait names, and `agent.translate_traits({"experience": {"new": "new user"}})` recodes values. `agent.select(...)` and `agent.drop(...)` keep or remove traits. `agent.prompt()` shows the traits presentation text; use `job.prompts()` to inspect the complete model prompt. | Task | AgentList methods | | - | - | | Filter and sample profiles | `filter(expression)`, `sample(n=2, seed="pilot")`, `shuffle(seed="pilot")`, and `split(frac_left=0.5, seed=42)`. | | Transform traits | `select(...)`, `drop(...)`, `rename("old", "new")`, `add_trait("condition", values)`, and `remove_trait("condition")`. Supply one value per agent to `add_trait()`. | | Clean imported traits | `filter_na(["role"])` removes missing required traits; `numberify()` converts numeric strings. | | Set names | `with_names("profile_id", remove_traits=False)` returns a named copy. `give_names(...)` modifies the list's agents; both remove naming traits by default unless told otherwise. | | Set shared instructions | `set_instruction(text)` changes every agent; `set_traits_presentation_template(template)` changes how traits are presented to models. These modify agents in place. | | Select traits by question | `set_dynamic_traits_from_question_map({"q1": ["role"], "q2": ["experience"]})` limits the traits supplied for each question. | | Summarize and export | `tally("role")`, `sql(query)`, `to_dataset()`, `to_scenario_list()`, and the [table export methods](/en/latest/dataset#export-and-convert-tables). | `AgentList.from_list("role", values)` creates one agent per value. `AgentList.from_results(results, question_names=["role"])` turns selected answers into traits for a subsequent study; `Agent.from_result(result)` does this for one interview. ## Supply deterministic answers For a simulation with a known response rule, attach a Python function to an Agent: ```python theme={null} from edsl import Agent def answer_directly(self, question, scenario): return "Yes" if self.traits["uses_service"] else "No" agent = Agent(traits={"uses_service": True}) agent.add_direct_question_answering_method(answer_directly) # Use with a question whose allowed answers are Yes and No. # agent.remove_direct_question_answering_method() restores model-based answering. ``` The function supplies the answer without an LLM response. Its return value must suit the question type. Preserve the function with your study so the rule can be inspected and reproduced. # Model access and API keys Source: https://docs.expectedparrot.com/en/latest/api_keys Choose hosted execution or direct provider access, and configure credentials. Constructing questions, surveys, and other objects does not require credentials. Generating responses requires access to the selected model. The test model used in the [Run your first survey](/en/latest/starter_tutorial) tutorial runs locally without credentials or paid calls. ## Choose an execution path | Path | Credentials | Execution | | - | - | - | | Expected Parrot hosted execution | Expected Parrot API key and sufficient usage credits | Expected Parrot runs the job and provides access to supported models. | | Direct provider access | API key for each selected provider | EDSL runs on your machine and calls the provider; provider charges apply. | | Local model server | Endpoint configuration and any server credentials | EDSL calls a server you operate. See [Local models](/en/latest/language_models#connect-to-your-own-model-server). | Local execution describes where the job is orchestrated. A locally orchestrated job can still send requests to a remote provider. ## Expected Parrot ### Log in from a notebook Run this in a notebook cell: ```python theme={null} from edsl import login login() ``` Follow the sign-in link in the cell output to log in to [Expected Parrot](https://www.expectedparrot.com). EDSL retrieves your API key, saves it in a local `.env` file, and makes it available to the current notebook session. You do not need to copy or paste the key. If a valid key is already configured, `login()` reports the signed-in user. These optional arguments let you sign in again or allow more time to finish: ```python theme={null} # Sign in again, for example to use a different account: # login(force=True) # Allow five minutes to finish signing in (the default is two minutes): # login(timeout=300) ``` ### Use an existing API key You can also obtain a key from your Expected Parrot account settings and supply it through the `EXPECTED_PARROT_API_KEY` environment variable or your environment's secret manager. Keep the key and any `.env` file containing it private. For automation, configure the environment variable before starting the process. These CLI commands check authentication and connectivity: ```bash theme={null} ep auth status ep check ``` Check each JSON response's `status` and error details. See [Remote execution](/en/latest/remote_inference) for running and retrieving a job. ## Direct provider access Configure the selected provider's key in its environment variable, such as `OPENAI_API_KEY` for OpenAI or `ANTHROPIC_API_KEY` for Anthropic. Select current identifiers from the model catalog: ```python theme={null} from edsl import Model available = Model.available(service_name="openai") ``` Catalog discovery may require network access. Use [Provider configuration](/en/latest/guides/providers) for Azure and model-specific parameters. Do not assume every provider supports the same model options. For a job constructed with your selected model, call the provider directly from Python: ```python theme={null} results = job.run(disable_remote_inference=True, disable_remote_cache=True) ``` These settings run the job from your environment and keep it from using Expected Parrot's hosted response cache. For a saved job, the CLI alternative is: ```bash theme={null} ep run jobs.ep --local --output results.ep ``` See [Jobs](/en/latest/jobs) for constructing a job and [Local caching](/en/latest/cache) for saving and reusing responses. ## Loading a local .env file If you use a local `.env` file, exclude it from version control and load it before constructing clients: ```python theme={null} from dotenv import load_dotenv load_dotenv() ``` The file contains environment variable assignments; use the names above with your own secret values. In Colab, use its [Secrets panel](/en/latest/colab_setup) instead of embedding keys in cells. ## Troubleshooting access Use `ep info` for redacted diagnostics. For Expected Parrot, use `ep auth status` and `ep check`. For a direct provider, confirm that its key is configured and that the account has access to the selected model. Correct authentication errors before retrying a job. # Shared object methods Source: https://docs.expectedparrot.com/en/latest/base Understand common serialization and persistence methods on EDSL objects. Surveys, agents, scenarios, and results share methods for copying, saving, and inspecting their contents. Use this reference when you want to revise an independent copy, examine an object's definition as Python data, or move it between local files and Expected Parrot. For the everyday save-and-reopen workflow, start with [Saving objects](/en/latest/guides/saving_objects). ## Serialize and reconstruct **Serialization** means representing an object as data that can be stored or transferred. Here `to_dict()` produces a dictionary containing the survey's questions and settings, and `from_dict()` turns it back into a usable Survey. This is useful when another Python tool needs to inspect or exchange the definition; use `save()` and `load()` for ordinary files. ```python theme={null} from edsl import QuestionFreeText, Survey survey = Survey([ QuestionFreeText(question_name="feedback", question_text="What should improve?") ]) payload = survey.to_dict() restored = Survey.from_dict(payload) assert restored.to_dict() == payload ``` `restored` contains the same questionnaire. Reconstructing it generates no answers; it is ready to edit, attach to a job, or publish for people to answer. ## Copies, text representations, and files Use `object.duplicate()` on core study objects when you want an independent object to revise. Methods such as `question.copy()` and `survey.copy()` are also available. Collection transformations can return a new object or mutate the original; the object guides call out the distinction. `object.to_json()` and `object.to_yaml()` provide readable serialized representations on objects that support them. `to_dict()` produces Python data; use the matching class's `from_dict()` to reconstruct the object. `Survey.from_jsonl(...)`, `ScenarioList.from_jsonl(...)`, `AgentList.from_jsonl(...)`, `ModelList.from_jsonl(...)`, `Jobs.from_jsonl(...)`, and `Results.from_jsonl(...)` read their EDSL JSONL representations. These are structured EDSL formats, not interchangeable readers for arbitrary row data. `object.save("study.ep")` and `ObjectClass.load("study.ep")` preserve core study objects as packages. Use an explicit `.json` or `.json.gz` filename for JSON storage. `save_json()` and `load_json()` are the explicit JSON-only methods. A data export such as `results.select(...).to_csv(...)` preserves selected table columns; it does not preserve the entire study object. ## Remote object shortcuts For supported object classes, `ObjectClass.list(page=1, page_size=20, search_query="pilot")` lists remote objects of that type. `ObjectClass.pull(uuid)` downloads one object, and `object.push(description="...", visibility="private")` uploads it. These call Expected Parrot and require the appropriate access. Use `ObjectClass.patch(uuid, value=updated_object)` to replace remote content, `ObjectClass.patch_metadata(uuid, description="...")` for metadata, and `ObjectClass.share(uuid, username_or_email="collaborator@example.com")` for access. `unshare()` removes a recipient's access and `delete()` removes the remote object. The [Coop guide](/en/latest/coop#update-and-share) explains the corresponding client methods and when each operation is appropriate. Jobs have different remote semantics: `Jobs.list(job_status="completed", page=1, page_size=20)` lists executions, and `Jobs.pull(job_uuid)` retrieves a submitted job's specification. The metadata and access shortcuts above are not supported for Jobs; use [background job methods](/en/latest/guides/background_jobs) to manage executions and Results object methods to share completed data. For durable files, use [Saving objects](/en/latest/guides/saving_objects). For version history and package internals, see [.ep packages](/en/latest/survey_ep_format). For uploads and shared-object operations, see [Expected Parrot objects](/en/latest/coop). # Local caching Source: https://docs.expectedparrot.com/en/latest/cache Reuse model responses locally and preserve the cache alongside a study. A `Cache` stores model responses so you can rerun a job using the same responses when the inputs match. This is useful when extending a study: matching calls can reuse earlier responses while new inputs receive new answers. A cache matches individual model calls using their prompts, model settings, and iteration. It differs from a saved Results object, which you reopen to analyze the already-collected dataset without running the job again. ## Run with an explicit local cache This example uses the test model, makes no paid model calls, and disables hosted execution, API proxying, and remote caching. It writes a cache file in the current directory. ```python theme={null} from edsl import Cache, Model, QuestionFreeText cache = Cache() question = QuestionFreeText( question_name="feedback", question_text="What would improve this service?" ) job = question.by(Model("test", canned_response="A clearer export button.")) settings = dict( cache=cache, disable_remote_inference=True, disable_remote_cache=True, progress_bar=False, ) first = job.run(**settings) second = job.run(**settings) assert first.select("answer.feedback").to_list() == second.select("answer.feedback").to_list() cache.write_jsonl("feedback-cache.jsonl") saved_cache = Cache.from_jsonl("feedback-cache.jsonl") assert len(saved_cache) == len(cache) ``` The first run fills the empty cache; the second reuses its entry. Both return the fixed test answer in this example. Inspect `second.select("cache_used.*")` to see the reuse indicator; identical answer text alone would not tell you whether a new call occurred. `saved_cache` contains the entries written to `feedback-cache.jsonl`. Use a configured provider model in place of `Model("test", ...)` for real responses. Locally orchestrated execution can still call a remote provider and incur costs. An empty Cache starts without stored responses; it does not make a real model free or offline. ## Preserve and reload `write_jsonl()` and `from_jsonl()` provide a portable cache file. SQLite persistence is available through `write_sqlite_db()` and `from_sqlite_db()`. Load a saved cache and pass it to `job.run(cache=cache)` to reuse its responses. Save and load completed Results with `results.save()` and `Results.load()`; see [Saving objects](/en/latest/guides/saving_objects). ## Fresh responses and remote cache To request new model responses with local orchestration, pass a new `Cache()` together with `disable_remote_inference=True` and `disable_remote_cache=True`. For hosted execution, `run(offload_execution=True, fresh=True)` requests new responses. In this version, `fresh=True` alone does not bypass entries already in a supplied local cache. Expected Parrot's [remote cache](/en/latest/remote_caching) is a separate hosted service. `disable_remote_cache=True` controls remote cache access; it does not disable local reuse or hosted execution by itself. ## Common methods | Call | Arguments and result | | - | - | | `Cache()` | Create an empty local cache. | | `len(cache)` | Count stored entries. | | `cache.write_jsonl("responses.jsonl")` | Write cache entries to a portable JSONL file. | | `Cache.from_jsonl("responses.jsonl")` | Restore a Cache from that file. | | `cache.write_sqlite_db("responses.db")` | Write entries to a SQLite database. | | `Cache.from_sqlite_db("responses.db")` | Load a Cache from the database. | To reuse the cache saved by the example above in another session: ```python theme={null} from edsl import Cache cache = Cache.from_jsonl("feedback-cache.jsonl") len(cache) # In a script: print(len(cache)) # Pass cache=cache to your next job.run(...) call to reuse matching entries. ``` ## Inspect, select, and combine entries `cache.select("model", "user_prompt", "output")` returns a Dataset for inspection. `to_scenario_list()`, `to_dataset()`, and `to_pandas()` provide other representations. Each stored entry contains model parameters, system and user prompts, the raw model output, and iteration information. | Task | Method | | - | - | | Inspect keys and entries | `cache.keys()`, `cache.values()`, and `cache.items()` provide keys, CacheEntry objects, and key-entry pairs. `entry.to_dict()` exposes an entry's fields. | | Keep particular entries | `cache.subset(keys)` returns a Cache with the requested keys. | | Merge entries | `cache.add_from_dict({entry.key: entry for entry in entries})` adds CacheEntry objects. A conflicting value for an existing key raises an error. | | Merge a saved cache | `cache.add_from_jsonl("other.jsonl")` or `cache.add_from_sqlite("other.db")` adds entries to this cache. | | Access the configured local cache | `Cache.from_local_cache()` opens the configured SQLite cache. Use an explicit Cache when a study needs isolated storage. | | Keep entries added during use | `cache.new_entries_cache()` returns a Cache containing tracked new and fetched entries. | See [Remote caching](/en/latest/remote_caching#download-cache-entries-for-a-job) for downloading hosted entries by job UUID or cache key. # Citation Source: https://docs.expectedparrot.com/en/latest/citation If you use the EDSL package in your research or projects, please use the following citation: ```bibtex theme={null} @techreport{Horton2024EDSL, title = {EDSL: Expected Parrot Domain Specific Language for AI Powered Social Science}, author = {John Horton and Robin Horton}, year = {2024}, institution = {Expected Parrot}, type = {Whitepaper} } ``` # EDSL CLI Guide Source: https://docs.expectedparrot.com/en/latest/cli Create surveys, run jobs, collect human responses, and inspect saved data from the command line, scripts, or coding agents. The EDSL command-line interface lets you create and validate study objects, run surveys with language models, publish surveys for human participants, and inspect or export responses. Use it from a terminal, a script, or a coding agent. This guide uses `ep`. The commands `edsl` and `python -m edsl` are equivalent entry points. An EDSL `Agent` describes a survey respondent; a coding agent is software that can use the CLI to build and run a study. ## Install and discover commands Install EDSL in your Python environment: ```bash theme={null} pip install edsl ``` Check the installed version and available commands: ```bash theme={null} ep --help ep info ep run --help ep jobs --help ep results --help ep schema list ``` Use the installed CLI's help to check flags before constructing a command. `ep info` reports configuration diagnostics and redacts credentials. See [Installation](/en/latest/installation) for environment setup. ## Try a local example Run these commands in a working directory where you want to save the example files. They create a survey and execute it with EDSL's test model, which returns a fixed answer. This example requires no model credentials and makes no paid model calls; its output is test data. Create a question and save it as a survey: ```bash theme={null} ep surveys create \ --question-type free_text \ --question-name feedback \ --question-text "What would improve this service?" \ --output survey.ep ``` Create the test model configuration and combine it with the survey in a saved job: ```bash theme={null} ep models create \ --model test \ --canned-response "A test response." \ --output test-models.ep ep jobs build \ --survey survey.ep \ --models test-models.ep \ --output test-job.ep ``` Validate the job, run it locally, and inspect the answer: ```bash theme={null} ep validate --file test-job.ep ep run test-job.ep --local --output test-results.ep ep results select --file test-results.ep --column answer.feedback --limit 5 ``` The selected answer is `A test response.` The completed responses are saved in `test-results.ep`; the `run` command returns execution metadata rather than response rows. Read each command's JSON response and check `status` before proceeding to the next command. ## Read command responses By default, operational commands write one JSON document to stdout. Diagnostics go to stderr, and output captured from lower-level operations may appear in `warnings`. Help output from `--help` is plain text. Options that explicitly request raw output, such as `--output -` on object-creation commands, bypass the usual response envelope. A successful response has this structure: ```json theme={null} { "status": "ok", "data": {}, "warnings": [] } ``` `data` contains command-specific information. When a command saves a file, inspect the returned save metadata for its path and format. An error response has this structure; `suggestion` is included when available: ```json theme={null} { "status": "error", "error": { "code": "USAGE_ERROR", "message": "Completed run results must be written to a file.", "suggestion": "Use --output results.ep or --output results.json." } } ``` For automation, parse stdout as JSON, inspect `status`, read warnings, and check the process exit code. Do not extract identifiers or URLs from progress messages. For example: ```python theme={null} import json import subprocess import sys process = subprocess.run( ["ep", "info"], capture_output=True, text=True, ) response = json.loads(process.stdout) for warning in response.get("warnings", []): print(warning, file=sys.stderr) if response["status"] == "error": error = response["error"] raise RuntimeError( f"{error['code']}: {error['message']} " f"{error.get('suggestion', '')}" ) process.check_returncode() data = response["data"] ``` If stdout cannot be parsed as JSON, treat it as a command failure and inspect stderr. Do not continue with missing or guessed output values. ## Create and validate study objects Prefer `.ep` packages for objects you want to save, inspect, reuse, or share. The CLI also accepts serialized JSON for many commands. A `Jobs` object combines the survey with its agents, scenarios, and models. Inspect schemas before constructing serialized objects by hand: ```bash theme={null} ep schema show --class Survey ep schema show --class Jobs ep schema show --question_type multiple_choice ``` You can also use object-creation commands directly. For example, add a question to the survey from the local example: ```bash theme={null} ep surveys add-question survey.ep \ --question-type multiple_choice \ --question-name recommend \ --question-text "Would you recommend this service?" \ --option Yes \ --option No ep validate --file survey.ep ep inspect survey.ep ``` `add-question` updates the survey package in place. A job built earlier retains its saved survey, so rebuild the job to include the new question. Use `ep agents create --help` and `ep scenarios create --help` for creating respondent profiles and scenario data from files. If validation fails, correct the reported problem before running the object. ## Run with language models For hosted execution, configure an Expected Parrot account and API key as described in [Managing Keys](/en/latest/api_keys). Check authentication and connectivity: ```bash theme={null} ep auth status ep check ``` Discover available models: ```bash theme={null} ep models --text ``` Replace `MODEL_NAME` below with a model returned by the catalog. These commands build and estimate a job; the final command runs it and may incur model usage costs: ```bash theme={null} ep models create --model MODEL_NAME --output models.ep ep jobs build --survey survey.ep --models models.ep --output jobs.ep ep validate --file jobs.ep ep jobs cost jobs.ep ep run jobs.ep --output results.ep ``` Blocking runs require an output file. Use `.ep` for a Results package or `.json` for serialized data. To disable remote inference, add `--local`. This runs execution on your machine but can still call a model provider over the network and incur charges. Provider credentials or a configured local model server are required for real model calls. See [Connect to your own model server](/en/latest/language_models#connect-to-your-own-model-server) for endpoint configuration. When overriding a saved job, `--agent_list`, `--scenario_list`, and `--model_list` replace the corresponding component. `--model` replaces the model list with one model; it cannot be combined with `--model_list`. ## Submit a background job Use background execution for long remote jobs: ```bash theme={null} ep run jobs.ep --background ``` The response includes job metadata under `data.meta.remote_job` when available. Read its `job_uuid` and use that value in follow-up commands. Replace `JOB_UUID` below with the returned identifier: ```bash theme={null} ep jobs status JOB_UUID ep jobs results JOB_UUID --output results.ep ep jobs errors JOB_UUID --output error.md ``` Fetch results after the job completes; fetch the error report when investigating failures. To submit, wait, and save in one command: ```bash theme={null} ep run jobs.ep --background --wait \ --poll_interval 10 --timeout 3600 --output results.ep ``` `--background` alone returns before responses are ready. Do not combine `--background` with `--output` unless you also use `--wait`. ## Collect human responses After configuring Expected Parrot authentication, you can publish the saved survey for human participants: ```bash theme={null} ep validate --file survey.ep ep humanize create --survey survey.ep --name "Service feedback" ``` `humanize create` creates a hosted survey. Read the returned survey identifier and respondent URL from the JSON response. Share the respondent URL with participants when you are ready to collect responses. Replace `HUMAN_SURVEY_UUID` with the returned identifier to inspect the survey and save responses: ```bash theme={null} ep humanize status HUMAN_SURVEY_UUID ep humanize responses HUMAN_SURVEY_UUID --output human-results.ep ep results columns --file human-results.ep ep results select --file human-results.ep --column "answer.*" --limit 5 ``` You can also publish from a Jobs package with `--jobs`. Jobs used for human surveys must not include models. If they contain scenarios, supply `--scenario_method`; use `ep humanize create --help` to see the available assignment methods. See [Humanize](/en/latest/humanize) for survey configuration and [Prolific studies](/en/latest/prolific) for participant recruitment. ## Inspect and export saved responses Use results commands on saved files: ```bash theme={null} ep results columns --file results.ep ep results select --file results.ep \ --column answer.feedback --column model.model --limit 5 ep results export results.ep \ --column answer.feedback --output answers.csv ``` Repeat `--column` for multiple columns; do not pass a comma-separated list. Inspect available columns before selecting them. Use a bounded `select` to review responses and `export` to save larger datasets. Results queries support `.ep`, `.json`, and `.json.gz` files. ## Automation conventions * Discover commands and schemas from the installed version before generating inputs. * Validate inputs, check every response's `status`, and read `warnings` before continuing. * Keep credentials out of logs and shared files. Use the configuration described in [Managing Keys](/en/latest/api_keys). * Use saved objects and returned identifiers to continue work. A background submission is not a completed run. * Pass explicit pagination, for example `ep jobs list --page 1 --page_size 20`, and inspect returned pagination metadata. * Retry remote operations only when doing so will not duplicate work. Check job status before submitting another run. * Obtain explicit user direction before destructive actions such as `ep delete` or `ep jobs cancel`; both require `--yes`. ### Exit codes | Exit code | Meaning | | - | - | | `0` | Success | | `1` | General error | | `2` | Usage error | | `3` | Resource not found | | `4` | Authentication error | | `5` | Validation error | | `6` | Remote service error | Use the JSON error's `code`, `message`, and optional `suggestion` to decide the next step. For example, correct flags after `USAGE_ERROR`, inspect schemas after `VALIDATION_ERROR`, and check credentials after `AUTH_ERROR`. # Sharing a Colab notebook Source: https://docs.expectedparrot.com/en/latest/colab_notebooks Save a Colab notebook as supporting material for a study. First use [Colab setup](/en/latest/colab_setup) to install EDSL and configure any required credentials. Sharing a notebook is separate from saving the survey, job, and completed results; preserve those objects with [Saving objects](/en/latest/guides/saving_objects). ## Download and inspect the notebook In Colab, download the notebook as an `.ipynb` file. Review its code, cell outputs, and embedded data before sharing. Remove credentials and any private data that should not be included. From an environment with EDSL and that file available: ```python theme={null} from edsl import Notebook notebook = Notebook(path="study.ipynb") ``` Replace `study.ipynb` with the downloaded file's path. This reads the notebook locally; it does not execute its cells. ## Upload supporting material With Expected Parrot authentication configured: ```python theme={null} info = notebook.push( description="Notebook accompanying the study", visibility="private", ) ``` This uploads the notebook. Retain the returned identifier and configure sharing deliberately. See [Notebooks and scripts](/en/latest/notebooks) for updating, downloading, and sharing notebook objects. # Google Colab setup Source: https://docs.expectedparrot.com/en/latest/colab_setup Install EDSL and load credentials in a Colab notebook. [Google Colab](https://colab.research.google.com) runs Python notebooks in a hosted environment. In a new notebook, install EDSL in a code cell: ```text theme={null} %pip install edsl ``` Then follow the [Run your first survey](/en/latest/starter_tutorial) tutorial. Its local test model needs no credentials. Re-run installation when you start a new runtime. ## Log in to Expected Parrot To use Expected Parrot's model access, run this in a cell: ```python theme={null} from edsl import login login() ``` Follow the sign-in link in the output. EDSL retrieves your API key and configures the notebook session automatically. Run `login()` again when you start a new runtime. See [Notebook login](/en/latest/api_keys#log-in-from-a-notebook) for options. ## Use Colab secrets as an alternative If you already have an Expected Parrot API key, you can instead add `EXPECTED_PARROT_API_KEY` to Colab's Secrets panel and enable access for this notebook. Load it without displaying the value: ```python theme={null} import os from google.colab import userdata os.environ["EXPECTED_PARROT_API_KEY"] = userdata.get("EXPECTED_PARROT_API_KEY") ``` For direct provider access, store and load the provider's environment variable instead. See [Model access](/en/latest/api_keys) to choose the execution path. Do not paste keys into notebook cells or saved output. ## Preserve your work The runtime's local files can disappear when the session ends. Save study packages and results to durable storage or download them before closing the runtime. [Sharing a Colab notebook](/en/latest/colab_notebooks) explains how to save the notebook itself as supporting material. # Contributing Source: https://docs.expectedparrot.com/en/latest/contributing Report problems, suggest features, and share EDSL code or examples. EDSL is open source. We welcome bug reports, feature suggestions, and useful code or research examples through the [GitHub repository](https://github.com/expectedparrot/edsl). ## Report a problem Open an issue with your EDSL version, a small example that reproduces the problem, what you expected, and what happened. `ep info` provides environment diagnostics with credentials redacted. Remove private study data and credentials from anything you share. ## Suggest a feature or share an example Describe what you want to accomplish and how the feature or example would help. You can open an issue to discuss an idea or a pull request with proposed code. Include a short explanation of how to use it. For questions or help with a study, email [info@expectedparrot.com](mailto:info@expectedparrot.com). # Expected Parrot objects Source: https://docs.expectedparrot.com/en/latest/coop Store, retrieve, update, and share EDSL objects through the Expected Parrot platform. Expected Parrot is a platform for model access, remote execution, caching, human research, and sharing study materials. The `Coop` client exposes its services in Python. This page covers storing and retrieving EDSL objects; see [Remote Inference](/en/latest/remote_inference), [Remote Caching](/en/latest/remote_caching), and [Human studies](/en/latest/guides/human_studies) for execution and collection workflows. Use these methods to reopen a study from another computer, reuse a collaborator's questionnaire, or share completed results. `push()` uploads an object and `pull()` retrieves it as a usable EDSL object. The returned **UUID** is its unique identifier, which you can keep instead of searching for the object each time. ## Authentication Create an Expected Parrot account and configure your key using [Managing Keys](/en/latest/api_keys). Remote operations below require authentication. Saving a local [`.ep` package](/en/latest/guides/saving_objects) does not upload it. ## Upload and retrieve a survey This example creates a local survey, then uploads a private copy. The upload stores the definition; it does not run the survey or generate responses. ```python theme={null} from edsl import QuestionFreeText, Survey survey = Survey([ QuestionFreeText(question_name="feedback", question_text="What should improve?") ]) info = survey.push(description="Feature feedback survey", visibility="private") survey_uuid = info["uuid"] restored = Survey.pull(survey_uuid) ``` Keep the returned UUID with the study. Choose visibility explicitly. Sharing a survey definition is separate from publishing a respondent link; use `survey.humanize()` when collecting human responses. ## Use the client directly The equivalent client operations let you work with multiple object types. Continuing with the UUID returned above: ```python theme={null} from edsl import Coop coop = Coop() restored = coop.get(survey_uuid, expected_object_type="survey") ``` Use `coop.push(object, description=..., visibility=...)` to upload another object. Objects such as surveys, agents, scenarios, and results preserve different parts of a study; saving one does not automatically preserve every related artifact. ## Find objects with filters and pagination `coop.list()` returns a `CoopRegularObjects` collection of object metadata. For example, find private surveys and results whose descriptions match a study label: ```python theme={null} from edsl import Coop coop = Coop() objects = coop.list( object_type=["survey", "results"], visibility="private", search_query="Customer feedback", page=1, page_size=20, sort_ascending=False, ) objects.select("uuid", "object_type", "description", "visibility") # In a script, append .print() to display the selected metadata. ``` | Argument | Meaning | | - | - | | `object_type` | One type or a list, such as `"survey"`, `"results"`, `"agent_list"`, or `"scenario_list"`. Omit to include all types. | | `visibility` | One value or a list drawn from `private`, `unlisted`, and `public`. | | `search_query` | Search descriptions; this does not search the contents of questions or answers. | | `page` | Page number, starting at **1**. | | `page_size` | Objects per page, from 1 to 100. | | `sort_ascending` | `False` returns the most recently created objects first. | | `community` | `False` searches objects you own or that are shared with you. `True` searches public objects owned by other users. | The returned collection exposes `current_page`, `total_pages`, `page_size`, and `total_count`. To retrieve another page, call `coop.list()` again with the same filters and a different `page` value. Fetching one page does not fetch all matches. You can also filter the metadata you have already downloaded: ```python theme={null} surveys_on_page = objects.filter("object_type == 'survey'") surveys_on_page.select("uuid", "description") ``` This local filter returns a ScenarioList and only examines the current page. Use `object_type="survey"` in `coop.list()` to filter the search across all matching objects. ## Retrieve the listed objects Listing metadata does not download each object's contents. Call `fetch()` on the original collection to download all objects on that page: ```python theme={null} downloaded = objects.fetch() # A Python list of EDSL objects. ``` To download only the surveys selected above, retrieve each by its object UUID: ```python theme={null} surveys = [ coop.pull(item["uuid"], expected_object_type="survey") for item in surveys_on_page ] ``` For a known object UUID or URL, `coop.pull(identifier, expected_object_type="results")` retrieves the Results object directly; `Results.pull(identifier)` is the class-specific alternative. `coop.get()` also retrieves and deserializes objects. These identifiers refer to stored objects. Use the [background jobs guide](/en/latest/guides/background_jobs#find-jobs-with-filters-and-pagination) to find execution records and obtain their results UUIDs. ## Update and share `coop.patch(survey_uuid, description="Revised description")` updates metadata. Supply `value=updated_survey` to replace object content deliberately. Preserve a local version before updating a shared definition that collaborators may rely on. Read metadata without downloading the object with `coop.get_metadata(survey_uuid)`. `coop.patch_metadata(survey_uuid, description="Revised description", visibility="private")` changes metadata without replacing the object contents. Use Python methods for access management: ```python theme={null} coop.get_object_shared_users(survey_uuid) # To grant access to a particular person: # coop.share_object(survey_uuid, username_or_email="collaborator@example.com") # To remove their access later: # coop.unshare_object(survey_uuid, username_or_email="collaborator@example.com") ``` Sharing and unsharing change who can access the remote object. Private, unlisted, and public visibility have different access implications; choose the appropriate setting before sharing a link. The [CLI alternatives](/en/latest/cli) are `ep shared`, `ep share`, and `ep unshare`. ## Delete a remote object `coop.delete(survey_uuid)` deletes the remote object. Use it only when that deletion is intended. It is separate from removing a local file. The CLI equivalent is `ep delete`, which requires explicit confirmation in automated use. ## Research materials and provenance Keep questions, logic, source data, model settings, prompts, results, and the EDSL version needed to interpret a run. Use [Notebooks](/en/latest/notebooks) to share supporting notebooks or scripts and [Saving objects](/en/latest/guides/saving_objects) for local versioning. ## Common methods These client methods contact Expected Parrot and require configured access. `url_or_uuid` can be a stored object's URL or UUID. | Call | Arguments and result | | - | - | | `coop.push(object, description=..., visibility="private")` | Upload an EDSL object and return its remote metadata, including its UUID. | | `coop.pull(url_or_uuid, expected_object_type="results")` | Download and deserialize a stored object; optionally require a particular type. | | `coop.get(url_or_uuid, expected_object_type="survey")` | Retrieve and deserialize an object; optionally require a particular type. | | `coop.list(object_type="survey", page=1, page_size=10)` | Retrieve a page of accessible objects, optionally filtered by type. | | `coop.patch(url_or_uuid, description=...)` | Update supplied metadata fields. Use `value=updated_object` to replace the stored content. | | `coop.delete(url_or_uuid)` | Delete the remote object when you intend to remove it. | # Estimating and inspecting costs Source: https://docs.expectedparrot.com/en/latest/costs Estimate a job in USD and credits, inspect its cost breakdown, and review completed results. EDSL provides methods to estimate a job before running it and inspect recorded costs afterward. Cost estimation can retrieve pricing information or request an estimate from Expected Parrot; it does not run the survey or generate model responses. ## Estimate a prepared job Combine a survey with the agents, scenarios, and models you plan to use. For example: ```python theme={null} from edsl import Agent, Model, QuestionFreeText, Survey survey = Survey([ QuestionFreeText( question_name="favorite_flower", question_text="What is your favorite flower?", ), QuestionFreeText( question_name="flower_color", question_text="What color is {{ favorite_flower.answer }}?", ), ]) agent = Agent(traits={"occupation": "botanist"}) model = Model() # Use your configured default, or specify a model from the catalog. job = survey.by(agent).by(model) # Alternatively: from edsl import Jobs; job = Jobs.load("jobs.ep") ``` Call `estimate_job_cost()` to get a dictionary of estimated costs and token counts: ```python theme={null} estimate = job.estimate_job_cost(iterations=3) estimate # Omit iterations for one run; use iterations=3 to estimate a run with n=3. ``` | Returned field | Meaning | | - | - | | `estimated_total_cost_usd` | Estimated cost in USD for all requested iterations. | | `total_credits_hold` | Corresponding estimated credits. | | `estimated_total_input_tokens` | Estimated input tokens across the job. | | `estimated_total_output_tokens` | Estimated output tokens across the job. | | `detailed_costs` | Breakdown by inference service, model, and input/output token type, including token counts, prices, and costs. | To inspect the breakdown as a table: ```python theme={null} from edsl import ScenarioList ScenarioList.from_list_of_dicts(estimate["detailed_costs"]) ``` This method estimates tokens from rendered prompt lengths and applies the pricing table. Its text approximation uses four characters per input token and output tokens equal to 75% of input tokens, rounded up. Prompt text containing unresolved Jinja placeholders receives a multiplier of two before token estimation. These are estimation assumptions; no answers are generated. For question-specific estimates and adjustable assumptions, see [Detailed estimates](#detailed-estimates). ## Estimate hosted execution in credits and USD With [Expected Parrot authentication](/en/latest/api_keys#log-in-from-a-notebook) configured, request the server's estimate for the same job: ```python theme={null} from edsl import Coop coop = Coop() remote_estimate = coop.remote_inference_cost(job, iterations=3) remote_estimate ``` The returned dictionary includes `usd` and `credits_hold`, along with estimate metadata such as `estimate_kind`, `assumptions`, and `warnings` when available. `remote_inference_cost()` also accepts a Survey; pass a Jobs object when you want the estimate to use your selected agents, scenarios, and models. The CLI equivalent is `ep jobs cost jobs.ep --iterations 3`. See [Credits](/en/latest/credits) for account balances and usage. ## Inspect prompts and their estimated costs `job.prompts()` returns the rendered prompts and an estimated cost for each question in each planned interview: ```python theme={null} job.prompts().select("question_name", "model", "estimated_cost", "user_prompt", "system_prompt") # To print only the prompts: job.show_prompts() # To print all prompt-table columns: job.show_prompts(all=True) ``` Each `estimated_cost` is in USD for one question in one interview, before multiplying by iterations. The preview includes scenario values and agent instructions; answers to earlier questions become available when the survey runs. See [Prompts and instructions](/en/latest/prompts) for inspecting the prompts recorded with completed responses. ## Detailed estimates For a breakdown that accounts for question types, file inputs, and survey memory, use `estimate_remote_job_cost()`: ```python theme={null} detailed = job.estimate_remote_job_cost() detailed.detail.select("question_name", "model", "total_input_tokens", "total_output_tokens", "cost_usd") ``` Despite its name, this method calculates the estimate locally, using retrieved prices unless you supply a `price_lookup`. It returns a `JobCostEstimate` for one iteration; it does not submit a remote job. | Attribute or method | What it provides | | - | - | | `detailed.total_cost_usd`, `detailed.total_credits` | Total estimated cost. | | `detailed.total_input_tokens`, `detailed.total_output_tokens` | Total estimated token counts. | | `detailed.detail` | Dataset with a row for each question in each planned interview. | | `detailed.summary_by_question()` | List of dictionaries aggregating costs by question. | | `detailed.summary_by_model()` | List of dictionaries aggregating costs by model and service. | | `detailed.warnings` | Notes about approximations and unsupported inputs. | | `detailed.to_markdown("cost-estimate.md")` | Save a report of costs, assumptions, and warnings; omit the path to return the Markdown without writing a file. | You can supply your own expected response lengths. For example, estimate a 100-token answer to the `favorite_flower` question: ```python theme={null} from edsl.jobs.cost_estimation import TokenOverride adjusted = job.estimate_remote_job_cost( token_overrides={"favorite_flower": TokenOverride(answer_tokens=100)}, ) adjusted ``` `TokenOverride` also accepts `comment_tokens` and `thinking_tokens`; omitted fields retain their estimates. Optional `service` and `model` fields restrict an override to a particular model. These values change the estimate, not the model's output limits. For surveys with branching, `branch_weights={("from_question", "to_question"): probability}` supplies expected transition probabilities. Without branch weights, the estimator assumes every question is reached. `chars_per_token` adjusts the text-to-token approximation. ## Reuse a pricing table `Coop.fetch_prices()` returns a pricing dictionary keyed by `(inference_service, model)`. You can reuse that table across estimates: ```python theme={null} prices = coop.fetch_prices() job.estimate_job_cost_from_external_prices(price_lookup=prices, iterations=3) # The detailed estimator accepts the same table: # job.estimate_remote_job_cost(price_lookup=prices) ``` The table contains `input` and `output` price entries. If a model is missing, the estimator falls back to available service prices or default rates; inspect the prices used in the breakdown. ## Inspect recorded costs Load completed results and compute the recorded job cost: ```python theme={null} from edsl import Results results = Results.load("results.ep") results.compute_job_cost() # Include recorded costs for cached responses as well: # results.compute_job_cost(include_cached_responses_in_cost=True) ``` By default this excludes cached responses. `include_cached_responses_in_cost=True` includes their recorded costs for comparison; those costs are not necessarily new charges. Missing provider usage or pricing information can limit the estimate. Use your account or provider's billing record to reconcile actual charges. ## Keep a cost ledger The CLI can record an estimate or the cost from a saved result package: ```bash theme={null} ep costs log --output costs.jsonl --estimated 1.25 --note "Budget estimate for pilot" ep costs log --output costs.jsonl --actual-from results.ep --note "Completed pilot" ``` The first amount is an illustrative manual estimate. Recording it does not reserve credits or enforce a limit. See [Credits](/en/latest/credits) for Expected Parrot account usage, [Token limits](/en/latest/token_usage) for execution constraints, and [Caching](/en/latest/cache) for response reuse. # Expected Parrot credits Source: https://docs.expectedparrot.com/en/latest/credits Check usage credits for hosted model execution and platform services. Expected Parrot uses usage credits for hosted services. Direct calls made with a model provider's own API key are billed by that provider. Constructing EDSL objects and running the local test model do not require usage credits. ## Check your account With Expected Parrot authentication configured: ```python theme={null} from edsl import Coop coop = Coop() coop.get_balance() ``` `get_balance()` returns account balance information. Use `coop.get_profile()` to inspect the authenticated account. The CLI alternatives are: ```bash theme={null} ep balance ep credits ``` `ep balance` returns the account balance in a JSON response. `ep credits` opens or returns the credits page; inspect the response instead of assuming a browser opened. You can also visit [account credits](https://www.expectedparrot.com/home/credits). Review current pricing and the estimated workload before submitting a job. The [Costs guide](/en/latest/costs) explains estimates, recorded costs, and cost ledgers. A balance check is not a reservation of funds or a limit on future spending. ## Academic research and teaching We support academic researchers, instructors, and students using EDSL for research and coursework. Contact [info@expectedparrot.com](mailto:info@expectedparrot.com) about available support and usage credits for your project or class. See the [Teaching guide](/en/latest/teaching) for assignment workflows. # Custom CSS Source: https://docs.expectedparrot.com/en/latest/custom_css CSS classes and HTML structure for styling human-facing surveys with custom CSS. Custom CSS changes the appearance of a human survey, such as text size, spacing, colors, and logo dimensions. Use it to adapt the questionnaire to your study's visual design. For choices such as dropdowns, optional questions, or grouped pages, use the [Humanize schema](/en/latest/humanize_schema). The examples below use CSS selectors to identify parts of the survey: `.edsl-question-text`, for example, selects question wording. The HTML reference shows the structure EDSL generates; you do not need to write that HTML yourself. ## Applying custom CSS ### When creating a human survey The following example gives the survey a light-gray background and makes question text bold and dark gray. Pass the CSS string under `survey.custom_css`; the other schema entries make the rating a dropdown and feedback optional. ```python theme={null} from edsl import Survey, QuestionFreeText, QuestionMultipleChoice survey = Survey([ QuestionMultipleChoice( question_name="rating", question_text="Rate your experience.", question_options=["Good", "OK", "Bad"], ), QuestionFreeText( question_name="feedback", question_text="Any additional feedback you'd like to share?", ), ]) custom_css = """ .edsl-survey-container { background-color: #f9f9f9; } .edsl-question-text { font-weight: bold; color: #333; } """ humanize_schema = { "questions": { "rating": {"format": {"type": "dropdown"}}, "feedback": {"optional": True}, }, "survey": {"custom_css": custom_css}, } survey.humanize(humanize_schema=humanize_schema) ``` Use `survey.preview(humanize_schema=humanize_schema)` to inspect the appearance before publishing, including how long questions and answer options fit on a phone. Pass `"survey": {"custom_css": None}` (or omit `custom_css`) to create the survey without custom styling. ### Updating CSS on an existing human survey Use `Coop.patch_human_survey_css` to set or clear the CSS on a human survey that already exists. ```python theme={null} from edsl import Coop coop = Coop() # Set or replace CSS coop.patch_human_survey_css( human_survey_uuid="your-human-survey-uuid", css=".edsl-survey-container { background-color: #fff; }", ) # Clear existing CSS coop.patch_human_survey_css( human_survey_uuid="your-human-survey-uuid", css=None, ) ``` ## Survey-level classes | Class | Element | Notes | | - | - | - | | `edsl-survey-container` | Outermost container of the entire survey | Use to set background color for the whole survey. Carries `data-group-name` when the survey is presented by question group; see below | | `edsl-survey-banner` | `
` holding the survey's logo, at the top of the survey | Only present when the survey has a logo. Give it a `background` for a banner; see below | | `edsl-survey-banner-content` | Column inside `edsl-survey-banner` holding the logo | Lined up with `edsl-survey-content` | | `edsl-logo` | The logo `` inside `edsl-survey-banner` | 40px tall by default, width following the image. Resize with `height`; see below | | `edsl-survey-content` | Column holding the page's questions and `edsl-survey-nav` | Questions are spaced by its `gap` | | `edsl-survey-item` | `
` wrapping each question or instruction on the page | Carries `data-question-name` or `data-instruction-name`; see below | | `edsl-instruction` | `
` holding an instruction's text | Inside `edsl-survey-item` | | `edsl-survey-nav` | Row `
` holding the page's navigation buttons | Below the questions | | `edsl-next-button` | `

…

``` ### EDSL object ```html theme={null}
Select the agent list to use.
Selected Agent List

My agents - Q1 2025

UUID: a1b2c3d4-…

…

``` ### File upload ```html theme={null}
Upload supporting documents.
…

Upload files

Drag & drop or click to browse

All file types · Max 10 files · Up to 1 GB per file

Files (2)
…

doc1.pdf

24 KB

Uploading… Done Failed
Name Size Upload Status Actions
… doc1.pdf 24 KB
Done Failed

…

``` ### Free text ```html theme={null}
Tell us about yourself.

…

``` ### Linear scale **Radio** When humanize format is `radio` (default): ```html theme={null}
Rate from 1 to 5.

…

``` **Dropdown** When humanize format is `dropdown`: ```html theme={null}
Rate from 1 to 5.

…

``` ### Likert five **Radio** When humanize format is `radio` (default): ```html theme={null}
How satisfied are you?

…

``` **Dropdown** When humanize format is `dropdown`: ```html theme={null}
How satisfied are you?

…

``` ### List ```html theme={null}
List your top hobbies.

Please enter one item per line.

…

``` ### Matrix **Table** When humanize format is `table` (default): ```html theme={null}
Rate each item.
Price
Poor 1
Price

…

``` Both views are always in the page, and CSS picks which one shows. To keep the table at every width: ```css theme={null} .edsl-matrix-stack-view { display: none; } .edsl-matrix-table-view { display: block; } ``` Row labels take 25% of the table's width by default, and the option columns share the rest. To change that, set `--edsl-matrix-item-column` to a percentage, on `.edsl-matrix-table` or on `.edsl-survey-container`: ```css theme={null} .edsl-matrix-table { --edsl-matrix-item-column: 40%; } ``` **Carousel** When humanize format is `carousel`, rows are shown one at a time: ```html theme={null}
Rate each item.
Price
1 of 8

…

``` ### Multiple choice **Radio** When humanize format is `radio` (default): ```html theme={null}
Which fruit do you prefer?

… Please select an option.

``` **Dropdown** When humanize format is `dropdown`: ```html theme={null}
Which fruit do you prefer?

…

``` ### Multiple choice with other ```html theme={null}
How did you hear about us?

…

``` ### Numerical **Input** When humanize format is `input` (default): ```html theme={null}
Enter your age.

…

``` **Slider** When humanize format is `slider`: ```html theme={null}
On a scale of 0 to 100, how likely are you to recommend?
0 50 100

…

``` ### Rank ```html theme={null}
Rank these items from most to least preferred.
Item A
Item B
…
Item C
…

…

``` ### Survey message ```html theme={null}
Thanks! The next few questions are about your work.
``` ### Top K ```html theme={null}
Pick your top 2.

…

``` ### Yes/no **Radio** When humanize format is `radio` (default): ```html theme={null}
Do you agree?

…

``` **Dropdown** When humanize format is `dropdown`: ```html theme={null}
Do you agree?

…

``` ## Comment field If a question has `comment` configured in the humanize schema, an additional comment input is rendered under that question. ```html theme={null}
``` # Custom JavaScript Source: https://docs.expectedparrot.com/en/latest/custom_javascript Run your own JavaScript on a human survey's questions, and record what respondents do. Custom JavaScript lets you attach a script to a question in a human survey. The script runs in the respondent's browser once the question is on the page, so you can build interactive tasks, measure timing and behaviour, and record events with the survey. > [!NOTE] > > Custom JavaScript is available on approved accounts only. Write to [info@expectedparrot.com](mailto:info@expectedparrot.com) to request access. Until your account is approved, a humanize schema that includes a script is rejected when you save it, and no part of that schema is saved. ## Checking access ```python theme={null} from edsl import Coop coop = Coop() coop.get_custom_js_access() # True once your account is approved ``` Access is per account: once yours is approved, the human surveys you create can include custom JavaScript. ## Adding a script Scripts go in the humanize schema, under `questions[question_name].javascript.hooks`. A hook maps an event to the script that runs on it. The only event is `"question.ready"`, which fires once the question has been rendered and its inputs are on the page. This example records milliseconds from the question appearing to the participant's first selection. It illustrates collecting a behavioral measure alongside a multiple-choice answer. It measures the first selection, including time spent reading; it does not measure time to final submission or subsequent answer changes. ```python theme={null} from edsl import Survey, QuestionMultipleChoice survey = Survey([ QuestionMultipleChoice( question_name="choice", question_text="Which option do you prefer?", question_options=["A", "B", "C"], ), ]) time_to_first_answer = """ const container = this.getQuestionContainer(); const shownAt = performance.now(); container.addEventListener("change", () => { ep.log("first_answer", { ms: Math.round(performance.now() - shownAt) }); }, { once: true }); """ humanize_schema = { "questions": { "choice": {"javascript": {"hooks": {"question.ready": time_to_first_answer}}}, }, } survey.humanize(humanize_schema=humanize_schema) ``` To add or change a script on a survey that already exists, use `Coop.patch_human_survey_humanize_schema`: ```python theme={null} coop.patch_human_survey_humanize_schema( "your-human-survey-uuid", {"questions": {"choice": {"javascript": {"hooks": {"question.ready": time_to_first_answer}}}}}, ) ``` Set `"javascript": None` on a question to remove its script. Most question types support `javascript`, including survey messages; see [Humanize schema](/en/latest/humanize_schema) for the list. `compute` and `image_generation` questions do not: they run in the background and never appear on the page. ## Writing a hook A hook is the *body* of a function, not a function. Write the statements you want to run; don't wrap them in `function () { ... }`. Inside a hook you have: | Name | What it is | | - | - | | `this.getQuestionContainer()` | The question's element on the page. Find your question's inputs and add your own elements inside it. | | `context.questionName` | The name of the question the hook is running for. | | `context.questionType` | The question's type, for example `"multiple_choice"`. | | `context.isPreview` | `true` when the survey was opened through a preview link. A preview's answers don't appear in your results, but its `ep.log` events are kept, marked `is_preview`. | | `ep.log(eventName, payload)` | Records an event with the survey. See below. | `context` carries nothing that identifies the respondent. Each question's hook runs once when that question appears. If the same script is attached to several questions, it runs once for each, and `context.questionName` tells them apart. ### Recording events with `ep.log` `ep.log(eventName, payload)` records an event against the respondent's response. Use it for anything you want to analyse later: timings, clicks, which condition a respondent saw. * `eventName` must be a non-empty string. Names longer than 200 characters are cut to 200. * `payload` is optional and must be convertible to JSON: objects, arrays, strings, numbers, booleans and `null`. A payload that isn't (a page element, an object that refers to itself) is not recorded, and a warning appears in the browser console. * A payload larger than 4,000 characters, measured as JSON, is not recorded. * Each response records at most 10,000 events. `ep.log` returns nothing. Events are sent in batches in the background, so don't wait on it. ### What a script may do Scripts can build interface inside the survey, read and write answers, control when respondents can move on, measure behaviour, randomise conditions, read parameters from the page URL, load images, audio, video and other media to show, and use the camera, microphone or location for a task the respondent is told about. A script can also load a well-known, widely used library, such as d3, Chart.js or lodash, from a public CDN like cdnjs, jsDelivr or unpkg, at a pinned version (for example `https://cdn.jsdelivr.net/npm/d3@7.9.0/dist/d3.min.js`, not `d3@latest`). Your own code must still be plain JavaScript written in the script itself. A script can't be saved if it: * Loads code from anywhere else: an unfamiliar or personal site, a paste site, a little-known or custom package, source hosted somewhere other than a named library, or a URL put together while the script runs. * Builds code at runtime, for example by passing a constructed string to `eval`, `new Function`, `setTimeout` or `setInterval`, or by inserting markup that contains a script. * Makes any request to Expected Parrot, or sends respondent data anywhere else. `ep.log` is how data leaves a script. * Reads cookies, tokens, credentials, or storage it didn't create. * Is obfuscated or minified so that what it does can't be read from the source. This applies to your script, not to a well-known library it loads. * Asks for passwords or payment details, or imitates a login or other dialog. * Collects anything unrelated to taking the survey, or uses the camera, microphone or location covertly. * Keeps running after the respondent has finished or left the survey. * Misleads or traps the respondent, for example by opening popups, preventing them from leaving, or hiding the survey's own notices. * Uses the respondent's device for something other than the survey. If a script can't be saved, no part of that humanize schema is saved, and the survey keeps what it had. If you think a script was refused by mistake, write to [info@expectedparrot.com](mailto:info@expectedparrot.com). ### Limits * Each hook can be at most 64,000 characters, and all hooks on one question at most 256,000. * One save can add at most 25 different new scripts. A script used unchanged on several questions counts once, and scripts the survey already has don't count. Save larger surveys in smaller batches. ## Retrieving events Use `Coop.get_all_human_survey_events` to get every event a survey has recorded, oldest first: ```python theme={null} from edsl import Coop coop = Coop() events = coop.get_all_human_survey_events("your-human-survey-uuid") respondent_events = [e for e in events if not e["is_preview"]] ``` Events from preview links are included and marked `is_preview`, so leave them out of an analysis. Events that arrive while the download runs are included, and no event comes back twice. For the timing example above, turn the matching events into a table: ```python theme={null} from edsl import ScenarioList timings = ScenarioList.from_list_of_dicts([ {"response_uuid": event["response_uuid"], "session_uuid": event["log_session_uuid"], "first_answer_ms": event["payload"]["ms"]} for event in respondent_events if event["event_name"] == "first_answer" and event["question_name"] == "choice" ]) timings ``` Match `response_uuid` to `agent.response_uuid` in the [human response data](/en/latest/humanize#retrieving-responses) to examine timing alongside choices. A participant can reopen the survey and create another session, so inspect repeated response identifiers before treating the timing table as one row per participant. To fetch new events later without downloading everything again, pass the `id` of the last event you downloaded as `after`. Take it from the full download, before filtering out previews: ```python theme={null} last_id = events[-1]["id"] if events else None new_events = coop.get_all_human_survey_events( "your-human-survey-uuid", after=last_id ) ``` > [!NOTE] > > Fetching only new events is useful for checking your scripts while a survey is collecting, but it can occasionally miss an event that was still being saved when you downloaded. For analysis, download the full log once fielding has ended, at least an hour after the last response. To fetch one batch at a time, use `Coop.get_human_survey_events`. It takes `after` and `limit` (at most 200, the default), and returns `{"events", "next_cursor", "has_more"}`. Pass `next_cursor` as `after` to get the next batch while `has_more` is `True`. `next_cursor` is returned even when there is nothing more, so you can keep it and use it later. To see how many events a survey has without downloading them, use `Coop.count_human_survey_events`. The count is as of the call; on a survey still collecting, a download started afterwards can return more. Each event includes: | Field | Description | | - | - | | `id` | The event's unique id | | `received_at` | When Expected Parrot received it (ISO 8601, UTC) | | `client_ts` | When the respondent's browser logged it (ISO 8601, UTC) | | `response_uuid` | The response the event belongs to | | `is_preview` | Whether the event came from a preview link | | `log_session_uuid` | The page load the event was logged in. Each time a respondent opens or reloads the survey starts a new session, so one response can have several. | | `session_state` | Whether that session's log is whole. See below. | | `client_seq` | The event's position in its session, counting from 0. Sort by it to get the order events were logged in. A gap means an event in between was not recorded. | | `question_name` | The question whose script logged it | | `event_type` | What kind of event it is. Always `"ep.log"` for now. | | `event_name` | The `eventName` passed to `ep.log` | | `hook_event` | The hook that was running when the event was logged, for example `"question.ready"` | | `source` | What logged the event. Always `"author_script"` for now. | | `payload` | The `payload` passed to `ep.log`, or `None` | `session_state` is one of: * `"complete"`: the session ended normally and no events are missing. * `"truncated"`: some events from this session are known to be missing, for example because they broke the limits above. * `"unknown"`: no events are known to be missing, but the session never signalled that it ended, for example because the respondent's browser closed or lost its connection. Its log may or may not be whole. `session_state` describes the whole session, so every event from one session carries the same value. ## From the command line The `ep` CLI covers the same steps. `--javascript` takes `QUESTION:HOOK=FILE`, naming the question, the hook, and a file holding the script: ```bash theme={null} # Check access ep humanize custom-js-access # Attach a script while building a schema, or to a survey that already exists ep humanize schema create --survey survey.ep --javascript choice:question.ready=choice.js --output humanize.json ep humanize schema set --javascript choice:question.ready=choice.js # Remove a question's script ep humanize schema set --clear-javascript choice # Get the first batch of events, or save every event to a file ep humanize events ep humanize events --all --output events.jsonl # Later, save only the events since the last download ep humanize events --all --after --output new.jsonl # Count events without downloading them ep humanize events --count ``` `--all` requires `--output`, which can end in `.json` or `.jsonl`. Every result includes `next_cursor`, the value to pass as `--after` next time. If a download fails part way, no output file is written. As with the Python methods, use `--after` for checking a survey while it collects, and a full download once fielding has ended for analysis. # Dataset Source: https://docs.expectedparrot.com/en/latest/dataset Work with selected columns as tabular data and export them for analysis. A `Dataset` is a table of columns. A `Results` object retains interview and study information; calling `results.select(...)` produces a Dataset containing the selected columns. Use Dataset methods or pandas to transform and export that table. For example, select a treatment label and a rating from Results, filter to the observations you want, then export a CSV for your statistical software. You can do these steps directly in EDSL; converting with `.to_pandas()` is useful when you want pandas analyses or plotting. A [ScenarioList](/en/latest/scenarios) is the corresponding collection of named records used as inputs to questions. ## A complete local example This example uses four fabricated scores and makes no model calls. It creates a Dataset from a DataFrame, filters and sorts rows, adds a derived column, and writes a CSV. ```python theme={null} import pandas as pd from edsl import Dataset frame = pd.DataFrame({ "condition": ["short", "explicit", "short", "explicit"], "score": [3, 5, 2, 4], }) dataset = Dataset.from_pandas_dataframe(frame) selected = dataset.filter("score >= 4").order_by("score", reverse=True) with_flag = dataset.mutate("high_score = score >= 4") # Display in a notebook: put selected on the last line of a cell. # Display in a script: selected.print() # Convert to pandas when you need DataFrame operations. analysis_frame = selected.to_pandas() selected.to_csv("selected_scores.csv") ``` `analysis_frame` contains the two `explicit` rows with scores 5 and 4, in that order. `with_flag` contains all four rows plus the Boolean `high_score` column. `selected_scores.csv` contains only the selected rows. ## Constructors and conversions Use `Dataset.from_pandas_dataframe(frame)` for a pandas DataFrame. You can also construct a column-oriented table directly: ```python theme={null} from edsl import Dataset dataset = Dataset([{"record_id": ["r1", "r2"]}, {"score": [3, 5]}]) frame = dataset.to_pandas() ``` Keep columns the same length. `to_pandas()` preserves prefixed EDSL column names by default. Use qualified names when joining data so unrelated columns are not confused. ## Common methods | Call | Arguments and result | | - | - | | `dataset.select("condition", "score")` | Return a Dataset with the selected columns. | | `dataset.filter("score >= 4")` | Return rows matching a Boolean expression. | | `dataset.order_by("score", reverse=True)` | Return rows sorted by the named column; descending when `reverse=True`. | | `dataset.mutate("high_score = score >= 4")` | Return a Dataset with a new or updated column computed from the expression. | | `dataset.to_list()` | Return a list of values for one column, or a list of row tuples for multiple columns. | | `dataset.to_pandas(remove_prefix=False)` | Return a pandas DataFrame; set `remove_prefix=True` to strip namespaces such as `answer.`. | | `dataset.to_csv("scores.csv")` | Write a CSV file and return a FileStore containing the CSV. Column prefixes are retained by default. | | `dataset.print()` | Display the table explicitly in a script. A final notebook expression displays it automatically. | This example filters a table and converts a single selected column to a Python list: ```python theme={null} from edsl import Dataset scores = Dataset([{"score": [3, 5, 2, 4]}]) high_scores = scores.filter("score >= 4").order_by("score", reverse=True) high_scores # In a script: high_scores.print() # For further processing: high_scores.select("score").to_list() returns [5, 4]. ``` Conversion is optional when you only want to inspect the table. See [Results](/en/latest/results) for selecting columns from a completed study. ## Reshape and inspect a table | Task | Methods and behavior | | - | - | | Preview rows | `head(n=5)`, `tail(n=5)`, or `table(max_rows=5)` returns a smaller table. `first()` returns the first value in the first column. | | Inspect structure | `keys()` lists column names; `summary()` describes the table. | | Rename or remove columns | `rename({"old": "new"})`, `drop("column")`, and `remove_prefix()` return new Datasets with renamed, removed, or unprefixed columns. Check for collisions before removing prefixes. | | Sample rows | `sample(n=2, with_replacement=False, seed=42)` returns a sample; `shuffle(seed=42)` returns a shuffled table. Dataset sampling defaults to replacement, so choose explicitly. | | Deduplicate | `unique()` returns distinct rows. | | Reshape lists | `expand("tags")` creates rows from list items; `collapse("text", separator="\n")` combines values while grouping by the other columns. | | Reshape all columns | `long()` produces `row`, `key`, and `value` columns; `wide()` reconstructs that layout. | | Join tables | `merge(other, by_x="record_id", by_y="id")` performs a left join. Check for duplicate keys first. | | Format output | `to_markdown_table()` returns a Markdown table string; `latex()` returns a LaTeX table. `print()` explicitly displays the table in scripts. | For example, inspect the long representation of a small table: ```python theme={null} from edsl import Dataset table = Dataset([{"record_id": ["r1", "r2"]}, {"score": [3, 5]}]) long_table = table.long() long_table # long_table.wide() reconstructs the original columns. ``` ## Export and convert tables These methods also apply to Results, ScenarioList, and AgentList. Select the fields you want before exporting: | Output | Call | | - | - | | Row dictionaries | `table.to_dicts(remove_prefix=False)` | | Python values | `table.select("score").to_list()` returns the values of one column. Multiple columns produce row tuples. | | pandas or Polars | `table.to_pandas()` / `table.to_polars()`; prefixes are retained by default. | | CSV | `table.to_csv("scores.csv")` | | Excel | `table.to_excel("scores.xlsx", sheet_name="Scores")` | | Word table | `table.to_docx("scores.docx")` | | EDSL inputs | `table.to_scenario_list()` / `table.to_agent_list()`; these remove column prefixes by default. | The CSV, Excel, and Word export methods above return `FileStore` objects. `table.write("scores.csv")` writes a file directly, choosing its format from the filename extension. `table.to_json()` returns a Python list of column dictionaries. Use `json.dumps()` when you need JSON text: ```python theme={null} import json json.dumps(table.to_json()) ``` This differs from `table.to_dict()`, which returns the EDSL object structure that `Dataset.from_dict()` can reconstruct. Install the corresponding optional library for conversions such as `to_polars()` or Excel output. For database exports, convert to a pandas DataFrame and use its database-writing methods with your chosen connection and replacement policy. ## Embed text Text embeddings represent passages as lists of numbers that can be used to search for similar passages or group related responses. `EmbeddingModel` creates these representations; `Dataset.embed()` and `ScenarioList.embed()` add them as a column beside the source text. Embedding the text does not itself assign themes or create groups. Use [qualitative coding](/en/latest/research/qualitative) when you want named themes from a codebook. This example uses fixed test vectors to show the table structure without a provider call. Their values do not represent semantic similarity: ```python theme={null} from edsl import Dataset, EmbeddingCache, EmbeddingModel texts = Dataset([{"text": ["The tool is easy to use.", "Exporting takes too long."]}]) embedding_model = EmbeddingModel(service_name="test", dimensions=4) embedding_cache = EmbeddingCache() embedded = texts.embed("text", model=embedding_model, cache=embedding_cache) embedded ``` For actual text embeddings, select an embedding model and service explicitly: `EmbeddingModel(model_name="EMBEDDING_MODEL_NAME", service_name="openai")` uses direct provider access by default. Configure the provider's credentials. The `sentence_transformers` service runs locally with its dependencies and model weights installed; loading a model may download weights. `remote=True` routes supported embedding requests through Expected Parrot. Embedding models are separate from the language models selected with `Model()`. For text outside a Dataset, `embedding_model.embed(["first text", "second text"], cache=embedding_cache)` returns an `EmbeddingResult`. Its `embeddings`, `input`, `model`, `service_name`, `usage`, and `cache_used` attributes retain the vectors and available metadata. `to_dataset()` and `to_scenario_list()` convert it to EDSL tables; `to_dict()` and `EmbeddingResult.from_dict(data)` serialize and restore it. In asynchronous Python, use `await embedding_model.async_embed(texts, cache=embedding_cache)`. `EmbeddingCache(filename="embeddings.jsonl")` loads existing embedding entries and writes new ones to that file. With `immediate_write=False`, call `flush()` to persist pending entries. This cache is separate from the response Cache used by survey runs. Embeddings and provider calls can incur costs; the test service above makes no paid calls. # Discovering types and services Source: https://docs.expectedparrot.com/en/latest/enums Inspect supported question types and inference services in the installed version. Discover types from the installed package instead of maintaining a copied list. This is useful for tools that construct EDSL objects or validate generated inputs. ## Question types ```python theme={null} from edsl import Question question_types = Question.list_question_types() example = Question.example("multiple_choice") parameters = example.to_dict() ``` These operations construct local objects and do not call models. To inspect a particular type's schema, use `ep schema show --question_type multiple_choice`. See [Questions](/en/latest/questions#question-type-details) for response formats and construction parameters. ## Inference services ```python theme={null} from edsl import Model services = Model.services() ``` A registered service is not a guarantee that every model is currently available to your account. Use [Language Models](/en/latest/language_models) and [Provider configuration](/en/latest/guides/providers) for current model discovery and access. # FileStore Source: https://docs.expectedparrot.com/en/latest/filestore Represent files as EDSL inputs and distinguish file attachments from tabular rows. `FileStore` represents a file and its contents for use in EDSL or storage on Expected Parrot. A scenario can contain a FileStore attachment, or rows extracted from a file can become separate scenarios. For complete examples of asking questions about PDFs, images, video, and audio, see [Files and media as scenarios](/en/latest/scenarios#data-and-files). ## Rows from a CSV and a whole-file attachment This self-contained example writes a two-row CSV using Python's CSV writer, constructs a FileStore, and loads the rows as scenarios. It makes no model calls or uploads. ```python theme={null} import csv from pathlib import Path from edsl import FileStore, Scenario, ScenarioList, QuestionMultipleChoice path = Path("feedback.csv") with path.open("w", newline="", encoding="utf-8") as handle: writer = csv.DictWriter(handle, fieldnames=["record_id", "feedback"]) writer.writeheader() writer.writerows([ {"record_id": "f1", "feedback": "The setup was easy."}, {"record_id": "f2", "feedback": "The export button is missing."}, ]) file = FileStore(str(path)) # A CSV row becomes one scenario, retaining its record_id. rows = ScenarioList.from_source("csv", str(path)) question = QuestionMultipleChoice( question_name="sentiment", question_text="Classify this feedback: {{ scenario.feedback }}", question_options=["Positive", "Neutral", "Negative"], ) job = question.by(rows) # A whole file can instead be attached under a scenario key for a compatible model. attachment = Scenario({"document": file}) ``` `rows` has two scenarios, each with `record_id` and `feedback`. The `attachment` scenario instead holds the complete file under `document`. Loading a CSV into rows is useful for one classification per record; attaching a whole document is useful when the model needs the document itself. ## Images, PDFs, and other media Use a FileStore value in a scenario and reference its field in the question. For example, `Scenario({"image": FileStore("image.png")})` attaches an image that the question can reference with `{{ scenario.image }}`. A plain path string carries only the filename. The [Scenarios page](/en/latest/scenarios#data-and-files) contains worked examples for: * [Images](/en/latest/scenarios#images), including [several attachments in one question](/en/latest/scenarios#multiple-attachments-in-one-scenario). * [Whole PDFs](/en/latest/scenarios#pdf-attachments), [PDF text by page](/en/latest/scenarios#pdf-text), and [PDF pages rendered as images](/en/latest/scenarios#pdf-page-images). * [Video](/en/latest/scenarios#video) and [audio](/en/latest/scenarios#audio). * [A directory of files](/en/latest/scenarios#a-collection-of-files), with one scenario per file. ## Storage and sharing Creating a FileStore reads local data. Calling `push()` uploads it to Expected Parrot and requires authentication. Set visibility explicitly and retain the returned identifier. See [Saving and sharing objects](/en/latest/guides/saving_objects) for posting and retrieving research materials. For example, save a PDF on Expected Parrot and retrieve it as a scenario attachment: ```python theme={null} from edsl import FileStore report = FileStore("report.pdf") info = report.push(description="Study report", visibility="private") retrieved_report = FileStore.pull(info["uuid"]) document = retrieved_report.to_scenario(key_name="document") ``` Reference `{{ scenario.document }}` in your question, then pass `document` to `question.by(document)`. You can also pass a shared file URL to `FileStore.pull()`. Access follows the object's sharing permissions. To extract PDF text after retrieval, use `ScenarioList.from_source("pdf", retrieved_report.to_tempfile())` after importing `ScenarioList`. ## Common methods | Call | Arguments and result | | - | - | | `FileStore("document.pdf")` | Read a local file and create an object containing its data. | | `file.to_scenario(key_name="document")` | Return a Scenario with the file under the named key. | | `file.to_scenario_list()` | Parse supported file formats into rows; available conversions depend on the file type. | | `file.write("copy.pdf")` | Write the stored content to disk and return the filename. | | `file.push(description="...", visibility="private")` | Upload the file object and return remote metadata. Requires an account. | | `FileStore.pull(url_or_uuid)` | Retrieve a stored file object from Expected Parrot. | Using the CSV created in the example above: ```python theme={null} from edsl import FileStore file = FileStore("feedback.csv") attachment = file.to_scenario(key_name="document") rows = file.to_scenario_list() rows # In a script: rows.print() ``` `attachment` contains one complete file, while `rows` contains the CSV records. Choose the input that matches what each respondent should evaluate. ## Read, preview, and convert files | Task | Method and result | | - | - | | Download a file | `FileStore.from_url(url, download_path=absolute_path)` downloads content to the supplied path and returns a FileStore. | | Extract text | `file.extract_text()` returns extracted text when the file handler supports it. Inspect the text for extraction errors. | | Read through a file object | `file.open()` returns a Python file-like object; close it when finished. | | Create a temporary copy | `file.to_tempfile()` writes the stored content to a temporary file. | | Preview | `file.view()` displays supported content. `file.download_link()` creates a notebook download link. | | Read a table | `file.to_pandas()` uses the file's supported tabular reader. | | Convert Markdown | `file.to_docx(output_path="report.docx")` or `file.to_pdf(output_path="report.pdf")` converts a Markdown FileStore. Conversion dependencies must be installed. | | Inspect media | `file.is_image()`, `is_video()`, `get_image_dimensions()`, and `get_video_metadata()` identify supported media and inspect dimensions or video metadata. | `FileStore.from_file_upload_answer(file_info)` retrieves one uploaded file from a human survey answer; see [file upload responses](/en/latest/humanize#working-with-file-upload-responses). ## Generate image inputs `ImageGeneration` produces FileStore objects that can be attached to scenarios. This example uses a local test service that returns a tiny placeholder image, without a provider request: ```python theme={null} from edsl import ImageGeneration generator = ImageGeneration(model="test", service_name="test") image = generator.generate("A product concept illustration") image.write("test-image.png") ``` For actual generation, choose a model supported by the image service, for example `ImageGeneration(model="MODEL_NAME", service_name="google")`, with the provider credential configured. Image generation can incur costs. Pass `input_images=[file]` when the chosen service supports editing or image-conditioned generation. Provider-specific parameters can be supplied at construction or to `generate()`. In a notebook with an active event loop or an asynchronous function, use `await generator.async_generate(prompt)` instead of `generate()`. A single generated image returns one FileStore; multiple images return a list. With `return_type="generated_image"`, the return value instead contains `GeneratedImage` metadata. Use its `to_filestore()` to access the file, `token_usage()` for usage information, and `cost()` for the model-price-based cost calculation; missing usage can be estimated rather than measured. # Background jobs Source: https://docs.expectedparrot.com/en/latest/guides/background_jobs Submit remote jobs, find runs, and retrieve results with Python or the CLI. A background job starts a remote study and returns control to your notebook or script while Expected Parrot runs the interviews. Use it for a large set of documents, repeated simulations, or a comparison across several models when you want to continue other work and collect the results later. The workflow has three steps: submit the job, keep its identifier, and retrieve the completed Results. The examples use Python's `Coop` client; CLI equivalents follow at the end. Remote execution requires [Expected Parrot access](/en/latest/api_keys). ## Submit from Python Start with a Jobs object containing your survey, agents, scenarios, and models. You can construct it with `survey.by(...)` or load a saved package: ```python theme={null} from edsl import Jobs job = Jobs.load("jobs.ep") pending = job.run( background=True, offload_execution=True, remote_inference_description="Customer feedback pilot", ) job_uuid = pending.job_info.job_uuid job_uuid # In a script: print(job_uuid) ``` This returns a Results object whose data is pending. Keep `job_uuid` so you can retrieve the run from another session. The description helps you find it later. Review the inputs and [estimated cost](/en/latest/costs) before submitting. ## Check status and retrieve results ```python theme={null} from edsl import Coop coop = Coop() status = coop.remote_inference_get(job_uuid=job_uuid) status # In a new session, first set job_uuid to the saved job identifier. # In a script: print(status) ``` Inspect `status["status"]` and `status["latest_job_run_details"]`. The job UUID identifies the execution; `status["results_uuid"]` identifies the stored Results object and may be absent while the job is pending. After the status is `completed` and a results UUID is available, retrieve and save the Results object: ```python theme={null} results = coop.pull(status["results_uuid"], expected_object_type="results") results.save("results.ep") results.select("answer.*") # Alternative: Results.pull(status["results_uuid"]) after importing Results. # In a script, append .print() to display the selected answers. ``` If you still have `pending` in the original session, `pending.fetch()` can also load the completed results into that object. Calling it before completion waits for the run. Use the status query above to check failed, cancelled, or partially failed runs explicitly. For a failed or partially failed run, retrieve the error report with `coop.get_error_report_markdown(job_uuid)`. Compare completion counts with the planned study before analyzing any available results. ## Find jobs with filters and pagination `remote_inference_list()` returns a `CoopJobsObjects` collection containing job metadata. For example, find completed runs whose description contains your study label: ```python theme={null} completed_jobs = coop.remote_inference_list( status="completed", search_query="Customer feedback", page=1, page_size=20, sort_ascending=False, ) completed_jobs.select("uuid", "description", "status", "results_uuid") # In a script, append .print() to display the selected metadata. ``` | Argument | Meaning | | - | - | | `status` | One status or a list, such as `["queued", "running"]`. Also accepts `completed`, `failed`, `cancelled`, `cancelling`, and `partial_failed`. | | `search_query` | Search the job description. | | `page` | Page number, starting at **1**. Increase it while keeping the same filters to retrieve subsequent pages. | | `page_size` | Jobs per page, from 1 to 100. | | `sort_ascending` | `False` returns the most recently created jobs first. | Listing metadata does not download the study or its answers. Fetch the objects represented by the current page when you need them: ```python theme={null} result_sets = completed_jobs.fetch_results() # A Python list of Results objects. job_specs = completed_jobs.fetch() # A Python list of Jobs objects. ``` `fetch_results()` skips entries without a results UUID. Both methods operate on the current collection, so request additional pages explicitly. `completed_jobs.filter(...)` can narrow metadata already downloaded, but does not search other pages; use the arguments to `remote_inference_list()` for server-side filtering. For saved surveys, agents, scenarios, and Results objects, use [`Coop.list()`](/en/latest/coop#find-objects-with-filters-and-pagination). That searches stored objects; `remote_inference_list()` searches execution records. ## Read interview data by page To inspect a bounded part of a run without downloading its entire stored Results object, use the result manifest and page methods. With a completed `job_uuid`: ```python theme={null} manifest = coop.remote_inference_results_manifest(job_uuid, page_size=100) manifest ``` The manifest includes `status`, `total_interviews`, `page_size`, and `page_count`. If `page_count` is greater than zero, retrieve a page: ```python theme={null} page_data = coop.remote_inference_results_page( job_uuid, page=0, page_size=manifest["page_size"], ) interviews = page_data["interviews"] interviews # In a script: print(interviews) ``` Interview pages start at **0**, unlike job and object listings. Use page numbers from `0` through `manifest["page_count"] - 1`, keeping the same page size. These methods return dictionaries of raw interview data, including `agent`, `scenario`, `model`, and `answers`; they do not return a Results object. Use `coop.pull()` when you want Results methods such as `select()` and `filter()`. ## Cancel a job To stop an execution you no longer want, call `coop.cancel_remote_inference_job(job_uuid)`. Check its status again with `remote_inference_get()`; a cancellation request may take time to finish. Cancelling a job does not undo costs already incurred or delete saved Results objects. The CLI alternative is `ep jobs cancel JOB_UUID --yes` when cancellation is intended. ## Use the CLI Save and validate a Jobs package before submitting: ```bash theme={null} ep validate --file jobs.ep ep jobs cost jobs.ep ep run jobs.ep --background ``` Read `data.meta.remote_job.job_uuid` from the JSON response. Replace `JOB_UUID` below with that value: ```bash theme={null} ep jobs status JOB_UUID ep jobs results JOB_UUID --output results.ep ep results columns --file results.ep ``` Retrieve results after completion. For failures, use `ep jobs errors JOB_UUID --output error.md`. A submission acknowledgment is not a completed dataset. Inspect the finished observation count and any partial results before analysis. To wait and save in one command, use `ep run jobs.ep --background --wait --output results.ep`. `--background --output` without `--wait` is invalid because there are no completed results to save yet. See [CLI Guide](/en/latest/cli#submit-a-background-job), [Jobs](/en/latest/jobs), and [Remote Inference](/en/latest/remote_inference). # Label data with survey questions Source: https://docs.expectedparrot.com/en/latest/guides/data_labeling Generate labels from an existing codebook, or develop the labels through exploratory questions first. In EDSL, **the answers to your survey questions become the labels for your data**. Write questions about the characteristics you want to identify, choose their answer formats, and ask a model to answer them about each item. For example, questions about topic, sentiment, and suggested improvements can produce three labels for each open-ended survey response. The questions form a **Survey**, and the items being labeled become **Scenarios**. For one model and iteration, each item produces one result row, with an answer column for each question. The same approach works with text passages, documents, images, and other [scenario inputs](/en/latest/scenarios#data-and-files). There are two starting points: * **You already have a codebook:** turn its categories and definitions into structured questions, then apply them to your data. * **You want to develop the labels from the data:** use exploratory questions first, organize their answers into a codebook, then use that codebook to construct structured labeling questions. ## Start with an existing codebook A **codebook** names your labels and explains when each applies. For a multiple-choice question, its category names become `question_options`, and its definitions become part of `question_text`. A checkbox question can use the same approach when several labels may apply to one item. This example labels four service requests as Billing, Access, Feature request, or Other. It also asks for a supporting quotation. The codebook distinguishes a payment problem from an access problem, including when the inaccessible page concerns billing. The sample requests have no preassigned labels: ```python theme={null} from edsl import QuestionMultipleChoice, QuestionFreeText, Scenario, ScenarioList, Survey # Sample inputs: no existing labels are needed to run the survey. records = ScenarioList([ Scenario({"record_id": "t1", "text": "I was charged twice."}), Scenario({"record_id": "t2", "text": "I cannot reset my password."}), Scenario({"record_id": "t3", "text": "Please add a dark theme."}), Scenario({"record_id": "t4", "text": "The invoice page won't open."}), ]) # Translate the codebook into allowed answers and instructions for using them. codebook = { "Billing": "A charge or payment issue.", "Access": "Inability to enter or use a page, including a billing page.", "Feature request": "A request for new functionality.", "Other": "A request that does not fit any of the categories above.", } definitions = "\n".join(f"{label}: {meaning}" for label, meaning in codebook.items()) category = QuestionMultipleChoice( question_name="category", question_text="What kind of request is this: {{ scenario.text }}?\n" + definitions, question_options=list(codebook), ) evidence = QuestionFreeText( question_name="evidence", question_text="Quote the words supporting your code: {{ scenario.text }}", ) survey = Survey([category, evidence]).add_targeted_memory(evidence, category) job = survey.by(records) ``` `QuestionMultipleChoice` produces the category label in `answer.category`; `QuestionFreeText` produces supporting text in `answer.evidence`. `add_targeted_memory(evidence, category)` supplies the category question and its answer to the evidence question, so it knows which choice to support. Add more questions to label other characteristics of the same request. ### Generate and inspect the labels With [model access configured](/en/latest/api_keys), run the job and view each classification beside its source and evidence: ```python theme={null} results = job.run() results.select("scenario.record_id", "scenario.text", "answer.category", "answer.evidence") ``` This produces four rows, one per request. For example, an illustrative row would be: | record\_id | text | answer.category | answer.evidence | | - | - | - | - | | t1 | I was charged twice. | Billing | charged twice | Actual answers come from your chosen model. Read the labels and quotations alongside the source, especially where category boundaries matter. To save the labeled data: ```python theme={null} results.select( "scenario.record_id", "scenario.text", "answer.category", "answer.evidence" ).to_csv("labeled_requests.csv") ``` ## Develop labels through exploratory questions When you do not yet know which categories will be useful, start by asking open-ended questions about your material. Their answers help you decide what the later structured questions should ask and which choices they should offer. The code below reuses `records` from the example above; it does not use the existing codebook. ### 1. Explore the material This first survey proposes issue categories and describes what each person needs. For a larger dataset, begin with a selection of records covering the range of material you want to label: ```python theme={null} from edsl import QuestionList candidate_labels = QuestionList( question_name="candidate_labels", question_text=( "What issue categories would help organize this service request: " "{{ scenario.text }}? Propose short category names grounded in the text." ), max_list_items=3, ) need = QuestionFreeText( question_name="need", question_text="What does this person need help with: {{ scenario.text }}?", ) exploration = Survey([candidate_labels, need]).by(records).run() notes = exploration.select( "scenario.record_id", "scenario.text", "answer.candidate_labels", "answer.need" ) notes ``` Each record now has proposed categories and a description of the need. Different records may produce overlapping names such as “Payment problem” and “Billing.” These suggestions help develop a common set of labels for the next stage. ### 2. Consolidate the suggestions into a codebook You can choose and define the labels yourself, making this a two-stage workflow: explore, then label. Or add a question that considers the exploratory answers together. Here `QuestionExtract` asks for a draft with two named fields: a list of labels and their definitions, including rules for ambiguous cases: ```python theme={null} from edsl import QuestionExtract propose_codebook = QuestionExtract( question_name="codebook", question_text=( "Develop a codebook for classifying service requests by their main issue. " "Use these source records and exploratory answers: {{ scenario.notes }}. " "Merge overlapping suggestions into 2 to 4 categories, including Other. " "Give each label a definition and explain how to choose between overlapping " "categories. Other should cover requests outside the named categories." ), answer_template={ "labels": ["Category name", "Other"], "definitions": "Definitions and boundary rules for all labels.", }, ) proposal = propose_codebook.by(Scenario({"notes": notes.to_list()})).run() draft_codebook = proposal.select("answer.codebook").to_list()[0] draft_codebook ``` Here the answers from one survey become the input to another question. Review the draft against the source material and your research question. You can edit `draft_codebook["labels"]` and `draft_codebook["definitions"]`, ask further questions to refine them, or explore additional records. For large collections, consolidate suggestions in batches before combining them into one codebook. ### 3. Use the chosen labels in structured questions The next question takes its answer options directly from the codebook you developed. The definitions explain how to choose among those options. Apply this shared question to all records, including any used during exploration: ```python theme={null} category_from_draft = QuestionMultipleChoice( question_name="category", question_text=( "What kind of request is this: {{ scenario.text }}?\n" + draft_codebook["definitions"] ), question_options=draft_codebook["labels"], ) labeling_survey = Survey([category_from_draft, evidence]).add_targeted_memory( evidence, category_from_draft ) labeled_results = labeling_survey.by(records).run() labeled_results.select( "scenario.record_id", "scenario.text", "answer.category", "answer.evidence" ) ``` The exploratory questions helped determine the labeling scheme; this final survey generates labels using that scheme consistently across the dataset. You can use `QuestionCheckBox` instead when several discovered themes may apply, or construct several questions for different characteristics. For example, exploration might reveal that topic and urgency should be labeled separately. ## Choose the right labeling question The response type determines what kind of labeling you can do: | Labeling task | EDSL approach | | - | - | | Exactly one category per record | `QuestionMultipleChoice`, as above. Include an Other option if your categories do not cover every case. | | Several categories can apply | `QuestionCheckBox`; see [Qualitative coding](/en/latest/research/qualitative) for themes and supporting quotations. | | Extract named fields such as dates or organizations | `QuestionExtract` with an answer template; see [Question types](/en/latest/questions). | | Rate an attribute such as intensity or relevance | `QuestionLinearScale` with defined scale values. | For a larger dataset, [import records from a file or DataFrame](/en/latest/guides/importing_data) and reuse the same survey. [Qualitative coding](/en/latest/research/qualitative) develops the example of assigning overlapping themes to interview excerpts. ## Evaluate the labels Generating labels does not require already-labeled examples. When you have independent reference labels, you can compare them with the answers to assess how well the questions and model perform your task. [Human validation](/en/latest/research/human_validation) shows how to collect or import those labels, match them by record identifier, and calculate agreement. To test whether the labeling depends on the model, attach a ModelList and [compare answers on the same records](/en/latest/guides/model_comparisons). To refine the instructions themselves, use the [Prompt sensitivity](/en/latest/research/prompt_sensitivity) pattern. If you revise the codebook after reviewing disagreements, evaluate it on additional independently labeled records. # Experimental conditions Source: https://docs.expectedparrot.com/en/latest/guides/experimental_conditions Compare messages across respondent profiles and understand which combinations EDSL runs. An experimental condition is a version of the material or situation you want to compare: a different message, price, policy, or product description. In EDSL, scenarios hold those versions, while a shared question defines how each version will be evaluated. Keeping the question and response options the same makes the answers easier to compare. This page builds a small message test: **does an instruction that names the file format receive different clarity ratings from a shorter instruction?** Four agent profiles evaluate both messages. You will learn how to combine scenarios and agents, run all their combinations, and keep the condition and profile information alongside each answer. ## Define what varies The example represents two kinds of variation separately. The `messages` ScenarioList supplies what is being evaluated: a short message and a more explicit message. The AgentList supplies the perspective used to evaluate it: new or experienced users, each acting as an individual contributor or team lead. ```python theme={null} from edsl import Agent, AgentList, QuestionMultipleChoice, Scenario, ScenarioList agents = AgentList([ Agent(traits={"experience": experience, "role": role}, instruction="Answer using only the supplied profile and message.") for experience in ["new user", "experienced user"] for role in ["individual contributor", "team lead"] ]) messages = ScenarioList([ Scenario({"condition": "short", "message": "Export your report."}), Scenario({"condition": "explicit", "message": "Download your report as a CSV file."}), ]) question = QuestionMultipleChoice( question_name="clarity", question_text="How clear is this instruction: {{ scenario.message }}?", question_options=["Clear", "Unclear", "Unsure"], ) job = question.by(agents).by(messages) # Four profiles crossed with two messages gives eight interviews per model. ``` ## Execute the design Chaining `.by(agents).by(messages)` gives every profile both messages, in separate interviews. Four profiles times two messages produces eight interviews per model and iteration, each answering the same `clarity` question. This is a **crossed design**: EDSL runs every combination of the supplied lists. With [model access configured](/en/latest/api_keys), the following code uses the default model. You can replace `Model()` with a [model you select](/en/latest/language_models). First view the answers together with the variables that produced them: ```python theme={null} from edsl import Model results = job.by(Model()).run() table = results.select("scenario.condition", "agent.experience", "agent.role", "answer.clarity") table ``` The resulting table has eight rows. `scenario.condition` identifies the message, `agent.experience` and `agent.role` identify the profile, and `answer.clarity` contains its rating. To count the ratings for each message in pandas: ```python theme={null} frame = table.to_pandas() counts = frame.groupby(["scenario.condition", "answer.clarity"]).size() counts ``` These counts summarize the four configured profiles for each message. Keep the profile columns when you want to see whether a difference is concentrated among new users or team leads, rather than looking only at the totals. The responses describe these model-generated evaluations; the profiles are not recruited human participants. ## Choose a different assignment pattern Crossing is useful when every profile should evaluate every condition. Other designs call for a different relationship between the inputs: | What you want to compare | EDSL approach | | - | - | | Conditions without specified respondent profiles | Use `question.by(messages)` and let the selected model answer without the custom agents. | | One scenario for each corresponding agent | Use `job.zip_assign(over=("agents", "scenarios"))` with equally sized lists; see [Job assignments](/en/latest/jobs). | | Only particular agent–scenario combinations | Use `job.assign(...)` for explicit assignments or `job.include_when(...)` for a condition on their fields. | | Several message versions within one interview | Use `question.loop(messages)` to create a separate question for each message, then put those questions in a Survey. Add memory if later answers should see earlier ones. | | Human participants assigned to scenarios | Publish with `survey.by(messages).humanize(...)` and choose a scenario assignment method; see [Human studies](/en/latest/guides/human_studies). | For additional factors such as price and message wording, construct scenarios containing each combination you want to test. To compare the same design across models, attach a ModelList as shown in [Comparing models](/en/latest/guides/model_comparisons). Each added profile, condition, or model increases the number of interviews, which you can inspect and [estimate costs for](/en/latest/costs) before running. # AI-led interviews with people Source: https://docs.expectedparrot.com/en/latest/guides/human_interviews Personalize AI interview guides and combine conversations with structured survey questions. An AI-led interview uses a guide to ask a human participant questions and relevant follow-ups. The participant supplies the responses; the AI acts as interviewer. In EDSL, this is a `QuestionInterview`: a question type that contains a conversation and returns its transcript. Use it when you want participants to describe experiences in their own words and clarify details as the conversation develops. An interview can be the whole survey or sit alongside [multiple-choice questions, ratings, and other question types](/en/latest/questions). This lets you collect comparable measures and explore the experiences behind them in the same study. ## Combine an interview with structured questions This survey first asks how often the participant exports reports, then explores their most recent experience in an interview, and finishes with an ease rating. The interview guide uses the first answer to put the conversation in context: ```python theme={null} from edsl import QuestionInterview, QuestionLinearScale, QuestionMultipleChoice, Survey frequency = QuestionMultipleChoice( question_name="export_frequency", question_text="How often do you export reports?", question_options=["Daily", "Weekly", "Less often"], ) interview = QuestionInterview( question_name="export_experience", question_text="Describe your experience exporting a report.", interview_guide=( # Pipe the participant's earlier answer into the interviewer's guide. "The participant exports reports {{ export_frequency.answer }}. " "Ask about the participant's most recent attempt to export a report. " "Explore how that experience fits their usual reporting routine. " "Ask what they tried, where they encountered difficulty, and what helped. " "Use neutral follow-ups to clarify details; do not suggest problems." ), max_turns=8, ) ease = QuestionLinearScale( question_name="export_ease", question_text="Overall, how easy was your most recent attempt to export a report?", question_options=[1, 2, 3, 4, 5], option_labels={1: "Very difficult", 5: "Very easy"}, ) # The interview is one question in the same survey as the structured questions. survey = Survey([frequency, interview, ease]) humanize_schema = { "questions": {"export_experience": {"interview_mode": "text"}} } ``` The interview occupies one position in the survey, but can contain multiple conversational turns. `question_text` introduces the topic and `interview_guide` directs the interviewer. `max_turns=8` limits interviewer turns; it does not define eight fixed questions. After the conversation, the participant proceeds to the rating question. The schema selects text participation. See [Humanize schema](/en/latest/humanize_schema) for voice and other interview options. ## Personalize the interview guide Piping works inside `interview_guide` as well as `question_text`. In this example, `{{ export_frequency.answer }}` inserts the participant's answer to the earlier question. Someone who selects “Weekly” gives the interviewer the context “The participant exports reports Weekly.” The guide then directs the interviewer to explore how the experience fits that person's routine. You can use the same approach with an earlier free-text answer, such as a task the participant wants to discuss. Place questions whose answers you pipe before the interview. [Survey logic and piping](/en/latest/guides/survey_logic) explains how to combine this with branching, for example to ask an interview question only of participants with relevant experience. Guides can also use information supplied by your study through [Scenarios](/en/latest/scenarios). For example, `"Ask about exporting reports in {{ scenario.product }}."` makes the product a variable in the guide. Attach the scenarios with `survey.by(scenarios)` and choose how participants receive them when publishing; see [Human surveys with scenarios](/en/latest/humanize#with-scenarios). This lets you reuse a guide across products, settings, or experimental conditions. ## Preview and publish The following calls use Expected Parrot and require authentication. AI interviewing can incur usage costs. First obtain a preview and pilot the conversation with people: ```python theme={null} preview_url = survey.preview(humanize_schema=humanize_schema) ``` Check that the guide uses the earlier answer, follow-ups stay on topic, and participants reach the final rating. When the instrument is ready, create the hosted survey: ```python theme={null} info = survey.humanize( human_survey_name="Report export interviews", humanize_schema=humanize_schema, ) respondent_url = info["respondent_url"] human_survey_uuid = info["uuid"] ``` Preserve the UUID and distribute the respondent URL through the planned recruitment workflow. See [Human surveys](/en/latest/humanize#retrieving-responses) for response retrieval and [Prolific](/en/latest/prolific) for recruitment. ## Analyze the responses After participants have responded, retrieve the structured answers and interview transcript together: ```python theme={null} from edsl import Coop results = Coop().get_human_survey_responses(human_survey_uuid) results.select( "agent.response_uuid", "answer.export_frequency", "answer.export_experience", "answer.export_ease", ) ``` Each participant's transcript sits alongside their frequency and rating answers, so you can compare themes across frequent and occasional users or relate interview accounts to ease ratings. The interview answer contains messages with interviewer and respondent roles. Keep those roles when extracting passages so interviewer questions are not counted as participant statements. The [Qualitative coding example](/en/latest/research/qualitative) shows how to analyze excerpts with themes and supporting quotations; use the response identifier to connect excerpts back to their interview. For help developing the interview guide from a brief or source material, use the [Research Agent](/en/latest/research_agent), then inspect and refine the EDSL study it produces. # Human studies Source: https://docs.expectedparrot.com/en/latest/guides/human_studies Publish a survey, collect responses, and retrieve data for analysis. Use EDSL to collect responses from people by turning a Survey into a hosted web questionnaire. Participants answer through a link; you retrieve their responses as data in Python. You can conduct a human study on its own or use the same instrument to compare human and model responses. This example asks about an export feature. Participants who have not used it finish after the first question; those who have used it describe a problem and what would resolve it. The code first defines that questionnaire: ```python theme={null} from edsl import QuestionYesNo, QuestionFreeText, Survey used = QuestionYesNo(question_name="used", question_text="Have you used the export feature?") problem = QuestionFreeText( question_name="problem", question_text="What problem, if any, did you encounter when exporting?", ) followup = QuestionFreeText( question_name="followup", question_text="What would resolve this problem: {{ problem.answer }}?", ) survey = ( Survey([used, problem, followup]) .add_stop_rule(used, "{{ used.answer }} == 'No'") .add_targeted_memory(followup, problem) ) ``` ## Publish and collect With [Expected Parrot login configured](/en/latest/api_keys#log-in-from-a-notebook), publish it and retain the two identifiers used in the collection workflow: ```python theme={null} info = survey.humanize(human_survey_name="Export feature feedback") respondent_url = info["respondent_url"] human_survey_uuid = info["uuid"] ``` `respondent_url` is the link participants open. `human_survey_uuid` identifies this hosted collection when you retrieve responses later. Calling `humanize()` creates the survey; it does not recruit participants or generate their answers. Use `survey.preview()` to inspect the questionnaire before sharing its respondent link. See [Humanize](/en/latest/humanize) for presentation settings and [Prolific](/en/latest/prolific) for recruitment. ## Retrieve responses After responses have been collected: ```python theme={null} from edsl import Coop, Results human_results = Coop().get_human_survey_responses(human_survey_uuid) if not isinstance(human_results, Results): raise TypeError("Inspect returned response data before applying Results methods.") table = human_results.select("answer.*") table ``` Each completed response is a row, with columns such as `answer.used`, `answer.problem`, and `answer.followup`. Participants routed past a question have no answer to it. Use `table.to_csv("human-responses.csv")` to export the selected answers, or `table.to_pandas()` for analysis in a notebook. The type check handles the service's fallback to raw response data when conversion to Results is unavailable. To show different material to participants, attach scenarios with `survey.by(scenarios).humanize(...)` and choose an [assignment method](/en/latest/humanize#scenario-list-methods). To invite a known panel, attach [respondent profiles and email addresses](/en/latest/humanize#with-agent-lists-for-email-notifications). The [Human validation example](/en/latest/research/human_validation) demonstrates matching human classifications with model predictions. For adaptive interviews with human participants, use [Human interviews](/en/latest/guides/human_interviews). # Importing data Source: https://docs.expectedparrot.com/en/latest/guides/importing_data Import tables, documents, and web content into scenarios and respondent profiles. Importing data turns material you already have—such as spreadsheet rows, interview excerpts, or respondent profiles—into objects you can use in an EDSL survey. You do not need to copy each record into a separate question. Instead, write one question with a placeholder and let EDSL fill it with a different record for each interview. The first choice is what the imported data represents. Use **scenarios** for material a model will read or evaluate, such as a customer message. Use **agents** for the traits of the respondent the model should portray, such as their experience with a product. A file containing the questions themselves is a third case, covered under [Existing questionnaires](#existing-questionnaires). ## Import messages and respondent profiles The example creates two small CSV files so you can try the imports without downloading a dataset. `messages.csv` has one message per row; `profiles.csv` has one respondent profile per row. The last lines read them into a ScenarioList and an AgentList. If you already have CSV files, start with those import calls and substitute your filenames. ```python theme={null} import csv from pathlib import Path from edsl import AgentList, ScenarioList def write_fixture(path, rows): with Path(path).open("w", newline="", encoding="utf-8") as handle: writer = csv.DictWriter(handle, fieldnames=list(rows[0])) writer.writeheader() writer.writerows(rows) write_fixture("messages.csv", [ {"record_id": "m1", "message": "Your report is ready to download."}, {"record_id": "m2", "message": "Download your report with the Export button."}, ]) write_fixture("profiles.csv", [ {"profile_id": "p1", "experience": "new user"}, {"profile_id": "p2", "experience": "frequent user"}, ]) records = ScenarioList.from_source("csv", "messages.csv") agents = AgentList.from_source("csv", "profiles.csv", name_field="profile_id") assert len(records) == 2 assert len(agents) == 2 ``` Each row of `messages.csv` becomes one Scenario in `records`. Its column names become fields: `record_id` identifies the original row, and `message` supplies text to a question. Each row of `profiles.csv` becomes one Agent; `name_field="profile_id"` uses the identifier as the agent's name, while the experience field describes the profile. Importing profiles prepares agents for model responses; it does not collect answers from the people described in the file. ## Use an imported column in a question The following question evaluates the messages from the CSV. The placeholder `{{ scenario.message }}` reads the `message` field from each scenario. Attaching both lists asks each profile to evaluate each message: ```python theme={null} from edsl import QuestionMultipleChoice question = QuestionMultipleChoice( question_name="clarity", question_text="How clear is this message: {{ scenario.message }}?", question_options=["Clear", "Unclear", "Unsure"], ) job = question.by(records).by(agents) # With model access configured, collect and display the ratings. results = job.run() results.select("scenario.record_id", "agent.agent_name", "answer.clarity") ``` Two messages times two profiles gives four result rows for one model and iteration. The identifiers let you match each rating back to its message and profile. Calling `by()` crosses the lists; it does not pair their rows by position. If each profile should receive only its corresponding record, use `zip_assign()` as described in [Experimental conditions](/en/latest/guides/experimental_conditions). If the task needs no custom profiles, use `question.by(records)` alone. ## Other Python input formats The same approach works when your data comes from a spreadsheet, database, or Python object. `ScenarioList.from_source()` reads the source into scenarios so the question and job code can stay the same. For tabular data, each row becomes a scenario: | Source | Python call | | - | - | | CSV or TSV | `ScenarioList.from_source("csv", "records.csv")` or `from_source("tsv", "records.tsv")` | | Excel sheet | `ScenarioList.from_source("excel", "records.xlsx", sheet_name="Responses")` | | pandas DataFrame | `ScenarioList.from_source("pandas", frame)` | | Dictionary of columns | `ScenarioList.from_source("dict", {"record_id": ["r1", "r2"], "text": ["A", "B"]})` | | List of row dictionaries | `ScenarioList.from_list_of_dicts([{"record_id": "r1", "text": "A"}])` | | SQLite table | `ScenarioList.from_source("sqlite", "records.db", table="responses")` | | Parquet or Stata file | `ScenarioList.from_source("parquet", "records.parquet")` or `from_source("dta", "records.dta")` | `from_source()` normalizes column names to snake case by default. Pass `snakify=False` to preserve the source names. Some readers require optional format-specific dependencies. `ScenarioList.from_jsonl("scenarios.jsonl")` restores EDSL's serialized JSONL representation; it is not a general JSON row reader. For arbitrary JSON records, read them with Python's `json` module and use `from_list_of_dicts()`. `AgentList.from_source()` supports the same source selection and accepts `instructions`, `codebook`, and `name_field` for profile imports. Instructions tell agents how to respond; a codebook explains what their trait fields mean. `AgentList.from_csv()` is a direct CSV alternative. To turn rows you have already prepared into profiles, use `AgentList.from_scenario_list(records)` or `records.to_agent_list()`. Choose the import method based on where your data already lives. For example, use the pandas source if you have cleaned a DataFrame in a notebook; there is no need to save it to CSV and read it back. See [Scenarios](/en/latest/scenarios) for selecting, renaming, filtering, and transforming fields after import. ## Documents and web content For documents, decide what one interview should receive: the text of a whole document, one page at a time, or the original file with its visual content. This choice determines both the scenario you construct and what the model can evaluate. Extracted text is useful for analyzing wording; the original PDF or page images are useful when charts, layout, or scanned content matter. For documents, `Scenario.from_pdf("report.pdf")` and `Scenario.from_docx("report.docx")` extract text into one Scenario. `ScenarioList.from_source("pdf", "report.pdf")` extracts one scenario per PDF page. The [PDF text examples](/en/latest/scenarios#pdf-text) show the resulting fields, page filtering, and how to run a question over the pages. To send the actual file to a model, use `Scenario.from_file(path, field_name="document")` or put a FileStore in a Scenario. See the worked examples for [whole PDFs](/en/latest/scenarios#pdf-attachments), [images](/en/latest/scenarios#images), [video](/en/latest/scenarios#video), and [audio](/en/latest/scenarios#audio). For scanned pages and visual analysis, [render PDF pages as images](/en/latest/scenarios#pdf-page-images). `Scenario.from_url(url)` retrieves one URL, `Scenario.from_html(url)` retains HTML and extracted text, and `ScenarioList.from_urls(urls)` retrieves several. These make network requests. `ScenarioList.from_source("directory", "materials", recursive=True)` creates file-backed scenarios from a directory; see [A collection of files](/en/latest/scenarios#a-collection-of-files) for filtering file types and retaining filenames with answers. Keep filenames, page numbers, or source URLs as scenario fields when you want them alongside the answers. These fields make it possible to trace a finding back to the source passage or file. [FileStore](/en/latest/filestore) explains how EDSL represents file attachments; the [Scenarios examples](/en/latest/scenarios#data-and-files) show how to reference them in questions. ## Existing questionnaires If your source contains questions to ask rather than records to analyze, you want to create a Survey. [Builder](/en/latest/survey_builder) supports questionnaire creation and import through a visual interface. Importing an existing questionnaire requires its question types, wording, options, and routing to be represented in the resulting survey. If your ScenarioList contains EDSL question definitions, `definitions.to_survey()` converts each row into a question. Supply fields such as `question_type`, `question_name`, `question_text`, and the parameters required by that type. For example: ```python theme={null} from edsl import ScenarioList definitions = ScenarioList.from_list_of_dicts([ {"question_type": "free_text", "question_name": "feedback", "question_text": "What should improve?"}, {"question_type": "multiple_choice", "question_name": "priority", "question_text": "Which matters most?", "question_options": ["Speed", "Reliability"]}, ]) survey = definitions.to_survey() ``` Here each row defines a question, so the result is a two-question Survey rather than two interviews about source records. The questions follow the row order; add any branching or memory with the [Survey methods](/en/latest/surveys). Use this route when question definitions already live in a table, or construct Question objects directly when writing a survey in Python. # Comparing models Source: https://docs.expectedparrot.com/en/latest/guides/model_comparisons Apply selected models to matched inputs and inspect disagreement. Different models can interpret the same request differently, even when they must choose from the same answer options. A model comparison sends matched inputs to several models and puts their answers side by side. This is useful when choosing a model for a labeling task, examining whether a finding depends on the model, or identifying records that deserve closer review. Here you will ask two models to classify the sentiment of the same three feedback messages. EDSL handles the repeated interviews; the comparison table lets you see where their labels agree or differ. ## Prepare one task for every model The example below creates a multiple-choice question and three scenarios. Each scenario carries a `record_id` so its answers can be matched across models, and a `feedback` field that fills the placeholder in the question. The `job` asks the question separately for each record: ```python theme={null} from edsl import QuestionMultipleChoice, Scenario, ScenarioList, Survey question = QuestionMultipleChoice( question_name="sentiment", question_text="Classify the sentiment of this feedback: {{ scenario.feedback }}", question_options=["Positive", "Neutral", "Negative"], ) feedback = ScenarioList([ Scenario({"record_id": "f1", "feedback": "The setup was easy."}), Scenario({"record_id": "f2", "feedback": "I couldn't find the export button."}), Scenario({"record_id": "f3", "feedback": "It worked as expected."}), ]) job = question.by(feedback) # by(): one interview per input, with the same answer.sentiment column. # loop(): three distinct questions in one survey, with the text substituted now. variants = question.loop(feedback) variant_survey = Survey(variants) ``` Use `job` for the comparison below. The final lines also demonstrate `question.loop(feedback)`, which creates a `variant_survey` containing three distinct questions in one interview. That alternative is useful when the records belong together in one questionnaire; the separate-interview design here keeps one answer column and one row per record and model. ## Select two available models With [model access configured](/en/latest/api_keys), display the model catalog: ```python theme={null} from edsl import Model, ModelList Model.available() ``` Choose two model identifiers and their inference services from the catalog, then replace the placeholders below. The models can be from the same service or different services. A ModelList groups them so you can attach both to the existing job in one call: ```python theme={null} models = ModelList([ Model("MODEL_A", service_name="SERVICE_A"), Model("MODEL_B", service_name="SERVICE_B"), ]) comparison_job = job.by(models) ``` You can also discover models from the terminal with `ep models --text`. See [Language models](/en/latest/language_models) for catalog filters and model parameters. ## Compare answers for each record Three records times two models gives six interviews for one iteration. The following code runs them, selects the record identifier, model name, and answer, then rearranges those six rows into a table with one column per model: ```python theme={null} results = comparison_job.run() frame = results.select("scenario.record_id", "model.model", "answer.sentiment").to_pandas() comparison = frame.pivot(index="scenario.record_id", columns="model.model", values="answer.sentiment") comparison ``` Use two distinct model names and one iteration for this pivot. Its three rows line up the answers for `f1`, `f2`, and `f3`, so a disagreement is visible on the same row. Read those rows alongside the original feedback: differences may reveal ambiguous text, different interpretations of the categories, or a model error. Agreement tells you whether the models made the same choice; [human reference labels](/en/latest/research/human_validation) let you assess whether that choice matches your intended classification. ## Adapt the comparison You can reuse this pattern with any EDSL survey, including numerical ratings, free-text critiques, and questions about [files or images](/en/latest/scenarios#data-and-files). Choose models that support the input types your survey uses. To compare configurations of one model, put multiple instances of that model in a ModelList with different supported parameters. Retain the parameter columns in the results: the model name alone will no longer identify a configuration. Similarly, `comparison_job.run(n=3)` produces three iterations for each record and model; retain the `iteration` field instead of using the single-answer pivot above. [Results](/en/latest/results) explains selecting and grouping these fields. If the question is how instructions affect an answer, keep the model fixed and use the [Prompt sensitivity](/en/latest/research/prompt_sensitivity) design. If you simply want to run one chosen model, attach it directly with `job.by(model)`; a ModelList is only needed for the multi-model comparison. # Provider and model configuration Source: https://docs.expectedparrot.com/en/latest/guides/providers Discover current services and configure model-specific parameters. A provider supplies access to a model; EDSL's `service_name` selects the connection to that provider. Use this page when configuring model-specific settings or an institutional deployment such as Azure. For ordinary model selection, start with [Language models](/en/latest/language_models); [Model access](/en/latest/api_keys) explains Expected Parrot access and using your own provider keys. After choosing access, inspect the available services and models, then the configuration you will use: ```python theme={null} from edsl import Model Model.services() Model.available() model = Model() {"model": model.model, "parameters": model.parameters} ``` Catalog discovery may use network access. Creating a configuration does not generate answers. For an explicit model, use `Model(model_name, service_name=provider)` with identifiers selected from the catalog. ## Reasoning parameters These settings matter when your task needs a different generation budget or behavior—for example, a lengthy document analysis rather than a short category label. Set supported parameters on `Model(...)`; they apply to the model calls in the job, rather than changing your survey questions. A provider's supported parameters depend on the model. For OpenAI reasoning models, `reasoning_effort` is forwarded for provider validation. The Responses adapter also accepts a `reasoning` dictionary; an explicitly supplied effort overrides the effort inside that dictionary. Check the adapter and provider documentation for the selected model before choosing values. Do not assume temperature or token-limit semantics are shared across models. ## Azure deployments The current Azure adapter reads `AZURE_ENDPOINT_URL_AND_KEY` as comma-separated `endpoint:key` pairs. Configure this credential securely without printing it. For Azure OpenAI, the endpoint includes the deployment path and its `api-version` query parameter. Select the `azure:` identifier reported for that configuration and pass `service_name="azure"`. Deployment names and endpoint versions come from your Azure account. See the [Azure adapter source](https://github.com/expectedparrot/edsl/blob/main/edsl/inference_services/services/azure_ai.py) for the format supported by your installed EDSL version. ## Local and cloud execution Local orchestration can still make network calls to a provider. See [Connect to your own model server](/en/latest/language_models#connect-to-your-own-model-server) to point EDSL at a local server. For Azure or other cloud-hosted models, use the service and credential configuration documented by the installed adapter and [Managing Keys](/en/latest/api_keys); endpoints and deployment identifiers are account-specific. [Model comparisons](/en/latest/guides/model_comparisons) keeps matched inputs across providers. Use [Costs](/en/latest/costs) before expanding a run, and inspect recorded prompts and errors when troubleshooting adapter behavior. # Saving and sharing objects Source: https://docs.expectedparrot.com/en/latest/guides/saving_objects Save, reopen, and share EDSL objects. Save EDSL objects to continue a study in another session or give a collaborator the materials needed to inspect and reuse it. An `.ep` file reopens as the same kind of EDSL object; a CSV is a table for analysis in other software. Choose what to save based on what you want to do next: | What you want to preserve | Save | Reopen with | | - | - | - | | The questionnaire and its logic | `survey.save("survey.ep")` | `Survey.load("survey.ep")` | | The full planned model run, including its inputs | `job.save("job.ep")` | `Jobs.load("job.ep")` | | Collected answers, prompts, and study information | `results.save("results.ep")` | `Results.load("results.ep")` | | Selected columns for a report or statistical software | `results.select("answer.*").to_csv("answers.csv")` | Your usual CSV reader. | Loading a saved Results object lets you analyze the collected answers without making new model calls. Saving a job preserves what to run; it does not substitute for saving the completed results. ## Save and reopen a survey This example writes a local package and reads it back. It makes no model calls or uploads. ```python theme={null} from edsl import QuestionFreeText, Survey survey = Survey([QuestionFreeText(question_name="feedback", question_text="What should improve?")]) survey.save("feedback.ep") restored = Survey.load("feedback.ep") restored.question_names # ['feedback']; the loaded survey is ready to use. ``` Use explicit `.json` or `.json.gz` filenames when you need JSON instead of a package. For keeping named versions such as a pilot and a final instrument, see [Survey version history](/en/latest/survey_ep_format). Basic saving and loading do not require Git commands. ## Share through Expected Parrot With authentication configured, `survey.push(description="Feedback survey", visibility="private")` uploads the survey and returns its identifier. Choose visibility explicitly and retain the returned identifier for retrieval or updates. Sharing a survey does not automatically share every referenced dataset or grant access to all related material. Use [Expected Parrot objects](/en/latest/coop) and the [CLI Guide](/en/latest/cli) for remote operations. See [Notebooks](/en/latest/notebooks) for sharing a notebook or script as supporting material. ## Preserve a directory of study materials `Study` captures files in a directory, including subdirectories. Use it when a study includes scripts, notes, and other files alongside individual EDSL objects: ```python theme={null} from edsl import Study study = Study("study-materials", exclude=[".git/*", "__pycache__/*", "*.tmp"]) study.to_jsonl("study-materials.jsonl") restored = Study.from_jsonl("study-materials.jsonl") restored.extract("restored-materials") ``` This assumes `study-materials` already exists. The archive contains file contents; it does not execute scripts or install dependencies. `exclude` uses relative-path glob patterns and no exclusions are automatic. Review the included files before sharing the archive. `extract()` writes beneath the original directory name (`restored-materials/study-materials` here) and overwrites existing files with matching paths. `study.to_yaml()` returns a readable inventory of the captured files. # Survey logic and piping Source: https://docs.expectedparrot.com/en/latest/guides/survey_logic Build surveys that choose relevant questions and use earlier answers in follow-ups. Survey logic lets an interview respond to the answers it receives. You can screen out respondents who have not used a product, ask a follow-up about a problem they described, or repeat a question for each item they name. These patterns are useful whenever asking everyone the same fixed sequence would produce irrelevant questions. Three EDSL features work together here. **Branching** decides which question comes next. **Piping** inserts an earlier answer into a later question's wording. **Memory** gives the model earlier questions and answers as context when it answers a later question. A survey can use any combination of these features. ## Build a follow-up interview The example below asks whether a respondent has used an export feature. A “No” ends the interview. A “Yes” leads to a question about problems, followed by a question about how to resolve the reported problem. It demonstrates all three features in a three-question survey: ```python theme={null} from edsl import QuestionYesNo, QuestionFreeText, Survey used = QuestionYesNo(question_name="used", question_text="Have you used the export feature?") problem = QuestionFreeText( question_name="problem", question_text="What problem, if any, did you encounter when exporting?", ) followup = QuestionFreeText( question_name="followup", question_text="What would resolve this problem: {{ problem.answer }}?", ) survey = ( Survey([used, problem, followup]) .add_stop_rule(used, "{{ used.answer }} == 'No'") .add_targeted_memory(followup, problem) ) ``` ## Check the paths `add_stop_rule()` evaluates the answer to `used` after that question is answered. If the answer is `No`, neither `problem` nor `followup` is asked. Otherwise, the survey continues in its original order. In the last question, `{{ problem.answer }}` is replaced by the respondent's actual answer. If they answered “The export button is hidden,” the question becomes “What would resolve this problem: The export button is hidden?” `add_targeted_memory(followup, problem)` also supplies the preceding question and its answer as context. This illustrates both mechanisms; piping alone would be enough if you only wanted to insert the problem text. With [model access configured](/en/latest/api_keys), run the survey and view the answers together: ```python theme={null} results = survey.run() results.select("answer.used", "answer.problem", "answer.followup") ``` This returns one interview with an answer column for each question; skipped questions have no substantive answer. To ask the same survey of several profiles or about several products, attach [agents](/en/latest/agents) or [scenarios](/en/latest/scenarios) with `survey.by(...)` before running it. For a closing message on an early-exit path, see [Survey messages](/en/latest/survey_messages). ## Choose which prior answers a model sees Use memory when a later answer should take earlier exchanges into account without copying them into the question text. For example, a final recommendation can consider several earlier answers while keeping its wording simply “What should we improve first?” Choose how much context to include: | Method | Context supplied to later questions | | - | - | | `survey.add_targeted_memory("followup", "problem")` | Include one earlier question and answer. | | `survey.add_memory_collection("followup", ["used", "problem"])` | Include several named earlier questions and answers. | | `survey.set_lagged_memory(lags=2)` | Include the preceding two questions and answers for each question. | | `survey.set_full_memory_mode()` | Include all preceding questions and answers. | These methods modify the survey and return it for chaining. Targeted memory is useful for a specific dependency, lagged memory for a moving conversation, and full memory for an interview in which every earlier exchange is relevant. More memory also means more text in later model prompts. ## Jump to another question Ending the interview is only one routing choice. In a longer survey, you might want non-users to skip the feature questions but still answer a final summary question. If that survey contains a later question named `summary`, use `survey.add_rule("used", "{{ used.answer }} == 'No'", next_question="summary")` to jump to it. Use `add_skip_rule()` to skip a particular question before it is asked, `add_rule()` to choose a destination after an answer, and `add_stop_rule()` to end the interview. Conditions can refer to available scenario values or earlier answers; an answer from a question that has not been asked cannot supply the intended routing information. `survey.clear_non_default_rules()` removes custom routing and restores the default forward sequence. See [Surveys](/en/latest/surveys) for the complete routing methods. ## Ask follow-ups for items in a list answer Sometimes you do not know the follow-up topics in advance. `Survey.loop_over()` can take a list answer and ask the same follow-up about each item. The example below first requests up to three features, then asks what would improve each named feature. The follow-up template must not already be in the survey: ```python theme={null} from edsl import QuestionFreeText, QuestionList, Survey topics = QuestionList( question_name="topics", question_text="List up to three features you use.", max_list_items=3, ) detail = QuestionFreeText( question_name="detail", question_text="What would improve {{ item }}?", ) survey = Survey([topics]).loop_over("topics", ask=detail, max_items=3) survey.question_names ``` `{{ item }}` receives one list item at a time. If the answer lists “exports” and “search,” the respondent gets a follow-up about each; the unused third position is skipped. EDSL builds the three possible follow-up questions in advance, so each answer has its own question name in the results. `survey.loop_question_groups()` reports which questions belong to each loop. For conditional follow-ups within the loop, attach `detail.skip_when(expression)` or `detail.jump_when(expression, to="another_template_name")` before calling `loop_over()`. For a fixed set of multiple-choice options, `survey.add_followup_questions(reference_question, followup_template, answer_template_var="answer")` creates a follow-up for each option and routes according to the selected answer. Use `{{ answer }}` in that template for the option text. If you already know the items before the interview, `question.loop(scenarios)` instead creates a question for each supplied scenario. To ask each item in a separate interview, use `question.by(scenarios)`. See [Experimental conditions](/en/latest/guides/experimental_conditions) for choosing between those designs. ## Compute scores within a survey A follow-up can also use a calculated value. For a survey containing numeric questions named `score_a` and `score_b`, `survey.add_summation_question(question_name="total", include=["score_a", "score_b"])` adds a question that computes their sum. `add_weighted_linear_scale_sum()` creates a score from linear-scale questions with specified weights. These calculations use the earlier answers directly and do not ask a model to do the arithmetic. Use them when the score belongs in the interview itself; use [Results transformations](/en/latest/results) when you only need it during analysis. # Human surveys Source: https://docs.expectedparrot.com/en/latest/humanize EDSL comes with built-in methods for generating web versions of your surveys and collecting and analyzing responses from humans. `humanize()` turns an EDSL Survey into a web questionnaire that people can complete. It preserves the questions and survey logic while adding a respondent interface, an administration page, and response collection. Start with [Human studies](/en/latest/guides/human_studies) for a short end-to-end example; use this page to configure scenarios, respondent lists, and data retrieval. > [!NOTE] > > This page covers code methods for launching surveys, gathering human responses, and analyzing results. For interactive survey building, see [Survey Builder](/en/latest/survey_builder). For Prolific recruitment, see [Prolific studies](/en/latest/prolific). For email notifications and scheduled reminders, see [Survey notifications](/en/latest/humanize_notifications). To let AI agents take a survey, see [AI agent access](/en/latest/humanize_agent_access). ## How it works **Step 1** **Create** [Surveys](/en/latest/surveys) with desired types of [Questions](/en/latest/questions). **Step 2** **Humanize** the survey to generate a web-hosted version on Expected Parrot with a respondent link and an admin dashboard. **Step 3** **Share** the respondent link with human participants (or recruit via [Prolific](/en/latest/prolific)). **Step 4** **Collect** responses back as a standard EDSL `Results` object. **Step 5** **Analyze** your human results alongside any LLM results. *** ## Basic usage The example collects three measures of workplace experience: work-from-home frequency, satisfaction, and a suggested improvement. After defining the questions, `.humanize()` publishes them using your [Expected Parrot account](/en/latest/api_keys#log-in-from-a-notebook). ```python theme={null} from edsl import Survey, QuestionMultipleChoice, QuestionFreeText, QuestionLinearScale q1 = QuestionMultipleChoice( question_name="frequency", question_text="How often do you work from home?", question_options=["Never", "1-2 days a week", "3-4 days a week", "Every day"], ) q2 = QuestionLinearScale( question_name="satisfaction", question_text="How satisfied are you with your current work arrangement?", question_options=[1, 2, 3, 4, 5], option_labels={1: "Very dissatisfied", 5: "Very satisfied"}, ) q3 = QuestionFreeText( question_name="improvement", question_text="What one change would most improve your day-to-day work experience?", ) survey = Survey([q1, q2, q3]) info = survey.humanize( human_survey_name="Workplace Experience Survey", survey_description="A short survey about remote and in-office work preferences.", survey_visibility="unlisted", ) info # Contains the respondent link, admin link, and collection UUID. ``` ### `.humanize()` parameters | Parameter | Type | Default | Description | | - | - | - | - | | `human_survey_name` | str | `"New survey"` | Display name on Expected Parrot | | `survey_description` | str \| None | None | Optional description | | `survey_alias` | str \| None | None | URL-friendly alias | | `survey_visibility` | str | `"private"` | `"private"`, `"public"`, or `"unlisted"` | | `humanize_schema` | dict \| None | None | Controls question styling and optionality — see [Humanize schema](/en/latest/humanize_schema) | ### Return value `survey.humanize()` returns a `Scenario` (dict-like) with these keys: | Key | Description | | - | - | | `uuid` | Human survey UUID — needed to fetch responses later | | `respondent_url` | Public link for respondents | | `admin_url` | Admin dashboard link | | `name` | Survey name | | `n_responses` | Response count (0 at creation) | | `survey_uuid` | UUID of the pushed survey object | | `scenario_list_uuid` | UUID of the pushed scenario list (if any) | | `asset_substitutions` | Old asset uuid → new one, when the schema named assets that were copied into your library. Empty when nothing was copied | **Save the `uuid`** — you need it to retrieve responses. *** ## Previewing before deployment Use `.preview()` to get a preview URL without creating a live survey: ```python theme={null} preview_url = survey.preview() # print() also shows output in scripts. In a notebook, you can instead # put the value you want to inspect on the last line of a cell. print(preview_url) # With schema customization schema = {"questions": {"improvement": {"optional": True}}} preview_url = survey.preview(humanize_schema=schema) ``` *** ## Present questions in groups To request one group per page, define contiguous question groups on the survey and pass `humanize_schema={"survey": {"presentation": "group"}}` to `.humanize()`. See [Grouped surveys](/en/latest/humanize_groups) for a complete example, branching behavior, and availability requirements. *** ## Add your logo To show your own logo above the questions, upload the image once and reference it from the humanize schema: ```python theme={null} from edsl import Coop logo = Coop().upload_human_survey_asset("lab_logo.png") survey.humanize( human_survey_name="Customer feedback", humanize_schema={ "survey": { "branding": { "logo": { "source": {"type": "asset", "asset_uuid": logo.uuid}, "alt": "Acme Research logo", } } } }, ) ``` Uploading the same file again returns the asset you already have, so re-running a script does not fill your library with copies. `Coop().list_human_survey_assets(page=1, page_size=20)` lists a page of image-library metadata. `Coop().get_human_survey_asset(asset_uuid)` returns a `HumanSurveyAsset`; its `download("logo.png")` method saves the image locally. `Coop().delete_human_survey_asset(asset_uuid)` removes an asset when you intend to delete it. From the CLI, upload once and apply it to a survey that already exists: ```bash theme={null} ep humanize assets upload lab_logo.png ep humanize schema set --logo-asset --logo-alt "Acme Research logo" ``` Or do both in one step: ```bash theme={null} ep humanize schema set --logo-file lab_logo.png --logo-alt "Acme Research logo" ``` See [Humanize schema](/en/latest/humanize_schema) for the `branding` fields, and [Custom CSS](/en/latest/custom_css) for resizing the logo or putting it on a colored banner. *** ## With scenarios When using scenarios, call `.humanize()` on a `Jobs` object (not directly on the survey). You must specify a `scenario_list_method`. Use scenarios for stimulus material or treatment conditions. This example assigns each respondent one of three policy descriptions and asks for a support rating and its reason. Random assignment can select the same policy for several respondents: ```python theme={null} from edsl import Survey, QuestionFreeText, QuestionLinearScale, Scenario, ScenarioList scenarios = ScenarioList([ Scenario({"policy": "Four-day workweek with no change in pay"}), Scenario({"policy": "Flexible start times between 7am and 11am"}), Scenario({"policy": "Monthly remote-work stipend of $100"}), ]) q1 = QuestionLinearScale( question_name="support", question_text="How much do you support this policy: '{{ scenario.policy }}'?", question_options=[1, 2, 3, 4, 5], option_labels={1: "Strongly oppose", 5: "Strongly support"}, ) q2 = QuestionFreeText( question_name="reasoning", question_text="What is the main reason for your rating?", ) survey = Survey([q1, q2]) info = survey.by(scenarios).humanize( human_survey_name="Workplace Policy Reactions", scenario_list_method="randomize", ) ``` ### Scenario list methods | Method | Behavior | | - | - | | `"randomize"` | Each respondent gets a random scenario (with replacement) | | `"ordered"` | Scenarios are assigned sequentially to respondents | | `"loop"` | Survey is expanded — every question is repeated for every scenario | | `"single_scenario"` | Exactly one scenario is used for all respondents (list must have length 1) | **Rules:** * If you attach scenarios, you **must** specify `scenario_list_method`. * If you specify a method, you **must** attach scenarios. * Do not attach language models to a human-survey job. An AgentList can supply participant metadata and delivery addresses, as described below; it does not generate simulated responses. *** ## With agent lists (for email notifications) Attach an `AgentList` via `.by()` to associate respondents with email addresses. Pass a `DeliveryMap` to tell Coop which agent trait holds the email address. ```python theme={null} from edsl import Survey, Agent, AgentList from edsl.coop.coop_humanize_notifications import ChannelConfig, DeliveryMap agents = AgentList([ Agent(traits={"name": "Jordan", "email": "jordan@example.com"}), Agent(traits={"name": "Taylor", "email": "taylor@example.com"}), Agent(traits={"name": "Morgan", "email": "morgan@example.com"}), ]) info = survey.by(agents).humanize( human_survey_name="Panel Study — Wave 1", agent_list_description="Recruited panel participants", delivery_map=DeliveryMap(email=ChannelConfig(col_name="email")), ) ``` `col_name` must match the trait key that holds each respondent's email address. Once deployed, use [Survey notifications](/en/latest/humanize_notifications) to send invitations, schedule reminders, and track delivery status. *** ## Retrieving responses ```python theme={null} from edsl import Coop coop = Coop() results = coop.get_human_survey_responses("your-human-survey-uuid") # In a notebook, the last expression displays the selected table. # In a script, append .print() to display it. results.select("answer.frequency", "answer.satisfaction", "answer.improvement") ``` ### What the results contain Each completed response is one result, oldest first. Retrieving data reads the recorded answers; it does not rerun the questionnaire. Skipped questions stay skipped. Column names match the responses table on the survey's page. Select `scenario.*` alongside answers when you need to compare the material or condition each respondent received. | Field | Description | | - | - | | `answer.{question}`, `comment.{question}_comment` | The respondent's answer and comment. Empty for a question they didn't see. | | `agent.response_uuid`, `agent.agent_name` | The response's ID. | | `agent.respondent_uuid`, `agent.agent_index` | The respondent's ID and position in the attached agent list. Empty without an agent list. | | `agent.{trait}` | The traits of the respondent's agent, when an agent list is attached. | | `raw_model_response.{question}_started_at`, `_answered_at` | When the respondent reached and answered the question. | | `raw_model_response.{question}_question_presented` | `False` if survey logic skipped the question, so a skipped question can be told from one left blank. | | `question_options.{question}_question_options` | The options as the respondent saw them, after piping and any randomized order. | | `agent.external_platform`, `agent.prolific_*` | The Prolific submission, study and participant IDs, when any response came through Prolific. | Fields that show how a response was collected, such as `agent.collection_interface`, are listed under [AI agent access](/en/latest/humanize_agent_access#in-results). The `model.*` columns describe a placeholder model: a human response has no model, but every result records one. To get only some responses, pass `started_after` and/or `started_before` (inclusive; a response is "started" when the respondent opened the survey). ### Checking response count ```python theme={null} status = coop.get_human_survey("your-human-survey-uuid") print(f"Responses received: {status['n_responses']}") ``` ### Find surveys and inspect respondent progress Use Python pagination when retrieving account records: ```python theme={null} surveys = coop.list_human_surveys(search_query="Workplace", page=1, page_size=20) respondents = coop.get_human_survey_respondents( "your-human-survey-uuid", page=1, page_size=50, ) respondents ``` These calls return response dictionaries with records and pagination information; inspect the returned metadata before requesting another page. `coop.get_human_survey_agent_list(uuid)` retrieves the attached respondent list configuration. To change its delivery mapping or participation settings, use `coop.patch_human_survey_agent_list(...)`; see [notifications](/en/latest/humanize_notifications) and [AI agent access](/en/latest/humanize_agent_access). For events logged by a survey's [custom JavaScript](/en/latest/custom_javascript), `coop.get_human_survey_events(uuid, limit=200)` retrieves a batch. While `has_more` is true, pass the returned `next_cursor` as `after` to continue, or use `coop.get_all_human_survey_events(uuid)` to collect all batches. `coop.count_human_survey_events(uuid)` counts those events. Preview events are included and marked `is_preview`; exclude them from participant analyses. Use `get_human_survey_responses()` for answer data. ### Working with file upload responses If your survey includes a `QuestionFileUpload` question, each response's answer for that question is a list of file-info dicts. Use `FileStore.from_file_upload_answer` (single file) or `FileStoreList.from_file_upload_answers` (all files at once) to download and wrap them: ```python theme={null} from edsl import Coop from edsl.scenarios import FileStore, FileStoreList coop = Coop() results = coop.get_human_survey_responses("your-human-survey-uuid") # Each answer is a list of uploaded file dicts uploaded_files = results[0]["answer"]["my_upload_question"] # Download all files as a FileStoreList fsl = FileStoreList.from_file_upload_answers(uploaded_files) # Or download a single file fs = FileStore.from_file_upload_answer(uploaded_files[0]) ``` The returned `FileStore` and `FileStoreList` objects behave like any other — you can inspect, filter, or pass them into further EDSL workflows. *** ## QR code Generate a QR code for the respondent link using `get_human_survey_qr_code()`. Requires the `qrcode` library (`pip install "edsl[full]"` or `pip install "qrcode[pil]"`). ```python theme={null} from edsl import Coop coop = Coop() qr = coop.get_human_survey_qr_code("your-human-survey-uuid") qr.save("qr_code.png") # save to file ``` From the CLI, generate a QR code for the shared respondent URL: ```bash theme={null} ep humanize qr --output survey-qr.png ``` When the human survey has an attached agent list, export the roster together with each respondent's personal link: ```python theme={null} links = coop.get_human_survey_respondent_links( "your-human-survey-uuid", save_path="respondent-links.csv", ) # links is a Dataset; the CSV contains the personal respondent URLs. ``` The CLI alternative is: ```bash theme={null} ep humanize links --output respondent-links.csv ``` Add `--qr-dir` to generate one personal QR code per respondent and record each PNG path in the CSV: ```bash theme={null} ep humanize links \ --output respondent-links.csv \ --qr-dir respondent-qr ``` Personal links and QR codes contain access tokens and act as bearer credentials. Store the export securely and share each link or code only with its intended respondent. The CLI returns only file metadata on stdout, never the links. *** ## Prolific integration For paid recruitment, see [Prolific recruitment](/en/latest/prolific). That guide covers account limits, cost estimation, draft creation, preflight checks, publication, and response retrieval. # AI agent access Source: https://docs.expectedparrot.com/en/latest/humanize_agent_access Let AI agents take a human survey through an agent link, and control who answers and what guidance they get. A human survey can let AI agents take it: a participant hands their AI agent the survey's agent link, and the agent answers through Expected Parrot's API instead of a browser. Agent access is off until the survey's owner turns it on, and only the owner can read or change it. The agent link is the respondent link with `/respond/human-surveys/` replaced by `/respond/agent/`: ```text theme={null} https://www.expectedparrot.com/respond/agent/ ``` It serves the agent its instructions. Each agent attempt is recorded as a response, alongside browser responses. Choose this workflow when participants want to use their own assistant to complete a hosted survey, or when you deliberately want autonomous assistant responses. This is separate from defining an EDSL `Agent` profile and running a simulation with `survey.by(agents).run()`. The participation mode below determines whether answers come from the participant, authorized context, or the assistant itself; keep that mode in comparisons of the collected responses. *** ## The config | Field | Type | Default | Description | | - | - | - | - | | `enabled` | bool | `false` | Whether the agent link accepts new attempts. | | `participation_mode` | string | `"human_assisted"` | Who produces the answers (see below). | | `instructions` | string \| null | `null` | Guidance for agents on the whole survey, at most 4,000 characters. In autonomous mode, this is how the agent should answer. Advisory: nothing enforces it. | | `question_settings` | dict | `{}` | Per-question settings, keyed by question name. Each may have `instructions` (at most 2,000 characters). | ### Participation modes | Mode | Who answers | | - | - | | `human_assisted` | The participant. The agent asks each question and records their answers, and doesn't answer for them. | | `authorized_context` | The agent, from context the participant has authorized it to use, asking them whenever it's unsure. | | `autonomous` | The agent alone, as described in `instructions`. | In `autonomous` mode there's no participant, so: * The agent can't hand a page to a browser (see [What agents can't answer](#what-agents-cant-answer)). * A start with Prolific parameters is refused: autonomous responses can't be used for a paid panel. *** ## Reading and changing the config ```python theme={null} from edsl import Coop coop = Coop() access = coop.get_human_survey_agent_access("your-human-survey-uuid") # {"configured": True, "enabled": True, "participation_mode": "human_assisted", # "instructions": None, "question_settings": {}} ``` `configured` is `False` if agent access has never been set; the other fields are then their defaults. To change it, send only the fields you want to change: ```python theme={null} coop.patch_human_survey_agent_access( "your-human-survey-uuid", { "enabled": True, "participation_mode": "autonomous", "instructions": ( "Keep free-text answers to one or two sentences, and use the " "comment box to flag any answer that's an estimate." ), "question_settings": { "improvements": { "instructions": "Name at least one specific change, not a general comment." } }, }, ) ``` This example assumes the hosted survey has a question named `improvements`. Pass `survey=your_survey` when you also have the local Survey and want to validate the question names before sending the patch. The patch is merged into the stored config: * Fields left out are unchanged. * `question_settings` merges by question name: naming `improvements` changes only its settings. * `"instructions": None` clears the survey-wide instructions. * A question set to `None` in `question_settings` has its settings removed, e.g. `{"question_settings": {"improvements": None}}`. It returns the stored config, in the same shape as `get_human_survey_agent_access`. From the CLI, `ep humanize agent-access get ` prints the config, and `patch` changes it: ```bash theme={null} ep humanize agent-access patch --enabled --participation_mode autonomous ep humanize agent-access patch --question_instructions "improvements=Name at least one specific change, not a general comment." --survey survey.ep ``` `patch` also takes the patch as a JSON file with `--config`, and `--survey` checks question names as `survey=` does. *** ## Validation EDSL checks a patch before sending it, and raises `AgentAccessValidationError` if: * It has a key the config doesn't define. * A value is the wrong type or out of range, or `participation_mode` isn't one of the three modes. * `enabled`, `participation_mode` or `question_settings` is `None`. Only `instructions` and a single question's settings can be cleared that way. * With `survey=` (or `--survey`), it gives settings to a question that isn't in the survey. *** ## What agents can't answer Some questions need the participant themselves: * **Voice-only interviews.** In `human_assisted` and `authorized_context` mode, the agent hands the page to the participant: it gives them a one-time link to answer that page in their browser, then carries on. In `autonomous` mode there's no one to hand it to, so the survey can't be completed by an agent. * **Timed questions.** A question with a fixed time limit, in a survey that shows one question per page (where the browser runs the timer), can't be answered by an agent: a time limit can't be enforced over the API. The agent link tells the agent to stop and give the participant the browser link instead. Text interviews and file uploads are answered by the agent itself. *** ## In results Every response carries these fields, so you can tell agent responses apart: | Field | Description | | - | - | | `agent.collection_interface` | `browser`, or `agent_http` for an agent response. | | `agent.participation_mode` | The survey's mode when the attempt started. Empty for browser responses. | | `agent.client_name_self_reported` | The agent's client, as the agent reported it (e.g. `Claude Code`). Not verified. | | `agent.model_name_self_reported` | The agent's model, as the agent reported it. Not verified. | | `agent.handed_off_questions` | The questions the participant answered in their browser through a handoff. Empty for browser responses. | The survey's page on Expected Parrot also has an **AI agents** tab showing the current config. # Grouped surveys Source: https://docs.expectedparrot.com/en/latest/humanize_groups Define question groups and request one group per page in a human survey. Grouped presentation puts related questions on the same page—for example, background questions together and feedback questions on the next page. Participants can review the questions in a group before submitting that page. Define the page boundaries on your `Survey`, then pass `{"survey": {"presentation": "group"}}` to `.humanize()`. > [!NOTE] > > Grouped presentation requires support in both your installed EDSL version and the hosted survey interface. Preview with the same schema to check that the intended questions appear together. ## Create a two-page survey This example puts two background questions on the first page and two feedback questions on the second. ```python theme={null} from edsl import Survey, QuestionFreeText, QuestionMultipleChoice survey = Survey([ QuestionMultipleChoice( question_name="work_location", question_text="Where do you usually work?", question_options=["At home", "In an office", "A mix of both"], ), QuestionFreeText( question_name="role", question_text="What is your role?", ), QuestionMultipleChoice( question_name="satisfaction", question_text="How satisfied are you with your work arrangement?", question_options=["Satisfied", "Neutral", "Dissatisfied"], ), QuestionFreeText( question_name="improvement", question_text="What would improve your work arrangement?", ), ]) # First question, last question (inclusive), group name. survey.add_question_group("work_location", "role", "background") survey.add_question_group("satisfaction", "improvement", "feedback") humanize_schema = { "survey": {"presentation": "group"}, "questions": {"improvement": {"optional": True}}, } ``` | Page | Group name | Questions | | - | - | - | | 1 | `background` | `work_location`, `role` | | 2 | `feedback` | `satisfaction`, `improvement` | Preview with the same schema you intend to use when creating the survey: ```python theme={null} preview_url = survey.preview(humanize_schema=humanize_schema) # print() also shows output in scripts. In a notebook, you can instead # put the value you want to inspect on the last line of a cell. print(preview_url) ``` When ready, create the human survey: ```python theme={null} info = survey.humanize( human_survey_name="Workplace experience", humanize_schema=humanize_schema, ) print(info["respondent_url"]) print(info["admin_url"]) ``` The groups live on the survey; the schema selects how to present them. Adding groups alone does not enable grouped presentation. Omit `presentation`, or set it to `"question"`, for the default question-by-question mode. ## Set group boundaries `survey.add_question_group(start_question, end_question, group_name)` accepts question names or question objects and returns the survey for chaining. * Each group includes both endpoints and every question between them. * Groups must not overlap. A one-question page uses the same question for both endpoints. * Names must be unique Python identifiers, such as `background`, and must not match a question name. * Pages follow question order in the survey. * Assign every question you want displayed to a group. With groups defined, the current grouped navigator skips questions outside those groups. For example, to make a page containing only `work_location`, define its group with `survey.add_question_group("work_location", "work_location", "location_page")` on a survey where that question is not already grouped. ## Inspect navigation locally Use `survey.next_group()` to inspect which questions belong on the next page for a given set of answers, without publishing the survey. This is useful when checking a route that should skip a page. For a visual check of the participant experience, use `survey.preview()` instead. With the two-page survey constructed above: ```python theme={null} first_page = survey.next_group() assert first_page["group_name"] == "background" assert first_page["question_names"] == ["work_location", "role"] second_page = survey.next_group( current_group="background", answers={ "work_location.answer": "At home", "role.answer": "Researcher", }, ) assert second_page["group_name"] == "feedback" ``` Pass all answers collected so far, including those from the just-submitted group. Use `include_instructions=False` to return questions only. | Returned field | Meaning | | - | - | | `group_name` | Selected group name, or `None` at the end | | `items` | Question and instruction objects; `[EndOfSurvey]` at the end | | `question_names` | Names of the questions selected for this page | | `is_end` | `True` when the survey has finished | See [Humanize](/en/latest/humanize) for retrieving responses and [Humanize schema](/en/latest/humanize_schema) for styling and other presentation options. ## Plan branching around pages The navigator uses the answers supplied when a page is requested to filter skipped questions. After a page is submitted, it applies the normal navigation rules from that group's **last renderable question** to choose the next page. Put page-level jump or stop rules on that question. If a later question depends on an earlier answer, put the dependent question on a later page so that answer is available when the page is selected. The navigation API does not establish live, within-page updates to conditional questions. For example, starting with the two-page survey above, omit the improvement question for satisfied respondents: ```python theme={null} # Keep the controlling answer on an earlier page. survey.question_groups.clear() survey.add_question_group("work_location", "satisfaction", "background") survey.add_question_group("improvement", "improvement", "improvements") survey.add_skip_rule( "improvement", "{{ satisfaction.answer }} == 'Satisfied'", ) ``` Now the first page collects work location, role, and satisfaction. A satisfied respondent skips the improvements page and reaches the end; other respondents continue to it. Add the rule before previewing or creating the human survey. Additional navigation behavior: * A group with no remaining questions is skipped. * A jump into the middle of a group starts at the target question; earlier questions in that group are omitted. * Relevant instructions before or within the selected group are included by default. # Survey notifications Source: https://docs.expectedparrot.com/en/latest/humanize_notifications Send emails, schedule deliveries, and react to respondent events for human surveys. Use survey notifications to invite a known respondent list, remind people who have not started, or send a transcript after completion. An immediate **delivery** sends messages now, a **schedule** sends them at a chosen time, and a **callback** sends them in response to an event such as survey completion. `HumanSurveyNotificationHandler` keeps these operations associated with one hosted survey. Creating the handler does not send a message; the send, schedule, and callback methods below perform the corresponding action. > [!NOTE] > > The survey must have an agent list with an email delivery channel configured before any notifications can be sent. Build a job with `survey.by(agents)` and pass `delivery_map` to its `.humanize()` method. See [Respondent lists](/en/latest/humanize#with-agent-lists-for-email-notifications) for the construction workflow. The examples below operate on an existing hosted survey. Send and schedule calls contact the service and can deliver real messages to its respondents. Replace placeholder identifiers with values returned by your own survey workflow. ## Getting a handler ```python theme={null} from edsl.coop import HumanSurveyNotificationHandler handler = HumanSurveyNotificationHandler("your-human-survey-uuid") ``` A `Coop` client is created automatically. Pass one explicitly if you need a specific API key or base URL: ```python theme={null} from edsl import Coop from edsl.coop import HumanSurveyNotificationHandler coop = Coop() handler = HumanSurveyNotificationHandler("your-human-survey-uuid", coop=coop) ``` *** ## Statuses A delivery can contain one email task per selected respondent. Its dispatch and delivery statuses describe the message; the respondent's response status describes participation. A delivered invitation therefore does not mean a completed survey. ### Dispatch status Controls whether the platform has attempted to hand off the message to the downstream channel (e.g. the email provider). | Value | Meaning | | - | - | | `pending` | The task has been created but not yet processed. | | `dispatched` | The message was successfully handed off to the channel. | | `failed` | The dispatch attempt failed (e.g. missing address, provider error). | ### Delivery status Reflects what the email provider reports back after dispatch. | Value | Meaning | | - | - | | `pending` | Waiting for a delivery event from the provider. | | `sent` | The provider accepted the message for delivery. | | `delivered` | The provider confirmed the message reached the recipient. | | `bounced` | The message was rejected by the recipient's mail server. | | `failed` | The provider reported an error sending the email. | ### Response status Tracks whether a respondent has engaged with the survey itself. | Value | Meaning | | - | - | | `not_started` | The respondent has not opened the survey link. | | `completed` | The respondent submitted a complete response. | *** ## Quickstart examples ### Send an email right now Trigger an immediate delivery to all respondents. ```python theme={null} result = handler.send_respondent_email(name="Initial outreach") # print() also shows output in scripts. In a notebook, you can instead # put the value you want to inspect on the last line of a cell. print(result) # {"delivery_uuid": "...", "routes": [...]} ``` Pass a custom subject, HTML email body, or restrict which respondents receive it: ```python theme={null} from edsl.coop.coop_humanize_notifications import ( HumanizeRespondentFilter, RespondentCondition, ResponseStatus, ) # Only email respondents who haven't started yet condition = RespondentCondition( response_status=[ResponseStatus.not_started] ) respondent_filter = HumanizeRespondentFilter(conditions=[condition]) result = handler.send_respondent_email( name="Reminder to non-starters", subject="Don't forget to complete our survey", delivery_template='

Please complete our survey.

', respondent_filter=respondent_filter, ) ``` *** ### Schedule a one-time email Schedule a single delivery at a specific time. `run_at` must be a timezone-aware datetime or an ISO 8601 string. ```python theme={null} from datetime import datetime, timedelta, timezone run_at = datetime.now(timezone.utc) + timedelta(days=1) schedule = handler.create_one_time_schedule( name="Reminder tomorrow", run_at=run_at, ) print(schedule["schedule_uuid"]) ``` Update the time before it fires: ```python theme={null} handler.update_one_time_schedule( schedule_uuid=schedule["schedule_uuid"], run_at=run_at + timedelta(days=1), ) ``` *** ### Schedule recurring emails (cron) Send emails on a repeating schedule using standard cron syntax. Provide either `max_jobs` (fire at most N times) or `deadline` (stop firing after this datetime) as the termination condition. Use `start_at` to delay when the schedule begins firing; omit it to start immediately. ```python theme={null} from datetime import datetime, timedelta, timezone # Every Monday at 9 AM Eastern, up to 4 times schedule = handler.create_cron_schedule( name="Weekly Monday reminder", cron_expression="0 9 * * MON", timezone="America/New_York", max_jobs=4, ) ``` With a deadline instead: ```python theme={null} schedule = handler.create_cron_schedule( name="Daily reminders for one week", cron_expression="0 8 * * *", timezone="America/Chicago", deadline=datetime.now(timezone.utc) + timedelta(days=7), ) ``` With a delayed start: ```python theme={null} schedule = handler.create_cron_schedule( name="Weekly reminders starting next week", cron_expression="0 9 * * MON", timezone="America/New_York", max_jobs=4, start_at=datetime.now(timezone.utc) + timedelta(days=7), ) ``` Pause and resume a schedule: ```python theme={null} handler.deactivate_schedule(schedule["schedule_uuid"]) handler.activate_schedule(schedule["schedule_uuid"]) ``` *** ### React when someone completes the survey (callback) A **respondent** is a person with a corresponding entry in the survey's agent list. Each respondent has a unique survey link and a tracked response status. Submissions from people outside the agent list (e.g. anonymous links) are not associated with a respondent. `create_callback` sends emails to respondents in your agent list. Two respondent event types are supported: | Event | When it fires | | - | - | | `"human_survey_respondent.completed"` | When a respondent's status is set to completed. | | `"human_survey_respondent.response_submitted"` | Each time a respondent submits a response. | ```python theme={null} callback = handler.create_callback( name="Completion thank-you", callback_type="human_survey_respondent.completed", ) print(callback["callback_uuid"]) ``` Limit how many times the callback fires (e.g. only notify for the first 100 completions): ```python theme={null} callback = handler.create_callback( name="First 100 completions", callback_type="human_survey_respondent.completed", max_fires=100, ) ``` ### Notify the owner on every new response Use `create_owner_response_callback` to email the survey owner each time a response is submitted. This fires for all responses, including anonymous ones — not just respondents in the agent list. ```python theme={null} callback = handler.create_owner_response_callback(name="New response notification") print(callback["callback_uuid"]) ``` ### Send a transcript after each submission Use `create_transcript_callback` to automatically email a transcript each time a response is submitted. Pass `recipient` to control who receives it: * `"respondent"` (default) — emails each respondent a copy of their own answers. * `"owner"` — emails the survey owner a copy of the answers. Fires for all submissions, including anonymous ones. ```python theme={null} # Email the respondent their own transcript callback = handler.create_transcript_callback( name="Send respondent transcript on submission", recipient="respondent", ) # Email the owner a transcript on every submission callback = handler.create_transcript_callback( name="Send owner transcript on submission", recipient="owner", ) print(callback["callback_uuid"]) ``` Activate and deactivate: ```python theme={null} handler.deactivate_callback(callback["callback_uuid"]) handler.activate_callback(callback["callback_uuid"]) ``` *** ### Filter which respondents receive a delivery `HumanizeRespondentFilter` lets you compose conditions to target specific subsets of respondents. Conditions can be nested with `and` / `or` operators. ```python theme={null} from edsl.coop.coop_humanize_notifications import ( HumanizeRespondentFilter, RespondentCondition, DeliveryStatus, DispatchStatus, ResponseStatus, FilterOperator, ) # Target respondents who were never contacted never_contacted = RespondentCondition(never_contacted=True) # OR respondents whose last email bounced bounced = RespondentCondition( most_recent_delivery_status=[DeliveryStatus.bounced] ) # Combine: never contacted OR previously bounced respondent_filter = HumanizeRespondentFilter( operator=FilterOperator.or_, conditions=[never_contacted, bounced], ) handler.send_respondent_email( name="Re-engagement", respondent_filter=respondent_filter, ) ``` **Available condition fields:** | Field | Type | Filters to respondents where… | | - | - | - | | `respondent_uuids` | `list[str]` | UUID is in the provided list | | `response_status` | `list[ResponseStatus]` | Survey response status matches | | `never_contacted` | `bool` | Has never received a delivery | | `any_dispatch_status` | `list[DispatchStatus]` | Any past task has this dispatch status | | `any_delivery_status` | `list[DeliveryStatus]` | Any past task has this delivery status | | `most_recent_dispatch_status` | `list[DispatchStatus]` | Most recent task has this dispatch status | | `most_recent_delivery_status` | `list[DeliveryStatus]` | Most recent task has this delivery status | *** ### Check delivery status List all delivery jobs for the survey: ```python theme={null} deliveries = handler.list_deliveries() for d in deliveries["deliveries"]: print(d["delivery_uuid"], d["status"], d["sent_count"], d["failed_count"]) ``` Inspect a specific delivery job: ```python theme={null} delivery = handler.get_delivery(delivery_uuid="...") print(delivery) # { # "delivery_uuid": "...", # "status": "completed", # "total_respondents": 50, # "processed_respondents": 50, # "sent_count": 48, # "failed_count": 2, # "started_at": "...", # "completed_at": "..." # } ``` Drill into individual tasks (one per respondent per delivery): ```python theme={null} tasks = handler.list_delivery_tasks(delivery_uuid="...") for task in tasks["tasks"]: print(task["identifier"], task["dispatch_status"], task["delivery_status"]) ``` Get a single delivery task by UUID: ```python theme={null} task = handler.get_delivery_task(task_uuid="...") print(task) # { # "task_uuid": "...", # "channel": "email", # "identifier": "respondent@example.com", # "dispatch_status": "dispatched", # "delivery_status": "delivered", # "respondent": {"respondent_uuid": "...", "response_status": "completed"}, # ... # } ``` *** ## Route types Each delivery or schedule sends email to survey respondents via a `RespondentEmailRouteConfig` route. A delivery template is always required; convenience methods like `send_respondent_email` default to the built-in invitation template when none is provided. `RespondentEmailRouteConfig` options: | Field | Description | | - | - | | `delivery_template` | The email body. Pass an HTML string to use a custom template, or omit to use the built-in invitation template. | | `respondent_filter` | `HumanizeRespondentFilter` to restrict which respondents receive the email. | | `subject` | Email subject line (1-200 characters). Uses the default subject when omitted. | ### Template variables Custom HTML templates support the following variables, which are filled in per-respondent at send time: | Variable | Value | | - | - | | `{{ survey_name }}` | The name of the human survey. | | `{{ url }}` | The respondent's unique survey link (includes their access token). | Example: ```python theme={null} handler.send_respondent_email( name="Initial outreach", subject="You're invited to take our survey", delivery_template="""

Hi there,

You've been invited to take {{ survey_name }}.

Click here to begin

""", ) ``` Update the subject and template on an existing respondent email route on a schedule: ```python theme={null} handler.patch_schedule_respondent_email_route( schedule_uuid="...", route_uuid="...", subject="A quick reminder about our survey", delivery_template='

Please complete our survey.

', ) ``` Update the subject and template on an existing respondent email route on a callback: ```python theme={null} handler.patch_callback_respondent_email_route( callback_uuid="...", route_uuid="...", subject="Thanks for completing our survey", delivery_template="

Hi! Thank you for completing {{ survey_name }}.

", ) ``` *** ## Find and revise schedules and callbacks The handler keeps the human survey UUID, so these methods only need the identifier of the schedule, callback, or route: | Task | Method | | - | - | | List schedules or callbacks | `handler.list_schedules(page=1, page_size=50)` / `handler.list_callbacks(page=1, page_size=50)` | | Inspect one | `handler.get_schedule(schedule_uuid)` / `handler.get_callback(callback_uuid)` | | Reschedule | `handler.update_one_time_schedule(schedule_uuid, run_at=timestamp)`; for recurring schedules, `handler.update_cron_schedule(schedule_uuid, cron_expression=expression, timezone=timezone)` | | Rename a callback | `handler.patch_callback(callback_uuid, name="New name")` | | Add a route | `handler.add_schedule_route(schedule_uuid, route_config)` / `handler.add_callback_route(callback_uuid, route_config)` | | Remove a route | `handler.delete_schedule_route(schedule_uuid, route_uuid)` / `handler.delete_callback_route(callback_uuid, route_uuid)` | | Remove a schedule or callback | `handler.delete_schedule(schedule_uuid)` / `handler.delete_callback(callback_uuid)` | Deactivating a schedule or callback pauses future triggers without deleting its configuration. Deletion removes it. Adding or changing active routes affects future deliveries, so review recipients and templates first. Listing methods return one page; use their pagination metadata to request subsequent pages. `list_deliveries()`, `list_delivery_tasks()`, and `get_respondents()` also accept `page` and `page_size`. ## Managing respondents List all respondents and their current response status: ```python theme={null} respondents = handler.get_respondents() for r in respondents["respondents"]: print(r["respondent_uuid"], r["response_status"]) ``` Check and update the agent list delivery configuration (the trait column that holds each respondent's email address): ```python theme={null} from edsl.coop.coop_humanize_notifications import ChannelConfig, DeliveryMap # See current config config = handler.get_agent_list() print(config["agent_list_config"]) # Update the email column mapping handler.patch_agent_list( delivery_map=DeliveryMap(email=ChannelConfig(col_name="email_address")) ) ``` # Humanize schema Source: https://docs.expectedparrot.com/en/latest/humanize_schema Reference for the humanize schema: a dict that configures how survey questions are presented in web surveys with human respondents. The **humanize schema** is a Python dictionary of settings for the participant experience: which questions are optional, how answer controls look, and whether to offer comments, timers, or voice interviews. Your Survey still defines question wording, answer choices, and routing. Include only the settings you want to change; the rest use their defaults. ## Make a question optional and offer a comment This example keeps the rating required, adds a place to explain it, and makes the final free-text question optional. Settings under `questions` use the question names from the Survey: ```python theme={null} from edsl import QuestionFreeText, QuestionMultipleChoice, Survey survey = Survey([ QuestionMultipleChoice( question_name="rating", question_text="How was your experience?", question_options=["Good", "OK", "Bad"], ), QuestionFreeText(question_name="feedback", question_text="What should improve?"), ]) humanize_schema = { "questions": { "rating": {"comment": {"label": "What led to your rating?"}}, "feedback": {"optional": True}, } } preview_url = survey.preview(humanize_schema=humanize_schema) preview_url ``` Previewing requires [Expected Parrot access](/en/latest/api_keys). Pass the same `humanize_schema` to `survey.humanize(...)` when you publish. The rating and optional feedback remain answer columns; the accompanying explanation is recorded as `comment.rating_comment`. Use the field reference below for additional settings, or [Custom CSS](/en/latest/custom_css) for colors and spacing. ## Fields **questions** `dict` Map of question name (string) -> `HumanizeQuestionSchema`. **survey** `dict` Optional survey-level options (e.g. custom CSS). **properties** **survey.presentation** `string` (default: question) `"question"` requests question-by-question presentation (the default); `"group"` requests one question group per page. Define groups on the survey with `add_question_group()` and include every question you want displayed. See [Grouped surveys](/en/latest/humanize_groups) for setup and branching between pages. Question time limits (`time_limit`) are ignored under `"group"`. **survey.custom\_css** `string` Optional custom CSS for the survey. Can be `null` or omitted. See [Custom CSS](/en/latest/custom_css) for the `edsl-*` CSS classes and HTML structure per question type. **survey.progress** `dict` How the respondent is shown their position in the survey. Omit to keep the progress bar, which is what surveys render by default. **properties** **type** `string` (default: bar) Which progress indicator to render. * `"bar"` — a filled bar, optionally labelled (default). * `"hidden"` — no progress indicator anywhere in the survey. * `"steps"` — markers joined by a connector, one per step, with the current step highlighted. **label** `dict` **`bar` only.** The label beneath the bar. Set to `null` to keep the bar and drop the label. **properties** **type** `string` (default: percent) Must be `"percent"` — the share of the survey completed, e.g. `"33%"`. **marker** `string` (default: number) **`steps` only.** Whether each marker carries its step number (`"number"`) or is a bare dot (`"dot"`). **steps** `array` *required* **`steps` only.** The steps, in order. At least two are required. A step is a *boundary*, not a bucket: it covers every survey item from the end of the previous step through its `complete_after`. Questions can be added or removed inside a step without touching this config, and items a respondent skips past inside a step don't move the marker. Each step is an object with: * `label` (string, optional) — the word under the marker, 1–60 characters. Omit or set to `null` to render the marker alone. * `complete_after` (string, optional) — the name of the survey item (question or instruction) after which this step is complete. Only the final step may omit it, which runs that step to the end of the survey; every earlier step must name an item. The same item must not be named by two steps. **survey.branding** `dict` The author's brand on the respondent page. Omit or set to `null` for no banner, which is how a survey without this field renders. **properties** **logo** `dict` The logo drawn in the survey banner, above the questions. Set to `null` to remove it. Upload the image first, with `Coop().upload_human_survey_asset("lab_logo.png")` or `ep humanize assets upload lab_logo.png`, and reference the uuid it returns. A schema never carries image bytes. There is no size field: the banner picks a default height. Style `.edsl-logo` in `custom_css` to change it. See [Custom CSS](/en/latest/custom_css). **properties** **source** `dict` *required* Where the image comes from. Must be `{"type": "asset", "asset_uuid": ""}`, naming an image in your asset library. **alt** `string` *required* Alt text for screen readers, up to 200 characters. Required, so that leaving it out is a decision rather than an accident. Use an empty string to mark the logo decorative, which renders it with `alt=""`. **position** `string` (default: left) Where the logo sits in the banner: `"left"`, `"center"`, or `"right"`. ## HumanizeQuestionSchema (per question) Each `questions[question_name]` value is a `HumanizeQuestionSchema` object. **optional** `boolean` (default: false) Whether the question is optional. Default is `false` when omitted. **Supported question types:** `free_text`, `budget`, `checkbox`, `checkbox_with_other`, `file_upload`, `interview`, `likert_five`, `linear_scale`, `list`, `matrix`, `multiple_choice`, `multiple_choice_with_other`, `numerical`, `rank`, `top_k`, `yes_no`. Note: `optional` currently has no effect for `interview` and `rank`. **time\_limit** `dict` A limit on how long the respondent may spend answering the question. Omit or set to `null` for no limit. While the limit runs, a countdown is shown above the question. When it reaches zero the answer locks: the respondent can no longer change it and clicks **Next** to continue. They are not moved on automatically. Whatever they had entered is submitted as their answer, and a question they had not answered is recorded as unanswered, even if it is not `optional`. A time limit applies only while the question is alone on its page. When `survey.presentation` is `"group"`, every question's `time_limit` is ignored, including a question that is the only one in its group. The countdown runs in the respondent's browser, so reloading the page starts it again. **Supported question types:** `free_text`, `budget`, `checkbox`, `checkbox_with_other`, `file_upload`, `likert_five`, `linear_scale`, `list`, `matrix`, `multiple_choice`, `multiple_choice_with_other`, `numerical`, `rank`, `top_k`, `yes_no`. **properties** **duration** `dict` *required* How long the respondent has. Must be `{"type": "fixed", "seconds": }`, the same number of seconds for every respondent. `seconds` must be between 30 and 7200 (two hours). `type` is required, even though `"fixed"` is its only value. **format** `dict` Display format. Varies by question type: * **MC-style** (`likert_five`, `linear_scale`, `multiple_choice`, `yes_no`): `"radio"` (list of radio buttons) or `"dropdown"` (single `