Importing data

Importing data turns material you already have—such as spreadsheet rows, interview excerpts, or respondent profiles—into objects you can use in an EDSL survey. You do not need to copy each record into a separate question. Instead, write one question with a placeholder and let EDSL fill it with a different record for each interview. The first choice is what the imported data represents. Use scenarios for material a model will read or evaluate, such as a customer message. Use agents for the traits of the respondent the model should portray, such as their experience with a product. A file containing the questions themselves is a third case, covered under Existing questionnaires.

Import messages and respondent profiles

The example creates two small CSV files so you can try the imports without downloading a dataset. messages.csv has one message per row; profiles.csv has one respondent profile per row. The last lines read them into a ScenarioList and an AgentList. If you already have CSV files, start with those import calls and substitute your filenames.

import csv
from pathlib import Path
from edsl import AgentList, ScenarioList

def write_fixture(path, rows):
    with Path(path).open("w", newline="", encoding="utf-8") as handle:
        writer = csv.DictWriter(handle, fieldnames=list(rows[0]))
        writer.writeheader()
        writer.writerows(rows)

write_fixture("messages.csv", [
    {"record_id": "m1", "message": "Your report is ready to download."},
    {"record_id": "m2", "message": "Download your report with the Export button."},
])
write_fixture("profiles.csv", [
    {"profile_id": "p1", "experience": "new user"},
    {"profile_id": "p2", "experience": "frequent user"},
])

records = ScenarioList.from_source("csv", "messages.csv")
agents = AgentList.from_source("csv", "profiles.csv", name_field="profile_id")
assert len(records) == 2
assert len(agents) == 2

Each row of messages.csv becomes one Scenario in records. Its column names become fields: record_id identifies the original row, and message supplies text to a question. Each row of profiles.csv becomes one Agent; name_field="profile_id" uses the identifier as the agent’s name, while the experience field describes the profile. Importing profiles prepares agents for model responses; it does not collect answers from the people described in the file.

Use an imported column in a question

The following question evaluates the messages from the CSV. The placeholder {{ scenario.message }} reads the message field from each scenario. Attaching both lists asks each profile to evaluate each message:

from edsl import QuestionMultipleChoice

question = QuestionMultipleChoice(
    question_name="clarity",
    question_text="How clear is this message: {{ scenario.message }}?",
    question_options=["Clear", "Unclear", "Unsure"],
)
job = question.by(records).by(agents)

# With model access configured, collect and display the ratings.
results = job.run()
results.select("scenario.record_id", "agent.agent_name", "answer.clarity")

Two messages times two profiles gives four result rows for one model and iteration. The identifiers let you match each rating back to its message and profile. Calling by() crosses the lists; it does not pair their rows by position. If each profile should receive only its corresponding record, use zip_assign() as described in Experimental conditions. If the task needs no custom profiles, use question.by(records) alone.

Other Python input formats

The same approach works when your data comes from a spreadsheet, database, or Python object. ScenarioList.from_source() reads the source into scenarios so the question and job code can stay the same. For tabular data, each row becomes a scenario:

Source	Python call
CSV or TSV	`ScenarioList.from_source("csv", "records.csv")` or `from_source("tsv", "records.tsv")`
Excel sheet	`ScenarioList.from_source("excel", "records.xlsx", sheet_name="Responses")`
pandas DataFrame	`ScenarioList.from_source("pandas", frame)`
Dictionary of columns	`ScenarioList.from_source("dict", {"record_id": ["r1", "r2"], "text": ["A", "B"]})`
List of row dictionaries	`ScenarioList.from_list_of_dicts([{"record_id": "r1", "text": "A"}])`
SQLite table	`ScenarioList.from_source("sqlite", "records.db", table="responses")`
Parquet or Stata file	`ScenarioList.from_source("parquet", "records.parquet")` or `from_source("dta", "records.dta")`

from_source() normalizes column names to snake case by default. Pass snakify=False to preserve the source names. Some readers require optional format-specific dependencies. ScenarioList.from_jsonl("scenarios.jsonl") restores EDSL’s serialized JSONL representation; it is not a general JSON row reader. For arbitrary JSON records, read them with Python’s json module and use from_list_of_dicts(). AgentList.from_source() supports the same source selection and accepts instructions, codebook, and name_field for profile imports. Instructions tell agents how to respond; a codebook explains what their trait fields mean. AgentList.from_csv() is a direct CSV alternative. To turn rows you have already prepared into profiles, use AgentList.from_scenario_list(records) or records.to_agent_list(). Choose the import method based on where your data already lives. For example, use the pandas source if you have cleaned a DataFrame in a notebook; there is no need to save it to CSV and read it back. See Scenarios for selecting, renaming, filtering, and transforming fields after import.

Documents and web content

For documents, decide what one interview should receive: the text of a whole document, one page at a time, or the original file with its visual content. This choice determines both the scenario you construct and what the model can evaluate. Extracted text is useful for analyzing wording; the original PDF or page images are useful when charts, layout, or scanned content matter. For documents, Scenario.from_pdf("report.pdf") and Scenario.from_docx("report.docx") extract text into one Scenario. ScenarioList.from_source("pdf", "report.pdf") extracts one scenario per PDF page. The PDF text examples show the resulting fields, page filtering, and how to run a question over the pages. To send the actual file to a model, use Scenario.from_file(path, field_name="document") or put a FileStore in a Scenario. See the worked examples for whole PDFs, images, video, and audio. For scanned pages and visual analysis, render PDF pages as images. Scenario.from_url(url) retrieves one URL, Scenario.from_html(url) retains HTML and extracted text, and ScenarioList.from_urls(urls) retrieves several. These make network requests. ScenarioList.from_source("directory", "materials", recursive=True) creates file-backed scenarios from a directory; see A collection of files for filtering file types and retaining filenames with answers. Keep filenames, page numbers, or source URLs as scenario fields when you want them alongside the answers. These fields make it possible to trace a finding back to the source passage or file. FileStore explains how EDSL represents file attachments; the Scenarios examples show how to reference them in questions.

Existing questionnaires

If your source contains questions to ask rather than records to analyze, you want to create a Survey. Builder supports questionnaire creation and import through a visual interface. Importing an existing questionnaire requires its question types, wording, options, and routing to be represented in the resulting survey. If your ScenarioList contains EDSL question definitions, definitions.to_survey() converts each row into a question. Supply fields such as question_type, question_name, question_text, and the parameters required by that type. For example:

from edsl import ScenarioList

definitions = ScenarioList.from_list_of_dicts([
    {"question_type": "free_text", "question_name": "feedback", "question_text": "What should improve?"},
    {"question_type": "multiple_choice", "question_name": "priority",
     "question_text": "Which matters most?", "question_options": ["Speed", "Reliability"]},
])
survey = definitions.to_survey()

Here each row defines a question, so the result is a two-question Survey rather than two interviews about source records. The questions follow the row order; add any branching or memory with the Survey methods. Use this route when question definitions already live in a table, or construct Question objects directly when writing a survey in Python.

Introduction

Setup and tutorials

EDSL objects

How-to guides

Research examples

Human research

Execution and costs

Saving and sharing

Research and teaching

Expected Parrot platform

Contributing

Import messages and respondent profiles

Use an imported column in a question

Other Python input formats

Documents and web content

Existing questionnaires

​Import messages and respondent profiles

​Use an imported column in a question

​Other Python input formats

​Documents and web content

​Existing questionnaires

Import messages and respondent profiles

Use an imported column in a question

Other Python input formats

Documents and web content

Existing questionnaires