Dataset

A Dataset is a table of columns. A Results object retains interview and study information; calling results.select(...) produces a Dataset containing the selected columns. Use Dataset methods or pandas to transform and export that table. For example, select a treatment label and a rating from Results, filter to the observations you want, then export a CSV for your statistical software. You can do these steps directly in EDSL; converting with .to_pandas() is useful when you want pandas analyses or plotting. A ScenarioList is the corresponding collection of named records used as inputs to questions.

A complete local example

This example uses four fabricated scores and makes no model calls. It creates a Dataset from a DataFrame, filters and sorts rows, adds a derived column, and writes a CSV.

import pandas as pd
from edsl import Dataset

frame = pd.DataFrame({
    "condition": ["short", "explicit", "short", "explicit"],
    "score": [3, 5, 2, 4],
})
dataset = Dataset.from_pandas_dataframe(frame)
selected = dataset.filter("score >= 4").order_by("score", reverse=True)
with_flag = dataset.mutate("high_score = score >= 4")
# Display in a notebook: put selected on the last line of a cell.
# Display in a script: selected.print()
# Convert to pandas when you need DataFrame operations.
analysis_frame = selected.to_pandas()
selected.to_csv("selected_scores.csv")

analysis_frame contains the two explicit rows with scores 5 and 4, in that order. with_flag contains all four rows plus the Boolean high_score column. selected_scores.csv contains only the selected rows.

Constructors and conversions

Use Dataset.from_pandas_dataframe(frame) for a pandas DataFrame. You can also construct a column-oriented table directly:

from edsl import Dataset

dataset = Dataset([{"record_id": ["r1", "r2"]}, {"score": [3, 5]}])
frame = dataset.to_pandas()

Keep columns the same length. to_pandas() preserves prefixed EDSL column names by default. Use qualified names when joining data so unrelated columns are not confused.

Common methods

Call	Arguments and result
`dataset.select("condition", "score")`	Return a Dataset with the selected columns.
`dataset.filter("score >= 4")`	Return rows matching a Boolean expression.
`dataset.order_by("score", reverse=True)`	Return rows sorted by the named column; descending when `reverse=True`.
`dataset.mutate("high_score = score >= 4")`	Return a Dataset with a new or updated column computed from the expression.
`dataset.to_list()`	Return a list of values for one column, or a list of row tuples for multiple columns.
`dataset.to_pandas(remove_prefix=False)`	Return a pandas DataFrame; set `remove_prefix=True` to strip namespaces such as `answer.`.
`dataset.to_csv("scores.csv")`	Write a CSV file and return a FileStore containing the CSV. Column prefixes are retained by default.
`dataset.print()`	Display the table explicitly in a script. A final notebook expression displays it automatically.

This example filters a table and converts a single selected column to a Python list:

from edsl import Dataset

scores = Dataset([{"score": [3, 5, 2, 4]}])
high_scores = scores.filter("score >= 4").order_by("score", reverse=True)
high_scores
# In a script: high_scores.print()
# For further processing: high_scores.select("score").to_list() returns [5, 4].

Conversion is optional when you only want to inspect the table. See Results for selecting columns from a completed study.

Reshape and inspect a table

Task	Methods and behavior
Preview rows	`head(n=5)`, `tail(n=5)`, or `table(max_rows=5)` returns a smaller table. `first()` returns the first value in the first column.
Inspect structure	`keys()` lists column names; `summary()` describes the table.
Rename or remove columns	`rename({"old": "new"})`, `drop("column")`, and `remove_prefix()` return new Datasets with renamed, removed, or unprefixed columns. Check for collisions before removing prefixes.
Sample rows	`sample(n=2, with_replacement=False, seed=42)` returns a sample; `shuffle(seed=42)` returns a shuffled table. Dataset sampling defaults to replacement, so choose explicitly.
Deduplicate	`unique()` returns distinct rows.
Reshape lists	`expand("tags")` creates rows from list items; `collapse("text", separator="\n")` combines values while grouping by the other columns.
Reshape all columns	`long()` produces `row`, `key`, and `value` columns; `wide()` reconstructs that layout.
Join tables	`merge(other, by_x="record_id", by_y="id")` performs a left join. Check for duplicate keys first.
Format output	`to_markdown_table()` returns a Markdown table string; `latex()` returns a LaTeX table. `print()` explicitly displays the table in scripts.

For example, inspect the long representation of a small table:

from edsl import Dataset

table = Dataset([{"record_id": ["r1", "r2"]}, {"score": [3, 5]}])
long_table = table.long()
long_table
# long_table.wide() reconstructs the original columns.

Export and convert tables

These methods also apply to Results, ScenarioList, and AgentList. Select the fields you want before exporting:

Output	Call
Row dictionaries	`table.to_dicts(remove_prefix=False)`
Python values	`table.select("score").to_list()` returns the values of one column. Multiple columns produce row tuples.
pandas or Polars	`table.to_pandas()` / `table.to_polars()`; prefixes are retained by default.
CSV	`table.to_csv("scores.csv")`
Excel	`table.to_excel("scores.xlsx", sheet_name="Scores")`
Word table	`table.to_docx("scores.docx")`
EDSL inputs	`table.to_scenario_list()` / `table.to_agent_list()`; these remove column prefixes by default.

The CSV, Excel, and Word export methods above return FileStore objects. table.write("scores.csv") writes a file directly, choosing its format from the filename extension. table.to_json() returns a Python list of column dictionaries. Use json.dumps() when you need JSON text:

import json

json.dumps(table.to_json())

This differs from table.to_dict(), which returns the EDSL object structure that Dataset.from_dict() can reconstruct. Install the corresponding optional library for conversions such as to_polars() or Excel output. For database exports, convert to a pandas DataFrame and use its database-writing methods with your chosen connection and replacement policy.

Embed text

Text embeddings represent passages as lists of numbers that can be used to search for similar passages or group related responses. EmbeddingModel creates these representations; Dataset.embed() and ScenarioList.embed() add them as a column beside the source text. Embedding the text does not itself assign themes or create groups. Use qualitative coding when you want named themes from a codebook. This example uses fixed test vectors to show the table structure without a provider call. Their values do not represent semantic similarity:

from edsl import Dataset, EmbeddingCache, EmbeddingModel

texts = Dataset([{"text": ["The tool is easy to use.", "Exporting takes too long."]}])
embedding_model = EmbeddingModel(service_name="test", dimensions=4)
embedding_cache = EmbeddingCache()
embedded = texts.embed("text", model=embedding_model, cache=embedding_cache)
embedded

For actual text embeddings, select an embedding model and service explicitly: EmbeddingModel(model_name="EMBEDDING_MODEL_NAME", service_name="openai") uses direct provider access by default. Configure the provider’s credentials. The sentence_transformers service runs locally with its dependencies and model weights installed; loading a model may download weights. remote=True routes supported embedding requests through Expected Parrot. Embedding models are separate from the language models selected with Model(). For text outside a Dataset, embedding_model.embed(["first text", "second text"], cache=embedding_cache) returns an EmbeddingResult. Its embeddings, input, model, service_name, usage, and cache_used attributes retain the vectors and available metadata. to_dataset() and to_scenario_list() convert it to EDSL tables; to_dict() and EmbeddingResult.from_dict(data) serialize and restore it. In asynchronous Python, use await embedding_model.async_embed(texts, cache=embedding_cache). EmbeddingCache(filename="embeddings.jsonl") loads existing embedding entries and writes new ones to that file. With immediate_write=False, call flush() to persist pending entries. This cache is separate from the response Cache used by survey runs. Embeddings and provider calls can incur costs; the test service above makes no paid calls.

Introduction

Setup and tutorials

EDSL objects

How-to guides

Research examples

Human research

Execution and costs

Saving and sharing

Research and teaching

Expected Parrot platform

Contributing

A complete local example

Constructors and conversions

Common methods

Reshape and inspect a table

Export and convert tables

Embed text

​A complete local example

​Constructors and conversions

​Common methods

​Reshape and inspect a table

​Export and convert tables

​Embed text

A complete local example

Constructors and conversions

Common methods

Reshape and inspect a table

Export and convert tables

Embed text