Skip to main content

Overview

Dataset is a fundamental data structure in EDSL that provides a column-oriented representation of tabular data. It offers methods for manipulating, analyzing, visualizing, and exporting data, similar to tools like pandas or dplyr. Key features:
  1. Flexible data manipulation (filtering, sorting, transformation)
  2. Visualization capabilities with multiple rendering options
  3. Export to various formats (CSV, Excel, Pandas, etc.)
  4. Integration with other EDSL components

Creating Datasets

Datasets can be created from various sources: From dictionaries:
From existing EDSL objects:

Displaying and Visualizing Data

The Dataset class provides multiple ways to display and visualize data: Basic display:

Table Display Options

You can control the table formatting using the tablefmt parameter:

Rich Terminal Output

The Dataset class supports displaying tables with enhanced formatting using the Rich library, which provides beautiful terminal formatting with colors, styles, and more:
This creates a nicely formatted table in the terminal with automatically sized columns, bold headers, and grid lines. Example:

Data Manipulation

The Dataset class provides numerous methods for data manipulation: Filtering:
Creating new columns:
Sorting:
Reshaping:

Exporting Data

Export to various formats:

Dataset Methods

class edsl.dataset.Dataset(data: list [dict [str, Any ] ] = None, print _parameters: dict | None = None) [source ]

Bases: UserList, DatasetOperationsMixin, PersistenceMixin, HashingMixin A versatile data container for tabular data with powerful manipulation capabilities. The Dataset class is a fundamental data structure in EDSL that represents tabular data in a column-oriented format. It provides a rich set of methods for data manipulation, transformation, analysis, visualization, and export through the DatasetOperationsMixin. Key features:
  1. Column-oriented data structure optimized for LLM experiment results
  2. Rich data manipulation API similar to dplyr/pandas (filter, select, mutate, etc.)
  3. Visualization capabilities including tables, plots, and reports
  4. Export to various formats (CSV, Excel, SQLite, pandas, etc.)
  5. Serialization for storage and transport
  6. Tree-based data exploration
A Dataset typically contains multiple columns, each represented as a dictionary with a single key-value pair. The key is the column name and the value is a list of values for that column. All columns must have the same length. The Dataset class inherits from: - UserList: Provides list-like behavior for storing column data - DatasetOperationsMixin: Provides data manipulation methods - PersistenceMixin: Provides serialization capabilities - HashingMixin: Provides hashing functionality for comparison and storage Datasets are typically created by transforming other EDSL container types like Results, AgentList, or ScenarioList, but can also be created directly from data.

collapse(field: str, separator: str | None = None) → Dataset [source ]

Collapse multiple values in a field into a single value using a separator.
Args:
field: The name of the field to collapse. separator: Optional string to use as a separator between values.
Defaults to a space if not specified.
Examples:

drop(field _name) [source ]

Returns a new Dataset with the specified field removed.
Args:
field _name (str): The name of the field to remove.
Returns:
Dataset: A new Dataset instance without the specified field.
Raises:
KeyError: If the field _name doesn’t exist in the dataset.
Examples:

classmethod example(n: int = None) → Dataset [source ]

Return an example dataset.
Examples:

expand(field: str, number _field: bool = False) → Dataset [source ]

Expand a field containing lists into multiple rows.
Args:
field: The field containing lists to expand number _field: If True, adds a number field indicating the position in the original list
Returns:
A new Dataset with the expanded rows
Examples:

expand _field(field) [source ]

Expand a field in the dataset. Renamed to avoid conflict with the expand method defined earlier.

filter(expression) → Dataset [source ]

Filter the dataset based on a boolean expression.
Args:
expression: A string expression that evaluates to a boolean value. Can reference column names in the dataset.
Examples:

first() → dict [str, Any ] [source ]

Get the first value of the first key in the first dictionary.
Examples:

classmethod from _dict(data: dict) → Dataset [source ]

Convert a dictionary to a dataset.
Examples:

classmethod from _edsl _object(object) [source ]

classmethod from _pandas _dataframe(df) [source ]

get _sort _indices(lst: list [Any ], reverse: bool = False) → list [int ] [source ]

Return the indices that would sort the list, using either numpy or pure Python. None values are placed at the end of the sorted list.
Args:
lst: The list to be sorted reverse: Whether to sort in descending order use _numpy: Whether to use numpy implementation (falls back to pure Python if numpy is unavailable)
Returns:
A list of indices that would sort the list

head(n: int = 5) → Dataset [source ]

Return the first n observations in the dataset.

keys() → list [str ] [source ]

Return the keys of the dataset.
Examples:

latex(* * kwargs) [source ]

Return a LaTeX representation of the dataset.
Args:
* *kwargs: Additional arguments to pass to the table formatter.

long(exclude _fields: list [str ] = None) → Dataset [source ]

Convert the dataset from wide to long format.
Examples:

merge(other: Dataset, by _x, by _y) → Dataset [source ]

Merge the dataset with another dataset on the given keys.
Examples:

order _by(sort _key: str, reverse: bool = False) → Dataset [source ]

Return a new dataset with the observations sorted by the given key.
Args:
sort _key: The key to sort the observations by reverse: Whether to sort in reverse order
Examples:
Print the dataset in a formatted way.
Args:
pretty _labels: A dictionary mapping column names to their display names * *kwargs: Additional arguments format: The output format (“html”, “markdown”, “rich”, “latex”)
Returns:
TableDisplay object
Examples:

remove _prefix() → Dataset [source ]

Remove the prefix from column names that contain dots.
Examples:

rename(rename _dic) → Dataset [source ]

Rename columns in the dataset according to the provided dictionary.
Examples:

sample(n: int = None, frac: float = None, with _replacement: bool = True, seed: str | int | float = None) → Dataset [source ]

Return a new dataset with a sample of the observations.
Examples:

select(* keys) → Dataset [source ]

Return a new dataset with only the selected keys.
Examples:

shuffle(seed =None) → Dataset [source ]

Return a new dataset with the observations shuffled.
Examples:

summary() → Dataset [source ]

Return a summary of the dataset.
Examples:

table(* fields, tablefmt: str | None = ‘rich’, max _rows: int | None = None, pretty _labels =None, print _parameters: dict | None = None) [source ]

tail(n: int = 5) → Dataset [source ]

Return the last n observations in the dataset.

to(survey _or _question: ‘Survey’ | ‘QuestionBase’) → Job [source ]

Transform the dataset using a survey or question.
Args:
survey _or _question: Either a Survey or QuestionBase object to apply to the dataset.
Examples:

to _dict() → dict [source ]

Convert the dataset to a dictionary.
Examples:

to _docx(output _file: str, title: str = None) → None [source ]

Convert the dataset to a Word document.
Args:
output _file (str): Path to save the Word document title (str, optional): Title for the document
Examples:

to _json() [source ]

Return a JSON representation of the dataset.
Examples:

tree(node _order: list [str ] | None = None) → Tree [source ]

Return a tree representation of the dataset.

unique() → Dataset [source ]

Remove duplicate rows from the dataset.
Returns:
A new Dataset with duplicate rows removed.
Examples:

wide() → Dataset [source ]

Convert a long-format dataset (with row, key, value columns) to wide format.
Examples:

write(filename: str, tablefmt: str | None = None) → None [source ]

Write the dataset to a file in the specified format.
Args:
filename: The name of the file to write to. tablefmt: Optional format for the table (e.g., ‘csv’, ‘html’, ‘latex’).