> ## Documentation Index
> Fetch the complete documentation index at: https://docs.expectedparrot.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data cleaning

> This notebook provides sample [EDSL](/en/latest/index) code for using a language model to conduct a data cleaning task. In a series of steps we use EDSL to prompt a language model to generate appropriate sense checks for a dataset and then run the sense checks in the form of a survey about the data, returning a new dataset consisting of the data failing the checks.

[EDSL is an open-source library](https://github.com/expectedparrot/edsl) for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below please see instructions on [getting started](https://www.expectedparrot.com/getting-started) and tips and tutorials at our [documentation page](/en/latest/index).

## Example data

EDSL allows us to generate data or [import it from other sources](/en/latest/scenarios) (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). Here we construct a dataset for our exercise: a random list of ages between 22 and 85 with some bad values mixed in. Our goal is to identify them:

```python expandable theme={null}
ages = [84, 62, 79, 57, 59, 55, 68, 66, 47, 54, 76, 33, 74, 56, 47, 24, 23, 38, 38, 54, 51, 84, 71,
        46, 38, 26, 50, 56, 62, 39, 31, 52, 69, 84, 69, 48, 48, 23, 65, 54, 78, 51, 69, 77, 75, 76,
        26, 44, 61, 32, 70, 24, 74, 22, 32, 24, 80, 65, 36, 42, 84, 66, 40, 85, 28, 22, 67, 25, 70,
        77, 53, 69, 64, 27, 61, 68, 68, 78, 0.99, 83, 58, 33, 46, 43, 50, 85, 28, 82, 50, 61, 66, 32,
        45, 70, 56, 50, 43, 30, 43, 55, 33, 72, 43, 43, -5, 32, 43, 45, 67, 84, 37, 63, 52, 53, 58,
        79, 79, 80, 62, 75, 57, 60, 39, 79, 49, 60, 60, 37, 45, 36, 1050, 73, 70, 56, 39, 58, 69, 77,
        68, 84, 78, 48, 31, 74, 27, 55, 56, 66, 35, 39, 57, 47, 29, 24, 47, 60, 43, 37, 84, 64, 28,
        22, 37, 71, 77, 76, 84, 63, 76, 58, 41, 72, 22, 63, 78, 49, 82, 69, "old", 37, 27, 29, 54, 83,
        80, 74, 48, 76, 49, 26, 38, 35, 36, 25, 23, 71, 33, 39, 40, 35, 85, 24, 57, 85, 63, 53, 62,
        47, 69, 76, 71, 48, 62, 23, 25, 84, 32, 63, 75, 31, 25, 50, 85, 36, 58, 85, 34, 62, 43, 2,
        50, 83, 44, 73, 81, 44, 43, 82, 84, 30, 24, 63, 63, 59, 46, 30, 62, 25, 52, 23, 100, 1.3, 3]
```

## Quick question

With a small dataset, we may be able to design the entire task as a single question where we prompt a model to review all the data at once and flag bad data:

```python expandable theme={null}
from edsl import QuestionList, Scenario, Model

q = QuestionList(
    question_name = "bad_ages",
    question_text = """
    Review the following list of observations of human ages
    and return a list of all the unrealistic ages: {{ scenario.ages }}
    """
)

s = Scenario({"ages":ages})

m = Model("gemini-1.5-flash", service_name = "google")

results = q.by(s).by(m).run()

results.select("bad_ages", "bad_ages_comment")
```

|    | answer.bad\_ages              | comment.bad\_ages\_comment                                                                                                                                              |
| :- | :---------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0  | \[0.99, -5, 1050, 1.3, 'old'] | # These values are not realistic human ages because they are either negative, less than 1, or far exceed the maximum human lifespan. "old" is also not a numerical age. |

This approach may be feasible for a small dataset that is easily checked. For larger datasets, we may encounter problems with input token limits, a model’s ability to accurately check a large volume of data at once, and responses that are not usefully formatted.

Below we demonstrate some ways of approaching the task in an iterative manner instead.

## Constructing a question

We start by creating a question to prompt a model to draft sense check questions for our data. EDSL comes with a [variety of question types](/en/latest/questions) that we can choose from based on the desired form of the response (multiple choice, free text, etc.). Here we use `QuestionList` in order to prompt the model to format its response as a list. We use a `{{ placeholder }}` for content that we will add to the question when we run it (a description of the data and a sample); this allows us to re-use the question with other contexts as desired:

```python theme={null}
from edsl import QuestionList

q = QuestionList(
    question_name = "sense_check_questions",
    question_text = """
    You are being asked to suggest sense checks for a dataset consisting of {{ scenario.data_description }}.
    Here is a small sample of the data (to demonstrate the format): {{ scenario.sample_data }}.
    Return the sense checks as a list of questions to be answered about each item in the dataset individually.
    """,
    max_list_items = 3 # optional
)
```

## Adding context to the question

Next we create `Scenario` objects representing the content that we want to add to the question when we run it. Here we create a single scenario for our example data:

```python theme={null}
import random

sample_data = random.sample(ages, 10)
```

```python theme={null}
from edsl import Scenario

s = Scenario({
    "data_description": "a list of realistic human ages (in years)",
    "sample_data": sample_data
})
s
```

```python theme={null}
Scenario
```

|    | key               | value                                     |
| :- | :---------------- | :---------------------------------------- |
| 0  | data\_description | a list of realistic human ages (in years) |
| 1  | sample\_data:0    | 54                                        |
| 2  | sample\_data:1    | 48                                        |
| 3  | sample\_data:2    | 57                                        |
| 4  | sample\_data:3    | 43                                        |
| 5  | sample\_data:4    | 80                                        |
| 6  | sample\_data:5    | 58                                        |
| 7  | sample\_data:6    | 57                                        |
| 8  | sample\_data:7    | 57                                        |
| 9  | sample\_data:8    | 23                                        |
| 10 | sample\_data:9    | 72                                        |

## Running the question

We administer the question to a model by adding the scenarios and calling the `run` method. This generates a formatted dataset of `Results` that we can access with [built-in methods for analysis](/en/latest/results). Here we inspect the answer:

```python theme={null}
results = q.by(s).by(m).run()
```

```python theme={null}
results.select("sense_check_questions")
```

|    | answer.sense\_check\_questions                                                                                                                                                                                          |
| :- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 0  | \['Is the age a non-negative integer?', 'Is the age within a plausible range for a living human (e.g., 0-125)?', 'Is the age consistent with other ages in the dataset (considering potential context, if available)?'] |

## Conducting the task

Next we want a model to answer each sense check question about each piece of data in the dataset. This can be done by using the sense check questions as scenarios of a new question explaining the task. We can use `QuestionYesNo` to easily filter the responses:

```python theme={null}
from edsl import QuestionYesNo

q2 = QuestionYesNo(
    question_name = "check_data",
    question_text = """
    You are being asked to sense check a dataset consisting of {{ scenario.data_description }}.
    Consider the following item in the dataset: {{ scenario.age }}
    {{ scenario.sense_check_question }}
    """
)
```

We need to create a new set of scenarios for the question. We use `ScenarioList` objects to create all the combinations of values to add to the question (learn more about [constructing scenarios from different data sources](/en/latest/scenarios)):

```python theme={null}
from edsl import ScenarioList

sl = ScenarioList(
    Scenario({
        "data_description": "a list of realistic human ages (in years)",
        "age": age,
        "sense_check_question": sense_check_question
    }) for age in ages for sense_check_question in results.select("sense_check_questions").to_list()[0]
)
```

We can inspect the scenarios that we created:

```python theme={null}
sl.sample(3)
```

[ScenarioList](/en/latest/scenarios) scenarios: 3; keys: \['sense\_check\_question', 'age', 'data\_description'];

|    | data\_description                         | age | sense\_check\_question                                                                              |
| :- | :---------------------------------------- | :-- | :-------------------------------------------------------------------------------------------------- |
| 0  | a list of realistic human ages (in years) | 22  | Is the age a non-negative integer?                                                                  |
| 1  | a list of realistic human ages (in years) | 58  | Is the age consistent with other ages in the dataset (considering potential context, if available)? |
| 2  | a list of realistic human ages (in years) | 38  | Is the age a non-negative integer?                                                                  |

Same as with a single scenario, we add all the scenarios to the question at once when we run it:

```python theme={null}
results = q2.by(sl).by(m).run()
```

We can filter, sort, select and print any components of the results that are generated:

```python theme={null}
(
    results
    .filter("check_data == 'No'")
    .sort_by("sense_check_question")
    .select("sense_check_question", "age")
)
```

|    | scenario.sense\_check\_question                                                                     | scenario.age |
| :- | :-------------------------------------------------------------------------------------------------- | :----------- |
| 0  | Is the age a non-negative integer?                                                                  | 0.99         |
| 1  | Is the age a non-negative integer?                                                                  | -5           |
| 2  | Is the age a non-negative integer?                                                                  | 1050         |
| 3  | Is the age a non-negative integer?                                                                  | old          |
| 4  | Is the age a non-negative integer?                                                                  | 1.3          |
| 5  | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 0.99         |
| 6  | Is the age consistent with other ages in the dataset (considering potential context, if available)? | -5           |
| 7  | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 1050         |
| 8  | Is the age consistent with other ages in the dataset (considering potential context, if available)? | old          |
| 9  | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 2            |
| 10 | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 100          |
| 11 | Is the age consistent with other ages in the dataset (considering potential context, if available)? | 1.3          |
| 12 | Is the age within a plausible range for a living human (e.g., 0-125)?                               | -5           |
| 13 | Is the age within a plausible range for a living human (e.g., 0-125)?                               | 1050         |
| 14 | Is the age within a plausible range for a living human (e.g., 0-125)?                               | old          |
| 15 | Is the age within a plausible range for a living human (e.g., 0-125)?                               | 1.3          |

## Further exploration

This notebook can be readily edited and expanded for other data cleaning and data labeling purposes, or to add personas for AI agents answering the questions with relevant background and expertise. [Learn more about using AI agents for your EDSL surveys](/en/latest/agents).

Please see our [documentation page](/en/latest/index) for examples of other methods and use cases and let us know if you have any questions!

## Posting to Expected Parrot

[Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop).

Here we post this notebook:

```python theme={null}
# from edsl import Notebook

# nb = Notebook(path = "data_cleaning.ipynb")

# nb.push(
#     description = "Example code for data cleaning",
#     alias = "data-cleaning-notebook",
#     visibility = "public"
# )
```

To update an object at Expected Parrot:

```python theme={null}
from edsl import Notebook

nb = Notebook(path = "data_cleaning.ipynb")

nb.patch("https://www.expectedparrot.com/content/RobinHorton/data-cleaning-notebook", value = nb)
```