Data cleaning

This notebook provides sample EDSL code for using a language model to conduct a data cleaning task. In a series of steps we use EDSL to prompt a language model to generate appropriate sense checks for a dataset and then run the sense checks in the form of a survey about the data, returning a new dataset consisting of the data failing the checks.

EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Example data

Here we construct a dataset for our exercise: a random list of ages between 22 and 85 with some bad values mixed in. Our goal is to identify them:

[1]:

ages = [84, 62, 79, 57, 59, 55, 68, 66, 47, 54, 76, 33, 74, 56, 47, 24, 23, 38, 38, 54, 51, 84, 71,
        46, 38, 26, 50, 56, 62, 39, 31, 52, 69, 84, 69, 48, 48, 23, 65, 54, 78, 51, 69, 77, 75, 76,
        26, 44, 61, 32, 70, 24, 74, 22, 32, 24, 80, 65, 36, 42, 84, 66, 40, 85, 28, 22, 67, 25, 70,
        77, 53, 69, 64, 27, 61, 68, 68, 78, 0.99, 83, 58, 33, 46, 43, 50, 85, 28, 82, 50, 61, 66, 32,
        45, 70, 56, 50, 43, 30, 43, 55, 33, 72, 43, 43, -5, 32, 43, 45, 67, 84, 37, 63, 52, 53, 58,
        79, 79, 80, 62, 75, 57, 60, 39, 79, 49, 60, 60, 37, 45, 36, 1050, 73, 70, 56, 39, 58, 69, 77,
        68, 84, 78, 48, 31, 74, 27, 55, 56, 66, 35, 39, 57, 47, 29, 24, 47, 60, 43, 37, 84, 64, 28,
        22, 37, 71, 77, 76, 84, 63, 76, 58, 41, 72, 22, 63, 78, 49, 82, 69, "old", 37, 27, 29, 54, 83,
        80, 74, 48, 76, 49, 26, 38, 35, 36, 25, 23, 71, 33, 39, 40, 35, 85, 24, 57, 85, 63, 53, 62,
        47, 69, 76, 71, 48, 62, 23, 25, 84, 32, 63, 75, 31, 25, 50, 85, 36, 58, 85, 34, 62, 43, 2,
        50, 83, 44, 73, 81, 44, 43, 82, 84, 30, 24, 63, 63, 59, 46, 30, 62, 25, 52, 23, 100, 1.3, 3]

Quick question

With a small dataset, we may be able to design the entire task as a single question where we prompt a model to review all the data at once and flag bad data:

[2]:

from edsl import QuestionList, Scenario

q = QuestionList(
    question_name = "bad_ages",
    question_text = """
    Review the following list of observations of human ages
    and return a list of all the problematic ages: {{ scenario.ages }}
    """
)

s = Scenario({"ages":ages})

results = q.by(s).run()

results.select("bad_ages", "bad_ages_comment")

▼ Job Status (2025-03-03 09:08:45)

Job UUID	3daaed84-62da-466b-92b2-5e746e097a14
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/3daaed84-62da-466b-92b2-5e746e097a14
Exceptions Report URL	None
Results UUID	b2052857-410d-4ed0-b932-d7c1323fa2fe
Results URL	https://www.expectedparrot.com/content/b2052857-410d-4ed0-b932-d7c1323fa2fe

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/b2052857-410d-4ed0-b932-d7c1323fa2fe

[2]:

	answer.bad_ages	comment.bad_ages_comment
0	[0.99, -5, 1050, 'old', 1.3]	# Problematic ages include non-integer values like 0.99 and 1.3, negative values like -5, an excessively high value like 1050, and a non-numeric string "old".

This approach may be feasible for a small dataset that is easily checked. For larger datasets, we may encounter problems with input token limits, a model’s ability to accurately check a large volume of data at once, and responses that are not usefully formatted.

Below we demonstrate some ways of approaching the task in an iterative manner instead.

Constructing a question

We start by creating a question to prompt a model to draft sense check questions for our data. EDSL comes with a variety of question types that we can choose from based on the desired form of the response (multiple choice, free text, etc.). Here we use QuestionList in order to prompt the model to format its response as a list. We use a {{ placeholder }} for content that we will add to the question when we run it (a description of the data and a sample); this allows us to re-use the question with other contexts as desired:

[3]:

from edsl import QuestionList

q1 = QuestionList(
    question_name = "sense_check_questions",
    question_text = """
    You are being asked to suggest a list of sense checks for a dataset consisting of {{ scenario.data_description }}.
    Here is a sample of the data: {{ scenario.sample_data }}.
    Format the sense checks as a list of questions to be answered about each item in the dataset individually,
    using '<data>' as a placeholder for an item being reviewed in each question text.
    """,
    max_list_items = 4 # optional
)

Adding context to the question

Next we create Scenario objects representing the content that we want to add to the question when we run it. Here we create a single scenario for our example data:

[4]:

import random

sample_data = random.sample(ages, 10)

[5]:

from edsl import Scenario

s = Scenario({
    "data_description": "a list of human ages (in years)",
    "sample_data": sample_data
})
s

[5]:

Scenario

	key	value
0	data_description	a list of human ages (in years)
1	sample_data:0	43
2	sample_data:1	24
3	sample_data:2	58
4	sample_data:3	43
5	sample_data:4	56
6	sample_data:5	24
7	sample_data:6	54
8	sample_data:7	79
9	sample_data:8	77
10	sample_data:9	68

Running the question

We administer the question to a model by adding the scenarios and calling the run method. This generates a formatted dataset of Results that we can access with built-in methods for analysis. Here we inspect the answer:

[6]:

results = q1.by(s).run()

▼ Job Status (2025-03-03 09:11:25)

Job UUID	ad1b6f29-3fb1-408c-a871-ae9d7616cc69
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/ad1b6f29-3fb1-408c-a871-ae9d7616cc69
Exceptions Report URL	None
Results UUID	0bed643d-34e6-444c-a463-18ba151b0ade
Results URL	https://www.expectedparrot.com/content/0bed643d-34e6-444c-a463-18ba151b0ade

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/0bed643d-34e6-444c-a463-18ba151b0ade

[7]:

results.select("sense_check_questions")

[7]:

	answer.sense_check_questions
0	['Is a non-negative integer?', 'Is less than or equal to 122?', 'Is consistent with the expected age range for the dataset?', 'Is free from obvious errors or outliers?']

Conducting the task

Next we want a model to answer each sense check question about each piece of data in the dataset. This can be done by using the sense check questions as scenarios of a new question explaining the task. We can use QuestionYesNo to easily filter the responses:

[8]:

from edsl import QuestionYesNo

q2 = QuestionYesNo(
    question_name = "check_data",
    question_text = """
    You are being asked to sense check a dataset consisting of {{ scenario.data_description }}.
    Consider the following item in the dataset: {{ scenario.age }}
    {{ scenario.sense_check_question }}
    """
)

We need to create a new set of scenarios for the question. We use ScenarioList objects to create all the combinations of values to add to the question (learn more about constructing scenarios from different data sources):

[9]:

from edsl import ScenarioList

sl = ScenarioList(
    Scenario({
        "data_description": "a list of human ages (in years)",
        "sample_data": sample_data,
        "age": age,
        "sense_check_question": sense_check_question
    }) for age in ages for sense_check_question in results.select("sense_check_questions").to_list()[0]
)

We can inspect the scenarios that we created:

[10]:

sl.sample(3)

[10]:

ScenarioList scenarios: 3; keys: ['sample_data', 'age', 'data_description', 'sense_check_question'];

	data_description	sample_data	age	sense_check_question
0	a list of human ages (in years)	[43, 24, 58, 43, 56, 24, 54, 79, 77, 68]	50	Is free from obvious errors or outliers?
1	a list of human ages (in years)	[43, 24, 58, 43, 56, 24, 54, 79, 77, 68]	52	Is consistent with the expected age range for the dataset?
2	a list of human ages (in years)	[43, 24, 58, 43, 56, 24, 54, 79, 77, 68]	36	Is free from obvious errors or outliers?

Same as with a single scenario, we add all the scenarios to the question at once when we run it:

[11]:

results = q2.by(sl).run()

▼ Job Status (2025-03-03 09:12:29)

Job UUID	6f91ab81-5e93-42db-b133-f2f42a3838a4
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/6f91ab81-5e93-42db-b133-f2f42a3838a4
Exceptions Report URL	None
Results UUID	8e5b10c1-ad7e-4a84-ae6b-2b96da40e01c
Results URL	https://www.expectedparrot.com/content/8e5b10c1-ad7e-4a84-ae6b-2b96da40e01c

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/8e5b10c1-ad7e-4a84-ae6b-2b96da40e01c

We can filter, sort, select and print any components of the results that are generated:

[12]:

(
    results
    .filter("check_data == 'No'")
    .sort_by("sense_check_question")
    .select("sense_check_question", "age")
)

[12]:

	scenario.sense_check_question	scenario.age
0	Is a non-negative integer?	0.99
1	Is a non-negative integer?	-5
2	Is a non-negative integer?	old
3	Is a non-negative integer?	1.3
4	Is consistent with the expected age range for the dataset?	-5
5	Is consistent with the expected age range for the dataset?	1050
6	Is consistent with the expected age range for the dataset?	old
7	Is consistent with the expected age range for the dataset?	1.3
8	Is free from obvious errors or outliers?	-5
9	Is free from obvious errors or outliers?	1050
10	Is free from obvious errors or outliers?	old
11	Is free from obvious errors or outliers?	1.3
12	Is less than or equal to 122?	1050
13	Is less than or equal to 122?	old

Further exploration

This notebook can be readily edited and expanded for other data cleaning and data labeling purposes, or to add personas for AI agents answering the questions with relevant background and expertise. Learn more about using AI agents for your EDSL surveys.

Please see our documentation page for examples of other methods and use cases and let us know if you have any questions!

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

Here we post this notebook:

[15]:

from edsl import Notebook

nb = Notebook(path = "data_cleaning.ipynb")

if refresh := False:
    nb.push(
        description = "Example code for data cleaning",
        alias = "data-cleaning-notebook",
        visibility = "public"
    )
else:
    nb.patch("https://www.expectedparrot.com/content/RobinHorton/data-cleaning-notebook", value = nb)