> ## Documentation Index
> Fetch the complete documentation index at: https://docs.expectedparrot.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Validating classifications with humans

> Collect independent labels and compare them with model predictions.

Human validation compares model-generated labels with labels people assign using the same definitions. It helps answer a practical question: does the model classify your records in the way your task requires? Disagreements can reveal a model error, an unclear category boundary, or a record that people also interpret differently.

This page takes the service-request task from [Data labeling](https://docs.expectedparrot.com/en/latest/guides/data_labeling.md) through three stages: generate model predictions, collect independent human classifications, and match both by record identifier. You can use EDSL to collect the human responses, or bring in labels you already have.

## Start with the same records and codebook

The example below defines four sample requests, a codebook, a category question, and a supporting-evidence question. The requests have no preassigned labels. Both models and human reviewers will answer the same questions using the same category definitions.

```python theme={null}
from edsl import QuestionMultipleChoice, QuestionFreeText, Scenario, ScenarioList, Survey

# Sample inputs: no existing labels are needed to run the survey.
records = ScenarioList([
    Scenario({"record_id": "t1", "text": "I was charged twice."}),
    Scenario({"record_id": "t2", "text": "I cannot reset my password."}),
    Scenario({"record_id": "t3", "text": "Please add a dark theme."}),
    Scenario({"record_id": "t4", "text": "The invoice page won't open."}),
])

# Translate the codebook into allowed answers and instructions for using them.
codebook = {
    "Billing": "A charge or payment issue.",
    "Access": "Inability to enter or use a page, including a billing page.",
    "Feature request": "A request for new functionality.",
    "Other": "A request that does not fit any of the categories above.",
}
definitions = "\n".join(f"{label}: {meaning}" for label, meaning in codebook.items())
category = QuestionMultipleChoice(
    question_name="category",
    question_text="What kind of request is this: {{ scenario.text }}?\n" + definitions,
    question_options=list(codebook),
)
evidence = QuestionFreeText(
    question_name="evidence", question_text="Quote the words supporting your code: {{ scenario.text }}",
)
survey = Survey([category, evidence]).add_targeted_memory(evidence, category)
job = survey.by(records)
```

## Generate model predictions

With [model access configured](https://docs.expectedparrot.com/en/latest/api_keys.md), run the survey on the records:

```python theme={null}
predictions = job.run()
predictions.select("scenario.record_id", "scenario.text", "answer.category", "answer.evidence")
```

For one model and iteration, this gives one row per request. `record_id` is the link between a prediction and a later human judgment. Keeping that identifier avoids relying on row order, which may change when responses arrive or data is filtered.

## Collect human labels

EDSL can present the same questions to people through a hosted survey. The code below attaches the source records to the survey and publishes it with random scenario assignment. Each participant receives a request and answers the category and evidence questions:

```python theme={null}
info = survey.by(records).humanize(
    human_survey_name="Service request coding",
    scenario_list_method="randomize",
)
respondent_url = info["respondent_url"]
human_survey_uuid = info["uuid"]
```

Calling `humanize()` creates the hosted survey using your Expected Parrot account. Give `respondent_url` to the people who will label the requests. They receive the source text and codebook, without the model's chosen categories or evidence. This lets the comparison reflect independent judgments.

Random assignment can give some records more responses than others, so inspect which records have been labeled. [Human studies](https://docs.expectedparrot.com/en/latest/guides/human_studies.md) and [Humanize](https://docs.expectedparrot.com/en/latest/humanize.md) cover collection and other scenario assignment options.

## Retrieve and review the human responses

After participants have responded, retrieve the completed responses with the survey UUID:

```python theme={null}
from edsl import Coop

human_results = Coop().get_human_survey_responses(human_survey_uuid)
human_results
```

The method normally returns a Results object, so you can use `human_results.select("scenario.record_id", "answer.category", "answer.evidence")` to inspect the annotations. If responses cannot be converted to Results, it returns a ScenarioList of response data instead; inspect those fields before selecting columns.

When several people label one request, first decide how their judgments will become a reference label. For example, reviewers can discuss disagreements and agree on a final category, a process called **adjudication**. Keep the individual judgments if you also want to examine how often the human reviewers agree with one another. A disagreement between reviewers may point to a codebook definition that needs clarification.

## Match predictions to reviewed labels

For the comparison below, prepare `human_labels.csv` with two columns: `record_id` and `human_label`. Include one reviewed label for each record that has a completed review. You can prepare this file from the collected responses, or start here with labels from an existing annotation project.

```python theme={null}
from edsl import ScenarioList

labels = ScenarioList.from_source("csv", "human_labels.csv").to_pandas()
predicted = predictions.select("scenario.record_id", "answer.category").to_pandas()
comparison = predicted.merge(
    labels,
    left_on="scenario.record_id",
    right_on="record_id",
    how="left",
    validate="many_to_one",
)
reviewed = comparison.dropna(subset=["human_label"]).copy()
reviewed["matches_human"] = reviewed["answer.category"] == reviewed["human_label"]
agreement = reviewed["matches_human"].mean()
reviewed
```

The left join keeps every model prediction, including records that have no human label yet. `validate="many_to_one"` checks that the reference file contains at most one label for each record. `reviewed` includes only the records with a reference label, and `agreement` is the fraction of those records with an exact match. If no records have been reviewed, there is no agreement estimate yet.

Inspect unmatched records in `comparison` to see what still needs human review. Inspect disagreements in `reviewed` alongside the original text and evidence to understand their cause. Looking separately at Billing, Access, and Feature request can show a category-specific problem hidden by the overall agreement.

## Adapt the validation to your task

For multiple-choice classifications, exact agreement is a simple starting point. For checkbox themes, compare the selected sets or each theme separately; for numerical ratings, compare the size and direction of the differences. For free-text outputs, create evaluation questions that capture what you want people to judge, such as factual correctness or usefulness.

You can also [compare several models](https://docs.expectedparrot.com/en/latest/guides/model_comparisons.md) against the same reviewed labels. If the disagreements lead you to revise the prompt or codebook, use additional independently labeled records to assess the revised version. This separates the examples that helped you improve the task from the ones you use to evaluate it.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.