Concept induction

This notebook provides example EDSL code for using language models to perform “concept induction”: identify concepts in unstructured texts; generate criteria for the concepts; and then apply the criteria to evaluate the texts. This idea is inspired by the recent paper: Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM

Before running the code below, please see instructions on getting started using EDSL.

Identify concepts

We start by creating a general question prompting the respondent (a language model) to identify concepts in a given text.

EDSL comes with a variety of question types that we can choose from based on the form of the response that we want to get back from the model. QuestionList may be appropriate where we want the response to be formatted as a list of strings:

[1]:

from edsl import QuestionList

q_concepts = QuestionList(
    question_name="concepts",
    question_text="Identify the key concepts in the following text: {{ scenario.text }}",
    # max_list_items = # Optional,
    # min_list_items = # Optional
)

We might also want to ask some other questions about our data at the same time (a data labeling task). For example:

[2]:

from edsl import QuestionMultipleChoice

q_sentiment = QuestionMultipleChoice(
    question_name="sentiment",
    question_text="Identify the sentiment of this text: {{ scenario.text }}",
    question_options=["Negative", "Neutral", "Positive"],
)

We parameterize the questions in order to run them for each of our texts. This is done with Scenario objects that we can create using EDSL or import from other sources (CSV, PDF, PNG, MP4, DOC, tables, lists, dicts, etc.). Here we import a setof recent tweets by President Biden:

[3]:

# Replace with your data
texts = [  # POTUS recent tweets
    "Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C.",
    "We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom.",
    "Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share.",
    "Today, the Army Black Knights are taking home West Point’s 10th Commander-in-Chief Trophy. They should be proud. I’m proud of them too – not for the wins, but because after every game they hang up their uniforms and put on another: one representing the United States.",
    "This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.'",
    "The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow.",
    "Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day.",
    "Jill and I send warm wishes to Orthodox Christian communities around the world as they celebrate Easter. May the Lord bless and keep you this Easter Sunday and in the year ahead.",
    "Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients.",
    "With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.",
]
len(texts)

[3]:

[4]:

from edsl import ScenarioList

scenarios = ScenarioList.from_source("list", "text", texts)
scenarios

[4]:

ScenarioList scenarios: 10; keys: ['text'];

	text
0	Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C.
1	We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom.
2	Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share.
3	Today, the Army Black Knights are taking home West Point’s 10th Commander-in-Chief Trophy. They should be proud. I’m proud of them too – not for the wins, but because after every game they hang up their uniforms and put on another: one representing the United States.
4	This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.'
5	The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow.
6	Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day.
7	Jill and I send warm wishes to Orthodox Christian communities around the world as they celebrate Easter. May the Lord bless and keep you this Easter Sunday and in the year ahead.
8	Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients.
9	With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.

Next we combine the questions into a survey in order to administer them together (asynchronously by default, or according to any skip/stop rules or other logic that we want to add–learn more about Survey methods in our documentation):

[5]:

from edsl import Survey

survey = Survey(questions=[q_concepts, q_sentiment])

We add the scenarios to the survey and then run it with the default model (currently gpt-4o) to generate a dataset of results:

[6]:

results = survey.by(scenarios).run()

⌃ Job Status 🦜

Completed (10 completed, 0 failed)

Job Links

Results

Progress Report

Content

Remote Jobs

Remote Cache

Identifiers

Results UUID:

8ac9a9f7...6a82

Use Results.pull(uuid) to fetch results.

Job UUID:

50c6cc61...f66f

Use Jobs.pull(uuid) to fetch job.