Concept induction

This notebook offers sample EDSL code for using language models to identify concepts in unstructured texts, then generate criteria for the concepts, and then apply the criteria to evaluate the texts.

This idea is inspired by the recent paper: Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM

Technical setup

Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Identify concepts

We start by creating a general question prompting the respondent (a language model) to identify concepts in a given text.

EDSL comes with a variety of question types that we can choose from based on the form of the response that we want to get back from the model. QuestionList may be appropriate where we want the response to be formatted as a list of strings:

[1]:

from edsl import QuestionList

q_concepts = QuestionList(
    question_name="concepts",
    question_text="Identify the key concepts in the following text: {{ text }}",
    # max_list_items = # Optional
)

We might also want to ask some other questions about our data at the same time (a data labeling task). For example:

[2]:

from edsl import QuestionMultipleChoice

q_sentiment = QuestionMultipleChoice(
    question_name="sentiment",
    question_text="Identify the sentiment of this text: {{ text }}",
    question_options=["Negative", "Neutral", "Positive"],
)

We parameterize the questions in order to run them for each of our texts. This is done with Scenario objects that we create for our data (here, some recent tweets by Pres. Biden):

[3]:

# Replace with your data
texts = [  # POTUS recent tweets
    "Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C.",
    "We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom.",
    "Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share.",
    "Today, the Army Black Knights are taking home West Point’s 10th Commander-in-Chief Trophy. They should be proud. I’m proud of them too – not for the wins, but because after every game they hang up their uniforms and put on another: one representing the United States.",
    "This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.'",
    "The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow.",
    "Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day.",
    "Jill and I send warm wishes to Orthodox Christian communities around the world as they celebrate Easter. May the Lord bless and keep you this Easter Sunday and in the year ahead.",
    "Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients.",
    "With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.",
]
len(texts)

[3]:

10

[4]:

from edsl import ScenarioList

scenarios = ScenarioList.from_list("text", texts)
scenarios

[4]:

ScenarioList scenarios: 10; keys: ['text'];

text
Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C.
We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom.
Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share.
Today, the Army Black Knights are taking home West Point’s 10th Commander-in-Chief Trophy. They should be proud. I’m proud of them too – not for the wins, but because after every game they hang up their uniforms and put on another: one representing the United States.
This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.'
The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow.
Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day.
Jill and I send warm wishes to Orthodox Christian communities around the world as they celebrate Easter. May the Lord bless and keep you this Easter Sunday and in the year ahead.
Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients.
With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.

Next we combine the questions into a survey in order to administer them together (asynchronously by default, or according to any skip/stop rules or other logic that we want to add–learn more about Survey methods in our documentation):

[5]:

from edsl import Survey

survey = Survey(questions=[q_concepts, q_sentiment])

We add the scenarios to the survey and then run it to generate a dataset of results:

[6]:

results = survey.by(scenarios).run()
▼ Remote Job Log (2024-12-14 10:17:42)
Remote inference activated. Sending job to server...
Your survey is running at the Expected Parrot server...
Job sent to server. (Job uuid=4ffe3597-09b1-4fdc-92fb-d64a8da65f8c).
Job status: queued - last update: 2024-12-14 10:17:25 AM
Job status: running - last update: 2024-12-14 10:17:28 AM
Job status: running - last update: 2024-12-14 10:17:32 AM
Job status: running - last update: 2024-12-14 10:17:35 AM
Job status: running - last update: 2024-12-14 10:17:38 AM

EDSL comes with built-in methods for working with results in a variety of forms (data tables, SQL queries, dataframes, JSON, CSV). We can call the columns method to see a list of all the components that we can analyze:

[7]:

results.columns

[7]:
0
agent.agent_instruction
agent.agent_name
answer.concepts
answer.sentiment
comment.concepts_comment
comment.sentiment_comment
generated_tokens.concepts_generated_tokens
generated_tokens.sentiment_generated_tokens
iteration.iteration
model.frequency_penalty
model.logprobs
model.max_tokens
model.model
model.presence_penalty
model.temperature
model.top_logprobs
model.top_p
prompt.concepts_system_prompt
prompt.concepts_user_prompt
prompt.sentiment_system_prompt
prompt.sentiment_user_prompt
question_options.concepts_question_options
question_options.sentiment_question_options
question_text.concepts_question_text
question_text.sentiment_question_text
question_type.concepts_question_type
question_type.sentiment_question_type
raw_model_response.concepts_cost
raw_model_response.concepts_one_usd_buys
raw_model_response.concepts_raw_model_response
raw_model_response.sentiment_cost
raw_model_response.sentiment_one_usd_buys
raw_model_response.sentiment_raw_model_response
scenario.text

We can select and print specific components to inspect in a table:

[8]:

results.select("text", "concepts", "sentiment")

[8]:
scenario.text answer.concepts answer.sentiment
Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. ['keynote address', 'U.S. Holocaust Memorial Museum', 'Annual Days of Remembrance', 'Washington, D.C.'] Neutral
We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. ['nation of immigrants', 'nation of dreamers', 'Cinco de Mayo', 'nation of freedom'] Positive
Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. ['Medicare', 'Social Security', 'economic plan', 'solvency', 'rich pay their fair share'] Positive
Today, the Army Black Knights are taking home West Point’s 10th Commander-in-Chief Trophy. They should be proud. I’m proud of them too – not for the wins, but because after every game they hang up their uniforms and put on another: one representing the United States. ['Army Black Knights', 'West Point', '10th Commander-in-Chief Trophy', 'United States'] Positive
This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' ['Holocaust Remembrance Day', 'six million Jews', 'Nazis', 'darkest chapters in human history', 'lessons of the Shoah', "responsibility of 'Never Again'"]Neutral
The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow. ['Presidential Medal of Freedom', 'freedom', "America's faith", 'better tomorrow'] Positive
Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day. ['Teaching as a calling', 'Educators', 'Teacher State Dinner', 'White House'] Positive
Jill and I send warm wishes to Orthodox Christian communities around the world as they celebrate Easter. May the Lord bless and keep you this Easter Sunday and in the year ahead. ['Jill', 'Orthodox Christian communities', 'Easter', 'Lord', 'blessings', 'year ahead'] Positive
Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. ['Dreamers', 'health care', 'Affordable Care Act', 'DACA recipients', 'affordable health coverage'] Positive
With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.['New Jobs', 'American Comeback', 'Congressional Republicans', 'Tax Cuts', 'Special Interests', 'Job Creation', 'Economy for Families'] Positive

If our concepts lists are too long, we can run another question prompting a model to condense it. We can specify the number of concepts that we want to get:

[9]:

# Flattening our list of lists for all the texts to use in a follow-on question:
concepts_list = results.select("concepts").to_list(flatten=True)
concepts_list

[9]:
0
keynote address
U.S. Holocaust Memorial Museum
Annual Days of Remembrance
Washington, D.C.
nation of immigrants
nation of dreamers
Cinco de Mayo
nation of freedom
Medicare
Social Security
economic plan
solvency
rich pay their fair share
Army Black Knights
West Point
10th Commander-in-Chief Trophy
United States
Holocaust Remembrance Day
six million Jews
Nazis
darkest chapters in human history
lessons of the Shoah
responsibility of 'Never Again'
Presidential Medal of Freedom
freedom
America's faith
better tomorrow
Teaching as a calling
Educators
Teacher State Dinner
White House
Jill
Orthodox Christian communities
Easter
Lord
blessings
year ahead
Dreamers
health care
Affordable Care Act
DACA recipients
affordable health coverage
New Jobs
American Comeback
Congressional Republicans
Tax Cuts
Special Interests
Job Creation
Economy for Families 
[10]:

q_condense = QuestionList(
    question_name="condense",
    question_text="Return a condensed list of the following list of concepts: "
    + ", ".join(concepts_list),
    max_list_items=10,
)

Note that we can call the run() method on either a survey of questions or an individual question:

[11]:

results = q_condense.run()
▼ Remote Job Log (2024-12-14 10:19:00)
Remote inference activated. Sending job to server...
Your survey is running at the Expected Parrot server...
Job sent to server. (Job uuid=7a32a73e-c461-473b-848c-0257b7cafe32).
Job status: running - last update: 2024-12-14 10:18:47 AM
Job status: running - last update: 2024-12-14 10:18:50 AM
Job status: running - last update: 2024-12-14 10:18:53 AM
Job status: running - last update: 2024-12-14 10:18:57 AM
[12]:

results.select("condense")

[12]:
answer.condense
['Holocaust Remembrance', 'Nation of Immigrants', 'Economic Plan', 'U.S. Military', 'Freedom and Democracy', 'Education and Teaching', 'Health Care', 'Immigration and Dreamers', 'American Economy', 'Religious and Cultural Observances']

Identify criteria for each concept

Similar to our first step, next we can run a question prompting the model to generate criteria for each concept. We could use QuestionFreeText to generate criteria in an unstructured narrative:

[13]:

from edsl import QuestionFreeText

q_criteria = QuestionFreeText(
    question_name="criteria",
    question_text="""Describe key criteria for determining whether a text is primarily about the
    following concept: {{ concept }}""",
)

For this question, the scenarios are the concepts that we generated:

[14]:

condensed_concepts_list = results.select("condense").to_list(flatten=True)

scenarios = ScenarioList.from_list("concept", condensed_concepts_list)
scenarios

[14]:

ScenarioList scenarios: 10; keys: ['concept'];

concept
Holocaust Remembrance
Nation of Immigrants
Economic Plan
U.S. Military
Freedom and Democracy
Education and Teaching
Health Care
Immigration and Dreamers
American Economy
Religious and Cultural Observances
 
[15]:

results = q_criteria.by(scenarios).run()
▼ Remote Job Log (2024-12-14 10:19:43)
Remote inference activated. Sending job to server...
Your survey is running at the Expected Parrot server...
Job sent to server. (Job uuid=16be9e9d-4b5f-4bc8-9b96-cda2ce4ea6bd).
Job status: queued - last update: 2024-12-14 10:19:17 AM
Job status: queued - last update: 2024-12-14 10:19:20 AM
Job status: running - last update: 2024-12-14 10:19:23 AM
Job status: running - last update: 2024-12-14 10:19:27 AM
Job status: running - last update: 2024-12-14 10:19:30 AM
Job status: running - last update: 2024-12-14 10:19:33 AM
Job status: running - last update: 2024-12-14 10:19:36 AM
Job status: running - last update: 2024-12-14 10:19:40 AM
[16]:

results.select("concept", "criteria")

[16]:
scenario.concept answer.criteria
Holocaust Remembrance Determining whether a text is primarily about Holocaust Remembrance involves evaluating several key criteria: 1. **Subject Matter**: The text should focus on the Holocaust, discussing its historical events, impact, and the importance of remembering it. This includes narratives about survivors, victims, and the atrocities committed during this period. 2. **Purpose and Intent**: The text should aim to honor and remember the victims and survivors of the Holocaust. It may emphasize the importance of memory in preventing future genocides or discuss the moral and ethical lessons learned from the Holocaust. 3. **Commemorative Elements**: The text might describe or promote activities and events related to Holocaust remembrance, such as memorial services, educational programs, museum exhibits, or Holocaust Remembrance Day (Yom HaShoah). 4. **Educational Focus**: The text should aim to educate readers about the Holocaust, its history, and its lasting impact. This might include discussions on the significance of teaching the Holocaust in schools and how it is remembered in different cultures and countries. 5. **Reflective and Emotional Tone**: The text often carries a reflective and solemn tone, highlighting personal stories, reflections, and testimonies from survivors, as well as the emotional and psychological impact of the Holocaust on individuals and communities. 6. **Use of Symbolic Language**: The text may use language and symbols commonly associated with Holocaust remembrance, such as references to concentration camps, the Star of David, or iconic phrases like "Never Again." 7. **Historical Context**: The text should provide historical context about the Holocaust, including dates, locations, key figures, and events that are central to understanding the significance of remembrance. 8. **Call to Action**: The text may include a call to action, urging readers to participate in remembrance activities, combat Holocaust denial, or support Holocaust education and awareness initiatives.
Nation of Immigrants Determining whether a text is primarily about the concept of a "Nation of Immigrants" involves analyzing several key criteria: 1. **Historical Context**: The text should discuss the historical aspects of immigration, such as waves of immigration, key events, and policies that have shaped the nation. It might reference specific periods of significant immigration and how these have influenced the nation's development. 2. **Cultural Diversity**: The text should emphasize the cultural diversity brought about by immigration, detailing how various immigrant groups have contributed to the cultural, social, and economic fabric of the nation. It might explore the blending of traditions, languages, and customs. 3. **Economic Impact**: A focus on the economic contributions of immigrants, such as labor market influences, entrepreneurship, and innovation, is often central to this concept. The text might analyze how immigrants have driven economic growth and filled critical roles in various industries. 4. **Policy and Legislation**: The text should cover immigration policies and legislation, discussing how laws have evolved to manage immigration and the impact of these policies on the nation. It might include debates over immigration reform and border control. 5. **Identity and National Narrative**: The text should engage with the idea of national identity, exploring how the concept of being a "Nation of Immigrants" is woven into the national narrative and identity. It might discuss public perception and the symbolic significance of immigration in shaping the nation's self-image. 6. **Challenges and Controversies**: The text might address challenges and controversies related to immigration, such as integration, social tensions, or political debates. It could explore opposing viewpoints and the complexities of immigration in the national discourse. 7. **Personal Stories and Experiences**: The inclusion of personal narratives or case studies of immigrants can be a key indicator. These stories illustrate the human aspect of immigration and its impact on individuals and communities. 8. **Comparative Analysis**: The text might compare the nation with others in terms of immigration patterns, policies, and cultural integration, highlighting what makes the nation's experience unique.
Economic Plan Determining whether a text is primarily about an "Economic Plan" involves evaluating several key criteria. Here's a breakdown of these criteria: 1. **Purpose and Goals**: The text should clearly outline specific economic objectives or goals. This could include targets for growth, employment, inflation, or fiscal balance. The presence of specific, measurable outcomes related to the economy is a strong indicator. 2. **Strategies and Policies**: Look for detailed descriptions of strategies or policies designed to achieve the stated economic goals. This might include tax reforms, monetary policy adjustments, investment in infrastructure, or trade policies. 3. **Scope and Framework**: The text should define the scope of the economic plan, including the sectors of the economy it addresses (e.g., agriculture, industry, services) and the time frame over which it aims to operate (short-term, medium-term, long-term). 4. **Stakeholder Involvement**: An economic plan often involves various stakeholders, including government entities, private sector participants, and possibly international organizations. The text should mention who is responsible for implementation and who the plan affects. 5. **Financial Projections and Resources**: The text should include financial projections, such as budget allocations, expected revenues, or investment levels. It may also discuss the resources required and how they will be mobilized. 6. **Challenges and Risks**: A comprehensive economic plan will typically address potential challenges and risks, along with contingency measures or risk management strategies. 7. **Evaluation and Monitoring**: The text might outline mechanisms for evaluating the plan's progress and effectiveness, including key performance indicators or regular review processes. 8. **Contextual Background**: The text should provide a background on the current economic situation, explaining why the plan is necessary. It might reference existing economic conditions, historical data, or comparative analysis with other economies. 9. **Language and Terminology**: The use of specific economic terminology and jargon is a strong indicator. Terms like "fiscal policy," "GDP growth," "monetary tightening," or "structural reform" suggest a focus on economic planning. 10. **Intended Audience**: Consider who the text is aimed at. An economic plan is often directed at policymakers, economists, investors, or the general public to inform or persuade them about the proposed economic direction.
U.S. Military Determining whether a text is primarily about the U.S. Military involves evaluating several key criteria: 1. **Subject Matter**: The text should predominantly focus on topics directly related to the U.S. Military, such as its branches (Army, Navy, Air Force, Marine Corps, Space Force, and Coast Guard), operations, missions, or roles. 2. **Terminology**: The use of specific military terminology, jargon, or acronyms commonly associated with the U.S. Military can indicate that the text is centered on this subject. 3. **Personnel and Leadership**: Discussions about military personnel, including soldiers, officers, and leadership figures like the Secretary of Defense or Joint Chiefs of Staff, often signify a focus on the U.S. Military. 4. **Historical Context**: The text may cover historical events, conflicts, or wars involving the U.S. Military, such as World War II, the Vietnam War, or the War on Terror. 5. **Policy and Strategy**: Analysis or discussion of U.S. defense policies, military strategies, budget allocations, or legislative actions related to military operations can be a strong indicator. 6. **Geopolitical Role**: The text might explore the U.S. Military's role in international relations, alliances (e.g., NATO), or its presence and activities in various regions around the world. 7. **Technology and Equipment**: Coverage of military technology, weaponry, equipment, or advancements specific to the U.S. Military can also highlight the primary focus. 8. **Training and Exercises**: Descriptions of military training programs, exercises, or readiness assessments specific to U.S. forces. 9. **Cultural and Social Impact**: Discussions about the cultural, social, or economic impact of the U.S. Military on American society or its influence abroad. 10. **Legal and Ethical Issues**: Examination of legal, ethical, or human rights issues involving the U.S. Military, such as military justice, rules of engagement, or treatment of veterans.
Freedom and Democracy Determining whether a text is primarily about the concept of "Freedom and Democracy" involves analyzing several key criteria. Here are some factors to consider: 1. **Core Themes and Topics**: - **Freedom**: Look for discussions on individual liberties, civil rights, autonomy, freedom of speech, religion, and expression. The text may explore the balance between freedom and security or the impact of freedom on personal and societal levels. - **Democracy**: Check for references to democratic principles such as popular sovereignty, electoral processes, political participation, equality, and representation. The text might cover democratic institutions, governance, and the rule of law. 2. **Language and Terminology**: - Identify specific language that emphasizes freedom and democratic values, such as “liberty,” “rights,” “justice,” “equality,” “participation,” “elections,” “representation,” and “government by the people.” 3. **Purpose and Intent**: - Determine whether the text aims to promote, defend, critique, or analyze freedom and democracy. The intent might be to advocate for democratic reforms, highlight threats to freedom, or explore historical or philosophical perspectives on these concepts. 4. **Context and Setting**: - Consider the historical, political, or cultural context in which the text is situated. Texts set in democratic societies or those undergoing democratic transitions are more likely to focus on these themes. Similarly, discussions about authoritarian regimes may highlight the absence or suppression of freedom and democracy. 5. **Key Arguments and Perspectives**: - Analyze the main arguments or perspectives presented. Texts focusing on the benefits, challenges, or evolution of freedom and democracy often provide detailed analysis or critique of these concepts. 6. **Examples and Case Studies**: - Look for examples or case studies that illustrate democratic practices or the exercise of freedom. These could include historical events, political movements, or significant legal cases related to civil liberties. 7. **Audience and Impact**: - Consider who the intended audience is and what impact the text seeks to have. Texts aimed at policymakers, activists, or educators might prioritize themes of freedom and democracy to inspire action or inform policy. 8. **Comparative Analysis**: - If the text compares different political systems or ideologies, assess whether it contrasts democratic systems with non-democratic ones, highlighting the differences in freedom and governance.
Education and Teaching When determining whether a text is primarily about Education and Teaching, consider the following key criteria: 1. **Subject Matter**: The text should focus on topics related to education systems, teaching methods, learning theories, curriculum development, educational policies, or classroom management. It might also discuss specific subjects taught in schools or universities. 2. **Audience**: The intended audience often includes educators, students, policymakers, or individuals interested in educational theory and practice. Texts aimed at parents or guardians about educational involvement may also fit this category. 3. **Purpose**: The primary purpose of the text should be to inform, educate, or provide insights into educational practices, challenges, innovations, or reforms. It may aim to improve teaching effectiveness or student learning outcomes. 4. **Setting**: References to educational settings such as schools, universities, online learning platforms, or training centers can indicate a focus on education and teaching. 5. **Language and Terminology**: The use of specialized language and terminology related to pedagogy, such as "curriculum," "assessment," "pedagogical strategies," or "learning outcomes," suggests an educational focus. 6. **Discussion of Roles**: The text may discuss the roles of teachers, educators, administrators, or students, highlighting their experiences, responsibilities, or challenges within the educational context. 7. **Case Studies or Examples**: It might include case studies, examples, or anecdotes from educational environments, illustrating teaching techniques or learning experiences. 8. **Research and Data**: The presence of educational research, data, or statistics that support discussions on educational trends, student performance, or teaching efficacy can indicate a primary focus on this field. 9. **Historical or Cultural Context**: Some texts might explore the history of education, cultural influences on teaching practices, or comparative education studies. 10. **Policy and Reform**: Discussions on educational policy, reform initiatives, funding, or legislation are strong indicators of a focus on education and teaching.
Health Care Determining whether a text is primarily about health care involves analyzing several key criteria. Here are some important factors to consider: 1. **Topic and Content**: The central theme of the text should revolve around health care-related subjects. This includes discussions on medical services, health care systems, patient care, public health, health policies, and medical research. 2. **Terminology and Language**: The text should include terminology specific to health care, such as "medical treatment," "health insurance," "patient care," "clinical practice," "healthcare providers," "hospitals," and "public health." 3. **Stakeholders and Entities**: The text should mention key stakeholders in the health care sector, such as doctors, nurses, patients, health care organizations, insurance companies, and government health agencies. 4. **Issues and Challenges**: The text should address current issues, challenges, or debates within the health care sector, such as access to care, health care costs, quality of care, health care reform, and technological advancements in medicine. 5. **Purpose and Intent**: The intent of the text should be to inform, analyze, or discuss aspects of health care. This could be through providing information, proposing solutions, or critiquing existing systems. 6. **Scope and Relevance**: The scope of the text should be relevant to health care, whether it’s a broad overview or a specific focus on a particular aspect of health care, such as mental health services, preventive care, or health care legislation. 7. **Data and Evidence**: The text should utilize data, statistics, or research findings related to health care, such as health outcomes, patient demographics, or economic impacts of health policies. 8. **Context and Setting**: The context or setting of the text should be related to health care environments, such as hospitals, clinics, health departments, or health-related conferences and events.
Immigration and Dreamers Determining whether a text is primarily about "Immigration and Dreamers" involves analyzing the content for several key criteria: 1. **Focus on Immigration Policies**: The text should discuss immigration laws, policies, or reforms, especially those related to undocumented immigrants and pathways to citizenship. 2. **Mention of Dreamers**: The text should explicitly reference "Dreamers," a term commonly used to describe young undocumented immigrants who were brought to the United States as children and are eligible for the Deferred Action for Childhood Arrivals (DACA) program. 3. **Discussion of DACA**: The text should address the DACA program, its implications, legal challenges, or changes in policy affecting Dreamers. 4. **Personal Stories or Case Studies**: The text may include personal narratives or case studies of Dreamers, highlighting their experiences, challenges, and contributions to society. 5. **Socioeconomic Impact**: The text might explore the impact of immigration and Dreamers on the economy, education, and social services. 6. **Advocacy and Activism**: The text could cover advocacy efforts, protests, or movements supporting Dreamers and comprehensive immigration reform. 7. **Legal and Political Context**: The text may provide analysis or commentary on the legal and political landscape surrounding immigration and Dreamers, including court rulings or legislative proposals. 8. **Cultural and Social Integration**: The text might discuss issues related to the cultural assimilation, identity, and social integration of Dreamers and immigrant communities.
American Economy To determine whether a text is primarily about the American economy, you should look for several key criteria that indicate the text's focus on this specific topic: 1. **Focus on Economic Indicators**: The text should discuss economic indicators such as GDP growth, unemployment rates, inflation, trade balances, consumer spending, and other metrics that measure the economic health of the United States. 2. **Discussion of Economic Policies**: Look for analysis or discussion of U.S. government policies that impact the economy, such as fiscal policy (taxation and government spending), monetary policy (Federal Reserve actions), and regulatory changes. 3. **Sector Analysis**: The text should cover major sectors of the American economy, such as manufacturing, technology, services, agriculture, and energy, and how these sectors contribute to overall economic performance. 4. **Historical Context**: There may be references to historical economic events or trends that have shaped the current state of the American economy, such as the Great Depression, the 2008 financial crisis, or the COVID-19 pandemic. 5. **Comparison with Other Economies**: The text might compare the American economy with other national economies, discussing its relative strengths, weaknesses, and global competitiveness. 6. **Impact of International Trade**: Discussions about trade policies, tariffs, trade agreements, and their impact on the U.S. economy are critical indicators that the text is focused on the American economy. 7. **Labor Market Analysis**: Examination of employment trends, wage growth, labor force participation, and job creation in the United States can signal a focus on the American economy. 8. **Investment and Financial Markets**: Analysis of the U.S. stock market, bond market, and investment trends, including foreign investment in the U.S., can also indicate a focus on the American economy. 9. **Socioeconomic Issues**: The text may address socioeconomic issues such as income inequality, poverty rates, and their implications for economic growth and stability in the United States. 10. **Expert Opinions and Forecasts**: Inclusion of expert analyses, economic forecasts, and opinions from economists and policymakers about the future trajectory of the American economy.
Religious and Cultural ObservancesWhen determining whether a text is primarily about Religious and Cultural Observances, consider the following key criteria: 1. **Subject Matter**: The text should focus on specific religious or cultural events, rituals, ceremonies, or practices. This can include descriptions of holidays, festivals, rites of passage, or other significant observances within a particular religious or cultural tradition. 2. **Terminology and Language**: Look for the use of language and terminology specific to religious or cultural practices. This might include references to sacred texts, religious figures, cultural symbols, or traditional practices. 3. **Purpose and Function**: The text should explore the purpose or function of the observances, such as their role in community building, spiritual fulfillment, cultural identity, or moral instruction. 4. **Historical and Cultural Context**: The text may provide historical background or cultural context that explains the origins or significance of the observance. This can include discussions of how these practices have evolved over time or their impact on society. 5. **Participants and Practices**: The text should detail who participates in these observances and how they are practiced. This can involve descriptions of rituals, customs, attire, food, music, or other elements that are integral to the observance. 6. **Impact and Significance**: Consider whether the text discusses the impact of these observances on individuals and communities, including their spiritual, cultural, or social significance. 7. **Comparative Analysis**: The text might compare different religious or cultural observances, highlighting similarities and differences in practices, beliefs, or significance across cultures or religions. 8. **Narratives and Stories**: The inclusion of narratives or stories related to religious or cultural observances can indicate that the text is focused on this concept, particularly if these stories are used to illustrate the meaning or importance of the observance.

Identify the concepts in each text and evaluate based on the criteria

Finally, we can use the concepts and the criteria to run another question where we prompt the model to evaulate each text. Question types QuestionLinearScale, QuestionRank or QuestionNumerical may be appropriate where we want to return a score:

[17]:

from edsl import QuestionLinearScale

q_score = QuestionLinearScale(
    question_name="score",
    question_text="""Consider the following concept and criteria for determining whether
    a given text addresses this concept. Then score how well the following text satisfies
    the criteria for the concept.
    Concept: {{ concept }}
    Criteria: {{ criteria }}
    Text: {{ text }}""",
    question_options=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    option_labels={0: "Not at all", 10: "Very well"},  # Optional
)

Here we want to use both the texts and the concepts and corresponding criteria together as scenarios of the question:

[18]:

concepts_criteria = [
    list(pair)
    for pair in zip(
        results.select("concept").to_list(), results.select("criteria").to_list()
    )
]
len(concepts_criteria)

[18]:

10

[19]:

from edsl import ScenarioList, Scenario

scenarios = ScenarioList(
    Scenario({"text": text, "concept": concept, "criteria": criteria})
    for text in texts
    for [concept, criteria] in concepts_criteria
)

[20]:

results = q_score.by(scenarios).run()
▼ Remote Job Log (2024-12-14 10:20:27)
Remote inference activated. Sending job to server...
Your survey is running at the Expected Parrot server...
Job sent to server. (Job uuid=9d10477f-7bb1-438c-860b-023e5bd42d75).
Job status: queued - last update: 2024-12-14 10:20:02 AM
Job status: queued - last update: 2024-12-14 10:20:06 AM
Job status: running - last update: 2024-12-14 10:20:09 AM
Job status: running - last update: 2024-12-14 10:20:13 AM
Job status: running - last update: 2024-12-14 10:20:16 AM
Job status: running - last update: 2024-12-14 10:20:19 AM
Job status: running - last update: 2024-12-14 10:20:23 AM

We can filter the results based on the responses–e.g., here we just show the non-zero scores:

[21]:

(
    results.filter("score > 0")
    .select("text", "concept", "score")
)

[21]:
scenario.text scenario.concept answer.score
Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. Holocaust Remembrance 5
Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. Holocaust Remembrance 5
Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. Freedom and Democracy 1
Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. Freedom and Democracy 1
Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. Religious and Cultural Observances 3
Tune in as I deliver the keynote address at the U.S. Holocaust Memorial Museum’s Annual Days of Remembrance ceremony in Washington, D.C. Religious and Cultural Observances 3
We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. Nation of Immigrants 1
We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. Freedom and Democracy 2
We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. Freedom and Democracy 2
We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. Immigration and Dreamers 1
We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. Immigration and Dreamers 1
We’re a nation of immigrants. A nation of dreamers. And as Cinco de Mayo represents, a nation of freedom. Religious and Cultural Observances 1
Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. Economic Plan 3
Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. Economic Plan 3
Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. Freedom and Democracy 1
Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. Health Care 3
Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. Health Care 3
Medicare is stronger and Social Security remains strong. My economic plan has helped extend Medicare solvency by a decade. And I am committed to extending Social Security solvency by making the rich pay their fair share. American Economy 1
Today, the Army Black Knights are taking home West Point’s 10th Commander-in-Chief Trophy. They should be proud. I’m proud of them too – not for the wins, but because after every game they hang up their uniforms and put on another: one representing the United States. U.S. Military 3
This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' Holocaust Remembrance 7
This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' Holocaust Remembrance 7
This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' Freedom and Democracy 1
This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' Religious and Cultural Observances 3
This Holocaust Remembrance Day, we mourn the six million Jews who were killed by the Nazis during one of the darkest chapters in human history. And we recommit to heeding the lessons of the Shoah and realizing the responsibility of 'Never Again.' Religious and Cultural Observances 3
The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow. Freedom and Democracy 3
The recipients of the Presidential Medal of Freedom haven't just kept faith in freedom. They kept all of America's faith in a better tomorrow. Freedom and Democracy 3
Like Jill says, 'Teaching isn’t just a job. It’s a calling.' She knows that in her bones, and I know every educator who joined us at the White House for the first-ever Teacher State Dinner lives out that truth every day. Education and Teaching 2
Jill and I send warm wishes to Orthodox Christian communities around the world as they celebrate Easter. May the Lord bless and keep you this Easter Sunday and in the year ahead. Religious and Cultural Observances 1
Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. Nation of Immigrants 3
Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. Nation of Immigrants 3
Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. Freedom and Democracy 1
Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. Health Care 5
Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. Health Care 5
Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. Immigration and Dreamers 5
Dreamers are our loved ones, nurses, teachers, and small business owners – they deserve the promise of health care just like all of us. Today, my Administration is making that real by expanding affordable health coverage through the Affordable Care Act to DACA recipients. Immigration and Dreamers 5
With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.Economic Plan 1
With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.Freedom and Democracy 1
With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.American Economy 3
With today’s report of 175,000 new jobs, the American comeback continues. Congressional Republicans are fighting to cut taxes for billionaires and let special interests rip folks off, I'm focused on job creation and building an economy that works for the families I grew up with.American Economy 3

Posting to the Coop

The Coop is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Coop account page. Learn more about creating an account and using the Coop.

Here we post the scenarios, survey and results from above, and this notebook:

[22]:

scenarios.push(description = "Example scenarios", visibility = "public")

[22]:

{'description': 'Example scenarios',
 'object_type': 'scenario_list',
 'url': 'https://www.expectedparrot.com/content/81641a37-dc54-4413-845b-0443e1bb5f99',
 'uuid': '81641a37-dc54-4413-845b-0443e1bb5f99',
 'version': '0.1.39.dev1',
 'visibility': 'public'}

[23]:

survey.push(description = "Example survey", visibility = "public")

[23]:

{'description': 'Example survey',
 'object_type': 'survey',
 'url': 'https://www.expectedparrot.com/content/069c9b88-12c3-4f1c-8ff3-cb6219d36243',
 'uuid': '069c9b88-12c3-4f1c-8ff3-cb6219d36243',
 'version': '0.1.39.dev1',
 'visibility': 'public'}

[24]:

results.push(description = "Example results", visibility = "public")

[24]:

{'description': 'Example results',
 'object_type': 'results',
 'url': 'https://www.expectedparrot.com/content/7b2978ea-1a9e-43b2-9c78-f0714c89ad97',
 'uuid': '7b2978ea-1a9e-43b2-9c78-f0714c89ad97',
 'version': '0.1.39.dev1',
 'visibility': 'public'}

We can also post this notebook:

[25]:

from edsl import Notebook

[27]:

n = Notebook(path = "concept_induction.ipynb")

[28]:

info = n.push(description = "Example code for concept induction", visibility = "public")
info

[28]:

{'description': 'Example code for concept induction',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/6d261050-b51b-49fa-98cc-df714ddfbec2',
 'uuid': '6d261050-b51b-49fa-98cc-df714ddfbec2',
 'version': '0.1.39.dev1',
 'visibility': 'public'}

To update an object at the Coop:

[33]:

n = Notebook(path = "concept_induction.ipynb") # resave it

[34]:

n.patch(uuid = info["uuid"], value = n)

[34]:

{'status': 'success'}