Results
A Results object is the result of running a survey. It is a list of individual Result objects, each of which represents a response to a Survey for each combination of Agent, Model and Scenario objects that were used with the survey. For example, the Results of a survey administered to 2 agents and 2 language models with no question scenarios will contain 4 individual Result objects. If the survey questions are parameterized with 2 scenarios then the survey Results will include 8 Result objects.
A Results object is not typically instantiated directly, but is returned by calling the run() method of a survey after any agents, language models and scenarios are added to it. To inspect the form of an example Results we can call the example() method (it is long – we show it at the end of this page):
from edsl import Results
example_results = Results.example()
We can see the number of Result objects created by inspecting the length of the Results:
len(example_results)
Output:
4
We can verify that object types:
type(example_results)
Output
edsl.results.Results.Results
And the 4 Result objects:
type(example_results[0])
Output:
edsl.results.Results.Result
Note: You must have API keys for language models in order to generate results. Please see the API Keys section for instructions on activating Remote Inference from your coop account or storing your own API keys.
For purposes of demonstrating how to unpack and interact with results, we’ll use the following code to generate results for a simple survey. Note that specifying agent traits, scenarios (question parameter values) and language models is optional, and we include those steps here for illustrative purposes:
# Create questions
from edsl import QuestionLinearScale, QuestionFreeText, QuestionMultipleChoice
q1 = QuestionLinearScale(
question_name = "important",
question_text = "On a scale from 1 to 5, how important to you is {{ topic }}?",
question_options = [0, 1, 2, 3, 4, 5],
option_labels = {0:"Not at all", 5:"Very much"}
)
q2 = QuestionFreeText(
question_name = "opinions",
question_text = "What are your opinions on {{ topic }}?"
)
q3 = QuestionMultipleChoice(
question_name = "read",
question_text = "Have you read any books about {{ topic }}?",
question_options = ["Yes", "No", "I do not know"]
)
# Optionally parameterize the questions with scenarios
from edsl import ScenarioList
scenarios = ScenarioList.from_list("topic", ["climate change", "data privacy"])
# Optionally create agents with traits
from edsl import AgentList, Agent
agents = AgentList(
Agent(traits = {"persona": p}) for p in ["student", "celebrity"]
)
# Optionally specify language models
from edsl import ModelList, Model
models = ModelList(
Model(m) for m in ["claude-3-5-sonnet-20240620", "gpt-4o"]
)
# Create a survey with the questions
from edsl import Survey
survey = Survey([q1, q2, q3])
# Run the survey with the scenarios, agents and models
results = survey.by(scenarios).by(agents).by(models).run()
For more details on each of the above steps, please see the relevant sections of the docs.
Result objects
We can check the number of Result objects created by inspecting the length of the Results:
len(results)
This will count 2 (scenarios) x 2 (agents) x 2 (models) = 8 Result objects:
8
Generating multiple results
If we want to generate multiple results for a survey–i.e., more than 1 result for each combination of Agent, Model and Scenario objects used–we can pass the desired number of iterations when calling the run() method. For example, the following code will generate 3 results for our survey (n=3):
results = survey.by(scenarios).by(agents).by(models).run(n=3)
We can verify that the number of Result objects created is now 24 = 3 iterations x 2 scenarios x 2 agents x 2 models:
len(results)
24
We can readily inspect a result:
results[0]
{
"agent": {
"traits": {
"persona": "student"
},
"edsl_version": "0.1.33.dev1",
"edsl_class_name": "Agent"
},
"scenario": {
"topic": "climate change",
"edsl_version": "0.1.33.dev1",
"edsl_class_name": "Scenario"
},
"model": {
"model": "claude-3-5-sonnet-20240620",
"parameters": {
"temperature": 0.5,
"max_tokens": 1000,
"top_p": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
"logprobs": false,
"top_logprobs": 3
},
"edsl_version": "0.1.33.dev1",
"edsl_class_name": "LanguageModel"
},
"iteration": 0,
"answer": {
"important": 4,
"opinions": "As a student, I'm still learning about climate change and forming my views on it. From what I've studied so far in my science classes, the scientific consensus seems to be that climate change is a real phenomenon and human activities are contributing to it. I find the topic really interesting and important to understand. I try to stay up to date by reading articles and reports from reputable scientific sources. At the same time, I know there's still a lot of debate around the specific impacts and best solutions. I'm eager to continue learning more as I progress in my studies.",
"read": "Yes"
},
"prompt": {
"important_user_prompt": {
"text": "On a scale from 1 to 5, how important to you is climate change?\n\n0 : Not at all\n\n1 : \n\n2 : \n\n3 : \n\n4 : \n\n5 : Very much\n\nOnly 1 option may be selected.\n\nRespond only with the code corresponding to one of the options. E.g., \"1\" or \"5\" by itself.\n\nAfter the answer, you can put a comment explaining why you chose that option on the next line.",
"class_name": "Prompt"
},
"important_system_prompt": {
"text": "You are answering questions as if you were a human. Do not break character. You are an agent with the following persona:\n{'persona': 'student'}",
"class_name": "AgentInstruction"
},
"opinions_user_prompt": {
"text": "What are your opinions on climate change?",
"class_name": "Prompt"
},
"opinions_system_prompt": {
"text": "You are answering questions as if you were a human. Do not break character. You are an agent with the following persona:\n{'persona': 'student'}",
"class_name": "AgentInstruction"
},
"read_user_prompt": {
"text": "\nHave you read any books about climate change?\n\n \nYes\n \nNo\n \nI do not know\n \n\nOnly 1 option may be selected.\n\nRespond only with a string corresponding to one of the options.\n\n\nAfter the answer, you can put a comment explaining why you chose that option on the next line.",
"class_name": "Prompt"
},
"read_system_prompt": {
"text": "You are answering questions as if you were a human. Do not break character. You are an agent with the following persona:\n{'persona': 'student'}",
"class_name": "AgentInstruction"
}
},
"raw_model_response": {
"important_raw_model_response": {
"id": "msg_01HGaPNDtj6fkCLdbdy3h4HA",
"content": [
{
"text": "4\n\nAs a student, I'm quite concerned about climate change and its long-term impacts on our planet and future generations. It's a major issue we learn about in school, and I feel it's important to be informed and take action where we can.",
"type": "text"
}
],
"model": "claude-3-5-sonnet-20240620",
"role": "assistant",
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message",
"usage": {
"input_tokens": 152,
"output_tokens": 57
}
},
"important_cost": 0.001310994813023199,
"important_one_usd_buys": 762.779524423873,
"opinions_raw_model_response": {
"id": "msg_017mQNAmbkvzuLLtpe7HzhiS",
"content": [
{
"text": "As a student, I'm still learning about climate change and forming my views on it. From what I've studied so far in my science classes, the scientific consensus seems to be that climate change is a real phenomenon and human activities are contributing to it. I find the topic really interesting and important to understand. I try to stay up to date by reading articles and reports from reputable scientific sources. At the same time, I know there's still a lot of debate around the specific impacts and best solutions. I'm eager to continue learning more as I progress in my studies.",
"type": "text"
}
],
"model": "claude-3-5-sonnet-20240620",
"role": "assistant",
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message",
"usage": {
"input_tokens": 49,
"output_tokens": 119
}
},
"opinions_cost": 0.0019319907810452126,
"opinions_one_usd_buys": 517.6008135292432,
"read_raw_model_response": {
"id": "msg_01VwAiiNMiwTZQ6Q4jU5hPof",
"content": [
{
"text": "Yes\n\nAs a student, I've likely had to read at least one book about climate change for a science or environmental studies class. It's a major topic covered in many curricula these days.",
"type": "text"
}
],
"model": "claude-3-5-sonnet-20240620",
"role": "assistant",
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message",
"usage": {
"input_tokens": 114,
"output_tokens": 43
}
},
"read_cost": 0.000986996091017493,
"read_one_usd_buys": 1013.1752385859008
},
"question_to_attributes": {
"important": {
"question_text": "On a scale from 1 to 5, how important to you is {{ topic }}?",
"question_type": "linear_scale",
"question_options": [
0,
1,
2,
3,
4,
5
]
},
"opinions": {
"question_text": "What are your opinions on {{ topic }}?",
"question_type": "free_text",
"question_options": null
},
"read": {
"question_text": "Have you read any books about {{ topic }}?",
"question_type": "multiple_choice",
"question_options": [
"Yes",
"No",
"I do not know"
]
}
},
"generated_tokens": {
"important_generated_tokens": "4\n\nAs a student, I'm quite concerned about climate change and its long-term impacts on our planet and future generations. It's a major issue we learn about in school, and I feel it's important to be informed and take action where we can.",
"opinions_generated_tokens": "As a student, I'm still learning about climate change and forming my views on it. From what I've studied so far in my science classes, the scientific consensus seems to be that climate change is a real phenomenon and human activities are contributing to it. I find the topic really interesting and important to understand. I try to stay up to date by reading articles and reports from reputable scientific sources. At the same time, I know there's still a lot of debate around the specific impacts and best solutions. I'm eager to continue learning more as I progress in my studies.",
"read_generated_tokens": "Yes\n\nAs a student, I've likely had to read at least one book about climate change for a science or environmental studies class. It's a major topic covered in many curricula these days."
}
}
We can use the rich_print method to display the Result object in a more readable format:
results[0].rich_print()
Result
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Attribute ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ agent │ Agent Attributes │
│ │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ _name │ None │ │
│ │ │ _traits │ {'persona': 'student'} │ │
│ │ │ _codebook │ {} │ │
│ │ │ _instruction │ 'You are answering questions as if you were a human. │ │
│ │ │ │ Do not break character.' │ │
│ │ │ set_instructions │ False │ │
│ │ │ dynamic_traits_function │ None │ │
│ │ │ has_dynamic_traits_function │ False │ │
│ │ │ current_question │ Question('multiple_choice', question_name = │ │
│ │ │ │ """read""", question_text = """Have you read any │ │
│ │ │ │ books about {{ topic }}?""", question_options = │ │
│ │ │ │ ['Yes', 'No', 'I do not know']) │ │
│ │ └─────────────────────────────┴──────────────────────────────────────────────────────┘ │
│ scenario │ Scenario Attributes │
│ │ ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ data │ {'topic': 'climate change'} │ │
│ │ │ name │ None │ │
│ │ └───────────┴─────────────────────────────┘ │
│ model │ Language Model │
│ │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ model │ 'claude-3-5-sonnet-20240620' │ │
│ │ │ parameters │ {'temperature': 0.5, 'max_tokens': 1000, 'top_p': 1, │ │
│ │ │ │ 'frequency_penalty': 0, 'presence_penalty': 0, │ │
│ │ │ │ 'logprobs': False, 'top_logprobs': 3} │ │
│ │ │ remote │ False │ │
│ │ │ omit_system_prompt_if_empty │ True │ │
│ │ │ temperature │ 0.5 │ │
│ │ │ max_tokens │ 1000 │ │
│ │ │ top_p │ 1 │ │
│ │ │ frequency_penalty │ 0 │ │
│ │ │ presence_penalty │ 0 │ │
│ │ │ logprobs │ False │ │
│ │ │ top_logprobs │ 3 │ │
│ │ │ _LanguageModel__rate_limits │ {'rpm': 10000, 'tpm': 2000000} │ │
│ │ └─────────────────────────────┴──────────────────────────────────────────────────────┘ │
│ iteration │ 0 │
│ answer │ ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ important │ 4 │ │
│ │ │ opinions │ "As a student, I'm still learning about climate change and forming my │ │
│ │ │ │ views on it. From what I've studied so far in my science classes, the │ │
│ │ │ │ scientific consensus seems to be that climate change is a real │ │
│ │ │ │ phenomenon and human activities are contributing to it. I find the │ │
│ │ │ │ topic really interesting and important to understand. I try to stay up │ │
│ │ │ │ to date by reading articles and reports from reputable scientific │ │
│ │ │ │ sources. At the same time, I know there's still a lot of debate around │ │
│ │ │ │ the specific impacts and best solutions. I'm eager to continue │ │
│ │ │ │ learning more as I progress in my studies." │ │
│ │ │ read │ 'Yes' │ │
│ │ └───────────┴────────────────────────────────────────────────────────────────────────┘ │
│ prompt │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ important_user_prompt │ Prompt(text="""On a scale from 1 to 5, how important to │ │
│ │ │ │ you is climate change? │ │
│ │ │ │ │ │
│ │ │ │ 0 : Not at all │ │
│ │ │ │ │ │
│ │ │ │ 1 : │ │
│ │ │ │ │ │
│ │ │ │ 2 : │ │
│ │ │ │ │ │
│ │ │ │ 3 : │ │
│ │ │ │ │ │
│ │ │ │ 4 : │ │
│ │ │ │ │ │
│ │ │ │ 5 : Very much │ │
│ │ │ │ │ │
│ │ │ │ Only 1 option may be selected. │ │
│ │ │ │ │ │
│ │ │ │ Respond only with the code corresponding to one of the │ │
│ │ │ │ options. E.g., "1" or "5" by itself. │ │
│ │ │ │ │ │
│ │ │ │ After the answer, you can put a comment explaining why │ │
│ │ │ │ you chose that option on the next line.""") │ │
│ │ │ important_system_prompt │ Prompt(text="""You are answering questions as if you │ │
│ │ │ │ were a human. Do not break character. You are an agent │ │
│ │ │ │ with the following persona: │ │
│ │ │ │ {'persona': 'student'}""") │ │
│ │ │ opinions_user_prompt │ Prompt(text="""What are your opinions on climate │ │
│ │ │ │ change?""") │ │
│ │ │ opinions_system_prompt │ Prompt(text="""You are answering questions as if you │ │
│ │ │ │ were a human. Do not break character. You are an agent │ │
│ │ │ │ with the following persona: │ │
│ │ │ │ {'persona': 'student'}""") │ │
│ │ │ read_user_prompt │ Prompt(text=""" │ │
│ │ │ │ Have you read any books about climate change? │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ Yes │ │
│ │ │ │ │ │
│ │ │ │ No │ │
│ │ │ │ │ │
│ │ │ │ I do not know │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ Only 1 option may be selected. │ │
│ │ │ │ │ │
│ │ │ │ Respond only with a string corresponding to one of the │ │
│ │ │ │ options. │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ After the answer, you can put a comment explaining why │ │
│ │ │ │ you chose that option on the next line.""") │ │
│ │ │ read_system_prompt │ Prompt(text="""You are answering questions as if you │ │
│ │ │ │ were a human. Do not break character. You are an agent │ │
│ │ │ │ with the following persona: │ │
│ │ │ │ {'persona': 'student'}""") │ │
│ │ └─────────────────────────┴──────────────────────────────────────────────────────────┘ │
│ raw_model_response │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ important_raw_model_response │ {'id': 'msg_01HGaPNDtj6fkCLdbdy3h4HA', 'content': │ │
│ │ │ │ [{'text': "4\n\nAs a student, I'm quite concerned │ │
│ │ │ │ about climate change and its long-term impacts on │ │
│ │ │ │ our planet and future generations. It's a major │ │
│ │ │ │ issue we learn about in school, and I feel it's │ │
│ │ │ │ important to be informed and take action where we │ │
│ │ │ │ can.", 'type': 'text'}], 'model': │ │
│ │ │ │ 'claude-3-5-sonnet-20240620', 'role': 'assistant', │ │
│ │ │ │ 'stop_reason': 'end_turn', 'stop_sequence': None, │ │
│ │ │ │ 'type': 'message', 'usage': {'input_tokens': 152, │ │
│ │ │ │ 'output_tokens': 57}} │ │
│ │ │ important_cost │ 0.001310994813023199 │ │
│ │ │ important_one_usd_buys │ 762.779524423873 │ │
│ │ │ opinions_raw_model_response │ {'id': 'msg_017mQNAmbkvzuLLtpe7HzhiS', 'content': │ │
│ │ │ │ [{'text': "As a student, I'm still learning about │ │
│ │ │ │ climate change and forming my views on it. From │ │
│ │ │ │ what I've studied so far in my science classes, the │ │
│ │ │ │ scientific consensus seems to be that climate │ │
│ │ │ │ change is a real phenomenon and human activities │ │
│ │ │ │ are contributing to it. I find the topic really │ │
│ │ │ │ interesting and important to understand. I try to │ │
│ │ │ │ stay up to date by reading articles and reports │ │
│ │ │ │ from reputable scientific sources. At the same │ │
│ │ │ │ time, I know there's still a lot of debate around │ │
│ │ │ │ the specific impacts and best solutions. I'm eager │ │
│ │ │ │ to continue learning more as I progress in my │ │
│ │ │ │ studies.", 'type': 'text'}], 'model': │ │
│ │ │ │ 'claude-3-5-sonnet-20240620', 'role': 'assistant', │ │
│ │ │ │ 'stop_reason': 'end_turn', 'stop_sequence': None, │ │
│ │ │ │ 'type': 'message', 'usage': {'input_tokens': 49, │ │
│ │ │ │ 'output_tokens': 119}} │ │
│ │ │ opinions_cost │ 0.0019319907810452126 │ │
│ │ │ opinions_one_usd_buys │ 517.6008135292432 │ │
│ │ │ read_raw_model_response │ {'id': 'msg_01VwAiiNMiwTZQ6Q4jU5hPof', 'content': │ │
│ │ │ │ [{'text': "Yes\n\nAs a student, I've likely had to │ │
│ │ │ │ read at least one book about climate change for a │ │
│ │ │ │ science or environmental studies class. It's a │ │
│ │ │ │ major topic covered in many curricula these days.", │ │
│ │ │ │ 'type': 'text'}], 'model': │ │
│ │ │ │ 'claude-3-5-sonnet-20240620', 'role': 'assistant', │ │
│ │ │ │ 'stop_reason': 'end_turn', 'stop_sequence': None, │ │
│ │ │ │ 'type': 'message', 'usage': {'input_tokens': 114, │ │
│ │ │ │ 'output_tokens': 43}} │ │
│ │ │ read_cost │ 0.000986996091017493 │ │
│ │ │ read_one_usd_buys │ 1013.1752385859008 │ │
│ │ └──────────────────────────────┴─────────────────────────────────────────────────────┘ │
│ survey │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Questions ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ │ │
│ │ │ ┃ Question Name ┃ Question Type ┃ Question Text ┃ Options ┃ │ │
│ │ │ ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │ │
│ │ │ │ important │ linear_scale │ On a scale from 1 to 5, how │ 0, 1, 2, 3, 4, 5 │ │ │
│ │ │ │ │ │ important to you is {{ │ │ │ │
│ │ │ │ │ │ topic }}? │ │ │ │
│ │ │ └───────────────┴───────────────┴─────────────────────────────┴──────────────────┘ │ │
│ │ │ ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓ │ │
│ │ │ ┃ Question Name ┃ Question Type ┃ Question Text ┃ Options ┃ │ │
│ │ │ ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩ │ │
│ │ │ │ opinions │ free_text │ What are your opinions on {{ topic │ None │ │ │
│ │ │ │ │ │ }}? │ │ │ │
│ │ │ └───────────────┴───────────────┴──────────────────────────────────────┴─────────┘ │ │
│ │ │ ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ │ │
│ │ │ ┃ Question Name ┃ Question Type ┃ Question Text ┃ Options ┃ │ │
│ │ │ ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ │ │
│ │ │ │ read │ multiple_choice │ Have you read any │ Yes, No, I do not │ │ │
│ │ │ │ │ │ books about {{ topic │ know │ │ │
│ │ │ │ │ │ }}? │ │ │ │
│ │ │ └───────────────┴─────────────────┴──────────────────────┴───────────────────────┘ │ │
│ │ └────────────────────────────────────────────────────────────────────────────────────┘ │
│ question_to_attributes │ ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ important │ {'question_text': 'On a scale from 1 to 5, how important to you is {{ │ │
│ │ │ │ topic }}?', 'question_type': 'linear_scale', 'question_options': [0, │ │
│ │ │ │ 1, 2, 3, 4, 5]} │ │
│ │ │ opinions │ {'question_text': 'What are your opinions on {{ topic }}?', │ │
│ │ │ │ 'question_type': 'free_text', 'question_options': None} │ │
│ │ │ read │ {'question_text': 'Have you read any books about {{ topic }}?', │ │
│ │ │ │ 'question_type': 'multiple_choice', 'question_options': ['Yes', 'No', │ │
│ │ │ │ 'I do not know']} │ │
│ │ └───────────┴────────────────────────────────────────────────────────────────────────┘ │
│ generated_tokens │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ important_generated_tokens │ "4\n\nAs a student, I'm quite concerned about climate │ │
│ │ │ │ change and its long-term impacts on our planet and │ │
│ │ │ │ future generations. It's a major issue we learn about │ │
│ │ │ │ in school, and I feel it's important to be informed │ │
│ │ │ │ and take action where we can." │ │
│ │ │ opinions_generated_tokens │ "As a student, I'm still learning about climate │ │
│ │ │ │ change and forming my views on it. From what I've │ │
│ │ │ │ studied so far in my science classes, the scientific │ │
│ │ │ │ consensus seems to be that climate change is a real │ │
│ │ │ │ phenomenon and human activities are contributing to │ │
│ │ │ │ it. I find the topic really interesting and important │ │
│ │ │ │ to understand. I try to stay up to date by reading │ │
│ │ │ │ articles and reports from reputable scientific │ │
│ │ │ │ sources. At the same time, I know there's still a lot │ │
│ │ │ │ of debate around the specific impacts and best │ │
│ │ │ │ solutions. I'm eager to continue learning more as I │ │
│ │ │ │ progress in my studies." │ │
│ │ │ read_generated_tokens │ "Yes\n\nAs a student, I've likely had to read at │ │
│ │ │ │ least one book about climate change for a science or │ │
│ │ │ │ environmental studies class. It's a major topic │ │
│ │ │ │ covered in many curricula these days." │ │
│ │ └────────────────────────────┴───────────────────────────────────────────────────────┘ │
│ comments_dict │ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ important_comment │ "As a student, I'm quite concerned about climate change and │ │
│ │ │ │ its long-term impacts on our planet and future generations. │ │
│ │ │ │ It's a major issue we learn about in school, and I feel it's │ │
│ │ │ │ important to be informed and take action where we can." │ │
│ │ │ opinions_comment │ '' │ │
│ │ │ read_comment │ "As a student, I've likely had to read at least one book about │ │
│ │ │ │ climate change for a science or environmental studies class. │ │
│ │ │ │ It's a major topic covered in many curricula these days." │ │
│ │ └───────────────────┴────────────────────────────────────────────────────────────────┘ │
│ _combined_dict │ None │
│ _problem_keys │ None │
│ interview_hash │ 1646262796627658719 │
└────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────┘
Results fields
Results contain fields that can be accessed and analyzed individually or collectively. We can see a list of these fields by calling the columns method:
results.columns
The following list will be returned for the results generated by the above code:
['agent.agent_instruction',
'agent.agent_name',
'agent.persona',
'answer.important',
'answer.opinions',
'answer.read',
'comment.important_comment',
'comment.opinions_comment',
'comment.read_comment',
'generated_tokens.important_generated_tokens',
'generated_tokens.opinions_generated_tokens',
'generated_tokens.read_generated_tokens',
'iteration.iteration',
'model.frequency_penalty',
'model.logprobs',
'model.max_tokens',
'model.model',
'model.presence_penalty',
'model.temperature',
'model.top_logprobs',
'model.top_p',
'prompt.important_system_prompt',
'prompt.important_user_prompt',
'prompt.opinions_system_prompt',
'prompt.opinions_user_prompt',
'prompt.read_system_prompt',
'prompt.read_user_prompt',
'question_options.important_question_options',
'question_options.opinions_question_options',
'question_options.read_question_options',
'question_text.important_question_text',
'question_text.opinions_question_text',
'question_text.read_question_text',
'question_type.important_question_type',
'question_type.opinions_question_type',
'question_type.read_question_type',
'raw_model_response.important_cost',
'raw_model_response.important_one_usd_buys',
'raw_model_response.important_raw_model_response',
'raw_model_response.opinions_cost',
'raw_model_response.opinions_one_usd_buys',
'raw_model_response.opinions_raw_model_response',
'raw_model_response.read_cost',
'raw_model_response.read_one_usd_buys',
'raw_model_response.read_raw_model_response',
'scenario.topic']
The columns include information about each agent, model and corresponding prompts used to simulate the answer to each question and scenario in the survey, together with each raw model response. If the survey was run multiple times (run(n=<integer>)) then the iteration.iteration column will show the iteration number for each result.
Agent information:
agent.instruction: The instruction for the agent. This field is the optional instruction that was passed to the agent when it was created.
agent.agent_name: This field is always included in any Results object. It contains a unique identifier for each Agent that can be specified when an agent is is created (Agent(name=<name>, traits={<traits_dict>})). If not specified, it is added automatically when results are generated (in the form Agent_0, etc.).
agent.persona: Each of the traits that we pass to an agent is represented in a column of the results. Our example code created a “persona” trait for each agent, so our results include a “persona” column for this information. Note that the keys for the traits dictionary should be a valid Python keys.
Answer information:
answer.important: Agent responses to the linear scale important question.
answer.opinions: Agent responses to the free text opinions question.
answer.read: Agent responses to the multiple choice read question.
A “comment” field is automatically included for every question in a survey other than free text questions, to allow the agent to optionally provide additional information about its response to the question (unless the parameter include_comment=False is passed to a question when constructed): * comment.important_comment: Agent commentary on responses to the important question. * comment.opinions_comment: Agent commentary on responses to the opinion question. Note that this field is empty because the question type is `free_text`. * comment.read_comment: Agent commentary on responses to the read question.
Generated tokens information:
generated_tokens.important_generated_tokens: The generated tokens for the important question.
generated_tokens.opinions_generated_tokens: The generated tokens for the opinions question.
generated_tokens.read_generated_tokens: The generated tokens for the read question.
Iteration information:
The iteration column shows the number of the run (run(n=<integer>)) for the combination of components used (scenarios, agents and models).
Model information:
Each of model columns is a modifiable parameter of the models used to generate the responses.
model.frequency_penalty: The frequency penalty for the model.
model.logprobs: The logprobs for the model.
model.max_tokens: The maximum number of tokens for the model.
model.model: The name of the model used.
model.presence_penalty: The presence penalty for the model.
model.temperature: The temperature for the model.
model.top_logprobs: The top logprobs for the model.
model.top_p: The top p for the model.
model.use_cache: Whether the model uses cache.
Prompt information:
prompt.important_system_prompt: The system prompt for the important question.
prompt.important_user_prompt: The user prompt for the important question.
prompt.opinions_system_prompt: The system prompt for the opinions question.
prompt.opinions_user_prompt: The user prompt for the opinions question.
prompt.read_system_prompt: The system prompt for the read question.
prompt.read_user_prompt: The user prompt for the read question.
For more details about prompts, please see the Prompts section.
Question information:
question_options.important_question_options: The options for the important question, if any.
question_options.opinions_question_options: The options for the opinions question, if any.
question_options.read_question_options: The options for the read question, if any.
question_text.important_question_text: The text of the important question.
question_text.opinions_question_text: The text of the opinions question.
question_text.read_question_text: The text of the read question.
question_type.important_question_type: The type of the important question.
question_type.opinions_question_type: The type of the opinions question.
question_type.read_question_type: The type of the read question.
Raw model response information:
raw_model_response.important_cost: The cost of the result for the important question, applying the token quanities & prices.
raw_model_response.important_one_usd_buys: The number of identical results for the important question that 1USD would cover.
raw_model_response.important_raw_model_response: The raw model response for the important question.
raw_model_response.opinions_cost: The cost of the result for the opinions question, applying the token quanities & prices.
raw_model_response.opinions_one_usd_buys: The number of identical results for the opinions question that 1USD would cover.
raw_model_response.opinions_raw_model_response: The raw model response for the opinions question.
raw_model_response.read_cost: The cost of the result for the read question, applying the token quanities & prices.
raw_model_response.read_one_usd_buys: The number of identical results for the read question that 1USD would cover.
raw_model_response.read_raw_model_response: The raw model response for the read question.
Note that the cost of a result for a question is specific to the components (scenario, agent, model used with it).
Scenario information:
scenario.topic: The values provided for the “topic” scenario for the questions.
Creating tables by selecting/dropping and printing
Each of these columns can be accessed directly by calling the select() method and passing the column names. Alternatively, we can specify the columns to exclude by calling the drop() method. These methods can be chained together with the print() method to display the specified columns in a table format.
For example, the following code will print a table showing the answers for read and important together with model, persona and topic columns (because the column names are unique we can drop the model, agent, scenario and answer prefixes when selecting them):
results = survey.by(scenarios).by(agents).by(models).run() # Running the survey once
results.select("model", "persona", "topic", "read", "important").print(format="rich")
A table with the selected columns will be printed:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ data privacy │ No │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└────────────────────────────┴───────────┴────────────────┴────────┴────────────┘
Sorting results
We can sort the columns by calling the sort_by method and passing it the column name to sort by:
(
results
.sort_by("model", reverse=False)
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
The following table will be printed:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ data privacy │ No │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└────────────────────────────┴───────────┴────────────────┴────────┴────────────┘
The sort_by method can be applied multiple times:
(
results
.sort_by("model", reverse=False)
.sort_by("persona", reverse=True)
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
The following table will be printed:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ data privacy │ No │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└────────────────────────────┴───────────┴────────────────┴────────┴────────────┘
Labeling results
We can also add some table labels by passing a dictionary to the pretty_labels argument of the print method (note that we need to include the column prefixes when specifying the table labels, as shown below):
(
results
.sort_by("model", reverse=False)
.sort_by("persona", reverse=True)
.select("model", "persona", "topic", "read", "important")
.print(pretty_labels={
"model.model": "LLM",
"agent.persona": "Agent",
"scenario.topic": "Topic",
"answer.read": q3.question_text,
"answer.important": q1.question_text
}, format="rich")
)
The following table will be printed:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ On a scale from 1 to 5, ┃
┃ ┃ ┃ ┃ Have you read any books ┃ how important to you is ┃
┃ LLM ┃ Agent ┃ Topic ┃ about {{ topic }}? ┃ {{ topic }}? ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ claude-3-5-sonnet-202406… │ student │ climate change │ Yes │ 4 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ claude-3-5-sonnet-202406… │ student │ data privacy │ No │ 3 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ claude-3-5-sonnet-202406… │ celebrity │ climate change │ Yes │ 4 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ claude-3-5-sonnet-202406… │ celebrity │ data privacy │ No │ 4 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└───────────────────────────┴───────────┴────────────────┴────────────────────────────┴───────────────────────────┘
Filtering results
Results can be filtered by using the filter method and passing it a logical expression identifying the results that should be selected. For example, the following code will filter results where the answer to important is “5” and then just print the topic and important_comment columns:
(
results
.filter("important == 5")
.select("topic", "important_comment")
.print(format="rich")
)
This will return an abbreviated table:
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ scenario ┃ comment ┃
┃ .topic ┃ .important_comment ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ climate change │ Climate change is a critical issue that affects everyone and everything on the planet. As a │
│ │ student, I believe it's essential to address it for our future. │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ data privacy │ Data privacy is crucial to me because it protects my personal information and ensures that my │
│ │ data is not misused or accessed without my consent. │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ climate change │ Climate change is one of the most pressing issues of our time, and as a public figure, I │
│ │ believe it's crucial to use my platform to raise awareness and advocate for sustainable │
│ │ practices. │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ data privacy │ Data privacy is crucial, especially as a public figure. Protecting personal information is │
│ │ essential to maintaining security and trust. │
└────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────┘
Note: The filter method allows us to pass the unique short names of the columns (without the prefixes) when specifying the logical expression. However, because the model.model column name is also a prefix, we need to include the prefix when filtering by this column, as shown in the example below:
(
results
.filter("model.model == 'gpt-4o'")
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
This will return a table of results where the model is “gpt-4o”:
┏━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└────────┴───────────┴────────────────┴────────┴────────────┘
Limiting results
We can select and print a limited number of results by passing the desired number of max_rows to the print() method. This can be useful for quickly checking the first few results:
(
results
.select("model", "persona", "topic", "read", "important")
.print(max_rows=4, format="rich")
)
This will return a table of the selected components of the first 4 results:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼──────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼──────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼──────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
└────────────────────────────┴──────────┴────────────────┴────────┴────────────┘
Sampling results
We can select a sample of n results by passing the desired number of random results to the sample() method. This can be useful for checking a random subset of the results with different parameters:
sample_results = results.sample(2)
(
sample_results
.sort_by("model", reverse=False)
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
This will return a table of the specified number of randomly selected results:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼──────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
└────────────────────────────┴──────────┴────────────────┴────────┴────────────┘
Shuffling results
We can shuffle results by calling the shuffle() method. This can be useful for quickly checking the first few results:
shuffle_results = results.shuffle()
(
shuffle_results
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
This will return a table of shuffled results:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ celebrity │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ data privacy │ No │ 4 │
└────────────────────────────┴───────────┴────────────────┴────────┴────────────┘
Adding results
We can add results together straightforwardly by using the + operator:
add_results = results + results
We can see that the results have doubled:
len(add_results)
This will return the number of results:
16
Displaying results in a tree format
We can display the results in a tree format using the tree method, which displays the results in a nested format for each of the components: model, scenario, agent, answer, question, iteration. The method takes a list parameter fold_attributes for the attributes to be folded, and an option list parmeter drop for fields to be excluded from the display.
For example, the following code will display the results in a tree format for the model and scenario components, excluding the iteration component:
results.tree(fold_attributes=["model", "scenario"], drop=["iteration"])
This will display the results in a tree format:
model: Model(model_name = 'claude-3-5-sonnet-20240620', temperature = 0.5, max_tokens = 1000, top_p = 1, frequency_penalty = 0, presence_penalty = 0, logprobs = False, top_logprobs = 3)
model: Model(model_name = 'gpt-4o', temperature = 0.5, max_tokens = 1000, top_p = 1, frequency_penalty = 0, presence_penalty = 0, logprobs = False, top_logprobs = 3)
Generating HTML reports
We can generate an HTML report of the results by calling the generate_html and save_html methods. The generate_html method will create an HTML report of the results, and the save_html method will save the report to a specified file path (default filename: output.html).
results.generate_html().save_html("output.html")
Interacting via SQL
We can interact with the results via SQL using the sql method. This is done by passing a SQL query and a shape (“long” or “wide”) for the resulting table, where the table name in the query is “self”.
The “wide” shape will return a table with each result as a row and columns for the selected columns of the results. For example, the following code will return a table showing the model, persona, read and important columns for the first 4 results:
results.sql("select model, persona, read, important from self limit 4", shape="wide")
This following table will be displayed:
model persona read important
0 claude-3-5-sonnet-20240620 student Yes 4
1 gpt-4o student Yes 5
2 claude-3-5-sonnet-20240620 student No 3
3 gpt-4o student Yes 5
The “long” shape lets us instead treat the components of the results as rows. There are 4 columns in the resulting table:
data_type: The component type within the results (i.e., the column prefixes referred to above).
key: The name of the component (e.g., the prefix question_text).
value: The actual component (e.g., the individual question texts).
id: The number of the Result object within the Results.
Because a Result includes answers for all of the questions in a survey, the all of the questions of a Result share the same id.
For example, the following code will return a table showing the question_text data for all of the results:
results.sql("select * from self where data_type = 'question_text'", shape="long")
This following table will be displayed:
id data_type key value
0 0 question_text important_question_text On a scale from 1 to 5, how important to you i...
1 0 question_text opinions_question_text What are your opinions on {{ topic }}?
2 0 question_text read_question_text Have you read any books about {{ topic }}?
3 1 question_text important_question_text On a scale from 1 to 5, how important to you i...
4 1 question_text opinions_question_text What are your opinions on {{ topic }}?
5 1 question_text read_question_text Have you read any books about {{ topic }}?
6 2 question_text important_question_text On a scale from 1 to 5, how important to you i...
7 2 question_text opinions_question_text What are your opinions on {{ topic }}?
8 2 question_text read_question_text Have you read any books about {{ topic }}?
9 3 question_text important_question_text On a scale from 1 to 5, how important to you i...
10 3 question_text opinions_question_text What are your opinions on {{ topic }}?
11 3 question_text read_question_text Have you read any books about {{ topic }}?
12 4 question_text important_question_text On a scale from 1 to 5, how important to you i...
13 4 question_text opinions_question_text What are your opinions on {{ topic }}?
14 4 question_text read_question_text Have you read any books about {{ topic }}?
15 5 question_text important_question_text On a scale from 1 to 5, how important to you i...
16 5 question_text opinions_question_text What are your opinions on {{ topic }}?
17 5 question_text read_question_text Have you read any books about {{ topic }}?
18 6 question_text important_question_text On a scale from 1 to 5, how important to you i...
19 6 question_text opinions_question_text What are your opinions on {{ topic }}?
20 6 question_text read_question_text Have you read any books about {{ topic }}?
21 7 question_text important_question_text On a scale from 1 to 5, how important to you i...
22 7 question_text opinions_question_text What are your opinions on {{ topic }}?
23 7 question_text read_question_text Have you read any books about {{ topic }}?
Dataframes
We can also export results to other formats. The to_pandas method will turn our results into a Pandas dataframe:
results.to_pandas()
For example, here we use it to create a dataframe consisting of the models, personas and the answers to the important question:
results.to_pandas()[["model.model", "agent.persona", "answer.important"]]
This will display our new dataframe:
model.model agent.persona answer.important
0 claude-3-5-sonnet-20240620 student 4
1 gpt-4o student 5
2 claude-3-5-sonnet-20240620 student 3
3 gpt-4o student 5
4 claude-3-5-sonnet-20240620 celebrity 4
5 gpt-4o celebrity 5
6 claude-3-5-sonnet-20240620 celebrity 4
7 gpt-4o celebrity 5
Exporting to CSV or JSON
The to_csv method will write the results to a CSV file:
results.to_pandas().to_csv("results.csv")
The to_json method will write the results to a JSON file:
results.to_pandas().to_json("results.json")
Exceptions
If any exceptions are raised when the survey is run a detailed exceptions report is generated and will open automatically. See the Exceptions & Debugging section for more information on exceptions.
Result class
- class edsl.results.Result.PromptDict(dict=None, /, **kwargs)[source]
A dictionary that is used to store the prompt for a given result.
- pop(k[, d]) v, remove specified key and return the corresponding value.[source]
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair[source]
as a 2-tuple; but raise KeyError if D is empty.
- class edsl.results.Result.Result(agent: Agent, scenario: Scenario, model: Type['LanguageModel'], iteration: int, answer: str, prompt: dict[str, str] = None, raw_model_response=None, survey: 'Survey' | None = None, question_to_attributes: dict | None = None, generated_tokens: dict | None = None, comments_dict: dict | None = None, cache_used_dict: dict | None = None)[source]
This class captures the result of one interview.
The answer dictionary has the structure:
>>> import warnings >>> warnings.simplefilter("ignore", UserWarning) >>> Result.example().answer == {'how_feeling_yesterday': 'Great', 'how_feeling': 'OK'} True
Its main data is an Agent, a Scenario, a Model, an Iteration, and an Answer. These are stored both in the UserDict and as attributes.
- __init__(agent: Agent, scenario: Scenario, model: Type['LanguageModel'], iteration: int, answer: str, prompt: dict[str, str] = None, raw_model_response=None, survey: 'Survey' | None = None, question_to_attributes: dict | None = None, generated_tokens: dict | None = None, comments_dict: dict | None = None, cache_used_dict: dict | None = None)[source]
Initialize a Result object.
- Parameters:
agent – The Agent object.
scenario – The Scenario object.
model – The LanguageModel object.
iteration – The iteration number.
answer – The answer string.
prompt – A dictionary of prompts.
raw_model_response – The raw model response.
- property combined_dict: dict[str, Any][source]
Return a dictionary that includes all sub_dicts, but also puts the key-value pairs in each sub_dict as a key_value pair in the combined dictionary.
>>> r = Result.example() >>> r.combined_dict['how_feeling'] 'OK'
- classmethod delete(uuid: str | UUID | None = None, url: str | None = None)[source]
Delete the object from coop.
- classmethod from_dict(json_dict: dict) Result[source]
Return a Result object from a dictionary representation.
- get_value(data_type: str, key: str) Any[source]
Return the value for a given data type and key.
>>> r = Result.example() >>> r.get_value("answer", "how_feeling") 'OK'
data types can be “agent”, “scenario”, “model”, or “answer”
keys are relevant attributes of the Objects the data types represent
- property key_to_data_type: dict[str, str][source]
Return a dictionary where keys are object attributes and values are the data type (object) that the attribute is associated with.
>>> r = Result.example() >>> r.key_to_data_type["how_feeling"] 'answer'
- classmethod patch(uuid: str | UUID | None = None, url: str | None = None, description: str | None = None, value: Any | None = None, visibility: str | None = None)[source]
Patch an uploaded objects attributes. - description changes the description of the object on Coop - value changes the value of the object on Coop. has to be an EDSL object - visibility changes the visibility of the object on Coop
- pop(k[, d]) v, remove specified key and return the corresponding value.[source]
If key is not found, d is returned if given, otherwise KeyError is raised.
- popitem() (k, v), remove and return some (key, value) pair[source]
as a 2-tuple; but raise KeyError if D is empty.
- classmethod pull(uuid: str | UUID | None = None, url: str | None = None, expected_parrot_url: str | None = None)[source]
Pull the object from coop.
- push(description: str | None = None, visibility: str | None = 'unlisted', expected_parrot_url: str | None = None)[source]
Post the object to coop.
- save(filename, compress=True)[source]
Save the object to a file as zippped JSON.
>>> obj.save("obj.json.gz")
- score(scoring_function: Callable) Any[source]
Score the result using a passed-in scoring function.
>>> def f(status): return 1 if status == 'Joyful' else 0 >>> Result.example().score(f) 1
- property sub_dicts: dict[str, dict][source]
Return a dictionary where keys are strings for each of the main class attributes/objects.
- to_dict(add_edsl_version=True) dict[str, Any][source]
Return a dictionary representation of the Result object.
>>> r = Result.example() >>> r.to_dict()['scenario'] {'period': 'morning', 'edsl_version': '...', 'edsl_class_name': 'Scenario'}
Results class
The Results object is the result of running a survey. It is not typically instantiated directly, but is returned by the run method of a Job object.
- class edsl.results.Results.Results(survey: Survey | None = None, data: list[Result] | None = None, created_columns: list[str] | None = None, cache: Cache | None = None, job_uuid: str | None = None, total_results: int | None = None, task_history: TaskHistory | None = None)[source]
This class is a UserList of Result objects.
It is instantiated with a Survey and a list of Result objects. It can be manipulated in various ways with select, filter, mutate, etc. It also has a list of created_columns, which are columns that have been created with mutate and are not part of the original data.
- __init__(survey: Survey | None = None, data: list[Result] | None = None, created_columns: list[str] | None = None, cache: Cache | None = None, job_uuid: str | None = None, total_results: int | None = None, task_history: TaskHistory | None = None)[source]
Instantiate a Results object with a survey and a list of Result objects.
- Parameters:
survey – A Survey object.
data – A list of Result objects.
created_columns – A list of strings that are created columns.
job_uuid – A string representing the job UUID.
total_results – An integer representing the total number of results.
- add_column(column_name: str, values: list) Results[source]
Adds columns to Results
>>> r = Results.example() >>> r.add_column('a', [1,2,3, 4]).select('a') Dataset([{'answer.a': [1, 2, 3, 4]}])
- add_columns_from_dict(columns: List[dict]) Results[source]
Adds columns to Results from a list of dictionaries.
>>> r = Results.example() >>> r.add_columns_from_dict([{'a': 1, 'b': 2}, {'a': 3, 'b': 4}, {'a':3, 'b':2}, {'a':3, 'b':2}]).select('a', 'b') Dataset([{'answer.a': [1, 3, 3, 3]}, {'answer.b': [2, 4, 2, 2]}])
- property agent_keys: list[str][source]
Return a set of all of the keys that are in the Agent data.
Example:
>>> r = Results.example() >>> r.agent_keys ['agent_instruction', 'agent_name', 'status']
- property agents: AgentList[source]
Return a list of all of the agents in the Results.
Example:
>>> r = Results.example() >>> r.agents AgentList([Agent(traits = {'status': 'Joyful'}), Agent(traits = {'status': 'Joyful'}), Agent(traits = {'status': 'Sad'}), Agent(traits = {'status': 'Sad'})])
- property all_keys: list[str][source]
Return a set of all of the keys that are in the Results.
Example:
>>> r = Results.example() >>> r.all_keys ['agent_instruction', 'agent_name', 'frequency_penalty', 'how_feeling', 'how_feeling_yesterday', 'logprobs', 'max_tokens', 'model', 'period', 'presence_penalty', 'status', 'temperature', 'top_logprobs', 'top_p']
- property answer_keys: dict[str, str][source]
Return a mapping of answer keys to question text.
Example:
>>> r = Results.example() >>> r.answer_keys {'how_feeling': 'How are you this {{ period }}?', 'how_feeling_yesterday': 'How were you feeling yesterday {{ period }}?'}
- answer_truncate(column: str, top_n=5, new_var_name=None) Results[source]
Create a new variable that truncates the answers to the top_n.
- Parameters:
column – The column to truncate.
top_n – The number of top answers to keep.
new_var_name – The name of the new variable. If None, it is the original name + ‘_truncated’.
- answers_to_themes(field, context: str, themes: list, progress_bar=False, print_exceptions=False) dict[source]
- auto_theme(field: str, context: str, themes: list[str], newfield: str = None, progress_bar=False, print_exceptions=False) tuple[source]
- Parameters:
field – The field to be themed.
context – The context of the field.
themes – The list of themes.
newfield – The new field name.
- backup_db_to_file(shape: Literal['wide', 'long'], filename: str)[source]
Backup the in-memory database to a file.
- Parameters:
shape – The shape of the data in the database (wide or long)
filename – The filename to save the database to
>>> from edsl.results import Results >>> r = Results.example() >>> r.backup_db_to_file(filename="backup.db", shape="long")
- property columns: list[str][source]
Return a list of all of the columns that are in the Results.
Example:
>>> r = Results.example() >>> r.columns ['agent.agent_instruction', ...]
- classmethod delete(uuid: str | UUID | None = None, url: str | None = None)[source]
Delete the object from coop.
- download_link(pretty_labels: dict | None = None) str[source]
Return a download link for the results.
- Parameters:
pretty_labels – A dictionary of pretty labels for the columns.
>>> from edsl.results import Results >>> r = Results.example() >>> r.select('how_feeling').download_link() '<a href="data:file/csv;base64,YW5zd2VyLmhvd19mZWVsaW5nDQpPSw0KR3JlYXQNClRlcnJpYmxlDQpPSw0K" download="my_data.csv">Download CSV file</a>'
- classmethod example(randomize: bool = False) Results[source]
Return an example Results object.
Example usage:
>>> r = Results.example()
- Parameters:
debug – if False, uses actual API calls
- export_sql_dump(shape: Literal['wide', 'long'], filename: str)[source]
Export the SQL database to a file.
- Parameters:
shape – The shape of the data in the database (wide or long)
filename – The filename to save the database to
- filter(expression: str) Results[source]
Filter based on the given expression and returns the filtered Results.
- Parameters:
expression – A string expression that evaluates to a boolean. The expression is applied to each element in Results to determine whether it should be included in the filtered results.
The expression parameter is a string that must resolve to a boolean value when evaluated against each element in Results. This expression is used to determine which elements to include in the returned Results.
Example usage: Create an example Results instance and apply filters to it:
>>> r = Results.example() >>> r.filter("how_feeling == 'Great'").select('how_feeling').print() ┏━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .how_feeling ┃ ┡━━━━━━━━━━━━━━┩ │ Great │ └──────────────┘
Example usage: Using an OR operator in the filter expression.
>>> r = Results.example().filter("how_feeling = 'Great'").select('how_feeling').print() Traceback (most recent call last): ... edsl.exceptions.results.ResultsFilterError: You must use '==' instead of '=' in the filter expression. ...
>>> r.filter("how_feeling == 'Great' or how_feeling == 'Terrible'").select('how_feeling').print() ┏━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .how_feeling ┃ ┡━━━━━━━━━━━━━━┩ │ Great │ ├──────────────┤ │ Terrible │ └──────────────┘
- first() Result[source]
Return the first observation in the results.
Example:
>>> r = Results.example() >>> r.first() Result(agent...
- classmethod from_dict(data: dict[str, Any]) Results[source]
Convert a dictionary to a Results object.
- Parameters:
data – A dictionary representation of a Results object.
Example:
>>> r = Results.example() >>> d = r.to_dict() >>> r2 = Results.from_dict(d) >>> r == r2 True
- get_themes(field: str, context: str, max_values=100, num_themes: int = 10, seed=None, progress_bar=False, print_exceptions=False) list[source]
- ggplot2(ggplot_code: str, filename: str = None, shape='wide', sql: str = None, remove_prefix: bool = True, debug: bool = False, height=4, width=6, format='svg', factor_orders: dict | None = None)[source]
Create a ggplot2 plot from a DataFrame.
- Parameters:
ggplot_code – The ggplot2 code to execute.
filename – The filename to save the plot to.
shape – The shape of the data in the DataFrame (wide or long).
sql – The SQL query to execute beforehand to manipulate the data.
remove_prefix – Whether to remove the prefix from the column names.
debug – Whether to print the R code instead of executing it.
height – The height of the plot in inches.
width – The width of the plot in inches.
format – The format to save the plot in (png or svg).
factor_orders – A dictionary of factor columns and their order.
- property model_keys: list[str][source]
Return a set of all of the keys that are in the LanguageModel data.
>>> r = Results.example() >>> r.model_keys ['frequency_penalty', 'logprobs', 'max_tokens', 'model', 'presence_penalty', 'temperature', 'top_logprobs', 'top_p']
- property models: ModelList[source]
Return a list of all of the models in the Results.
Example:
>>> r = Results.example() >>> r.models[0] Model(model_name = ...)
- mutate(new_var_string: str, functions_dict: dict | None = None) Results[source]
Creates a value in the Results object as if has been asked as part of the survey.
- Parameters:
new_var_string – A string that is a valid Python expression.
functions_dict – A dictionary of functions that can be used in the expression. The keys are the function names and the values are the functions themselves.
It splits the new_var_string at the “=” and uses simple_eval
Example:
>>> r = Results.example() >>> r.mutate('how_feeling_x = how_feeling + "x"').select('how_feeling_x') Dataset([{'answer.how_feeling_x': ...
- num_observations()[source]
Return the number of observations in the dataset.
>>> from edsl.results import Results >>> Results.example().num_observations() 4
- order_by(*columns: str, reverse: bool = False) Results[source]
Sort the results by one or more columns.
- Parameters:
columns – One or more column names as strings.
reverse – A boolean that determines whether to sort in reverse order.
Each column name can be a single key, e.g. “how_feeling”, or a dot-separated string, e.g. “answer.how_feeling”.
Example:
>>> r = Results.example() >>> r.sort_by('how_feeling', reverse=False).select('how_feeling').print() ┏━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .how_feeling ┃ ┡━━━━━━━━━━━━━━┩ │ Great │ ├──────────────┤ │ OK │ ├──────────────┤ │ OK │ ├──────────────┤ │ Terrible │ └──────────────┘ >>> r.sort_by('how_feeling', reverse=True).select('how_feeling').print() ┏━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .how_feeling ┃ ┡━━━━━━━━━━━━━━┩ │ Terrible │ ├──────────────┤ │ OK │ ├──────────────┤ │ OK │ ├──────────────┤ │ Great │ └──────────────┘
- classmethod patch(uuid: str | UUID | None = None, url: str | None = None, description: str | None = None, value: Any | None = None, visibility: str | None = None)[source]
Patch an uploaded objects attributes. - description changes the description of the object on Coop - value changes the value of the object on Coop. has to be an EDSL object - visibility changes the visibility of the object on Coop
- print(pretty_labels: dict | None = None, filename: str | None = None, format: Literal['rich', 'html', 'markdown', 'latex'] | None = None, interactive: bool = False, split_at_dot: bool = True, max_rows=None, tee=False, iframe=False, iframe_height: int = 200, iframe_width: int = 600, web=False, return_string: bool = False) None | str | Results[source]
Print the results in a pretty format.
- Parameters:
pretty_labels – A dictionary of pretty labels for the columns.
filename – The filename to save the results to.
format – The format to print the results in. Options are ‘rich’, ‘html’, ‘markdown’, or ‘latex’.
interactive – Whether to print the results interactively in a Jupyter notebook.
split_at_dot – Whether to split the column names at the last dot w/ a newline.
max_rows – The maximum number of rows to print.
tee – Whether to return the dataset.
iframe – Whether to display the table in an iframe.
iframe_height – The height of the iframe.
iframe_width – The width of the iframe.
web – Whether to display the table in a web browser.
return_string – Whether to return the output as a string instead of printing.
- Returns:
None if tee is False and return_string is False, the dataset if tee is True, or a string if return_string is True.
Example: Print in rich format at the terminal
>>> from edsl.results import Results >>> r = Results.example() >>> r.select('how_feeling').print(format = "rich") ┏━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .how_feeling ┃ ┡━━━━━━━━━━━━━━┩ │ OK │ ├──────────────┤ │ Great │ ├──────────────┤ │ Terrible │ ├──────────────┤ │ OK │ └──────────────┘
>>> r = Results.example() >>> r2 = r.select("how_feeling").print(format = "rich", tee = True, max_rows = 2) ┏━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .how_feeling ┃ ┡━━━━━━━━━━━━━━┩ │ OK │ ├──────────────┤ │ Great │ └──────────────┘ >>> r2 Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}])
>>> r.select('how_feeling').print(format = "rich", max_rows = 2) ┏━━━━━━━━━━━━━━┓ ┃ answer ┃ ┃ .how_feeling ┃ ┡━━━━━━━━━━━━━━┩ │ OK │ ├──────────────┤ │ Great │ └──────────────┘
>>> r.select('how_feeling').print(format = "rich", split_at_dot = False) ┏━━━━━━━━━━━━━━━━━━━━┓ ┃ answer.how_feeling ┃ ┡━━━━━━━━━━━━━━━━━━━━┩ │ OK │ ├────────────────────┤ │ Great │ ├────────────────────┤ │ Terrible │ ├────────────────────┤ │ OK │ └────────────────────┘
Example: using the pretty_labels parameter
>>> r.select('how_feeling').print(format="rich", pretty_labels = {'answer.how_feeling': "How are you feeling"}) ┏━━━━━━━━━━━━━━━━━━━━━┓ ┃ How are you feeling ┃ ┡━━━━━━━━━━━━━━━━━━━━━┩ │ OK │ ├─────────────────────┤ │ Great │ ├─────────────────────┤ │ Terrible │ ├─────────────────────┤ │ OK │ └─────────────────────┘
Example: printing in markdown format
>>> r.select('how_feeling').print(format='markdown') | answer.how_feeling | |--| | OK | | Great | | Terrible | | OK | ...
>>> r.select('how_feeling').print(format='latex') \begin{tabular}{l} ... \end{tabular}
- print_long()[source]
Print the results in a long format. >>> from edsl.results import Results >>> r = Results.example() >>> r.select(‘how_feeling’).print_long() answer.how_feeling: OK answer.how_feeling: Great answer.how_feeling: Terrible answer.how_feeling: OK
- classmethod pull(uuid: str | UUID | None = None, url: str | None = None, expected_parrot_url: str | None = None)[source]
Pull the object from coop.
- push(description: str | None = None, visibility: str | None = 'unlisted', expected_parrot_url: str | None = None)[source]
Post the object to coop.
- property question_names: list[str][source]
Return a list of all of the question names.
Example:
>>> r = Results.example() >>> r.question_names ['how_feeling', 'how_feeling_yesterday']
- recode(column: str, recode_function: Callable | None, new_var_name=None) Results[source]
Recode a column in the Results object.
>>> r = Results.example() >>> r.recode('how_feeling', recode_function = lambda x: 1 if x == 'Great' else 0).select('how_feeling', 'how_feeling_recoded') Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling_recoded': [0, 1, 0, 0]}])
- relevant_columns(data_type: str | None = None, remove_prefix=False) list[source]
Return the set of keys that are present in the dataset.
- Parameters:
data_type – The data type to filter by.
remove_prefix – Whether to remove the prefix from the column names.
>>> from edsl.results.Dataset import Dataset >>> d = Dataset([{'a.b':[1,2,3,4]}]) >>> d.relevant_columns() ['a.b']
>>> d.relevant_columns(remove_prefix=True) ['b']
>>> d = Dataset([{'a':[1,2,3,4]}, {'b':[5,6,7,8]}]) >>> d.relevant_columns() ['a', 'b']
>>> from edsl.results import Results; Results.example().select('how_feeling', 'how_feeling_yesterday').relevant_columns() ['answer.how_feeling', 'answer.how_feeling_yesterday']
>>> from edsl.results import Results >>> sorted(Results.example().select().relevant_columns(data_type = "model")) ['model.frequency_penalty', 'model.logprobs', 'model.max_tokens', 'model.model', 'model.presence_penalty', 'model.temperature', 'model.top_logprobs', 'model.top_p']
>>> Results.example().relevant_columns(data_type = "flimflam") Traceback (most recent call last): ... ValueError: No columns found for data type: flimflam. Available data types are: ...
- rename(old_name: str, new_name: str) Results[source]
Rename an answer column in a Results object.
>>> s = Results.example() >>> s.rename('how_feeling', 'how_feeling_new').select('how_feeling_new') Dataset([{'answer.how_feeling_new': ['OK', 'Great', 'Terrible', 'OK']}])
# TODO: Should we allow renaming of scenario fields as well? Probably.
- sample(n: int | None = None, frac: float | None = None, with_replacement: bool = True, seed: str | None = 'edsl') Results[source]
Sample the results.
- Parameters:
n – An integer representing the number of samples to take.
frac – A float representing the fraction of samples to take.
with_replacement – A boolean representing whether to sample with replacement.
seed – An integer representing the seed for the random number generator.
Example:
>>> r = Results.example() >>> len(r.sample(2)) 2
- save(filename, compress=True)[source]
Save the object to a file as zippped JSON.
>>> obj.save("obj.json.gz")
- property scenario_keys: list[str][source]
Return a set of all of the keys that are in the Scenario data.
>>> r = Results.example() >>> r.scenario_keys ['period']
- property scenarios: ScenarioList[source]
Return a list of all of the scenarios in the Results.
Example:
>>> r = Results.example() >>> r.scenarios ScenarioList([Scenario({'period': 'morning'}), Scenario({'period': 'afternoon'}), Scenario({'period': 'morning'}), Scenario({'period': 'afternoon'})])
- score(f: Callable) list[source]
Score the results using in a function.
- Parameters:
f – A function that takes values from a Resul object and returns a score.
>>> r = Results.example() >>> def f(status): return 1 if status == 'Joyful' else 0 >>> r.score(f) [1, 1, 0, 0]
- select(*columns: str | list[str]) Results[source]
Select data from the results and format it.
- Parameters:
columns – A list of strings, each of which is a column name. The column name can be a single key, e.g. “how_feeling”, or a dot-separated string, e.g. “answer.how_feeling”.
Example:
>>> results = Results.example() >>> results.select('how_feeling') Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}])
>>> results.select('how_feeling', 'model', 'how_feeling') Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'model.model': ['...', '...', '...', '...']}, {'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}])
>>> from edsl import Results; r = Results.example(); r.select('answer.how_feeling_y') Dataset([{'answer.how_feeling_yesterday': ['Great', 'Good', 'OK', 'Terrible']}])
- show_schema(shape: Literal['wide', 'long'], remove_prefix: bool = False) None[source]
Show the schema of the Results database.
- Parameters:
shape – The shape of the data in the database (wide or long)
remove_prefix – Whether to remove the prefix from the column names
>>> from edsl.results import Results >>> r = Results.example() >>> r.show_schema(shape="long") Type: table, Name: self, SQL: CREATE TABLE self ( ...
- shuffle(seed: str | None = 'edsl') Results[source]
Shuffle the results.
Example:
>>> r = Results.example() >>> r.shuffle(seed = 1)[0] Result(...)
- sql(query: str, shape: Literal['wide', 'long'] = 'wide', remove_prefix: bool = True, transpose: bool = None, transpose_by: str = None, csv: bool = False, to_list=False, to_latex=False, filename: str | None = None) pd.DataFrame | str[source]
Execute a SQL query and return the results as a DataFrame.
- Parameters:
query – The SQL query to execute
shape – The shape of the data in the database (wide or long)
remove_prefix – Whether to remove the prefix from the column names
transpose – Whether to transpose the DataFrame
transpose_by – The column to use as the index when transposing
csv – Whether to return the DataFrame as a CSV string
Example usage:
>>> from edsl.results import Results >>> r = Results.example() >>> d = r.sql("select data_type, key, value from self where data_type = 'answer' order by value limit 3", shape="long") >>> sorted(list(d['value'])) ['Good', 'Great', 'Great']
We can also return the data in wide format. Note the use of single quotes to escape the column names, as required by sql.
>>> from edsl.results import Results >>> Results.example().sql("select how_feeling from self", shape = 'wide', remove_prefix=True) how_feeling 0 OK 1 Great 2 Terrible 3 OK
- tally(*fields: str | None, top_n: int | None = None, output='Dataset') dict | Dataset[source]
Tally the values of a field or perform a cross-tab of multiple fields.
- Parameters:
fields – The field(s) to tally, multiple fields for cross-tabulation.
>>> from edsl.results import Results >>> r = Results.example() >>> r.select('how_feeling').tally('answer.how_feeling', output = "dict") {'OK': 2, 'Great': 1, 'Terrible': 1} >>> r.select('how_feeling').tally('answer.how_feeling', output = "Dataset") Dataset([{'value': ['OK', 'Great', 'Terrible']}, {'count': [2, 1, 1]}]) >>> r.select('how_feeling', 'period').tally('how_feeling', 'period', output = "dict") {('OK', 'morning'): 1, ('Great', 'afternoon'): 1, ('Terrible', 'morning'): 1, ('OK', 'afternoon'): 1}
- to_agent_list(remove_prefix: bool = True)[source]
Convert the results to a list of dictionaries, one per agent.
- Parameters:
remove_prefix – Whether to remove the prefix from the column names.
>>> from edsl.results import Results >>> r = Results.example() >>> r.select('how_feeling').to_agent_list() AgentList([Agent(traits = {'how_feeling': 'OK'}), Agent(traits = {'how_feeling': 'Great'}), Agent(traits = {'how_feeling': 'Terrible'}), Agent(traits = {'how_feeling': 'OK'})])
- to_csv(filename: str | None = None, remove_prefix: bool = False, download_link: bool = False, pretty_labels: dict | None = None)[source]
Export the results to a CSV file.
- Parameters:
filename – The filename to save the CSV file to.
remove_prefix – Whether to remove the prefix from the column names.
download_link – Whether to display a download link in a Jupyter notebook.
Example:
>>> from edsl.results import Results >>> r = Results.example() >>> r.select('how_feeling').to_csv() 'answer.how_feeling\r\nOK\r\nGreat\r\nTerrible\r\nOK\r\n'
>>> r.select('how_feeling').to_csv(pretty_labels = {'answer.how_feeling': "How are you feeling"}) 'How are you feeling\r\nOK\r\nGreat\r\nTerrible\r\nOK\r\n'
>>> import tempfile >>> filename = tempfile.NamedTemporaryFile(delete=False).name >>> r.select('how_feeling').to_csv(filename = filename) >>> import os >>> import csv >>> with open(filename, newline='') as f: ... reader = csv.reader(f) ... for row in reader: ... print(row) ['answer.how_feeling'] ['OK'] ['Great'] ['Terrible'] ['OK']
- to_dict(sort=False, add_edsl_version=False) dict[str, Any][source]
This method should be implemented by subclasses.
- to_dicts(remove_prefix: bool = True) list[dict][source]
Convert the results to a list of dictionaries.
- Parameters:
remove_prefix – Whether to remove the prefix from the column names.
>>> from edsl.results import Results >>> r = Results.example() >>> r.select('how_feeling').to_dicts() [{'how_feeling': 'OK'}, {'how_feeling': 'Great'}, {'how_feeling': 'Terrible'}, {'how_feeling': 'OK'}]
- to_list(flatten=False, remove_none=False, unzipped=False) list[list][source]
Convert the results to a list of lists.
- Parameters:
flatten – Whether to flatten the list of lists.
remove_none – Whether to remove None values from the list.
>>> from edsl.results import Results >>> Results.example().select('how_feeling', 'how_feeling_yesterday') Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling_yesterday': ['Great', 'Good', 'OK', 'Terrible']}])
>>> Results.example().select('how_feeling', 'how_feeling_yesterday').to_list() [('OK', 'Great'), ('Great', 'Good'), ('Terrible', 'OK'), ('OK', 'Terrible')]
>>> r = Results.example() >>> r.select('how_feeling').to_list() ['OK', 'Great', 'Terrible', 'OK']
>>> from edsl.results.Dataset import Dataset >>> Dataset([{'a.b': [[1, 9], 2, 3, 4]}]).select('a.b').to_list(flatten = True) [1, 9, 2, 3, 4]
>>> from edsl.results.Dataset import Dataset >>> Dataset([{'a.b': [[1, 9], 2, 3, 4]}, {'c': [6, 2, 3, 4]}]).select('a.b', 'c').to_list(flatten = True) Traceback (most recent call last): ... ValueError: Cannot flatten a list of lists when there are multiple columns selected.
- to_pandas(remove_prefix: bool = False, lists_as_strings=False) DataFrame[source]
Convert the results to a pandas DataFrame, ensuring that lists remain as lists.
- Parameters:
remove_prefix – Whether to remove the prefix from the column names.
- to_scenario_list(remove_prefix: bool = True) list[dict][source]
Convert the results to a list of dictionaries, one per scenario.
- Parameters:
remove_prefix – Whether to remove the prefix from the column names.
>>> from edsl.results import Results >>> r = Results.example() >>> r.select('how_feeling').to_scenario_list() ScenarioList([Scenario({'how_feeling': 'OK'}), Scenario({'how_feeling': 'Great'}), Scenario({'how_feeling': 'Terrible'}), Scenario({'how_feeling': 'OK'})])