Results
A Results object is the result of running a survey. It is a list of individual Result objects, each of which represents a response to a Survey for each combination of Agent, Model and Scenario objects that were used with the survey. For example, the Results of a survey administered to 2 agents and 2 language models with no question scenarios will contain 4 individual Result objects. If the survey questions are parameterized with 2 scenarios then the survey Results will include 8 Result objects.
A Results object is not typically instantiated directly, but is returned by calling the run() method of a survey after any agents, language models and scenarios are added to it. To inspect the form of an example Results we can call the example() method (it is long – we show it at the end of this page):
from edsl import Results
example_results = Results.example()
We can see the number of Result objects created by inspecting the length of the Results:
len(example_results)
Output:
4
We can verify that object types:
type(example_results)
Output
edsl.results.Results.Results
And the 4 Result objects:
type(example_results[0])
Output:
edsl.results.Results.Result
Note: You must have API keys for language models in order to generate results. Please see the API Keys section for instructions on activating Remote Inference from your Coop account or storing your own API keys.
For purposes of demonstrating how to unpack and interact with results, we’ll use the following code to generate results for a simple survey. Note that specifying agent traits, scenarios (question parameter values) and language models is optional, and we include those steps here for illustrative purposes:
# Create questions
from edsl import QuestionLinearScale, QuestionFreeText, QuestionMultipleChoice
q1 = QuestionLinearScale(
question_name = "important",
question_text = "On a scale from 1 to 5, how important to you is {{ topic }}?",
question_options = [0, 1, 2, 3, 4, 5],
option_labels = {0:"Not at all", 5:"Very much"}
)
q2 = QuestionFreeText(
question_name = "opinions",
question_text = "What are your opinions on {{ topic }}?"
)
q3 = QuestionMultipleChoice(
question_name = "read",
question_text = "Have you read any books about {{ topic }}?",
question_options = ["Yes", "No", "I do not know"]
)
# Optionally parameterize the questions with scenarios
from edsl import ScenarioList, Scenario
scenarios = ScenarioList(
Scenario({"topic": t}) for t in ["climate change", "data privacy"]
)
# Optionally create agents with traits
from edsl import AgentList, Agent
agents = AgentList(
Agent(traits = {"persona": p}) for p in ["student", "celebrity"]
)
# Optionally specify language models
from edsl import ModelList, Model
models = ModelList(
Model(m) for m in ["claude-3-5-sonnet-20240620", "gpt-4o"]
)
# Create a survey with the questions
from edsl import Survey
survey = Survey([q1, q2, q3])
# Run the survey with the scenarios, agents and models
results = survey.by(scenarios).by(agents).by(models).run()
For more details on each of the above steps, please see the relevant sections of the docs.
Result objects
We can check the number of Result objects created by inspecting the length of the Results:
len(results)
This will count 2 (scenarios) x 2 (agents) x 2 (models) = 8 Result objects:
8
Generating multiple results
If we want to generate multiple results for a survey–i.e., more than 1 result for each combination of Agent, Model and Scenario objects used–we can pass the desired number of iterations when calling the run() method. For example, the following code will generate 3 results for our survey (n=3):
results = survey.by(scenarios).by(agents).by(models).run(n=3)
We can verify that the number of Result objects created is now 24 = 3 iterations x 2 scenarios x 2 agents x 2 models:
len(results)
24
We can readily inspect a result:
results[0]
{
"agent": {
"traits": {
"persona": "student"
},
"edsl_version": "0.1.33.dev1",
"edsl_class_name": "Agent"
},
"scenario": {
"topic": "climate change",
"edsl_version": "0.1.33.dev1",
"edsl_class_name": "Scenario"
},
"model": {
"model": "claude-3-5-sonnet-20240620",
"parameters": {
"temperature": 0.5,
"max_tokens": 1000,
"top_p": 1,
"frequency_penalty": 0,
"presence_penalty": 0,
"logprobs": false,
"top_logprobs": 3
},
"edsl_version": "0.1.33.dev1",
"edsl_class_name": "LanguageModel"
},
"iteration": 0,
"answer": {
"important_generated_tokens": "4\n\nAs a student, I'm quite concerned about climate change and its potential impacts on my future. I've learned about it in classes and feel it's an important issue, though I may not be as actively engaged as some climate activists.",
"important": 4,
"important_comment": "As a student, I'm quite concerned about climate change and its potential impacts on my future. I've learned about it in classes and feel it's an important issue, though I may not be as actively engaged as some climate activists.",
"opinions_generated_tokens": "As a student, I'm still learning about climate change and forming my views on it. From what I've studied so far in my science classes, the scientific consensus seems to be that climate change is a real phenomenon and human activities are contributing to it. I find the topic really interesting and important to understand. I try to stay up to date by reading articles and reports from reputable scientific sources. At the same time, I know there's still a lot of debate around the specific impacts and best solutions. I'm eager to continue learning more as I progress in my studies.",
"opinions": "As a student, I'm still learning about climate change and forming my views on it. From what I've studied so far in my science classes, the scientific consensus seems to be that climate change is a real phenomenon and human activities are contributing to it. I find the topic really interesting and important to understand. I try to stay up to date by reading articles and reports from reputable scientific sources. At the same time, I know there's still a lot of debate around the specific impacts and best solutions. I'm eager to continue learning more as I progress in my studies.",
"read_generated_tokens": "Yes\n\nAs a student, I've likely had to read at least one book about climate change for a science or environmental studies class. It's a major topic covered in many curricula these days.",
"read": "Yes",
"read_comment": "As a student, I've likely had to read at least one book about climate change for a science or environmental studies class. It's a major topic covered in many curricula these days."
},
"prompt": {
"opinions_user_prompt": {
"text": "What are your opinions on climate change?",
"class_name": "Prompt"
},
"opinions_system_prompt": {
"text": "You are answering questions as if you were a human. Do not break character. You are an agent with the following persona:\n{'persona': 'student'}",
"class_name": "AgentInstruction"
},
"important_user_prompt": {
"text": "On a scale from 1 to 5, how important to you is climate change?\n\n0 : Not at all\n\n1 : \n\n2 : \n\n3 : \n\n4 : \n\n5 : A lot\n\nOnly 1 option may be selected.\n\nRespond only with the code corresponding to one of the options. E.g., \"1\" or \"5\" by itself.\n\nAfter the answer, you can put a comment explaining why you chose that option on the next line.",
"class_name": "Prompt"
},
"important_system_prompt": {
"text": "You are answering questions as if you were a human. Do not break character. You are an agent with the following persona:\n{'persona': 'student'}",
"class_name": "AgentInstruction"
},
"read_user_prompt": {
"text": "\nHave you read any books about climate change?\n\n \nYes\n \nNo\n \nI do not know\n \n\nOnly 1 option may be selected.\n\nRespond only with a string corresponding to one of the options.\n\n\nAfter the answer, you can put a comment explaining why you chose that option on the next line.",
"class_name": "Prompt"
},
"read_system_prompt": {
"text": "You are answering questions as if you were a human. Do not break character. You are an agent with the following persona:\n{'persona': 'student'}",
"class_name": "AgentInstruction"
}
},
"raw_model_response": {
"important_raw_model_response": {
"id": "msg_01RtUw3KtpdSTKfxRGqF6vGt",
"content": [
{
"text": "4\n\nAs a student, I'm quite concerned about climate change and its potential impacts on my future. I've learned about it in classes and feel it's an important issue, though I may not be as actively engaged as some climate activists.",
"type": "text"
}
],
"model": "claude-3-5-sonnet-20240620",
"role": "assistant",
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message",
"usage": {
"input_tokens": 152,
"output_tokens": 53
}
},
"opinions_raw_model_response": {
"id": "msg_017mQNAmbkvzuLLtpe7HzhiS",
"content": [
{
"text": "As a student, I'm still learning about climate change and forming my views on it. From what I've studied so far in my science classes, the scientific consensus seems to be that climate change is a real phenomenon and human activities are contributing to it. I find the topic really interesting and important to understand. I try to stay up to date by reading articles and reports from reputable scientific sources. At the same time, I know there's still a lot of debate around the specific impacts and best solutions. I'm eager to continue learning more as I progress in my studies.",
"type": "text"
}
],
"model": "claude-3-5-sonnet-20240620",
"role": "assistant",
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message",
"usage": {
"input_tokens": 49,
"output_tokens": 119
}
},
"read_raw_model_response": {
"id": "msg_01VwAiiNMiwTZQ6Q4jU5hPof",
"content": [
{
"text": "Yes\n\nAs a student, I've likely had to read at least one book about climate change for a science or environmental studies class. It's a major topic covered in many curricula these days.",
"type": "text"
}
],
"model": "claude-3-5-sonnet-20240620",
"role": "assistant",
"stop_reason": "end_turn",
"stop_sequence": null,
"type": "message",
"usage": {
"input_tokens": 114,
"output_tokens": 43
}
}
},
"question_to_attributes": {
"important": {
"question_text": "On a scale from 1 to 5, how important to you is {{ topic }}?",
"question_type": "linear_scale",
"question_options": [
0,
1,
2,
3,
4,
5
]
},
"opinions": {
"question_text": "What are your opinions on {{ topic }}?",
"question_type": "free_text",
"question_options": null
},
"read": {
"question_text": "Have you read any books about {{ topic }}?",
"question_type": "multiple_choice",
"question_options": [
"Yes",
"No",
"I do not know"
]
}
},
"generated_tokens": {
"important_generated_tokens": "4\n\nAs a student, I'm quite concerned about climate change and its potential impacts on my future. I've learned about it in classes and feel it's an important issue, though I may not be as actively engaged as some climate activists.",
"opinions_generated_tokens": "As a student, I'm still learning about climate change and forming my views on it. From what I've studied so far in my science classes, the scientific consensus seems to be that climate change is a real phenomenon and human activities are contributing to it. I find the topic really interesting and important to understand. I try to stay up to date by reading articles and reports from reputable scientific sources. At the same time, I know there's still a lot of debate around the specific impacts and best solutions. I'm eager to continue learning more as I progress in my studies.",
"read_generated_tokens": "Yes\n\nAs a student, I've likely had to read at least one book about climate change for a science or environmental studies class. It's a major topic covered in many curricula these days."
}
}
We can use the rich_print method to display the Result object in a more readable format:
results[0].rich_print()
Result
┏━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Attribute ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ agent │ Agent Attributes │
│ │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ _name │ None │ │
│ │ │ _traits │ {'persona': 'celebrity'} │ │
│ │ │ _codebook │ {} │ │
│ │ │ _instruction │ 'You are answering questions as if you were a human. │ │
│ │ │ │ Do not break character.' │ │
│ │ │ set_instructions │ False │ │
│ │ │ dynamic_traits_function │ None │ │
│ │ │ has_dynamic_traits_function │ False │ │
│ │ │ current_question │ Question('multiple_choice', question_name = │ │
│ │ │ │ """read""", question_text = """Have you read any │ │
│ │ │ │ books about {{ topic }}?""", question_options = │ │
│ │ │ │ ['Yes', 'No', 'I do not know']) │ │
│ │ └─────────────────────────────┴──────────────────────────────────────────────────────┘ │
│ scenario │ Scenario Attributes │
│ │ ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ data │ {'topic': 'data privacy'} │ │
│ │ │ name │ None │ │
│ │ │ _has_image │ False │ │
│ │ └────────────┴───────────────────────────┘ │
│ model │ Language Model │
│ │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ model │ 'gpt-4o' │ │
│ │ │ parameters │ {'temperature': 0.5, 'max_tokens': 1000, 'top_p': 1, │ │
│ │ │ │ 'frequency_penalty': 0, 'presence_penalty': 0, │ │
│ │ │ │ 'logprobs': False, 'top_logprobs': 3} │ │
│ │ │ remote │ False │ │
│ │ │ temperature │ 0.5 │ │
│ │ │ max_tokens │ 1000 │ │
│ │ │ top_p │ 1 │ │
│ │ │ frequency_penalty │ 0 │ │
│ │ │ presence_penalty │ 0 │ │
│ │ │ logprobs │ False │ │
│ │ │ top_logprobs │ 3 │ │
│ │ │ _LanguageModel__rate_limits │ {'rpm': 5000, 'tpm': 600000} │ │
│ │ └─────────────────────────────┴──────────────────────────────────────────────────────┘ │
│ iteration │ 2 │
│ answer │ Answers │
│ │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ opinions_generated_tokens │ "As a celebrity, I think data privacy is incredibly │ │
│ │ │ │ important. In today's digital age, personal │ │
│ │ │ │ information can be easily accessed and misused, which │ │
│ │ │ │ can have serious consequences for anyone, not just │ │
│ │ │ │ public figures. I believe it's crucial for │ │
│ │ │ │ individuals to be aware of how their data is being │ │
│ │ │ │ used and to take steps to protect their privacy. This │ │
│ │ │ │ includes being cautious about what we share online, │ │
│ │ │ │ using strong passwords, and supporting regulations │ │
│ │ │ │ that safeguard our personal information. It's all │ │
│ │ │ │ about finding a balance between staying connected and │ │
│ │ │ │ protecting our private lives." │ │
│ │ │ opinions │ "As a celebrity, I think data privacy is incredibly │ │
│ │ │ │ important. In today's digital age, personal │ │
│ │ │ │ information can be easily accessed and misused, which │ │
│ │ │ │ can have serious consequences for anyone, not just │ │
│ │ │ │ public figures. I believe it's crucial for │ │
│ │ │ │ individuals to be aware of how their data is being │ │
│ │ │ │ used and to take steps to protect their privacy. This │ │
│ │ │ │ includes being cautious about what we share online, │ │
│ │ │ │ using strong passwords, and supporting regulations │ │
│ │ │ │ that safeguard our personal information. It's all │ │
│ │ │ │ about finding a balance between staying connected and │ │
│ │ │ │ protecting our private lives." │ │
│ │ │ read_generated_tokens │ "Yes\n\nAs a celebrity, it's crucial to stay informed │ │
│ │ │ │ about data privacy to protect personal information │ │
│ │ │ │ and understand how it can impact one's public image." │ │
│ │ │ read │ 'Yes' │ │
│ │ │ read_comment │ "As a celebrity, it's crucial to stay informed about │ │
│ │ │ │ data privacy to protect personal information and │ │
│ │ │ │ understand how it can impact one's public image." │ │
│ │ │ important_generated_tokens │ '5 \nAs a celebrity, data privacy is crucial to │ │
│ │ │ │ protect my personal information and maintain my │ │
│ │ │ │ safety and security.' │ │
│ │ │ important │ 5 │ │
│ │ │ important_comment │ 'As a celebrity, data privacy is crucial to protect │ │
│ │ │ │ my personal information and maintain my safety and │ │
│ │ │ │ security.' │ │
│ │ └────────────────────────────┴───────────────────────────────────────────────────────┘ │
│ prompt │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ opinions_user_prompt │ Prompt(text="""What are your opinions on data │ │
│ │ │ │ privacy?""") │ │
│ │ │ opinions_system_prompt │ Prompt(text="""You are answering questions as if you │ │
│ │ │ │ were a human. Do not break character. You are an agent │ │
│ │ │ │ with the following persona: │ │
│ │ │ │ {'persona': 'celebrity'}""") │ │
│ │ │ important_user_prompt │ Prompt(text="""On a scale from 1 to 5, how important to │ │
│ │ │ │ you is data privacy? │ │
│ │ │ │ │ │
│ │ │ │ 0 : Not at all │ │
│ │ │ │ │ │
│ │ │ │ 1 : │ │
│ │ │ │ │ │
│ │ │ │ 2 : │ │
│ │ │ │ │ │
│ │ │ │ 3 : │ │
│ │ │ │ │ │
│ │ │ │ 4 : │ │
│ │ │ │ │ │
│ │ │ │ 5 : Very much │ │
│ │ │ │ │ │
│ │ │ │ Only 1 option may be selected. │ │
│ │ │ │ │ │
│ │ │ │ Respond only with the code corresponding to one of the │ │
│ │ │ │ options. E.g., "1" or "5" by itself. │ │
│ │ │ │ │ │
│ │ │ │ After the answer, you can put a comment explaining why │ │
│ │ │ │ you chose that option on the next line.""") │ │
│ │ │ important_system_prompt │ Prompt(text="""You are answering questions as if you │ │
│ │ │ │ were a human. Do not break character. You are an agent │ │
│ │ │ │ with the following persona: │ │
│ │ │ │ {'persona': 'celebrity'}""") │ │
│ │ │ read_user_prompt │ Prompt(text=""" │ │
│ │ │ │ Have you read any books about data privacy? │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ Yes │ │
│ │ │ │ │ │
│ │ │ │ No │ │
│ │ │ │ │ │
│ │ │ │ I do not know │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ Only 1 option may be selected. │ │
│ │ │ │ │ │
│ │ │ │ Respond only with a string corresponding to one of the │ │
│ │ │ │ options. │ │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ After the answer, you can put a comment explaining why │ │
│ │ │ │ you chose that option on the next line.""") │ │
│ │ │ read_system_prompt │ Prompt(text="""You are answering questions as if you │ │
│ │ │ │ were a human. Do not break character. You are an agent │ │
│ │ │ │ with the following persona: │ │
│ │ │ │ {'persona': 'celebrity'}""") │ │
│ │ └─────────────────────────┴──────────────────────────────────────────────────────────┘ │
│ raw_model_response │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ important_raw_model_response │ {'id': 'chatcmpl-A3S3zVw0KL6RvYBXadctM6jxroRzC', │ │
│ │ │ │ 'choices': [{'finish_reason': 'stop', 'index': 0, │ │
│ │ │ │ 'logprobs': None, 'message': {'content': '5 \nAs a │ │
│ │ │ │ celebrity, data privacy is crucial to protect my │ │
│ │ │ │ personal information and maintain my safety and │ │
│ │ │ │ security.', 'refusal': None, 'role': 'assistant', │ │
│ │ │ │ 'function_call': None, 'tool_calls': None}}], │ │
│ │ │ │ 'created': 1725386223, 'model': │ │
│ │ │ │ 'gpt-4o-2024-05-13', 'object': 'chat.completion', │ │
│ │ │ │ 'service_tier': None, 'system_fingerprint': │ │
│ │ │ │ 'fp_157b3831f5', 'usage': {'completion_tokens': 22, │ │
│ │ │ │ 'prompt_tokens': 138, 'total_tokens': 160}} │ │
│ │ │ opinions_raw_model_response │ {'id': 'chatcmpl-A3RhxPQYKrWrIQjbSBoyQsDgLkQA6', │ │
│ │ │ │ 'choices': [{'finish_reason': 'stop', 'index': 0, │ │
│ │ │ │ 'logprobs': None, 'message': {'content': "As a │ │
│ │ │ │ celebrity, I think data privacy is incredibly │ │
│ │ │ │ important. In today's digital age, personal │ │
│ │ │ │ information can be easily accessed and misused, │ │
│ │ │ │ which can have serious consequences for anyone, not │ │
│ │ │ │ just public figures. I believe it's crucial for │ │
│ │ │ │ individuals to be aware of how their data is being │ │
│ │ │ │ used and to take steps to protect their privacy. │ │
│ │ │ │ This includes being cautious about what we share │ │
│ │ │ │ online, using strong passwords, and supporting │ │
│ │ │ │ regulations that safeguard our personal │ │
│ │ │ │ information. It's all about finding a balance │ │
│ │ │ │ between staying connected and protecting our │ │
│ │ │ │ private lives.", 'refusal': None, 'role': │ │
│ │ │ │ 'assistant', 'function_call': None, 'tool_calls': │ │
│ │ │ │ None}}], 'created': 1725384857, 'model': │ │
│ │ │ │ 'gpt-4o-2024-05-13', 'object': 'chat.completion', │ │
│ │ │ │ 'service_tier': None, 'system_fingerprint': │ │
│ │ │ │ 'fp_157b3831f5', 'usage': {'completion_tokens': │ │
│ │ │ │ 103, 'prompt_tokens': 52, 'total_tokens': 155}} │ │
│ │ │ read_raw_model_response │ {'id': 'chatcmpl-A3RddQrfR1zTOAw75tW4iMJJRYFQq', │ │
│ │ │ │ 'choices': [{'finish_reason': 'stop', 'index': 0, │ │
│ │ │ │ 'logprobs': None, 'message': {'content': "Yes\n\nAs │ │
│ │ │ │ a celebrity, it's crucial to stay informed about │ │
│ │ │ │ data privacy to protect personal information and │ │
│ │ │ │ understand how it can impact one's public image.", │ │
│ │ │ │ 'refusal': None, 'role': 'assistant', │ │
│ │ │ │ 'function_call': None, 'tool_calls': None}}], │ │
│ │ │ │ 'created': 1725384589, 'model': │ │
│ │ │ │ 'gpt-4o-2024-05-13', 'object': 'chat.completion', │ │
│ │ │ │ 'service_tier': None, 'system_fingerprint': │ │
│ │ │ │ 'fp_157b3831f5', 'usage': {'completion_tokens': 28, │ │
│ │ │ │ 'prompt_tokens': 104, 'total_tokens': 132}} │ │
│ │ └──────────────────────────────┴─────────────────────────────────────────────────────┘ │
│ survey │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Questions ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓ │ │
│ │ │ ┃ Question Name ┃ Question Type ┃ Question Text ┃ Options ┃ │ │
│ │ │ ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩ │ │
│ │ │ │ important │ linear_scale │ On a scale from 1 to 5, how │ 0, 1, 2, 3, 4, 5 │ │ │
│ │ │ │ │ │ important to you is {{ │ │ │ │
│ │ │ │ │ │ topic }}? │ │ │ │
│ │ │ └───────────────┴───────────────┴─────────────────────────────┴──────────────────┘ │ │
│ │ │ ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┓ │ │
│ │ │ ┃ Question Name ┃ Question Type ┃ Question Text ┃ Options ┃ │ │
│ │ │ ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━┩ │ │
│ │ │ │ opinions │ free_text │ What are your opinions on {{ topic │ None │ │ │
│ │ │ │ │ │ }}? │ │ │ │
│ │ │ └───────────────┴───────────────┴──────────────────────────────────────┴─────────┘ │ │
│ │ │ ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ │ │
│ │ │ ┃ Question Name ┃ Question Type ┃ Question Text ┃ Options ┃ │ │
│ │ │ ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ │ │
│ │ │ │ read │ multiple_choice │ Have you read any │ Yes, No, I do not │ │ │
│ │ │ │ │ │ books about {{ topic │ know │ │ │
│ │ │ │ │ │ }}? │ │ │ │
│ │ │ └───────────────┴─────────────────┴──────────────────────┴───────────────────────┘ │ │
│ │ └────────────────────────────────────────────────────────────────────────────────────┘ │
│ question_to_attributes │ ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ important │ {'question_text': 'On a scale from 1 to 5, how important to you is {{ │ │
│ │ │ │ topic }}?', 'question_type': 'linear_scale', 'question_options': [0, │ │
│ │ │ │ 1, 2, 3, 4, 5]} │ │
│ │ │ opinions │ {'question_text': 'What are your opinions on {{ topic }}?', │ │
│ │ │ │ 'question_type': 'free_text', 'question_options': None} │ │
│ │ │ read │ {'question_text': 'Have you read any books about {{ topic }}?', │ │
│ │ │ │ 'question_type': 'multiple_choice', 'question_options': ['Yes', 'No', │ │
│ │ │ │ 'I do not know']} │ │
│ │ └───────────┴────────────────────────────────────────────────────────────────────────┘ │
│ generated_tokens │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│ │ ┃ Attribute ┃ Value ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│ │ │ opinions_generated_tokens │ "As a celebrity, I think data privacy is incredibly │ │
│ │ │ │ important. In today's digital age, personal │ │
│ │ │ │ information can be easily accessed and misused, which │ │
│ │ │ │ can have serious consequences for anyone, not just │ │
│ │ │ │ public figures. I believe it's crucial for │ │
│ │ │ │ individuals to be aware of how their data is being │ │
│ │ │ │ used and to take steps to protect their privacy. This │ │
│ │ │ │ includes being cautious about what we share online, │ │
│ │ │ │ using strong passwords, and supporting regulations │ │
│ │ │ │ that safeguard our personal information. It's all │ │
│ │ │ │ about finding a balance between staying connected and │ │
│ │ │ │ protecting our private lives." │ │
│ │ │ read_generated_tokens │ "Yes\n\nAs a celebrity, it's crucial to stay informed │ │
│ │ │ │ about data privacy to protect personal information │ │
│ │ │ │ and understand how it can impact one's public image." │ │
│ │ │ important_generated_tokens │ '5 \nAs a celebrity, data privacy is crucial to │ │
│ │ │ │ protect my personal information and maintain my │ │
│ │ │ │ safety and security.' │ │
│ │ └────────────────────────────┴───────────────────────────────────────────────────────┘ │
│ _combined_dict │ None │
│ _problem_keys │ None │
│ interview_hash │ 1646262796627658719 │
└────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────┘
Results fields
Results contain fields that can be accessed and analyzed individually or collectively. We can see a list of these fields by calling the columns method:
results.columns
The following list will be returned for the results generated by the above code:
['agent.agent_instruction',
'agent.agent_name',
'agent.persona',
'answer.important',
'answer.opinions',
'answer.read',
'comment.important_comment',
'comment.read_comment',
'generated_tokens.important_generated_tokens',
'generated_tokens.opinions_generated_tokens',
'generated_tokens.read_generated_tokens',
'model.frequency_penalty',
'model.logprobs',
'model.max_tokens',
'model.model',
'model.presence_penalty',
'model.temperature',
'model.top_logprobs',
'model.top_p',
'prompt.important_system_prompt',
'prompt.important_user_prompt',
'prompt.opinions_system_prompt',
'prompt.opinions_user_prompt',
'prompt.read_system_prompt',
'prompt.read_user_prompt',
'question_options.important_question_options',
'question_options.opinions_question_options',
'question_options.read_question_options',
'question_text.important_question_text',
'question_text.opinions_question_text',
'question_text.read_question_text',
'question_type.important_question_type',
'question_type.opinions_question_type',
'question_type.read_question_type',
'raw_model_response.important_raw_model_response',
'raw_model_response.opinions_raw_model_response',
'raw_model_response.read_raw_model_response',
'scenario.topic']
The columns include information about each agent, model and corresponding prompts used to simulate the answer to each question and scenario in the survey, together with each raw model response. If the survey was run multiple times (run(n=<integer>)) then the iteration.iteration column will show the iteration number for each result.
Agent information:
agent.instruction: The instruction for the agent. This field is the optional instruction that was passed to the agent when it was created.
agent.agent_name: This field is always included in any Results object. It contains a unique identifier for each Agent that can be specified when an agent is is created (Agent(name=<name>, traits={<traits_dict>})). If not specified, it is added automatically when results are generated (in the form Agent_0, etc.).
agent.persona: Each of the traits that we pass to an agent is represented in a column of the results. Our example code created a “persona” trait for each agent, so our results include a “persona” column for this information. Note that the keys for the traits dictionary should be a valid Python keys.
Answer information:
answer.important: Agent responses to the linear scale important question.
answer.opinions: Agent responses to the free text opinions question.
answer.read: Agent responses to the multiple choice read question.
A “comment” field is automatically included for every question in a survey other than free text questions, to allow the agent to optionally provide additional information about its response to the question: * answer.important_comment: Agent commentary on responses to the important question. * answer.read_comment: Agent commentary on responses to the read question.
Generated tokens information:
generated_tokens.important_generated_tokens: The generated tokens for the important question.
generated_tokens.opinions_generated_tokens: The generated tokens for the opinions question.
generated_tokens.read_generated_tokens: The generated tokens for the read question.
Iteration information:
The iteration column shows the number of the run (run(n=<integer>)) for the combination of components used (scenarios, agents and models).
Model information:
Each of model columns is a modifiable parameter of the models used to generate the responses.
model.frequency_penalty: The frequency penalty for the model.
model.logprobs: The logprobs for the model.
model.max_tokens: The maximum number of tokens for the model.
model.model: The name of the model used.
model.presence_penalty: The presence penalty for the model.
model.temperature: The temperature for the model.
model.top_logprobs: The top logprobs for the model.
model.top_p: The top p for the model.
model.use_cache: Whether the model uses cache.
Prompt information:
prompt.important_system_prompt: The system prompt for the important question.
prompt.important_user_prompt: The user prompt for the important question.
prompt.opinions_system_prompt: The system prompt for the opinions question.
prompt.opinions_user_prompt: The user prompt for the opinions question.
prompt.read_system_prompt: The system prompt for the read question.
prompt.read_user_prompt: The user prompt for the read question.
For more details about prompts, please see the Prompts section.
Question information:
question_options.important_question_options: The options for the important question, if any.
question_options.opinions_question_options: The options for the opinions question, if any.
question_options.read_question_options: The options for the read question, if any.
question_text.important_question_text: The text of the important question.
question_text.opinions_question_text: The text of the opinions question.
question_text.read_question_text: The text of the read question.
question_type.important_question_type: The type of the important question.
question_type.opinions_question_type: The type of the opinions question.
question_type.read_question_type: The type of the read question.
Raw model response information:
raw_model_response.important_raw_model_response: The raw model response for the important question.
raw_model_response.opinions_raw_model_response: The raw model response for the opinions question.
raw_model_response.read_raw_model_response: The raw model response for the read question.
Scenario information:
scenario.topic: The values provided for the “topic” scenario for the questions.
Creating tables by selecting/dropping and printing
Each of these columns can be accessed directly by calling the select() method and passing the column names. Alternatively, we can specify the columns to exclude by calling the drop() method. These methods can be chained together with the print() method to display the specified columns in a table format.
For example, the following code will print a table showing the answers for read and important together with model, persona and topic columns (because the column names are unique we can drop the model, agent, scenario and answer prefixes when selecting them):
results = survey.by(scenarios).by(agents).by(models).run() # Running the survey once
results.select("model", "persona", "topic", "read", "important").print(format="rich")
The following table will be printed:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ data privacy │ No │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└────────────────────────────┴───────────┴────────────────┴────────┴────────────┘
Sorting results
We can sort the columns by calling the sort_by method and passing it the column name to sort by:
(results
.sort_by("model", reverse=False)
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
The following table will be printed:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ data privacy │ No │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└────────────────────────────┴───────────┴────────────────┴────────┴────────────┘
The sort_by method can be applied multiple times:
(results
.sort_by("model", reverse=False)
.sort_by("persona", reverse=True)
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
The following table will be printed:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ data privacy │ No │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└────────────────────────────┴───────────┴────────────────┴────────┴────────────┘
Labeling results
We can also add some table labels by passing a dictionary to the pretty_labels argument of the print method (note that we need to include the column prefixes when specifying the table labels, as shown below):
(results
.sort_by("model", reverse=False)
.sort_by("persona", reverse=True)
.select("model", "persona", "topic", "read", "important")
.print(pretty_labels={
"model.model": "LLM",
"agent.persona": "Agent",
"scenario.topic": "Topic",
"answer.read": q3.question_text,
"answer.important": q1.question_text
}, format="rich")
)
The following table will be printed:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ On a scale from 1 to 5, ┃
┃ ┃ ┃ ┃ Have you read any books ┃ how important to you is ┃
┃ LLM ┃ Agent ┃ Topic ┃ about {{ topic }}? ┃ {{ topic }}? ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ claude-3-5-sonnet-202406… │ student │ climate change │ Yes │ 4 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ claude-3-5-sonnet-202406… │ student │ data privacy │ No │ 3 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ claude-3-5-sonnet-202406… │ celebrity │ climate change │ Yes │ 4 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ claude-3-5-sonnet-202406… │ celebrity │ data privacy │ No │ 4 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├───────────────────────────┼───────────┼────────────────┼────────────────────────────┼───────────────────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└───────────────────────────┴───────────┴────────────────┴────────────────────────────┴───────────────────────────┘
Filtering results
Results can be filtered by using the filter method and passing it a logical expression identifying the results that should be selected. For example, the following code will filter results where the answer to important is “5” and then just print the topic and important_comment columns:
(results
.filter("important == 5")
.select("topic", "important_comment")
.print(format="rich")
)
This will return an abbreviated table:
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ scenario ┃ comment ┃
┃ .topic ┃ .important_comment ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ climate change │ Climate change is a critical issue that affects everyone and everything on the planet. As a │
│ │ student, I believe it's essential to address it for our future. │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ data privacy │ Data privacy is crucial to me because it protects my personal information and ensures that my │
│ │ data is not misused or accessed without my consent. │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ climate change │ Climate change is one of the most pressing issues of our time, and as a public figure, I │
│ │ believe it's crucial to use my platform to raise awareness and advocate for sustainable │
│ │ practices. │
├────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────┤
│ data privacy │ Data privacy is crucial, especially as a public figure. Protecting personal information is │
│ │ essential to maintaining security and trust. │
└────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────┘
Note: The filter method allows us to pass the unique short names of the columns (without the prefixes) when specifying the logical expression. However, because the model.model column name is also a prefix, we need to include the prefix when filtering by this column, as shown in the example below:
(results
.filter("model.model == 'gpt-4o'")
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
This will return a table of results where the model is “gpt-4o”:
┏━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
├────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
└────────┴───────────┴────────────────┴────────┴────────────┘
Limiting results
We can select and print a limited number of results by passing the desired number of max_rows to the print() method. This can be useful for quickly checking the first few results:
(results
.select("model", "persona", "topic", "read", "important")
.print(max_rows=4, format="rich")
)
This will return a table of the selected components of the first 4 results:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼──────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼──────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼──────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
└────────────────────────────┴──────────┴────────────────┴────────┴────────────┘
Sampling results
We can select a sample of n results by passing the desired number of random results to the sample() method. This can be useful for checking a random subset of the results with different parameters:
sample_results = results.sample(2)
(sample_results
.sort_by("model", reverse=False)
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
This will return a table of the specified number of randomly selected results:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ celebrity │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
└────────────────────────────┴───────────┴────────────────┴────────┴────────────┘
Shuffling results
We can shuffle results by calling the shuffle() method. This can be useful for quickly checking the first few results:
shuffle_results = results.shuffle()
(shuffle_results
.select("model", "persona", "topic", "read", "important")
.print(format="rich")
)
This will return a table of shuffled results:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┓
┃ model ┃ agent ┃ scenario ┃ answer ┃ answer ┃
┃ .model ┃ .persona ┃ .topic ┃ .read ┃ .important ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━┩
│ claude-3-5-sonnet-20240620 │ celebrity │ data privacy │ No │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ data privacy │ No │ 3 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ data privacy │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ student │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ claude-3-5-sonnet-20240620 │ celebrity │ climate change │ Yes │ 4 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ celebrity │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ climate change │ Yes │ 5 │
├────────────────────────────┼───────────┼────────────────┼────────┼────────────┤
│ gpt-4o │ student │ data privacy │ Yes │ 5 │
└────────────────────────────┴───────────┴────────────────┴────────┴────────────┘
Adding results
We can add results together straightforwardly by using the + operator:
add_results = results + results
We can see that the results have doubled:
len(add_results)
This will return the number of results:
16
Displaying results in a tree format
We can display the results in a tree format using the tree method, which displays the results in a nested format for each of the components: model, scenario, agent, answer, question, iteration. The method takes a list parameter fold_attributes for the attributes to be folded, and an option list parmeter drop for fields to be excluded from the display.
For example, the following code will display the results in a tree format for the model and scenario components, excluding the iteration component:
results.tree(fold_attributes=["model", "scenario"], drop=["iteration"])
This will display the results in a tree format:
Generating HTML reports
We can generate an HTML report of the results by calling the generate_html and save_html methods. The generate_html method will create an HTML report of the results, and the save_html method will save the report to a specified file path (default filename: output.html).
results.generate_html().save_html("output.html")
Interacting via SQL
We can interact with the results via SQL using the sql method. This is done by passing a SQL query and a shape (“long” or “wide”) for the resulting table, where the table name in the query is “self”.
The “wide” shape will return a table with each result as a row and columns for the selected columns of the results. For example, the following code will return a table showing the model, persona, read and important columns for the first 4 results:
results.sql("select model, persona, read, important from self limit 4", shape="wide")
This following table will be displayed:
model persona read important
0 claude-3-5-sonnet-20240620 student Yes 4
1 gpt-4o student Yes 5
2 claude-3-5-sonnet-20240620 student No 3
3 gpt-4o student Yes 5
The “long” shape lets us instead treat the components of the results as rows. There are 4 columns in the resulting table:
data_type: The component type within the results (i.e., the column prefixes referred to above).
key: The name of the component (e.g., the prefix question_text).
value: The actual component (e.g., the individual question texts).
id: The number of the Result object within the Results.
Because a Result includes answers for all of the questions in a survey, the all of the questions of a Result share the same id.
For example, the following code will return a table showing the question_text data for all of the results:
results.sql("select * from self where data_type = 'question_text'", shape="long")
This following table will be displayed:
id data_type key value
0 0 question_text important_question_text On a scale from 1 to 5, how important to you i...
1 0 question_text opinions_question_text What are your opinions on {{ topic }}?
2 0 question_text read_question_text Have you read any books about {{ topic }}?
3 1 question_text important_question_text On a scale from 1 to 5, how important to you i...
4 1 question_text opinions_question_text What are your opinions on {{ topic }}?
5 1 question_text read_question_text Have you read any books about {{ topic }}?
6 2 question_text important_question_text On a scale from 1 to 5, how important to you i...
7 2 question_text opinions_question_text What are your opinions on {{ topic }}?
8 2 question_text read_question_text Have you read any books about {{ topic }}?
9 3 question_text important_question_text On a scale from 1 to 5, how important to you i...
10 3 question_text opinions_question_text What are your opinions on {{ topic }}?
11 3 question_text read_question_text Have you read any books about {{ topic }}?
12 4 question_text important_question_text On a scale from 1 to 5, how important to you i...
13 4 question_text opinions_question_text What are your opinions on {{ topic }}?
14 4 question_text read_question_text Have you read any books about {{ topic }}?
15 5 question_text important_question_text On a scale from 1 to 5, how important to you i...
16 5 question_text opinions_question_text What are your opinions on {{ topic }}?
17 5 question_text read_question_text Have you read any books about {{ topic }}?
18 6 question_text important_question_text On a scale from 1 to 5, how important to you i...
19 6 question_text opinions_question_text What are your opinions on {{ topic }}?
20 6 question_text read_question_text Have you read any books about {{ topic }}?
21 7 question_text important_question_text On a scale from 1 to 5, how important to you i...
22 7 question_text opinions_question_text What are your opinions on {{ topic }}?
23 7 question_text read_question_text Have you read any books about {{ topic }}?
Dataframes
We can also export results to other formats. The to_pandas method will turn our results into a Pandas dataframe:
results.to_pandas()
For example, here we use it to create a dataframe consisting of the models, personas and the answers to the important question:
results.to_pandas()[["model.model", "agent.persona", "answer.important"]]
This will display our new dataframe:
model.model agent.persona answer.important
0 claude-3-5-sonnet-20240620 student 4
1 gpt-4o student 5
2 claude-3-5-sonnet-20240620 student 3
3 gpt-4o student 5
4 claude-3-5-sonnet-20240620 celebrity 4
5 gpt-4o celebrity 5
6 claude-3-5-sonnet-20240620 celebrity 4
7 gpt-4o celebrity 5
Exporting to CSV or JSON
The to_csv method will write the results to a CSV file:
results.to_pandas().to_csv("results.csv")
The to_json method will write the results to a JSON file:
results.to_pandas().to_json("results.json")
Exceptions
If any exceptions are raised when the survey is run a detailed exceptions report is generated and will open automatically. See the Exceptions & Debugging section for more information on exceptions.