Results

A Results object represents the outcome of running a Survey. It contains a list of individual Result objects, where each Result corresponds to a response to the survey for a unique combination of Agent, Model, and Scenario object used in the survey.

For example, if a survey is administered to 2 agents and 2 language models without any scenarios, the Results will contain 4 Result objects (one for each combination of agent and model). If the survey includes parameterized questions with 2 scenarios, the Results will expand to include 8 Result objects (accounting for all combinations of agents, models, and scenarios).

Generating results

A Results object is not typically instantiated directly, but is returned by calling the run() method of a Survey after any agents, language models and scenarios are added to it.

In order to demonstrate how to access and interact with results, we use the following code to generate results for a simple survey. Note that specifying agent traits, scenarios (question parameter values) and language models is optional, and we include those steps here for illustrative purposes. (See the Agents, Scenarios and models sections for more details on these components.)

Note: You must store API keys for language models in order to generate results. Please see the API Keys section for instructions on activating Remote Inference from your coop account or storing your own API keys from service providers.

To construct a survey we start by creating questions:

from edsl import QuestionLinearScale, QuestionMultipleChoice

q1 = QuestionLinearScale(
   question_name = "important",
   question_text = "On a scale from 1 to 5, how important to you is {{ topic }}?",
   question_options = [0, 1, 2, 3, 4, 5],
   option_labels = {0:"Not at all important", 5:"Very important"}
)

q2 = QuestionMultipleChoice(
   question_name = "read",
   question_text = "Have you read any books about {{ topic }}?",
   question_options = ["Yes", "No", "I do not know"]
)

We combine them in a survey to administer them together:

from edsl import Survey

survey = Survey([q1, q2])

We have parameterized our questions, so we can use them with different scenarios:

from edsl import ScenarioList

scenarios = ScenarioList.from_list("topic", ["climate change", "house prices"])

We can optionally create agents with personas or other relevant traits to answer the survey:

from edsl import AgentList, Agent

agents = AgentList(
   Agent(traits = {"persona": p}) for p in ["student", "celebrity"]
)

We can specify the language models that we want to use to generate responses:

from edsl import ModelList, Model

models = ModelList(
   Model(m) for m in ["gemini-1.5-flash", "gpt-4o"]
)

Finally, we generate results by adding the scenarios, agents and models to the survey and calling the run() method:

results = survey.by(scenarios).by(agents).by(models).run()

For more details on each of the above steps, please see the Agents, Scenarios and models sections of the docs.

Result objects

We can check the number of Result objects created by inspecting the length of the Results:

len(results)

This will count 2 (scenarios) x 2 (agents) x 2 (models) = 8 Result objects:

8

Generating multiple results

If we want to generate multiple results for a survey–i.e., more than 1 result for each combination of Agent, Model and Scenario objects used–we can pass the desired number of iterations when calling the run() method. For example, the following code will generate 3 results for our survey (n=3):

results = survey.by(scenarios).by(agents).by(models).run(n=3)

We can verify that the number of Result objects created is now 24 = 3 iterations x 2 scenarios x 2 agents x 2 models:

len(results)

24

We can readily inspect a result:

results[0]

Output:

key	value
agent:traits	{‘persona’: ‘student’}
scenario:topic	climate change
model:model	gemini-1.5-flash
model:parameters	{‘temperature’: 0.5, ‘topP’: 1, ‘topK’: 1, ‘maxOutputTokens’: 2048, ‘stopSequences’: []}
iteration	0
answer:important	5
answer:read	Yes
prompt:important_user_prompt	{‘text’: ‘On a scale from 1 to 5, how important to you is climate change?nn0 : Not at all importantnn1 : nn2 : nn3 : nn4 : nn5 : Very importantnnOnly 1 option may be selected.nnRespond only with the code corresponding to one of the options. E.g., “1” or “5” by itself.nnAfter the answer, you can put a comment explaining why you chose that option on the next line.’, ‘class_name’: ‘Prompt’}
prompt:important_system_prompt	{‘text’: “You are answering questions as if you were a human. Do not break character. Your traits: {‘persona’: ‘student’}”, ‘class_name’: ‘Prompt’}
prompt:read_user_prompt	{‘text’: ‘nHave you read any books about climate change?nn nYesn nNon nI do not known nnOnly 1 option may be selected.nnRespond only with a string corresponding to one of the options.nnnAfter the answer, you can put a comment explaining why you chose that option on the next line.’, ‘class_name’: ‘Prompt’}
prompt:read_system_prompt	{‘text’: “You are answering questions as if you were a human. Do not break character. Your traits: {‘persona’: ‘student’}”, ‘class_name’: ‘Prompt’}
raw_model_response:important_raw_model_response	{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “5nnIt’s, like, a huge deal. The future of the planet is at stake, and that affects everything – from the environment to the economy to social justice. It’s something I worry about a lot.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.12477729758437799, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 129, ‘candidates_token_count’: 49, ‘total_token_count’: 178, ‘cached_content_token_count’: 0}}
raw_model_response:important_cost	1.78e-12
raw_model_response:important_one_usd_buys	561797752808.9888
raw_model_response:read_raw_model_response	{‘candidates’: [{‘content’: {‘parts’: [{‘text’: “YesnnI’ve read a few articles and some chapters from textbooks for my environmental science class, which covered climate change extensively. It’s not exactly the same as reading a whole book dedicated to the subject, but I’ve definitely learned about it.n”}], ‘role’: ‘model’}, ‘finish_reason’: 1, ‘safety_ratings’: [{‘category’: 8, ‘probability’: 1, ‘blocked’: False}, {‘category’: 10, ‘probability’: 1, ‘blocked’: False}, {‘category’: 7, ‘probability’: 1, ‘blocked’: False}, {‘category’: 9, ‘probability’: 1, ‘blocked’: False}], ‘avg_logprobs’: -0.14903209827564382, ‘token_count’: 0, ‘grounding_attributions’: []}], ‘usage_metadata’: {‘prompt_token_count’: 96, ‘candidates_token_count’: 54, ‘total_token_count’: 150, ‘cached_content_token_count’: 0}}
raw_model_response:read_cost	1.5e-12
raw_model_response:read_one_usd_buys	666666666666.6666
question_to_attributes:important	{‘question_text’: ‘On a scale from 1 to 5, how important to you is {{ topic }}?’, ‘question_type’: ‘linear_scale’, ‘question_options’: [0, 1, 2, 3, 4, 5]}
question_to_attributes:read	{‘question_text’: ‘Have you read any books about {{ topic }}?’, ‘question_type’: ‘multiple_choice’, ‘question_options’: [‘Yes’, ‘No’, ‘I do not know’]}
generated_tokens:important_generated_tokens	5 It’s, like, a huge deal. The future of the planet is at stake, and that affects everything – from the environment to the economy to social justice. It’s something I worry about a lot.
generated_tokens:read_generated_tokens	Yes I’ve read a few articles and some chapters from textbooks for my environmental science class, which covered climate change extensively. It’s not exactly the same as reading a whole book dedicated to the subject, but I’ve definitely learned about it.
comments_dict:important_comment	It’s, like, a huge deal. The future of the planet is at stake, and that affects everything – from the environment to the economy to social justice. It’s something I worry about a lot.
comments_dict:read_comment	I’ve read a few articles and some chapters from textbooks for my environmental science class, which covered climate change extensively. It’s not exactly the same as reading a whole book dedicated to the subject, but I’ve definitely learned about it.

Results fields

Results contain fields that can be accessed and analyzed individually or collectively. We can see a list of these fields by calling the columns method:

results.columns

The following list will be returned for the results generated by the above code:

The columns include information about each agent, model and corresponding prompts used to simulate the answer to each question and scenario in the survey, together with each raw model response. If the survey was run multiple times (run(n=<integer>)) then the iteration.iteration column will show the iteration number for each result.

Agent information:

• agent.instruction: The instruction for the agent. This field is the optional instruction that was passed to the agent when it was created.

• agent.agent_name: This field is always included in any Results object. It contains a unique identifier for each Agent that can be specified when an agent is is created (Agent(name=<name>, traits={<traits_dict>})). If not specified, it is added automatically when results are generated (in the form Agent_0, etc.).

• agent.persona: Each of the traits that we pass to an agent is represented in a column of the results. Our example code created a “persona” trait for each agent, so our results include a “persona” column for this information. Note that the keys for the traits dictionary should be a valid Python keys.

Answer information:

• answer.important: Agent responses to the linear scale important question.

• answer.read: Agent responses to the multiple choice read question.

Comment information:

A “comment” field is automatically included for every question in a survey other than free text questions, to allow the model to provide additional information about its response. The default instruction for the agent to provide a comment is included in user_prompt for a question, and can be modified or omitted when creating the question. (See the Prompts section for details on modifying user and system prompts, and information about prompts in results below. Comments can also be automatically excluded by passing a parameter include_comment=False a question when creating it.)

• comment.important_comment: Agent commentary on responses to the important question.

• comment.read_comment: Agent commentary on responses to the read question.

Generated tokens information:

• generated_tokens.important_generated_tokens: The generated tokens for the important question.

• generated_tokens.read_generated_tokens: The generated tokens for the read question.

Iteration information:

The iteration column shows the number of the run (run(n=<integer>)) for the combination of components used (scenarios, agents and models).

Model information:

Each of model columns is a modifiable parameter of the models used to generate the responses.

• model.frequency_penalty: The frequency penalty for the model.

• model.logprobs: The logprobs for the model.

• model.max_tokens: The maximum number of tokens for the model.

• model.model: The name of the model used.

• model.presence_penalty: The presence penalty for the model.

• model.temperature: The temperature for the model.

• model.top_logprobs: The top logprobs for the model.

• model.top_p: The top p for the model.

• model.use_cache: Whether the model uses cache.

Prompt information:

• prompt.important_system_prompt: The system prompt for the important question.

• prompt.important_user_prompt: The user prompt for the important question.

• prompt.read_system_prompt: The system prompt for the read question.

• prompt.read_user_prompt: The user prompt for the read question.

For more details about prompts, please see the Prompts section.

Question information:

• question_options.important_question_options: The options for the important question, if any.

• question_options.read_question_options: The options for the read question, if any.

• question_text.important_question_text: The text of the important question.

• question_text.read_question_text: The text of the read question.

• question_type.important_question_type: The type of the important question.

• question_type.read_question_type: The type of the read question.

Raw model response information:

• raw_model_response.important_cost: The cost of the result for the important question, applying the token quanities & prices.

• raw_model_response.important_one_usd_buys: The number of identical results for the important question that 1USD would cover.

• raw_model_response.important_raw_model_response: The raw model response for the important question.

• raw_model_response.read_cost: The cost of the result for the read question, applying the token quanities & prices.

• raw_model_response.read_one_usd_buys: The number of identical results for the read question that 1USD would cover.

• raw_model_response.read_raw_model_response: The raw model response for the read question.

Note that the cost of a result for a question is specific to the components (scenario, agent, model used with it).

Scenario information:

• scenario.topic: The values provided for the “topic” scenario for the questions.

Creating tables by selecting/dropping and printing

Each of these columns can be accessed directly by calling the select() method and passing the column names. Alternatively, we can specify the columns to exclude by calling the drop() method. These methods can be chained together to display the specified columns in a table format.

For example, the following code will print a table showing the answers for read and important together with model, persona and topic columns (because the column names are unique we can drop the model, agent, scenario and answer prefixes when selecting them):

results = survey.by(scenarios).by(agents).by(models).run() # Running the survey once
results.select("model", "persona", "topic", "read", "important")

A table with the selected columns will be printed:

model.model	agent.persona	scenario.topic	answer.read	answer.important
gemini-1.5-flash	student	climate change	Yes	5
gpt-4o	student	climate change	Yes	5
gemini-1.5-flash	student	house prices	No	1
gpt-4o	student	house prices	No	3
gemini-1.5-flash	celebrity	climate change	Yes	5
gpt-4o	celebrity	climate change	Yes	5
gemini-1.5-flash	celebrity	house prices	Yes	3
gpt-4o	celebrity	house prices	No	3

Sorting results

We can sort the columns by calling the sort_by method and passing it the column names to sort by:

(
   results
   .sort_by("model", "persona", reverse=False)
   .select("model", "persona", "topic", "read", "important")
)

The following table will be printed:

model.model	agent.persona	scenario.topic	answer.read	answer.important
gemini-1.5-flash	celebrity	climate change	Yes	5
gemini-1.5-flash	celebrity	house prices	Yes	3
gemini-1.5-flash	student	climate change	Yes	5
gemini-1.5-flash	student	house prices	No	1
gpt-4o	celebrity	climate change	Yes	5
gpt-4o	celebrity	house prices	No	3
gpt-4o	student	climate change	Yes	5
gpt-4o	student	house prices	No	3

Labeling results

We can also add some table labels by passing a dictionary to the pretty_labels argument of the print method (note that we need to include the column prefixes when specifying the table labels, as shown below):

(
   results
   .sort_by("model", "persona", reverse=True)
   .select("model", "persona", "topic", "read", "important")
   .print(pretty_labels={
      "model.model": "LLM",
      "agent.persona": "Agent",
      "scenario.topic": "Topic",
      "answer.read": q2.question_text,
      "answer.important": q1.question_text
      }, format="rich")
)

The following table will be printed:

LLM	Agent	Topic	Have you read any books about {{ topic }}?	On a scale from 1 to 5, how important to you is {{ topic }}?
gpt-4o	student	climate change	Yes	5
gpt-4o	student	house prices	No	3
gpt-4o	celebrity	climate change	Yes	5
gpt-4o	celebrity	house prices	No	3
gemini-1.5-flash	student	climate change	Yes	5
gemini-1.5-flash	student	house prices	No	1
gemini-1.5-flash	celebrity	climate change	Yes	5
gemini-1.5-flash	celebrity	house prices	Yes	3

Filtering results

Results can be filtered by using the filter method and passing it a logical expression identifying the results that should be selected. For example, the following code will filter results where the answer to important is “5” and then just print the topic and important_comment columns:

(
   results
   .filter("important == 5")
   .select("topic", "important", "important_comment")
)

This will return an abbreviated table:

scenario.topic	answer.important	comment.important_comment
climate change	5	It’s, like, a huge deal. The future of the planet is at stake, and that affects everything – from the environment to the economy to social justice. It’s something I worry about a lot.
climate change	5	As a student, I’m really concerned about climate change because it affects our future and the planet we’ll inherit. It’s crucial to understand and address it to ensure a sustainable world for generations to come.
climate change	5	It’s a huge issue, you know? We only have one planet, and if we don’t take care of it, what kind of world are we leaving for future generations? It’s not just about polar bears; it’s about everything. It’s my responsibility, as someone with a platform, to speak out about it.
climate change	5	Climate change is a critical issue that affects everyone globally, and as a public figure, I believe it’s important to use my platform to raise awareness and advocate for sustainable practices.

Note: The filter method allows us to pass the unique short names of the columns (without the prefixes) when specifying the logical expression. However, because the model.model column name is also a prefix, we need to include the prefix when filtering by this column, as shown in the example below:

(
   results
   .filter("model.model == 'gpt-4o'")
   .select("model", "persona", "topic", "read", "important")
)

This will return a table of results where the model is “gpt-4o”:

model.model	agent.persona	scenario.topic	answer.read	answer.important
gpt-4o	student	climate change	Yes	5
gpt-4o	student	house prices	No	3
gpt-4o	celebrity	climate change	Yes	5
gpt-4o	celebrity	house prices	No	3

Limiting results

We can select and print a limited number of results by passing the desired number of max_rows to the print() method. This can be useful for quickly checking the first few results:

(
   results
   .select("model", "persona", "topic", "read", "important")
   .print(max_rows=4, format="rich")
)

This will return a table of the selected components of the first 4 results:

model.model	agent.persona	scenario.topic	answer.read	answer.important
gemini-1.5-flash	student	climate change	Yes	5
gpt-4o	student	climate change	Yes	5
gemini-1.5-flash	student	house prices	No	1
gpt-4o	student	house prices	No	3

Sampling results

We can select a sample of n results by passing the desired number of random results to the sample() method. This can be useful for checking a random subset of the results with different parameters:

sample_results = results.sample(2)

(
   sample_results
   .sort_by("model")
   .select("model", "persona", "topic", "read", "important")
)

This will return a table of the specified number of randomly selected results:

model.model	agent.persona	scenario.topic	answer.read	answer.important
gpt-4o	celebrity	house prices	No	3
gpt-4o	celebrity	climate change	Yes	5

Shuffling results

We can shuffle results by calling the shuffle() method. This can be useful for quickly checking the first few results:

shuffle_results = results.shuffle()

(
   shuffle_results
   .select("model", "persona", "topic", "read", "important")
)

This will return a table of shuffled results:

model.model	agent.persona	scenario.topic	answer.read	answer.important
gemini-1.5-flash	celebrity	climate change	Yes	5
gpt-4o	student	house prices	No	3
gemini-1.5-flash	celebrity	house prices	Yes	3
gemini-1.5-flash	student	house prices	No	1
gpt-4o	celebrity	house prices	No	3
gpt-4o	celebrity	climate change	Yes	5
gpt-4o	student	climate change	Yes	5
gemini-1.5-flash	student	climate change	Yes	5

Adding results

We can add results together straightforwardly by using the + operator:

add_results = results + results

We can see that the results have doubled:

len(add_results)

This will return the number of results:

16

Interacting via SQL

We can interact with the results via SQL using the sql method. This is done by passing a SQL query and a shape (“long” or “wide”) for the resulting table, where the table name in the query is “self”.

For example, the following code will return a table showing the model, persona, read and important columns for the first 4 results:

results.sql("select model, persona, read, important from self limit 4")

This following table will be displayed

model	persona	read	important
gemini-1.5-flash	student	Yes	5
gpt-4o	student	Yes	5
gemini-1.5-flash	student	No	1
gpt-4o	student	No	3

Dataframes

We can also export results to other formats. The to_pandas method will turn our results into a Pandas dataframe:

results.to_pandas()

For example, here we use it to create a dataframe consisting of the models, personas and the answers to the important question:

results.to_pandas()[["model.model", "agent.persona", "answer.important"]]

Exporting to CSV or JSON

The to_csv method will write the results to a CSV file:

results.to_pandas().to_csv("results.csv")

The to_json method will write the results to a JSON file:

results.to_pandas().to_json("results.json")

Exceptions

If any exceptions are raised when the survey is run a detailed exceptions report is generated and can be opened in your browser. See the Exceptions & Debugging section for more information on exceptions.

Result class

class edsl.results.Result.AgentNamer

Maintains a registry of agent names to ensure unique naming.

__init__()

get_name(agent: Agent) → str

Get or create a unique name for an agent.

class edsl.results.Result.Result(agent: Agent, scenario: Scenario, model: LanguageModel, iteration: int, answer: dict[QuestionName, AnswerValue], prompt: dict[QuestionName, str] = None, raw_model_response: dict | None = None, survey: 'Survey' | None = None, question_to_attributes: dict[QuestionName, Any] | None = None, generated_tokens: dict | None = None, comments_dict: dict | None = None, cache_used_dict: dict[QuestionName, bool] | None = None, indices: dict | None = None)

This class captures the result of one interview.

__init__(agent: Agent, scenario: Scenario, model: LanguageModel, iteration: int, answer: dict[QuestionName, AnswerValue], prompt: dict[QuestionName, str] = None, raw_model_response: dict | None = None, survey: 'Survey' | None = None, question_to_attributes: dict[QuestionName, Any] | None = None, generated_tokens: dict | None = None, comments_dict: dict | None = None, cache_used_dict: dict[QuestionName, bool] | None = None, indices: dict | None = None)

Initialize a Result object.

Parameters:

• agent – The Agent object.

• scenario – The Scenario object.

• model – The LanguageModel object.

• iteration – The iteration number.

• answer – The answer string.

• prompt – A dictionary of prompts.

• raw_model_response – The raw model response.

• survey – The Survey object.

• question_to_attributes – A dictionary of question attributes.

• generated_tokens – A dictionary of generated tokens.

• comments_dict – A dictionary of comments.

• cache_used_dict – A dictionary of cache usage.

• indices – A dictionary of indices.

property agent: Agent

Return the Agent object.

property answer: dict[str, Any]

Return the answers.

check_expression(expression: str) → None

clear() → None.  Remove all items from D.

code()

Return a string of code that can be used to recreate the Result object.

property combined_dict: dict[str, Any]

Return a dictionary that includes all sub_dicts, but also puts the key-value pairs in each sub_dict as a key_value pair in the combined dictionary.

>>> r = Result.example()
>>> r.combined_dict['how_feeling']
'OK'

copy() → Result

Return a copy of the Result object.

>>> r = Result.example()
>>> r2 = r.copy()
>>> r == r2
True
>>> id(r) == id(r2)
False

create_download_link()

classmethod delete(uuid: str | UUID | None = None, url: str | None = None)

Delete the object from coop.

display_dict()

duplicate(add_edsl_version=False)

Return a duplicate of the object.

classmethod example()

Return an example Result object.

>>> Result.example()
Result(...)

classmethod from_dict(json_dict: dict) → Result

Return a Result object from a dictionary representation.

classmethod from_interview(interview, extracted_answers, model_response_objects) → Result

Return a Result object from an interview dictionary.

classmethod from_yaml(yaml_str: str | None = None, filename: str | None = None)

classmethod fromkeys(iterable, value=None)

get(k[, d]) → D[k] if k in D, else d.  d defaults to None.

get_value(data_type: str, key: str) → Any

Return the value for a given data type and key.

>>> r = Result.example()
>>> r.get_value("answer", "how_feeling")
'OK'

• data types can be “agent”, “scenario”, “model”, or “answer”

• keys are relevant attributes of the Objects the data types represent

help()

Extract all public instance methods and their docstrings from a class instance.

Args:

obj: The instance to inspect

Returns:

dict: A dictionary where keys are method names and values are their docstrings

items() → a set-like object providing a view on D's items

json()

property key_to_data_type: dict[str, str]

Return a dictionary where keys are object attributes and values are the data type (object) that the attribute is associated with.

>>> r = Result.example()
>>> r.key_to_data_type["how_feeling"]
'answer'

keys()

Return the keys of the object.

classmethod load(filename)

Load the object from a file.

>>> obj = cls.load("obj.json.gz")

property model: LanguageModel

Return the LanguageModel object.

static open_compressed_file(filename)

static open_regular_file(filename)

classmethod patch(uuid: str | UUID | None = None, url: str | None = None, description: str | None = None, alias: str | None = None, value: Any | None = None, visibility: str | None = None)

Patch an uploaded objects attributes. - description changes the description of the object on Coop - value changes the value of the object on Coop. has to be an EDSL object - visibility changes the visibility of the object on Coop

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

print(format='rich')

property problem_keys: list[str]

Return a list of keys that are problematic.

classmethod pull(uuid: str | UUID | None = None, url: str | None = None, expected_parrot_url: str | None = None)

Pull the object from coop.

push(description: str | None = None, alias: str | None = None, visibility: str | None = 'unlisted', expected_parrot_url: str | None = None)

Post the object to coop.

save(filename, compress=True)

Save the object to a file as zippped JSON.

>>> obj.save("obj.json.gz")

property scenario: Scenario

Return the Scenario object.

score(scoring_function: Callable) → int | float

Score the result using a passed-in scoring function.

>>> def f(status): return 1 if status == 'Joyful' else 0
>>> Result.example().score(f)
1

classmethod search(query)

Search for objects on coop.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D

show_methods(show_docstrings=True)

Show the methods of the object.

store(d: dict, key_name: str | None = None)

property sub_dicts: dict[str, dict]

Return a dictionary where keys are strings for each of the main class attributes/objects.

to_dataset()

to_dict(add_edsl_version: bool = True, include_cache_info: bool = False) → dict[str, Any]

Return a dictionary representation of the Result object.

>>> r = Result.example()
>>> r.to_dict()['scenario']
{'period': 'morning', 'scenario_index': 0, 'edsl_version': '...', 'edsl_class_name': 'Scenario'}

to_json()

to_yaml(add_edsl_version=False, filename: str = None) → str | None

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

values()

Return the values of the object.

view()

Displays an interactive / perspective view of the object

edsl.results.Result.agent_namer(agent: Agent) → str

Get or create a unique name for an agent.

Results class

The Results object is the result of running a survey. It is not typically instantiated directly, but is returned by the run method of a Job object.

class edsl.results.Results.Results(survey: Survey | None = None, data: list[Result] | None = None, created_columns: list[str] | None = None, cache: Cache | None = None, job_uuid: str | None = None, total_results: int | None = None, task_history: TaskHistory | None = None)

This class is a UserList of Result objects.

It is instantiated with a Survey and a list of Result objects. It can be manipulated in various ways with select, filter, mutate, etc. It also has a list of created_columns, which are columns that have been created with mutate and are not part of the original data.

__init__(survey: Survey | None = None, data: list[Result] | None = None, created_columns: list[str] | None = None, cache: Cache | None = None, job_uuid: str | None = None, total_results: int | None = None, task_history: TaskHistory | None = None)

Instantiate a Results object with a survey and a list of Result objects.

Parameters:

• survey – A Survey object.

• data – A list of Result objects.

• created_columns – A list of strings that are created columns.

• job_uuid – A string representing the job UUID.

• total_results – An integer representing the total number of results.

add_column(column_name: str, values: list) → Results

Adds columns to Results

>>> r = Results.example()
>>> r.add_column('a', [1,2,3, 4]).select('a')
Dataset([{'answer.a': [1, 2, 3, 4]}])

add_columns_from_dict(columns: List[dict]) → Results

Adds columns to Results from a list of dictionaries.

>>> r = Results.example()
>>> r.add_columns_from_dict([{'a': 1, 'b': 2}, {'a': 3, 'b': 4}, {'a':3, 'b':2}, {'a':3, 'b':2}]).select('a', 'b')
Dataset([{'answer.a': [1, 3, 3, 3]}, {'answer.b': [2, 4, 2, 2]}])

property agent_keys: list[str]

Return a set of all of the keys that are in the Agent data.

Example:

>>> r = Results.example()
>>> r.agent_keys
['agent_index', 'agent_instruction', 'agent_name', 'status']

property agents: AgentList

Return a list of all of the agents in the Results.

Example:

>>> r = Results.example()
>>> r.agents
AgentList([Agent(traits = {'status': 'Joyful'}), Agent(traits = {'status': 'Joyful'}), Agent(traits = {'status': 'Sad'}), Agent(traits = {'status': 'Sad'})])

property all_keys: list[str]

Return a set of all of the keys that are in the Results.

Example:

>>> r = Results.example()
>>> r.all_keys
['agent_index', ...]

property answer_keys: dict[str, str]

Return a mapping of answer keys to question text.

Example:

>>> r = Results.example()
>>> r.answer_keys
{'how_feeling': 'How are you this {{ period }}?', 'how_feeling_yesterday': 'How were you feeling yesterday {{ period }}?'}

answer_truncate(column: str, top_n: int = 5, new_var_name: str = None) → Results

Create a new variable that truncates the answers to the top_n.

Parameters:

• column – The column to truncate.

• top_n – The number of top answers to keep.

• new_var_name – The name of the new variable. If None, it is the original name + ‘_truncated’.

Example: >>> r = Results.example() >>> r.answer_truncate(‘how_feeling’, top_n = 2).select(‘how_feeling’, ‘how_feeling_truncated’) Dataset([{‘answer.how_feeling’: [‘OK’, ‘Great’, ‘Terrible’, ‘OK’]}, {‘answer.how_feeling_truncated’: [‘Other’, ‘Other’, ‘Other’, ‘Other’]}])

code()

This method should be implemented by subclasses.

property columns: list[str]

Return a list of all of the columns that are in the Results.

Example:

>>> r = Results.example()
>>> r.columns
['agent.agent_index', ...]

compare(other_results: Results) → dict

Compare two Results objects and return the differences.

compute_job_cost(include_cached_responses_in_cost: bool = False) → float

Computes the cost of a completed job in USD.

create_download_link()

classmethod delete(uuid: str | UUID | None = None, url: str | None = None)

Delete the object from coop.

display_dict()

duplicate(add_edsl_version=False)

Return a duplicate of the object.

classmethod example(randomize: bool = False) → Results

Return an example Results object.

Example usage:

>>> r = Results.example()

Parameters:

debug – if False, uses actual API calls

filter(expression: str) → Results

Filter based on the given expression and returns the filtered Results.

Parameters:

expression – A string expression that evaluates to a boolean. The expression is applied to each element in Results to determine whether it should be included in the filtered results.

The expression parameter is a string that must resolve to a boolean value when evaluated against each element in Results. This expression is used to determine which elements to include in the returned Results.

Example usage: Create an example Results instance and apply filters to it:

>>> r = Results.example()
>>> r.filter("how_feeling == 'Great'").select('how_feeling')
Dataset([{'answer.how_feeling': ['Great']}])

Example usage: Using an OR operator in the filter expression.

>>> r = Results.example().filter("how_feeling = 'Great'").select('how_feeling')
Traceback (most recent call last):
...
edsl.exceptions.results.ResultsFilterError: You must use '==' instead of '=' in the filter expression.
...

>>> r.filter("how_feeling == 'Great' or how_feeling == 'Terrible'").select('how_feeling')
Dataset([{'answer.how_feeling': ['Great', 'Terrible']}])

first() → Result

Return the first observation in the results.

Example:

>>> r = Results.example()
>>> r.first()
Result(agent...

classmethod from_dict(data: dict[str, Any]) → Results

Convert a dictionary to a Results object.

Parameters:

data – A dictionary representation of a Results object.

Example:

>>> r = Results.example()
>>> d = r.to_dict()
>>> r2 = Results.from_dict(d)
>>> r == r2
True

classmethod from_yaml(yaml_str: str | None = None, filename: str | None = None)

ggplot2(ggplot_code: str, filename: str = None, shape='wide', sql: str = None, remove_prefix: bool = True, debug: bool = False, height=4, width=6, format='svg', factor_orders: dict | None = None)

Create a ggplot2 plot from a DataFrame.

Parameters:

• ggplot_code – The ggplot2 code to execute.

• filename – The filename to save the plot to.

• shape – The shape of the data in the DataFrame (wide or long).

• sql – The SQL query to execute beforehand to manipulate the data.

• remove_prefix – Whether to remove the prefix from the column names.

• debug – Whether to print the R code instead of executing it.

• height – The height of the plot in inches.

• width – The width of the plot in inches.

• format – The format to save the plot in (png or svg).

• factor_orders – A dictionary of factor columns and their order.

property has_unfixed_exceptions: bool

property hashes: set

help()

Extract all public instance methods and their docstrings from a class instance.

Args:

obj: The instance to inspect

Returns:

dict: A dictionary where keys are method names and values are their docstrings

html(filename: str | None = None, cta: str = 'Open in browser', return_link: bool = False)

interactive_tree(fold_attributes: List[str] | None = None, drop: List[str] | None = None, open_file=True) → dict

Return the results as a tree.

json()

keys()

Return the keys of the object.

leaves()

classmethod load(filename)

Load the object from a file.

>>> obj = cls.load("obj.json.gz")

long()

property model_keys: list[str]

Return a set of all of the keys that are in the LanguageModel data.

>>> r = Results.example()
>>> r.model_keys
['frequency_penalty', 'logprobs', 'max_tokens', 'model', 'model_index', 'presence_penalty', 'temperature', 'top_logprobs', 'top_p']

property models: ModelList

Return a list of all of the models in the Results.

Example:

>>> r = Results.example()
>>> r.models[0]
Model(model_name = ...)

mutate(new_var_string: str, functions_dict: dict | None = None) → Results

Creates a value in the Results object as if has been asked as part of the survey.

Parameters:

• new_var_string – A string that is a valid Python expression.

• functions_dict – A dictionary of functions that can be used in the expression. The keys are the function names and the values are the functions themselves.

It splits the new_var_string at the “=” and uses simple_eval

Example:

>>> r = Results.example()
>>> r.mutate('how_feeling_x = how_feeling + "x"').select('how_feeling_x')
Dataset([{'answer.how_feeling_x': ...

num_observations()

Return the number of observations in the dataset.

>>> from edsl.results.Results import Results
>>> Results.example().num_observations()
4

static open_compressed_file(filename)

static open_regular_file(filename)

order_by(*columns: str, reverse: bool = False) → Results

Sort the results by one or more columns.

Parameters:

• columns – One or more column names as strings.

• reverse – A boolean that determines whether to sort in reverse order.

Each column name can be a single key, e.g. “how_feeling”, or a dot-separated string, e.g. “answer.how_feeling”.

Example:

>>> r = Results.example()
>>> r.sort_by('how_feeling', reverse=False).select('how_feeling')
Dataset([{'answer.how_feeling': ['Great', 'OK', 'OK', 'Terrible']}])

>>> r.sort_by('how_feeling', reverse=True).select('how_feeling')
Dataset([{'answer.how_feeling': ['Terrible', 'OK', 'OK', 'Great']}])

classmethod patch(uuid: str | UUID | None = None, url: str | None = None, description: str | None = None, alias: str | None = None, value: Any | None = None, visibility: str | None = None)

Patch an uploaded objects attributes. - description changes the description of the object on Coop - value changes the value of the object on Coop. has to be an EDSL object - visibility changes the visibility of the object on Coop

print(format='rich')

print_long()

Print the results in a long format. >>> from edsl.results import Results >>> r = Results.example() >>> r.select(‘how_feeling’).print_long() answer.how_feeling: OK answer.how_feeling: Great answer.how_feeling: Terrible answer.how_feeling: OK

classmethod pull(uuid: str | UUID | None = None, url: str | None = None, expected_parrot_url: str | None = None)

Pull the object from coop.

push(description: str | None = None, alias: str | None = None, visibility: str | None = 'unlisted', expected_parrot_url: str | None = None)

Post the object to coop.

property question_names: list[str]

Return a list of all of the question names.

Example:

>>> r = Results.example()
>>> r.question_names
['how_feeling', 'how_feeling_yesterday']

recode(column: str, recode_function: Callable | None, new_var_name=None) → Results

Recode a column in the Results object.

>>> r = Results.example()
>>> r.recode('how_feeling', recode_function = lambda x: 1 if x == 'Great' else 0).select('how_feeling', 'how_feeling_recoded')
Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling_recoded': [0, 1, 0, 0]}])

relevant_columns(data_type: str | None = None, remove_prefix=False) → list

Return the set of keys that are present in the dataset.

Parameters:

• data_type – The data type to filter by.

• remove_prefix – Whether to remove the prefix from the column names.

>>> from edsl.results.Dataset import Dataset
>>> d = Dataset([{'a.b':[1,2,3,4]}])
>>> d.relevant_columns()
['a.b']

>>> d.relevant_columns(remove_prefix=True)
['b']

>>> d = Dataset([{'a':[1,2,3,4]}, {'b':[5,6,7,8]}])
>>> d.relevant_columns()
['a', 'b']

>>> from edsl.results import Results; Results.example().select('how_feeling', 'how_feeling_yesterday').relevant_columns()
['answer.how_feeling', 'answer.how_feeling_yesterday']

>>> from edsl.results import Results
>>> sorted(Results.example().select().relevant_columns(data_type = "model"))
['model.frequency_penalty', ...]

>>> Results.example().relevant_columns(data_type = "flimflam")
Traceback (most recent call last):
...
ValueError: No columns found for data type: flimflam. Available data types are: ...

rename(old_name: str, new_name: str) → Results

Rename an answer column in a Results object.

>>> s = Results.example()
>>> s.rename('how_feeling', 'how_feeling_new').select('how_feeling_new')
Dataset([{'answer.how_feeling_new': ['OK', 'Great', 'Terrible', 'OK']}])

# TODO: Should we allow renaming of scenario fields as well? Probably.

rich_print()

Display an object as a table.

sample(n: int | None = None, frac: float | None = None, with_replacement: bool = True, seed: str | None = None) → Results

Sample the results.

Parameters:

• n – An integer representing the number of samples to take.

• frac – A float representing the fraction of samples to take.

• with_replacement – A boolean representing whether to sample with replacement.

• seed – An integer representing the seed for the random number generator.

Example:

>>> r = Results.example()
>>> len(r.sample(2))
2

save(filename, compress=True)

Save the object to a file as zippped JSON.

>>> obj.save("obj.json.gz")

property scenario_keys: list[str]

Return a set of all of the keys that are in the Scenario data.

>>> r = Results.example()
>>> r.scenario_keys
['period', 'scenario_index']

property scenarios: ScenarioList

Return a list of all of the scenarios in the Results.

Example:

>>> r = Results.example()
>>> r.scenarios
ScenarioList([Scenario({'period': 'morning', 'scenario_index': 0}), Scenario({'period': 'afternoon', 'scenario_index': 1}), Scenario({'period': 'morning', 'scenario_index': 0}), Scenario({'period': 'afternoon', 'scenario_index': 1})])

score(f: Callable) → list

Score the results using in a function.

Parameters:

f – A function that takes values from a Resul object and returns a score.

>>> r = Results.example()
>>> def f(status): return 1 if status == 'Joyful' else 0
>>> r.score(f)
[1, 1, 0, 0]

classmethod search(query)

Search for objects on coop.

select(*columns: str | list[str]) → Results

Select data from the results and format it.

Parameters:

columns – A list of strings, each of which is a column name. The column name can be a single key, e.g. “how_feeling”, or a dot-separated string, e.g. “answer.how_feeling”.

Example:

>>> results = Results.example()
>>> results.select('how_feeling')
Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}])

>>> results.select('how_feeling', 'model', 'how_feeling')
Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'model.model': ['...', '...', '...', '...']}, {'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}])

>>> from edsl import Results; r = Results.example(); r.select('answer.how_feeling_y')
Dataset([{'answer.how_feeling_yesterday': ['Great', 'Good', 'OK', 'Terrible']}])

show_exceptions(traceback=False)

Print the exceptions.

show_methods(show_docstrings=True)

Show the methods of the object.

shuffle(seed: str | None = 'edsl') → Results

Shuffle the results.

Example:

>>> r = Results.example()
>>> r.shuffle(seed = 1)[0]
Result(...)

sort_by(*columns: str, reverse: bool = False) → Results

Sort the results by one or more columns.

sql(query: str, transpose: bool = None, transpose_by: str = None, remove_prefix: bool = True) → pd.DataFrame | str

Execute a SQL query and return the results as a DataFrame.

Args:

query: The SQL query to execute shape: The shape of the data in the database (wide or long) remove_prefix: Whether to remove the prefix from the column names transpose: Whether to transpose the DataFrame transpose_by: The column to use as the index when transposing csv: Whether to return the DataFrame as a CSV string to_list: Whether to return the results as a list to_latex: Whether to return the results as LaTeX filename: Optional filename to save the results to

Returns:

DataFrame, CSV string, list, or LaTeX string depending on parameters

store(d: dict, key_name: str | None = None)

table(*fields, tablefmt: str | None = None, pretty_labels: dict | None = None, print_parameters: dict | None = None)

tally(*fields: str | None, top_n: int | None = None, output='Dataset') → dict | Dataset

Tally the values of a field or perform a cross-tab of multiple fields.

Parameters:

fields – The field(s) to tally, multiple fields for cross-tabulation.

>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').tally('answer.how_feeling', output = "dict")
{'OK': 2, 'Great': 1, 'Terrible': 1}
>>> from edsl.results.Dataset import Dataset
>>> expected = Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible']}, {'count': [2, 1, 1]}])
>>> r.select('how_feeling').tally('answer.how_feeling', output = "Dataset") == expected
True

to_agent_list(remove_prefix: bool = True)

Convert the results to a list of dictionaries, one per agent.

Parameters:

remove_prefix – Whether to remove the prefix from the column names.

>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_agent_list()
AgentList([Agent(traits = {'how_feeling': 'OK'}), Agent(traits = {'how_feeling': 'Great'}), Agent(traits = {'how_feeling': 'Terrible'}), Agent(traits = {'how_feeling': 'OK'})])

to_csv(filename: str | None = None, remove_prefix: bool = False, pretty_labels: dict | None = None) → FileStore | None

Export the results to a FileStore instance containing CSV data.

to_dataset()

to_dict(sort: bool = False, add_edsl_version: bool = False, include_cache: bool = False, include_task_history: bool = False, include_cache_info: bool = True) → dict[str, Any]

This method should be implemented by subclasses.

to_dicts(remove_prefix: bool = True) → list[dict]

Convert the results to a list of dictionaries.

Parameters:

remove_prefix – Whether to remove the prefix from the column names.

>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_dicts()
[{'how_feeling': 'OK'}, {'how_feeling': 'Great'}, {'how_feeling': 'Terrible'}, {'how_feeling': 'OK'}]

to_excel(filename: str | None = None, remove_prefix: bool = False, pretty_labels: dict | None = None, sheet_name: str | None = None) → FileStore | None

Export the results to a FileStore instance containing Excel data.

to_json()

to_jsonl(filename: str | None = None) → FileStore | None

Export the results to a FileStore instance containing JSONL data.

to_list(flatten=False, remove_none=False, unzipped=False) → list[list]

Convert the results to a list of lists.

Parameters:

• flatten – Whether to flatten the list of lists.

• remove_none – Whether to remove None values from the list.

>>> from edsl.results import Results
>>> Results.example().select('how_feeling', 'how_feeling_yesterday')
Dataset([{'answer.how_feeling': ['OK', 'Great', 'Terrible', 'OK']}, {'answer.how_feeling_yesterday': ['Great', 'Good', 'OK', 'Terrible']}])

>>> Results.example().select('how_feeling', 'how_feeling_yesterday').to_list()
[('OK', 'Great'), ('Great', 'Good'), ('Terrible', 'OK'), ('OK', 'Terrible')]

>>> r = Results.example()
>>> r.select('how_feeling').to_list()
['OK', 'Great', 'Terrible', 'OK']

>>> from edsl.results.Dataset import Dataset
>>> Dataset([{'a.b': [[1, 9], 2, 3, 4]}]).select('a.b').to_list(flatten = True)
[1, 9, 2, 3, 4]

>>> from edsl.results.Dataset import Dataset
>>> Dataset([{'a.b': [[1, 9], 2, 3, 4]}, {'c': [6, 2, 3, 4]}]).select('a.b', 'c').to_list(flatten = True)
Traceback (most recent call last):
...
ValueError: Cannot flatten a list of lists when there are multiple columns selected.

to_pandas(remove_prefix: bool = False, lists_as_strings=False) → DataFrame

Convert the results to a pandas DataFrame, ensuring that lists remain as lists.

Parameters:

remove_prefix – Whether to remove the prefix from the column names.

to_polars(remove_prefix: bool = False, lists_as_strings=False) → pl.DataFrame

Convert the results to a Polars DataFrame.

Parameters:

remove_prefix – Whether to remove the prefix from the column names.

to_scenario_list(remove_prefix: bool = True) → list[dict]

Convert the results to a list of dictionaries, one per scenario.

Parameters:

remove_prefix – Whether to remove the prefix from the column names.

>>> from edsl.results import Results
>>> r = Results.example()
>>> r.select('how_feeling').to_scenario_list()
ScenarioList([Scenario({'how_feeling': 'OK'}), Scenario({'how_feeling': 'Great'}), Scenario({'how_feeling': 'Terrible'}), Scenario({'how_feeling': 'OK'})])

to_sqlite(filename: str | None = None, remove_prefix: bool = False, pretty_labels: dict | None = None, table_name: str = 'results', if_exists: str = 'replace') → FileStore | None

Export the results to a SQLite database file.

to_yaml(add_edsl_version=False, filename: str = None) → str | None

tree(node_list: List[str] | None = None)

values()

Return the values of the object.

view()

Displays an interactive / perspective view of the object