> ## Documentation Index > Fetch the complete documentation index at: https://docs.expectedparrot.com/llms.txt > Use this file to discover all available pages before exploring further. # Starter Tutorial > This tutorial provides step-by-step instructions for getting started using EDSL (*Expected Parrot Domain-Specific Language*), an open-source Python library for simulating surveys, experiments and other research tasks using AI agents and large language models. EDSL is developed by [Expected Parrot](https://www.expectedparrot.com/about) and available under the MIT License. The source code is hosted on [GitHub](https://github.com/expectedparrot/edsl). ## Goals of this tutorial We begin with technical setup: instructions for installing the EDSL library and storing API keys to access language models. Then we demonstrate some of the basic features of EDSL, with examples for constructing and running surveys with agents and models, analyzing responses as datasets, and validating results with human respondents. By the end of this tutorial, you will be able to use EDSL to do each of the following: * Construct various types of [questions](/en/latest/questions) tailored to your research objectives. * Combine questions into [surveys](/en/latest/surveys) and integrate logical rules to control the survey flow. * Add context to questions by piping answers, adding memory of prior questions and answers, and using [scenarios](/en/latest/scenarios) to add data or content to questions. * Design personas for [AI agents](/en/latest/agents) to simulate responses to your surveys. * Choose and deploy [large language models](/en/latest/language_models) to generate responses for AI agents. * Analyze [results](/en/latest/results) as datasets with built-in analytical tools. * Validate LLM answers with [human respondents](/en/latest/humanize). ## Storing & sharing your work We also introduce [Expected Parrot](https://www.expectedparrot.com/login): a platform for creating, storing and sharing AI-based research and launching hybrid human/AI surveys. Expected Parrot is fully integrated with EDSL and free to use. At the end of the tutorial we show how to use EDSL with Expected Parrot by posting content created in this tutorial for anyone to view at the web app and launching a web-based survey to compare LLM and human responses. ## Further reading & questions Please see our [documentation page](/en/latest/index) for more details on each of the topics covered in this notebook. If you encounter any issues or have questions, please email us at [info@expectedparrot.com](mailto:info@expectedparrot.com) or post a question at our [Discord channel](https://discord.com/invite/mxAYkjfy9m). ## Pre-requisites EDSL is compatible with Python 3.9 - 3.12. Before starting this tutorial, please ensure that you have a Python environment set up on your machine or in a cloud-based environment, such as Google Colab. You can find instructions for installing Python at the [Python Software Foundation](https://www.python.org/downloads/). ## Recommendations The code examples in this tutorial are designed to be run in a Jupyter notebook or another Python environment, or in a cloud-based environment such as Google Colab. If you are using Google Colab, please see additional instructions for setting up EDSL in the [Colab setup](/en/latest/colab_setup) page in the documentation. We also recommend using a virtual environment when installing and using EDSL in order to avoid conflicts with other Python packages. You can find instructions for setting up a virtual environment at the [Python Packaging Authority](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/). ## Installation To begin using EDSL, you first need to install the library. This can either be done locally on your machine or in a cloud-based environment, such as Google Colab. Once you have decided where to install EDSL, you can choose to whether install it from [PyPI](https://pypi.org/project/edsl/) or [GitHub](https://github.com/expectedparrot/edsl): ### From PyPI Install EDSL directly using `pip`, which is straightforward and recommended for most users. We also recommend using a virtual environment to manage your Python packages (see *Recommendations* above). Uncomment and run the following command to install EDSL from PyPI: ```python theme={null} # ! uv pip install edsl -q ``` If you have already installed EDSL, you can uncomment and run the following code to check that your version is up to date (compare it to the version at [PyPI](https://pypi.org/project/edsl/)): ```python theme={null} # pip show edsl ``` If your version of EDSL is not up to date, uncomment and run the following code to update it: ```python theme={null} # pip install --upgrade edsl ``` ### From GitHub You can find the source code for EDSL and contribute to the project at [GitHub](https://github.com/expectedparrot/edsl). Installing from GitHub allows you to get the latest updates to EDSL before they are released to a new version at PyPI. This is recommended if you are using new features or contributing to the project. Uncomment and run the following command to install EDSL from GitHub: ```python theme={null} # pip install git+https://github.com/expectedparrot/edsl.git@main ``` ## Create an account Creating an account allows you to run survey jobs at Expected Parrot using language models of your choice, and automatically cache your results. Your account also allows you to launch human surveys and share your content and workflows with other users. Your account comes with \$25 in credits for API calls to LLMs for getting started and a referal code for earning more credits. [Create an account](https://www.expectedparrot.com/login) with an email address and password, or uncomment and run the following code to be prompted automatically: ```python theme={null} # from edsl import login # login() ``` ## Accessing LLMs The next step is to decide how you want to access language models for running surveys. EDSL works with many popular language models that you can choose from to generate responses to your surveys. These models are hosted by various service providers, such as Anthropic, Azure, Bedrock, Deep Infra, Google, Groq, Mistral, OpenAI, Replicate, Together and Xai. In order to run a survey, you need to provide API keys for the service providers of models that you want to use. There are two methods for providing API keys to EDSL: * Use your Expected Parrot API key to access all available models * Provide your own API keys from service providers ## Managing keys To manage your keys, navigate to your [Keys](https://www.expectedparrot.com/home/keys) page and use the options to add keys and optionally share access to them with other users. You can specify which keys to use at any time, and check the current priority of your keys. Your Expected Parrot API key is used by default when a private key is not provided for a selected model. Please see [instructions](https://www.expectedparrot.com/getting-started/edsl-api-keys) for alternative methods of storing your own API keys. **Note**: If you try to run a survey without storing a required API key, you will be provided a link to activate remote inference and use your Expected Parrot API key. ## Credits & tokens Running surveys with language models requires tokens. If you are using your own API keys, service providers will bill you directly. If you are using your Expected Parrot API key to access models, you will need to purchase credits to cover token costs. Please see the model pricing page for details on available models and their current prices. **Note**: Your account comes with 100 free credits. You can purchase more credits at any time at your [Credits](https://www.expectedparrot.com/home/credits) page. After installing EDSL and storing API keys you are ready to run some examples! ## Example: Running a simple question EDSL comes with a [variety of question types](/en/latest/questions) that we can choose from based on the form of the response that we want to get back from a model. To see a list of all question types: ```python theme={null} from edsl import Question Question.available() ``` | | question\_type | question\_class | example\_question | | :- | :---------------------------- | :------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | checkbox | QuestionCheckBox | Question('checkbox', question\_name = """never\_eat""", question\_text = """Which of the following foods would you eat if you had to?""", min\_selections = 2, max\_selections = 5, question\_options = \['soggy meatpie', 'rare snails', 'mouldy bread', 'panda milk custard', 'McDonalds'], include\_comment = False) | | 1 | compute | QuestionCompute | Question('compute', question\_name = """computed\_greeting""", question\_text = """Hello `{{name}}`!""") | | 2 | dict | QuestionDict | Question('dict', question\_name = """example""", question\_text = """Please provide a simple recipe for hot chocolate.""", answer\_keys = \['title', 'ingredients', 'num\_ingredients', 'instructions'], value\_types = \['str', 'list\[str]', 'int', 'str'], value\_descriptions = \['The title of the recipe.', 'A list of ingredients.', 'The number of ingredients.', 'The instructions for making the recipe.'], question\_presentation = """Please provide a simple recipe for hot chocolate.""", answering\_instructions = """Please respond with a dictionary using the following keys: title, ingredients, num\_ingredients, instructions. Here are descriptions of the values to provide: - "title": "The title of the recipe." - "ingredients": "A list of ingredients." - "num\_ingredients": "The number of ingredients." - "instructions": "The instructions for making the recipe." The values should be formatted in the following types: - "title": "str" - "ingredients": "list\[str]" - "num\_ingredients": "int" - "instructions": "str" If you do not have a value for a given key, use "null". After the answer, you can put a comment explaining your response on the next line. """) | | 3 | extract | QuestionExtract | Question('extract', question\_name = """extract\_name""", question\_text = """My name is Moby Dick. I have a PhD in astrology, but I'm actually a truck driver""", answer\_template = `{{'name': 'John Doe', 'profession': 'Carpenter'}}`) | | 4 | free\_text | QuestionFreeText | Question('free\_text', question\_name = """how\_are\_you""", question\_text = """How are you?""") | | 5 | functional | QuestionFunctional | Question('functional', question\_name = """sum\_and\_multiply""", question\_text = """Calculate the sum of the list and multiply it by the agent trait multiplier.""") | | 6 | likert\_five | QuestionLikertFive | Question('likert\_five', question\_name = """happy\_raining""", question\_text = """I'm only happy when it rains.""", question\_options = \['Strongly disagree', 'Disagree', 'Neutral', 'Agree', 'Strongly agree']) | | 7 | linear\_scale | QuestionLinearScale | Question('linear\_scale', question\_name = """ice\_cream""", question\_text = """How much do you like ice cream?""", question\_options = \[1, 2, 3, 4, 5], option\_labels = `{{1: 'I hate it', 5: 'I love it'}}`) | | 8 | list | QuestionList | Question('list', question\_name = """list\_of\_foods""", question\_text = """What are your favorite foods?""", max\_list\_items = None, min\_list\_items = None) | | 9 | matrix | QuestionMatrix | Question('matrix', question\_name = """child\_happiness""", question\_text = """How happy would you be with different numbers of children?""", question\_items = \['No children', '1 child', '2 children', '3 or more children'], question\_options = \[1, 2, 3, 4, 5], option\_labels = `{{1: 'Very sad', 3: 'Neutral', 5: 'Extremely happy'}}`) | | 10 | multiple\_choice | QuestionMultipleChoice | Question('multiple\_choice', question\_name = """how\_feeling""", question\_text = """How are you?""", question\_options = \['Good', 'Great', 'OK', 'Bad'], include\_comment = False) | | 11 | multiple\_choice\_with\_other | QuestionMultipleChoiceWithOther | Question('multiple\_choice\_with\_other', question\_name = """how\_feeling\_with\_other""", question\_text = """How are you?""", question\_options = \['Good', 'Great', 'OK', 'Bad'], include\_comment = False) | | 12 | numerical | QuestionNumerical | Question('numerical', question\_name = """age""", question\_text = """You are a 45 year old man. How old are you in years?""", min\_value = 0, max\_value = 86.7, include\_comment = False) | | 13 | rank | QuestionRank | Question('rank', question\_name = """rank\_foods""", question\_text = """Rank your favorite foods.""", question\_options = \['Pizza', 'Pasta', 'Salad', 'Soup'], num\_selections = 2) | | 14 | top\_k | QuestionTopK | Question('top\_k', question\_name = """two\_fruits""", question\_text = """Which of the following fruits do you prefer?""", min\_selections = 2, max\_selections = 2, question\_options = \['apple', 'banana', 'carrot', 'durian'], use\_code = True) | | 15 | yes\_no | QuestionYesNo | Question('yes\_no', question\_name = """is\_it\_equal""", question\_text = """Is 5 + 5 equal to 11?""", question\_options = \['No', 'Yes']) | We can see the components of a particular question type by importing the question type class and calling the `example` method on it: ```python theme={null} from edsl import ( # QuestionCheckBox, # QuestionExtract, # QuestionFreeText, # QuestionFunctional, # QuestionLikertFive, # QuestionLinearScale, # QuestionList, QuestionMultipleChoice, # QuestionNumerical, # QuestionRank, # QuestionTopK, # QuestionYesNo ) q = QuestionMultipleChoice.example() # substitute any question type class name q ``` ``` QuestionMultipleChoice ``` | | key | value | | :- | :------------------ | :--------------- | | 0 | question\_name | how\_feeling | | 1 | question\_text | How are you? | | 2 | question\_options:0 | Good | | 3 | question\_options:1 | Great | | 4 | question\_options:2 | OK | | 5 | question\_options:3 | Bad | | 6 | include\_comment | False | | 7 | question\_type | multiple\_choice | Here we create a simple multiple choice question of our own: ```python theme={null} from edsl import QuestionMultipleChoice q = QuestionMultipleChoice( question_name = "smallest_prime", question_text = "Which is the smallest prime number?", question_options = [0, 1, 2, 3] ) ``` We can administer the question to a language model by calling the `run` method on it. If you have activated remote inference and stored your Expected Parrot API key (see instructions above), the question will be run remotely at the Expected Parrot server. Results are stored at an unlisted page by default; we can also set the visibility to `public` or `private` either when we run it or by updating the object (demonstrated in later examples). We can also view a progress report for the job: ```python theme={null} results = q.run() ``` ### Inspecting results This generates a dataset of `Results` that we can readily access with [built-in methods for analysis](/en/latest/results). Here we `select()` the response to inspect it, together with the model that was used and the model’s “comment” about its response–a field that is automatically added to all question types other than free text: ```python theme={null} results.select("model", "smallest_prime", "smallest_prime_comment") ``` | | model.model | answer.smallest\_prime | comment.smallest\_prime\_comment | | :- | :---------- | :--------------------- | :-------------------------------------------------------------------------------------------------------- | | 0 | gpt-4o | 2 | 2 is the smallest prime number because it is the only even number that is divisible by only 1 and itself. | The `Results` also include information about the question, model parameters, prompts, generated tokens and raw responses. To see a list of all the components: ```python theme={null} results.columns ``` | | 0 | | :- | :------------------------------------------------------------------------ | | 0 | agent.agent\_index | | 1 | agent.agent\_instruction | | 2 | agent.agent\_name | | 3 | answer.smallest\_prime | | 4 | cache\_keys.smallest\_prime\_cache\_key | | 5 | cache\_used.smallest\_prime\_cache\_used | | 6 | comment.smallest\_prime\_comment | | 7 | generated\_tokens.smallest\_prime\_generated\_tokens | | 8 | iteration.iteration | | 9 | model.frequency\_penalty | | 10 | model.inference\_service | | 11 | model.logprobs | | 12 | model.max\_tokens | | 13 | model.model | | 14 | model.model\_index | | 15 | model.presence\_penalty | | 16 | model.temperature | | 17 | model.top\_logprobs | | 18 | model.top\_p | | 19 | prompt.smallest\_prime\_system\_prompt | | 20 | prompt.smallest\_prime\_user\_prompt | | 21 | question\_options.smallest\_prime\_question\_options | | 22 | question\_text.smallest\_prime\_question\_text | | 23 | question\_type.smallest\_prime\_question\_type | | 24 | raw\_model\_response.smallest\_prime\_cost | | 25 | raw\_model\_response.smallest\_prime\_input\_price\_per\_million\_tokens | | 26 | raw\_model\_response.smallest\_prime\_input\_tokens | | 27 | raw\_model\_response.smallest\_prime\_one\_usd\_buys | | 28 | raw\_model\_response.smallest\_prime\_output\_price\_per\_million\_tokens | | 29 | raw\_model\_response.smallest\_prime\_output\_tokens | | 30 | raw\_model\_response.smallest\_prime\_raw\_model\_response | | 31 | reasoning\_summary.smallest\_prime\_reasoning\_summary | | 32 | scenario.scenario\_index | | 33 | validated.smallest\_prime\_validated | ## Example: Conducting a survey with agents and models In the next example we construct a more complex survey consisting of multiple questions, and design personas for AI agents to answer the survey. Then we select specific language models to generate the answers. We start by creating questions in different types and passing them to a `Survey`: ```python theme={null} from edsl import QuestionLinearScale, QuestionFreeText q_enjoy = QuestionLinearScale( question_name = "enjoy", question_text = "On a scale from 1 to 5, how much do you enjoy reading?", question_options = [1, 2, 3, 4, 5], option_labels = {{1:"Not at all", 5:"Very much"} ) q_favorite_place = QuestionFreeText( question_name = "favorite_place", question_text = "Describe your favorite place for reading." ) ``` We construct a `Survey` by passing a list of questions: ```python theme={null} from edsl import Survey survey = Survey(questions = [q_enjoy, q_favorite_place]) ``` ### Agents An important feature of EDSL is the ability to create AI agents to answer questions. This is done by passing dictionaries of relevant “traits” to `Agent` objects that are used by language models to generate responses. Learn more about [designing agents](/en/latest/agents). Here we construct several simple agent personas to use with our survey: ```python theme={null} from edsl import AgentList, Agent agents = AgentList( Agent(traits = {{"persona":p}) for p in ["artist", "mechanic", "sailor"] ) ``` ### Language models EDSL works with many popular large language models that we can select to use with a survey. This makes it easy to compare responses among models in the results that are generated. See a current list of available models at our [model pricing and performance page](https://www.expectedparrot.com/getting-started/coop-pricing). You can also check available service providers: ```python theme={null} from edsl import Model Model.services() ``` [ScenarioList](/en/latest/scenarios) scenarios: 15; keys: \['service']; | | service | | :- | :----------- | | 0 | anthropic | | 1 | azure | | 2 | bedrock | | 3 | deep\_infra | | 4 | deepseek | | 5 | google | | 6 | groq | | 7 | mistral | | 8 | ollama | | 9 | open\_router | | 10 | openai | | 11 | openai\_v2 | | 12 | perplexity | | 13 | together | | 14 | xai | To check the default model that will be used if no models are specified for a survey (e.g., as in the first example above): ```python theme={null} Model() ``` ``` LanguageModel ``` | | key | value | | :- | :---------------------------- | :------- | | 0 | model | gpt-4o | | 1 | parameters:temperature | 0.500000 | | 2 | parameters:max\_tokens | 1000 | | 3 | parameters:top\_p | 1 | | 4 | parameters:frequency\_penalty | 0 | | 5 | parameters:presence\_penalty | 0 | | 6 | parameters:logprobs | False | | 7 | parameters:top\_logprobs | 3 | | 8 | inference\_service | openai | **Note**: (Note that the output may be different if the default model has changed since this page was last updated.) Here we select some models to use with our survey: ```python theme={null} from edsl import ModelList, Model models = ModelList([ Model("gpt-4o", service_name = "openai"), Model("gemini-1.5-flash", service_name = "google"), Model("claude-3-7-sonnet-20250219", service_name = "anthropic") ]) ``` ### Running a survey We add agents and models to a survey using the `by` method. Then we administer a survey the same way that we do an individual question, by calling the `run` method on it: ```python theme={null} results = survey.by(agents).by(models).run() ``` We can pass an expression to `filter()` the results and list the components to `sort_by()`: ```python theme={null} ( results .filter("persona != 'artist'") .order_by("persona", "model") .select("model", "persona", "enjoy", "favorite_place") ) ``` | | model.model | agent.persona | answer.enjoy | answer.favorite\_place | | :- | :------------------------- | :------------ | :----------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | claude-3-7-sonnet-20250219 | mechanic | 3 | As a mechanic, I'd have to say my favorite place for reading is actually in my workshop during lunch breaks or after hours when things quiet down. There's this old, worn leather chair I salvaged and fixed up that sits in the corner by a window. The natural light is perfect during the day, and I've got a good shop lamp nearby for evening reading. I usually have service manuals, repair guides, or car magazines stacked on a small side table, but I also enjoy a good thriller or history book there too. There's something calming about being surrounded by my tools and projects while taking a mental break with a book. The faint smell of oil and metal gives it a comfortable familiarity that I find relaxing. Plus, if I read something interesting about a new repair technique, I'm right where I need to be to try it out! | | 1 | gemini-1.5-flash | mechanic | 3 | My favorite place to read? Gotta be my garage, actually. Not the whole thing, mind you. Just that little corner by the workbench, where the light's just right – not too harsh, you know? I've got a comfy old stool there, worn smooth from years of use, and a little side table I’ve cobbled together from scrap parts. Got a good lamp on it, too, one of those adjustable ones so I can get the perfect angle. The air smells of oil and grease, sure, but it's a familiar smell, comforting even. Plus, there's always something interesting to look at – a half-finished project, a neat tool I haven't used in a while... it keeps my mind occupied even when I'm supposed to be concentrating on the book. It's peaceful in its own way, you know? The quiet hum of the fridge in the back, the occasional drip from a leaky faucet… it’s my sanctuary. | | 2 | gpt-4o | mechanic | 3 | As a mechanic, my favorite place for reading is actually my garage. It's not your typical cozy reading nook, but there's something about the smell of oil and the sound of tools clinking that feels comforting. I've got a sturdy workbench where I can prop up a manual or a good book about classic cars. The lighting is bright enough to read by, and when I need a break, I can glance up at the projects I’m working on. Plus, it's quiet, especially when the garage door is down, so I can really focus on whatever I'm reading. | | 3 | claude-3-7-sonnet-20250219 | sailor | 3 | Ah, me favorite spot for readin'? That'd be the fo'c'sle (forecastle) of me ship when we're anchored in a calm bay. There's somethin' special about sittin' on a sea chest with me back against the hull, lantern swingin' gently overhead, and the soft sounds of water lappin' against the sides. The gentle rockin' of the vessel puts ye in a perfect state of mind for gettin' lost in a tale. I like to read in the early mornin' when the air is crisp and most of the crew is still snorin' away, or at dusk when the day's work is done. When I'm ashore, I fancy findin' a quiet spot near the harbor where I can still see the ships and smell the salt air. A good book, the distant cry of gulls, and the promise of the open water - that's all this old salt needs for a perfect readin' nook. | | 4 | gemini-1.5-flash | sailor | 3 | Ahoy there! My favorite place for a good read? That's easy. The crow's nest, of course! Up there, high above the deck, with the wind whipping through my hair and the spray of the ocean kissing my face... It's the perfect spot. The rocking of the ship is a bit of a distraction sometimes, but the view\... the endless horizon... it's inspiring. Makes even the dullest sea shanty seem like a thrilling adventure. Plus, nobody bothers you up there! Just me, my book, and the vast, beautiful ocean. | | 5 | gpt-4o | sailor | 3 | Ah, my favorite place for reading has to be the deck of a ship, right under the open sky. There's something about the gentle sway of the sea and the salty breeze that makes the words come alive. I like to find a quiet spot, maybe near the stern where the sound of the waves is a bit more pronounced. The horizon stretches out endlessly, and with the sun setting, casting golden hues across the water, it feels like I'm part of the stories I'm reading. It's a place where adventure and tranquility meet, perfect for diving into a good book. | ## Example: Adding context to questions EDSL provides a variety of ways to add data or content to survey questions. These methods include: * [Piping](/en/latest/surveys#id2) answers to questions into follow-on questions * [Adding “memory”](/en/latest/surveys#question-memory) of prior questions and answers in a survey when presenting other questions to a model * [Parameterizing questions with data](/en/latest/scenarios), e.g., content from PDFs, CSVs, docs, images or other sources that you want to add to questions ### Piping question answers Here we demonstrate how to pipe the answer to a question into the text of another question. This is done by using a placeholder `{{ .answer }}` in the text of the follow-on question where the answer to the prior question is to be inserted when the survey is run. This causes the questions to be administered in the required order (survey questions are administered asynchronously by default). Learn more about [piping question answers](/en/latest/surveys#id2). Here we insert the answer to a numerical question into the text of a follow-on yes/no question: ```python theme={null} from edsl import QuestionNumerical, QuestionYesNo, Survey q1 = QuestionNumerical( question_name = "random_number", question_text = "Pick a random number between 1 and 1,000." ) q2 = QuestionYesNo( question_name = "prime", question_text = "Is this a prime number: {{ random_number.answer }}" ) survey = Survey([q1, q2]) results = survey.run() ``` We can check the `user_prompt` for the `prime` question to verify that the answer to the `random_number` question was piped into it: ```python theme={null} results.select("random_number", "prime_user_prompt", "prime", "prime_comment") ``` | | answer.random\_number | prompt.prime\_user\_prompt | answer.prime | comment.prime\_comment | | :- | :-------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------- | :------------------------------------------------------------------------ | | 0 | 487 | Is this a prime number: 487 No Yes Only 1 option may be selected. Please respond with just your answer. After the answer, you can put a comment explaining your response. | Yes | 487 is a prime number because it has no divisors other than 1 and itself. | ### Adding “memory” of questions and answers Here we instead add a “memory” of the first question and answer to the context of the second question. This is done by calling a memory rule and identifying the question(s) to add. Instead of just the answer, information about the full question and answer are presented with the follow-on question text, and no placeholder is used. Learn more about [question memory rules](/en/latest/surveys#survey-rules-logic). Here we demonstrate the `add_targeted_memory` method (we could also use `set_full_memory_mode` or other memory rules): ```python theme={null} from edsl import QuestionNumerical, QuestionYesNo, Survey q1 = QuestionNumerical( question_name = "random_number", question_text = "Pick a random number between 1 and 1,000." ) q2 = QuestionYesNo( question_name = "prime", question_text = "Is the number you picked a prime number?" ) survey = Survey([q1, q2]).add_targeted_memory(q2, q1) results = survey.run() ``` We can again use the `user_prompt` to verify the context that was added to the follow-on question. To view the results in a long table, we can call the `table()` and `long()` methods to modify the default table view: ```python theme={null} results.select("random_number", "prime_user_prompt", "prime", "prime_comment") ``` | | answer.random\_number | prompt.prime\_user\_prompt | answer.prime | comment.prime\_comment | | :- | :-------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------- | :------------------------------------------------------------------------ | | 0 | 487 | Is the number you picked a prime number? No Yes Only 1 option may be selected. Please respond with just your answer. After the answer, you can put a comment explaining your response. Before the question you are now answering, you already answered the following question(s): Question: Pick a random number between 1 and 1,000. Answer: 487 | Yes | 487 is a prime number because it has no divisors other than 1 and itself. | *Related topic: Learn more about exploring and simulating “randomness” with AI agents and LLMs in*[this notebook](/en/latest/notebooks/random_numbers)*.* ## Scenarios We can also add external data or content to survey questions. This can be useful when you want to efficiently create and administer multiple versions of questions at once, e.g., for conducting data labeling tasks. This is done by creating `Scenario` dictionaries for the data or content to be used with a survey, where the keys match `{{ placeholder }}` names used in question texts (or question options) and the values are the content to be added. Scenarios can also be used to [add metadata to survey results](/en/latest/notebooks/adding_metadata), e.g., data sources or other information that you may want to include in the results for reference but not necessarily include in question texts. In the next example we revise the prior survey questions about reading to take a parameter for other activities that we may want to add to the questions, and create simple scenarios for some activities. EDSL provides methods for automatically generating scenarios from a variety of data sources, including PDFs, CSVs, docs, images, tables and dicts. We use the `from_list` method to convert a list of activities into scenarios. Then we demonstrate how to use scenarios to create multiple versions of our questions either (i) when constructing a survey or (ii) when running it: * In the latter case, the `by` method is used to add scenarios to a survey of questions with placeholders at the time that it is run (the same way that agents and models are added to a survey). This adds a `scenario` column to the results with a row for each answer to each question for each scenario. * In the former case, the `loop` method is used to create a list of versions of a question with the scenarios already added to it; when the questions are passed to a survey and it is run, the results include columns for each individual question; there is no `scenario` column and a single row for each agent’s answers to all the questions. Learn more about [using scenarios](/en/latest/scenarios). Here we create scenarios for a simple list of activities: ```python theme={null} from edsl import ScenarioList scenarios = ScenarioList.from_list("activity", ["reading", "running", "relaxing"]) ``` ### Adding scenarios using the `by` method Here we add the scenarios to the survey when we run it, together with any desired agents and models: ```python theme={null} from edsl import QuestionLinearScale, QuestionFreeText, Survey q_enjoy = QuestionLinearScale( question_name = "enjoy", question_text = "On a scale from 1 to 5, how much do you enjoy {{ scenario.activity }}?", question_options = [1, 2, 3, 4, 5], option_labels = {{1:"Not at all", 5:"Very much"} ) q_favorite_place = QuestionFreeText( question_name = "favorite_place", question_text = "In a brief sentence, describe your favorite place for {{ scenario.activity }}." ) survey = Survey([q_enjoy, q_favorite_place]) results = survey.by(scenarios).by(agents).by(models).run() ``` We can optionally drop the prefixes `agent`, `scenario`, `answer`, etc., when fields are unique: ```python theme={null} ( results .filter("model.model == 'gpt-4o'") .order_by("activity", "persona") .select("activity", "persona", "enjoy", "favorite_place") ) ``` | | scenario.activity | agent.persona | answer.enjoy | answer.favorite\_place | | :- | :---------------- | :------------ | :----------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | reading | artist | 4 | My favorite place for reading is a cozy corner of my art studio, surrounded by vibrant canvases and the soft glow of afternoon light filtering through the window. | | 1 | reading | mechanic | 3 | My favorite place for reading is the cozy corner of my garage, surrounded by tools and the smell of motor oil, where I can escape into a good book during breaks. | | 2 | reading | sailor | 3 | My favorite place for reading is the deck of a ship, with the sound of waves lapping against the hull and a gentle sea breeze in the air. | | 3 | relaxing | artist | 4 | My favorite place for relaxing is a quiet, sun-dappled corner of my art studio, surrounded by canvases and the soft hum of creativity. | | 4 | relaxing | mechanic | 4 | My favorite place for relaxing is my garage, surrounded by the familiar scent of motor oil and the satisfying hum of engines. | | 5 | relaxing | sailor | 3 | My favorite place for relaxing is on the deck of a ship, watching the horizon as the sun sets over the endless ocean. | | 6 | running | artist | 2 | My favorite place for running is a serene forest trail where the dappled sunlight dances through the leaves and the air is filled with the earthy scent of nature. | | 7 | running | mechanic | 1 | I don't run much, but I'd imagine a quiet trail through the woods would be a nice spot for a jog. | | 8 | running | sailor | 2 | As a sailor, my favorite place for running is along the beach at sunrise, with the sound of the waves and the salty sea breeze filling the air. | ### Adding scenarios using the `loop` method Here we add scenarios to questions when constructing a survey, as opposed to when running it. When we run the survey the results will include columns for each question and no `scenario` field. Note that we can also optionally use the scenario key in the question names (they are otherwise incremented by default): ```python theme={null} from edsl import QuestionLinearScale, QuestionFreeText q_enjoy = QuestionLinearScale( question_name = "enjoy_{{ scenario.activity }}", # optional use of scenario key question_text = "On a scale from 1 to 5, how much do you enjoy {{ scenario.activity }}?", question_options = [1, 2, 3, 4, 5], option_labels = {{1:"Not at all", 5:"Very much"} ) q_favorite_place = QuestionFreeText( question_name = "favorite_place_{{ scenario.activity }}", # optional use of scenario key question_text = "In a brief sentence, describe your favorite place for {{ scenario.activity }}." ) ``` Looping the scenarios to create lists of questions: ```python theme={null} enjoy_questions = q_enjoy.loop(scenarios) enjoy_questions ``` ``` [Question('linear_scale', question_name = """enjoy_reading""", question_text = """On a scale from 1 to 5, how much do you enjoy reading?""", question_options = [1, 2, 3, 4, 5], option_labels = {{1: 'Not at all', 5: 'Very much'}), Question('linear_scale', question_name = """enjoy_running""", question_text = """On a scale from 1 to 5, how much do you enjoy running?""", question_options = [1, 2, 3, 4, 5], option_labels = {{1: 'Not at all', 5: 'Very much'}), Question('linear_scale', question_name = """enjoy_relaxing""", question_text = """On a scale from 1 to 5, how much do you enjoy relaxing?""", question_options = [1, 2, 3, 4, 5], option_labels = {{1: 'Not at all', 5: 'Very much'})] ``` ```python theme={null} favorite_place_questions = q_favorite_place.loop(scenarios) favorite_place_questions ``` ``` [Question('free_text', question_name = """favorite_place_reading""", question_text = """In a brief sentence, describe your favorite place for reading."""), Question('free_text', question_name = """favorite_place_running""", question_text = """In a brief sentence, describe your favorite place for running."""), Question('free_text', question_name = """favorite_place_relaxing""", question_text = """In a brief sentence, describe your favorite place for relaxing.""")] ``` Combining the questions in a survey: ```python theme={null} survey = Survey(questions = enjoy_questions + favorite_place_questions) results = survey.by(agents).by(models).run() ``` We can see that there are additional question fields and no “scenario” field: ```python theme={null} results.columns ``` | | 0 | | :-- | :---------------------------------------------------------------------------------- | | 0 | agent.agent\_index | | 1 | agent.agent\_instruction | | 2 | agent.agent\_name | | 3 | agent.persona | | 4 | answer.enjoy\_reading | | 5 | answer.enjoy\_relaxing | | 6 | answer.enjoy\_running | | 7 | answer.favorite\_place\_reading | | 8 | answer.favorite\_place\_relaxing | | 9 | answer.favorite\_place\_running | | 10 | cache\_keys.enjoy\_reading\_cache\_key | | 11 | cache\_keys.enjoy\_relaxing\_cache\_key | | 12 | cache\_keys.enjoy\_running\_cache\_key | | 13 | cache\_keys.favorite\_place\_reading\_cache\_key | | 14 | cache\_keys.favorite\_place\_relaxing\_cache\_key | | 15 | cache\_keys.favorite\_place\_running\_cache\_key | | 16 | cache\_used.enjoy\_reading\_cache\_used | | 17 | cache\_used.enjoy\_relaxing\_cache\_used | | 18 | cache\_used.enjoy\_running\_cache\_used | | 19 | cache\_used.favorite\_place\_reading\_cache\_used | | 20 | cache\_used.favorite\_place\_relaxing\_cache\_used | | 21 | cache\_used.favorite\_place\_running\_cache\_used | | 22 | comment.enjoy\_reading\_comment | | 23 | comment.enjoy\_relaxing\_comment | | 24 | comment.enjoy\_running\_comment | | 25 | comment.favorite\_place\_reading\_comment | | 26 | comment.favorite\_place\_relaxing\_comment | | 27 | comment.favorite\_place\_running\_comment | | 28 | generated\_tokens.enjoy\_reading\_generated\_tokens | | 29 | generated\_tokens.enjoy\_relaxing\_generated\_tokens | | 30 | generated\_tokens.enjoy\_running\_generated\_tokens | | 31 | generated\_tokens.favorite\_place\_reading\_generated\_tokens | | 32 | generated\_tokens.favorite\_place\_relaxing\_generated\_tokens | | 33 | generated\_tokens.favorite\_place\_running\_generated\_tokens | | 34 | iteration.iteration | | 35 | model.frequency\_penalty | | 36 | model.inference\_service | | 37 | model.logprobs | | 38 | model.maxOutputTokens | | 39 | model.max\_tokens | | 40 | model.model | | 41 | model.model\_index | | 42 | model.presence\_penalty | | 43 | model.stopSequences | | 44 | model.temperature | | 45 | model.topK | | 46 | model.topP | | 47 | model.top\_logprobs | | 48 | model.top\_p | | 49 | prompt.enjoy\_reading\_system\_prompt | | 50 | prompt.enjoy\_reading\_user\_prompt | | 51 | prompt.enjoy\_relaxing\_system\_prompt | | 52 | prompt.enjoy\_relaxing\_user\_prompt | | 53 | prompt.enjoy\_running\_system\_prompt | | 54 | prompt.enjoy\_running\_user\_prompt | | 55 | prompt.favorite\_place\_reading\_system\_prompt | | 56 | prompt.favorite\_place\_reading\_user\_prompt | | 57 | prompt.favorite\_place\_relaxing\_system\_prompt | | 58 | prompt.favorite\_place\_relaxing\_user\_prompt | | 59 | prompt.favorite\_place\_running\_system\_prompt | | 60 | prompt.favorite\_place\_running\_user\_prompt | | 61 | question\_options.enjoy\_reading\_question\_options | | 62 | question\_options.enjoy\_relaxing\_question\_options | | 63 | question\_options.enjoy\_running\_question\_options | | 64 | question\_options.favorite\_place\_reading\_question\_options | | 65 | question\_options.favorite\_place\_relaxing\_question\_options | | 66 | question\_options.favorite\_place\_running\_question\_options | | 67 | question\_text.enjoy\_reading\_question\_text | | 68 | question\_text.enjoy\_relaxing\_question\_text | | 69 | question\_text.enjoy\_running\_question\_text | | 70 | question\_text.favorite\_place\_reading\_question\_text | | 71 | question\_text.favorite\_place\_relaxing\_question\_text | | 72 | question\_text.favorite\_place\_running\_question\_text | | 73 | question\_type.enjoy\_reading\_question\_type | | 74 | question\_type.enjoy\_relaxing\_question\_type | | 75 | question\_type.enjoy\_running\_question\_type | | 76 | question\_type.favorite\_place\_reading\_question\_type | | 77 | question\_type.favorite\_place\_relaxing\_question\_type | | 78 | question\_type.favorite\_place\_running\_question\_type | | 79 | raw\_model\_response.enjoy\_reading\_cost | | 80 | raw\_model\_response.enjoy\_reading\_input\_price\_per\_million\_tokens | | 81 | raw\_model\_response.enjoy\_reading\_input\_tokens | | 82 | raw\_model\_response.enjoy\_reading\_one\_usd\_buys | | 83 | raw\_model\_response.enjoy\_reading\_output\_price\_per\_million\_tokens | | 84 | raw\_model\_response.enjoy\_reading\_output\_tokens | | 85 | raw\_model\_response.enjoy\_reading\_raw\_model\_response | | 86 | raw\_model\_response.enjoy\_relaxing\_cost | | 87 | raw\_model\_response.enjoy\_relaxing\_input\_price\_per\_million\_tokens | | 88 | raw\_model\_response.enjoy\_relaxing\_input\_tokens | | 89 | raw\_model\_response.enjoy\_relaxing\_one\_usd\_buys | | 90 | raw\_model\_response.enjoy\_relaxing\_output\_price\_per\_million\_tokens | | 91 | raw\_model\_response.enjoy\_relaxing\_output\_tokens | | 92 | raw\_model\_response.enjoy\_relaxing\_raw\_model\_response | | 93 | raw\_model\_response.enjoy\_running\_cost | | 94 | raw\_model\_response.enjoy\_running\_input\_price\_per\_million\_tokens | | 95 | raw\_model\_response.enjoy\_running\_input\_tokens | | 96 | raw\_model\_response.enjoy\_running\_one\_usd\_buys | | 97 | raw\_model\_response.enjoy\_running\_output\_price\_per\_million\_tokens | | 98 | raw\_model\_response.enjoy\_running\_output\_tokens | | 99 | raw\_model\_response.enjoy\_running\_raw\_model\_response | | 100 | raw\_model\_response.favorite\_place\_reading\_cost | | 101 | raw\_model\_response.favorite\_place\_reading\_input\_price\_per\_million\_tokens | | 102 | raw\_model\_response.favorite\_place\_reading\_input\_tokens | | 103 | raw\_model\_response.favorite\_place\_reading\_one\_usd\_buys | | 104 | raw\_model\_response.favorite\_place\_reading\_output\_price\_per\_million\_tokens | | 105 | raw\_model\_response.favorite\_place\_reading\_output\_tokens | | 106 | raw\_model\_response.favorite\_place\_reading\_raw\_model\_response | | 107 | raw\_model\_response.favorite\_place\_relaxing\_cost | | 108 | raw\_model\_response.favorite\_place\_relaxing\_input\_price\_per\_million\_tokens | | 109 | raw\_model\_response.favorite\_place\_relaxing\_input\_tokens | | 110 | raw\_model\_response.favorite\_place\_relaxing\_one\_usd\_buys | | 111 | raw\_model\_response.favorite\_place\_relaxing\_output\_price\_per\_million\_tokens | | 112 | raw\_model\_response.favorite\_place\_relaxing\_output\_tokens | | 113 | raw\_model\_response.favorite\_place\_relaxing\_raw\_model\_response | | 114 | raw\_model\_response.favorite\_place\_running\_cost | | 115 | raw\_model\_response.favorite\_place\_running\_input\_price\_per\_million\_tokens | | 116 | raw\_model\_response.favorite\_place\_running\_input\_tokens | | 117 | raw\_model\_response.favorite\_place\_running\_one\_usd\_buys | | 118 | raw\_model\_response.favorite\_place\_running\_output\_price\_per\_million\_tokens | | 119 | raw\_model\_response.favorite\_place\_running\_output\_tokens | | 120 | raw\_model\_response.favorite\_place\_running\_raw\_model\_response | | 121 | reasoning\_summary.enjoy\_reading\_reasoning\_summary | | 122 | reasoning\_summary.enjoy\_relaxing\_reasoning\_summary | | 123 | reasoning\_summary.enjoy\_running\_reasoning\_summary | | 124 | reasoning\_summary.favorite\_place\_reading\_reasoning\_summary | | 125 | reasoning\_summary.favorite\_place\_relaxing\_reasoning\_summary | | 126 | reasoning\_summary.favorite\_place\_running\_reasoning\_summary | | 127 | scenario.scenario\_index | | 128 | validated.enjoy\_reading\_validated | | 129 | validated.enjoy\_relaxing\_validated | | 130 | validated.enjoy\_running\_validated | | 131 | validated.favorite\_place\_reading\_validated | | 132 | validated.favorite\_place\_relaxing\_validated | | 133 | validated.favorite\_place\_running\_validated | ```python theme={null} ( results .filter("model.model == 'gpt-4o'") .order_by("persona") .select("persona", "enjoy_reading", "enjoy_running", "enjoy_relaxing", "favorite_place_reading", "favorite_place_running", "favorite_place_relaxing") ) ``` | | agent.persona | answer.enjoy\_reading | answer.enjoy\_running | answer.enjoy\_relaxing | answer.favorite\_place\_reading | answer.favorite\_place\_running | answer.favorite\_place\_relaxing | | :- | :------------ | :-------------------- | :-------------------- | :--------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------- | | 0 | artist | 4 | 2 | 4 | My favorite place for reading is a cozy corner of my art studio, surrounded by vibrant canvases and the soft glow of afternoon light filtering through the window. | My favorite place for running is a serene forest trail where the dappled sunlight dances through the leaves and the air is filled with the earthy scent of nature. | My favorite place for relaxing is a quiet, sun-dappled corner of my art studio, surrounded by canvases and the soft hum of creativity. | | 1 | mechanic | 3 | 1 | 4 | My favorite place for reading is the cozy corner of my garage, surrounded by tools and the smell of motor oil, where I can escape into a good book during breaks. | I don't run much, but I'd imagine a quiet trail through the woods would be a nice spot for a jog. | My favorite place for relaxing is my garage, surrounded by the familiar scent of motor oil and the satisfying hum of engines. | | 2 | sailor | 3 | 2 | 3 | My favorite place for reading is the deck of a ship, with the sound of waves lapping against the hull and a gentle sea breeze in the air. | As a sailor, my favorite place for running is along the beach at sunrise, with the sound of the waves and the salty sea breeze filling the air. | My favorite place for relaxing is on the deck of a ship, watching the horizon as the sun sets over the endless ocean. | ## Exploring `Results` EDSL comes with [built-in methods for analyzing and visualizing survey results](/en/latest/language_models). For example, you can call the `to_pandas` method to convert results into a dataframe: ```python theme={null} df = results.to_pandas(remove_prefix=True) df ``` | | enjoy\_reading | enjoy\_running | enjoy\_relaxing | favorite\_place\_reading | favorite\_place\_running | favorite\_place\_relaxing | scenario\_index | agent\_instruction | persona | agent\_index | ... | enjoy\_relaxing\_reasoning\_summary | enjoy\_reading\_reasoning\_summary | enjoy\_running\_reasoning\_summary | favorite\_place\_running\_reasoning\_summary | enjoy\_running\_validated | enjoy\_relaxing\_validated | enjoy\_reading\_validated | favorite\_place\_running\_validated | favorite\_place\_reading\_validated | favorite\_place\_relaxing\_validated | | :- | :------------- | :------------- | :-------------- | :------------------------------------------------- | :------------------------------------------------ | :------------------------------------------------ | :-------------- | :------------------------------------------------ | :------- | :----------- | :-- | :---------------------------------- | :--------------------------------- | :--------------------------------- | :------------------------------------------- | :------------------------ | :------------------------- | :------------------------ | :---------------------------------- | :---------------------------------- | :----------------------------------- | | 0 | 4 | 2 | 4 | My favorite place for reading is a cozy corner... | My favorite place for running is a serene fore... | My favorite place for relaxing is a quiet, sun... | 0 | You are answering questions as if you were a h... | artist | 0 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True | | 1 | 3 | 1 | 5 | Honestly? Anywhere the light's good and I've ... | Anywhere with a good view—preferably overlooki... | My favorite place to relax is nestled in my st... | 0 | You are answering questions as if you were a h... | artist | 0 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True | | 2 | 5 | 3 | 4 | My favorite place for reading is my sun-drench... | As an artist, I find running along the coastal... | My favorite place for relaxing is my sun-drenc... | 0 | You are answering questions as if you were a h... | artist | 0 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True | | 3 | 3 | 1 | 4 | My favorite place for reading is the cozy corn... | I don't run much, but I'd imagine a quiet trai... | My favorite place for relaxing is my garage, s... | 0 | You are answering questions as if you were a h... | mechanic | 1 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True | | 4 | 3 | 1 | 3 | My favorite place to read? Gotta be my garage... | Anywhere with a good, solid, well-maintained r... | My garage, with a cold beer and a good engine ... | 0 | You are answering questions as if you were a h... | mechanic | 1 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True | | 5 | 3 | 3 | 4 | My favorite place for reading is in my small w\... | As a mechanic, I'd say my favorite place for r... | My favorite place to relax is my garage worksh... | 0 | You are answering questions as if you were a h... | mechanic | 1 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True | | 6 | 3 | 2 | 3 | My favorite place for reading is the deck of a... | As a sailor, my favorite place for running is ... | My favorite place for relaxing is on the deck ... | 0 | You are answering questions as if you were a h... | sailor | 2 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True | | 7 | 3 | 1 | 4 | The crow's nest, of course! The wind in my ha... | Anywhere the wind whips off the ocean and the ... | A quiet cove, sheltered from the wind, with th... | 0 | You are answering questions as if you were a h... | sailor | 2 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True | | 8 | 3 | 3 | 3 | I love reading in the ship's crow's nest at du... | I'd say the long stretch of beach at sunrise, ... | Nothing beats the gentle sway of a hammock on ... | 0 | You are answering questions as if you were a h... | sailor | 2 | ... | NaN | NaN | NaN | NaN | True | True | True | True | True | True | 9 rows × 134 columns The `Results` object also supports SQL-like queries with the the `sql` method: ```python theme={null} results.sql(""" select model, persona, enjoy_reading, favorite_place_reading from self order by 1,2,3 """) ``` | | model | persona | enjoy\_reading | favorite\_place\_reading | | :- | :------------------------- | :------- | :------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | claude-3-7-sonnet-20250219 | artist | 5 | My favorite place for reading is my sun-drenched studio corner, where the natural light perfectly illuminates the pages while inspiring my artistic sensibilities. | | 1 | claude-3-7-sonnet-20250219 | mechanic | 3 | My favorite place for reading is in my small workshop corner, where the familiar smell of motor oil and the soft hum of the shop fan create a surprisingly peaceful atmosphere after a long day of working on engines. | | 2 | claude-3-7-sonnet-20250219 | sailor | 3 | I love reading in the ship's crow's nest at dusk, with the gentle rocking of the vessel and the vast ocean stretching to the horizon - it's peaceful above the bustle of the deck below. | | 3 | gemini-1.5-flash | artist | 3 | Honestly? Anywhere the light's good and I've got a strong cup of coffee nearby. A sun-drenched cafe, my messy studio, even a quiet park bench will do. | | 4 | gemini-1.5-flash | mechanic | 3 | My favorite place to read? Gotta be my garage, surrounded by the comforting smell of engine grease and the quiet hum of the air compressor. | | 5 | gemini-1.5-flash | sailor | 3 | The crow's nest, of course! The wind in my hair, the sea stretching out...perfect. | | 6 | gpt-4o | artist | 4 | My favorite place for reading is a cozy corner of my art studio, surrounded by vibrant canvases and the soft glow of afternoon light filtering through the window. | | 7 | gpt-4o | mechanic | 3 | My favorite place for reading is the cozy corner of my garage, surrounded by tools and the smell of motor oil, where I can escape into a good book during breaks. | | 8 | gpt-4o | sailor | 3 | My favorite place for reading is the deck of a ship, with the sound of waves lapping against the hull and a gentle sea breeze in the air. | ## Validating results with humans We can use the `humanize` method to launch a web-based version of a survey to collect responses from humans. Responses are immediately available at your Expected Parrot account, where you can launch surveys with LLMs and human responsents interactively. Here we use a method for generating a web-based version of the above survey, answer it, and then inspect the new results in code. Learn more about [launching hybrid surveys](/en/latest/humanize) about collecting responses with [participant platform](/en/latest/prolific) integrations. ```python theme={null} web_info = survey.humanize() web_info ``` ```json theme={null} { 'project_name': 'Project', 'uuid': '2504145b-329d-4970-82fb-10659e21e62c', 'admin_url': 'https://www.expectedparrot.com/home/projects/2504145b-329d-4970-82fb-10659e21e62c', 'respondent_url': 'https://www.expectedparrot.com/respond/projects/2504145b-329d-4970-82fb-10659e21e62c/runs/d6e66568-7921-479e-bb44-5b8880f3d9a2' } ``` ```python theme={null} from edsl import Coop coop = Coop() human_results = coop.get_project_human_responses(web_info["uuid"]) ``` ```python theme={null} human_results ``` ## Posting to Expected Parrot [Expected Parrot](https://www.expectedparrot.com/login) is a platform for creating, storing and sharing LLM-based research. It is fully integrated with EDSL and accessible from your workspace or Expected Parrot account page. Learn more about [creating an account](https://www.expectedparrot.com/login) and [the platform](/en/latest/coop). We can post any EDSL object to Expected Parrot by call the `push` method on it, optionally passing a `description` for the object, a convenient `alias` for the URL, and a `visibility` status (*public*, *private* or *unlisted* by default). For example, the results above are already posted to Expected Parrot because they were generated using remote inference (see links). The following code will post them manually: ```python theme={null} results.push( description = "Starter tutorial sample survey results", alias = "starter-tutorial-example-survey-results", visibility = "public" ) ``` We can also post this notebook: ```python theme={null} notebook.push( description = "Starter Tutorial", alias = "starter-tutorial-notebook", visibility = "public" ) ``` To update an object: ```python theme={null} from edsl import Notebook notebook = Notebook(path = "starter_tutorial.ipynb") # resave notebook.patch("https://www.expectedparrot.com/content/RobinHorton/starter-tutorial-notebook", value = notebook) ``` ```json theme={null} {'status': 'success', 'message': None, 'requires_upload': False, 'object_uuid': None} ```