Piping questions and answers
This notebook demonstrates how to pipe components of a question and answer into follow-on questions. When piping is used, survey questions are automatically administered in the order required to facilitate the piping (i.e., a question with components piped from another question will be administered later instead of asynchronously by default).
We also compare piping to methods for adding memory to surveys, which allow you to automatically add the context of specified questions to other question in a survey. Please see the surveys section of the docs for more details on each of the methods shown below.
Before running the code below, please see instructions on installing the EDSL library and storing API keys for language models.
We start by importing tools and constructing an initial question:
[1]:
from edsl import QuestionMultipleChoice, QuestionFreeText, Survey
[2]:
q1 = QuestionMultipleChoice(
question_name = "season",
question_text = "What is your favorite season?",
question_options = ["Spring", "Summer", "Fall", "Winter"]
)
In our next question, we create {{ placeholders }} for the components of the first question that we want to include. The placeholders must include the question_name of the piped question and the name of the component (e.g., answer or comment):
[3]:
q2_piping = QuestionFreeText(
question_name = "piping",
question_text = """
You were previously asked '{{ season.question_text }}' and you answered: '{{ season.answer }}'
You also provided this comment about your response: '{{ season.comment }}'
What is your favorite activity during this season?
"""
)
We combine the questions in a survey as usual:
[4]:
survey_piping = Survey([q1, q2_piping])
Before running the survey, we can inspect the prompts for the questions by calling the show_prompts() method on the survey. We can see that the multiple choice question automatically includes answering instructions for the question type, and an instruction for the model to add a comment about its response which will be stored as a separate component in results (we will see this when the question is run). (The instruction to add a comment is automatically added to all question types other
than free text and can be omitted by passing include_comment=False).
In the second question, we can see that the components of the first question text, answer and comment will be inserted. Note that there is no system prompt displayed in the prompts because we have not added any agents:
[5]:
survey_piping.show_prompts()
[5]:
| user_prompt | system_prompt | interview_index | question_name | scenario_index | agent_index | model | estimated_cost | cache_key | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | What is your favorite season? Spring Summer Fall Winter Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | nan | 0 | season | 0 | 0 | gpt-4o | 0.000692 | 71fe04dc8be8ab7fcb9d69f58781a9df |
| 1 |
You were previously asked 'What is your favorite season?' and you answered: '< |
nan | 0 | piping | 0 | 0 | gpt-4o | 0.000652 | d0653eecfd755ed52a7484e655eec02d |
Inspecting completed prompts in results
Here we run the survey and inspect results, which also include the prompts that were used with the piped components:
[6]:
results = survey_piping.run()
| Job UUID | 31fdc313-2b78-43ff-ba39-1bc0e4bd2836 |
| Progress Bar URL | https://www.expectedparrot.com/home/remote-job-progress/31fdc313-2b78-43ff-ba39-1bc0e4bd2836 |
| Exceptions Report URL | None |
| Results UUID | f8d79b51-cca8-4d0e-889a-7a2dd2feeed2 |
| Results URL | https://www.expectedparrot.com/content/f8d79b51-cca8-4d0e-889a-7a2dd2feeed2 |
To see a list of all the components of results:
[7]:
results.columns
[7]:
| 0 | |
|---|---|
| 0 | agent.agent_index |
| 1 | agent.agent_instruction |
| 2 | agent.agent_name |
| 3 | answer.piping |
| 4 | answer.season |
| 5 | cache_keys.piping_cache_key |
| 6 | cache_keys.season_cache_key |
| 7 | cache_used.piping_cache_used |
| 8 | cache_used.season_cache_used |
| 9 | comment.piping_comment |
| 10 | comment.season_comment |
| 11 | generated_tokens.piping_generated_tokens |
| 12 | generated_tokens.season_generated_tokens |
| 13 | iteration.iteration |
| 14 | model.frequency_penalty |
| 15 | model.inference_service |
| 16 | model.logprobs |
| 17 | model.max_tokens |
| 18 | model.model |
| 19 | model.model_index |
| 20 | model.presence_penalty |
| 21 | model.temperature |
| 22 | model.top_logprobs |
| 23 | model.top_p |
| 24 | prompt.piping_system_prompt |
| 25 | prompt.piping_user_prompt |
| 26 | prompt.season_system_prompt |
| 27 | prompt.season_user_prompt |
| 28 | question_options.piping_question_options |
| 29 | question_options.season_question_options |
| 30 | question_text.piping_question_text |
| 31 | question_text.season_question_text |
| 32 | question_type.piping_question_type |
| 33 | question_type.season_question_type |
| 34 | raw_model_response.piping_cost |
| 35 | raw_model_response.piping_one_usd_buys |
| 36 | raw_model_response.piping_raw_model_response |
| 37 | raw_model_response.season_cost |
| 38 | raw_model_response.season_one_usd_buys |
| 39 | raw_model_response.season_raw_model_response |
| 40 | scenario.scenario_index |
To inspect the user prompts, which now include the piped components:
[8]:
results.select("season_user_prompt", "piping_user_prompt")
[8]:
| prompt.season_user_prompt | prompt.piping_user_prompt | |
|---|---|---|
| 0 | What is your favorite season? Spring Summer Fall Winter Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | You were previously asked 'What is your favorite season?' and you answered: 'Fall' You also provided this comment about your response: 'I chose fall because of the beautiful foliage, the crisp air, and the cozy atmosphere it brings with activities like apple picking and pumpkin carving.' What is your favorite activity during this season? |
Compare to memory rules
EDSL provides a variety of rules for automatically adding the context of one or more specified questions and answers to other questions in a survey. See the docs page for examples of all of these methods.
Here we create a different version of the second question (omitting piped components and context) and add a memory of the first question after combining them in a survey. Then we show how this impacts the prompts tha are generated:
[9]:
q2_memory = QuestionFreeText(
question_name = "memory",
question_text = """
What is your favorite activity during this season?
"""
)
[10]:
survey_memory = Survey([q1, q2_memory]).set_full_memory_mode()
[11]:
survey_memory.show_prompts()
[11]:
| user_prompt | system_prompt | interview_index | question_name | scenario_index | agent_index | model | estimated_cost | cache_key | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | What is your favorite season? Spring Summer Fall Winter Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | nan | 0 | season | 0 | 0 | gpt-4o | 0.000692 | 71fe04dc8be8ab7fcb9d69f58781a9df |
| 1 | What is your favorite activity during this season? Before the question you are now answering, you already answered the following question(s): Question: What is your favorite season? Answer: None | nan | 0 | memory | 0 | 0 | gpt-4o | 0.000560 | 44fa7d3f2f20e05afaa038a1f4a09c38 |
We can see that the following context has been added to the second question: “Before the question you are now answering, you already answered the following question(s): Question: What is your favorite season? Answer: None”
Note that this context does not include the comment field!
If multiple prior questions are added to memory they are added in the same way:
[12]:
q3 = QuestionFreeText(
question_name = "poem",
question_text = "Write a poem about your favorite season."
)
Survey([q1, q2_memory, q3]).set_full_memory_mode().show_prompts()
[12]:
| user_prompt | system_prompt | interview_index | question_name | scenario_index | agent_index | model | estimated_cost | cache_key | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | What is your favorite season? Spring Summer Fall Winter Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | nan | 0 | season | 0 | 0 | gpt-4o | 0.000692 | 71fe04dc8be8ab7fcb9d69f58781a9df |
| 1 | What is your favorite activity during this season? Before the question you are now answering, you already answered the following question(s): Question: What is your favorite season? Answer: None | nan | 0 | memory | 0 | 0 | gpt-4o | 0.000560 | 44fa7d3f2f20e05afaa038a1f4a09c38 |
| 2 | Write a poem about your favorite season. Before the question you are now answering, you already answered the following question(s): Question: What is your favorite season? Answer: None Prior questions and answers: Question: What is your favorite activity during this season? Answer: None | nan | 0 | poem | 0 | 0 | gpt-4o | 0.000800 | 4b9d1157e101ecb8c3e5fb2a19dd63e2 |
Note that full memory can results in long question contexts when a survey consists of many questions. Before using a memory rule, we commend investigating impacts to the accuracy of responses by testing examples, and whether memory is needed for any particular question.
Posting to Coop
Here we post this notebook to Coop, and then show how to update the object at Coop. Learn more about using Coop to store files, surveys, results and projects.
[ ]:
from edsl import Notebook
nb = Notebook(path = "piping_comments.ipynb")
if refresh := False:
nb.push(
description = "Piping example",
alias = "piping-comments-notebook",
visibility = "public"
)
else:
nb.patch("https://www.expectedparrot.com/content/RobinHorton/piping-comments-notebook", value = nb)