Skip to main content

​
Default question instructions

Each EDSL question type includes default instructions to the model about how to answer the question. We can view these instructions by inspecting the user prompt for a question that has been created (the other type of prompt–systen prompt–is for agent instructions). For example, here we see that the default instruction for multiple choice questions is: “Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line.” This text is automatically appended to the question text: 
from edsl import QuestionMultipleChoice, Survey, Model

q = QuestionMultipleChoice(
    question_name = "primary_color",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"]
)

survey = Survey([q])
survey.show_prompts()
user_promptsystem_promptinterview_indexquestion_namescenario_indexagent_indexmodelestimated_costcache_keys
0What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line.nan0primary_color00gpt-4o0.000678[‘20e75009c72f3e88c490c58bf13d6a72’]
We can isolate the user prompt: 
survey.by(Model()).prompts().select("user_prompt")
user_prompt
0What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line.
We can compare this with default instructions for other question types: 
from edsl import QuestionCheckBox, Survey, Model

q = QuestionCheckBox(
    question_name = "primary_colors",
    question_text = "Which colors are 'primary'?",
    question_options = ["Red", "Orange", "Yellow", "Green", "Blue", "Purple"]
)

survey = Survey([q])
survey.by(Model()).prompts().select("user_prompt")
user_prompt
0Which colors are ‘primary’? Red Orange Yellow Green Blue Purple Please respond only with a comma-separated list of the options that apply, with square brackets. E.g., [‘Good’, ‘Bad’, ‘Ugly’] After the answer, you can put a comment explaining your choice on the next line.
from edsl import QuestionRank, Survey, Model

q = QuestionRank(
    question_name = "primary_colors_rank",
    question_text = "Rank the primary colors in terms of popularity.",
    question_options = ["Red", "Yellow", "Blue"]
)

survey = Survey([q])
survey.by(Model()).prompts().select("user_prompt")
user_prompt
0Rank the primary colors in terms of popularity. The options are 0: Red 1: Yellow 2: Blue You have to include 3 options in your answer. Please respond only with a comma-separated list of the code of the raked options, with square brackets. E.g., [0, 1, 3] After the answer, you can put a comment explaining your choice on the next line.
from edsl import QuestionLinearScale, Survey, Model

q = QuestionLinearScale(
    question_name = "primary_color_scale",
    question_text = "Most people know what the primary colors are.",
    question_options = [1,2,3,4,5],
    option_labels = {
        1:"This statement is completely inaccurate",
        5:"This statement is completely accurate."
    }
)

survey = Survey([q])
survey.by(Model()).prompts().select("user_prompt")
user_prompt
0Most people know what the primary colors are. 1 : This statement is completely inaccurate 2 : 3 : 4 : 5 : This statement is completely accurate. Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., “1” or “5” by itself. After the answer, you can put a comment explaining why you chose that option on the next line.

​
Formatting answers & comments

We can see that each default instruction includes directions on (1) formatting the answer and (2) providing a comment about the answer. When a question is administered, the contents of the comment that is returned are automatically stored in a separate field of the results. We can see this when we run a question and inspect the columns of the results that have been created. Here we run the multiple choice question created above: 
from edsl import QuestionMultipleChoice, Survey, Model

q = QuestionMultipleChoice(
    question_name = "primary_color",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"]
)

r = q.run() # default model will be used
We can see that the results include a comment field: 
r.columns
0
0agent.agent_index
1agent.agent_instruction
2agent.agent_name
3answer.primary_color
4cache_keys.primary_color_cache_key
5cache_used.primary_color_cache_used
6comment.primary_color_comment
7generated_tokens.primary_color_generated_tokens
8iteration.iteration
9model.frequency_penalty
10model.inference_service
11model.logprobs
12model.max_tokens
13model.model
14model.model_index
15model.presence_penalty
16model.temperature
17model.top_logprobs
18model.top_p
19prompt.primary_color_system_prompt
20prompt.primary_color_user_prompt
21question_options.primary_color_question_options
22question_text.primary_color_question_text
23question_type.primary_color_question_type
24raw_model_response.primary_color_cost
25raw_model_response.primary_color_input_price_per_million_tokens
26raw_model_response.primary_color_input_tokens
27raw_model_response.primary_color_one_usd_buys
28raw_model_response.primary_color_output_price_per_million_tokens
29raw_model_response.primary_color_output_tokens
30raw_model_response.primary_color_raw_model_response
31reasoning_summary.primary_color_reasoning_summary
32scenario.scenario_index
We can display it with any other fields: 
r.select("model", "primary_color", "primary_color_comment")
model.modelanswer.primary_colorcomment.primary_color_comment
0gpt-4oRedRed is often considered the most common primary color due to its prominence in nature, its use in art and design, and its psychological impact.

​
Turning off comments

If desired, we can omit the instruction to provide a comment by passing a parameter include_comment=False to the question constructor. This may be desired if comments are not necessary or to save tokens. Here we inspect how the question prompt has been modified and verify that the comment field in the results is blank: 
from edsl import QuestionMultipleChoice, Survey, Model

q = QuestionMultipleChoice(
    question_name = "primary_color",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"],
    include_comment = False # optional
)

q.by(Model()).prompts().select("user_prompt")
user_prompt
0What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options.
r = q.run() # default model will be used

r.select("model", "primary_color", "primary_color_comment")
model.modelanswer.primary_colorcomment.primary_color_comment
0gpt-4oRednan

​
Modifying comments

We can also modify the default instruction if we want to use the comment field in a different way. This can be done by passing an optional parameter answering_instruction to the question constructor. For example, here we pass an instruction that preserves the directions about the format of the answer to a multiple choice question (”Respond only with a string corresponding to one of the options.”) but replace the comments part of the instruction with a new instruction for the model to instead note it’s second choice answer. We include the original question in the survey as well for ease of comparison: 
from edsl import QuestionMultipleChoice, Survey, Model

q1 = QuestionMultipleChoice(
    question_name = "primary_color_v1",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"]
)

q2 = QuestionMultipleChoice(
    question_name = "primary_color_v2",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"],
    answering_instructions = """
    Respond only with a string corresponding to one of the options.
    After the answer, please provide your second choice on the next line.
    """
)

survey = Survey([q1, q2])

survey.by(Model()).prompts().select("user_prompt")
user_prompt
0What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line.
1What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, please provide your second choice on the next line.
r = survey.run() # default model will be used

r.select("model", "primary_color_v1", "primary_color_v1_comment", "primary_color_v2", "primary_color_v2_comment")
model.modelanswer.primary_color_v1comment.primary_color_v1_commentanswer.primary_color_v2comment.primary_color_v2_comment
0gpt-4oRedRed is often considered the most common primary color due to its prominence in nature, its use in art and design, and its psychological impact.RedYellow

​
Further reading

Please see the questions page of the documentation for more examples and details on all of the required and optional parameters for question types! Here we post this notebook to Coop for reference: 
from edsl import Notebook

nb = Notebook(path = "answering_instructions_example.ipynb")

nb.push(
    description = "Example answering instructions",
    alias = "answering-instructions-notebook",
    visibility = "public"
)
⌘I