Answering instructions

This notebook demonstrates how you can modify default instructions for question types and answer comments.

Default question instructions

Each EDSL question type includes default instructions to the model about how to answer the question. We can view these instructions by inspecting the user prompt for a question that has been created (the other type of prompt–systen prompt–is for agent instructions).

For example, here we see that the default instruction for multiple choice questions is:

“Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line.”

This text is automatically appended to the question text:

[1]:

from edsl import QuestionMultipleChoice, Survey, Model

q = QuestionMultipleChoice(
    question_name = "primary_color",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"]
)

survey = Survey([q])
survey.show_prompts()

[1]:

	user_prompt	system_prompt	interview_index	question_name	scenario_index	agent_index	model	estimated_cost	cache_keys
0	What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line.	nan	0	primary_color	0	0	gpt-4o	0.000678	['20e75009c72f3e88c490c58bf13d6a72']

We can isolate the user prompt:

[2]:

survey.by(Model()).prompts().select("user_prompt")

[2]:

	user_prompt
0	What is the most common primary color? Red Yellow Blue Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line.

We can compare this with default instructions for other question types:

[3]:

from edsl import QuestionCheckBox, Survey, Model

q = QuestionCheckBox(
    question_name = "primary_colors",
    question_text = "Which colors are 'primary'?",
    question_options = ["Red", "Orange", "Yellow", "Green", "Blue", "Purple"]
)

survey = Survey([q])
survey.by(Model()).prompts().select("user_prompt")

[3]:

	user_prompt
0	Which colors are 'primary'? Red Orange Yellow Green Blue Purple Please respond only with a comma-separated list of the options that apply, with square brackets. E.g., ['Good', 'Bad', 'Ugly'] After the answer, you can put a comment explaining your choice on the next line.

[4]:

from edsl import QuestionRank, Survey, Model

q = QuestionRank(
    question_name = "primary_colors_rank",
    question_text = "Rank the primary colors in terms of popularity.",
    question_options = ["Red", "Yellow", "Blue"]
)

survey = Survey([q])
survey.by(Model()).prompts().select("user_prompt")

[4]:

	user_prompt
0	Rank the primary colors in terms of popularity. The options are 0: Red 1: Yellow 2: Blue You have to include 3 options in your answer. Please respond only with a comma-separated list of the code of the raked options, with square brackets. E.g., [0, 1, 3] After the answer, you can put a comment explaining your choice on the next line.

[5]:

from edsl import QuestionLinearScale, Survey, Model

q = QuestionLinearScale(
    question_name = "primary_color_scale",
    question_text = "Most people know what the primary colors are.",
    question_options = [1,2,3,4,5],
    option_labels = {
        1:"This statement is completely inaccurate",
        5:"This statement is completely accurate."
    }
)

survey = Survey([q])
survey.by(Model()).prompts().select("user_prompt")

[5]:

	user_prompt
0	Most people know what the primary colors are. 1 : This statement is completely inaccurate 2 : 3 : 4 : 5 : This statement is completely accurate. Only 1 option may be selected. Respond only with the code corresponding to one of the options. E.g., "1" or "5" by itself. After the answer, you can put a comment explaining why you chose that option on the next line.

Formatting answers & comments

We can see that each default instruction includes directions on (1) formatting the answer and (2) providing a comment about the answer. When a question is administered, the contents of the comment that is returned are automatically stored in a separate field of the results. We can see this when we run a question and inspect the columns of the results that have been created. Here we run the multiple choice question created above:

[6]:

from edsl import QuestionMultipleChoice, Survey, Model

q = QuestionMultipleChoice(
    question_name = "primary_color",
    question_text = "What is the most common primary color?",
    question_options = ["Red", "Yellow", "Blue"]
)

r = q.run() # default model will be used

⌃ Job Status 🦜

Completed (1 completed, 0 failed)

Job Links

Results

Progress Report

Content

Remote Jobs

Remote Cache

Identifiers

Results UUID:

3c23c10e...c24e

Use Results.pull(uuid) to fetch results.

Job UUID:

cad4a50f...21fb

Use Jobs.pull(uuid) to fetch job.