Using images in a survey
This notebook provides sample code for using images with an EDSL survey.
EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL and Coop.
Scenarios
A Scenario
is a dictionary containing a key/value pair that is used to add data or content to questions in an EDSL survey. Scenarios allow you create variations and versions of questions efficiently, and with data or content from different sources.
EDSL provides a variety of methods for automatically generating scenarios from PDFs, CSVs, docs, tables, lists, dicts – and images. In the steps below we demonstrate how to create a scenario for an image and use it in a survey.
Note: When using images with questions it is necessary to specify a vision model, and to ensure that the model is capable of viewing each image. Always run test questions to ensure that each image is actually readable by the selected models.
Learn more about working with scenarios.
Creating a scenario
We start by creating a Scenario
for an image. For purposes of demonstration, we use the FileStore
module to post a PNG image to the Coop, and then retrieve it and pass it to a Scenario
(this can be done by any user with a Coop account). Note that FileStore
can be used to post and retrieve all types of files, and will automatically infer the file type.
Here we post a file to the Coop:
[1]:
from edsl import FileStore
filename = "parrot_logo.png" # file stored locally
fs = FileStore(filename)
info = fs.push()
info
[1]:
{'description': 'File: parrot_logo.png',
'object_type': 'scenario',
'url': 'https://www.expectedparrot.com/content/dc2e3ede-54bb-4470-a51e-927d45d91293',
'uuid': 'dc2e3ede-54bb-4470-a51e-927d45d91293',
'version': '0.1.42.dev1',
'visibility': 'unlisted'}
Here we retrieve the file (can be replaced with the UUID of any posted object):
[2]:
png_file = FileStore.pull(info["uuid"])
Here we use the retrieved file in a Scenario
by creating a key and passing the file as the value. We also (optionally) create a key/value for metadata about the file that we want to keep with the survey results (more on this below):
[3]:
from edsl import Scenario
[4]:
s = Scenario({
"parrot_logo":png_file,
"filename":filename
})
Creating questions using the image
Next we construct questions with the image scenario. Note that we use a {{ placeholder }}
for the scenario key for the image file. This will cause the image to be automatically be inserted when the survey is run with the scenario. We also pipe the answer to one question into a follow-on question:
[5]:
from edsl import QuestionYesNo, QuestionMultipleChoice, QuestionList, Survey
[6]:
q1 = QuestionYesNo(
question_name = "animal",
question_text = "Is there an animal in this image? {{ parrot_logo }}"
)
[7]:
q2 = QuestionMultipleChoice(
question_name = "identify",
question_text = "Identify the animal in this image: {{ parrot_logo }}",
question_options = ["dog", "cat", "bird", "something else"]
)
[8]:
q3 = QuestionList(
question_name = "colors",
question_text = "What color(s) is this {{ identify.answer }}? {{ parrot_logo }}",
)
[9]:
survey = Survey(questions = [q1, q2, q3])
Next we add a rule to stop the survey if the answer to the first question is “No”. This rule and the piping in the questions that follow will cause the questions to be administered in the required order, instead of asynchronously by default (learn more about piping and applygin survey rules):
[10]:
survey = (
survey
.add_stop_rule(q1, "animal == 'No'")
)
Next we select a model to generate the responses. Note that we need to use a vision model. You can check available vision models at the Coop model pricing page.
[11]:
from edsl import Model
m = Model("gemini-1.5-flash")
We administer the survey in the same way that we do with any other scenarios:
[12]:
results = survey.by(s).by(m).run()
Job UUID | 0fd3ad6e-b01a-449d-ac7c-f90c557cecd3 |
Progress Bar URL | https://www.expectedparrot.com/home/remote-job-progress/0fd3ad6e-b01a-449d-ac7c-f90c557cecd3 |
Exceptions Report URL | None |
Results UUID | e1a2d0c9-1aff-47f7-be63-5d0cfd47b633 |
Results URL | https://www.expectedparrot.com/content/e1a2d0c9-1aff-47f7-be63-5d0cfd47b633 |
We can select any scenario key/value to access in the results that have been generated (e.g., image metadata created):
[13]:
results.select("model", "filename", "animal", "identify", "colors", "colors_comment")
[13]:
model.model | scenario.filename | answer.animal | answer.identify | answer.colors | comment.colors_comment | |
---|---|---|---|---|---|---|
0 | gemini-1.5-flash | parrot_logo.png | Yes | bird | ['green', 'orange', 'red', 'blue'] | The parrot is primarily green, with orange on its beak, red and blue on its underside. |
Posting to Coop
The results of the survey were automatically posted to Coop using remote inference (see link in the job summary above). Here we also post this notebook, as we can any local objects that we want to push to Coop:
[14]:
from edsl import Notebook
n = Notebook(path = "image_scenario_example.ipynb")
[16]:
n.push(description = "Using an image scenario", visibility = "public")
[16]:
{'description': 'Using an image scenario',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/e82e61b1-8563-42d3-af3a-e3eae2b27804',
'uuid': 'e82e61b1-8563-42d3-af3a-e3eae2b27804',
'version': '0.1.39.dev2',
'visibility': 'public'}
To update an object at Coop:
To update an object at Coop:
[15]:
n.patch(uuid = "b72f3990-0630-4aa7-99c8-2230d51376d1", value = n)
[15]:
{'status': 'success'}