Using images in a survey

This notebook provides sample code for using images with an EDSL survey.

EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL and Coop.

Scenarios

A Scenario is a dictionary containing a key/value pair that is used to add data or content to questions in an EDSL survey. Scenarios allow you create variations and versions of questions efficiently, and with data or content from different sources.

EDSL provides a variety of methods for automatically generating scenarios from PDFs, CSVs, docs, tables, lists, dicts – and images. In the steps below we demonstrate how to create a scenario for an image and use it in a survey.

Note: When using images with questions it is necessary to specify a vision model, and to ensure that the model is capable of viewing each image. Always run test questions to ensure that each image is actually readable by the selected models.

Learn more about working with scenarios.

Creating a scenario

We start by creating a Scenario for an image. For purposes of demonstration, we use the FileStore module to post a PNG image to the Coop, and then retrieve it and pass it to a Scenario (this can be done by any user with a Coop account). Note that FileStore can be used to post and retrieve all types of files, and will automatically infer the file type.

Here we post a file to the Coop:

from edsl import FileStore

filename = "parrot_logo.png" # file stored locally

fs = FileStore(filename)
info = fs.push()
info

{'description': 'File: parrot_logo.png',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/c08e9ec6-cb24-49fc-bb88-dc77e3edde14',
 'uuid': 'c08e9ec6-cb24-49fc-bb88-dc77e3edde14',
 'version': '0.1.43.dev1',
 'visibility': 'unlisted'}

Here we retrieve the file (can be replaced with the UUID of any posted object):

png_file = FileStore.pull(info["uuid"])

Here we use the retrieved file in a Scenario by creating a key and passing the file as the value. We also (optionally) create a key/value for metadata about the file that we want to keep with the survey results (more on this below):

from edsl import Scenario

s = Scenario({
    "parrot_logo":png_file,
    "filename":filename
})

Creating questions using the image

Next we construct questions with the image scenario. Note that we use a {{ placeholder }} for the scenario key for the image file. This will cause the image to be automatically be inserted when the survey is run with the scenario. We also pipe the answer to one question into a follow-on question:

from edsl import QuestionYesNo, QuestionMultipleChoice, QuestionList, Survey

q1 = QuestionYesNo(
    question_name = "animal",
    question_text = "Is there an animal in this image? {{ parrot_logo }}"
)

q2 = QuestionMultipleChoice(
    question_name = "identify",
    question_text = "Identify the animal in this image: {{ parrot_logo }}",
    question_options = ["dog", "cat", "bird", "something else"]
)

q3 = QuestionList(
    question_name = "colors",
    question_text = "What color(s) is this {{ identify.answer }}? {{ parrot_logo }}",
)

survey = Survey(questions = [q1, q2, q3])

Next we add a rule to stop the survey if the answer to the first question is “No”. This rule and the piping in the questions that follow will cause the questions to be administered in the required order, instead of asynchronously by default (learn more about piping and applygin survey rules):

survey = (
    survey
    .add_stop_rule(q1, "animal == 'No'")
)

Next we select a model to generate the responses. Note that we need to use a vision model. You can check available vision models at the Coop model pricing page.

from edsl import Model

m = Model("gemini-1.5-flash")

We administer the survey in the same way that we do with any other scenarios:

results = survey.by(s).by(m).run()

▼ Job Status (2025-02-07 19:25:34)

Job UUID	c2d11906-e903-43b5-ac18-c4a990955481
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/c2d11906-e903-43b5-ac18-c4a990955481
Exceptions Report URL	None
Results UUID	e714554d-d9f9-48ca-8e24-efc4b7dbe5cd
Results URL	https://www.expectedparrot.com/content/e714554d-d9f9-48ca-8e24-efc4b7dbe5cd

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/e714554d-d9f9-48ca-8e24-efc4b7dbe5cd

We can select any scenario key/value to access in the results that have been generated (e.g., image metadata created):

results.select("model", "filename", "animal", "identify", "colors", "colors_comment")

	model.model	scenario.filename	answer.animal	answer.identify	answer.colors	comment.colors_comment
0	gemini-1.5-flash	parrot_logo.png	Yes	bird	['green', 'orange', 'red', 'blue']	The parrot is primarily green, with orange on its beak, red and blue on its lower body.

Posting to Coop

The results of the survey were automatically posted to Coop using remote inference (see link in the job summary above). Here we also post this notebook, as we can any local objects that we want to push to Coop:

from edsl import Notebook

n = Notebook(path = "image_scenario_example.ipynb")

info = n.push(description = "Using an image scenario", visibility = "public")

info

{'description': 'Using an image scenario',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/14167a50-ff40-446c-a879-ceb1c70e7aa1',
 'uuid': '14167a50-ff40-446c-a879-ceb1c70e7aa1',
 'version': '0.1.43.dev1',
 'visibility': 'public'}

To update an object at Coop:

from edsl import Notebook

n = Notebook(path = "image_scenario_example.ipynb")

n.patch(uuid = info["uuid"], value = n)

{'status': 'success'}