Using images in a survey

This notebook provides sample code for using images with an EDSL survey.

EDSL is an open-source library for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL and Coop.

Scenarios

A Scenario is a dictionary containing a key/value pair that is used to add data or content to questions in an EDSL survey. Scenarios allow you create variations and versions of questions efficiently, and with data or content from different sources.

EDSL provides a variety of methods for automatically generating scenarios from PDFs, CSVs, docs, tables, lists, dicts – and images. In the steps below we demonstrate how to create a scenario for an image and use it in a survey.

Note: When using images with questions it is necessary to specify a vision model, and to ensure that the model is capable of viewing each image. Always run test questions to ensure that each image is actually readable by the selected models.

Learn more about working with scenarios.

Creating a scenario

We start by creating a Scenario for an image. For purposes of demonstration, we use the FileStore module to post a PNG image to the Coop, and then retrieve it and pass it to a Scenario (this can be done by any user with a Coop account). Note that FileStore can be used to post and retrieve all types of files, and will automatically infer the file type.

Here we post a file to the Coop:

from edsl import FileStore

fs = FileStore("parrot_logo.png") # replace with your own file
info = fs.push()
info

{'description': 'File: parrot_logo.png',
 'object_type': 'scenario',
 'url': 'https://www.expectedparrot.com/content/04b7fa81-7d57-4431-9bfd-ee853fa319ee',
 'uuid': '04b7fa81-7d57-4431-9bfd-ee853fa319ee',
 'version': '0.1.39.dev2',
 'visibility': 'unlisted'}

Here we retrieve the file (can be replaced with the UUID of any posted object):

png_file = FileStore.pull(info["uuid"])

Here we use the retrieved file in a Scenario by creating a key and passing the file as the value:

from edsl import Scenario

s = Scenario({"parrot_logo":png_file})

Creating questions using the image

Next we construct questions with the image scenario. Note that we use a {{ placeholder }} for the scenario key. We also demonstrate how to pipe an answer into a follow-on question:

from edsl import QuestionYesNo, QuestionFreeText, QuestionList, Survey

q1 = QuestionYesNo(
    question_name = "animal",
    question_text = "Is there an animal in this image? {{ parrot_logo }}"
)

q2 = QuestionFreeText(
    question_name = "identify",
    question_text = "Identify the animal in this image: {{ parrot_logo }}"
)

q3 = QuestionList(
    question_name = "colors",
    question_text = "What color(s) is the animal?",
)

survey = Survey(questions = [q1, q2, q3])

Adding logic and rules for adminitering the questions:

survey = (
    survey
    .add_stop_rule(q1, "animal == 'No'")
    .add_targeted_memory(q3, q2)
)

We use the scenario by adding it to the survey when we run it, as with any other scenarios. Note that we need to use a vision model with the questions.

from edsl import Model

m = Model("gemini-1.5-flash")

results = survey.by(s).by(m).run()

▼ Job Status (2024-12-26 21:13:20)

Job UUID	478aefcc-1ef5-41d9-9c07-f93c1178e46d
Progress Bar URL	https://www.expectedparrot.com/home/remote-job-progress/478aefcc-1ef5-41d9-9c07-f93c1178e46d
Error Report URL	None
Results UUID	a00b5262-6173-457a-b79a-d9397152d9ae
Results URL	https://www.expectedparrot.com/content/a00b5262-6173-457a-b79a-d9397152d9ae

✓Current Status: Job completed and Results stored on Coop: https://www.expectedparrot.com/content/a00b5262-6173-457a-b79a-d9397152d9ae

results.select("model", "animal", "identify", "colors", "colors_comment")

	model.model	answer.animal	answer.identify	answer.colors	comment.colors_comment
0	gemini-1.5-flash	Yes	That's a parrot. More specifically, the stylized image appears to be based on a species of parrot, but it's not precisely identifiable to a specific species from this simplified illustration.	['Green', 'Yellow', 'Blue']	The parrot in the image is primarily green, with yellow and blue accents.

from edsl import Notebook

n = Notebook(path = "image_scenario_example.ipynb")

n.push(description = "Using an image scenario", visibility = "public")

{'description': 'Using an image scenario',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/e82e61b1-8563-42d3-af3a-e3eae2b27804',
 'uuid': 'e82e61b1-8563-42d3-af3a-e3eae2b27804',
 'version': '0.1.39.dev2',
 'visibility': 'public'}

To update an object at Coop:

n.patch(uuid = "b72f3990-0630-4aa7-99c8-2230d51376d1", value = n)

{'status': 'success'}