Google Form -> EDSL

This notebook provides example code for using EDSL to reformat non-EDSL survey questions in EDSL.

Learn more about using EDSL to simulate surveys, experiments and market research with AI agents and large language models.

Designing the task as an EDSL question

We start by creating a question that will prompt a language model to extract questions from the text of a survey form and reformat them as EDSL questions. EDSL comes with many common question types that we can choose from based on the form of the response that we want to get back from the model (multiple choice, checkbox, free text, linear scale, etc.). Learn more about EDSL question types.

Here we use question type QuestionList in order to return the reformatted questions as a list that we can then pass to a Survey. We can use a {{ placeholder }} for the text that we want to add to the question, in order to reuse it with different survey forms:

from edsl import QuestionList

q = QuestionList(
    question_name="reformat_edsl",
    question_text="""
    You are being asked to extract and reformat questions in a survey form.

    Each question should be reformatted as a dictionary that contains:
    - key: "question_type", value: "multiple_choice", "checkbox", "linear_scale", "numerical", or "free_text"
    - key: "question_name", value: a unique Python identifier
    - key: "question_text", value: a string containing the question
    - key: "question_options", value: a list of unique answer option strings (required for all types except "free_text").
    For "linear_scale" type questions:
    - "question_options" must be a list of integers.
    - The dictionary can include an optional key: "option_labels", value: a dictionary integer keys and corresponding labels as values.

    Return the question dictionaries in a list.
    Here is the text of the survey form containing the questions for you to extract and reformat: {{ text }}
    """
)

Adding content to questions

Next we create Scenario objects for the text or other content that we want to add to the question. This allows us to efficiently administer multiple versions of the question at once. Learn more about using scenarios to scale data labeling and other tasks.

Here we create a single scenario for a Google Form seeking feedback on remote inference – features for running EDSL surveys remotely on the Expected Parrot server, as we do in this notebook):

from edsl import Scenario

s_pdf = Scenario.from_pdf("google_form_remote_inference.pdf") # https://forms.gle/iye2i8KSWQ3xKCKs7
s_pdf

{
    "filename": "google_form_remote_inference.pdf",
    "text": "Feedback on remote inference\n\nPlease let us know about your experience using remote inference to run your EDSL surveys!\n\n1.\nEmail *\n\nFirst learned about remote inference\n\n2.\n\nHow did you first learn about the option to use remote inference with EDSL surveys?\u00a0\n\nMark only one oval.\n\nDiscord\nSkip to question 4\n\nX/Twitter\nSkip to question 4\n\nEDSL documentation\nSkip to question 4\n\nCoop notebook/example\nSkip to question 4\n\nNone of these\n\nFirst learned about remote inference\n\n3.\n\nPlease describe how you first learned about the option to use remote inference with\nEDSL surveys.\n\nSurveys run with remote inference\n\n4.\n\nHow many different EDSL surveys have you tried running with remote inference?\n\nMark only one oval.\n\nNone\nSkip to question 6\n\n1-10\n\nMore than 10\n\nOverall experience\n\n5.\n\nOverall, how has your experience with remote inference been?\n\nMark only one oval.\n\nTerrible\n\nOk\n\nGood\n\nGreat\n\nCost estimate\n\n6.\n\nHow important is it to you to get an accurate estimate of the credits required to run\nyour survey?\n\nMark only one oval.\n\n1\n2\n3\n4\n5\n\nNot important\n\nVery important\n\nDocumentation\n\n7.\n\nHave you read or used the documentation page on remote inference?\u00a0\n(https://docs.expectedparrot.com/en/latest/remote_inference.html)\n\nMark only one oval.\n\nYes\n\nNo\nSkip to question 10\n\nDocumentation\n\n8.\n\nHow useful was the documentation page on remote inference?\u00a0\n\nMark only one oval.\n\nNot useful\n\nSomewhat useful\n\nVery useful\n\n9.\n\nHow clear / easy to follow was the documentation page on remote inference?\u00a0\n\nMark only one oval.\n\nNot clear\n\nSomewhat clear\n\nVery clear\n\nImprovements\n\n10.\n\nWhat is confusing or unclear about remote inference?\n\n11.\n\nHow can we make remote inference more useful to you?\n\nThis content is neither created nor endorsed by Google.\n\n\u00a0Forms\n\n"
}

Selecting language models

EDSL works with many popular language models that we can select to use in generating survey responses. You can provide your own API keys for models or activate remote inference to run surveys at the Expected Parrot server with any available models. Learn more about working with language models and using remote inference.

Here we select a model to add to the survey when we run it:

from edsl import ModelList, Model

To see a list of all available models:

# Model.available()

model = Model("claude-3-5-sonnet-20240620")

Running a survey

Next we add the scenario and model to the survey and run. This generates a dataset of Results that we can access with built-in methods for analysis. Learn more about working with results.

results = q.by(s_pdf).by(model).run()

Remote inference started (Job uuid=d296fc7f-bc9d-4542-be17-7614b23796d8).
Job completed and Results stored on Coop (Results uuid=6326186e-d02c-4bf0-bcef-01a648a9ad32).

Viewing results at the Coop

The job and results uuids indicate that remote inference is activated, and are the unique identifiers for inspecting and modifying the objects at the Coop: a new platform for creating, storing and sharing LLM-based research using EDSL. Learn more about using the Coop.

The results can be viewed at: https://www.expectedparrot.com/content/6326186e-d02c-4bf0-bcef-01a648a9ad32

Creating an EDSL survey

Now we select the list of reformatted questions from the results, create Question objects for them, and then pass them to a Survey object:

questions_list = results.select("reformat_edsl").to_list()[0]
questions_list

[{'question_type': 'free_text',
  'question_name': 'email',
  'question_text': 'Email'},
 {'question_type': 'multiple_choice',
  'question_name': 'first_learned_about_remote_inference',
  'question_text': 'How did you first learn about the option to use remote inference with EDSL surveys?',
  'question_options': ['Discord',
   'X/Twitter',
   'EDSL documentation',
   'Coop notebook/example',
   'None of these']},
 {'question_type': 'free_text',
  'question_name': 'describe_first_learned_about_remote_inference',
  'question_text': 'Please describe how you first learned about the option to use remote inference with EDSL surveys.'},
 {'question_type': 'multiple_choice',
  'question_name': 'number_of_surveys_run',
  'question_text': 'How many different EDSL surveys have you tried running with remote inference?',
  'question_options': ['None', '1-10', 'More than 10']},
 {'question_type': 'multiple_choice',
  'question_name': 'overall_experience',
  'question_text': 'Overall, how has your experience with remote inference been?',
  'question_options': ['Terrible', 'Ok', 'Good', 'Great']},
 {'question_type': 'linear_scale',
  'question_name': 'importance_of_cost_estimate',
  'question_text': 'How important is it to you to get an accurate estimate of the credits required to run your survey?',
  'question_options': [1, 2, 3, 4, 5],
  'option_labels': {'1': 'Not important', '5': 'Very important'}},
 {'question_type': 'multiple_choice',
  'question_name': 'read_documentation',
  'question_text': 'Have you read or used the documentation page on remote inference? (https://docs.expectedparrot.com/en/latest/remote_inference.html)',
  'question_options': ['Yes', 'No']},
 {'question_type': 'multiple_choice',
  'question_name': 'documentation_usefulness',
  'question_text': 'How useful was the documentation page on remote inference?',
  'question_options': ['Not useful', 'Somewhat useful', 'Very useful']},
 {'question_type': 'multiple_choice',
  'question_name': 'documentation_clarity',
  'question_text': 'How clear / easy to follow was the documentation page on remote inference?',
  'question_options': ['Not clear', 'Somewhat clear', 'Very clear']},
 {'question_type': 'free_text',
  'question_name': 'confusing_aspects',
  'question_text': 'What is confusing or unclear about remote inference?'},
 {'question_type': 'free_text',
  'question_name': 'improvement_suggestions',
  'question_text': 'How can we make remote inference more useful to you?'}]

Creating EDSL questions:

from edsl import Question

questions = [Question(**d) for d in questions_list]
questions

[Question('free_text', question_name = """email""", question_text = """Email"""),
 Question('multiple_choice', question_name = """first_learned_about_remote_inference""", question_text = """How did you first learn about the option to use remote inference with EDSL surveys?""", question_options = ['Discord', 'X/Twitter', 'EDSL documentation', 'Coop notebook/example', 'None of these']),
 Question('free_text', question_name = """describe_first_learned_about_remote_inference""", question_text = """Please describe how you first learned about the option to use remote inference with EDSL surveys."""),
 Question('multiple_choice', question_name = """number_of_surveys_run""", question_text = """How many different EDSL surveys have you tried running with remote inference?""", question_options = ['None', '1-10', 'More than 10']),
 Question('multiple_choice', question_name = """overall_experience""", question_text = """Overall, how has your experience with remote inference been?""", question_options = ['Terrible', 'Ok', 'Good', 'Great']),
 Question('linear_scale', question_name = """importance_of_cost_estimate""", question_text = """How important is it to you to get an accurate estimate of the credits required to run your survey?""", question_options = [1, 2, 3, 4, 5], option_labels = {1: 'Not important', 5: 'Very important'}),
 Question('multiple_choice', question_name = """read_documentation""", question_text = """Have you read or used the documentation page on remote inference? (https://docs.expectedparrot.com/en/latest/remote_inference.html)""", question_options = ['Yes', 'No']),
 Question('multiple_choice', question_name = """documentation_usefulness""", question_text = """How useful was the documentation page on remote inference?""", question_options = ['Not useful', 'Somewhat useful', 'Very useful']),
 Question('multiple_choice', question_name = """documentation_clarity""", question_text = """How clear / easy to follow was the documentation page on remote inference?""", question_options = ['Not clear', 'Somewhat clear', 'Very clear']),
 Question('free_text', question_name = """confusing_aspects""", question_text = """What is confusing or unclear about remote inference?"""),
 Question('free_text', question_name = """improvement_suggestions""", question_text = """How can we make remote inference more useful to you?""")]

Passing the questions to a new survey:

from edsl import Survey

survey = Survey(questions)
# survey

Survey rules & logic

We can add skip/stop or other logic as desired (even if not allowed in the original survey – e.g., Google Forms allows skip rules for multiple choice questions only). Learn about adding rules to EDSL surveys.

survey = (survey
          .add_skip_rule("overall_experience", "surveys_run_with_remote_inference == 'No'")
          .add_skip_rule("documentation_usefulness", "seen_documentation == 'No'")
          .add_skip_rule("documentation_clarity", "seen_documentation == 'No'")
         )

Posting content to the Coop

The results of our reformat_edsl question were automatically posted to the Coop using remote inference. We can post the new survey to the Coop as well:

survey.push(description = "Google Form reformatted in EDSL: Feedback on remote inference", visibility = "public")

{'description': 'Google Form reformatted in EDSL: Feedback on remote inference',
 'object_type': 'survey',
 'url': 'https://www.expectedparrot.com/content/caef1dcf-1f85-401f-8e16-bda040185e3b',
 'uuid': 'caef1dcf-1f85-401f-8e16-bda040185e3b',
 'version': '0.1.33.dev1',
 'visibility': 'public'}

We can also post this notebook:

from edsl import Notebook

n = Notebook(path = "google_form_to_edsl.ipynb")

n.push(description = "Example code for reformatting Google Form questions in EDSL", visibility = "public")

{'description': 'Example code for reformatting Google Form questions in EDSL',
 'object_type': 'notebook',
 'url': 'https://www.expectedparrot.com/content/ffc53ff9-dad3-4c9f-aea7-68576247a923',
 'uuid': 'ffc53ff9-dad3-4c9f-aea7-68576247a923',
 'version': '0.1.33.dev1',
 'visibility': 'public'}