Survey logic checklist

This page provides a checklist for reviewing the logic and contents of an EDSL survey and ensuring that it will run as you intend it to.

Support

Please reach out to us with any questions or issues! Send an email to info@expectedparrot.com or post a message at Discord.

How EDSL works

At its core, EDSL is built on the concept of Questions being answered by AI Agents, using Language Models to generate responses that are returned as formatted datasets of Results.

A typical workflow consists of the following steps:

  1. Construct questions of different types (multiple choice, free text, etc.) and combine them in a survey. Learn about question types.

  2. Optional: Add data or content to your questions (from CSVs, PDFs, docs, images, etc.). Learn about scenarios.

  3. Optional: Add rules or logic to specify how questions should be presented (e.g., skip/stop rules, or including context of other questions). Learn about survey rules.

  4. Optional: Design or import personas for AI agents to answer the questions. Learn about agents.

  5. Select language models to generate responses. EDSL works with hundreds of popular models.

  6. Run the survey and get a formatted dataset of results. Use built-in methods for analyzing them.

  7. Optional: Store and share your work at the Coop: a platform for AI research that is fully integrated with EDSL. Learn more.

Issues checklist

Below are some issues to consider before running a survey, with links to examples of EDSL features for handling them.

Are you using appropriate question types?

Ensure that the question type you are using is appropriate for the question you are presenting.

  • Use QuestionYesNo when you only want to allow “Yes” or “No” responses.

  • Use QuestionMultipleChoice when you want to require that only one option be selected.

  • Use QuestionCheckBox when you want to allow multiple (or no) selections or specify a number of selections.

Are question options correct and complete?

Ensure that the question options make sense for the question text.

  • Does the question text ask for options not presented, or a different number of options?

  • Should the question options include an option for non-responses (e.g., “I do not know.”)?

Do any questions require context of other questions?

Survey questions are administered asynchronously by default, meaning that the presentation of one question does not include context of any other questions in the survey unless you specify otherwise. This default functionality reduces costs and runtime. It also allows you to fine-tune and readily compare responses to versions of questions with different contexts.

If a question depends on or requires information about other questions in a survey, you need to add a rule specifying the logic to be applied. This can be done in a variety of ways:

  • Use piping to add components of a question in another question (e.g., insert the answer to a question in the text of a follow-up question).

  • Use question memory to include the entire context of a question/answer in the presentation of a different question (“You were previously asked…”). There are separate rules for adding the context of a single question, a set of questions, all prior questions, or a lagged number of questions.

Testing a model’s ability to answer a question

Running test questions and examining the answers and “comment” fields can help you understand whether a question needs to be improved. You can also run questions about your questions for cognitive testing (e.g., “How is the following question confusing: {{ question }}”)

Tools for integrating data

Scenario objects

EDSL provides many methods for importing and formatting data or content to be used with a survey. This can be particularly useful for surveys that involve data labeling or data cleaning tasks, such as using questions to extract information from unstructured texts or apply labels to images or other content. This is done by creating Scenerio <https://docs.expectedparrot.com/en/latest/scenarios.html>`_ objects representing data or content to be added to survey questions (e.g., “What is the main topic of this text? {{ text }}?”, “What is in this image? {{ image }}”). See examples for working with PDFs, CSVs, images, docs, tables and other data types. You can also use scenarios to store metadata for surveys.

Converting questions

Many models are highly capable of reformatting non-EDSL surveys based on a prompt that includes basic instructions or examples of EDSL code. This can be an efficient way to reconstruct an existing survey in EDSL and then modify as needed with above-mentioned methods. See examples: CES data; GoogleForms.