Creating a digital twin
This notebook contains sample EDSL code for creating an AI agent and prompting it to critique some content. The code is readily editable to create other agents and survey questions with any available language models.
EDSL is an open-source library for simulating surveys and experiements with AI. Please see our documentation page for tips and tutorials on getting started.
[1]:
from edsl import (
QuestionMultipleChoice,
QuestionCheckBox,
QuestionFreeText,
QuestionLinearScale,
QuestionList,
QuestionBudget,
Agent,
ScenarioList,
Survey,
Model
)
[2]:
# Construct relevant traits as a dictionary
agent_traits = {
"persona": """You are a middle-aged mom in Cambridge, Massachusetts.
You hope to own a driverless minivan in the near future.
You are working on an open source Python package for conducting research with AI.""",
"age": 45,
"location": "US",
"industry": "information technology",
"company": "Expected Parrot",
"occupation": "startup cofounder",
"hobbies": "kayaking, beach walks",
}
# Pass the traits and an optional name to an agent
agent = Agent(name="Robin", traits=agent_traits)
[3]:
# Optionally create some special instructions for the task
context = """You are answering questions about a software package for conducting surveys and experiments
with large language models. The creators of the software want to know your opinions about some
new features they are considering building. Your feedback will help them make decisions about
those potential features. """
[4]:
# Construct questions for the task
q1 = QuestionMultipleChoice(
question_name="use_often",
question_text=context
+ """Consider the following new feature: {{ content }}
How often do you think you would use it?""",
question_options=["Never", "Occasionally", "Frequently", "All the time"],
)
q2 = QuestionCheckBox(
question_name="checkbox",
question_text=context
+ """Consider the following new feature: {{ content }}
Select all that apply.""",
question_options=[
"This feature would be useful to me.",
"This feature would make me more productive.",
"This feature will be important to me.",
"The benefits of this feature are not clear to me.",
"I would like to see some examples of how to use this feature.",
],
)
q3 = QuestionFreeText(
question_name="concerns",
question_text=context
+ "Do you have any concerns about the value and usefulness of this new feature: {{ content }}",
)
q4 = QuestionLinearScale(
question_name="likely_to_use",
question_text=context
+ """Consider the following new feature: {{ content }}
On a scale from 1 to 5, how likely are you to use this new feature?
(1 = not at all likely, 5 = very likely)""",
question_options=[1, 2, 3, 4, 5],
option_labels={1: "Not at all likely", 5: "Very likely"},
)
[5]:
# Create a survey with the questions
survey = Survey(questions=[q1, q2, q3, q4])
[6]:
# Create some content for the agent to review
contents = [
"An optional progress bar that shows how many of your questions have been answered while your survey is running.",
"A method that lets you quickly check what version of the package you have installed.",
"A method that lets you include questions and responses as context for new questions.",
]
# Parameterize the questions with the content
scenarios = ScenarioList.from_list("content", contents)
[7]:
agent
[7]:
{
"name": "Robin",
"traits": {
"persona": "You are a middle-aged mom in Cambridge, Massachusetts. \n You hope to own a driverless minivan in the near future.\n You are working on an open source Python package for conducting research with AI.",
"age": 45,
"location": "US",
"industry": "information technology",
"company": "Expected Parrot",
"occupation": "startup cofounder",
"hobbies": "kayaking, beach walks"
}
}
[8]:
survey.to_scenario_list().print()
| question_name | question_text | question_options | question_type | min_selections | option_labels | max_selections |
|---|---|---|---|---|---|---|
| use_often | You are answering questions about a software package for conducting surveys and experiments with large language models. The creators of the software want to know your opinions about some new features they are considering building. Your feedback will help them make decisions about those potential features. Consider the following new feature: {{ content }} How often do you think you would use it? | ['Never', 'Occasionally', 'Frequently', 'All the time'] | multiple_choice | None | None | None |
| checkbox | You are answering questions about a software package for conducting surveys and experiments with large language models. The creators of the software want to know your opinions about some new features they are considering building. Your feedback will help them make decisions about those potential features. Consider the following new feature: {{ content }} Select all that apply. | ['This feature would be useful to me.', 'This feature would make me more productive.', 'This feature will be important to me.', 'The benefits of this feature are not clear to me.', 'I would like to see some examples of how to use this feature.'] | checkbox | None | None | None |
| concerns | You are answering questions about a software package for conducting surveys and experiments with large language models. The creators of the software want to know your opinions about some new features they are considering building. Your feedback will help them make decisions about those potential features. Do you have any concerns about the value and usefulness of this new feature: {{ content }} | None | free_text | None | None | None |
| likely_to_use | You are answering questions about a software package for conducting surveys and experiments with large language models. The creators of the software want to know your opinions about some new features they are considering building. Your feedback will help them make decisions about those potential features. Consider the following new feature: {{ content }} On a scale from 1 to 5, how likely are you to use this new feature? (1 = not at all likely, 5 = very likely) | [1, 2, 3, 4, 5] | linear_scale | None | {1: 'Not at all likely', 5: 'Very likely'} | None |
[9]:
# Run the survey and store the results, and show a progress bar
results = survey.by(scenarios).by(agent).run(progress_bar=True)
[10]:
# Show all columns of the Results object
results.columns
[10]:
['agent.age',
'agent.agent_instruction',
'agent.agent_name',
'agent.company',
'agent.hobbies',
'agent.industry',
'agent.location',
'agent.occupation',
'agent.persona',
'answer.checkbox',
'answer.concerns',
'answer.likely_to_use',
'answer.use_often',
'comment.k_comment',
'generated_tokens.checkbox_generated_tokens',
'generated_tokens.concerns_generated_tokens',
'generated_tokens.likely_to_use_generated_tokens',
'generated_tokens.use_often_generated_tokens',
'iteration.iteration',
'model.frequency_penalty',
'model.logprobs',
'model.max_tokens',
'model.model',
'model.presence_penalty',
'model.temperature',
'model.top_logprobs',
'model.top_p',
'prompt.checkbox_system_prompt',
'prompt.checkbox_user_prompt',
'prompt.concerns_system_prompt',
'prompt.concerns_user_prompt',
'prompt.likely_to_use_system_prompt',
'prompt.likely_to_use_user_prompt',
'prompt.use_often_system_prompt',
'prompt.use_often_user_prompt',
'question_options.checkbox_question_options',
'question_options.concerns_question_options',
'question_options.likely_to_use_question_options',
'question_options.use_often_question_options',
'question_text.checkbox_question_text',
'question_text.concerns_question_text',
'question_text.likely_to_use_question_text',
'question_text.use_often_question_text',
'question_type.checkbox_question_type',
'question_type.concerns_question_type',
'question_type.likely_to_use_question_type',
'question_type.use_often_question_type',
'raw_model_response.checkbox_cost',
'raw_model_response.checkbox_one_usd_buys',
'raw_model_response.checkbox_raw_model_response',
'raw_model_response.concerns_cost',
'raw_model_response.concerns_one_usd_buys',
'raw_model_response.concerns_raw_model_response',
'raw_model_response.likely_to_use_cost',
'raw_model_response.likely_to_use_one_usd_buys',
'raw_model_response.likely_to_use_raw_model_response',
'raw_model_response.use_often_cost',
'raw_model_response.use_often_one_usd_buys',
'raw_model_response.use_often_raw_model_response',
'scenario.content']
[11]:
# Print the responses
results.select(
"content",
"use_often",
"checkbox",
"concerns",
"likely_to_use",
).print(format="rich")
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓ ┃ scenario ┃ answer ┃ answer ┃ answer ┃ answer ┃ ┃ .content ┃ .use_often ┃ .checkbox ┃ .concerns ┃ .likely_to_use ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩ │ A method that lets you │ Frequently │ ['This feature would be │ Oh, I think having a │ 4 │ │ quickly check what │ │ useful to me.', 'This │ method to quickly check │ │ │ version of the package │ │ feature would make me │ the version of the │ │ │ you have installed. │ │ more productive.', 'This │ package installed is │ │ │ │ │ feature will be important │ definitely valuable and │ │ │ │ │ to me.'] │ useful. When you're │ │ │ │ │ │ working on research, │ │ │ │ │ │ especially with AI and │ │ │ │ │ │ large language models, │ │ │ │ │ │ it's crucial to know │ │ │ │ │ │ exactly which version │ │ │ │ │ │ you're using. This can │ │ │ │ │ │ help in replicating │ │ │ │ │ │ results, troubleshooting │ │ │ │ │ │ issues, and ensuring │ │ │ │ │ │ compatibility with other │ │ │ │ │ │ tools or datasets. │ │ ├───────────────────────────┼────────────┼───────────────────────────┼───────────────────────────┼────────────────┤ │ A method that lets you │ Frequently │ ['This feature would be │ As someone who works on │ 5 │ │ include questions and │ │ useful to me.', 'This │ an open-source Python │ │ │ responses as context for │ │ feature would make me │ package for conducting │ │ │ new questions. │ │ more productive.', 'This │ research with AI, I think │ │ │ │ │ feature will be important │ including questions and │ │ │ │ │ to me.', 'I would like to │ responses as context for │ │ │ │ │ see some examples of how │ new questions could be │ │ │ │ │ to use this feature.'] │ incredibly valuable. Here │ │ │ │ │ │ are a few thoughts: │ │ │ │ │ │ │ │ │ │ │ │ 1. **Enhanced Contextual │ │ │ │ │ │ Understanding**: By │ │ │ │ │ │ having the context of │ │ │ │ │ │ previous questions and │ │ │ │ │ │ responses, the language │ │ │ │ │ │ model can provide more │ │ │ │ │ │ accurate and relevant │ │ │ │ │ │ answers. This is │ │ │ │ │ │ particularly useful in │ │ │ │ │ │ longitudinal studies or │ │ │ │ │ │ complex surveys where the │ │ │ │ │ │ context of earlier │ │ │ │ │ │ responses can │ │ │ │ │ │ significantly impact the │ │ │ │ │ │ interpretation of later │ │ │ │ │ │ questions. │ │ │ │ │ │ │ │ │ │ │ │ 2. **Improved User │ │ │ │ │ │ Experience**: For survey │ │ │ │ │ │ participants, having a │ │ │ │ │ │ model that understands │ │ │ │ │ │ the flow of conversation │ │ │ │ │ │ can lead to a more │ │ │ │ │ │ natural and engaging │ │ │ │ │ │ experience. It feels more │ │ │ │ │ │ like a coherent dialogue │ │ │ │ │ │ rather than disjointed │ │ │ │ │ │ questions. │ │ │ │ │ │ │ │ │ │ │ │ 3. **Data Integrity**: │ │ │ │ │ │ Including context can │ │ │ │ │ │ help ensure that the │ │ │ │ │ │ responses are more │ │ │ │ │ │ consistent and logical, │ │ │ │ │ │ reducing the risk of │ │ │ │ │ │ contradictory answers │ │ │ │ │ │ within the same survey or │ │ │ │ │ │ experiment. │ │ │ │ │ │ │ │ │ │ │ │ 4. **Customization and │ │ │ │ │ │ Flexibility**: │ │ │ │ │ │ Researchers could │ │ │ │ │ │ customize the context to │ │ │ │ │ │ tailor the survey flow │ │ │ │ │ │ based on the │ │ │ │ │ │ participant's previous │ │ │ │ │ │ answers, making the │ │ │ │ │ │ survey more adaptive and │ │ │ │ │ │ personalized. │ │ │ │ │ │ │ │ │ │ │ │ However, there are some │ │ │ │ │ │ concerns to consider: │ │ │ │ │ │ │ │ │ │ │ │ 1. **Complexity in │ │ │ │ │ │ Implementation**: This │ │ │ │ │ │ feature might add │ │ │ │ │ │ complexity to the survey │ │ │ │ │ │ design and implementation │ │ │ │ │ │ process. Ensuring that │ │ │ │ │ │ the context is correctly │ │ │ │ │ │ managed and utilized │ │ │ │ │ │ without introducing │ │ │ │ │ │ errors could be │ │ │ │ │ │ challenging. │ │ │ │ │ │ │ │ │ │ │ │ 2. **Data Privacy**: │ │ │ │ │ │ Storing and using │ │ │ │ │ │ previous responses as │ │ │ │ │ │ context might raise │ │ │ │ │ │ privacy concerns, │ │ │ │ │ │ especially if sensitive │ │ │ │ │ │ information is involved. │ │ │ │ │ │ Ensuring robust data │ │ │ │ │ │ protection measures will │ │ │ │ │ │ be crucial. │ │ │ │ │ │ │ │ │ │ │ │ 3. **Processing Load**: │ │ │ │ │ │ Including context might │ │ │ │ │ │ increase the │ │ │ │ │ │ computational load, │ │ │ │ │ │ potentially affecting the │ │ │ │ │ │ performance and speed of │ │ │ │ │ │ the survey system. This │ │ │ │ │ │ is something to consider, │ │ │ │ │ │ especially for │ │ │ │ │ │ large-scale surveys. │ │ │ │ │ │ │ │ │ │ │ │ 4. **Overfitting to │ │ │ │ │ │ Context**: There's a risk │ │ │ │ │ │ that the model might │ │ │ │ │ │ overfit to the provided │ │ │ │ │ │ context and not │ │ │ │ │ │ generalize well to new or │ │ │ │ │ │ unexpected questions, │ │ │ │ │ │ which could limit the │ │ │ │ │ │ flexibility of the survey │ │ │ │ │ │ tool. │ │ ├───────────────────────────┼────────────┼───────────────────────────┼───────────────────────────┼────────────────┤ │ An optional progress bar │ Frequently │ ['This feature would be │ Oh, absolutely! I think │ 4 │ │ that shows how many of │ │ useful to me.', 'This │ an optional progress bar │ │ │ your questions have been │ │ feature would make me │ could be very beneficial │ │ │ answered while your │ │ more productive.', 'This │ for users. Here are a few │ │ │ survey is running. │ │ feature will be important │ thoughts: │ │ │ │ │ to me.', 'I would like to │ │ │ │ │ │ see some examples of how │ 1. **User Engagement**: A │ │ │ │ │ to use this feature.'] │ progress bar can help │ │ │ │ │ │ keep participants engaged │ │ │ │ │ │ by giving them a clear │ │ │ │ │ │ sense of how far along │ │ │ │ │ │ they are in the survey. │ │ │ │ │ │ This can be especially │ │ │ │ │ │ important for longer │ │ │ │ │ │ surveys, where │ │ │ │ │ │ participants might │ │ │ │ │ │ otherwise drop out before │ │ │ │ │ │ completing it. │ │ │ │ │ │ │ │ │ │ │ │ 2. **Completion Rates**: │ │ │ │ │ │ By providing a visual │ │ │ │ │ │ indicator of progress, │ │ │ │ │ │ you might see higher │ │ │ │ │ │ completion rates. People │ │ │ │ │ │ are more likely to finish │ │ │ │ │ │ something when they can │ │ │ │ │ │ see the end in sight. │ │ │ │ │ │ │ │ │ │ │ │ 3. **User Experience**: │ │ │ │ │ │ It generally enhances the │ │ │ │ │ │ user experience by │ │ │ │ │ │ reducing uncertainty. │ │ │ │ │ │ Participants know exactly │ │ │ │ │ │ how much more they need │ │ │ │ │ │ to do, which can reduce │ │ │ │ │ │ frustration. │ │ │ │ │ │ │ │ │ │ │ │ 4. **Optional Nature**: │ │ │ │ │ │ Since it’s optional, │ │ │ │ │ │ users who don’t find it │ │ │ │ │ │ helpful can simply turn │ │ │ │ │ │ it off. This flexibility │ │ │ │ │ │ is great because it │ │ │ │ │ │ caters to different │ │ │ │ │ │ preferences. │ │ │ │ │ │ │ │ │ │ │ │ However, a couple of │ │ │ │ │ │ concerns might include: │ │ │ │ │ │ │ │ │ │ │ │ 1. **Implementation │ │ │ │ │ │ Complexity**: Depending │ │ │ │ │ │ on how your survey │ │ │ │ │ │ platform is currently │ │ │ │ │ │ built, adding a progress │ │ │ │ │ │ bar might require │ │ │ │ │ │ significant changes to │ │ │ │ │ │ the codebase. │ │ │ │ │ │ │ │ │ │ │ │ 2. **Accuracy**: Ensuring │ │ │ │ │ │ the progress bar │ │ │ │ │ │ accurately reflects the │ │ │ │ │ │ participant's progress │ │ │ │ │ │ might be tricky, │ │ │ │ │ │ especially if the survey │ │ │ │ │ │ has branching logic or │ │ │ │ │ │ conditional questions. │ │ │ │ │ │ │ │ │ │ │ │ 3. **Distraction**: For │ │ │ │ │ │ some users, a progress │ │ │ │ │ │ bar might be distracting │ │ │ │ │ │ or create pressure, which │ │ │ │ │ │ could affect the quality │ │ │ │ │ │ of their responses. │ │ └───────────────────────────┴────────────┴───────────────────────────┴───────────────────────────┴────────────────┘
[12]:
# Post the notebook on the Coop
from edsl import Notebook
n = Notebook(path = "digital_twin.ipynb")
n.push(description="Digital Twin", visibility="public")
[12]:
{'description': 'Digital Twin',
'object_type': 'notebook',
'url': 'https://www.expectedparrot.com/content/3fb6fb6f-0adf-4db1-a23c-193909162b2d',
'uuid': '3fb6fb6f-0adf-4db1-a23c-193909162b2d',
'version': '0.1.33.dev1',
'visibility': 'public'}
[15]:
# Update an object at the Coop
n = Notebook(path = "digital_twin.ipynb")
n.patch(uuid = "3fb6fb6f-0adf-4db1-a23c-193909162b2d", value = n)
[15]:
{'status': 'success'}