Language Models

Language models are used to generate responses to survey questions. EDSL works with many models from a variety of popular inference service providers, including Anthropic, Azure, Bedrock, Deep Infra, DeepSeek, Google, Mistral, OpenAI, Perplexity and Together. Current model pricing and performance information can be found at the Coop model pricing page. The same information can also be retrieved at your workspace by running the Model.check_working_models() method (see example code below).

We also recommend checking providers’ websites for the most up-to-date information on models and service providers’ terms of use. Links to providers’ websites can be found at the Coop model pricing page. If you need assistance checking whether a model is working or to report a missing model or price, please send a message to info@expectedparrot.com or post a message on Discord.

This page provides examples of methods for specifying models for surveys using the Model and ModelList classes.

API keys

In order to use a model, you need to have an API key for the relevant service provider. EDSL allows you to choose whether to provide your own keys from service providers or use an Expected Parrot API key to access all available models at once. See the Managing Keys page for instructions on storing and prioritizing keys.

Available services

The following code will return a table of inference service providers:

from edsl import Model

Model.services()

Output:

Service Name
anthropic
azure
bedrock
deep_infra
deepseek
google
groq
mistral
ollama
openai
perplexity
together

Available models

The following code will return a table of models for all service providers that have been used with EDSL (output omitted here for brevity).

This list should be used together with the model pricing page to check current model performance with test survey questions. We also recommend running your own test questions with any models that you want to use before running a large survey.

from edsl import Model

Model.available()

To see a list of all models for a specific service, pass the service name as an argument:

Model.available(service = "google")

Output (this list will vary based on the models that have been used when the code is run):

Model Name	Service Name
gemini-pro	google
gemini-1.0-pro	google
gemini-1.0-flash	google
gemini-1.0-flash-8b	google
gemini-1.5-pro	google
gemini-2.0-flash	google

Check working models

You can check current performance and pricing for models by running the following code:

from edsl import Model

Model.check_working_models()

This will return the same information available at the model pricing page: Service, Model, Works with text, Works with images, Price per 1M input tokens (USD), Price per 1M output tokens (USD). It can also be used to check a particular service provider (output omitted here for brevity):

from edsl import Model

Model.check_working_models(service = "google")

Specifying a model

To specify a model to use with a survey, create a Model object and pass it the name of the model. You can optionally set other model parameters at the same time (temperature, etc.).

For example, the following code creates a Model object for gpt-4o with default model parameters that we can inspect:

from edsl import Model

m = Model("gpt-4o")
m

Output:

key	value
model	gpt-4o
parameters:temperature	0.5
parameters:max_tokens	1000
parameters:top_p	1
parameters:frequency_penalty	0
parameters:presence_penalty	0
parameters:logprobs	False
parameters:top_logprobs	3
inference_service	openai

We can see that the object consists of a model name and a dictionary of the default parameters of the model, together with the name of the inference service (some models are provided by multiple services).

Here we also specify the temperature when creating the Model object:

from edsl import Model

m = Model("gpt-4o", temperature = 1.0)
m

Output:

key	value
model	gpt-4o
parameters:temperature	1.0
parameters:max_tokens	1000
parameters:top_p	1
parameters:frequency_penalty	0
parameters:presence_penalty	0
parameters:logprobs	False
parameters:top_logprobs	3
inference_service	openai

Creating a list of models

To create a list of models at once, pass a list of model names to a ModelList object.

For example, the following code creates a Model for each of gpt-4o and gemini-pro:

from edsl import Model, ModelList

ml = ModelList([Model("gpt-4o"), Model("gemini-1.5-flash")])

This code is equivalent to the following:

from edsl import Model, ModelList

ml = ModelList(Model(model) for model in ["gpt-4o", "gemini-1.5-flash"])

We can also use a special method to pass a list of names instead:

from edsl import Model, ModelList

model_names = ['gpt-4o', 'gemini-1.5-flash']

ml = ModelList.from_names(model_names)

ml

Output:

topK	presence_penalty	top_logprobs	topP	temperature	stopSequences	maxOutputTokens	logprobs	max_tokens	frequency_penalty	model	top_p	inference_service
nan	0.000000	3.000000	nan	0.500000	nan	nan	False	1000.000000	0.000000	gpt-4o	1.000000	openai
1.000000	nan	nan	1.000000	0.500000	[]	2048.000000	nan	nan	nan	gemini-1.5-flash	nan	google

Running a survey with models

Similar to how we specify Agents and Scenarios to use with a survey, we specify the models to use by adding them to a survey with the by() method when the survey is run. We can pass either a single Model object or a list of models to the by() method. If multiple models are to be used they are passed as a list or as a ModelList object.

For example, the following code specifies that a survey will be run with each of gpt-4o and gemini-1.5-flash:

from edsl import Model, QuestionFreeText, Survey

m = [Model("gpt-4o"), Model("gemini-1.5-flash")]

q = QuestionFreeText(
   question_name = "example",
   question_text = "What is the capital of France?"
)

survey = Survey(questions = [q])

results = survey.by(m).run()

This code uses ModelList instead of a list of Model objects:

from edsl import Model, ModelList, QuestionFreeText, Survey

ml = ModelList(Model(model) for model in ["gpt-4o", "gemini-1.5-flash"])

q = QuestionFreeText(
   question_name = "example",
   question_text = "What is the capital of France?"
)

survey = Survey(questions = [q])

results = survey.by(ml).run()

This will generate a result for each question in the survey with each model. If agents and/or scenarios are also specified, the responses will be generated for each combination of agents, scenarios and models. Each component is added with its own by() method, the order of which does not matter. The following commands are equivalent:

# add code for creating survey, scenarios, agents, models here ...

results = survey.by(scenarios).by(agents).by(models).run()

# this is equivalent:
results = survey.by(models).by(agents).by(scenarios).run()

Default model

If no model is specified, a survey is automatically run with the default model. Run Model() to check the current default model.

For example, the following code runs the above survey with the default model (and no agents or scenarios) without needing to import the Model class:

results = survey.run() # using the survey from above

# this is equivalent
results = survey.by(Model()).run()

We can verify the model that was used:

results.select("model.model") # selecting only the model name

Output:

model
gpt-4o

Inspecting model parameters

We can also inspect parameters of the models that were used by calling the models of the Results object.

For example, we can verify the default model when running a survey without specifying a model:

results.models # using the results from above

This will return the same information as running results.select(“model.model”) in the example above.

To learn more about all the components of a Results object, please see the Results section.

Troubleshooting

Newly released models of service providers are automatically made available to use with your surveys whenever possible (not all service providers facilitate this).

If you do not see a model that you want to work with or are unable to instantiate it using the standard method, please send us a request to add it to info@expectedparrot.com.

ModelList class

class edsl.language_models.ModelList.ModelList(data: LanguageModel | None = None)

Bases: Base, UserList

__init__(data: LanguageModel | None = None)

Initialize the ScenarioList class.

>>> from edsl import Model
>>> m = ModelList(Model.available())

code()

This method should be implemented by subclasses.

classmethod example(randomize: bool = False) → ModelList

Returns an example ModelList instance.

Parameters:

randomize – If True, uses Model’s randomize method.

classmethod from_available_models(available_models_list: AvailableModels)

Create a ModelList from an AvailableModels object

classmethod from_dict(data)

Create a ModelList from a dictionary.

>>> newm = ModelList.from_dict(ModelList.example().to_dict())
>>> assert ModelList.example() == newm

classmethod from_names(*args, **kwargs)

A a model list from a list of names

property names

>>> ModelList.example().names
{'...'}

table(*fields, tablefmt: str | None = None, pretty_labels: dict | None = None)

>>> ModelList.example().table('model')
model
-------
gpt-4o
gpt-4o
gpt-4o

to_dict(sort=False, add_edsl_version=True)

This method should be implemented by subclasses.

to_list() → list

to_scenario_list()

tree(node_list: List[str] | None = None)

LanguageModel class

This module contains the LanguageModel class, which is an abstract base class for all language models.

Terminology:

raw_response: The JSON response from the model. This has all the model meta-data about the call.

edsl_augmented_response: The JSON response from model, but augmented with EDSL-specific information, such as the cache key, token usage, etc.

generated_tokens: The actual tokens generated by the model. This is the output that is used by the user. edsl_answer_dict: The parsed JSON response from the model either {‘answer’: …} or {‘answer’: …, ‘comment’: …}

class edsl.language_models.LanguageModel.LanguageModel(tpm: float | None = None, rpm: float | None = None, omit_system_prompt_if_empty_string: bool = True, key_lookup: 'KeyLookup' | None = None, **kwargs)

Bases: PersistenceMixin, RepresentationMixin, HashingMixin, ABC

ABC for Language Models.

DEFAULT_RPM = 100

DEFAULT_TPM = 1000

__init__(tpm: float | None = None, rpm: float | None = None, omit_system_prompt_if_empty_string: bool = True, key_lookup: 'KeyLookup' | None = None, **kwargs)

Initialize the LanguageModel.

property api_token: str

ask_question(question: QuestionBase) → str

Ask a question and return the response.

Parameters:

question – The question to ask.

abstract async async_execute_model_call(system_prompt: str)

Execute the model call and returns a coroutine.

async async_get_response(user_prompt: str, system_prompt: str, cache: Cache, iteration: int = 1, files_list: List[FileStore] | None = None, **kwargs) → dict

Get response, parse, and return as string.

Parameters:

• user_prompt – The user’s prompt.

• system_prompt – The system’s prompt.

• cache – The cache to use.

• iteration – The iteration number.

• files_list – The list of files to use.

cost(raw_response: dict[str, Any]) → float | str

Return the dollar cost of a raw response.

Parameters:

raw_response – The raw response from the model.

classmethod example(test_model: bool = False, canned_response: str = 'Hello world', throw_exception: bool = False) → LanguageModel

Return a default instance of the class.

>>> from edsl.language_models import LanguageModel
>>> m = LanguageModel.example(test_model = True, canned_response = "WOWZA!")
>>> isinstance(m, LanguageModel)
True
>>> from edsl import QuestionFreeText
>>> q = QuestionFreeText(question_text = "What is your name?", question_name = 'example')
>>> q.by(m).run(cache = False, disable_remote_cache = True, disable_remote_inference = True).select('example').first()
'WOWZA!'
>>> m = LanguageModel.example(test_model = True, canned_response = "WOWZA!", throw_exception = True)
>>> r = q.by(m).run(cache = False, disable_remote_cache = True, disable_remote_inference = True, print_exceptions = True)
Exception report saved to ...

execute_model_call(**kwargs)

from_cache(cache: Cache) → LanguageModel

classmethod from_dict(data: dict) → Type[LanguageModel]

Convert dictionary to a LanguageModel child instance.

NB: This method does not use the stores inference_service but rather just fetches a model class based on the name.

classmethod get_generated_token_string(raw_response: dict[str, Any]) → str

Return the generated token string from the raw response.

>>> m = LanguageModel.example(test_model = True)
>>> raw_response = m.execute_model_call("Hello, model!", "You are a helpful agent.")
>>> m.get_generated_token_string(raw_response)
'Hello world'

get_response(user_prompt: str, system_prompt: str, cache: Cache, iteration: int = 1, files_list: List[FileStore] | None = None, **kwargs) → dict

Get response, parse, and return as string.

Parameters:

• user_prompt – The user’s prompt.

• system_prompt – The system’s prompt.

• cache – The cache to use.

• iteration – The iteration number.

• files_list – The list of files to use.

classmethod get_usage_dict(raw_response: dict[str, Any]) → dict[str, Any]

Return the usage dictionary from the raw response.

has_valid_api_key() → bool

Check if the model has a valid API key.

>>> LanguageModel.example().has_valid_api_key() : 
True

This method is used to check if the model has a valid API key.

hello(verbose=False)

Runs a simple test to check if the model is working.

key_sequence = None

classmethod parse_response(raw_response: dict[str, Any]) → EDSLOutput

Parses the API response and returns the response text.

async remote_async_execute_model_call(user_prompt: str, system_prompt: str)

Execute the model call and returns the result as a coroutine, using Coop.

response_handler = <edsl.language_models.RawResponseHandler.RawResponseHandler object>

property rpm

set_key_lookup(key_lookup: KeyLookup) → None

Set the key lookup, later

simple_ask(question: QuestionBase, system_prompt='You are a helpful agent pretending to be a human.', top_logprobs=2)

Ask a question and return the response.

to_dict(add_edsl_version: bool = True) → dict[str, Any]

Convert instance to a dictionary

Parameters:

add_edsl_version – Whether to add the EDSL version to the dictionary.

>>> m = LanguageModel.example()
>>> m.to_dict()
{'model': '...', 'parameters': {'temperature': ..., 'max_tokens': ..., 'top_p': ..., 'frequency_penalty': ..., 'presence_penalty': ..., 'logprobs': False, 'top_logprobs': ...}, 'inference_service': 'openai', 'edsl_version': '...', 'edsl_class_name': 'LanguageModel'}

property tpm

class edsl.language_models.LanguageModel.classproperty(method)

Bases: object

__init__(method)

edsl.language_models.LanguageModel.handle_key_error(func)

Handle KeyError exceptions.

Other methods