Language Models

Language models are used to generate agent responses to questions and can be specified when running a survey. API keys are required in order to access the available models, and should be stored in your private .env file. See the Starter Tutorial section for instructions on storing your API keys.

Available models

We can see all of the available models by calling the available() method of the Model class:

from edsl import Model

Model.available()

This will return a list of the models we can choose from:

[['01-ai/Yi-34B-Chat', 'deep_infra', 0],
['Austism/chronos-hermes-13b-v2', 'deep_infra', 1],
['Gryphe/MythoMax-L2-13b', 'deep_infra', 2],
['Gryphe/MythoMax-L2-13b-turbo', 'deep_infra', 3],
['HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1', 'deep_infra', 4],
['Phind/Phind-CodeLlama-34B-v2', 'deep_infra', 5],
['bigcode/starcoder2-15b', 'deep_infra', 6],
['bigcode/starcoder2-15b-instruct-v0.1', 'deep_infra', 7],
['claude-3-haiku-20240307', 'anthropic', 8],
['claude-3-opus-20240229', 'anthropic', 9],
['claude-3-sonnet-20240229', 'anthropic', 10],
['codellama/CodeLlama-34b-Instruct-hf', 'deep_infra', 11],
['codellama/CodeLlama-70b-Instruct-hf', 'deep_infra', 12],
['cognitivecomputations/dolphin-2.6-mixtral-8x7b', 'deep_infra', 13],
['databricks/dbrx-instruct', 'deep_infra', 14],
['deepinfra/airoboros-70b', 'deep_infra', 15],
['gemini-pro', 'google', 16],
['google/codegemma-7b-it', 'deep_infra', 17],
['google/gemma-1.1-7b-it', 'deep_infra', 18],
['gpt-3.5-turbo', 'openai', 19],
['gpt-3.5-turbo-0125', 'openai', 20],
['gpt-3.5-turbo-0301', 'openai', 21],
['gpt-3.5-turbo-0613', 'openai', 22],
['gpt-3.5-turbo-1106', 'openai', 23],
['gpt-3.5-turbo-16k', 'openai', 24],
['gpt-3.5-turbo-16k-0613', 'openai', 25],
['gpt-3.5-turbo-instruct', 'openai', 26],
['gpt-3.5-turbo-instruct-0914', 'openai', 27],
['gpt-4', 'openai', 28],
['gpt-4-0125-preview', 'openai', 29],
['gpt-4-0613', 'openai', 30],
['gpt-4-1106-preview', 'openai', 31],
['gpt-4-1106-vision-preview', 'openai', 32],
['gpt-4-turbo', 'openai', 33],
['gpt-4-turbo-2024-04-09', 'openai', 34],
['gpt-4-turbo-preview', 'openai', 35],
['gpt-4-vision-preview', 'openai', 36],
['gpt-4o', 'openai', 37],
['gpt-4o-2024-05-13', 'openai', 38],
['lizpreciatior/lzlv_70b_fp16_hf', 'deep_infra', 39],
['llava-hf/llava-1.5-7b-hf', 'deep_infra', 40],
['meta-llama/Llama-2-13b-chat-hf', 'deep_infra', 41],
['meta-llama/Llama-2-70b-chat-hf', 'deep_infra', 42],
['meta-llama/Llama-2-7b-chat-hf', 'deep_infra', 43],
['meta-llama/Meta-Llama-3-70B-Instruct', 'deep_infra', 44],
['meta-llama/Meta-Llama-3-8B-Instruct', 'deep_infra', 45],
['microsoft/WizardLM-2-7B', 'deep_infra', 46],
['microsoft/WizardLM-2-8x22B', 'deep_infra', 47],
['mistralai/Mistral-7B-Instruct-v0.1', 'deep_infra', 48],
['mistralai/Mistral-7B-Instruct-v0.2', 'deep_infra', 49],
['mistralai/Mixtral-8x22B-Instruct-v0.1', 'deep_infra', 50],
['mistralai/Mixtral-8x22B-v0.1', 'deep_infra', 51],
['mistralai/Mixtral-8x7B-Instruct-v0.1', 'deep_infra', 52],
['openchat/openchat_3.5', 'deep_infra', 53]]

Available models are updated regularly. A current list is also viewable at edsl.enums.LanguageModelType.

If you don’t see a model that you want to work with, please send us a feature request to add it!

Check models

We can check the models that for which we have already properly stored API keys by calling the check_models() method:

Model.check_models()

This will return a list of the available models and a confirmation message whether a valid key exists.

Specifying a model

We specify a model to use with a survey by creating a Model object and passing it the name of an available model. We can optionally set other model parameters as well (temperature, etc.). For example, the following code creates a Model object for Claude 3 with default model parameters:

from edsl import Model

model = Model('claude-3-opus-20240229')

We can see that the object consists of a model name and a dictionary of parameters:

model

This will return the following:

{
"model": "claude-3-opus-20240229",
"parameters": {
   "temperature": 0.5,
   "max_tokens": 1000,
   "top_p": 1,
   "frequency_penalty": 0,
   "presence_penalty": 0,
   "logprobs": false,
   "top_logprobs": 3
}
}

We can also print the model name and parameters in a readable table with the print() method:

model.print()

This will print the following table:

                                    Language Model
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Attribute         ┃ Value                                                               ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ model             │ 'claude-3-opus-20240229'                                            │
│ parameters        │ {'temperature': 0.5, 'max_tokens': 1000, 'top_p': 1,                │
│                   │ 'frequency_penalty': 0, 'presence_penalty': 0, 'logprobs': False,   │
│                   │ 'top_logprobs': 3}                                                  │
│ temperature       │ 0.5                                                                 │
│ max_tokens        │ 1000                                                                │
│ top_p             │ 1                                                                   │
│ frequency_penalty │ 0                                                                   │
│ presence_penalty  │ 0                                                                   │
│ logprobs          │ False                                                               │
│ top_logprobs      │ 3                                                                   │
└───────────────────┴─────────────────────────────────────────────────────────────────────┘

We can also inspect the default parameters of the model by calling the parameters method on it:

model.parameters

This will return the following dictionary of parameters:

{'temperature': 0.5,
'max_tokens': 1000,
'top_p': 1,
'frequency_penalty': 0,
'presence_penalty': 0,
'logprobs': False,
'top_logprobs': 3}

Running a survey with a model

Similar to how we specify Agents and Scenarios in running a survey, we specify the models to use by adding them to a survey with the by() method when the survey is run. If a single model is specified, it is the only item passed to the by() method. If multiple models are to be used, they are passed as a list. For example, the following code specifies that a survey be run with each of GPT 4 and Llama 2:

from edsl import Model

models = [Model('gpt-4-1106-preview'), Model('llama-2-70b-chat-hf')]

from edsl import Survey

survey = Survey.example()

results = survey.by(models).run()

This will generate a result for each question in the survey with each model. If agents and/or scenarios are also specified, the responses will be generated for each combination of agents, scenarios and models. Each component is added with its own by() method, the order of which does not matter. The following commands are equivalent:

results = survey.by(scenarios).by(agents).by(models).run()

results = survey.by(models).by(agents).by(scenarios).run()

If we only want to use a single model it can be passed directly to the by() method:

results = survey.by(Model('gpt-4-1106-preview')).run()

Default model

If no model is specified, a survey is automatically run with the default model (GPT 4). For example, the following code runs a survey with the default model (and no agents or scenarios) without needing to import the Model class:

from edsl import Survey

results = survey.run()

Inspecting model details in results

After running a survey, we can inspect the models used by calling the models method on the result object. For example, we can verify the default model when running a survey without specifying a model:

from edsl import Survey

survey = Survey.example()

results = survey.run()

results.models

This will return the following information about the default model that was used:

{
"model": "gpt-4-1106-preview",
"parameters": {
   "temperature": 0.5,
   "max_tokens": 1000,
   "top_p": 1,
   "frequency_penalty": 0,
   "presence_penalty": 0,
   "logprobs": false,
   "top_logprobs": 3
}
}

To learn more about all the components of a Results object, please see the Results section.

Printing model attributes

If multiple models were used to generate results, we can print the attributes in a table. For example, the following code prints a table of the model names and temperatures for some results:

from edsl import Model

models = [Model('gpt-4-1106-preview'), Model('llama-2-70b-chat-hf')]

from edsl.questions import QuestionMultipleChoice, QuestionFreeText

q1 = QuestionMultipleChoice(
   question_name = "favorite_day",
   question_text = "What is your favorite day of the week?",
   question_options = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
)

q2 = QuestionFreeText(
   question_name = "favorite_color",
   question_text = "What is your favorite color?"
)

from edsl import Survey

survey = Survey([q1, q2])

results = survey.by(models).run()

results.select("model.model", "model.temperature").print()

The table will look like this:

model.model

model.temperature

gpt-4-1106-preview

0.5

llama-2-70b-chat-hf

0.5

We can also print model attributes together with other components of results. We can see a list of all components by calling the columns method on the results:

results.columns

For the above example, this will display the following list of components (note that no agents were specified, so there are no agent fields listed other than the default agent_name that is generated when a job is run):

['agent.agent_name',
'answer.favorite_color',
'answer.favorite_day',
'answer.favorite_day_comment',
'iteration.iteration',
'model.frequency_penalty',
'model.logprobs',
'model.max_new_tokens',
'model.max_tokens',
'model.model',
'model.presence_penalty',
'model.stopSequences',
'model.temperature',
'model.top_k',
'model.top_logprobs',
'model.top_p',
'prompt.favorite_color_system_prompt',
'prompt.favorite_color_user_prompt',
'prompt.favorite_day_system_prompt',
'prompt.favorite_day_user_prompt',
'raw_model_response.favorite_color_raw_model_response',
'raw_model_response.favorite_day_raw_model_response']

The following code will display a table of the model names together with the simulated answers:

(results
.select("model.model", "answer.favorite_day", "answer.favorite_color")
.print()
)

The table will look like this:

model.model

answer.favorite_day

answer.favorite_color

gpt-4-1106-preview

Sat

My favorite color is blue.

llama-2-70b-chat-hf

Sat

My favorite color is blue. It reminds me of the ocean on a clear summer day, full of possibilities and mystery.

To learn more about methods of inspecting and printing results, please see the Results section.

LanguageModel class

This module contains the LanguageModel class, which is an abstract base class for all language models.

class edsl.language_models.LanguageModel.LanguageModel(**kwargs)[source]

Bases: RichPrintingMixin, PersistenceMixin, ABC

ABC for LLM subclasses.

property RPM[source]

Model’s requests-per-minute limit.

property TPM[source]

Model’s tokens-per-minute limit.

>>> m = LanguageModel.example()
>>> m.TPM
1600000.0
__init__(**kwargs)[source]

Initialize the LanguageModel.

property api_token[source]
abstract async async_execute_model_call(system_prompt)[source]

Execute the model call and returns the result as a coroutine.

async async_get_raw_response(user_prompt: str, system_prompt: str, cache, iteration: int = 0) dict[str, Any][source]

Handle caching of responses.

Parameters:
  • user_prompt – The user’s prompt.

  • system_prompt – The system’s prompt.

  • iteration – The iteration number.

  • cache – The cache to use.

If the cache isn’t being used, it just returns a ‘fresh’ call to the LLM, but appends some tracking information to the response (using the _update_response_with_tracking method). But if cache is being used, it first checks the database to see if the response is already there. If it is, it returns the cached response, but again appends some tracking information. If it isn’t, it calls the LLM, saves the response to the database, and returns the response with tracking information.

If self.use_cache is True, then attempts to retrieve the response from the database; if not in the DB, calls the LLM and writes the response to the DB.

async async_get_response(user_prompt: str, system_prompt: str, cache: Cache, iteration: int = 1) dict[source]

Get response, parse, and return as string.

Parameters:
  • user_prompt – The user’s prompt.

  • system_prompt – The system’s prompt.

  • iteration – The iteration number.

  • cache – The cache to use.

cost(raw_response: dict[str, Any]) float[source]

Return the dollar cost of a raw response.

classmethod example()[source]

Return a default instance of the class.

execute_model_call(**kwargs)[source]
classmethod from_dict(data: dict) Type[LanguageModel][source]

Convert dictionary to a LanguageModel child instance.

get_raw_response(user_prompt: str, system_prompt: str, cache, iteration: int = 0) dict[str, Any][source]

Handle caching of responses.

Parameters:
  • user_prompt – The user’s prompt.

  • system_prompt – The system’s prompt.

  • iteration – The iteration number.

  • cache – The cache to use.

If the cache isn’t being used, it just returns a ‘fresh’ call to the LLM, but appends some tracking information to the response (using the _update_response_with_tracking method). But if cache is being used, it first checks the database to see if the response is already there. If it is, it returns the cached response, but again appends some tracking information. If it isn’t, it calls the LLM, saves the response to the database, and returns the response with tracking information.

If self.use_cache is True, then attempts to retrieve the response from the database; if not in the DB, calls the LLM and writes the response to the DB.

get_response(user_prompt: str, system_prompt: str, cache: Cache, iteration: int = 1) dict[source]

Get response, parse, and return as string.

Parameters:
  • user_prompt – The user’s prompt.

  • system_prompt – The system’s prompt.

  • iteration – The iteration number.

  • cache – The cache to use.

has_valid_api_key() bool[source]

Check if the model has a valid API key.

>>> LanguageModel.example().has_valid_api_key()
True

This method is used to check if the model has a valid API key.

hello(verbose=False)[source]

Runs a simple test to check if the model is working.

abstract parse_response() str[source]

Parse the API response and returns the response text.

What is returned by the API is model-specific and often includes meta-data that we do not need. For example, here is the results from a call to GPT-4: To actually tract the response, we need to grab data[“choices[0]”][“message”][“content”].

print()[source]

Print the object to the console.

async remote_async_execute_model_call(user_prompt, system_prompt)[source]

Execute the model call and returns the result as a coroutine, using Coop.

rich_print()[source]

Display an object as a table.

set_rate_limits(rpm=None, tpm=None) None[source]

Set the rate limits for the model.

>>> m = LanguageModel.example()
>>> m.set_rate_limits(rpm=100, tpm=1000)
>>> m.RPM
80.0
simple_ask(question: QuestionBase, system_prompt='You are a helpful agent pretending to be a human.', top_logprobs=2)[source]

Ask a question and return the response.

to_dict() dict[str, Any][source]

Convert instance to a dictionary.

class edsl.language_models.LanguageModel.RegisterLanguageModelsMeta(name, bases, namespace, /, **kwargs)[source]

Bases: ABCMeta

Metaclass to register output elements in a registry i.e., those that have a parent.

REQUIRED_CLASS_ATTRIBUTES = ['_model_', '_parameters_', '_inference_service_'][source]
__init__(name, bases, dct)[source]

Register the class in the registry if it has a _model_ attribute.

static check_required_class_variables(candidate_class: LanguageModel, required_attributes: List[str] = None)[source]

Check if a class has the required attributes.

>>> class M:
...     _model_ = "m"
...     _parameters_ = {}
>>> RegisterLanguageModelsMeta.check_required_class_variables(M, ["_model_", "_parameters_"])
>>> class M2:
...     _model_ = "m"
>>> RegisterLanguageModelsMeta.check_required_class_variables(M2, ["_model_", "_parameters_"])
Traceback (most recent call last):
...
Exception: Class M2 does not have required attribute _parameters_
classmethod get_registered_classes()[source]

Return the registry.

classmethod model_names_to_classes()[source]

Return a dictionary of model names to classes.

static verify_method(candidate_class: LanguageModel, method_name: str, expected_return_type: Any, required_parameters: List[tuple[str, Any]] = None, must_be_async: bool = False)[source]

Verify that a method is defined in a class, has the correct return type, and has the correct parameters.

edsl.language_models.LanguageModel.handle_key_error(func)[source]

Handle KeyError exceptions.

Other methods

class edsl.language_models.registry.Meta[source]

Bases: type

class edsl.language_models.registry.Model(model_name=None, registry=None, *args, **kwargs)[source]

Bases: object

classmethod available(search_term=None, name_only=False, registry=None)[source]
classmethod check_models(verbose=False)[source]
default_model = 'gpt-4-1106-preview'[source]
edsl.language_models.registry.get_model_class(model_name, registry=None)[source]