- Options for setting requests per minute (RPM) and tokens per minute (TPM) for models at your account.
- Methods for setting RPM and TPM from your workspace.
- Methods for turning off default prompt features to reduce token usage.
- Methods for calculating next token probabilities.
Token limits
Token limits refer to the maximum number of tokens that a language model can process in a single input prompt or output generation. A token limit affects how much text you can send to a model in one go. A language model provider should provide information about the token limits for each model that is associated with your account and API key. When running a big job in EDSL, you may encounter token limits, which can be managed by adjusting the token limits for a model.RPM: Requests Per Minute
RPM stands for Requests Per Minute, which measures the number of API requests that a user can make to a language model within a minute. This is a metric for managing the load and traffic that a model can handle.TPM: Tokens Per Minute
TPM stands for Tokens Per Minute, which is a metric for tracking the volume of tokens processed by a language model within a minute. This metric typically tracks usage for billing purposes.Default token limits
You can inspect and modify token limits for a model at the Keys page of your account:
key | value |
---|---|
model | gpt-4o |
parameters:temperature | 0.5 |
parameters:max_tokens | 1000 |
parameters:top_p | 1 |
parameters:frequency_penalty | 0 |
parameters:presence_penalty | 0 |
parameters:logprobs | False |
parameters:top_logprobs | 3 |
Modifying token limits
We can reset the default RPM and TPM and then check the new values:Methods for reducing token usage
There are several ways to reduce the tokens required to run a question or survey.Turning off question commments
Each question type (other than free_text) automatically includes a comment field that gives the answering model a place to put additional information about its response to a question. This serves as an outlet for a chatty model to return context about an answer without violating formatting instructions (e.g., a model may want to provide an explanation for a mutiple choice response but the answer to the question must only be one of the answer options). Question comments can also be useful when used with survey “memory” rules, giving a model an opportunity to simulate a “chain of thought” across multiple survey questions. (By default, questions are administered asynchronously; a model does not have context of other questions and answers in a survey unless memory rules are applied.) Comments can also provide insight into non-responsive (None) answers: a model may use the comments field to describe a point of confusion about a question. Because the question comment field requires additional tokens, it can sometimes be cost-effective to exclude the field from question prompts, especially when the comment is unlikely to be useful. This is done by passing a boolean parameter include_comment = False when constructing a question. For example, here we compare a question with comments left on and turned off:scenario.number_1 | scenario.number_2 | answer.sum | comment.sum_comment | answer.sum_silent | comment.sum_silent_comment |
---|---|---|---|---|---|
0 | 5 | 5 | The sum of 0 and 5 is 5. | 5 | None |
1 | 4 | 5 | The sum of 1 and 4 is 5. | 5 | None |
2 | 3 | 5 | The sum of 2 and 3 is 5. | 5 | None |
3 | 2 | 5 | The sum of 3 and 2 is 5. | 5 | None |
4 | 1 | 5 | The sum of 4 and 1 is 5. | 5 | None |
Coding question options
Question instructions can be modified to prompt a model to use codes (integers) in lieu of text responses for answer options, reducing generated tokens. This is done by passing a boolean parameter use_code = True to a Question when it is constructed. For example:question_index | user_prompt |
---|---|
income_pref | Which of the following is more important to you: Financial stability Moving up the income ladder Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. |
income_pref_coded | Which of the following is more important to you: 0: Financial stability 1: Moving up the income ladder Only 1 option may be selected. Respond only with the code corresponding to one of the options. |
prompt.income_pref_user_prompt | generated_tokens.income_pref_generated_tokens | prompt.income_pref_coded_user_prompt | generated_tokens.income_pref_coded_generated_tokens |
---|---|---|---|
Which of the following is more important to you: Financial stability Moving up the income ladder Only 1 option may be selected. Respond only with a string corresponding to one of the options. After the answer, you can put a comment explaining why you chose that option on the next line. | Financial stability provides a secure foundation and peace of mind, allowing for better long-term planning and resilience against unexpected challenges. | Which of the following is more important to you: 0: Financial stability 1: Moving up the income ladder Only 1 option may be selected. Respond only with the code corresponding to one of the options. | 0 |