Rate limits

Every key has limits per minute. A workspace on an API plan also has a cap in tokens per second, shared by all its API keys in each region.

On an API plan

The plan sets the workspace's tokens per second: API Small 25, API Medium 100 and API Large 400.

Output is paced to the plan's tokens per second across all the workspace's requests, and a request is never refused for the plan. Over the cap, responses slow down; requests at once share it. A prompt does not delay its own first token. It counts when the request ends, so a long uncached prompt slows the workspace's tokens after it.

Up to 8 requests at once per workspace. One more waits up to 60 seconds for a place, then returns 429 with the code rate_limited and a retry-after.

Stream on a plan. A paced answer arrives as it is generated; a request without stream waits for the whole answer at the plan's cap, which on a small tier can be minutes, and counts against your client's timeout.

Output tokens count in full. An uncached input token counts as 0.33. Cached input tokens do not count. The cap is shared by every API key in the workspace, and applies in each region separately. Sail keys are paid for by the Sail plan and never count against it.

A key on an API plan has no tokens-per-minute limit: the plan's pace governs its tokens. Its requests per minute still apply.

Inference responses to a plan workspace's API keys carry the plan's cap beside the per-key headers:

x-ratelimit-limit-plan-tokens-per-second: 25
x-ratelimit-plan-queued-ms: 0

Queued is how long the request waited for a place, in milliseconds. Plans has the tiers.

Per-key limits

Every key has these limits, per region, except that a key on an API plan or a Sail key on a Sail plan has no tokens-per-minute limit, and its requests per minute still apply. Pay per token, they are the only ones.

Default limits. Contact us to raise them.

Tokens are completion tokens and the prompt tokens the engine did not read from its prefix cache. They are counted when the response ends, however it ends. Close a stream early and what was served still counts. While a request runs, the most it could use is held against the minute: its whole prompt, since the cache is not known until the engine answers, and your max_tokens, or the model's completion limit when you name none. Whatever it does not use comes back as it ends. One long request can take the minute's tokens below zero. The next request waits until they are back.

A request carries the whole conversation. The part the engine reads from the prefix cache is billed at the cache read price and does not come off the minute. An agent that re-sends a 150k-token conversation with 2k new tokens and a 1k-token answer spends about 3k tokens of the minute, but needs room for the whole 150k while the turn runs.

Every response carries the current state. Reset is the seconds until the limit is whole again:

x-ratelimit-limit-requests: 5000
x-ratelimit-remaining-requests: 47
x-ratelimit-reset-requests: 13
x-ratelimit-limit-tokens: 100000000
x-ratelimit-remaining-tokens: 99841808
x-ratelimit-reset-tokens: 5
retry-after: 12            # only on 429 and 503

Over the limit, the request returns 429 with the code rate_limited. Wait for retry-after seconds and send it again.

These docs as plain text for coding agents: llms.txt