Model Rate Limits and Throughput Capacity Planning

1. Configuration Goal

AOS Work and Runtime capacity planning provides capacity references for the AOS Work, Runtime, and Sandbox layers. Production throughput is also constrained by upstream model-service rate limits, so it cannot be estimated from the number of Runtime instances alone.

The usable throughput is the minimum of the following bottlenecks:

Usable throughput = min(
  AOS Work API capacity,
  Runtime and Sandbox execution capacity,
  model-platform request quota,
  model-platform token throughput quota,
  purchased or provisioned dedicated throughput
)

This chapter explains how to record the effective rate-limit scope for each model, calculate peak demand, and configure Runtime throttling, queuing, retry, and monitoring before launch. Platform rules and account quotas can change. Use the live values shown for the target account, region, model, version, and deployment type as the final configuration authority.

2. Rate-limit Metrics to Distinguish

A model's Context Window limits the size of one request; it is not a per-minute throughput allowance. Credits are BasilOS billing and usage-accounting units; they are not cloud-platform quota units.

3. Current Azure Environment Record

The supplied Azure portal page shows the following model deployments. The record date is September 8, 2026, and the values are only a snapshot of this environment:

The page does not show the subscription, region, deployment type, or RPM. It therefore does not establish the complete throughput. Before launch, record those fields from the Azure portal and confirm whether the quota is allocated to one Deployment or shared by the same subscription, region, model, and deployment type.

Azure OpenAI defines TPM and RPM by subscription, region, model, and deployment type. The TPM/RPM mapping of a Capacity Unit varies by model, so a ratio for one model must not be applied to another. For higher and more predictable throughput, evaluate a quota increase or Provisioned Throughput Units (PTU). See the Microsoft quotas and limits documentation.

4. Rate-limit Strategies on Other Cloud Platforms

4.1 Amazon Bedrock

The rate-limit method depends on the Bedrock endpoint. bedrock-runtime uses combined TPM by model, account, and Region for On-demand and Cross-Region calls, and can also enforce RPM, TPD, and model-specific burndown. bedrock-mantle separates input TPM from output TPM and does not expose a separate per-model RPM.

The following bedrock-mantle default quotas were listed in the AWS General Reference on September 8, 2026. Each row is scoped by AWS account, supported Region, and model, and is shared across all Mantle APIs for that model:

Do not apply this table to bedrock-runtime. For Cross-Region calls, for example, AWS currently lists combined TPM of 6,000,000 for Claude Sonnet 5, 20,000,000 for GPT-5.6 Luna, 10,000,000 for GPT-5.6 Sol, and 20,000,000 for GPT-5.6 Terra. Applicable RPM and TPD must still be read from the target account's Service Quotas console.

References: AWS Bedrock endpoints and quotas, Bedrock Runtime quotas, and token quota calculation.

4.2 Google Cloud Vertex AI

Vertex AI uses two fundamentally different rate-limit methods for newer Gemini models. Pay-as-you-go uses Dynamic Shared Quota (DSQ), with no fixed RPM/TPM value that can be entered in advance. Workloads that require a defined allowance must purchase Provisioned Throughput measured in GSUs.

The following model-specific values were verified on September 8, 2026. The “quota tokens/minute” column is derived from the official tokens/s/GSU × 60; it represents quota tokens after burndown, not unweighted raw input and output tokens:

For example, one GSU of gemini-2.5-pro can consume 39,000 quota tokens per minute when input is no more than 200K. Generating 1,000 output tokens consumes 8,000 quota tokens under the 8× output burndown. The actual request count depends on the input, output, and modality mix, so a GSU cannot be converted into one fixed RPM value.

A 429 RESOURCE_EXHAUSTED response under DSQ means that the shared pool does not currently have enough capacity; it does not mean that the workload crossed a fixed TPM number. Smooth traffic, use truncated exponential backoff, and evaluate the global endpoint. Provisioned Throughput is bound to a project, Region, model, and version, and cannot be combined across other Regions or models.

References: Vertex AI quotas and system limits, 429 handling, Provisioned Throughput supported models, and Provisioned Throughput sizing.

4.3 Alibaba Cloud Model Studio

Model Studio aggregates limits by primary account and model, combining traffic from RAM users, workspaces, and API Keys under the account. Fixed-limit models are governed by both RPM and TPM and can also be enforced at RPM/60 RPS and TPM/60 TPS. A sudden increase can additionally trigger Traffic Burst protection.

The following fixed allowances for text models in China (Beijing) were verified on September 8, 2026. TPM includes input and output tokens:

Models such as qwen3.8-max and qwen3.8-flash use dynamic soft rate limiting in China (Beijing). RPM is high and normally requires no separate configuration. The TPM values below are guaranteed baselines; actual available TPM can be higher when the platform has spare capacity:

An overage generally returns 429. Use the error details to distinguish RPM, TPM, RPS/TPS, and Traffic Burst. Fixed values, dynamic tiers, and temporary increases can change. At launch, read the target API Key's model_limit and workspace_limit from the console or quota API.

References: Model Studio rate limits, dynamic rate limiting, list model quotas, and rate-limit handling best practices.

5. Convert Business Demand into Quota Demand

Collect the following inputs separately for every model:

• peak business requests per second;
• P95 input tokens per request;
• maximum output tokens or P95 output tokens per request;
• expected model calls per user task, including tool loops;
• expected retry ratio and batch workload;
• the platform's quota conversion rules for cached tokens, output tokens, and multimodal input.

5.1 Observed Task-consumption Reference

The current test result provides an initial capacity baseline for document-generation tasks:

This range is an observation from the current test scenario, not a fixed value for every Excel or PPT task. Actual consumption depends on Excel file size and complexity, target slide count, content per slide, model selection, repeated context, tool-loop count, revision rounds, and failed-request retries.

The value is cumulative across multiple model calls in one complete task. It must not be treated as tokens for one API request or as one minute of TPM. For initial capacity planning, convert task volume into token demand as follows:

Task-level token demand/minute = peak completed tasks/minute
                               × 100,000–500,000 tokens/task
                               × platform burndown factor

Until more samples are available, use 500,000 tokens/task as the upper bound for production peak planning, then add retry and safety headroom. Continue recording P50/P95 token consumption by attachment size, target slide count, and model, and replace this initial range when enough measurements are available.

Estimate baseline demand as follows:

Required RPM = peak business requests/second × 60
             × model calls per business request

Required TPM = peak business requests/second × 60
             × (P95 input tokens + P95 output tokens)
             × platform burndown factor

Calculate input and output separately when the platform maintains separate pools; otherwise calculate the combined token total. For sustained production load, use only 70%–80% of the platform quota as the operating budget, leaving headroom for traffic variation, retries, and long-tail output. This percentage is a capacity-planning recommendation, not a fixed rule of any cloud platform.

The model-side request capacity must satisfy all applicable constraints:

Model-side requests/second = min(
  RPM / 60,
  TPM / (60 × P95 quota tokens/request),
  requests/second supported by dedicated throughput
)

Take the minimum of this result and the AOS Work, Runtime, and Sandbox capacities in AOS Work and Runtime capacity planning to obtain the end-to-end capacity conclusion.

6. Runtime Traffic-control Requirements

1. Use platform account or subscription + region + model + version + deployment type or endpoint as the rate-limit key. Do not use one global QPS value.
2. Configure both a request token bucket and a token-usage bucket. Add request smoothing for platforms that enforce RPS, TPS, or burst protection.
3. Give interactive requests and batch jobs separate queues and budgets so that background work cannot consume all online quota.
4. Reserve tokens before submission using the input and maximum output. Reconcile the reservation after actual usage is returned. Apply the model-specific burndown rate for platforms such as Bedrock.
5. On 429, first inspect the platform error type and Retry-After. When no delay is supplied, use truncated exponential backoff with jitter and a bounded retry count.
6. Retries create new traffic and must be included in capacity calculations. Do not allow every Runtime instance to retry immediately at the same time.
7. Model fallback is allowed only when an administrator has preconfigured a compatible and authorized fallback. A rate limit must not silently replace the model selected by the user.
8. If queue time exceeds the interactive deadline, return a clear model-busy/retry-later result. Move delay-tolerant work to an asynchronous queue.

7. Monitoring and Alerts

At minimum, record the following by platform, region, model, and deployment:

• request count, success rate, 429 count, and 429 ratio;
• input, output, cached, and quota-accounting tokens;
• current RPM/TPM utilization and remaining allowance;
• queue depth, queue time, retry count, and retry success rate;
• P50, P95, and P99 time to first token and total response time;
• dedicated-throughput utilization, requests overflowing to Pay-as-you-go, and overflow cost.

A practical starting point is to alert at 70% sustained utilization and restrict new batch work at 80%. Adjust these thresholds for business variability and quota-increase lead time.

8. Launch Checklist and Completion Criteria

• Platform, account or subscription, region, model ID, version, and deployment type or endpoint are recorded for every production model.
• Applicable RPM, TPM, RPS/TPS, TPD, concurrency, or dedicated-throughput values have been read from the target account console and recorded with the verification date.
• Peak RPM and TPM have been calculated using P95 tokens and tool-loop call counts, and the result has been combined with the AOS Work and Runtime capacity planning infrastructure capacity using the minimum bottleneck.
• Runtime implements isolated dual rate limiting, queuing, jittered backoff, and retry limits for each rate-limit key.
• Interactive and batch traffic are isolated, and throttling never silently switches to an unauthorized model.
• Alerts cover approaching quota, increasing 429s, queue backlog, and dedicated-throughput overflow.
• A staged load test has validated sustained and burst traffic, recording success rate, 429s, P95/P99 latency, token usage, and recovery time.
• Changes to platform quota or model version trigger a review of this chapter's records and Runtime rate-limit configuration.