Operations and Runtime: capacity, rate limits, and acceptance (administrators and operators only)
Evaluate operating capacity
For administrators, implementation teams, and operators.
Before launch, estimate how many AOS Work requests, ordinary Runtime tasks, and Sandbox tasks your infrastructure can handle. Check whether model rate limits constrain expected throughput, and use load tests to verify monitoring and recovery procedures.
The figures below are planning baselines for separate AOS Work and Runtime hosts with approximately 15%–25% production headroom. They are not a customer-environment commitment. Final capacity must be proven with target-environment tests and actual model quotas.
Quick pre-launch check
- Run representative load tests in the target environment and record success rate, response time, and resource usage.
- Confirm model quotas and rate limits cover the expected traffic.
- Check that CPU, memory, and disk still have enough headroom during peak load.
- Prepare an executable recovery plan for failures, rate limiting, or service interruptions.
Evaluation measures
| Measure | Purpose | Acceptance use |
|---|---|---|
| QPS / req/s | AOS Work interface requests per second | Stable under target load without persistent errors or saturation |
| Run/s | Ordinary Runtime tasks per second without Sandbox | Validate with actual model calls and external dependencies |
| Session/s | Complete AI tasks per second with Sandbox | Measure light, medium, and heavy work separately |
| Success rate | Successful requests/tasks divided by all requests/tasks | Capacity conclusion requires at least 99% |
| P50 / P95 / P99 | Median, typical-tail, and long-tail latency | Record all three; set thresholds through the business SLA |
| CPU, memory, disk | Service and Sandbox headroom | Retain safe headroom during peak tests; CPU count alone is not concurrency |
Review usage and execution quality
Open 【Overview】 → 【Usage Overview】, select a time range and refresh, then review activity, token and Credits consumption, data coverage, and execution quality. This example uses local sample data. Usage metrics do not replace QPS, latency, CPU, and memory load tests in the target environment.
Infrastructure capacity baseline
| Machine | Stable AOS Work requests | Ordinary Runtime without Sandbox | Sandbox light / medium / heavy | Recommended use |
|---|---|---|---|---|
| 4 vCPU / 16 GB | 180–250 req/s | 20–35 Run/s | 1–3 / 0.5–1.5 / 0.1–0.5 Session/s | Development, demos, low traffic |
| 8 vCPU / 32 GB | 320–430 req/s | 45–75 Run/s | 3–7 / 1–3 / 0.2–0.8 Session/s | Small-to-medium production, light AI tools |
| 16 vCPU / 64 GB | 450–500 req/s | 140–160 Run/s | 8–20 / 3–8 / 0.5–2 Session/s | Current recommended production baseline |
| 32 vCPU / 128 GB | 480–600 req/s | 220–300 Run/s | 16–36 / 6–15 / 1–4 Session/s | Larger production, more concurrent tool tasks |
Light Sandbox work performs limited state queries and file I/O; medium work includes roughly 8 MB file writes and multi-round calculation; heavy work includes roughly 32 MB writes and intensive repeated calculation. Long-running commands, large files, complex code, and document generation should be planned as medium or heavy work.
Deployment and concurrency baseline
| Machine | Without Sandbox | With Sandbox | Primary risk |
|---|---|---|---|
| 4 vCPU / 16 GB | 1 Runtime, 16–32 ordinary concurrent tasks | Light 2; medium/heavy 1 | CPU and memory headroom |
| 8 vCPU / 32 GB | 1–2 Runtimes, about 32 ordinary concurrent tasks each | Light 4–6; medium 2–3; heavy 1–2 | Disk and memory under medium/heavy work |
| 16 vCPU / 64 GB | 4 Runtimes, maximum 64 ordinary concurrent tasks each | 4 Runtimes; light 12, medium 6, heavy 3 | CPU contention when co-hosted with AOS Work |
| 32 vCPU / 128 GB | 4–8 Runtimes, maximum 64 ordinary concurrent tasks each | 8 Runtimes; light 24, medium 12, heavy 6 | Database, messaging and disk throughput limits |
Never apply the no-Sandbox concurrency limit directly to Sandbox work. Prove absence of CPU, memory, and I/O contention if AOS Work and Runtime share a host.
Include model limits in capacity calculations
End-to-end throughput is the minimum of AOS Work capacity, Runtime/Sandbox capacity, model RPM/RPS, model TPM/TPS, and model concurrency or provisioned throughput.
Record applicable limits separately for each platform account/subscription, region, model, version, and endpoint/deployment type. Context Window and Credits do not replace provider rate limits.
Required RPM = peak business requests/s × 60 × model calls per business request
Required TPM = peak business requests/s × 60 × (P95 input tokens + P95 output tokens) × provider conversion factorPlan sustained production load at only 70%–80% of provider quota to reserve capacity for bursts, retries, and long-tail output. Current document-generation observations are approximately 100,000–500,000 cumulative tokens per complete task; use 500,000 as an initial upper bound when samples are unavailable, then replace it with measured P50/P95 values.
Runtime System Tool and Action boundary
Discover business actions using workbench_action_search, then execute the Action Key and Schema returned for that run. Search is neither authorization nor business-completion evidence. access=user is limited to current-user-accessible data; admin actions also require an administrator or super-administrator seat.