---
title: "Performance and Scalability"
url: "https://docs.caracal.run/v1.0/operations/performance-benchmarks/"
markdown_url: "https://docs.caracal.run/markdown/v1.0/operations/performance-benchmarks.md"
description: "Measured throughput, latency, resource, failure, and sizing results from the v1.0.0 production validation, under stated test conditions."
page_type: "reference"
concepts: []
requires: []
---

# Performance and Scalability

Canonical URL: https://docs.caracal.run/v1.0/operations/performance-benchmarks/
Markdown URL: https://docs.caracal.run/markdown/v1.0/operations/performance-benchmarks.md
Description: Measured throughput, latency, resource, failure, and sizing results from the v1.0.0 production validation, under stated test conditions.
Page type: reference
Concepts: none
Requires: none

---

These are results from the Caracal **v1.0.0** production validation: a primary bench on a production cloud VM, and an extended bench covering component costs, failure injection, startup, and storage behavior. Every figure below was observed on the bench described in [Test Conditions](#test-conditions); none is extrapolated or theoretical.

Treat these as a **reference baseline for sizing and evaluation, not a service-level guarantee.** Throughput and latency depend on your hardware, workload shape, and upstreams. Establish your own baseline with [Scale Capacity](/v1.0/operations/scale-capacity/) before committing to a limit.

## Test Conditions

| Attribute | Primary cloud bench | Extended bench |
| --- | --- | --- |
| Deployment | Packaged Docker Compose via `caracal up`, single host, loopback-bound, stock configuration | Packaged Docker Compose, single host, loopback-bound |
| Host | Azure `Standard_D4as_v5`: 4 vCPU, 16 GiB RAM, StandardSSD, Ubuntu 24.04 | 16 vCPU, 60 GiB RAM, NVMe SSD, Linux (Fedora 43) |
| Load model | Closed-loop client, published TypeScript SDK, one in-flight governed request per worker, 16 long-lived application identities | Closed-loop client, published TypeScript SDK, one in-flight governed request per worker |
| Upstream | Mock HTTP service on the compose network (isolates Caracal overhead) | Local mock HTTP service (isolates Caracal overhead) |
| Credential path | Long-lived identities: verification served from the credential cache, the steady state for persistent agent fleets | Includes full Argon2id credential derivation |

A governed request performs one single-use, replay-protected mandate mint at the STS plus one Gateway hop. An application transport provisions its session and delegation once per credential lifetime (four control-plane calls), then mints per request.

## Throughput and Scaling

On the primary cloud bench, governed throughput scaled linearly with offered concurrency until the host CPU saturated, reaching **33 governed requests per second per host vCPU** with credential verification served from cache:

| Concurrent workers | Sustained governed req/s | p50 | p95 |
| --- | --- | --- | --- |
| 1 | 8.2 (paced) | 20 ms | 30 ms |
| 4 | 36.5 | 24 ms | 33 ms |
| 8 | 63.2 | 22 ms | 33 ms |
| 16 | 138.2 | 34 ms | 71 ms |

An **8-minute sustained soak** at 16 workers held **132.6 governed requests per second** — 63,984 requests with **zero transport or 5xx errors** — for a daily capacity of **11.5 million governed calls on one 4-vCPU host**. Peak measured throughput was **139 req/s**.

Two ceilings apply together:

* **Per resource:** a single `(zone, resource, application)` pair is capped at **1000 mints/minute (16.6 req/s)** by default. The cap is a deployment setting: raise `STS_MINT_RATE_LIMIT_PER_MIN` for a higher ceiling, or set a lower working limit from the web console (Settings → Preferences → Mint rate limit). Spreading load across resources or applications also multiplies the budget; the soak above spread 16 identities.
* **Per host:** CPU-bound, with the STS and Postgres as the first components to saturate. Add cores, or split the stores onto their own hosts, before tuning any Caracal setting.

Session lifecycle is independent of the mint path: on the extended bench the Coordinator sustained **192 session create-and-close cycles per second** (p50 20 ms) before lock contention flattened throughput.

## Latency

Measured on the primary cloud bench. The governance decision itself is negligible; cost is concentrated in the cryptographic mint, and it degrades gracefully rather than collapsing as the host saturates.

| Operation | p50 | p95 | p99 |
| --- | --- | --- | --- |
| Governed request (warm, below saturation) | 17-24 ms | 23-33 ms | 34-43 ms |
| Governed request (at full host saturation) | 45 ms | 85 ms | 113 ms |
| Governed request (cold, first call incl. authority provisioning) | 287 ms | — | — |
| Policy (OPA) evaluation, mean over 128,000 evaluations under load | 1.04 ms | — | — |
| List applications / resources | 4 ms | 6 ms | — |
| Audit query (50 rows) | 6 ms | 11 ms | — |

Component costs measured on the extended bench: Gateway mandate verification **3.7 ms**, delegation create **17.5 ms**, provider create **8.1 ms** p50.

The mint dominates because it verifies identity and traverses the delegation graph cryptographically. Policy evaluation and Gateway verification are sub-millisecond to low-single-digit milliseconds.

## Resource Usage and Stability

Measured during the primary cloud bench's **8-minute sustained soak** at full host saturation: **132.6 req/s, 63,984 requests, zero hard errors.**

| Signal | Observation |
| --- | --- |
| Host memory | 2 GiB used of 16 at full load; memory was never a limiting factor |
| Service restarts during soak | 0 |
| Postgres connections | 27-43 of the 100 ceiling; no connection leak |
| Audit events dropped | 0 — dead-letter queue empty at soak end, hash chain continuous |
| Policy evaluation errors | 0 across 128,000 evaluations |

On the extended bench, STS memory under sustained load showed a healthy garbage-collection sawtooth (floor 337 MB / peak 792 MB on a 2 GB limit) with no leak and no unbounded growth.

:::note[Size STS memory deliberately]
The STS hashes credentials with Argon2id, which allocates **64 MB per in-flight verification**. Two mechanisms bound that cost: verified credentials are cached against their stored hash (rotation invalidates the entry), so steady-state traffic skips the derivation entirely, and cold verifications run under a fixed concurrency budget (`STS_SECRET_VERIFY_CONCURRENCY`, default 2) that caps peak verification memory. In this validation, a 512 MB STS driven with a full derivation on every request exhausted its container limit under sustained multi-agent load and was restarted by the container runtime (fail-closed, automatic, ~2-5 s); raising the limit to **2 GB eliminated this entirely**. The packaged deployment ships the STS at **2 vCPU / 1 GB**; provision **2 GB** when raising the verification budget or running many distinct applications.
:::

## Failure and Recovery

On the extended bench, each dependency was failed in isolation while a continuous governed probe ran. Every failure was **fail-closed** — no request ever succeeded without valid authority — and every recovery was **automatic**, with no manual intervention.

| Injected failure | Live governed traffic | Recovery | Data loss |
| --- | --- | --- | --- |
| Redis unavailable (brief) | Requests block, then drain on return | Automatic, immediate | None |
| Postgres unavailable (brief) | Requests block, then drain on return | Automatic, immediate | None |
| STS process killed | Immediate fail-closed (connection refused) | Next request after restart; restart-to-healthy 0.6 s | None |
| Gateway process killed | Requests fail closed | Automatic; restart-to-healthy 0.6 s | None |
| Coordinator process killed | **Continues unbroken** (established transports mint against the STS) | No impact to live traffic | None |
| Audit process killed | **Continues unbroken** (asynchronous write path) | Events buffer and drain; hash chain verified continuous | None |

This separates the request path from the control and evidence paths:

* **Required for live governed traffic:** STS, Gateway, Postgres, Redis.
* **Not required for live governed traffic:** Coordinator (needed only to provision new sessions and delegations) and Audit (asynchronous). Their outages are invisible to established transports.

Brief database or cache interruptions shorter than the request timeout are absorbed as **latency, not errors**. For diagnosis and recovery order, see [Recover from Failures](/v1.0/operations/failure-modes/).

## Startup and Operational Timings

Measured on the extended bench.

| Operation | Time |
| --- | --- |
| Download, checksum-verify, and install the runtime | 9.0 s |
| Cold start (`caracal up`: image pull, migrate, readiness gate) | 17.7 s |
| Warm restart (`caracal down` then `up`) | 4.1 s + 3.7 s |
| Single service restart to healthy | 0.6 s |
| `caracal status` | 37-87 ms |
| Web console response | 3.6 ms |

## Storage and Capacity Growth

Measured on the extended bench.

| Metric | Measured |
| --- | --- |
| Audit storage per event | 1,843 bytes (including indexes) |
| Audit events per governed call | ~1.4 |
| Audit query scaling | Flat with total volume (keyset pagination over monthly partitions) |
| Database footprint (end of run) | 176 MB for 49,498 audit events, 8,450 sessions, 186 applications, 186 providers, 33 resources |
| Postgres connections | 100 ceiling; 50 peak under full load; no lock contention |
| Audit retention | 365-day default ceiling (`AUDIT_RETENTION_DAYS`), console-adjustable below it; hourly Parquet export to S3-compatible storage for longer archival |

At a sustained governed rate, audit disk grows at approximately `rate × 1.4 × 1843` bytes per second — for example, ~62 KB/s (~5.3 GB/day) at 24 req/s. Plan audit storage and [retention](/v1.0/operations/backup-retention/) for your rate.

## Known Product Limits

Defaults are deliberately conservative safety limits. The throughput figures above were reached by raising the capacity-shaping limits for the test workload.

| Limit | Default | Behavior when exceeded |
| --- | --- | --- |
| STS mint rate, per `(zone, resource, app)` | 1000 / minute (`STS_MINT_RATE_LIMIT_PER_MIN`); console working limit below it | Denied, fail-closed; recorded as `rate_limited` in audit |
| STS repeated authentication failures, per application | 60 / minute | Temporary block |
| Coordinator requests, per client IP | 600 / minute (`COORDINATOR_RATE_LIMIT_PER_MIN`) | `429 rate_limited` |
| Live sessions per zone | 50 (`MAX_AGENTS_PER_ZONE`) | `429 session_zone_limit_exceeded` |
| Live sessions per application | 200 (`MAX_AGENTS_PER_APP`) | `429 session_limit_exceeded` |
| Control API calls, per client | 60 / minute | `429` |

To operate at the throughput and concurrency in this report, raise `MAX_AGENTS_PER_ZONE` to your concurrent-agent count and `STS_MINT_RATE_LIMIT_PER_MIN` to your target rate, then confirm the change with [Scale Capacity](/v1.0/operations/scale-capacity/). Configure limits and container resources in [Configure Service Environment](/v1.0/operations/env-vars/); the console-managed working limits live under Settings → Preferences.

## Recommended Starting Profiles

Sizing is driven by the STS (throughput and memory) and Postgres (durability). These profiles derive from the extended-bench measurements, which include full credential derivation on the load path; fleets of long-lived identities served from the credential cache reach substantially higher throughput on the same profile, as the primary-bench results above show. Size the supporting stores with headroom above the measured footprint and validate before production.

| Workload target | STS | Gateway | Postgres | Redis |
| --- | --- | --- | --- | --- |
| Small — ≤ 10 req/s, ≤ 50 sessions | 1 vCPU / 1 GB | 1 vCPU / 512 MB | 2 vCPU / 2 GB | 0.5 vCPU / 512 MB |
| Standard — ~25 req/s, hundreds of sessions | 2 vCPU / 2 GB | 2 vCPU / 1 GB | 4 vCPU / 4 GB (SSD) | 1 vCPU / 768 MB |
| Large — ~40 req/s per instance | 4 vCPU / 4 GB | 2 vCPU / 2 GB | 8 vCPU / 8 GB (NVMe) | 1 vCPU / 1 GB |
| High throughput | STS replicas at 4 vCPU / 4 GB, sharded across resources | 4 vCPU / 2 GB | 8 vCPU / 16 GB + read replica | 2 vCPU / 2 GB |

The packaged deployment ships the STS at 2 vCPU / 1 GB, between the Small and Standard profiles. Move to the Standard profile's 2 GB when sustaining ~25 req/s. Postgres used up to roughly one vCPU-equivalent and 461 MB with 50 connections at 24 req/s; provision it with headroom above that and prefer SSD or NVMe storage.

## Production Recommendations

* **Size STS memory at or above the 1 GB packaged default.** The credential cache removes Argon2id from the steady state and the verification budget bounds cold bursts; provision 2 GB when raising `STS_SECRET_VERIFY_CONCURRENCY` or running many distinct applications.
* **Raise the per-zone session cap** (`MAX_AGENTS_PER_ZONE`) to your concurrent-agent count before load; the default is 50.
* **Raise `STS_MINT_RATE_LIMIT_PER_MIN`** when a shared resource needs more than 16.6 req/s, or shard across resources or applications; then add STS replicas to scale beyond one instance.
* **Provision Postgres as the second bottleneck** after STS CPU: SSD or NVMe storage, connection headroom below the 100 default, and roughly twice the STS vCPU.
* **Alert on the STS memory limit, Postgres connections, and audit lag**, and validate every limit change against your own workload.

## Next Step

Turn these results into a sized deployment with [Choose a Deployment Profile](/v1.0/operations/deployment-profiles/), then validate your own baseline in [Scale Capacity](/v1.0/operations/scale-capacity/).