# Caracal > Caracal gives agents and automated workloads short-lived, policy-approved authority for protected resources. Caracal is an open-source system built by Garudex Labs. Applications request scoped authority, STS evaluates policy, Gateway or an in-process verifier enforces the issued Mandate, Coordinator owns Sessions and Delegations, and Audit records decisions and outcomes. --- --- # Overview # URL: https://docs.caracal.run/v1.0/get-started/ # Markdown: https://docs.caracal.run/markdown/v1.0/get-started.md # Type: landing # Concepts: # Requires: --- Caracal is a self-hosted, open-source system that controls what AI agents and automated programs are allowed to do, without giving them your API keys. This page explains the problem it solves and how it works before you install anything. By the end you will know every term the rest of Get Started uses. ## The Problem If you run agents or automation today, each program that calls an external API usually holds a credential for it, an API key or token in an environment variable or config file. That works until you look closely at what the program can now do: * The key usually grants far more than the program needs. An agent that should only read reports can often also write, delete, or administer. * The key lives inside the program's process. A prompt injection, dependency compromise, or leaked log exposes it, and whoever has it has everything it grants. * Turning access off means rotating the key and redeploying every workload that shares it. There is no central off switch. * When something goes wrong, there is no reliable record of which program did what, with which permission, and why it was allowed. These problems get worse with agents, because agents decide at runtime what to call. A static, all-powerful key in the hands of software that improvises is a standing risk. ## The Solution Caracal puts a decision point between your program and the things it calls. Your program never holds the upstream API key. Instead, each time it wants to act: 1. The program proves its identity to Caracal. 2. It asks for a narrow, specific permission, such as "read from the reports service". 3. Caracal checks the rules you wrote. This happens before the action runs, not after. 4. If the rules allow it, Caracal issues a short-lived signed pass for exactly that permission. 5. The request travels through a checkpoint that verifies the pass, attaches the real upstream credential if one is needed, and forwards the request. 6. The decision and the result are written to a tamper-evident log. If you revoke access, the pass stops working at the checkpoint. There is no key to rotate and no redeploy, because the program never had the key. :::note[Key takeaway] Your program never holds the upstream credential. It holds a short-lived, narrowly scoped pass that Caracal can refuse to renew or revoke centrally, and every use of it is recorded. ::: ## The Building Blocks Caracal gives each part of that flow a name. Seven terms cover the rest of Get Started, and each one maps to something you already know: | Term | Plain meaning | Familiar comparison | | --- | --- | --- | | Application | The registered identity your program acts as when it talks to Caracal. | An OAuth client with an ID and secret. | | Resource | A thing you protect: an API, tool, model provider, or data service. | An upstream service behind a proxy. | | Provider | Caracal's sealed custody of a resource's real upstream credential, attached only after a request is approved. | A vault entry only the proxy can use. | | Policy | The rules that decide which application may use which resource, with which permissions. | An allow/deny ruleset evaluated before execution. | | Mandate | The short-lived signed pass Caracal issues when policy allows a request. | A scoped access token with a tight expiry. | | Gateway | The HTTP checkpoint in front of your resources. It verifies each mandate, attaches the upstream credential when one is needed, forwards the request, and records the result. | A reverse proxy that authenticates every request. | | Audit | The append-only record of every decision and result. | A structured, tamper-evident event log. | One more term appears when you first sign in: a **Zone** is an isolated workspace, like a project or tenant. Applications, resources, policies, and audit records all live inside one zone and never leak across zones. ## What Happens on a Request ```mermaid sequenceDiagram participant App as Your program participant Caracal as Caracal (policy check) participant GW as Gateway participant API as Protected service App->>Caracal: Who I am + what I want to do Caracal->>Caracal: Evaluate active policy Caracal-->>App: Short-lived mandate (or denial) App->>GW: Request + mandate GW->>GW: Verify mandate, attach upstream credential GW->>API: Forwarded request API-->>GW: Response GW-->>App: Response Note over Caracal,GW: Decision and result recorded in audit ``` Read it once as a story: your program asks, policy decides, a mandate is issued, the Gateway verifies and forwards, and audit records both the decision and the outcome. Every page that follows walks a real request through exactly this path. ## When to Use Caracal Caracal fits when programs, not humans, need controlled access: | Question | Use Caracal when | | --- | --- | | Do programs hold credentials? | Agents, services, or jobs call APIs, tools, providers, or data systems, and you do not want raw keys inside them. | | Must access be decided up front? | An action should be allowed or denied before it runs, based on rules you control centrally. | | Do you need a fast off switch? | Access must end centrally, without rotating keys or restarting every workload. | | Do you need evidence? | You must be able to show which program did what, with which permission, under which rule, and what happened. | Caracal is not an LLM framework, agent scheduler, or general-purpose API gateway. If you already use adjacent tools, here is where it differs: | You already use | How Caracal differs | | --- | --- | | A secrets manager | It stores and rotates keys, but your program still holds the key at call time. Caracal keeps the key out of the program entirely and decides each request against policy. | | An API gateway | It routes and rate-limits, but it does not decide per-request permission from your rules or issue short-lived passes per action. | | An identity provider | It authenticates people. Caracal governs what programs do. Keep your IdP for logins; you can federate it into Caracal later for attribution. If your only need is human users behind a normal login, an identity provider alone is the right tool. | ## What You Will Do in This Section ```mermaid flowchart LR Install[Install Caracal] Call[First Protected Call] SDK[Add SDK to Your App] Fix[Troubleshoot if needed] Install --> Call --> SDK -.-> Fix ``` 1. [Install Caracal](./install-caracal/) - install one `caracal` executable, start the local stack with Docker, and enable console sign-in. 2. [First Protected Call](./first-protected-call/) - give an AI agent scoped authority to call an LLM provider from the browser console, then send one request through the full path and read its audit trail. 3. [Add SDK to Your App](./add-sdk-to-your-app/) - turn the first call into an application integration with a configuration profile. 4. [First-Run Troubleshooting](./first-run-troubleshooting/) - a boundary-by-boundary checklist if any step fails. You are done when `caracal status --ready` succeeds, an agent's request reaches the protected provider through the Gateway, and the audit view shows the matching allow decision and result. A successful local run proves the enforcement path; it does not make the local stack production-ready. ## Prerequisites You need a supported Linux, macOS, or Windows machine, Docker 25 or later with Compose v2, and permission to run containers. No Caracal source checkout is required. ## Where to Go Deeper Get Started needs only the terms on this page. When you want the full model, including delegation between agents, revocation mechanics, and the audit chain, read [Concepts](/v1.0/concepts/) after your first successful call. Contributors who want to build Caracal from source should use [Set Up Locally](/v1.0/contributing/setup/) instead of this path. ## Next Step Continue with [Install Caracal](./install-caracal/). --- # Install Caracal # URL: https://docs.caracal.run/v1.0/get-started/install-caracal/ # Markdown: https://docs.caracal.run/markdown/v1.0/get-started/install-caracal.md # Type: workflow # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' Running Caracal locally takes two pieces: * **The `caracal` CLI** - a single executable that starts and stops the local stack and launches your programs under Caracal's control. * **Docker** - the local stack runs as containers: Caracal's services, a PostgreSQL database, a Redis cache, and the web console, the browser interface where you will create and manage everything in the next page. You install one binary; Docker provides the rest. No source checkout, package build, or separate database install is required. ## Prerequisites * a supported Linux, macOS, or Windows machine; * Docker 25 or later with Compose v2; * `curl` or a browser for downloading a release; * permission to install one executable on your `PATH`. ## Install the CLI The install script downloads the archive for your platform, verifies its checksum, and places `caracal` on your `PATH`. The commands below pin this release; drop `--version` (or set `CARACAL_VERSION`) to track the latest stable release instead. ```sh curl -fsSL https://raw.githubusercontent.com/Garudex-Labs/caracal/main/install.sh | sh -s -- --version v1.0.0 ``` ```powershell $installer = "$env:TEMP\install.ps1" iwr -useb https://raw.githubusercontent.com/Garudex-Labs/caracal/main/install.ps1 -OutFile $installer powershell -ExecutionPolicy Bypass -File $installer -Version v1.0.0 ``` The installer places `caracal.exe` in `%LOCALAPPDATA%\Programs\caracal` and adds it to the user `Path`; open a new shell afterward. Prefer to download and verify manually? Each [release](https://github.com/Garudex-Labs/caracal/releases) ships platform archives (`caracal-runtime-*`), a `manifest.json`, and a `SHA256SUMS` file. Pick the archive matching your platform (`linux-amd64`, `linux-arm64`, `darwin-amd64`, `darwin-arm64`, or `windows-amd64`): ```sh tag=v1.0.0 platform=linux-amd64 # or linux-arm64, darwin-amd64, darwin-arm64 curl -fsSLO "https://github.com/Garudex-Labs/caracal/releases/download/${tag}/caracal-runtime-${platform}-${tag}.tar.gz" curl -fsSLO "https://github.com/Garudex-Labs/caracal/releases/download/${tag}/SHA256SUMS" sha256sum --ignore-missing --check SHA256SUMS # macOS: shasum -a 256 --ignore-missing --check SHA256SUMS tar -xzf "caracal-runtime-${platform}-${tag}.tar.gz" install -m 0755 caracal ~/.local/bin/caracal ``` Make sure the destination directory is on your `PATH`. ```powershell $tag = 'v1.0.0' Invoke-WebRequest "https://github.com/Garudex-Labs/caracal/releases/download/$tag/caracal-runtime-windows-amd64-$tag.zip" -OutFile caracal.zip Invoke-WebRequest "https://github.com/Garudex-Labs/caracal/releases/download/$tag/SHA256SUMS" -OutFile SHA256SUMS Get-FileHash .\caracal.zip -Algorithm SHA256 # compare with the matching SHA256SUMS entry Expand-Archive .\caracal.zip -DestinationPath $env:LOCALAPPDATA\Programs\caracal ``` Add `%LOCALAPPDATA%\Programs\caracal` to the user `Path` and open a new shell. Both install scripts always verify checksums. With the GitHub CLI `gh` installed they also verify build provenance; set `CARACAL_REQUIRE_PROVENANCE=1` to make a missing provenance check fail the install instead of skipping it. For deeper supply-chain checks, signatures, and container image verification, see [Verify a Release](/v1.0/security/verify-releases/). ## Verify the Installation The same command works in every shell: ```sh caracal --version ``` It should print the installed version without errors. ## Verify Docker ```sh docker compose version ``` If this fails, install or start Docker Desktop (macOS, Windows) or Docker Engine with the Compose plugin (Linux) before continuing. ## Start the Local Stack ```sh caracal up caracal status --ready ``` `caracal up` pulls and starts the containers; `caracal status --ready` waits until every service reports healthy. When readiness succeeds, the web console is being served at [http://localhost:3001](http://localhost:3001). ## Enable Console Sign-In The console ships locked down: nobody can register until you, from the machine that runs the stack, allow their email. Allow yours: ```sh caracal allowlist add ``` The change applies immediately; no restart is needed. The same command family later manages suspension, restoration, and removal - see [Control Console Access](/v1.0/runtime-console/console-access/) for the full lifecycle. An allowlisted email still needs a way to sign in. Pick one and add its settings to the operator env file the stack reads - `$CARACAL_HOME/caracal.env`, whose exact per-platform path and editor command are in [Configure Service Environment](/v1.0/operations/env-vars/#the-operator-env-file) - then rerun `caracal up`: | Sign-in method | Required variables | | --- | --- | | Google or GitHub | `GOOGLE_CLIENT_ID` + `GOOGLE_CLIENT_SECRET` or `GITHUB_CLIENT_ID` + `GITHUB_CLIENT_SECRET`. OAuth callback URL: `http://localhost:3001/api/auth/callback/google` or `.../github`. | | Email and password | `CARACAL_PASSWORD_SIGNUP=true`, plus `CARACAL_SMTP_URL` and `CARACAL_SMTP_FROM` so the required verification email can be delivered; the console fails closed without a mail transport. | For OAuth, create an OAuth app in the [Google Cloud console](https://console.cloud.google.com/apis/credentials) or [GitHub developer settings](https://github.com/settings/developers), set its callback URL to the value above, and paste the client ID and secret into `caracal.env`. With email/password sign-up, registration sends a verification link over SMTP and the account signs in after the link is confirmed. The full sign-in reference lives in [Configure Service Environment](/v1.0/operations/env-vars/#web-console-bff). :::note[Why is sign-in this strict on a local machine?] The packaged runtime runs the console in published mode, which enforces the same fail-closed sign-in posture you would run in production: host-controlled admission plus a verified sign-in method. This is a one-time setup; everything after it happens in the browser. ::: Do not sign in yet - the next page walks through onboarding and your first protected setup in one continuous flow. ## Common Mistakes * If the shell cannot find `caracal` after installation, reopen the shell or add the install directory to `PATH`. * Do not use `caracal purge` to recover from a normal startup error; it intentionally removes all local state. * If you have a Caracal source checkout for contribution work, keep it separate: this path uses the released `caracal` executable, not the source-tree `pnpm caracal` command. ## Expected Outcome `caracal status --ready` exits successfully, [http://localhost:3001](http://localhost:3001) responds, your email is allowlisted, and one sign-in method is configured. The console's sign-in and registration pages live at `/sign-in` and `/sign-up` under that origin. Readiness proves the local services are up; nothing is protected yet - that is the next page. ## Platform Notes | Platform | Architectures | Notes | | --- | --- | --- | | Linux | `amd64`, `arm64` | Requires `curl` or `wget`, `tar`, and `sha256sum` or `shasum`. | | macOS | `amd64`, `arm64` | If Gatekeeper quarantines the binary, remove quarantine from the installed file. | | Windows | `amd64` | Open a new shell after the installer updates the user `Path`. | ## Next Step Continue with [First Protected Call](/v1.0/get-started/first-protected-call/). --- # First Protected Call # URL: https://docs.caracal.run/v1.0/get-started/first-protected-call/ # Markdown: https://docs.caracal.run/markdown/v1.0/get-started/first-protected-call.md # Type: workflow # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' In this walkthrough an AI agent makes its first protected call. The agent asks Caracal for authority, receives a short-lived scoped mandate, and calls an LLM provider through the Gateway - which checks policy, attaches the provider's real API key on the way through, and records everything. The agent never sees that key. ```mermaid flowchart LR Agent[AI agent] -->|scoped mandate| GW[Gateway] GW -->|provider key attached| LLM[LLM provider API] Policy[Your policy] -.decides.-> GW Audit[Audit trail] -.records.-> GW ``` This is the workflow Caracal is built for, and it inverts traditional service authentication. A conventional service holds a long-lived API key and uses it whenever it likes; you find out what it did afterward, if at all. An agent under Caracal starts with nothing: authority is requested per run, granted by policy in exactly the scope you allowed, expires on its own, and leaves evidence. That difference matters most for agents because agents decide at runtime what to call. Concretely, you will create four objects in the browser, then make the call with a small script: * an **Application** - the identity your agent acts as; * a **Provider** - Caracal's sealed custody of the LLM API key, so the key lives in Caracal, not in the agent; * a **Resource** - Caracal's record of the LLM API and where to forward verified requests; * a **Policy** - one rule allowing that application to list the provider's models. ## Prerequisites * Complete [Install Caracal](/v1.0/get-started/install-caracal/), including the sign-in setup at the end of that page. * An API key for an OpenAI-compatible LLM provider. The walkthrough's only upstream call is `GET /v1/models`, which is free and consumes no tokens. * One script runtime: Node.js 22+, Python 3.12+, or Go 1.26+. * Keep a terminal open. :::note[No provider key handy?] Any OpenAI-compatible endpoint works: point the resource's upstream URL at a local model runner instead and create the provider with kind **None** (the Gateway then enforces policy and records audit without attaching a credential). Every other step is identical. One rule to respect: the upstream URL is resolved from inside the Gateway's container, where `localhost` means the Gateway itself - use a hostname the Gateway can reach. ::: ## Start Caracal The same commands work in every shell: ```sh caracal up caracal status --ready ``` If readiness fails, run `caracal status --ready --json`; the output names the service that is not ready. Create nothing until readiness succeeds. ## Sign In and Create Your First Zone Open the web console at [http://localhost:3001](http://localhost:3001) and sign in with the method you configured during [installation](/v1.0/get-started/install-caracal/#enable-console-sign-in). A first sign-in walks through a short onboarding flow: 1. **Profile** - your name and avatar. 2. **Zone** - create your first zone. As introduced in the Overview, a zone is an isolated workspace: everything you create next lives inside it. 3. **Review** - confirm and finish. Onboarding drops you into the console for that zone. The browser address follows an `account → org → zone` hierarchy (an org groups zones; onboarding creates a default one for you). If you later create more zones, the zone selector switches between them. ## Create the Agent's Access Chain The console now shows **Guided setup**, a checklist that explains each building block and opens its real create form. Work through it in order: | Step | What you do | Why | | --- | --- | --- | | Application | Create **Anton** as a *confidential* application - confidential means it runs server-side and can keep a secret, as opposed to code running in a browser. Copy the application ID and the client secret it shows you. | This is the identity your agent acts as. | | Provider | Create a provider named **OpenAI** with kind **Bearer** and paste your provider API key. The key is sealed server-side on save. | This moves the LLM key out of your agent and into Caracal's custody. The Gateway will attach it to verified requests; the agent never receives it. | | Resource | Create **OpenAI** with identifier `resource://openai`, scope `openai:models`, upstream URL `https://api.openai.com`, and the provider you just created. | This tells Caracal what it is protecting and where the Gateway should forward verified requests. A *scope* is a named permission - `openai:models` is the one permission this walkthrough grants: listing the provider's models. | | Policy | Create and activate the starter policy allowing Anton to request `openai:models`. | Caracal denies everything not explicitly allowed. Without an active policy, every request in the zone is refused. | Guided setup ticks each step off from live zone data; you can also reach the same forms from the **Applications**, **Providers**, **Resources**, and **Policies** pages in the navigation. :::tip[Lost the secret?] The application client secret stays recoverable: reveal it again from the application's detail panel in the console. Every reveal is recorded in audit, so recovery never bypasses the evidence trail. The provider API key is different - it is sealed for the Gateway's use and is never returned to anyone. ::: ## Give the Agent Its Identity Your rules exist; now the agent needs the identity you just registered. It takes exactly three values from guided setup - the zone ID (shown in zone settings and in the console URL), the application ID, and the client secret: ```sh export CARACAL_ZONE_ID= export CARACAL_APPLICATION_ID= export CARACAL_APP_CLIENT_SECRET= ``` ```powershell $env:CARACAL_ZONE_ID = "" $env:CARACAL_APPLICATION_ID = "" $env:CARACAL_APP_CLIENT_SECRET = "" ``` On the local stack the SDK's built-in defaults already point at the right service URLs, so these three variables are the whole configuration. Notice what is *not* here: the OpenAI key. The agent's environment never contains it. ## The Agent's First Protected Call Install the SDK for one language and run the smallest possible agent - a script that asks Caracal for authority to list the provider's models, then sends that one request through the Gateway. On the local stack the Gateway listens at `http://localhost:8081`; the SDK builds the request and attaches the mandate for you. ```sh npm install @caracalai/sdk ``` ```typescript // agent.mjs - run with: node agent.mjs import { Caracal } from '@caracalai/sdk' const caracal = new Caracal() const governedFetch = caracal.applicationTransport('resource://openai', { scopes: ['openai:models'], }) const target = caracal.gatewayRequest('resource://openai', '/v1/models') try { const response = await governedFetch(target.url, { method: 'GET' }) if (!response.ok) throw new Error(`protected call failed: ${response.status}`) console.log(await response.text()) } finally { await caracal.close() } ``` ```sh pip install caracalai-sdk ``` ```python # agent.py - run with: python agent.py import asyncio from caracalai import Caracal async def main(): caracal = Caracal() target = caracal.gateway_request("resource://openai", "/v1/models") try: async with caracal.application_transport( "resource://openai", scopes=["openai:models"], ) as governed: response = await governed.get(target.url) response.raise_for_status() print(response.text) finally: await caracal.aclose() asyncio.run(main()) ``` ```sh go mod init agent && go get github.com/garudex-labs/caracal/packages/sdk/go ``` ```go // agent.go - run with: go run agent.go package main import ( "fmt" "io" caracal "github.com/garudex-labs/caracal/packages/sdk/go" ) func main() { client, err := caracal.New() if err != nil { panic(err) } defer client.Close() // nil base: the SDK constructs its own HTTP client. governed, err := client.ApplicationTransport(nil, "resource://openai", caracal.ApplicationTransportOptions{ Scopes: []string{"openai:models"}, }) if err != nil { panic(err) } target, err := client.GatewayRequest("resource://openai", "/v1/models") if err != nil { panic(err) } resp, err := governed.Get(target.URL) if err != nil { panic(err) } defer resp.Body.Close() if resp.StatusCode < 200 || resp.StatusCode >= 300 { panic(fmt.Sprintf("protected call failed: %s", resp.Status)) } body, _ := io.ReadAll(resp.Body) fmt.Println(string(body)) } ``` A JSON list of models means every link in the chain worked: the agent authenticated as Anton, policy allowed `openai:models`, a short-lived mandate was issued, and the Gateway verified it, swapped the mandate for the sealed provider key, forwarded the request, and returned the provider's answer. The agent authenticated to the provider without ever possessing its credential. Here is what happened in that moment. `applicationTransport()` asked Caracal for exactly the authority you listed - the `openai:models` scope on `resource://openai` - and attached a fresh, short-lived, single-use mandate to the request. `gatewayRequest()` built the Gateway URL and the routing header that names the resource. The Gateway did the rest. A real agent does exactly this on every protected action - request scoped authority, present it to the Gateway - whether the operation is listing models, a chat completion under an `openai:chat` scope you add later, or any other API you protect. :::caution[Failure point: the mandate is not a proxy pass] The Gateway is not an open relay with extra steps. Let the mandate expire, replay one, or request a scope policy does not allow, and the request is rejected before the provider is contacted - and the rejection is recorded. ::: ## Read the Audit Trail Every step you just triggered was recorded. In the web console: 1. Open **Audit**. 2. Find the request you just made. 3. Open the event detail to follow its full decision trace. The explanation shows which application asked, which resource and scopes were requested, which policy version decided, what the Gateway did, and the final result. If a request fails, this same view tells you whether to fix configuration, policy, resource routing, or the upstream provider - diagnose from here rather than guessing. ## Common Mistakes * The agent presents a Caracal mandate to the Gateway, never a provider key. If your agent's code asks for `OPENAI_API_KEY`, point it at the governed transport instead - the [next page](/v1.0/get-started/add-sdk-to-your-app/) and [Provider Recipes](/v1.0/guides/provider-recipes/) show how existing provider clients adopt it. * Resource identifiers and scopes must match the active policy exactly - `openai:models` and `openai-models` are different strings. * The one secret in the agent's environment is Anton's client secret - its identity, not the provider key. That key is sealed in Caracal and never leaves it. * Signing in to the console does not connect your identity to the agent's calls. The call you made was made by the application identity. (Federating user identities in for attribution is possible later; nothing in Get Started needs it.) ## Clean Up ```sh caracal down ``` `caracal down` stops the stack and keeps your data. Use `caracal purge` only when you intentionally want to erase local containers, volumes, config, runtime state, and caches. ## Next Step Your agent made one protected call from a throwaway script. Next, wire the same call into a real application with a configuration profile: [Add SDK to Your App](/v1.0/get-started/add-sdk-to-your-app/). --- # Add SDK to Your App # URL: https://docs.caracal.run/v1.0/get-started/add-sdk-to-your-app/ # Markdown: https://docs.caracal.run/markdown/v1.0/get-started/add-sdk-to-your-app.md # Type: workflow # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' In [First Protected Call](/v1.0/get-started/first-protected-call/) a throwaway script proved the enforcement path with three environment variables. This page turns that into the shape real applications use: a durable **configuration profile** instead of pasted variables, and an example parameterized so the same code reaches any resource you protect. Pick one language; the three examples are equivalent. ## Prerequisites * Complete [First Protected Call](/v1.0/get-started/first-protected-call/). * Keep Caracal running. * Install one supported runtime from the table below. ## Write the Configuration Profile Inline environment variables work for a script; an application needs configuration it can deploy. The SDK reads a small TOML file called a profile, named by the `CARACAL_CONFIG` environment variable. You write this file yourself, using the values from First Protected Call - the zone ID, the application ID, and the application client secret you copied during guided setup: ```toml zone_id = "" application_id = "" app_client_secret_file = "/path/to/anton-client-secret" [[credentials]] resource = "resource://openai" upstream_prefix = "https://api.openai.com" ``` Reading it line by line: * `zone_id` and `application_id` identify your program: it acts as the Anton application inside your zone. * `app_client_secret_file` points to a file containing the client secret, readable only by your user. The SDK never searches for credentials on its own; the path must be explicit. If you lost the secret, reveal it again from the application's detail panel in the console (each reveal is audited). * The `[[credentials]]` entry declares which resource this program uses. Its key is the resource identifier, not the upstream URL. On the local stack the SDK's built-in defaults already point at the right service URLs, so this is the whole file. For cloud or custom deployments you would add `sts_url`, `coordinator_url`, and `gateway_url`; the full schema is in [Configure Workloads](/v1.0/runtime-console/config-file/). :::note[Key takeaway] One profile serves your whole service. It names the application your program acts as - not each agent inside it. Every run executes under this one application identity, so you do not register an application per agent. ::: Point the examples at the profile and choose what they call. When `CARACAL_CONFIG` is set, the SDK reads exactly that profile - the identity variables from the previous page are no longer needed: ```sh export CARACAL_CONFIG=/path/to/caracal.toml export CARACAL_RESOURCE_ID=resource://openai export CARACAL_RESOURCE_PATH=/v1/models export CARACAL_RESOURCE_SCOPE=openai:models ``` ```powershell $env:CARACAL_CONFIG = "C:\path\to\caracal.toml" $env:CARACAL_RESOURCE_ID = "resource://openai" $env:CARACAL_RESOURCE_PATH = "/v1/models" $env:CARACAL_RESOURCE_SCOPE = "openai:models" ``` ## Install the SDK Install the package for your application language. | Language | Install command | Runtime requirement | | ---------- | --------------------------------------------------------- | ------------------- | | TypeScript | `npm install @caracalai/sdk` or `pnpm add @caracalai/sdk` | Node.js 22+ | | Python | `pip install caracalai-sdk` | Python 3.12+ | | Go | `go get github.com/garudex-labs/caracal/packages/sdk/go` | Go 1.26+ | ## Make the Call ```typescript import { Caracal } from '@caracalai/sdk' const caracal = new Caracal() const resourceId = process.env.CARACAL_RESOURCE_ID const resourcePath = process.env.CARACAL_RESOURCE_PATH const resourceScope = process.env.CARACAL_RESOURCE_SCOPE if (!resourceId) throw new Error('CARACAL_RESOURCE_ID is required') if (!resourcePath) throw new Error('CARACAL_RESOURCE_PATH is required') if (!resourceScope) throw new Error('CARACAL_RESOURCE_SCOPE is required') const governedFetch = caracal.applicationTransport(resourceId, { scopes: [resourceScope], }) const target = caracal.gatewayRequest(resourceId, resourcePath) try { const response = await governedFetch(target.url, { method: 'GET', }) if (!response.ok) throw new Error(`protected call failed: ${response.status}`) console.log(await response.text()) } finally { await caracal.close() } ``` ```python import asyncio import os from caracalai import Caracal caracal = Caracal() resource_id = os.environ["CARACAL_RESOURCE_ID"] resource_path = os.environ["CARACAL_RESOURCE_PATH"] resource_scope = os.environ["CARACAL_RESOURCE_SCOPE"] async def main(): target = caracal.gateway_request(resource_id, resource_path) try: async with caracal.application_transport( resource_id, scopes=[resource_scope], ) as governed: response = await governed.get(target.url) response.raise_for_status() print(response.text) finally: await caracal.aclose() asyncio.run(main()) ``` ```go package main import ( "fmt" "io" "os" caracal "github.com/garudex-labs/caracal/packages/sdk/go" ) func main() { c, err := caracal.New() if err != nil { panic(err) } resourceID := os.Getenv("CARACAL_RESOURCE_ID") resourcePath := os.Getenv("CARACAL_RESOURCE_PATH") resourceScope := os.Getenv("CARACAL_RESOURCE_SCOPE") if resourceID == "" || resourcePath == "" || resourceScope == "" { panic("CARACAL_RESOURCE_ID, CARACAL_RESOURCE_PATH, and CARACAL_RESOURCE_SCOPE are required") } defer c.Close() // nil base: the SDK constructs its own HTTP client. governed, err := c.ApplicationTransport(nil, resourceID, caracal.ApplicationTransportOptions{ Scopes: []string{resourceScope}, }) if err != nil { panic(err) } target, err := c.GatewayRequest(resourceID, resourcePath) if err != nil { panic(err) } resp, err := governed.Get(target.URL) if err != nil { panic(err) } defer resp.Body.Close() if resp.StatusCode < 200 || resp.StatusCode >= 300 { panic(fmt.Sprintf("protected call failed: %s", resp.Status)) } body, _ := io.ReadAll(resp.Body) fmt.Println(string(body)) } ``` ## Expected Outcome Run the example in your selected language. It should print the provider's JSON model list - the same response your first script returned - and create matching authorization and Gateway action-result events in **Audit**. A constructor or profile error means no request reached Caracal; a policy denial appears in Audit with a request ID. ## What Each Call Does All three examples follow the same four calls; only the naming convention changes per language: * `new Caracal()` / `Caracal()` / `New()` loads the profile named by `CARACAL_CONFIG` (or, without one, `CARACAL_*` environment variables, as in First Protected Call). It fails immediately if the configuration is missing or unreadable. * `applicationTransport()` / `application_transport()` / `ApplicationTransport()` returns an HTTP transport pinned to one resource. Each call asks Caracal's token service (the **STS**, the component that evaluates policy and issues mandates) for authority with exactly the scopes you list and gets a fresh, short-lived, replay-protected mandate for the request. To do this it opens a **session** - Caracal's record of one governed run of your program - and narrows the requested scope through a **delegation**, a hand-off that can only ever shrink permissions, never grow them. The SDK manages both for you; you will meet them properly in [Concepts](/v1.0/concepts/). * `gatewayRequest()` / `gateway_request()` / `GatewayRequest()` builds the Gateway URL and the `X-Caracal-Resource` routing header that names the resource. * `close()` / `aclose()` / `Close()` ends SDK-owned sessions and background work cleanly. `applicationTransport()` is the smallest application-owned call path and the right starting point. Agent frameworks that already run under delegated authority use `session()` with a scoped transport instead; read your language's SDK guide ([TypeScript](/v1.0/guides/sdk-typescript/), [Python](/v1.0/guides/sdk-python/), [Go](/v1.0/guides/sdk-go/)) before switching paths. If policy denies the exchange, the SDK surfaces the error from the STS. Open **Audit** in the web console with the request ID to see the determining policy and diagnostics. ## Common Mistakes * Use the resource identifier (`resource://openai`), not the upstream URL, as the `[[credentials]]` entry key. * Give `gatewayRequest` a relative path; it rejects absolute URLs and dot segments. * Request at least one scope, and keep every scope within what the active policy allows. * Close the client on shutdown so sessions and background work terminate cleanly. ## Next Step If anything failed - the profile would not load, the exchange was denied, the Gateway refused the call, or audit shows nothing - walk [First-Run Troubleshooting](/v1.0/get-started/first-run-troubleshooting/). Otherwise you have completed Get Started: continue to [Tutorials](/v1.0/tutorials/) to protect a real API, or read [Concepts](/v1.0/concepts/) for the full model behind what you just built. --- # First-Run Troubleshooting # URL: https://docs.caracal.run/v1.0/get-started/first-run-troubleshooting/ # Markdown: https://docs.caracal.run/markdown/v1.0/get-started/first-run-troubleshooting.md # Type: workflow # Concepts: # Requires: --- Use this page when a Get Started step fails. Every request you make crosses the same boundaries in the same order, so diagnose in that order and stop at the first boundary that fails - do not change policy, credentials, and routing at the same time: ```mermaid flowchart LR Ready[Stack readiness] --> SignIn[Console sign-in] --> Identity[Application identity] --> STS[Token service] --> GW[Gateway] --> Up[Upstream service] --> Audit[Audit trail] ``` Run the failing step once and keep its request ID or exact error, then start at the matching section below. For production incidents and deeper operational diagnosis, use [Troubleshoot by Symptom](/v1.0/operations/troubleshooting/). ## Readiness Failures The same command works in every shell: ```sh caracal status --ready --json ``` | Symptom | Check | | ------------------------ | ------------------------------------------------------------------------------------------------------------------ | | Docker command fails | Confirm Docker Desktop or Docker Engine is running and `docker compose version` succeeds. | | A service is not ready | Wait for the dependency named in the JSON output, then rerun readiness. | | Ports are already in use | Stop the local process holding the port; the local port map is in [Defaults and Limits](/v1.0/reference/defaults-and-limits/#ports). | | Stack state looks stale | Run `caracal down`, then `caracal up`. Use `caracal purge` only when you intentionally want to remove local state. | ## Console Sign-In | Symptom | Check | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Sign-up is rejected or the access-denied page appears | The packaged console closes registration by default and shows one uniform page for every allowlist denial. On the runtime host, run `caracal allowlist list` to inspect entries, then `caracal allowlist add ` or `caracal allowlist unlock ` - see [Control Console Access](/v1.0/runtime-console/console-access/). Make sure a sign-in method is configured in `$CARACAL_HOME/caracal.env` - see [Enable Console Sign-In](/v1.0/get-started/install-caracal/#enable-console-sign-in). | | Google or GitHub buttons are missing | Set both the client ID and client secret for the provider, then rerun `caracal up`. | | The provider rejects sign-in with a redirect URI mismatch (Google shows `Error 400: redirect_uri_mismatch`) | The OAuth client does not list the console's callback URL. The packaged console signs in through `http://localhost:3001/api/auth/callback/google` (or `.../github`); a source-checkout `caracal web` session uses port `3002` instead, and a custom `CARACAL_WEB_URL` moves the origin with it. Add the packaged callback to the OAuth client - Google accepts several redirect URIs, while a GitHub OAuth app takes one callback URL per app - then retry; no stack restart is needed. | | Password sign-in is blocked pending verification | The packaged console requires a verified email. Confirm `CARACAL_SMTP_URL` and `CARACAL_SMTP_FROM` are set and the verification message was delivered. | ## Application Identity Issues | Symptom | Check | | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `Caracal.fromEnv: provide CARACAL_APP_CLIENT_SECRET` or a similar constructor error | Export `CARACAL_ZONE_ID`, `CARACAL_APPLICATION_ID`, and `CARACAL_APP_CLIENT_SECRET` exactly as in [Give the Agent Its Identity](/v1.0/get-started/first-protected-call/#give-the-agent-its-identity), or set `CARACAL_CONFIG` to a complete profile. | | SDK cannot load configuration | `CARACAL_CONFIG` must name an existing profile file; a missing file at that path is an error, not a fallthrough. | | Authentication is rejected (401) | The client secret no longer matches the application - reveal the current value from the application's detail panel, or rotate it and update your environment or secret file. | | Secret file is rejected | Ensure the file named by `app_client_secret_file` exists and is readable only by the current user. | ## Lost Application Secret The application client secret is held sealed server-side. If you lose your local copy, reveal it again from the application's detail panel in the web console - each reveal is recorded in the zone audit timeline - or rotate the secret and update your environment or secret file with the new value. ## Token Service Denials (STS 403) The STS is Caracal's token service: it checks policy and issues mandates. A 403 from it means your program authenticated successfully but policy did not allow what it asked for. | Check | Fix | | ----------------- | ------------------------------------------------------------------------------------------- | | Active policy set | Activate the starter policy set created by guided setup. | | Resource ID | Use the resource ID you created in guided setup. | | Scopes | Request only scopes covered by the starter policy. | | Audit request ID | Open web console **Audit** with the request ID to see the policy diagnostic. | ## Gateway 403 A Gateway 403 means policy already said yes and a mandate was issued, but the Gateway rejected the request before forwarding it upstream. | Check | Fix | | -------------------- | ------------------------------------------------------------------------- | | Authorization header | The SDK transport attaches `Authorization: Bearer `; send requests through it rather than a plain HTTP client. | | Resource header | The SDK sets `X-Caracal-Resource` from the resource ID you pass; confirm it matches guided setup. | | Mandate freshness | Mandates are short-lived and single-use by design; rerun the example to mint a fresh one. | | Revocation | Confirm the session, application, or delegation was not revoked. | | Route binding | Confirm the resource has the Gateway route and upstream URL you intended. | ## Upstream Unreachable | Symptom | Check | | --------------------- | -------------------------------------------------------------------------------------------------- | | Connection refused | The upstream service is not listening, or the Gateway cannot reach that host and port. | | DNS failure | Use a hostname visible from the Gateway container, not only from the host shell. | | Demo upstream missing | Confirm the resource's upstream URL is reachable from the Gateway container and the provider key sealed on the provider is current. | | Wrong path | Use a known-good path on the upstream before trying custom paths. | ## Missing Audit Events | Check | Fix | | ------------------------------------ | --------------------------------------------------------------------------------- | | Wrong request ID | Copy the request ID from the STS, Gateway, SDK, or web console output. | | Wrong zone | Select the zone used by guided setup. | | Request never reached STS or Gateway | Confirm the example used your configuration and the Gateway URL it derives. | | Audit ingestion lag | Wait briefly and refresh the web console audit view. | ## Expected Outcome Repeat only the failed step. Success means readiness passes, the protected call returns the upstream response, and Audit contains both the authorization decision and the Gateway result under the same request trace. ## Next Step After the first run succeeds, continue with [Tutorials](/v1.0/tutorials/) or the language-specific [SDK guides](/v1.0/guides/). --- # Tutorials # URL: https://docs.caracal.run/v1.0/tutorials/ # Markdown: https://docs.caracal.run/markdown/v1.0/tutorials.md # Type: workflow # Concepts: # Requires: --- In [Get Started](/v1.0/get-started/) your agent made one protected call to an LLM provider and you proved the full enforcement path. Tutorials make that setup yours. Working through them in order, you will: 1. **Protect a real API** - put Caracal in front of an HTTP service you actually run, and prove Caracal denies what policy does not allow. 2. **Make your runs identifiable** - label the calls your code makes, so you can tell one agent's work from another's in the audit trail. 3. **Debug on your own** - follow any request through the decision trail and answer "why was this allowed or denied?" without guessing. 4. **Choose your production path** - pick the one integration guide that matches how you will deploy for real. Each tutorial builds on the previous one and tells you what to expect after every step. None of them repeats installation or first-call setup. ## Tutorial Path ```mermaid flowchart LR API[Protect a real API] SDK[Make runs identifiable] Trace[Debug a request] Path[Choose production path] API --> SDK --> Trace --> Path ``` | You will be able to... | Tutorial | | --- | --- | | Put Caracal in front of one HTTP service you own, and prove both the allow and the deny. | [Protect Your First Real API](./protect-an-api/) | | Tell your agents' runs apart in the audit trail using labels. | [Make Runs Identifiable with Labels](./connect-an-agent/) | | Explain any request's outcome from its decision trace. | [Trace One Protected Request](./inspect-a-run/) | | Commit to one production integration boundary. | [Choose Your Production Integration Path](./choose-production-path/) | ## Before You Begin You need the working setup from Get Started: * a running stack - if you cleaned up earlier, run `caracal up` and wait for `caracal status --ready`; * your zone, the Anton application, and its active policy; * the working SDK example and `caracal.toml` profile from [Add SDK to Your App](/v1.0/get-started/add-sdk-to-your-app/). :::note If any prerequisite is missing, return to [Get Started](/v1.0/get-started/). Tutorials assume the evaluator stack already works. ::: ## After Tutorials Use [Guides](/v1.0/guides/) for task-specific implementation details, [SDKs](/v1.0/sdks/) for package APIs, and [Concepts](/v1.0/concepts/) when you need the reference model behind policy, delegation, revocation, or audit. --- # Protect Your First Real API # URL: https://docs.caracal.run/v1.0/tutorials/protect-an-api/ # Markdown: https://docs.caracal.run/markdown/v1.0/tutorials/protect-an-api.md # Type: workflow # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' In Get Started, your agent called an LLM provider through Caracal. In this tutorial you protect a service that matters to you - and, just as important, you watch Caracal *refuse* a request that policy does not allow. By the end you will have proven both halves of enforcement: the allow and the deny. You will: 1. pick one HTTP service the Gateway can reach; 2. register it in Caracal as a resource named `resource://pipernet`; 3. write one policy rule allowing your application to read it; 4. prove the allowed call works end to end; 5. prove a disallowed request is denied before it ever reaches your service. ## Prerequisites * Complete [Add SDK to Your App](/v1.0/get-started/add-sdk-to-your-app/). * Keep the stack running (`caracal status --ready` succeeds) and your `caracal.toml` profile from Get Started. ## 1. Pick the Service to Protect Choose one HTTP endpoint you own: an internal REST API, a staging service, or a route on a provider you use. Two things matter: * **The Gateway must be able to reach it.** The upstream URL is resolved from inside the Gateway's container, so `localhost` there means the Gateway itself. Use a hostname or IP the Gateway can resolve - the same rule you met in [First Protected Call](/v1.0/get-started/first-protected-call/). * **Know whether it needs its own credential.** In Get Started the LLM provider needed one, held by a Caracal provider. Internal services often accept any request. Note which kind yours is - it decides one form field in the next step. No suitable service handy? Start another disposable container on the stack's network and treat it as your "real" API; every step still teaches the same skills: ```sh docker run --rm -d --name pipernetUpstream --network caracalData nginx:stable-alpine ``` Its Gateway-visible URL is `http://pipernetUpstream:80`. **After this step:** you have one upstream URL and you know whether it needs a credential. ## 2. Register It as a Resource A resource is Caracal's record of what it protects and where to forward verified requests - you created one in guided setup; now you create one from the everyday form. In the web console at [http://localhost:3001](http://localhost:3001), open **Resources** and create: | Field | Value | Why | | --- | --- | --- | | Resource identifier | `resource://pipernet` | The stable name everything else refers to: policy, SDK code, Gateway headers, and audit events. | | Scopes | `pipernet:read` | The one named permission you will allow. Add more actions later, one at a time. | | Upstream URL | Your Gateway-reachable URL from step 1 | Where the Gateway forwards verified requests. | | Provider | See below | How the Gateway attaches your service's own credential, if it needs one. | You met providers in Get Started, where one held your LLM key: a **provider** is a credential source you configure once - the upstream's API key or OAuth client - so the Gateway can attach that credential to verified requests on the way through. Your program never sees it. * If your service needs no credential (including the disposable container), choose the `None` provider: the Gateway still enforces policy and records audit, it just attaches nothing. * If your service needs a key or token, create a provider for it first and select it on the resource. [Define Resources and Providers](/v1.0/guides/resources-providers/) covers the provider forms; you can also start with `None` now and attach a provider later. **After this step:** `resource://pipernet` appears in the Resources list. Nothing can call it yet - that is the point of the next step. ## 3. Allow Your Application to Read It Caracal denies everything not explicitly allowed, so a new resource starts unreachable. Open **Policies** and add the rule allowing your existing application (Anton from Get Started) to request `pipernet:read` on `resource://pipernet`, then activate the change. A good first rule names exactly four things - the application, the resource, the scopes, and the zone - and nothing more. [Author Policy Data](/v1.0/guides/author-policy/) and [Activate a Policy Set](/v1.0/guides/activate-policy-set/) go deeper when you need real policy structure. **After this step:** the active policy set includes your new rule. The Policies page shows which set is active. ## 4. Prove the Allowed Call Your SDK example from Get Started is already wired for this - it reads its target from environment variables. Point it at the new resource: ```sh export CARACAL_CONFIG=/path/to/caracal.toml export CARACAL_RESOURCE_ID=resource://pipernet export CARACAL_RESOURCE_PATH=/ export CARACAL_RESOURCE_SCOPE=pipernet:read ``` ```powershell $env:CARACAL_CONFIG = "C:\path\to\caracal.toml" $env:CARACAL_RESOURCE_ID = "resource://pipernet" $env:CARACAL_RESOURCE_PATH = "/" $env:CARACAL_RESOURCE_SCOPE = "pipernet:read" ``` Run the example. It should print your service's response - the same code that reached the LLM provider now reaches your real API, because authority comes from configuration and policy, not from the code. **After this step:** web console **Audit** shows two events for your request ID: the authorization decision and the Gateway's action result. ## 5. Prove the Deny Enforcement you have never seen fail is enforcement you cannot trust. Request a permission your policy does not allow: ```sh export CARACAL_RESOURCE_SCOPE=pipernet:write ``` ```powershell $env:CARACAL_RESOURCE_SCOPE = "pipernet:write" ``` Run the example again. This time it fails: policy allows `pipernet:read` only, so Caracal's token service refuses to issue a mandate for `pipernet:write`, and the request never reaches your service. The SDK surfaces the denial with a request ID. Set the scope back to `pipernet:read` afterward. **After this step:** Audit contains a deny decision with your request ID. Keep that ID - [Trace One Protected Request](../inspect-a-run/) dissects one just like it two steps from now. ## Expected Outcome The same SDK flow that reached the LLM provider now reaches your real API through the Gateway, and you have seen both outcomes in Audit: an allow with an action result, and a deny that stopped before the upstream. You did not change a line of code to switch targets - only configuration and policy. ## Common Mistakes * Do not reuse `resource://openai` for a different target; each protected service gets its own identifier. * Do not put an upstream secret in application code; bind a credential provider to the resource instead. * Do not treat a successful direct call to the upstream as enforcement proof - only the Gateway path checks anything. ## Next Step Continue with [Make Runs Identifiable with Labels](../connect-an-agent/) to make your app's runs identifiable in the audit trail. --- # Make Runs Identifiable with Labels # URL: https://docs.caracal.run/v1.0/tutorials/connect-an-agent/ # Markdown: https://docs.caracal.run/markdown/v1.0/tutorials/connect-an-agent.md # Type: workflow # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' Your SDK example works, but imagine it a month from now: one application identity backing a dozen agents, all making protected calls. When something misbehaves, which run was it? This tutorial solves that by adding a **label** - a short tag your code attaches to its runs - so the audit trail can tell your agents' work apart. You will: 1. point your working example at the resource from the previous tutorial; 2. add one label to the transport call; 3. find that label in the audit trail and understand what it does and does not do. ## Prerequisites * Complete [Protect Your First Real API](../protect-an-api/). * Keep the working profile and SDK example from Get Started. ## 1. Point the Example at Your Real Resource If your shell still has the variables from the previous tutorial, you are already set. Otherwise: ```sh export CARACAL_CONFIG=/path/to/caracal.toml export CARACAL_RESOURCE_ID=resource://pipernet export CARACAL_RESOURCE_PATH=/ export CARACAL_RESOURCE_SCOPE=pipernet:read ``` ```powershell $env:CARACAL_CONFIG = "C:\path\to\caracal.toml" $env:CARACAL_RESOURCE_ID = "resource://pipernet" $env:CARACAL_RESOURCE_PATH = "/" $env:CARACAL_RESOURCE_SCOPE = "pipernet:read" ``` Run the example once, unchanged, to confirm the baseline still works. **After this step:** the example prints your service's response, exactly as at the end of the previous tutorial. ## 2. Add a Label to the Transport Keep the complete example from [Add SDK to Your App](/v1.0/get-started/add-sdk-to-your-app/). Change only the transport options: add the label `pipernet-reader`. ```typescript const governedFetch = caracal.applicationTransport(resourceId, { scopes: [resourceScope], labels: ['pipernet-reader'], }) ``` ```python async with caracal.application_transport( resource_id, scopes=[resource_scope], labels=["pipernet-reader"], ) as governed: response = await governed.get(target.url) ``` ```go governed, err := c.ApplicationTransport(nil, resourceID, caracal.ApplicationTransportOptions{ Scopes: []string{resourceScope}, Labels: []string{"pipernet-reader"}, }) if err != nil { panic(err) } ``` Run the example again. **After this step:** the call succeeds exactly as before - the label changes nothing about what is allowed. It changes what is *recorded*. ## 3. Find the Label in Audit Open web console **Audit** and locate the request you just made. A complete Gateway-routed call produces two events - the authorization decision and the Gateway's action result - and the run now carries your `pipernet-reader` label alongside its session IDs. That is the payoff. In an app with many agents, give each agent role its own label (`report-writer`, `calendar-sync`, ...) and the audit trail stays separable while everything runs under one application identity. Two boundaries to keep straight: * **Labels describe; they never authorize.** Access still comes from scopes and the active policy. A label like `admin` grants nothing. * **Labels group; sessions identify.** As introduced in [What Each Call Does](/v1.0/get-started/add-sdk-to-your-app/#what-each-call-does), the SDK opens a session for each run. Many runs can share a label; each run's session ID is unique. When exact attribution matters, use the session ID from Audit. **After this step:** you can find a run by its label in Audit and read its session IDs from the event detail. ## Expected Outcome Audit shows your application, the `pipernet-reader` label, distinct session IDs, and the Gateway result for `resource://pipernet`. You know which of those facts identifies the run (the session ID) and which merely describes it (the label). ## Common Mistakes * Labels do not authorize access. Keep `pipernet:read` in the active policy. * Do not wrap `applicationTransport` in a session of your own expecting that session to grant resource authority; the transport provisions its own bounded path. * Do not reach for per-agent applications to get attribution; labels and session IDs exist so one application serves all your agents. ## Going Further If your app hands work from one agent to another and the receiving agent should hold *less* authority, that hand-off is a delegation - the narrowing mechanism you met briefly in Get Started. [Implement Multi-Agent Delegation](/v1.0/guides/delegation/) covers it after this tutorial path. ## Next Step Continue with [Trace One Protected Request](../inspect-a-run/) to dissect the request you just made - and the deny you produced earlier. --- # Trace One Protected Request # URL: https://docs.caracal.run/v1.0/tutorials/inspect-a-run/ # Markdown: https://docs.caracal.run/markdown/v1.0/tutorials/inspect-a-run.md # Type: workflow # Concepts: # Requires: --- The previous tutorials left you with two real events: an allowed call to `resource://pipernet` and a deliberate deny for `pipernet:write`. This tutorial teaches the skill you will use every time something behaves unexpectedly: proving *why* a request got its result, from evidence instead of guesswork. You will investigate both of your requests: 1. find each request in the audit trail; 2. read the pair of events an allowed call produces; 3. open the decision trace and answer "which rule decided this?"; 4. know where session and delegation context lives for deeper questions. ## Prerequisites * Complete [Make Runs Identifiable with Labels](../connect-an-agent/). * Have the request ID from your deny in [Protect Your First Real API](../protect-an-api/#5-prove-the-deny), or reproduce it - denials are free. * Keep the same zone selected in the web console. ## 1. Find Your Requests Open **Audit** in the web console at [http://localhost:3001](http://localhost:3001). The live view lists recent events immediately and supports filters, reload, and pause/resume. Every protected request carries a **request ID** - one identifier stamped on everything that request touched, across services. You can copy it from the SDK error, the audit event, or the trace view. Find two entries: * your labeled allowed call from the previous tutorial (search for the `pipernet-reader` label); * your `pipernet:write` deny. **After this step:** you have both events open and both request IDs in hand. ## 2. Read What an Allowed Call Produces Your allowed call produced two events, and the pair matters: | Event | Written by | What it proves | | --- | --- | --- | | Authorization decision | The token service (STS) | Policy was evaluated for this application, resource, scopes, and session context - and allowed it. | | Action result | The Gateway | The request was verified, forwarded, and the upstream answered with this result. | The pair is your completeness check. An authorization event *without* an action result means policy said yes but the request never completed through the Gateway - look at the Gateway URL, the `X-Caracal-Resource` header, or upstream reachability, not at policy. Now look at your deny: there is no action-result event at all. The request was stopped at the decision, before your service ever saw it - which is exactly what step 5 of the earlier tutorial claimed. **After this step:** given any request ID, you can say whether it was decided, whether it completed, and where it stopped. ## 3. Open the Decision Trace Press the trace action on your denied event. The decision trace is the full story of one request: * the application and its sessions; * the resource and the scopes it requested; * the active policy set version and the rules involved; * the decision - allow, deny, or partial - with policy diagnostics; * delegation constraints, when the request used delegated authority. For your deny, the trace shows the request asked for `pipernet:write` while the active rule allows only `pipernet:read`. That is the answer format the trace always gives you: not just *what* happened but *which rule and which version* decided it. Do the same for your allowed request and confirm the opposite: the rule that matched, the policy set version, and the label you attached. **After this step:** you can name the deciding rule and policy version for both of your requests. ## 4. Know Where the Deeper Context Lives Two follow-up questions come up in real investigations: * **"What else did this run do?"** Open **Sessions** and find the session IDs from your trace. A session is the record of one governed run, so its timeline groups everything that run touched. * **"Who handed authority to whom?"** The Sessions page also holds the delegation view. Each delegation edge records the source session, target session, narrowed scopes, lifetime, and resource constraints. Your labeled call has one - the SDK created it when it narrowed the transport's authority, as described in [What Each Call Does](/v1.0/get-started/add-sdk-to-your-app/#what-each-call-does). One more behavior worth knowing before production: revocation. Revoking a session, grant, delegation, or an application's emergency state invalidates the authority anchored to it - Gateway-routed calls check revocation before accepting a request and while streaming responses, and Audit records the revocation and marks interrupted results. You do not need to exercise it now; just know the evidence will be in the same trail you have been reading. ## Expected Outcome For any request ID you can now identify the application, session, resource, requested scopes, active policy version, final decision, and Gateway result - and for a failure, you can say which boundary stopped it. This replaces guesswork with the same evidence trail you will rely on in production. ## Common Mistakes * Do not debug from application logs alone; the decision trace is the source of truth for authorization outcomes. * Do not read a missing action result as a policy problem - policy already said yes; the request failed after the decision. * If a Federated user appears in a trace, treat it as attribution, not authorization - resource authority still comes from the application, policy, and delegation. ## Next Step Continue with [Choose Your Production Integration Path](../choose-production-path/). --- # Choose Your Production Integration Path # URL: https://docs.caracal.run/v1.0/tutorials/choose-production-path/ # Markdown: https://docs.caracal.run/markdown/v1.0/tutorials/choose-production-path.md # Type: workflow # Concepts: # Requires: --- You have protected a real service, made your runs identifiable, and traced both an allow and a deny. One decision remains before production work starts: **where enforcement happens** in your architecture. This page helps you make that decision once, deliberately. So far, every call was verified by the Gateway - the checkpoint in front of your service. That is one of two possible boundaries: | Boundary | How it works | Choose it when | | --- | --- | --- | | **Gateway-routed** (what you have been using) | Requests travel through the Gateway, which verifies each mandate, attaches upstream credentials, and records the action result centrally. | Your protected target is HTTP and you want enforcement, credential brokering, and audit in one place your services never have to implement. | | **In-process adapter** | Your own service verifies mandates inside its process using a framework adapter, then records its own action results. | You own the service's code, and requests should not detour through a central checkpoint - for latency, topology, or deployment reasons. | Both enforce the same contract; they differ in *where* verification runs and *who* writes the action-result audit. Start with the Gateway boundary unless you have a concrete reason not to - it is the path you have already proven end to end. :::caution[One boundary first] Pick one boundary and take it to production before adding a second pattern, delegation flows, or approvals. Every mixed-pattern debugging session starts with someone who skipped this step. ::: ## Prerequisite Complete [Trace One Protected Request](../inspect-a-run/) or be able to explain an equivalent request from application to resource. ## Choose by Boundary **Gateway-routed** - requests detour through the central checkpoint: | If your production path is... | Continue with | | --- | --- | | Route HTTP traffic through the Caracal Gateway | [Protect a Gateway-Routed HTTP API](/v1.0/guides/protect-gateway-http/) | **In-process adapter** - your service verifies mandates itself: | If your production path is... | Continue with | | --- | --- | | Protect an Express resource server in process | [Protect an Express App](/v1.0/guides/protect-express/) | | Protect a FastAPI or Starlette service in process | [Protect a FastAPI App](/v1.0/guides/protect-fastapi/) | | Protect a FastMCP server in process | [Protect a FastMCP App](/v1.0/guides/protect-fastmcp/) | | Protect a Go `net/http` service in process | [Protect a Go net/http Service](/v1.0/guides/protect-nethttp/) | | Protect an MCP server without a dedicated adapter | [Protect an MCP Server](/v1.0/guides/protect-mcp/) | Two paths sit **alongside** the boundary choice rather than replacing it - how your application code obtains authority, and how existing CLIs receive provider credentials: | If you also need to... | Continue with | | --- | --- | | Add Caracal sessions and Gateway calls to app code | [TypeScript SDK](/v1.0/guides/sdk-typescript/), [Python SDK](/v1.0/guides/sdk-python/), or [Go SDK](/v1.0/guides/sdk-go/) | | Launch an existing CLI or worker with injected provider credentials | [Run an Agent with caracal run](/v1.0/guides/runtime-run/) | ## Add Capabilities After the Boundary Works These build on a verified boundary; none of them replaces it: | When you need to... | Continue with | | --- | --- | | Model zones, apps, resources, and customer boundaries | [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/) | | Configure provider credentials or OAuth | [Define Resources and Providers](/v1.0/guides/resources-providers/) and [Provider Recipes](/v1.0/guides/provider-recipes/) | | Debug denies or unexpected allows | [Debug Authorization Decisions](/v1.0/guides/authorize-access/) | | Hand narrowed authority between agents | [Implement Multi-Agent Delegation](/v1.0/guides/delegation/) | | Export or query audit evidence | [Tail and Query the Audit Stream](/v1.0/guides/audit-stream/) | | Hold sensitive actions for human approval | [Human Approval](/v1.0/guides/human-approval/) | ## Choose by Team Role | Role | Start with | | --- | --- | | App engineer | SDK guide for your language, then the Gateway or adapter guide. | | Platform engineer | [Production Integration Patterns](/v1.0/guides/production-patterns/), then operations pages. | | Security reviewer | [Debug Authorization Decisions](/v1.0/guides/authorize-access/), [Audit and Request Traces](/v1.0/concepts/audit-ledger/), and [Review the Threat Model](/v1.0/security/threat-model/). | | Policy owner | [Author Policy Data](/v1.0/guides/author-policy/) and [Activate a Policy Set](/v1.0/guides/activate-policy-set/). | ## Expected Outcome You leave this page with one primary enforcement boundary and one implementation guide, recorded before anyone adds delegation, approvals, or alternative adapters. You have completed the tutorial path: a real protected API, identifiable runs, self-serve debugging, and a deliberate production direction. ## Next Step Open [Guides](/v1.0/guides/) and follow the path that matches your boundary. --- # Guides # URL: https://docs.caracal.run/v1.0/guides/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides.md # Type: page # Concepts: # Requires: --- Use Guides after [Get Started](/v1.0/get-started/) when you have a concrete integration job. These pages teach complete application-integrator and resource-server workflows; package and API pages remain the source for signatures and wire fields. ## When to use this section * **Application integrators** start with an SDK guide, then route outbound calls through Gateway or `caracal run`. * **Resource-server integrators** start with Gateway routing or the adapter matching their server framework. * **Platform integrators** use the modeling, resource/provider, policy, testing, and audit workflows before production traffic. ## Choose by Task | Task | Start with | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Map your architecture onto Caracal | [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/) | | Serve many of your own customers from one deployment | [Serve Your Own Customers](/v1.0/guides/serve-customers/) | | Define protected targets and upstream credentials | [Define Resources and Providers](/v1.0/guides/resources-providers/) and [Provider Recipes](/v1.0/guides/provider-recipes/) | | Write and activate authorization logic | [Author Policy Data](/v1.0/guides/author-policy/) and [Activate a Policy Set](/v1.0/guides/activate-policy-set/) | | Debug an authorization result | [Debug Authorization Decisions](/v1.0/guides/authorize-access/) | | Add Caracal to app code | [TypeScript SDK](/v1.0/guides/sdk-typescript/), [Python SDK](/v1.0/guides/sdk-python/), or [Go SDK](/v1.0/guides/sdk-go/) | | Run an existing process with Caracal tokens | [Run an Agent with caracal run](/v1.0/guides/runtime-run/) | | Protect a Gateway-routed HTTP upstream | [Protect a Gateway-Routed HTTP API](/v1.0/guides/protect-gateway-http/) | | Protect a resource server in process | [Express](/v1.0/guides/protect-express/), [FastAPI](/v1.0/guides/protect-fastapi/), [FastMCP](/v1.0/guides/protect-fastmcp/), [Go net/http](/v1.0/guides/protect-nethttp/), or [MCP server](/v1.0/guides/protect-mcp/) | | Add Delegation, audit export, or Approval | [Delegation](/v1.0/guides/delegation/), [Audit Stream](/v1.0/guides/audit-stream/), or [Human Approval](/v1.0/guides/human-approval/) | | Notify approvers when a hold is raised | [Approval Notifications](/v1.0/guides/approval-notifications/) | | Make retries safe for side-effecting actions | [Safe Retries and Idempotency](/v1.0/guides/idempotency/) | | Test an integration without a live stack | [Test Caracal Integrations](/v1.0/guides/testing/) | | Govern LangChain, LangGraph, or CrewAI | [Govern Agent Frameworks](/v1.0/guides/frameworks/) | | Plan a production integration | [Production Integration Patterns](/v1.0/guides/production-patterns/) | ## Recommended Order ```mermaid flowchart LR Model["Model app"] Resource["Define resources and providers"] Policy["Author policy"] Activate["Activate policy"] App["Integrate app"] Protect["Protect boundary"] Debug["Trace and debug"] Model --> Resource --> Policy --> Activate --> App --> Protect --> Debug ``` ## Surface Boundaries Use the right surface for each task: | Surface | Use for | | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `caracal up`, `down`, `status`, `upgrade`, `purge`, `allowlist`, and `run` | Local runtime lifecycle, Console sign-in admission, and subprocess injection. | | Console | Human-facing zone, application, provider, resource, policy, session, audit, explanation, delegation, and diagnostic workflows. | | Admin API and `@caracalai/admin` | Automation for the same control-plane objects. | | SDKs and adapters | Application integration, context propagation, mandate exchange, and mandate verification. | ## Before You Start You need a running Caracal runtime, a zone, an application, at least one resource, and an active policy set. [First Protected Call](/v1.0/get-started/first-protected-call/) creates that baseline. ## Expected Outcome After following one path through the table, an allowed call reaches exactly one protected resource, a denied call fails before protected work runs, and both outcomes can be found by request ID in **Audit**. :::caution[Common mistake] Use the linked [SDK and package reference](/v1.0/sdks/) for exact release signatures. Guides own sequencing, boundary choices, validation, and recovery - not duplicate API catalogs. ::: ## Next Step Choose the first unfinished job in **Choose by Task**. For a new integration, start with [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/). --- # Model Your Application in Caracal # URL: https://docs.caracal.run/v1.0/guides/modeling-recipes/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/modeling-recipes.md # Type: page # Concepts: # Requires: --- Use this guide before creating production objects or when an existing zone has become hard to reason about. It turns deployment boundaries into a reviewed model; the [concept pages](/v1.0/concepts/) remain the canonical definitions. ## Prerequisites * A list of workloads, protected upstreams, deployment environments, and policy owners. * A decision about which systems require separate signing keys and audit trails. * The stable actions each upstream exposes; do not start from hostnames or current credentials. The deliverable is a short model table naming each zone, application, resource, provider, and scope. Review it with application and resource-server owners before provisioning. Every recipe states the modeling decision, what maps to each noun, and the trade-off. Use the [Caracal Mental Model](/v1.0/concepts/model-overview/) as the vocabulary reference while you read. Three terms appear here before their own guides: a **grant** is policy data mapping application roles to resource scopes ([Resources and Grants](/v1.0/concepts/resource-grant/)); the **platform decision contract** is the fixed decision logic your policy data feeds ([Author Policy Data](/v1.0/guides/author-policy/)); a **DCR application** is a short-lived programmatically registered identity ([managed and DCR applications](/v1.0/concepts/principal/#managed-and-dcr-applications)). ## Choose the Zone Boundary The most common question is "what is a zone - an environment, a customer, a team, or a product area?" A zone is none of those by default. It is the isolation boundary that owns signing keys, policy sets, sessions, audit, and authority data. You decide what trust boundary it represents. ```mermaid flowchart TD Q1{"Independent signing keys,\npolicy activation, or audit?"} Q2{"Same trust boundary,\ndifferent upstreams?"} Q3{"Same upstream,\ndifferent actions?"} Zones[Separate zones] Resources[One zone,\nseparate resources] Scopes[One resource,\nseparate scopes] Q1 -- yes --> Zones Q1 -- no --> Q2 Q2 -- yes --> Resources Q2 -- no --> Q3 Q3 -- yes --> Scopes ``` :::note[FAQ] [What should a zone represent?](/v1.0/reference/faq/#faq-003) and [does this repository implement managed multi-tenancy?](/v1.0/reference/faq/#faq-004) ::: | Question | If yes, lean toward | | --------------------------------------------------------------------------- | ----------------------------- | | Must these workloads have independent signing keys and JWKS? | Separate zones | | Must a policy change for one never affect the other? | Separate zones | | Must audit and explain traces never mix? | Separate zones | | Do they share keys, policy owners, and audit, but call different upstreams? | One zone, separate resources | | Do they share an upstream but need different actions? | One resource, separate scopes | Keep [resource identifiers](/v1.0/concepts/resource-grant/) stable across zones so policy, grants, and audit refer to the same target even when upstream URLs differ. ## Single Application, Single Environment The simplest deployment: one team, one runtime, a handful of upstreams. | Noun | Mapping | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Zone | One zone for the whole deployment. | | Application | One managed application per durable service; one DCR application (a short-lived identity created through [Dynamic Client Registration](/v1.0/sdks/admin/#dynamic-client-registration-dcr)) per isolated, externally-launched identity (per tenant, job, or integration). | | Resource | One resource per protected upstream, with action-oriented scopes. | | Grant | One grant per application and Subject that may request a resource's scopes. | Trade-off: lowest operational overhead. Add zones only when you need key, policy, or audit isolation. ## Per-Environment Zones Separate production, staging, and development so a policy or key change in one cannot affect another. | Noun | Mapping | | ---------- | -------------------------------------------------------------------------------------------------------------------- | | Zone | One zone per environment: `prod`, `staging`, `dev`. | | Resource | The same stable identifier in each zone, such as `resource://pipernet`, pointing at that environment's upstream URL. | | Policy set | Authored and activated independently per zone. | | Keys | Each zone has its own signing key and JWKS. | Trade-off: clean blast-radius isolation and independent key rotation, at the cost of registering resources and activating policy in each zone. Automate this with the Admin API so environments stay consistent. In application code, one SDK client represents exactly one `(zone, application)` identity. A workload that acts as several applications, or talks to several zones, constructs one client per identity and routes work to the matching client - never swap the identity behind a live client, because its tokens, mandates, sessions, and shutdown state belong to that one identity. The mechanics live in the [SDK guides](/v1.0/guides/sdk-typescript/). ## Multiple Customers or Workspaces A platform with many customer workspaces and one shared agent service must choose how customer isolation maps onto Zones. This open-source product gives you the **Zone** as the isolation primitive; you provision and automate customer onboarding yourself through the Admin API. Managed tenant, team, and SSO lifecycle are not implemented in this repository. | Model | When to use | Trade-off | | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Zone per customer | Customers require isolated signing keys, isolated audit, and policy that one customer's change can never affect another's. | Strongest isolation; you automate per-zone provisioning and key rotation, and one shared agent service authenticates separately into each zone. | | Shared zone, resource per customer | Customers share a trust boundary and policy owner but target distinct upstreams or data sets. | Lower overhead; isolation is enforced by policy and grants, not by keys or audit separation. | | Shared zone, customer in policy input | Customer is a runtime attribute of the same resource, carried in the request and checked by policy. | Lowest overhead; relies entirely on policy correctness, so audit and keys are shared. | Decide on the strongest isolation a customer actually requires, then pick the least complex model that satisfies it. Do not encode customer identity into [scope names](/v1.0/concepts/resource-grant/); keep it in the zone, principal, or policy input. For the end-to-end pattern of serving many customers from one shared zone, see [Serve Your Own Customers](/v1.0/guides/serve-customers/). ## High-Sensitivity Resources For resources whose compromise is unacceptable - payouts, key material, production data deletion. | Option | Mapping | | -------------- | -------------------------------------------------------------------------------------------------------------- | | Dedicated zone | Put the sensitive resource in its own zone with distinct signing keys and a separate audit trail. | | Tight scopes | Split actions into the smallest scopes, such as `pipernet:read` and `pipernet:refund`, so grants stay minimal. | | Approval | Hold the sensitive action for a human decision through [Human Approval](/v1.0/guides/human-approval/). | Trade-off: a dedicated zone gives the cleanest audit and key separation; in-zone tight scopes plus approval are lighter and often enough. Combine them for the highest-risk targets. ## App-Only Agents and Delegated Upstream Accounts The principal is always your application; Sessions carry its labels and delegation context. The real fork is where the upstream credential lives. | Dimension | Shared broker credential | Connected upstream account | | ------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | Upstream credential | Shared credential the agent never sees: `oauth2_client_credentials`, `api_key`, or `bearer_token`. | A consented upstream account stored as a provider connection, shared across the Zone by default or bound to the exchange Subject: `oauth2_authorization_code`. | | Consent | None - the zone operator configures the credential once. | A human completes the provider's consent screen once for the shared account, or once per Subject when bound to a customer. | | Audit attribution | Application, Session, and delegation chain. | The same, plus the provider connection the exchange used. | Policy tells work apart with the principal registration method, Session ID, and labels documented in the [policy input contract](/v1.0/concepts/policy/#policy-input-contract). Authorization inputs are the application, labels, roles, Delegation, and confinement - never the Subject identifier. See [Identities and Applications](/v1.0/concepts/principal/). ## Connected Account or Shared Credential Choose how the upstream credential is held. | Choose | When | Provider kind | | -------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | Connected upstream account | The upstream call must act as a specific consented account rather than a shared credential. | `oauth2_authorization_code` | | Shared service credential | The agent acts as the application, not a specific account. | `oauth2_client_credentials`, `api_key`, or `bearer_token` | For connected accounts, Caracal owns `client_id`, `redirect_uri`, `state`, and PKCE; use the provider **Connect** action to mint a consent URL for a Subject, and revoke to disconnect it. Reconnecting the same Subject and provider replaces the active connection rather than duplicating it. The concrete field tables are in [Provider Recipes](/v1.0/guides/provider-recipes/). ## Validate the Model After you map your architecture, confirm it end to end before relying on it: * Author and activate a policy set that allows the intended application, Subject, resource, and scopes. * Run [Check Provider Readiness](/v1.0/examples/provider-preflight/) to verify resource-to-provider binding and that the active policy set returns `allow`. * Send a successful request, a denied request, and a revoked-session request, then confirm each has a clear audit trail in the web console. Expected result: every workload maps to one application identity at a time, every protected target has one stable resource identifier, credentials live on providers rather than applications, and each hard isolation requirement maps to a zone. :::caution[Failure point: invented user authority] Caracal does not authenticate customers or derive authorization from a user name. A Federated user exists only after an application exchanges a token from a registered Federated user issuer. For app-only work, use the application, Session labels, policy data, and Delegation described here. ::: ## Next Step Provision the reviewed model with [Define Resources and Providers](/v1.0/guides/resources-providers/), then author matching grant data with [Author Policy Data](/v1.0/guides/author-policy/). ## Related * [Zones](/v1.0/concepts/zone/) * [Identities and Applications](/v1.0/concepts/principal/) * [Resources and Grants](/v1.0/concepts/resource-grant/) * [Define Resources and Providers](/v1.0/guides/resources-providers/) * [Provider Recipes](/v1.0/guides/provider-recipes/) * [Production Integration Patterns](/v1.0/guides/production-patterns/) --- # Serve Your Own Customers # URL: https://docs.caracal.run/v1.0/guides/serve-customers/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/serve-customers.md # Type: page # Concepts: # Requires: --- [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/) compares the isolation models at a high level. This page is the end-to-end pattern for the most common case: your application is a single product that serves many of *its own* customers, and you want clean per-customer authority without standing up infrastructure for each customer. ## When to use this pattern Use one shared zone only when customers share signing keys, policy ownership, audit storage, and operational rate limits. If any customer requires cryptographic or audit isolation, use a zone per customer and automate provisioning yourself. ## Prerequisites * An application-owned, authenticated customer ID that is stable and non-personal. * A reviewed label vocabulary and grant/confinement data. * Capacity and audit filters tested against expected customer concurrency. * The policy-data vocabulary from [Author Policy Data](/v1.0/guides/author-policy/) - this page uses `grants` and `confinement` documents and the platform decision contract without re-introducing them. The short answer: run **one zone for your deployment** and carry **each customer on the work itself** through a `customer:` label that policy confines and audit can filter. Keep `customer_id` metadata only when Coordinator inspection needs an additional business key. Customer separation lives below the zone - in Sessions, labels, and Delegation - which is exactly where Caracal models per-actor authority. You do not create a zone per customer. :::note[FAQ] [What should a zone represent?](/v1.0/reference/faq/#faq-003) and [does this repository implement managed multi-tenancy?](/v1.0/reference/faq/#faq-004) ::: ## Where Each Customer Lives in the Model | Layer | Owns | Per-customer? | | --------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | | Zone | Signing keys, policy set, resources, providers, audit trail | No - one per deployment | | Application | Your product's service identity | No - shared by all customers | | Customer label and metadata | The customer the work is for: a `customer:` label policy and audit can filter; optional `customer_id` metadata remains Coordinator inspection data | Yes - stamped on every Session | | Session | One agent run, labeled and attributed to one customer | Yes - one per customer task | | Delegation | The scoped authority that agent holds for one resource | Yes - least privilege per task | A zone is the trust boundary that owns keys, policy, and audit. A customer is not a trust boundary of its own here; it is an *attribute* of the work - carried as a label the platform decision contract confines and a metadata key the audit trail filters. Keep customer identity in labels and metadata - never in [scope names](/v1.0/concepts/resource-grant/). ## Step 1: One Zone, One Application Provision a single zone for the deployment and one managed application for your product, as in [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/). Set the application identity once in the environment; it is shared across all customers. ```bash CARACAL_ZONE_ID="" CARACAL_APPLICATION_ID="" CARACAL_APP_CLIENT_SECRET="" ``` The zone owns one policy set and one audit trail. Every customer's authority is decided and recorded inside it. ## Step 2: Choose a Stable Customer Identifier Caracal does not authenticate your customers, does not own your user directory, and never generates a customer identifier. Your application's existing authentication system - Auth0, Keycloak, Better Auth, a custom issuer - authenticates the customer; Caracal receives the identifier your application asserts and treats it as an opaque, stable string. It does not need to know what the identifier represents, only that the same customer always presents the same value. Choose an id that never changes for the life of the account - an account UUID, not an email or display name - so audit history stays attributable even after profile changes. Do not reuse one identifier across two customers: separation is only as strong as your identifier discipline. The identifier enters Caracal on every Session your application starts for that customer as a `customer:` label. Policy and audit consume that label. Add a `customer_id` metadata key only for direct Coordinator inspection; Session metadata is not STS policy or audit input. The `sub` recorded on the underlying STS Authority record is your application's identity from the client-credentials chain; the customer rides on the work itself. ## Step 3: Start a per-Customer Session with a Correlation Key Start the Session for a customer's request inside the one zone and stamp the customer ID into a label. The label travels into policy and audit, making per-customer attribution a direct lookup instead of a guess. Optional metadata can support direct Coordinator inspection but is not copied into decision audit events. ```python async with caracal.session( labels=[f"customer:{customer_id}"], metadata={"customer_id": customer_id}, ) as ctx: # Every gateway and provider call in this block carries a scoped, # non-root mandate for this customer's work. await do_work(ctx) ``` The Session ID and Authority record are the exact authority anchors; the `customer:` label is the business correlation key exposed to policy and audit. Keep the optional `customer_id` metadata key aligned with that label when Coordinator inspection needs the raw value. :::caution[Attribution is not enforcement] Session metadata is stored by Coordinator but does not enter STS policy or decision audit. The platform decision contract decides from Session labels (`input.principal.labels`) and the Delegation. Use the `customer:` label to restrict and find customer activity; use metadata only for direct Session inspection. ::: For fan-out work, give each child agent the least authority it needs with [delegation](/v1.0/guides/delegation/), keeping one customer's blast radius contained even though all customers share the zone. ### Customer Labels Drive Enforcement When policy must confine the work itself - a PiperNet report run, a retention cycle, or any job acting on one customer's records - carry the customer on the Session as a label: ```python async with caracal.session( authority=Authority.narrow(["pipernet:process"], ttl_seconds=600), labels=[role, f"customer:{customer_id}"], metadata={"customer_id": customer_id}, ) as ctx: await collect_overdue(ctx) ``` Labels drive the platform decision contract's confinement: a `confinement` data document caps every customer-labeled agent to the customer-record surface - whatever its role would otherwise allow: ```rego # caracal:data-document package caracal.authz import rego.v1 confinement := [{ "label_prefix": "customer:", "scopes": ["pipernet:process", "pipernet:read"], }] ``` Publish this `confinement` document and a worker Session started for one customer can never mint authority outside that surface, even by accident. The [Lynx Capital example](/v1.0/examples/lynx-capital/) ships this confinement data with tests. ## Step 4: Differentiate Authority per Customer All customers share one zone policy set, but authority is per-request and label-aware. The agent's role labels and the Delegation are in the platform decision contract's input, and your `grants` map roles to scopes per resource. Model plan or tier differences as roles: give the scale-plan role the resource scope, withhold it from the others, and label each customer's session with the role it earns. ```rego # caracal:data-document package caracal.authz import rego.v1 grants := { "resource://payouts": { "application": "pipernet", "roles": {"scale-plan": ["payouts:run"]}, }, } ``` A Session started for a scale-plan customer carries the `scale-plan` label and mints `payouts:run`; a starter-plan session never holds that role, so the platform contract denies it. Express customer differences as roles in `grants` and prefixes in `confinement` - not as a separate policy set per customer. There is one active policy set per zone. See [Author Policy Data](/v1.0/guides/author-policy/). ## Step 5: Attribute and Revoke per Customer Every decision is written to the zone audit ledger with the application, Authority record, Session, labels, and Delegation chain. The customer ID rides in the `customer:` label set in Steps 2 and 3, so you can answer "what did this customer do, with what authority" from the audit trail. See [Audit and Request Traces](/v1.0/concepts/audit-ledger/). To build a per-customer read-only view - for an internal support console or a customer-facing activity page - filter the shared Session and audit surfaces by the `customer:` label, then collect each Session's decisions, Delegations, and Gateway events by Session ID. This remains a filter over one audit trail, not a separate per-customer store. To cut a customer off, find Sessions by the `customer:` label, terminate them, and revoke their Delegations; cascade revocation tears down the chain beneath them. There is no single "purge everything for a customer" call, so iterate the matching Sessions. See [Sessions and Revocation](/v1.0/concepts/sessions-revocation/). ## Limits to Plan For One shared zone serves many customers well, with two ceilings to design around. Session concurrency caps and the rate-limit scope are in [Defaults and Limits](/v1.0/reference/defaults-and-limits/#session-and-delegation-limits). * **Concurrent agents per zone** are capped, so a single zone bounds how many customer agents can run at once. If you need more simultaneous customer agents than one zone allows, add applications within the zone, or move the busiest customers to their own zone. * **STS rate limiting is per zone, resource, and acting application** - not per customer. Because customers share your application identity, one heavy customer draws on the shared budget. Keep this in mind for fairness, and isolate a customer into its own zone if it must have a guaranteed independent budget. ## When to Use a Zone per Customer Instead Stay with one shared zone unless a customer genuinely requires hard isolation: independent signing keys, an audit trail that can never mix with others, or a policy change that can never affect another customer. Those needs are the **zone per customer** model in [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/#multiple-customers-or-workspaces). In this open-source product you provision and automate those zones yourself through the Admin API, and your application authenticates separately into each. A scoped Control key cannot create zones because it is bound to the zone that issued it. Automated managed-tenant lifecycle is not implemented in this repository. ## Validate the Pattern * Sign in as two different customers, run the same workflow, and confirm each produces a distinct Session carrying the correct `customer:` label in the web console. * Author grant data that gives one customer's role a resource and withholds it from another, then confirm both decisions in the audit trail. * Revoke one customer's session and confirm its in-flight agents lose authority while the other customer is unaffected. Expected result: customer A cannot mint customer B's resource authority, customer activity is discoverable by Session labels and IDs, and revoking A does not terminate B. :::caution[Failure point: metadata] Session metadata is not an authorization input. Enforce from reviewed labels and Delegation; use metadata only for Coordinator inspection. Caracal does not verify that a customer ID belongs to the caller - your application must authenticate and bind it before starting the Session. ::: ## Related * [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/) * [Zones](/v1.0/concepts/zone/) * [Identities and Applications](/v1.0/concepts/principal/) * [Implement Multi-Agent Delegation](/v1.0/guides/delegation/) * [Author Policy Data](/v1.0/guides/author-policy/) * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) * [Audit and Request Traces](/v1.0/concepts/audit-ledger/) ## Next Step Implement label and confinement tests in [Test Caracal Integrations](/v1.0/guides/testing/) before onboarding production customers. --- # Define Resources and Providers # URL: https://docs.caracal.run/v1.0/guides/resources-providers/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/resources-providers.md # Type: page # Concepts: # Requires: --- Use this guide after modeling an upstream and before writing policy or application code. The workflow creates the routing and credential boundary that Gateway enforces. The [Admin package](/v1.0/sdks/admin/) and [Admin API](/v1.0/api/control-plane/) hold exact automation signatures. ## Prerequisites * A zone and durable managed application for Gateway routing. * A stable resource identifier, action-oriented scopes, and an upstream URL reachable from Gateway. * The upstream's documented authentication method and a production secret source. ## When to create each object ```mermaid flowchart LR Caller[Verified request] --> GW[Gateway] Resource["Resource\nidentifier + scopes + upstream URL"] -->|routes| GW Provider["Provider\nsealed upstream credential"] -->|attaches| GW GW --> Upstream[Protected upstream] ``` | Object | Create it when | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Resource | You need a stable protected target, Caracal resource scopes, an upstream URL, a Gateway application, and one upstream credential provider binding. | | Provider | A resource needs an explicit upstream auth mode: none, Caracal mandate, OAuth 2.0 authorization code, OAuth 2.0 client credentials, API key, bearer-token, or HTTP Basic upstream auth. | | Gateway application | You want Gateway-originated upstream calls to use a specific Caracal application identity. | Resource fields should answer "what is being protected and where does Gateway send traffic?" Provider fields should answer "what credential or identity does Gateway attach upstream?" Keep credential details such as client secrets, token endpoints, API-key placement, bearer tokens, identity forwarding, and runtime injection on the provider. Keep target details such as resource identifier, Caracal resource scopes, upstream URL, Gateway application, and the selected upstream credential provider on the resource. :::note[FAQ] [What is the difference between a resource and a provider?](/v1.0/reference/faq/#faq-009) and [is an application secret the same as a provider credential?](/v1.0/reference/faq/#faq-013) ::: ## Web Console Workflow 1. Start the runtime for your deployment (locally, `caracal up`). 2. Open the web console and select **Resources**. 3. Create a resource with a stable identifier, such as `resource://pipernet`, and action-oriented scopes, such as `pipernet:read` and `pipernet:refund`. 4. Set the upstream URL and the Gateway application. 5. Attach exactly one upstream credential provider: `None` when the Gateway is the enforcement point and the upstream expects no credential, `Caracal mandate` when the upstream verifies Caracal tokens itself, or a provider-native kind when the upstream needs external credentials. 6. For a path-addressed REST upstream, declare the operations the Gateway may invoke and keep [operation authority](#operation-authority) `enforced`; choose **Any operation** for single-surface transports such as MCP. ## Provider auth modes | Provider type | Required main fields | Runtime behavior | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | None | No credential fields. | Gateway strips caller auth and forwards no upstream credential; use only when Gateway is the enforcement point and the upstream expects no credential. | | Caracal mandate | No credential fields. | Gateway forwards the Caracal resource mandate in `Authorization: Bearer ...`; the resource verifies issuer, audience, scopes, target, expiry, and revocation. | | OAuth 2.0 authorization code | Authorization endpoint, token endpoint, redirect URI, client ID, client secret, optional upstream OAuth scopes. | Creates a shared provider connection (optionally bound to a Subject) through a browser consent callback, then refreshes the brokered upstream tokens inside STS. | | OAuth 2.0 client credentials | Token endpoint, client ID, grant type, client secret or private key depending on the grant and authentication choices. | STS obtains and caches service-to-service provider tokens for Gateway upstream calls, via `client_credentials` or RFC 7523 `jwt_bearer` assertion grants. | | API key | Header name and API key. | Gateway forwards the sealed provider key in the configured header, with an optional auth scheme prefix. | | Bearer token | Bearer token. | Gateway forwards the sealed provider token as `Authorization: Bearer ...` unless Advanced routing configures another header or scheme. | | HTTP Basic | Username and password. | Gateway forwards `Authorization: Basic ...` built from the username and the sealed password; the pair is never exposed to callers or runtimes. | Rules shared by every kind: provider secrets are sealed at creation and never returned by list or detail APIs; every Gateway resource binds exactly one provider; one provider can serve many resources. Concrete per-upstream setups (OpenAI, Google, GitHub, Slack, Jira, internal APIs) live in [Provider Recipes](/v1.0/guides/provider-recipes/). ### OAuth 2.0 authorization code (delegated consent) * Register the exact Caracal callback URI as the provider redirect URI, such as `https://api.hooli.example/v1/zones/z1/provider-connections/oauth/callback`, then use the provider's **Connect** action to create an authorization URL for the shared upstream account, or bind it to a specific Subject for per-customer isolation. * A connection is one authenticated upstream account for the provider. By default it is shared across the Zone: every session policy authorizes through the provider uses it, and authorization stays per-resource through scopes, policies, and grants. A Zone that needs a distinct upstream account per customer can bind a connection to a specific Subject, which takes precedence over the shared account for that Subject. Reconnecting replaces the active connection for the same account. * The upstream scopes requested during consent come from the provider's configured upstream OAuth scopes. Advanced key/value entries add provider-specific browser parameters (`access_type=offline`, `prompt=consent`) and token endpoint parameters; Caracal always owns `client_id`, `redirect_uri`, `state`, PKCE, credentials, connections, and refresh tokens. * Connections follow the upstream token's lifecycle: refresh tokens are refreshed inside STS shortly before expiry (rotating refresh tokens are stored under optimistic locking); a lapsed connection without a refresh token becomes `expired` and callers receive a precise `credential_expired_not_renewable` denial. The provider's Connections panel reads this state live; `GET /v1/zones/{zoneId}/provider-connections` is the same listing for automation. Token responses are held to RFC 6749: a `token_type` other than `bearer` is rejected unless the provider's upstream auth scheme explicitly asserts it. * Revoking a connection always succeeds inside Caracal and immediately stops STS from brokering its tokens; when the provider advertises an RFC 7009 revocation endpoint, Caracal also revokes upstream best-effort and reports the result. To switch the upstream account, **Revoke** then **Connect** again, or just **Connect** again to replace the active connection in place. * The callback endpoint is intentionally public so providers can redirect browsers back. Security comes from short-lived Redis-backed one-time state, PKCE, provider binding checks, sealed token storage, HTTPS-only exchange, redirect blocking, host allow-listing, and private-address rejection. In cloud deployments, all API replicas must share Redis so any replica can validate callback state. ### OAuth 2.0 client credentials (machine-to-machine) * The grant type selects how STS obtains tokens: `client_credentials` posts the client's own credentials to the token endpoint; `jwt_bearer` signs an RFC 7523 assertion with the provider private key - the pattern behind Google service accounts and Salesforce's JWT bearer flow. * For `jwt_bearer`, client authentication defaults to `none` because the assertion is the credential. Upstream OAuth scopes ride inside the assertion's `scope` claim; the assertion subject overrides `sub` (Google domain-wide delegation) and the assertion audience overrides `aud` (Salesforce). * For `client_credentials`, use upstream OAuth scopes, OAuth token audience, or an OAuth resource indicator as the provider documents; reserve OAuth token parameters for documented provider-specific extras. With `private_key_jwt` client authentication, an optional client certificate adds the `x5t`/`x5t#S256` thumbprint headers Microsoft Entra ID certificate credentials require. * OAuth token endpoint hosts constrain outbound token acquisition; the web console infers the host from the token endpoint when the Advanced field is blank. Endpoints must resolve publicly unless the exact private hostname is granted through `CARACAL_PRIVATE_EGRESS_HOSTS` on API and STS. * Both OAuth kinds support endpoint autofill: paste the issuer URL and the control plane resolves its OIDC or RFC 8414 metadata (MCP servers publish the same metadata), fills the endpoints, and rejects metadata whose issuer does not match. ### API key * One required routing field: the exact header the upstream expects. Use `X-API-Key` or another vendor header for raw key values, `Authorization` plus the Advanced auth scheme `Bearer` for bearer-style keys, or a documented vendor scheme such as `Token`. * Query-parameter placement is supported for upstreams that require it, but a key in the query string may be logged by systems outside Caracal's control - Caracal's own Gateway and STS audit events never include query strings. Prefer header placement. ### Bearer token * For static, pre-issued access tokens Caracal does not mint or refresh. The main form needs only the token; Advanced options cover non-default headers or schemes. List and detail APIs expose only `secret_config_keys`, and the Gateway replaces caller-supplied auth before forwarding. ### HTTP Basic * For username/password or username/API-token pairs (Atlassian, Twilio, Elasticsearch). The username stays readable configuration; the password is sealed, and the Gateway composes `Authorization: Basic ...` at forward time. Because the credential is a two-part pair, HTTP Basic providers are Gateway-forwarded only and never eligible for runtime credential injection. ### None and Caracal mandate * None providers have no secret and send no credential. Caracal mandate providers have no provider-native secret and no `forward_caracal_identity` setting because the mandate is already the upstream credential - use it for internal services, partner services, and integrations that verify Caracal mandates through a verifier or adapter. ## Provider connectivity checks OAuth providers own a token endpoint Caracal can genuinely verify before anything is saved, so creating one in the web console ends with **Connect** instead of a plain create. Client-credentials providers perform a real token request against the allow-listed HTTPS token endpoint, including a signed client assertion when OAuth client authentication is `private_key_jwt` or a signed assertion grant when the grant type is `jwt_bearer`. Authorization-code providers submit a placeholder code that a healthy endpoint rejects as an invalid grant after accepting the client credentials; the probe carries a placeholder PKCE verifier so endpoints that mandate PKCE classify the same way. Checks run from the control plane with DNS pinned to approved addresses; private space is accepted only for exact names in `CARACAL_PRIVATE_EGRESS_HOSTS`, while metadata, link-local, loopback, multicast, and unspecified addresses stay forbidden. Upstream responses are reduced to a fixed classification, and any issued token is discarded, never stored or returned. A failing check blocks creation and reports a precise classification: authentication failed, endpoint unreachable, unexpected endpoint response, or configuration incomplete. **Skip for now** creates the OAuth provider without a passing check; it then carries a red **Failed** badge in the provider list and detail view until a check passes. Open the provider and run **Connect** from its Connectivity section after fixing the configuration - a passing check clears the badge automatically. The other kinds - none, Caracal mandate, API key, bearer token, and HTTP Basic - make no upstream credential request of their own, so no provider-level credential preflight exists before a resource uses them. The console creates them directly with **Create Provider** and shows no Connectivity section; their configuration is validated at creation and their credential is exercised once a resource uses it. Caracal never presents a check that cannot fail. Automation can run the OAuth check through `POST /v1/zones/{zoneId}/providers/{id}/test` or create with a check by setting `"check": true` on the create request; both reject non-OAuth kinds with `provider_check_unsupported`, and checks are rate limited per zone. ## Resource verification checks A Caracal mandate resource names an upstream that is expected to verify Caracal mandates through a verifier or adapter. Because the mandate is itself the upstream credential, Caracal can confirm the upstream actually enforces verification before traffic is routed. The check presents an untrusted mandate - signed by an ephemeral key that is not published in the zone JWKS - to the resource's upstream URL over `HEAD` and classifies the response: * **Verifier enforcing** - the upstream rejected the untrusted mandate with `401` or `403`, so a verifier is enforcing verification on this URL. * **Not verifying mandates** - the upstream accepted the untrusted mandate with a `2xx`. Add a Caracal verifier or adapter before routing traffic. * **Upstream unreachable** - the upstream did not answer. A public host must resolve; a private host must be granted egress through `CARACAL_PRIVATE_EGRESS_HOSTS` on the API. * **Unexpected response** - the upstream answered with another status, so the URL likely does not point at a route a verifier guards. The probe carries the same egress protection as provider checks: DNS is pinned to approved addresses at dial time, and private space is accepted only for exact names in `CARACAL_PRIVATE_EGRESS_HOSTS`, while metadata, link-local, loopback, multicast, and unspecified addresses stay forbidden. The untrusted mandate never reaches upstream business logic and is discarded after the probe; nothing is stored or returned. Creating a Caracal mandate resource that has an upstream URL runs the check: **Create resource** requires an enforcing result, while **Skip for now** creates the resource without one. Run the check any time from the resource's Verification section with **Test connection**. Automation can call `POST /v1/zones/{zoneId}/resources/{id}/test`, or create with a check by setting `"check": true` on the resource create request; both reject resources that are not bound to a Caracal mandate provider or have no upstream URL with `resource_verification_unsupported`, and probes are rate limited per zone. ## Operation authority A resource declares which upstream operations the Gateway may invoke and how strictly that surface is enforced. This authority is native to the platform: the Gateway and STS enforce it directly, so adopters configure data, not policy control flow. | Field | Meaning | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `operations` | The operations the Gateway may forward, each a `{method, path, scope}` triple. `method` and `path` match the upstream call; `scope` is the Caracal resource scope the mandate must carry for that operation, and must be one of the resource's declared scopes. | | `operation_enforcement` | `enforced` (**Listed operations only**) denies any Gateway operation not listed in `operations` and requires its scope on the mandate. `transport_uniform` (**Any operation**) treats the upstream as a single surface and relies on the mint-time scope check, for protocols such as MCP that address every call through one transport path. | When `operation_enforcement` is `enforced`, the Gateway authorizes only declared operations and denies everything else with `operation_not_permitted` before any upstream call. A resource that is `enforced` with an empty `operations` list is fully closed: every Gateway operation is denied until operations are declared. :::caution[Defaults differ by creation surface] Resources created through the Control API default to `enforced`, so authority stays closed until you describe it. Web console guided setup opens resources as **Any operation** because it does not collect per-operation detail - tighten them from the resource editor once the operation set is known. ::: Because the floor is enforced in the platform rather than in adopter policy, a policy can further restrict an operation but can never widen authority beyond the declared operations and their scopes. ## Identifier and scope guidance | Field | Good pattern | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Resource identifier | Stable audience URI, such as `resource://pipernet`; do not use the `provider://` namespace. | | Caracal resource scope | `domain:action`, such as `pipernet:read`. | | Upstream URL | The network URL Gateway can reach; it may change without changing policy audiences. | | Gateway application | The managed application this resource's route serves. Gateway exchanges as this identity, and STS only accepts callers whose mandates were minted under it - delegated calls must target a Session it owns. Policy then authorizes each call. DCR applications cannot be bound here - they are short-lived single-session credentials, not durable Gateway identities. | | Upstream credential provider | The provider record Gateway uses to attach no credential, a Caracal mandate, OAuth tokens, API keys, or bearer tokens. | | Authorized operation | `{method, path, scope}`, such as `{ "method": "POST", "path": "/api/create_payout", "scope": "pipernet:payout" }`; the scope must be one of the resource's Caracal resource scopes. | | Operation enforcement | `enforced` for path-addressed REST upstreams; **Any operation** for single-surface transports such as MCP. | | Provider identifier | `provider://lowercase-slug` stable name for the upstream credential system. | Do not put provider-specific credential details on a resource. Do not use mutable deployment hostnames as policy identifiers unless they are the actual authority boundary. Keep the resource identifier stable even if the upstream URL or upstream credential provider binding changes. :::note[FAQ] [Why must the resource identifier stay stable if the upstream URL can change?](/v1.0/reference/faq/#faq-010) ::: ## Validate the setup * For OAuth providers, run **Connect** and confirm the connectivity check passes; no provider should show a **Failed** badge. * Open the resource in the web console and confirm scopes are present. * Author and activate a policy that allows the application and Subject to request the scopes. * Send a Gateway request and inspect the audit event. For Caracal mandate upstreams, also confirm the resource uses a Caracal verifier or adapter. Expected result: OAuth provider checks pass where supported, the resource has one Gateway application and one provider binding, undeclared operations fail before the upstream, and an allowed request reaches only the configured host. :::caution[Failure point: operation mode] The API value is `enforced`, not `enforce`. An `enforced` resource with no operations is intentionally closed. Use **Any operation** only when every call shares one transport surface, such as MCP. ::: ## Protect an MCP server over the Gateway An MCP server is a standard Gateway upstream: it resolves to the same Resource and Provider objects with no MCP-specific primitive. MCP-over-HTTP is proxied like any other HTTP upstream, so only two choices are particular to MCP. **Resource.** Set operation enforcement to **Any operation**. MCP addresses every tool call through one transport surface, so there is no per-operation `{method, path, scope}` list to declare and the mint-time scope check is the authority boundary. Give the resource a stable identifier such as `resource://pipernet`, one or more tool scopes such as `mcp:tool:call`, the MCP server's upstream URL, and a Gateway application. **Provider.** Choose the credential kind by how the MCP server authenticates the request the Gateway forwards to it: | The MCP server authenticates with | Provider kind | | ------------------------------------------------------------------------------------------------ | ------------------------------------ | | Caracal mandates it verifies itself through a verifier or adapter | Caracal mandate | | Nothing, because the Gateway is the enforcement point and the server is network-restricted to it | None | | Delegated user consent through the MCP OAuth 2.1 authorization-code flow | OAuth 2.0 authorization code | | A machine-to-machine OAuth client | OAuth 2.0 client credentials | | A static API key, a pre-issued bearer token, or a username and password | API key, Bearer token, or HTTP Basic | For either OAuth kind, paste the MCP server's issuer URL and use endpoint autofill: MCP servers publish the OIDC and RFC 8414 metadata Caracal resolves, so the authorization and token endpoints fill in automatically. The Gateway proxies MCP-over-HTTP but does not proxy WebSocket upgrades. For a WebSocket MCP transport, verify mandates inside the server process with [Protect an MCP Server](/v1.0/guides/protect-mcp/) instead. ## Related Guides * [Provider Recipes](/v1.0/guides/provider-recipes/) * [Author Policy Data](/v1.0/guides/author-policy/) * [Debug Authorization Decisions](/v1.0/guides/authorize-access/) * [Protect a Gateway-Routed HTTP API](/v1.0/guides/protect-gateway-http/) * [Activate a Policy Set](/v1.0/guides/activate-policy-set/) ## Next Step Choose a concrete credential setup in [Provider Recipes](/v1.0/guides/provider-recipes/), then run [Check Provider Readiness](/v1.0/examples/provider-preflight/) before the first real call. --- # Provider Recipes # URL: https://docs.caracal.run/v1.0/guides/provider-recipes/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/provider-recipes.md # Type: page # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' Use this page while creating a provider for a known upstream. First select the enforcement boundary, then apply one recipe, bind the resource, wire one language-native transport, and validate the whole path. Field schemas stay canonical in [Define Resources and Providers](/v1.0/guides/resources-providers/) and the [Admin package reference](/v1.0/sdks/admin/). ## Prerequisites * The resource, scopes, Gateway application, and active policy already exist. * The upstream's official token or authentication documentation is available. * Production secrets are ready to enter through Console or a trusted Admin API process, never application source. Provider recipes live in the documentation, not the web console. The web console stays focused on the fields a given setup needs; this page is the reference you keep open beside it. ## Choose the enforcement boundary first The most common mistake is treating these four boundaries as interchangeable. They are not - they differ in where enforcement happens, who writes the action-result audit, and who holds the upstream credential. | Boundary | Enforcement point | Credential holder | Action-result audit | Use when | | ------------------------ | ------------------------------------------ | -------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | Gateway-mediated | Caracal Gateway | Gateway (sealed provider secret) | Gateway | The upstream is HTTP-routable and you want Caracal to broker the credential and enforce every call. | | Adapter-verified | Your service process | Your service | Your service after adapter verification | You own the service and must enforce mandates in process. | | Application-managed call | Your application code | Your application | Your application | You call the provider directly and only attach Caracal context for attribution. Not Caracal-enforced. | | Runtime token injection | Your runtime, eligibility gated by Caracal | Your runtime process | Your application | An existing client or CLI must receive a brokered token at runtime, and the provider sets `allow_runtime_injection=true`. | Prefer **Gateway-mediated** unless you have a specific reason to enforce elsewhere. The recipes below note which boundary they use. ## OpenAI API key (Gateway-mediated) Broker an OpenAI key so the agent never holds it. | Field | Value | | --------------------- | -------------------------------------------- | | Provider kind | `api_key` | | `header_name` | `Authorization` | | `auth_scheme` | `Bearer` | | `api_key` | Your OpenAI secret key (sealed at creation). | | Resource identifier | `resource://openai` | | Resource upstream URL | `https://api.openai.com` | | Resource scopes | `openai:chat`, `openai:embeddings` | The Gateway forwards `Authorization: Bearer ` upstream and strips caller auth. The agent calls the Caracal resource, not OpenAI directly. ## Google Workspace (OAuth authorization code) Delegated user consent for a Google API, refreshed inside STS. | Field | Value | | --------------------------------- | --------------------------------------------------------------------------------------------------------------- | | Provider kind | `oauth2_authorization_code` | | `authorization_endpoint` | `https://accounts.google.com/o/oauth2/v2/auth` | | `token_endpoint` | `https://oauth2.googleapis.com/token` | | `redirect_uri` | Your exact Caracal callback, e.g. `https://api.pipernet.example/v1/zones//provider-connections/oauth/callback` | | `client_id` / `client_secret` | From the Google Cloud OAuth client. | | `scopes` | e.g. `https://www.googleapis.com/auth/drive.readonly` | | `allowed_token_hosts` | `oauth2.googleapis.com` | | `authorization_params` (Advanced) | `access_type=offline`, `prompt=consent` for refresh tokens. | | Resource upstream URL | `https://www.googleapis.com` | Caracal owns `client_id`, `redirect_uri`, `state`, and PKCE. Use the provider `connect` action to mint the consent URL for the shared upstream account, or bind it to a specific subject for per-customer isolation. :::note[FAQ] [When should I use per-user OAuth instead of a shared provider credential?](/v1.0/reference/faq/#faq-014) ::: ## Google Cloud service account (jwt\_bearer) Machine access to Google APIs - Vertex AI, Cloud Storage, and every other `googleapis.com` surface - signed with a service-account key instead of user consent. | Field | Value | | ------------------------------ | ----------------------------------------------------------------------- | | Provider kind | `oauth2_client_credentials` | | `grant_type` | `jwt_bearer` | | `token_endpoint` | `https://oauth2.googleapis.com/token` | | `client_id` | The service account email, e.g. `agent@project.iam.gserviceaccount.com` | | `private_key` | The service account's PEM private key (sealed). | | `scopes` | e.g. `https://www.googleapis.com/auth/cloud-platform` | | `assertion_subject` (Advanced) | A user email, only for domain-wide delegation. | | Resource upstream URL | e.g. `https://aiplatform.googleapis.com` | STS signs the RFC 7523 assertion with the sealed key, carries the scopes inside the assertion's `scope` claim as Google expects, and caches the returned access token. The same shape covers Salesforce's JWT bearer flow: set `client_id` to the consumer key, `assertion_subject` to the Salesforce username, and `assertion_audience` to `https://login.salesforce.com`. ## Atlassian Jira (HTTP Basic) Jira Cloud REST authenticates with an email and API token as a Basic pair. | Field | Value | | --------------------- | --------------------------------------- | | Provider kind | `http_basic` | | `username` | The Atlassian account email. | | `password` | The Atlassian API token (sealed). | | Resource identifier | `resource://jira` | | Resource upstream URL | `https://your-site.atlassian.net` | | Resource scopes | `jira:issues-read`, `jira:issues-write` | Gateway composes `Authorization: Basic ...` at forward time; callers never see the pair. The same recipe covers Twilio (Account SID as username, auth token as password), Elasticsearch, Zendesk, and other Basic-authenticated REST APIs. ## GitHub (bearer token, with OAuth note) For automation, the simplest path is a pre-issued token. | Field | Value | | --------------------- | ----------------------------------------------------- | | Provider kind | `bearer_token` | | `bearer_token` | A GitHub fine-grained personal access token (sealed). | | `allowed_token_hosts` | `api.github.com` | | Resource identifier | `resource://github` | | Resource upstream URL | `https://api.github.com` | | Resource scopes | `github:repo-read`, `github:issues-write` | For a delegated, consented account instead of a shared token, use `oauth2_authorization_code` with `authorization_endpoint=https://github.com/login/oauth/authorize` and `token_endpoint=https://github.com/login/oauth/access_token`. ## Slack (OAuth client credentials or bearer) For a Slack bot token that does not expire, use a bearer provider. | Field | Value | | --------------------- | ----------------------------------------- | | Provider kind | `bearer_token` | | `bearer_token` | The Slack bot/user OAuth token (sealed). | | `allowed_token_hosts` | `slack.com` | | Resource identifier | `resource://slack` | | Resource upstream URL | `https://slack.com/api` | | Resource scopes | `slack:chat-write`, `slack:channels-read` | For rotating machine tokens, use `oauth2_client_credentials` with Slack's token endpoint and `scopes`. ## Internal API (Caracal mandate or none) For services you own, you usually do not need an upstream credential. | Setup | Provider kind | Use when | | ---------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Mandate-aware | `caracal_mandate` | The internal service verifies Caracal tokens directly (issuer, audience, scopes, expiry, revocation). The Gateway forwards the resource mandate as `Authorization: Bearer ...`. | | No upstream credential | `none` | The Gateway is the enforcement point and the upstream expects no credential. | | Field | Value | | --------------------- | ---------------------------------- | | Resource identifier | `resource://pipernet` | | Resource upstream URL | `https://api.pipernet.example` | | Resource scopes | `pipernet:read`, `pipernet:submit` | ## Runtime token injection When an existing CLI or client cannot route through the Gateway but still needs a brokered credential, set `allow_runtime_injection=true` on the provider. This is a distinct security boundary: the token is injected into your runtime, and enforcement of the actual call is your application's responsibility, not the Gateway's. Use it deliberately and confirm eligibility with the preflight below. ## Wire the transport into provider clients Once a provider and resource exist, the client side of a Gateway-mediated setup is two lines: a resource binding and the SDK transport. The binding maps the provider's natural upstream prefix to the resource, so agent code keeps calling the provider's real URL - the transport rewrites matching requests through the Gateway, attaches the caller's mandate, and the Gateway injects the sealed provider credential upstream. ```sh export CARACAL_RESOURCES="resource://openai=https://api.openai.com" ``` Bindings can also live in the runtime profile as `[[credentials]]` entries; see [Configure Workloads](/v1.0/runtime-console/config-file/). Provider clients still require an API key argument at construction - pass a placeholder such as `caracal-gateway`. The transport replaces the `Authorization` header on Gateway-routed calls, so the placeholder never reaches the provider. :::caution[Failure point: snippet scopes] The wiring snippets below all request one illustrative scope, `inference:invoke`. The recipe tables above declare provider-specific scopes such as `openai:chat`. Pick one convention and use it in **both** the resource's declared scopes and the transport call - a scope the resource does not declare, or policy does not grant, is denied. ::: Each recipe below assumes a constructed SDK client (`const caracal = new Caracal()`, `caracal = Caracal()`, `client, _ := caracal.New()`) and runs inside a governed Session context - `transport()` borrows the ambient Session's authority, unlike `applicationTransport()`, which provisions its own (see [Call Protected Resources](/v1.0/sdks/typescript/#call-protected-resources)). ### OpenAI ```ts import OpenAI from 'openai' const openai = new OpenAI({ apiKey: 'caracal-gateway', fetch: caracal.transport({ scopes: ['inference:invoke'], propagation: 'gateway-only' }), }) ``` ```python from openai import AsyncOpenAI openai = AsyncOpenAI( api_key="caracal-gateway", http_client=caracal.transport(scopes=["inference:invoke"], propagation="gateway-only"), ) ``` ```go import ( "github.com/openai/openai-go" "github.com/openai/openai-go/option" ) openaiClient := openai.NewClient( option.WithAPIKey("caracal-gateway"), option.WithHTTPClient(client.Transport(nil, caracal.CallOptions{ Scopes: []string{"inference:invoke"}, Propagation: caracal.PropagationGatewayOnly, })), ) ``` ### Anthropic Binding: `resource://anthropic=https://api.anthropic.com`. ```ts import Anthropic from '@anthropic-ai/sdk' const anthropic = new Anthropic({ apiKey: 'caracal-gateway', fetch: caracal.transport({ scopes: ['inference:invoke'], propagation: 'gateway-only' }), }) ``` ```python from anthropic import AsyncAnthropic anthropic = AsyncAnthropic( api_key="caracal-gateway", http_client=caracal.transport(scopes=["inference:invoke"], propagation="gateway-only"), ) ``` ```go import ( "github.com/anthropics/anthropic-sdk-go" "github.com/anthropics/anthropic-sdk-go/option" ) anthropicClient := anthropic.NewClient( option.WithAPIKey("caracal-gateway"), option.WithHTTPClient(client.Transport(nil, caracal.CallOptions{ Scopes: []string{"inference:invoke"}, Propagation: caracal.PropagationGatewayOnly, })), ) ``` ### Google Gemini Binding: `resource://gemini=https://generativelanguage.googleapis.com`. ```python from google import genai from google.genai import types gemini = genai.Client( api_key="caracal-gateway", http_options=types.HttpOptions( httpx_async_client=caracal.transport( scopes=["inference:invoke"], propagation="gateway-only" ) ), ) ``` ```go import "google.golang.org/genai" gemini, err := genai.NewClient(ctx, &genai.ClientConfig{ APIKey: "caracal-gateway", HTTPClient: client.Transport(nil, caracal.CallOptions{ Scopes: []string{"inference:invoke"}, Propagation: caracal.PropagationGatewayOnly, }), }) ``` The `@google/genai` JavaScript SDK does not accept a custom `fetch`. In TypeScript, call Gemini through its OpenAI-compatible surface with the OpenAI wiring above and `baseURL: "https://generativelanguage.googleapis.com/v1beta/openai/"`; the binding stays the same. ### LiteLLM The LiteLLM Python library sends traffic through module-level httpx sessions: ```python import litellm litellm.aclient_session = caracal.transport( scopes=["inference:invoke"], propagation="gateway-only" ) litellm.client_session = caracal.sync_transport( scopes=["inference:invoke"], propagation="gateway-only" ) ``` Bind one resource per upstream LiteLLM reaches, or govern a self-hosted LiteLLM proxy as a single resource (for example `resource://litellm=https://litellm.internal.example`). The proxy speaks the OpenAI protocol, so TypeScript and Go clients use the OpenAI wiring above with `baseURL` pointed at the proxy. ### Ollama Binding: `resource://ollama=http://ollama.internal.example:11434`. ```ts import { Ollama } from 'ollama' const ollama = new Ollama({ host: 'http://ollama.internal.example:11434', fetch: caracal.transport({ scopes: ['inference:invoke'], propagation: 'gateway-only' }), }) ``` ```go import "github.com/ollama/ollama/api" base, _ := url.Parse("http://ollama.internal.example:11434") ollamaClient := api.NewClient(base, client.Transport(nil, caracal.CallOptions{ Scopes: []string{"inference:invoke"}, Propagation: caracal.PropagationGatewayOnly, })) ``` The Ollama Python client constructs its own httpx client internally. In Python, call Ollama through its OpenAI-compatible `/v1` endpoint with the OpenAI wiring above and `base_url="http://ollama.internal.example:11434/v1"`. Self-hosted upstreams like LiteLLM and Ollama usually need no upstream credential: use provider kind `none` and let the Gateway be the enforcement point. ## Validate before the first call After you create the provider, bind it to the resource, and activate a policy, run [Check Provider Readiness](/v1.0/examples/provider-preflight/). It validates control-plane and Gateway readiness, the resource-to-provider binding, application validity, provider configuration completeness, scope coverage, token endpoint and upstream reachability, runtime-injection eligibility, and that the active policy set actually returns `allow` for your application, resource, and scopes - with a concrete remediation for every failure. Then make one allowed call and one scope-denied call. The allowed call must appear as an STS decision followed by a Gateway result under one request ID; the denied call must not reach the upstream. :::caution[Failure point: transport propagation] Pass `propagation: 'gateway-only'` (or the language equivalent) for third-party provider clients. Do not send Caracal context to unrelated hosts. ::: :::caution[Failure point: retries] Gateway does not make an unsafe upstream operation idempotent. Supply a destination-supported idempotency key for mutations and follow [Safe Retries and Idempotency](/v1.0/guides/idempotency/). ::: ## Related * [Define Resources and Providers](/v1.0/guides/resources-providers/) * [Govern Agent Frameworks](/v1.0/guides/frameworks/) * [Author Policy Data](/v1.0/guides/author-policy/) * [Debug Authorization Decisions](/v1.0/guides/authorize-access/) * [Protect a Gateway-Routed HTTP API](/v1.0/guides/protect-gateway-http/) ## Next Step Integrate the caller with the [TypeScript](/v1.0/guides/sdk-typescript/), [Python](/v1.0/guides/sdk-python/), or [Go](/v1.0/guides/sdk-go/) workflow. --- # Author Policy Data # URL: https://docs.caracal.run/v1.0/guides/author-policy/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/author-policy.md # Type: page # Concepts: # Requires: --- Use this guide after resources and applications exist and before activating production access. ## Why data, not decisions Authorization logic - delegation narrowing, role and grant checks, label confinement, bootstrap isolation - is identical for every adopter and is the part most dangerous to get wrong. A single typo in a hand-written rule (`if { false }` → `if { true }`) silently turns a deny into an allow-all. Caracal removes that footgun by owning the logic in a signed, versioned [platform decision contract](/v1.0/concepts/policy/#decision-contract) and letting you supply only **data**. Every policy you author is a data document marked with `# caracal:data-document` on its first line. It carries only data tables and is **forbidden from defining `result`**, so it can never decide an authorization on its own. `restrict` entries can only subtract authority and `confinement` can only narrow it - a careless data change fails closed. ## Prerequisites * A zone, application, and resource. * Access to the web console or Admin API. * The resource identifier and scopes you want to grant. * The real application ID and the Session labels the application emits. * Representative allow and deny inputs for simulation. ## Start from a grant The core document is `grants`: it names the application that owns a resource view and the scopes each role may hold. Pair it with `app_ids`, which binds the application key you use in `grants` to the control-plane id the STS sees as `input.principal.id`. ```rego # caracal:data-document package caracal.authz import rego.v1 app_ids := { "pipernet": "app-pipernet", } grants := { "resource://pipernet": { "application": "pipernet", "roles": {"reader": ["pipernet:read", "pipernet:write"]}, }, } ``` This is the canonical "application A may call resource B with scopes C" pattern, expressed as data. The platform contract allows a mint only when the acting application owns the view, the agent's role label grants the scope, and the Delegation narrows to it. You declare the grant; the contract enforces the narrowing. Several `app_ids` keys may bind to one application id - the contract treats them as one identity - but one binding key per application keeps grant review straightforward, and the Admin SDK's grant helpers author exactly that. ## Gate scopes on human approval Two further documents add an optional human gate on top of a grant. `risk` names a tier for each sensitive scope, and `approval_tiers` declares which tiers hold the mint until a person decides it: ```rego # caracal:data-document package caracal.authz import rego.v1 risk := [ {"scope": "pipernet:refund", "tier": "high"}, ] approval_tiers := [ {"tier": "high", "approver": "operator", "ttl_seconds": 1800, "privacy": "identified"}, ] ``` Like `restrict` and `confinement`, an approval declaration can only add a gate, never widen authority, and a malformed declaration fails the gated mint closed. [Human Approval](/v1.0/guides/human-approval/) covers the tier fields, both decision planes, and the agent-side wait-and-retry flow. ## Start from a template You do not have to write each document by hand. Caracal ships a built-in catalog of data-document starters, served at `/v1/policy-templates` and through the Admin SDK: ```ts import { AdminClient } from '@caracalai/admin' const admin = new AdminClient({ apiUrl: process.env.CARACAL_API_URL!, adminToken: process.env.CARACAL_ADMIN_TOKEN!, }) const templates = await admin.policyTemplates.list() const starter = await admin.policyTemplates.get('resource-grants') ``` | Template | Use it for | | ---------------------- | ------------------------------------------------------------------------------ | | `application-bindings` | Map each application key used in `grants` to its control-plane application id. | | `resource-grants` | Declare the owning application and per-role scope sets for a resource view. | | `label-confinement` | Cap every session carrying a label prefix to a fixed scope set. | | `zone-restriction` | A deny overlay that freezes the zone while an entry is present. | For assisted authoring, describe the outcome to the [Caracal Operator](/v1.0/concepts/operator/#authoring-policy). Its policy author models the use case as grant, binding, and confinement data, validates and previews each document against the platform contract, and proposes a governed create you review and approve - so the policy that lands is already contract-valid. ## How your data maps to the request The decision contract evaluates a fixed input contract and resolves it against your data. The acting application is the principal - there is no `input.application` or `input.grant` object. See the full [Policy Input Contract](/v1.0/concepts/policy/#policy-input-contract) for every field. | Input the contract reads | Resolved against | | -------------------------------- | ---------------------------------------------------------- | | `input.principal.id` | `app_ids` - to find the application key used in `grants`. | | `input.principal.labels` | `grants[...].roles` and `confinement` label prefixes. | | `input.resource.identifier` | the top-level key in `grants`. | | `input.context.requested_scopes` | the role's scope set and any matching `confinement` rule. | | `input.delegation_edge.scopes` | the narrowing floor every requested scope must sit inside. | ## Validate before versioning Use the web console policy workflow to paste the document and run validation. For automation, validate through the Admin API or `@caracalai/admin`: ```ts import { AdminClient } from '@caracalai/admin' const admin = new AdminClient({ apiUrl: process.env.CARACAL_API_URL!, adminToken: process.env.CARACAL_ADMIN_TOKEN!, }) const validation = await admin.policies.validate(policySource) if (!validation.valid) { throw new Error('policy failed validation') } ``` Validation enforces the data-document contract: the package must be `caracal.authz`, the first line must carry the `# caracal:data-document` directive, the document must define at least one data rule, and it must **not** define `result`. Validation also checks the schema version, balanced syntax, and forbidden built-ins. Because the platform contract owns the decision, a data document can never authorize on its own. ### Preview how the document parses A successful validation returns a `preview` describing exactly what the engine parsed, so you can confirm the backend reads your data the way you intend before activating it: ```ts const { preview } = await admin.policies.validate(policySource) // preview = { // package: "caracal.authz", // rules: ["app_ids", "grants"], // the data documents you defined // default_result: false, // data documents never define result // decisions: [], // the platform contract owns every decision // inputs_referenced: [], // data_referenced: [], // } ``` Use `rules` to confirm the document defines the data tables you intended. The preview is a static read of the source; for an end-to-end decision run a [simulation](/v1.0/guides/activate-policy-set/) with representative input against the platform decision contract. ## Iterate from a denied request When a real request is denied, you do not have to guess the input: the audit explain endpoint reconstructs a redaction-safe policy input for every denied decision, and you replay it against a candidate policy-set version before activating the fix. The workflow, snippet, and caveats live in [Iterate from real denials](/v1.0/guides/activate-policy-set/#iterate-from-real-denials); [Iterate Policy Safely](/v1.0/examples/policy-iterate/) automates the whole loop. ## Keep policies reviewable * Default to deny. * Keep resource identifiers stable and scopes action-oriented. * Keep grant data normalized: one `grants` entry per resource view, rather than duplicating the same scope sets across documents. * Split policies by ownership only when separate review or activation is useful. Next, [activate the policy in a policy set](/v1.0/guides/activate-policy-set/). ## Validate the authored data Validation must return `valid: true`; preview must list only intended data rules; simulation must allow the intended role and deny a missing role, extra scope, wrong resource, and confinement escape. Syntax validity alone is not production readiness. :::caution[Failure point: identity fields] The acting application is `input.principal.id`; Session labels are `input.principal.labels`. Do not invent `input.user`, `input.application`, or an authoritative subject field. A Federated user is a separate explicit identity flow, not default policy authority. ::: ## Next Step [Activate a Policy Set](/v1.0/guides/activate-policy-set/) with the allow and deny simulations used here. --- # Activate a Policy Set # URL: https://docs.caracal.run/v1.0/guides/activate-policy-set/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/activate-policy-set.md # Type: page # Concepts: # Requires: --- Use this guide when validated policy data must become live. The job is complete only after simulation, activation, propagation, and a real audited exchange; creating a policy version alone changes no authorization result. ## Prerequisites * A validated policy version and representative allow and deny inputs. * A policy set in the target zone and the current active version recorded for rollback. * Console access or an Admin client in a trusted automation environment. ## The mental model Four immutable layers separate authoring from what runs: | Layer | What it is | Mutable? | | --- | --- | --- | | Policy | A named Rego document. | The name; content is versioned. | | Policy version | One immutable snapshot of policy content. | No. | | Policy-set version | An immutable bundle of policy versions. | No. | | Active policy-set version | The one bundle the STS evaluates for a zone. | The pointer moves on activation. | You edit by creating a **new** version and moving the active pointer to it. Nothing already evaluated is ever rewritten, so every audit decision ties back to the exact policy-set version and manifest hash that produced it. A zone is governed by exactly one policy set at a time. Activating a version of one set deactivates any other set in the zone in the same transaction, so which policies govern a zone is never ambiguous. Saving or activating a version also compiles the full policy bundle on the STS runtime engine, so a manifest whose policies conflict (for example, two documents defining the same data key) is rejected before it can be pinned or promoted. ## Activation flow ```mermaid flowchart LR Policy["Create policy"] --> Version["Create immutable policy version"] Version --> Set["Add policy-set version"] Set --> Simulate["Simulate"] Simulate --> Activate["Activate"] Activate --> Audit["Audit policy decisions"] ``` ## Web Console Workflow 1. Open the web console for your deployment. 2. Select **Policies** and create or update the Rego policy. 3. Select **Policy Sets** and create a set for the zone. 4. Add the policy version to a new policy-set version. 5. Simulate the version with a representative input. 6. Activate the version when the simulated decision matches the intended behavior. 7. Use **Audit** to follow the decision trace after the first real request. ## Automation workflow ```ts import { AdminClient } from "@caracalai/admin"; const admin = new AdminClient({ apiUrl: process.env.CARACAL_API_URL!, adminToken: process.env.CARACAL_ADMIN_TOKEN!, }); const policy = await admin.policies.create(process.env.CARACAL_ZONE_ID!, { name: "pipernet-read", content: policySource, }); const set = await admin.policySets.create(process.env.CARACAL_ZONE_ID!, "pipernet"); const version = await admin.policySets.addVersion(process.env.CARACAL_ZONE_ID!, set.id, [ { policy_version_id: policy.version.id }, ]); const simulation = await admin.policySets.simulate(process.env.CARACAL_ZONE_ID!, set.id, version.id, sampleInput); if (!simulation.would_activate) { throw new Error(simulation.explanation.reason); } await admin.policySets.activate(process.env.CARACAL_ZONE_ID!, set.id, version.id); ``` ## Wait for propagation Activation returns `202 Accepted`: the active pointer moves durably, then the STS runtime reloads the bundle through a durable invalidation stream. Poll the activation status until the runtime reports the version as loaded before treating the rollout as live: ```ts let status = await admin.policySets.activationStatus(zoneId, set.id, version.id); while (status.propagation_status !== "loaded") { if (status.propagation_status === "failed") { throw new Error(status.outbox.last_error ?? "policy rollout failed"); } await new Promise((resolve) => setTimeout(resolve, 2000)); status = await admin.policySets.activationStatus(zoneId, set.id, version.id); } ``` `propagation_status` moves through `waiting_for_outbox` → `waiting_for_sts` → `loaded`. With several STS replicas, the replica that receives the invalidation reloads within about a second; the rest converge through a periodic database poll (60 seconds by default), so a brief mixed-version window across replicas is expected during rollout. ## Validation checklist | Check | Expected result | | --- | --- | | Policy validation | `valid: true` and no blocking warnings. | | Simulation | Representative input returns the intended decision. | | Activation | Zone has the expected active policy-set version. | | Propagation | `activationStatus` reports `propagation_status: "loaded"`. | | First exchange | Audit shows the new policy as a determining policy. | If activation changes expected access, keep the old policy-set version ID in the rollout notes so you can promote it again if needed. ## Iterate from real denials Every denied decision links to the policy-set version that produced it, and the audit explain endpoint reconstructs the policy input for that denial. Stage the fix as a new policy-set version, then simulate the denied input against it: ```ts const trace = await admin.audit.explain(zoneId, requestId); const input = trace.denied[0]?.policy_input; const candidate = await admin.policySets.addVersion(zoneId, set.id, manifest); const check = await admin.policySets.simulate(zoneId, set.id, candidate.id, input); if (check.result?.decision === "allow") { await admin.policySets.activate(zoneId, set.id, candidate.id); } ``` The reconstructed input is redaction-safe: actor and subject claims are never written to audit, so add any claim-dependent fields before simulating. The [Iterate Policy Safely example](/v1.0/examples/policy-iterate/) wraps this loop as a runnable script. ## Likely failure points | Failure | Response | | --- | --- | | Simulation warns or denies the intended input | Correct the data document and create another immutable version. | | Activation is accepted but status never reaches `loaded` | Inspect the outbox identifier and STS readiness; do not assume propagation. | | Real decisions still name the previous manifest | Treat the replica as unconverged and keep rollout monitoring active. | | Rollback is needed | Activate the recorded prior version; never mutate the failed version. | ## Next Step Run [Iterate Policy Safely](/v1.0/examples/policy-iterate/) for the first real denial, or [Debug Authorization Decisions](/v1.0/guides/authorize-access/) when live behavior differs from simulation. --- # Debug Authorization Decisions # URL: https://docs.caracal.run/v1.0/guides/authorize-access/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/authorize-access.md # Type: page # Concepts: # Requires: --- Use this guide when a request is denied, unexpectedly allowed, missing audit evidence, or routed to the wrong protected resource. For production incidents and dependency failures, use [Troubleshoot by Symptom](/v1.0/operations/troubleshooting/). ## Prerequisites * The request ID, zone, application, resource, scopes, and approximate timestamp. * Access to Audit and active policy-set status. * A safe way to reproduce without changing production data. ## Debug Flow ```mermaid flowchart LR Request["Request ID"] --> Trace["Request trace"] Trace --> Resource["Resource and scopes"] Resource --> Policy["Active policy set"] Policy --> Session["Session and revocation"] Session --> Route["Gateway route"] Route --> Audit["Audit evidence"] ``` ## Start with the Request ID Find the request ID from the SDK error, STS response, Gateway response, web console audit event, or application log. Open web console **Audit**, filter by the request ID, and open its decision trace. The trace should show: * application and execution session; * requested resource and scopes; * policy set version; * determining policies; * diagnostics; * final decision; * Gateway result when the request reached Gateway. If you do not have a request ID, reproduce the request with your runtime configuration and capture the SDK, STS, or Gateway output. ## Common Authorization Failures | Symptom | Check | Fix | | --------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | Exchange is denied | Active policy does not allow the application, Subject, resource, or scopes. | Update the Rego policy, simulate the denied input, and activate a new policy-set version. | | Scope is missing | Resource does not define the scope, or policy allowlist omits it. | Add the scope to the resource and policy deliberately. | | Wrong resource is evaluated | App used the wrong resource ID or Gateway header. | Use the resource ID from Console and `X-Caracal-Resource`. | | Policy change has no effect | New policy version is not in the active policy-set version. | Create and activate a new policy-set version. | | Access continues after revocation | Resource server is not consuming revocation state, or Gateway route is not used. | Confirm Gateway-mediated routing or shared revocation consumers for adapters. | | Gateway returns 403 | Mandate is expired, missing, revoked, or scoped for another resource. | Rerun the workload for a fresh mandate and confirm resource bindings. | | Upstream succeeds without audit | Request bypassed Gateway or adapter result audit. | Route through Gateway or emit service-side action-result audit after adapter verification. | | No audit event appears | Request never reached STS/Gateway, wrong zone is selected, or audit ingestion is delayed. | Confirm profile, selected zone, and request ID; refresh audit after a short wait. | ## Iterate from a Denial Every denied decision links to the policy-set version that produced it, and the audit explain endpoint reconstructs a redaction-safe policy input you can replay against a candidate version before activating the fix. Follow [Iterate from real denials](/v1.0/guides/activate-policy-set/#iterate-from-real-denials) for the snippet and caveats, or run [Iterate Policy Safely](/v1.0/examples/policy-iterate/) to automate the loop. ## Authorization Design Rules * Allow only the scopes an application and Subject need. * Keep policies resource-specific instead of allowing broad cross-resource access. * Express context-sensitive conditions in policy rather than widening scopes. * Revoke active sessions when a Subject leaves a workflow or an application no longer needs a resource. * Use diagnostics for expected denial paths so request traces explain what to change. ## Related Pages * [Define Resources and Providers](/v1.0/guides/resources-providers/) * [Author Policy Data](/v1.0/guides/author-policy/) * [Activate a Policy Set](/v1.0/guides/activate-policy-set/) * [Resources and Grants](/v1.0/concepts/resource-grant/) * [Policies and Policy Sets](/v1.0/concepts/policy/) * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) ## Validation and expected result Reproduce once after correction. The trace must name the intended resource, Session/Delegation, active manifest, determining data, final decision, and Gateway result. A deny fix is complete only when nearby negative cases still deny. :::caution[Failure point: changing several layers] Do not edit resource scopes, policy data, application labels, and Gateway routing simultaneously. Trace from request ID inward and change the first incorrect boundary; otherwise the new allow cannot be explained. ::: ## Next Step Use [Iterate Policy Safely](/v1.0/examples/policy-iterate/) for policy-data correction or [Check Provider Readiness](/v1.0/examples/provider-preflight/) for route/provider failures. --- # Integrate the TypeScript SDK # URL: https://docs.caracal.run/v1.0/guides/sdk-typescript/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/sdk-typescript.md # Type: page # Concepts: # Requires: --- Use `@caracalai/sdk` in a Node application that must create governed Sessions, narrow authority, call Gateway-routed resources, or propagate verified Caracal context. Resource servers that only verify inbound mandates should use [the Express adapter](/v1.0/guides/protect-express/) or [verify package](/v1.0/sdks/verify/) instead. ## Prerequisites * A managed application credential, active policy, resource, provider binding, and Gateway route. * `CARACAL_CONFIG` or a complete `CARACAL_*` environment configuration. * A destination timeout and idempotency plan for mutating calls. ## Install ```bash npm install @caracalai/sdk ``` ## Configure `new Caracal()` loads exactly the profile named by `CARACAL_CONFIG` when set; otherwise it loads `CARACAL_*` environment variables. It does not search default profile or credential paths, and conflicting credential modes fail at startup. Use `Caracal.fromClientSecret(...)` for complete static credentials supplied in code. Power-user loaders and resolver-backed credentials live in `@caracalai/sdk/advanced`. ## Connect and call a protected resource The smallest protected call needs no session code at all: `applicationTransport()` pins a fetch to one resource and calls as the application's own identity, provisioning the required Session and Delegation for you. ```ts import { Caracal } from '@caracalai/sdk' const caracal = new Caracal() const governedFetch = caracal.applicationTransport('resource://pipernet', { scopes: ['pipernet:read'], }) const target = caracal.gatewayRequest('resource://pipernet', '/reports') const response = await governedFetch(target.url) ``` When your code already runs inside a governed Session - an agent step, a worker task - call through the Session instead: `fetch()` mints a one-shot Gateway mandate under the Session's authority. ```ts import { Authority, Caracal } from '@caracalai/sdk' const caracal = new Caracal() const resourceId = 'resource://pipernet' await caracal.session(async () => { await caracal.session( async () => { await caracal.fetch(resourceId, '/reports', { scopes: ['pipernet:read'] }) }, { authority: Authority.narrow(['pipernet:read'], { resourceId, ttlSeconds: 600, }), }, ) }) ``` Why two levels? The outer `session()` is the lifecycle parent; the narrowed child creates the positive-TTL Delegation that resource authority requires. Both Sessions terminate when their callbacks exit. `applicationTransport()` builds exactly this structure internally, which is why it is the right starting point for application-owned calls. To make agents distinguishable in policy and audit, pass `labels`. These become `input.principal.labels`, so several agents under one application stay separable without one application per agent. `labels` are descriptive, for policy and audit, not grants; authority always comes from scopes and Delegation. The Session lifecycle is handled for you: `session()` records a `task` Session, while `startSession()` records a heartbeat-leased `service` Session. ```ts await caracal.session( async (ctx) => { console.log('refund Session', ctx.sessionId) }, { labels: ['refund-agent'] }, ) ``` See [If many agents share one managed application, can policy and audit still tell them apart?](/v1.0/reference/faq/#faq-008). ## Long-lived Sessions Daemons and workers that outlive a single request use `startSession()` instead of `session()`. It returns a handle you own: the SDK renews the lease from an independent background timer by default, and you retire the Session with `close()`. Each renewal extends the lease. If heartbeats stop before the lease expires, Coordinator suspends the Session for explicit recovery or termination. The stored protocol lifecycle is `service`; unlike `task`, it is not subject to the wall-clock TTL sweeper. ```ts const svc = await caracal.startSession({ labels: ['fiona-worker'] }) try { while (running) { await caracal.bind(svc.context, async () => { await doWork(await caracal.headersAsync()) }) // lease renews in the background } } finally { await svc.close() } ``` The renewal cadence is controlled by `heartbeatIntervalMs`: | `heartbeatIntervalMs` | Behavior | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | unset | Cadence derives from the server lease: renew at roughly a third of the remaining lease, with jitter. The right default for almost every worker. | | positive | Fixed cadence. Use when you need deterministic renewal timing, for example under test clocks. | | `0` or negative | No background timer; the lease is renewed only by your own `heartbeat()` calls, and a missed renewal lets the lease lapse. | The handle exposes `deadlineAt` and `leaseGeneration`. The generation is a monotonic ownership token: every heartbeat and `close()` sends it, so a process holding an earlier generation cannot renew or terminate the Session after another process takes ownership. Because renewal runs on an independent timer, the lease stays current even while your code is blocked on a long `await` (a streaming response, a slow tool). Transient renewal errors are logged and retried on the next tick rather than crashing the worker. If Coordinator reports the Session permanently gone or the generation has been fenced by a new holder, the timer stops and `onLeaseLost` fires once so the worker can resign instead of spinning. An expired lease instead suspends the Session and reports `suspended` through `onStateChange`. The handle's `status` follows Coordinator heartbeat responses. If it becomes `suspended`, automatic heartbeat stops and `onStateChange` reports the transition. Stop taking work, resolve the cause, resume the Session through the control plane, and call `attachSession()` to restart lease renewal. `onLeaseLost` remains reserved for terminal loss. ```ts const svc = await caracal.startSession({ labels: ['voice-worker'], onLeaseLost: () => shutdown(), }) ``` A renewal cannot run while the event loop is blocked synchronously; in that case the lease correctly lapses, which is the liveness signal working as intended. A worker that restarts does not need a fresh session: persist `svc.sessionId` and re-attach with `attachSession()`. Attach atomically acquires a new lease generation and renews the deadline, fencing every older handle. A Session the Coordinator no longer holds live fails with `CoordinatorError`; a suspended Session must be resumed through the control plane before attachment. The returned handle behaves like one from `startSession()`: ```ts const svc = await caracal.attachSession(persistedSessionId, { onLeaseLost: () => shutdown(), }) ``` The rebuilt context carries the Session identity only; Delegations bound by the previous holder are re-presented with `acceptDelegation()`. ## Start a narrowed child The nested pattern from the [connect example](#connect-and-call-a-protected-resource) is how narrowing always looks; add typed constraints when the child's authority should carry explicit bounds: ```ts authority: Authority.narrow(['pipernet:read'], { resourceId: 'resource://pipernet', constraints: { maxHops: 1 }, ttlSeconds: 600, }), ``` A plain `session()` runs the child under its parent's effective authority - the application's authority for a root parent, or the parent's narrowed slice when the parent was itself narrowed (transitive least-privilege). Pass `authority: Authority.narrow(...)` only when the child should hold a smaller subset, or `Authority.none()` for a child with no inherited authority. Use `delegate()` when you need to grant authority to a Session that already exists, typically in another application: it returns the delegation, and the receiving session presents it with `acceptDelegation(delegationId, fn)` - the full two-sided flow is in [Implement Multi-Agent Delegation](/v1.0/guides/delegation/). ## Handle approvals A mint whose scope is approval-gated throws `ApprovalRequiredError`. `withApproval` runs the whole flow - catch the hold, wait for the decision, retry with the approval id - in one call: ```ts const mandate = await caracal.withApproval((approvalId) => caracal.mintMandate('resource://pipernet', ['pipernet:admin'], { approvalId }), ) ``` A rejected, expired, or already-consumed decision rethrows the original `ApprovalRequiredError`; its `approvalId` lets you resume waiting later with `waitForApproval`, which returns the typed final state (`approved`, `rejected`, `expired`, `consumed`, or `pending`). ## Route through the Gateway ```ts await caracal.fetch('resource://pipernet', '/reports', { scopes: ['pipernet:read'], }) ``` For provider SDKs that accept a custom `fetch`, pass `caracal.transport({ scopes: ['pipernet:read'], propagation: 'gateway-only' })` so Caracal mints a Gateway-ingress mandate and applies routing automatically. `gatewayRequest()` only builds a URL and routing header; it does not authenticate the request. ## Call as the application Background jobs with no inbound user context use `applicationTransport(resourceId, { scopes })` - the same call the [connect example](#connect-and-call-a-protected-resource) opens with. There is no ambient authority to borrow, so each mint cycle builds the authority the platform requires: a source and target Session pair plus a narrowing Delegation, which keeps every request session-attributed, policy-checked, and delegation-bounded in audit. Provisioning costs and caching behavior are in the [TypeScript SDK reference](/v1.0/sdks/typescript/#call-protected-resources); tune `mandateTtlSeconds` for authority lifetime, not to eliminate per-request exchange. ```ts const llm = new OpenAI({ baseURL: 'https://api.pipernet.example/v1', apiKey: 'unused-gateway-injects-credentials', fetch: caracal.applicationTransport('resource://pipernet', { scopes: ['pipernet:chat'] }), }) ``` ## Shut down cleanly `await caracal.close()` terminally releases client-held state: cached application mandates and in-flight mint cycles are dropped, the credential exchanger's cached lifecycle token is invalidated, and the Sessions backing released application transports are terminated best-effort (anything missed retires on its own TTL). Repeated close is safe. Construct a new client for later work; operations on the closed client fail deterministically. Two things `close()` deliberately does **not** do. It does not retire `startSession()`/`attachSession()` handles - those Sessions are yours, so call `handle.close()` on each before the process exits, or persist the Session ID and re-attach after restart. It also does not abort in-flight requests on transports you handed to provider SDKs; bound them with `timeoutMs` or an `AbortSignal` and let them drain. ## Troubleshooting | Symptom | Check | | ------------------------------- | -------------------------------------------------------------------------------------------- | | `Caracal: missing ...` | Confirm the named profile or required `CARACAL_*` variables are present. | | Root headers rejected | Call `headersAsync({ asApplication: true })` only when service-root identity is intentional. | | Delegation fails | Ensure the call runs inside `session()` or another bound context. | | Gateway request misses resource | Confirm `gateway_url`, resource bindings, and `X-Caracal-Resource`. | ## Validate the integration Run an allowed call, a call with one extra scope, a revoked-Session call, and a timeout. Expect only the allowed call to reach the upstream; expect the SDK error or Gateway response to carry a request ID for Audit. Close every owned Session handle and then `await caracal.close()` during shutdown. :::caution[Failure point: automatic retries] The SDK retries selected Coordinator and STS transient failures with stable operation identity. It does not retry Gateway side effects. Use [Safe Retries and Idempotency](/v1.0/guides/idempotency/) before retrying a mutation. ::: For exact constructors, options, and return types, use [TypeScript SDK reference](/v1.0/sdks/typescript/). ## Next Step Add [multi-agent delegation](/v1.0/guides/delegation/) only if one Session must hand a narrower slice to another; otherwise validate the resource boundary with [Protect a Gateway-Routed HTTP API](/v1.0/guides/protect-gateway-http/). --- # Integrate the Python SDK # URL: https://docs.caracal.run/v1.0/guides/sdk-python/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/sdk-python.md # Type: page # Concepts: # Requires: --- Use `caracalai-sdk` in an async Python application that must create governed Sessions, narrow authority, route HTTP through Gateway, or bind verified inbound context. Resource servers that only verify inbound mandates should use [the ASGI adapter](/v1.0/guides/protect-fastapi/) or [verify package](/v1.0/sdks/verify/). ## Prerequisites * A managed application credential, active policy, resource/provider binding, and Gateway route. * An async runtime; move CPU-bound work off the event loop so service heartbeats can run. * A timeout and destination idempotency plan for mutating requests. ## Install ```bash pip install caracalai-sdk ``` ## Connect ```python from caracalai import Caracal caracal = Caracal() ``` `Caracal()` loads exactly the profile named by `CARACAL_CONFIG` when set; otherwise it loads `CARACAL_*` environment variables. It does not search default profile or credential paths, and conflicting credential modes fail at startup. Use `Caracal.from_client_secret(...)` for complete static credentials supplied in code. Power-user loaders and resolver-backed credentials live in `caracalai.advanced`. ## Start a Session and call a protected resource The smallest protected call needs no session code at all: `application_transport()` pins an `httpx.AsyncClient` to one resource and calls as the application's own identity, provisioning the required Session and Delegation for you. ```python from caracalai import Caracal caracal = Caracal() target = caracal.gateway_request("resource://pipernet", "/reports") async with caracal.application_transport( "resource://pipernet", scopes=["pipernet:read"], ) as governed: response = await governed.get(target.url) ``` When your code already runs inside a governed Session - an agent step, a worker task - call through the Session instead: `fetch()` mints a one-shot Gateway mandate under the Session's authority. ```python from caracalai import Authority, Caracal caracal = Caracal() resource_id = "resource://pipernet" async with caracal.session() as parent: async with caracal.session( parent_ctx=parent, authority=Authority.narrow( ["pipernet:read"], resource_id=resource_id, ttl_seconds=600, ), ): await caracal.fetch(resource_id, "/reports", scopes=["pipernet:read"]) ``` Why two levels? The outer Session is the lifecycle parent; the narrowed child creates the positive-TTL Delegation that resource authority requires. Both async context managers retire their Sessions on exit. `application_transport()` builds exactly this structure internally, which is why it is the right starting point for application-owned calls. To make AI-agent executions distinguishable in policy and audit, pass `labels` when starting a Session. These become `input.principal.labels`, so several agents under one application stay separable without one application per agent. Labels describe work; scopes and Delegations bound authority. `session()` retires the task Session when the block exits, while `start_session()` starts a heartbeat-leased long-lived Session whose protocol lifecycle is `service`. ```python async with caracal.session(labels=["refund-agent"]) as ctx: print("refund Session", ctx.session_id) ``` See [If many agents share one managed application, can policy and audit still tell them apart?](/v1.0/reference/faq/#faq-008). ## Long-lived Sessions Daemons and workers that outlive a single request use `start_session()` instead of `session()`. It returns a handle you own: the SDK renews the lease from an independent background task by default, and you retire the Session with `aclose()`. If heartbeats stop before the lease expires, Coordinator suspends the Session for explicit recovery or termination. The stored protocol lifecycle is `service`; unlike `task`, it is not subject to the wall-clock TTL sweeper. ```python svc = await caracal.start_session(labels=["fiona-worker"]) try: while running: async with caracal.bind(svc.context): await do_work(caracal.headers()) finally: await svc.aclose() ``` Narrowing requires an active parent. For a long-lived hierarchy, bind a parent service handle while starting its narrowed service child. The child context carries the edge, and closing each handle retires its own Session. ```python parent = await caracal.start_session(labels=["pipernet-orchestrator"]) async with caracal.bind(parent.context): svc = await caracal.start_session( labels=["fiona-worker"], authority=Authority.narrow( ["pipernet:read"], ttl_seconds=600, resource_id="resource://pipernet", ), ) ``` The renewal cadence follows the server lease, renewing at roughly a third of the remaining lease with jitter. Pass a positive `heartbeat_interval` to fix the cadence, or `0` to disable the background task and call `heartbeat()` yourself. Because renewal runs on an independent task, the lease stays current even while your code is blocked on a long `await` (a streaming response, a slow tool). The handle exposes `heartbeat_deadline_at` and `lease_generation`. Every heartbeat and `aclose()` sends that monotonic ownership token, so a process holding an earlier generation cannot renew or terminate the Session after another process takes ownership. Transient renewal errors are logged and retried on the next tick rather than crashing the worker. If Coordinator reports the Session permanently gone or the generation has been fenced by a new holder, the task stops and `on_lease_lost` fires once so the worker can resign instead of spinning. An expired lease instead suspends the Session and reports `suspended` through `on_state_change`. The handle's `status` follows Coordinator heartbeat responses. If it becomes `suspended`, automatic heartbeat stops and `on_state_change` reports the transition. Stop taking work, resolve the cause, resume the Session through the control plane, and call `attach_session()` to restart lease renewal. `on_lease_lost` remains reserved for terminal loss. ```python svc = await caracal.start_session( labels=["voice-worker"], on_lease_lost=lambda exc: shutdown(), ) ``` A renewal cannot run while the event loop is blocked synchronously (CPU-bound work with no `await`); in that case the lease correctly lapses, which is the liveness signal working as intended. A worker that restarts does not need a fresh session: persist `svc.session_id` and re-attach with `attach_session()`. Attach atomically acquires a new lease generation and renews the deadline, fencing every older handle. A Session the Coordinator no longer holds live fails with `CoordinatorError`; a suspended Session must be resumed through the control plane before attachment. The returned handle behaves like one from `start_session()`: ```python svc = await caracal.attach_session( persisted_session_id, on_lease_lost=lambda exc: shutdown(), ) ``` The rebuilt context carries the Session identity only; Delegations bound by the previous holder are re-presented with `accept_delegation()`. ## Start a narrowed child ```python from caracalai import Authority, DelegationConstraints async with caracal.session() as parent: async with caracal.session( parent_ctx=parent, authority=Authority.narrow( ["pipernet:read"], resource_id="resource://pipernet", constraints=DelegationConstraints(max_hops=1), ttl_seconds=600, ), ): headers = caracal.headers() ``` A plain `caracal.session()` runs the child under its parent's effective authority - the application's authority for a root parent, or the parent's narrowed slice when the parent was itself narrowed (transitive least-privilege). Pass `authority=Authority.narrow(...)` only when the child should hold a smaller subset, or `authority=Authority.none()` for a child with no inherited authority. Use `await caracal.delegate(...)` when you need to grant authority to a Session that already exists, typically in another application: it returns the edge, and the receiving session presents it with `async with caracal.accept_delegation(edge_id)`. Use `async with caracal.bind(ctx)` before handing a captured context to a background task. ## Handle approvals A mint whose scope is approval-gated raises `ApprovalRequired`. `with_approval` runs the whole flow - catch the hold, wait for the decision, retry with the approval id - in one call. Its callback must return an awaitable; `mint_mandate` is synchronous, so wrap it with `asyncio.to_thread`: ```python import asyncio mandate = await caracal.with_approval( lambda approval_id: asyncio.to_thread( caracal.mint_mandate, "resource://pipernet", ["pipernet:admin"], approval_id=approval_id, ) ) ``` A rejected, expired, or already-consumed decision re-raises the original `ApprovalRequired`; its approval id lets you resume waiting later with `wait_for_approval`, which returns the typed final state (`approved`, `rejected`, `expired`, `consumed`, or `pending`). [Human Approval](/v1.0/guides/human-approval/) covers the tiers and decision paths. ## Use httpx transport injection ```python async with caracal.session(): async with caracal.transport( scopes=["pipernet:read"], propagation="gateway-only" ) as client: await client.get("https://api.pipernet.example/reports") ``` The transport mints the scoped Gateway-ingress mandate, injects Caracal envelope headers, and rewrites configured resource-bound URLs through the Gateway. Calls without `scopes` require an already Gateway-class bound token. ## ASGI propagation After a verifier boundary, use the SDK middleware to bind an inbound Caracal envelope into request context: ```python from fastapi import FastAPI from caracalai import Caracal caracal = Caracal() app = FastAPI() app.add_middleware(caracal.context_middleware(trusted_propagation=True)) ``` This propagation-only form is valid only when a Gateway or adapter already verified the request. At an untrusted ingress, pass a complete `verifier=` instead; production mode rejects propagation that is neither verified nor explicitly trusted. ## Troubleshooting | Symptom | Check | | --------------------------------- | --------------------------------------------------------------------------------- | | `Caracal: missing ...` | Confirm the named profile or required environment variables. | | `headers()` refuses root identity | Bind a context or pass `as_application=True` only for trusted service-root calls. | | Background task loses context | Capture the context and rebind with `async with caracal.bind(ctx)`. | | Gateway routing misses | Confirm resource bindings and `gateway_url`. | ## Validate the integration Exercise allow, extra-scope deny, revoked Session, cancellation, and clean shutdown. Only the allow case may reach the upstream. Close `httpx` transports and owned service handles, then `await caracal.aclose()` when the client is no longer used. :::caution[Failure point: client lifetime] Do not create one SDK or `httpx` client per request. Reuse them, bind the current Session context around each unit of work, and close them during application shutdown. ::: For exact Python signatures and sync/async variants, use [Python SDK reference](/v1.0/sdks/python/). ## Next Step Protect inbound FastAPI traffic with [the ASGI adapter](/v1.0/guides/protect-fastapi/) or implement [multi-agent delegation](/v1.0/guides/delegation/) for cross-Session authority. --- # Integrate the Go SDK # URL: https://docs.caracal.run/v1.0/guides/sdk-go/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/sdk-go.md # Type: page # Concepts: # Requires: --- Use the Go SDK in an application that must carry governed authority in `context.Context`, create Sessions, narrow Delegation, or route HTTP through Gateway. A resource server that only verifies inbound mandates should use [the net/http adapter](/v1.0/guides/protect-nethttp/). ## Prerequisites * A managed application credential, active policy, resource/provider binding, and Gateway route. * Request deadlines propagated through `context.Context`. * A destination idempotency strategy for mutating requests. ## Install ```bash go get github.com/garudex-labs/caracal/packages/sdk/go ``` ## Connect The smallest protected call needs no session code at all: `ApplicationTransport` pins an `*http.Client` to one resource and calls as the application's own identity, provisioning the required Session and Delegation for you. ```go package main import ( "io" "log" caracal "github.com/garudex-labs/caracal/packages/sdk/go" ) func main() { client, err := caracal.New() if err != nil { log.Fatal(err) } defer client.Close() // nil base: the SDK constructs its own HTTP client. governed, err := client.ApplicationTransport(nil, "resource://pipernet", caracal.ApplicationTransportOptions{ Scopes: []string{"pipernet:read"}, }) if err != nil { log.Fatal(err) } target, err := client.GatewayRequest("resource://pipernet", "/reports") if err != nil { log.Fatal(err) } resp, err := governed.Get(target.URL) if err != nil { log.Fatal(err) } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) log.Println(string(body)) } ``` `New()` loads exactly the profile named by `CARACAL_CONFIG` when set; otherwise it loads `CARACAL_*` environment variables. It does not search default profile or credential paths. ## Use an HTTP client ```go httpClient := client.Transport(nil, caracal.CallOptions{ Scopes: []string{"pipernet:read"}, Propagation: caracal.PropagationGatewayOnly, }) err := client.Session(context.Background(), func(ctx context.Context) error { req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.pipernet.example/reports", nil) if err != nil { return err } _, err = httpClient.Do(req) return err }) ``` The transport mints the scoped Gateway-ingress mandate and injects Caracal envelope headers from `context.Context`. For explicit routing metadata, use `GatewayRequest()` and send the resulting request through the scoped transport; the helper alone does not authenticate it. ## Long-lived Sessions Daemons and workers that outlive a single request use `StartSession` instead of `Session`. It returns a handle you own: the SDK renews the lease from a background goroutine by default, and you retire the Session with `Close`. If heartbeats stop before the lease expires, Coordinator suspends the Session for explicit recovery or termination; its stored protocol lifecycle is `service`. ```go svc, err := client.StartSession(context.Background(), caracal.StartSessionOptions{ Labels: []string{"fiona-worker"}, }) if err != nil { return err } defer svc.Close(context.Background()) for running { doWork(svc.Context) // lease renews in the background } ``` The renewal cadence follows the server lease, renewing at roughly a third of the remaining lease with jitter. Set `StartSessionOptions.HeartbeatInterval` to a positive duration to fix the cadence, or a negative one to disable the background goroutine and call `Heartbeat` yourself. `LeaseGeneration()` exposes the monotonic ownership token sent by every heartbeat and `Close`, so an earlier holder cannot renew or terminate the Session after another process takes ownership. Transient renewal errors are logged and retried on the next tick; if the Coordinator reports the Session permanently gone or fenced, the goroutine stops and `StartSessionOptions.OnLeaseLost` fires once so the worker can resign instead of spinning. `Status()` follows Coordinator heartbeat responses. If it becomes `suspended`, automatic heartbeat stops and `OnStateChange` reports the transition. Stop taking work, resolve the cause, resume the Session through the control plane, and call `AttachSession` to restart lease renewal. `OnLeaseLost` remains reserved for terminal loss. A worker that restarts does not need a fresh session: persist `svc.SessionID()` and re-attach with `AttachSession`. Attach atomically acquires a new lease generation and renews the deadline, fencing every older handle. A Session the Coordinator no longer holds live fails with `*CoordinatorError`; a suspended Session must be resumed through the control plane before attachment. The returned handle behaves like one from `StartSession`: ```go svc, err := client.AttachSession(ctx, persistedSessionID, caracal.AttachSessionOptions{ OnLeaseLost: func(error) { shutdown() }, }) ``` The rebuilt context carries the Session identity only; Delegations bound by the previous holder are re-presented with `AcceptDelegation`. ## Start a narrowed child ```go err := client.Session(context.Background(), func(ctx context.Context) error { return client.Session(ctx, func(child context.Context) error { _, _ = client.Current(child) return nil }, caracal.SessionOptions{ Authority: caracal.AuthorityNarrow([]string{"pipernet:read"}, caracal.NarrowOptions{ Constraints: &caracal.DelegationConstraints{MaxHops: 1}, TTLSeconds: 600, }), }) }) ``` A plain `client.Session` runs the child under its parent's effective authority - the application's authority for a root parent, or the parent's narrowed slice when the parent was itself narrowed (transitive least-privilege). Set `SessionOptions.Authority` to `caracal.AuthorityNarrow([]string{...})` only when the child should hold a least-privilege subset, or `caracal.AuthorityNone()` for a child with no inherited authority. Use `client.Delegate` to grant authority to a Session that already exists, typically in another application: it returns the edge, and the receiving session presents it with `client.AcceptDelegation(ctx, edgeID)` - the full two-sided flow is in [Implement Multi-Agent Delegation](/v1.0/guides/delegation/). Use `Current(ctx)` to inspect the bound Caracal context and `Headers(ctx)` to project it to outbound HTTP headers. ## Handle approvals A mint whose scope is approval-gated returns `*oauth.ApprovalRequiredError`. `caracal.WithApproval` runs the whole flow - catch the hold, wait for the decision, retry with the approval id - in one call: ```go mandate, err := caracal.WithApproval(ctx, client, 10*time.Minute, func(ctx context.Context, approvalID string) (oauth.MintedMandate, error) { return client.MintMandate(ctx, "resource://pipernet", []string{"pipernet:admin"}, caracal.MintMandateOptions{ApprovalID: approvalID}) }) ``` A rejected, expired, or already-consumed decision returns the original `*oauth.ApprovalRequiredError`; its `ApprovalID` lets you resume waiting later with `client.WaitForApproval(ctx, approvalID, timeout)`, which returns the typed final state. [Human Approval](/v1.0/guides/human-approval/) covers the tiers and decision paths. ## Troubleshooting | Symptom | Check | | ------------------------------- | -------------------------------------------------------------------------------------------- | | Missing config error | Confirm the runtime profile or required environment variables. | | `Headers` without context fails | Call inside `Session`, `Delegate`, or pass `CallOptions{AsApplication: true}` intentionally. | | Delegation fails | Ensure delegation runs from a context with an active Session. | | Gateway URL error | Confirm the runtime profile includes `gateway_url`. | ## Validate the integration Run allow, extra-scope deny, revoked Session, canceled context, and shutdown cases. Only the allow case may reach the upstream. Close every `SessionHandle`, then call `client.Close()` after transports have drained. :::caution[Failure point: context] Do not replace the callback or handle context with `context.Background()` inside governed work. That drops Session and Delegation authority even though the process still has application credentials. ::: For exact exported types and options, use [Go SDK reference](/v1.0/sdks/go/). ## Next Step Protect inbound handlers with [the net/http adapter](/v1.0/guides/protect-nethttp/) or implement [multi-agent delegation](/v1.0/guides/delegation/). --- # Run an Agent with caracal run # URL: https://docs.caracal.run/v1.0/guides/runtime-run/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/runtime-run.md # Type: page # Concepts: # Requires: --- `caracal run` starts a local subprocess with short-lived **provider credentials** injected as environment variables: the brokered OAuth token or sealed API key of each bound resource's provider, released only after Caracal authorizes the launch. The child process then calls the provider directly with its native credential - the Gateway is not in this path. The workload carries only its workload ID and secret; the credential bindings live in the web console. Use it for development, demos, and controlled local runs of existing CLIs that read provider-native environment variables such as `OPENAI_API_KEY`. Do not use it for a daemon that must renew credentials, for request-level Gateway enforcement, or as a process supervisor. When you want per-request policy checks, Gateway brokering, and action-result audit, use an SDK transport instead. ## Prerequisites * A ready runtime, Launcher workload, owner-only workload secret, and at least one launch binding. * A provider with `allow_runtime_injection=true` and an active policy permitting binding scopes. * A child process that reads the configured environment variable and can finish before credential expiry. ## What a Binding Injects A binding names an environment variable, a resource, and the scopes the policy decision is made against. What lands in the variable is the resource's **provider credential**: | Provider kind | Injected value | | --- | --- | | `api_key`, `bearer_token` | The sealed static key or token itself. Scope selection gates whether Caracal releases it - the value still carries the provider's full authority. | | `oauth2_client_credentials` | A brokered short-lived provider access token. | | `none`, `caracal_mandate`, `http_basic`, `oauth2_authorization_code` | Never injected - the launch is refused for these kinds (`http_basic` is a two-part pair, authorization-code connections belong to a consenting user, and the other two have no injectable credential). | Every eligible kind also requires `allow_runtime_injection=true` on the provider; it is off by default because injection moves enforcement of the actual call from the Gateway into your process. ```mermaid flowchart LR Run[caracal run] -->|workload proof| STS STS -->|policy decision| STS STS -->|provider credential| Run Run -->|inject env var| Child[Child process] Child -->|provider-native call| Provider[Upstream provider] ``` ## Prepare the workload 1. Run `caracal up`. 2. Sign in to the web console and use **Guided setup** to create the zone, provider, resource, and policy. Enable **runtime injection** on the provider. 3. On **Services → Launcher**, create a workload. Store its secret in the owner-only file at `/runtime//secret`, or export `CARACAL_WORKLOAD_SECRET`; the secret stays retrievable from the Launcher page, with every reveal audited. 4. On the same page, bind an environment variable to each resource the workload needs and select the scopes each policy decision should evaluate. The page then shows the exact launch commands. See [Configure Workloads](/v1.0/runtime-console/config-file/). ## Run a command ```bash export CARACAL_WORKLOAD_ID= caracal run -- npm start ``` The launcher fetches the workload's launch bindings from STS, requests the provider credential for each binding after a policy decision on only its selected scopes, and injects the results into the configured environment variables. The child environment is otherwise scrubbed: `CARACAL_*` configuration variables stay with the launcher, and only a small allowlist such as `PATH`, `HOME`, locale, and `XDG_*` directories is inherited. Credentials are obtained once at launch and never renewed; long-running workloads should use an SDK. If policy requires an Approval for a binding, the launch pauses and emits an `approval_required` line on stderr until the hold is decided. See [Run Workloads](/v1.0/runtime-console/runtime/) for the full contract. ## Validate the run | Check | Command or surface | | --- | --- | | Runtime is ready | `caracal status --ready` | | Credential is injected | `caracal run -- printenv OPENAI_API_KEY` (use your configured `env` name) | | First request succeeds | Run the child once; it calls the provider directly with the injected credential. | | Audit captured the launch | Web console **Audit** shows the credential-injection decision. | Expected result: only configured binding variables and optional `_EXPIRES_AT` values enter the child; workload identity and other `CARACAL_*` variables do not. The launcher exits with the child process's exit code. ## Troubleshooting | Symptom | Fix | | --- | --- | | `workload identity not found` | Set `CARACAL_WORKLOAD_ID` and a workload secret source. | | `invalid workload credentials` | The workload ID or secret is wrong or was rotated; copy the current values from **Services → Launcher**. | | `no credential bindings configured` | Define launch bindings for this workload on **Services → Launcher**. | | `does not allow runtime credential injection` | Enable runtime injection on the provider, or switch the resource to an injectable provider kind - see [What a Binding Injects](#what-a-binding-injects). | | Secret file rejected | Restrict file permissions and avoid setting both inline and file secrets. | | Launch pauses on `approval_required` | Decide the Approval in the web console, or adjust the approval tier in policy data. | | Launch denied | Check policy-set activation, scopes, and audit diagnostics. | :::caution[Failure point: scope versus provider credential] Binding scopes govern whether STS releases a credential. A static upstream API key still carries its full provider-native power once injected. Prefer Gateway brokering when the upstream cannot issue a truly scoped credential. ::: ## Next Step Run [Launch Research Agent](/v1.0/examples/research-agent/) for the fixed example, or migrate a long-lived process to the matching SDK guide ([TypeScript](/v1.0/guides/sdk-typescript/), [Python](/v1.0/guides/sdk-python/), [Go](/v1.0/guides/sdk-go/)). --- # Protect a Gateway-Routed HTTP API # URL: https://docs.caracal.run/v1.0/guides/protect-gateway-http/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/protect-gateway-http.md # Type: page # Concepts: # Requires: --- Use Gateway-routed protection when the upstream is HTTP-routable and you want Caracal to enforce every request before it reaches the target. The agent receives a short-lived Caracal mandate; Gateway verifies it, resolves the resource route, attaches the configured upstream credential when needed, forwards the request, and writes action-result audit. ## Prerequisites * Gateway, STS, Redis revocation state, audit ingestion, and the upstream ready. * A resource with stable identifier, Gateway application, provider, scopes, and declared operations. * An active policy granting only the calling role and an application SDK configured for the same resource. ## When to Use Gateway | Use Gateway when | Use another guide when | | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------- | | The upstream is HTTP, REST, MCP-over-HTTP, or another Gateway-routable target. | The service must verify mandates inside its own process. | | You want provider credentials to stay inside the trusted Gateway/STS boundary. | The application must call the provider directly and only needs attribution. | | You want action-result audit from the central Gateway. | The resource server owns result audit after adapter verification. | | Responses are request/response or streamed HTTP, such as SSE. | The upstream requires WebSocket connections, which Gateway does not proxy. | For in-process enforcement, use [Protect an Express App](/v1.0/guides/protect-express/), [Protect a FastMCP App](/v1.0/guides/protect-fastmcp/), [Protect a Go net/http Service](/v1.0/guides/protect-nethttp/), or [Protect an MCP Server](/v1.0/guides/protect-mcp/). ## Create or Select the Resource Open the web console for your deployment (`http://localhost:3001` on the packaged local stack). Create or select a resource with: | Field | Guidance | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------- | | Resource identifier | Stable audience URI, such as `resource://pipernet`. | | Scopes | Action-oriented scopes, such as `pipernet:read` and `pipernet:submit`. | | Upstream URL | Network URL the Gateway can reach. | | Gateway application | The Caracal application identity Gateway uses for upstream exchanges. | | Upstream credential provider | The provider record Gateway uses to attach no credential, a Caracal mandate, OAuth token, API key, or bearer token. | Keep the resource identifier stable even when upstream hosts or provider bindings change. ## Choose Provider Auth Mode | Provider type | Use when | | ------------------------ | ----------------------------------------------------------------------- | | None | Gateway enforces Caracal access and the upstream expects no credential. | | Caracal mandate | The upstream verifies Caracal mandates directly. | | OAuth client credentials | Gateway needs a machine OAuth token. | | OAuth authorization code | Gateway needs a connected upstream account (delegated OAuth consent). | | API key | Gateway attaches a sealed static API key in a configured header. | | Bearer token | Gateway attaches a sealed pre-issued bearer token. | Use [Define Resources and Providers](/v1.0/guides/resources-providers/) for field definitions and [Provider Recipes](/v1.0/guides/provider-recipes/) for concrete OpenAI, Google, GitHub, Slack, and internal API examples. ## Route Calls Through Gateway Gateway-routed requests include: | Request part | Value | | -------------------- | ----------------------------------------------------------------------------------- | | URL | Gateway URL from your runtime profile (`gateway_url`; `http://localhost:8081` on the local stack). | | `Authorization` | `Bearer `. | | `X-Caracal-Resource` | Resource identifier. | | Path | The protected path you configured for the upstream. | SDKs can build the Gateway request and inject headers for you: * TypeScript: `caracal.gatewayRequest()` plus `caracal.transport({ scopes })` * Python: `caracal.gateway_request()` plus `caracal.transport(scopes=[...])` * Go: `GatewayRequest()` plus `Transport(nil, CallOptions{Scopes: ...})` The request helper supplies only the Gateway URL and `X-Caracal-Resource`; the scoped transport mints and attaches the required `use=gateway` mandate. Do not combine the helper with raw lifecycle headers. ## Verify Authorization and Audit Run these checks before treating the route as ready: | Check | Expected result | | --------------------- | ------------------------------------------------------------- | | Allowed request | Gateway forwards to upstream and records action-result audit. | | Missing mandate | Gateway rejects the request. | | Missing scope | STS or Gateway rejects the request. | | Wrong resource header | Gateway rejects or routes to a different configured resource. | | Revoked session | Gateway rejects the old mandate. | Open the web console **Audit** and filter by the request ID. A complete Gateway-routed call has both authorization evidence from STS and action-result evidence from Gateway. Gateway preserves the HTTP method, path, query, body, and streaming response. It replaces caller authorization with the selected provider credential, strips upgrade behavior, does not proxy WebSockets, and does not retry an upstream mutation. A stale revocation snapshot fails closed with `503` before forwarding. ## Troubleshooting | Symptom | Check | | ------------------------------------------------------------- | ------------------------------------------------------------------------ | | Gateway returns 403 | Token expiry, resource ID, scopes, revocation, and `X-Caracal-Resource`. | | Upstream is unreachable | Upstream URL must be reachable from the Gateway container or deployment. | | Provider credential not attached | Resource must bind exactly one upstream credential provider. | | Authorization audit exists but action-result audit is missing | The request did not reach Gateway or failed before route execution. | :::caution[Failure point: raw headers] `X-Caracal-Resource` routes the request; it does not authorize it. Send a scoped `use=gateway` bearer minted by the SDK transport. Do not send lifecycle headers or trust a client-supplied application ID. ::: ## Next Step Run [Check Provider Readiness](/v1.0/examples/provider-preflight/), then use [Debug Authorization Decisions](/v1.0/guides/authorize-access/) for any deny. --- # Protect an Express App # URL: https://docs.caracal.run/v1.0/guides/protect-express/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/protect-express.md # Type: page # Concepts: # Requires: --- Use `@caracalai/express` when an Express app should verify Caracal mandates before route handlers run. ## Prerequisites * A Caracal-mandate resource whose audience matches this service. * The zone STS issuer, zone ID, and a shared production revocation store. * A route-by-route scope and target map. ## Install ```bash npm install express @caracalai/express @caracalai/verify @caracalai/revocation ``` Use a Redis-backed revocation store in production. The in-memory store is only suitable for local development and tests. ## Add middleware ```ts import express from 'express' import { caracalAuth, type CaracalRequest } from '@caracalai/express' import { createMandateVerifier } from '@caracalai/verify' import { InMemoryRevocationStore } from '@caracalai/revocation' const app = express() const verifier = createMandateVerifier({ issuer: process.env.CARACAL_ISSUER!, audience: 'resource://pipernet', zoneId: process.env.CARACAL_ZONE_ID!, revocations: new InMemoryRevocationStore(), }) await verifier.warmup() app.use( '/reports', caracalAuth( { verifier }, { requiredScopes: ['pipernet:read'], requiredTargets: ['resource://pipernet'], }, ), ) app.get('/reports', (req: CaracalRequest, res) => { res.json({ principal: req.caracalClaims?.sub, reports: ['market-risk', 'quarterly'], }) }) ``` The middleware attaches verified claims to `req.caracal` and `req.caracalClaims`, plus the propagation context at `req.caracalContext`. ## Enforce the right boundary | Option | Use it for | | ------------------- | --------------------------------- | | `requiredScopes` | Tool or route-level scope checks. | | `requiredTargets` | Resource-target checks. | | `requireSession` | Require a governed Session. | | `requireDelegation` | Require delegated authority. | | `maxHopCount` | Limit delegation depth. | ## Validate 1. Exchange for a mandate that targets the resource. 2. Call the protected route with `Authorization: Bearer `. 3. Remove a required scope and confirm the route returns `403`. 4. Revoke the session and confirm the route rejects the old mandate. Expected result: missing or invalid credentials return `401`, valid but insufficient authority returns `403`, and protected handlers run only after `req.caracalClaims` and `req.caracalContext` are populated. :::caution[Failure point: revocation] `InMemoryRevocationStore` proves local behavior only. Every production replica must share the Redis-backed store and consume revocation events; otherwise one replica can continue accepting a revoked Session. ::: For middleware option signatures, use [Express Adapter reference](/v1.0/sdks/adapters/express/). ## Next Step Add route-specific requirements, then test the boundary with [Test Caracal Integrations](/v1.0/guides/testing/). --- # Protect a FastAPI App # URL: https://docs.caracal.run/v1.0/guides/protect-fastapi/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/protect-fastapi.md # Type: page # Concepts: # Requires: --- Use `caracalai-asgi` when a FastAPI, Starlette, or other ASGI app should verify Caracal mandates before request handlers run. This is the provider-side boundary: a partner serving Caracal-governed customers verifies each inbound mandate against the zone's keys before doing any work. ## Prerequisites * A Caracal-mandate resource whose audience is `resource://pipernet` or another stable resource ID. * STS issuer and zone configuration plus a shared production revocation store. * Public health paths identified before middleware is enabled. ## Install ```bash pip install caracalai-asgi caracalai-revocation ``` Use a Redis-backed revocation store in production. The in-memory store is only suitable for local development and tests. ## Add middleware ```python from caracalai_asgi import CaracalASGIAuth from caracalai_revocation import InMemoryRevocationStore from fastapi import FastAPI, Request app = FastAPI() app.add_middleware( CaracalASGIAuth, audience="resource://pipernet", revocations=InMemoryRevocationStore(), required_scopes=["pipernet:read"], routes={ "/payouts": {"required_scopes": ["pipernet:payout"], "require_delegation": True}, }, exclude=["/healthz"], ) @app.post("/payouts/create") async def create_payout(request: Request): principal = request.state.caracal return {"subject": principal.sub} ``` With `CARACAL_STS_URL` and `CARACAL_ZONE_ID` set - the standard Caracal workload variables - the middleware resolves the issuer and zone itself; you state only your own audience and revocation store. Requests reach your handlers only after the mandate's signature, issuer, audience, zone, token use, scopes, and revocation anchors all verify. ## Enforce the right boundary | Option | Use it for | | -------------------- | ------------------------------------------------------------------------- | | `required_scopes` | Route-level scope checks. | | `required_targets` | Resource-target checks. | | `require_session` | Require a governed Session. | | `require_delegation` | Require delegated authority. | | `max_hop_count` | Limit delegation depth. | | `routes` | Apply any of the above per path prefix; the longest matching prefix wins. | ## Validate 1. Exchange for a mandate that targets the resource. 2. Call the protected route with `Authorization: Bearer `. 3. Remove a required scope and confirm the route returns `403`. 4. Revoke the session and confirm the route rejects the old mandate. Expected result: missing or invalid credentials return `401`, valid but insufficient authority returns `403`, excluded probes remain reachable, and handlers receive `request.state.caracal` only after verification. :::caution[Failure point: prefix matching] Route configuration uses path prefixes and the longest match wins. Test overlapping prefixes so a broad rule cannot hide a stricter operation rule. ::: For every middleware field, use [ASGI Adapter reference](/v1.0/sdks/adapters/asgi/). ## Next Step Replace the in-memory store with the Redis backend and run the revocation case in [Test Caracal Integrations](/v1.0/guides/testing/). --- # Protect a FastMCP App # URL: https://docs.caracal.run/v1.0/guides/protect-fastmcp/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/protect-fastmcp.md # Type: page # Concepts: # Requires: --- Use the FastMCP adapter when a FastMCP app should reject invalid, expired, insufficient, or revoked Caracal mandates. Python uses `caracalai-fastmcp`; TypeScript uses `@caracalai/fastmcp`. ## Prerequisites * A mandate-aware MCP resource, stable audience, tool scopes, issuer, and zone ID. * A shared production revocation store. * A FastMCP hook that can reject before tool code runs; verification after dispatch is too late. ## Install ```bash pip install "caracalai-fastmcp[fastmcp]" caracalai-revocation ``` ```bash npm install @caracalai/fastmcp @caracalai/verify @caracalai/revocation ``` Use `caracalai-revocation-redis` (Python) or the Redis store in `@caracalai/revocation` (TypeScript) for shared production revocation. The in-memory store is only for local development and tests. ## Create an authenticator (Python) ```python from caracalai_fastmcp import CaracalAuth from caracalai_revocation import InMemoryRevocationStore auth = CaracalAuth( issuer="https://sts.pipernet.example", audience="resource://pipernet", expected_zone_id="0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f", required_scopes=["mcp:tool:call"], required_targets=["resource://pipernet"], revocations=InMemoryRevocationStore(), require_session=True, ) async def startup(): await auth.warmup() ``` ## Verify before running a tool (Python) ```python from caracalai_fastmcp import CaracalAuthError async def handle_tool_call(token: str, payload: dict): try: claims = await auth(token) except CaracalAuthError as err: return {"error": err.code, "error_description": err.description, "error_hint": err.hint} return { "subject": claims.sub, "result": await run_tool(payload), } ``` Wire this check into the FastMCP auth or request hook used by your server. The important boundary is that the mandate is verified before the tool handler performs work. ## Verify before running a tool (TypeScript) ```ts import { extractBearer, verifyFastMcpToken, FastMcpAuthError } from '@caracalai/fastmcp' import { createMandateVerifier } from '@caracalai/verify' import { InMemoryRevocationStore } from '@caracalai/revocation' const verifier = createMandateVerifier({ issuer: 'https://sts.pipernet.example', audience: 'resource://pipernet', zoneId: '0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f', revocations: new InMemoryRevocationStore(), }) async function handleToolCall(authorization: string, payload: unknown) { const token = extractBearer(authorization) if (!token) return { error: 'missing_token' } try { const context = await verifyFastMcpToken(token, verifier, { requiredScopes: ['mcp:tool:call'], requiredTargets: ['resource://pipernet'], requireSession: true, }) return { subject: context.sub, result: await runTool(payload) } } catch (err) { if (err instanceof FastMcpAuthError) return { error: err.code } throw err } } ``` The boundary is the same in both languages: verify the mandate before the tool handler performs work. ## Validate | Test | Expected result | | ---------------------------------------------------------------- | --------------------------------- | | Missing bearer token | `missing_token` or framework 401. | | Wrong audience | `invalid_token`. | | Missing scope | `insufficient_scope`. | | Revoked session | `session_revoked`. | | Mandate without a Session with `require_session=True` / `requireSession: true` | `session_required`. | Expected result: the tool handler runs only for a mandate matching issuer, audience, zone, target, scope, Session requirements, and current revocation state. :::caution[Failure point: adapter scope] The adapter verifies one token; it does not start a Session, mint authority, or authorize a later outbound call. Use the SDK inside verified context for governed downstream work. ::: For exact call signatures, use [FastMCP Adapter reference](/v1.0/sdks/adapters/fastmcp/). ## Next Step Use [Protect an MCP Server](/v1.0/guides/protect-mcp/) only when the framework hook cannot use this dedicated adapter. --- # Protect a Go net/http Service # URL: https://docs.caracal.run/v1.0/guides/protect-nethttp/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/protect-nethttp.md # Type: page # Concepts: # Requires: --- Use the Go net/http adapter when a Go service should verify Caracal mandates at the handler boundary. ## Prerequisites * A mandate-aware resource with a stable audience and route scopes. * STS issuer, zone ID, and a production Redis revocation consumer. * Request deadlines on inbound handlers so JWKS or revocation work is cancelable. ## Install ```bash go get github.com/garudex-labs/caracal/packages/adapters/nethttp/go go get github.com/garudex-labs/caracal/packages/revocation/go ``` ## Wrap a handler ```go package main import ( "encoding/json" "net/http" "time" nethttp "github.com/garudex-labs/caracal/packages/adapters/nethttp/go" revocation "github.com/garudex-labs/caracal/packages/revocation/go" verify "github.com/garudex-labs/caracal/packages/verify/go" ) func main() { revocations := revocation.NewInMemoryStore(24 * time.Hour) verifier := verify.NewVerifier(verify.Options{ Issuer: "https://sts.pipernet.example", Audience: "resource://pipernet", ZoneID: "0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f", Revocations: revocations, }) protected := nethttp.VerifierMiddleware(verifier.Require(verify.Options{ RequiredScopes: []string{"pipernet:read"}, RequiredTargets: []string{"resource://pipernet"}, }))(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { claims, ok := nethttp.ClaimsFromContext(r.Context()) if !ok { http.Error(w, "missing claims", http.StatusUnauthorized) return } _ = json.NewEncoder(w).Encode(map[string]string{"subject": claims.Sub}) })) http.Handle("/reports", protected) _ = http.ListenAndServe(":8080", nil) } ``` ## Enforce constraints | Option | Use it for | | ---------------------- | ------------------------------ | | `RequiredScopes` | Route or operation permission. | | `RequiredTargets` | Resource target matching. | | `RequireSession` | Session-bound endpoints. | | `RequireDelegation` | Delegated-only endpoints. | | `RequireChainContains` | Application path requirements. | | `MaxHopCount` | Delegation depth limit. | ## Production revocation The in-memory store does not share revocations across instances. Use the Redis revocation backend and consume `caracal.sessions.revoke` for production resource servers. ## Validate 1. Call without a bearer token and expect `401`. 2. Call with a valid mandate and expect the handler response. 3. Remove a required scope and expect `403`. 4. Mark the session revoked and expect `session_revoked`. Expected result: `401` identifies an unaccepted credential, `403` identifies accepted but insufficient authority, and `ClaimsFromContext` succeeds only inside protected handlers. :::caution[Failure point: middleware order] Place Caracal verification before handlers that read bodies or perform work. Logging may wrap it, but authentication-dependent middleware must run after it. ::: For exact Go fields, use [net/http Adapter reference](/v1.0/sdks/adapters/nethttp/). ## Next Step Connect the Redis revocation consumer and validate it with [Test Caracal Integrations](/v1.0/guides/testing/). --- # Protect an MCP Server # URL: https://docs.caracal.run/v1.0/guides/protect-mcp/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/protect-mcp.md # Type: page # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' Use the verify packages when your MCP framework does not have a dedicated Caracal adapter or when you want to build your own boundary. :::note[Two ways to protect an MCP server] This guide covers in-process verification, where the MCP server checks mandates itself. To route an MCP-over-HTTP server through the Caracal Gateway with no code in the server, model it as a resource with operation enforcement set to **Any operation**, bound to a credential provider - see [Protect an MCP server over the Gateway](/v1.0/guides/resources-providers/#protect-an-mcp-server-over-the-gateway). ::: ## Prerequisites * A resource mandate audience and tool-scope map. * Issuer, zone ID, shared revocation storage, and a hook that runs before tool dispatch. * A consistent mapping from verification errors to the framework's unauthorized/forbidden response. ## Build the Verifier ```bash npm install @caracalai/verify @caracalai/revocation ``` ```ts import { createMandateVerifier } from '@caracalai/verify' import { InMemoryRevocationStore } from '@caracalai/revocation' const verifier = createMandateVerifier({ issuer: 'https://sts.pipernet.example', audience: 'resource://pipernet', zoneId: '0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f', revocations: new InMemoryRevocationStore(), }) export async function verifyToolRequest(authorization: string | undefined) { const result = await verifier.authorization(authorization, { requiredScopes: ['mcp:tool:call'], requiredTargets: ['resource://pipernet'], requireSession: true, }) if (!result.ok) { throw new Error(`${result.error.code}: ${result.error.description}`) } return result.principal } ``` ```bash pip install caracalai-verify caracalai-revocation ``` ```python from caracalai_verify import authenticate, extract_bearer from caracalai_revocation import InMemoryRevocationStore revocations = InMemoryRevocationStore() async def verify_tool_request(authorization: str | None): token = extract_bearer(authorization) result = await authenticate( token or "", issuer="https://sts.pipernet.example", audience="resource://pipernet", required_scopes=["mcp:tool:call"], expected_zone_id="0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f", revocations=revocations, require_session=True, required_targets=["resource://pipernet"], ) if result.error is not None: raise RuntimeError(f"{result.error.code}: {result.error.description}") return result.principal ``` ## Verification checklist | Check | Why it matters | | ---------------------------------- | ---------------------------------------------------------- | | Issuer and audience | Prevents accepting mandates from the wrong zone or target. | | Required scopes | Enforces tool-level authority. | | Required targets | Prevents cross-resource token reuse. | | Session or Delegation requirements | Keeps application-root and delegated calls separate. | | Revocation store | Rejects revoked sessions and Delegations. | ## Production revocation Use Redis-backed revocation packages for multi-instance MCP servers: * TypeScript: `@caracalai/revocation-redis` * Python: `caracalai-revocation-redis` * Go: `github.com/garudex-labs/caracal/packages/backends/redis/go` Run a consumer for the `caracal.sessions.revoke` stream so every resource server instance learns about revoked anchors. ## Validate the boundary Test missing token, wrong audience, wrong target, missing scope, required Session, required Delegation, hop limit, and revoked Session before enabling traffic. Expected result: no rejected call reaches tool code and every allowed principal is derived from verified claims rather than request metadata. :::caution[Failure point: unsigned context] Never authorize from baggage, trace headers, tool arguments, or a caller-provided Subject. Verified mandate claims are authoritative; other fields are correlation only. ::: For exact verifier functions and error types, use [Verify Package reference](/v1.0/sdks/verify/). ## Next Step Wrap the framework hook, then run [Test Caracal Integrations](/v1.0/guides/testing/) with a revoked-anchor case. --- # Tail and Query the Audit Stream # URL: https://docs.caracal.run/v1.0/guides/audit-stream/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/audit-stream.md # Type: page # Concepts: # Requires: --- Use Audit when a request is denied, Approval is required, Delegation behaves unexpectedly, or an operator needs evidence for a run. ## Prerequisites * A zone and known time window or request ID. * A trusted Admin client for automation; application credentials cannot read zone audit. * Durable event-ID deduplication if exporting continuously. ## Web Console Workflow 1. Open the web console for your deployment. 2. Select **Audit**. 3. Filter by zone, decision, event type, or request ID. 4. Open the event detail to inspect metadata, determining policies, and diagnostics. 5. Open the decision trace from the event for a full request-level view. ## Automation workflow ```ts import { AdminClient } from '@caracalai/admin' const admin = new AdminClient({ apiUrl: process.env.CARACAL_API_URL!, adminToken: process.env.CARACAL_ADMIN_TOKEN!, }) const events = await admin.audit.list(process.env.CARACAL_ZONE_ID!, { decision: 'deny', limit: 25, }) const trace = await admin.audit.explain(process.env.CARACAL_ZONE_ID!, events[0].request_id!) console.log(trace.final_decision, trace.denied) ``` ## Useful filters | Filter | Use it for | | ----------- | ---------------------------------------------------------------- | | Request ID | Stitch together STS, Gateway, adapter, and explain events. | | Decision | Find denies, allows, or partial decisions. | | Event type | Focus on Policy, Session, Delegation, Approval, or admin changes. | | Time window | Investigate a run, incident, or rollout. | ## Ship events to a SIEM Poll `admin.audit.list` with a bounded time window and persist the last processed event identity. Caracal does not expose a supported external audit-stream subscription API; Redis topics are runtime internals, not an application integration contract. Use Admin API list/export surfaces and forward events after deduplication. | Field | Type | Meaning | | ----------------------------- | -------------- | --------------------------------------------------------------------------------- | | `id` | string | Unique event id; use as the dedup key. | | `zone_id` | string | Zone the event belongs to. | | `event_type` | string | What happened: exchange, decision, replay\_detected, revocation, lifecycle events. | | `request_id` | string | null | Correlates every event from one request across STS, Gateway, and adapters. | | `decision` | string | null | `allow`, `deny`, or `partial` for decision-bearing events. | | `evaluation_status` | string | null | How policy evaluation concluded. | | `metadata_json` | object | null | Event-specific attributes: session, resource, scopes, client identity. | | `occurred_at` / `ingested_at` | string | Event time and audit-service ingest time (RFC 3339). | Event detail (`admin.audit.explain`) adds the determining policies, policy-set version, manifest hash, and diagnostics for decision events - fetch it lazily from the SIEM for alerts rather than shipping it wholesale. Alert first on `decision: "deny"` spikes, `replay_detected`, and delegation events for sessions outside your expected label set. ## What to capture in incidents * request ID; * zone ID; * application ID; * resource identifier; * requested scopes; * final decision; * determining policies; * diagnostics; * session or Delegation IDs when present. Related page: [Audit and Request Traces](/v1.0/concepts/audit-ledger/). ## Validate the export Generate one allow, deny, approval, Delegation, and Gateway result. Confirm request correlation, ordering in the destination, replay-safe deduplication by event ID, and credential redaction. Expected result: an operator can move from a SIEM alert to the Caracal request trace without querying internal Redis. :::caution[Failure point: polling cursor] Do not use wall-clock time alone as an exactly-once cursor. Re-read an overlap window and deduplicate by event ID so late ingestion does not create gaps. ::: ## Next Step Use [Debug Authorization Decisions](/v1.0/guides/authorize-access/) to turn a correlated deny into a focused correction. --- # Implement Multi-Agent Delegation # URL: https://docs.caracal.run/v1.0/guides/delegation/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/delegation.md # Type: page # Concepts: # Requires: --- Use Delegation when one Session needs to hand a narrower slice of authority to another Session. The SDK creates the Delegation and carries the context; the web console shows the graph and impact. ## Prerequisites * Two live Sessions and a parent that already holds the resource authority being delegated. * A specific resource, scope subset, positive TTL, and intended maximum hop count. * An authenticated work channel for transferring the opaque Delegation ID to the target Session. `session()` always starts the child under the **same application** as the parent. Narrow the child's authority only when it should hold less than the parent. To hand authority across applications, use `delegate()` with a peer Session under the other application; it returns the Delegation, and the receiver presents it with `acceptDelegation()`. ## Implementation flow ```mermaid flowchart LR Parent["Parent Session"] --> Start["Start child with narrowed authority"] Start --> Delegation["Bounded Delegation"] Delegation --> Child["Run child with delegated context"] Child --> Exchange["Exchange for resource mandate"] Exchange --> Audit["Inspect audit and graph"] ``` Python uses `Authority.narrow(...)`, `await caracal.delegate(...)`, and `async with caracal.accept_delegation(edge_id, validate=True)`. Go uses `AuthorityNarrow(...)`, `Delegate`, and `AcceptDelegation`. The [language SDK references](/v1.0/sdks/) define exact option types; the workflow and wire authority are equivalent. ## Narrow a child in the same application Starting a child with less authority than its parent needs no explicit hand-off - pass `Authority.narrow(...)` when starting the child Session, exactly as shown in your [SDK guide](/v1.0/guides/sdk-typescript/#start-a-narrowed-child). This page's subject is the explicit flow between two Sessions that already exist. ## Delegate across applications The issuer holds the authority and knows the receiver's Session ID; the receiver presents the returned Delegation ID. Two sides, one edge: ```ts // Issuer side: Anton delegates read-only PiperNet access // to a peer Session owned by the Fiona application. const delegation = await caracal.delegate({ toSessionId: receiverSessionId, // shared by the receiver up front toApplicationId: fionaApplicationId, // omit for same-application peers resourceId: 'resource://pipernet', scopes: ['pipernet:read'], constraints: { maxHops: 1 }, ttlSeconds: 600, }) await sendOverWorkChannel({ delegationId: delegation.delegationId }) ``` ```ts // Receiver side (Fiona): present the offered Delegation inside the // target Session's context. validate: true confirms it is live first. await caracal.session(async () => { const { delegationId } = await readFromWorkChannel() await caracal.acceptDelegation( delegationId, async () => { const governed = caracal.transport({ scopes: ['pipernet:read'] }) const target = caracal.gatewayRequest('resource://pipernet', '/reports') await governed(target.url) }, { validate: true }, ) }) ``` Every mint inside the accepted block presents the Delegation, so the STS enforces its scopes, resource bound, TTL, and hop budget - and revoking the edge cuts the receiver off mid-run. ## Review the graph 1. Open the web console for your deployment. 2. Select **Delegation**. 3. Inspect active edges, inbound edges, outbound edges, and traversal. 4. Use impact before revoking an edge. 5. Check **Audit** for the delegated exchange and resource decision. ## Hand Over the Delegation ID Safely The issuer and receiver share the opaque Delegation ID over their authenticated work channel, such as a queue message, RPC field, or task payload. Creation is only an offer and does not mutate receiver context. Possessing and presenting this target-Session-bound ID through `acceptDelegation()` is receiver consent; STS also checks the receiving application and live target Session. Use `acceptDelegation(delegationId, fn, { validate: true })` to confirm that exact ID with Coordinator before work runs under it. Every presentation emits a `delegation.accept` event. ## Safe constraints | Constraint | Good default | | ---------- | ----------------------------------------- | | Scopes | Small subset of parent authority. | | TTL | Required; use minutes, not days. | | Hop count | `1` unless a deeper graph is intentional. | | Resource | Single resource whenever possible. | ## Troubleshooting | Symptom | Check | | ------------------------------------- | -------------------------------------------------------------------- | | `Delegate requires an active Session` | Call delegation inside `session()` or a bound Caracal context. | | Resource denies delegated request | Confirm the edge, ancestry, policy grant, and required scopes remain live. | | Chain validation fails | Confirm authoritative parent links are continuous; configure `requireChainContains` separately at verifiers when needed. | | Revocation did not affect child | Confirm cascade revocation and resource-server revocation consumers. | ## Validate the delegation Mint the allowed subset, attempt one broader scope, revoke the edge, and retry. The subset must succeed; broadening must fail server-side; after revocation neither the child nor descendants may mint or pass resource verification. :::caution[Failure point: lifetime] A Delegation can expire before its target Session. The Session may remain observable but cannot mint through the expired edge. Size the edge TTL to the work and handle terminal authorization failure rather than widening authority. ::: ## Next Step Inspect the graph and request trace in [Tail and Query the Audit Stream](/v1.0/guides/audit-stream/). --- # Approval Notifications # URL: https://docs.caracal.run/v1.0/guides/approval-notifications/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/approval-notifications.md # Type: page # Concepts: # Requires: --- Approval holds are decided fastest when the right person hears about them immediately. A notification sink is a zone-scoped webhook endpoint that Caracal pushes approval lifecycle events to: your relay turns the delivery into a page, a chat message, or a ticket in whatever system your team watches. Sinks are optional - a zone without one loses nothing except the push, since the web console badges pending holds and the [Approvals page](/v1.0/guides/human-approval/#decide-as-an-operator) remains the decision surface. ## Prerequisites * A public HTTPS receiver with raw-body access, durable delivery-ID deduplication, and a secret manager. * An operational owner for retries, abandoned deliveries, and secret rotation. * Approval tiers already validated end to end without notifications. ## Create a sink Open **Settings → Notifications** in the web console and add a sink with a name, an HTTPS endpoint, and the event types it should receive. The response reveals the signing secret exactly once - store it in your receiver's secret manager before closing the dialog, because Caracal keeps only a sealed copy it cannot show again. For automation, the Admin API exposes the same lifecycle: ```bash curl -X POST "$CARACAL_API_URL/v1/zones/$CARACAL_ZONE_ID/notification-sinks" \ -H "Authorization: Bearer $CARACAL_ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "Pied Piper on-call relay", "url": "https://hooks.hooli.example/caracal-approvals", "event_types": ["step_up_issued", "step_up_decided"] }' ``` Endpoints must be HTTPS; plain HTTP is accepted only for loopback addresses during local development, and URLs may never embed credentials. Delivery resolves and pins DNS for each connection and rejects loopback, metadata, link-local, multicast, mapped, and NAT64-embedded addresses outright; private, unique-local, and CGNAT ranges are rejected too unless the receiver's host is listed in `CARACAL_PRIVATE_EGRESS_HOSTS`, the same egress allowlist that governs provider token endpoints - so an on-premises relay stays reachable without ever opening a path to cloud metadata. Authentication is the signature, not the URL. A zone holds at most 20 sinks. | Event type | Fires when | | ------------------ | ------------------------------------------------------------------------------------------------------- | | `step_up_issued` | A hold is created and waits for a decision. This is the event to page on. | | `step_up_decided` | An approver settles the hold - the payload's `decision` field says whether it was approved or rejected. | | `step_up_consumed` | The approved hold releases its mandate to the retrying agent. | The `step_up_` event-type prefix is the audit stream's stable taxonomy: routing rules and SIEM pipelines keyed on these types stay valid across releases. ## Verify every delivery Each delivery is an HTTP `POST` with a JSON body and these headers: | Header | Content | | --------------------- | ----------------------------------------------------------------------------------- | | `X-Caracal-Event` | The event type, for routing before parsing. | | `X-Caracal-Delivery` | Unique delivery id. Deliveries are at-least-once; use this id to deduplicate. | | `X-Caracal-Sink` | The sink id, so one receiver can serve several sinks. | | `X-Caracal-Timestamp` | Unix seconds when the delivery was signed. Reject stale timestamps to stop replays. | | `X-Caracal-Signature` | `v1=` followed by hex HMAC-SHA256 of `timestamp.body` under the sink secret. | Verify the signature over the raw request body before trusting anything in it: ```ts import { createHmac, timingSafeEqual } from 'node:crypto' function verifySink(secret: string, headers: Record, rawBody: string): boolean { const timestamp = headers['x-caracal-timestamp'] if (Math.abs(Date.now() / 1000 - Number(timestamp)) > 300) return false const expected = `v1=${createHmac('sha256', secret).update(`${timestamp}.${rawBody}`).digest('hex')}` const received = headers['x-caracal-signature'] ?? '' return received.length === expected.length && timingSafeEqual(Buffer.from(received), Buffer.from(expected)) } ``` The body carries the audit event verbatim - `data` is the same metadata the zone audit stream records for the hold, so a relay can render a complete approval prompt without calling back. The requesting run rides along as `agent_session_id`, and any labels its developer annotated at session start ride as `agent_labels` - the business context, such as a case or settlement reference, that lets an approver correlate the hold to the work behind it: ```json { "id": "0195f2aa-7c31-7d4e-a01b-3f6c2d9e8b10", "type": "step_up_issued", "zone_id": "0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f", "decision": "pending", "occurred_at": "2026-01-01T12:00:00.000Z", "data": { "challenge_id": "0195f2aa-6d20-7b3c-8e1a-2f4d6c8b0a9e", "tier": "high", "approver_class": "operator", "privacy_mode": "identified", "binding": "9f3c…", "expires_at": "2026-01-01T12:30:00Z", "application_id": "0195f2a9-4e55-7a1b-bc2d-3e4f5a6b7c8d", "session_id": "0195f2aa-2c11-7e9f-8a0b-1c2d3e4f5a6b", "agent_session_id": "0195f2aa-9f10-7c2d-8b3e-4a5c6d7e8f90", "agent_labels": ["case:CASE-4021", "payout-execution"] } } ``` Respond with any `2xx` status within ten seconds. The response body is ignored - do your downstream work asynchronously and acknowledge fast. ## Delivery semantics Delivery is at-least-once with per-sink ordering by the zone audit sequence. A non-`2xx` response or a timeout schedules a retry with exponential backoff from 30 seconds to a 15-minute ceiling; after 8 attempts the delivery is abandoned and the failure is visible on the sink's delivery record in the web console. Pausing a sink stops deliveries without losing anything: the sink's cursor into the audit stream holds its place, and resuming catches up from where it stopped. A run of consecutive failures flags the sink as failing in the web console but never disables it - fix the receiver and the backlog drains on its own. Rotating the secret (Console or `POST …/notification-sinks/{id}/rotate-secret`) invalidates the old value immediately and reveals the replacement once; update the receiver in the same change window. Sink creation, edits, rotations, and deletions are themselves recorded in the zone audit stream, with secret material redacted. Related pages: [Human Approval](/v1.0/guides/human-approval/) and [Tail and Query the Audit Stream](/v1.0/guides/audit-stream/). ## Validate the sink Send a test hold, verify the raw-body signature and timestamp, return a temporary failure, and confirm retry plus delivery-ID deduplication. Rotate the secret and confirm the old signature immediately fails. Expected result: the receiver acknowledges within ten seconds and performs slow paging or ticket creation asynchronously. :::caution[Failure point: notification versus decision] A valid webhook proves Caracal emitted an event; it is not authority to approve it. Decisions still require the web console or Admin operator plane or a valid Federated user session mandate. ::: ## Next Step Create alerts for abandoned deliveries and test the operator decision path in [Human Approval](/v1.0/guides/human-approval/). --- # Production Integration Patterns # URL: https://docs.caracal.run/v1.0/guides/production-patterns/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/production-patterns.md # Type: workflow # Concepts: # Requires: --- Use this review after one application and one protected resource work end to end. Production deployments combine a primary enforcement boundary with SDK lifecycle, shared revocation, policy rollout, audit export, and operational ownership. ## Prerequisites * A tested allow, deny, revoke, and request-trace path. * Named owners for application identity, resource verification, policy rollout, revocation, audit, and incidents. * TLS, secret storage, shared revocation state, and timeout budgets appropriate to the deployment. ## Choose the Enforcement Boundary | Boundary | Use when | Enforcement and result evidence | | --- | --- | --- | | Gateway-routed HTTP | Caracal can route the request before it reaches the upstream. | Gateway verifies, exchanges, routes, and records the upstream result. | | Framework adapter | You own the resource server and must verify inside Express, ASGI, FastMCP, or Go `net/http`. | The service verifies before work and must emit its action result. | | Verify engine | No supported adapter fits the service boundary. | Custom middleware uses the shared verifier and owns result evidence. | | Runtime credential injection | An existing process needs a credential once at launch. | `caracal run` gates launch; it does not renew or enforce later provider calls. | Use [Protect a Gateway-Routed HTTP API](/v1.0/guides/protect-gateway-http/) for the default HTTP path. Use the framework-specific protection guides when enforcement must live in the resource server. ## Reference Flow ```mermaid flowchart LR Workload[Application workload] --> SDK[Caracal SDK] SDK --> Coordinator[Coordinator] SDK --> STS[STS] STS --> Policy[Active policy set] SDK --> Gateway[Gateway] Gateway --> Upstream[Protected upstream] SDK --> Verifier[In-process verifier] Verifier --> Service[Protected service] STS --> Audit[Audit] Gateway --> Audit Service --> Audit ``` ## Identity and Configuration Boundaries One SDK client represents one `(zone, application)` identity during concurrent use. Keep one client and one credential source per application boundary. Do not swap identities through a shared resolver while Sessions or transports are active. On a shared host, give each service its own application, profile, and owner-only secret file: ```toml zone_id = "0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f" application_id = "anton" app_client_secret_file = "/etc/caracal/anton.secret" sts_url = "https://sts.pipernet.example" gateway_url = "https://gateway.pipernet.example" [[credentials]] resource = "resource://pipernet" ``` Use a dynamic credential resolver only when one application credential is rotated or resolved externally. The next exchange can use the replacement credential; active identities still remain separate clients. ## Propagation into Other Protocols Caracal SDKs project authority and trace context into HTTP headers. When another protocol carries equivalent metadata, inject the SDK-produced fields per call and verify the Mandate before the server performs work. For gRPC, use a language-native client interceptor to copy the authorization and Caracal metadata into call metadata, then build server middleware around the verify package. Caracal does not ship gRPC middleware, service-mesh identity, or protocol-specific result audit. A service mesh authenticates network peers; it does not replace Caracal resource scopes, policy decisions, Mandate verification, or revocation. ## OpenTelemetry Correlation SDK transports preserve valid W3C trace context and add Caracal baggage. Gateway audit includes trace and request correlation. Validate that one APM trace can pivot to the corresponding Caracal request trace without treating tracing as an authorization control. ## Queue and Batch Work Use a Session per governed unit of work. Supply an explicit idempotency key only when a durable source provides a stable delivery identifier. Caracal makes supported Coordinator creations replay-safe; it does not make arbitrary callbacks, queue consumption, or upstream mutations exactly once. For long-running workers, use SDK-managed service Sessions and lease callbacks. Do not use `caracal run` credentials beyond their issued lifetime because the launcher never renews them. ## Production Checklist | Area | Required evidence | | --- | --- | | Enforcement | A request cannot reach the protected action without Gateway or verifier acceptance. | | Identity | Each application boundary has distinct credentials, clients, policy attribution, and rotation ownership. | | Revocation | Shared consumers receive Session and Delegation invalidation; stale state fails according to the documented boundary. | | Policy | Candidate versions validate and simulate, activation reaches every STS replica, and rollback is rehearsed. | | Audit | Authorization and action-result evidence correlate by request or trace ID. | | Approval | Gated scopes reach the intended operator or Federated user decision plane. | | Recovery | Dependency outages, audit replay, restore, and credential rotation are tested. | ## Validate After Rollout Run a successful request, policy denial, verifier denial, revoked-Session request, replayed Gateway request, and Approval-gated request. Confirm expected HTTP or SDK errors and complete audit evidence. In staging, interrupt STS, Redis revocation freshness, Audit delivery, and the upstream. Confirm access fails closed where authority cannot be proven and application retries remain bounded and mutation-safe. ## Next Step Complete the focused guide for the selected enforcement boundary, then use [Operate Caracal](/v1.0/operations/) to deploy and monitor it. --- # Understand the Model # URL: https://docs.caracal.run/v1.0/concepts/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts.md # Type: page # Concepts: # Requires: --- Read this section after Get Started when you need to design an integration or explain a decision. Get Started gave you the working vocabulary - application, resource, policy, mandate, Gateway, audit, zone. This section completes the model with the pieces behind them: how a **Session** bounds one run, how an optional **Subject** adds end-user attribution, how a **Delegation** narrows authority between Sessions, and how revocation and audit tie it all together. One sentence holds the whole picture: an Application acts in a Zone, a Session bounds execution, Delegation narrows authority, policy approves a Resource request, and a Mandate carries the result. ## Read This Section in Order | Start here | Use it to understand | | --------------------------------------------------------- | ----------------------------------------------------------------------------- | | [Caracal Mental Model](/v1.0/concepts/model-overview/) | The smallest useful picture of Caracal. | | [Authority and Enforcement](/v1.0/concepts/authority-model/) | Where decisions happen before requests reach a target. | | [Zones](/v1.0/concepts/zone/) | The tenant boundary that owns keys, policies, resources, sessions, and audit. | | [Identities and Applications](/v1.0/concepts/principal/) | Application credentials, Subjects and Federated users, Authority records, and Sessions. | | [Resources and Grants](/v1.0/concepts/resource-grant/) | What can be accessed and which scopes are granted. | | [Providers](/v1.0/concepts/provider/) | The credential Caracal attaches to the upstream target. | | [Policies and Policy Sets](/v1.0/concepts/policy/) | Rego rules evaluated by the STS during token exchange. | | [Mandates](/v1.0/concepts/mandate/) | The short-lived JWT that carries approved authority. | | [Approvals](/v1.0/concepts/approvals/) | How sensitive actions are held for a human decision. | | [Session Delegation](/v1.0/concepts/delegation/) | How one agent passes bounded authority to another. | | [Delegation Constraints](/v1.0/concepts/constraint/) | The limits attached to delegated authority. | | [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) | How active authority is ended and propagated. | | [Audit and Request Traces](/v1.0/concepts/audit-ledger/) | The event trail behind decisions and runs. | | [Caracal Operator](/v1.0/concepts/operator/) | An optional governed assistant for reviewed console changes. | ## Core Flow ```mermaid flowchart LR App["Application"] --> Session["Session"] Subject["Federated user (optional)"] -. attribution .-> Session Session --> SDK["SDK / Gateway request"] SDK --> STS["STS token exchange"] STS --> Policy["Active policy set"] Policy --> Mandate["Mandate JWT"] Mandate --> Gateway["Gateway or adapter"] Gateway --> Resource["Protected resource"] STS --> Audit["Audit ledger"] Gateway --> Audit ``` The same model appears across the product: * Onboarding uses the web console guided setup to create the first zone, application, provider, resource, and policy, then makes the first protected call with the application's identity through an SDK transport. * Guides use the SDKs, web console, Admin API, and adapters to build repeatable integrations. * Operations pages use the same terms when explaining keys, revocation, audit, and runtime health. ## Term Map | Term | Short definition | Canonical page | | ----------------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | Zone | Tenant boundary for authority data and signing keys. | [Zones](/v1.0/concepts/zone/) | | Application | Registered client or agent workload. | [Identities and Applications](/v1.0/concepts/principal/) | | Subject | The identity work is done for: the Application itself by default, or a Federated user for attribution and supported approval flows. | [Identities and Applications](/v1.0/concepts/principal/) | | Authority record | Immutable record of identity and authority context created by an exchange. | [Identities and Applications](/v1.0/concepts/principal/) | | Resource | Protected API, MCP server, tool group, or upstream target. | [Resources and Grants](/v1.0/concepts/resource-grant/) | | Grant | Policy data that describes which Application roles may request Resource scopes. | [Resources and Grants](/v1.0/concepts/resource-grant/) | | Provider | Credential mode Caracal uses toward the upstream target. | [Providers](/v1.0/concepts/provider/) | | Policy | Versioned Rego logic evaluated at token exchange. | [Policies and Policy Sets](/v1.0/concepts/policy/) | | Policy set | Versioned bundle of policies activated for a zone. | [Policies and Policy Sets](/v1.0/concepts/policy/) | | Mandate | Short-lived JWT issued by the STS after policy approval. | [Mandates](/v1.0/concepts/mandate/) | | Delegation | Bounded authority transfer between Sessions. | [Session Delegation](/v1.0/concepts/delegation/) | | Revocation anchor | Authority record ID, Root authority record ID, Session ID, or Delegation ID checked by resource servers. | [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) | | Caracal Operator | Governed natural-language assistant that turns intent into reviewed control-plane changes. | [Caracal Operator](/v1.0/concepts/operator/) | | System zone | Reserved `caracal.sys/` zone for the infrastructure that runs Caracal. | [Zones](/v1.0/concepts/zone/#system-zone) | ## What to Read Next After the concepts, use [Guides](/v1.0/guides/) for task-focused procedures or [SDKs](/v1.0/sdks/) for language-specific reference. --- # Caracal Mental Model # URL: https://docs.caracal.run/v1.0/concepts/model-overview/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/model-overview.md # Type: page # Concepts: # Requires: --- Caracal answers one question: **should this Application receive scoped authority for this Resource right now?** It answers that question during token exchange, records the decision, and returns a mandate only when the active policy set allows the request. ```mermaid flowchart TD Zone["Zone"] --> App["Application"] Zone --> Resource["Resource"] Zone --> PolicySet["Active policy set"] App --> Session["Session"] Session --> Exchange["Token exchange"] Resource --> Exchange PolicySet --> Exchange Exchange -->|"allow"| Mandate["Mandate"] Exchange -->|"deny / approval"| Audit["Audit event"] Mandate --> Audit ``` ## Before You Begin Know the Application and Resource from Get Started. No protocol knowledge is required. ## Eight Core Nouns | Noun | What it means | | ----------- | ---------------------------------------------------------------------------------- | | Zone | The trust boundary for configuration, signing keys, policy, Sessions, and audit. | | Application | Registered software that authenticates to Caracal. | | Subject | The identity work is done for: the Application itself by default, or a Federated user from your identity provider. | | Session | One governed execution under an Application. | | Resource | The protected target: API, MCP server, tool group, or upstream service. | | Provider | Sealed custody of a Resource's upstream credential, attached only after approval. | | Policy | Versioned data the platform decision contract evaluates during exchange. | | Mandate | The short-lived signed proof accepted by the Gateway or a verified service. | Get Started introduced Application, Resource, Provider, Policy, Mandate, Gateway, Audit, and Zone; this model adds the two runtime nouns behind them - Session and Subject. Most of these are configured directly. A *grant* is a permission binding for resource scopes that policy can read during evaluation; it is not the mandate a resource server verifies. ## One Application, Many Sessions An application and a Session are different layers, and they scale differently: * An **application** is the credentialed security boundary - operator-provisioned or dynamically registered, holding the secret Caracal authenticates. It is created deliberately. * A **Session** is the governed execution unit - started the moment your software acts, with no secret and no registration step. One application backs many Sessions. A long-running service registers **one managed application**, then starts, delegates, and fans out as many Sessions as it needs under that single credential. You do not create an application per AI agent; per-execution attribution comes from the Session, not a new application. See [Should I create one application per agent?](/v1.0/reference/faq/#faq-006). ## Three Runtime Verbs | Verb | Meaning | | -------- | ---------------------------------------------------------------------------------------------------------- | | Exchange | Ask the STS to convert existing identity into a resource mandate. | | Start | Open a governed Session, optionally attaching a Subject authority record ID for attribution and lifecycle. | | Delegate | Pass constrained authority from one Session to another. | ## One Decision Point Caracal evaluates the active policy with the Application, Authority record, Session, Resource, scopes, Delegation, and Approval context. If policy allows the request, Caracal signs a Mandate. If policy denies it, no Mandate is issued. A Federated user can be attached for attribution and supported Federated user approval flows. Federation does not create a separate per-Subject permission system: Resource authority still comes from the Application, policy, and any Delegation. Resource servers still verify mandates locally through the Gateway or adapters. That keeps every request protected even after token exchange succeeds. ## Why This Model Matters * Long-lived provider secrets stay out of agents. * Authority expires quickly and can be revoked. * Delegation carries typed constraints instead of informal trust. * Every allow, deny, Approval, and revocation path is auditable. ## Outcome You should be able to separate durable Application identity, Subject attribution (the Application itself or a Federated user), temporary Session execution, and short-lived Mandates. Next, read [Authority and Enforcement](/v1.0/concepts/authority-model/) to see where each enforcement layer fits. --- # Authority and Enforcement # URL: https://docs.caracal.run/v1.0/concepts/authority-model/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/authority-model.md # Type: page # Concepts: # Requires: --- Use this page to choose where Caracal makes and enforces an authorization decision. First understand [Caracal Mental Model](/v1.0/concepts/model-overview/). Caracal separates **granting authority** from **using authority**. * The STS grants authority by issuing a short-lived mandate after policy evaluation. * The Gateway or adapter uses authority by verifying the mandate before forwarding a request or running a tool. ## Enforcement Flow ```mermaid sequenceDiagram participant Client as Agent or app participant STS as STS participant Policy as Active policy set participant Guard as Gateway or adapter participant Resource as Protected resource participant Audit as Audit ledger Client->>STS: Exchange subject token for resource scopes STS->>Policy: Evaluate Application, Session, grant data, Resource, Delegation Policy-->>STS: allow, deny, diagnostics, or Approval requirement STS->>Audit: Record decision alt allowed STS-->>Client: Mandate JWT Client->>Guard: Request with mandate Guard->>Guard: Verify signature, claims, expiry, scopes, revocation Guard->>Resource: Forward approved request Guard->>Audit: Record resource decision else denied or Approval required STS-->>Client: Error or interaction_required hold end ``` ## Control Layers | Layer | Responsibility | Source of truth | | ------------------ | ---------------------------------------------------------------- | ---------------------------------- | | Zone | Owns keys, policies, resources, sessions, and audit data. | Console or Admin API | | Grant data | Declares which Application roles may request Resource scopes. | Console or Admin API | | Policy set | Makes the final allow, deny, or Approval decision. | Rego policy versions | | Mandate | Carries approved authority as a short-lived signed JWT. | STS | | Gateway or adapter | Verifies mandate claims and revocation before use. | Runtime and resource server config | | Audit ledger | Records decisions, diagnostics, and request correlation. | API, Console, and storage | ## Where Resource Authority Comes From A Session does not receive Resource authority merely because it exists. Application-owned calls use the SDK's bounded call path. Agent work receives a narrowed Delegation. Both paths still require active policy approval before Caracal issues a Mandate. ```mermaid flowchart TD Zone["Zone grant + policy set
(what the application MAY hold)"] --> App["Application identity"] App --> Session["Session
(lifecycle-only authority)"] Session -- "Authority.narrow(...)
parent issues a bounded edge" --> Edge["Delegation
(scopes x resource x TTL)"] Peer["Peer session"] -- "delegate(...) + acceptDelegation" --> Edge App -- "applicationTransport
(SDK builds its own session pair + edge)" --> Edge Edge --> Mint["STS mint: policy evaluates
grant + edge + session"] Mint --> Mandate["Resource mandate
(short-lived, resource-audienced JWT)"] ``` Common mistake: treating a Session as a permission grant. A Session supplies lifecycle and attribution. Use an Application-owned call for ordinary app work, or attach a narrowed Delegation when a child or peer must hold less authority. ## Default Posture Caracal should be treated as deny-by-default: 1. A resource must be registered. 2. The Application must have an applicable grant-data or Delegation path. 3. The active policy set must allow the requested resource and scopes. 4. The resource server must verify the mandate and revocation anchors. Missing configuration, invalid signatures, expired mandates, insufficient scopes, revoked sessions, and failed delegation checks all stop the request. ## Gateway and Adapter Roles Use the Gateway when you want Caracal to front an HTTP upstream and mediate requests centrally. Use an adapter when the resource server should verify mandates inside its own framework, such as Express, FastMCP, or net/http, or use the verify engine directly for custom boundaries. Both patterns share the same authority model. The difference is only where mandate verification runs. ## Enforce, Propagate, or Attribute The SDK exposes several call paths. They are not interchangeable: each does a different job, and only some **enforce** authority. Pick by what you need at that boundary. | Call path | Role | Verifies the mandate? | Use when | | -------------------------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------ | -------------------------------------------------------------------------------- | | SDK request through the Gateway | **Enforce** | Yes - the Gateway verifies claims and revocation before forwarding | Caracal fronts an HTTP upstream and mediates centrally | | Direct Gateway call (`curl` to the Gateway URL) | **Enforce** | Yes - same Gateway verification | Non-SDK clients or debugging the enforced path | | Adapter verify (`context_middleware(verifier=…)`, Express, FastMCP, net/http, verify engine) | **Enforce** | Yes - the adapter verifies inside the resource server | The resource server should enforce in its own framework | | `caracal.transport` / `context_middleware()` (no verifier) | **Propagate** | No - carries and binds the envelope only | A Gateway or adapter already enforced upstream and you only need context to flow | | App-managed provider call (your code calls the provider SDK directly) | **Attribute** | No - Caracal records who acted, but does not gate the call | You want audit attribution without routing through the Gateway | Rule of thumb: **use the Gateway or an adapter verifier to enforce; use transport/propagation to carry identity after enforcement; treat app-managed provider calls as attribution only.** If a path says "no" under *verifies the mandate*, something upstream must have already enforced it. ## Outcome You should be able to name the decision point, the enforcement boundary, and whether a proposed call path enforces, propagates, or only attributes authority. ## Next Step Read [Zones](/v1.0/concepts/zone/) to understand the tenant boundary that owns authority data. ## Related Pages * [Policies and Policy Sets](/v1.0/concepts/policy/) explains the decision contract. * [Mandates](/v1.0/concepts/mandate/) explains the issued token. * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) explains how active authority ends. --- # Zones # URL: https://docs.caracal.run/v1.0/concepts/zone/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/zone.md # Type: page # Concepts: # Requires: --- Use this page when deciding what must be isolated from what. A Zone is Caracal's main trust boundary: it groups the identities, policy, Resources, keys, runtime authority, and audit evidence that must share one administrative boundary. ## What a Zone Owns | Area | Zone-owned data | | ------------- | ---------------------------------------------------------------------------- | | Identity | Applications, credentials, Authority records, and Sessions. | | Authorization | Resources, Providers, grants, policies, policy sets, and Approvals. | | Cryptography | Zone signing keys and JWKS used to verify mandates. | | Delegation | Delegations, constraints, depth limits, and cascade revocation state. | | Audit | Decision events, diagnostics, request IDs, and explain traces. | ## Why Zones Exist Zones let teams run separate environments, tenants, or trust domains without mixing authority data. Common zone boundaries include: * production, staging, and development environments; * separate customers in a hosted deployment; * isolated product areas with different policy owners; * high-sensitivity resources that need distinct keys and audit trails. When several boundaries could apply at once, use [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/) to choose between a separate zone, a shared zone with separate resources, and a customer attribute in policy input. To serve many of your own customers from one zone, follow [Serve Your Own Customers](/v1.0/guides/serve-customers/). :::note[FAQ] [What should a zone represent?](/v1.0/reference/faq/#faq-003) ::: ## Zone Lifecycle ```mermaid flowchart LR Create["Create zone"] --> Register["Register applications and resources"] Register --> Policy["Author and activate policy set"] Policy --> Run["Issue mandates and run agents"] Run --> Audit["Review audit and explain traces"] Audit --> Tune["Update grants, policy, and resources"] Tune --> Run ``` Zone setup is normally managed through the web console. The Admin API exposes the same objects for automation. ## Key and Policy Isolation Each zone has its own signing-key and JWKS context. Resource servers verify mandates against the issuer, audience, and expected zone. Policy activation is also zone-scoped: activating a policy set in one zone does not affect another zone. ## Advanced Application Registration A Zone can allow programmatic registration of auto-expiring **DCR Applications** (Dynamic Client Registration). This is off by default and is for separate credential boundaries, not ordinary Session fan-out. The console does not create these Applications. Disabling registration while live DCR Applications exist requires an explicit decision: | Choice | Effect | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Keep live | Blocks new registrations; existing DCR applications stay valid until their own expiry. | | Revoke live | Blocks new registrations and immediately archives live DCR applications, revoking their sessions and terminating related agent access. | The console prompts for this choice. Use the Admin API reference when automating it. ## System Zone Caracal reserves one **system zone** for the infrastructure that runs the platform itself, distinct from the tenant zones you create. It carries the reserved `caracal.sys/` namespace and is provisioned by the bootstrap admin identity, so a tenant can never create, author, or impersonate the objects inside it. The [Caracal Operator](/v1.0/concepts/operator/) self-governs through this zone: its reserved control identity, least-privilege control grants, and the providers and resources that route its model calls all live there. The Operator will not open a session in, or execute against, the system zone, so its delegated authority stays away from the infrastructure that runs Caracal. An administrator can also mark additional zones as system zones to place them out of the Operator's reach. ## Operational Guidance * Keep production and non-production authority in separate zones. * Name zones after the trust boundary, not a single service. * Keep resource identifiers stable because policies, grants, and audit traces refer to them. * Rotate zone signing keys using the operations workflow, then confirm resource servers load the current JWKS. ## Next Step Read [Identities and Applications](/v1.0/concepts/principal/) to understand who acts inside a zone. ## Related Pages * [Resources and Grants](/v1.0/concepts/resource-grant/) * [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/) * [Audit and Request Traces](/v1.0/concepts/audit-ledger/) --- # Identities and Applications # URL: https://docs.caracal.run/v1.0/concepts/principal/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/principal.md # Type: page # Concepts: # Requires: --- Use this page to decide which identity is durable, which identity is optional, and which record identifies one execution. An **Application** is registered software that authenticates to Caracal. A **Subject** is the identity work is done for - the JWT `sub` recorded on Authority records and Mandates. Every exchange has a Subject, and a Subject is one of two kinds: the **Application itself** (the default, when software acts as itself) or a **Federated user** (an external identity supplied by a trusted identity provider). An **Authority record** is an immutable record of identity and authority context. A **Session** is one governed execution under an Application. ## Identity and Execution Objects When you need the operator-facing view of these objects side by side - including how their IDs appear on the wire - the identity table in [Manage Runtime Authority](/v1.0/runtime-console/agents/#keep-the-identities-distinct) is the quick reference. | Public object | Typical source | What it controls | | ---------------- | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | Application | Workload credential or Application secret | The durable credential and policy boundary for software. | | Subject | The application's own identity by default; a Federated user's token when one is exchanged | Attribution, provider connection ownership, revocation anchor, and supported Federated user approvals; not scope authority alone. | | Federated user | Token from your registered identity provider | One kind of Subject: an external end-user identity Caracal federates and records but never authenticates itself. | | Authority record | Successful identity or authority exchange | Immutable audit and revocation context. | | Session | SDK runtime primitive | The lifetime and exact attribution of one governed execution. | No identity object authorizes a Resource by itself. A request still needs Resource scopes, applicable policy, and any required Delegation or Approval. ## Application Roles Applications represent software that can participate in Caracal flows: * an agent runtime that starts child Sessions; * a backend service that requests mandates; * a Gateway application that fronts protected upstreams; * an adapter-protected resource server; * a managed or dynamically registered client. Applications have registration metadata, a server-owned credential, and a registration method. **Managed** Applications are durable and operator-provisioned for known software, including runtimes that start child Sessions. **DCR** Applications (Dynamic Client Registration) are auto-expiring and created programmatically when a separate temporary credential boundary is needed, not for ordinary Session fan-out. ## Applications Are the Credential Boundary; Sessions Are the Runtime Unit ```mermaid flowchart TD App["Managed application
(credential boundary)"] --> S1["Session"] App --> S2["Session"] App --> S3["Session"] S2 --> C1["Started child Session"] S3 --> C2["Delegated Session"] ``` An application is registered, holds a server-owned secret, and is the identity Caracal authenticates. A Session is started at runtime by the process that already holds that secret; it carries parent, Subject authority record ID, labels, and delegation context. One application backs many Sessions, so a long-running service uses **one managed application** and starts, delegates, and fans out as many Sessions as it needs. You do not register an application per AI agent - see [Should I create one application per agent?](/v1.0/reference/faq/#faq-006). ## Managed and DCR Applications | Kind | Use it for | Rule | | ------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | Managed (durable) | A service, orchestrator, Gateway, or agent runtime | Create once and reuse across many Sessions for the same service. | | DCR (auto-expiring) | An isolated credential boundary for a tenant or integration | Create programmatically; it expires, binds to one task Session, and cannot participate in a child tree. | Sessions use one of two lifecycles: a **task Session** ends with a bounded unit of work, while a **service Session** stays active through a heartbeat lease. Lifecycle describes time, not actor type or authority. A task Session may have a wall-clock TTL. A service Session ends when its heartbeat lease lapses or it is closed. Keeping the lease alive does not widen authority. Two structural rules prevent invalid trees: * A `task` parent cannot start a `service` child. The protocol reports `task_session_cannot_start_service`. A `service` parent may start either lifecycle. * A DCR Application's Session is a **leaf**: it cannot parent or be a child Session. A short-lived worker is an ordinary task Session with a TTL. Model an orchestrator and its workers under one managed Application unless they need separate credentials or mutual distrust boundaries. * the orchestrator is the top-level Session, or a [`start_session()`](/v1.0/sdks/python/) handle when it needs a heartbeat lease, * each manager is a plain `session()` that inherits the application's authority, * each task worker is `session(authority=Authority.narrow([...]), ttl_seconds=…)` - least-privilege and auto-terminated on block exit, with the TTL sweeper as a backstop. Use a DCR Application for credential isolation, not Session fan-out. It authenticates independently; it is not started as a child Session. :::note[FAQ] [What is the difference between an application, principal, and Session?](/v1.0/reference/faq/#faq-005) and [when should I use a managed application versus DCR?](/v1.0/reference/faq/#faq-007) ::: Policy and audit can distinguish Application kind, Session lifecycle, labels, parentage, and Delegation context. ## Telling Sessions Apart Every Session has one canonical Session ID. SDK context exposes it as `sessionId`, `session_id`, or `SessionID`. It is returned when `session()` or `startSession()` starts the Session and is stamped onto its token exchanges and audit events. `labels` are descriptors, not identity. Many Sessions under one application can intentionally share labels. Use labels, metadata, or a trace ID for business correlation; use Session ID for exact attribution. The Admin API audit endpoint filters by `session_id` for one Session or `label` for a role across many Sessions. ## Federated Users Are Attribution, Not a Permission Store A Federated user is one kind of Subject: an external end-user identity supplied by a trusted identity provider. Caracal never authenticates Federated users itself; it verifies their identity tokens only from a Federated user issuer registered in the Zone (API resource: `subject-issuers`), then federates and records the identity. The resulting Authority record can be attached to a Session as immutable attribution and as a revocation anchor. Supported Federated user approval flows can also require that identity. When no external identity is exchanged, the Subject is simply the Application's own identity - there is no Federated user, and nothing is missing. Caracal does **not** derive per-Subject Resource scopes from federation. Do not claim that attaching Richard Hendricks authorizes a call. The Application, active policy, requested scopes, and any Delegation still determine authority. ## Sessions Bind Identity to Time ```mermaid flowchart TD App["Application"] --> Authority["Authority record
(Subject: application or Federated user)"] Fed["Federated user token (optional)"] --> Authority Authority --> Session["Session"] Session --> Delegate["Delegation"] Authority --> Mandate["Mandate exchange"] Session --> Mandate Delegate --> Mandate ``` Authority records, Sessions, and Delegations make authority revocable. Mandates carry their identifiers as revocation anchors. ## Naming Guidance * Use **Application** for registered software and **Subject** for the identity work is done for in user-facing material. * Use **Subject** for the JWT `sub` identity. Every exchange has one: the Application itself by default, or a Federated user. * Use **Federated user** for the external end-user kind of Subject; never describe a Subject as only a federated identity. * Use **Authority record** for an STS exchange record. * Use **Session** for a governed Coordinator execution. * Avoid using "client" unless you are describing OAuth protocol fields. ## Next Step Read [Resources and Grants](/v1.0/concepts/resource-grant/) to understand what identities can request. ## Related Pages * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) * [Session Delegation](/v1.0/concepts/delegation/) * [Integrate the TypeScript SDK](/v1.0/guides/sdk-typescript/) --- # Resources and Grants # URL: https://docs.caracal.run/v1.0/concepts/resource-grant/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/resource-grant.md # Type: page # Concepts: # Requires: --- Use this page to separate the protected target from the policy data used to evaluate access. A Resource is something Caracal protects. Grant data maps an Application role to scopes for that Resource. ## Resources Resources describe protected targets such as: * HTTP APIs behind the Gateway; * MCP servers and tool groups; * internal services protected by Express, FastMCP, or net/http adapters; * provider-backed targets that need credential mediation. | Resource field | Purpose | | --- | --- | | Identifier | Stable policy and token audience target. Always use the `resource://` convention, such as `resource://pipernet`; keep it stable even when the upstream URL changes. | | Upstream URL | Gateway forwarding target. | | Scopes | Named Caracal resource actions that policies and mandates can constrain. | | Gateway application | Managed application identity used by Gateway-mediated resources. | | Upstream credential provider | Resource binding to the provider record used when Gateway attaches no credential, a Caracal mandate, OAuth tokens, API keys, or bearer tokens. | :::note[FAQ] [What is the difference between a resource and a provider?](/v1.0/reference/faq/#faq-009) and [why must the resource identifier stay stable?](/v1.0/reference/faq/#faq-010) ::: ## Grants A grant data entry binds an Application key and role to a Resource and one or more scopes. It is not a Mandate and it is not a per-Subject permission assignment. Grants are not the final decision. They are one input to Policy. The active Policy set can still deny, require Approval, or constrain the exchange. Caracal also stores administrative records associated with Subjects for lifecycle and revocation workflows. Those records do not feed per-exchange scope decisions. Subject-specific upstream accounts (often one per Federated user) are credential Provider connections, not Caracal authorization grants. ## Exchange Relationship ```mermaid flowchart LR App["Application"] --> Grant["Grant"] Role["Application role"] --> Grant Resource["Resource"] --> Grant Grant --> STS["STS policy input"] Policy["Active policy set"] --> STS STS -->|"allow"| Mandate["Mandate with resource and scopes"] ``` ## Scope Design Prefer small, action-oriented scopes: | Good | Avoid | | --- | --- | | `pipernet:read` | `admin` | | `piperchat:comment` | `write_all` | | `nucleus:tool:call` | `tools` | Use Resource identifiers for targets and scopes for actions. Do not encode environment, tenant, or Subject identity into scope names when those belong in the Zone or decision context. :::note[FAQ] [How should I design scopes?](/v1.0/reference/faq/#faq-011) and [do I manage grants directly?](/v1.0/reference/faq/#faq-012) ::: ## Next Step Read [Providers](/v1.0/concepts/provider/) to understand the credential Caracal attaches when it calls the upstream target. ## Related Pages * [Define Resources and Providers](/v1.0/guides/resources-providers/) * [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/) * [Debug Authorization Decisions](/v1.0/guides/authorize-access/) --- # Providers # URL: https://docs.caracal.run/v1.0/concepts/provider/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/provider.md # Type: page # Concepts: # Requires: --- A credential Provider answers one question: after Caracal approves a call, what credential does the upstream target receive? Read [Resources and Grants](/v1.0/concepts/resource-grant/) first. Every Gateway-routed Resource binds one Provider; one Provider can serve many Resources. ## Why Providers Exist Callers never hold upstream credentials. The agent presents a Caracal mandate; Gateway verifies it, then attaches the upstream credential the provider describes. Provider secrets are sealed at creation and are not returned by list or detail APIs. ## Auth Modes | Mode | Upstream receives | Use when | | --- | --- | --- | | None | No credential. | Gateway is the enforcement point and the upstream expects nothing. | | Caracal mandate | The Caracal mandate as a bearer token. | The upstream verifies Caracal tokens itself with a verifier or adapter. | | OAuth 2.0 authorization code | A consented upstream account's token. | The upstream needs delegated account consent. | | OAuth 2.0 client credentials | A service-to-service token. | The upstream uses machine-to-machine OAuth, via the standard grant or an RFC 7523 signed-assertion grant such as a Google service account. | | API key | A static key in a configured header. | The upstream uses vendor API keys. | | Bearer token | A static pre-issued token. | The upstream expects a fixed bearer credential Caracal does not mint. | | HTTP Basic | `Authorization: Basic` from a username and sealed password. | The upstream authenticates with a username/password or username/API-token pair. | ## How a Provider Is Used ```mermaid flowchart LR Agent["Agent"] -->|mandate| Gateway Provider["Provider record"] --> Gateway Gateway -->|verify + exchange| STS Gateway -->|upstream credential| Upstream["Protected target"] ``` Gateway strips the caller's authorization and forwards the provider credential instead. For OAuth modes, STS obtains and refreshes the upstream tokens; delegated consent is stored as a provider connection - by default one shared upstream account for the provider that serves every session policy authorizes, with an optional per-Subject binding when a Zone needs a distinct upstream account per customer. A connection is distinct from Caracal grants, which express authorization. ## Common Mistakes * A resource says what is protected; its provider says how the upstream is authenticated. Keep credential detail on the provider and target detail on the resource. * Name providers with stable `provider://` identifiers, such as `provider://hooli-oidc`. * OAuth providers support a real connectivity check before creation. The other modes are validated at creation and exercised when a resource first uses them. * A Provider controls upstream authentication, not whether Caracal issues a Mandate. * A Provider connection identifies the upstream account Gateway uses; it does not grant Caracal scopes. :::note[FAQ] [What is the difference between a resource and a provider?](/v1.0/reference/faq/#faq-009) and [is an application secret the same as a provider credential?](/v1.0/reference/faq/#faq-013) ::: ## Next Step Read [Policies and Policy Sets](/v1.0/concepts/policy/) to understand how requests against a Resource are allowed, denied, or held for Approval. ## Related Pages * [Define Resources and Providers](/v1.0/guides/resources-providers/) * [Provider Recipes](/v1.0/guides/provider-recipes/) * [Resources and Grants](/v1.0/concepts/resource-grant/) --- # Policies and Policy Sets # URL: https://docs.caracal.run/v1.0/concepts/policy/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/policy.md # Type: page # Concepts: # Requires: --- Use this page to understand what policy owners configure and what Caracal guarantees. Caracal owns the decision contract. You author versioned policy data describing Application bindings, Resource grants, confinement, restrictions, risk, and Approval tiers. One policy-set version is active per Zone. ## Policy Objects | Object | Purpose | | ------------------------- | --------------------------------------------------- | | Policy | Named policy data document with immutable versions. | | Policy version | A specific content hash and schema version. | | Policy set | A named bundle of policy versions. | | Policy set version | A specific manifest of policy versions. | | Active policy set version | The version the STS evaluates for a zone. | ## Decision Contract The decision contract is deny by default. You never replace its decision result; you supply the data it reads. This keeps structural checks, Delegation narrowing, and Approval behavior consistent across Zones. Under the hood the contract is Rego evaluated by an embedded [Open Policy Agent](https://www.openpolicyagent.org/) engine inside the STS - which is why policy content is Rego syntax and offline policy tests use `opa test` - but the decision rules are platform-owned; adopters ship data. ```rego # caracal:data-document package caracal.authz import rego.v1 grants := { "resource://pipernet": { "application": "anton", "roles": {"operator": ["pipernet:read"]}, }, } ``` The contract reads six adopter documents: | Document | Shape | Effect | | ------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `app_ids` | `{binding_key: application_id}` | Binds a readable Application key to its registered ID. | | `grants` | `{resource: {application, roles: {role: [scopes]}}}` | Declares which application owns a resource view and which scopes each role may hold. | | `confinement` | `[{label_prefix, scopes}]` | Caps every Session with a matching label prefix to a fixed scope set. Optional; omitting it confines nothing. | | `restrict` | set of reasons | Deny overlay. Any entry denies every exchange in the zone. Optional; keep it empty to authorize normally. | | `risk` | `[{scope, tier}]` | Names a risk tier for each sensitive scope. Optional; feeds the approval gate. | | `approval_tiers` | `[{tier, approver, ttl_seconds, privacy}]` | Declares which tiers hold a mint for a human decision. Optional; a zone without it never raises a hold. | A zone that supplies no data authorizes nothing: every allow rule collapses to the default deny. ## Policy Input Contract Every evaluation receives documented Application, Resource, action, Session, Delegation, requested-scope, Subject-claim, Approval, and trace context. Use [Author Policy Data](/v1.0/guides/author-policy/) for the current field-level contract. | Context area | What policy can reason about | | --- | --- | | Application | Registered ID, kind, and Zone. | | Session | ID, task or service lifecycle, and labels. | | Resource and action | Stable Resource identifier, declared scopes, requested scopes, and Gateway HTTP operation when present. | | Delegation | Source, target, narrowed Resource and scopes, and verified chain context. | | Subject | Verified claims for structural binding and audit, not an independent scope grant. | | Approval and trace | Whether the matching Approval was resolved and the request trace ID. | Verified Subject claims are available for structural binding and audit. The built-in allow rules do not turn those claims into per-Subject scope grants. If a product needs user-specific business authorization, keep that in the product's own authorization system rather than claiming Caracal federation provides it. ## Policy Outcomes | Outcome | Effect | | ------------- | -------------------------------------------------------------------------- | | `allow` | STS signs a mandate if token and session checks also pass. | | `deny` | STS refuses the exchange and records diagnostics. | | approval gate | STS holds the mint and returns `interaction_required` with an approval ID. | Approval gates are declared as data: `risk` names a tier for each sensitive scope and `approval_tiers` declares which tiers require a human decision. When a mint requests a gated scope, STS creates a durable hold and releases the Mandate only after an authorized approver decides it. See [Approvals](/v1.0/concepts/approvals/). ## Authoring Rules * Author data documents only; mark each with `# caracal:data-document`. The platform decision contract owns every `result`. * Validate through the web console or Admin API before activation - a data document must define data and must not define `result`. * Activate through a policy set version, not by editing active content in place. * Keep `grants`, `app_ids`, `confinement`, and `restrict` in separate documents so ownership and review stay clear. * Tighten authority through `confinement` and `restrict`; neither can widen what the contract already allows. ## Next Step Read [Mandates](/v1.0/concepts/mandate/) to understand the short-lived proof an allowed exchange produces. ## Related Pages * [Author Policy Data](/v1.0/guides/author-policy/) * [Activate a Policy Set](/v1.0/guides/activate-policy-set/) --- # Mandates # URL: https://docs.caracal.run/v1.0/concepts/mandate/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/mandate.md # Type: page # Concepts: # Requires: --- Read this page after [Policies and Policy Sets](/v1.0/concepts/policy/). A Mandate is the short-lived signed proof Caracal issues after policy allows a request. The Gateway or a verified service checks it before the action runs. ## What a Mandate Proves A valid mandate proves: * which zone issued it; * which Application is acting and which Subject the work is attributed to - the Application itself, or a Federated user; * which session anchors are active; * which resource targets and scopes were approved; * which Authority record, Session, and Delegation supplied the authority; * when the authority expires. ## Issuance Path ```mermaid flowchart LR Subject["Subject token or Session context"] --> Exchange["OAuth token exchange"] Exchange --> Policy["Policy evaluation"] Policy -->|"allow"| Mandate["Mandate JWT"] Mandate --> Verify["Gateway or adapter verification"] Verify --> Resource["Protected resource"] ``` ## Mandate Use Classes Every mandate carries a `use` claim naming the boundary that may accept it. Pages across the docs refer to these classes as `use=session`, `use=gateway`, and `use=resource`: | `use` claim | Minted when | Accepted by | | --- | --- | --- | | `session` | An application's lifecycle bootstrap exchange, with no Session or Delegation context. Reusable within its TTL. | Coordinator lifecycle operations and later exchanges as a subject token. | | `gateway` | A direct mint with Session and Delegation context - what SDK transports send to the Gateway. Single-use. | Gateway ingress only. | | `resource` | The Gateway's own authenticated exchange of an inbound `use=gateway` mandate. | The upstream path: `caracal_mandate` resources and in-process verifiers. | A verifier must accept only its own class: resource servers require `use=resource`, the Gateway requires `use=gateway`, and neither accepts a Session or lifecycle mandate. ## Where Mandates Are Verified | Boundary | Verification focus | | ------------------ | ------------------------------------------------------------------------------ | | Gateway request | Issuer, audience, zone, resource, scopes, expiry, replay, revocation. | | MCP tool call | Bearer token, required scopes, required targets, Session and Delegation constraints. | | SDK outbound call | Context propagation and mandate header injection. | | Delegated exchange | Session, Delegation, scopes, hop count, and constraints. | ## Mandates Are Not Credentials to Store Mandates are intentionally short lived and context bound. Do not store them as durable credentials, copy them into configuration files, or reuse them across unrelated Resources. Resource servers should always verify a mandate at request time. Verification includes signature and claim checks plus revocation checks for the Authority record ID, Root authority record ID, Session ID, and Delegation ID when those claims are present. See the [parsed claim mapping](/v1.0/sdks/identity/#parsed-claim-names) for language-level and raw JWT names. ## Failure Modes | Failure | Meaning | | ------------------------- | ------------------------------------------------------------------------- | | `invalid_token` | Signature, issuer, audience, required claim, or expiry validation failed. | | `scope_insufficient` | The mandate does not contain a required scope. In-process verifiers report the same condition as `insufficient_scope`. | | `session_revoked` | One of the mandate revocation anchors has been revoked. | | `session_required` | The resource requires authority from a governed Session. | | `delegation_required` | The resource requires delegated authority. | | `chain_mismatch` | The delegation chain does not include the required application. | | `hop_count_exceeded` | The delegation path exceeds the configured hop limit. | ## Next Step Read [Approvals](/v1.0/concepts/approvals/) to understand how sensitive requests wait for a human decision before a Mandate is issued. ## Related Pages * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) * [Protect an MCP Server](/v1.0/guides/protect-mcp/) * [Run an Agent with caracal run](/v1.0/guides/runtime-run/) --- # Session Delegation # URL: https://docs.caracal.run/v1.0/concepts/delegation/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/delegation.md # Type: page # Concepts: # Requires: --- Read this page when one Session must give a child or peer less authority than the Application could otherwise request. A Delegation is a revocable, expiring authority relationship between Sessions. Authority follows the **application**. Sessions started under the same application already act under that application's authority. Create a Delegation in exactly two cases: * To **narrow** authority, so a child holds only a subset of what its parent can do (least privilege). * To carry authority **across applications**, when a receiving Session belongs to a different application and explicitly presents the offered Delegation ID. A Delegation is represented as a graph of directed edges. Each edge connects a source session to a target session, carries scopes and constraints, and can be revoked independently. ## Graph Model ```mermaid flowchart LR Authority["Subject authority record"] --> A["Session A"] A -->|"delegation: read tickets"| B["Session B"] B -->|"delegation: summarize only"| C["Session C"] A -. revoke .-> B B -. cascade .-> C ``` ## Delegation Fields | Field | Purpose | | -------------------- | ---------------------------------------------------------------- | | Source session | The session that delegates authority. | | Target session | The child or receiving Session. | | Issuer application | Application creating the delegation. | | Receiver application | Application receiving authority. | | Resource | Optional resource boundary for the edge. | | Scopes | Subset of authority being delegated. | | Constraints | Typed resource, scope, TTL, and hop limits plus audit metadata. | | Status | Active, expired, or revoked lifecycle state. | ## Rules * Delegation should narrow authority, not expand it. * Every Delegation must have a positive TTL; unbounded edges are rejected before creation. * Delegation paths must not cycle. * Hop count should be bounded. * Revoking an upstream Delegation should invalidate downstream authority. * Resource servers should verify delegation claims when they require delegated access. Expiry removes the edge's authority without terminating either endpoint Session. STS rejects the expired edge and caps every mandate minted through it so the mandate cannot outlive the edge. The target Session remains active but cannot use that Delegation again; it must receive another live Delegation before it can mint delegated authority. Revocation is different: it is an explicit monotonic action that cascades through downstream Delegations and affected Session subtrees. ## What Delegation Bounds A Delegation bounds every Mandate issued through it and every later Delegation chained from it. It never raises the Application's policy ceiling. * A narrowed child receives no more scopes, Resources, lifetime, or hop budget than its parent. * Inheriting from a narrowed parent preserves that narrowing for descendants. * Inheriting from an Application-root Session does not manufacture delegated Resource authority. Worked example: A starts B with `Authority.narrow([pipernet:read])`, then B starts C. * If B starts C with **inherit**, Coordinator records a `B → C` edge mirroring B's `pipernet:read` slice, so C remains bounded by B's narrowing. * If B starts C with narrower authority, Coordinator records a `B → C` edge and rejects it unless `C ⊆ B`. The Application and active policy remain the hard ceiling. Use a separate Application when the receiver needs a separate credential or trust boundary. A cross-Application receiver must explicitly accept the offered Delegation before using it. ## SDK Relationship The SDKs expose one primitive for creating children, one for granting a peer, and one for presenting a received grant: | Language | Start a child | Delegate to an existing peer | Present a received Delegation | | ---------- | ---------------------------------------- | ---------------------------- | ----------------------------- | | TypeScript | `session()` / `session({ authority })` | `delegate()` / `revokeDelegation()` | `acceptDelegation()` | | Python | `session()` / `session(authority=…)` | `delegate()` / `revoke_delegation()` | `accept_delegation()` | | Go | `Session()` / `Session` with `Authority` | `Delegate()` / `RevokeDelegation()` | `AcceptDelegation()` | `session()` returns a child running under the **same application's** authority; it never moves the child into another application. Pass `Authority.narrow(...)` to bind a least-privilege Delegation or `Authority.none()` for no inherited authority. To hand authority across applications, use `delegate()` with an existing peer Session. The issuer's context is unchanged, and the receiver presents the Delegation with `acceptDelegation()`. These helpers propagate session and delegation context so later token exchanges include the correct graph proof. ## Where You Interact With Delegation Create Delegations at runtime through the SDK. The console is an inspection and revocation surface, not a Delegation authoring form. Audit records the Delegation ID, chain, scopes, and outcome. ## Next Step Read [Delegation Constraints](/v1.0/concepts/constraint/) to understand the limits carried by each edge. ## Related Pages * [Implement Multi-Agent Delegation](/v1.0/guides/delegation/) * [Audit and Request Traces](/v1.0/concepts/audit-ledger/) --- # Delegation Constraints # URL: https://docs.caracal.run/v1.0/concepts/constraint/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/constraint.md # Type: page # Concepts: # Requires: --- Read [Session Delegation](/v1.0/concepts/delegation/) first. Constraints state how far, how long, and for what a Delegation may be used. Caracal validates them when the Delegation is created and again before issuing a Mandate through it. ## Constraint Types | Constraint | Use it to limit | | ---------------- | ----------------------------------------------------------- | | Resource | Which protected target can be reached. | | Scopes | Which actions can be requested. | | TTL | How long the delegated edge remains useful. | | Hop count | How deep the delegation chain may become. | | Approval metadata | Audit/display annotation; it does not authorize the edge. | | Broad reason | Audit/display note for an elevated resource-unbounded edge. | ## Example Shape ```json { "resource": "resource://piperchat", "scopes": ["piperchat:read", "piperchat:comment"], "max_hops": 2, "expires_at": "2026-06-01T12:00:00Z", "policy_approved": true } ``` Delegation constraints bound authority, lifetime, and propagation. They are not quotas: the wire accepts a `budget` field, but it is a narrowing bound - a child edge's budget can never exceed its parent's - not a per-call counter, and no constraint performs usage accounting. Use a domain-specific store when calls, work, or cost must be consumed atomically. Approval metadata and broad-reason fields are audit annotations; they do not authorize the Delegation. Application-chain requirements are verifier options rather than edge constraints. Use `requireChainContains` at a resource boundary when a specific application must appear in the signed delegation chain. ## Where Constraints Are Enforced ```mermaid flowchart TD SDK["SDK creates edge"] --> Coordinator["Coordinator stores constraints"] Coordinator --> STS["STS validates live authoritative lineage"] STS --> Policy["Policy evaluates its documented input"] Policy --> Mandate["Mandate carries delegation claims"] Mandate --> Adapter["Gateway or adapter checks claims"] ``` ## Design Guidance * Put durable business rules in policy data documents. * Put per-edge runtime limits in constraints. * Prefer positive allowlists over open-ended deny lists. * Keep constraints small enough to review in audit traces. * Use consistent field names across agents so policies stay readable. ## Next Step Read [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) to understand how active authority ends. ## Related Pages * [Session Delegation](/v1.0/concepts/delegation/) * [Policies and Policy Sets](/v1.0/concepts/policy/) * [Implement Multi-Agent Delegation](/v1.0/guides/delegation/) --- # Sessions and Revocation # URL: https://docs.caracal.run/v1.0/concepts/sessions-revocation/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/sessions-revocation.md # Type: page # Concepts: # Requires: --- Read this page after [Delegation Constraints](/v1.0/concepts/constraint/). Authority records, Sessions, and Delegations make authority temporary and revocable. A Mandate refers to these records so the Gateway or verified service can reject authority that has ended. ## Identity and Execution Records | Record | Role | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Subject | Opaque JWT `sub` identity - the application itself by default, or a Federated user. The web console's **Subjects** page groups Authority records by this value. | | Authority record | Immutable record created by an identity or authority exchange; an audit and revocation anchor. | | Root Authority record | First record in an authority ancestry; revoking it can end the descendant authority chain. | | Session | Governed execution. A delegated Session is still a Session, with an inbound Delegation. | ## A Subject Can Be a Federated User **Subjects** is not a login surface, and a Subject is not only a federated identity. When an application exchanges as itself, the Subject is the application's own identity. When a zone registers an external identity system as a Federated user issuer and the application exchanges an end user's identity token, the Subject is that Federated user: the `sub` is recorded verbatim from the exchanged token, and the resulting Authority record carries no resource authority. Caracal never authenticates Federated users; it federates and records them. A Session may attach the Federated user's Authority record only while presenting proof of control. That Federated user remains immutable attribution and a revocation anchor for the Session. It does not automatically replace the Application identity on later Resource Mandates and does not create per-Subject scopes. ## Revocation Anchors Resource servers check every relevant anchor: Authority record ID, Root authority record ID, Session ID, and Delegation ID. The [parsed claim mapping](/v1.0/sdks/identity/#parsed-claim-names) lists the canonical language-level names and raw JWT fields. If any anchor is revoked, the mandate should be rejected as `session_revoked`. ## Revocation Flow ```mermaid sequenceDiagram participant Control as Console or Admin API participant State as Authority state participant Distribution as Revocation distribution participant Resource as Gateway or adapter Control->>State: Revoke Authority record, Session, or Delegation State->>Distribution: Publish revocation anchor Resource->>Distribution: Consume revocation Resource->>Resource: Cache revoked anchor Resource-->>Resource: Reject matching mandates ``` Suspension is reversible Session state, not permanent revocation. Gateway-routed requests perform a fresh STS exchange and reject a suspended Session immediately through authoritative Session validation. Already issued mandates checked directly by a resource verifier can remain usable until their mandate TTL expires; keep mandate TTLs within the documented 15-minute cap when suspension latency matters. Termination and Delegation revocation remain monotonic revocation events. ## Cascade Behavior Revocation should follow authority: * revoking an Authority record invalidates authority descended from it; * revoking a Session invalidates its child Delegations; * revoking a Delegation invalidates downstream delegated authority; * revoking a grant prevents future exchange and can invalidate active Authority records and Sessions depending on workflow. ## Resource-Server Responsibility The Gateway and adapters must be configured with a revocation store. For development, an in-memory store can be useful. For production, use a shared store and stream consumer so revocations propagate across resource-server instances. ## Next Step Read [Audit and Request Traces](/v1.0/concepts/audit-ledger/) to understand how decisions and requests are explained. ## Related Pages * [Mandates](/v1.0/concepts/mandate/) * [Protect an MCP Server](/v1.0/guides/protect-mcp/) * [Tail and Query the Audit Stream](/v1.0/guides/audit-stream/) --- # Audit and Request Traces # URL: https://docs.caracal.run/v1.0/concepts/audit-ledger/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/audit-ledger.md # Type: page # Concepts: # Requires: --- Use this page when you need to explain one decision or prove one action. Audit records what Caracal decided, the context used, and which request or Session produced the event. ## What Gets Audited | Event area | Examples | | ----------------------- | --------------------------------------------------------------------- | | Token exchange | Allow, deny, Approval required, and policy diagnostics. | | Gateway and adapter use | Resource decision, mandate verification failure, request correlation. | | Policy lifecycle | Policy creation, validation, policy-set activation, simulation. | | Delegation | Creation, traversal, impact, and revocation. | | Sessions | Start, terminate, revoke, expire. | | Administration | Zone, application, resource, provider, grant, and approval changes. | ## Evidence Flow ```mermaid flowchart LR Decision["Authority decision"] --> Evidence["Audit evidence"] Enforcement["Gateway or verified service"] --> Evidence Admin["Administrative change"] --> Evidence Evidence --> Trace["Request trace"] Trace --> Console["Console"] Trace --> API["Admin API"] ``` ## How to Use Audit | Question | Where to look | | --------------------------------------------- | -------------------------------------------------------------------- | | Why was a request denied? | Console `request trace` or Admin API explain endpoint by request ID. | | Which policy caused the decision? | Determining policies and diagnostics. | | Did revocation propagate? | Session, delegation, and resource decision events. | | Which run made a request? | Request ID, Authority record ID, Session ID, and trace context. | | Was an Approval required, and who decided it? | The `step_up_issued`, `step_up_decided`, and `step_up_consumed` events; filter the audit list by those `event_type` values. | ## Request IDs Request IDs tie multiple events together. Keep the request ID from an SDK, Gateway, STS error, or Console trace whenever debugging. The explain view uses it to collect related decision events and diagnostics. ## Integrity and Retention Tamper evidence is mechanical, not aspirational. Each ingested event is stored with a `content_sha256` of its payload, an HMAC computed under the deployment's `AUDIT_HMAC_KEY`, and the previous event's content hash - forming a per-zone hash chain. A background sweeper continuously recomputes hashes and HMACs and checks chain continuity; any mismatch or chain break surfaces as a tamper metric and readiness signal, and is treated as a security incident, never a retryable formatting issue. The database role that writes evidence cannot update or delete rows. Retention defaults to 365 days (`AUDIT_RETENTION_DAYS`), and complete hourly partitions can be exported to S3-compatible storage - see [Export Audit Evidence](/v1.0/operations/compliance-audit-integration/). Configure retention, export, and SIEM forwarding according to your deployment requirements, and do not rely on local process logs as the only authority trail. ## Outcome For one request ID, you should be able to identify the Application, the Subject (including the Federated user when one was attached), Session, Resource, scopes, policy version, Approval or Delegation context, decision, and enforcement result. ## Next Step Use [Guides](/v1.0/guides/) when you are ready to apply the model. ## Related Pages * [Tail and Query the Audit Stream](/v1.0/guides/audit-stream/) * [Trace One Protected Request](/v1.0/tutorials/inspect-a-run/) * [Operations](/v1.0/operations/) --- # Caracal Operator # URL: https://docs.caracal.run/v1.0/concepts/operator/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/operator.md # Type: page # Concepts: # Requires: --- Read this page only if you plan to use the optional Caracal Operator in the web console. It turns natural-language intent into a previewed plan and applies that plan through the same guarded APIs available to the human operator. It introduces no new authority. ## Why It Exists Most control-plane work is a sequence of small, related changes: register an application, connect a provider, define a resource and its scopes, then activate the policy that ties them together. The Operator collapses that into a described outcome while keeping every safety property - validation, least privilege, approval, and audit - in the platform rather than the model. The Operator works against a **capability catalog**: the set of control-plane actions it can take, grouped by the objects you operate - zones, applications, providers, resources, access, and policy. Each capability is classified as read-only or state-changing, so a request that only inspects state is always distinguishable from one that changes it. ## The Governed Lifecycle A change never applies directly from natural language. Within a session the Operator follows a fixed lifecycle, and the language model only ever produces a draft that enters it: ```mermaid flowchart LR Intent["Natural-language intent"] --> Propose["Propose: draft validated against the catalog"] Propose --> Preview["Preview: read-only dry run against live state"] Preview --> Decide["Decide: you approve or reject"] Decide --> Apply["Apply: re-validated, executed step by step"] Apply --> Audit["Audit: change attributed to you"] ``` 1. **Propose.** Intent becomes a plan whose every step is validated against the capability catalog. A step that names an unknown action or invalid arguments is rejected before anything runs. 2. **Preview.** The plan is resolved against your live state as a read-only dry run, so each step is marked as a create, an update, a no-op, or blocked when a referenced object is missing. Nothing is written. 3. **Decide.** You approve or reject the plan. A plan is decided once, and only an approved plan is eligible to apply. 4. **Apply.** An approved plan is re-validated and re-previewed, then executed step by step. A plan applies only once, and any secret it produces is surfaced in the apply response, never written to the conversation or the audit log; issued credentials stay retrievable from Secret Store custody through the owning object's audited reveal. ## Authority and Isolation The Operator runs as a reserved Application, distinct from the human operator who approves a plan. Audit records both the human decision and the Operator Application that executes the change. That delegated authority is least-privilege: the Operator may execute only the capabilities explicitly granted to it. A plan that asks for a capability outside its grant is refused as forbidden, before execution. The Operator is also bounded by zone isolation - it will not open a session in, or execute against, a [system zone](/v1.0/concepts/zone/#system-zone), keeping its authority away from the infrastructure that runs Caracal itself. ## Ask and Agent Modes Every conversation runs in one of two modes, enforced by Caracal and never chosen by the model: | Mode | What the Operator can do | | ----- | ----------------------------------------------------------------------------------------------------------- | | Agent | Answer questions, read state, and propose plans that apply changes after your approval. | | Ask | Strictly read-only: explain, investigate, and diagnose. It never produces a plan and cannot apply anything. | Ask mode is enforced in two independent places - the planning skill is never selected, and the change endpoints refuse outright - so a read-only conversation is provably write-incapable. Each durable message run records its input and output token totals with a breakdown for every provider and model that served one of its completions. This usage survives the live response for accurate historical cost analysis even when a later call fails over; provider credentials and prompts are never stored in the usage record. ## Autopilot In agent mode you can engage **autopilot**, which lets Caracal auto-satisfy the approval step for every plan in the conversation. Engaging it is an explicit opt-in to acting without a human in the loop; it is off by default and per conversation, and a platform-level master switch must also be on. Auto-approval never widens authority - the governed execute path still enforces the capability allowlist, the least-privilege executor token, and zone isolation on every apply. Autopilot defers while a plan still needs credentials from the console's secure prompt and stops when a preview shows the plan cannot apply. A deployment can also bound an engaged conversation with a **write budget**: once the cumulative auto-approved write operations would exceed it, autopilot pauses, records the pause in the conversation ledger, and the plan waits for explicit human approval. The master switch is a single kill switch that stops all auto-approval on the next turn. ## Authoring Policy Policy is the densest control-plane object to write by hand: a decision rests on grant, binding, and confinement data that must parse as valid Rego and satisfy the platform decision contract. The Operator includes a dedicated policy author for exactly this. Describe the access you want - which application owns a resource, which roles hold which scopes, how to confine a label - and it drafts the matching [data documents](/v1.0/concepts/policy/), explains each one, and reports the least-privilege posture, the risks it detected, ready-to-run simulations, and activation readiness. Every draft is validated and previewed against the same contract the platform enforces, so a document the Operator emits is already contract-valid; if it cannot produce a valid document it fails closed rather than returning broken Rego. A draft is not a change. Turning one into a policy runs the ordinary [governed lifecycle](#the-governed-lifecycle) - you review the proposed create, approve it, and the create is re-validated on apply and attributed to you. Policies authored this way carry provenance marking them AI-assisted, and their later version, simulation, and activation steps stay under the same review and audit as any other policy work. ## Natural-Language Model Endpoint Turning words into a plan requires a model endpoint supplied by the operator. Open **Settings → AI Operator → Models** to add an OpenAI chat-completions-compatible endpoint, model IDs, optional context window, and key placement. The API key is accepted only when the model endpoint is created or rotated, sealed into a credential Provider in the reserved `caracal.sys` Zone, and never returned to the browser. Operator model calls use the governed Gateway route for that Resource. The settings page can edit model endpoint metadata, rotate the sealed key, delete a model endpoint, and run a real connectivity check. Multiple model endpoints and models participate in failover. Each logical completion gives its current model endpoint one SDK-managed retry for transient failures, including rate limits and server errors; terminal failures move directly to the next endpoint, and the endpoint timeout bounds the retry as well as the initial request. Direct API-process `API_OPERATOR_AI_*` configuration also remains implemented, but the packaged-runtime workflow is console management; see [Configure Service Environment](/v1.0/operations/env-vars/#api-operator-and-control) for that narrower path. When an endpoint fails, the API process temporarily moves it behind endpoints without a recent failure. It remains available as a fallback, a later success restores it immediately, and its configured priority returns automatically after the recovery window. This ordering memory is local to each API process and resets on restart. ### Provider Health Signals Caracal records the outcome of model requests the Operator already makes. For every configured provider, `GET /v1/operator/ai/status` includes `last_ok_at`, `last_error_at`, and a bounded `last_error_class`. A successful request advances `last_ok_at` but deliberately preserves the previous failure and its timestamp, so operators can see that a provider recovered without losing the recent incident. A provider with null timestamps has no recorded observation in Redis; null does not assert that the endpoint is healthy or unhealthy. The error classes are `auth_failed`, `rate_limited`, `timeout`, `unreachable`, `endpoint_error`, `config_error`, `invalid_response`, `stream_interrupted`, and `unknown_error`. Caracal stores only the provider ID, timestamps, and this bounded class in Redis. It never stores prompts, responses, SDK error messages, URLs, model IDs, or credentials in the health record. The same observations are exposed as `caracal_operator_ai_provider_last_success_timestamp_seconds` and `caracal_operator_ai_provider_last_failure_timestamp_seconds` gauges on `/metrics`. These are passive and traffic-dependent: an old success timestamp may only mean no request has used that provider recently. The **Test connection** action remains an explicit real completion and therefore consumes provider quota. Caracal does not run background model probes, and `/ready` never calls or reads an AI provider; a provider outage cannot restart an otherwise healthy control plane. ## Where to Use It Open **Caracal Operator** from the console utility rail or command palette. For the workspace and model-endpoint management paths, see [Caracal Operator](/v1.0/runtime-console/console/#caracal-operator) in the web console reference. ## Common Mistakes * Ask mode is read-only; it cannot produce or apply a change plan. * Approval of one plan does not widen the Operator Application's capability grant. * Autopilot changes who satisfies the plan Approval; it does not bypass validation, Zone isolation, or audit. ## Related Pages * [Zones](/v1.0/concepts/zone/) for the system zone the Operator self-governs. * [Policies and Policy Sets](/v1.0/concepts/policy/) for the data documents a plan can author. * [Audit and Request Traces](/v1.0/concepts/audit-ledger/) for the trail every applied change leaves. --- # Operate Caracal # URL: https://docs.caracal.run/v1.0/operations/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations.md # Type: landing # Concepts: # Requires: --- Use this section when you own a Caracal deployment. These runbooks describe behavior present in this repository; they do not promise an availability level, managed service, or compliance outcome. ## Choose a Runbook The runbooks follow the deployment lifecycle. Work down the phases for a new deployment; jump straight to a phase for day-to-day operations. **Deploy** - pick a substrate and stand the stack up: | Need | Runbook | | --- | --- | | Choose a substrate | [Choose a Deployment Profile](/v1.0/operations/deployment-profiles/) | | Review measured performance, sizing, and failure results | [Performance and Scalability](/v1.0/operations/performance-benchmarks/) | | Run one Docker host | [Deploy with Docker Compose](/v1.0/operations/docker-compose/) | | Deploy the repository chart | [Deploy with Helm](/v1.0/operations/kubernetes-helm/) | | Provision chart or VM bootstrap declaratively | [Provision with OpenTofu](/v1.0/operations/opentofu/) | | Map managed cloud services to the chart | [Choose a Cloud Profile](/v1.0/operations/cloud-native-profiles/) and [Deploy on Managed Kubernetes](/v1.0/operations/cloud-reference-deployments/) | | Hand a reviewed deployment package to another team | [Package an Install Kit](/v1.0/operations/install-kit/) | **Configure and secure** - before real credentials or traffic: | Need | Runbook | | --- | --- | | Set service variables and secrets | [Configure Service Environment](/v1.0/operations/env-vars/) | | Choose where sealed credentials are stored | [Configure Secret Backends](/v1.0/operations/secret-backends/) | | Pass the pre-production hardening checklist | [Harden Production](/v1.0/operations/tls-hardening/) | | Rotate keys, HMACs, and service tokens | [Rotate Keys and Secrets](/v1.0/operations/key-management/) | | Operate the durable stores | [Operate PostgreSQL](/v1.0/operations/postgres/) and [Operate Redis Streams](/v1.0/operations/redis/) | | Size and scale services | [Scale Capacity](/v1.0/operations/scale-capacity/) | **Observe** - know the deployment is healthy before users tell you it is not: | Need | Runbook | | --- | --- | | Wire health, readiness, and metrics | [Monitor Health and Metrics](/v1.0/operations/observability/) | | Alert on measured thresholds | [Configure Alerts](/v1.0/operations/alerts/) | | Diagnose a failed request | [Troubleshoot by Symptom](/v1.0/operations/troubleshooting/) | | Diagnose unhealthy infrastructure | [Debug Infrastructure Issues](/v1.0/operations/debugging/) | **Recover** - when something is wrong or before it can be: | Need | Runbook | | --- | --- | | Recover a known failure | [Recover from Failures](/v1.0/operations/failure-modes/) | | Prove recovery works before you need it | [Run Failure Drills](/v1.0/operations/failure-drills/) | | Back up or restore Compose state | [Back Up and Retain Data](/v1.0/operations/backup-retention/) | | Handle a security or availability incident | [Respond to Incidents](/v1.0/operations/incident-response/) | **Change** - roll out versions, policy, and responsibility: | Need | Runbook | | --- | --- | | Change versions | [Upgrade Caracal](/v1.0/operations/upgrade/) | | Roll out infrastructure changes | [Plan a Platform Rollout](/v1.0/operations/platform-rollout-kit/) | | Activate policy changes safely | [Deploy Policy Changes](/v1.0/operations/policy-deployment/) | | Export audit evidence for compliance tooling | [Export Audit Evidence](/v1.0/operations/compliance-audit-integration/) | | Transfer operational ownership | [Hand Off to Platform Teams](/v1.0/operations/platform-team-handoff/) | ## Operating Invariants * Postgres is the durable system of record. Redis carries streams and correctness-critical revocation state. * `dev` is local-development posture. `rc` and `stable` enforce published-mode configuration; `rc` is not a production stability claim. * Compose publishes service ports on loopback. Expose them only through an operator-owned TLS proxy. * Helm and OpenTofu are deployment assets, not a managed Kubernetes, HA, backup, or SLO service. * Runtime lifecycle belongs to `caracal up`, `down`, `status`, `upgrade`, and `purge`. Product administration belongs to the web console, Admin SDK, or Control API. ## Baseline Verification For Compose, run `caracal status --ready`. For Kubernetes, inspect Jobs and pod readiness, then test required service endpoints. `/health` proves liveness; only `/ready` is a traffic gate. ## Recovery Boundary Before a risky change, retain the current version, configuration, data backup, and secrets backup. Database migrations are forward-only; application rollback is safe only when the older version accepts the migrated schema. ## Next Step Choose the substrate in [Choose a Deployment Profile](/v1.0/operations/deployment-profiles/). --- # Choose a Deployment Profile # URL: https://docs.caracal.run/v1.0/operations/deployment-profiles/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/deployment-profiles.md # Type: reference # Concepts: # Requires: --- Choose a profile from requirements you can verify. Caracal ships deployment mechanics; it does not certify capacity, high availability, multi-region operation, or a cloud service. ## Decision | Requirement | Use | Do not infer | | --- | --- | --- | | Local development from source | `caracal up` with development Compose | Production hardening | | One Docker host with bundled stores | Installed runtime and packaged Compose | Host redundancy or zero downtime | | Kubernetes 1.30+ with operator dependencies | Helm chart | A tested SLO or managed stores | | A managed runtime with no cluster to operate | `infra/containerPlatform` render targets | Network policy, disruption budgets, or container hardening | | Declarative chart installation | `caracalStack` OpenTofu module | Resources beyond namespace, optional Secret, and Helm release | | Provider-neutral VM bootstrap | `caracalHost` OpenTofu module | VM, firewall, TLS, backup, or monitoring creation | ## Prerequisites Define ingress, recovery objectives, storage ownership, secret delivery, monitoring, and maintenance policy. If availability matters, prove it in your environment; replicas, PDBs, HPAs, and atomic upgrades are mechanisms, not guarantees. ## Choose Public Hostnames Decide these before provisioning anything. The STS origin becomes the `iss` claim in every mandate and is configured in every resource verifier and SDK that validates one, so changing it later invalidates that configuration everywhere, silently. Choose it once and keep it for the life of the deployment. Which shape is right depends on whether the domain belongs to Caracal or to you. | Domain | Shape | Example | | --- | --- | --- | | Dedicated to this deployment | Flat service names | `sts.caracal.example` | | Your organisation's existing domain | Nested under a product label | `sts.caracal.example.com` | Nest on an organisation domain because `api`, `console`, and `gateway` are almost certainly already in use there. A product label keeps every Caracal name inside one delegable subtree, so a platform team can hand out `caracal.example.com` once instead of reviewing four records. Nest deeper only for environment or region, such as `sts.caracal.staging.example.com`. | Service | Public | Notes | | --- | --- | --- | | Web console | Yes | The only origin that holds session cookies | | STS | Yes | Resource verifiers and SDKs outside the deployment resolve the issuer | | Gateway | Yes | Only when clients outside the network call protected resources | | API | Optional | Only when automation outside the network uses the Admin API | | Audit, Coordinator | Never | Internal by design | Two consequences worth planning for. A wildcard certificate covers one label, so `*.caracal.example.com` covers `sts.caracal.example.com` but not `sts.caracal.staging.example.com`; each extra level needs its own certificate. And the console must set host-only cookies: a cookie scoped to the parent domain would also be sent to the STS and Gateway origins. ## Safe Procedure 1. Use `dev` only on a local development host. 2. Pin a release and use `stable` for production evaluation. 3. Keep Compose ports loopback-bound; add an operator-owned TLS proxy for remote access. 4. For Helm, provide Postgres, Redis, runtime Secret, ingress, and network egress explicitly. 5. Establish backup, restore, metrics, alerts, and incident ownership before production traffic. ## Verify Confirm assets render, secrets resolve, migrations finish, and `/ready` passes. Run a canary token exchange and Gateway request, then locate its audit evidence. ## Rollback or Recovery Keep the prior release and values. Restore data only from a tested backup and restore secrets separately. Helm rollback never reverses database migrations. ## Next Step Use [Deploy with Docker Compose](/v1.0/operations/docker-compose/), [Deploy with Helm](/v1.0/operations/kubernetes-helm/), or [Deploy on a Managed Container Platform](/v1.0/operations/managed-container-platforms/). --- # Deploy with Docker Compose # URL: https://docs.caracal.run/v1.0/operations/docker-compose/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/docker-compose.md # Type: workflow # Concepts: # Requires: --- Use Compose for local development or one Docker host. It is not a multi-host or HA deployment. ## Prerequisites * Docker with Compose support. * For repository development, installed workspace dependencies. * For a released runtime, an installed Caracal binary and writable `CARACAL_HOME`. * Host backup and TLS exposure plans before production use. ## Procedure ```bash caracal up caracal status --ready ``` Development builds `infra/docker/docker-compose.yml`; installed `rc` and `stable` runtimes use the embedded release topology. API `3000`, STS `8080`, Gateway `8081`, Audit `9090`, Coordinator `4000`, and web `3001` bind to `127.0.0.1`. The web container listens internally on `3002`. `caracal up` creates missing managed secret files and preserves non-empty values. Keep `$CARACAL_HOME/secrets` or `CARACAL_SECRETS_DIR` outside source and agent workspaces. Put overrides in `$CARACAL_HOME/caracal.env`; do not edit the installed compose file. ## Verify `caracal status` checks `/health`; `caracal status --ready` checks dependencies. From a checkout, `bash infra/scripts/smokeTest.sh` probes the five core services on loopback. Open `http://localhost:3001` only after readiness passes. ## Stop and Recover Use `caracal down` to stop services while retaining volumes. :::danger[Data loss] `caracal down -v` and purge operations can remove state. Do not use them until Postgres, Redis, replay state, and separately stored secrets have recoverable backups. ::: If startup fails, preserve volumes and secrets, inspect `caracal status --json` and container logs, then use [Debug Infrastructure Issues](/v1.0/operations/debugging/). ## Next Step Review [Configure Service Environment](/v1.0/operations/env-vars/) before exposing an endpoint. --- # Deploy with Helm # URL: https://docs.caracal.run/v1.0/operations/kubernetes-helm/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/kubernetes-helm.md # Type: workflow # Concepts: # Requires: --- Use the chart when you already operate Kubernetes, Postgres, Redis, secrets, ingress, and monitoring. It does not provision a cluster or managed dependencies and makes no availability claim. ## Prerequisites * Kubernetes `>=1.30`, Helm, and namespace privileges. * A pinned chart version or reviewed checkout. * External Postgres and Redis for `stable` mode. * A complete runtime Secret; production values set `secrets.create=false`. * Explicit DNS, ingress-controller, dependency, provider, and object-store network rules. ## Safe Procedure 1. Create an environment-owned values file from `values.production.yaml`. 2. Set the image tag, service URLs, `services.web.publicUrl`, storage hosts, Secret name, and only required ingress. 3. Render before apply: ```bash helm lint infra/helm/caracal helm template caracal infra/helm/caracal --namespace caracal --values > caracal.rendered.yaml ``` 4. Review Secret references, NetworkPolicies, Ingresses, migration Jobs, image tags, PVCs, and security contexts. 5. Install: ```bash helm upgrade --install caracal infra/helm/caracal --namespace caracal --create-namespace --atomic --wait --wait-for-jobs --values ``` STS and Gateway render as StatefulSets when replay persistence is enabled. PDBs, HPAs, ServiceMonitor, and PrometheusRule require compatible controllers and do not prove HA. ## Verify Check migration Jobs, pods, services, and rollout status. Confirm `/ready`, run a canary token exchange and Gateway call, and locate audit evidence. Verify the scraper authenticates with `METRICS_BEARER`. ## Rollback and Recovery ```bash helm -n caracal history caracal helm -n caracal rollback caracal ``` Rollback does not reverse Postgres migrations. Verify schema compatibility or roll forward. Preserve replay PVCs and the runtime Secret. ## Next Step Use [Choose a Cloud Profile](/v1.0/operations/cloud-native-profiles/) to integrate operator-owned dependencies. --- # Choose a Cloud Profile # URL: https://docs.caracal.run/v1.0/operations/cloud-native-profiles/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/cloud-native-profiles.md # Type: reference # Concepts: # Requires: --- Caracal does not ship provider resources or a managed-cloud service. Use this page to connect services you operate to standard Helm interfaces. ## Use Criteria Proceed only when the platform provides Kubernetes, TLS ingress, Postgres, Redis with Streams, secret projection, replay storage, and metrics collection. Provider product names are examples, not certifications. ## Dependency Contract | Dependency | Required behavior | | --- | --- | | Postgres | TLS connection, migration privileges, capacity, and operator-owned backup/restore | | Redis | Streams, authentication, persistence, `noeviction`, and observable pending entries | | Secrets | Complete runtime Secret; no plaintext in Git or plans | | Ingress | HTTPS, exact origins and paths, explicit NetworkPolicy ingress | | Monitoring | Authenticated metrics scraping and actionable alert routing | ## Procedure 1. Provision dependencies with platform tooling. 2. Project runtime credentials into the namespace. 3. Set external hosts and internal URLs in environment values. 4. Allow only required DNS, storage, identity-provider, upstream, and export egress. 5. Render and review the chart before installation. ## Verify and Recover Confirm secret synchronization, TLS, network reachability, migrations, `/ready`, metrics auth, canary authorization, revocation, and audit capture. Test provider-native backup restore. Retain the prior values and release; never rely on provider marketing or chart defaults for recovery behavior. ## Next Step Use [Deploy on Managed Kubernetes](/v1.0/operations/cloud-reference-deployments/) for the repository examples and their limits. --- # Deploy on Managed Kubernetes # URL: https://docs.caracal.run/v1.0/operations/cloud-reference-deployments/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/cloud-reference-deployments.md # Type: workflow # Concepts: # Requires: --- Files under `infra/helm/caracal/examples` demonstrate External Secrets Operator and Helm interfaces. They are templates, not complete or certified AWS, Azure, or Google Cloud deployments. ## Prerequisites Operate Kubernetes 1.30+, Postgres, Redis with Streams, External Secrets Operator, a TLS issuer, ingress controller, and Prometheus Operator when using chart monitoring resources. ## Procedure 1. Adapt one `external-secrets/secretstore-*.yaml`; replace every identity and store reference. 2. Store every value referenced by `externalsecret-runtime.yaml` in the provider secret manager. 3. Apply the adapted SecretStore and ExternalSecret. 4. Copy `values.cloud-managed.yaml`; replace hosts, domains, storage class, ingress class, issuer, and egress selectors. 5. Render. Reject plaintext Secret data, default hosts, broad ingress, and unintended public API, Gateway, or STS endpoints. 6. Install the reviewed overlay. ## Verify Confirm Secret synchronization, migration Jobs, pod readiness, HTTPS origins, STS issuer, NetworkPolicy paths, metrics collection, alerts, and a canary protected call with audit evidence. ## Rollback and Recovery Retain the previous Helm revision and values. Roll back only when schema-compatible. Use provider-native storage recovery and restore runtime secrets separately; examples do not create or test backups. ## Next Step Apply [Harden Production](/v1.0/operations/tls-hardening/) before admitting traffic. --- # Provision with OpenTofu # URL: https://docs.caracal.run/v1.0/operations/opentofu/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/opentofu.md # Type: workflow # Concepts: # Requires: --- `caracalHost` renders cloud-init; it does not create a VM. `caracalStack` creates a namespace, optionally a Secret, and a Helm release; it does not create a cluster, Postgres, Redis, ingress controller, or secret manager. ## Prerequisites Use OpenTofu `>=1.8`. Kubernetes also requires version `>=1.30`, kubeconfig access, external dependencies, and a complete runtime Secret. ## Kubernetes Procedure 1. Copy the production tfvars example outside source control. 2. Pin `chartVersion`, set database and Redis hosts, and deliver `caracal-runtime` before apply. 3. Add reviewed chart overlays through `extraValues`. 4. Run: ```bash cd infra/tofu/envs/production tofu init tofu plan -out caracal.plan tofu apply caracal.plan ``` The module waits for Jobs and uses atomic Helm behavior by default. Remote state and locking are operator responsibilities; never put plaintext runtime secrets in variables or state. ## VM Procedure Attach `caracalHost.userData` or `userDataBase64` to an operator-owned VM. It installs Docker when absent, installs a pinned release, writes non-secret overrides, and starts Caracal. Runtime secrets are generated on the host. The packaged stack publishes every port on loopback, so a host without a proxy serves no external traffic. Set `tlsProxy` to terminate HTTPS in front of it: ```hcl tlsProxy = { email = "ops@example.com" routes = { "console.example.com" = "web" "sts.example.com" = "sts" } } ``` The console origin, the trusted-proxy flag, and the STS issuer are derived from these routes, so they cannot drift from the names the proxy terminates. Certificate issuance needs inbound 80 and 443 open in your cloud firewall, and every hostname must already resolve to the VM. `proxyImage` is pinned by digest; override it to track a different proxy. ## Verify and Recover Run `bash infra/tofu/scripts/validate.sh` in a checkout. After apply, verify Jobs, readiness, and audit evidence; on a VM run `caracal status --ready`. Use Helm revision rollback only when schema-compatible. Restore VM data and secrets separately before rebuilding. ## Next Step Read [Deploy with Helm](/v1.0/operations/kubernetes-helm/) or [Deploy with Docker Compose](/v1.0/operations/docker-compose/). --- # Package an Install Kit # URL: https://docs.caracal.run/v1.0/operations/install-kit/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/install-kit.md # Type: workflow # Concepts: # Requires: --- Use an install kit when another team needs a reviewed, reproducible handoff for a self-hosted Caracal deployment. The kit packages artifacts already implemented in this repository; it does not create a managed service, support entitlement, availability guarantee, or certification. ## Kit Contents | Artifact | Required content | | --- | --- | | Release record | Product version, image digests, chart version, checksums, and verification output. | | Deployment values | Environment-specific Compose or Helm inputs with no secret data. | | Secret inventory | Every required key, owning team, external location, delivery method, and rotation procedure. | | Network record | Public endpoints, TLS termination, private dependencies, ingress, egress, and firewall or NetworkPolicy rules. | | Runbooks | Install, readiness, canary, monitoring, backup, restore, upgrade, rollback or roll-forward, and incident response. | | Acceptance evidence | Rendered output, migration result, readiness, negative tests, audit trace, alerts, restore, and owner approval. | Keep customer values and all secret material outside the repository and outside the kit archive unless the receiving team's approved secret-delivery system encrypts and controls them. ## Procedure 1. Pin the release, chart, image digests, and required package versions. 2. Verify the release and retain the output. 3. Render the exact deployment configuration the receiving team will use. 4. Review public exposure, service identity, resource requests, persistence, secret references, and published mode. 5. Build the secret inventory without recording secret values. 6. Include the relevant operations and incident procedures, with named owners and escalation paths. 7. Install the kit in staging and run readiness, canary allow/deny/revoke, audit, alert, backup, and isolated restore checks. 8. Record unsupported or adopter-owned requirements explicitly. 9. Obtain platform and security acceptance for the immutable kit revision. ## Acceptance Criteria * A clean environment can reproduce the rendered deployment and artifact verification. * No plaintext secret appears in values, manifests, logs, or the kit archive. * The migration job and every required readiness endpoint pass. * A protected canary request produces authorization and action-result evidence. * A revoked Session is rejected at the chosen enforcement boundary. * Alert routing reaches the documented owner. * The data backup and separately protected key material restore successfully together. * The previous kit revision and recovery decision remain available. ## Recovery Keep every accepted kit immutable. If validation fails, return the candidate to review instead of patching the deployed values out of band. After a production change, issue a new kit revision with updated render, evidence, and recovery notes. ## Next Step Use [Plan a Platform Rollout](/v1.0/operations/platform-rollout-kit/) for deployment and [Hand Off to Platform Teams](/v1.0/operations/platform-team-handoff/) for ownership acceptance. --- # Configure Service Environment # URL: https://docs.caracal.run/v1.0/operations/env-vars/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/env-vars.md # Type: reference # Concepts: # Requires: --- Use this page before first start or after a release adds configuration. Service variables and workload SDK variables are separate concerns. ## The Operator Env File Every `caracal` command - `up`, `web`, and `run` - loads one operator env file at startup. It holds everything you enter by hand: the web console's sign-in settings, without which `caracal web` cannot authenticate anyone; the runtime launcher's workload identity and secret; the STS URL; and any service override. A variable already set in the process environment always wins over the file, and for a supported secret `NAME_FILE` is resolved before `NAME`. On an installed host the file is `$CARACAL_HOME/caracal.env`, created `0600` on the first `caracal up`. It lives outside your project, so open it in an editor - it is plain `KEY=VALUE`: ```bash ${EDITOR:-nano} ~/.local/share/caracal/caracal.env # Linux ${EDITOR:-nano} ~/Library/Application\ Support/caracal/caracal.env # macOS notepad $env:LOCALAPPDATA\caracal\caracal.env # Windows (PowerShell) ``` Set `CARACAL_ENV_FILE` to point at a different path. In development the file is `.env` at the repository root - copy the committed `.env.example` - and `infra/docker/dev.env` holds the Compose defaults. Apply a change by rerunning the affected command. Use `CARACAL_MODE=dev` only locally. `rc` and `stable` share fail-closed configuration checks; `rc` denotes release maturity, not weaker security. ## Fixed Endpoints Every service binds a fixed local port with `/health`, `/ready`, and `/metrics` endpoints; the canonical map is in [Monitor Health and Metrics](/v1.0/operations/observability/#endpoint-map). Compose publishes the web console on host port `3001` (container `3002`) and other services on their service ports, all on loopback; bare port numbers are listed in [Defaults and Limits](/v1.0/reference/defaults-and-limits/#ports). ## Required Secret Classes Published deployments require storage URLs, admin and Coordinator credentials, `SECRET_STORE_KEK`, `AUDIT_HMAC_KEY`, `STREAMS_HMAC_KEY`, `IDEMPOTENCY_HMAC_KEY`, `GATEWAY_STS_HMAC_KEY`, and `METRICS_BEARER` where consumed. Use the release-matched Compose/chart secret mapping rather than guessing `_FILE` support. ## Web Console BFF | Variable | Purpose | | --- | --- | | `CARACAL_AUTH_URL` | Public auth/Web BFF URL; packaged local default is `http://localhost:3001`. | | `CARACAL_OPEN_REGISTRATION` | Opens registration beyond the host allowlist posture when explicitly enabled. | | `CARACAL_OPERATOR_ALLOWLIST` | Comma-separated operator emails or `@domain` suffixes admitted declaratively; entries managed with `caracal allowlist` override these per address. | | `CARACAL_PASSWORD_SIGNUP` | Enables email/password signup; published mode also requires working verification mail. | | `CARACAL_SMTP_URL`, `CARACAL_SMTP_FROM` | SMTP transport and sender for verification and reset messages. | | `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET` | Enables Google sign-in when both values are present. | | `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET` | Enables GitHub sign-in when both values are present. | | `CARACAL_AUTH_TRUST_PROXY` | Trusts forwarded client/protocol headers only when the deployment has an approved proxy boundary. | Provider credentials and SMTP URLs support their implemented `_FILE` variants; the packaged Compose stack mounts `$CARACAL_HOME/secrets/console` at `/run/caracalConsoleSecrets` so a client secret can be a file on the host instead of an environment value. Registration admission and authentication method are separate: an allowlisted email still needs one configured sign-in method. In published modes the web service refuses to start with no method at all - configure Google, GitHub, or SMTP before first start. ## STS Egress Trust | Variable | Purpose | | --- | --- | | `CARACAL_PRIVATE_EGRESS_HOSTS` | Comma-separated hosts on private address ranges that Federated user issuer JWKS, provider token endpoints, and notification sink deliveries may reach; empty blocks private-range egress. | | `CARACAL_TLS_EXTRA_CA_FILE` | PEM bundle appended to system trust for STS egress TLS, so internal-PKI Federated user issuers and provider endpoints verify without replacing public trust. | The packaged Compose stack wires `CARACAL_TLS_EXTRA_CA_FILE` for you: drop a PEM bundle at `$CARACAL_HOME/ca/extra-ca.pem` and restart with `caracal up`. An absent bundle leaves system trust untouched; an unreadable or unparseable bundle fails STS egress closed rather than silently ignoring the stated trust intent. ## STS Capacity | Variable | Purpose | | --- | --- | | `STS_MINT_RATE_LIMIT_PER_MIN` | Deployment ceiling for mandate mints per minute for each zone, resource, and acting application; default 1000. Set it on the STS and API services together. The web console's Preferences page manages a working limit below this ceiling. | | `STS_SECRET_VERIFY_CONCURRENCY` | Concurrent Argon2id credential verifications; default 2. Each in-flight verification allocates 64 MB, and verified credentials are cached, so this bounds cold-start bursts, not steady-state throughput. | | `CARACAL_STS_CPU_LIMIT`, `CARACAL_STS_MEM_LIMIT` | Packaged Compose STS container resources; defaults 2.0 CPUs and 1G. Size per [Performance and Scalability](/v1.0/operations/performance-benchmarks/). | ## API Operator and Control | Variable | Purpose | | --- | --- | | `API_OPERATOR_ENABLED` | Registers the Operator capability and routes; default is enabled. | | `API_OPERATOR_ALLOWED_CAPABILITIES` | Optional comma-separated ceiling over executable Operator capabilities. | | `API_OPERATOR_AUTOPILOT_ENABLED` | Master switch that permits conversation-level automatic plan Approval. | | `API_OPERATOR_AUTOPILOT_WRITE_BUDGET` | Optional cumulative write-operation budget for an autopilot conversation. | | `API_OPERATOR_AI_MAX_OUTPUT_TOKENS` | Per-call model output ceiling. | | `API_OPERATOR_AI_MAX_CALLS_PER_TURN` | Per-turn model-call budget. | | `CARACAL_CONTROL_ENABLED`, `CONTROL_GATE_FILE` | Build-time Control mount and runtime invoke gate. | The API process also accepts `API_OPERATOR_AI_PROVIDERS` plus per-ID `API_OPERATOR_AI__BASE_URL`, `_MODEL`, optional `_API_KEY`, `_TIMEOUT_MS`, and `_CONTEXT_WINDOW`. IDs are tried in listed order before console-managed model endpoints. Those variables are a direct API-process configuration path; the installed-runtime Compose and Helm surfaces do not forward model-endpoint-specific entries. For the packaged workflow, configure model endpoints under **Settings → AI Operator → Models**, where keys are sealed into `caracal.sys` and never returned. ## Safe Procedure 1. Start from the shipped env template or chart values for the same release. 2. Put non-secret overrides in `caracal.env` or environment-owned Helm values. 3. Put secrets in owner-only files or a Kubernetes Secret projection. 4. Set external origins and issuer to exact HTTPS URLs. 5. Enable Control, public ingress, private egress exceptions, password signup, or proxy trust only when required. 6. Restart changed services and gate on `/ready`. ## Verify and Recover Run `caracal status --ready` or inspect Kubernetes readiness. Published metrics must reject a missing bearer and accept the configured one. If startup fails, revert the last override; do not replace generated secrets merely to clear validation errors. ## Next Step Choose a [Configure Secret Backends](/v1.0/operations/secret-backends/) and apply [Harden Production](/v1.0/operations/tls-hardening/). --- # Harden Production # URL: https://docs.caracal.run/v1.0/operations/tls-hardening/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/tls-hardening.md # Type: workflow # Concepts: # Requires: --- Use this checklist before a `stable` deployment receives real credentials or traffic. ## Prerequisites Own the TLS endpoint, DNS, runtime Secret, ingress or proxy, dependency firewall rules, and monitoring path. Caracal does not configure host firewalls or issue Compose certificates. ## Procedure 1. Set `CARACAL_MODE=stable`; never expose `dev`. 2. Keep Postgres, Redis, API, Coordinator, Audit, and Control private unless explicitly required. 3. Expose the same-origin web BFF through HTTPS and set its exact public origin. Trust proxy headers only from a controlled direct proxy. 4. Deliver secrets through files/projections and deny agents access to runtime secrets and Docker socket. 5. Retain non-root, dropped-capability, read-only-filesystem, and `no-new-privileges` settings. 6. Permit only required ingress and egress. Pin Gateway destinations with `UPSTREAM_HOST_ALLOWLIST` when appropriate. 7. Require authenticated metrics and route critical audit/revocation alerts. Gateway blocks dangerous address classes and does not follow redirects, but operator-provisioned private upstreams are intentionally supported. Network policy remains the outer boundary. ## Verify Test HTTPS and issuer/JWKS identity, denied direct storage access, rejected cross-origin browser writes, rejected unauthenticated metrics, blocked upstreams, and revoked sessions denied before dispatch. ## Rollback or Recovery Revert the narrowest policy change while keeping services private. Never recover by switching to `dev`, disabling revocation safety, publishing storage, or placing secrets inline. ## Next Step Complete [Rotate Keys and Secrets](/v1.0/operations/key-management/) and [Configure Alerts](/v1.0/operations/alerts/). --- # Rotate Keys and Secrets # URL: https://docs.caracal.run/v1.0/operations/key-management/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/key-management.md # Type: workflow # Concepts: # Requires: --- Use this runbook on a rotation schedule, after suspected exposure, or when a person with secret access leaves. Rotate one class at a time. Inventory the KEK, zone signing keys, integrity HMACs, idempotency HMAC, Gateway-STS HMAC, service tokens, storage credentials, and web auth secret. ## Prerequisites Keep current data and secret backups, owners for every producer/consumer, canary requests, and a documented overlap window where supported. ## Safe Procedure 1. Generate replacement material cryptographically. 2. Use `SECRET_STORE_KEK_PREVIOUS` for envelope re-sealing and `IDEMPOTENCY_HMAC_KEY_PREVIOUS` for receipt overlap. 3. Update every producer and consumer of a shared HMAC before retirement. 4. Roll affected services and wait for readiness. 5. Check audit, stream, replay, revocation, JWKS, and canary signals. 6. Retire old material only after applicable token, cache, message, replay, or receipt windows. Keep the old idempotency key for at least `IDEMPOTENCY_RETENTION_SECONDS`. ## Verify Confirm retired credentials fail, envelopes decrypt, JWKS is expected, new stream/audit messages verify, pending entries drain, and Gateway-to-STS exchange succeeds. ## Recovery Restore the retiring value before deletion if verification fails. Preserve logs, DLQ, and replay state. Never regenerate a lost KEK; restore its backup. ## Next Step Verify [PostgreSQL](/v1.0/operations/postgres/) and [Redis Streams](/v1.0/operations/redis/). --- # Operate PostgreSQL # URL: https://docs.caracal.run/v1.0/operations/postgres/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/postgres.md # Type: workflow # Concepts: # Requires: --- Postgres holds product, authority, policy, session, delegation, outbox, key-reference, and audit state. ## Prerequisites Use release-matched migrations, an administrative migration role, service-specific credentials, and a tested backup. Do not grant service roles ownership or `BYPASSRLS`. ## Procedure Compose and Helm run `infra/postgres/scripts/migrate.sh` in a one-shot workload. Files apply in order, one transaction each, under an advisory lock. Production operation is forward-only. In a disposable database, run: ```bash bash infra/postgres/scripts/validateMigrations.sh ``` Monitor pools, long transactions, timeouts, storage, audit partitions, and outbox age. Increase database capacity before aggregate service pools. ## Verify Confirm migrations, `schema_migrations`, service roles, RLS, append-only audit permissions, and dependent readiness. ## Recovery Restore connectivity/capacity, then verify migrations, pools, outboxes, and readiness. For data loss, use complete restore; do not reconstruct authority or audit rows manually. :::danger[Schema and evidence damage] Do not run down migrations, drop databases, disable RLS, or mutate audit rows as an outage shortcut. ::: ## Next Step Verify event delivery with [Operate Redis Streams](/v1.0/operations/redis/). --- # Operate Redis Streams # URL: https://docs.caracal.run/v1.0/operations/redis/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/redis.md # Type: workflow # Concepts: # Requires: --- Redis carries audit, invalidation, revocation, agent, invocation, and delegation streams. It is correctness-critical, not an evictable cache. ## Prerequisites Use Streams, authentication, persistence, `maxmemory-policy noeviction`, and enough memory. The bundled image uses AOF with `appendfsync everysec`; validate external services separately. ## Procedure Provisioning is automatic in the bundled stack. From a checkout: ```bash bash infra/redis/provision-streams.sh bash infra/redis/scripts/verify.sh ``` Provide Redis host, port, and password file when needed. Monitor stream length, groups, pending entries, reclaim/retry, DLQ, memory, persistence, and latency. ## Verify Confirm expected streams/groups, idempotent provisioning, `noeviction`, persistence, and policy/revocation/audit delivery. ## Recovery Restore Redis, verify groups, then drain consumers and replay. Reconcile Postgres outboxes and replay directories. Never delete pending entries or reset groups until durable handling is proven. ## Next Step Use [Scale Capacity](/v1.0/operations/scale-capacity/) when lag or memory grows. --- # Scale Capacity # URL: https://docs.caracal.run/v1.0/operations/scale-capacity/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/scale-capacity.md # Type: workflow # Concepts: # Requires: --- Caracal publishes signals and Helm controls, but no universal throughput, latency, or availability figure. Establish a baseline with your workload. For measured reference results from the v0.2.0 validation under stated test conditions, see [Performance and Scalability](/v1.0/operations/performance-benchmarks/). ## Decision Criteria | Signal | Investigate first | | --- | --- | | Database pool pressure | Slow queries, transactions, connections, I/O | | Audit lag, DLQ, replay age | Audit write path, Redis, Postgres, replay storage | | Gateway STS circuit | STS readiness and exchange latency | | Revocation lag | Redis latency, pending entries, snapshot refresh | | Readiness flapping | CPU, memory, dependencies, probes | ## Prerequisites Collect authenticated metrics, latency, dependency metrics, logs, and repeatable load. Know storage connection/memory ceilings. ## Safe Procedure 1. Record baseline and first saturated resource. 2. Correct dependencies or slow queries before replicas. 3. Change one limit or replica count at a time. 4. Keep aggregate pools within database capacity. 5. Repeat load and compare readiness, errors, lag, DLQ, replay, and revocation. 6. Set environment thresholds from measured bounds. Chart replica/HPA defaults are not capacity recommendations. ## Rollback Restore the previous value if pressure or errors increase and retain both measurements. ## Next Step Build gates in [Monitor Health and Metrics](/v1.0/operations/observability/). --- # Monitor Health and Metrics # URL: https://docs.caracal.run/v1.0/operations/observability/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/observability.md # Type: workflow # Concepts: # Requires: --- Use `/health` for liveness and `/ready` for traffic and rollout gates. ## Endpoint Map | Service | Port | Health | Readiness | Metrics | | --- | ---: | --- | --- | --- | | API | 3000 | `/health` | `/ready` | `/metrics` | | STS | 8080 | `/health` | `/ready` | `/metrics`, `/metrics.json` | | Gateway | 8081 | `/health` | `/ready` | `/metrics`, `/metrics.json` | | Audit | 9090 | `/health` | `/ready` | `/metrics`, `/metrics.json` | | Coordinator | 4000 | `/health` | `/ready` | `/metrics` | Published metrics require `Authorization: Bearer `. Keep them private even when authenticated. ## Procedure 1. Scrape every enabled service with the metrics secret. 2. Track readiness, DB pool, policy age/compile errors, provider refresh, audit lag/DLQ/tamper/replay, outbox, STS circuit, and revocation freshness. 3. Correlate logs and audit by request ID. 4. Gate deployment on readiness plus canary exchange, Gateway call, and audit lookup. For Compose run `caracal status --ready`; from a checkout use `bash infra/scripts/smokeTest.sh`. For Helm, verify Jobs, workloads, ServiceMonitor discovery, and rules. ## Recovery If health passes but readiness fails, preserve the readiness reason and inspect its dependency/safety latch. Restore the cause and drain backlogs. Preserve evidence before replacing an `audit_evidence_lost` process. ## Next Step Route signals with [Configure Alerts](/v1.0/operations/alerts/). --- # Configure Alerts # URL: https://docs.caracal.run/v1.0/operations/alerts/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/alerts.md # Type: workflow # Concepts: # Requires: --- Configure alerts after [monitoring](/v1.0/operations/observability/) is scraping and before production traffic, so failures page someone instead of waiting to be noticed. Use the chart PrometheusRule only with a compatible rule importer. Tune thresholds from measured behavior. ## Required Routes | Alerts | Route and first action | | --- | --- | | Audit tamper/evidence loss | Security incident; stop risky traffic and preserve evidence | | Revocation freshness/reload/lag | Access-safety incident; remove affected Gateway from traffic | | Gateway STS circuit/exchange | Restore STS exchange before protected traffic | | Audit DLQ/lag/replay or API outbox | Freeze changes; restore Redis/Postgres/Audit path | | Policy/provider alerts | Restore last known-good policy/provider behavior | | Pool saturation/readiness flapping | Platform on-call; diagnose capacity/dependencies | ## Procedure 1. Render PrometheusRule and inspect expressions, thresholds, `for` durations, and labels. 2. Confirm every metric is scraped. 3. Route critical safety alerts to staffed security/on-call and warnings to owners. 4. Link rules to incident or recovery runbooks. 5. Fire each rule safely in non-production and record detection time. ## Verify Confirm loading, firing, intended receiver, resolution, and valid runbook links. ## Rollback Revert only a noisy threshold/route, not collection. Keep audit, revocation, and policy safety alerts enabled. ## Alert Runbooks Each rule the chart ships sets a `runbook_url` that resolves to one of the sections below. When an alert fires, open its runbook, take the first action, and escalate on the routes above. ## CaracalSTSOPACompileErrors A policy bundle failed compilation, so STS fails closed for affected policy loads. Inspect STS logs and policy activation history, then restore the last known-good policy version. ## CaracalSTSPolicyBundleStale The active policy bundle is older than the configured threshold, so invalidation or PostgreSQL polling may be delayed. Check the Redis policy-invalidation stream and STS readiness. ## CaracalSTSProviderRefreshErrors Provider credential refresh coordination is reporting Redis lease or result errors. Agents using provider-backed grants may fail closed until Redis and STS recover. ## CaracalSTSProviderCircuitOpen A provider refresh circuit is rejecting attempts after repeated provider failures. Provider-backed resources fail closed until the provider or credential issue is corrected. ## CaracalAuditDLQNonEmpty Audit events are failing ingestion or verification. Inspect Audit logs, DLQ age, producer HMAC failures, and Redis stream health. ## CaracalAuditDLQGrowth The audit failure backlog is increasing and protected-action evidence may be delayed. Freeze risky rollouts and recover Audit, Redis, and PostgreSQL before continuing. ## CaracalAuditConsumerLagHigh Audit ingestion is behind the Redis stream. Scale Audit, check PostgreSQL latency, and confirm Redis memory is not constrained. ## CaracalGatewayAuditReplayBacklogOld Gateway has audit replay files waiting on disk beyond the configured threshold. Recover Redis and Audit, then confirm replay files drain before continuing risky rollouts. ## CaracalSTSAuditReplayBacklogOld STS has audit replay files waiting on disk beyond the configured threshold. Recover Redis and Audit, then confirm replay files drain before continuing risky rollouts. ## CaracalGatewayAuditEvidenceLost Gateway lost audit evidence after both Redis delivery and durable replay failed. Preserve service and storage evidence, stop risky traffic, and replace the affected pod only after recording the incident scope. ## CaracalSTSAuditEvidenceLost STS lost audit evidence after both Redis delivery and durable replay failed. Preserve evidence, stop token issuance through the affected pod, and replace it only after recording the incident scope. ## CaracalAuditTamperDetected Audit chain verification detected a mismatch, ordering break, or stored HMAC failure. Treat it as a security incident: preserve evidence and stop risky traffic. ## CaracalAPIOutboxDeadMessages Control-plane events exhausted delivery attempts and may not have reached Redis consumers. Recover Redis or the stream consumers, inspect API logs, and reconcile affected rows before continuing rollouts. ## CaracalAPIOutboxPendingOldest Control-plane events are delayed before Redis publication. Check Redis health, API outbox workers, database pool saturation, and stream memory pressure. ## CaracalGatewaySTSExchangeErrors Gateway cannot reliably exchange mandates with STS, so protected requests may fail before upstream dispatch. Check Gateway and STS readiness, route bindings, and service-to-service auth. ## CaracalGatewaySTSCircuitOpen Gateway is fast-failing STS exchanges after repeated STS-unavailable failures, so protected requests that need exchanged authority fail closed. Restore STS exchange before protected traffic. ## CaracalGatewayRevocationSnapshotStale Gateway cannot prove it has a fresh revocation baseline. Treat it as an access-safety incident and restore PostgreSQL/Redis snapshot loading before relying on protected routes. ## CaracalGatewayRevocationPropagationLag Revocation stream messages are reaching Gateway later than the configured safety window. Restore Redis consumer health and confirm pending entries are reclaimed. ## CaracalGatewayRevocationReloadErrors Gateway cannot refresh revocation state from PostgreSQL. Treat it as an access-safety incident until reloads succeed and snapshot freshness returns. ## CaracalPostgresPoolSaturation A service is holding most of its PostgreSQL pool. Inspect long-running queries, statement timeouts, and connection leaks before the pool exhausts and requests start failing. ## CaracalReadinessFlapping A pod is repeatedly toggling Ready and NotReady. Check dependency health (PostgreSQL and Redis), CPU throttling, OOM, and readiness probe timeouts. ## Next Step Start [Troubleshoot by Symptom](/v1.0/operations/troubleshooting/) or rehearse [Run Failure Drills](/v1.0/operations/failure-drills/). --- # Troubleshoot by Symptom # URL: https://docs.caracal.run/v1.0/operations/troubleshooting/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/troubleshooting.md # Type: workflow # Concepts: # Requires: --- Use this when an SDK, HTTP request, console action, or protected call failed. It identifies the failed surface; it does not repair infrastructure. ## Prerequisites Capture timestamp, request ID, zone, application, resource, operation, status, error code, and version. Redact credentials. ## Triage Procedure 1. Check readiness. If any service is NotReady, use [Debug Infrastructure Issues](/v1.0/operations/debugging/). 2. For client startup failure, verify explicit profile, endpoints, IDs, and credential file. 3. For `401`, verify credential type, issuer, expiry, and intended surface. 4. For STS `403`, inspect Grants, Resource/scopes, Policy set, Session, Delegation, and Approval. 5. For resource/Gateway `403`, verify mandate issuer, audience, scope, `X-Caracal-Resource`, binding, revocation, and verifier. 6. Search Audit by request ID. If absent, confirm the request reached enforcement, then inspect the audit path. ## Verification Repeat one safe request and confirm its expected status and audit Subject/resource/scopes/policy/result. ## Recovery Boundary Do not widen policy, bypass Gateway, disable revocation, or replace credentials until the surface is identified. ## Diagnostic Bundle Caracal exposes diagnosis through existing supported surfaces rather than a separate `doctor` command: | Evidence | Surface | | --- | --- | | Runtime health and readiness | `caracal status --json` and `caracal status --ready --json` | | Service, Zone, and provider checks | Web console **Diagnostics** | | Recent decisions and operational events | Web console **Audit** or the Admin API audit list | | One correlated authorization path | Request trace by request ID | Capture those outputs with the timestamp, version, Zone, and request ID. Redact credentials before attaching the bundle to an incident. ## Next Step Use [Debug Infrastructure Issues](/v1.0/operations/debugging/) or [Debug Authorization Decisions](/v1.0/guides/authorize-access/). --- # Debug Infrastructure Issues # URL: https://docs.caracal.run/v1.0/operations/debugging/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/debugging.md # Type: workflow # Concepts: # Requires: --- Use this after readiness, rollout, or dependency failure. It diagnoses infrastructure; recovery is separate. ## Prerequisites Preserve the first failing readiness response, deployment/config diff, workload status, logs, and alert timeline before restart. ## Diagnosis Procedure 1. Inspect Compose containers or Kubernetes pods, Jobs, events, and rollout. 2. Compare mode, Secret keys, URLs, ports, origins, and release pin with the release. 3. Test Postgres connectivity, migrations, pools, and outbox age. 4. Test Redis connectivity, `noeviction`, persistence, groups, and pending entries. 5. Read exact readiness reason and correlated logs. 6. Inspect policy age, STS circuit, revocation snapshot, audit DLQ/replay, and evidence-loss latch. For an Operator model failure, inspect `GET /v1/operator/ai/status` and compare each provider's `last_ok_at`, `last_error_at`, and `last_error_class`. The matching `/metrics` timestamp gauges make the transition alertable without attaching high-cardinality error messages. These signals come from real requests, so stale or null values are not active reachability checks. Use **Test connection** only when a fresh, quota-consuming completion is appropriate. Do not use `/ready` to diagnose a model endpoint: readiness intentionally checks platform dependencies without calling or reading an LLM provider. Start with `caracal status --json`, Compose status/logs, or Kubernetes get/describe/logs. ## Verification Diagnosis is complete when one dependency, value, ceiling, or safety invariant explains readiness with log/metric evidence. ## Recovery Boundary Do not delete volumes, reset streams, restore data, or clear evidence before selecting recovery. ## Next Step Apply [Recover from Failures](/v1.0/operations/failure-modes/). --- # Recover from Failures # URL: https://docs.caracal.run/v1.0/operations/failure-modes/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/failure-modes.md # Type: workflow # Concepts: # Requires: --- Use this only after diagnosis. Restore durable state, then services, then evidence and access-safety freshness. ## Recovery Order 1. Freeze rollouts and risky traffic. 2. Restore Postgres, then Redis. 3. Restore API, STS, Gateway, Audit, Coordinator, and web readiness. 4. Drain outboxes, pending entries, audit replay, and DLQ. 5. Prove policy and revocation freshness. 6. Run canary allow/deny and locate audit evidence. ## Failure Procedures | Failure | Recovery | | --- | --- | | Postgres | Restore connectivity/capacity, migrations and pools, then outboxes | | Redis | Restore persistence/`noeviction`, groups and pending entries, then replay | | STS | Restore stores, policy bundle, signing/KEK, JWKS, Gateway HMAC | | Gateway | Restore STS exchange, binding, revocation, egress, replay | | Audit | Restore stores/HMAC, drain replay/DLQ, check tamper state | | Coordinator | Restore stores, service token, outbox, workers | `audit_replay_unavailable` can clear after storage recovers. `audit_evidence_lost` is latched: preserve evidence, scope the interval, then replace the process as an incident action. ## Verification Readiness passes; queues are drained/understood; revoked sessions deny; current policy applies; audit records canaries. ## Rollback Return to preserved config/dependency snapshot if repair worsens state. Database restore is separate destructive recovery. ## Next Step Rehearse [Run Failure Drills](/v1.0/operations/failure-drills/). --- # Run Failure Drills # URL: https://docs.caracal.run/v1.0/operations/failure-drills/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/failure-drills.md # Type: workflow # Concepts: # Requires: --- Run only in isolated non-production with restorable data and secrets. ## Prerequisites Establish green readiness, understood backlogs, tested alert routing, recovery owner, stop condition, backup, and baseline canary/audit results. ## Procedure 1. Select one fault: block Redis/Postgres, isolate Gateway from STS, pause Audit, delay revocation, or activate invalid test policy. 2. Inject only that fault with a time limit. 3. Record readiness, traffic, alerts, logs, lag, and detection time. 4. Remove the fault. 5. Execute [Recover from Failures](/v1.0/operations/failure-modes/). 6. Record readiness, drain, freshness, canary, and audit recovery times. :::danger[Destructive drill] Audit tampering, restore, Redis replacement, volume deletion, and key loss require disposable infrastructure and explicit approval. ::: ## Pass Criteria Alert reaches the owner; services fail closed where expected; recovery follows the runbook; readiness and canary evidence return without unexplained queues. ## Recovery If a stop condition is exceeded, terminate injection, preserve evidence, restore last known-good, and open an incident. ## Next Step Update thresholds/runbooks, then validate [Back Up and Retain Data](/v1.0/operations/backup-retention/). --- # Back Up and Retain Data # URL: https://docs.caracal.run/v1.0/operations/backup-retention/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/backup-retention.md # Type: workflow # Concepts: # Requires: --- Repository scripts support Compose only. Kubernetes/external stores require operator tooling. ## Bundle Scope `backup.sh` writes Postgres globals/database dumps, Redis AOF after rewrite, and available STS/Gateway replay state. It excludes secrets. ## Backup Procedure ```bash CARACAL_BACKUP_DIR=/var/backups/caracal CARACAL_BACKUP_RETAIN=7 bash infra/scripts/backup.sh ``` Set the Compose project when non-default. Encrypt/copy off-host and back up matching secrets separately. ## Verify Backup Inspect manifest/dumps/AOF/replay, then restore with copied secrets into an isolated project and verify readiness, policy, sessions, revocation, authorization, and audit. ## Restore Procedure :::danger[Destructive restore] Restore stops apps, drops/recreates databases, replaces Redis, and overwrites replay state. ::: ```bash CARACAL_COMPOSE_PROJECT= CARACAL_RESTORE_CONFIRM=yes bash infra/scripts/restore.sh ``` Postgres/Redis containers and matching secrets must exist. ## Recovery On failure keep isolated, preserve output/state, recreate clean volumes, and retry from original bundle/secrets. Never reopen partial restore. ## Next Step Use [Respond to Incidents](/v1.0/operations/incident-response/) after data loss or tampering. --- # Respond to Incidents # URL: https://docs.caracal.run/v1.0/operations/incident-response/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/incident-response.md # Type: workflow # Concepts: # Requires: --- Use for policy bypass, credential exposure, unsafe routing, audit integrity/loss, malicious release, stale revocation, or serious availability failure. Canonical detail is in `governance/INCIDENT_RESPONSE.md`. ## Prerequisites Assign Incident Lead, Driver, and Reviewer. Use a private advisory for OSS security issues and approved private evidence storage. ## Procedure 1. Record source, time, version, mode, boundary, assets, and severity. 2. Contain severe incidents first: revoke/rotate, deny, block, disable, or remove traffic. 3. Preserve logs, metrics, audit, Redis pending/DLQ, replay, config, images, and request IDs. 4. Reproduce minimally in isolation and identify the missing guard/failure. 5. Remove the path, validate reproduction and a negative variant, and search sibling boundaries. 6. Recover and reconcile Postgres, Redis, outbox, revocation, sessions, and audit. 7. Communicate affected versions/actions after containment or fix. ## Verification Close only when behavior fails safely, targeted validation passes, readiness/safety signals recover, communication completes, and follow-ups have owners. ## Recovery Boundary Keep containment until validation. Do not delete evidence or publish exploit detail to accelerate closure. ## Next Step Use [Plan a Platform Rollout](/v1.0/operations/platform-rollout-kit/) for correction. --- # Plan a Platform Rollout # URL: https://docs.caracal.run/v1.0/operations/platform-rollout-kit/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/platform-rollout-kit.md # Type: workflow # Concepts: # Requires: --- Use this for infrastructure changes. Policy activation has a separate [Deploy Policy Changes](/v1.0/operations/policy-deployment/) procedure. ## Prerequisites Retain current version, rendered config, values, secrets, data backup, and schema-compatibility decision. Prepare readiness, authorization, revocation, and audit checks. ## Procedure 1. Verify provenance and render exact configuration. 2. Review migrations, Secret keys, endpoints, NetworkPolicies, volumes, resources, and alerts. 3. Confirm storage/event health. 4. Apply migrations, then roll via supported runtime or Helm path. 5. Stop on migration, readiness, audit, outbox, policy, revocation, or STS-exchange failure. 6. Resume only after canary authorization and audit evidence. ## Verify Record migration, rollout, `/ready`, queues/replay, metrics, and allow/deny canary request IDs. ## Rollback or Recovery Use prior app/config only when schema-compatible; otherwise roll forward. Restore data only for declared disaster recovery, not ordinary rollback. ## Next Step Complete [Hand Off to Platform Teams](/v1.0/operations/platform-team-handoff/). --- # Deploy Policy Changes # URL: https://docs.caracal.run/v1.0/operations/policy-deployment/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/policy-deployment.md # Type: workflow # Concepts: # Requires: --- Use web console, Admin SDK, or API. Runtime CLI does not manage policy. ## Prerequisites Record active version, allow/deny cases, Resources/scopes, Grants, Approval behavior, rollback version, and STS/Audit readiness. ## Procedure 1. Create version and compile merged bundle. 2. Simulate allow, deny, missing-scope, revoked-Session, and Approval cases. 3. Activate in a test zone/cohort. 4. Poll activation status until `propagation_status` is `loaded` and STS version matches. 5. Run canaries and inspect audit. 6. Expand only after pass. Allow the configured STS poll interval for multi-replica convergence. ## Verify Canaries match simulation, STS reports intended bundle, no compile/staleness alert fires, Gateway matches, and audit identifies determining policy. ## Rollback or Recovery Activate the last known-good version and repeat convergence/canaries. Do not edit stored versions or weaken unrelated grants. ## Next Step Use [Upgrade Caracal](/v1.0/operations/upgrade/) only for runtime/chart/schema changes. --- # Upgrade Caracal # URL: https://docs.caracal.run/v1.0/operations/upgrade/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/upgrade.md # Type: workflow # Concepts: # Requires: --- Migrations are forward-only. Runtime stages images, migrates, rolls services, then waits up to two minutes for readiness. This is not a zero-downtime guarantee. ## Prerequisites Read release notes, use supported sequential upgrades, [back up data and secrets](/v1.0/operations/backup-retention/), record current config, verify audit/streams, and prepare canaries. Install the new runtime binary before upgrading. ## Runtime Procedure ```bash caracal upgrade ``` It refreshes assets, preserves env/non-empty secrets, stages, migrates, rolls, and gates readiness. `--no-pull` uses staged images. Attempts append to `$CARACAL_HOME/upgrade.log` outside dev. Re-run after interruption. ## Helm Procedure Render/diff exact values, run pinned Helm upgrade with wait/atomic behavior, and inspect migration Jobs/rollout. Use a diff plugin only if installed. ## Verify Confirm migrations, readiness, queues/replay, exchange, protected request, revocation denial, and audit evidence. ## Rollback Prefer roll forward. Use older apps only after schema/config compatibility review. Never delete secrets or bypass the runtime version guard casually. ## Next Step Record result in [Plan a Platform Rollout](/v1.0/operations/platform-rollout-kit/). --- # Export Audit Evidence # URL: https://docs.caracal.run/v1.0/operations/compliance-audit-integration/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/compliance-audit-integration.md # Type: workflow # Concepts: # Requires: --- Audit records are technical evidence, not certification or a complete organizational trail. Audit consumes Redis, writes append-only Postgres rows, checks tamper state, and can export complete hourly Parquet partitions. ## Prerequisites Define evidence scope, retention, access, encryption, legal hold, and archive ownership. Configure `AUDIT_EXPORT_S3_*` only when needed and size `AUDIT_EXPORT_TMP_DIR` for the largest hourly partition. ## Procedure 1. Verify Audit readiness, HMAC, stream group, DLQ, and tamper metrics. 2. Configure an operator-owned S3-compatible endpoint and least-privilege credentials. 3. Enable export; monitor watermark, scratch space, upload failures, and objects. 4. Correlate a canary request across producer, stream, Postgres, and export. 5. Retain applicable HMAC material with verification evidence. The watermark advances only after successful upload. ## Verify and Recover Compare counts for a closed interval, inspect DLQ/lag, and verify intended reader access. On failure preserve Postgres, pending entries, DLQ, replay, scratch output, and logs before replay or deletion. ## Next Step Generate an [Generate an Evidence Pack](/v1.0/security/evidence-pack/) or complete [Hand Off to Platform Teams](/v1.0/operations/platform-team-handoff/). --- # Hand Off to Platform Teams # URL: https://docs.caracal.run/v1.0/operations/platform-team-handoff/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/platform-team-handoff.md # Type: workflow # Concepts: # Requires: --- This transfers operational responsibility; it does not create support or availability guarantees. ## Prerequisites Name service, storage, security, release, and on-call owners. Provide deployed values, secret inventory, monitoring, backups, and private incident access. ## Acceptance Procedure 1. Reproduce release verification and rendering. 2. Start or upgrade staging. 3. Verify migrations, health/readiness, metrics, authorization, revocation, and audit. 4. Trigger alerts and confirm routing. 5. Produce data and separate secret backups; restore both in isolation. 6. Rehearse dependency failure and security intake. 7. Record known limits and absent enterprise assurances. ## Acceptance Evidence Retain version/digests, values, migration/readiness output, canary IDs, alert tests, restore results, owners, escalation, and risks. ## Recovery Reject handoff when an owner, secret, backup, alert, or recovery test is missing. Continue staging ownership until closed. ## Next Step Use [Respond to Incidents](/v1.0/operations/incident-response/) as the on-call entry point. --- # Understand Architecture # URL: https://docs.caracal.run/v1.0/architecture/ # Markdown: https://docs.caracal.run/markdown/v1.0/architecture.md # Type: landing # Concepts: # Requires: --- You do not need this section to complete normal console setup. Use it when selecting an enforcement boundary, tracing a failed request, planning dependencies, or recovering state. ## Operational Model ```mermaid flowchart LR Human[Operator] --> Console[Web console] Automation[Trusted automation] --> Control[Control API or Admin SDK] Console --> API[API] Control --> API Workload[SDK or caracal run] --> STS[STS] Caller[Protected request] --> Gateway[Gateway] SDK[Session-aware SDK] --> Coordinator[Coordinator] API & STS & Gateway & Coordinator --> PG[(Postgres)] API & STS & Gateway & Coordinator --> Redis[(Redis Streams)] Redis --> Audit[Audit] Gateway --> Upstream[Configured upstream] ``` The web console, through its auth backend, is the human management surface; the API applies product and policy changes for it and for trusted automation. Postgres is durable state. Redis Streams propagates events and invalidations. STS decides and issues authority. Gateway enforces before an upstream call. Coordinator makes execution lineage explicit. Audit turns signed events into evidence. That separation creates useful failure boundaries: management can be unavailable without changing already-issued token expiry; Audit can lag while requests continue to emit replayable evidence; Redis can recover propagation from durable outboxes; Gateway denies when it cannot establish current authority. ## Choose the Flow You Need | Question | Read | | ----------------------------------------------------------- | ----------------------------------------------------- | | Which clients may call which surfaces? | [Map the System](/v1.0/architecture/system-topology/) | | What happens before a mandate is issued? | [Exchange Tokens](/v1.0/architecture/token-exchange-flow/) | | How do Sessions and Delegations affect authority? | [Coordinate Sessions](/v1.0/architecture/delegation-flow/) | | Why can state be current in Postgres but delayed elsewhere? | [Propagate Events](/v1.0/architecture/event-streams/) | | What must be backed up or restored first? | [Store State](/v1.0/architecture/storage-model/) | | Which key protects which trust transition? | [Manage Keys](/v1.0/architecture/crypto-keys/) | | Where must a deployment fail closed? | [Enforce Boundaries](/v1.0/architecture/trust-boundaries/) | ## Supported Integration Surfaces Integrators call the SDKs, STS token endpoint, Gateway proxy, documented Coordinator API, Admin API, or optional Control API according to the task. Internal endpoints, databases, Redis topics, replay files, and retained schema names are implementation boundaries, not application APIs. ## Next Step [Map the System](/v1.0/architecture/system-topology/). --- # Map the System # URL: https://docs.caracal.run/v1.0/architecture/system-topology/ # Markdown: https://docs.caracal.run/markdown/v1.0/architecture/system-topology.md # Type: architecture # Concepts: # Requires: --- Use this map before exposing a service or diagnosing a cross-service failure. ## Caller Map | Caller | Supported destination | Avoid | | ----------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------ | | Human operator | Web console on local `3001` or deployed console URL | Direct database, Redis, or internal routes | | Trusted management automation | Admin API/Admin SDK, or optional Control API on API `3000` | Runtime CLI product-management commands | | Workload needing mandates | STS `POST /oauth/2/token` through an SDK or documented client | API admin credentials | | Protected HTTP client | Gateway `8081` with a mandate and resource header | Direct protected upstream when Gateway is the enforcement boundary | | Session-aware application | Coordinator `4000` through an SDK or documented API | Coordinator operator tokens in workload source | | Verifier | STS JWKS and revocation backend through verification packages | Private signing keys | Ports are local Compose defaults, bound to loopback. A production deployment normally places supported public surfaces behind TLS and keeps internal and operator-only routes private. ## Dependency Map ```mermaid flowchart TB Console --> API Console --> Coordinator SDK --> STS Client --> Gateway Gateway --> STS Gateway --> Upstream API & STS & Gateway & Coordinator & Audit --> PG[(Postgres)] API & STS & Gateway & Coordinator & Audit --> Redis[(Redis)] ``` | Visible failure | Dependency to check | | --------------------------------------------- | --------------------------------------------------------------------------- | | Console cannot list or mutate product objects | API auth, API readiness, Postgres, then Redis/outbox | | Exchange denies or cannot load policy | STS, Postgres policy/product state, Redis invalidation, signing/secret keys | | Gateway denies before reaching upstream | Inbound mandate, resource binding, revocation, STS, or upstream safety | | Sessions or Delegations appear stale | Coordinator, Postgres, Redis, outbox, leases/sweepers | | Audit search lags behind requests | Redis consumer state, Audit readiness, DLQ, replay volumes, Postgres | ## Deployment Implications * API, STS, Gateway, Audit, and Coordinator all need Postgres and Redis in the packaged topology; readiness captures more than process liveness. * Gateway also depends synchronously on STS for per-request exchange. * Console product views depend on API and Coordinator through the auth backend-for-frontend. * Control is an optional API plugin, not a separate service or port. Use [Choose a Deployment Profile](/v1.0/operations/deployment-profiles/) for deployment choices and [Understand Services](/v1.0/services/) for service-specific failure posture. ## Next Step [Exchange Tokens](/v1.0/architecture/token-exchange-flow/). --- # Exchange Tokens # URL: https://docs.caracal.run/v1.0/architecture/token-exchange-flow/ # Markdown: https://docs.caracal.run/markdown/v1.0/architecture/token-exchange-flow.md # Type: architecture # Concepts: # Requires: --- Use this flow when an SDK, `caracal run`, or Gateway cannot obtain authority. ## Request Flow ```mermaid sequenceDiagram participant Client as SDK / caracal run / Gateway participant STS participant PG as Postgres participant Redis participant Audit Client->>STS: POST /oauth/2/token or run credential request STS->>PG: authenticate client; load resource, policy, authority, session, delegation STS->>Redis: use invalidation and revocation state STS->>STS: evaluate requested resource, scopes, operation, constraints alt approval required STS-->>Client: interaction_required + hold id + expiry else denied STS-->>Client: typed denial STS->>Redis: signed audit event else allowed STS-->>Client: scoped, short-lived mandate or provider credential STS->>Redis: signed audit event Redis->>Audit: ingest evidence end ``` ## What the Integrator Supplies The caller authenticates an application or workload, identifies the zone and target resource, and requests scopes. A subject token can carry a Federated user's `sub` into the chain; a prior Caracal mandate can be narrowed. Session and Delegation identifiers supply execution lineage when the SDK flow created them. The protocol fields `session_id`, `agent_session_id`, and `delegation_edge_id` refer respectively to an Authority record, a Coordinator Session, and a Delegation. They are not interchangeable IDs. ## Approval Branch An Approval is a hold, not a credential. An eligible operator or Federated user decides it, then the waiting client retries with the hold ID. STS consumes an approved hold once while issuing the credential. Rejected, expired, undecided, or already-consumed holds deny. `caracal run` waits and retries once. SDK applications should follow their language SDK's interaction-required contract. See [Approvals](/v1.0/concepts/approvals/) for the canonical model. ## Failure and TTL Implications * Resource mandates are capped at 15 minutes; Session mandates are capped at 60 minutes. * Gateway rejects an inbound token that is too close to expiry before proxying. * Invalid client proof, Federated user issuer, resource, policy, Session, Delegation, operation, scope, or approval fails before issuance. * Gateway-authenticated exchanges are signed over the exact form body with a timestamp and nonce; query ambiguity and duplicate singleton fields are rejected. * Audit delivery is asynchronous, but STS and Gateway use replay storage when immediate Redis publication is unavailable. For exact request fields and errors, use [Use STS Endpoint](/v1.0/api/sts/). Do not call STS internal policy or key-rotation routes; the API service owns those calls. ## Next Step [Coordinate Sessions](/v1.0/architecture/delegation-flow/). --- # Coordinate Sessions # URL: https://docs.caracal.run/v1.0/architecture/delegation-flow/ # Markdown: https://docs.caracal.run/markdown/v1.0/architecture/delegation-flow.md # Type: architecture # Concepts: # Requires: --- Use this flow when a request depends on execution lineage, long-lived leases, or authority delegated between Sessions. ## Session and Delegation Flow ```mermaid sequenceDiagram participant SDK participant Coord as Coordinator participant PG as Postgres participant Redis participant STS SDK->>Coord: start Session Coord->>PG: persist Session, parent, lifecycle, lease Coord->>Redis: publish lifecycle from durable outbox SDK->>Coord: create bounded Delegation Coord->>PG: persist edge, constraints, expiry, graph epoch Coord->>Redis: publish delegation invalidation SDK->>STS: exchange with Authority record, Session, Delegation IDs STS->>PG: validate live lineage and bounds STS-->>SDK: mandate or denial ``` ## Choose a Session Lifecycle Task Sessions use a TTL and expire. Service Sessions use heartbeat leases and become unhealthy when renewal stops. The process that owns a service Session must continue heartbeating; container liveness alone does not renew authority. A Session tree records parent-child execution. Suspending or terminating a subtree affects governed execution state. A Delegation separately records exactly which resource, scopes, constraints, and expiry cross from one Session to another. ## Consistency and Failure Coordinator writes state and outbox records durably in Postgres, then publishes lifecycle and invalidation events through Redis. A short propagation delay is possible. STS validates authoritative Session and Delegation state before issuing delegated authority, so a stale consumer view does not grant authority by itself. Operational jobs expire stale task Sessions and Delegations, detect missed service leases, enforce invocation deadlines, publish outbox rows, and clean retained terminal data. Backlogged or dead outbox rows mean downstream views and revocation consumers can lag; surface that in Diagnostics before assuming the SDK failed. ## Operator and Integrator Surfaces Applications use an SDK or the documented Coordinator API. Human operators use console **Sessions**, Delegation views, and Audit. Do not mutate Coordinator tables or publish lifecycle topics directly. Top-level `caracal` commands do not manage Sessions or Delegations. ## Next Step [Propagate Events](/v1.0/architecture/event-streams/). --- # Propagate Events # URL: https://docs.caracal.run/v1.0/architecture/event-streams/ # Markdown: https://docs.caracal.run/markdown/v1.0/architecture/event-streams.md # Type: architecture # Concepts: # Requires: --- Redis Streams carries propagation, not the durable product model. This distinction explains why a committed change can exist before every consumer has observed it. ## Delivery Paths ```mermaid flowchart LR Tx[API or Coordinator transaction] --> Outbox[(Postgres outbox)] Outbox --> Redis[Signed Redis stream message] Redis --> Consumers[Consumer groups] Consumers --> Effects[Reload, revoke, relay, audit] Emit[STS or Gateway audit emit] --> Replay[Replay volume] Replay --> Redis ``` API and Coordinator changes enqueue outbox rows in the same Postgres transaction as state. A dispatcher retries publication. Consumers use groups, pending-entry recovery, deduplication, and signed messages in published modes. STS and Gateway cannot put audit evidence in the same database transaction as every decision. They write replay files when Redis delivery is unavailable and drain them after recovery. ## What Lag Means | Observation | Interpretation | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | Product read shows a change but a consumer has not reacted | Outbox or stream propagation is delayed. | | Gateway or verifier still accepts revoked authority | Check revocation snapshot, consumer lag, readiness, and verifier fail posture immediately. | | Audit event appears later | Check Audit consumer lag and replay backlog; do not assume evidence was never emitted. | | Dead outbox rows or old pending entries | Recovery needs operator action; readiness thresholds may fail. | Postgres remains the recovery anchor. Redis is operationally important but is not a replacement for database backups. ## Canonical Topic Reference [Use Event Topics](/v1.0/api/event-topics/) owns the topic, producer, and consumer-group list. Do not publish those topics from application code. They are service integration contracts, not a public event bus for workloads. ## Next Step [Store State](/v1.0/architecture/storage-model/). --- # Store State # URL: https://docs.caracal.run/v1.0/architecture/storage-model/ # Markdown: https://docs.caracal.run/markdown/v1.0/architecture/storage-model.md # Type: architecture # Concepts: # Requires: --- ## Data Classes | State | Store | Operational implication | | -------------------------------------------------------------------------------- | --------------------------------------- | ---------------------------------------------------------------------------- | | Product, policy, authority, Session, Delegation, admin-audit, and audit evidence | Postgres | Primary durable backup and restore target. | | Invalidation, revocation, lifecycle, audit delivery, consumer coordination | Redis Streams | Preserve when possible; recover propagation from durable state and outboxes. | | Undelivered STS/Gateway audit events | Replay volumes | Restore before discarding Redis state so evidence can drain. | | Provider, application, workload, signing, HMAC, and encryption secrets | Secret backend and runtime secret files | Restore with matching key material; ciphertext alone is insufficient. | ## Ownership Boundaries The API owns product and policy data. STS owns issuance behavior and Authority records. Coordinator owns Sessions and Delegations. Audit owns append-only evidence ingestion. Shared Postgres does not make one service's tables a supported API for another client. Some schema names retain protocol history, including `agent_services`, `agent_invocations`, and `delegation_edges`. Public surfaces expose Sessions, invocations, and Delegations. Integrations must use public names and APIs rather than retained table names. ## Integrity Guarantees * Production migrations move forward and are recorded in `schema_migrations`. * Policy versions are immutable. * Zone-scoped reads use fail-closed row-level security. * The Audit database role cannot update or delete evidence rows. * Outboxes couple a state change to eventual event publication. * Secret envelopes bind encrypted values to key fingerprints and purposes. ## Restore Order 1. Restore Postgres and the runtime/secret-backend keys needed to decrypt it. 2. Restore STS and Gateway audit replay volumes. 3. Restore Redis streams and pending entries when available, or allow durable publishers and snapshots to rebuild propagation state. 4. Reconnect audit exports and verify retention watermarks. 5. Run readiness, Diagnostics, a policy simulation, a protected request, and an Audit trace. Use [Back Up and Retain Data](/v1.0/operations/backup-retention/) for procedures. Do not use direct SQL as a management or migration shortcut. ## Next Step [Manage Keys](/v1.0/architecture/crypto-keys/). --- # Manage Keys # URL: https://docs.caracal.run/v1.0/architecture/crypto-keys/ # Markdown: https://docs.caracal.run/markdown/v1.0/architecture/crypto-keys.md # Type: architecture # Concepts: # Requires: --- Caracal separates keys so compromise or rotation at one boundary does not silently authorize another. ## Key-to-Boundary Map | Material | Boundary | Failure implication | | ------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | Zone signing keys | STS mandate issuance to verifiers | Missing private material blocks issuance; stale JWKS caches can reject a rotated signer. | | `SECRET_STORE_KEK` | Services to sealed provider, signing, connection, sink, application, and workload secrets | Wrong key fingerprint prevents decryption; restore ciphertext and keyring together. | | `AUDIT_HMAC_KEY` | Producers to Audit evidence | Mismatch creates HMAC failures and blocks trusted ingestion. | | `STREAMS_HMAC_KEY` | Stream producers to consumers | Mismatch blocks trusted propagation in published modes. | | `GATEWAY_STS_HMAC_KEY` | Gateway to STS exchange | Mismatch denies every Gateway-authenticated exchange. | | Admin/Coordinator/Control credentials | Operator clients to management surfaces | Missing or insufficient scope yields `401` or `403`; do not distribute to workloads. | ## Mandate Verification Flow ```mermaid sequenceDiagram participant STS participant Client participant Verifier Verifier->>STS: GET /.well-known/jwks.json?zone_id=... STS-->>Verifier: public keys + cache headers STS->>Client: signed mandate Client->>Verifier: bearer mandate Verifier->>Verifier: verify signature, issuer, audience, use, scopes, expiry, revocation ``` Keep old public keys available until verifier caches and outstanding token TTLs no longer need them. Never copy private signing material to a verifier. ## Secret Envelope Rotation Secret Store uses a fresh data-encryption key per value and wraps it with the configured KEK. The envelope records a key fingerprint and purpose-specific associated data. During KEK rotation, `SECRET_STORE_KEK_PREVIOUS` allows reads under the retiring key while data is rewrapped. Removing it too early makes old envelopes unreadable. Published `rc` and `stable` modes require validated HMAC material rather than silently disabling integrity checks. A missing or short required key fails startup or readiness. Use [Rotate Keys and Secrets](/v1.0/operations/key-management/) for the operational procedure and [Configure Secret Backends](/v1.0/operations/secret-backends/) for custody options. ## Next Step [Enforce Boundaries](/v1.0/architecture/trust-boundaries/). --- # Enforce Boundaries # URL: https://docs.caracal.run/v1.0/architecture/trust-boundaries/ # Markdown: https://docs.caracal.run/markdown/v1.0/architecture/trust-boundaries.md # Type: architecture # Concepts: # Requires: --- ## Boundary Decisions | Boundary | Accepted caller | Must remain outside | | --------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------- | | Runtime CLI | Local operator managing lifecycle or launching a workload | Product-management state and admin credentials | | Web console/BFF | Authenticated, admitted operator with zone access | Anonymous callers and direct browser possession of root service tokens | | Admin API | Trusted Admin SDK or console backend | Workload code | | Control API | Enabled gate plus short-lived, scoped, zone-bound Control token | Replayed tokens, root admin token distribution, lifecycle commands | | STS | Valid application/workload/Gateway proof and well-formed exchange | End-user password authentication and unsupported token fields | | Gateway | Mandate, `X-Caracal-Resource`, binding, safe path and upstream | Caller-selected upstreams and caller-provided upstream credentials | | Coordinator | SDK authority or operator credential appropriate to the route | Direct table mutation and unsigned lifecycle publication | | Audit | Signed events and authenticated operator search | Mutable evidence and untrusted stream payloads | ## Protected Request Boundary Gateway denies before upstream dispatch when it cannot establish a valid, unexpired, unreplayed, unrevoked mandate; a resource header and binding; an allowed operation and scope; a safe path and host; and a successful STS exchange. It strips caller authorization before applying the configured upstream credential. Gateway is an HTTP proxy. It streams HTTP responses, including SSE, while rechecking revocation. It strips hop-by-hop upgrade headers and does not proxy WebSockets. Protect WebSockets in process with the verification engine or an adapter. ## Management Boundary The console is the human management surface. Admin SDK and Control API are automation surfaces. The runtime CLI remains local lifecycle and workload launch only. A Control gate or API outage must not prevent `caracal status`, `down`, or `up` from operating locally. The system-zone viewer adds a console-side read-only boundary: mutation controls are disabled and non-read requests are blocked. It is a transparency surface, not an alternative management route. ## Fail-Closed Expectations * Gateway and STS deny when authority, revocation, policy, or service proof cannot be established. * Published modes do not permit `JTI_FAIL_OPEN` and require integrity keys. * Postgres row-level security denies cross-zone access without valid context. * Control denies when disabled, unauthenticated, replayed, rate-limited, out of scope, or unable to record required audit evidence. * Audit rejects integrity failures rather than treating them as trustworthy evidence. For deployment controls, use [Harden Production](/v1.0/operations/tls-hardening/). For exact Gateway behavior, use [Proxy Through Gateway](/v1.0/api/gateway/). ## Next Step The architecture journey ends here. Use [Understand Services](/v1.0/services/) for each service's configuration, readiness, and failure posture. --- # Operate Runtime and the Web Console # URL: https://docs.caracal.run/v1.0/runtime-console/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console.md # Type: landing # Concepts: # Requires: --- Use this section after installation. Operating Caracal involves two surfaces with a deliberate split: * **The `caracal` runtime CLI** manages the local stack's lifecycle - starting, stopping, checking, upgrading - and launches workload processes with injected credentials. It never touches product state. * **The web console** is the browser interface for everything product-shaped: zones, applications, providers, resources, policies, workloads, audit, and live intervention. The pages below follow the work in the order an operator normally performs it, without mixing runtime lifecycle with product management. ## Complete an Operator Journey 1. [Choose the Right Surface](/v1.0/runtime-console/cli-and-console/). Use `caracal` for local lifecycle and process launch, the web console for human management, and the Control API or Admin SDK for automation. 2. [Start and Check the Stack](/v1.0/runtime-console/stack/). Start services, wait for readiness, and open the packaged console. 3. [Control Console Access](/v1.0/runtime-console/console-access/). Admit the people who may sign in. 4. [Use the Web Console](/v1.0/runtime-console/console/). Finish account onboarding, create a zone, and complete guided setup. 5. [Configure Workloads](/v1.0/runtime-console/config-file/). Create a Launcher workload and credential bindings, or configure an SDK profile. 6. [Run Workloads](/v1.0/runtime-console/runtime/). Launch a command with short-lived credentials. 7. [Inspect Diagnostics and Audit](/v1.0/runtime-console/observability/). Read dashboard posture, health, decisions, and request traces. 8. [Manage Product Objects](/v1.0/runtime-console/admin/) and [Manage Runtime Authority](/v1.0/runtime-console/agents/). Maintain configuration and intervene in Subjects, Authority records, Sessions, Delegations, and Approvals. ## Keep the Boundary Clear | Goal | Surface | | ------------------------------------------------------------------------------------------------ | ------------------------ | | Start, stop, check, upgrade, or purge the local stack | `caracal` | | Launch one process with injected credentials | `caracal run` | | Manage access to the console host | `caracal allowlist` | | Create or change zones, applications, providers, resources, policies, workloads, or Control keys | Web console | | Inspect and intervene in runtime authority | Web console | | Automate product management | Control API or Admin SDK | The runtime CLI does not provide zone, policy, session, delegation, approval, audit, or Control management commands. This keeps local lifecycle independent of admin tokens, selected zones, and product credentials. ## Local Addresses After `caracal up`, open the packaged console at [http://localhost:3001](http://localhost:3001). The full loopback port map lives in [Start and Check the Stack](/v1.0/runtime-console/stack/#open-the-packaged-console). Do not call service ports merely because they are reachable locally. The console, SDKs, Gateway, Control API, and documented API surfaces provide the supported paths for their respective tasks. ## Next Step Start with [Choose the Right Surface](/v1.0/runtime-console/cli-and-console/). --- # Choose the Right Surface # URL: https://docs.caracal.run/v1.0/runtime-console/cli-and-console/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console/cli-and-console.md # Type: workflow # Concepts: # Requires: --- Choose a surface from the outcome you need, not from where a similarly named object happens to appear. ## Decision Table | You need to | Use | Why | | ------------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------ | | Start, stop, inspect, upgrade, or reset a local runtime | `caracal` | These operations concern local processes and storage. | | Launch a command with scoped credentials | `caracal run` | It authenticates a Launcher workload and injects its configured bindings. | | Admit, suspend, or remove a console user | `caracal allowlist` | Access is host-level auth configuration, not zone product state. | | Create or change product objects | Web console | It provides selected-zone context, validation, guided setup, and audited secret handling. | | Investigate or intervene in live authority | Web console | Subjects, Authority records, Sessions, Delegations, Approvals, Audit, and Diagnostics are structured operator workflows. | | Manage product state from CI or another trusted host | Control API or Admin SDK | These are authenticated automation surfaces. | ## Runtime CLI Boundary The top-level runtime commands are lifecycle and launch commands: ```text up down status upgrade purge allowlist run web ``` `web` is a development launcher for the console UI and auth backend. It is not a product-management command. The packaged console already runs as part of `caracal up` and is opened in a browser at `http://localhost:3001`. Do not look for top-level commands to create zones, policies, applications, resources, workloads, Sessions, Delegations, Approvals, or Control keys. Use the console for human work. Use the Admin SDK or Control API for automation. ## Console Versus Automation Use the web console when a person needs to follow guided setup, review selected-zone validation, reveal a secret with an audit record, activate policy, inspect a trace, decide an eligible Approval, or intervene in a Session. Use automation only from a trusted operator environment. The Admin SDK provides direct management APIs. The optional Control API provides zone-bound, scoped, replay-protected remote invocation. Neither surface starts Docker or launches workload processes. Never give workload code root admin, Coordinator, Control, or secret-store credentials. Workloads authenticate with application, SDK, or Launcher workload credentials. ## If the Wrong Surface Seems Necessary | Symptom | Action | | ------------------------------------------------------ | ----------------------------------------------------------------------------------- | | A zone or policy command is absent from `caracal` | Open the web console; the absence is intentional. | | A lifecycle command asks for a zone or admin token | Treat it as a boundary violation. | | CI needs repeatable configuration | Use the Admin SDK or Control API, not browser scripting. | | The console is unavailable but lifecycle must continue | Use `caracal status`, `down`, or `up`; lifecycle does not depend on console access. | ## Next Step [Start and Check the Stack](/v1.0/runtime-console/stack/). --- # Start and Check the Stack # URL: https://docs.caracal.run/v1.0/runtime-console/stack/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console/stack.md # Type: workflow # Concepts: # Requires: --- The stack is the set of containers `caracal up` manages: Caracal's services, PostgreSQL, Redis, and the packaged web console. This page covers its day-to-day lifecycle; [Install Caracal](/v1.0/get-started/install-caracal/) covers getting the CLI and Docker in place. ## Start ```bash caracal up caracal status --ready ``` `caracal up` starts the stack in the background. Development mode builds local images; release modes use the installed runtime assets. Continue only when `status --ready` exits `0`. `caracal status` checks liveness. `caracal status --ready` also checks service dependencies. A service can be healthy while Postgres, Redis, policy state, or another required dependency is not ready. For machine-readable checks: ```bash caracal status --ready --json ``` ## Open the Packaged Console Open [http://localhost:3001](http://localhost:3001). Compose maps loopback port `3001` to the combined web/auth service on container port `3002`. The other loopback bindings are API `3000`, Coordinator `4000`, STS `8080`, Gateway `8081`, and Audit `9090`. They are useful for documented clients and probes, not substitutes for console workflows. ## Stop or Reset ```bash caracal down ``` This stops the stack while preserving volumes. `caracal down -v` also removes Compose volumes. Use `caracal purge` only when intentionally removing selected local artifacts or all local runtime state. A destructive reset can invalidate generated tokens and remove product data; rebuild zones and product objects through the console afterward. ## Recover Common Startup Failures | Result | Meaning | Next action | | ------------------------------------------ | -------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | `status` fails | At least one service is not live. | Inspect container/service logs and port conflicts. | | `status` passes but `status --ready` fails | A live service cannot satisfy a dependency or readiness threshold. | Open Diagnostics if available; otherwise inspect the failing readiness probe. | | Console does not open but readiness passes | Host port `3001` is occupied or the web service failed separately. | Check the web service and local listener. | | Credentials fail after reset | A shell variable or file contains a credential from the prior state. | Use freshly generated local secret files or rotate the affected credential. | For infrastructure diagnosis, use [Debug Infrastructure Issues](/v1.0/operations/debugging/). ## Development Console Launcher `caracal web` is for console development. It stops the packaged web container to avoid a port collision, starts the local UI and auth backend, and leaves the packaged container stopped when it exits. Run `caracal up` afterward to restore the packaged console. `caracal web --allow-offline` permits UI or sign-in development while runtime services are unavailable. Product views still cannot perform their backend work. ## Next Step [Control Console Access](/v1.0/runtime-console/console-access/). --- # Use the Web Console # URL: https://docs.caracal.run/v1.0/runtime-console/console/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console/console.md # Type: workflow # Concepts: # Requires: --- The web console is where people manage Caracal: it carries account onboarding, guided setup, every product form, audit, and live intervention. This page orients you in it; the pages after it go deep on each workflow. ## Open the Correct Console For normal local operation, open [http://localhost:3001](http://localhost:3001) after `caracal up`. This is the packaged console served with its session-guarded backend-for-frontend. Use `caracal web` only while developing the console. That development launcher is unrelated to the console's **Launcher** workload page: | Name | Purpose | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `caracal web` | Starts local web-development processes. It does not create or run governed workloads. | | **Services → Launcher** | Creates workload identities and credential bindings consumed by `caracal run`. It does not start a development server. | ## Sign In and Finish Account Onboarding After admission through the host allowlist, sign in with a configured method at `/sign-in` (registration at `/sign-up`) under the console origin. A first account visit collects the operator profile and creates or selects the first ordinary zone. This account onboarding is distinct from in-zone guided setup. The console keeps the active zone in the URL and profile state. Zone-scoped pages require a selected zone. ## Complete Guided Setup Guided setup reads live zone inventory and opens the real product forms. It teaches one deny-by-default path: 1. **Register an application** such as Anton. This is the identity requesting access. 2. **Connect a provider** such as Hooli OIDC. This supplies upstream credentials at runtime. 3. **Define a resource** such as `resource://pipernet`, including its scopes and upstream. 4. **Activate a policy** that authorizes the intended application and scopes. 5. **Verify** from the dashboard, policy simulation, Sessions, and Audit. The guide marks a step complete from actual zone state. It does not create a Launcher workload. After access is enforcing, open **Services → Launcher** to configure `caracal run`. For field-level guidance, use [Define Resources and Providers](/v1.0/guides/resources-providers/) and [Activate a Policy Set](/v1.0/guides/activate-policy-set/). ## Read the Console by Task | Task | Console area | | ------------------------------------------------ | ----------------------------- | | Review posture and recent activity | **Dashboard** | | Configure applications, providers, and resources | **Access** | | Author, simulate, and activate policy | **Policy** | | Investigate Subjects and authority | **Subjects** | | Intervene in Sessions, Delegations, or Approvals | **Runtime** | | Search decisions and run health checks | **Audit** and **Diagnostics** | | Configure workload launch | **Services → Launcher** | | Configure scoped remote automation | **Services → Control** | ## System-Zone Read-Only View Caracal's reserved system zone is not offered in the normal active-zone switcher. The Settings entry opens it in a separate transparency view. In that tab, editable fields and mutating controls are disabled, every non-GET/HEAD console API request is blocked with `system_zone_read_only`, and settings, Operator, and other hide-locked routes are unavailable. Do not use the system-zone viewer for customer or application configuration. Return to an ordinary zone to make changes. ## Secret Handling Application and Launcher workload secrets are held in Secret Store custody. Copy a secret at creation or use the object's reveal action later. Every reveal is audited. Rotation invalidates the current secret, so update the consuming secret store or file immediately. Provider secrets are entered during supported create or rotation flows and remain masked. The console does not write revealed values to the workload host. ## Caracal Operator Open **Caracal Operator** from the utility rail or command palette. Each conversation runs in **Ask** mode for read-only investigation or **Agent** mode for answers, live reads, and change plans. A mutating plan is validated and previewed against current state, then waits for Approval before it is revalidated and applied. Conversation memory records activity; live reads and execution previews remain the source of truth for current state. Configure natural-language models under **Settings → AI Operator → Models**. The page can add, edit, test, rotate, and delete OpenAI-compatible model endpoints. A new or rotated key is sent once, sealed into the reserved `caracal.sys` Zone, and never returned. Caracal routes model calls through the governed Gateway. The Operator can still expose deterministic catalog and plan behavior when no model endpoint is configured, but natural-language assistance remains unavailable. ## Next Step [Configure Workloads](/v1.0/runtime-console/config-file/). --- # Control Console Access # URL: https://docs.caracal.run/v1.0/runtime-console/console-access/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console/console-access.md # Type: workflow # Concepts: # Requires: --- The runtime host controls who may register and sign in: after the stack starts, nobody can use the console until you admit their email from the machine that runs Caracal. Zone roles and product authority are separate concerns applied after authentication. ## Admit a Person or Domain ```bash caracal allowlist add richard.hendricks@piedpiper.example caracal allowlist add @piedpiper.example caracal allowlist list ``` Entries are exact email addresses or `@domain` suffixes. An exact entry takes precedence over a domain entry. While any active or locked entries exist, the allowlist is the sign-in authority. With an empty list, registration follows deployment configuration: open in development and closed in production unless configured otherwise. ## Suspend, Restore, or Remove Access ```bash caracal allowlist lock monica.hall@piedpiper.example caracal allowlist unlock monica.hall@piedpiper.example caracal allowlist remove gavin.belson@hooli.example ``` | Action | Sign-in effect | Account effect | | ------------------- | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `lock` | Blocks sign-in and console requests. | Revokes sessions; retains the account and product data. | | `unlock` | Restores access. | Retains the existing account. | | `remove` | Blocks access until explicitly re-admitted. | Revokes sessions and erases sign-in records on the person's next contact; zone data and audit evidence remain. | | `add` after removal | Allows registration again. | Starts a fresh admission. | Removal is stored as an explicit marker. A missing, empty, or unreadable allowlist may deny access, but it does not trigger account erasure. ## Interpret Access Denial The browser deliberately shows the same access-denied page when an address was never admitted, is locked, or was removed. This prevents the UI from revealing account status. Runtime auth logs retain the concrete reason for operators. To resolve a denial: 1. Run `caracal allowlist list` on the runtime host. 2. Unlock a locked entry, or add an intended address. 3. Check the configured email/password or OAuth sign-in method. 4. If the allowlist is empty, check `CARACAL_OPEN_REGISTRATION` and the deployment environment. Being admitted to the console does not make a person a workload Subject. Console accounts authenticate operators. Subjects are the identities applications act for - each application itself, or its Federated users - and are observed inside a selected zone. ## Next Step [Use the Web Console](/v1.0/runtime-console/console/). --- # Configure Workloads # URL: https://docs.caracal.run/v1.0/runtime-console/config-file/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console/config-file.md # Type: workflow # Concepts: # Requires: --- Caracal supports two configuration paths. Choose one from the credential lifecycle your process needs. | Need | Configure | | ------------------------------------------------------------------- | -------------------------------------------------- | | Start a bounded command with environment-injected credentials | A workload and bindings on **Services → Launcher** | | Renew credentials during a long-running process or use SDK Sessions | An SDK profile or SDK environment configuration | ## Configure a Launcher Workload 1. Open **Services → Launcher** in the intended zone. 2. Create a workload identity, for example Fiona. 3. Add bindings. Each binding selects an environment variable, a resource, the scopes for the policy decision, and - for optional bindings - the unavailable behavior. The injected value is the resource provider's credential; the provider must allow runtime injection. See [Run an Agent with caracal run](/v1.0/guides/runtime-run/#what-a-binding-injects). 4. Reveal the workload secret. The reveal is recorded in admin audit. 5. Store the secret on the host that will run the command. 6. Copy the exact launch command shown by the page. The launching host carries only workload identity and proof: | Variable | Meaning | | ------------------------------ | ---------------------------------------------- | | `CARACAL_WORKLOAD_ID` | Required workload identifier. | | `CARACAL_WORKLOAD_SECRET` | Inline secret for local development. | | `CARACAL_WORKLOAD_SECRET_FILE` | Explicit mounted secret-file path. | | `CARACAL_STS_URL` | STS override for custom or remote deployments. | Set only one secret source. In local dev and stable modes, omitting both secret variables makes the runtime read `/runtime//secret`. Production launches require an explicit secret source. ```bash mkdir -p ~/.config/caracal/runtime/ printf '%s' '' > ~/.config/caracal/runtime//secret chmod 600 ~/.config/caracal/runtime//secret ``` The config directory defaults to `$XDG_CONFIG_HOME/caracal` or `~/.config/caracal` on Linux, `~/Library/Application Support/Caracal` on macOS, and `%APPDATA%\Caracal` on Windows. `CARACAL_CONFIG_HOME` overrides it. ### One file for every command Workload settings share the operator env file that every `caracal` command - `up`, `web`, and `run` - loads at startup: workload identity and STS URL sit alongside the web console's sign-in settings and any stack override. See [Configure Service Environment](/v1.0/operations/env-vars/#the-operator-env-file) for the file's per-platform location, editor commands, and precedence. This table maps where the workload secret comes from in each deployment: | Deployment | Config file or mechanism | Workload secret | | --- | --- | --- | | **Dev (repo)** | `.env` at the repo root - copy the committed `.env.example` | Inline in `.env`, or the default `/runtime//secret` | | **Released (host)** | `$CARACAL_HOME/caracal.env` (created `0600` on first `caracal up`), or a path you set with `CARACAL_ENV_FILE` | Inline in that file, or `CARACAL_WORKLOAD_SECRET_FILE` | | **Cloud** | Orchestrator env - a Kubernetes `env` block (Helm `web.extraEnv`) or an ECS task definition | A secret manager mounted read-only (`defaultMode: 0400`) via `CARACAL_WORKLOAD_SECRET_FILE`, or `valueFrom.secretKeyRef` | For release and cloud, set `CARACAL_ENV=production`: it disables the auto-detected local secret file so a launch fails closed unless it names an explicit secret source. Caracal never auto-loads a `.env` from the current working directory (a footgun for a credential tool) and requires secret files to be owner-only. A binding with no selected scopes requests the resource's configured scope set. Required binding failure prevents launch. Configure an optional binding's unavailable behavior on the Launcher page when the process may continue without it. ## Configure an SDK Profile SDK loaders use `CARACAL_CONFIG` when it names an existing file; otherwise they read explicit runtime environment configuration. They do not search the current directory or home directory for a profile. ```toml zone_id = "" application_id = "" default_ttl_seconds = 900 [[credentials]] resource = "resource://pipernet" upstream_prefix = "https://api.pipernet.example/v1" ``` Common profile fields are `sts_url`, `gateway_url`, `coordinator_url`, `zone_id`, `application_id`, `app_client_secret_file`, `app_client_secret`, `default_ttl_seconds`, `credentials`, and `optional_credentials`. Launcher binding fields such as environment name and unavailable behavior do not belong in an SDK profile. For language-specific setup, use [Choose an SDK or Package](/v1.0/sdks/). ## Security Checks * Keep workload and application secret files owner-only and explicitly mounted in production. * Do not mount operator secret directories into workload containers. * Non-local insecure STS URLs are rejected unless the deployment explicitly permits them. * Binding targets cannot be process-loader variables such as `NODE_OPTIONS`, `LD_PRELOAD`, `LD_LIBRARY_PATH`, or `DYLD_*`. ## Next Step [Run Workloads](/v1.0/runtime-console/runtime/). --- # Run Workloads # URL: https://docs.caracal.run/v1.0/runtime-console/runtime/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console/runtime.md # Type: workflow # Concepts: # Requires: --- `caracal run` authenticates a Launcher workload, fetches its bindings, obtains each binding's provider credential after a policy decision, injects the credentials into a child environment, and returns the child's result. The workload identity and bindings come from [Configure Workloads](/v1.0/runtime-console/config-file/); this page is the launch contract - what happens at spawn, and how approvals, expiry, signals, and exits behave. ```bash caracal run -- python3 agent.py ``` Use `--` before a child command that accepts flags. ## What Happens Before Spawn ```text workload proof → fetch bindings → obtain provider credentials → inject environment → spawn child ``` For each binding, the runtime validates the environment name and secret-file permissions, then requests the credential for only the configured resource and scopes. The injected value is the resource provider's own credential - a brokered OAuth token or the sealed static key - released only when the provider allows runtime injection and policy approves; the eligible provider kinds are listed in [What a Binding Injects](/v1.0/guides/runtime-run/#what-a-binding-injects). A required failure prevents the child from starting. Optional bindings follow their configured unavailable behavior. Static upstream credentials such as API keys remain as powerful as the upstream value itself even when the Caracal release decision was scope-bounded. Prefer short-lived provider tokens or Gateway brokering where supported. ## Approval Holds If STS returns `interaction_required`, the runtime prints an `approval_required` notice containing the hold and binding, long-polls the hold, and retries the credential request once after approval. When the response has no usable expiry, it uses a five-minute fallback. Rejection, expiry, timeout, or consumption by another request prevents spawn. An operator can inspect eligible holds in the console **Approvals** page. Holds reserved for the Federated user are visible there but can be decided only by the application's own user. ## Child Environment The child receives configured credential variables and, when the provider reports expiry, `_EXPIRES_AT` in epoch seconds. It also receives a narrow allowlist of ordinary process variables such as `PATH`, home, temporary-directory, locale, terminal, CI, environment name, and OS system variables. Launcher configuration variables are removed. In particular, `CARACAL_WORKLOAD_SECRET`, admin credentials, and most `CARACAL_*` values do not propagate. `CARACAL_ENV` is retained because it identifies the deployment environment. ## Credential Lifetime `caracal run` obtains credentials once at startup and does not renew them. Use it for bounded processes whose work completes before the injected credentials expire. Use a Caracal SDK when a service must re-exchange credentials, manage Sessions, or continue indefinitely. Every manifest fetch and credential release is correlated in audit. Rotating the workload secret blocks later launches that still use the prior value. Revoking Caracal authority does not recall a credential that was already injected: a static provider key remains valid until rotated at the provider, and a brokered token until its own expiry. ## Signals and Exit Results The runtime forwards `SIGINT`, `SIGTERM`, `SIGHUP`, and `SIGQUIT`, waits for the child, and does not leave it running. | Exit result | Meaning | | --------------- | --------------------------------------------------------------------- | | Child exit code | The child started and its result is returned unchanged. | | `1` | Configuration, exchange, validation, or Approval failed before spawn. | | `127` | The command could not be spawned. | | `128 + N` | The child ended because of signal number `N`. | ## Next Step [Inspect Diagnostics and Audit](/v1.0/runtime-console/observability/). --- # Manage Product Objects # URL: https://docs.caracal.run/v1.0/runtime-console/admin/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console/admin.md # Type: workflow # Concepts: # Requires: --- Use the console for human management after guided setup. Keep work inside the active ordinary zone. ## Maintain the Access Chain ```text application + provider + resource + active policy → authorized access ``` | Area | Operator task | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------ | | **Applications** | Register managed application identities and rotate or reveal client secrets. | | **Providers** | Configure the upstream credential mode and rotate provider secrets. | | **Resources** | Define `resource://` identifiers, operations/scopes, upstream URLs, Gateway application bindings, and provider bindings. | | **Policies** | Author and validate policy versions. | | **Policy Sets** | Compose, simulate, and activate the policy that STS evaluates. | | **Services → Launcher** | Create workload identities and launch bindings. | | **Services → Control** | Enable scoped remote automation and manage Control credentials. | Follow [Author Policy Data](/v1.0/guides/author-policy/) and [Activate a Policy Set](/v1.0/guides/activate-policy-set/) for the canonical policy workflow rather than duplicating it here. ## Treat Secrets as Audited Operations Application and workload secrets are sealed after creation. A reveal action retrieves the current value from Secret Store custody and writes an admin-audit record. A rotation invalidates the previous value. Copy the replacement directly into the consuming secret store; do not place it in source, screenshots, chat, or audit annotations. Provider secret fields remain masked and are accepted only by the supported credential create or rotation flow. ## Hand Off to Automation Use the Admin SDK for trusted direct management clients. Use **Services → Control** when remote automation needs a zone-bound credential and the optional Control endpoint is enabled. Control credentials are scoped to management nouns and verbs. Control calls are replay-protected, rate-limited, and audited. They do not receive the root admin token. Control is not a top-level `caracal` command and does not manage stack lifecycle. See [Automate Management](/v1.0/services/control/) for dependency and failure implications, and [Use the Admin API](/v1.0/api/control-plane/) for endpoint reference. ## System Zone The reserved system-zone viewer is read-only. It exists to expose Caracal-owned state for transparency, not as a management target. Return to an ordinary selected zone before creating, editing, revealing, rotating, deciding, or deleting anything. ## Next Step [Manage Runtime Authority](/v1.0/runtime-console/agents/) for Subjects, Authority records, Sessions, Delegations, and Approvals. --- # Inspect Diagnostics and Audit # URL: https://docs.caracal.run/v1.0/runtime-console/observability/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console/observability.md # Type: workflow # Concepts: # Requires: --- Start at the dashboard, then choose Diagnostics for platform posture or Audit for a specific decision. ## Read the Dashboard The selected-zone dashboard summarizes product readiness, pending Approvals, object counts, and recent activity. Use it to decide which detail workspace to open next. A count is navigation context, not proof of correctness; verify policy with simulation and verify a real request in Audit. ## Run Diagnostics **Diagnostics** runs shared Doctor checks across: | Group | What it establishes | | --------- | -------------------------------------------------------------------------------- | | Health | API reachability, management authentication, and clock alignment. | | Readiness | Service readiness and operator metrics for STS, Gateway, Audit, and Coordinator. | | Zones | Visible zones, resources, active policy state, and audit queryability. | | Preflight | Local secrets, keys, TLS material, Postgres, and Redis reachability. | `caracal status --ready` answers whether the stack can accept work. Diagnostics goes deeper: policy compilation, audit integrity, event backlog, and clock skew. Treat audit chain mismatch as a security incident. Treat dead outbox rows, DLQ entries, stale pending events, and large lag as propagation failures even if HTTP health still passes. ## Trace a Request 1. Open **Audit** in the correct zone. 2. Choose **Activity** for authority/resource events or **Admin** for management changes. 3. Filter by request ID when one is available. Otherwise narrow by time, decision, event type, application, Authority record, or Session. 4. Open the event group or decision trace. 5. Read the resource, Subject, requested scopes, determining policy, diagnostics, and result. Audit evidence and live state answer different questions. Audit shows what happened and why. Subjects, Sessions, Delegations, and Approvals show what authority remains live. Secret reveal and rotation actions appear in admin audit. Secret values do not belong in the event payload. ## Follow Dependency Failures | Symptom | First check | Then | | ------------------------------ | ------------------------------- | ---------------------------------------------- | | Management pages fail | API readiness and auth | Postgres, Redis, and API outbox | | Mandate issuance fails | STS readiness and policy state | Postgres, Redis invalidation, keys | | Gateway denies before upstream | Gateway trace and STS readiness | Binding, mandate, revocation, upstream safety | | Audit event is delayed | Audit readiness | Redis lag, pending entries, DLQ, replay volume | | Session state is stale | Coordinator readiness | Postgres, Redis, outbox, lease sweepers | Use [Troubleshoot by Symptom](/v1.0/operations/troubleshooting/) for the canonical symptom-first workflow. ## Next Step [Manage Runtime Authority](/v1.0/runtime-console/agents/). --- # Manage Runtime Authority # URL: https://docs.caracal.run/v1.0/runtime-console/agents/ # Markdown: https://docs.caracal.run/markdown/v1.0/runtime-console/agents.md # Type: workflow # Concepts: # Requires: --- Use these console workspaces when access is already configured and a person needs to understand or interrupt live authority. They are not runtime CLI workflows. ## Keep the Identities Distinct | Object | Meaning | Identifier relationship | | ---------------- | -------------------------------------------------- | ------------------------------------------------------------------------------- | | Console account | A human authenticated to operate the console. | Not a workload Subject. | | Application | A confidential client that asks STS for authority. | May act as itself or for a Federated user. | | Subject | The stable JWT `sub` an application acts for: the application itself or a Federated user. | Grouping key in the Subjects workspace. | | Authority record | One STS-issued authority anchor. | Its ID is carried as protocol `session_id`; it is not a Coordinator Session ID. | | Session | A governed execution owned by Coordinator. | Its ID is carried as `agent_session_id`. | | Delegation | Bounded authority from one Session to another. | Anchored by source and target Session IDs. | | Approval | A human-decision hold raised for a requested action. | May link to Subject, Authority record, Session, Resource, and scopes. | ## Investigate a Subject Open **Subjects** and pick a view: **All Subjects**, **Application Subjects**, or **Federated users**. Search by identity, sort by standing or recency, and open a Subject detail to see its Authority record history, governed Sessions, pending Approvals, and provider connections. Use the Subject kill switch for credential compromise or offboarding when individual cleanup would be unsafe. The implemented cascade revokes live Authority records, terminates linked Sessions, revokes Delegations, and revokes provider connections. Confirm the selected Subject and zone before acting. The kill switch ends authority that already exists; it does not disable the credential behind it. A Subject backed by an application can establish fresh authority on its next exchange, so containing a compromised application credential also requires rotating that application's secret from its detail panel in the same intervention. Every Subject is one of two kinds. An application Subject appears when the application exchanges as itself. A Federated user appears when the application exchanges that user's identity token from a registered Federated user issuer; the console records the issuer but never authenticates those users. ## Intervene in Sessions Open **Sessions** to filter active, suspended, or terminated executions and inspect their tree, lifecycle, leases, invocations, Authority linkage, and inbound or outbound Delegations. * **Suspend** pauses an active Session subtree. * **Resume** returns a suspended subtree to active operation. * **Terminate** is terminal and stops the Session subtree. Task Sessions expire by TTL. Service Sessions use heartbeat leases. A Session can reference an Authority record for attribution without changing the IDs into the same object. ## Review and Revoke Delegation The Sessions workspace exposes Delegation views and lineage. Inspect source and target Sessions, resource and scope bounds, constraints, expiry, and traversal state before revocation. Revocation ends the delegated authority represented by that edge. Requests must still pass the consuming verifier's revocation checks; use Audit to confirm denial after intervention. ## Decide Approvals **Approvals** lists holds in these implemented states: | State | Meaning | | ---------- | -------------------------------------------------------------------------------- | | `pending` | Waiting for an eligible approver until expiry. | | `approved` | An approver satisfied the hold; the requesting exchange has not consumed it yet. | | `consumed` | A retry used the approval; it cannot be reused. | | `rejected` | An approver denied the request. | | `expired` | The decision window ended. | A pending hold is decidable in the console only when its approver class is **Zone operator** or **Operator or federated user**. A **Federated user only** hold remains visible but has no console approve/reject action. These labels map to the policy data values `operator`, `any`, and `subject` in `approval_tiers`. Before deciding, inspect the requester, privacy mode, resource, scopes, Session lineage, expiry, and recent matching decisions. Approval does not itself issue a credential. The waiting client retries the mint, which consumes the hold exactly once. A second consumer receives an already-consumed failure. ## Correlate Every Intervention Use links from Subject, Session, Delegation, and Approval details into Audit. Filter by request ID, Authority record ID, or Session ID to confirm the original decision and the effect of a revoke, suspend, terminate, approve, or reject action. ## Related Pages * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) * [Session Delegation](/v1.0/concepts/delegation/) * [Approvals](/v1.0/concepts/approvals/) * [Inspect Diagnostics and Audit](/v1.0/runtime-console/observability/) --- # Choose an SDK or Package # URL: https://docs.caracal.run/v1.0/sdks/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks.md # Type: page # Concepts: # Requires: --- Use SDKs when you need package-level details after a guide. The [Guides](/v1.0/guides/) section shows task workflows; this section explains package names, install commands, public APIs, runtime requirements, and where each package fits. :::caution[Three separate boundaries] Application SDKs create and carry authority, the Admin package changes product state, and verification packages enforce inbound mandates. Do not use an application credential for administration or treat context propagation as authentication. ::: ## Choose by What You Are Building | I am building... | Start here | Then read | | ------------------------------------ | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | TypeScript app or agent workflow | [TypeScript SDK](./typescript/) | [OAuth Package](./oauth/), [Verify Package](./verify/) | | Python app, agent, or ASGI service | [Python SDK](./python/) | [ASGI Adapter](./adapters/asgi/), [FastMCP Adapter](./adapters/fastmcp/), [Verify Package](./verify/) | | Go service or agent workflow | [Go SDK](./go/) | [Go net/http Adapter](./adapters/nethttp/) | | Express resource server | [Express Adapter](./adapters/express/) | [Verify Package](./verify/), [Redis Revocation Store](./backends/redis/) | | FastAPI or Starlette resource server | [ASGI Adapter](./adapters/asgi/) | [Verify Package](./verify/), [Redis Revocation Store](./backends/redis/) | | FastMCP server | [FastMCP Adapter](./adapters/fastmcp/) | [Verify Package](./verify/) | | Custom verification boundary | [Verification Layer Overview](./verification-layer/) | [Identity Package](./identity/), [Revocation Package](./revocation/) | | Admin or provisioning automation | [Admin Package](./admin/) | [Use the Admin API](/v1.0/api/control-plane/) | | Shared revocation storage | [Redis Revocation Store](./backends/redis/) | [Revocation Package](./revocation/) | ## Package Map | Area | TypeScript / Node | Python | Go | | ---------------------- | ----------------------------- | ---------------------------- | -------------------------------------------------------------- | | App SDK | `@caracalai/sdk` | `caracalai-sdk` | `github.com/garudex-labs/caracal/packages/sdk/go` | | Identity verification | `@caracalai/identity` | `caracalai-identity` | `github.com/garudex-labs/caracal/packages/identity/go` | | OAuth token exchange | `@caracalai/oauth` | `caracalai-oauth` | `github.com/garudex-labs/caracal/packages/oauth/go` | | Revocation store | `@caracalai/revocation` | `caracalai-revocation` | `github.com/garudex-labs/caracal/packages/revocation/go` | | Verification engine | `@caracalai/verify` | `caracalai-verify` | `github.com/garudex-labs/caracal/packages/verify/go` | | Express adapter | `@caracalai/express` | Not applicable | Not applicable | | FastMCP adapter | `@caracalai/fastmcp` | `caracalai-fastmcp` | Not applicable | | ASGI adapter | Not applicable | `caracalai-asgi` | Not applicable | | net/http adapter | Not applicable | Not applicable | `github.com/garudex-labs/caracal/packages/adapters/nethttp/go` | | Redis revocation store | `@caracalai/revocation-redis` | `caracalai-revocation-redis` | `github.com/garudex-labs/caracal/packages/backends/redis/go` | ## Runtime Requirements | Ecosystem | Current package target | | --------- | --------------------------------------------------------------------------------------------------------------------- | | Node.js | Node `>=22` for published TypeScript packages that declare an engine. | | Python | Python `>=3.12` for published Python packages. | | Go | Module paths under `github.com/garudex-labs/caracal/packages/...`; current modules declare Go `1.26` where specified. | Each Go submodule has a path-prefixed release tag at the same commit as the product tag, so an exact Caracal SemVer resolves without a pseudo-version. Adapter availability is intentionally ecosystem-specific; parity covers behavior exposed by all three application facades, not identical framework packages. ## Related Guides * [Add SDK to Your App](/v1.0/get-started/add-sdk-to-your-app/) * [Make Runs Identifiable with Labels](/v1.0/tutorials/connect-an-agent/) * [Protect a Gateway-Routed HTTP API](/v1.0/guides/protect-gateway-http/) * [Implement Multi-Agent Delegation](/v1.0/guides/delegation/) --- # TypeScript SDK # URL: https://docs.caracal.run/v1.0/sdks/typescript/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/typescript.md # Type: page # Concepts: # Requires: --- `@caracalai/sdk` is the main TypeScript package for Session lifecycle, Delegation, Gateway routing, and Caracal context propagation. Use it in application code. Do not use it to create zones, applications, policies, resources, or grants; those operations belong to [Admin Package](/v1.0/sdks/admin/). ## Install ```bash npm install @caracalai/sdk ``` The package is ESM-only and targets Node `>=22`. To consume it from a CommonJS project, use a dynamic `await import('@caracalai/sdk')` or set `"type": "module"` in your `package.json`. ## Connect and Configure | API | Use it when | | ----------------------------------- | -------------------------------------------------------------------------------------------------------- | | `new Caracal()` | Use normal deployment configuration: exactly `CARACAL_CONFIG` when set, otherwise `CARACAL_*` variables. | | `Caracal.fromClientSecret(options)` | Supply one complete static client-secret configuration directly. | ```ts import { Caracal } from '@caracalai/sdk' const caracal = new Caracal() ``` The constructor never searches home directories or default profile paths. Multiple credential modes fail at startup instead of using precedence. Explicit environment mappings, profile loading, dynamic credential resolvers, custom transports, and raw configuration live in `@caracalai/sdk/advanced`. `Caracal.fromClientSecret` requires the static `zoneId`/`applicationId`/`clientSecret` triple. Resources are optional for per-resource application transports; Session and lifecycle operations fail clearly when no lifecycle resource audience is configured. It refreshes application subject tokens automatically. ## Make Your First Protected Call The smallest complete integration pins a transport to one resource and sends a request through the Gateway: ```ts import { Caracal } from '@caracalai/sdk' const caracal = new Caracal() const governedFetch = caracal.applicationTransport('resource://pipernet', { scopes: ['pipernet:read'], }) const target = caracal.gatewayRequest('resource://pipernet', '/reports') try { const response = await governedFetch(target.url, { method: 'GET' }) if (!response.ok) throw new Error(`protected call failed: ${response.status}`) console.log(await response.text()) } finally { await caracal.close() } ``` The zone, application, and resource behind this call come from your runtime profile; [Add SDK to Your App](/v1.0/get-started/add-sdk-to-your-app/) walks that setup end to end. The sections below group the client API by task. ## Run Work in Sessions | Method | Purpose | | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `session(fn, options?)` | Run `fn` inside a governed Session; `fn` receives the bound context, including its Session ID. Pass `authority: Authority.narrow(...)` to bound authority and `task` to describe the work. Retry protection is automatic; ordinary code should omit `idempotencyKey`. See [Safe Retries and Idempotency](/v1.0/guides/idempotency/) for durable-source redelivery. | | `startSession(options?)` | Start a governed Session that outlives a block; auto-renews its generation-fenced lease and returns a handle with `heartbeat()`, `deadlineAt`, `leaseGeneration`, and `close()`. Service lifetime is lease-only. Pass `authority: Authority.narrow(...)` with a positive Delegation TTL to bound the handle's authority. Retry protection is automatic. | | `attachSession(sessionId, options?)` | Re-attach to an active persisted long-lived Session after a restart: atomically acquires a new generation, fences older holders, and returns the same handle `startSession` does. | Session patterns for long-lived services and restart recovery are walked through in [Integrate the TypeScript SDK](/v1.0/guides/sdk-typescript/). ## Hand Off Authority Between Agents | Method | Purpose | | ---------------------------------------------- | ------- | | `delegate(options)` | Delegate a slice of the current Session's authority to an existing peer Session; returns the Delegation for the receiver to accept. A transient Coordinator failure is retried once under an idempotency key, so no duplicate Delegation is issued. | | `revokeDelegation(delegationId)` | Revoke a Delegation issued by this application. | | `acceptDelegation(delegationId, fn, options?)` | Present a received Delegation: bind a derived context carrying it while `fn` runs. Pass `{ validate: true }` to confirm with the Coordinator that the Delegation is live for the bound Session first. | The full hand-off workflow, including safe constraint choices, lives in [Implement Multi-Agent Delegation](/v1.0/guides/delegation/). ## Propagate Context Across Services | Method | Purpose | | ---------------------------------------------- | ------- | | `headers(options?)` | Project the current bound context into HTTP headers synchronously. | | `headersAsync(options?)` | Project headers when a root token may require async refresh. | | `bindFromHeaders(headers, fn, options?)` | Bind inbound Caracal envelope headers to the current async context. | ## Call Protected Resources | Method | Purpose | | ---------------------------------------------- | ------- | | `transport(options?)` | Return a fetch-compatible function that mints the `use=gateway` mandate from `scopes` and applies Gateway routing. Also accepts `approvalId` and `timeoutMs`. | | `applicationTransport(resourceId, options)` | Return a fetch pinned to one resource, calling as the application's own identity; provisions its own Session pair and Delegation. | | `mintMandate(resourceId, scopes, options?)` | Mint a cached resource mandate carrying the bound Session and Delegation; returns `{ token, expiresInSeconds }`. Requires client-secret credentials. | | `fetch(resourceId, path, init?)` | One-call Gateway request to a resource: builds the Gateway URL, mints the scoped mandate from `init.scopes`, and sends the request with context and authority injected. | Use `applicationTransport()` when your service calls as itself; use `transport()` inside a Session when work already runs under session or delegated authority. Propagation defaults to `"gateway-only"`; use `"always"` only for a known Caracal-aware direct service chain, and note that Gateway redirects are surfaced without automatic replay. Application-transport provisioning starts one source/target Session pair and one narrowed Delegation per cache key: a cold call costs four provisioning calls plus the final SDK mint and Gateway STS exchange, a warm call still performs both per-request STS exchanges, and `close()` retires backing Sessions best-effort. Wiring these transports into OpenAI, Anthropic, and other provider clients is covered in [Provider Recipes](/v1.0/guides/provider-recipes/#wire-the-transport-into-provider-clients). ## Act for Federated Users and Approvals | Method | Purpose | | ---------------------------------------------- | ------- | | `federateSubject(idToken, options?)` | Exchange a Federated user's identity token for an Authority record and return `{ subjectAuthorityRecordId, token, expiresInSeconds }`. Start attributed work with both `subjectAuthorityRecordId` and `subjectAuthorityRecordToken: token`; Coordinator verifies the signed proof before binding the record. This does not by itself propagate the Federated user's `sub` into later resource mandates. The returned mandate also remains the Federated user's credential for supported approval and exchange paths. | | `waitForApproval(approvalId, options?)` | Long-poll an approval raised by an approval-gated mint; returns the final `ApprovalState` (`approved`, `rejected`, `expired`, `consumed`, or `pending`). | | `withApproval(fn, options?)` | Run an approval-gated operation end to end: on `ApprovalRequiredError` the client waits for the decision and, once approved, invokes `fn` again with the approval id. | Approval gating is a policy feature; [Human Approval](/v1.0/guides/human-approval/) covers the tiers and the operator decision path. ## Build Requests and Manage the Client | Method | Purpose | | ---------------------------------------------- | ------- | | `gatewayRequest(resourceId, path?)` | Build a Gateway URL and `X-Caracal-Resource` header. | | `identity()` | The zone and application the client acts as, for logging and metric labels. | | `close()` | Terminally close the client: drop cached application mandates, invalidate credentials, and terminate application-transport Sessions best-effort. Repeated close is safe; later operations fail. | | `current()` | Return the currently bound `CaracalContext`, if present. | ## Context Propagation ```ts import { Authority } from '@caracalai/sdk' await caracal.session( async () => { await fetch('https://api.pipernet.example/reports', { headers: await caracal.headersAsync(), }) }, { authority: Authority.narrow(['pipernet:read'], { resourceId: 'resource://pipernet', constraints: { maxHops: 1, policyApproved: true, }, ttlSeconds: 600, }), }, ) ``` `DelegationConstraints` uses camelCase fields: `resources`, `maxDepth`, `maxHops`, `ttlSeconds`, `policyApproved`, `expiresAt`, and `broadReason`. `policyApproved` and `broadReason` are audit/display metadata, not authorization decisions. Gateway-bound requests sent through `transport()` carry the context envelope: W3C `traceparent`/`tracestate` plus `caracal.*` baggage entries for Session, Delegation, and Subject authority record correlation. These are visible correlation identifiers, never credentials. Direct non-Gateway requests omit the envelope by default. Use `propagation: "always"` only for a known Caracal-aware direct service chain. The bearer is attached only inside the configured Gateway origin and base path, and Gateway strips `caracal.*` baggage before forwarding upstream. Gateway-bound calls without `scopes` are valid only when the bound token is already a `use=gateway` mandate; lifecycle and resource tokens fail locally. ## Protect Inbound Requests Use `bindFromHeaders()` to propagate a Caracal context after an upstream Gateway, adapter, or verify-engine verifier has accepted the inbound mandate. Pass `{ verify }` to enforce the bearer token at the boundary itself. The callback must throw on failure and return a complete authoritative `VerifiedClaims` projection. Zone, application, and hop are required. Optional Session, Delegation, parent Delegation, and Subject authority record fields omitted from the result are authoritatively absent; they never fall back to unsigned caller baggage after verification. `caracal.contextMiddleware()` is the Express-style wrapper over this binding - it mounts with `app.use(...)`. It is not the same job as `caracalAuth` from [`@caracalai/express`](/v1.0/sdks/adapters/express/): `caracalAuth` **enforces** inbound mandates (401/403 before your handler); `contextMiddleware` **propagates or binds** context, enforcing only when you pass a `verify` callback. ```ts import { verify as verifyToken } from '@caracalai/identity' app.use( caracal.contextMiddleware({ verify: async (token) => { const claims = await verifyToken(token, { issuer: ISSUER, audience: AUDIENCE, zoneId: ZONE_ID, }) return { zoneId: claims.zoneId, applicationId: claims.clientId, sessionId: claims.sessionId, delegationId: claims.delegationId, subjectAuthorityRecordId: claims.authorityRecordId, hop: claims.hopCount ?? 0, } }, }), ) ``` Trace context and non-Caracal baggage remain propagation data. Use [Verification Layer Overview](/v1.0/sdks/verification-layer/) when the TypeScript service must also enforce revocation, scopes, targets, Session identity, or Delegation requirements. The advanced `caracalFastifyHook()` entrypoint is intended for Fastify's `onRequest` hook; ambient context is inherited by asynchronous work created during request dispatch, so detached background work must capture only the context it intentionally retains. Production propagation-only ingress must pass `{ trustedPropagation: true }` to state that an upstream Gateway or verifier already enforced the request. Omitting both `verify` and `trustedPropagation` fails closed in production. Header and transport helpers refuse to fall back to the application root token unless you pass `{ asApplication: true }`. Use that option only for trusted service-root ingress or setup calls. When inbound middleware injects application identity, caller-supplied Caracal authority baggage is discarded. Normal agent work should run inside `session()`, `delegate()`, or `bindFromHeaders()`. ## Errors and Observability STS denials throw `CaracalError` carrying `code`, `httpStatus`, and `requestId`, so callers branch on the machine-readable code instead of matching message text. Approval-gated exchanges throw `ApprovalRequiredError`, a `CaracalError` subclass that adds the approval fields, and coordinator failures throw `CoordinatorError` with `status`, `method`, and `path`. A client built on a `credentials` resolver throws `CredentialsUnavailableError` while the resolver returns no usable credential, so a not-yet-provisioned or expired identity fails closed rather than reaching the wire. ```ts import { Caracal, CaracalError } from '@caracalai/sdk' try { await caracal.mintMandate('resource://pipernet', ['pipernet:read']) } catch (err) { if (err instanceof CaracalError && err.code === 'access_denied') { console.warn(`denied by policy (request ${err.requestId})`) } } caracal.onEvent((event) => { metrics.timing(`caracal.${event.type}`, event.durationMs, { ok: event.ok }) }) ``` `onEvent(hook)` reports every control-plane operation: `token.exchange` (with `resources`, `scopes`, and `cached` for cache hits), `approval.wait` (with `approvalId` and the final `state`), `coordinator.call` (with `method`, `path`, and `status`), and `delegation.accept` (with `delegationId` and `sessionId`, so delegation presentations are auditable client-side). Each event carries `ok` and `durationMs`, ready to bridge into any metrics or tracing system. A hook that throws is ignored and never disturbs the operation that emitted the event, and the call returns a disposer that removes the hook. Errors the platform reports carry `isRetryable`, a hint that transport-level congestion and availability failures are worth retrying while policy denials are not. Bridging into Prometheus takes one hook; the same shape feeds an OpenTelemetry meter: ```ts import { Counter, Histogram } from 'prom-client' const operations = new Counter({ name: 'caracal_operations_total', help: 'Caracal control-plane operations', labelNames: ['type', 'ok'], }) const latency = new Histogram({ name: 'caracal_operation_duration_ms', help: 'Caracal control-plane operation latency', labelNames: ['type'], }) caracal.onEvent((event) => { operations.inc({ type: event.type, ok: String(event.ok) }) latency.observe({ type: event.type }, event.durationMs) }) ``` Operational warnings - lease loss, cleanup failures, an unverified inbound boundary in production - go to `console.warn` by default; pass `logger` in `CaracalConfig` to route them into your logging system instead. ## Retries and Cleanup Session creation retries transient Coordinator failures twice under one generated idempotency key; Delegation creation retries once. Reusing the key with different input is a conflict. STS exchange deliberately performs one network attempt because a lost response may already represent a minted one-shot mandate. Gateway transports do not replay redirects or request bodies. Use `session()` for bounded work. For long-lived work, close the `SessionHandle`, then call `await caracal.close()` during shutdown. `close()` is terminal and idempotent. A canceled local wait does not prove the remote operation did not commit. Approval-gated flows resolve in one call with `withApproval`, which encapsulates the catch-wait-retry dance: ```ts const mandate = await caracal.withApproval((approvalId) => caracal.mintMandate('resource://pipernet', ['pipernet:admin'], { approvalId }), ) ``` When a policy denies a mint for a session that carries no delegation, the error message appends a hint naming the cause - the session holds lifecycle-only authority - and the three remediations: `Authority.narrow`, `acceptDelegation`, or `applicationTransport`. ## Advanced Surface `@caracalai/sdk/advanced` is the low-level entrypoint for integrations that outgrow the facade: the envelope codec (`decodeEnvelope`, `encodeEnvelope`, header and baggage constants), bound-context plumbing (`bind`, `current`, `captureContext`), the raw Coordinator client (`startCoordinatorSession`, `acquireSessionLease`, `createDelegation`, `heartbeatSession`, `listInboundDelegations`, `terminateSession`), and the Session primitives (`session`, `startSession`, `attachSession`, `delegate`, and the context-deriving `acceptDelegation`) that take explicit inputs instead of client configuration. Everything in it is supported API; reach for it when building middleware, custom transports, or another layer's Caracal integration, and stay on `Caracal` for application code. ## Transport Security and Credential Handling Every control-plane client accepts a custom fetch through `fetchImpl` (on `CaracalConfig.coordinator` and `Caracal.fromClientSecret`), so deployments that require mutual TLS or a private CA toward the STS and coordinator inject a fetch bound to an `https.Agent` (for example via `undici.Agent` with client certificates) instead of patching globals. Production configuration refuses plaintext `http` control-plane URLs outside loopback; `CARACAL_ALLOW_INSECURE_CONFIG_URLS=true` overrides that gate and logs a warning banner at startup so the exception stays visible until TLS is in place. Tokens, client secrets, and minted mandates live in ordinary process memory for their lifetime: JavaScript strings are immutable and garbage-collected, so the SDK cannot zeroize them, and anything able to read the process heap (a debugger, a core dump, a compromised dependency) can read them. Keep secrets out of logs and error trackers - the SDK never logs them and caps error bodies for the same reason - rely on short token lifetimes and the `credentials` resolver for rotation, and treat heap access as full credential compromise in your threat model. ## Related Pages * [Integrate the TypeScript SDK](/v1.0/guides/sdk-typescript/) * [Verification Layer Overview](/v1.0/sdks/verification-layer/) * [Enforce, propagate, or attribute](/v1.0/concepts/authority-model/#enforce-propagate-or-attribute) * [Session Delegation](/v1.0/concepts/delegation/) --- # Python SDK # URL: https://docs.caracal.run/v1.0/sdks/python/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/python.md # Type: page # Concepts: # Requires: --- `caracalai-sdk` is the main Python package for async Session lifecycle, Delegation, Gateway routing, and ASGI context propagation. Use it in application code. Product management belongs to [Admin Package](/v1.0/sdks/admin/), and direct inbound enforcement belongs to an adapter or the verify package. ## Install ```bash pip install caracalai-sdk ``` The package requires Python `>=3.12`. ## Connect and Configure | API | Use it when | | --------------------------------- | -------------------------------------------------------------------------------------------------------- | | `Caracal()` | Use normal deployment configuration: exactly `CARACAL_CONFIG` when set, otherwise `CARACAL_*` variables. | | `Caracal.from_client_secret(...)` | Supply one complete static client-secret configuration directly. | ```python from caracalai import Caracal caracal = Caracal() ``` The constructor never searches home directories or default profile paths. Multiple credential modes fail at startup. Explicit environment mappings, profile loading, dynamic credential resolvers, and raw configuration live in `caracalai.advanced`. ## Make Your First Protected Call The smallest complete integration pins a transport to one resource and sends a request through the Gateway: ```python import asyncio from caracalai import Caracal async def main() -> None: caracal = Caracal() target = caracal.gateway_request("resource://pipernet", "/reports") try: async with caracal.application_transport( "resource://pipernet", scopes=["pipernet:read"], ) as governed: response = await governed.get(target.url) response.raise_for_status() print(response.text) finally: await caracal.aclose() asyncio.run(main()) ``` The zone, application, and resource behind this call come from your runtime profile; [Add SDK to Your App](/v1.0/get-started/add-sdk-to-your-app/) walks that setup end to end. The sections below group the client API by task. ## Run Work in Sessions | Method | Purpose | | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `async with caracal.session(...)` | Run the block inside a governed Session - a bounded identity Caracal establishes around whatever the block executes; pass `authority=Authority.narrow(...)` to bound its authority and `task=` to record what the Session is for. Retry protection is automatic; ordinary code should omit `idempotency_key`. See [Safe Retries and Idempotency](/v1.0/guides/idempotency/) for durable-source redelivery. | | `await caracal.start_session(...)` | Start a governed Session that outlives a block; auto-renews its generation-fenced lease and returns a handle with `heartbeat()`, `heartbeat_deadline_at`, `lease_generation`, and `aclose()`. Service lifetime is lease-only. Pass `authority=Authority.narrow(...)` with a positive Delegation TTL to bound the handle's authority. Retry protection is automatic. | | `await caracal.attach_session(session_id, ...)` | Re-attach to an active persisted long-lived Session after a restart: atomically acquires a new generation, fences older holders, and returns the same handle `start_session` does. | Session patterns for long-lived services and restart recovery are walked through in [Integrate the Python SDK](/v1.0/guides/sdk-python/). ## Hand Off Authority Between Agents | Method | Purpose | | --- | --- | | `await caracal.delegate(...)` | Delegate a slice of the bound Session's authority to an existing peer Session; returns the Delegation for the receiver to accept. A transient Coordinator failure is retried once under an idempotency key, so no duplicate Delegation is issued. | | `await caracal.revoke_delegation(delegation_id)` | Revoke a Delegation issued by this application. | | `async with caracal.accept_delegation(delegation_id, validate=False)` | Present a received Delegation: bind a derived context carrying it. Pass `validate=True` to confirm with the Coordinator that the Delegation is live for the bound Session first. | The full hand-off workflow, including safe constraint choices, lives in [Implement Multi-Agent Delegation](/v1.0/guides/delegation/). ## Propagate Context Across Services | Method | Purpose | | --- | --- | | `caracal.headers(as_application=False, ctx=None)` | Project the bound context (or an explicit `ctx`) into HTTP headers. | | `async with caracal.bind(ctx)` | Rebind a captured context into a new async task. | | `async with caracal.bind_from_headers(headers, as_application=False, verifier=None, trusted_propagation=False)` | Bind inbound Caracal envelope headers; pass `verifier=` to enforce the bearer token or `trusted_propagation=True` when an upstream boundary already enforced it. | ## Call Protected Resources | Method | Purpose | | --- | --- | | `caracal.transport(as_application=False, ctx=None, scopes=None, approval_id=None, propagation="gateway-only", **kwargs)` | Return an `httpx.AsyncClient` that mints the `use=gateway` mandate from `scopes=` and applies Gateway routing. Pass `ctx=` from thread pools or executors and `approval_id=` to consume an approved hold. | | `caracal.sync_transport(as_application=False, ctx=None, scopes=None, approval_id=None, **kwargs)` | Synchronous `httpx.Client` counterpart. | | `caracal.application_transport(resource_id, scopes=[...], approval_id=None, labels=None, mandate_ttl_seconds=None, **kwargs)` | Return an `httpx.AsyncClient` pinned to one resource, calling as the application's own identity; provisions its own Session pair and Delegation. `sync_application_transport(...)` is the synchronous counterpart. | | `await caracal.fetch(resource_id, path, ctx=None, scopes=None, ...)` | One-call Gateway request to a resource with context and authority injected. | | `caracal.gateway_request(resource_id, path="/")` | Build a Gateway URL and `X-Caracal-Resource` header. | | `caracal.mint_mandate(resource_id, scopes, ctx=None, ttl_seconds=None)` | Mint a cached resource mandate carrying the bound Session and Delegation; returns a `MintedMandate` with `token` and `expires_in_seconds`. Requires client-secret credentials. | Use `application_transport()` when your service calls as itself; use `transport()` inside a Session when work already runs under session or delegated authority. Propagation defaults to `"gateway-only"`; use `propagation="always"` only for a known Caracal-aware direct service chain, and note that automatic redirects are rejected because request mandates are single-use. Application-transport provisioning starts one source/target Session pair and one narrowed Delegation per cache key: a cold call costs four provisioning calls plus the final SDK mint and Gateway STS exchange, and a warm call still performs both per-request STS exchanges. Wiring these transports into OpenAI, Anthropic, and other provider clients is covered in [Provider Recipes](/v1.0/guides/provider-recipes/#wire-the-transport-into-provider-clients). ## Enforce Inbound Requests, Federated Users, and Approvals | Method | Purpose | | --- | --- | | `caracal.context_middleware(verifier=None)` | ASGI middleware factory: propagates context, and enforces at the boundary when a `verifier` is passed. | | `caracal.federate_subject(id_token, ttl_seconds=None)` | Exchange a Federated user's identity token for an Authority record and return a `FederatedSubject` with `subject_authority_record_id`, `token`, and `expires_in_seconds`. Start attributed work with both `subject_authority_record_id` and `subject_authority_record_token=token`; Coordinator verifies the signed proof before binding the record. This does not by itself propagate the Federated user's `sub` into later resource mandates. The returned mandate also remains the Federated user's credential for supported approval and exchange paths. | | `caracal.wait_for_approval(approval_id, timeout_seconds=300.0)` | Long-poll an approval raised by an approval-gated mint; returns the final `ApprovalState` (`approved`, `rejected`, `expired`, `consumed`, or `pending`). | | `await caracal.with_approval(fn, timeout_seconds=300.0)` | Run an approval-gated operation end to end: on `ApprovalRequired` the client waits for the decision and, once approved, awaits `fn` again with the approval id. | Approval gating is a policy feature; [Human Approval](/v1.0/guides/human-approval/) covers the tiers and the operator decision path. ## Client Lifecycle | Method | Purpose | | --- | --- | | `await caracal.aclose()` | Terminally close the client, owned HTTP pools, and application-transport Sessions best-effort. Repeated close is safe; later operations fail. | ## Context Propagation ```python from caracalai import DelegationConstraints constraints = DelegationConstraints( max_hops=1, policy_approved=True, ) ``` `DelegationConstraints` uses Python field names: `resources`, `max_depth`, `max_hops`, `ttl_seconds`, `policy_approved`, `expires_at`, and `broad_reason`. `policy_approved` and `broad_reason` are audit/display metadata, not authorization decisions. ## Protect Inbound Requests `context_middleware()` is framework-agnostic and runs on any ASGI app (FastAPI, Starlette, Quart, Django ASGI). Without a verifier it only **propagates**: it binds the inbound Caracal envelope into request context but does not check JWT signatures, audience, scopes, token use, or revocation. In production, pass `trusted_propagation=True` to state explicitly that a Gateway already enforced the mandate upstream; omitting both modes fails closed. Pass `verifier=` to **enforce at the boundary**. The callable receives the bearer token, must raise on failure, and must return a complete authoritative `VerifiedClaims` projection. Zone, application, and hop are required. Optional authority fields omitted from the projection are authoritatively absent and never fall back to unsigned caller baggage. The SDK never inspects token internals itself. ```python from caracalai_identity import verify_token from caracalai import Caracal, VerifiedClaims caracal = Caracal() app = FastAPI() async def verify(token: str) -> VerifiedClaims: claims = await verify_token( token, issuer=ISSUER, audience=AUDIENCE, expected_zone_id=ZONE_ID, ) return VerifiedClaims( zone_id=str(claims["zone_id"]), application_id=str(claims["client_id"]), session_id=str(claims["agent_session_id"]) if claims.get("agent_session_id") else None, delegation_id=str(claims["delegation_edge_id"]) if claims.get("delegation_edge_id") else None, subject_authority_record_id=str(claims["sid"]), hop=int(claims.get("hop_count") or 0), ) app.add_middleware(caracal.context_middleware(verifier=verify)) ``` Trace context and non-Caracal baggage remain propagation data. Middleware that uses `as_application=True` discards inbound Caracal authority baggage before binding the application credential. See [Enforce, propagate, or attribute](/v1.0/concepts/authority-model/#enforce-propagate-or-attribute) for which call path verifies authority. ## Errors and Observability STS denials raise typed subclasses of `CaracalError` (`AccessDenied`, `ScopeInsufficient`, `ZoneMismatch`, and the rest of the taxonomy), each carrying `code`, `http_status`, and `request_id`, so callers branch on the exception type instead of matching message text. Approval-gated exchanges raise `ApprovalRequired` with the approval fields, and coordinator failures raise `CoordinatorError` with `status`, `method`, and `path`. ```python from caracalai import AccessDenied, Caracal try: caracal.mint_mandate("resource://pipernet", ["pipernet:read"]) except AccessDenied as err: print(f"denied by policy (request {err.request_id})") caracal.on_event( lambda event: metrics.timing(f"caracal.{event.type}", event.duration_ms) ) ``` `on_event(hook)` reports every control-plane operation: `token.exchange` (with `resources`, `scopes`, and `cached` for cache hits), `approval.wait` (with `approval_id` and the final `state`), `coordinator.call` (with `method`, `path`, and `status`), and `delegation.accept` (with `delegation_id` and `session_id`, so delegation presentations are auditable client-side). Each event carries `ok` and `duration_ms`, ready to bridge into any metrics or tracing system. A hook that raises is ignored and never disturbs the operation that emitted the event, and the call returns a disposer that removes the hook. Errors the platform reports carry `is_retryable`, a hint that transport-level congestion and availability failures are worth retrying while policy denials are not. Bridging into Prometheus takes one hook; the same shape feeds an OpenTelemetry meter: ```python from prometheus_client import Counter, Histogram operations = Counter( "caracal_operations_total", "Caracal control-plane operations", ["type", "ok"] ) latency = Histogram( "caracal_operation_duration_ms", "Caracal control-plane operation latency", ["type"] ) caracal.on_event( lambda event: ( operations.labels(type=event.type, ok=str(event.ok)).inc(), latency.labels(type=event.type).observe(event.duration_ms), ) ) ``` ## Retries and Cleanup Session creation retries transient Coordinator failures twice with one generated idempotency key; Delegation creation retries once. STS exchange makes one network attempt, and Gateway transports do not replay redirects or request bodies. Use `async with caracal.session()` for bounded work. Close long-lived handles with `await handle.aclose()`, close owned synchronous `httpx` clients, and finish shutdown with `await caracal.aclose()`. Repeated facade close is safe; later operations fail. ## Advanced Surface `caracalai.advanced` is the low-level entrypoint for adapter authors and tests that deliberately own lifecycle wiring: envelope codecs, bound-context plumbing, raw Coordinator calls including `acquire_session_lease()`, middleware classes, and Session primitives with explicit dependencies. Application code should use `caracalai.Caracal`; advanced symbols are not duplicated in the package root. ## Transport Security and Credential Handling Every control-plane client accepts a custom HTTP client. `from_client_secret` takes the synchronous `http_client` used by STS exchange and the asynchronous `coordinator_http_client` used by lifecycle calls; the advanced `CoordinatorClient` also exposes its async client directly. Supply both from the same TLS/proxy policy when deployments require mutual TLS or a private CA. Standard `httpx` keyword arguments (`verify=`, `cert=`, `transport=`) configure data-plane transports. Production configuration refuses plaintext `http` control-plane URLs outside loopback; `CARACAL_ALLOW_INSECURE_CONFIG_URLS=true` overrides that gate and logs a warning banner at startup so the exception stays visible until TLS is in place. Tokens, client secrets, and minted mandates live in ordinary process memory for their lifetime: Python strings are immutable and garbage-collected, so the SDK cannot zeroize them, and anything able to read the process heap (a debugger, a core dump, a compromised dependency) can read them. Keep secrets out of logs and error trackers - the SDK never logs them and caps error bodies for the same reason - rely on short token lifetimes and the credentials resolver for rotation, and treat heap access as full credential compromise in your threat model. ## Related Pages * [Integrate the Python SDK](/v1.0/guides/sdk-python/) * [Protect a FastMCP App](/v1.0/guides/protect-fastmcp/) * [Verification Layer Overview](/v1.0/sdks/verification-layer/) * [Verify Package](/v1.0/sdks/verify/) --- # Go SDK # URL: https://docs.caracal.run/v1.0/sdks/go/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/go.md # Type: page # Concepts: # Requires: --- The Go SDK provides Session lifecycle, Delegation, Gateway routing, and context propagation for Go services. Use it for application authority, not management CRUD. Use [Admin Package](/v1.0/sdks/admin/) for control-plane automation. ## Install ```bash go get github.com/garudex-labs/caracal/packages/sdk/go ``` ## Connect and Configure ```go import caracal "github.com/garudex-labs/caracal/packages/sdk/go" client, err := caracal.New() if err != nil { return err } ``` `New()` loads exactly `CARACAL_CONFIG` when set; otherwise it loads `CARACAL_*` environment variables. It never searches home directories or default profile paths. Multiple credential modes fail at startup. | Constructor | Use it when | | ------------------------------------------------------- | ----------------------------------------------------------------- | | `New()` | Use normal deployment configuration. | | `FromClientSecret(options)` | Supply one complete static client-secret configuration directly. | | `FromEnv()`, `FromConfig(path)`, `FromCredentials(...)` | Advanced explicit loading, profile, and dynamic credential paths. | ## Make Your First Protected Call The smallest complete integration pins a transport to one resource and sends a request through the Gateway: ```go package main import ( "fmt" "io" caracal "github.com/garudex-labs/caracal/packages/sdk/go" ) func main() { client, err := caracal.New() if err != nil { panic(err) } defer client.Close() governed, err := client.ApplicationTransport(nil, "resource://pipernet", caracal.ApplicationTransportOptions{ Scopes: []string{"pipernet:read"}, }) if err != nil { panic(err) } target, err := client.GatewayRequest("resource://pipernet", "/reports") if err != nil { panic(err) } resp, err := governed.Get(target.URL) if err != nil { panic(err) } defer resp.Body.Close() if resp.StatusCode < 200 || resp.StatusCode >= 300 { panic(fmt.Sprintf("protected call failed: %s", resp.Status)) } body, _ := io.ReadAll(resp.Body) fmt.Println(string(body)) } ``` The zone, application, and resource behind this call come from your runtime profile; [Add SDK to Your App](/v1.0/get-started/add-sdk-to-your-app/) walks that setup end to end. The sections below group the client API by task. ## Run Work in Sessions | API | Purpose | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `client.Session(ctx, fn, opts...)` | Run `fn` inside a governed Session with a bound `context.Context`; set `SessionOptions.Authority` to bound its authority and `Task` to record what the Session is for. Retry protection is automatic; ordinary code should leave `IdempotencyKey` empty. See [Safe Retries and Idempotency](/v1.0/guides/idempotency/) for durable-source redelivery. | | `client.StartSession(ctx, opts...)` | Start a governed Session that outlives a block; auto-renews its generation-fenced lease and returns a `*SessionHandle` with `Heartbeat`, `DeadlineAt`, `LeaseGeneration`, and `Close`. Retry protection is automatic. | | `client.AttachSession(ctx, sessionID, opts...)` | Re-attach to an active persisted long-lived Session after a restart: atomically acquires a new generation, fences older holders, and returns the same handle `StartSession` does. | Session patterns for long-lived services and restart recovery are walked through in [Integrate the Go SDK](/v1.0/guides/sdk-go/). ## Hand Off Authority Between Agents | API | Purpose | | --- | --- | | `client.Delegate(ctx, opts)` | Delegate a slice of the current Session's authority to a peer Session; returns the Delegation for the receiver to accept. A transient Coordinator failure is retried once under an idempotency key, so no duplicate Delegation is issued. | | `client.RevokeDelegation(ctx, delegationID)` | Revoke a Delegation issued by this application. | | `client.AcceptDelegation(ctx, delegationID, opts...)` | Present a received Delegation: derive a context carrying it. Pass `AcceptDelegationOptions{Validate: true}` to confirm with the Coordinator that the Delegation is live for the bound Session first. | The full hand-off workflow, including safe constraint choices, lives in [Implement Multi-Agent Delegation](/v1.0/guides/delegation/). ## Propagate Context Across Services | API | Purpose | | --- | --- | | `client.Headers(ctx, opts...)` | Project the bound context to `http.Header`. | | `client.BindFromRequest(ctx, req, opts...)` | Bind inbound Caracal envelope headers. | ## Call Protected Resources | API | Purpose | | --- | --- | | `client.Transport(base, opts...)` | Return an `*http.Client` that mints the `use=gateway` mandate from `CallOptions.Scopes` and applies Gateway routing. Set `CallOptions.ApprovalID` to consume an approved hold. | | `client.ApplicationTransport(base, resourceID, opts)` | Return an `*http.Client` pinned to one resource, calling as the application's own identity; provisions its own Session pair and Delegation. Set `ApplicationTransportOptions.ApprovalID` to consume an approved hold. | | `client.MintMandate(ctx, resourceID, scopes, opts...)` | Mint a cached resource mandate carrying the bound Session and Delegation; returns an `oauth.MintedMandate` with `Token` and `ExpiresInSeconds`. Requires client-secret credentials. | Use `ApplicationTransport()` when your service calls as itself; use `Transport()` inside a Session when work already runs under session or delegated authority. Propagation defaults to `PropagationGatewayOnly`; use `PropagationAlways` only for a known Caracal-aware direct service chain, and note that Gateway redirects are surfaced without automatically replaying the mandate. Application-transport provisioning starts one source/target Session pair and one narrowed Delegation per cache key: a cold call costs four provisioning calls plus the final SDK mint and Gateway STS exchange, a warm call still performs both per-request STS exchanges, and `Close()` retires backing Sessions best-effort. Wiring these transports into OpenAI, Anthropic, and other provider clients is covered in [Provider Recipes](/v1.0/guides/provider-recipes/#wire-the-transport-into-provider-clients). ## Act for Federated Users and Approvals | API | Purpose | | --- | --- | | `client.FederateSubject(ctx, idToken, opts...)` | Exchange a Federated user's identity token for an Authority record and return a `FederatedSubject` with `SubjectAuthorityRecordID`, `Token`, and `ExpiresInSeconds`. Start attributed work with both `SubjectAuthorityRecordID` and `SubjectAuthorityRecordToken: Token`; Coordinator verifies the signed proof before binding the record. This does not by itself propagate the Federated user's `sub` into later resource mandates. The returned mandate also remains the Federated user's credential for supported approval and exchange paths. | | `client.WaitForApproval(ctx, approvalID, timeout)` | Long-poll an approval raised by an approval-gated mint; returns the final `oauth.ApprovalState` (`ApprovalApproved`, `ApprovalRejected`, `ApprovalExpired`, `ApprovalConsumed`, or `ApprovalPending`). | | `sdk.WithApproval(ctx, client, timeout, fn)` | Run an approval-gated operation end to end: on `*oauth.ApprovalRequiredError` the client waits for the decision and, once approved, invokes `fn` again with the approval id. | Approval gating is a policy feature; [Human Approval](/v1.0/guides/human-approval/) covers the tiers and the operator decision path. ## Build Requests and Manage the Client | API | Purpose | | --- | --- | | `client.GatewayRequest(resourceID, path)` | Build explicit Gateway routing metadata. | | `client.Identity(ctx)` | The zone and application the client acts as, for logging and metric labels. | | `client.Current(ctx)` | Inspect the current Caracal context. | | `client.Close()` | Terminally close the client and application-transport Sessions best-effort. Repeated close is safe; later operations fail. | ## Context Propagation ```go constraints := &caracal.DelegationConstraints{ MaxHops: 1, PolicyApproved: true, } ``` Available fields are `Resources`, `MaxDepth`, `MaxHops`, `TTLSeconds`, `PolicyApproved`, `ExpiresAt`, and `BroadReason`. `PolicyApproved` and `BroadReason` are audit/display metadata, not authorization decisions. Delegation constraints do not implement usage accounting; enforce consumable quotas in a durable domain transaction. ## Protect Inbound Requests Use `BindFromRequest()` to propagate a Caracal context after an upstream Gateway, adapter, or verify-engine verifier has accepted the inbound mandate. Set `CallOptions.Verify` to enforce the bearer token at the boundary itself. The callback receives the inbound token, returns an error on failure, and must return a non-nil complete `*VerifiedClaims` projection on success. `ZoneID`, `ApplicationID`, and `Hop` are authoritative. Empty optional authority fields are authoritatively absent and never fall back to unsigned caller baggage. In production propagation-only mode, set `TrustedPropagation: true` to state that an upstream boundary already enforced the request; omitting both modes fails closed. Use [Verification Layer Overview](/v1.0/sdks/verification-layer/) when the Go service must enforce mandate signatures, scopes, targets, Session identity, Delegation, or revocation at its own boundary. Trace context and non-Caracal baggage remain propagation data. `Headers` and `Transport` require a bound Caracal context by default. Pass `caracal.CallOptions{AsApplication: true}` only when the call should intentionally use the application subject token; inbound application binding discards caller-supplied Caracal authority baggage. ## Errors and Observability STS denials return `*oauth.CaracalError` carrying `Code`, `HTTPStatus`, and `RequestID`, so callers branch with `errors.As` instead of matching message text. Approval-gated exchanges return `*oauth.ApprovalRequiredError` with the approval fields, and coordinator failures return `*caracal.CoordinatorError` with `StatusCode`, `Method`, and `Path`. ```go import oauth "github.com/garudex-labs/caracal/packages/oauth/go" _, err := client.MintMandate(ctx, "resource://pipernet", []string{"pipernet:read"}) var denied *oauth.CaracalError if errors.As(err, &denied) && denied.Code == "access_denied" { log.Printf("denied by policy (request %s)", denied.RequestID) } client.OnEvent(func(event oauth.Event) { metrics.Timing("caracal."+event.Type, event.Duration, event.Ok) }) ``` `OnEvent(hook)` reports every control-plane operation: `token.exchange` (with `Resources`, `Scopes`, and `Cached` for cache hits), `approval.wait` (with `ApprovalID` and the final `State`), `coordinator.call` (with `Method`, `Path`, and `Status`), and `delegation.accept` (with `DelegationID` and `SessionID`, so delegation presentations are auditable client-side). Each event carries `Ok` and `Duration`, ready to bridge into any metrics or tracing system. A hook that panics is recovered and never disturbs the operation that emitted the event, and the call returns a disposer that removes the hook. Errors the platform reports carry `Retryable()`, a hint that transport-level congestion and availability failures are worth retrying while policy denials are not. Bridging into Prometheus takes one hook; the same shape feeds an OpenTelemetry meter: ```go operations := prometheus.NewCounterVec(prometheus.CounterOpts{ Name: "caracal_operations_total", Help: "Caracal control-plane operations", }, []string{"type", "ok"}) latency := prometheus.NewHistogramVec(prometheus.HistogramOpts{ Name: "caracal_operation_duration_ms", Help: "Caracal control-plane operation latency", }, []string{"type"}) client.OnEvent(func(event oauth.Event) { operations.WithLabelValues(event.Type, strconv.FormatBool(event.Ok)).Inc() latency.WithLabelValues(event.Type).Observe(float64(event.Duration.Milliseconds())) }) ``` The Go module ships as a single flat package: the envelope codec, context plumbing, Coordinator functions (`StartCoordinatorSession`, `AcquireSessionLease`, `CreateDelegation`, `HeartbeatSession`, `ListInboundDelegations`, `TerminateSession`), and Session primitives (`Session`, `StartSession`, `AttachSession`, `Delegate`) sit beside the `Caracal` client rather than behind a separate advanced entrypoint. ## Retries and Cleanup Session creation retries transient Coordinator failures twice under one generated idempotency key; Delegation creation retries once. STS exchange and Gateway requests are not automatically replayed. Always use the callback context passed to `Session`, close long-lived handles, and call `Close()` during shutdown. `Close()` is idempotent and terminal. ## Transport Security and Credential Handling Every control-plane client accepts a custom `*http.Client` - `HTTPClient` on `CoordinatorClient` and `ClientSecretOptions`, and the `base` argument of `Transport` - so deployments that require mutual TLS or a private CA toward the STS and coordinator inject a client with a `tls.Config` carrying certificates instead of patching defaults. Production configuration refuses plaintext `http` control-plane URLs outside loopback; `CARACAL_ALLOW_INSECURE_CONFIG_URLS=true` overrides that gate and logs a warning banner at startup so the exception stays visible until TLS is in place. Tokens, client secrets, and minted mandates live in ordinary process memory for their lifetime: Go strings are immutable and garbage-collected, so the SDK cannot zeroize them, and anything able to read the process heap (a debugger, a core dump, a compromised dependency) can read them. Keep secrets out of logs and error trackers - the SDK never logs them and caps error bodies for the same reason - rely on short token lifetimes and the credentials resolver for rotation, and treat heap access as full credential compromise in your threat model. ## Related Pages * [Integrate the Go SDK](/v1.0/guides/sdk-go/) * [Verification Layer Overview](/v1.0/sdks/verification-layer/) * [Enforce, propagate, or attribute](/v1.0/concepts/authority-model/#enforce-propagate-or-attribute) * [Protect a Go net/http Service](/v1.0/guides/protect-nethttp/) * [Session Delegation](/v1.0/concepts/delegation/) --- # Verification Layer Overview # URL: https://docs.caracal.run/v1.0/sdks/verification-layer/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/verification-layer.md # Type: page # Concepts: # Requires: --- Use this page when you are protecting an inbound resource-server boundary and need to choose the right package layer. Start with the highest-level adapter that fits your framework; use lower-level packages only when you are building a custom boundary. The verification layer consumes `use=resource` mandates at a resource server. Gateway ingress consumes `use=gateway` mandates instead. A Session or lifecycle mandate must not be accepted at either boundary. The classes are defined in [Mandate Use Classes](/v1.0/concepts/mandate/#mandate-use-classes). ## Which Layer Should I Use? | Need | Use | | --------------------------------------------------------- | ------------------------------------------------------------------------------------ | | Express route middleware | [Express Adapter](/v1.0/sdks/adapters/express/) | | FastAPI/Starlette (ASGI) middleware | [ASGI Adapter](/v1.0/sdks/adapters/asgi/) | | FastMCP server or tool authentication | [FastMCP Adapter](/v1.0/sdks/adapters/fastmcp/) | | Go `net/http` middleware | [Go net/http Adapter](/v1.0/sdks/adapters/nethttp/) | | Framework-neutral bearer parsing and mandate verification | [Verify Package](/v1.0/sdks/verify/) | | Custom JWT claim verification | [Identity Package](/v1.0/sdks/identity/) | | Shared revocation checks | [Revocation Package](/v1.0/sdks/revocation/) plus [Redis Revocation Store](/v1.0/sdks/backends/redis/) | ## Framework Adapters Adapters bind the shared verify and identity packages to common server frameworks. They should reject failed requests before your handler or tool runs, attach verified claims to framework context, and preserve the same 401/403 behavior across languages. Use adapters first when your framework is supported. They reduce boilerplate and keep error mapping consistent with the rest of Caracal. ## Verify Package [Verify Package](/v1.0/sdks/verify/) is the reusable verification engine under the adapters. Use it when your framework is unsupported or when you need direct control over bearer parsing, verifier defaults, route-level scopes, targets, Session requirements, Delegation requirements, hop limits, and safe error hints. ## Identity Package [Identity Package](/v1.0/sdks/identity/) verifies mandate JWT claims directly. Use it when you are composing a custom verifier or adapter. It does not provide the full verify-engine error mapping or revocation-store integration by itself. ## Revocation and Shared State Resource servers must reject mandates anchored to a revoked Authority record ID, Root authority record ID, Session ID, or Delegation ID. Use in-memory revocation stores for local development only. Use [Redis Revocation Store](/v1.0/sdks/backends/redis/) for multi-instance resource servers that consume the protocol stream `caracal.sessions.revoke`. ## Failure Behavior All HTTP verification layers preserve one status contract - `401` when the credential itself was not accepted, `403` when a verified mandate lacks the route's required authority. The canonical per-code mapping and the shared status function live in [Framework Adapters](/v1.0/sdks/adapters/#boundary-semantics); the code catalog is in [Error Codes](/v1.0/reference/errors/). JWT verification establishes signed claims; revocation checks establish whether their authority remains active. Both are required where revocation-before-expiry is part of the resource's security contract. ## Related Pages * [Protect an MCP Server](/v1.0/guides/protect-mcp/) * [Protect an Express App](/v1.0/guides/protect-express/) * [Protect a FastAPI App](/v1.0/guides/protect-fastapi/) * [Protect a FastMCP App](/v1.0/guides/protect-fastmcp/) * [Protect a Go net/http Service](/v1.0/guides/protect-nethttp/) --- # Framework Adapters # URL: https://docs.caracal.run/v1.0/sdks/adapters/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/adapters.md # Type: page # Concepts: # Requires: --- Framework adapters bind the lower-level identity and verify packages to common server frameworks. Use them before reaching for lower-level verification APIs. Adapters are inbound enforcement only. They do not start Sessions, mint mandates, proxy to Gateway, or manage product state. ## Adapter Map | Adapter | Package | Use it for | | --- | --- | --- | | [Express](/v1.0/sdks/adapters/express/) | `@caracalai/express` | Protecting Express 5 routes with Caracal mandate verification. | | [ASGI](/v1.0/sdks/adapters/asgi/) | `caracalai-asgi` | Protecting FastAPI, Starlette, and other Python ASGI apps with Caracal mandate verification. | | [FastMCP](/v1.0/sdks/adapters/fastmcp/) | `@caracalai/fastmcp`, `caracalai-fastmcp` | Verifying FastMCP bearer tokens before tool execution. | | [Go net/http](/v1.0/sdks/adapters/nethttp/) | `github.com/garudex-labs/caracal/packages/adapters/nethttp/go` | Protecting Go HTTP handlers. | No adapter fits, or you need a custom boundary? Route through [Verification Layer Overview](/v1.0/sdks/verification-layer/) to choose between the verify engine, identity package, and revocation stores. ## Boundary Semantics Every HTTP adapter maps verification failures through one canonical status function in `@caracalai/verify` (`httpStatusForAuthError` in TypeScript, `verify.HTTPStatus` in Go), so the boundary behaves identically across frameworks and languages: * **401** - the credential itself was not accepted: `missing_token`, `invalid_token`, `invalid_zone`, `session_revoked`, `delegation_stale`. * **403** - the mandate verified but its authority is insufficient for the route: `insufficient_scope`, `session_required`, `delegation_required`, `chain_mismatch`, `hop_count_exceeded`. Adapters never re-derive these status codes; they consume the shared mapping. All adapters require trusted issuer, audience, zone, and route-authority requirements. In-memory revocation is suitable only for tests or one process; replicated deployments need a shared backend. ## Related State Backends * [Redis Revocation Store](/v1.0/sdks/backends/redis/) ## Related Guides * [Verification Layer Overview](/v1.0/sdks/verification-layer/) * [Protect an MCP Server](/v1.0/guides/protect-mcp/) * [Protect an Express App](/v1.0/guides/protect-express/) * [Protect a FastAPI App](/v1.0/guides/protect-fastapi/) * [Protect a FastMCP App](/v1.0/guides/protect-fastmcp/) * [Protect a Go net/http Service](/v1.0/guides/protect-nethttp/) --- # Express Adapter # URL: https://docs.caracal.run/v1.0/sdks/adapters/express/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/adapters/express.md # Type: page # Concepts: # Requires: --- `@caracalai/express` protects Express routes by parsing the bearer token, verifying the mandate through `@caracalai/verify`, and attaching Caracal claims to the request. Use it for Express 5 ingress. Do not use it as outbound SDK middleware or as a replacement for Gateway. ## Install ```bash npm install @caracalai/express @caracalai/verify @caracalai/revocation-redis ``` The adapter has an Express `^5.0.0` peer dependency and targets Node `>=22`. ## Middleware ```ts import express from 'express' import { caracalAuth } from '@caracalai/express' import { createMandateVerifier } from '@caracalai/verify' import { RedisRevocationStore } from '@caracalai/revocation-redis' const app = express() const verifier = createMandateVerifier({ issuer: 'https://sts.pipernet.example', audience: 'resource://pipernet', zoneId: '0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f', revocations: new RedisRevocationStore(redis), }) app.use( '/mcp', caracalAuth( { verifier }, { requiredScopes: ['mcp:tool:call'], requiredTargets: ['resource://pipernet'], requireSession: true, }, ), ) ``` ## Request shape The middleware attaches Caracal claims to `req.caracal` and `req.caracalClaims` when verification succeeds. Use the exported `CaracalRequest` type when a handler needs typed access. `caracalAuth(options, routeOverrides?)` is the public middleware entry point. Pass either verify dependencies or `{ verifier }`. `bindContext` defaults to true and also sets `req.caracalContext`; set it false only when ambient SDK context is intentionally unwanted. ```ts import type { CaracalRequest } from '@caracalai/express' app.post('/mcp/tools/search', (req: CaracalRequest, res) => { res.json({ subject: req.caracal?.sub }) }) ``` ## Failure behavior Failed verification returns a verify-engine error code such as `missing_token`, `invalid_token`, `insufficient_scope`, or `session_revoked`, plus a safe `error_hint` field. Pair the adapter with a shared revocation store when multiple resource-server instances serve the same resource. ## Related Pages * [Protect an Express App](/v1.0/guides/protect-express/) * [Verify Package](/v1.0/sdks/verify/) * [Redis Revocation Store](/v1.0/sdks/backends/redis/) --- # ASGI Adapter # URL: https://docs.caracal.run/v1.0/sdks/adapters/asgi/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/adapters/asgi.md # Type: page # Concepts: # Requires: --- The ASGI adapter protects any Python ASGI application - FastAPI, Starlette, Quart, Django ASGI - with fail-closed Caracal mandate verification. It is pure ASGI: it imports no web framework and delegates every check to [Verify Package](/v1.0/sdks/verify/). `CaracalASGIAuth` is the only public package entry point. Use it for inbound ASGI HTTP or WebSocket authentication, not outbound SDK calls. ## Install ```bash pip install caracalai-asgi ``` ## Add middleware ```python from caracalai_asgi import CaracalASGIAuth from caracalai_revocation import InMemoryRevocationStore from fastapi import FastAPI app = FastAPI() app.add_middleware( CaracalASGIAuth, audience="resource://pipernet", revocations=InMemoryRevocationStore(), required_scopes=["pipernet:read"], routes={ "/payouts": {"required_scopes": ["pipernet:payout"], "require_delegation": True}, }, exclude=["/healthz"], ) @app.get("/balances") async def balances(request): principal = request.state.caracal return {"sub": principal.sub, "scopes": principal.scope} ``` `issuer` defaults to `CARACAL_STS_URL` and `expected_zone_id` to `CARACAL_ZONE_ID`, so a provider deployed with the standard Caracal workload variables only states its own audience and revocation store. Construction fails if no issuer or zone can be resolved. ## Options | Option | Use it for | | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | `audience` | The provider's own resource identifier; mandates minted for other resources are rejected. | | `revocations` | Revocation store consulted on every request. Use [Redis Revocation Store](/v1.0/sdks/backends/redis/) in production. | | `required_scopes` / `required_targets` | Default scope and target requirements for every route. | | `require_session` / `require_delegation` / `max_hop_count` | Session identity, delegated authority, and Delegation-depth requirements. | | `routes` | Per-route overrides by path prefix; the longest matching prefix wins. Any option above can be overridden. | | `exclude` | Path prefixes served without verification (health and readiness probes). | `required_use` defaults to `resource`; `require_session` and `require_delegation` default to false. Exclusions and route prefixes match path-segment boundaries, and the longest route prefix wins. ## Behavior * Verified claims are stored as `scope["state"]["caracal"]`, surfaced as `request.state.caracal` in Starlette and FastAPI. * Failed verification answers with the shared status mapping (`http_status_for_auth_error`): 401 for credential failures, 403 for insufficient authority, with the standard `error`/`error_description` JSON body. * WebSocket connections that fail verification are closed with policy code `1008`; lifespan events pass through. * Call `await middleware.warmup()` at startup to prefetch the zone JWKS before the first request. ## Boundary The adapter verifies inbound mandates; it does not create Sessions or Delegations. Use the [Python SDK](/v1.0/sdks/python/) to create Caracal context before making outbound calls. ## Related Pages * [Protect a FastAPI App](/v1.0/guides/protect-fastapi/) * [Verify Package](/v1.0/sdks/verify/) * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) --- # FastMCP Adapter # URL: https://docs.caracal.run/v1.0/sdks/adapters/fastmcp/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/adapters/fastmcp.md # Type: page # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' FastMCP adapters expose small verifier APIs that delegate to the verify packages. Use them when your FastMCP integration needs to authenticate a bearer token before running a tool. They verify one token and return a small principal projection; they do not install server lifecycle hooks, create Sessions, or mint outbound mandates. ## Install | Ecosystem | Package | | ---------- | ------------------------------------------------------------------------ | | TypeScript | `npm install @caracalai/fastmcp @caracalai/verify @caracalai/revocation` | | Python | `pip install caracalai-fastmcp` | ## Verify a Token ```ts import { extractBearer, verifyFastMcpToken } from '@caracalai/fastmcp' import { createMandateVerifier } from '@caracalai/verify' import { InMemoryRevocationStore } from '@caracalai/revocation' const verifier = createMandateVerifier({ issuer: 'https://sts.pipernet.example', audience: 'resource://pipernet', zoneId: '0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f', revocations: new InMemoryRevocationStore(), }) const token = extractBearer(request.headers.get('authorization') ?? '') if (!token) throw new Error('missing bearer token') const context = await verifyFastMcpToken(token, verifier, { requiredScopes: ['mcp:tool:call'], requiredTargets: ['resource://pipernet'], requireSession: true, }) console.log(context.sub, context.zoneId, context.scope) ``` `verifyFastMcpToken()` returns `{ sub, zoneId, scope }` or throws `FastMcpAuthError`. The public entries are `verifyFastMcpToken`, `extractBearer`, and `FastMcpAuthError`. ```python from caracalai_fastmcp import CaracalAuth, CaracalAuthError auth = CaracalAuth( issuer="https://sts.pipernet.example", audience="resource://pipernet", zone_id="0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f", required_scopes=["mcp:tool:call"], required_targets=["resource://pipernet"], require_session=True, revocations=revocations, ) try: context = await auth.verify_token(token) except CaracalAuthError as exc: raise RuntimeError(exc.code) from exc ``` The public entries are `CaracalAuth` and `CaracalAuthError`. There is no Go FastMCP adapter; use the framework-neutral Go verifier or the [net/http adapter](/v1.0/sdks/adapters/nethttp/). ## Boundary The adapter verifies tokens; it does not create Sessions or Delegations. Use the [Python SDK](/v1.0/sdks/python/) or [TypeScript SDK](/v1.0/sdks/typescript/) to create Caracal context before making outbound calls. ## Related Pages * [Protect a FastMCP App](/v1.0/guides/protect-fastmcp/) * [Verify Package](/v1.0/sdks/verify/) * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) --- # Go net/http Adapter # URL: https://docs.caracal.run/v1.0/sdks/adapters/nethttp/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/adapters/nethttp.md # Type: page # Concepts: # Requires: --- The Go net/http adapter wraps handlers with verify-engine verification and stores verified claims in the request context. Use it for inbound `net/http` handlers. It does not create outbound Caracal context or replace the Go application SDK. ## Install ```bash go get github.com/garudex-labs/caracal/packages/adapters/nethttp/go ``` ## Middleware ```go import ( "net/http" "time" nethttp "github.com/garudex-labs/caracal/packages/adapters/nethttp/go" revocation "github.com/garudex-labs/caracal/packages/revocation/go" verify "github.com/garudex-labs/caracal/packages/verify/go" ) revocations := revocation.NewInMemoryStore(24 * time.Hour) verifier := verify.NewVerifier(verify.Options{ Issuer: "https://sts.pipernet.example", Audience: "https://api.pipernet.example", ZoneID: "0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f", Revocations: revocations, }) handler := nethttp.VerifierMiddleware(verifier.Require(verify.Options{ RequiredScopes: []string{"pipernet:read"}, RequiredTargets: []string{"resource://pipernet"}, RequireSession: true, }))(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { claims, ok := nethttp.ClaimsFromContext(r.Context()) if !ok { http.Error(w, "missing claims", http.StatusUnauthorized) return } _, _ = w.Write([]byte(claims.Sub)) })) ``` ## APIs | API | Purpose | | --- | --- | | `Middleware(opts)` | Return middleware that verifies the bearer token and rejects failed requests. | | `VerifierMiddleware(verifier)` | Return middleware backed by a reusable verifier with shared defaults. | | `ClaimsFromContext(ctx)` | Retrieve verified Caracal claims inside a handler. | `Middleware` constructs a verifier from `verify.Options`; `VerifierMiddleware` reuses an existing verifier. A nil verifier produces a verifier with empty options and therefore cannot establish a properly configured production boundary; always pass trusted issuer, audience, zone, and revocation settings. ## Failure behavior The middleware maps verification errors to HTTP failures before the handler runs and includes a safe `error_hint` in JSON failures. Use a shared revocation store through `verify.Options` in production so revoked sessions are rejected consistently across service instances. ## Related Pages * [Protect a Go net/http Service](/v1.0/guides/protect-nethttp/) * [Go SDK](/v1.0/sdks/go/) * [Verify Package](/v1.0/sdks/verify/) --- # Verify Package # URL: https://docs.caracal.run/v1.0/sdks/verify/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/verify.md # Type: page # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' The verify packages authenticate Caracal mandates without tying the check to Express, FastMCP, or Go net/http. Framework adapters build on this engine. Use one long-lived verifier per trust boundary so JWKS and revocation state are reused. Do not use it to mint mandates or create Sessions. ## Install | Ecosystem | Package | | ---------- | ----------------------------------------------------------- | | TypeScript | `npm install @caracalai/verify` | | Python | `pip install caracalai-verify` | | Go | `go get github.com/garudex-labs/caracal/packages/verify/go` | ## Core APIs | API | Purpose | | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | `extractBearer` / `extract_bearer` / `ExtractBearer` | Parse a bearer token from an `Authorization` header. | | `createMandateVerifier` / `NewVerifier` | Build a reusable verifier with secure defaults, JWKS caching, revocation checks, and per-route overrides. | | `create_mandate_verifier` | Python reusable verifier with the same defaults, warmup, and per-route override model. | | `authenticate` / `Authenticate` | Verify one token against identity claims and revocation anchors. | | `authenticateRequest` / `unauthorizedResponse` | Authenticate a fetch-standard `Request` and render an `AuthError` as a JSON `Response` in WinterTC runtimes. | | `checkActiveAuthority` / `check_active_authority` / `CheckActiveAuthority` | Check expiry and revoked anchors for verified claims. | ## Verify One Request ```ts import { createMandateVerifier } from '@caracalai/verify' import { InMemoryRevocationStore } from '@caracalai/revocation' const verifier = createMandateVerifier({ issuer: 'https://sts.pipernet.example', audience: 'resource://pipernet', zoneId: '0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f', revocations: new InMemoryRevocationStore(), }) const result = await verifier.authorization(req.headers.authorization, { requiredScopes: ['mcp:tool:call'], requiredTargets: ['resource://pipernet'], requireSession: true, }) if (!result.ok) { throw new Error(`${result.error.code}: ${result.error.hint}`) } ``` Add route-level scopes or targets through `verifier.authorization(..., overrides)` or `verifier.require(overrides)`. ```python from caracalai_revocation import InMemoryRevocationStore from caracalai_verify import AuthOptions, create_mandate_verifier verifier = create_mandate_verifier( AuthOptions( issuer="https://sts.pipernet.example", audience="https://api.pipernet.example", expected_zone_id="0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f", revocations=InMemoryRevocationStore(), ) ) await verifier.warmup() result = await verifier.authorization( request.headers.get("authorization"), required_scopes=["pipernet:read"], required_targets=["resource://pipernet"], ) if not result.ok: raise PermissionError(f"{result.error.code}: {result.error.hint}") ``` Use one verifier per resource server. The verifier defaults to resource mandates, checks the STS issuer and audience, enforces zone, scope, target, Session, Delegation, and hop constraints, checks expiry, and queries the Authority record ID, Root authority record ID, Session ID, and Delegation ID parsed claims as revocation anchors. See the [parsed claim mapping](/v1.0/sdks/identity/#parsed-claim-names) for language-level and raw JWT names. ## Fetch-standard runtimes Runtimes built on WinterTC `Request`/`Response` globals - Node 18+, Deno, Bun, and edge workers - can authenticate without a framework adapter: ```ts import { authenticateRequest, unauthorizedResponse } from '@caracalai/verify' export default { async fetch(request: Request): Promise { const result = await authenticateRequest(request, deps) if (!result.ok) { return unauthorizedResponse(result.error) } return handle(request, result.principal) }, } ``` `unauthorizedResponse` maps error codes to 401 or 403 and renders the same JSON error body as the framework adapters. The same pair drops into any fetch-based framework. In Hono: ```ts app.use('/api/*', async (c, next) => { const result = await authenticateRequest(c.req.raw, deps) if (!result.ok) return unauthorizedResponse(result.error) c.set('principal', result.principal) await next() }) ``` And in a Next.js route handler: ```ts export async function GET(request: Request): Promise { const result = await authenticateRequest(request, deps) if (!result.ok) return unauthorizedResponse(result.error) return Response.json(await loadReports(result.principal)) } ``` Python and Go services cover this role with the [ASGI adapter](/v1.0/sdks/adapters/asgi/) and the [net/http adapter](/v1.0/sdks/adapters/nethttp/), which wrap their ecosystems' native request types. ## Error codes `authenticate` and reusable verifiers normalize failures into typed error codes: `missing_token`, `invalid_token`, `invalid_zone`, `insufficient_scope`, `session_revoked`, `delegation_stale`, `session_required`, `delegation_required`, `chain_mismatch`, and `hop_count_exceeded`. TypeScript, Python, and Go reusable verifiers include a safe debugging hint for operator logs and API error bodies. STS and Gateway report the zone and scope conditions with their own spellings, `zone_invalid` and `scope_insufficient`; see [Error Codes](/v1.0/reference/errors/) for the per-surface mapping. Authentication performs no retry of the protected operation. A JWKS or revocation-store failure fails verification rather than widening authority. Warm the verifier at process startup where the language API exposes `warmup`; close any caller-owned HTTP or Redis clients during service shutdown. ## Related Pages * [Verification Layer Overview](/v1.0/sdks/verification-layer/) * [Protect an MCP Server](/v1.0/guides/protect-mcp/) * [Express Adapter](/v1.0/sdks/adapters/express/) * [FastMCP Adapter](/v1.0/sdks/adapters/fastmcp/) * [Go net/http Adapter](/v1.0/sdks/adapters/nethttp/) --- # Identity Package # URL: https://docs.caracal.run/v1.0/sdks/identity/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/identity.md # Type: page # Concepts: # Requires: --- The identity packages verify Caracal mandate JWTs. Use them when you are building an adapter or custom resource-server boundary that accepts mandates directly. Do not use identity verification alone as a production authorization boundary when revocation must take effect before token expiry; compose it through [Verify Package](/v1.0/sdks/verify/) with a shared revocation store. ## Install | Ecosystem | Package | | ---------- | ------------------------------------------------------------- | | TypeScript | `npm install @caracalai/identity` | | Python | `pip install caracalai-identity` | | Go | `go get github.com/garudex-labs/caracal/packages/identity/go` | Node packages target Node `>=22`; Python packages require Python `>=3.12`. ## Verification inputs | Option | Meaning | | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `issuer` / `Issuer` | Expected STS issuer. | | `audience` / `Audience` | Expected audience for the mandate. | | `zoneId` / `expected_zone_id` / `ZoneID` | Required zone trust anchor. It fixes which zone's signing keyset verifies the token and must equal the `zone_id` claim, so key selection is never influenced by the unverified token. Verification fails closed when it is missing. | | `requiredScopes` / `required_scopes` / `RequiredScopes` | Scopes every accepted mandate must contain. | | `requiredTargets` / `required_targets` / `RequiredTargets` | Target resources every accepted mandate must include. | | `requiredUse` / `required_use` / `RequiredUse` | Token use, usually `resource`. | | `requireSession` / `require_session` / `RequireSession` | Require a governed Session identity. | | `requireDelegation` / `require_delegation` / `RequireDelegation` | Require a Delegation claim (`delegation_edge_id` on the wire). | | `requireChainContains` / `require_chain_contains` / `RequireChainContains` | Require an application in the delegation chain. | | `maxHopCount` / `max_hop_count` / `MaxHopCount` | Cap delegation chain depth. | ## TypeScript example ```ts import { verify } from '@caracalai/identity' const claims = await verify(token, { issuer: 'https://sts.pipernet.example', audience: 'https://api.pipernet.example', zoneId: '0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f', requiredScopes: ['pipernet:read'], requiredTargets: ['resource://pipernet'], }) ``` ## Parsed claim names The identity packages map raw JWT claims to canonical language-level fields: | Meaning | TypeScript | Python | Go | Raw JWT claim | | ------------------------ | ----------------------- | -------------------------- | ----------------------- | -------------------- | | Authority record ID | `authorityRecordId` | `authority_record_id` | `AuthorityRecordID` | `sid` | | Root authority record ID | `rootAuthorityRecordId` | `root_authority_record_id` | `RootAuthorityRecordID` | `root_sid` | | Session ID | `sessionId` | `session_id` | `SessionID` | `agent_session_id` | | Delegation ID | `delegationId` | `delegation_id` | `DelegationID` | `delegation_edge_id` | ## JWKS caching STS signing keysets are zone-scoped: every fetch hits `{issuer}/.well-known/jwks.json?zone_id={zone}` and is cached per issuer and zone. TypeScript verifiers use an in-memory JWKS cache by default. Build a cache explicitly when a resource server wants shorter local-development TTLs, a custom fetch implementation, or warmup during service boot. ```ts import { createJwksCache, verify } from '@caracalai/identity' const jwksCache = createJwksCache({ ttlMs: 300_000, fetchTimeoutMs: 5_000 }) await jwksCache.warm('https://sts.pipernet.example', '0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f') const claims = await verify(token, { issuer: 'https://sts.pipernet.example', audience: 'https://api.pipernet.example', zoneId: '0195f2a9-1b22-7c3d-9e4f-5a6b7c8d9e0f', jwksCache, }) ``` Go and Python identity packages also cache issuer JWKS metadata in memory and cap documents at 256 KiB. Python `JwksCache(http_client=...)` accepts an injected async client for private CAs, proxies, or mTLS. Go callers use `NewJWKSCache(httpClient, ttl)` and pass it through `Config.JWKSCache`; `GetJWKSContext` retains the bounded default. Python exposes `warm_jwks(issuer, zone_id)` for service boot. ## Failure classes | Error | Meaning | | ---------------------------- | -------------------------------------------------------------------------------------- | | `TokenInvalidError` | Signature, issuer, audience, expiry, use, or claim validation failed. | | `ZoneInvalidError` | The token zone did not match the expected zone. | | `ScopeInsufficientError` | A required scope is missing. | | `SessionRequiredError` | The verifier requires a governed Session identity. | | `DelegationRequiredError` | The verifier requires delegated authority. | | `ChainMismatchError` | The delegation chain is missing a required application. | | `HopCountExceededError` | The mandate exceeds the configured hop limit. | Verification is local except for JWKS retrieval. It does not call STS to refresh or exchange a token, and it does not retry application work. Zone ID, issuer, audience, required use, scopes, targets, Session, Delegation, chain, and hop requirements are security inputs and must come from trusted service configuration. ## When to Use Verify Package Use [Verify Package](/v1.0/sdks/verify/) when you also need revocation checks, reusable verifier defaults, bearer-header parsing, and safe debugging hints. Use the identity package directly when you are composing your own verifier or adapter boundary. --- # Revocation Package # URL: https://docs.caracal.run/v1.0/sdks/revocation/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/revocation.md # Type: page # Concepts: # Requires: --- The revocation packages define the store contract resource servers use to reject mandates after an Authority record, Root authority record, Session, or Delegation has been revoked. They do not consume streams, verify JWTs, or revoke product objects by themselves. Use the Redis backend and its consumers for distributed enforcement; use the Admin API to initiate revocation. ## Install | Ecosystem | Package | | ---------- | --------------------------------------------------------------- | | TypeScript | `npm install @caracalai/revocation` | | Python | `pip install caracalai-revocation` | | Go | `go get github.com/garudex-labs/caracal/packages/revocation/go` | ## Contract | Operation | Meaning | | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `isRevoked(anchorId)` / `is_revoked(anchor_id)` / `IsRevoked(anchorID)` | Return whether an Authority record, Root authority record, Session, or Delegation anchor is revoked. | | `markRevoked(anchorId, ttl)` / `mark_revoked(anchor_id, ttl)` / `MarkRevoked(anchorID, ttl)` | Record a revocation anchor for a TTL. | | `currentDelegationEpoch(zoneId)` / idiomatic equivalent | Return the newest observed Delegation graph epoch when the backend supports stale-edge detection. | | `markDelegationEpoch(zoneId, epoch, ttl)` / idiomatic equivalent | Advance that epoch without allowing delayed messages to regress it. | ## In-memory stores Use in-memory stores for local development, tests, and single-process examples: | Ecosystem | In-memory API | | ---------- | ----------------------------------------------------------- | | TypeScript | `new InMemoryRevocationStore({ defaultTtlMs, maxEntries })` | | Python | `InMemoryRevocationStore(default_ttl_ms=...)` | | Go | `revocation.NewInMemoryStore(defaultTTL)` | In-memory stores do not share revocation state across processes. Production resource servers should use a shared backend and consume `caracal.sessions.revoke`. ## Production path Use [Redis Revocation Store](/v1.0/sdks/backends/redis/) for multi-instance resource servers. The Redis backend reads signed revocation stream messages, marks every revocation anchor, and lets verifiers fail closed when Redis is unavailable. ## Related Pages * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) * [Verify Package](/v1.0/sdks/verify/) * [Protect an MCP Server](/v1.0/guides/protect-mcp/) --- # OAuth Package # URL: https://docs.caracal.run/v1.0/sdks/oauth/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/oauth.md # Type: page # Concepts: # Requires: --- The OAuth packages call the STS `/oauth/2/token` endpoint for RFC 8693 exchanges. The returned mandate class depends on the authenticated flow: lifecycle bootstrap returns `use=session`, a bound direct mint returns `use=gateway`, and Gateway's authenticated exchange returns `use=resource` privately to Gateway. Use this package only when composing a custom authority client. Normal applications should use the language SDK facade, which supplies the correct Authority-record, Session, and Delegation identifiers. ## Install | Ecosystem | Package | | ---------- | ---------------------------------------------------------- | | TypeScript | `npm install @caracalai/oauth` | | Python | `pip install caracalai-oauth` | | Go | `go get github.com/garudex-labs/caracal/packages/oauth/go` | ## Exchange inputs | Option | Meaning | | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | Subject token | Existing session mandate, or a Federated user's identity token from a registered issuer; omit to exchange as the application Subject. | | Resource | One or more resource identifiers. Clients trim, exact-deduplicate, and sort the set. | | Client secret | Application authentication. Public client assertions are not supported. | | Authority record ID | Sent in the STS form field `session_id`. It is distinct from a governed Session ID. | | Session ID | Sent in the STS form field `agent_session_id`. | | Delegation ID | Sent in the STS form field `delegation_edge_id`. | | Scopes | Requested resource scopes. | | TTL seconds | Requested mandate lifetime. | ## TypeScript example ```ts import { OAuthClient, ApprovalRequiredError } from '@caracalai/oauth' const oauth = new OAuthClient(stsUrl, zoneId, applicationId) try { const token = await oauth.exchange(subjectToken, 'resource://pipernet', { scopes: ['pipernet:read'], clientSecret: process.env.CARACAL_APP_CLIENT_SECRET, }) console.log(token.accessToken, token.expiresIn) } catch (error) { if (error instanceof ApprovalRequiredError) { console.log('approval required', error.approvalId) } else { throw error } } ``` ## Behavior * Successful responses are validated for `access_token`, `token_type`, and `expires_in`. * Public responses expose the granted `target_resources` subset; private Gateway upstream directives are not part of the SDK response type. * Cache-enabled responses are isolated by identity, canonical resources, scopes, TTL, and credential context. One-shot, approval-bearing, and cache-disabled exchanges bypass both cache and single-flight sharing. * Issuance uses one network attempt. A lost response may hide a successfully minted token, so the clients never retry STS exchange automatically. * STS `interaction_required` responses surface as `ApprovalRequiredError`. * Default exchange timeout is 30 seconds. `invalidate()` / `invalidate` clears the client cache; it does not revoke already issued mandates or cancel exchanges already in flight. * Never retry an exchange merely because the response was lost. If the surrounding business operation is replayable, mint a fresh mandate for that new attempt. ## Federated Users Every exchange acts for a Subject: the application itself by default, or a Federated user. When the zone registers the application's identity system as a Federated user issuer, the client exchanges an end user's identity token for a Caracal Authority record and can post that user's decision on an approval hold reserved for them: * `federateSubject(idToken, opts)` / `federate_subject(id_token, ...)` / `FederateSubject(ctx, idToken, opts)` - creates the Federated user's Authority record and returns its session mandate response. Never cached: each federation is an explicit identity event, and the record carries no resource authority. The high-level SDK wrappers also return the Authority record ID as `subjectAuthorityRecordId`, `subject_authority_record_id`, or `SubjectAuthorityRecordID`. * `decideApproval({...})` / `decide_approval(...)` / `DecideApproval(ctx, input)` - posts the Federated user's decision with their session mandate, echoing the hold's exact binding. See [Human Approval](/v1.0/guides/human-approval/#decide-as-the-applications-federated-user) for the end-to-end flow. ## Related Pages * [Human Approval](/v1.0/guides/human-approval/) * [Mandates](/v1.0/concepts/mandate/) * [Use STS Endpoint](/v1.0/api/sts/) --- # Admin Package # URL: https://docs.caracal.run/v1.0/sdks/admin/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/admin.md # Type: page # Concepts: # Requires: --- The admin package is the automation client for control-plane objects. It mirrors Console management workflows for scripts and services, and ships in three languages: `@caracalai/admin` (TypeScript), `caracalai-admin` (Python), and `github.com/garudex-labs/caracal/packages/admin/go` (Go). Use it from trusted operator automation. Do not embed admin tokens in agent or resource-server processes, and do not use this client for the SDK Session lifecycle. ## Install ```bash npm install @caracalai/admin ``` ```bash pip install caracalai-admin ``` ```bash go get github.com/garudex-labs/caracal/packages/admin/go ``` ## Create a client ```ts import { AdminClient } from '@caracalai/admin' const admin = new AdminClient({ apiUrl: process.env.CARACAL_API_URL!, coordinatorUrl: process.env.CARACAL_COORDINATOR_URL, adminToken: process.env.CARACAL_ADMIN_TOKEN!, coordinatorToken: process.env.CARACAL_COORDINATOR_TOKEN, }) ``` `apiUrl` and `adminToken` are required for API-backed resources. `coordinatorUrl` and `coordinatorToken` are required for Session lifecycle and Delegation methods. Python: ```python from caracalai_admin import AdminClient admin = AdminClient( api_url=os.environ["CARACAL_API_URL"], admin_token=os.environ["CARACAL_ADMIN_TOKEN"], coordinator_url=os.environ.get("CARACAL_COORDINATOR_URL"), coordinator_token=os.environ.get("CARACAL_COORDINATOR_TOKEN"), ) ``` Go: ```go import admin "github.com/garudex-labs/caracal/packages/admin/go" client := admin.NewAdminClient(admin.AdminClientOptions{ APIURL: os.Getenv("CARACAL_API_URL"), AdminToken: os.Getenv("CARACAL_ADMIN_TOKEN"), CoordinatorURL: os.Getenv("CARACAL_COORDINATOR_URL"), CoordinatorToken: os.Getenv("CARACAL_COORDINATOR_TOKEN"), }) ``` ## API groups | Group | Methods | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `zones` | `list`, `get`, `dcrStatus`, `create`, `patch`, `delete` | | `applications` | `list`, `get`, `create`, `patch`, `rotateSecret`, `getClientSecret`, `delete`, `dcr` | | `resources` | `list`, `get`, `create`, `patch`, `delete` | | `providers` | `list`, `get`, `create`, `patch`, `delete` | | `policies` | `list`, `get`, `create`, `validate`, `addVersion`, `delete` | | `policyTemplates` | `list`, `get` | | `policySets` | `list`, `get`, `create`, `addVersion`, `listVersions`, `simulate`, `activate`, `activationStatus`, `delete` | | `grants` | `list`, `get`, `create`, `revoke` | | `subjectIssuers` | `list`, `get`, `create`, `patch`, `delete` - manages Federated user issuers (the wire resource is `subject-issuers`). | | `providerConnections` | `create`, `authorizeOAuth`, `revoke`. To switch the upstream account, `revoke` then `authorizeOAuth` again, or re-run `authorizeOAuth` to replace the active connection in place. | | `workloads` | `list`, `get`, `create`, `update`, `rotateSecret`, `getSecret`, `delete` | | `authorityRecords` | `list` | | `subjects` | `revoke` - the kill switch: one call revokes every live Authority record for the Subject, terminates linked Sessions, revokes their Delegations and provider connections, and feeds the revocation stream so in-flight mandates die before `exp`. Idempotent. | | `sessions` | `list`, `get`, `children`, `suspend`, `resume`, `terminate`, `effectiveAuthority` | | `audit` | `list`, `byRequest`, `explain` | | `adminAudit` | `list` | | `approvals` | `list`, `get`, `approve`, `reject` | | `delegations` | `active`, `inbound`, `outbound`, `traverse`, `impact`, `revoke` | The same groups exist in every language with idiomatic naming: `admin.policySets.addVersion(...)` in TypeScript is `admin.policy_sets.add_version(...)` in Python and `client.PolicySets.AddVersion(...)` in Go. ## Policy activation example ```ts const policy = await admin.policies.create(zoneId, { name: 'pipernet-read', content: policySource, }) const set = await admin.policySets.create(zoneId, 'pipernet') const version = await admin.policySets.addVersion(zoneId, set.id, [{ policy_version_id: policy.version.id }]) await admin.policySets.activate(zoneId, set.id, version.id) let status = await admin.policySets.activationStatus(zoneId, set.id, version.id) while (status.propagation_status !== 'loaded' && status.propagation_status !== 'failed') { await new Promise((resolve) => setTimeout(resolve, 2000)) status = await admin.policySets.activationStatus(zoneId, set.id, version.id) } ``` ## Idempotent provisioning Alongside the API groups, the package exports `ensure*` reconcilers - `ensureApplication`, `ensureApiKeyProvider`, `ensureResource`, `ensureGrants`, and `ensureActivePolicySet` - that converge an object to a desired state: create it when absent, patch it only on drift, and return the live object. They are safe to rerun, so provisioning scripts and CI jobs declare state instead of scripting create-then-patch sequences. `ensureGovernedUpstreams` composes them for the most common declaration: a set of upstream APIs, each with a sealed credential provider, a gateway-routed resource, and its application grants, converged in dependency order in one call. ```ts import { ensureGovernedUpstreams } from '@caracalai/admin' const results = await ensureGovernedUpstreams(admin, zoneId, { upstreams: [ { provider: { name: 'OpenAI key', identifier: 'provider://openai', publicConfig: { auth_location: 'header', header_name: 'Authorization', auth_scheme: 'Bearer' }, apiKey: process.env.OPENAI_API_KEY, }, resource: { name: 'OpenAI', identifier: 'resource://openai', scopes: ['models:read', 'chat:write'], upstream_url: 'https://api.openai.com', }, grants: [{ applicationId: agentAppId, scopes: ['models:read', 'chat:write'] }], }, ], }) ``` Each run seals the provider key, binds the resource to it, and rewrites the zone's grant document to exactly the declared set - an upstream removed from the input loses its grants on the next run, which is the revocation. An upstream whose provider has no sealed key fails closed before any resource binds a dead credential. The same reconcilers ship in the Python package (`caracalai-admin`, as `ensure_governed_upstreams`) and the Go module (`github.com/garudex-labs/caracal/packages/admin/go`, as `EnsureGovernedUpstreams`). ## Credential custody Managed application client secrets and workload secrets are generated server-side, verified only by hash, and held sealed in the Secret Store. Create and rotate responses carry the plaintext for immediate delivery, and `applications.getClientSecret()` / `workloads.getSecret()` retrieve it later - each retrieval is recorded in the zone audit timeline as a credential reveal, so provisioning pipelines can fetch a credential at deploy time instead of persisting it at creation time. ```ts const { client_secret } = await admin.applications.getClientSecret(zoneId, appId) const { secret } = await admin.workloads.getSecret(zoneId, workloadId) ``` ## Dynamic Client Registration (DCR) DCR is the **only** way to create short-lived, self-registering client identities, and it is **programmatic-only** - Console creates managed applications, not DCR applications. Use `applications.dcr()` from a control-plane workload (per-tenant onboarding, a CI job, a per-integration identity) that already holds an admin token. ```ts const app = await admin.applications.dcr(zoneId, { name: 'tenant-hooli-job', expires_in: 900, // seconds; capped at 3600 }) // app.client_secret is returned ONCE and never retrievable again. ``` Creation is hardened server-side and cannot be misused as open self-registration: * **Admin token required** - the endpoint sits behind admin-bearer auth; a workload must hold a real, revocable, zone-scoped admin credential. Agent SDKs (`caracalai`) cannot create applications. * **Zone feature gate** - refused with `dcr_disabled` unless an operator enabled `dcr_enabled` on the zone. * **Rate-limited and capped** - per-actor request limiting plus a per-zone cap on live DCR applications (`dcr_rate_limit_exceeded` / `dcr_limit_exceeded`). * **Short-lived by construction** - `expires_in` is capped at one hour; expired applications are denied at token authentication and later archived by DCR cleanup. * **Secret hygiene** - the client secret is generated server-side, stored only as a hash, and returned exactly once. * **Authority is still policy-bound** - a DCR application is a credential, not a grant. It authenticates default-deny and receives no tool access until a policy (typically keyed on `registration_method == "dcr"`) grants scopes. DCR applications remain visible read-only in Console under the `dcr` method for audit and inspection. ## Rotating a managed application secret Two rotation paths exist and differ in who generates the secret. `applications.rotateSecret()` generates a strong secret server-side and returns the plaintext once - the path the web console's **rotate secret** action uses, and the right default. `applications.patch({ client_secret })` instead stores a secret you supply; the server keeps only its hash and the patch response does not echo it - use it only when an external system must own secret generation. ```ts await admin.applications.patch(zoneId, appId, { client_secret: newSecret, // store it yourself; the patch response does not echo it }) ``` Rotation is rejected with `client_secret_not_configured` if the application has no secret to replace. DCR applications are not rotated - they are short-lived and replaced by re-registration. ## Error handling Failed HTTP responses throw `AdminApiError` (TypeScript and Python) or return `*AdminAPIError` (Go) with `status`, `code`, parsed response details, and the base surface (`api` or `coordinator`). Non-idempotent write methods are not retried; read methods retry transient statuses. The default request timeout is 30 seconds and the default read retry budget is three. Retry handling honors `Retry-After` up to 30 seconds. Collection helpers follow `next_cursor` and stop after a bounded number of pages. Writes require caller-owned idempotency or reconciliation; the client does not replay them. Core groups and reconcilers are implemented in all three languages with idiomatic names. Do not infer that a newly introduced group exists in another language until its public package exports it; the raw [Admin API](/v1.0/api/control-plane/) remains the wire source of truth. ## Boundary Use the Admin package for automation. Use the web console for human workflows. Do not expose Admin operations as top-level `caracal` runtime commands. --- # Redis Revocation Store # URL: https://docs.caracal.run/v1.0/sdks/backends/redis/ # Markdown: https://docs.caracal.run/markdown/v1.0/sdks/backends/redis.md # Type: page # Concepts: # Requires: --- The Redis backends provide shared revocation state for resource servers. They let multiple service instances reject mandates after Authority records, root Authority records, Sessions, or Delegations are revoked. Use this backend only for verification state. It is not a general Caracal data client and must not receive Admin API credentials. The resource server does not need to share a network with Caracal services. For separate VPC, cluster, or cloud deployments, give the resource workload private TLS connectivity to the managed Redis endpoint that carries Caracal streams, authenticate it with a dedicated least-privilege Redis identity, and run the consumer beside the verifier. Do not expose Redis publicly or distribute Caracal administrative tokens to resource applications. ## Install | Ecosystem | Package | | ---------- | ------------------------------------------------------------------- | | TypeScript | `npm install @caracalai/revocation-redis` | | Python | `pip install caracalai-revocation-redis` | | Go | `go get github.com/garudex-labs/caracal/packages/backends/redis/go` | ## Defaults | Setting | Default | | ----------------- | ----------------------------------------------- | | Revocation stream | `caracal.sessions.revoke` | | Consumer group | `resource-revocation` | | Revocation TTL | 24 hours | | Signature support | Optional HMAC verification for stream messages. | | Dead-letter bound | Approximately 10,000 poison entries per `.dead` stream. | The store and consumers are long-lived resources. Start stream consumers with service startup, use unique consumer names, stop polling on shutdown, and close the caller-owned Redis connection. Verification should fail closed when the backend is unavailable. ## TypeScript store ```ts import { RedisRevocationStore } from '@caracalai/revocation-redis' const revocations = new RedisRevocationStore(redis, { defaultTtlMs: 24 * 60 * 60 * 1000, }) await revocations.markRevoked('sess_123') const blocked = await revocations.isRevoked('sess_123') ``` ## Stream consumer Use the stream consumer when your resource server should learn revocations from the audit/control plane instead of marking them locally. ```ts import { RedisRevocationConsumer } from '@caracalai/revocation-redis' const consumer = new RedisRevocationConsumer(redis, revocations, { consumer: 'api-1', // The key material is the deployment's STREAMS_HMAC_KEY; deliver it to this // workload through its own secret manager. streamHmacKey: Buffer.from(process.env.STREAMS_HMAC_KEY!, 'hex'), requireSignature: true, }) await consumer.ensureGroup() await consumer.pollOnce() ``` Stream events can revoke multiple anchors. Invalid signatures and permanently malformed messages are copied to the bounded `.dead` stream before acknowledgement. If the dead-letter write fails, the source message remains pending for recovery; transient application failures also remain pending. Delegated verification also needs the `caracal.delegations.invalidate` stream. Start `RedisDelegationInvalidationConsumer` beside `RedisRevocationConsumer` so the shared store receives the zone's current graph epoch; without it, signature and revocation-anchor checks still run, but `delegation_stale` cannot detect a mandate issued against an older graph epoch. Epoch writes use one atomic max-with-TTL operation, so delayed or concurrent messages cannot regress the recorded security state. The Python and Go Redis packages expose equivalent behavior. Each independently deployed resource instance uses a unique consumer name under deployment-specific groups for both streams. Supply `STREAMS_HMAC_KEY` through that workload's secret manager and require stream signatures in published deployments. Network policy should permit only DNS and the exact private Redis endpoint; the application needs no Postgres, API, Coordinator, or internal service access for revocation. ## Related Pages * [Revocation Package](/v1.0/sdks/revocation/) * [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) * [Verify Package](/v1.0/sdks/verify/) --- # Secure Caracal # URL: https://docs.caracal.run/v1.0/security/ # Markdown: https://docs.caracal.run/markdown/v1.0/security.md # Type: landing # Concepts: # Requires: --- This section covers OSS code and assets in this repository, not enterprise code, customer infrastructure, external identity providers/upstreams, or model behavior. ## The Security Model in Brief Caracal's core security idea is that programs hold **authority, not credentials**: instead of standing API keys, a program receives a short-lived signed pass for exactly one approved action, and every link in that chain fails closed. Four properties carry the model: | Property | What it means | Mechanics | | --- | --- | --- | | Deny by default | Nothing is callable until a resource is registered, a grant path exists, and the active policy allows the request. | [Authority and Enforcement](/v1.0/concepts/authority-model/) | | Fail closed | A policy, key, replay, revocation, Session, or Approval failure stops issuance; a verification failure stops the request before the upstream. | [Mandates](/v1.0/concepts/mandate/) | | Central revocation | Ending a Session, Delegation, or Authority record invalidates dependent authority without key rotation or redeploys. | [Sessions and Revocation](/v1.0/concepts/sessions-revocation/) | | Tamper-evident evidence | Decisions and results are recorded append-only with integrity checks, replay, and dead-letter paths. | [Audit and Request Traces](/v1.0/concepts/audit-ledger/) | ## What Is Enforced, Recommended, and Yours An honest evaluation separates three categories. Repository code enforces the first; documentation recommends the second; only you can provide the third. **Enforced by the code** (verifiable in source and tests): * STS fails closed on Policy, key, replay, revocation, Session, Approval, and signing failures. * Gateway authorizes before dispatch and applies binding, egress, redirect, replay, and revocation controls. * Published modes require integrity keys and authenticated metrics. * Audit is append-only/tamper-evident with replay and DLQ paths. * Runtime secrets remain outside untrusted application/agent workspaces. * OSS behavior does not depend on enterprise code. **Recommended practices** (documented, not forced): pin and [verify releases](/v1.0/security/verify-releases/), keep mandate TTLs short, use shared revocation stores in production, route provider credentials through Gateway brokering, and walk the [hardening checklist](/v1.0/security/hardening/) before real traffic. **Your responsibilities** (outside the repository's reach): host and network isolation, TLS termination, identity-provider governance, key custody and rotation cadence, backup and restore testing, monitoring ownership, and any compliance program. Repository controls contribute evidence to those programs; they never satisfy them alone. ## Choose a Review | Goal | Page | | --- | --- | | Assets, boundaries, threats, residual risk | [Review the Threat Model](/v1.0/security/threat-model/) | | Deployed environment review | [Harden Security Posture](/v1.0/security/hardening/) | | Artifact verification | [Verify a Release](/v1.0/security/verify-releases/) | | Repository evidence | [Generate an Evidence Pack](/v1.0/security/evidence-pack/) | | Private reporting | [Report a Vulnerability](/v1.0/security/disclosure/) | | Adoption controls and non-claims | [Review OSS Adoption Readiness](/v1.0/security/adoption-review/) | ## Review Completion Tie every claim to version, config, source/test evidence, and observed result. Repository controls do not replace host hardening, identity governance, recovery testing, or compliance programs. ## Next Step Start with [Review the Threat Model](/v1.0/security/threat-model/). --- # Review the Threat Model # URL: https://docs.caracal.run/v1.0/security/threat-model/ # Markdown: https://docs.caracal.run/markdown/v1.0/security/threat-model.md # Type: workflow # Concepts: # Requires: --- The canonical model is `governance/THREAT_MODEL.md`. This is a review procedure, not a duplicate inventory. ## Use Criteria Review before adoption and whenever auth, policy, tokens, keys, revocation, audit, streams, egress, services, ports, secrets, installers, releases, or product boundaries change. ## Prerequisites Pin source/deployed release. Gather rendered deployment, endpoints, secret flow, dependency topology, features, and targeted tests. ## Review Procedure 1. Confirm deployment fits the documented OSS scope. 2. Walk each boundary: untrusted input, mediation, credential, fail-closed behavior, evidence, owner. 3. Map assets to the canonical model's threat categories T1-T14 and run relevant negative checks. 4. Review every canonical known limit against deployment. 5. Record operator controls: TLS, host isolation, IdP, backup, monitoring, incident process. 6. Reject claims relying on enterprise-only or unverified platform controls. ## Verification Retain references/results for accepted threats and owner/containment/acceptance/date for residual risks. ## Recovery Disable or isolate unsupported boundaries. Report exploitable defects privately. ## Next Step Apply [Harden Security Posture](/v1.0/security/hardening/). --- # Harden Security Posture # URL: https://docs.caracal.run/v1.0/security/hardening/ # Markdown: https://docs.caracal.run/markdown/v1.0/security/hardening.md # Type: workflow # Concepts: # Requires: --- Use after operational hardening and before production traffic. [Harden Production](/v1.0/operations/tls-hardening/) configures the environment; this page verifies the security boundaries actually hold in it. ## Prerequisites Use a pinned verified release, stable mode, complete Secret, private storage, TLS ingress, authenticated metrics, and tested backup/incident paths. ## Procedure 1. Expose only required endpoints; keep storage, Audit, Coordinator, and Control private. 2. Retain non-root, read-only, dropped-capability, seccomp, and no-new-privileges settings. 3. Confirm secrets are projected, restricted, separately backed up, and inaccessible to agents. 4. Verify browser origins, cookies, registration allowlist, password/SMTP policy, and proxy trust. 5. Verify STS deny cases for credentials, scope, revocation, replay, and Approval. 6. Verify Gateway binding/header, egress, dangerous addresses, redirects, revocation, and pre-dispatch denial. 7. Verify stream/audit HMAC, tamper, DLQ/replay, and metrics auth. 8. Reserve bootstrap admin credentials for break-glass. ## Verification Retain negative-test IDs, readiness, rendered settings, network results, provenance, alert test, and restore evidence. ## Recovery Remove affected endpoint/workload from traffic, restore verified config, and open an incident for possible exposure. Never switch to dev or disable safety checks. ## Next Step Verify artifacts with [Verify a Release](/v1.0/security/verify-releases/). --- # Verify a Release # URL: https://docs.caracal.run/v1.0/security/verify-releases/ # Markdown: https://docs.caracal.run/markdown/v1.0/security/verify-releases.md # Type: workflow # Concepts: # Requires: --- The release workflow publishes archive checksums and GitHub Artifact Attestations; container builds request provenance and SBOM attestations. Verify only artifacts present for the selected release. ## Prerequisites Download from official GitHub/GHCR. Install/authenticate GitHub CLI. Record tag and digest/hash. ## Archive Procedure ```bash sha256sum --check SHA256SUMS gh attestation verify caracal-runtime--.tar.gz --repo Garudex-Labs/caracal jq '{release, sha, source, imageDigests}' manifest.json ``` Use `shasum -a 256 --check` on macOS or `Get-FileHash` on PowerShell. The manifest `sha` and `source.gitSha` must both equal the full commit behind the release tag, `source.dirty` must be `false`, and every deployed image must have an immutable `imageDigests` entry. Installers always verify checksum; provenance is opportunistic unless `CARACAL_REQUIRE_PROVENANCE=1`. ## Container Procedure ```bash gh attestation verify oci://ghcr.io/garudex-labs/caracal-go: --repo Garudex-Labs/caracal docker buildx imagetools inspect ghcr.io/garudex-labs/caracal-go: ``` Repeat for every deployed image and pin digests. Inspect available attestations; do not infer SBOM presence from provenance alone. Compare each inspected OCI index digest with the corresponding manifest `imageDigests` value and compare the chart digest with `helm.digest`. A matching tag without a matching digest is a different artifact and must not be admitted. ## Failure Recovery Do not run failed/unverifiable artifacts. Re-download and confirm tag/repository. Preserve hashes/output and report privately if persistent. ## Next Step Capture deployment evidence with [Generate an Evidence Pack](/v1.0/security/evidence-pack/). --- # Generate an Evidence Pack # URL: https://docs.caracal.run/v1.0/security/evidence-pack/ # Markdown: https://docs.caracal.run/markdown/v1.0/security/evidence-pack.md # Type: workflow # Concepts: # Requires: --- `evidencePack.sh` captures image provenance, schema validation, runtime smoke readiness, and the threat model. It is not compliance, penetration testing, an SBOM archive, config audit, or HA proof. ## Prerequisites Provide checkout, release version/authenticated `gh`, database credentials/`psql`, reachable runtime, and protected output as applicable. ## Procedure ```bash export CARACAL_VERSION= export PGHOST= PGPORT=5432 PGUSER= PGDATABASE= PGPASSWORD= export CARACAL_SMOKE_HOST= bash infra/scripts/evidencePack.sh ``` Output is under `evidence/caracal-evidence-/`. Executed failures exit non-zero; missing inputs produce non-failing `SKIPPED`. ## Verify Require relevant checks to be `PASS`; skipped means missing evidence. Add rendered deployment, digests, config review, canaries, alerts, and restore evidence separately. ## Recovery Preserve raw failure output, correct the boundary, and generate a new pack. Never edit generated status. ## Next Step Use [Review OSS Adoption Readiness](/v1.0/security/adoption-review/). --- # Review OSS Adoption Readiness # URL: https://docs.caracal.run/v1.0/security/adoption-review/ # Markdown: https://docs.caracal.run/markdown/v1.0/security/adoption-review.md # Type: workflow # Concepts: # Requires: --- Use this review when security, platform, compliance, procurement, or operations teams need a decision record for the product implemented in this repository. Caracal is a technical authority and audit component, not a compliance program or contractual assurance. This repository does not establish a managed service, service-level agreement, support entitlement, commercial edition, certification, regulatory conformity, or FIPS validation. Treat any requirement without implementation and evidence here as unavailable until a separate authoritative source is reviewed. ## What You Can Evaluate | Area | Evidence in this repository | Adopter-owned decision | | --- | --- | --- | | Authority enforcement | STS exchange, policy evaluation, Gateway and adapter verification, revocation, replay controls, tests | Which resources and actions require Caracal enforcement | | Deployment | Versioned Compose, Helm, and OpenTofu assets | Availability topology, infrastructure provider, ingress, and capacity | | Secrets and cryptography | Secret backends, envelope encryption, zone signing, HMAC boundaries, rotation procedures | Secret manager, key custody, rotation cadence, and required cryptographic validation | | Evidence | Request IDs, audit ingestion, tamper checks, replay, DLQ, export paths | Retention, legal hold, SIEM ownership, and evidence access | | Operations | Readiness, metrics, alerts, backup/restore, failure drills, upgrade flow | SLOs, RTO/RPO, on-call coverage, and escalation | | Supply chain | Lockfiles, checksums, release verification, build and publishing workflows | Artifact allowlisting, vulnerability acceptance, and deployment admission | | Human identity | Self-hosted Console sign-in and host-managed admission | Identity provider configuration, account lifecycle, privileged access review | ## Prerequisites Before the review, collect: * the exact Caracal release, image digests, chart or Compose assets, and configuration under review; * the protected resources, workload identities, Subjects, operators, and data classes in scope; * authoritative security and compliance requirements; * availability, recovery, retention, privacy, and incident objectives; * named owners for infrastructure, identity, secrets, policy, audit, and application integration. ## Review Procedure 1. **Verify the release.** Validate checksums, provenance, signatures or attestations that are actually present, image digests, and version alignment across packages. 2. **Review the threat model.** Map in-scope assets and trust boundaries to the deployed topology. Record assumptions the repository cannot enforce, including host and cloud controls. 3. **Render the deployment.** Inspect the exact Compose or Helm output for public endpoints, TLS termination, service accounts, NetworkPolicy, storage, secret mounts, resource limits, and published mode. 4. **Exercise authority failures.** Test expected allow, deny, insufficient scope, wrong audience, replay, expired Mandate, revoked Authority record, revoked Session, revoked Delegation, and unsafe upstream cases. 5. **Exercise human and administrative controls.** Verify Console admission, operator credential scope, Federated user federation where used, Approval decisions, and administrative audit attribution. 6. **Exercise evidence paths.** Trace request IDs from authorization through action result, test audit interruption and replay, inspect DLQ handling, and verify export or SIEM ingestion. 7. **Exercise operations.** Verify readiness, authenticated metrics, alerts, capacity assumptions, upgrade behavior, data-plus-secret restore, failure drills, and incident intake. 8. **Record every gap.** For each requirement, record Caracal evidence, deployment evidence, missing control, compensating control, owner, due date, and retest trigger. Framework mappings are review aids, not automatic coverage. A technical control can contribute evidence to an organizational requirement without satisfying the requirement by itself. ## Minimum Evidence Set Retain: * artifact verification output and immutable version identifiers; * rendered deployment manifests or resolved Compose configuration; * threat-model findings and accepted assumptions; * positive and negative test results with request IDs; * evidence-pack output, including the reason for every skipped check; * alert routing and failure-drill results; * backup and secret-custody evidence plus a successful isolated restore; * upgrade rehearsal and rollback or roll-forward decision; * final risks, owners, approval, expiry date, and review triggers. ## Decision Rules Approve only the exact release and environment reviewed. Do not convert the presence of a workflow, test, manifest, or documentation page into a guarantee about a deployed environment. Reject or conditionally approve when: * a required enforcement path can bypass Gateway or verifier checks; * revocation or audit freshness cannot be observed; * operator secrets can reach workload or agent environments; * a restore cannot recover both durable data and required key material; * critical alerts have no owner or tested response; * an organizational, contractual, or certification requirement is being inferred from repository intent alone. ## Review Triggers Repeat the review when a release, service boundary, public endpoint, identity flow, secret backend, storage platform, deployment substrate, protected data class, policy model, recovery objective, or authoritative requirement changes. Missing evidence means **unverified**. It does not mean planned, inherited, or satisfied. ## Next Step Complete [Harden Security Posture](/v1.0/security/hardening/), [Generate an Evidence Pack](/v1.0/security/evidence-pack/), and [Hand Off to Platform Teams](/v1.0/operations/platform-team-handoff/). --- # Report a Vulnerability # URL: https://docs.caracal.run/v1.0/security/disclosure/ # Markdown: https://docs.caracal.run/markdown/v1.0/security/disclosure.md # Type: workflow # Concepts: # Requires: --- Do not open public issues or pull requests for credential exposure, policy bypass, unsafe execution/routing, audit compromise, malicious artifacts, or exploitable failures. ## Choose a Channel | Scope | Channel | | --- | --- | | OSS code | `https://github.com/Garudex-Labs/caracal/security/advisories/new` | | Sensitive attachments/context | `support@caracal.run` | | Enterprise-only/private customer context | Email only; outside this workspace | ## Report Procedure Include summary, version/commit, boundary, prerequisites, minimal reproduction, observed/expected, impact, and mitigation if known. Redact credentials/customer data. Maintainers aim to respond within up to seven days; this is not a resolution SLA. ## Verify Submission Retain identifier and sanitized copy. Continue through the same private channel. ## Recovery and Disclosure Keep private until fix/mitigation or coordinated outcome. For active exploitation, contain with [Respond to Incidents](/v1.0/operations/incident-response/). ## Next Step Review [Review the Threat Model](/v1.0/security/threat-model/). --- # Use Examples # URL: https://docs.caracal.run/v1.0/examples/ # Markdown: https://docs.caracal.run/markdown/v1.0/examples.md # Type: landing # Concepts: # Requires: --- Examples show Caracal integrated into concrete applications and automation scripts. Use them after the first tutorials when you want runnable code that matches a specific integration job. All examples live in the dedicated [Caracal examples repository](https://github.com/Garudex-Labs/examples); clone it once and every guide below runs from that checkout: ```bash git clone https://github.com/Garudex-Labs/examples.git caracal-examples ``` ## Prerequisites * Complete [First Protected Call](/v1.0/get-started/first-protected-call/) and know whether you are integrating an application or a resource server. * Use an isolated development zone and synthetic credentials. * Run the example's offline tests before connecting it to a live runtime. ## Choose an example | Goal | Start here | Code path | | ------------------------------------------------------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | Need a local protected target for the first Gateway call. | [Run Echo Upstream](/v1.0/examples/echo-upstream/) | [echoUpstream](https://github.com/Garudex-Labs/examples/tree/main/echoUpstream) | | Want to automate zone setup from a script or pipeline through the Control API. | [Bootstrap Control State](/v1.0/examples/control-bootstrap/) | [controlBootstrap](https://github.com/Garudex-Labs/examples/tree/main/controlBootstrap) | | Need to prove a provider-backed resource is ready. | [Check Provider Readiness](/v1.0/examples/provider-preflight/) | [providerPreflight](https://github.com/Garudex-Labs/examples/tree/main/providerPreflight) | | Need to turn a denial into a safe policy fix. | [Iterate Policy Safely](/v1.0/examples/policy-iterate/) | [policyIterate](https://github.com/Garudex-Labs/examples/tree/main/policyIterate) | | Want to launch a plain CLI agent with injected provider credentials. | [Launch Research Agent](/v1.0/examples/research-agent/) | [ResearchAgent](https://github.com/Garudex-Labs/examples/tree/main/ResearchAgent) | | Want a full app reference lab with agents, providers, Gateway, STS, and Console inspection. | [Run Lynx Capital](/v1.0/examples/lynx-capital/) | [lynxCapital](https://github.com/Garudex-Labs/examples/tree/main/lynxCapital) | ## Recommended order 1. Start with [Run Echo Upstream](/v1.0/examples/echo-upstream/) if you have not completed a Gateway-mediated request yet. 2. Use [Bootstrap Control State](/v1.0/examples/control-bootstrap/) when automation should own zone setup instead of manual Console clicks. 3. Run [Check Provider Readiness](/v1.0/examples/provider-preflight/) before the first real provider-backed Gateway call. 4. Use [Iterate Policy Safely](/v1.0/examples/policy-iterate/) when an audit denial needs to become a tested policy change. 5. Try [Launch Research Agent](/v1.0/examples/research-agent/) to see `caracal run` inject provider-native credentials into an existing-style CLI process. 6. Study [Run Lynx Capital](/v1.0/examples/lynx-capital/) when you need a full app topology and live Console inspection path. ## Use examples safely * Start Caracal through the released runtime and Console path described by the example. * Use the web console for zones, applications, providers, resources, policies, control keys, and launch bindings. * Keep example fixtures inside their own example directory in the examples repository. * Run each example's offline tests before adapting it. * Do not commit provider secrets, admin tokens, or real third-party credentials. Expected result: each example proves one job and leaves a request ID, test output, or drift report. Examples are not production libraries and must be adapted with your own identity, secret, retry, timeout, and deployment controls. :::caution[Fixed example names] `ResearchAgent` and `lynxCapital` are checked-in source names in the examples repository; the walkthroughs use them as-is. Objects you create around them (zones, applications, providers) can use any names - these pages use the same sample names as the rest of the documentation. ::: ## Related Sections * [Get Started](/v1.0/get-started/) * [Tutorials](/v1.0/tutorials/) * [Guides](/v1.0/guides/) * [SDKs](/v1.0/sdks/) * [Runtime and Console](/v1.0/runtime-console/) ## Next Step Choose one row in **Choose an example**; do not run the full reference lab when a smaller example proves the required boundary. --- # Run Echo Upstream # URL: https://docs.caracal.run/v1.0/examples/echo-upstream/ # Markdown: https://docs.caracal.run/markdown/v1.0/examples/echo-upstream.md # Type: workflow # Concepts: # Requires: --- Echo Upstream is a zero-dependency HTTP service in the [Caracal examples repository](https://github.com/Garudex-Labs/examples) under `echoUpstream/`. It stands in for the API you would put behind Caracal, so you can verify a Gateway-mediated request end to end without hosting your own upstream. Every response states whether the call was brokered by the Gateway or hit the service directly, and shows the evidence behind that verdict. ## When to use it Use it as a disposable resource server while validating Gateway routing, provider injection, headers, and audit. It does not verify Caracal mandates itself and is not a production authorization boundary. ## Prerequisites * A ready Caracal runtime and Docker network access from Gateway. * A `resource://pipernet` resource, provider, operations, active policy, and scoped caller mandate. ## What it demonstrates | Area | Behavior | | --- | --- | | Brokered-call proof | Reports `viaGateway` plus the request ID, trace context, and forwarding metadata the Gateway stamped on the request. | | Credential handling | Confirms the Gateway injected the brokered credential and redacts credential values from the echoed headers. | | Gateway reachability | Joins the `caracalData` Docker network so Gateway can reach `http://echoUpstream:8088`. | | Local debugging | Exposes `http://127.0.0.1:8088` on the host and logs one line per request, marked `[gateway]` or `[direct]`. | ## Run with Docker ```bash git clone https://github.com/Garudex-Labs/examples.git caracal-examples cd caracal-examples/echoUpstream docker compose -f compose.yml up --build ``` Check the local health endpoint: ```bash curl http://127.0.0.1:8088/healthz ``` ## Run with Node ```bash cd caracal-examples/echoUpstream npm start ``` The server listens on port `8088`. Set `ECHO_PORT` when you need a different local port. ## Use as the protected upstream In web console guided setup or through the Control API, create a resource with this upstream URL: ```text http://echoUpstream:8088 ``` Then send a request through the Gateway with the SDK transport from [First Protected Call](/v1.0/get-started/first-protected-call/#the-agents-first-protected-call), pointing it at this resource. With the application identity from guided setup in the environment, the call has this shape: ```typescript import { Caracal } from '@caracalai/sdk' const caracal = new Caracal() const governedFetch = caracal.applicationTransport('resource://pipernet', { scopes: ['pipernet:read'], }) const target = caracal.gatewayRequest('resource://pipernet', '/v1/hello') const response = await governedFetch(target.url) console.log(await response.text()) await caracal.close() ``` ## Read the response A brokered call returns `"viaGateway": true` with a `gateway` section: | Field | What it proves | | --- | --- | | `viaGateway` | The request carried the Gateway's request ID and forwarding metadata. | | `gateway.requestId` | Audit handle - filter web console **Audit** by it to trace the policy decision. | | `gateway.credentialInjected` | The Gateway brokered a credential the client never held. | | `request.headers` | The headers the upstream actually received, with credentials redacted. | Calling `http://127.0.0.1:8088/` directly returns `"viaGateway": false`, which makes the difference between a protected and an unprotected path visible. ## Test ```bash cd caracal-examples/echoUpstream npm test ``` Expected result: direct traffic reports `viaGateway: false`; the Gateway path reports `viaGateway: true`, redacts credentials, and provides a request ID that resolves to STS and Gateway evidence. :::caution[Failure point: direct port] Host port `8088` deliberately bypasses Gateway. Never interpret a successful direct curl as proof of authorization, and do not expose an equivalent bypass in production. ::: ## Next Step Continue to [Bootstrap Control State](/v1.0/examples/control-bootstrap/) when you want repeatable setup for demo resources and policies. --- # Bootstrap Control State # URL: https://docs.caracal.run/v1.0/examples/control-bootstrap/ # Markdown: https://docs.caracal.run/markdown/v1.0/examples/control-bootstrap.md # Type: workflow # Concepts: # Requires: --- Control Bootstrap is the canonical Control API automation example in the [Caracal examples repository](https://github.com/Garudex-Labs/examples) under `controlBootstrap/`. A small CI/CD-style pipeline keeps one workload environment - Application, Provider, Resource, and Policy - matching a declared plan without adding product-management verbs to the runtime CLI. ## When to use it Use it when CI must reconcile objects inside one existing zone through scoped Control credentials. Use the Admin SDK from a trusted operator environment when automation must create zones or use management surfaces not exposed by Control. ## Prerequisites * A ready runtime and existing target zone. * Separate Control keys for apply, verify, and teardown with only documented scopes. * A reviewed plan containing no production secrets. ## What it demonstrates | Area | Behavior | | ------------------ | ----------------------------------------------------------------------------------------------------------- | | Automation surface | Calls `/v1/control/invoke` instead of using a root admin token. | | Identity model | Uses scoped, short-lived control keys created in the web console, one scope tier per pipeline stage. | | Reconciliation | `apply` creates missing objects, patches drifted ones, and publishes a new policy version on content drift. | | CI gating | `verify` is a read-only drift check that exits non-zero when the zone does not match the plan. | | Safety model | Uses replay protection, rate limits, scoped control permissions, and audit. | The plan describes the PiperNet reporter's environment: its Application, the `provider://pipernet-mandate` Provider, the `resource://pipernet` Resource wired to it, and the baseline Policy that allows `read`. ## Web Console Setup 1. Start the runtime and open the web console: ```bash caracal up caracal status --ready ``` Open the packaged web console at `http://localhost:3001`. 2. Create or select the target zone. 3. Create a control key with only the scopes the stage needs: read/write on app, provider, resource, and policy for `apply`; read for `verify`; read/delete for `teardown`. 4. Save the `client_id` and `client_secret`. STS resolves the zone from the bound control key; the secret stays retrievable from **Services → Control** through an audited reveal. ## Run the pipeline ```bash git clone https://github.com/Garudex-Labs/examples.git caracal-examples cd caracal-examples/controlBootstrap cp env.example .env $EDITOR .env . .env npm run apply npm run verify ``` `apply` is idempotent: rerunning it against an in-sync zone changes nothing, and rerunning it against a drifted zone converges the drift. Run teardown when you want to remove the environment: ```bash npm run teardown ``` ## Files to study | File | Purpose | | ------------------- | ----------------------------------------------------------------------------------- | | `controlClient.mjs` | Exchanges client credentials at STS and calls the Control API. | | `plan.mjs` | Declares the desired environment, drift checks, scope tiers, and env-driven config. | | `apply.mjs` | Reconciles the live zone with the plan. | | `verify.mjs` | Read-only drift gate for CI. | | `teardown.mjs` | Removes the environment in reverse dependency order. | ## Test ```bash cd caracal-examples/controlBootstrap npm test ``` The tests use a fake zone and mock transport and do not call a live Caracal stack. ## Validate the workflow Run `apply` twice, then `verify`, introduce one safe drift, and run `verify` again. Expect no second write on unchanged apply, a clean verification, then a non-zero drift result. Teardown must remove objects in dependency order without affecting objects outside the plan. :::caution[Failure point: credential boundary] Never give the example a root admin token or application credential. Control keys are zone-bound automation credentials; keep each pipeline stage least-privileged and short-lived. ::: ## Next Step Continue to [Check Provider Readiness](/v1.0/examples/provider-preflight/) before sending traffic through a provider-backed Gateway resource. --- # Check Provider Readiness # URL: https://docs.caracal.run/v1.0/examples/provider-preflight/ # Markdown: https://docs.caracal.run/markdown/v1.0/examples/provider-preflight.md # Type: workflow # Concepts: # Requires: --- Provider Preflight is an automatable readiness check in the [Caracal examples repository](https://github.com/Garudex-Labs/examples) under `providerPreflight/`. It walks the same chain a real Gateway request depends on - control plane, Gateway dependencies, application identity, provider binding and configuration, network paths, and the active policy decision - and reports exactly which link is broken and how to fix it. ## When to use it Run it after creating a provider-backed resource, before cutting traffic over, and from CI before each deploy. It proves the chain is ready; it does not replace the first real audited request as enforcement evidence. ## Prerequisites * A provisioned application, provider, resource, operation set, and active policy. * A read-capable Admin token stored only in the CI secret environment. * Network placement comparable to Gateway for meaningful reachability checks. ## What it checks Checks run in five phases: | Phase | Check | What it confirms | | ------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------- | | readiness | Admin API readiness | The control plane responds ready on `GET /ready`. | | readiness | Gateway readiness | The Gateway responds ready, which verifies its bindings, Postgres, Redis, revocation freshness, audit replay, and STS. | | dependencies | Resource binding | The resource resolves to exactly one credential provider and a Gateway application. | | dependencies | Application | The application exists, is not expired, and matches the resource's routing binding. | | configuration | Provider configuration | Kind-required fields are present and `allowed_token_hosts` covers the token endpoint. | | configuration | Scope coverage | Every requested scope is declared on the resource. | | configuration | Runtime injection | Runtime-injection eligibility is enabled when required. | | connectivity | Token endpoint host | OAuth token endpoints are HTTPS and publicly routable, or explicitly granted as private egress. | | connectivity | Callback reachability | Authorization-code callback origins are HTTPS and reachable. | | connectivity | Upstream reachability | The protected upstream is reachable, with a warning when it resolves to a private address. | | authorization | Policy authorization | Simulating the active policy set with the input shape STS uses for real token exchanges returns `allow`. | The preflight fails closed. Any failed check exits non-zero, and every failure or warning prints a `fix:` line with the concrete remediation. ## Run the preflight ```bash git clone https://github.com/Garudex-Labs/examples.git caracal-examples cd caracal-examples/providerPreflight CARACAL_API_URL=http://127.0.0.1:3000 \ CARACAL_ADMIN_TOKEN= \ PREFLIGHT_ZONE_ID= \ PREFLIGHT_RESOURCE_ID= \ PREFLIGHT_APPLICATION_ID= \ PREFLIGHT_SCOPES=pipernet:read,pipernet:write \ npm run preflight ``` Optional settings: | Variable | Purpose | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | | `PREFLIGHT_GATEWAY_URL` | Probes the Gateway's `/ready` endpoint; without it the Gateway phase reports a warning instead of validating. | | `PREFLIGHT_REQUIRE_RUNTIME_INJECTION=true` | Requires providers to allow runtime injection. | | `PREFLIGHT_OUTPUT=json` | Emits the full report as JSON for CI pipelines and dashboards. | Exit codes: `0` when all checks pass, `1` when any check fails, `2` when the preflight itself cannot run. ## Network position `Upstream reachability` and `Callback reachability` run from the host executing the script. Run it from a network position comparable to Gateway, or inside the cluster, when you need the reachability result to match production routing. ## Test ```bash cd caracal-examples/providerPreflight npm test ``` The tests inject network, DNS, and policy responses and do not call live systems. ## Validate the preflight Run once with valid config and once with a deliberately missing scope. Expect exit `0` and a clean summary, then exit `1` with a specific `fix:` line. JSON mode must remain parseable while narration stays off stdout. :::caution[Failure point: simulation is not traffic] A passing policy simulation and host probe do not prove credential injection or upstream behavior. Follow preflight with one real Gateway request and audit trace. ::: ## Next Step Continue to [Iterate Policy Safely](/v1.0/examples/policy-iterate/) when a denied request needs to become a tested policy change. --- # Iterate Policy Safely # URL: https://docs.caracal.run/v1.0/examples/policy-iterate/ # Markdown: https://docs.caracal.run/markdown/v1.0/examples/policy-iterate.md # Type: workflow # Concepts: # Requires: --- Policy Iterate is an audit-driven policy rollout loop in the [Caracal examples repository](https://github.com/Garudex-Labs/examples) under `policyIterate/`. It tests a candidate policy change against the exact redaction-safe audit input of a real denial - and proves the change does not alter other decisions - before activation. ## When to use it Use it when a real request was denied and the fix must be proven safe before rollout: the candidate has to repair that denial and keep every regression case's decision. Do not use it as a substitute for the [policy authoring contract](/v1.0/guides/author-policy/) or [activation workflow](/v1.0/guides/activate-policy-set/); it automates their verification loop. ## Prerequisites * A denied request ID with an explainable audit trace. * A staged immutable policy-set version and current active version recorded for rollback. * Regression cases covering nearby allow and deny behavior. ## Loop 1. **Diagnose** - a request is denied and you capture its audit `request_id`. The audit explain endpoint reconstructs the denied `policy_input` along with the diagnostics and determining policies. 2. **Simulate** - you edit the policy data, stage a candidate policy-set version, and the example replays the denied input against it through the same policy engine that serves live traffic. 3. **Regress** - the example replays your expected-decision cases against the same candidate, so loosening a policy for one caller cannot silently change decisions for everyone else. 4. **Decide** - activation is gated on evidence: the candidate allows the denied input, the rollout contract validates, simulation produced no warnings, and every regression case keeps its expected decision. 5. **Activate** - with `ACTIVATE=true` and a clean verdict, the example activates the version and polls activation status until the STS runtime reports it loaded. ## Run the iteration ```bash git clone https://github.com/Garudex-Labs/examples.git caracal-examples cd caracal-examples/policyIterate CARACAL_API_URL=http://127.0.0.1:3000 \ CARACAL_ADMIN_TOKEN= \ CARACAL_ZONE_ID= \ DENIED_REQUEST_ID= \ POLICY_SET_ID= \ CANDIDATE_VERSION_ID= \ REGRESSION_FILE=./regressions.json \ npm run iterate ``` The run is a dry run by default: it narrates each phase on stderr, prints a JSON report on stdout, and exits `0` only when the verdict is clean. Re-run with `ACTIVATE=true` to roll out the version and wait for propagation. `REGRESSION_FILE` is optional and points to a JSON array of `{ name, expect, input }` cases - see `regressions.example.json` in the example directory for the format. ## Claims note Actor and subject claims are not written to audit, so reconstructed input does not contain them. For a claim-dependent denial, add the relevant `context.actor_claims` to a regression case built from the printed `policyInput` and iterate with that case instead. ## Test ```bash cd caracal-examples/policyIterate npm test ``` The tests inject the Admin API transport and do not call a live Caracal stack. ## Validate the iteration Keep `ACTIVATE` unset first. Expect the candidate to repair the target denial while every regression retains its decision. Only then activate and wait for `loaded`; verify the first real exchange names the candidate manifest. :::caution[Failure point: reconstructed input] Audit reconstruction is redaction-safe and omits actor/subject claims. If a decision depends on implemented federated claims, supply them explicitly in a reviewed regression case; never guess them from a display name. ::: ## Next Step Continue to [Launch Research Agent](/v1.0/examples/research-agent/) to see runtime credential injection for a plain CLI agent. --- # Launch Research Agent # URL: https://docs.caracal.run/v1.0/examples/research-agent/ # Markdown: https://docs.caracal.run/markdown/v1.0/examples/research-agent.md # Type: workflow # Concepts: # Requires: --- Research Agent is a `caracal run` example in the [Caracal examples repository](https://github.com/Garudex-Labs/examples) under `ResearchAgent/`. It launches a normal Node.js CLI agent and injects provider-native credentials only into the child process after Caracal authorizes the run. ## When to use it Use it to evaluate one-shot runtime credential injection for an existing CLI that cannot accept an SDK transport. The launcher limits described in [Run an Agent with caracal run](/v1.0/guides/runtime-run/) apply: do not use it for a long-running service or as proof of request-level Gateway enforcement. ## Prerequisites * Three provider-backed resources with runtime injection enabled and least-privilege binding scopes. * A Launcher workload and owner-only workload secret. * Synthetic or development provider credentials; the offline test does not call third parties. ## What it demonstrates | Resource | Injected env | Used for | | --- | --- | --- | | `resource://google-drive` | `GOOGLE_DRIVE_ACCESS_TOKEN` | Searching and exporting Google Drive documents. | | `resource://google-calendar` | `GOOGLE_CALENDAR_ACCESS_TOKEN` | Reading relevant Calendar events. | | `resource://openai` | `OPENAI_API_KEY` | Answering the terminal question with model context. | The agent has no Caracal SDK dependency. It reads provider-native environment variables and behaves like an existing third-party terminal tool. ## Console setup Use the web console to create or select: | Object | Purpose | | --- | --- | | Zone | Owns the workload, providers, resources, and policies. | | Workload | Identity used by `caracal run` to call STS; created on the **Launcher** page. | | Google Drive provider | Returns a Drive read token. | | Google Calendar provider | Returns a Calendar read token. | | OpenAI provider | Returns an OpenAI-compatible credential. | | Resources | Map `resource://google-drive`, `resource://google-calendar`, and `resource://openai` to providers. | | Policy | Allows the workload to request all three resources. | Enable runtime injection on each provider. Then open the console's **Launcher** page, create a workload named `research agent`, and add three launch bindings: | Environment variable | Resource | | --- | --- | | `GOOGLE_DRIVE_ACCESS_TOKEN` | `resource://google-drive` | | `GOOGLE_CALENDAR_ACCESS_TOKEN` | `resource://google-calendar` | | `OPENAI_API_KEY` | `resource://openai` | Select the read-only scopes on the Drive and Calendar bindings so each injected credential carries only what the agent needs. The Launcher page then shows the exact launch commands. ## Store the workload secret The launcher needs one local file: the workload's secret at the runtime secret path. Copy it from the Launcher page - it stays retrievable there, with every reveal audited. ```bash export CARACAL_WORKLOAD_ID="" mkdir -p ~/.config/caracal/runtime/$CARACAL_WORKLOAD_ID printf '%s' '' > ~/.config/caracal/runtime/$CARACAL_WORKLOAD_ID/secret chmod 600 ~/.config/caracal/runtime/$CARACAL_WORKLOAD_ID/secret ``` Do not export `GOOGLE_DRIVE_ACCESS_TOKEN`, `GOOGLE_CALENDAR_ACCESS_TOKEN`, or `OPENAI_API_KEY` yourself. Caracal injects them into the child process after STS authorization. ## Launch the agent ```bash git clone https://github.com/Garudex-Labs/examples.git caracal-examples cd caracal-examples/ResearchAgent export CARACAL_WORKLOAD_ID="" caracal run -- node agent.mjs ``` The agent confirms the injected credentials (values masked) and opens an interactive prompt: ```text [agent] credential preflight (values masked, injected by launcher): [agent] GOOGLE_DRIVE_ACCESS_TOKEN present -> Google Drive (read-only scope) [agent] GOOGLE_CALENDAR_ACCESS_TOKEN present -> Google Calendar (read-only scope) [agent] OPENAI_API_KEY present -> OpenAI Caracal run research agent ready. Ask about Drive docs or Calendar events. Type "exit" to quit. > ``` Started directly with `node agent.mjs`, the preflight fails with exit code 2 before any network call. Credentials disappear with the child-process environment when the process exits. ## Test ```bash cd caracal-examples/ResearchAgent pnpm test ``` The tests do not contact Google, OpenAI, or Caracal. ## Validate the launch Run the agent directly and expect exit `2`, then run it through `caracal run` and expect all three masked preflight entries. Confirm binding fetch and credential mints in Audit, then exit and verify credentials are absent from the parent shell. :::caution[Failure point: static authority] Injected API keys or bearer tokens can carry more upstream authority than Caracal binding scopes suggest. Scopes gate release; they cannot narrow a static provider credential. Prefer Gateway brokering for strong per-request enforcement. ::: ## Next Step Continue to [Run Lynx Capital](/v1.0/examples/lynx-capital/) when you want a full app reference lab with live Console inspection. --- # Run Lynx Capital # URL: https://docs.caracal.run/v1.0/examples/lynx-capital/ # Markdown: https://docs.caracal.run/markdown/v1.0/examples/lynx-capital.md # Type: workflow # Concepts: # Requires: --- Lynx Capital is a runnable reference in the [Caracal examples repository](https://github.com/Garudex-Labs/examples) under `lynxCapital/`. It models a finance-operations platform: an LLM swarm of orchestrators, regional workflows, and thousands of ephemeral domain workers executes payout cycles across twenty partner providers, with every agent and every provider call governed by Caracal. It is the primary reference for modelling permission boundaries, Sessions for agents, providers, resources, and policies on Caracal. ## When to use it Use this fixed-name reference only after a smaller example works. Study it for multi-application permission boundaries, per-agent Sessions, provider views, policy data, and test organization; do not copy its finance domain or dependency pins blindly. ## Prerequisites * Python, Docker, OPA (for the offline policy-data tests), an isolated development zone, and a scoped provisioning Control key. * Capacity to run offline policy and Python suites before live provisioning. * Separate secrets for every managed application boundary. ## Architecture | Building block | Role | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Applications | `lynx-operations`, `lynx-intake`, `lynx-ledger`, `lynx-compliance`, `lynx-treasury`, `lynx-payments`, `lynx-audit` - one **managed application** per permission boundary, each holding only its own partner authority. | | Agents | Every agent execution - orchestrator or worker - is its own **Session** under its role's application, labeled `[role, lynx-swarm]` with run and agent metadata, narrowed by a Delegation to its role's scopes and views. | | Providers | Twenty partner **credential providers** (`provider://`), each registered in the exact config shape its kind supports: API key, bearer token, OAuth client credentials, OAuth authorization code, Caracal mandate, or none. | | Resources | Per-application **resource views** (`resource://-`). The Gateway binds each view to exactly one application, so shared partners expose one view per boundary, each carrying only that boundary's scopes. | | Policy set | `lynx-finance-ops`: generated policy data documents - application bindings, per-view grants, and label confinement - that the default-deny platform decision contract evaluates, allowing exactly each application's role-granted mandate mints and gateway calls. | The model is declared once in `config/tenancy.yaml`; the SDK seam, agent runner, provisioning, and policy all read from it. ### Why one application per permission boundary A Caracal application is a credential and trust boundary, and the Gateway binds each resource to exactly one application. Splitting the swarm by permission boundary means a payments worker and an audit worker can both reach the same partner - through different views, with different scopes - while a compromised intake agent can never present payment authority. Agent executions are Sessions, not applications: each started Session gets its own identity, labels, Delegation, and audit trail without minting new application credentials. ### Sessions for agents The swarm's runner gives every agent its own session via the SDK's `session()`: orchestrators inherit under the operations boundary; each domain worker starts under its application's per-run dispatcher root with `Authority.narrow(role scopes, views, max_hops=1, run TTL)`. Ad-hoc partner-integration workers resolve their boundary, scope, and view dynamically from the requested provider operation. Logs and policy decisions identify exactly which agent did what. ### Customer attribution and confinement Workers acting on one customer's records - invoicing, dunning, payment application - start Sessions with a `customer:` label and a `customer_id` metadata key. The metadata key makes per-customer audit a direct filter over the shared zone trail; the label is policy input, and the base policy confines customer-labeled agents to the customer-record scopes, so a worker dunning one customer can never mint treasury or payment-rail authority. This is the [Serve Your Own Customers](/v1.0/guides/serve-customers/) pattern applied to app-only work: one zone, customer separation carried by sessions, labels, metadata, and policy. ## Setup flow ```mermaid flowchart LR Install[Install Python deps] --> Zone[Console: zone + Control key] Zone --> Provision[scripts/provision.py] Provision --> Objects[Applications + providers + views + policy set] Objects --> Env[Export per-application credentials] Env --> Reference[scripts/reference.py] Reference --> Inspect[Inspect sessions and delegation in Console] ``` ## Commands ```bash git clone https://github.com/Garudex-Labs/examples.git caracal-examples cd caracal-examples/lynxCapital python -m venv .venv source .venv/bin/activate pip install -e ".[dev]" cp -n .env.example .env ``` The workload `.env` carries the zone and one `LYNX_CARACAL__APPLICATION_ID` / `_CLIENT_SECRET` pair per boundary. Provisioning uses a separate operator file and a scoped Control key created once in Console. ```bash cp -n .env.provision.example .env.provision # set CONTROL_CLIENT_ID / _SECRET . .env.provision python scripts/provision.py # applications, providers, views, policy set (idempotent) python scripts/reference.py # SDK walkthrough: labeled Sessions, narrowed authority, mandates python scripts/teardown.py # remove the provisioned objects ``` `provision.py` prints the per-application credential exports as it creates each application; each client secret is returned exactly once. It also renders the application-id bindings into the policy library before authoring it, so policy decisions key on the real control-plane UUIDs. ## Policies `policies/` is an importable, OPA-tested library of **policy data documents** - Caracal adopters author data, never decision logic, because the platform [decision contract](/v1.0/concepts/policy/#decision-contract) owns every `result`. The generated documents carry the application bindings (`app_ids`), the resource-view grants (`grants`), and the customer-label confinement; the contract then allows exactly the mandate mints (scope ∩ Delegation, role label granted, view owned by the caller) and gateway uses (mandate target includes the view) those documents describe, naming the deciding application boundary in every decision. Expected access behavior is documented in `policies/README.md`. The locally installed OPA from the prerequisites exists only for this offline test loop - it exercises the data documents against a vendored copy of the platform contract before anything is provisioned: ```bash opa test policies/ -v ``` ## SDK integration Application code uses two seams: `app/caracal.py` (per-application runtimes, worker authority, mandate minting, gateway calls) and `app/agents/runner.py` (per-Session lifecycle): ```python handle = await runner.aspawn("payment-execution", "payments.us", parent=fc, layer="worker") result = partners.call("meridian-pay", "create_payout", payload, authority=handle.authority) ``` Every partner call resolves the operation's scope from the model, verifies the calling agent's grant client-side, mints (or reuses) a resource mandate for the agent's view, and posts through the Gateway - which re-evaluates policy, natively enforces the resource's declared operation authority, injects the provider credential, and forwards upstream. Agents never hold partner secrets. ## Tests ```bash opa test policies/ -v pytest tests/ ``` The tests cover the policy decision suite, the identity-model and provisioning-plan builders, the runner and authority seams, and the provider transports, topology, and lifecycle of the bundled workload. Expected result: policy and Python suites pass offline; provisioning is idempotent; the reference run produces labeled Sessions and narrowed Delegations under expected application boundaries; teardown removes only example-owned state. :::caution[Failure point: source drift] The checked-in example is authoritative for its package versions and commands. If its manifest differs from this page, follow the repository manifest and report the ambiguity rather than forcing current workspace package assumptions onto the external example. ::: ## Bundled demo workload The repository ships the FastAPI and LangGraph swarm with a simulated payout cycle against local provider fixtures under `_mock/`. ```bash docker compose -f _mock/docker-compose.yml up -d --build --wait python -m uvicorn app.main:app --reload --port 8000 docker compose -f _mock/docker-compose.yml down ``` Open `http://localhost:8000`; the guided `/setup` wizard teaches the one-zone, per-boundary-application, provider, resource-view, and policy-library flow. ## Related Examples * [Run Echo Upstream](/v1.0/examples/echo-upstream/) * [Launch Research Agent](/v1.0/examples/research-agent/) ## Next Step Map one Lynx pattern - not the whole domain - onto [Model Your Application in Caracal](/v1.0/guides/modeling-recipes/) and add equivalent tests to your integration. --- # Use API Reference # URL: https://docs.caracal.run/v1.0/api/ # Markdown: https://docs.caracal.run/markdown/v1.0/api.md # Type: landing # Concepts: # Requires: --- Use this section when you need wire-level behavior. For task workflows, start in [Guides](/v1.0/guides/). For service ownership and operations, use [Understand Services](/v1.0/services/) and [Operations](/v1.0/operations/). Most integrations should use an SDK or the web console. Only the Admin API is a general external management API. Coordinator is a wire-facing SDK protocol, STS is an OAuth endpoint, Gateway is a protected reverse proxy, and event topics are deployment infrastructure rather than end-user HTTP APIs. ## API Surfaces | Surface | Base | Purpose | | ---------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------ | | [Use the Admin API](/v1.0/api/control-plane/) | API service `/v1` | Zones, applications, providers, resources, policies, policy sets, grants, Authority records, Sessions, audit, approvals, and templates. | | [Use Coordinator API](/v1.0/api/coordinator/) | Coordinator service | Sessions, long-lived Session services, invocations, Delegations, and SDK lifecycle endpoints. | | [Use STS Endpoint](/v1.0/api/sts/) | STS service | OAuth token exchange, JWKS, Approval status, and internal policy operations. | | [Proxy Through Gateway](/v1.0/api/gateway/) | Gateway service | Protected reverse-proxy behavior rather than CRUD endpoints. | | [Use Event Topics](/v1.0/api/event-topics/) | Redis Streams | Audit, invalidation, revocation, Session, invocation, and Delegation topics. | ## Service Ports Local Compose ports: API `3000`, Coordinator `4000`, STS `8080`, Gateway `8081`, Audit `9090`; Control shares the API port when enabled. The canonical port and endpoint maps are in [Defaults and Limits](/v1.0/reference/defaults-and-limits/#ports) and [Monitor Health and Metrics](/v1.0/operations/observability/#endpoint-map). ## Error Shape Caracal service errors use the shared OAuth-compatible shape: ```json { "error": "access_denied", "error_description": "policy denied request", "requestId": "018f..." } ``` See [Error Codes](/v1.0/reference/errors/) for canonical codes. Do not assume one status code across surfaces: STS follows OAuth semantics, adapters distinguish 401 from 403, and Gateway can return either a Caracal preflight error or the upstream response. Branch on the documented machine-readable `error` field where a Caracal JSON error is present. ## Stability The public Admin `/v1` routes and STS RFC 8693 exchange are supported integration surfaces. Coordinator field names such as `agents`, `agent_session_id`, and `delegation_edge_id` are protocol names retained for SDK interoperability; applications should use facade names. `/internal/*`, operator routes, private STS directives, and raw Redis topics are not general application APIs. ## Reading Path | Need | Page | | ----------------------------------------- | ---------------------------------------- | | Manage product objects over HTTP | [Use the Admin API](/v1.0/api/control-plane/) | | Manage Session and Delegation runtime state | [Use Coordinator API](/v1.0/api/coordinator/) | | Exchange authority for mandates | [Use STS Endpoint](/v1.0/api/sts/) | | Understand Gateway proxy requirements | [Proxy Through Gateway](/v1.0/api/gateway/) | | Consume or verify stream contracts | [Use Event Topics](/v1.0/api/event-topics/) | ## Next Step Start with [Use the Admin API](/v1.0/api/control-plane/) when automating product setup. --- # Use the Admin API # URL: https://docs.caracal.run/v1.0/api/control-plane/ # Markdown: https://docs.caracal.run/markdown/v1.0/api/control-plane.md # Type: api # Concepts: # Requires: --- The Admin API is served by the API service on port `3000`. Management routes are registered under `/v1` and are protected by admin authentication. This is the supported HTTP surface for trusted automation. Prefer [Admin Package](/v1.0/sdks/admin/) for typed pagination and errors. Never expose an admin token to an agent, browser, or protected upstream. ## Authentication Send an admin token as a bearer credential on every `/v1` request: ``` Authorization: Bearer ``` | Status | Error | Meaning | | ------ | --------------------------------------------- | ------------------------------------------------- | | `401` | `unauthorized`, `invalid_admin_token` | Token is missing, malformed, expired, or revoked. | | `403` | `admin_token_read_only` | Write attempted with a read-only token. | | `403` | `admin_token_zone_mismatch`, `zone_forbidden` | Token is scoped to a different zone. | | `403` | `system_zone_read_only` | Write attempted against the reserved system zone. | ## Error Shape Every non-2xx `/v1` response carries a single JSON envelope: ```json { "error": "invalid_body", "error_description": "Request body failed validation", "details": { "issues": [{ "path": ["name"], "message": "Required" }] } } ``` `error` is the stable machine code to branch on. `error_description`, when present, is a human-readable summary of the failure. `details`, when present, carries structured context: for example, the validation issues on `invalid_body`, the failed connectivity check on `provider_check_failed`, or the live dynamic-client count on `dcr_shutdown_required`. See [Error Codes](/v1.0/reference/errors/) for the shared code catalog. Malformed JSON, unknown fields in strict schemas, wrong types, missing required values, invalid identifiers, and zone mismatches are rejected before mutation. Success status depends on the operation: reads and updates return `200`, creates commonly return `201`, asynchronous policy activation returns `202`, and deletes commonly return `204`. Treat each endpoint response as authoritative rather than assuming every write returns an object. ## Health and Diagnostics | Method | Path | Purpose | | ------ | ---------- | --------------------------------- | | `GET` | `/health` | Liveness check. | | `GET` | `/ready` | Dependency and service readiness. | | `GET` | `/metrics` | Service metrics. | | `GET` | `/docs` | Optional API docs when enabled. | ## Core Resources | Resource | Collection | Item | Extra | | ------------ | -------------------------------------------- | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | Zones | `GET`, `POST /v1/zones` | `GET`, `PATCH`, `DELETE /v1/zones/:id` | `GET /v1/zones/:id/dcr-status`, `GET /v1/zones/:id/overview` | | Applications | `GET`, `POST /v1/zones/:zoneId/applications` | `GET`, `PATCH`, `DELETE /v1/zones/:zoneId/applications/:id` | `POST /v1/zones/:zoneId/applications/dcr`, `POST /v1/zones/:zoneId/applications/:id/rotate-secret`, `GET /v1/zones/:zoneId/applications/:id/client-secret` | | Providers | `GET`, `POST /v1/zones/:zoneId/providers` | `GET`, `PATCH`, `DELETE /v1/zones/:zoneId/providers/:id` | - | | Resources | `GET`, `POST /v1/zones/:zoneId/resources` | `GET`, `PATCH`, `DELETE /v1/zones/:zoneId/resources/:id` | - | | Workloads | `GET`, `POST /v1/zones/:zoneId/workloads` | `GET`, `PUT`, `DELETE /v1/zones/:zoneId/workloads/:id` | `POST /v1/zones/:zoneId/workloads/:id/rotate-secret`, `GET /v1/zones/:zoneId/workloads/:id/secret` | Workloads are the launcher identities `caracal run` uses. Create a workload, deliver its secret to the launch environment, and rotate the secret here when you automate runtime setup; the secret stays retrievable from Secret Store custody through an audited reveal endpoint. ## Policy and Access | Area | Read | Write | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Policy validation | - | `POST /v1/policies/validate` | | Policies | `GET /v1/zones/:zoneId/policies`, `GET /v1/zones/:zoneId/policies/:id` | `POST /v1/zones/:zoneId/policies`, `POST /v1/zones/:zoneId/policies/:id/versions`, `DELETE /v1/zones/:zoneId/policies/:id` | | Policy sets | `GET /v1/zones/:zoneId/policy-sets`, `GET /v1/zones/:zoneId/policy-sets/:id`, `GET /v1/zones/:zoneId/policy-sets/:id/versions/:versionId`, `GET /v1/zones/:zoneId/policy-sets/:id/activation-status` | `POST /v1/zones/:zoneId/policy-sets`, `POST /v1/zones/:zoneId/policy-sets/:id/versions`, `POST /v1/zones/:zoneId/policy-sets/:id/activate`, `POST /v1/zones/:zoneId/policy-sets/:id/simulate`, `DELETE /v1/zones/:zoneId/policy-sets/:id` | | Policy templates | `GET /v1/policy-templates` | - | | Grants | `GET /v1/zones/:zoneId/grants`, `GET /v1/zones/:zoneId/grants/:id` | `POST /v1/zones/:zoneId/grants`, `DELETE /v1/zones/:zoneId/grants/:id` | | Federated user issuers | `GET /v1/zones/:zoneId/subject-issuers`, `GET /v1/zones/:zoneId/subject-issuers/:id` | `POST /v1/zones/:zoneId/subject-issuers`, `PATCH /v1/zones/:zoneId/subject-issuers/:id`, `DELETE /v1/zones/:zoneId/subject-issuers/:id` | | Provider connections | `GET /v1/zones/:zoneId/provider-connections`, `GET /v1/zones/:zoneId/provider-connections/oauth/callback` (OAuth redirect target) | `POST /v1/zones/:zoneId/provider-connections`, `POST /v1/zones/:zoneId/provider-connections/oauth/authorize`, `POST /v1/zones/:zoneId/provider-connections/revoke` | Use the web console when you are performing these operations interactively. Use the Admin SDK or Control API when automation needs the same management behavior. ## Audit, Authority Records, Sessions, and Approvals | Resource | Methods and paths | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Audit | `GET /v1/zones/:zoneId/audit`, `GET /v1/zones/:zoneId/audit/by-request/:requestId`, `GET /v1/zones/:zoneId/audit/by-request/:requestId/explain` | | Admin audit | `GET /v1/zones/:zoneId/admin-audit` | | Authority records | `GET /v1/zones/:zoneId/authority-records`. Filter by `authority_record_id`, `subject_id`, `status`, or `kind` (`user` for Federated users, `application` for application Subjects); use `format=csv` to export. Rows carry `federated_user_issuer` only when the Subject is a Federated user. | | Subjects | `GET /v1/zones/:zoneId/subjects`, `GET /v1/zones/:zoneId/subjects/overview?subject_id=`, `POST /v1/zones/:zoneId/subjects/revoke` (the kill switch revokes every live Authority record for the Subject, terminates linked Sessions, revokes Delegations and provider connections, and feeds the revocation stream) | | Sessions | `GET /v1/zones/:zoneId/sessions`. Filter by `status`, `lifecycle`, `label`, `parent_session_id`, `application_id`, or `subject_id`; use `format=csv` to export. Rows resolve subject attribution (`subject_authority_record_id`, `subject_id`) and carry `federated_user_id` and `federated_user_issuer` only when the Session acts for a Federated user. | | Approvals | `GET /v1/zones/:zoneId/approvals`, `GET /v1/zones/:zoneId/approvals/counts`, `GET /v1/zones/:zoneId/approvals/:id`, `POST /v1/zones/:zoneId/approvals/:id/approve`, `POST /v1/zones/:zoneId/approvals/:id/reject`. | ## Platform Administration These routes require the global admin token. | Resource | Methods and paths | | --------------- | ------------------------------------------------------------- | | Admin tokens | `GET`, `POST /v1/admin-tokens`; `DELETE /v1/admin-tokens/:id` | | Audit retention | `GET`, `PUT /v1/audit-retention` | Use `POST /v1/admin-tokens` to mint read-only or zone-scoped tokens for automation, and keep the bootstrap token as break-glass. ## Listing and Filters * List endpoints accept `limit` (default `200`, max `500`) and an opaque `cursor`. Every list response is an envelope of `items` and `next_cursor`; pass `next_cursor` as the next request's `cursor` and stop when it is `null`. The Admin SDK list helpers for collection resources drain the cursor chain and return the complete collection. * `GET /v1/zones/:zoneId/audit` accepts `since`, `until`, `request_id`, `decision`, `event_type`, `application_id`, `session_id`, `authority_record_id`, and `label`. * Add `format=csv` to audit and admin-audit lists for exports; use `fields` to select columns. The `cursor` is opaque: do not parse, edit, or reuse it with different filters. Ordering is service-defined and stable only within the cursor chain. ```bash curl -s -H "Authorization: Bearer $CARACAL_ADMIN_TOKEN" \ "http://localhost:3000/v1/zones/$CARACAL_ZONE_ID/resources?limit=100" ``` ```json { "items": [...], "next_cursor": null } ``` ## Idempotency and Retries Admin writes do not expose a universal `Idempotency-Key` contract. Do not blindly retry a timed-out create, rotate, approve, revoke, or delete request. Read the object back, or use the Admin SDK `ensure*` reconcilers for declarative provisioning. GET/HEAD calls are safe to retry subject to normal rate limits. A `429` response can include `Retry-After`. Credential create, rotation, and reveal endpoints are security-sensitive. Plaintext secret responses must be delivered directly to the target secret store and excluded from logs, traces, and retry payload capture. Reveals are audited. ## Usage Notes * Use the web console for human workflows and Admin SDK or Control API for automation. * For scoped, non-interactive provisioning, drive the Control API with a control key. See the [supported Control management model](/v1.0/services/control/#supported-management-model) and [Bootstrap Control State](/v1.0/examples/control-bootstrap/). * Prefer declarative reconciliation over scripting individual writes. Control's [`state plan|verify|apply` workflows](/v1.0/services/control/#supported-management-model) converge a Zone from desired state with dry-run and CI-friendly verification. * Policy activation and simulation are API operations, but top-level `caracal` runtime commands do not expose them. * Caracal Operator routes under `/v1/operator` and zone-scoped operator paths back the web console. They are not a stable automation surface. * Writes that produce downstream state changes enqueue signed Redis stream events through the API outbox. ## Next Step Continue to [Use Coordinator API](/v1.0/api/coordinator/) when automation needs Session, invocation, or Delegation endpoints. ## Related Pages * [Admin Package](/v1.0/sdks/admin/) * [Manage Product Objects](/v1.0/runtime-console/admin/) * [Manage Product State](/v1.0/services/api/) --- # Use Coordinator API # URL: https://docs.caracal.run/v1.0/api/coordinator/ # Markdown: https://docs.caracal.run/markdown/v1.0/api/coordinator.md # Type: api # Concepts: # Requires: --- Coordinator is served on port `4000`. It owns governed Session and Delegation graph state. This page documents the raw protocol, whose stable paths and fields retain `agents`, `agent_session_id`, and `delegation_edge_id` names. **Wire-facing API:** application code normally should not call Coordinator directly. The SDK owns credential selection, idempotency, retries, context binding, generation fencing, and cleanup. ## Authentication and Errors Runtime routes require a Caracal bearer mandate whose zone matches `:zoneId`, whose application is active, and whose scope authorizes the operation. The configured operator token is accepted only on an allowlisted management subset; traversal and impact require `coordinator.admin`. `/v1/verify`, health, and readiness are exceptions documented below. Protocol errors are JSON with at least `error`; common statuses are `400` validation, `401` missing/invalid/expired bearer, `403` ownership or scope failure, `404` missing object, `409` lifecycle/idempotency conflict, `429` capacity or receipt limit, and `5xx` dependency failure. ## Health and Metrics | Method | Path | Purpose | | ------ | ---------- | ----------------------------- | | `GET` | `/health` | Liveness check. | | `GET` | `/ready` | Dependency and job readiness. | | `GET` | `/metrics` | Service metrics. | ## Session Management | Method | Path | Purpose | | -------- | ------------------------------------ | ------------------------------------------------ | | `POST` | `/zones/:zoneId/agents` | Start a Session. `/agents` is the protocol path. | | `GET` | `/zones/:zoneId/agents` | List Sessions. | | `GET` | `/zones/:zoneId/agents/:id` | Inspect one Session. | | `GET` | `/zones/:zoneId/agents/:id/children` | List child Sessions. | | `PATCH` | `/zones/:zoneId/agents/:id/suspend` | Suspend a Session subtree. | | `PATCH` | `/zones/:zoneId/agents/:id/resume` | Resume a suspended Session subtree. | | `DELETE` | `/zones/:zoneId/agents/:id` | Terminate a Session. | `POST /zones/:zoneId/agents` accepts `application_id`, protocol field `subject_session_id` (the Subject Authority-record ID), optional `parent_id`, `lifecycle` (`task` or `service`), `labels`, `ttl_seconds`, `parent_authority` (`inherit` or `none`), `inherit_parent_edge_id`, and `metadata`. Task TTL defaults to 3600 seconds and is bounded to 1–86400. Service lifetime is lease-based. The response carries protocol `agent_session_id`, optional inherited `delegation_edge_id`, and lifecycle/lease data. Lists accept `limit` (default 100, maximum 500), opaque `cursor`, and route-specific filters. Responses use `{ "items": [...], "next_cursor": string | null }`. A task Session start and its `201` response: ```json // POST /zones/{zoneId}/agents { "application_id": "0198f3e2-...", "lifecycle": "task", "ttl_seconds": 900 } ``` ```json { "agent_session_id": "0198f4a1-...", "zone_id": "0198f3d0-...", "application_id": "0198f3e2-...", "parent_id": null, "subject_authority_record_id": "0198f3f7-...", "lifecycle": "task", "labels": null, "status": "active", "depth": 0, "ttl_seconds": 900, "started_at": "2026-07-11T13:00:00.000Z", "last_heartbeat_at": null, "heartbeat_deadline_at": null, "lease_generation": 0, "delegation_edge_id": null } ``` ## Long-lived Sessions and Invocations | Method | Path | Purpose | | ------- | ----------------------------------------- | ------------------------------------------------------------------------------------- | | `POST` | `/zones/:zoneId/agent-services` | Register or update a long-lived Session lease. `agent-services` is the protocol path. | | `GET` | `/zones/:zoneId/agent-services` | List long-lived Session leases. | | `POST` | `/zones/:zoneId/agents/:id/heartbeat` | Refresh a long-lived Session lease. | | `POST` | `/zones/:zoneId/invocations` | Create an invocation. | | `GET` | `/zones/:zoneId/invocations/:id` | Inspect an invocation. | | `PATCH` | `/zones/:zoneId/invocations/:id/start` | Mark an invocation running. | | `PATCH` | `/zones/:zoneId/invocations/:id/cancel` | Cancel an invocation. | | `PATCH` | `/zones/:zoneId/invocations/:id/complete` | Complete an invocation. | `POST /zones/:zoneId/agents/:id/lease` acquires a new generation before an SDK attaches. Heartbeat must carry the current generation; an older holder is fenced with conflict. Terminal missing, expired, or fenced results stop SDK auto-heartbeat. ## Delegation Runtime credentials see only Delegations where their application is the issuer or receiver. Recursive traversal and impact analysis are operator-only because descendants can cross application boundaries and expose another application's topology or Authority record IDs. The managed operator token retains zone-wide inspection for web console and Admin SDK workflows. | Method | Path | Purpose | | ------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | | `POST` | `/zones/:zoneId/delegations` | Create a Delegation. The response field `delegation_edge_id` is the protocol name for Delegation ID. | | `GET` | `/zones/:zoneId/delegations/active` | List active Delegations. | | `GET` | `/zones/:zoneId/delegations/inbound/:sessionId` | List inbound edges for a session. | | `GET` | `/zones/:zoneId/delegations/inbound/:sessionId/:id` | Validate one opaque inbound Delegation ID for its receiver Session. | | `GET` | `/zones/:zoneId/delegations/outbound/:sessionId` | List outbound edges for a session. | | `GET` | `/zones/:zoneId/delegations/:id/traverse` | Traverse a Delegation; `coordinator.admin` required. | | `GET` | `/zones/:zoneId/delegations/:id/impact` | Compute revocation impact; `coordinator.admin` required. | | `GET` | `/zones/:zoneId/agents/:sessionId/effective-authority` | Compute effective authority for a Session. | | `PATCH` | `/zones/:zoneId/delegations/:id/revoke` | Revoke delegated authority. | Delegation creation requires `source_session_id`, `target_session_id`, issuer and receiver application IDs, and an expiry (`expires_at` or positive `ttl_seconds`). Optional narrowing includes `parent_edge_id`, `resource_id`, `scopes`, and strict constraints (`resources`, `max_depth`, `max_hops`, `budget`, `ttl_seconds`, `expires_at`, plus audit metadata). Self-Delegation is rejected. A resource-unbound edge requires an explicit broad-Delegation scope. ## Idempotency and Retries Session and Delegation creation accept `Idempotency-Key`. Repeating the same canonical request returns the recorded result; reusing the key with different input returns `409 idempotency_key_conflict`. SDK-generated keys also carry `Idempotency-Key-Kind: generated`. Keep one key across retries of one logical operation and generate a new key for a new operation. The SDK retries transient Session creation twice and Delegation creation once. Direct callers must implement the same bounded policy and must not retry non-idempotent writes without a key. Honor `Retry-After` on retryable congestion responses. ## Language-Neutral Verification | Method | Path | Purpose | | ------ | ------------ | --------------------------------------------------------------------------------------------- | | `POST` | `/v1/verify` | Verify a mandate through the configured identity verifier for language-neutral integrations. | `/v1/verify` is public in the HTTP routing sense but is a wire utility, not a replacement for local resource-server verification and shared revocation state. SDK lifecycle clients use the zone-scoped Session, lease, heartbeat, termination, and Delegation endpoints above. Coordinator exposes no parallel flat lifecycle or Delegation protocol. Delegation creation authenticates the issuer and records an offer. Receiver credentials are not distributed to the issuer. The receiver consents by presenting the opaque ID, while STS verifies its target Session and application binding. The web console intentionally has no creation form. ## Next Step Continue to [Use STS Endpoint](/v1.0/api/sts/) to see how Session IDs and Delegation IDs participate in token exchange. ## Related Pages * [Coordinate Session State](/v1.0/services/coordinator/) * [Coordinate Sessions](/v1.0/architecture/delegation-flow/) * [Manage Sessions and Delegation](/v1.0/runtime-console/agents/) --- # Use STS Endpoint # URL: https://docs.caracal.run/v1.0/api/sts/ # Markdown: https://docs.caracal.run/markdown/v1.0/api/sts.md # Type: api # Concepts: # Requires: --- STS is served on port `8080` and issues scoped Caracal mandate JWTs. Applications normally call it through the OAuth or application SDK. The public exchange follows RFC 8693 framing with Caracal parameters; internal policy and signing-key endpoints are service-to-service only. ## Public Endpoints | Method | Path | Purpose | | ------ | --------------------------------------- | ------------------------------------------------------------------------------------- | | `POST` | `/oauth/2/token` | OAuth token exchange for resource, session, Gateway, or delegated mandates. | | `GET` | `/.well-known/jwks.json?zone_id={zone}` | Public signing keys for mandate verification, scoped per zone. | | `GET` | `/approvals/{id}` | Approval hold state; `?wait={seconds}` long-polls until the state changes. | | `POST` | `/approvals/{id}/decision` | Federated user approval decision, authenticated with the Federated user's session mandate. | | `GET` | `/health` | Liveness check. | | `GET` | `/ready` | Readiness check. | | `GET` | `/metrics` | Prometheus metrics. | | `GET` | `/metrics.json` | JSON metrics. | ## Token Exchange Request `POST /oauth/2/token` accepts body-only `application/x-www-form-urlencoded` parameters. Query parameters, duplicate singleton fields, and other media types are rejected. Repeated `resource` fields are allowed; STS trims, exact-deduplicates, and sorts them before evaluation. | Parameter | Purpose | | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | `grant_type` | OAuth grant type. | | `subject_token`, `subject_token_type` | Existing authority to exchange: a session mandate this STS issued. | | `resource` | One or more target resource identifiers. | | `scope` | Requested scopes. | | `zone_id` | Zone boundary. | | `application_id` | Calling application. | | `client_secret` | Application authentication. | | `session_id`, `agent_session_id`, `delegation_edge_id` | Protocol names for Authority record ID, Session ID, and Delegation ID. They are distinct identifiers. | | `ttl_seconds` | Requested TTL. | | `approval_id` | Consumes an approved Approval hold during retry. | `grant_type` must be `urn:ietf:params:oauth:grant-type:token-exchange`. `subject_token_type` must match the presented token class. `scope` is a space-delimited string. `ttl_seconds` must be a positive integer within server limits. Resource identifiers, zone, application, Session, Authority-record, and Delegation bindings are validated server-side; caller-supplied identifiers never establish authority by themselves. RFC 8693 `actor_token` is not supported: a request that carries one is rejected with `invalid_token`. Public `client_assertion` and `client_assertion_type` fields are also rejected; application authentication uses `client_secret`. The acting application's identity reaches policy input as `input.context.actor_claims.caracal_client_id`. A minimal application-principal exchange: ```bash curl -s http://localhost:8080/oauth/2/token \ --data-urlencode 'grant_type=urn:ietf:params:oauth:grant-type:token-exchange' \ --data-urlencode "zone_id=$CARACAL_ZONE_ID" \ --data-urlencode "application_id=$CARACAL_APPLICATION_ID" \ --data-urlencode "client_secret=$CARACAL_APP_CLIENT_SECRET" \ --data-urlencode 'resource=resource://pipernet' \ --data-urlencode 'scope=pipernet:read' ``` ## Subject Identity Every exchange records a Subject - the identity work is done for. STS never invents one: the Subject is the `sub` claim of the presented `subject_token`, or the authenticated application's own id when no subject token is presented. The Subject is therefore one of two kinds: an **application Subject** (the default) or a **Federated user**. Two subject token classes are accepted: * `urn:ietf:params:oauth:token-type:access_token` - a session mandate this STS issued for the same zone; resource mandates are rejected as subject tokens (RFC 8693 subject-confusion mitigation). The chain keeps the Subject kind of the presented mandate. * `urn:ietf:params:oauth:token-type:id_token` - **Federated user federation**: an end user's identity token from a registered Federated user issuer (Admin API resource: `subject-issuers`). The application authenticates with its client credentials and relays the token; STS verifies it against the issuer's JWKS and creates an Authority record whose `sub` is copied verbatim. Caracal never authenticates the user itself. A federation exchange names no resources and mints no scopes: the record is an identity and revocation anchor, not resource authority. A Federated user's identifier is opaque to Caracal. It may be a user ID, UUID, employee ID, or customer ID. STS requires only that the same identity always presents the same value. The Authority record stores that `sub`; provider connections, approval bindings, revocation, and audit can attribute activity to it. Scope authority is decided by application bindings, resource grants, Delegation narrowing, and label confinement - never by the Subject identifier or kind. Minted mandates carry the Subject kind as the `sub_type` claim (`application` or `user`), and audit decision events record it as `subject_kind`, so downstream systems can distinguish the kinds without parsing identity out of identifiers. When an SDK starts a Session with `subjectAuthorityRecordId`, `subject_authority_record_id`, or `SubjectAuthorityRecordID`, that field supplies a **Subject authority record ID**. It links the Session to the Authority record for attribution and lifecycle. It does not, by itself, make later resource mandates carry that record's Federated user `sub`. ### Federated User Identifier Guarantees Any stable identifier format a trusted issuer signs is accepted - UUIDs, emails, numeric ids, prefixed ids such as `auth0|507f...`, URIs, ARNs, or unicode names. Caracal never parses, normalizes, case-folds, or trims the value: it is compared byte-for-byte everywhere, so two byte-distinct values are two Subjects, and the recorded value is always byte-identical to what the issuer signed. A federation exchange rejects only values that are unsafe to store, index, log, or display: | Rejected | Reason | | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | | Empty, or longer than 512 bytes | Bounded storage, indexing, and display. | | Inside the reserved `caracal:` namespace | Anchors Caracal-internal Subject sentinels such as the shared provider connection; an external issuer must never mint one. | | Invalid UTF-8, or containing U+FFFD | A replacement character would let two byte-distinct upstream values collapse into one stored identity. | | Control characters (including NUL, newlines, escapes) | Log forging and storage safety. | | Bidirectional-override characters (U+202A-U+202E, U+2066-U+2069) | Visual reordering of surrounding UI text. | | Leading or trailing whitespace | Rejected rather than trimmed, preserving the byte-exact identity. | ## Responses Successful exchanges return an OAuth-style token response with `access_token`, `token_type`, `expires_in`, and related fields. When policy gates the mint on human approval, the STS answers `401` with `error: interaction_required` plus the hold's `approval_id`, `approval_type`, `state`, `tier`, `binding`, and `approval_expires_at`, so the caller can wait on `/approvals/{id}` and retry. Other errors use the shared `error` and `error_description` shape. ```json { "access_token": "", "token_type": "Bearer", "expires_in": 900, "target_resources": ["resource://pipernet"] } ``` Public clients never receive Gateway's private upstream directive or provider credential. ## Retry and Idempotency The exchange endpoint has no public idempotency key. SDK OAuth clients make one network attempt because the STS may mint before a response is lost. Do not automatically replay a timed-out exchange. Approval retry is different: after an approved hold, send the exact `approval_id` with the same bound resource/scope request; consumed, rejected, expired, or mismatched holds fail. `GET /approvals/{id}?wait={seconds}` returns `pending`, `approved`, `rejected`, `expired`, or `consumed`; the server bounds the wait. The Federated user decision endpoint requires that user's session mandate and the exact approval binding. ## Internal Endpoints | Method | Path | Purpose | | ------ | --------------------------------------------- | --------------------------- | | `POST` | `/internal/policy/simulate` | Simulate policy input. | | `GET` | `/internal/policy/status/{zoneID}` | Inspect policy load status. | | `POST` | `/internal/zones/{zoneID}/signing-key/rotate` | Rotate zone signing key. | Internal endpoints are for service/admin integration, not normal application traffic. They require the service authentication configured for API-to-STS calls and carry no external stability guarantee. ## Next Step Use [Proxy Through Gateway](/v1.0/api/gateway/) to understand how Gateway validates inbound authority and exchanges with STS per request. ## Related Pages * [Exchange Tokens](/v1.0/architecture/token-exchange-flow/) * [OAuth Package](/v1.0/sdks/oauth/) * [Mandates](/v1.0/concepts/mandate/) --- # Proxy Through Gateway # URL: https://docs.caracal.run/v1.0/api/gateway/ # Markdown: https://docs.caracal.run/markdown/v1.0/api/gateway.md # Type: api # Concepts: # Requires: --- Gateway is served on port `8081`. It is not a CRUD API; it is a protected reverse proxy for configured resources. Use an SDK transport instead of constructing requests manually. Gateway does not offer pagination, management CRUD, or an application-visible idempotency API. ## Operator Endpoints | Method | Path | Purpose | | ------ | ------------------------------ | --------------------------- | | `GET` | `/health` | Liveness check. | | `GET` | `/ready` | Readiness check. | | `GET` | `/metrics` | Prometheus metrics. | | `GET` | `/metrics.json` | JSON metrics. | | `POST` | `/internal/revocations/reload` | Reload revocation snapshot. | ## Request Requirements | Input | Required | Purpose | | --------------------------- | -------- | ------------------------------------------------------------ | | `Authorization: Bearer ...` | yes | One-shot Caracal Gateway-ingress mandate (`use=gateway`). | | `X-Caracal-Resource` | yes | Resource identifier STS resolves to the routed upstream. | | Request path | yes | Forwarded to the configured upstream after traversal checks. | The original HTTP method, query, headers, and body are proxied after safety filtering. Gateway strips hop-by-hop headers, caller `Authorization`, internal Caracal routing headers, `Forwarded`, and `X-Real-IP`, then applies the resource's provider credential. `X-Caracal-Resource` is routing metadata, not authorization; the signed mandate and STS decision are authoritative. Gateway rejects any request that sets `X-Caracal-Client-ID`; the caller's application identity derives from the verified mandate's `client_id` claim. A proxied request: ```bash curl -s http://localhost:8081/v1/models \ -H "Authorization: Bearer $GATEWAY_MANDATE" \ -H 'X-Caracal-Resource: resource://pipernet' ``` ## Denial Checks Gateway rejects before upstream dispatch. Treat these checks as the safety gate in front of every protected upstream. | Stage | Checks | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | Request preflight | Bearer token exists, token size is bounded, `X-Caracal-Resource` exists, and request path is not traversal. | | Token validation | Token is well formed, not expiring inside the preflight window, signature-valid, not replayed, not revoked, and includes a zone. | | Resource resolution | STS resolves `(zone_id, resource)` to a gateway-routed resource, and the returned upstream passes host safety checks. | | STS exchange | Gateway's STS circuit is closed and the signed exchange succeeds; STS enforces the resource's declared scopes on the mandate. | ## Forwarding Behavior After checks pass, Gateway performs a signed STS exchange of the inbound `use=gateway` mandate, receives a `use=resource` mandate plus a private upstream directive, forwards the request to the resource's upstream, and emits audit evidence. Request and upstream timeouts are controlled by Gateway service config. Gateway proxies HTTP request/response traffic, including streamed responses such as SSE. Streamed bodies are flushed chunk by chunk, and revocation anchors are re-checked on every chunk boundary; a mid-stream revocation truncates the stream and sets the `X-Caracal-Revoked` trailer. Gateway does not proxy WebSocket connections: hop-by-hop headers, including `Upgrade`, are stripped before forwarding, so upgrade handshakes never reach the upstream. Protect WebSocket workloads with in-process verification at the service edge instead, using [Protect an MCP Server](/v1.0/guides/protect-mcp/) or a framework adapter. ## Replay Protection Every Gateway-ingress mandate is single-use. On first presentation Gateway records the token's `jti` in Redis with a lifetime equal to the token's remaining validity; presenting the same mandate again is rejected as a confirmed replay and emits a `replay_detected` audit event carrying the request id, resource, and client identity. Session mandates are not valid Gateway inputs. There is no client-side nonce to manage: the mandate's `jti` is the replay key. Scoped SDK transports therefore bypass mandate caching and single-flight for the final exchange, and application transports cache only their source/target Session and Delegation state before minting a fresh Gateway-ingress mandate per request. They never retry the data-plane request or replay its body. When Redis is unreachable, the production default rejects the request rather than silently widening the replay window; a deployment can opt into fail-open, which forwards the request and logs the tracker error. ## Responses and Failures After dispatch, Gateway streams the upstream status, headers, and body. Before dispatch, Caracal failures use JSON `error`, `error_description`, and request correlation where available. Typical classes are `400` malformed routing input, `401` missing/invalid/expired/replayed/revoked mandate, `403` operation or authority denial, `404` unresolved resource, `502` upstream or STS failure, and `503` open dependency circuit. Do not assume a non-2xx response came from Gateway after dispatch; it may be the upstream's response. Retries are a business-operation decision. Mint a fresh mandate for each new attempt and retry only when the HTTP method/body and upstream contract are idempotent. ## Next Step Continue to [Use Event Topics](/v1.0/api/event-topics/) to understand the Redis Stream topics that carry audit, invalidation, revocation, and Session lifecycle events. ## Related Pages * [Protect Upstreams](/v1.0/services/gateway/) * [Enforce Boundaries](/v1.0/architecture/trust-boundaries/) * [Protect an MCP Server](/v1.0/guides/protect-mcp/) --- # Use Event Topics # URL: https://docs.caracal.run/v1.0/api/event-topics/ # Markdown: https://docs.caracal.run/markdown/v1.0/api/event-topics.md # Type: api # Concepts: # Requires: --- Caracal uses Redis Streams for propagation. Published modes sign stream messages with `STREAMS_HMAC_KEY`. **Internal wire API:** applications normally should not consume these topics. They are deployment contracts for Caracal services, audit exporters, and revocation backends. Topic names and payload fields are protocol-facing and may retain `agents` terminology. ## Topics | Topic | Producers | Consumers | | -------------------------------- | --------------------------------------- | -------------------------------------------------- | | `caracal.audit.events` | API, STS, Gateway, Coordinator, Control | Audit `audit-ingestor`, SIEM exporters | | `caracal.audit.events.dlq` | Audit | DLQ observers | | `caracal.policy.invalidate` | API | STS policy loader | | `caracal.sessions.revoke` | API, Coordinator | STS, Gateway, resource-server revocation consumers | | `caracal.keys.invalidate` | API, STS | STS key caches | | `caracal.agents.lifecycle` | Coordinator | Coordinator lifecycle relay job | | `caracal.invocations.lifecycle` | Coordinator | Invocation observers | | `caracal.delegations.invalidate` | Coordinator | Delegation observers | | `caracal.providers.ratelimit` | Provisioner/provider coordination | Provider rate-limit coordination | ## Consumer Groups | Topic | Groups | | -------------------------------- | ------------------------------- | | `caracal.audit.events` | `audit-ingestor`, `siem-export` | | `caracal.audit.events.dlq` | `audit-dlq-observer` | | `caracal.policy.invalidate` | `opa-engine` | | `caracal.sessions.revoke` | `sts-revocation`, stable per-host `gateway-revocation:`, deployment-specific resource groups | | `caracal.keys.invalidate` | `sts-keys` | | `caracal.agents.lifecycle` | `coordinator-relay` | | `caracal.invocations.lifecycle` | `invocations-observer` | | `caracal.delegations.invalidate` | `delegations-observer` | ## Message Integrity Signed stream messages include the `_sig` field. Consumers in published modes must reject unsigned or mismatched messages for streams that require origin verification. Messages use Redis Stream IDs and field/value maps. Consumers must validate required fields and bounds before applying state, deduplicate by the event identity carried by the producer, and acknowledge only after durable application. Poison messages go to the stream's bounded dead-letter path where implemented. Pending messages must be claimed and retried with consumer-group semantics rather than copied as a new logical event. Revocation and Delegation epoch updates are monotonic security state: delayed delivery must never reduce the recorded epoch. Audit events are append-only evidence and must preserve `request_id`, occurrence time, and signature fields unchanged. ## Access and Stability Use a dedicated least-privilege Redis identity and private TLS connectivity. Do not expose Redis publicly or distribute Admin tokens to consumers. Consumer group names shown here are repository defaults, not a multi-tenant coordination API; independently deployed resource servers should use deployment-specific groups and unique consumer names. ## Related Pages * [Propagate Events](/v1.0/architecture/event-streams/) * [Operate Redis Streams](/v1.0/operations/redis/) * [Wire Contracts](/v1.0/reference/interoperability-contracts/) --- # Understand Services # URL: https://docs.caracal.run/v1.0/services/ # Markdown: https://docs.caracal.run/markdown/v1.0/services.md # Type: landing # Concepts: # Requires: --- Use this section for integration boundaries and operations. It is not a requirement for ordinary console setup. ## Service Selection | User-visible operation | Service | Supported caller | | ------------------------------------ | ------------------------------------- | ------------------------------------------------------------ | | Manage product and policy state | [API](/v1.0/services/api/) | Console BFF, Admin SDK, documented Admin API | | Start Sessions or create Delegations | [Coordinator](/v1.0/services/coordinator/) | SDK, documented Coordinator API, console operator views | | Exchange authority for a mandate | [STS](/v1.0/services/sts/) | SDK, `caracal run`, Gateway, documented token client | | Protect an HTTP upstream | [Gateway](/v1.0/services/gateway/) | Client presenting a Caracal mandate | | Ingest and retain decision evidence | [Audit](/v1.0/services/audit/) | Caracal stream producers; operators read through console/API | | Automate zone management remotely | [Control](/v1.0/services/control/) | Trusted automation with a scoped Control credential | ## Dependency Shape ```mermaid flowchart LR Console --> API SDK --> STS SDK --> Coordinator Client --> Gateway Gateway --> STS API & STS & Gateway & Coordinator & Audit --> Postgres[(Postgres)] API & STS & Gateway & Coordinator & Audit --> Redis[(Redis)] Control --> API ``` The packaged Compose and Helm topologies deploy all five runtime services together with Postgres and Redis; Control is the only optional management surface. All five runtime services expose health and readiness. Health means the process responds. Readiness includes dependencies and service-specific thresholds. Control has no separate process: it is an optional plugin on the API port. ## Direct-Call Rule Do not call `/internal/*` routes, write service tables, publish Redis topics, or manipulate replay directories from application code. Internal routes are authenticated service-to-service contracts. Use the console, SDKs, Admin API, Coordinator API, STS token endpoint, Gateway proxy, or Control API as documented. ## Next Step [Manage Product State](/v1.0/services/api/) for the management path, or [Issue Mandates](/v1.0/services/sts/) for the authority path. --- # Manage Product State # URL: https://docs.caracal.run/v1.0/services/api/ # Markdown: https://docs.caracal.run/markdown/v1.0/services/api.md # Type: reference # Concepts: # Requires: --- The API owns Zones, Applications, workloads, Providers, Resources, Policies, Policy sets, Grants, Federated user issuers, Subjects, Authority records, Approval holds, admin audit, and related management state. ## Who Calls It Human operators use the web console; its auth backend calls the API with derived operator credentials. Trusted automation uses the Admin SDK or documented `/v1` Admin API. The optional Control plugin dispatches into the same management implementation. Workloads must not receive the API admin token or use management routes as a data-plane credential source. ## Runtime Contract | Item | Local value | | ------------------------- | -------------------- | | Port | `3000` | | Liveness | `GET /health` | | Readiness | `GET /ready` | | Metrics | `GET /metrics` | | Management prefix | `/v1` | | Optional interactive docs | `/docs` when enabled | Use [Use the Admin API](/v1.0/api/control-plane/) for route and auth reference rather than treating this service page as an endpoint catalog. Service environment variables are cataloged in [Configure Service Environment](/v1.0/operations/env-vars/). ## Write and Propagation Flow ```mermaid sequenceDiagram participant Client as Console / Admin SDK / Control participant API participant PG as Postgres participant Redis Client->>API: authenticated management request API->>PG: validate zone and commit state + outbox API-->>Client: resource or typed error API->>Redis: dispatch signed outbox event ``` The state change and outbox row commit together. Redis publication follows asynchronously. A successful API response can therefore precede policy reload, revocation, or audit consumption by a short interval. ## Failure Implications | Failure | User impact | | --------------------------- | ------------------------------------------------------------------------------------------------ | | Postgres unavailable | Reads/writes and readiness fail. | | Redis unavailable | Readiness or outbox signals degrade; committed state remains in Postgres for retry. | | Dead/old outbox rows | Consumers may retain stale policy, revocation, or audit state; Diagnostics surfaces the backlog. | | Missing/invalid admin token | `401`; the console may report not connected or unauthorized. | | Valid token without scope | `403`; narrow the operation or grant the required operator scope. | | Secret backend unavailable | Secret-bearing create, reveal, rotation, or dependent issuance fails closed. | Do not call the API's STS-coordination or internal service routes directly. The owning console or Admin SDK workflow supplies the required authentication and validation. ## Next Step [Coordinate Session State](/v1.0/services/coordinator/) or [Manage Product Objects](/v1.0/runtime-console/admin/). --- # Coordinate Session State # URL: https://docs.caracal.run/v1.0/services/coordinator/ # Markdown: https://docs.caracal.run/markdown/v1.0/services/coordinator.md # Type: reference # Concepts: # Requires: --- Coordinator owns governed Sessions, service leases, invocations, Delegations, graph epochs, and their durable outbox. ## Who Calls It Applications use Caracal SDK Session and Delegation APIs or the documented Coordinator API. The console BFF uses an operator credential to inspect and intervene. Top-level runtime commands do not manage Coordinator state. Do not place the operator Coordinator token in workload code. Do not write Coordinator tables or Redis lifecycle topics directly. ## Runtime Contract | Item | Local value | | --------- | -------------- | | Port | `4000` | | Liveness | `GET /health` | | Readiness | `GET /ready` | | Metrics | `GET /metrics` | Use [Use Coordinator API](/v1.0/api/coordinator/) for routes and SDK contracts. Service environment variables are cataloged in [Configure Service Environment](/v1.0/operations/env-vars/). ## Operational Flow Session and Delegation writes commit to Postgres with outbox rows. Publishers relay lifecycle, invocation, delegation-invalidation, and revocation events through Redis. STS consults authoritative state before issuing authority tied to that lineage. Task Sessions expire by TTL. Service Sessions require heartbeat renewal. Sweepers mark stale leases, enforce invocation deadlines, expire Delegations, and clean terminal records according to retention settings. ## Failure Implications | Failure | User impact | | --------------------------- | ------------------------------------------------------------------------------------------- | | Postgres unavailable | Session/Delegation operations and readiness fail. | | Redis/outbox delayed | Downstream lifecycle and invalidation views lag; durable state remains available for retry. | | Service heartbeat stops | The Session becomes unhealthy or expires even if its process still exists. | | Sweeper unavailable | Stale state remains longer; alert on job/readiness metrics. | | Operator credential missing | Console Session and Delegation views fail; workload SDK credentials are a separate path. | ## Next Step [Issue Mandates](/v1.0/services/sts/) for the authority path, [Coordinate Sessions](/v1.0/architecture/delegation-flow/) for the cross-service flow, or [Manage Runtime Authority](/v1.0/runtime-console/agents/) for the operator workflow. --- # Issue Mandates # URL: https://docs.caracal.run/v1.0/services/sts/ # Markdown: https://docs.caracal.run/markdown/v1.0/services/sts.md # Type: reference # Concepts: # Requires: --- STS authenticates application or workload proof, evaluates current authority, and issues short-lived mandates or configured provider credentials. ## Supported Callers and Routes | Route | Caller | | ---------------------------------------------------- | --------------------------------------------- | | `POST /oauth/2/token` | SDK, documented token client, Gateway | | `POST /v1/run/manifest` | `caracal run` | | `POST /v1/run/credential` | `caracal run` | | `GET /.well-known/jwks.json?zone_id=...` | Verifiers | | `GET /approvals/{id}` | Waiting client with the required hold context | | `POST /approvals/{id}/decision` | Federated user decision flow | Routes under `/internal/` for policy simulation/status and zone signing-key rotation are service-to-service APIs. Do not call or expose them as workload APIs; the API service owns those operations. ## Runtime Contract | Item | Local value | | --------- | ------------------------------- | | Port | `8080` | | Liveness | `GET /health` | | Readiness | `GET /ready` | | Metrics | `GET /metrics`, `/metrics.json` | Service environment variables are cataloged in [Configure Service Environment](/v1.0/operations/env-vars/). ## Identity Boundary STS does not authenticate end users and does not invent a user `sub`. It accepts a Federated user's token only from a registered Federated user issuer, then treats its stable `sub` as opaque. Without a subject token, the application's own identity is the Subject. ## Synchronous Dependencies Issuance needs Postgres product, policy, authority, Session, Delegation, approval, signing, and secret state. It also consumes Redis invalidation and revocation state, requires the Secret Store KEK, verifies Gateway HMAC proof on the Gateway path, and emits audit evidence to Redis or replay storage. ## Failure Posture Invalid client proof, Federated user issuer, resource, policy, operation, scope, Authority record, Session, Delegation, approval, signing key, or Gateway signature denies issuance. There is no fallback token. Resource mandates are capped at 15 minutes and Session mandates at 60 minutes. An `interaction_required` response means policy raised an Approval hold. Approval alone is not issuance; the client retries and consumes the hold once. Use [Use STS Endpoint](/v1.0/api/sts/) for exact request and error contracts and [Exchange Tokens](/v1.0/architecture/token-exchange-flow/) for the trust flow. ## Next Step [Protect Upstreams](/v1.0/services/gateway/). --- # Protect Upstreams # URL: https://docs.caracal.run/v1.0/services/gateway/ # Markdown: https://docs.caracal.run/markdown/v1.0/services/gateway.md # Type: reference # Concepts: # Requires: --- Gateway is the HTTP enforcement boundary for configured upstream resources. ## Supported Request A client sends an authorization bearer mandate, `X-Caracal-Resource`, and the intended HTTP request. Gateway resolves the configured zone/resource binding; callers do not select an arbitrary upstream. ```mermaid sequenceDiagram participant Client participant Gateway participant STS participant Upstream Client->>Gateway: mandate + resource + HTTP request Gateway->>Gateway: verify token, replay, revocation, operation, path, binding Gateway->>STS: signed per-request exchange STS-->>Gateway: resource authority + private upstream directive Gateway->>Upstream: sanitized request + configured credential Upstream-->>Gateway: HTTP response or stream Gateway-->>Client: proxied response ``` ## Deny Before Upstream Gateway does not contact the upstream when the bearer is missing, malformed, oversized, expiring, replayed, revoked, or signature-invalid; the resource header or binding is missing; an enforced operation/scope is absent; the path traverses; STS fails or its circuit is open; or host safety rejects the destination. It rejects caller-supplied `X-Caracal-Client-ID`, strips hop-by-hop and caller authorization headers, and applies only the upstream credential returned through the trusted exchange. ## Runtime and Protocol Limits | Item | Behavior | | ------------------ | ------------------------------------------------------------------- | | Port | Local `8081` | | Liveness/readiness | `/health`, `/ready` | | Monitoring | `/metrics`, `/metrics.json` | | Request size | `MAX_REQUEST_BYTES`, 10 MiB by default | | HTTP streaming | Supported, including SSE; revocation is rechecked between chunks | | WebSocket upgrade | Not supported; upgrade headers are stripped | | Revocation reload | `POST /internal/revocations/reload`, service/operator-internal only | Protect WebSocket services in process with a verification package or framework adapter. Service environment variables are cataloged in [Configure Service Environment](/v1.0/operations/env-vars/). ## Dependency Implications Gateway depends synchronously on STS and needs Postgres/Redis-backed binding, key, replay, and revocation state. It buffers audit evidence in replay storage if Redis/Audit delivery is unavailable. In published modes, replay/JTI and authority uncertainty fail closed; `JTI_FAIL_OPEN` is forbidden. Use [Proxy Through Gateway](/v1.0/api/gateway/) for the client contract and [Harden Production](/v1.0/operations/tls-hardening/) for network placement. ## Next Step [Ingest Audit Evidence](/v1.0/services/audit/). --- # Ingest Audit Evidence # URL: https://docs.caracal.run/v1.0/services/audit/ # Markdown: https://docs.caracal.run/markdown/v1.0/services/audit.md # Type: reference # Concepts: # Requires: --- Audit consumes signed events from Redis, verifies them, writes append-only evidence to Postgres, manages failed delivery, and exposes operator search and metrics on local port `9090`. ## Supported Read Paths Human operators use the console **Audit** workspace. Zone-scoped automation uses the API service's documented audit routes. The Audit service's direct search and DLQ routes are operator endpoints protected by `AUDIT_ADMIN_TOKEN`; when that token is not configured, they return `404`. | Route | Purpose | | ------------------------------------------- | ------------------------------------- | | `GET /health`, `/ready` | Liveness and dependency/lag readiness | | `GET /metrics`, `/metrics.json` | Authorized monitoring | | `GET /api/audit/search` | Direct operator search | | `GET /api/audit/dlq`, `/api/audit/dlq/{id}` | Inspect failed events | | `POST /api/audit/dlq/replay` | Retry selected failed evidence | Do not send application audit records to these routes. Caracal services publish the signed event contract. Service environment variables are cataloged in [Configure Service Environment](/v1.0/operations/env-vars/). ## Delivery and Recovery Audit consumes `caracal.audit.events` in the `audit-ingestor` group. It drains its pending entries, claims orphaned work, retries failures, and moves events beyond the delivery limit to the DLQ. STS and Gateway replay volumes preserve events that could not reach Redis at emission time. ## Integrity and Readiness `AUDIT_HMAC_KEY` verifies producer signatures in published modes. Content-hash mismatch, chain breaks, and HMAC failures are security signals, not retryable formatting issues. The database role cannot update or delete evidence rows. Readiness considers DLQ size, consumer lag, and oldest pending-entry age. A healthy HTTP process can therefore be not ready because evidence is no longer being retained within the configured operating bounds. ## Operator Response 1. Check Audit readiness and metrics. 2. Check Redis connectivity, lag, pending entries, and replay volume growth. 3. Inspect DLQ detail before replaying. 4. Treat integrity failures as incidents; do not replay tampered payloads as trusted evidence. 5. Confirm recovery with a fresh protected request and console decision trace. Use [Audit and Request Traces](/v1.0/concepts/audit-ledger/) for evidence semantics and [Configure Alerts](/v1.0/operations/alerts/) for thresholds. ## Next Step [Automate Management](/v1.0/services/control/) when a trusted remote client needs product-management automation. --- # Automate Management # URL: https://docs.caracal.run/v1.0/services/control/ # Markdown: https://docs.caracal.run/markdown/v1.0/services/control.md # Type: reference # Concepts: # Requires: --- Control is the zone-bound remote automation counterpart to the web console. It is an optional plugin inside the API service, not a separate deployable or port. ## Choose Control or the Admin SDK Use the Admin SDK when trusted automation can call the Admin API directly with an appropriately scoped operator credential. Use Control when automation needs a self-describing, zone-bound invoke surface with short-lived Control tokens, replay protection, per-command scopes, rate limits, and mandatory audit. Neither surface manages local stack lifecycle or launches workloads. Control must never appear as a top-level `caracal` runtime command. ## Enablement and Route | Item | Contract | | -------------------- | ------------------------------ | | Host | API service, local port `3000` | | Invoke | `POST /v1/control/invoke` | | Build-time mount | `CARACAL_CONTROL_ENABLED=true` | | Runtime gate | `CONTROL_GATE_FILE` must exist | | Default Helm posture | Disabled | Removing the gate file makes invoke return `503` without restarting API. API health and readiness remain the service probes. ## Credential Flow ```mermaid sequenceDiagram participant Operator as Console operator participant STS participant Client as Trusted automation participant Control as API Control plugin Operator->>Control: create zone-bound Control key in console Client->>STS: exchange key for short-lived caracal-control token STS-->>Client: scoped one-use token Client->>Control: POST /v1/control/invoke Control->>Control: verify gate, JWT, scope, JTI, rate, audit Control-->>Client: result or structured error ``` The token's zone comes from the Control key. A caller must not select another zone. Each token is replay-protected, so mint a fresh token for each invoke. ## Supported Management Model Control exposes the shared management catalog, including imperative noun/verb operations and declarative `ensure` and `state plan|verify|apply` workflows. Prefer declarative reconciliation for repeatable CI: it is idempotent, reports per-object outcomes, authorizes each touched noun, and can dry-run before writes. Use `catalog describe` to discover commands, scopes, flags, and desired-state schema instead of hardcoding a copied catalog. Authority records and governed Sessions are separate nouns. For exact payload examples, scopes, reconciliation documents, and error envelopes, use [Use the Admin API](/v1.0/api/control-plane/) and [Bootstrap Control State](/v1.0/examples/control-bootstrap/). ## Failure Behavior | Condition | Result | | ----------------------------------------- | ------------------------------------------------------ | | Gate absent | `control_disabled`, `503` | | Token missing or invalid | `unauthorized`, `401` | | Token reused | `token_replay`, `401` | | Scope or policy insufficient | `denied`, `403` | | Bound zone conflicts with requested state | `zone_mismatch`, `409` | | Subject/source exceeds rate | `rate_limited`, `429` | | Required audit cannot be recorded | `audit_unavailable`, `500`; operation does not execute | | Body exceeds 64 KiB | `413` | Control depends on STS JWKS/issuer validation, Redis replay/rate state, the API's downstream credential, and durable audit. Keep it behind TLS and store the long-lived Control key in the automation platform's secret store. ## Manage Keys Only in Console Control keys are managed in **Services → Control**. They are zone-bound applications restricted to `control::` scopes. Control operations cannot mutate, rotate, or delete Control-key applications, so automation cannot use one key to take over another. Secret reveal is audited. ## Next Step [Choose the Right Surface](/v1.0/runtime-console/cli-and-console/) for the complete boundary or [Enforce Boundaries](/v1.0/architecture/trust-boundaries/) for trust placement. --- # Use Reference # URL: https://docs.caracal.run/v1.0/reference/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference.md # Type: landing # Concepts: # Requires: --- Use Reference when you need an exact value or contract. Use [Guides](/v1.0/guides/) for procedures and [API Reference](/v1.0/api/) for endpoint behavior. ## Find the Canonical Answer | Question | Canonical page | | -------------------------------------------------------- | -------------------------------------------------------- | | What does this public term mean? | [Glossary](/v1.0/reference/glossary/) | | Why did this request fail? | [Error Codes](/v1.0/reference/errors/) | | Which setting controls this behavior? | [Configuration Keys](/v1.0/reference/configuration/) | | Which configuration source wins? | [Configuration Order](/v1.0/reference/config-precedence/) | | What is the current default or limit? | [Defaults and Limits](/v1.0/reference/defaults-and-limits/) | | What exit status does automation receive? | [CLI Exit Codes](/v1.0/reference/runtime-exit-codes/) | | Which runtimes and version combinations are supported? | [Compatibility](/v1.0/reference/compatibility/) | | Which packages and images belong to one release? | [Release Map](/v1.0/reference/release-package-runtime-map/) | | Which bytes and JSON fields are interoperable? | [Wire Contracts](/v1.0/reference/interoperability-contracts/) | | How do I answer a likely product or operations question? | [FAQ](/v1.0/reference/faq/) | ## Read Values by Layer * **Product language** is the terminology shown in the web console and SDK facades. * **Runtime configuration** controls the CLI, SDK loaders, and deployed services. * **Protocol behavior** covers HTTP fields, raw JWT claims, status codes, and schemas. * **Release behavior** covers SemVer, package lockstep, images, and documentation versions. Do not substitute a protocol name for its public product term. Public surfaces use **Session ID** and **Delegation ID** while raw Coordinator and JWT contracts retain fields such as `agent_session_id` and `delegation_edge_id`. ## Source of Truth Code, package manifests, deployment configuration, and checked-in schemas are authoritative. This section explains those sources; it does not create a second contract. ## Next Step Search [FAQ](/v1.0/reference/faq/) for a question, or open [Glossary](/v1.0/reference/glossary/) before naming a product concept in code or documentation. --- # FAQ # URL: https://docs.caracal.run/v1.0/reference/faq/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/faq.md # Type: reference # Concepts: # Requires: --- import FaqRegistryScript from '../../../../components/FaqRegistryScript.astro'
32 results
FAQ-001 Platform

What problem does Caracal solve?

Caracal gives agents and automated workflows short-lived, policy-approved authority instead of long-lived credentials. The agent asks for scoped authority at the moment it acts, STS evaluates policy, Gateway or an adapter enforces the mandate, and Audit records the decision and result.

See Authority and Enforcement and the Caracal Mental Model.

Related: FAQ-002, FAQ-003, FAQ-016

FAQ-002 Platform

Is Caracal an identity provider, secrets manager, or API gateway?

No. Caracal is an authority broker for agent and workload actions. It can sit in front of HTTP resources like a protected Gateway, and it can broker provider credentials, but it does not replace your IdP, your static config store, or your general API management layer.

Use an IdP for human login, a secret manager for static application configuration, and Caracal when an agent or service needs scoped, auditable authority for a resource.

Related: FAQ-001, FAQ-009, FAQ-013

FAQ-003 Architecture

What should a zone represent?

A zone should represent a trust boundary: the set of resources, sessions, policies, signing keys, and audit records that are allowed to share authority state. Use a separate zone when two workloads need independent signing keys, policy activation, or audit trails. Use one zone with separate resources when the same trust boundary protects multiple upstreams.

For examples, see Model Your Application in Caracal.

Related: FAQ-004, FAQ-009, FAQ-011

FAQ-004 Architecture

Does this repository implement managed multi-tenancy?

No. The open-source product gives you Zones as an isolation primitive. You can model customers, environments, or trust tiers with Zones and automate them through the Admin API. Managed tenant, team, SSO, and hosted lifecycle are not implemented in this repository.

To serve many of your own customers from one deployment without per-customer zones, see Serve Your Own Customers.

Evaluate any separate offering from its own current source, contract, and demonstrated behavior; this documentation cannot verify it.

Related: FAQ-003, FAQ-019

FAQ-005 Identity

What is the difference between an application, Subject, Authority record, and Session?

An application is registered software that authenticates to Caracal. A Subject is the opaque JWT sub identity the work is attributed to - the application itself by default, or a Federated user supplied by a trusted identity provider. An Authority record is one STS exchange record. A Session is one governed Coordinator execution.

These identifiers are distinct: Authority record ID identifies one STS exchange, Root authority record ID identifies its exchange-chain root, and Session ID identifies one governed Coordinator execution.

Related: FAQ-006, FAQ-007, FAQ-008

FAQ-006 Identity

Should I create one application per agent?

No. This is the most common modeling mistake, and it does not match how Caracal scales. An application and an agent are different layers:

  • An application is the credentialed security boundary. It is operator-provisioned (managed) or dynamically registered (DCR), holds a server-owned secret, and is the identity Caracal authenticates. Creating one is a deliberate, secret-bearing act.
  • An Session is the scalable runtime unit. The process that already holds the application credential creates Sessions at runtime - no secret, no registration, no Console step. One application backs many concurrent Sessions (up to 200 per application by default).

The default model is one managed application per durable service, with many Sessions under it. When that service fans out across sub-agents or jobs, each execution is a new Session under the same application. Policy and audit tell them apart by Session ID, lifecycle, labels, and Delegation chain.

Use a separate application only when an independently launched agent or workload needs an isolated, expiring credential and registry-visible identity. DCR supplies that boundary and binds exactly one Session.

See Identities and Applications and the Caracal Mental Model.

Related: FAQ-005, FAQ-007, FAQ-008

FAQ-007 Identity

When should I use a managed application versus DCR?

Use a managed application for durable software you intentionally operate: a backend service, Gateway application, orchestrator, or agent runtime. Ordinary Session fan-out does not need DCR. Use DCR only when an independently launched identity needs an isolated, auto-expiring credential boundary, such as a per-tenant or per-integration process. DCR applications are registered through the Admin API, always expire, bind exactly one Session, and cannot parent further Sessions.

See Identities and Applications.

Related: FAQ-005, FAQ-006, FAQ-008

FAQ-008 Identity

If many agents share one managed application, can policy and audit still tell them apart?

Yes, with one important distinction between attribution and credential isolation.

Each execution has a unique Session ID. Policy and audit record it with lifecycle, labels, parentage, and Delegation context. The web console **Sessions** view shows the same execution records.

Authority, not labels, is the security boundary. Labels are asserted by the credentialed workload and help policy and audit classify work; scopes, Delegations, and policy contain compromised workloads.

Filter the Admin API audit endpoint by session\_id for one exact Session or by label for a role across many Sessions.

See Identities and Applications and Model Your Application in Caracal.

Related: FAQ-005, FAQ-006, FAQ-007

FAQ-009 Resources

What is the difference between a resource and a provider?

A resource is the protected target and policy audience: the thing a mandate authorizes access to. A provider describes how Gateway authenticates upstream: no credential, Caracal mandate, OAuth, API key, or bearer token.

Keep target identity, scopes, and upstream URL on the resource. Keep secrets, token endpoints, OAuth settings, API keys, and bearer tokens on the provider.

Related: FAQ-010, FAQ-011, FAQ-013

FAQ-010 Resources

Why must the resource identifier stay stable if the upstream URL can change?

Policies, grants, mandates, and audit records refer to the resource identifier. If you use a mutable deployment hostname as the identifier, changing infrastructure also changes your authority boundary and breaks audit continuity. Use a stable audience URI such as resource://pipernet, then change the upstream URL when routing changes.

Related: FAQ-003, FAQ-009, FAQ-011

FAQ-011 Resources

How should I design scopes?

Use small action-oriented scopes such as pipernet:read, piperchat:comment, or mcp\:tool:call. Do not encode environment, tenant, user, or hostname into scope names when that data belongs in the zone, principal, resource, or policy input.

Scopes answer "what action is allowed?" Resource identifiers answer "what target is protected?"

Related: FAQ-003, FAQ-009, FAQ-012

FAQ-012 Resources

Do I manage grants directly?

In the current web-console flow, you usually define Resources, scopes, Applications, Subjects, and Policy rather than managing Grants as a separate daily object. Policy data can declare role-to-scope grants, while managed delegated Grants are lifecycle and revocation records. The active Policy set still makes the final allow, deny, or Approval decision.

If access is denied, inspect the active policy, Subject, application, resource, and scopes through request trace.

Related: FAQ-011, FAQ-016

FAQ-013 Security

Is an application secret the same as a provider credential?

No. An application secret authenticates the application to Caracal. A provider credential authenticates Gateway or STS to an upstream provider such as Google, Slack, OpenAI, or an internal API. Agents should authenticate to Caracal and receive short-lived mandates; they should not receive long-lived provider credentials.

See Define Resources and Providers.

Related: FAQ-002, FAQ-009, FAQ-014

FAQ-014 Security

When should I use per-user OAuth instead of a shared provider credential?

Use a connected upstream account (oauth2\_authorization\_code) when the upstream call must act as a specific consented account: a human completes the provider's consent screen once for the shared account, or once per Subject when the connection is bound to a specific customer (typically a Federated user). Use a shared service credential (oauth2\_client\_credentials, api\_key, or bearer\_token) when the agent acts as the application with one operator-configured credential.

The concrete setup fields are in Provider Recipes.

Related: FAQ-013, FAQ-016

FAQ-015 Runtime

Why are zone and policy commands in the web console instead of the caracal CLI?

The top-level caracal CLI is intentionally limited to local runtime lifecycle, process execution, upgrades, Console sign-in admission, and web console launch: up, down, status, upgrade, purge, allowlist, run, and web. Product-management workflows such as zones, applications, providers, resources, policies, audit, diagnostics, agents, and delegation live in the web console, Control API, and Admin SDK so they use one management surface and do not drift into duplicated CLI commands.

See Choose the Right Surface.

Related: FAQ-017, FAQ-018

FAQ-016 Operations

What is the difference between a 403 from STS and a 403 from Gateway?

A 403 from STS means the exchange was authenticated but Policy did not allow the requested Resource scopes, or an Approval, Grant, or Session condition blocked issuance. A 403 from Gateway or a verifier means the request reached a protected boundary but the Mandate, Resource binding, scope check, revocation state, or route safety check failed.

Use request trace with the request ID to identify the surface before changing policy or resource configuration.

Related: FAQ-011, FAQ-012, FAQ-017

FAQ-017 Operations

Where is the diagnostic bundle or doctor command?

The diagnostic bundle is exposed through existing surfaces instead of a separate top-level command. Use caracal status --json for runtime status, web console Diagnostics for Doctor checks (health, readiness, zones, preflight), audit for recent decisions, and request trace for a known request ID.

See Troubleshoot by Symptom and Inspect Diagnostics and Audit.

Related: FAQ-015, FAQ-016, FAQ-018

FAQ-018 Operations

Why is an audit event missing?

First confirm the request reached a Caracal-protected boundary. If it did, check the selected zone, time window, request ID, Audit service readiness, Redis stream health, replay backlog, and DLQ. If the request failed before STS, Gateway, Coordinator, or an adapter emitted evidence, there may be no action-result event for that boundary.

Start with Inspect Diagnostics and Audit and Debug Infrastructure Issues.

Related: FAQ-016, FAQ-017

FAQ-019 Scope

What does this documentation verify?

This documentation verifies only behavior implemented by the self-hosted open-source product and supported deployment assets in this repository. It does not establish managed hosting, SSO, SCIM, organization RBAC, commercial support, an SLA, certification, or regulatory conformity.

Use Review OSS Adoption Readiness to separate repository evidence from deployment and organizational responsibilities.

Related: FAQ-004

FAQ-020 Identity

Two agents share the same application and labels - how do I tell which one acted?

Every governed execution has one canonical Session ID. It is returned when session() or startSession() starts the Session and is stamped onto its token exchanges and audit events.

Identical Sessions are interchangeable on purpose. A hundred \["pricing-worker"] Sessions fanned out under one application are meant to be fungible - that is how fan-out works, and it is why labels are a descriptor rather than a unique name. When you need to tell Sessions apart by meaning rather than by raw id, give them distinguishing labels, attach business correlation in metadata, or propagate a trace\_id through the work.

To investigate, the zone audit endpoint filters directly on these fields: query session\_id to follow one exact Session end to end, or label to scope to a role across a whole fleet of Sessions. The web console audit view exposes both filters.

To see Sessions that have ended, the Admin API exposes GET /v1/zones/\{zone}/sessions. It supports status, lifecycle, label, parent Session, and application filters plus CSV export.

Related: FAQ-006, FAQ-008, FAQ-021

FAQ-021 Security

A started Session uses application policy and narrowed authority - which wins?

Both apply, and they compose as a strict intersection - a logical AND - so the narrower of the two always wins. They are two independent layers with different jobs: a grant (the Delegation a narrowing session(grant=…) creates) caps which scopes the token may carry at all, while policy decides whether the action is allowed. Neither layer can ever add authority; each one can only subtract.

At token exchange a resource is released only if it passes every gate: the requested scopes must be within the resource's own scopes, within the grant edge's scopes (which is itself re-validated to be within the parent's authority), the resource must fall inside the delegation, and policy must return allow. The effective authority is therefore policy ∩ grant ∩ resource ∩ delegation. This holds in both directions: if policy is the narrower of the two, policy wins and the grant cannot widen past it; if the grant is the narrower, the grant wins, because the token cannot request scopes outside the grant and resources outside the delegation are rejected even when policy would have allowed them.

There is no clash. A plain session() under a top-level parent runs at the application's policy-bounded authority. Under a narrowed parent, Coordinator mirrors the parent's Delegation onto the child. Use session(authority=Authority.narrow(\[...])) when the child should hold less. Every layer is subtractive.

See Delegation and Policy.

Related: FAQ-006, FAQ-008, FAQ-011

FAQ-022 Identity

If child Sessions use the parent's application, when is DCR used?

A DCR application is used by authenticating as it, not by starting it as a child. Because session() uses the caller's application, a DCR application bound to exactly one Session cannot parent another Session.

The credential boundary between durable managed identities and short-lived DCR identities is named by registration\_method (managed vs DCR), not by an agent's lifecycle. A Session's lifecycle is either task (the default) or service (heartbeat-leased), and a DCR application cannot host a service Session, so its one Session is always a task Session. A short-lived worker is therefore not a separate lifecycle - it is an ordinary task Session with a TTL (see FAQ-023).

The Admin API registers the DCR application and returns a one-time client secret and short expiry. An orchestrator injects those credentials into an independently launched workload, which authenticates with client\_credentials and starts its single Session. STS creates Authority records as that workload exchanges.

The credential split is deliberate: minting a new credentialed identity is a privileged control-plane action, so a runtime cannot register applications for itself. The SDK consumes a DCR application by being configured with its credentials; only an operator or orchestrator with Admin API access mints one.

Because a DCR root has no parent and no Delegation, its authority is decided entirely by policy, not by inheritance - and policy is default-deny, so a DCR identity opens no tools until a policy grants it scopes. Policies receive input.principal.registration\_method, so you write one policy class targeting registration\_method == "dcr" (optionally narrowed by labels, resource, or zone) that covers every DCR application; you do not author a policy per DCR app. Pair that with per-tenant or per-job resources to keep each DCR identity scoped to its own data.

DCR provides a credential-isolated, independently revocable, auto-expiring, registry-visible identity for a per-tenant, per-job, or per-integration workload. It does not add execution attribution; Session ID already provides that.

See Identities and Applications and FAQ-007.

Related: FAQ-006, FAQ-007, FAQ-008

FAQ-023 Identity

How do I model an orchestrator, managers, and short-lived workers?

Model the runtime under one managed application. Every execution is a Session. A short-lived worker is a task Session; least privilege is expressed with narrowed authority, and ttl\_seconds adds an optional wall-clock cap.

  • The orchestrator uses a long-lived Session from startSession() when it needs a heartbeat lease.
  • Each manager is a plain session() that inherits the application's authority.
  • Each task worker is session(authority=Authority.narrow(\[...])) - bounded to a subset of authority and auto-terminated when its block exits - with ttl\_seconds=… added only when you also want a wall-clock cap.

A Session's authority comes from its application, bounded by policy. A parent matters when authority is narrowed: Authority.narrow(\[...]) creates a Delegation that the server validates as a subset of the parent's effective authority. Cross-application authority uses delegate(to=peer); the receiver consents by presenting the opaque, target-bound Delegation ID.

See Identities and Applications, Delegation, and FAQ-022.

Related: FAQ-006, FAQ-008, FAQ-022

FAQ-024 Security

If A narrows authority to B and B starts C, does C stay bounded?

Yes. inherit carries the parent's effective authority forward, so least privilege is transitive down a same-application Session tree. Suppose A starts B with Authority.narrow(\[pipernet:read]), then B starts C:

  • With inherit, Coordinator mirrors B's Delegation onto C. C stays within B's scopes, resource, constraints, and expiry.
  • With further narrowing, Coordinator rejects C unless C ⊆ B.
  • If B has no inbound Delegation, C runs under the application's authority bounded by policy.

Cross-application authority is never inherited automatically; it requires delegate(to=peer), followed by receiver presentation of the opaque, target-bound Delegation ID. Application plus policy remains the hard boundary.

See Delegation and FAQ-023.

Related: FAQ-021, FAQ-023, FAQ-008

FAQ-025 Identity

What is the difference between a task and a service lifecycle - and how do I model a task-and-die worker versus a time-limited one?

Every runtime actor is an Session; lifecycle only describes how it runs. There are two lifecycles, and they map directly onto the two SDK primitives:

  • Task - created with session(), recorded as lifecycle = "task". It lives for the duration of its task: when its block exits it is terminated automatically. This single behavior covers both of the cases you are distinguishing. A "do one task and die" worker (for example a search sub-agent) is just session() whose block returns when the task is done. A "live up to N seconds then expire" worker is the same session() with ttl\_seconds=N, which adds a hard wall-clock cap enforced by the TTL sweeper. The difference between "task-and-die" and "time-limited" is whether you set a TTL, not a different lifecycle.
  • Service - created with startSession(), start\_session(), or StartSession() and recorded as lifecycle = "service". This long-lived Session is governed by a renewable heartbeat lease rather than the task TTL sweeper.

The stored lifecycle column carries exactly these two protocol values. Use a long-lived Session when work must outlive one task; otherwise use session() with an optional TTL.

See Identities and Applications and FAQ-023.

Related: FAQ-022, FAQ-023, FAQ-007

FAQ-026 Identity

Can two DCR applications have different policies, and can a DCR Session start children?

Different policies per DCR app: yes. Policy evaluation receives the full principal, including the specific input.principal.id (the application id), input.principal.labels, and input.principal.registration\_method. Matching on registration\_method == "dcr" is just the convenient way to write one rule that covers every DCR app; when two DCR apps need different authority, target their distinct application ids or labels, or scope them to different resources. There is no requirement that all DCR apps share a policy.

Can a DCR application start child Sessions: no. It binds exactly one Session; a second start is rejected with the protocol error dcr\_application\_already\_bound.

Which Sessions can be parents: Sessions under managed applications can start child Sessions. A task parent cannot start a service child; the protocol reports task\_session\_cannot\_start\_service. DCR Sessions are isolated leaves, reported by the protocol as dcr\_application\_cannot\_start\_child and dcr\_application\_cannot\_be\_child.

See Identities and Applications and FAQ-022.

Related: FAQ-022, FAQ-025, FAQ-007

FAQ-027 Runtime

Where does Caracal read caracal.toml from?

SDK loaders read exactly the path in CARACAL\_CONFIG. If that variable is unset, they use SDK environment variables. They do not search the current directory, home directory, or OS Caracal config directory for caracal.toml. caracal run does not use an SDK profile; it loads a Workload identity locally and fetches launch bindings from STS.

See Configuration Order.

Related: FAQ-015, FAQ-028

FAQ-028 Runtime

Does caracal run renew injected credentials?

No. It fetches bindings and mints each credential once before starting the child. The child receives a scrubbed environment and exits with those credentials' existing expiry. Use a Caracal SDK for long-running software that must exchange on demand.

See Run Workloads.

Related: FAQ-015, FAQ-027

FAQ-029 Platform

Can Gateway proxy WebSockets?

No. Gateway proxies HTTP request/response traffic and streamed responses such as SSE, but strips hop-by-hop headers including Upgrade. Protect WebSocket services with an in-process verifier or framework adapter at the service edge.

See Proxy Through Gateway and Verification Layer Overview.

Related: FAQ-002, FAQ-016

FAQ-030 Operations

Why did retrying an STS exchange create uncertainty?

Mandate issuance uses one network attempt. If the response is lost, the server might have minted successfully, so the SDK cannot prove that a retry is the same issuance. Reconcile the operation or explicitly retry according to the protected action's idempotency contract. Session and Delegation creation are different: Coordinator uses durable idempotency receipts and can replay their creation response safely.

See Safe Retries and Idempotency.

Related: FAQ-016, FAQ-028

FAQ-031 Platform

Which Caracal versions can I mix?

None intentionally. Caracal packages, images, chart metadata, and binaries release in lockstep and are tested as one version. Pin exact versions and upgrade them together. On the pre-1.0 line, patch releases preserve documented public behavior; minor releases can change public contracts.

See Compatibility and Release Map.

Related: FAQ-002, FAQ-032

FAQ-032 Platform

Why does the documentation URL say v0.2 before v0.2.0 is released?

v0.2 is the configured first documentation target. Before stable v0.2.0, unversioned source is served on shareable /v0.2/ routes, but no snapshot exists. The stable release creates and registers that snapshot. RCs create no snapshot, and later v0.2.x patches update the same current minor.

See Documentation Versions.

Related: FAQ-031

## Next Step Use [Glossary](/v1.0/reference/glossary/) when you need canonical terms for concepts, API names, web console labels, and examples. --- # Glossary # URL: https://docs.caracal.run/v1.0/reference/glossary/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/glossary.md # Type: reference # Concepts: # Requires: --- Use these terms consistently across docs, API names, web console labels, and examples. | Term | Meaning | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Application | Registered client identity in a zone that software runs as; confidential applications hold a secret and can exchange credentials. | | Authority record | One STS exchange record. Its Authority record ID anchors revocation, ancestry, and audit for authority minted by that exchange. Stored in the `authority_records` table. | | Audit ledger | Append-only evidence stream and database records for decisions and operations. | | Approval | A policy-raised hold that pauses a request until a human with authority approves or rejects it; surfaced on the web console Approvals page and returned on the wire as `interaction_required`. SDKs identify it as `approvalId`; the wire carries the same value as `approval_id`. | | Caracal Operator | Governed natural-language console assistant that turns intent into reviewed, audited control-plane changes within your operator scope. | | Confinement | Policy data (`confinement`) that caps every Session carrying a matching label prefix to a fixed scope set; it can only narrow authority. | | Console | Browser-based management UI served by the packaged web tier in Compose and Helm; `caracal web` opens it during local development. | | Control API | Optional authenticated automation surface for remote management dispatch, authorized by scoped control keys. | | DCR application | Auto-expiring application created programmatically through Dynamic Client Registration for a separate temporary credential boundary; it binds to one task Session and is never created in the web console. | | Delegation | Bounded authority one session grants to another: scopes, optional resource, constraints, and expiry, revocable independently. Stored internally as an edge in the delegation graph. | | Delegation ID | Unique product identifier for a Delegation. | | Federated user | One kind of Subject: an external end-user identity supplied by a trusted identity provider. Caracal never authenticates Federated users; it verifies their tokens against a registered Federated user issuer, then federates and records the identity verbatim for attribution, connections, approvals, and revocation. | | Federated user issuer | Zone-registered trust declaration for an external identity system (issuer, JWKS URL, audience) whose end-user identity tokens the STS accepts to mint Federated users. The Admin API resource is `subject-issuers`. | | Gateway | Reverse proxy that verifies inbound authority, exchanges with STS, and forwards to upstreams. | | Grant | Access assignment for a resource. Policy `grants` data maps roles to allowed scopes per resource; managed delegated grants record Subject-level assignments and cascade revocation to the Subject's sessions. | | Guided setup | Web console checklist that walks a new zone through creating its first application, provider, resource, and active policy from live zone state. | | Launch binding | Instruction on a Workload naming an environment variable, a resource, and scopes; at launch, `caracal run` injects that resource provider's credential into the variable after a policy decision. | | Managed application | Durable, operator-provisioned application identity for known software; created in the web console or Admin API and reused across many Sessions. | | Mandate | Short-lived scoped access token (a JWT) carrying Caracal authority. | | Mandate use | The `use` claim classifying a mandate: `session` (reusable lifecycle authority for Coordinator operations), `gateway` (single-use Gateway-ingress pass), or `resource` (what the Gateway's own exchange hands the upstream path). | | Policy | Rego content that participates in allow/deny decisions. | | Policy set | Activated bundle of policy versions for a zone. | | Principal | User, service, application, or session identity participating in authority. | | Provider | Credential source that supplies what a protected upstream receives after Caracal approves a call. | | Resource | Protected API, tool, MCP server, provider target, or upstream identifier. Use a stable `resource://` URI such as `resource://pipernet`; the upstream URL can change without changing the identifier. | | Restrict | Policy data (`restrict`) forming a deny overlay: any entry denies every exchange in the zone until it is removed. | | Root authority record | Authority record at the root of an STS exchange ancestry chain. Its Root authority record ID is checked for revocation. | | Runtime profile | `caracal.toml` or environment configuration used by SDK credential loaders. | | Run manifest | Console-authored launch bindings served by STS to `caracal run`. | | Scope | Named, action-oriented permission declared on a resource (for example `pipernet:read`); policies grant scopes and mandates carry them. | | Session | Governed execution record the Coordinator holds while code runs under Caracal: it binds identity and delegated authority around whatever executes - an AI agent step, a job, a tool call - and anchors audit attribution. Started with the SDK `session()` (task, retired when the block exits) or `startSession()` (long-lived, heartbeat-leased, retired with `close`). | | Session handle | Holder-owned handle for a long-lived Session started with `startSession()`; it renews the heartbeat lease and is retired explicitly. | | Session ID | Unique product identifier for a governed Session. | | STS | Security Token Service that performs token exchange and mandate issuance. | | Subject | The identity work is done for: the JWT `sub` recorded on Authority records and mandates. Every exchange has a Subject of one of two kinds - the application itself (the default) or a Federated user. A Federated user's identifier arrives verbatim from the exchanged token and is owned by the application's own identity system. | | Subject authority record ID | Authority record ID attached to a Session for Subject attribution and lifecycle. SDK fields name it `subjectAuthorityRecordId`, `subject_authority_record_id`, or `SubjectAuthorityRecordID`; attaching it does not by itself make later resource mandates carry the Federated user's `sub`. | | System zone | Reserved `caracal.sys/` zone for the infrastructure that runs Caracal; the Operator self-governs through it and never executes against it. | | Workload | Launcher identity for software started with `caracal run`; it holds a client secret and the launch bindings that name which credentials are injected. | | Zone | Tenant and trust boundary for product state, policies, grants, sessions, and audit. | ## Naming Rules * Use `Caracal`, not informal product nicknames. * Use `Application` for the registered identity that authenticates to Caracal; use `AI agent` for the software acting under it. Authority always belongs to the Application - never write that an agent is registered, holds the credential, or is allowed by policy. * Use `Workload` only for the Launcher identity consumed by `caracal run`; describe long-running software generically as a service or process. * Use `mandate` for Caracal-issued JWT authority, not generic "token" when the distinction matters. * Use `Session` and `Session ID` for governed Coordinator executions. Use `Authority record`, `Authority record ID`, and `Root authority record ID` for STS exchange records and ancestry. * Use `Subject` only for the JWT `sub` identity. A Subject is not an Authority record or Session, and it is never only a federated identity: it is the application itself by default, or a Federated user. * Use `Federated user` for the external end-user kind of Subject; avoid "optional subject", "external subject", and "subject from IdP". * Use canonical parsed claim names in application code; reserve raw JWT names for explicit protocol tables. * Use `subjectAuthorityRecordId`, `subject_authority_record_id`, and `SubjectAuthorityRecordID` for the Subject authority record ID in SDK code. * Use `Delegation` and `Delegation ID` in product surfaces; reserve `delegation edge` and `delegation_edge_id` for storage and raw protocol references. * Use `web console` for the browser UI and `Control API` for automation. * Use `Admin API` for the `/v1` management REST surface; `control plane` describes the architecture layer, not an API name. * Use top-level `caracal` only for runtime lifecycle, `caracal run`, and `caracal web`. ## Next Step Use [Error Codes](/v1.0/reference/errors/) when a service, SDK, Gateway, or verifier returns a machine-readable error. --- # Error Codes # URL: https://docs.caracal.run/v1.0/reference/errors/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/errors.md # Type: reference # Concepts: # Requires: --- Caracal packages and services use a shared machine-readable error shape. ```json { "error": "invalid_token", "error_description": "bearer signature invalid", "requestId": "018f...", "details": {} } ``` `details` appears only when a service or package provides structured context. The same code can be emitted by different surfaces - an `access_denied` from STS is a policy decision, while an `access_denied` surfaced by an in-process verifier is a missing scope on an otherwise valid mandate. Use the `requestId` with the web console **Audit** decision trace to confirm which surface produced the error before acting. ## HTTP Classes | Status | Protocol meaning | | ------------ | ------------------------------------------------------------------------------------------------------- | | `400` | Invalid, denied, or held request; transport retry alone will not repair it. | | `401` | Credential was absent or not accepted: malformed, expired, wrong issuer/audience, revoked, or replayed. | | `403` | Identity was accepted but lacks authority for this operation or management scope. | | `404` | Object is absent, hidden by authorization, or an optional operator endpoint is disabled. | | `409` | Lifecycle transition conflicts with current state or idempotency history. | | `425`, `429` | Retryable timing or rate condition; honor `Retry-After` where provided. | | `5xx` | Service or dependency failed; use readiness, logs, and `requestId`. | ## Well-Known Codes The **Likely source** column names the surface that most often emits the code, and **First check** is the fastest thing to confirm. When a call fails, start from the surface, not the code. | Code | Meaning | Likely source | First check | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `access_denied` | Request is authenticated but not authorized. | STS policy decision, or an in-process verifier scope check. | Run `request trace`: confirm the active policy set allows the application, Subject, resource, and scopes. | | `invalid_token` | Token is missing, malformed, expired, invalid, replayed, or fails verification. | Gateway or a resource verifier. | Confirm the mandate is unexpired, the verifier trusts the issuing zone JWKS, and the token was not already consumed. | | `invalid_request` | Request is malformed or missing a required parameter. | STS exchange, Coordinator, or OAuth request parsing. | Compare the request against the endpoint contract in [API Reference](/v1.0/api/). | | `invalid_body` | Request body failed schema validation. | Admin API or Coordinator body validation. | Read `details.issues` for the exact field paths and messages. | | `resource_not_found` | Requested resource does not exist or is unavailable in the current scope. | API, or STS resource resolution. | Confirm the resource identifier and that the resource carries an upstream URL for `(zone, resource)`. | | `internal_error` | Service failed unexpectedly. | Any service. | Check service readiness and logs for the `requestId`. | | `policy_eval_failed` | Policy evaluation failed. | STS policy engine. | Confirm the active policy set compiles and that required `input` fields are present. | | `provider_rate_limited` | Provider or provider coordination rate limit denied work. | Gateway upstream or provider coordination. | Inspect provider limits and retry/backoff; confirm the provider grant is active. | | `interaction_required` | Policy holds the mint for human approval. | STS approval gate. | Wait for the hold to be decided, then retry with the approval id (`approval_id` on the wire); see [Human Approval](/v1.0/guides/human-approval/). | | `sts_unavailable` | STS could not satisfy an exchange. | STS, or Gateway's STS circuit. | Run `caracal status --ready`; confirm STS readiness, JWKS, and policy bundle freshness. | | `credential_expired_not_renewable` | Credential is too close to expiry or cannot be renewed. | STS provider-token refresh. | Reconnect the provider grant; confirm OAuth refresh configuration. | | `payload_too_large` | Request body or token exceeded the configured limit. | Gateway or API preflight. | Reduce payload size or adjust the configured limit. | | `zone_invalid` | Zone claim or route zone is invalid. Emitted as `zone_invalid` by STS and Gateway; in-process verifiers emit `invalid_zone` for the same condition. | Gateway or STS. | Confirm the request targets the correct zone and the mandate carries a matching zone claim. | | `scope_insufficient` | Required scope is missing. Emitted as `scope_insufficient` by STS and Gateway; in-process verifiers emit `insufficient_scope` for the same condition. | A resource verifier, or Gateway. | Confirm the resource defines the scope and the active policy authorizes it for this Subject. | | `operation_not_permitted` | The Gateway operation is not declared on an enforced resource, or its required scope is absent from the mandate. | STS native operation floor, on Gateway-authenticated mandate use. | Declare the operation on the resource (`operations`), or set `operation_enforcement` to `transport_uniform`; confirm the mandate carries the operation's scope. | | `session_required` | A verifier requires a governed Session identity. | In-process verifier or verify engine. | Run the call from a Session, not a bare Authority record. | | `session_lease_fenced` | A newer process generation owns this service Session lease. | Coordinator heartbeat or close. | Stop the stale holder; only the handle returned by the latest `attachSession` / `attach_session` / `AttachSession` may continue. | | `session_subject_inactive` | The Federated user's Authority record bound to the Session expired or was revoked. | Coordinator service lease operation. | Stop the Session and federate the user again before starting new attributed work. | | `delegation_required` | A verifier requires delegated authority. | In-process verifier. | Confirm the call carries a Delegation granting the required scopes. | | `chain_mismatch` | Delegation chain does not contain a required application/session. | Verifier delegation check. | Inspect the Delegation in the web console; confirm the chain includes the required hop. | | `hop_count_exceeded` | Delegation hop count exceeds the allowed maximum. | Coordinator or verifier. | Reduce delegation depth or raise the configured hop limit. | | `http_request_failed` | HTTP call failed before a valid response could be used. | SDK or Gateway upstream call. | Confirm endpoint URL, network reachability, and upstream allowlist. | | `config_missing` | Runtime or service configuration is missing. | SDK or service startup. | Confirm the runtime profile, secret file, and required environment variables. | | `approval_consumed` | Another matching retry already spent the Approval. | STS Approval consumption. | Reconcile the intended effect before requesting another Approval. | | `idempotency_key_conflict` | A durable operation key was reused with different security-relevant input. | Coordinator Session or Delegation creation. | Reuse the original input or create a deliberate new operation ID. | | `idempotency_result_inactive` | The recorded Session or Delegation result is no longer live. | Coordinator replay. | Reconcile, then use a newly versioned operation ID only for an intentional rerun. | | `session_lease_expired` | A long-lived Session lease lapsed and the Session is suspended. | Coordinator heartbeat or attach. | Resolve the cause, resume through the control plane, then attach. | | `dcr_application_already_bound` | A DCR Application already owns its single Session. | Coordinator Session start. | Reuse that Session or register another DCR Application. | | `task_session_cannot_start_service` | A task Session attempted to parent a long-lived service Session. | Coordinator Session start. | Use a service parent or create a task child. | Transport packages may map these codes into framework-specific HTTP responses. ## SDK Error Classes The SDKs surface failures through four typed classes, so callers branch on types and machine-readable fields instead of message text. The names below are the TypeScript spellings; Python and Go expose the same classes with idiomatic naming. | Class | Raised by | Carries | Handle it by | | ----------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CaracalError` | STS exchanges and mints | `code` (the table above), `httpStatus`, `requestId`, `details`, `isRetryable` | Branch on `code`; `isRetryable` hints that transport-level congestion and availability failures are worth retrying while policy outcomes are not. | | `ApprovalRequiredError` | Approval-gated mints | Everything `CaracalError` does, plus `approvalId`, `state`, `tier`, `binding`, `expiresAt` | Let `withApproval` run the flow, or persist `approvalId` and resume with `waitForApproval`. A `CaracalError` subclass, so generic handlers still catch it. | | `CoordinatorError` | Session, delegation, and heartbeat calls | `status`, `method`, `path`, `retryAfterSeconds` | Branch on `code`: fenced, terminal, and inactive-Subject errors retire the holder; `session_lease_expired` is resumable after control-plane resume; 401 means the SDK already attempted one credential refresh. | | `CredentialsUnavailableError` | Clients built on a credentials resolver | - | The resolver returned no usable credential; the call failed closed. Provision or repair the credential source - retrying without it cannot succeed. | ## Unknown Failure Surface If you only have an SDK error and not the surface, start from the symptom-first [Troubleshoot by Symptom](/v1.0/operations/troubleshooting/). It routes a denied or failing call to the right surface, the object to inspect, and the diagnostic tool to use. Retry transport congestion and availability failures only when the operation contract is idempotent. Do not automatically retry policy denial, invalid credentials, consumed Approvals, fenced leases, or changed-payload conflicts. STS issuance uses one network attempt because a lost response can hide a successful mint. :::note[FAQ] [What is the difference between a 403 from STS and a 403 from Gateway?](/v1.0/reference/faq/#faq-016) ::: ## Next Step Use [Configuration Keys](/v1.0/reference/configuration/) when an error points to missing runtime, service, or deployment configuration. ## Related Pages * [Troubleshoot by Symptom](/v1.0/operations/troubleshooting/) * [Debug Infrastructure Issues](/v1.0/operations/debugging/) * [Verify Package](/v1.0/sdks/verify/) * [Proxy Through Gateway](/v1.0/api/gateway/) --- # Configuration Keys # URL: https://docs.caracal.run/v1.0/reference/configuration/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/configuration.md # Type: config # Concepts: # Requires: --- Choose the consumer before choosing a key. Caracal has four configuration domains, and a key accepted by one layer is not automatically accepted by another. | Domain | Used by | Choose it when | | -------------------------- | --------------------------------------------------- | ------------------------------------------------------------- | | Workload identity | `caracal run`. | A Workload launches with server-side credential bindings. | | SDK profile | SDK clients. | An Application needs Session, Delegation, or Resource access. | | Service environment config | API, STS, Gateway, Audit, Coordinator, and web BFF. | A service needs URLs, secrets, limits, or readiness settings. | | Deployment values | Helm, Compose, Postgres, and Redis. | Operators size, schedule, expose, or secure infrastructure. | ```mermaid flowchart TD Need{What are you configuring?} Need -->|caracal run launch| Identity[Workload identity env plus console Launcher bindings] Need -->|SDK credentials| Profile[SDK profile and CARACAL_CONFIG] Need -->|service behavior| Env[Service environment variables] Need -->|deployment shape| Deploy[Helm values or Compose files] Identity --> Precedence[Config precedence reference] Profile --> Precedence Env --> Ops[Environment variables reference] Deploy --> Platform[Operations deployment pages] ``` ## Workload Identity Keys | Key | Meaning | | ------------------------------ | ------------------------------------------------------------------------ | | `CARACAL_WORKLOAD_ID` | Workload ID from the console's Launcher page; required by `caracal run`. | | `CARACAL_WORKLOAD_SECRET` | Inline local-development workload secret. | | `CARACAL_WORKLOAD_SECRET_FILE` | Cloud/custom mounted workload-secret file path. | | `CARACAL_STS_URL` | Cloud/custom STS URL override. | | `CARACAL_CONFIG_HOME` | Optional OS config-root override for the default workload-secret path. | Credential bindings, zone, scopes, and failure behavior come from the workload's launch bindings, authored in the web console. Local dev and stable launches can omit both secret variables and read the owner-only file at `/runtime//secret`. ## SDK Profile Fields | Field | Meaning | | ------------------------ | ----------------------------------------------------- | | `sts_url` | Cloud/custom STS URL for token exchange. | | `coordinator_url` | Cloud/custom SDK/Console Coordinator URL override. | | `gateway_url` | Cloud/custom Gateway URL override for SDK transports. | | `zone_id` | Zone identifier. | | `application_id` | Application identifier. | | `app_client_secret_file` | Cloud/custom secret-file path override. | | `app_client_secret` | Inline local-development secret. | | `default_ttl_seconds` | Default TTL for block-style Session calls. | | `credentials[]` | Resource audiences with optional upstream prefixes. | | `optional_credentials[]` | Additional resource audiences and upstream prefixes. | SDK credential entries use `resource` and optional `upstream_prefix`. SDKs do not read launcher fields or search the OS config directory. Set `CARACAL_CONFIG` to an explicit profile path. Environment loaders also support `CARACAL_BOOTSTRAP_TOKEN`, `CARACAL_RESOURCES_FILE`, `CARACAL_RESOURCES`, and `CARACAL_DEFAULT_TTL_SECONDS`. ## Core Service Environment Keys | Key | Services | | ---------------------------------------------------------------------- | ----------------------------------------------------------- | | `CARACAL_MODE` | All services. | | `DATABASE_URL` / `DATABASE_URL_FILE` | API, STS, Gateway, Audit, Coordinator. | | `REDIS_URL` / `REDIS_URL_FILE` | API, STS, Gateway, Audit, Coordinator. | | `STREAMS_HMAC_KEY` / `STREAMS_HMAC_KEY_FILE` | Stream producers and consumers. | | `AUDIT_HMAC_KEY` / `AUDIT_HMAC_KEY_FILE` | Audit producers and Audit service. | | `IDEMPOTENCY_HMAC_KEY` / `IDEMPOTENCY_HMAC_KEY_FILE` | Coordinator idempotency receipt key digest. | | `IDEMPOTENCY_HMAC_KEY_PREVIOUS` / `IDEMPOTENCY_HMAC_KEY_PREVIOUS_FILE` | Coordinator during one receipt-retention rotation window. | | `GATEWAY_STS_HMAC_KEY` / `GATEWAY_STS_HMAC_KEY_FILE` | API, STS, Gateway. | | `SECRET_STORE_KEK` / `SECRET_STORE_KEK_FILE` | API and STS. | | `SECRET_STORE_KEK_PREVIOUS` / `SECRET_STORE_KEK_PREVIOUS_FILE` | API and STS during a master-key rotation window. | | `CARACAL_SECRET_BACKEND` | API and STS; selects the secret backend, default `builtin`. | | `CARACAL_ADMIN_TOKEN` / `CARACAL_ADMIN_TOKEN_FILE` | API and management clients. | | `CARACAL_COORDINATOR_TOKEN` / `CARACAL_COORDINATOR_TOKEN_FILE` | Coordinator and Console Session/Delegation views. | | `METRICS_BEARER` / `METRICS_BEARER_FILE` | Metrics authentication in published modes. | ## Per-Service Tuning Keys Variables that appear in individual runbooks are cataloged here so every documented knob has one home. Defaults are in [Defaults and Limits](/v1.0/reference/defaults-and-limits/). | Key | Service | Controls | | --- | --- | --- | | `OPA_POLL_SECONDS` | STS | Policy-bundle database poll interval (default 60, max 300). | | `MAX_GRANT_TTL_SECONDS` | STS | Ceiling for requested mandate TTLs. | | `STS_MINT_RATE_LIMIT_PER_MIN` | API, STS | Deployment ceiling for mandate mints per minute for each zone, resource, and acting application (default 1000). The web console's Preferences page sets the working limit below this ceiling. | | `STS_SECRET_VERIFY_CONCURRENCY` | STS | Concurrent Argon2id credential verifications (default 2). Each in-flight verification allocates 64 MB; verified credentials are cached, so this bounds cold-start bursts only. | | `CARACAL_PRIVATE_EGRESS_HOSTS` | API, STS | Exact private hostnames granted to OAuth token endpoints, connectivity checks, and notification sink deliveries. | | `UPSTREAM_HOST_ALLOWLIST` | Gateway | Pins Gateway upstream destinations to an explicit host list. | | `MAX_REQUEST_BYTES` | Gateway | Proxied request size cap (default 10 MiB). | | `JTI_FAIL_OPEN` | Gateway | Replay-tracker failure posture; forbidden in published modes. | | `AUDIT_ADMIN_TOKEN` | Audit | Enables the direct operator search and DLQ routes; they return `404` when unset. | | `AUDIT_RETENTION_DAYS` | Audit | Evidence retention window (default 365). | | `AUDIT_EXPORT_S3_*`, `AUDIT_EXPORT_TMP_DIR` | Audit | Optional S3-compatible Parquet export endpoint, credentials, and scratch space. | | `MAX_AGENTS_PER_ZONE`, `MAX_AGENTS_PER_APP` | Coordinator | Concurrent Session ceilings (defaults 50 and 200). | | `IDEMPOTENCY_RETENTION_SECONDS`, `GENERATED_IDEMPOTENCY_RETENTION_SECONDS`, `IDEMPOTENCY_MAX_RECEIPTS_PER_SCOPE` | Coordinator | Idempotency receipt retention windows and per-scope cap. | | `COORDINATOR_BODY_LIMIT_BYTES` | Coordinator | Request body cap (default 256 KiB). | | `CARACAL_ALLOW_INSECURE_CONFIG_URLS` | SDK clients | Permits plaintext control-plane URLs outside loopback, with a startup warning. | | `CARACAL_REQUIRE_PROVENANCE` | Install scripts | Makes a missing provenance check fail the install instead of skipping. | ## Deployment Values Helm values live under `infra/helm/caracal/values.yaml`. Compose environment and secrets are defined by `infra/docker/docker-compose.yml` and `infra/docker/runtime-compose.yml`. This page is the canonical key inventory. [Configure Service Environment](/v1.0/operations/env-vars/) is the workflow for setting them, with precedence, secret-class, and published-mode requirements; the shipped Compose files and Helm values for a release map each key to its deployment surface. ## Next Step Use [Configuration Order](/v1.0/reference/config-precedence/) to understand which file, environment variable, or deployment value wins. ## Related Pages * [Configure Workloads](/v1.0/runtime-console/config-file/) * [Configure Service Environment](/v1.0/operations/env-vars/) * [Choose a Cloud Profile](/v1.0/operations/cloud-native-profiles/) --- # Configuration Order # URL: https://docs.caracal.run/v1.0/reference/config-precedence/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/config-precedence.md # Type: config # Concepts: # Requires: --- ## caracal run Workload Identity `caracal run` reads no profile file. It resolves only the workload identity: 1. `CARACAL_WORKLOAD_ID` names the workload created on the console's Launcher page. 2. The workload secret comes from `CARACAL_WORKLOAD_SECRET`, or `CARACAL_WORKLOAD_SECRET_FILE`, or (eligible local modes only) the owner-only file at `/runtime//secret`. Setting both variables is an error. 3. `CARACAL_STS_URL` overrides the STS endpoint; local dev and stable resolve it automatically. The credential bindings, zone, scopes, and failure behavior come from the workload's launch bindings, authored in the web console and served by STS at launch. ## SDK Profile Resolution SDK loaders use this order: 1. `CARACAL_CONFIG`, when set. A missing file at that path is an error, not a fallthrough. 2. Environment runtime config. SDKs do not read `./caracal.toml` from the current working directory. Environment config uses `CARACAL_ZONE_ID` and `CARACAL_APPLICATION_ID` with one explicit credential source. SDKs read secret and manifest files only when named by `CARACAL_APP_CLIENT_SECRET_FILE` or an explicit profile field. Setting `CARACAL_ENV=production` requires explicit service URLs. Cloud and custom deployments can provide `CARACAL_RESOURCES_FILE` values when mounted secret or config paths differ from the local convention. A deployment without a client secret can instead supply a pre-minted session mandate through `CARACAL_BOOTSTRAP_TOKEN`; the SDK rejects a bootstrap token that is already expired at startup. Resource bindings resolve from profile credentials, then `CARACAL_RESOURCES_FILE`, then `CARACAL_RESOURCES`. Later bindings with the same resource ID override earlier bindings, so a short environment override can replace one entry from a mounted JSON file without duplicating the file. Conflicting credential modes fail closed. A client-secret configuration and `CARACAL_BOOTSTRAP_TOKEN` are alternatives, not fallback layers. ## File-Secret Resolution Services support `*_FILE` variants for configured secret keys. File values are resolved before validation so deployment templates can mount secrets instead of placing sensitive material directly in environment variables. Previous-key variables used during KEK or idempotency-key rotation are separate overlap inputs, not precedence fallbacks. ## Deployment Values | Deployment path | Precedence model | | --------------- | --------------------------------------------------------------------------------------------------------------- | | Docker Compose | Shell environment overrides defaults in compose files; secrets are mounted from files. | | Helm | CLI `--set` and later values files override earlier chart values; runtime Secret keys feed mounted service env. | ## Troubleshooting | Symptom | Check | | -------------------------------------- | ----------------------------------------------------------------------------------------- | | Expected SDK profile is ignored | Set `CARACAL_CONFIG` to the exact profile path; SDKs do not search default directories. | | `caracal run` cannot find bindings | Bindings live on the **Launcher** page in the web console, not in a local file. | | Service fails with missing secret | Confirm the `*_FILE` variable name is supported by that service and the file is readable. | | Helm values render unexpected defaults | Check values file order and CLI overrides. | ## Next Step Use [Defaults and Limits](/v1.0/reference/defaults-and-limits/) to confirm ports, TTLs, request limits, and stream defaults. --- # Defaults and Limits # URL: https://docs.caracal.run/v1.0/reference/defaults-and-limits/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/defaults-and-limits.md # Type: reference # Concepts: # Requires: --- ## Ports | Component | Port | | ----------- | ------ | | API | `3000` | | Web console | `3001` | | STS | `8080` | | Gateway | `8081` | | Audit | `9090` | | Coordinator | `4000` | | Postgres | `5432` | | Redis | `6379` | ## Token and Authority Lifetimes | Limit | Default | | -------------------------------------------- | -------------------------------------------------------------------- | | STS resource mandate cap | 15 minutes | | STS session mandate cap | 60 minutes | | STS `MAX_GRANT_TTL_SECONDS` | `3600` | | DCR application lifetime default and maximum | 3600 seconds | | `caracal run` injected credential TTL | 900 seconds | | Runtime approval wait | until the hold expires; 5-minute fallback when expiry is unavailable | | Approval TTL | 1800 seconds default, clamped between 60 seconds and 7 days | | Approval decision reason | 500 characters maximum | | Gateway expiring-token preflight window | 35 seconds | ## Service Limits | Limit | Default | | --------------------------------- | -------------------- | | API body limit | `1_048_576` bytes | | API request timeout | `30_000` ms | | Coordinator body limit | 256 KiB | | Coordinator request timeout | 30 seconds | | STS request body limit | 64 KiB | | Gateway max request bytes | 10 MiB | | Gateway STS timeout | 5 seconds | | Gateway upstream timeout | 30 seconds | | Gateway non-stream write timeout | 60 seconds | | Gateway stream idle timeout | 60 seconds per chunk | | Gateway server idle timeout | 120 seconds | | Gateway STS circuit failure limit | 3 failures | | Gateway STS circuit open window | 10 seconds | | STS `OPA_POLL_SECONDS` | 60 seconds, max 300 | | Control body limit | 64 KiB | | Control rate capacity | 60 per window | | Control rate window | 60 seconds | | Control replay TTL | 3600 seconds | ## SDK Timeouts and Retries The SDKs share one timeout philosophy: **control-plane calls are bounded, data-plane calls are bounded by you.** Coordinator and STS operations carry defaults because a hung control plane must not hang the worker. Provider traffic through `transport()` carries no TypeScript/Go default because only the caller knows whether a request is a short lookup or a long stream; pass the language-specific timeout or cancellation primitive. | Operation | Default | Notes | | ------------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Coordinator call | 10 seconds | Per request, all SDKs. | | Session start | up to 2 retries | 5xx/network only. The SDK generates one operation id before the first attempt; Coordinator durably replays the same creation response and rejects changed inputs. This protects Session creation, not callback execution or downstream effects. | | Delegation create | 1 retry | Same durable creation-replay contract. | | Coordinator retry backoff | 250 ms escalating, jittered | A server `Retry-After` wins, capped at 10 seconds. | | STS token exchange | 30-second budget, one attempt | Issuance is never retried automatically because a lost response can hide a successfully minted token. Callers reconcile or explicitly retry according to their operation contract. | | TypeScript Control invoke | 30-second total budget | Covers one token mint and one invoke. Neither request is auto-retried; an invoke timeout is outcome-ambiguous. | | Approval wait | 300-second default | Long-polls in chunks; `pending` on timeout means waiting again is safe. | | Heartbeat renewal | 10-second bound per tick | Failures retry on the next tick; a session reported gone stops the timer and fires `onLeaseLost`. | | `transport()` / `fetch()` | none (TS/Go), httpx default (Python) | Bound per call: `timeoutMs`, `AbortSignal`, `timeout=`, or the injected HTTP client. | ## Session and Delegation Limits | Limit | Default | | ----------------------------------------------------------- | --------------- | | Concurrent Sessions per zone | 50 (`MAX_AGENTS_PER_ZONE`) | | Concurrent Sessions per application | 200 (`MAX_AGENTS_PER_APP`) | | Child Sessions per parent Session | 10 | | Delegation depth | 10 | | Session labels per Session | 32 | | Session label length | 64 characters | | STS request rate per zone, resource, and acting application | 1000 per minute (`STS_MINT_RATE_LIMIT_PER_MIN`) | The per-zone ceiling binds first: with defaults, no application can hold more than 50 concurrent Sessions because its zone caps out there. The higher per-application ceiling matters once `MAX_AGENTS_PER_ZONE` is raised. `STS_MINT_RATE_LIMIT_PER_MIN` is the deployment ceiling for the mint rate. Operators can set a lower working limit from the web console under Settings → Preferences → Mint rate limit; the STS applies a change within 30 seconds, and the working limit can never exceed the ceiling. `max_hops` defaults to `1` on a constrained Delegation when omitted. The server validates every child bound against its parent's remaining hops. ## Storage and Stream Defaults | Default | Value | | ------------------------------------------------------ | ---------- | | Audit retention | 365 days | | Audit max deliveries before DLQ | 8 | | Audit claim idle | 30 seconds | | Audit tamper rolling window | 4 hours | | Redis audit stream intended max length | 1,000,000 | | Redis audit DLQ intended max length | 100,000 | | Redis policy/revocation/key stream intended max length | 10,000 | ## Helm Defaults | Service | Replicas | Max HPA replicas | | ----------- | -------- | ---------------- | | API | 2 | 8 | | STS | 2 | 8 | | Gateway | 2 | 16 | | Audit | 2 | 8 | | Coordinator | 2 | 8 | | Control | disabled | 2 when enabled | ## Next Step Use [CLI Exit Codes](/v1.0/reference/runtime-exit-codes/) when automating top-level `caracal` runtime commands. --- # CLI Exit Codes # URL: https://docs.caracal.run/v1.0/reference/runtime-exit-codes/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/runtime-exit-codes.md # Type: reference # Concepts: # Requires: --- Top-level `caracal` commands use `0` for success and non-zero for failure. Commands that delegate to Compose or a child process can propagate that program's non-zero status. ## Exit Behavior | Command | Success | Failure | | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | `caracal up` | `0` after stack start succeeds. | Non-zero when Compose/build/start fails. | | `caracal down` | `0` after stack stop succeeds. | Non-zero when Compose stop fails. | | `caracal status` | `0` when health probes pass. | `1` when service health fails. | | `caracal status --ready` | `0` when readiness probes pass. | `1` when dependency readiness fails. | | `caracal upgrade` | `0` after images stage, migrations apply, services roll, and readiness passes. | Non-zero when image staging, migration, the roll, or the readiness gate fails. | | `caracal purge` | `0` after selected state is removed. | Non-zero when cleanup fails. | | `caracal allowlist [email]` | `0` after the entry change or listing succeeds, including no-op repeats. | `1` on an unknown subcommand, malformed entry, missing entry, locked entry on `add`, or an unreadable allowlist file. | | `caracal run -- ` | Child process exit code after credentials are acquired; `128 + N` when the child dies from signal N. | `1` before start when config, credential exchange, or Approval fails; `127` when the command cannot start. | | `caracal web` | `0` after the web console exits cleanly. | `127` when a required executable is absent; otherwise a propagated build, preflight, auth BFF, or child failure. | Help and version output exit `0`. Unknown commands and invalid usage exit non-zero. A child can return the same numeric status as a pre-start launcher failure, so automation that must distinguish them should preserve launcher logs or wrapper metadata. ## Structured Output `caracal status --json` emits machine-readable status. Web-console-owned diagnostics are available through the web console **Diagnostics** section and management command catalog, not as top-level runtime CLI workflows. ## Next Step Use [Compatibility](/v1.0/reference/compatibility/) before changing supported runtime, package manager, deployment, or docs build targets. ## Related Pages * [Choose the Right Surface](/v1.0/runtime-console/cli-and-console/) * [Start and Check the Stack](/v1.0/runtime-console/stack/) * [Run Workloads](/v1.0/runtime-console/runtime/) --- # Compatibility # URL: https://docs.caracal.run/v1.0/reference/compatibility/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/compatibility.md # Type: reference # Concepts: # Requires: --- ## Language Runtimes | Area | Current target | | ------------------------ | ----------------------------------------------- | | TypeScript/Node packages | Node `>=22` where package engines are declared. | | Python packages | Python `>=3.12`. | | Go modules | Go `1.26` where modules declare a version. | | Root package manager | `pnpm@11.1.1`. | ## Deployment Targets | Target | Source | | ------------------- | -------------------------------------------------------------- | | Local development | `caracal up` and `infra/docker/docker-compose.yml`. | | Self-hosted Compose | `infra/docker/runtime-compose.yml` with versioned GHCR images. | | Kubernetes | `infra/helm/caracal`. | | Docs site | Astro `^7.0.6` and Starlight `0.41.3`. | ## Release and Platform Targets * Release binaries target Linux x64/arm64, macOS x64/arm64, and Windows x64. * Current Kubernetes operations guidance targets Kubernetes 1.30 or newer. * Published modes are `rc` and `stable`. They use the same fail-closed security posture; `rc` denotes pre-release maturity, not relaxed security. ## Version Pinning Caracal packages release in lockstep: every `@caracalai/*` npm package, `caracalai*` Python distribution, and Go module tag carries the same version, and cross-package wire contracts are only validated within one release. Pin every Caracal package in a service to one exact version and upgrade them together - a mixed set (for example an older `@caracalai/oauth` under a newer `@caracalai/sdk`) is unsupported and can fail in ways that look like platform errors. Lockfiles make this the default; when bumping, change every Caracal entry in the same commit. ## v0.2 Stability Caracal is on a pre-1.0 SemVer line. Patch releases preserve documented public behavior while repairing defects. A minor release may change public APIs or operational contracts, so read its release notes and upgrade all Caracal artifacts together. Compatibility promises cover documented package exports, public product APIs, and raw protocol fields identified as wire contracts. Internal storage names, internal endpoints, unpublished exports, and implementation details are not public compatibility guarantees. When a public surface is deprecated, its notice names the replacement and intended removal minor; pre-1.0 consumers must not assume a multi-major deprecation window. This open-source compatibility policy does not imply a support SLA. ## Next Step Use [Release Map](/v1.0/reference/release-package-runtime-map/) to map product versions to packages, images, and release surfaces. ## Related Pages * [Release Map](/v1.0/reference/release-package-runtime-map/) * [Deploy with Helm](/v1.0/operations/kubernetes-helm/) * [Set Up Locally](/v1.0/contributing/setup/) --- # Release Map # URL: https://docs.caracal.run/v1.0/reference/release-package-runtime-map/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/release-package-runtime-map.md # Type: reference # Concepts: # Requires: --- ## Repository Release Surfaces `release.config.json` `product.version` is authoritative. The current source state is `1.0.0-rc.3`; successive release candidates increment `-rc.N`. | Surface | Current source | | ---------------------- | ------------------------------------------------------------------- | | Root package manager | `package.json` declares `pnpm@11.1.1`. | | Release scripts | `scripts/release.sh`, `scripts/release.mjs`, `release.config.json`. | | Docker runtime compose | `infra/docker/runtime-compose.yml`. | | Helm chart | `infra/helm/caracal/Chart.yaml` and values files. | | Docs build | `docs/package.json`. | ## Service Images Caracal shares built images across roles, but each role remains a separate process or pod. | Image name | Roles | Runtime | | ------------------ | -------------------------- | -------- | | `caracal-node` | API, Coordinator | Node | | `caracal-go` | STS, Gateway, Audit | Go | | `caracal-web` | Web console, auth BFF | Node | | `caracal-postgres` | Postgres, migrations | - | | `caracal-redis` | Redis, stream provisioning | - | | `caracal-runtime` | Runtime distribution | Node/Bun | Role selection: * `caracal-node` → `command: ["/app/api/dist/main.js"]` or `["/app/coordinator/dist/main.js"]`. * `caracal-go` → `command: ["/usr/local/bin/sts"]`, `["/usr/local/bin/gateway"]`, or `["/usr/local/bin/audit"]`. Runtime Compose references `${CARACAL_REGISTRY:-ghcr.io/garudex-labs/}:v${CARACAL_VERSION}`. ## Published Package Names | Area | TypeScript | Python | Go | | ------------- | ----------------------- | ---------------------- | -------------------------------------------------------- | | SDK | `@caracalai/sdk` | `caracalai-sdk` | `github.com/garudex-labs/caracal/packages/sdk/go` | | Admin | `@caracalai/admin` | `caracalai-admin` | `github.com/garudex-labs/caracal/packages/admin/go` | | Core | `@caracalai/core` | `caracalai-core` | `github.com/garudex-labs/caracal/packages/core/go` | | Identity | `@caracalai/identity` | `caracalai-identity` | `github.com/garudex-labs/caracal/packages/identity/go` | | OAuth | `@caracalai/oauth` | `caracalai-oauth` | `github.com/garudex-labs/caracal/packages/oauth/go` | | Revocation | `@caracalai/revocation` | `caracalai-revocation` | `github.com/garudex-labs/caracal/packages/revocation/go` | | Verify engine | `@caracalai/verify` | `caracalai-verify` | `github.com/garudex-labs/caracal/packages/verify/go` | ## Versioning Notes * One product release stamps every package, image, binary, chart, and generated release record with the same SemVer. * Every nested Go module uses `/vX.Y.Z[-rc.N]` at the same commit as the root product tag. * Pin exact image tags in production values. * Run migration and readiness validation after image or chart changes. * Do not use product-management runtime CLI aliases as release surfaces; broad automation belongs to the Admin API and zone-scoped dispatch belongs to the Control API. ## Documentation Mapping * Before stable `v0.2.0`, unversioned source is served at `/v0.2/`; no `v0.2` snapshot directory exists. * Stable `v0.2.0` creates the writable `v0.2` snapshot. * `v0.2.x` patches update that same current minor after release; they never create patch-specific documentation trees. * A later stable minor archives unversioned source as the new current snapshot, exposes ongoing work at `/next/`, and locks the superseded minor with its SHA-256 digest. ## Next Step Use [Wire Contracts](/v1.0/reference/interoperability-contracts/) when validating SDK, adapter, exporter, or connector compatibility. --- # Wire Contracts # URL: https://docs.caracal.run/v1.0/reference/interoperability-contracts/ # Markdown: https://docs.caracal.run/markdown/v1.0/reference/interoperability-contracts.md # Type: reference # Concepts: # Requires: --- Wire contracts are raw protocol behavior. Product and SDK facades deliberately map some names to the public terms in [Glossary](/v1.0/reference/glossary/). Authored schemas live under `docs/public/schemas/`; public copies are available at `/schemas/`. Cross-language fixtures live under `tests/shared/fixtures/interoperability/`. ## Schema Files | Contract | Schema | | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | Agent connector manifest | [`caracal-agent-connector-manifest-2026-05-21.schema.json`](/schemas/caracal-agent-connector-manifest-2026-05-21.schema.json) | | Audit event | [`caracal-audit-event-2026-05-21.schema.json`](/schemas/caracal-audit-event-2026-05-21.schema.json) | | Audit exporter manifest | [`caracal-audit-exporter-manifest-2026-05-21.schema.json`](/schemas/caracal-audit-exporter-manifest-2026-05-21.schema.json) | | Gateway upstream manifest | [`caracal-gateway-upstream-manifest-2026-05-21.schema.json`](/schemas/caracal-gateway-upstream-manifest-2026-05-21.schema.json) | | JWT claims | [`caracal-jwt-claims-2026-05-21.schema.json`](/schemas/caracal-jwt-claims-2026-05-21.schema.json) | | Policy input | [`caracal-policy-input-2026-05-20.schema.json`](/schemas/caracal-policy-input-2026-05-20.schema.json) | | Policy pack manifest | [`caracal-policy-pack-manifest-2026-05-21.schema.json`](/schemas/caracal-policy-pack-manifest-2026-05-21.schema.json) | | Policy result | [`caracal-policy-result-2026-05-20.schema.json`](/schemas/caracal-policy-result-2026-05-20.schema.json) | | Resource verifier manifest | [`caracal-resource-verifier-manifest-2026-05-21.schema.json`](/schemas/caracal-resource-verifier-manifest-2026-05-21.schema.json) | | Revocation event | [`caracal-revocation-event-2026-05-21.schema.json`](/schemas/caracal-revocation-event-2026-05-21.schema.json) | | Token response | [`caracal-token-response-2026-05-21.schema.json`](/schemas/caracal-token-response-2026-05-21.schema.json) | | W3C baggage | [`caracal-w3c-baggage-2026-05-21.schema.json`](/schemas/caracal-w3c-baggage-2026-05-21.schema.json) | Dates in schema filenames identify that wire contract; they are not product release versions. Preserve the full filename when pinning validation. ## Fixture Files Fixtures include valid examples for audit events, audit exporter manifests, Gateway upstream manifests, JWT claims, policy input/results, policy pack manifests, resource verifier manifests, revocation events, token responses, W3C baggage, trace context headers, and stream signature canonicalization vectors. ## Product-to-Wire Mapping | Product concept | Common SDK field | Raw protocol field | | ------------------------ | -------------------------------------------- | -------------------------------------------------- | | Authority record ID | `authorityRecordId` and language equivalents | JWT `sid`; STS form `session_id` | | Root authority record ID | `rootAuthorityRecordId` | JWT `root_sid` | | Session ID | `sessionId` | `agent_session_id` and Coordinator `/agents` paths | | Delegation ID | `delegationId` | `delegation_edge_id` and internal edge records | | Approval ID | `approvalId` | `approval_id` and `/approvals/{id}` | Approval audit events are the one deliberate exception: the audit taxonomy is a stable contract for SIEM pipelines and historical queries, so event types keep the `step_up_` prefix (`step_up_issued`, `step_up_decided`, `step_up_consumed`) and audit metadata keeps the `challenge_id` key. Do not rename raw fields in an interoperable implementation. Map them at the SDK or product boundary. ## Usage * Use schemas when building adapters, exporters, or connectors. * Use fixtures when adding SDK or interoperability tests. * Preserve schema filenames when documenting versioned contracts. * Add equivalent TypeScript, Python, and Go tests when a shared contract changes. * Treat a valid fixture as one accepted representation, not a substitute for endpoint validation and negative tests. ## Next Step Use [API Reference](/v1.0/api/) for transport semantics, then validate the implementation against the corresponding fixture. ## Related Pages * [Use Event Topics](/v1.0/api/event-topics/) * [Verify Package](/v1.0/sdks/verify/) * [Validate Changes](/v1.0/contributing/testing/) --- # Contribute to Caracal # URL: https://docs.caracal.run/v1.0/contributing/ # Markdown: https://docs.caracal.run/markdown/v1.0/contributing.md # Type: landing # Concepts: # Requires: --- Caracal welcomes bug reports, documentation fixes, tests, SDK improvements, and platform features. This section is the contributor journey from a fresh clone to a merged pull request; the canonical policy lives in [CONTRIBUTING.md](https://github.com/Garudex-Labs/caracal/blob/main/CONTRIBUTING.md). ## Repository Map The workspace is multi-language: TypeScript applications, Go services, multi-language SDK packages, Docker/Helm infrastructure, and Astro documentation. | Directory | What lives there | | ----------- | ------------------------------------------------------------------------------------------------------------------ | | `apps/` | TypeScript applications: the Admin API, Coordinator, runtime CLI, web console, and console auth backend. | | `services/` | Go data-plane services: STS, Gateway, and Audit. | | `packages/` | SDKs and shared libraries in TypeScript, Python, and Go: core, engine, SDK, identity, OAuth, verify, and adapters. | | `infra/` | Docker, Helm, Postgres, Redis, and OpenTofu deployment assets. | | `docs/` | This documentation site. | | `tests/` | Cross-language test suites and shared fixtures. | | `scripts/` | Repository automation for setup, style, tests, and releases. | Each service, app, and package directory self-documents its rules in an `instructions.md` file. ## Contributor Path Follow the path in order; each page ends where the next begins. | Need | Page | | ----------------------------------------------- | ------------------------------------------------ | | Prepare your machine | [Set Up Locally](/v1.0/contributing/setup/) | | Learn project boundaries and naming conventions | [Follow Project Standards](/v1.0/contributing/style/) | | Work on an issue or pull request | [Make a Change](/v1.0/contributing/workflow/) | | Run the right checks | [Validate Changes](/v1.0/contributing/testing/) | ## Maintainer Path | Need | Page | | -------------------------------------------------- | -------------------------------------------------- | | Understand review, ownership, and security process | [Understand Governance](/v1.0/contributing/governance/) | | Prepare, publish, or recover a release | [Release Caracal](/v1.0/contributing/release/) | ## Before You Start * File bugs, documentation gaps, and feature requests through the [issue templates](https://github.com/Garudex-Labs/caracal/issues/new/choose). A small focused fix can go straight to a pull request; medium and larger changes start with an issue or proposal before code, as sized in [Contribution Scale](/v1.0/contributing/governance/#contribution-scale). * Report suspected vulnerabilities through [Report a Vulnerability](/v1.0/security/disclosure/), never in public issues. * Interactions follow the repository [Code of Conduct](https://github.com/Garudex-Labs/caracal/blob/main/.github/CODE_OF_CONDUCT.md). * Toolchain versions are pinned in [Set Up Locally](/v1.0/contributing/setup/#prerequisites); no prior knowledge of the repository is assumed beyond those tools. ## Next Step Start with [Set Up Locally](/v1.0/contributing/setup/) before making source changes. --- # Set Up Locally # URL: https://docs.caracal.run/v1.0/contributing/setup/ # Markdown: https://docs.caracal.run/markdown/v1.0/contributing/setup.md # Type: workflow # Concepts: # Requires: --- ## Prerequisites | Tool | Version | | ------------------- | ---------- | | Node.js | 24+ | | pnpm | 11.1.1 | | Docker + Compose v2 | Docker 25+ | | Go | 1.26+ | | Python | 3.14+ | | Bun | 1.3.14 | The contributor toolchain is intentionally newer than the supported user runtime floors (Node 22+, Python 3.12+ in [Compatibility](/v1.0/reference/compatibility/)); packages are built with current tools while still targeting the published floors. ## Install Dependencies ```bash git clone https://github.com/Garudex-Labs/caracal.git cd caracal pnpm run setup ``` `pnpm run setup` installs the Node workspace, downloads Go modules, creates `.venv`, installs pinned Python test/style tools, and installs local Python packages in editable mode. Use plain `pnpm install` only when the Node workspace is the entire scope. Read [CONTRIBUTING.md](https://github.com/Garudex-Labs/caracal/blob/main/CONTRIBUTING.md) in the repository root for the contributor policy, review expectations, and testing requirements. ## Start the Local Stack ```bash pnpm caracal up pnpm caracal status --ready ``` The runtime initializes its managed development secrets during startup, builds local service images, and binds host ports to loopback. Run `pnpm secrets:init` only when a focused infrastructure task explicitly needs the standalone secret initializer. ## Open the Web Console ```bash open http://localhost:3001 ``` The packaged web console starts with the stack. Use `pnpm caracal web` only when developing the web console or auth BFF locally. ## Verify the Checkout ```bash pnpm run style scripts/testCi.sh --smoke ``` If either command fails on an untouched checkout, resolve the toolchain problem before editing source. ## Stop or Reset ```bash pnpm caracal down pnpm caracal purge ``` Use `purge` only when you intentionally want to remove local stack/runtime state. ## Next Step Read [Follow Project Standards](/v1.0/contributing/style/) before editing source files. ## Related Pages * [Runtime and Web Console](/v1.0/runtime-console/) * [Deploy with Docker Compose](/v1.0/operations/docker-compose/) * [Validate Changes](/v1.0/contributing/testing/) --- # Follow Project Standards # URL: https://docs.caracal.run/v1.0/contributing/style/ # Markdown: https://docs.caracal.run/markdown/v1.0/contributing/style.md # Type: reference # Concepts: # Requires: --- Caracal style favors small, explicit boundaries and source-aligned documentation. ## Language Style Guides | Language | Required style | Enforcement | | ------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | TypeScript and JavaScript | Existing repository patterns plus the pinned Prettier workspace dependency. | `pnpm run style` checks changed TS/JS source files with Prettier. | | Go | Effective Go with canonical `gofmt` formatting. | `pnpm run style` checks changed Go source files with `gofmt -l`. | | Python | PEP 8 layout as formatted by the pinned Ruff version. | `pnpm run style` checks changed Python source files with `ruff format --check`. | The pre-commit hook activated by `pnpm install` formats staged files automatically on every commit, and `pnpm run style:fix` formats changed files on demand. The gate always runs the Ruff version pinned in `scripts/pythonStyleRequirements.in`, so local formatting matches CI exactly. Pull requests run the same changed-file style gate automatically for primary-language source files. ## Code Conventions | Convention | Apply it | | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | Keep changes focused | Avoid unrelated refactors in feature or docs PRs. | | Prefer explicit validation | Fail closed on auth, policy, config, stream, and key errors. | | Preserve product boundaries | Do not couple open-source code to enterprise-only code. | | Respect command ownership | Runtime CLI is lifecycle and setup; the web console is the human surface, the Admin API is broad automation, and the Control API is Zone-scoped dispatch. | | Use existing shared layers | Reuse core config, errors, crypto, logging, engine dispatch, and SDK helpers. | | Preserve language parity | SDK capability and wire changes in shared packages must land idiomatically in TypeScript, Python, and Go with equivalent tests. | | Preserve file headers | Source files keep the repository copyright/product header in the language's comment syntax. | ## Documentation Conventions | Page type | Pattern | | ------------ | --------------------------------------------------------------------- | | Landing | Purpose, audience, map, recommended reading path. | | Workflow | Prerequisites, steps, validation, troubleshooting, related links. | | Reference | Exact names, defaults, tables, examples, source-of-truth links. | | Architecture | Diagram, component responsibilities, flow, boundaries, related pages. | Recurring sections use fixed names and casing: `## Prerequisites`, `## Expected Outcome`, `## Common Mistakes`, `## Troubleshooting`, `## Related Pages`, and `## Next Step`. Cross-reference link text matches the target page's title or sidebar label. Web console locations are written as breadcrumbs (**Services → Launcher**). Clause separators are spaced hyphens (`-`), not em dashes; quotes are straight. Example data uses the PiperNet/Hooli sample universe with reserved `.example` hosts and UUID-style zone IDs; scopes in a snippet must match the resource the snippet targets. Rego policy-data blocks are fenced as `rego`. Do not use docs to preserve stale command names, screenshots, package names, or workflows. Update the whole affected page coherently. ## Project Boundaries * Top-level `caracal` commands are limited to runtime lifecycle, upgrade, purge, console admission, `caracal run`, and the optional web development launcher. * Product-management workflows for Zones, Policy, Grants, audit, Sessions, Delegation, and Control belong in the web console, Admin SDK, or Control API docs. * Open-source code must not import, reference, or depend on enterprise-only code. ## Naming Use canonical terms from [Glossary](/v1.0/reference/glossary/). Keep raw names such as `agent_session_id`, `delegation_edge_id`, and `/agents` at explicit protocol boundaries only, and link the [Product-to-Wire Mapping](/v1.0/reference/interoperability-contracts/#product-to-wire-mapping) wherever a raw name must appear. Capitalize Caracal object nouns (Session, Zone, Delegation, Mandate, Approval) when naming the product object; keep fully generic uses lowercase, and never mix both styles in one page. ## Next Step Use [Make a Change](/v1.0/contributing/workflow/) to plan and submit a focused pull request. --- # Make a Change # URL: https://docs.caracal.run/v1.0/contributing/workflow/ # Markdown: https://docs.caracal.run/markdown/v1.0/contributing/workflow.md # Type: workflow # Concepts: # Requires: --- Match the process to the change size before writing code: a small focused fix can go straight to a pull request, a medium change starts with a [GitHub issue](https://github.com/Garudex-Labs/caracal/issues/new/choose), and a large one starts with a proposal, as sized in [Contribution Scale](/v1.0/contributing/governance/#contribution-scale). ## Standard Flow 1. Sync `main`, create a focused branch, and confirm the checkout passes its baseline targeted check. 2. Keep the change focused on one component or user workflow. 3. Read the `instructions.md` file in the directory you are editing. Every service, app, and package directory has one; it lists the required and forbidden patterns for that area. 4. Update docs when behavior, APIs, commands, config, examples, or operations change. Before `v0.2.0`, edit the unversioned source. Afterward, target either the current stable minor or the next unreleased minor as described below. 5. Add a regression test for every bug fix and tests for major new behavior. 6. Run the narrowest relevant check first, then broaden only when a shared or security boundary changed. 7. Review the diff for generated files, secrets, unrelated formatting, and documentation drift. 8. Open a pull request with the repository template: a typed title (`feat`, `fix`, `docs`, ...), the linked issue where one exists, and the evidence below. ## Choose the Area | Area | Common sources | | ----------------- | ----------------------------------------------------------------------------------- | | Runtime CLI | `apps/runtime`, `packages/engine`, runtime tests. | | Web console | `apps/web`, Admin SDK, web console tests. | | API | `apps/api`, migrations, Admin package, API tests. | | Coordinator | `apps/coordinator`, Coordinator tests, SDK tests. | | STS/Gateway/Audit | `services/*`, Go tests, operations docs. | | SDKs/adapters | `packages/*`, language-specific tests, interoperability fixtures. | | Infra | `infra/docker`, `infra/helm`, Postgres/Redis scripts. | | Docs | `docs/src/content/docs`, `docs/versions.json`, site config/plugins, and docs tests. | ## Choose the Documentation Line Documentation versions follow product minor releases, not individual patch releases. | Change target | Source to edit | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Before the first `v0.2.0` release | Unversioned pages under `docs/src/content/docs/`. | | Current stable patch line, such as `v0.2.1` or `v0.2.2` | The unlocked current snapshot under `docs/src/content/docs/v0.2/`. | | Next unreleased minor, such as work for `v0.3.0` | Unversioned pages under `docs/src/content/docs/`, published at `/next/` after versioning begins. | | Older minor | Do not edit it. Once superseded, its snapshot is locked and CI verifies its digest. | Do not create `v0.2.1`, `v0.2.2`, or other patch-specific documentation directories. If a documentation correction applies to both stable and unreleased behavior, make two deliberate edits: one in the unlocked stable snapshot and one in the unversioned next-minor source. Create snapshots only through `scripts/docsVersion.mjs` as part of the stable release flow. Never edit a locked snapshot or replace its digest. ## Pull Request Evidence Include: * what user or operator behavior changed; * which trust, command, language-parity, or product-isolation boundaries were reviewed; * exact targeted and broad checks that ran, with results; * deployment, migration, compatibility, or rollback impact; * unresolved ambiguity that a reviewer must decide. ## Command Boundary Do not add top-level runtime CLI commands for zones, policies, grants, audit, Sessions, Delegation, or Control. Human workflows belong to the web console; broad automation uses the Admin API, and zone-scoped dispatch uses the Control API. ## Security Reports Do not discuss suspected vulnerabilities in public issues. Use GitHub private advisories or the email path in [Report a Vulnerability](/v1.0/security/disclosure/). ## Next Step Use [Validate Changes](/v1.0/contributing/testing/) to choose the narrowest useful test command before opening a pull request. --- # Validate Changes # URL: https://docs.caracal.run/v1.0/contributing/testing/ # Markdown: https://docs.caracal.run/markdown/v1.0/contributing/testing.md # Type: reference # Concepts: # Requires: --- Run the smallest relevant suite first, then broaden when a change touches shared code, security boundaries, generated artifacts, or deployment behavior. ## Testing Policy This policy is mandatory and is enforced during review: * Major new functionality MUST add automated tests covering that functionality, in the same change that introduces it. * Every bug fix MUST add a regression test that fails without the fix and passes with it. * Reviewers MUST confirm the required tests exist and run in CI before approving; pull requests that omit them are not merged. ## Root Commands | Command | Purpose | | --------------------------- | ----------------------------------------------------------- | | `pnpm run build:typescript` | Build TypeScript apps and packages. | | `pnpm run lint` | Run package linters where present. | | `pnpm run typecheck` | Run TypeScript type checks. | | `pnpm run test` | Full TypeScript, Go, and Python test suite. | | `pnpm run test:typescript` | TypeScript app/package tests. | | `pnpm run test:go` | Go service/package and interoperability tests. | | `pnpm run test:python` | Python package tests. | | `pnpm run ci` | Build, lint, typecheck, and test sequence. | | `pnpm docs:version:verify` | Validate documentation version metadata and snapshot locks. | ## Targeted Examples | Area | Command | | -------------------------------- | --------------------------------------------------------------------------- | | Runtime CLI | `pnpm --dir apps/runtime test` | | Web console | `pnpm --dir apps/web test` | | Web backend-for-frontend | `pnpm --dir apps/auth test` | | API | `pnpm --dir apps/api test` | | Coordinator | `pnpm --dir apps/coordinator test` | | STS, Gateway, Audit, Go packages | `pnpm run test:go` (the repository harness owns its multi-module workspace) | | Docs | `pnpm --dir docs build` | | Docs versioning only | `pnpm exec vitest run tests/typescript/unit/docs/versioning.test.ts` | Runnable examples live in the [Caracal examples repository](https://github.com/Garudex-Labs/examples) and carry their own test suites and CI. ## CI Mirror ```bash scripts/testCi.sh scripts/testCi.sh --smoke scripts/testCi.sh --go scripts/testCi.sh --py scripts/testCi.sh --ts scripts/testCi.sh --docs ``` Use broader checks when a change affects auth, crypto, config, release, infra, shared packages, SDK contracts, or interoperability schemas. Record the exact commands and results in the pull request. Do not claim the full suite when only targeted checks ran. ## Next Step After validation, review [Understand Governance](/v1.0/contributing/governance/) for contribution scale, review ownership, and private security process. --- # Understand Governance # URL: https://docs.caracal.run/v1.0/contributing/governance/ # Markdown: https://docs.caracal.run/markdown/v1.0/contributing/governance.md # Type: reference # Concepts: # Requires: --- Caracal is maintained by Garudex Labs. `.github/MAINTAINERS` names maintainers; `.github/CODEOWNERS` routes path ownership and review requests. ## Contribution Scale | Change size | Expected process | | ----------------------------- | -------------------------------------------------------------------------------------------------- | | Small focused fix | Pull request with clear validation. | | Medium bug or feature | GitHub issue with context and expected outcome before implementation. | | Large or cross-cutting change | Proposal with problem statement, alternatives, trade-offs, open questions, and smaller sub-issues. | | Security-sensitive change | Private security process and maintainer coordination. | ## Maintainer Responsibilities * Review changes in owned areas. * Enforce repository standards and product boundaries. * Keep security reports private. * Approve releases and release workflow changes. * Preserve open-source and enterprise product isolation. ## Code Review Requirements Every change is proposed as a pull request and reviewed before merge or release. | Requirement | Expectation | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Independent review | At least one maintainer other than the author approves each pull request; authors do not approve or merge their own changes. | | Area ownership | `.github/CODEOWNERS` owners are requested automatically for their paths. | | What reviewers check | Correctness and edge cases, focused scope, Testing Policy compliance with passing CI, the `pnpm run style` gate, input validation and trust boundaries, secret hygiene, OSS/enterprise isolation, and updated docs. | | Acceptance bar | One approving non-author review, all required CI checks green, resolved comments, and a judgment that the change is worthwhile and free of known disqualifying defects. | | Release approval | Stable releases require `release-approval` from a maintainer other than the release preparer. | Maintainers own the decision to merge, not just the mechanics. They must reject changes whose security, compatibility, product-isolation, or operational risk is unresolved even when tests pass. ## Release Ownership * The release preparer stamps and validates the candidate but cannot supply the independent stable approval. * The approving maintainer verifies version consistency, release manifest evidence, documentation version plan, registry preflight, and rollback posture. * Published tags are immutable. Recovery is a roll-forward release, not tag deletion or artifact replacement. The full contributor-facing policy lives in [`./CONTRIBUTING.md`](https://github.com/Garudex-Labs/caracal/blob/main/CONTRIBUTING.md#code-review). ## Community Standards The project follows the repository [Code of Conduct](https://github.com/Garudex-Labs/caracal/blob/main/.github/CODE_OF_CONDUCT.md). Harassment, private-information disclosure, and disruptive behavior are not acceptable. ## Security Governance Security concerns must be reported through [Report a Vulnerability](/v1.0/security/disclosure/). Public issues are not appropriate for vulnerabilities, credential exposure, unsafe execution, or exploitable operational failures. ## Next Step Maintainers preparing a cut should use [Release Caracal](/v1.0/contributing/release/). ## Related Pages * [Make a Change](/v1.0/contributing/workflow/) * [Review the Threat Model](/v1.0/security/threat-model/) * [Respond to Incidents](/v1.0/operations/incident-response/) --- # Release Caracal # URL: https://docs.caracal.run/v1.0/contributing/release/ # Markdown: https://docs.caracal.run/markdown/v1.0/contributing/release.md # Type: reference # Concepts: # Requires: --- Every Caracal release artifact shares one Semantic Version from `release.config.json` `product.version`. The current source state is `1.0.0-rc.3`; successive release candidates increment `-rc.N`. `scripts/release.sh stamp` propagates the configured version to owned artifact metadata. ## Release Surfaces | Surface | Source | | --------------- | ----------------------------------------------------------------------------------------- | | Product version | `release.config.json` `product.version`, currently `1.0.0-rc.3`. | | Runtime binary | `apps/runtime/dist/caracal-*`. | | Containers | Shared Go/Node role images, Web, Postgres, Redis, and Runtime from `release.config.json`. | | Helm | `infra/helm/caracal`. | | npm packages | Public `@caracalai/*` packages from `release.config.json`. | | PyPI packages | Public `caracalai-*` packages from `release.config.json`. | ## Build Targets Runtime release builds compile Linux x64/arm64, macOS x64/arm64, and Windows x64 binaries through Bun compile scripts. Go-based service container builds strip debug symbols by default (`GO_LDFLAGS` defaults to `-s -w`) and honor native build arguments passed to Docker: `CGO_ENABLED`, `CC`, `CFLAGS`, `CXX`, `CXXFLAGS`, `LDFLAGS`, `GOFLAGS`, `GO_BUILDFLAGS`, and `GO_LDFLAGS`. The Dockerfiles add `-mod=readonly` and `-trimpath`; override `GO_LDFLAGS` for diagnostic builds that need symbol tables. ## Release Flow | Stage | Command pattern | | -------------- | ---------------------------------------------------------------------------- | | RC prepare | Set `product.version` to `X.Y.Z-rc.N`, then `scripts/release.sh rc prepare` | | RC dry run | `scripts/release.sh rc dry-run --local`, then `scripts/release.sh rc dry-run` | | RC publish | `scripts/release.sh rc publish [--watch]` | | Stable dry run | Add `--local` first, then run `scripts/release.sh stable --dry-run` remotely | | Stable publish | `scripts/release.sh stable [--watch]` | | Prepare stable | `scripts/release.sh promote --from vX.Y.Z-rc.N`, review, and commit | Run `pnpm release:plan` and `pnpm release:stamp:check` before publication. Preparation generates a source-neutral release plan and the docs release record. On its first invocation, the publish command atomically creates the root tag and every nested Go module tag, then queues a dry run from that immutable tag. Invoke the same publish command again after the dry run succeeds, or pass `--watch` to track the dry run and dispatch the publication automatically in one command; only then can CI publish and finalize the customer manifest with the full tag commit. Do not author release evidence as an independent source of truth. Publication requires a successful release-workflow dry run for the exact commit. A green branch test run is necessary but does not replace the archive, image, package, and documentation preflight performed by that dry run. The root release workflow owns production publication. npm package workflow dispatches are dry-run only. PyPI production publication is dispatched by the release orchestrator through the protected `publishPypi.yml` Trusted Publisher workflow with an exact release tag and source SHA. Every release publishes all publishable packages at the shared version, and Python distributions publish registry attestations. A retry reuses an existing package, image, chart, or GitHub Release only after its digest and provenance verify against the exact release tag and commit. Any mismatch consumes the version and requires a roll-forward release. PyPI publication is dispatched directly through `publishPypi.yml`, matching its Trusted Publisher identity. If publication stops after immutable artifacts exist, `resumeRelease.yml` verifies the retained release-assets artifact and every npm, PyPI, OCI, and Helm artifact before creating the missing GitHub Release. The resume path never rebuilds or replaces an existing artifact. ## Pipeline Safeguards Release workflows execute from the immutable tag snapshot, so a workflow defect can only be corrected on `main` and recovered through `resumeRelease.yml`. To keep defects out of tags, CI statically validates release-workflow invariants on every change: ```bash node scripts/validateWorkflows.mjs ``` The validator enforces reusable-workflow permission coverage, declared `workflow_call` inputs, SHA-pinned actions, job timeouts, repository guards, explicit job permissions, `GH_REPO` on checkout-free `gh` usage, and the run-name contracts that release tooling matches against. Run titles are single-sourced in `scripts/lib/releaseSpec.mjs`; unit tests reject drift between workflow `run-name` templates and the formats the scripts expect. Every publish step checks published state before acting and verifies digests and provenance afterward, so rerunning a failed job never duplicates or replaces an artifact. A failed run appends a failure report to the workflow summary naming the failed jobs and steps with log links and recovery guidance; a successful publication appends a summary of everything that shipped. ## Documentation Versions Open Source documentation is versioned by product minor release: * `v0.2`, `v0.3`, `v0.4`, and `v1.0` are documentation versions. * Patch releases such as `v0.2.1` and `v0.2.2` update the existing `v0.2` documentation; they never create new documentation versions. * Release candidates never create documentation versions. * The latest stable minor is the default at `https://docs.caracal.run/`. The unversioned source is available at `/next/` after the first stable documentation release. * Superseded minors remain available at `/vX.Y/`, lose their edit link, and are protected by a committed SHA-256 digest. `v0.2.0` is the first versioned documentation release. `docs/versions.json` records `v0.2` as the current snapshot, and its source lives in `docs/src/content/docs/v0.2/`. Patch documentation changes for the active release line go directly into `docs/src/content/docs/v0.2/`. Work intended for the next minor continues in the unversioned source. When `v1.0.0` is published, release automation performs one transaction: 1. Archive the unversioned source as `v1.0` and preserve its sidebar and referenced assets. 2. Make `v1.0` the default stable documentation. 3. Lock `v0.2` and record its content digest. 4. Leave the unversioned source in place as the starting point for `/next/`. The same process repeats for every later minor. `scripts/release.sh stable` and `scripts/release.sh promote` invoke it automatically. Maintainers can inspect the decision without changing files: ```bash node scripts/docsVersion.mjs plan 1.0.0 node scripts/docsVersion.mjs verify ``` Do not edit `docs/versions.json` to bypass the release command. CI compares every previously locked entry and snapshot with the base branch, so replacing both a historical page and its digest is rejected. ## Rollback Rule Do not delete published tags. Roll forward with a new SemVer tag. Pinned `vX.Y.Z` tags are immutable; the floating `vX.Y` series tag moves with the new cut. ## Registry Cleanup After the SemVer Cutover The migration from CalVer to lockstep SemVer leaves registry state that only a maintainer with registry access can retire. Complete these steps manually after the first `v0.2.0` stable release is live: 1. Deprecate dead npm names with a pointer to their successors - never unpublish: ```bash npm deprecate @caracalai/transport-mcp@'*' 'renamed: use the current @caracalai packages' npm deprecate @caracalai/transport-a2a@'*' 'renamed: use the current @caracalai packages' npm deprecate @caracalai/mcp-express@'*' 'renamed: use @caracalai/express' npm deprecate @caracalai/mcp-fastmcp@'*' 'renamed: use @caracalai/fastmcp' npm deprecate @caracalai/tokenstate-postgres@'*' 'renamed: use the current @caracalai packages' ``` 2. Deprecate the pre-0.2.0 CalVer and `0.1.x` versions on live npm names the same way. 3. Yank (PEP 592) the pre-0.2.0 versions on live PyPI projects and all versions of dead-name PyPI projects. Yanking keeps files installable by exact pin while removing them from resolution. 4. Delete the CalVer `ghcr.io` image tags (`v2026.*`, `2026.*`) and CalVer OCI Helm chart versions. CalVer sorts above SemVer, so any surviving `2026.*` tag would outrank `0.2.0` in version resolution. 5. Mark pre-0.2.0 GitHub Releases as pre-release with a note pointing to the current release line. Do this only after `v0.2.0` is published so `releases/latest` never resolves to nothing. 6. Keep all git tags: deleting them breaks reproducibility and Go module caching. 7. Refresh the `lynxCapital` dependency pins in the [examples repository](https://github.com/Garudex-Labs/examples) to the published `0.2.0` packages. ## Related Pages * [Release Map](/v1.0/reference/release-package-runtime-map/) * [Upgrade Caracal](/v1.0/operations/upgrade/) --- # Page not found # URL: https://docs.caracal.run/v1.0/404/ # Markdown: https://docs.caracal.run/markdown/v1.0/404.md # Type: page # Concepts: # Requires: --- import LandingFooter from '../../../components/LandingFooter.astro'

Error · 404

This docs path is not available.

Check the selected documentation version and use site search for the page title. If the link came from code or an older release, open the matching stable minor before assuming the behavior moved.

New evaluator: Get Started. Operator: Operations. Integrator:{' '} Guides. Contributor: Contributing.

--- # Approvals # URL: https://docs.caracal.run/v1.0/concepts/approvals/ # Markdown: https://docs.caracal.run/markdown/v1.0/concepts/approvals.md # Type: page # Concepts: # Requires: --- Read this page after [Policies and Policy Sets](/v1.0/concepts/policy/). An Approval pauses a sensitive authority request until an eligible human decides it. Policy data maps scopes to risk tiers and identifies whether an operator, the application's Federated user, or either may decide. Approval is an optional security primitive. A zone that declares no `approval_tiers` data never creates a hold, and nothing in the platform requires one. One name carries through every surface: the SDKs call the decision's identifier `approvalId` and the wire calls it `approval_id` under `/approvals` paths; only the audit stream keeps the historical `step_up_` event-type prefix and `challenge_id` metadata key - see the [wire-name mapping](/v1.0/reference/interoperability-contracts/#product-to-wire-mapping). ## Approval Flow ```mermaid sequenceDiagram participant App as App or agent participant STS as STS participant Policy as Active policy set participant Approver as Console, Admin API, or Federated user App->>STS: Exchange for gated scope STS->>Policy: Evaluate request Policy-->>STS: allow, gated by matched approval tiers STS-->>App: interaction_required with approval_id + binding App->>STS: GET /approvals/{id} (long-poll) Approver->>STS: approve or reject the hold STS-->>App: state: approved App->>STS: Retry exchange with approval_id STS-->>App: Mandate (approval consumed) ``` ## Two Decision Planes | Plane | Approver | Surface | | -------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | Operator | A control-plane admin holding an `approve`-capable token. | Console Approvals page or `POST /v1/zones/{zone}/approvals/{id}/approve` / `/reject`. | | Federated user | The requesting application's own Federated user. | `POST /approvals/{id}/decision` on the STS, requiring a user-type session mandate and the hold's binding. | The tier's `approver` declaration picks the plane: `operator`, `subject` (the Federated user plane's wire value), or `any`. On the operator plane, approval authority is a distinct admin capability - a `write` token cannot decide a hold. An `any` hold admits either plane, so operators can always decide it. An Approval reserved for the Federated user requires the Application to federate that user through a registered Federated user issuer. Without that federation, the Approval can only expire. This is a decision mechanism, not per-Subject Resource authorization: the original Application, policy, scopes, and Delegation still bound the resulting Mandate. ## Components | Component | Responsibility | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Policy data | Declares `risk` tiers per scope and `approval_tiers` gates; the platform fixes no tier taxonomy. | | STS | Creates the hold, serves its state to long-polling agents, records decisions, verifies the binding, and consumes the approval at mint. | | Console or Admin API | Lists, inspects, and decides operator-plane holds. | | Application | Relays Federated user holds to its own user and posts the decision with that user's session mandate. | | SDK or OAuth client | Surfaces `interaction_required` and waits on the hold (`waitForApproval`); `caracal run` parks and retries automatically. | ## Approval Lifecycle | State | Meaning | | ---------- | ---------------------------------------------------------------------------------------- | | `pending` | The hold is live and awaiting a decision. | | `approved` | An approver granted the hold; the next matching exchange mints. | | `rejected` | An approver refused the hold; terminal. | | `expired` | The approval window closed without a decision, or an approval lapsed before consumption. | | `consumed` | The approval released its one mandate; terminal. | An Approval releases at most one Mandate. Its binding covers the Application, Authority record, Session, Delegation, Resource, scopes, and active policy version. A policy rollout or execution-context change therefore requires a fresh decision. Because the Authority record is minted per client-credentials exchange, a requester that restarts before its hold is decided returns with a new Authority record: persisting the approval id lets it observe the final state through `waitForApproval`, but consumption stays bound to the run that asked, so a restarted requester raises a fresh hold rather than consuming one approved for the prior process. Consumed and rejected are terminal. ## Privacy Modes The tier's `privacy` declaration controls what the decision record retains of a Federated user approver: `identified` stores the identity verbatim, `pseudonymous` a stable zone-scoped pseudonym, and `anonymous` a redaction marker. The approver's Authority record ID is always kept as the forensic and revocation anchor. Operator-plane decisions always record the deciding admin identity. Caracal stores authorization facts, never business context. ## Design Guidance * Gate high-risk scopes, not everything: approval latency is a person, so reserve it for authority worth a pause. * Keep the decision outside policy; policy declares that a decision is needed, never performs it. * Use `subject` tiers when the risk belongs to the application's Federated user and the application federates its users through a registered Federated user issuer; use `operator` tiers when the risk belongs to the zone. * Give automation credentials `write` without `approve`, so no pipeline can silently settle a hold. * Cross-check the binding: the agent prints it beside the approval id, and the web console shows it on the hold. ## Next Step Read [Session Delegation](/v1.0/concepts/delegation/) to understand how approved authority can be narrowed for another Session. ## Related Pages * [Human Approval](/v1.0/guides/human-approval/) * [Policies and Policy Sets](/v1.0/concepts/policy/) * [Audit and Request Traces](/v1.0/concepts/audit-ledger/) --- # Govern Agent Frameworks # URL: https://docs.caracal.run/v1.0/guides/frameworks/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/frameworks.md # Type: page # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' Caracal does not ship framework connectors for LangChain, LangGraph, or CrewAI, because it does not need them. Frameworks orchestrate model and tool calls through the provider clients you construct; Caracal governs those clients through the [transport wiring](/v1.0/guides/provider-recipes/#wire-the-transport-into-provider-clients). Configure the client once, and every chain, graph node, agent, and tool call the framework issues is authenticated, policy-checked, and audited - with no framework-version coupling to maintain. ## When to use this guide Use it after a direct SDK-to-Gateway call works and before handing the governed client to an agent framework. It is not an onboarding path and does not govern arbitrary framework-local tool calls. ## Prerequisites * One working provider recipe and resource binding per network host the framework can call. * A scoped SDK transport tested without the framework. * A Session and Delegation design for parallel agents and subagents. The pattern is always the same: 1. Bind the provider upstream to a Caracal resource. 2. Build the provider client with a scoped transport and `gateway-only` propagation. 3. Hand that client to the framework. 4. Run framework work inside `session()` so each agent execution gets its own Session ID. ## LangChain LangChain model classes accept the same custom HTTP client hooks as the underlying provider SDKs. ```python from caracalai import Caracal from langchain_openai import ChatOpenAI caracal = Caracal() llm = ChatOpenAI( model="gpt-4o", api_key="caracal-gateway", http_client=caracal.sync_transport( scopes=["inference:invoke"], propagation="gateway-only" ), http_async_client=caracal.transport( scopes=["inference:invoke"], propagation="gateway-only" ), ) ``` ```ts import { Caracal } from '@caracalai/sdk' import { ChatOpenAI } from '@langchain/openai' const caracal = new Caracal() const llm = new ChatOpenAI({ model: 'gpt-4o', apiKey: 'caracal-gateway', configuration: { fetch: caracal.transport({ scopes: ['inference:invoke'], propagation: 'gateway-only' }), }, }) ``` The `configuration` object forwards to the OpenAI SDK, so the governed `fetch` carries every LangChain invocation. Anthropic and Gemini model classes take the corresponding hooks from their provider recipes. ## LangGraph LangGraph nodes call whatever model client the graph was built with, so a governed client governs the whole graph. What LangGraph adds is concurrency: parallel branches and subgraphs fan out, and each logical agent should be distinguishable in policy and audit. Start a labeled Session around each run, or around each branch that represents a distinct agent: ```python from langgraph.prebuilt import create_react_agent agent = create_react_agent(llm, tools) async with caracal.session(labels=["research-agent"]): result = await agent.ainvoke({"messages": [("user", question)]}) ``` Every model call and governed tool call inside the run carries that Session's identity. To narrow a subagent below its parent, start its Session with reduced authority; see [Implement Multi-Agent Delegation](/v1.0/guides/delegation/). ## CrewAI CrewAI routes model traffic through LiteLLM, so the LiteLLM session hooks govern an entire crew: ```python import litellm from caracalai import Caracal caracal = Caracal() litellm.client_session = caracal.sync_transport( scopes=["inference:invoke"], propagation="gateway-only" ) litellm.aclient_session = caracal.transport( scopes=["inference:invoke"], propagation="gateway-only" ) ``` Set the sessions before constructing agents. Each provider LiteLLM reaches needs its own resource binding, exactly as in the [LiteLLM recipe](/v1.0/guides/provider-recipes/#litellm). ## Framework tools Tools are where agent traffic actually touches your systems, and they follow the same rule as models: build the tool's HTTP client from the transport. ```python pipernet = caracal.transport(scopes=["pipernet:read"]) async def fetch_report(report_id: str) -> dict: response = await pipernet.get(f"https://api.pipernet.example/reports/{report_id}") return response.json() ``` Build the transport once and share it: identity is resolved per request from the bound context, so concurrent agents can safely use the same client. Passing `scopes` mints the narrowed `use=gateway` mandate required for each call, so a tool holds only the authority it needs. For a complete multi-agent LangChain and LangGraph deployment governed this way, see the [Lynx Capital example](/v1.0/examples/lynx-capital/). ## Related * [Provider Recipes](/v1.0/guides/provider-recipes/) * [Integrate the Python SDK](/v1.0/guides/sdk-python/) * [Integrate the TypeScript SDK](/v1.0/guides/sdk-typescript/) * [Implement Multi-Agent Delegation](/v1.0/guides/delegation/) ## Validate the framework integration Capture every outbound host in a test run. Each governed host must route through Gateway under the expected resource; an unknown host must not receive Caracal headers. Run parallel branches and confirm distinct labeled Sessions in Audit. :::caution[Failure point: framework tools] Passing a governed model client does not govern a tool that constructs its own HTTP client. Inject a Caracal transport into every network client or protect the destination as a resource server. ::: ## Next Step Add [multi-agent delegation](/v1.0/guides/delegation/) only after single-Session transport and audit validation pass. --- # Human Approval # URL: https://docs.caracal.run/v1.0/guides/human-approval/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/human-approval.md # Type: page # Concepts: # Requires: --- Human approval pauses a token exchange until a person decides it. A policy data document classifies scopes into risk tiers and declares which tiers need a human decision; when an agent requests a gated scope, the STS parks the exchange on a durable hold and returns `interaction_required` instead of a mandate. The gate is optional: a zone that declares no approval tiers never sees a hold. :::note[One object, one name] An Approval is called an Approval on every surface: SDKs expose it as `approvalId`, the wire carries `approval_id`, and the STS and Admin API serve it under `/approvals` paths. This page says *Approval* for the object and *hold* for its undecided state. Only the audit stream retains the historical `step_up_` event-type prefix and `challenge_id` metadata key, so SIEM pipelines and saved queries stay stable; the [wire-name mapping](/v1.0/reference/interoperability-contracts/#product-to-wire-mapping) records that exception. ::: ## When to use approval Use it for infrequent high-impact authority where waiting is safer than automatic issuance. Do not use approval as user authentication, a substitute for least privilege, or a repair for an overbroad provider credential. ## Prerequisites * Valid grant data for the underlying scope; approval can gate authority but cannot create it. * An operator approval path, or a registered Federated user issuer plus implemented federation for decisions reserved to the Federated user. * A durable place to persist `approvalId` and operation identity across worker restarts. ## Flow ```mermaid flowchart LR Exchange["Exchange for gated scope"] --> Hold["interaction_required with approval_id + binding"] Hold --> Wait["Agent waits on the hold"] Decide["Approver decides: Console, Admin API, or Federated user"] --> Wait Wait --> Retry["Retry exchange with approval_id"] Retry --> Mandate["Mandate issued, approval consumed"] ``` ## Declare the gate in policy data Approval is declared as data, like every other policy input. `risk` names a tier for each sensitive scope; `approval_tiers` declares which tiers hold a mint for a decision: ```rego # caracal:data-document package caracal.authz import rego.v1 risk := [ {"scope": "pipernet:refund", "tier": "high"}, ] approval_tiers := [ {"tier": "high", "approver": "operator", "ttl_seconds": 1800, "privacy": "identified"}, ] ``` | Field | Meaning | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `tier` | Your tier name; the platform fixes no taxonomy. Required - a declaration without a tier fails the mint closed. | | `approver` | Who may decide: `operator` (approve-capable admin credential), `subject` (the application's own Federated user - see below), or `any`. Defaults to `operator`. | | `ttl_seconds` | How long the hold stays decidable. Federated-user-decidable holds (`subject` or `any`) default to 15 minutes and cap at 24 hours; operator-only holds default to 4 hours and cap at 7 days. The floor is 60 seconds everywhere. | | `privacy` | How much approver identity the decision record retains: `identified`, `pseudonymous`, or `anonymous`. Defaults to `identified`. | When one mint matches several compatible tiers, the shortest window and most protective privacy mode win, and a specific `operator` or `subject` requirement wins over `any`. A request that combines an operator-only tier with a Federated-user-only tier is denied and must be split into separate mints: one decision cannot satisfy two independent approver roles. Invalid approver, privacy, or TTL declarations fail the mint closed. An approval's lifetime is its own: it never inherits or extends a session's lifetime. A Federated user decision is an interactive consent moment, so its window is short by default; an operator decision is an administrative review and gets hours. Tune `ttl_seconds` to the tier's real decision window: it bounds how long an approved-but-unconsumed hold stays spendable, so a shorter TTL shrinks the window in which a leaked approval could be consumed, while a TTL shorter than your approvers' actual response time just converts legitimate requests into expiries. Expiry is cheap by design - an expired hold never resumes anything, the requester simply raises a fresh one, and terminal rows age out of the store automatically. ## Handle the hold in the agent The `interaction_required` error carries the approval id, tier, expiry, and an opaque `binding`. The binding covers the principal, Authority record, governed Session, Delegation, application, canonical resource and scope sets, active policy-set manifest, and platform decision contract. A retry after policy or execution context changes cannot spend an earlier decision. The SDK runs the whole flow - catch the hold, wait for the decision, retry with the approval id so the retried mint consumes it - in one call: ```ts import { Caracal } from '@caracalai/sdk' const caracal = new Caracal() const mandate = await caracal.withApproval((approvalId) => caracal.mintMandate('resource://pipernet', ['pipernet:refund'], { approvalId })) ``` When the wait outcome is anything but `approved` - rejected, expired, already consumed, or the wait timed out - `withApproval` rethrows the original `ApprovalRequiredError`. Its `approvalId` is durable: persist it and resume later with `waitForApproval`, which long-polls and returns the typed final state without re-raising the hold: ```ts const state = await caracal.waitForApproval(persistedApprovalId, { timeoutMs: 600_000 }) // 'approved' -> retry the mint with { approvalId } to consume the decision // 'pending' -> nobody has decided; waiting again is safe // 'rejected' | 'expired' | 'consumed' -> terminal; re-request the operation ``` An approval is **single-use**: it releases exactly one authority. The first mint that presents the approval id under its bound Authority record spends it, and the consumption records the authority it created. When two in-flight attempts from the same run race one approval - a retried queue message picked up while the first mint is still outstanding - one wins and the loser receives the typed `approval_consumed` error. This is not a new `ApprovalRequired` hold and is not an error to wait out: check whether the intended effect already happened before requesting another approval. A lost success response never costs a second decision. When the network drops a successful mint's response, the retry - same approval id, same complete binding, same credentials - arriving within a two-minute window receives a fresh bearer for the very authority the consumption created: nothing new is minted into existence, nothing needs revoking, and the human is not asked again. Outside that window the retry receives `approval_consumed` and the requester raises a fresh hold. A worker that has since restarted returns with a new Authority record and cannot spend a decision approved for the prior run; it re-requests the operation and raises a fresh hold - expiry and re-request are the expected shape of long gaps, not a failure. A rejection is final. The rejected hold stays authoritative for its remaining window, during which identical re-asks are refused rather than re-raised, so a declined agent cannot nag its approver into fatigue. After the window closes, requesting the operation again raises an entirely fresh hold with a fresh decision - rejection never permanently revokes the underlying grant, and no API can flip, cancel, or reopen a decided hold. The Python and Go clients expose the same surface as `with_approval`/`wait_for_approval` and `WithApproval`/`WaitForApproval`. The lower-level `@caracalai/oauth` client offers `waitForApproval` directly for integrations that drive the exchange themselves. Under `caracal run` none of this is hand-written: the engine prints an `approval_required` notice with the approval id and binding, waits on the hold, and retries the exchange itself. ## Decide as an operator Open the zone's **Approvals** page in the web console to review pending holds - each shows the requesting principal, the tier, the binding to cross-check against the agent's notice, and the approval window - and approve or reject with an optional reason. For automation, the Admin API exposes the same decision: ```bash curl -X POST \ "$CARACAL_API_URL/v1/zones/$CARACAL_ZONE_ID/approvals/$APPROVAL_ID/approve" \ -H "Authorization: Bearer $CARACAL_ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{"reason": "PiperNet refund reviewed against baseline v3"}' ``` Use `/reject` to settle the hold terminally. Deciding requires an admin token minted with the `approve` capability - `write` alone cannot decide a hold, so day-to-day automation credentials never carry approval authority. The approver is recorded from the authenticated actor, never from the request body. To hear about new holds without watching the web console, add a [notification sink](/v1.0/guides/approval-notifications/) that pushes approval events to your team's own systems. ## Decide as the application's Federated user A hold declared `"approver": "subject"` reserves the decision for the application's own Federated user and refuses every operator decision with `subject_approval_required`. Deciding it takes a user session mandate minted through federation: register the application's identity system as a Federated user issuer in the zone, exchange the user's identity token (`subject_token_type=urn:ietf:params:oauth:token-type:id_token`) for that user's session mandate, then post the decision to the STS: ```bash curl -X POST "$CARACAL_STS_URL/approvals/$APPROVAL_ID/decision" \ -H "Authorization: Bearer $USER_SESSION_MANDATE" \ -H "Content-Type: application/json" \ -d '{"decision": "approved", "binding": "'$BINDING'", "reason": "refund confirmed"}' ``` The SDKs wrap both steps: `caracal.federateSubject(idToken)` creates the Federated user's Authority record and returns its `subjectAuthorityRecordId` plus mandate; Python and Go expose `subject_authority_record_id` and `SubjectAuthorityRecordID`. The OAuth client's `decideApproval({...})` posts the decision. The decision must echo the hold's binding exactly. The Session that raised the hold cannot approve itself, and the deciding Federated user must have federated through the application that raised it. Without a registered Federated user issuer, a hold reserved for the Federated user can only expire. When the gated execution acts for a known Federated user - the exchange carried that user's token, or the requesting Session was started with the Federated user's authority record - the hold anchors to that exact person and the decision is reserved for them. A session mandate for any other user is refused, so one user of an application can never approve authority requested on behalf of another. Because the requester's own token lineage still cannot decide the hold, approving means the same human authenticating freshly through the application's identity system. Holds raised by executions whose Subject is the application itself - a workload credential, an application-only Session - carry no anchor and stay decidable by any of the application's Federated users; attribute Sessions to their Federated users at start when the tier's decision must be personal. ## Retry the exchange Retry with `approval_id` once the hold is approved. STS verifies the complete binding against the current execution and policy context, then consumes it: an approval releases at most one mandate for the held capability and cannot be replayed. STS emits issuance, decision, and consumption lifecycle events to the zone audit pipeline. ## Troubleshooting | Symptom | Check | | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Deny without an `approval_id` | The deny was not an approval gate. Confirm the scope appears in `risk` and its tier in `approval_tiers`. | | `subject_approval_required` on approve | The tier declares `"approver": "subject"`: only the application's own Federated user can decide it. Relay the hold to that user, or redeclare the tier as `operator` or `any`. | | Reserved decision refused | The hold anchors to the Federated user the requesting agent acts for, and the presented session mandate belongs to someone else. Relay the approval to that user; no one else can decide it. | | `approval_not_decidable` | The hold was already decided, consumed, or expired; the response names its current state. | | `approve_capability_required` | The admin token carries `write`, not `approve`. Mint a token with the `approve` capability. | | Retry still denied | The approval may have expired, or the retry changed its principal, Authority record, Session, Delegation, application, resource, scopes, policy set, or decision contract. Request a fresh approval for the current context. | ## Validate the flow Test approve, reject, expiry, concurrent consumption, changed binding, a second user deciding a hold anchored to another Federated user, and a missing issuer for a Federated-user-only tier. Expected result: exactly one matching retry consumes an approval; no decision widens resource, scopes, Session, Delegation, application, or policy binding; an anchored hold is decidable only by its own Federated user. :::caution[Failure point: Federated user authority] A Federated user approval works only when the application registers a trusted Federated user issuer, federates the user's token, and presents the resulting user session mandate. A caller-supplied Subject identifier or Session label cannot approve a hold. ::: ## Next Step Add [Approval Notifications](/v1.0/guides/approval-notifications/) if operators need push delivery, then add all terminal states to [Test Caracal Integrations](/v1.0/guides/testing/). --- # Safe Retries and Idempotency # URL: https://docs.caracal.run/v1.0/guides/idempotency/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/idempotency.md # Type: page # Concepts: # Requires: --- Most developers can ignore idempotency. Caracal automatically protects its own session and delegation creation calls against the network failures and transient server errors that ordinary applications encounter. This guide explains the exceptional case: work delivered by a queue, webhook provider, workflow engine, or scheduler that can arrive again after the original process has stopped. ## Prerequisites * The source system's immutable delivery, event, workflow, or schedule identifier. * The destination's idempotency or transactional deduplication mechanism. * A retry budget that distinguishes network/`5xx`/`408`/`425`/`429` from terminal authorization and validation failures. ## What Idempotency Means An operation is **idempotent** when repeating the same request has the same effect as performing it once. For example, asking Caracal twice to create the same governed Session should not create two Session records. Idempotency does not mean that arbitrary application code runs exactly once. A process can call an external API successfully and crash before recording success. After restart, no generic SDK can determine whether the external API committed the change. ## The Default: Do Nothing For a normal request handler, command, or in-process task, omit `idempotencyKey`: ```ts await caracal.session( async () => { await caracal.fetch('resource://pipernet', '/reports/market-risk') }, { labels: ['research-agent'], task: 'Prepare the market risk report', }, ) ``` The SDK generates a cryptographically random operation identifier before contacting the Coordinator. If the connection drops, the Coordinator returns a transient error, or the SDK refreshes a rejected bearer, every retry reuses that identifier. The Coordinator stores a durable receipt in PostgreSQL, so a committed session or delegation creation is replayed instead of created again. The receipt contains: * a keyed digest of the operation identifier, never the plaintext value; * the authenticated tenant and application scope; * a canonical fingerprint of every security-relevant request field; * the original response and created resource id; * an explicit expiry time. Receipts store only bounded operational response fields. They do not duplicate session metadata, invocation parameters, credentials, or bearer tokens. The default receipt window is seven days for explicit stable identifiers (`IDEMPOTENCY_RETENTION_SECONDS`) and one day for automatically generated retry identifiers (`GENERATED_IDEMPOTENCY_RETENTION_SECONDS`). The shorter generated window contains storage growth without weakening process-local retry safety. Each application and operation can retain at most 10,000 live receipts by default (`IDEMPOTENCY_MAX_RECEIPTS_PER_SCOPE`), preventing a compromised workload from growing the receipt index without bound. The Coordinator also caps request bodies at 256 KiB (`COORDINATOR_BODY_LIMIT_BYTES`). ## When an Explicit Key Helps Supply a key only when the system delivering work already has a stable identifier that survives process restart: | Source | Use | | --------------- | --------------------------------------------------------- | | Queue | Queue or subscription namespace plus immutable message id | | Webhook | Verified provider namespace plus delivery or event id | | CloudEvents | `source` plus `id` | | Workflow engine | Workflow namespace, run id, and step or activity id | | Scheduler | Schedule namespace plus intended fire time | ```ts async function handleWorkItem(message: QueueMessage) { const ticket = message.body await caracal.session( async () => { await caracal.fetch('resource://pipernet', '/reports/market-risk') }, { labels: ['research-agent'], task: `${ticket.title} (${ticket.key})`, idempotencyKey: `ticket-queue:v1:${message.id}`, }, ) } ``` Python uses `idempotency_key=`. Go uses `SessionOptions{IdempotencyKey: ...}`. The namespace and version prevent unrelated integrations from colliding. Change the version only when intentionally defining a different operation. ## Exact Guarantee With the same key and the same request fields, Caracal replays the original **Coordinator creation response** during the configured retention window. A replay carries the `Idempotency-Replayed: true` response header and appears as `replayed: true` on SDK coordinator events. With the same key and different fields, Caracal returns `409 idempotency_key_conflict`. This includes changes to the Subject authority record ID, parent Session, application, lifecycle, labels, task or metadata, authority mode, TTL, Delegation endpoints, resource, scopes, constraints, or invocation parameters. If the receipt's governed Session or Delegation has already terminated, expired, or been revoked, Caracal returns `409 idempotency_result_inactive` instead of replaying an unusable identity. This prevents a delayed sequential redelivery from entering the callback. Use a newly versioned operation id only for an intentional rerun. While the original session remains active, a valid replay still enters your callback. The key does not claim a queue message, serialize consumers, cache the callback result, or make downstream effects exactly once. Two concurrent consumers can both execute application code. Use the queue's own visibility/lease mechanism to prevent concurrent delivery. ## Protect External Side Effects Pass a stable operation id to every destination that supports idempotency: ```ts const operationId = `market-risk:v1:${message.id}` await caracal.session( async () => { await caracal.fetch('resource://pipernet', '/reports/market-risk', { method: 'POST', headers: { 'Idempotency-Key': operationId }, }) }, { task: `Prepare ${ticket.key}`, labels: ['research-agent'], idempotencyKey: `session:${operationId}`, }, ) ``` For a database mutation, use a transactional inbox or unique operation-id constraint in the same database transaction as the mutation. For asynchronous publication, use a transactional outbox. For APIs that do not support idempotency, reconcile the destination state before retrying. The unavoidable failure case is: 1. The destination commits the effect. 2. The process crashes before recording completion. 3. The source redelivers the work. Only the destination, a shared transaction, or a durable workflow engine can close that gap. ## What Caracal Does Not Infer Caracal does not derive stable keys from payload hashes, task descriptions, trace ids, timestamps, user ids, URLs, or handler arguments. Those values are not reliable operation identities: identical payloads can represent legitimate separate work, and descriptions can change between retries. Framework adapters do not guess provider-specific message ids. When an integration has a verified stable source id, map it explicitly at the queue, webhook, workflow, or scheduler boundary. Ordinary HTTP resource-server adapters only verify authority and bind context; they do not create governed Sessions or deduplicate handlers. ## Validation and Security Explicit keys must be non-empty, at most 255 UTF-8 bytes, contain no control characters, and have no surrounding whitespace. Both SDK and Coordinator enforce the contract. Treat a key as an operational identifier, not a credential: * never put tokens, secrets, passwords, prompts, request bodies, email addresses, or URLs with credentials in it; * prefer provider-generated opaque ids; * if creating an id yourself, use at least 128 random bits and persist it with the source operation; * do not log raw keys; * do not reuse one key for different operations; * do not use a mutable title or task description as the key. The Coordinator stores only an HMAC-SHA-256 digest under `IDEMPOTENCY_HMAC_KEY`. During key rotation, set `IDEMPOTENCY_HMAC_KEY_PREVIOUS` for at least one full receipt-retention window, then remove it. ## Common Mistakes | Mistake | Why it fails | Use instead | | ------------------------------------------------ | -------------------------------------------------------- | ---------------------------------------------------------- | | Generate a random key inside every queue attempt | Restart creates a different key | Use the immutable delivery or workflow id | | Hash the payload | Separate legitimate messages may have identical payloads | Use the source event id | | Use `ticket.key` for every operation on a ticket | Different operations collide | Namespace and version each operation | | Change labels or task while reusing the key | Fingerprint conflict | Use the original request or define a new operation version | | Assume the callback is skipped on replay | Session creation and callback execution are separate | Use source leases and destination-side deduplication | | Put personal data in the key | Keys may reach headers and diagnostics | Use an opaque provider id | ## Operations Monitor: * `caracal_idempotency_requests_total{outcome="created|replayed|conflicts|invalid|expired"}`; * `caracal_idempotency_receipts`; * `caracal_idempotency_oldest_seconds`; * retention-cleaner failures and database storage growth. A conflict usually indicates a programming error or an operation namespace reused across different work. A sudden replay spike usually indicates source redelivery or network instability. Raw keys never appear in these metric labels. ## Validate the retry design Deliver the same work twice with identical fields, then with one changed security field, then after the governed result becomes inactive. Expect a replayed Coordinator response, `409 idempotency_key_conflict`, and `409 idempotency_result_inactive` respectively. Separately prove the destination effect occurs once. :::caution[Failure point: Gateway calls] Caracal's Coordinator receipt does not deduplicate an upstream HTTP mutation. Forward the stable operation ID to the destination's `Idempotency-Key` or use a transactional inbox/outbox. ::: ## Next Step Encode replay, conflict, and destination-dedup cases in [Test Caracal Integrations](/v1.0/guides/testing/). --- # Test Caracal Integrations # URL: https://docs.caracal.run/v1.0/guides/testing/ # Markdown: https://docs.caracal.run/markdown/v1.0/guides/testing.md # Type: page # Concepts: # Requires: --- import { Tabs, TabItem } from '@astrojs/starlight/components' Caracal integrations test cleanly because every control-plane dependency enters through an injectable HTTP client: `fetchImpl` in TypeScript, `http_client` and `httpx.MockTransport` in Python, `HTTPClient` and `httptest` in Go. Fake the wire, not the SDK - your test exercises the same request construction, retry, and caching code that runs in production, and nothing reaches a network. ## When to use this guide Use it before production rollout and whenever Session, Delegation, transport, policy, approval, or resource-server requirements change. ## Prerequisites * The exact application workflow and enforcement boundary to test. * One allow case and explicit deny, expiry, revoke, timeout, replay, and cleanup expectations. * Injectable clients or a local runtime dedicated to the test. ## What to stub A governed Session touches a handful of endpoints. Stub the ones your code path uses and let anything unexpected fail loudly. The wire paths and fields retain protocol names - Sessions are `/agents` and `agent_session_id` on the Coordinator wire; the [Product-to-Wire Mapping](/v1.0/reference/interoperability-contracts/#product-to-wire-mapping) is the translation table: | Call | Respond with | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `POST /zones/{zone}/agents` | Protocol Session-start response `{"agent_session_id": "agent-1"}`; long-lived Sessions also require `heartbeat_deadline_at` and `lease_generation`. | | `POST /zones/{zone}/delegations` | Protocol Delegation response `{"delegation_edge_id": "edge-1"}`. | | `POST .../agents/{id}/heartbeat` | `{"agent": {"status": "active", "heartbeat_deadline_at": "...", "lease_generation": 1}}` | | `DELETE .../agents/{id}` | `204` | | `POST /oauth/2/token` | `{"access_token": "tok", "token_type": "Bearer", "expires_in": 900}` | Return an error status to exercise failure paths: a `503` drives the idempotent Session-start retry, a `429` from STS proves issuance is surfaced after one attempt, and a `401` from an `interaction_required` body raises the approval hold. Make scoped-transport Gateway fakes replay-sensitive: remember each presented bearer and reject a duplicate with `token_replayed`. Repeated and concurrent requests must carry distinct mandates, while an application transport must still create only one source/target Session pair and one delegation per authority-cache key. ## Fake the Coordinator in Each Language ```ts import { describe, expect, it, vi } from 'vitest' import { Caracal } from '@caracalai/sdk' it('runs work inside a governed Session', async () => { const calls: { url: string; method: string }[] = [] const fetchImpl = (async (input: RequestInfo | URL, init: RequestInit = {}) => { calls.push({ url: String(input), method: init.method ?? 'GET' }) if (init.method === 'POST' && String(input).endsWith('/agents')) { return new Response(JSON.stringify({ agent_session_id: 'agent-1' }), { status: 200 }) } return new Response(null, { status: 204 }) }) as typeof fetch const caracal = new Caracal({ coordinator: { baseUrl: 'http://coord.test', fetchImpl }, zoneId: 'z', applicationId: 'app', subjectToken: 'test-token', }) const result = await caracal.session(async (ctx) => ctx.sessionId) expect(result).toBe('agent-1') expect(calls.at(-1)?.method).toBe('DELETE') // the session was retired }) ``` ```python import httpx from caracalai import Caracal, CaracalConfig from caracalai.coordinator import CoordinatorClient async def test_runs_work_inside_a_governed_session() -> None: async def handler(request: httpx.Request) -> httpx.Response: if request.method == "POST" and str(request.url).endswith("/agents"): return httpx.Response(200, json={"agent_session_id": "agent-1"}) return httpx.Response(204) coordinator = CoordinatorClient( base_url="http://coord.test", http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)), ) caracal = Caracal( CaracalConfig( coordinator=coordinator, zone_id="z", application_id="app", subject_token="test-token", ) ) async with caracal.session() as ctx: assert ctx.session_id == "agent-1" ``` ```go func TestRunsWorkInsideAGovernedSession(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/agents") { w.Header().Set("Content-Type", "application/json") fmt.Fprint(w, `{"agent_session_id":"agent-1"}`) return } w.WriteHeader(http.StatusNoContent) })) defer srv.Close() client := &caracal.Caracal{ Coordinator: &caracal.CoordinatorClient{BaseURL: srv.URL}, ZoneID: "z", ApplicationID: "app", SubjectToken: "test-token", } err := client.Session(context.Background(), func(ctx context.Context) error { cur, _ := caracal.Current(ctx) if cur.SessionID != "agent-1" { t.Fatalf("unexpected session: %s", cur.SessionID) } return nil }) if err != nil { t.Fatal(err) } } ``` ## Assert on behavior, not internals Two seams make assertions precise without reaching into SDK state. The fake transport records every request, so you can assert the wire contract: an `idempotency-key` header on Session starts, the Delegation body, and termination on Session exit. `onEvent` reports control-plane operations as data, so tests can assert the expected `coordinator.call`, `token.exchange`, or `delegation.accept` events. Client-secret flows need the STS stub too: answer `POST /oauth/2/token` with a bearer and `expires_in`, and mint distinct tokens per call when the test asserts caching (a second exchange means a cache miss). For approval flows, return the `interaction_required` error body once and a token on the retry - `withApproval` completes end to end against the fake. ## Integration tier Fakes prove your code; a local runtime proves the contract. `caracal up` starts the full platform on localhost, and the same test binary points at it by swapping the injected client for real URLs from `caracal status`. Keep this tier thin - a handful of end-to-end paths per integration - and let the fake-backed tests carry the matrix. Caracal's source CI runs [the protected-resource contract](https://github.com/Garudex-Labs/caracal/blob/main/tests/typescript/e2e/protected-resource.mjs) against a freshly built stack and a real HTTP upstream. It provisions a zone, application, provider, resource, grant policy, and active policy set; opens a lifecycle parent and narrowed child Session; calls the upstream through Gateway; waits for correlated STS and Gateway audit events; and removes every object it created. This is the canonical deployable proof for the safe path: ```text application credential -> parent Session -> narrowed child Delegation -> one-shot Gateway mandate -> upstream -> audit ``` The workflow owns the admin-token file and upstream container. Application tests should continue using injected transports unless they intentionally run in an isolated operator environment. Related pages: [Integrate the TypeScript SDK](/v1.0/guides/sdk-typescript/), [Integrate the Python SDK](/v1.0/guides/sdk-python/), and [Integrate the Go SDK](/v1.0/guides/sdk-go/). ## Production acceptance result The fake-backed suite proves request construction and failure handling in TypeScript, Python, or Go; the thin runtime suite proves one real exchange, Gateway call, revocation, and correlated audit trace. No test depends on production secrets or a shared developer zone. :::caution[Failure point: over-mocking] Do not mock your own wrapper and conclude Caracal works. Record and assert actual Coordinator, STS, Gateway, and adapter wire behavior, including cleanup and stable idempotency keys. ::: ## Next Step Add acceptance cases to deployment gating, then use [Debug Authorization Decisions](/v1.0/guides/authorize-access/) for any runtime-only difference. --- # Add a Cloud Provider # URL: https://docs.caracal.run/v1.0/operations/add-a-cloud-provider/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/add-a-cloud-provider.md # Type: workflow # Concepts: # Requires: --- Caracal's deployment logic contains no cloud provider. Each deployment model has a provider-neutral core that owns the whole deployment shape, and a small adapter per provider that translates generic concepts into that provider's resources. Supporting a new cloud, or an internal platform, means writing adapters. It never means changing the core. ## What the Core Owns The core decides everything that is a property of Caracal rather than of a cloud: which images run, which start command selects a service role, which ports they listen on, which credentials each service needs, how readiness is probed, how far each service scales, and that schema migrations complete before any service rolls. An adapter never restates these. If you find yourself encoding a port, an image, or a service's environment in an adapter, that fact belongs in the core. ## Deployment Models | Model | Core | Adapter | | --- | --- | --- | | Virtual machine | `infra/tofu/modules/caracalHost` renders cloud-init | `infra/tofu/providers//host` creates the instance, exposure, identity, and DNS | | Managed container platform | `infra/containerPlatform/topology.yaml` and `render.mjs` | `infra/containerPlatform/targets/.mjs` renders platform manifests | | Kubernetes | `infra/helm/caracal` | A values overlay in `infra/helm/caracal/examples` | ## Virtual Machine Adapter Implement one OpenTofu module that satisfies the host contract. The contract is the same on every cloud, so a caller swaps providers by changing a module source and nothing else. Inputs: `name`, `region`, `machineSize`, `diskGb`, `userData`, `adminUsername`, `adminPublicKey`, `ingressCidrs`, `adminCidrs`, `networkCidr`, `dnsZone`, `hostnames`, `dnsTtl`, `tags`. Outputs: `publicIp`, `hostId`, `identityId`, `hostnames`. Group any input only your provider needs after the shared contract, under a comment naming the provider. Create the instance, its network exposure, a cloud identity, and the DNS records; nothing else. Open inbound 80 alongside 443, because certificate issuance answers its challenge over plain HTTP before a certificate exists. ```hcl module "bootstrap" { source = "../../modules/caracalHost" caracalVersion = "v0.2.1" tlsProxy = { email = "ops@example.com" routes = { "console.example.com" = "web" } } } module "host" { source = "../../providers/myCloud/host" name = "caracal-prod" region = "region-1" userData = module.bootstrap.userData adminPublicKey = file("~/.ssh/id_ed25519.pub") hostnames = ["console.example.com"] } ``` `bash infra/tofu/scripts/validate.sh` validates every adapter against its real provider schema and fails if any contract input or output is missing. ## Managed Container Platform Adapter Implement one module exporting three members, then register it in `render.mjs`. | Member | Purpose | | --- | --- | | `secretDelivery` | `file` when the platform can project a secret onto a filesystem, `env` when it can only bind a variable | | `internalUrl(service, config)` | The address other services reach this one on inside the deployment. Receives `{ name, port }` | | `render(plan, config)` | Returns a map of file name to file body | | `experimental` | Set `true` until the adapter has run against a live account | `secretDelivery` is the whole of the secrets abstraction. Every Caracal service accepts a credential either as a variable or through its `_FILE` form; the core reads your declaration and binds whichever the platform supports. Declare `file` and the core sets `DATABASE_URL_FILE` and hands you the list of files to project. Declare `env` and it sets `DATABASE_URL` and hands you the secret name to reference. Your adapter never chooses a variable name. Name rendered files so they sort in apply order, with migration jobs ahead of services: `10-job-migrate`, `20-app-sts`, and so on. Ship an apply flow that runs the jobs to completion before rolling any service, because no platform sequences that for you. Add an example deployment config under `examples/`, then run `bash infra/containerPlatform/scripts/validate.sh`. It renders every adapter through the same assertions: stock images selected by command, credentials resolved from a secret manager and never materialised, internal services not publicly exposed, and jobs sorted ahead of services. ## Kubernetes Adapter The chart is already provider-neutral. A cloud is four values: | Value | Concept | | --- | --- | | `replayPersistence.storageClassName` | Durable per-replica storage | | `ingress.*.className` and annotations | Ingress and certificates | | `serviceAccount.annotations` | Keyless cloud identity | | `global.podLabels` | Any label the provider's identity webhook requires | Copy an existing overlay from `infra/helm/caracal/examples`, substitute those values and the matching External Secrets store, and add it to the loop in the chart's validation script. A cloud that needs a chart change is a bug in the chart, not in the overlay. ## Maturity Mark a new adapter experimental until it has been exercised against a live account. For a container platform adapter that is `experimental: true`, which makes the renderer warn on every run; elsewhere it is a note at the top of the file and a row in [Cloud Support Matrix](/v1.0/operations/cloud-support-matrix/). Rendering cleanly proves a manifest is well formed. It does not prove a deployment works. ## Next Step Read [Cloud Support Matrix](/v1.0/operations/cloud-support-matrix/) for what each provider currently implements. --- # Cloud Support Matrix # URL: https://docs.caracal.run/v1.0/operations/cloud-support-matrix/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/cloud-support-matrix.md # Type: reference # Concepts: # Requires: --- Caracal ships three deployment models. The stack itself is provider-neutral: the same released images, the same environment contract, the same schema migrations. Each model has a provider-neutral core and a small adapter per cloud, so a provider is a translation layer rather than a branch in the deployment logic. To add one, see [Add a Cloud Provider](/v1.0/operations/add-a-cloud-provider/). ## Status | Model | Core | Azure | AWS | Google Cloud | | --- | --- | --- | --- | --- | | Virtual machine | `caracalHost` cloud-init | Supported | Experimental | Experimental | | Managed container platform | `containerPlatform` topology | Supported (Container Apps) | Experimental (ECS on Fargate) | Experimental (Cloud Run) | | Kubernetes | Helm chart | Supported (AKS overlay) | Experimental (EKS overlay) | Experimental (GKE overlay) | **Supported** means the adapter is implemented, validated in CI, and exercised against a live account. It is not a capacity, availability, or compliance claim. **Experimental** means the adapter renders, validates against the provider's own schema, and satisfies the shared contract, but has not been deployed to a live account. Azure is the reference implementation; treat the others as a starting point you verify in your own environment. ## What Never Changes These hold on every cloud and every model, and no adapter may alter them: * The published images, with the container start command selecting the service role. * The environment and secret contract in [Configuration Reference](/v1.0/reference/configuration/). * Expand-only schema migrations applied to completion before a rollout. * Postgres with migration privileges, and Redis with Streams and `noeviction`. * `/health` and `/ready`, and the same readiness semantics. ## Virtual Machines `caracalHost` renders cloud-init; a `providers//host` adapter creates the instance, its inbound exposure, its cloud identity, and its DNS records. Every adapter takes the same inputs and returns the same outputs, so changing cloud means changing a module source. Set `tlsProxy` on the core to terminate HTTPS and derive the console origin and token issuer from its routes. | Concept | Azure | AWS | Google Cloud | | --- | --- | --- | --- | | Instance | Linux virtual machine | EC2 instance | Compute Engine instance | | Public address | Static public IP | Elastic IP | Static external address | | Inbound exposure | Network security group | Security group | VPC firewall rule | | Identity | User-assigned managed identity | IAM role and instance profile | Attached service account | | DNS | Azure DNS A record | Route 53 record | Cloud DNS record set | **Caveat.** The bootstrap installs Docker through `get.docker.com`, which supports Debian, Ubuntu, RHEL, and Fedora. It does not support Amazon Linux 2023 or Container-Optimized OS. The adapters use Ubuntu images; if you change the image, install the container runtime with `extraRuncmd`. ## Managed Container Platforms `topology.yaml` describes the deployment once; an adapter maps it onto a provider. Each adapter declares how its platform delivers secrets and how services address each other, and the core does the rest. | Neutral concept | Azure Container Apps | AWS ECS on Fargate | Google Cloud Run | | --- | --- | --- | --- | | Role selection | `command` override | `entryPoint` and `command` | `command` override | | Secret delivery | Projected file | Environment variable | Environment variable | | Secret reference | Key Vault URL with a managed identity | Secrets Manager ARN with a task role | Secret Manager version with a service account | | Registry auth | Managed identity with `AcrPull` | Task execution role | Runtime service account | | Internal address | `https://.internal.` | Service Connect discovery name | Internal-ingress service URL | | One-shot migration | Container Apps Job | Task definition run with `RunTask` | Cloud Run Job | | Autoscaling | `http` concurrency rule | Service desired count | Concurrency and instance bounds | | Durable spill storage | NFS Azure Files share | EFS access point | None available | | Resource shape | Fixed vCPU-to-memory ladder, 4 vCPU ceiling | Fargate task sizes | Up to 8 vCPU and 32 GiB | Two consequences are worth knowing before choosing a provider: * **Only Container Apps projects credentials as files.** ECS cannot mount a secret at all, and Cloud Run refuses two secret volumes at one mount path, so neither can give a service the several credentials it needs as files. On both, credentials are bound to environment variables and resolved before the instance starts. They never appear in a task definition or service manifest, but a principal who can describe the running workload can read them; scope that permission accordingly. * **Cloud Run has no durable per-instance volume.** STS and Gateway spill audit evidence to disk when Redis is unreachable; on Cloud Run that evidence survives only while the instance lives. ## Kubernetes The chart is cloud-neutral. Four values carry the entire provider surface: | Value | AKS | EKS | GKE | | --- | --- | --- | --- | | `replayPersistence.storageClassName` | `managed-csi` | `gp3` | `premium-rwo` | | `ingress.*.className` | `webapprouting.kubernetes.azure.com` | `alb` | `gce` | | `serviceAccount.annotations` | `azure.workload.identity/client-id` | `eks.amazonaws.com/role-arn` | `iam.gke.io/gcp-service-account` | | `global.podLabels` | `azure.workload.identity/use: "true"` | not required | not required | `infra/helm/caracal/examples` carries a worked overlay for each. A cloud that needs a chart change is a bug in the chart. ## Database Prerequisites The baseline migration creates the `pgcrypto` extension and six service roles, and grants `CREATEDB` to the role that owns the console's auth database. Managed Postgres restricts all three, so confirm them before the first deployment on any provider. A migration job that fails here fails the whole rollout, by design. | Requirement | Azure Database for PostgreSQL | Amazon RDS and Aurora | Cloud SQL | | --- | --- | --- | --- | | `pgcrypto` | **Must be added to the `azure.extensions` server parameter first.** `CREATE EXTENSION` fails until it is, and the parameter change needs a server restart | Available to the master user | Available to the default user | | Create roles | Administrative user is a member of `azure_pg_admin` | Master user holds `rds_superuser` | Default user holds `cloudsqlsuperuser` | | Grant `CREATEDB` | Permitted for the administrative user | Permitted for the master user | Permitted for the default user | Run the migration job against the managed instance before provisioning anything else. It is the cheapest possible failure and it exercises connectivity, TLS, credentials, and privileges in one step. ## Choosing Use [Choose a Deployment Profile](/v1.0/operations/deployment-profiles/) to pick a model, then the workflow page for it: [Docker Compose](/v1.0/operations/docker-compose/), [a managed container platform](/v1.0/operations/managed-container-platforms/), or [Helm](/v1.0/operations/kubernetes-helm/). --- # Deploy on a Managed Container Platform # URL: https://docs.caracal.run/v1.0/operations/managed-container-platforms/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/managed-container-platforms.md # Type: workflow # Concepts: # Requires: --- The published images are role-agnostic: one Go image carries the STS, Gateway, and Audit binaries, one Node image carries API and Coordinator, and the container start command selects the role. Any platform that runs an OCI image with a custom start command, file-projected secrets, and HTTP probes can therefore run Caracal from released artifacts with no build step of your own. `infra/containerPlatform` renders that deployment. `topology.yaml` describes the stack once, in platform-neutral terms; a target module maps it onto a provider. Azure Container Apps ships today. ## Use Criteria Choose this path when you want a managed runtime, per-service scaling, and revision-based rollout without operating a cluster, and when the traded guarantees below are acceptable. Choose [Deploy with Helm](/v1.0/operations/kubernetes-helm/) when any of them are not. ## What You Trade Away The Helm chart enforces controls that container platforms do not expose. None of them are optional in a hardened deployment; on this path you replace them or accept their absence. | Control | Helm chart | Container platform | Consequence | | --- | --- | --- | --- | | East-west network policy | Default-deny NetworkPolicy per service | No pod-level policy; subnet NSGs only | Any workload in the environment can reach Audit and Coordinator | | Container hardening | `readOnlyRootFilesystem`, dropped capabilities, `runAsNonRoot`, seccomp | Not configurable | The image's own non-root user is the only remaining boundary | | Audit spill durability | Per-replica persistent volume | Replica-local ephemeral storage by default | A replaced replica loses audit evidence it had not drained to Redis | | Disruption budgets | PodDisruptionBudget per service | None | Platform maintenance can take replicas below your intended floor | | Failure-domain spread | Topology spread constraints and anti-affinity | Platform-chosen placement | Replica distribution across zones is not something you assert | | Scheduling priority | PriorityClass keeps the control plane above best-effort work | None | No eviction ordering under pressure | | Migration ordering | `pre-install`/`pre-upgrade` hook, automatic | Explicit job run before the rollout | Ordering depends on the deploy flow, not the platform | | Vertical headroom | Node-sized | 4 vCPU and 8 GiB per replica | Scale out rather than up; STS and Gateway hit this first | | Metrics collection | ServiceMonitor and PrometheusRule | Not consumed by the platform | Alerting must be rebuilt on the provider's monitor | Kubernetes remains the recommended path for production because those controls are the deployment's security and availability posture, not decoration. This path is appropriate for evaluation, staging, lower-traffic production, and teams with no cluster to run. What does not change: horizontal autoscaling, replica floors and ceilings, health-gated rolling updates, TLS ingress, and per-service resource limits are all expressed natively. Revision traffic splitting is a stronger canary primitive than the chart's rolling update. ## Prerequisites Operate managed Postgres reachable with migration privileges, managed Redis with Streams and `noeviction`, a secret manager holding the runtime credentials, a registry, and a workload identity with pull and secret-read access. Review [Choose a Cloud Profile](/v1.0/operations/cloud-native-profiles/) for the dependency contract, which is identical here. ## Procedure 1. Import the released images into your registry so deployments pull from a private, in-region source. No rebuild is involved, and the digest and its provenance are preserved: ```bash for image in caracal-go caracal-node caracal-web caracal-postgres; do az acr import --name \ --source ghcr.io/garudex-labs/${image}:v0.2.1 \ --image ${image}:v0.2.1 done ``` 2. Store the runtime credentials in the secret manager. `render.mjs` writes `secrets.txt` listing every required name in the platform's naming form; the key material itself is the same set the Helm chart consumes, described in [Rotate Keys and Secrets](/v1.0/operations/key-management/). 3. Download the deployment tooling for the release you are deploying. It is a published, checksummed, attested artifact, so no source checkout is involved: ```bash tag=v0.2.1 gh release download "$tag" --repo Garudex-Labs/caracal \ --pattern "caracal-deploy-${tag}.tar.gz" --pattern SHA256SUMS grep " caracal-deploy-${tag}.tar.gz$" SHA256SUMS | sha256sum --check --strict - gh attestation verify "caracal-deploy-${tag}.tar.gz" --repo Garudex-Labs/caracal tar -xzf "caracal-deploy-${tag}.tar.gz" npm --prefix caracal-deploy install --omit=dev ``` 4. Copy `caracal-deploy/containerPlatform/examples/azure.yaml`, set your endpoints, public origins, operator admission, and provider wiring, and render: ```bash node caracal-deploy/containerPlatform/render.mjs --config deployment.yaml --out ./rendered ``` The `console` block carries operator sign-in: `operatorEmails` lists the addresses or `@domain` suffixes admitted to the console, and `console.auth` names at least one sign-in method - a Google or GitHub client id with the name of its client secret in the secret manager, or an SMTP transport. Rendering fails without a method, on a plaintext public origin, debug logging in `stable` mode, a missing identity, or an unresolvable value. Review the manifests before applying them. 5. Apply in order. Schema migrations must finish before any service revision rolls, because a release's migrations are expand-only against the version still serving: ```bash CARACAL_RESOURCE_GROUP= bash caracal-deploy/containerPlatform/scripts/deployAzure.sh ./rendered ``` 6. Restore what the platform does not provide: subnet-level network restrictions, provider-native metrics and alert rules equivalent to the chart's, and backup and restore for Postgres and Redis. ## Durable Audit Spill STS, Gateway, and Audit write audit evidence to disk when Redis is unreachable and drain it on the next start. With ephemeral storage that evidence does not survive replica replacement. To keep it, attach an NFS file share and set `azure.stateStorageName`. Use NFS rather than SMB: the spill path fsyncs each file and its parent directory, an SMB share does not reliably honor that, and a failed fsync is accounted as lost evidence and fails readiness. ## Verify Confirm secrets resolve into the container, migrations completed, and every service answers `/ready`. Run `caracal-deploy/smokeTest.sh` against the ingress hosts, then a canary token exchange and Gateway request, and locate the resulting audit evidence. Confirm internal-only services are not externally reachable. ## Rollback or Recovery Roll back by activating the previous revision; it is immediate and does not rebuild. Migrations are never reversed by a revision rollback, which is why they stay expand-only. Keep the previous rendered manifests alongside the release they describe. ## Teardown Which teardown is correct depends on whether the environment holds anything you need to keep. For a **long-lived environment**, `caracal-deploy/containerPlatform/scripts/destroyAzure.sh` removes only the container apps and jobs it created. The database, cache, vault, registry, and the environment itself survive, so audit evidence and credentials are not destroyed along with the compute. For a **temporary environment**, put every resource in one resource group and delete the group: ```bash az group delete --name --yes ``` This is deliberately manual and deliberately not automated. It is atomic and complete, whereas removing only the applications leaves the database, cache, registry, and environment billing indefinitely. Confirm the result in Cost Analysis rather than trusting the command's exit code: a teardown is proven by the spend reaching zero. ## Next Step Use [Monitor Health and Metrics](/v1.0/operations/observability/) to rebuild the signals the chart's ServiceMonitor and alert rules would have provided. --- # Performance and Scalability # URL: https://docs.caracal.run/v1.0/operations/performance-benchmarks/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/performance-benchmarks.md # Type: reference # Concepts: # Requires: --- These are results from the Caracal **v1.0.0** production validation: a primary bench on a production cloud VM, and an extended bench covering component costs, failure injection, startup, and storage behavior. Every figure below was observed on the bench described in [Test Conditions](#test-conditions); none is extrapolated or theoretical. Treat these as a **reference baseline for sizing and evaluation, not a service-level guarantee.** Throughput and latency depend on your hardware, workload shape, and upstreams. Establish your own baseline with [Scale Capacity](/v1.0/operations/scale-capacity/) before committing to a limit. ## Test Conditions | Attribute | Primary cloud bench | Extended bench | | --- | --- | --- | | Deployment | Packaged Docker Compose via `caracal up`, single host, loopback-bound, stock configuration | Packaged Docker Compose, single host, loopback-bound | | Host | Azure `Standard_D4as_v5`: 4 vCPU, 16 GiB RAM, StandardSSD, Ubuntu 24.04 | 16 vCPU, 60 GiB RAM, NVMe SSD, Linux (Fedora 43) | | Load model | Closed-loop client, published TypeScript SDK, one in-flight governed request per worker, 16 long-lived application identities | Closed-loop client, published TypeScript SDK, one in-flight governed request per worker | | Upstream | Mock HTTP service on the compose network (isolates Caracal overhead) | Local mock HTTP service (isolates Caracal overhead) | | Credential path | Long-lived identities: verification served from the credential cache, the steady state for persistent agent fleets | Includes full Argon2id credential derivation | A governed request performs one single-use, replay-protected mandate mint at the STS plus one Gateway hop. An application transport provisions its session and delegation once per credential lifetime (four control-plane calls), then mints per request. ## Throughput and Scaling On the primary cloud bench, governed throughput scaled linearly with offered concurrency until the host CPU saturated, reaching **33 governed requests per second per host vCPU** with credential verification served from cache: | Concurrent workers | Sustained governed req/s | p50 | p95 | | --- | --- | --- | --- | | 1 | 8.2 (paced) | 20 ms | 30 ms | | 4 | 36.5 | 24 ms | 33 ms | | 8 | 63.2 | 22 ms | 33 ms | | 16 | 138.2 | 34 ms | 71 ms | An **8-minute sustained soak** at 16 workers held **132.6 governed requests per second** — 63,984 requests with **zero transport or 5xx errors** — for a daily capacity of **11.5 million governed calls on one 4-vCPU host**. Peak measured throughput was **139 req/s**. Two ceilings apply together: * **Per resource:** a single `(zone, resource, application)` pair is capped at **1000 mints/minute (16.6 req/s)** by default. The cap is a deployment setting: raise `STS_MINT_RATE_LIMIT_PER_MIN` for a higher ceiling, or set a lower working limit from the web console (Settings → Preferences → Mint rate limit). Spreading load across resources or applications also multiplies the budget; the soak above spread 16 identities. * **Per host:** CPU-bound, with the STS and Postgres as the first components to saturate. Add cores, or split the stores onto their own hosts, before tuning any Caracal setting. Session lifecycle is independent of the mint path: on the extended bench the Coordinator sustained **192 session create-and-close cycles per second** (p50 20 ms) before lock contention flattened throughput. ## Latency Measured on the primary cloud bench. The governance decision itself is negligible; cost is concentrated in the cryptographic mint, and it degrades gracefully rather than collapsing as the host saturates. | Operation | p50 | p95 | p99 | | --- | --- | --- | --- | | Governed request (warm, below saturation) | 17-24 ms | 23-33 ms | 34-43 ms | | Governed request (at full host saturation) | 45 ms | 85 ms | 113 ms | | Governed request (cold, first call incl. authority provisioning) | 287 ms | — | — | | Policy (OPA) evaluation, mean over 128,000 evaluations under load | 1.04 ms | — | — | | List applications / resources | 4 ms | 6 ms | — | | Audit query (50 rows) | 6 ms | 11 ms | — | Component costs measured on the extended bench: Gateway mandate verification **3.7 ms**, delegation create **17.5 ms**, provider create **8.1 ms** p50. The mint dominates because it verifies identity and traverses the delegation graph cryptographically. Policy evaluation and Gateway verification are sub-millisecond to low-single-digit milliseconds. ## Resource Usage and Stability Measured during the primary cloud bench's **8-minute sustained soak** at full host saturation: **132.6 req/s, 63,984 requests, zero hard errors.** | Signal | Observation | | --- | --- | | Host memory | 2 GiB used of 16 at full load; memory was never a limiting factor | | Service restarts during soak | 0 | | Postgres connections | 27-43 of the 100 ceiling; no connection leak | | Audit events dropped | 0 — dead-letter queue empty at soak end, hash chain continuous | | Policy evaluation errors | 0 across 128,000 evaluations | On the extended bench, STS memory under sustained load showed a healthy garbage-collection sawtooth (floor 337 MB / peak 792 MB on a 2 GB limit) with no leak and no unbounded growth. :::note[Size STS memory deliberately] The STS hashes credentials with Argon2id, which allocates **64 MB per in-flight verification**. Two mechanisms bound that cost: verified credentials are cached against their stored hash (rotation invalidates the entry), so steady-state traffic skips the derivation entirely, and cold verifications run under a fixed concurrency budget (`STS_SECRET_VERIFY_CONCURRENCY`, default 2) that caps peak verification memory. In this validation, a 512 MB STS driven with a full derivation on every request exhausted its container limit under sustained multi-agent load and was restarted by the container runtime (fail-closed, automatic, ~2-5 s); raising the limit to **2 GB eliminated this entirely**. The packaged deployment ships the STS at **2 vCPU / 1 GB**; provision **2 GB** when raising the verification budget or running many distinct applications. ::: ## Failure and Recovery On the extended bench, each dependency was failed in isolation while a continuous governed probe ran. Every failure was **fail-closed** — no request ever succeeded without valid authority — and every recovery was **automatic**, with no manual intervention. | Injected failure | Live governed traffic | Recovery | Data loss | | --- | --- | --- | --- | | Redis unavailable (brief) | Requests block, then drain on return | Automatic, immediate | None | | Postgres unavailable (brief) | Requests block, then drain on return | Automatic, immediate | None | | STS process killed | Immediate fail-closed (connection refused) | Next request after restart; restart-to-healthy 0.6 s | None | | Gateway process killed | Requests fail closed | Automatic; restart-to-healthy 0.6 s | None | | Coordinator process killed | **Continues unbroken** (established transports mint against the STS) | No impact to live traffic | None | | Audit process killed | **Continues unbroken** (asynchronous write path) | Events buffer and drain; hash chain verified continuous | None | This separates the request path from the control and evidence paths: * **Required for live governed traffic:** STS, Gateway, Postgres, Redis. * **Not required for live governed traffic:** Coordinator (needed only to provision new sessions and delegations) and Audit (asynchronous). Their outages are invisible to established transports. Brief database or cache interruptions shorter than the request timeout are absorbed as **latency, not errors**. For diagnosis and recovery order, see [Recover from Failures](/v1.0/operations/failure-modes/). ## Startup and Operational Timings Measured on the extended bench. | Operation | Time | | --- | --- | | Download, checksum-verify, and install the runtime | 9.0 s | | Cold start (`caracal up`: image pull, migrate, readiness gate) | 17.7 s | | Warm restart (`caracal down` then `up`) | 4.1 s + 3.7 s | | Single service restart to healthy | 0.6 s | | `caracal status` | 37-87 ms | | Web console response | 3.6 ms | ## Storage and Capacity Growth Measured on the extended bench. | Metric | Measured | | --- | --- | | Audit storage per event | 1,843 bytes (including indexes) | | Audit events per governed call | ~1.4 | | Audit query scaling | Flat with total volume (keyset pagination over monthly partitions) | | Database footprint (end of run) | 176 MB for 49,498 audit events, 8,450 sessions, 186 applications, 186 providers, 33 resources | | Postgres connections | 100 ceiling; 50 peak under full load; no lock contention | | Audit retention | 365-day default ceiling (`AUDIT_RETENTION_DAYS`), console-adjustable below it; hourly Parquet export to S3-compatible storage for longer archival | At a sustained governed rate, audit disk grows at approximately `rate × 1.4 × 1843` bytes per second — for example, ~62 KB/s (~5.3 GB/day) at 24 req/s. Plan audit storage and [retention](/v1.0/operations/backup-retention/) for your rate. ## Known Product Limits Defaults are deliberately conservative safety limits. The throughput figures above were reached by raising the capacity-shaping limits for the test workload. | Limit | Default | Behavior when exceeded | | --- | --- | --- | | STS mint rate, per `(zone, resource, app)` | 1000 / minute (`STS_MINT_RATE_LIMIT_PER_MIN`); console working limit below it | Denied, fail-closed; recorded as `rate_limited` in audit | | STS repeated authentication failures, per application | 60 / minute | Temporary block | | Coordinator requests, per client IP | 600 / minute (`COORDINATOR_RATE_LIMIT_PER_MIN`) | `429 rate_limited` | | Live sessions per zone | 50 (`MAX_AGENTS_PER_ZONE`) | `429 session_zone_limit_exceeded` | | Live sessions per application | 200 (`MAX_AGENTS_PER_APP`) | `429 session_limit_exceeded` | | Control API calls, per client | 60 / minute | `429` | To operate at the throughput and concurrency in this report, raise `MAX_AGENTS_PER_ZONE` to your concurrent-agent count and `STS_MINT_RATE_LIMIT_PER_MIN` to your target rate, then confirm the change with [Scale Capacity](/v1.0/operations/scale-capacity/). Configure limits and container resources in [Configure Service Environment](/v1.0/operations/env-vars/); the console-managed working limits live under Settings → Preferences. ## Recommended Starting Profiles Sizing is driven by the STS (throughput and memory) and Postgres (durability). These profiles derive from the extended-bench measurements, which include full credential derivation on the load path; fleets of long-lived identities served from the credential cache reach substantially higher throughput on the same profile, as the primary-bench results above show. Size the supporting stores with headroom above the measured footprint and validate before production. | Workload target | STS | Gateway | Postgres | Redis | | --- | --- | --- | --- | --- | | Small — ≤ 10 req/s, ≤ 50 sessions | 1 vCPU / 1 GB | 1 vCPU / 512 MB | 2 vCPU / 2 GB | 0.5 vCPU / 512 MB | | Standard — ~25 req/s, hundreds of sessions | 2 vCPU / 2 GB | 2 vCPU / 1 GB | 4 vCPU / 4 GB (SSD) | 1 vCPU / 768 MB | | Large — ~40 req/s per instance | 4 vCPU / 4 GB | 2 vCPU / 2 GB | 8 vCPU / 8 GB (NVMe) | 1 vCPU / 1 GB | | High throughput | STS replicas at 4 vCPU / 4 GB, sharded across resources | 4 vCPU / 2 GB | 8 vCPU / 16 GB + read replica | 2 vCPU / 2 GB | The packaged deployment ships the STS at 2 vCPU / 1 GB, between the Small and Standard profiles. Move to the Standard profile's 2 GB when sustaining ~25 req/s. Postgres used up to roughly one vCPU-equivalent and 461 MB with 50 connections at 24 req/s; provision it with headroom above that and prefer SSD or NVMe storage. ## Production Recommendations * **Size STS memory at or above the 1 GB packaged default.** The credential cache removes Argon2id from the steady state and the verification budget bounds cold bursts; provision 2 GB when raising `STS_SECRET_VERIFY_CONCURRENCY` or running many distinct applications. * **Raise the per-zone session cap** (`MAX_AGENTS_PER_ZONE`) to your concurrent-agent count before load; the default is 50. * **Raise `STS_MINT_RATE_LIMIT_PER_MIN`** when a shared resource needs more than 16.6 req/s, or shard across resources or applications; then add STS replicas to scale beyond one instance. * **Provision Postgres as the second bottleneck** after STS CPU: SSD or NVMe storage, connection headroom below the 100 default, and roughly twice the STS vCPU. * **Alert on the STS memory limit, Postgres connections, and audit lag**, and validate every limit change against your own workload. ## Next Step Turn these results into a sized deployment with [Choose a Deployment Profile](/v1.0/operations/deployment-profiles/), then validate your own baseline in [Scale Capacity](/v1.0/operations/scale-capacity/). --- # Configure Secret Backends # URL: https://docs.caracal.run/v1.0/operations/secret-backends/ # Markdown: https://docs.caracal.run/markdown/v1.0/operations/secret-backends.md # Type: workflow # Concepts: # Requires: --- `CARACAL_SECRET_BACKEND` selects storage for user-entered credentials. It does not replace runtime secret delivery. Values are envelope-encrypted under `SECRET_STORE_KEK` before backend storage. ## Choose a Backend Implemented identifiers are `builtin`, `vault`, `infisical`, `azurekeyvault`, `awssecretsmanager`, `gcpsecretmanager`, and `custom`. `builtin` stores envelopes in Postgres. External backends require their documented endpoint, identity, project/region, or token variables on API and STS. ## Safe Procedure 1. Back up Postgres, runtime secrets, and external backend data. 2. Configure one backend on API and STS with the same KEK. 3. Restart API and STS; wait for readiness. 4. Create a disposable provider credential and perform a token exchange. 5. Verify backend operation/error metrics and delete the disposable object. STS caches external reads for 60 seconds and may serve stale cached data for up to 10 minutes during errors. It never falls back to another backend. ## KEK Rotation Deploy the replacement as `SECRET_STORE_KEK` and retiring value as `SECRET_STORE_KEK_PREVIOUS`. Run `node apps/api/scripts/rotate-secret-store-kek.mjs` from a checkout with the API service's environment (the script ships beside the API service), verify zero failures, then remove the previous key and restart API and STS. :::danger[Permanent credential loss] Removing the retiring KEK before all envelopes are re-sealed makes remaining envelopes unreadable. ::: ## Backend Migration and Recovery Use `node apps/api/scripts/migrate-secret-backend.mjs ` from the same environment while the source remains configured. Switch services only after copying succeeds. On failure, keep the source selected and rerun; do not create a fallback chain. ## Next Step Plan dependent rotations in [Rotate Keys and Secrets](/v1.0/operations/key-management/).