Skip to content

Troubleshoot by Symptom

Most Caracal failures are easy to fix once you know where they happened. This page starts from the symptom you can see — usually an SDK error or an HTTP status — and routes you to the failing surface, the object to inspect, and the tool that confirms it. It complements Debug Infrastructure Issues, which works from infrastructure inward; start here when you have an application-level failure and an error or request ID.

flowchart TD
  Start[SDK or HTTP call failed] --> HasID{Have a request ID?}
  HasID -->|Yes| Trace[Open Console request trace]
  HasID -->|No| Status{caracal status --ready healthy?}
  Status -->|No| Infra[Infrastructure or readiness issue]
  Status -->|Yes| Symptom{What did you get back?}
  Symptom -->|Config or startup error| Config[SDK / runtime config]
  Symptom -->|401| AuthN[Authentication surface]
  Symptom -->|403 at token exchange| STS[STS policy decision]
  Symptom -->|403 at the resource| Verifier[Gateway or in-process verifier]
  Symptom -->|Call succeeded, no audit| Audit[Audit pipeline]
  Trace --> Symptom

The Console request trace is the fastest disambiguator: it ties one request ID to the resource, subject, scopes, determining policies, diagnostics, and result, so you can see whether the denial came from STS policy or from a resource verifier.

SymptomSurfaceInspectTool
SDK throws config_missing or cannot startSDK / runtime configRuntime profile, secret file, and required environment variables.caracal status --json, profile file
401 from API or ConsoleAuthenticationAdmin or Control token, or workload credential source.Use the Console credential inputs
403 from a token exchange (access_denied)STS policyActive policy-set activation, application ID, subject, requested scopes, and grant.request trace, Console audit
403 at the resource (invalid_token, scope_insufficient)Gateway or in-process verifierMandate validity, issuer/JWKS trust, X-Caracal-Resource, binding, and verifier scope checks.Proxy Through Gateway, verifier logs
scope_insufficient even after a policy allowResource definitionWhether the resource defines the scope and the policy authorizes it.Console resource, request trace
Call works but no audit event appearsAudit pipelineSelected zone, time window, Redis stream health, and Audit readiness.Console diagnostics, audit
Access continues after revocationResource verifierWhether revocation consumers run on the resource server.Sessions and Revocation

When an SDK call returns access denied and you are not sure why, walk these in order. Each step either fixes the problem or hands you to the next surface.

  1. Confirm the runtime is healthy. Run caracal status --ready. If a dependency is down, fix that first.
  2. Confirm configuration. Check the runtime profile, secret file, selected zone, and resource identifier the SDK is using. A wrong resource ID or zone produces denials that look like policy failures.
  3. Trace the request. Open Console request trace with the request ID. It shows the resource, subject, requested scopes, determining policies, and result.
  4. Check the grant and policy. Confirm a grant binds the application and subject to the resource scopes, and that the active policy set authorizes them. See Debug Authorization Decisions.
  5. Check policy-set activation. A policy that exists but is not the active set will not take effect. See Activate a Policy Set.
  6. Check the requested scopes. Confirm the SDK requests scopes the resource defines and the policy allows; a typo’d scope denies silently.
  7. Check the Gateway route or verifier. If STS allowed the exchange but the resource rejected it, inspect the Gateway binding and upstream allowlist, or the in-process verifier’s scope and identity checks. See Proxy Through Gateway.
  8. Confirm the audit trace. Open Console audit for the request ID to read the final decision and diagnostics.

Caracal exposes the triage bundle through existing surfaces rather than a separate command. Gather these together when you open an incident or escalate:

ItemWhere
Config summary and selected zonecaracal status --json, runtime profile
Service readiness and dependenciesConsole diagnostics (Doctor: health, readiness)
Visible zones, resources, and policy setsConsole diagnostics (Doctor: zones)
Deployment and operator environment checksConsole diagnostics (Doctor: preflight)
Resource-to-provider binding and policy allowCheck Provider Readiness (examples/providerPreflight)
Last denial and request traceConsole audit and request trace

The Console diagnostics Doctor view runs health, readiness, zones, and preflight in one place, including strict readiness for automation gates. Use it as the single bundle the rest of this page points back to.

Use Debug Infrastructure Issues when the symptom points to readiness, storage, streams, service configuration, or deployment state.