Harden Production

Production Caracal should run in published mode (CARACAL_MODE=stable, or rc for staging), use TLS at external boundaries, and keep stateful dependencies private. rc and stable enforce the same fail-closed trust boundary; rc is a release-candidate build that may be functionally unstable, so use stable for production.

Hardening Checklist

Area	Required stance
External TLS	Terminate TLS at ingress, load balancer, service mesh, or Gateway TLS files. The web console public edge must be HTTPS.
Internal services	Prefer private ClusterIP or bridge-network traffic.
Secrets	Use mounted secret files, not inline env vars.
Containers	Drop Linux capabilities, set no-new-privileges, use read-only filesystems where supported.
NetworkPolicy	Admit only required ingress and egress, including explicit ingress-controller traffic to the web tier.
Gateway upstreams	Keep private upstreams disabled unless explicitly allowed and allowlisted.
Runtime CLI	Do not expose product management as top-level runtime CLI commands.

Web Console Edge

The production web tier is a same-origin deployment: the caracal-web image serves the built SPA and the backend-for-frontend from the same HTTPS origin. The BFF proxies /api/console/* to the internal API and Coordinator, so browser traffic does not reach those services directly and no cross-origin CORS or CSRF surface is needed for the standard deployment.

Use HTTPS at the public edge and set CARACAL_AUTH_URL to that origin. The BFF pins Secure, HttpOnly, and SameSite session cookies, sets hardened response headers (Content-Security-Policy with frame-ancestors 'none', X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer, and HSTS when HTTPS is in use), enforces the request Origin on state-changing proxied calls, rate-limits credential endpoints, and validates proxied paths.

For a split deployment where the browser origin differs from the BFF origin, set CARACAL_WEB_ORIGIN to the trusted browser origins and use SameSite=None; Secure cookies. Keep this shape explicit; the packaged Helm and Compose paths serve the SPA same-origin.

With Helm NetworkPolicy enabled, add a networkPolicy.extraIngress rule for the ingress controller to reach the web service on port 3002. API, Coordinator, and Gateway should remain private unless a deployment requires their endpoints externally.

Gateway Upstream Safety

Upstream hosts are operator-provisioned through the Control API, never client-supplied, so the Gateway routes to private and on-prem upstreams (internal tools, local MCPs) by default. Dangerous infrastructure ranges that are never a legitimate upstream — cloud metadata (169.254.0.0/16), loopback, carrier-grade NAT, and multicast — are always blocked, including under DNS rebinding.

To pin egress to an explicit set of hosts as a hardening control, set:

UPSTREAM_HOST_ALLOWLIST=internal-api.example.svc,provider.example.com

HMAC and Signing Keys

Published modes require production-grade keys:

Key	Used for
AUDIT_HMAC_KEY	Audit event integrity.
STREAMS_HMAC_KEY	Redis stream message signatures.
GATEWAY_STS_HMAC_KEY	Gateway-to-STS service exchange.
ZONE_KEK	Zone secret/key encryption.

HMAC keys must be hex-encoded and at least 32 bytes where validated.

Kubernetes Controls

The Helm chart sets non-root pods, RuntimeDefault seccomp, dropped capabilities, read-only root filesystem for service containers, optional NetworkPolicy, PDBs, HPAs, and optional Ingress TLS.

Troubleshooting

Symptom	Check
Published-mode startup fails	Missing or short HMAC key, unsafe URL, forbidden fail-open setting, or wrong service port.
Gateway cannot reach upstream	NetworkPolicy egress, DNS egress, HTTPS egress, and upstream allowlist.
Metrics endpoint denied	Confirm METRICS_BEARER and scraper configuration.
TLS works externally but clients fail token validation	Confirm STS ISSUER_URL matches the URL clients use for JWKS and token issuer checks.

Next Step

Use Rotate Keys and Secrets to plan secret storage, key overlap, and rotation evidence.