---
title: "Harden Production"
url: "https://docs.caracal.run/operations/tls-hardening/"
markdown_url: "https://docs.caracal.run/markdown/operations/tls-hardening.md"
description: "Harden network exposure, secrets, service processes, and upstream access for production Caracal."
page_type: "workflow"
concepts: []
requires: []
---

# Harden Production

Canonical URL: https://docs.caracal.run/operations/tls-hardening/
Markdown URL: https://docs.caracal.run/markdown/operations/tls-hardening.md
Description: Harden network exposure, secrets, service processes, and upstream access for production Caracal.
Page type: workflow
Concepts: none
Requires: none

---

Production Caracal should run in published mode (`CARACAL_MODE=stable`, or `rc` for staging), use TLS at external boundaries, and keep stateful dependencies private. `rc` and `stable` enforce the same fail-closed trust boundary; `rc` is a release-candidate build that may be functionally unstable, so use `stable` for production.

## Hardening Checklist

| Area | Required stance |
| --- | --- |
| External TLS | Terminate TLS at ingress, load balancer, service mesh, or Gateway TLS files. The web console public edge must be HTTPS. |
| Internal services | Prefer private ClusterIP or bridge-network traffic. |
| Secrets | Use mounted secret files, not inline env vars. |
| Containers | Drop Linux capabilities, set no-new-privileges, use read-only filesystems where supported. |
| NetworkPolicy | Admit only required ingress and egress, including explicit ingress-controller traffic to the web tier. |
| Gateway upstreams | Keep private upstreams disabled unless explicitly allowed and allowlisted. |
| Runtime CLI | Do not expose product management as top-level runtime CLI commands. |

## Web Console Edge

The production web tier is a same-origin deployment: the `caracal-web` image serves the built SPA and the backend-for-frontend from the same HTTPS origin. The BFF proxies `/api/console/*` to the internal API and Coordinator, so browser traffic does not reach those services directly and no cross-origin CORS or CSRF surface is needed for the standard deployment.

Use HTTPS at the public edge and set `CARACAL_AUTH_URL` to that origin. The BFF pins `Secure`, `HttpOnly`, and `SameSite` session cookies, sets hardened response headers (`Content-Security-Policy` with `frame-ancestors 'none'`, `X-Frame-Options: DENY`, `X-Content-Type-Options: nosniff`, `Referrer-Policy: no-referrer`, and HSTS when HTTPS is in use), enforces the request `Origin` on state-changing proxied calls, rate-limits credential endpoints, and validates proxied paths.

For a split deployment where the browser origin differs from the BFF origin, set `CARACAL_WEB_ORIGIN` to the trusted browser origins and use `SameSite=None; Secure` cookies. Keep this shape explicit; the packaged Helm and Compose paths serve the SPA same-origin.

With Helm NetworkPolicy enabled, add a `networkPolicy.extraIngress` rule for the ingress controller to reach the web service on port `3002`. API, Coordinator, and Gateway should remain private unless a deployment requires their endpoints externally.

## Gateway Upstream Safety

Upstream hosts are operator-provisioned through the Control API, never client-supplied, so the
Gateway routes to private and on-prem upstreams (internal tools, local MCPs) by default. Dangerous
infrastructure ranges that are never a legitimate upstream — cloud metadata (`169.254.0.0/16`),
loopback, carrier-grade NAT, and multicast — are always blocked, including under DNS rebinding.

To pin egress to an explicit set of hosts as a hardening control, set:

```bash
UPSTREAM_HOST_ALLOWLIST=internal-api.example.svc,provider.example.com
```

## HMAC and Signing Keys

Published modes require production-grade keys:

| Key | Used for |
| --- | --- |
| `AUDIT_HMAC_KEY` | Audit event integrity. |
| `STREAMS_HMAC_KEY` | Redis stream message signatures. |
| `GATEWAY_STS_HMAC_KEY` | Gateway-to-STS service exchange. |
| `ZONE_KEK` | Zone secret/key encryption. |

HMAC keys must be hex-encoded and at least 32 bytes where validated.

## Kubernetes Controls

The Helm chart sets non-root pods, RuntimeDefault seccomp, dropped capabilities, read-only root filesystem for service containers, optional NetworkPolicy, PDBs, HPAs, and optional Ingress TLS.

## Troubleshooting

| Symptom | Check |
| --- | --- |
| Published-mode startup fails | Missing or short HMAC key, unsafe URL, forbidden fail-open setting, or wrong service port. |
| Gateway cannot reach upstream | NetworkPolicy egress, DNS egress, HTTPS egress, and upstream allowlist. |
| Metrics endpoint denied | Confirm `METRICS_BEARER` and scraper configuration. |
| TLS works externally but clients fail token validation | Confirm STS `ISSUER_URL` matches the URL clients use for JWKS and token issuer checks. |

## Next Step

Use [Rotate Keys and Secrets](/operations/key-management/) to plan secret storage, key overlap, and rotation evidence.