Debug Infrastructure Issues

Use this after readiness, rollout, or dependency failure. It diagnoses infrastructure; recovery is separate.

Prerequisites

Preserve the first failing readiness response, deployment/config diff, workload status, logs, and alert timeline before restart.

Diagnosis Procedure

1. Inspect Compose containers or Kubernetes pods, Jobs, events, and rollout.
2. Compare mode, Secret keys, URLs, ports, origins, and release pin with the release.
3. Test Postgres connectivity, migrations, pools, and outbox age.
4. Test Redis connectivity, noeviction, persistence, groups, and pending entries.
5. Read exact readiness reason and correlated logs.
6. Inspect policy age, STS circuit, revocation snapshot, audit DLQ/replay, and evidence-loss latch.

For an Operator model failure, inspect GET /v1/operator/ai/status and compare each provider’s last_ok_at, last_error_at, and last_error_class. The matching /metrics timestamp gauges make the transition alertable without attaching high-cardinality error messages. These signals come from real requests, so stale or null values are not active reachability checks. Use Test connection only when a fresh, quota-consuming completion is appropriate. Do not use /ready to diagnose a model endpoint: readiness intentionally checks platform dependencies without calling or reading an LLM provider.

Start with caracal status --json, Compose status/logs, or Kubernetes get/describe/logs.

Verification

Diagnosis is complete when one dependency, value, ceiling, or safety invariant explains readiness with log/metric evidence.

Recovery Boundary

Do not delete volumes, reset streams, restore data, or clear evidence before selecting recovery.

Next Step

Apply Recover from Failures.