---
title: "Debug Infrastructure Issues"
url: "https://docs.caracal.run/v1.0/operations/debugging/"
markdown_url: "https://docs.caracal.run/markdown/v1.0/operations/debugging.md"
description: "Diagnose an unhealthy deployment from orchestration through storage, streams, services, and safety state."
page_type: "workflow"
concepts: []
requires: []
---

# Debug Infrastructure Issues

Canonical URL: https://docs.caracal.run/v1.0/operations/debugging/
Markdown URL: https://docs.caracal.run/markdown/v1.0/operations/debugging.md
Description: Diagnose an unhealthy deployment from orchestration through storage, streams, services, and safety state.
Page type: workflow
Concepts: none
Requires: none

---

Use this after readiness, rollout, or dependency failure. It diagnoses infrastructure; recovery is separate.

## Prerequisites

Preserve the first failing readiness response, deployment/config diff, workload status, logs, and alert timeline before restart.

## Diagnosis Procedure

1. Inspect Compose containers or Kubernetes pods, Jobs, events, and rollout.
2. Compare mode, Secret keys, URLs, ports, origins, and release pin with the release.
3. Test Postgres connectivity, migrations, pools, and outbox age.
4. Test Redis connectivity, `noeviction`, persistence, groups, and pending entries.
5. Read exact readiness reason and correlated logs.
6. Inspect policy age, STS circuit, revocation snapshot, audit DLQ/replay, and evidence-loss latch.

For an Operator model failure, inspect `GET /v1/operator/ai/status` and compare each provider's `last_ok_at`, `last_error_at`, and `last_error_class`. The matching `/metrics` timestamp gauges make the transition alertable without attaching high-cardinality error messages. These signals come from real requests, so stale or null values are not active reachability checks. Use **Test connection** only when a fresh, quota-consuming completion is appropriate. Do not use `/ready` to diagnose a model endpoint: readiness intentionally checks platform dependencies without calling or reading an LLM provider.

Start with `caracal status --json`, Compose status/logs, or Kubernetes get/describe/logs.

## Verification

Diagnosis is complete when one dependency, value, ceiling, or safety invariant explains readiness with log/metric evidence.

## Recovery Boundary

Do not delete volumes, reset streams, restore data, or clear evidence before selecting recovery.

## Next Step

Apply [Recover from Failures](/v1.0/operations/failure-modes/).