Recover from Failures
Use this only after diagnosis. Restore durable state, then services, then evidence and access-safety freshness.
Recovery Order
Section titled “Recovery Order”- Freeze rollouts and risky traffic.
- Restore Postgres, then Redis.
- Restore API, STS, Gateway, Audit, Coordinator, and web readiness.
- Drain outboxes, pending entries, audit replay, and DLQ.
- Prove policy and revocation freshness.
- Run canary allow/deny and locate audit evidence.
Failure Procedures
Section titled “Failure Procedures”| Failure | Recovery |
|---|---|
| Postgres | Restore connectivity/capacity, migrations and pools, then outboxes |
| Redis | Restore persistence/noeviction, groups and pending entries, then replay |
| STS | Restore stores, policy bundle, signing/KEK, JWKS, Gateway HMAC |
| Gateway | Restore STS exchange, binding, revocation, egress, replay |
| Audit | Restore stores/HMAC, drain replay/DLQ, check tamper state |
| Coordinator | Restore stores, service token, outbox, workers |
audit_replay_unavailable can clear after storage recovers. audit_evidence_lost is latched: preserve evidence, scope the interval, then replace the process as an incident action.
Verification
Section titled “Verification”Readiness passes; queues are drained/understood; revoked sessions deny; current policy applies; audit records canaries.
Rollback
Section titled “Rollback”Return to preserved config/dependency snapshot if repair worsens state. Database restore is separate destructive recovery.
Next Step
Section titled “Next Step”Rehearse Run Failure Drills.