Rotate Keys and Secrets

Use this runbook on a rotation schedule, after suspected exposure, or when a person with secret access leaves. Rotate one class at a time. Inventory the KEK, zone signing keys, integrity HMACs, idempotency HMAC, Gateway-STS HMAC, service tokens, storage credentials, and web auth secret.

Prerequisites

Keep current data and secret backups, owners for every producer/consumer, canary requests, and a documented overlap window where supported.

Safe Procedure

1. Generate replacement material cryptographically.
2. Use SECRET_STORE_KEK_PREVIOUS for envelope re-sealing and IDEMPOTENCY_HMAC_KEY_PREVIOUS for receipt overlap.
3. Update every producer and consumer of a shared HMAC before retirement.
4. Roll affected services and wait for readiness.
5. Check audit, stream, replay, revocation, JWKS, and canary signals.
6. Retire old material only after applicable token, cache, message, replay, or receipt windows.

Keep the old idempotency key for at least IDEMPOTENCY_RETENTION_SECONDS.

Verify

Confirm retired credentials fail, envelopes decrypt, JWKS is expected, new stream/audit messages verify, pending entries drain, and Gateway-to-STS exchange succeeds.

Recovery

Restore the retiring value before deletion if verification fails. Preserve logs, DLQ, and replay state. Never regenerate a lost KEK; restore its backup.

Next Step

Verify PostgreSQL and Redis Streams.