Upgrade Caracal

Caracal upgrades are forward-only at the database layer. Every release ships expand-only schema changes: new migrations are backward compatible with the previous application version, so the database can be migrated while the old version is still serving. This removes the maintenance window — the schema moves first, then services roll onto it.

Contract-phase changes (dropping, renaming, retyping, or tightening a column to NOT NULL) are split into a later release and must be tagged -- caracal:phase contract. CI (validateMigrations.sh) rejects untagged contract changes so the no-window guarantee cannot silently regress.

One-Command Upgrade (Runtime)

For the Compose-based runtime, caracal upgrade performs the full expand-then-roll sequence in order:

caracal upgrade

It stages the new images (pull, or build in dev), applies the expand-phase migrations against the running database, then rolls the services and waits for readiness. If the migration step fails it aborts before touching services, leaving the previous version running. Pass --no-pull to roll already-staged images without fetching.

Pre-Upgrade

1. Read release notes and changed chart values.
2. Render the new Helm/Compose configuration.
3. Confirm secrets and required env vars still match service config.
4. Back up Postgres and runtime secrets.
5. Confirm Redis streams/groups and audit health.
6. Prepare canary token exchange and Gateway request.

Helm Upgrade

helm -n caracal diff upgrade caracal infra/helm/caracal -f values.production.yaml
helm -n caracal upgrade caracal infra/helm/caracal -f values.production.yaml
kubectl -n caracal get jobs
kubectl -n caracal rollout status deploy/caracal-api

The migration Job runs as a pre-install/pre-upgrade hook when migrations are enabled.

Compose Upgrade (Manual)

caracal upgrade automates this. To run it by hand:

export CARACAL_VERSION=<new-version>
docker compose -f infra/docker/runtime-compose.yml pull
docker compose -f infra/docker/runtime-compose.yml run --rm dbMigrate
docker compose -f infra/docker/runtime-compose.yml up -d
bash infra/scripts/smokeTest.sh

Post-Upgrade Validation

Check	Expected
/ready endpoints	API, STS, Gateway, Audit, Coordinator pass.
Migrations	schema_migrations includes the expected versions.
Redis	Streams and consumer groups exist; pending entries are bounded.
Audit	DLQ empty or understood; replay backlog drains.
Token exchange	STS exchange succeeds for canary application/resource.
Gateway	Canary protected request reaches upstream and produces audit evidence.

Rollback

Prefer roll-forward fixes after migrations. If rolling back app images, verify the previous version understands the current schema and config values.

Troubleshooting

Symptom	Action
Migration hook fails	Stop rollout, inspect Job logs, restore only if no migration committed; otherwise roll forward.
New pods fail config validation	Compare env vars and mounted secret keys with current service config.
Readiness fails after app rollback	Schema or Redis stream expectations may no longer match the old app.

Next Step

Use Export Audit Evidence when audit retention, SIEM export, or compliance evidence needs a durable path.