Operate Redis Streams

Redis Streams move operational events between Caracal services. Redis is not the durable source of product state; Postgres remains authoritative.

Streams and Groups

Stream	Consumer groups
`caracal.audit.events`	`audit-ingestor`, `siem-export`
`caracal.audit.events.dlq`	`audit-dlq-observer`
`caracal.policy.invalidate`	`opa-engine`
`caracal.sessions.revoke`	`sts-revocation`
`caracal.keys.invalidate`	`sts-keys`
`caracal.agents.lifecycle`	`coordinator-relay`
`caracal.invocations.lifecycle`	`invocations-observer`
`caracal.delegations.invalidate`	`delegations-observer`
`caracal.providers.ratelimit`	initialized as a one-entry stream

Provision

REDIS_HOST=localhost REDIS_PORT=6379 \
REDIS_PASSWORD_FILE="${CARACAL_SECRETS_DIR:-${CARACAL_HOME:-$HOME/.local/share/caracal}/secrets}/redisPassword" \
bash infra/redis/provision-streams.sh

The provisioner is idempotent, creates consumer groups with MKSTREAM, and warns when an existing stream length exceeds the intended bound.

Verify

REDIS_HOST=localhost REDIS_PORT=6379 \
REDIS_PASSWORD_FILE="${CARACAL_SECRETS_DIR:-${CARACAL_HOME:-$HOME/.local/share/caracal}/secrets}/redisPassword" \
bash infra/redis/scripts/verify.sh

Verification checks stream existence, consumer group existence, and provisioner idempotency.

Event Path

flowchart LR
  API[API transaction] --> Outbox[event_outbox]
  Outbox --> Redis[Redis Streams]
  Coordinator[Coordinator outbox] --> Redis
  STS --> Redis
  Gateway --> Redis
  Redis --> Audit[Audit ingestor]
  Redis --> STSConsumers[STS invalidation/revocation]
  Redis --> GatewayConsumers[Gateway revocation]

Stream messages are signed with STREAMS_HMAC_KEY in published modes.

Durability and Eviction

Redis is a correctness-critical store, not a disposable cache: revocation entries and unacknowledged stream messages must survive memory pressure and restarts. Configure every Caracal Redis with:

Setting	Required value	Why
`maxmemory-policy`	`noeviction`	Any `allkeys-`/`volatile-` policy can silently drop revocation entries and pending stream messages, weakening revocation correctness.
`appendonly`	`yes`	Persists stream and revocation state across restarts.
`appendfsync`	`everysec`	Bounds data loss on crash to roughly one second.

The bundled Redis image (infra/redis/redis.conf) already sets these. External or managed Redis is your responsibility — many managed tiers default to allkeys-lru. On startup each consumer service (STS, Gateway, Audit) reads the live maxmemory-policy and logs a warning when it is not noeviction, so a misconfigured store is visible in service logs immediately. The check never blocks startup and is skipped when the provider restricts CONFIG GET.

Troubleshooting

Symptom	Check
Audit DLQ grows	HMAC failures, malformed producers, Redis memory pressure, and Audit database writes.
Policy changes do not take effect	`caracal.policy.invalidate` group health and STS policy age metrics.
Revocation is delayed	`caracal.sessions.revoke` pending entries and Gateway/STS consumer health.
Provisioner warns about length	Plan an explicit stream retention/reset; provisioning does not rewrite stream policy silently.

Next Step

Use Scale Capacity when storage, streams, or service pools approach operational limits.