ADR 021: Multi-instance API through the Redis Streams adapter
- Status: Accepted
- Owner: Eng lead
- Date: 2026-11-13
Context
The API ran as a single instance because Socket.IO kept rooms in process memory. A message sent through one instance never reached sockets on another, and suspending a user (disconnectSockets) only disconnected sockets on the instance that handled the request; elsewhere the user stayed connected until the 30-second revalidation. Everything else was already safe to run on several instances: presence leases, rate limits and worker ticks coordinate through Redis locks, queue claims use FOR UPDATE SKIP LOCKED, and usage counters are additive upserts.
Options considered
@socket.io/redis-adapter(pub/sub). Mature, but pub/sub drops every message published while an instance is disconnected from Redis, and its last release is from 2024.@socket.io/redis-streams-adapter. Messages go through a capped Redis stream that each instance reads from its last offset, so a short Redis disconnect does not lose fan-out; supportsdisconnectSocketsand room operations across nodes; maintained (0.3.1, March 2026); works with theredisclient the API already uses.- A message broker (NATS, Kafka). A new infrastructure component for a problem the existing Redis/Valkey already solves.
Decision
Use the Redis Streams adapter on the existing Redis/Valkey, with one stream per deployment environment (lynkoria:<staging|production>:socket.io, capped at about 10,000 entries). The adapter gets its own Redis connection with the offline queue enabled. Its read loop retries immediately after an error; with the main client's disableOfflineQueue: true, a Redis outage made that loop reject instantly and spin, which pinned a CPU core and starved the event loop in a local test. With the offline queue on, the same outage cost 20 ms of CPU over 4 seconds.
Connection state recovery stays off: clients already recover missed messages from PostgreSQL by sequence number (ADR 005), so the stream is only a fan-out path, never a source of truth.
Consequences
instance_countabove 1 is supported. A two-instance integration test (apps/api/test/multi-instance.test.ts) covers cross-instance message delivery and immediate cross-instance disconnect on suspension, and fails without the adapter.- Local emits do not wait on Redis. If Redis is unavailable, other instances miss those events and clients catch up by resync; rate-limited routes already fail closed in that case.
- Each broadcast adds one
XADDand is read by every instance. Load across replicas is unmeasured; the deployment keeps one instance until capacity tests run (LOAD-01). - Per-socket revalidation remains per instance (SEC-10); it costs database reads per event, not per replica.