Operations and maintenance
Release, rollback, restore and rotate a running platform.
For first installation use deployment; infrastructure is OpenTofu code in infra/opentofu/, applied by GitHub Actions; for local work use getting started. Staging and production must have separate databases, Redis credentials, buckets, media nodes, keys and DNS. The v1 topology has single points of failure.
Release trains
API and static sites deploy from the reviewed Git revision after CI. The API image is scanned and pinned by digest; a PRE_DEPLOY job runs migrations before the new service receives traffic. The static console embeds VITE_API_ORIGIN at build time. The documentation is an exported site without a runtime backend.
SDKs use Changesets and restricted npm publishing behind NPM_PUBLISH_ENABLED; registry ownership and trusted publishing must be configured by the operator. LiveKit uses its own pinned image/configuration release. Do not update it implicitly during an API rollout.
Before every release: review dependency/security changes, run checks, verify staging behavior, inspect migration compatibility, record previous API and media digests, and confirm current backup/restore coverage. Use expand → migrate → contract: retain old columns/contracts until the rollback window and old clients have passed.
The runtime image excludes npm and applies a pinned Debian zlib security update on top of the pinned Node base. Review this package pin when refreshing the base image; retain the patch until the base includes the fix. A removed package version should fail the build rather than silently select a different version. Build and scan the replacement image before promoting its digest.
Rollback
The API is deployed by image digest through OpenTofu. To roll back, run Actions → Deploy → Run workflow for the environment with api_image_digest set to the previous digest (from the previous run's images job, or doctl registry repository list-tags lynkoria-api). OpenTofu changes only the image of the api service and the migrate/bootstrap-owner jobs; secrets and settings stay as the code defines them. The workflow then checks /health/ready and the origin lock.
Never edit the app spec by hand to roll back: the next apply would revert it. Image rollback does not reverse database migrations. If the previous image cannot read the current schema, stop and use the reviewed forward fix or an isolated restore/cutover plan. Never automatically run destructive down-migrations on live data.
For the console/docs, redeploy the previous Pages artifact/commit with the correct API origin. SDK consumers pin versions; publish a corrective patch rather than replacing a published version. Keep old image digests in the registry through the rollback window.
Rotation and compromise
| Credential | Procedure |
|---|---|
| Service key | Rotate → update product secret → deploy → verify new last-use → revoke old. On compromise, revoke immediately. |
| Webhook secret | Rotate in console; product temporarily accepts old and new via webhookSecret: [old, new]; remove old after queued/retried deliveries are accounted for. |
| Messaging key | OpenTofu -replace=tls_private_key.messaging issues a new key and kid; for zero-downtime rotation instead generate a new P-256 private key and kid by hand, retain old public key in previous-key map for at least 10 minutes plus clock tolerance; remove after expiry. Never reuse the same kid. |
| LiveKit environment key | Generated by OpenTofu per livekit_key_refs entry; rotate with -replace (operate). The media Droplet is replaced and the API updated in one apply; active calls end. |
| Database/Redis/Spaces | Managed by OpenTofu (-replace on the user or key resource) or rotate via provider; update only the affected components; verify TLS, pooling, object permissions and workers; revoke old credentials. |
| Pepper/webhook master | Emergency operation requiring coordinated replacement of all affected credentials; restore the same values during disaster recovery. |
| User suspension | Stop new tokens and disconnect messaging/remove connected media. End/recreate a call for a hard media ban. |
| Console offboarding | pnpm admin:user disable --email … (revokes sessions), remove console_roles, revoke IdP/Access sessions; inspect audit history and rotate credentials the user could have copied. |
Database backup and restore
Enable provider backups/PITR and verify the actual retention/restore limits for the selected plan. Back up encrypted platform secrets separately; restoring rows without the matching pepper, webhook master and signing material does not restore a usable platform.
Quarterly and before risky migrations: restore into a new isolated cluster, run integrity/count checks and migrations, connect a staging API using restored secrets, exercise messages/files/auth, and record recovery time and data loss window. Ensure the restored environment cannot send webhooks to production product endpoints. Do not test recovery by overwriting the live cluster.
A cutover needs a maintenance window, stopped writers, final backup/replication catch-up, reviewed URLs/CA updates, runtime grants, and verification before traffic resumes. Keep the former cluster read-only until the rollback window closes. Do not promise an RPO/RTO until measured.
Upgrading LiveKit
Choose an exact tag and digest, read its release notes, and change local.images.livekit in infra/opentofu/platform/media.tf (Caddy and Redis images are pinned there too). Merge to deploy staging and verify unknown-room denial, tokens, moderation, browser media, reconnect, UDP/TCP/TURN and intended load; native clients require device testing. Then deploy production in a quiet window.
The change replaces the media Droplet: OpenTofu creates the new one first, points rtc and turn at it, then destroys the old one. Active calls end; the reconciler closes sessions that disappeared. Caddy requests new certificates on the new Droplet (mind Let's Encrypt's duplicate-certificate rate limit if you replace it many times a week). Rollback is a revert of the commit, which replaces the Droplet again.
Monitoring and logging
Metrics: the API exposes Prometheus metrics on the private METRICS_PORT (request rate/latency by route, auth failures, sockets, LiveKit webhooks, attachment scans, outbox and push queues); labels never include keys, users or rooms. The monitoring worker (OpenTofu, infra/opentofu/platform/app.tf) scrapes the API and LiveKit and remote-writes to your backend. infra/monitoring/alerts.yml defines ApiHigh5xxRate, ApiLatencyP95High, AuthFailureSpike, TokenMintingAnomaly, OutboxBacklog, WebhooksDeadLettered, PresenceSweeperLagging, ResyncRateRising, ReconcilerCorrections, MediaHostCpuHigh, MediaHostCpuSteal and MediaEgressHigh. Set-up steps are in the monitoring step.
Health: /health/live (process) and /health/ready (PostgreSQL and Redis, 2 s budget each; 503 with draining: true during shutdown) on the API; /health/ready on the scanner also fails when signatures are stale. The console's Operations tab shows push deliveries, recordings and scan verdicts per project. Alerts, SLIs and what to do for each alert: runbooks.
Logs: JSON (pino) to stdout, collected by App Platform. Tokens, secrets, authorization headers, cookies and message bodies are redacted (apps/api/src/lib/logger.ts); configuration errors print variable names only. Every response carries x-request-id, which also appears in error bodies and audit rows, so support can correlate a user report with logs. Traces are exported when OTEL_EXPORTER_OTLP_ENDPOINT is set. Keep LOG_LEVEL=info in production.
Capacity and maintenance
The API can run several instances: Socket.IO fan-out and disconnects cross instances through the Redis Streams adapter (ADR 021). On shutdown App Platform drains the instance for 15 s, then the API closes sockets (clients reconnect elsewhere) and exits within 15 s; behind other ingresses set SHUTDOWN_DRAIN_SECONDS. Raise instance_count only after capacity tests; watch Redis CPU and the lynkoria:<environment>:socket.io stream when you do. PostgreSQL locks serialize writes within each conversation; this preserves order but makes a very busy room a contention boundary. Use pagination and monitor pool pressure, outbox backlog, retention throughput and event-table growth. Index changes and archival policy need real query/load evidence.
Retention processes bounded batches each hourly tick; a large backlog needs an operator-reviewed increase/frequency change. Product webhooks are at least once and dead-letter after 12 attempts. Daily request/token/message counters flush every 15 seconds and are not billing-grade. Monitor LiveKit outbound bitrate, CPU, packet loss and actual provider transfer usage; viewer count alone cannot predict cost.
Apply OS security updates in a media maintenance window. Check disk/certificate/backup health, registry retention, alert receivers, leaked credentials, and restore readiness regularly. See troubleshooting for incident paths.