Runbooks
Service-level indicators and what to do when each alert fires.
Every rule in infra/monitoring/alerts.yml carries a runbook annotation pointing to a section here. Severity page needs a person now; warn within working hours; info is a trend to review. Use the request id (x-request-id, returned on every API response and logged by the API and the scanner) to follow one request across logs.
Service-level indicators
| SLI | Measured by | Pilot target |
|---|---|---|
| API availability | 1 - rate(lynkoria_http_requests_total{status=~"5.."}) / rate(lynkoria_http_requests_total) | ≥ 99.5% |
| API latency | p95 of lynkoria_http_request_duration_seconds | < 250 ms |
| Realtime connection success | lynkoria_socket_connections_total{result="accepted"} / all attempts | ≥ 95% |
| Message persist latency | p95 of lynkoria_message_persist_duration_seconds | < 100 ms |
| Realtime delivery health | lynkoria_resync_requests_total rate (clients recovering missed events) | Stable baseline |
| Media join success and latency | Not measured server-side yet: needs client SDK reporting or LiveKit join metrics | Planned |
| Reconnect rate | Not measured yet (same dependency) | Planned |
Targets are pilot goals, not commitments; none has been measured on deployed infrastructure (blockers).
First checks for any alert
curl -s https://api.<domain>/health/readyreturns{ ok, checks: { database, redis } };draining: truemeans the instance is shutting down.- App Platform: recent deployments, restarts and logs for the
apicomponent (doctl apps logs <app-id> api --type run). - Media host:
cd /opt/livekit && sudo docker compose ps && sudo docker compose logs --tail=150. - Was anything released in the last hour (API image, LiveKit, console)? If so, consider rollback first.
ApiHigh5xxRate
More than 2% of API requests fail with 5xx. Check /health/ready: a failing database or redis check explains most cases (managed database maintenance, connection limits, expired credentials). Search logs for request failed with the route label to find the failing endpoint. If it started with a deploy, roll back. If Redis is down, rate-limited routes fail closed by design.
ApiLatencyP95High
p95 above 250 ms for 10 minutes. Compare by route in lynkoria_http_request_duration_seconds. Usual causes: database CPU or slow queries (check the managed database's query insights), connection pool exhaustion (DATABASE_POOL_MAX), or CPU saturation on the single API instance (App Platform CPU alert). Scale the instance size if CPU-bound; look for missing indexes if one route dominates.
AuthFailureSpike
Sustained authentication failures for one kind. A product with a rotated or revoked service key causes a burst of service_key failures; confirm with the product team. Unknown sources suggest probing: check client IPs in the audit log and tighten ALLOWED_ORIGINS or rate limits. If a key may have leaked, rotate it (rotation).
TokenMintingAnomaly
Token issuance is 3× the same hour yesterday. Legitimate causes: launches, events, retry loops in a product backend. Check lynkoria_tokens_issued_total by type, then the audit log for the service key responsible. A runaway retry loop should be fixed in the product; a leaked key must be rotated.
OutboxBacklog
More than 500 product webhook/push items waiting. Check whether one product endpoint is failing (webhook delivery logs, lynkoria_webhook_deliveries_total by result) or workers stopped (WORKERS_ENABLED, worker errors in logs). Deliveries retry with backoff; the backlog drains once the endpoint recovers.
WebhooksDeadLettered
Webhook deliveries exhausted their retries. The product endpoint rejected or timed out repeatedly. Tell the product owner, confirm the endpoint is healthy, then retry the dead-lettered deliveries from the console.
PresenceSweeperLagging
Presence leases are not expiring, so users can appear online after disconnecting. The sweeper runs under a Redis lock; check Redis health and worker errors. A crashed worker loop recovers on restart.
ResyncRateRising
Clients request missed messages more often than usual, which means realtime events are being dropped or connections are flapping. Check RealtimeConnectionRejectionsHigh, API restarts (each restart disconnects every socket) and Redis health. Messages are not lost: clients recover them through after_seq.
ReconcilerCorrections
The reconciler closed many calls or participants that LiveKit no longer knows about. LiveKit webhooks are probably not arriving: check LiveKitWebhooksRejected, the webhook URL in livekit.yaml, and that the media host can reach the API.
ApiTargetDown
Prometheus cannot scrape api:9464. Either the API instance is down (check App Platform health and restarts) or the metrics worker lost access. If the public API is healthy, fix the metrics worker; all other API alerts are blind until then.
RealtimeConnectionRejectionsHigh
More than 20% of Socket.IO connection attempts fail authentication. Common causes: an expired or rotated messaging signing key (MESSAGING_JWT_*), clock skew, a product sending tokens for the wrong environment, or an origin missing from ALLOWED_ORIGINS. lynkoria_auth_failures_total by kind narrows it down.
AttachmentScansFailing
Attachment scans returned no verdict, so uploads stay quarantined (fail closed). Check the scanner host:
docker compose -f infra/scanner/compose.yaml ps
curl -s --cacert /etc/lynkoria/scanner-tls/ca.crt https://<scanner-private-ip>:3344/health/ready # reason: e.g. "signatures are 52h old"
docker compose -f infra/scanner/compose.yaml logs --tail=100 scanner clamd updater
openssl x509 -enddate -noout -in /etc/lynkoria/scanner-tls/tls.crt # certificate expiryIf the API logs Scanner unreachable with a certificate error, the scanner certificate expired, was issued by a CA that is not in the API's SCANNER_CA_CERT, or does not name the address in SCANNER_URL. OpenTofu issues the certificate (tls_locally_signed_cert.scanner) and renews it on any apply in its last 30 days, replacing the scanner Droplet; run the Deploy workflow, or force it with -replace=tls_locally_signed_cert.scanner (operate). Also check that scanner.internal.<domain> resolves to the scanner's current private IP.
Stale signatures mean the updater cannot reach the ClamAV mirrors (egress network, rate limiting). A down clamd usually means memory pressure (it needs about 3 GiB). Failed uploads can be retried by clients after the scanner recovers.
LiveKitWebhooksRejected
LiveKit webhooks fail signature verification. After a key rotation, LIVEKIT_WEBHOOK_KEY/LIVEKIT_WEBHOOK_SECRET in the API must match webhook.api_key and its secret in livekit.yaml. Requests that are not from the media host are blocked by the signature check; confirm the source if the rate is high.
LiveKitTargetDown
Prometheus cannot scrape LiveKit on the media host. Treat as an SFU outage until proven otherwise: calls and livestreams fail. Check the container (docker compose ps, logs), the Droplet, and the VPC firewall for port 6789. Restart LiveKit only in a window if it is still serving calls.
MediaHostExporterDown
node_exporter on the media host is unreachable, so CPU and bandwidth alerts are blind. Restart the exporter; check the VPC firewall for port 9100.
MediaHostCpuHigh
The SFU host is above 80% CPU. Quality degrades for everyone on the node before it fails. Short term: cap new rooms or livestream quality in the products. Longer term: a larger CPU-Optimized Droplet (see load testing for measured capacity).
MediaHostCpuSteal
The hypervisor is taking CPU from the SFU. Media is sensitive to steal; move to a dedicated-CPU (CPU-Optimized) Droplet or resize to migrate hosts.
MediaEgressHigh
Projected monthly bandwidth exceeds the Droplet allowance; overage is billed. Review livestream bitrates and simulcast layers, and plan capacity before the allowance is reached.