Production checklist
Verify a deployment end to end. Automated checks first, then the manual checks no script can do.
Run after every deployment to a new environment and after infrastructure changes. The Deploy workflow already checks API readiness and the origin lock on every run; tofu plan showing no changes after a deploy confirms there is no drift. Record results, dates and who ran them in the release ticket. A green build is not a production verification.
Automated checks
| Check | Command | Proves |
|---|---|---|
| Services healthy | curl -sf https://api.<domain>/health/live and /health/ready | API up; PostgreSQL and Redis reachable from the API |
| Migrations applied | App Platform deployment log shows the migrate job succeeded | Schema matches the image |
| Control plane | node --env-file=/secure/verify.env scripts/verify-production.mjs | Service-key auth, call create/end, join and messaging tokens, readiness, anonymous denial |
| Scanner | ssh root@<scanner-ip> 'docker compose -f /opt/lynkoria-scanner/compose.yaml ps' shows scanner healthy | clamd reachable, signatures fresh |
| Static sites | curl -sI https://console.<domain> shows Content-Security-Policy and Strict-Transport-Security | Security headers deployed |
| Metrics | Query up{job="lynkoria-api"} and up{job="livekit"} | Scraping works |
| CI on the deployed commit | GitHub Actions run for the exact SHA | Tests, scans and OpenAPI drift passed |
verify-production.mjs needs PUBLIC_API_ORIGIN and a disposable LYNKORIA_SERVICE_KEY with calls and messaging scopes. Revoke the key afterwards. It does not publish media.
Infrastructure checks
| Component | Command | Expected |
|---|---|---|
| Origin lock | curl -s -o /dev/null -w '%{http_code}' https://<app>.ondigitalocean.app/openapi.json | 403 |
| Cloudflare path | curl -sI https://api.<domain>/health/ready | grep -i cf-ray | A cf-ray header |
| Media TLS | curl -sI https://rtc.<domain> and openssl s_client -connect turn.<domain>:443 | Valid Let's Encrypt certificates |
| Media DNS | dig +short rtc.<domain> turn.<domain> | The media IP, not Cloudflare addresses |
| Unknown room | curl -s -o /dev/null -w '%{http_code}' "https://rtc.<domain>/rtc/validate?access_token=<token for a room the API did not create>" | Rejected (not 200) |
| Database access | doctl databases firewalls list <id> for PostgreSQL and Valkey | Only the VPC range (no leftover runner IPs) |
| Runtime role | As lynkoria_app: SHOW statement_timeout; and CREATE TABLE x(id int); | 30s; permission denied |
| Bucket | Anonymous curl of an object URL | 403 |
| Scanner exposure | curl -m 5 -k https://<scanner-public-ip>:3344/health/ready from outside the VPC | Times out |
| Media firewall | doctl compute firewall get <id> | Only 22 (admin IPs), 80, 443, 7881, UDP 3478, UDP 50000–60000 public |
Manual end-to-end checks
Use two separate networks (for example office Wi-Fi and a phone hotspot) and two people or browser profiles. The playground can drive most of these against staging.
| Area | Check |
|---|---|
| Authentication | Console sign-in works; a wrong password is rejected; a disabled account cannot sign in; sign-out ends the session. |
| API authentication | A revoked service key fails on the next request; a lynk_test_ key is rejected by production. |
| WebSockets | Two clients connect; a non-allowlisted origin is refused; a suspended user is disconnected. |
| Video and audio | One-to-one and group calls connect over UDP, forced TCP and forced TURN/TLS; mute, camera and device switching work; reconnect after a network change. |
| Screen sharing | Desktop browser shares a screen and a tab; the share ends cleanly; viewers cannot share. |
| Livestreaming | Host publishes; viewers watch and cannot publish; promoting a viewer to speaker works; the stream ends for all. |
| Messaging | Send, edit, delete, react, read receipts and typing; disconnect, reconnect and replay history without gaps or duplicates. |
| Presence | Status changes appear on another client; closing a tab marks the user offline within the lease (about 60 s). |
| Attachments | A clean image uploads and downloads; EICAR is rejected; a non-member cannot download. |
| Recording (if Egress is deployed) | A composite recording finishes, downloads through the API and is deleted after retention. |
| Webhooks | LiveKit webhooks arrive (call usage updates); a product webhook verifies its signature and retries on failure. |
| Console | Project, environment and key changes persist and appear in the audit log; analytics show the test traffic. |
| SDKs | A sample product backend (@lynkoria/server) and web app (@lynkoria/client) complete the quickstart against the deployment. |
| Recovery | Restore a database backup into an isolated cluster; roll back the API to the previous digest; an alert reaches a person. |
Mobile native media, push delivery and capacity targets need devices and staging load tests; they are tracked in blockers and testing completion criteria.