Troubleshooting
Fix common deployment problems, then diagnose setup, authentication, media and data-path failures.
Deployment problems
Read the failing job's log first: OpenTofu names the resource and the provider's error. A rerun of the Deploy workflow continues where the last one stopped.
| Symptom | Likely cause | Fix |
|---|---|---|
preflight fails or staging deploys are skipped | Environment lacks DIGITALOCEAN_ACCESS_TOKEN, TOFU_STATE_PASSPHRASE or TOFU_STATE_BUCKET | Add them (step 4) |
tofu init fails with access denied / NoSuchBucket | Wrong Spaces key or bucket name, or bucket not in sgp1 | Check SPACES_* secrets and TOFU_STATE_BUCKET; the backend endpoint is sgp1 |
decryption failed / encryption is enforced | Wrong state passphrase | Use the passphrase the state was written with; zone state uses TOFU_ZONE_STATE_PASSPHRASE |
Unable to find remote state / no origin_secrets | Zone stack not applied yet | Apply it (step 5) |
Variable errors (No value for required variable, invalid list) | A GitHub variable is missing, or a list is not JSON | Lists must look like ["a@example.com"] |
Cloudflare Authentication error / 403 on a resource | API token lacks a permission | Add the permission named in the error (step 2.5) |
| Ruleset create fails: a ruleset for the phase already exists | Rules created in the dashboard | Delete them or import the ruleset (operate) |
| Access application fails | Zero Trust organization not created, or wrong team domain | Open Zero Trust once; set CF_ACCESS_TEAM_DOMAIN |
| Droplet create fails with a size or limit error | Account Droplet limit or size not available in sgp1 | Request a limit increase or change media_droplet_size / scanner_droplet_size |
terraform_data.database_grants fails | Runner could not reach the database, or doctl/psql missing locally | Check the log; a temporary trusted source is added and removed by the script. If it says to remove a rule by hand, delete it in the cluster's trusted sources |
App deploy fails in the migrate job | Grants not applied, or wrong CA | doctl apps logs <app> migrate --type deploy; rerun Deploy (grants rerun when the SQL changes; -replace=terraform_data.database_grants forces it) |
| API starts then fails health checks | Configuration validation failed or a dependency is unreachable | doctl apps logs <app> api --type run; the API logs variable names only |
| Domain stuck in Pending on App Platform | Record proxied before the certificate was issued | Set API_DNS_PROXIED=false, deploy, wait for Active, remove it, deploy (step 6) |
| Smoke test: origin lock answered 200 instead of 403 | Zone rule or CF_ORIGIN_SECRET mismatch, or an old app spec | Re-apply the zone stack, then Deploy |
Every API request returns 403 forbidden | Record not proxied (header never added) or secret mismatch | Remove API_DNS_PROXIED=false once the domain is Active; re-apply zone, then Deploy |
rtc. has no certificate | DNS not resolving yet or record proxied | Wait a few minutes after the first apply; records are DNS only by code. On the Droplet: cd /opt/livekit && docker compose logs caddy |
| Calls: signaling connects but no media | UDP range or TCP 7881 blocked by the client network | Test forced TURN/TLS; check the media firewall in the DigitalOcean panel |
| Call create returns 503 | Key reference mismatch | The project environment's LiveKit key reference must be one of livekit_key_refs |
| Scanner not ready | First signature download still running | docker compose -f /opt/lynkoria-scanner/compose.yaml logs signatures-init updater; it can take several minutes |
| Scanner image pull fails at boot | Registry credentials or digest | cloud-init status --long on the Droplet; check DOCR_REGISTRY, then Deploy again |
API logs Scanner unreachable (certificate error) | DNS for the scanner name not updated yet, or certificate expired | dig +short scanner.internal.lynkoria.com must show the scanner's private IP; any apply renews the certificate in its last 30 days |
| Console requests blocked in the browser (CSP) | API host not in apps/admin/public/_headers | Add it and deploy |
| Console: CORS errors or redirect loops to Access | No Access cookie for the API host | Visit https://api.<domain>/v1/admin/me once, then the console |
| Console sign-in rejected with the right password | Access email differs from the console account | Use the same email in the IdP and admin-users.js create |
| Uploads fail in the browser | Product origin missing from allowed_origins | Add it to the environment's .tfvars and deploy |
| A plan shows changes nobody made | Someone edited a managed resource in a dashboard | Apply to restore the code's version, or move the change into code |
Local migration connection timeout
If pnpm db:migrate reports a connection timeout at CREATE SCHEMA IF NOT EXISTS "drizzle", check connectivity first: the query can fail before PostgreSQL executes any SQL. In root .env, make LYNKORIA_PG_PORT match the port in DATABASE_URL (and DATABASE_URL_DIRECT if set; it takes precedence for migrations). Likewise, match LYNKORIA_REDIS_PORT and REDIS_URL.
Run pnpm infra:up, then pnpm db:migrate. For direct Compose commands, include --env-file .env; without it, this nested Compose project may use default ports instead of your root configuration. Check published ports with docker compose --env-file .env -f infra/local/compose.yaml ps. Container health alone does not prove the host port is reachable. Recreating containers keeps their named data volumes; do not use down --volumes to fix connectivity.
Start with the request ID, API readiness, recent structured logs and relevant metrics. Do not paste tokens, service keys or full connection strings into tickets. Keep the failure’s environment/project ID and time window.
| Symptom | Checks and recovery |
|---|---|
| Local tests cannot connect | Confirm Compose host ports and matching DATABASE_URL/REDIS_URL; LiveKit and attachments profiles must be running; test DB user needs CREATE DATABASE. |
| API fails startup | Read variable names in validation errors; check all-or-none Spaces config, distinct pepper/master, PEM escapes, current kid vs previous keys, production TLS/CA/direct connection. |
| Migration fails | Use direct migration-role connection, correct database, trusted source and CA; verify schema ownership/grants. Do not give runtime DDL rights to bypass it. |
| Service request 401/403 | Correct staging/live prefix, deployment environment, active project/environment, key expiry/revocation and required scope. |
| Console returns HTML/401 | Sign into API admin hostname, verify shared Access audience/team, exact console origin and active DB role. Check API Access OPTIONS bypass and browser cookie blocking. |
| Socket rejected/disconnected | Exact Origin in global and project allowlists, messaging enabled, valid unexpired user JWT, active user/project, WebSocket support at ingress. Product callbacks must obtain fresh tokens. |
| 503 during Redis failure | Restore the private TLS endpoint/trusted sources/credentials. Rate limits fail closed; PostgreSQL data survives but request availability is degraded. Never expose Redis publicly as a workaround. |
| LiveKit WebSocket error | DNS-only media records, Caddy certificate/443, matching key reference, existing room, clock synchronization and token expiry. |
| Connected but no audio/video | Device permissions, published tracks, viewer role, public ICE candidate, UDP range, TCP 7881, TURN 3478/TLS 443, firewall and mobile-network test. |
| Livestream slow | Outbound bitrate/NIC, CPU/steal, simulcast, publishers, TURN share and load-generator limits. Measure before changing quotas. |
| Missing/duplicate messages | Stable send ID, client history watermark and after_seq replay, active membership, commit errors. Do not manually edit sequence numbers; inspect for writers bypassing the service. |
| Presence stale | 25-second heartbeat/60-second lease, disconnected devices, manual/invisible status, sweeper lag and Redis health. |
| Upload denied | Both attachment switches enabled, membership/uploader match, exact Spaces CORS, presigned URL expiry, actual file size/type and private bucket credentials. |
| Webhooks failing | Inbound raw body/key/signature and dedupe; outbound approved HTTPS origin, signature receiver, no redirects, dead letters and product idempotency. Retry after fixing the cause. |
| Usage delayed | 15-second flush for buffered counters; participant seconds appear on closure; inspect worker and reconciler health. |
| Docs page absent | Frontmatter, docs/meta.json, case-sensitive filename and static build output. Regenerate MDX types, run docs tests/build and inspect broken-link diagnostics. |
For bad releases follow rollback. For backup/recovery follow the restore drill before pointing traffic at recovered data. Keep an incident timeline and verify recovery from a real client, not only a health endpoint.