Skip to Content
LYNKORIADocs

Troubleshooting

Fix common deployment problems, then diagnose setup, authentication, media and data-path failures.

Deployment problems

Read the failing job's log first: OpenTofu names the resource and the provider's error. A rerun of the Deploy workflow continues where the last one stopped.

SymptomLikely causeFix
preflight fails or staging deploys are skippedEnvironment lacks DIGITALOCEAN_ACCESS_TOKEN, TOFU_STATE_PASSPHRASE or TOFU_STATE_BUCKETAdd them (step 4)
tofu init fails with access denied / NoSuchBucketWrong Spaces key or bucket name, or bucket not in sgp1Check SPACES_* secrets and TOFU_STATE_BUCKET; the backend endpoint is sgp1
decryption failed / encryption is enforcedWrong state passphraseUse the passphrase the state was written with; zone state uses TOFU_ZONE_STATE_PASSPHRASE
Unable to find remote state / no origin_secretsZone stack not applied yetApply it (step 5)
Variable errors (No value for required variable, invalid list)A GitHub variable is missing, or a list is not JSONLists must look like ["a@example.com"]
Cloudflare Authentication error / 403 on a resourceAPI token lacks a permissionAdd the permission named in the error (step 2.5)
Ruleset create fails: a ruleset for the phase already existsRules created in the dashboardDelete them or import the ruleset (operate)
Access application failsZero Trust organization not created, or wrong team domainOpen Zero Trust once; set CF_ACCESS_TEAM_DOMAIN
Droplet create fails with a size or limit errorAccount Droplet limit or size not available in sgp1Request a limit increase or change media_droplet_size / scanner_droplet_size
terraform_data.database_grants failsRunner could not reach the database, or doctl/psql missing locallyCheck the log; a temporary trusted source is added and removed by the script. If it says to remove a rule by hand, delete it in the cluster's trusted sources
App deploy fails in the migrate jobGrants not applied, or wrong CAdoctl apps logs <app> migrate --type deploy; rerun Deploy (grants rerun when the SQL changes; -replace=terraform_data.database_grants forces it)
API starts then fails health checksConfiguration validation failed or a dependency is unreachabledoctl apps logs <app> api --type run; the API logs variable names only
Domain stuck in Pending on App PlatformRecord proxied before the certificate was issuedSet API_DNS_PROXIED=false, deploy, wait for Active, remove it, deploy (step 6)
Smoke test: origin lock answered 200 instead of 403Zone rule or CF_ORIGIN_SECRET mismatch, or an old app specRe-apply the zone stack, then Deploy
Every API request returns 403 forbiddenRecord not proxied (header never added) or secret mismatchRemove API_DNS_PROXIED=false once the domain is Active; re-apply zone, then Deploy
rtc. has no certificateDNS not resolving yet or record proxiedWait a few minutes after the first apply; records are DNS only by code. On the Droplet: cd /opt/livekit && docker compose logs caddy
Calls: signaling connects but no mediaUDP range or TCP 7881 blocked by the client networkTest forced TURN/TLS; check the media firewall in the DigitalOcean panel
Call create returns 503Key reference mismatchThe project environment's LiveKit key reference must be one of livekit_key_refs
Scanner not readyFirst signature download still runningdocker compose -f /opt/lynkoria-scanner/compose.yaml logs signatures-init updater; it can take several minutes
Scanner image pull fails at bootRegistry credentials or digestcloud-init status --long on the Droplet; check DOCR_REGISTRY, then Deploy again
API logs Scanner unreachable (certificate error)DNS for the scanner name not updated yet, or certificate expireddig +short scanner.internal.lynkoria.com must show the scanner's private IP; any apply renews the certificate in its last 30 days
Console requests blocked in the browser (CSP)API host not in apps/admin/public/_headersAdd it and deploy
Console: CORS errors or redirect loops to AccessNo Access cookie for the API hostVisit https://api.<domain>/v1/admin/me once, then the console
Console sign-in rejected with the right passwordAccess email differs from the console accountUse the same email in the IdP and admin-users.js create
Uploads fail in the browserProduct origin missing from allowed_originsAdd it to the environment's .tfvars and deploy
A plan shows changes nobody madeSomeone edited a managed resource in a dashboardApply to restore the code's version, or move the change into code

Local migration connection timeout

If pnpm db:migrate reports a connection timeout at CREATE SCHEMA IF NOT EXISTS "drizzle", check connectivity first: the query can fail before PostgreSQL executes any SQL. In root .env, make LYNKORIA_PG_PORT match the port in DATABASE_URL (and DATABASE_URL_DIRECT if set; it takes precedence for migrations). Likewise, match LYNKORIA_REDIS_PORT and REDIS_URL.

Run pnpm infra:up, then pnpm db:migrate. For direct Compose commands, include --env-file .env; without it, this nested Compose project may use default ports instead of your root configuration. Check published ports with docker compose --env-file .env -f infra/local/compose.yaml ps. Container health alone does not prove the host port is reachable. Recreating containers keeps their named data volumes; do not use down --volumes to fix connectivity.

Start with the request ID, API readiness, recent structured logs and relevant metrics. Do not paste tokens, service keys or full connection strings into tickets. Keep the failure’s environment/project ID and time window.

SymptomChecks and recovery
Local tests cannot connectConfirm Compose host ports and matching DATABASE_URL/REDIS_URL; LiveKit and attachments profiles must be running; test DB user needs CREATE DATABASE.
API fails startupRead variable names in validation errors; check all-or-none Spaces config, distinct pepper/master, PEM escapes, current kid vs previous keys, production TLS/CA/direct connection.
Migration failsUse direct migration-role connection, correct database, trusted source and CA; verify schema ownership/grants. Do not give runtime DDL rights to bypass it.
Service request 401/403Correct staging/live prefix, deployment environment, active project/environment, key expiry/revocation and required scope.
Console returns HTML/401Sign into API admin hostname, verify shared Access audience/team, exact console origin and active DB role. Check API Access OPTIONS bypass and browser cookie blocking.
Socket rejected/disconnectedExact Origin in global and project allowlists, messaging enabled, valid unexpired user JWT, active user/project, WebSocket support at ingress. Product callbacks must obtain fresh tokens.
503 during Redis failureRestore the private TLS endpoint/trusted sources/credentials. Rate limits fail closed; PostgreSQL data survives but request availability is degraded. Never expose Redis publicly as a workaround.
LiveKit WebSocket errorDNS-only media records, Caddy certificate/443, matching key reference, existing room, clock synchronization and token expiry.
Connected but no audio/videoDevice permissions, published tracks, viewer role, public ICE candidate, UDP range, TCP 7881, TURN 3478/TLS 443, firewall and mobile-network test.
Livestream slowOutbound bitrate/NIC, CPU/steal, simulcast, publishers, TURN share and load-generator limits. Measure before changing quotas.
Missing/duplicate messagesStable send ID, client history watermark and after_seq replay, active membership, commit errors. Do not manually edit sequence numbers; inspect for writers bypassing the service.
Presence stale25-second heartbeat/60-second lease, disconnected devices, manual/invisible status, sweeper lag and Redis health.
Upload deniedBoth attachment switches enabled, membership/uploader match, exact Spaces CORS, presigned URL expiry, actual file size/type and private bucket credentials.
Webhooks failingInbound raw body/key/signature and dedupe; outbound approved HTTPS origin, signature receiver, no redirects, dead letters and product idempotency. Retry after fixing the cause.
Usage delayed15-second flush for buffered counters; participant seconds appear on closure; inspect worker and reconciler health.
Docs page absentFrontmatter, docs/meta.json, case-sensitive filename and static build output. Regenerate MDX types, run docs tests/build and inspect broken-link diagnostics.

For bad releases follow rollback. For backup/recovery follow the restore drill before pointing traffic at recovered data. Keep an incident timeline and verify recovery from a real client, not only a health endpoint.

On this page