Skip to Content
LYNKORIADocs

Production recommendations

Security, scaling, backups, logging and maintenance practices for running Lynkoria after the first deployment.

These go beyond "it runs". Day-2 procedures (release, rollback, rotation, restore) are in operations; alert responses are in runbooks.

Security

  • Separate everything per environment. The platform stack gives staging and production their own VPC, databases, Valkey, bucket, Droplets, keys, secrets, Pages projects and state file. Consider separate DigitalOcean tokens per environment too.
  • Change infrastructure only through code. Dashboard edits are drift that the next apply reverts. Keep main protected and production behind a required reviewer.
  • Guard the state. The state bucket is versioned and state is encrypted; the passphrases and the bucket key are the keys to every generated secret. Restrict who can read the GitHub environments.
  • Least privilege. The runtime database role cannot change the schema; only the migration job holds migration credentials. The API's Spaces key is limited to one bucket. GitHub and Cloudflare tokens are scoped to what the workflow needs.
  • No public data stores. PostgreSQL and Valkey accept only the app (and VPC sources you need). Remove your admin IP after each maintenance task. The scanner listens only on the private IP.
  • Keep the origin lock and Access on. Without the origin lock, anyone can reach *.ondigitalocean.app directly and forge their IP. Access is a second layer in front of console passwords.
  • SSH. Admin IPs only, keys only. Consider a bastion or Tailscale-style VPN instead of a public SSH rule. Apply OS updates on the Droplets (unattended-upgrades for security patches) and plan media reboots in a window.
  • Secrets. Keep them in a secret manager; back up the pepper, webhook master, messaging keys and the scanner CA key with the database backups. Follow rotation.
  • Run the security checklist before real traffic. Review security findings and the Fidescript isolation decision (ADR 008) before sensitive traffic.
  • HSTS preload only after every subdomain, including staging, serves HTTPS (Cloudflare zone).

Scalability

ComponentHow it scalesBefore you scale
APIRaise api_instance_count in the environment's .tfvars; Socket.IO and workers coordinate through Valkey (ADR 021)Run the load tests; watch Valkey CPU
MediaVertical: change media_droplet_size (replaces the Droplet). One node only in v1Benchmark with lk load-test; outbound bandwidth decides cost
PostgreSQLVertical resize; transaction pool size (10 by default)Watch pool pressure and slow queries
ValkeyVertical resizeMemory with noeviction must never fill
ScannerVertical (clamd memory) or more hosts behind one URLWatch scan latency and failures
StaticCDN, nothing to do—

Usage counters are not billing-grade. Do not raise quotas without measurements.

Backups and recovery

  • Managed PostgreSQL has daily backups and point-in-time recovery; check the retention for your plan in the cluster's Backups tab.
  • Restore drill, quarterly and before risky migrations: fork/restore into a new cluster (doctl databases fork or the control panel), point a staging API at it with the restored secrets, test, and record how long it took. Never test by overwriting production.
  • Valkey holds no source-of-truth data; losing it drops presence and rate-limit state only.
  • Spaces: consider versioning or a periodic copy of the bucket to another region if attachments must survive operator error.
  • The LiveKit and Caddy configuration, LiveKit keys and the scanner CA are in code and encrypted state; the state bucket's versioning is their backup.

Logging and monitoring

  • Forward App Platform logs to a destination with retention you choose (step 10).
  • Keep LOG_LEVEL=info. Logs are JSON with x-request-id; secrets and message bodies are redacted.
  • Every alert in infra/monitoring/alerts.yml should reach a person; test the route after each change.
  • Watch media Droplet CPU steal and outbound transfer against your bandwidth allowance.

Maintenance calendar

FrequencyTask
Every releaseCI green, staging checklist, record previous digests, review migrations
WeeklyReview alerts and dead-lettered webhooks; check scanner signature age
MonthlyOS updates on Droplets (maintenance window for media); dependency and image updates
QuarterlyRestore drill; rollback drill; access review of console users and Cloudflare Access policy
Before expiryScanner certificate (397 days); Cloudflare and DigitalOcean API tokens
On LiveKit releasesFollow upgrading LiveKit on staging first

On this page