Production recommendations
Security, scaling, backups, logging and maintenance practices for running Lynkoria after the first deployment.
These go beyond "it runs". Day-2 procedures (release, rollback, rotation, restore) are in operations; alert responses are in runbooks.
Security
- Separate everything per environment. The platform stack gives staging and production their own VPC, databases, Valkey, bucket, Droplets, keys, secrets, Pages projects and state file. Consider separate DigitalOcean tokens per environment too.
- Change infrastructure only through code. Dashboard edits are drift that the next apply reverts. Keep
mainprotected and production behind a required reviewer. - Guard the state. The state bucket is versioned and state is encrypted; the passphrases and the bucket key are the keys to every generated secret. Restrict who can read the GitHub environments.
- Least privilege. The runtime database role cannot change the schema; only the migration job holds migration credentials. The API's Spaces key is limited to one bucket. GitHub and Cloudflare tokens are scoped to what the workflow needs.
- No public data stores. PostgreSQL and Valkey accept only the app (and VPC sources you need). Remove your admin IP after each maintenance task. The scanner listens only on the private IP.
- Keep the origin lock and Access on. Without the origin lock, anyone can reach
*.ondigitalocean.appdirectly and forge their IP. Access is a second layer in front of console passwords. - SSH. Admin IPs only, keys only. Consider a bastion or Tailscale-style VPN instead of a public SSH rule. Apply OS updates on the Droplets (
unattended-upgradesfor security patches) and plan media reboots in a window. - Secrets. Keep them in a secret manager; back up the pepper, webhook master, messaging keys and the scanner CA key with the database backups. Follow rotation.
- Run the security checklist before real traffic. Review security findings and the Fidescript isolation decision (ADR 008) before sensitive traffic.
- HSTS preload only after every subdomain, including staging, serves HTTPS (Cloudflare zone).
Scalability
| Component | How it scales | Before you scale |
|---|---|---|
| API | Raise api_instance_count in the environment's .tfvars; Socket.IO and workers coordinate through Valkey (ADR 021) | Run the load tests; watch Valkey CPU |
| Media | Vertical: change media_droplet_size (replaces the Droplet). One node only in v1 | Benchmark with lk load-test; outbound bandwidth decides cost |
| PostgreSQL | Vertical resize; transaction pool size (10 by default) | Watch pool pressure and slow queries |
| Valkey | Vertical resize | Memory with noeviction must never fill |
| Scanner | Vertical (clamd memory) or more hosts behind one URL | Watch scan latency and failures |
| Static | CDN, nothing to do | — |
Usage counters are not billing-grade. Do not raise quotas without measurements.
Backups and recovery
- Managed PostgreSQL has daily backups and point-in-time recovery; check the retention for your plan in the cluster's Backups tab.
- Restore drill, quarterly and before risky migrations: fork/restore into a new cluster (
doctl databases forkor the control panel), point a staging API at it with the restored secrets, test, and record how long it took. Never test by overwriting production. - Valkey holds no source-of-truth data; losing it drops presence and rate-limit state only.
- Spaces: consider versioning or a periodic copy of the bucket to another region if attachments must survive operator error.
- The LiveKit and Caddy configuration, LiveKit keys and the scanner CA are in code and encrypted state; the state bucket's versioning is their backup.
Logging and monitoring
- Forward App Platform logs to a destination with retention you choose (step 10).
- Keep
LOG_LEVEL=info. Logs are JSON withx-request-id; secrets and message bodies are redacted. - Every alert in
infra/monitoring/alerts.ymlshould reach a person; test the route after each change. - Watch media Droplet CPU steal and outbound transfer against your bandwidth allowance.
Maintenance calendar
| Frequency | Task |
|---|---|
| Every release | CI green, staging checklist, record previous digests, review migrations |
| Weekly | Review alerts and dead-lettered webhooks; check scanner signature age |
| Monthly | OS updates on Droplets (maintenance window for media); dependency and image updates |
| Quarterly | Restore drill; rollback drill; access review of console users and Cloudflare Access policy |
| Before expiry | Scanner certificate (397 days); Cloudflare and DigitalOcean API tokens |
| On LiveKit releases | Follow upgrading LiveKit on staging first |