Skip to Content
LYNKORIADocs

Step 9: Operate

Change infrastructure by pull request, release and roll back, rotate generated secrets, run OpenTofu locally and import or remove resources.

Everything in infra/opentofu/ is the source of truth. Changes made in the DigitalOcean or Cloudflare dashboards are drift: the next apply reverts them. Make changes in code.

Change infrastructure

  1. Edit infra/opentofu/ on a branch and run pnpm infra:tofu.
  2. Open a pull request. The Infrastructure workflow posts the staging tofu plan in its log. Read it: look for must be replaced and destroy.
  3. Merge. CI passes, then Deploy applies staging.
  4. Run Deploy for production when staging looks right.
ChangeEffect
App env, instance size or countApp Platform redeploys the API (rolling, no downtime)
LiveKit configuration (keys, hosts)Media Droplet replaced (new one first, DNS follows); active calls end
Scanner image, compose file, certificateScanner Droplet replaced; uploads stay quarantined for a few minutes
Database sizeManaged resize; follow DigitalOcean's notes on downtime for the plan
HostnamesDNS, Access, Pages domains and the app domain change together; re-check certificates

Resources holding data (postgres cluster, attachments bucket, pepper, webhook master) have prevent_destroy: OpenTofu refuses any plan that would delete them.

Release and roll back

A release is a merge to main (staging) and an approved Deploy run (production). The workflow deploys the API by image digest; migrations run in the PRE_DEPLOY job before traffic moves.

To roll back the API, find the previous digest (the previous Deploy run's images job log, or doctl registry repository list-tags lynkoria-api), then Deploy → Run workflow with api_image_digest set to it. Image rollback does not undo migrations; if the old image cannot work with the new schema, ship a forward fix (operations). For the console and docs, use Rollback on an earlier deployment in the Pages project.

Run OpenTofu locally

For troubleshooting, imports or one-off operations, run the same stacks from your machine with the same values the workflow uses:

for v in DIGITALOCEAN_ACCESS_TOKEN SPACES_ACCESS_KEY_ID SPACES_SECRET_ACCESS_KEY CLOUDFLARE_API_TOKEN TF_VAR_state_passphrase TF_VAR_zone_state_passphrase; do
  read -rsp "$v: " "$v"; echo; export "$v"
done
export TOFU_STATE_BUCKET=YOUR_STATE_BUCKET
export TF_VAR_cloudflare_account_id=… TF_VAR_cloudflare_zone_id=… TF_VAR_cf_access_team_domain=…
export TF_VAR_access_allowed_emails='["ana@example.com"]' TF_VAR_owner_email=ana@example.com
export TF_VAR_docr_registry=… TF_VAR_ssh_key_fingerprints='["…"]' TF_VAR_admin_ips='["203.0.113.10/32"]'
infra/opentofu/run.sh staging output
infra/opentofu/run.sh staging plan -var "api_image_digest=$(infra/opentofu/run.sh staging output -raw api_image_digest)" \
  -var "scanner_image_digest=$(infra/opentofu/run.sh staging output -raw scanner_image_digest)"

Run this in bash: the loop asks for the six secrets without echoing them or writing them to history. Applying locally needs doctl, psql and curl (for the grants step). Do not apply locally while a workflow run is in progress: the workflows serialize their own runs, not yours.

Rotate secrets

Generated secrets rotate by replacing their resource. Read rotation first; some need a careful order.

SecretCommand (-replace= on a plan/apply)Notes
Scanner tokenrandom_password.scanner_tokenScanner Droplet and API update in one apply; brief upload failures
LiveKit key for one product'random_password.livekit["vibelynk:production"]'Media Droplet replaced; calls end
Messaging signing keytls_private_key.messagingNew MESSAGING_JWT_KID. Tokens signed with the old key fail until clients refresh (up to 10 minutes)
Scanner certificateautomatic within 30 days of expiry on any applyOr tls_locally_signed_cert.scanner
Origin-lock secretzone stack: 'random_password.origin_secret["staging"]', then Deploy stagingRequests get 403 between the zone apply and the app deploy
Pepper, webhook masterprotected (prevent_destroy)Emergency only; replacing them invalidates every service key and webhook secret
Database passwords, Spaces keydigitalocean_database_user.app, digitalocean_spaces_key.apiApp redeploys with the new value

Example, locally: infra/opentofu/run.sh staging apply -replace=random_password.scanner_token -var api_image_digest=… -var scanner_image_digest=….

To change a state passphrase, follow OpenTofu's key rollover with a fallback block; never just change the secret.

Import something made by hand

If a resource already exists (for example a Cloudflare ruleset made in the dashboard), import it instead of deleting it:

infra/opentofu/run.sh zone import cloudflare_ruleset.cache zones/YOUR_ZONE_ID/YOUR_RULESET_ID

Each resource's import ID format is in its provider documentation (DigitalOcean, Cloudflare).

Tear down an environment

destroy stops at the protected resources. To really delete staging, remove the prevent_destroy lines in a branch, run infra/opentofu/run.sh staging destroy locally, and expect to empty the attachments bucket first. Back up the database before you do.

On this page