Step 9: Operate
Change infrastructure by pull request, release and roll back, rotate generated secrets, run OpenTofu locally and import or remove resources.
Everything in infra/opentofu/ is the source of truth. Changes made in the DigitalOcean or Cloudflare dashboards are drift: the next apply reverts them. Make changes in code.
Change infrastructure
- Edit
infra/opentofu/on a branch and runpnpm infra:tofu. - Open a pull request. The Infrastructure workflow posts the staging
tofu planin its log. Read it: look formust be replacedanddestroy. - Merge. CI passes, then Deploy applies staging.
- Run Deploy for production when staging looks right.
| Change | Effect |
|---|---|
| App env, instance size or count | App Platform redeploys the API (rolling, no downtime) |
| LiveKit configuration (keys, hosts) | Media Droplet replaced (new one first, DNS follows); active calls end |
| Scanner image, compose file, certificate | Scanner Droplet replaced; uploads stay quarantined for a few minutes |
| Database size | Managed resize; follow DigitalOcean's notes on downtime for the plan |
| Hostnames | DNS, Access, Pages domains and the app domain change together; re-check certificates |
Resources holding data (postgres cluster, attachments bucket, pepper, webhook master) have prevent_destroy: OpenTofu refuses any plan that would delete them.
Release and roll back
A release is a merge to main (staging) and an approved Deploy run (production). The workflow deploys the API by image digest; migrations run in the PRE_DEPLOY job before traffic moves.
To roll back the API, find the previous digest (the previous Deploy run's images job log, or doctl registry repository list-tags lynkoria-api), then Deploy → Run workflow with api_image_digest set to it. Image rollback does not undo migrations; if the old image cannot work with the new schema, ship a forward fix (operations). For the console and docs, use Rollback on an earlier deployment in the Pages project.
Run OpenTofu locally
For troubleshooting, imports or one-off operations, run the same stacks from your machine with the same values the workflow uses:
for v in DIGITALOCEAN_ACCESS_TOKEN SPACES_ACCESS_KEY_ID SPACES_SECRET_ACCESS_KEY CLOUDFLARE_API_TOKEN TF_VAR_state_passphrase TF_VAR_zone_state_passphrase; do
read -rsp "$v: " "$v"; echo; export "$v"
done
export TOFU_STATE_BUCKET=YOUR_STATE_BUCKET
export TF_VAR_cloudflare_account_id=… TF_VAR_cloudflare_zone_id=… TF_VAR_cf_access_team_domain=…
export TF_VAR_access_allowed_emails='["ana@example.com"]' TF_VAR_owner_email=ana@example.com
export TF_VAR_docr_registry=… TF_VAR_ssh_key_fingerprints='["…"]' TF_VAR_admin_ips='["203.0.113.10/32"]'
infra/opentofu/run.sh staging output
infra/opentofu/run.sh staging plan -var "api_image_digest=$(infra/opentofu/run.sh staging output -raw api_image_digest)" \
-var "scanner_image_digest=$(infra/opentofu/run.sh staging output -raw scanner_image_digest)"Run this in bash: the loop asks for the six secrets without echoing them or writing them to history. Applying locally needs doctl, psql and curl (for the grants step). Do not apply locally while a workflow run is in progress: the workflows serialize their own runs, not yours.
Rotate secrets
Generated secrets rotate by replacing their resource. Read rotation first; some need a careful order.
| Secret | Command (-replace= on a plan/apply) | Notes |
|---|---|---|
| Scanner token | random_password.scanner_token | Scanner Droplet and API update in one apply; brief upload failures |
| LiveKit key for one product | 'random_password.livekit["vibelynk:production"]' | Media Droplet replaced; calls end |
| Messaging signing key | tls_private_key.messaging | New MESSAGING_JWT_KID. Tokens signed with the old key fail until clients refresh (up to 10 minutes) |
| Scanner certificate | automatic within 30 days of expiry on any apply | Or tls_locally_signed_cert.scanner |
| Origin-lock secret | zone stack: 'random_password.origin_secret["staging"]', then Deploy staging | Requests get 403 between the zone apply and the app deploy |
| Pepper, webhook master | protected (prevent_destroy) | Emergency only; replacing them invalidates every service key and webhook secret |
| Database passwords, Spaces key | digitalocean_database_user.app, digitalocean_spaces_key.api | App redeploys with the new value |
Example, locally: infra/opentofu/run.sh staging apply -replace=random_password.scanner_token -var api_image_digest=… -var scanner_image_digest=….
To change a state passphrase, follow OpenTofu's key rollover with a fallback block; never just change the secret.
Import something made by hand
If a resource already exists (for example a Cloudflare ruleset made in the dashboard), import it instead of deleting it:
infra/opentofu/run.sh zone import cloudflare_ruleset.cache zones/YOUR_ZONE_ID/YOUR_RULESET_IDEach resource's import ID format is in its provider documentation (DigitalOcean, Cloudflare).
Tear down an environment
destroy stops at the protected resources. To really delete staging, remove the prevent_destroy lines in a branch, run infra/opentofu/run.sh staging destroy locally, and expect to empty the attachments bucket first. Back up the database before you do.