Operator runbook

How to stand up, run and recover a deployment. This file holds the standing procedures. What changed in a particular release is in DEPLOY_NOTES.md, and .env.example documents every variable. Where this file and the code disagree, the code wins. Fix this file.

The binding requirements come from §11A of the spec (Malaysian statutory compliance) and §15 (deployment). They have not yet been reviewed by a tax agent. That review must happen before any paying customer’s data lands.

---

1. Provisioning checklist: records stay in Malaysia

ITA 1967 s.82(8): records relating to a business in Malaysia must be kept and retained in Malaysia (spec §11A.1). Nothing in the code can enforce this, so it is checked here, when an environment is created and again whenever any item below changes. Record the date, the person who checked, and the provider and location of each item.

A backup is a record (§11A.1). An offshore backup bucket breaches s.82(8) just as an offshore primary does.

1.1 Where everything lives

Every item must be in a Malaysian datacenter, and the provider’s contract must say so. A provider that “may replicate for durability” to another region fails this check.

• The VPS that runs the api and worker containers.
• The Postgres data volume, including any block-storage volume, snapshot or replica attached to it.
• Database backups: every pg_dump / pg_basebackup output and every WAL archive used for point-in-time recovery (§15). The archive target (disk, NAS, S3-compatible bucket) is in Malaysia, and so is anything that copies it onward.
• The object store: MinIO or another S3-compatible store, on the same VPS or in the same datacenter (DEPLOY_NOTES, Plan 5, item 2). OBJECT_STORE_ENDPOINT points at it. The code has no default endpoint on purpose: an S3 SDK’s default is AWS’s global endpoint.
• The bucket named by OBJECT_STORE_BUCKET is on that store, with versioning and object lock (compliance mode, 8-year default retention) turned on before the first upload. Compliance, not governance: in governance mode anyone holding the bypass permission can delete a locked record early, so a leaked admin key could erase them; in compliance mode nobody can. It cannot be undone, so try the settings in governance mode with a 1-day retention first (object-lock research §3). MinIO accepts object lock only when the bucket is created (mc mb --with-lock). Check the mc syntax for your MinIO version. api and worker refuse to start if the bucket is missing, unless OBJECT_STORE_CREATE_BUCKET=true (local development only), so they never create an unlocked one.
• Bucket backups and replicas: any mc mirror, site replication or offsite copy of the bucket goes to storage in Malaysia.
• The disaster-recovery copy, if it is separate from the above.
• Key backups (§4): PAYOUT_DETAILS_KEY, JWT_KEYS, TOTP_ENCRYPTION_KEY. These are not records, but losing PAYOUT_DETAILS_KEY destroys records. Keep them in Malaysia as well so the recovery path never crosses a border.

1.2 No offshore processor

• OCR_PROVIDER is empty or none. No provider is built in, and any other value stops the api from booting (internal/kernel/config/ocr.go). A provider added later must process documents in Malaysia (spec §6.6, Ruling R35). Until one exists, OCR stays off.
• BNM_RATES_URL is either unset or https://api.bnm.gov.my. Only a bnm.gov.my host is accepted. This is an inbound fetch, not an outbound processor.
• No log shipper, APM agent, error tracker or managed backup service sends record data (request bodies, document bytes, database dumps) to a host outside Malaysia. Structured logs carry ids, not record text. Anything that ships dumps or objects is a backup target and falls under §1.1.

---

2. Required configuration

.env.example is the reference. It explains every variable and ships placeholders that are refused at boot. This section lists which binary needs which variable.

Variable	api	worker	migrate up/down/status	migrate provision, verify-audit	migrate reseal-payout-details	orgadmin create-org
DATABASE_URL (app role erp_app)	required	required		required	required	required
DATABASE_MIGRATE_URL (owner / superuser)			required		required
JWT_KEYS	required	required
PUBLIC_BASE_URL	required	required				required
PAYOUT_DETAILS_KEY	required				required
PAYOUT_DETAILS_KEY_PREVIOUS	during a rotation				during a rotation
WEBHOOK_SECRET_KEY	required	required
WEBHOOK_SECRET_KEY_PREVIOUS	during a rotation	during a rotation
OBJECT_STORE_ENDPOINT, _BUCKET, _ACCESS_KEY, _SECRET_KEY	required
OBJECT_STORE_REGION, _PRESIGN_TTL	optional
TRUSTED_PROXIES	set to your proxy	set to your proxy
TOTP_ENCRYPTION_KEY	optional; 32 bytes if set	optional
REDIS_URL	optional (cache only)	optional
BNM_RATES_URL		production: set
OCR_PROVIDER	empty or none
PORT	default 3000

api and worker both load configuration through config.Load, so the worker also refuses to boot without DATABASE_URL, JWT_KEYS and a valid PUBLIC_BASE_URL.

---

3. Retention: seven years, and restores that are actually rehearsed

ITA 1967 s.82A: records are kept at least seven years from the end of the year of assessment (spec §11A.2). Retention outranks erasure (§17.3, as amended).

What already enforces it:

• The app role cannot delete statutory rows. Migration 00015_retention_grants withholds DELETE from erp_app on every record-bearing table, and internal/kernel/db/retention_test.go fails the build if a table is on neither list.
• The ledger, the audit trail and the workflow decision log are append-only.
• The bucket’s object lock (§1.1) stops documents from being overwritten or deleted.

What the operator owns:

• Backups are kept for at least seven years after the end of the year of assessment they relate to. Do not add a lifecycle rule that expires backups sooner.
  • The object lock is eight years from upload. The statutory period runs seven years from the end of the year of assessment, so a document uploaded on the first day of a year needs just under eight years. A 7-year lock would expire early.
  • The application never deletes documents in any case. The lock guards against an operator or a compromised credential.
  • Backups follow the same rule: keep each one at least eight years from when it was taken, or seven years from the end of the latest year of assessment it covers, whichever is later.
• Never prune audit_log, and never restore it from an older backup than the rest of the database. Every period close records the audit chain head, so a trail cut short blocks the next close (DEPLOY_NOTES, Plan 4, item 5).
• No anonymization or erasure path exists yet. A tenant deletion request therefore cannot be executed. Record it and escalate it. The retention-gated path is built with tenant export and erasure (§17.3).

3.1 Restore test

Spec §15: “An ERP whose backups have never been restored does not have backups.”

Cadence: at least quarterly, after any change to the backup tooling or target, and before the first paying customer’s data lands. Record each test (date, backup used, result).

Each time:

1. Restore the latest base backup plus WAL to a scratch Postgres, in Malaysia, to a chosen point in time.
2. Re-apply the TEMPORARY revoke. It is a database-level privilege that pg_restore without --create does not carry (DEPLOY_NOTES, Plan 4, item 1):

   REVOKE TEMPORARY ON DATABASE <db> FROM PUBLIC;
   SELECT has_database_privilege('erp_app', current_database(), 'TEMPORARY');  -- must be false

3. Run migrate status against it. Every migration the source had applied must show as applied.
4. Verify each organization’s audit chain. The command exits non-zero and names the first broken row if the chain is broken:

   DATABASE_URL=<restored db, app role> migrate verify-audit --org <uuid>

5. Restore a sample of objects from the bucket backup, and check that a claim’s document downloads and matches its stored checksum.
6. Confirm that PAYOUT_DETAILS_KEY from the key backup opens a restored settlement snapshot (for example, view a settled claim through a scratch api). A backup whose key is lost is a backup of unreadable records.

---

4. Key custody: PAYOUT_DETAILS_KEY

The key encrypts members’ payout details and the masked values in every claim’s settlement snapshot. The snapshots are statutory records (DEPLOY_NOTES, Plan 6).

• Generate one per environment: echo "p1:$(openssl rand -base64 32)". Never reuse a kid, and never commit the key.
• Store it apart from the database backups. A dump alone yields no account number only while the key does not sit beside it. Use two custodians or two locations, both in Malaysia.
• Losing it loses every saved payout detail (members re-enter them) and every masked value in seven years of claim snapshots. That loss cannot be repaired.
• Only the api container and migrate reseal-payout-details read it. The worker, migrate up and orgadmin do not.
• Treat JWT_KEYS and TOTP_ENCRYPTION_KEY the same way (generate per environment, keep out of the database backup). Losing either one breaks sessions or second factors, not records.

4.1 Rotating it

Rotation uses one active key plus at most one previous key (Ruling P32). Full detail is in DEPLOY_NOTES, Plan 5 (“Rotating the payout-details key”).

1. Generate a key with a new kid. Deploy with PAYOUT_DETAILS_KEY=<new kid>:<secret> and PAYOUT_DETAILS_KEY_PREVIOUS=<old kid>:<secret>.
2. Re-encrypt every member profile and claim snapshot, with the same environment plus DATABASE_MIGRATE_URL:

   migrate reseal-payout-details            # every organization; safe to interrupt and re-run
   migrate reseal-payout-details --check    # exits 0 only when nothing remains under an old key

   --org <uuid> limits the run to one organization, and --batch N sets the batch size.
3. Only after --check exits 0, clear PAYOUT_DETAILS_KEY_PREVIOUS and redeploy. Keep the old key in custody until the next restore test has proved that nothing needs it.

Never drop the previous key before --check passes. Anything still sealed under it becomes unreadable forever.

4.2 WEBHOOK_SECRET_KEY

Seals each webhook endpoint’s signing secret (public API design D13). It is a different key from PAYOUT_DETAILS_KEY, kept in the same custody (apart from the database backups, in Malaysia). Losing it does not lose records: every endpoint’s secret becomes unreadable, so each endpoint must be rotated (a new secret issued to its integrator) before deliveries resume. Generate with echo "w1:$(openssl rand -base64 32)"; rotate like 4.1 with a new kid and WEBHOOK_SECRET_KEY_PREVIOUS. There is no bulk re-seal command: a secret moves to the active key only when its endpoint’s secret is rotated, so keep _PREVIOUS until no endpoint still opens under it.

---

5. Migrations

Migrations run as a separate job under the privileged role (DATABASE_MIGRATE_URL), never from the api on boot (spec §15). Take a backup first.

migrate status    # what is applied, and in which order
migrate up        # apply everything not yet recorded, whatever its number

• up runs with allow-missing. Plans own migration number ranges and merge in any order, so a database already at 00090 still applies a newly merged 00034 (DEPLOY_NOTES, “Wave 3 base”). goose applies each file in its own transaction. A failed file leaves the database at the previous state with nothing half-applied, and the next up retries it.
• Some migrations must run as the database owner (00052 revokes TEMPORARY and fails if the revoke does not take effect). Some fail on purpose when they find bad data (00021 refuses cross-tenant references). Read that release’s DEPLOY_NOTES entry before the window.

5.1 Rolling back

Rolling back a schema change is an emergency tool, not a routine step. The ledger and the audit trail are append-only, so the fix for a bad release is usually a forward migration.

1. Run migrate status first, every time. After an out-of-order merge, the most recently applied migration need not be the highest-numbered one.
2. migrate down rolls back the most recently applied migration, one at a time. Check status again after each step.
3. Do not use migrate down-to N after an out-of-order apply. down-to stops as soon as the most recently applied version is at or below N. If 00034 was applied after 00090, migrate down-to 60 does nothing at all, even though 00090 is above 60, and it reports no error. Use repeated migrate down with status in between instead.
4. Never run reset against a deployment that holds records.

---

6. Organizations: creating them and re-running provisioning

6.1 Creating an organization

There is no public sign-up. The operator creates each organization with orgadmin (go build -o bin/orgadmin ./cmd/orgadmin, because make build does not build it):

orgadmin create-org \
  -name "Contoh Sdn Bhd" -owner-email owner@contoh.com.my -owner-name "Nur Aisyah binti Ahmad"

Three fields only. The organization starts setup pending: it has no company profile, chart or journals. The owner accepts the invitation, signs in and completes the setup wizard themselves (country, base currency, fiscal year start, account code format and the company code in it, which never change afterwards). Until they press Finish, every screen sends them to the wizard and the API answers setup_required. See docs/superpowers/specs/2026-10-09-two-phase-onboarding-design.md.

It prints organization <uuid> and invitation <link>. The link is a credential, and it expires after seven days (DEPLOY_NOTES, Plan 2b, item 3).

6.2 After a release that adds workflow kinds or provisioning steps

Provisioning (spec §20.2) is ordered and idempotent: a step that is already done is skipped. orgadmin create-org runs it for a new organization. Nothing runs it for existing organizations on deploy.

So when a release registers a new seeded workflow kind (step 60 workflow_definitions, for example Plan 7’s advance and Plan 8’s petty_cash_spend) or adds a provisioning step, every existing organization lacks it until someone re-runs provisioning for that organization. Until then, starting a document of the new kind fails (workflow.Start returns ErrNoDefinition), and the organization’s readiness report shows the step as missing.

For each existing organization, do one of the following:

• Through the API, as a member holding identity.org.administer:

  GET  /v1/org/readiness          # which steps are missing
  POST /v1/org/provision          # run them; returns the report after the run

  The body is optional. Send {"owner_member_id": "<uuid>"} only if the report shows step org_structure as missing. Without it, a run that still has to seat the Owner is refused with a 422. An organization that was provisioned before needs no body.
• From the CLI, with DATABASE_URL set to the app role (RLS applies exactly as it does for the api):

  migrate provision --org <org uuid> --owner-member <Owner's organization_member uuid>

  --owner-member is always required here, even when the Owner is already seated. The command prints each step as done, MISSING or FAILED and runs in one transaction.

orgadmin has no re-provision command; it only creates organizations. There is no all-organizations batch yet, so loop over the organizations yourself. A release that needs this says so in its DEPLOY_NOTES entry.

7. Operator console: what a red or amber cell means

Every red or amber cell on the console names a code and the section below. The console shows counts and pass/fail only; finding the records behind a count means working inside the organization (with its permission) or on the database, never in the console.

7.1 Organizations

• org_suspended (amber): an operator suspended the organization (Actions, A5). Its members cannot sign in or change anything; its jobs still run and nothing is deleted. The organization’s audit trail records who did it. Reactivate from its page once the reason is resolved.
• org_setup_pending (amber): the organization was created (orgadmin create-org or the console) and its owner has not finished company setup, so it has no company profile, chart or journals yet. Nothing is wrong. Background jobs skip it, the integrity checks do not run for it, and every screen sends its owner to the setup wizard. If it stays pending for weeks, check the owner received the invitation (it expires after seven days) and reissue it. Do not delete it: its creation is already in the audit trail.

7.2 Background jobs

• job_failed: a job is retryable (River will try again) or discarded (it gave up). Read the worker’s logs for the job kind. Once the cause is fixed, retry it from Actions → Failed jobs (A4); retrying before the fix only fails again.
• job_stuck: a job has been running for over 30 minutes. Check that the worker is up (Health) and read its logs. A worker that died mid-job leaves it running until River’s rescuer picks it up.

7.3 Ledger integrity

Nightly (and on “Run checks now”) the worker runs every reconciliation check in the organization’s own transaction and records pass/fail and a finding count.

• trial_balance: total debits and credits differ.
• entry_balance: a journal entry’s lines do not balance.
• entry_period: an entry is dated outside its fiscal period.
• reversal_mirror: a reversal does not exactly mirror the entry it reverses.
• audit_chain: the organization’s audit log hash chain is broken, meaning a row was changed or removed outside the application. Treat it as an incident: stop, preserve the database, and do not “repair” the chain.
• stock_valuation: an inventory account’s ledger balance differs from the value of the stock that names it. Usually a manual journal on an inventory account the organization chose itself (the provisioned Inventory account refuses manual lines). The organization’s accountant reverses that journal, or posts a stock adjustment for the difference.
• asset_register_gl: a fixed-asset class account’s ledger balance differs from the register. Per account, it compares the cost of assets not disposed with the class’s cost account, and their accumulated depreciation with minus the class’s accumulated-depreciation account. Usually a cost posted to a class cost account but never registered (Fixed assets → Register asset), a registered source later voided or reversed, or a depreciation run’s journal reversed by hand. The organization’s accountant registers the asset, or reverses or re-posts the stray entry; a period close is refused while the check fails unless it is closed unreconciled.
• run_failed: the run itself failed, so nothing was checked. Read the worker’s logs.
• not_checked (amber): no run has been recorded yet, for example a new organization before its first night. Run the checks now; absence of a result is never a pass.

The ledger is append-only, so every fix to the first four is a reversal and re-post by the organization’s accountant, never an UPDATE.

7.4 Compliance flags

• period_overdue: fiscal periods are still open after their end date. Remind the organization to close them; closing is theirs to do.
• einvoice_missing: receipt lines are recorded without a MyInvois e-Invoice. The organization should obtain or issue the e-Invoice; the count falls as they do.

7.5 Backups

• backup_stale: the last successful backup is over 26 hours old. Read the backup script’s log and run it by hand (§3).
• restore_test_stale: the last restore test is over 100 days old. Run one (§3.1).
• backup_status_unknown (amber): the status file is not configured, missing or malformed, so nothing is known. Unknown is never a pass; fix the backup script’s output.

7.6 Security

• token_replay: a refresh token was used twice, so the session family was revoked. One can be a flaky client; several suggest a stolen token. Check the organization’s audit trail and tell its administrator.
• roles_manage_changed: the org.roles.manage permission was granted or removed in the last 30 days. Confirm with the organization that the change was intended.

---

8. Public API and webhooks

Design: docs/superpowers/specs/2026-10-03-public-api-webhooks-design.md. An organization’s public API is off until an administrator turns it on (Integrations screen), and every key and webhook secret is shown once. The platform operator can also suspend an organization’s public API (public_api_setting.operator_suspended).

8.1 A leaked API key

A leaked key is a potential personal-data incident for the customer: tell their administrator.

1. Revoke it, do not rotate it (Integrations, API keys, Revoke). It stops authenticating at once. Rotation keeps the old key working for the overlap and is for planned changes only.
2. Evidence: last_used_at and last_used_ip on the key (the last use only, never a history), and every audited write the key made: audit_log joined to audit_log_credential on (org_id, seq = audit_seq) for that credential_id names “member M via key K”. The access log carries credential_id for every request.
3. To stop the whole surface at once: turn the organization’s public API off (Integrations), or set public_api_setting.operator_suspended = true for it.
4. Find the key in the wild: keys start mck_, which GitHub and trufflehog secret scanners can be told to match.

8.2 A leaked webhook secret

Rotate the endpoint’s secret and tick “stop the old secret at once” (no overlap). Deliveries signed with the leaked secret can be forged until then; receivers dedupe on webhook-id and reject a timestamp more than 5 minutes old, which limits replay but not forgery. If the secret key itself (WEBHOOK_SECRET_KEY) is suspected, section 4.2.

8.3 A hostile webhook URL: erp_webhook_attempts_total{outcome="blocked_address"} above 0

An administrator registered a URL that resolves to an internal or reserved address (the guard refused the connection; nothing was sent). Find it: webhook_delivery rows with last_error_class = 'blocked_address' name the endpoint; disable that endpoint, then tell the organization. A rise right after a DNS change is usually a receiver that moved onto a private address. The same guard runs when an endpoint is saved and at every attempt.

8.4 Replaying a dead letter

A delivery becomes dead after nine attempts over about 45 hours (or on a 410). From the endpoint’s page, Replay (one delivery) or Replay all dead (since a date, at most 1000) puts it back to pending with the SAME event id, so a receiver that already processed it dedupes. An endpoint that was disabled for failures can be resumed with “also send what it missed”. A dead delivery stays replayable for 90 days; a succeeded one is pruned at 30.

8.5 Webhook deliveries are stuck

erp_webhook_oldest_pending_seconds above 900 for 10 minutes: due deliveries are not being attempted. Check that a worker serving the webhook queue is up (WORKER_QUEUES includes webhook:2; compose worker), then the job-count gauge for a growing available backlog. The reconciler re-enqueues a due delivery that has no live job once it is two minutes overdue.