Operator console: progress and next steps

As of: 2026-09-29 (first written 2026-09-25 @ 7de0036) Design: 2026-09-25-operator-console-design.md Roadmap: Phase 2, “Operator console”

The console is built (Stages 0–4; P1–P11, P14, P15), merged in PR #8. Still open: P12 support access (A7), P13 tenant export (A6), and the server side (§4.1; a test deployment is in deploy/vps). The rest of this file is kept as the record of how it started. Groundwork from §5: version stamping (item 1), /metrics (item 3), the object lock decision (items 5 and 6), and two storage fixes from the object-lock research. This file records what the codebase already provides, what is missing, and what can start now. It separates infrastructure (set up on the server) from programming (code in this repo).

---

1. What already exists and can be reused

Console item	What exists	Where	Reuse
V2 health	/v1/health and /v1/ready, with a 2-second database timeout	internal/kernel/httpx/server.go:107	As is
V4 integrity	10 built-in checks in one reconciliation set, plus an on-demand route per organization	internal/platform/accounting/reconcile/builtin.go, internal/modules/accounting/routes.go:82	The worker calls Set.Run for each organization
A3 audit chain	audit.VerifyChain and migrate verify-audit	internal/kernel/audit/audit.go:422, cmd/migrate/verifyaudit.go	As is
A1 create organization	orgadmin create-org	cmd/orgadmin	Wrap it
A2 provisioning	migrate provision	cmd/migrate	Wrap it
V7 failed logins	login_failed is written to the audit log; logins are throttled	internal/modules/identity/handlers.go:429, internal/platform/identity/throttle	Count them
P5 operator sign-in	TOTP already exists for users, keyed by TOTP_ENCRYPTION_KEY	internal/platform/identity/totp	Reuse it; the design’s separate OPERATOR_TOTP_KEY is not needed
V3 jobs, P8 checks	River worker client and the reports queue	cmd/worker/exports.go	One more AddWorker per job kind
V6 backups	Residency checklist and restore-test procedure, on paper only	docs/RUNBOOK.md §1.1, §3.1	The console shows the recorded results

2. What is missing

Gap	Evidence	Needed for
No metrics	Done (P1): /metrics on a private listener in api (API_METRICS_ADDR) and worker (WORKER_METRICS_ADDR); request latency by route template, pool gauges, River job counts; drift test internal/app/metrics_test.go	V2, Stage 1
Version not stamped	Done: make build stamps git describe into api and worker (-X main.version); /v1/health reports it	V8
Nightly reconciliation not scheduled Done: ops_reconcile	The worker runs only the session-prune ticker and exports. Already an open roadmap item (Plan 9, Ruling R30)	V4
No organization status Done: migration 00240	organization holds only id, org_id, name, created_at	A5 suspend
No ops schema or operator roles Done: 00230–00233	Only erp_app and the migration owner exist	Design §3.3
No deployment files (a test stack is in deploy/vps; no backup script)	Only web/Dockerfile and web/nginx.conf; no compose file, unit files or backup script	I1–I15

3. Object lock mode — decided: compliance (2026-09-28)

RUNBOOK §1.1 specifies MinIO object lock in governance mode. Hosting advice given alongside this design said compliance mode.

• Governance: a user with the bypass permission can delete a locked object early.
• Compliance: nobody can, including root, until the lock expires.

Compliance mode fits a seven-year statutory record better. Decided: compliance, tried first in governance mode with a 1-day retention. RUNBOOK §1.1 and DEPLOY_NOTES are updated.

Two code changes came with it:

• The thumbnail upload sends Content-MD5; S3 rejects a PUT into a locked bucket without an integrity header (s3store.Put, TestPutSendsContentMD5).
• api and worker refuse to start against a missing bucket instead of creating an unlocked one, unless OBJECT_STORE_CREATE_BUCKET=true (local development; scripts/dev-stack.sh sets it).

---

4. Work split: infrastructure vs programming

4.1 Infrastructure (set up on the VPS, no ERP code)

#	Item	What to set up	Design ref
I1	Malaysian VPS	Sized for api, worker, Postgres, MinIO, Prometheus, Grafana	§1
I2	Private network	VPN, Tailscale or SSH tunnel; the console port is not on the public proxy	§3.4
I3	Firewall	Public: 443 only. Private: console port and /metrics. Postgres never exposed	§3.2, §3.6
I4	TLS certificates	HTTPS for the public site and the console, auto-renewed	V7
I5	Postgres storage	Data volume, WAL archiving on	V6
I6	Database backups	Kept 8+ years, locked against early deletion, copy at a second Malaysian site	V6, RUNBOOK §3
I7	Document storage	MinIO with object lock (see §3 above), bucket backups	RUNBOOK §1.1
I8	Restore test	Quarterly, recorded	V6, RUNBOOK §3.1
I9	Prometheus	Scrapes /metrics over the private network	Stage 1
I10	Grafana	Self-hosted, service-health dashboard	V2
I11	Host metrics	node_exporter, postgres_exporter	V2
I12	Alerts	Disk nearly full, backup failed, certificate expiring, app down	V2, V6
I13	Residency record	Server, disks, backups, logs, bucket all in Malaysia; OCR off	V6, §1
I14	Backup status file	The backup script writes “last success at” for the console to read	V6
I15	Secrets	OPERATOR_DATABASE_URL and friends, readable only by the service	§3.8

4.2 Programming (code in this repo)

#	Item	Where	Design ref
P1	/metrics, route-template labels only	internal/kernel/httpx, cmd/api, cmd/worker	Stage 1
P2	Roles erp_ops_owner, erp_operator	migrations 00230+	§3.3
P3	ops.operator_account, ops.operator_session, hash-chained ops.operator_audit	migrations	§3.4
P4	cmd/operator, server-rendered, private listen address	cmd/operator, internal/operator	§3.2
P5	Operator sign-in: password + TOTP (reuse identity/totp)	internal/operator	Stage 2
P6	Operator action log, mirrored to the organization’s audit_log where it affects them	internal/operator, kernel/audit	§2.2
P7	ops.org_overview, ops.job_summary, ops.compliance_flags, ops.security_summary	migrations	V1, V3, V5, V7
P8	Nightly reconciliation job writing ops.check_result	cmd/worker	V4, §3.5
P9	Pages for V1–V8	internal/operator	V1–V8
P10	Actions A1–A4	internal/operator	A1–A4
P11	organization.status, enforced at sign-in and on writes	migration, Authenticate	A5
P12	Support access	API, web/, internal/operator	A7, §3.7
P13	Tenant export	with spec §17.3	A6
P14	The six drift tests	tests	§6
P15	New env vars in .env.example	repo root	§3.8

4.3 Items that need both

Feature	Infrastructure	Programming
Service health (V2)	I9, I10, I11	P1
Backups (V6)	I6, I8, I14	P9 reads the status file
Console stays private	I2, I3	P4 binds to a private address
Security (V7)	I4	P7

---

5. What can start now

None of these is blocked by Phase 2 or by the Stage 0 decisions.

#	Item	Type	Size	Why now
1	Stamp the build version into api and worker Done	Code	Tiny	Every later stage uses it
2	Nightly reconciliation as a River job, pass/fail per organization Done (B5, ops_reconcile)	Code	Small	Closes R30 and is P8
3	/metrics on api and worker with its no-tenant-label drift test Done. Prometheus and Grafana themselves are infrastructure (I9–I11)	Code	Small	Stage 1
4	Deployment files: compose for api, worker, Postgres, MinIO, Prometheus, Grafana; backup script with status file	Infrastructure as code	Medium	The repo has none; needs the host decision (VPS or AWS)
5	Decide the object lock mode; update RUNBOOK §1.1 Done: compliance	Docs	Tiny	§3 above
6	Update the design doc: reuse identity/totp, drop OPERATOR_TOTP_KEY, link R30 Done	Docs	Tiny	Keeps the design accurate
7	Stage 0 decisions D1, D2, D4, D5 decided 2026-09-28; D3 (the private network path) is still open	Decision	—	Blocks Stage 2 onward

Suggested order: 5 and 6 (docs) → 1 and 2 on a branch → 3 → 7 → 4 once the host is chosen.