Operator console — guide, architecture and roadmap

Status: Stages 0–4 built (plan 2026-09-28-operator-console.md), except the network-path decision (D3) and the server side of Stage 1. Stage 5 and A6 are not built. Date: 2026-09-25 Audience: whoever runs the service for paying organizations (the operator), and whoever builds this console.

This is the screen the ERP provider uses to run the service for its customers. It is not a screen for a customer’s management. It answers “is the service healthy, and is each organization’s data intact and compliant?” It does not answer “what is in their books?”

---

1. Why it exists, and what it must not become

Today every operator task is a command-line call with a privileged database URL (orgadmin create-org, migrate provision, migrate verify-audit, RUNBOOK §6). That works for one operator and a few organizations. It does not scale, and nothing shows cross-tenant health in one place.

The console must stay inside four lines. Each one drives a design decision below.

Constraint	Source	Consequence for the console
The provider is a data processor. Each organization is the data user of its own records	PDPA 2010, 2024 amendments	The console shows counts and pass/fail, never record contents, names or bank details
Records and backups stay in Malaysia	ITA s.82(8), RUNBOOK §1	Hosted on the same Malaysian VPS. External monitoring receives metrics only
Records are kept for seven years, and the ledger is append-only	ITA s.82A, spec §11A	No delete and no edit actions. “Suspend” never removes data
Nobody grants themselves authority	AGENTS.md invariant	An operator can never grant themselves access to an organization’s data. The organization grants support access, and it is recorded in their audit log

---

2. What the console contains

2.1 Views (read-only)

#	Area	Shows	Source	Customer data?
V1	Organizations	name, status, created, member count, last activity, storage used	organization, membership counts	Account details only
V2	Service health	/v1/health, /v1/ready, error rate, latency, DB connections	Probes exist; metrics are new	No
V3	Background jobs	queued / running / failed / stuck River jobs per org, by kind	River tables (kernel/jobs)	Job kind and state only
V4	Ledger integrity	pass/fail per built-in check: audit_chain, trial_balance, entry_balance, entry_period, reversal_mirror, expense_claim_accrual, expense_claim_ap, float_integrity, float_ledger, float_commitment	platform/accounting/reconcile, audit.VerifyChain	Pass/fail only
V5	Compliance flags	periods open past their end; count of claim lines missing supplier TIN or e-Invoice UUID	fiscal periods, claim tax columns	Counts only
V6	Backups and residency	last good backup, last restore test (RUNBOOK §3.1), WAL archive lag, residency checklist	backup tooling	No
V7	Security	failed / throttled logins per org, role-permission changes (esp. org.roles.manage, DEPLOY_NOTES Plan 10 §6), cert expiry	auth audit rows, audit_log event types	Counts and event types
V8	Releases	build version, goose version, pending migrations	build info, goose table	No

2.2 Actions

#	Action	Today	Rule
A1	Create organization and invite Owner	orgadmin create-org	The invitation link is a credential: show it once
A2	Re-run provisioning	migrate provision	Idempotent already; the console only calls it
A3	Verify audit chain on demand	migrate verify-audit	Result is a pass/fail plus the break position, never row contents
A4	Retry a failed job	console (B6)	Only jobs River marks retryable
A5	Suspend / reactivate an organization	console (B6)	Blocks sign-in and writes. Never deletes. Needs a new organization.status
A6	Tenant export	not built (spec §17.3)	Follows the retention rules; lands with §17.3, not before
A7	Support access	not built	See §3.7. The organization grants it, time-limited, visible to them

Every action is written to the operator audit log (§3.4). An action that affects an organization (A1, A2, A5, A7) is also written to that organization’s own audit_log, so the customer can see what the provider did.

2.3 Never in the console

• Browsing an organization’s ledger, claims, advances, petty cash, invoices or reports.
• Employee names, emails, payout details, receipts or document images.
• Editing or deleting any record.
• Any path by which an operator grants themselves access to an organization.
• A database role with BYPASSRLS held by the console process.

---

3. Architecture

3.1 Shape

flowchart LR
  subgraph VPS["Private network (VPN / Tailscale), Malaysian VPS"]
    OP[Operator browser] -->|HTTPS + MFA| CON[cmd/operator<br/>html/template, own port]
    CON -->|erp_operator role<br/>EXECUTE only| OPS[(ops schema<br/>SECURITY DEFINER functions)]
    CON -->|enqueue| RIVER[(River jobs)]
    WORKER[cmd/worker] --> RIVER
    WORKER -->|writes results| OPSR[(ops.check_result)]
    PROM[Prometheus] -->|scrape /metrics| API[cmd/api]
    PROM --> CON
    GRAF[Grafana, self-hosted] --> PROM
  end
  OPS -->|aggregates only| DB[(business tables<br/>forced RLS)]
  API -->|erp_app, per-org tx| DB

3.2 A separate binary, not a route group in the API

cmd/operator is a fourth binary beside api, worker, migrate and orgadmin.

• Why separate: the customer-facing API is on the public internet. The console must not be. A separate process binds to a private interface, so a routing mistake in cmd/api cannot expose it, and TestEveryRouteIsAuthorized keeps meaning what it means today.
• Why server-rendered: it is a small internal tool. Go html/template needs no second frontend build, no generated client, and nothing in the public web/ bundle can reveal its routes. Charts come from Grafana (§3.6), not from hand-built components.
• Not in api/openapi.yaml. The contract is the customer API. The console’s routes are internal, declared in the console package, and covered by the console’s own “every route is authorized” test (§6).

3.3 Reading across organizations without bypassing RLS

This is the hardest part and the one to review most carefully.

The console has to count things across every organization. Forced RLS lets a transaction see one organization. The options:

Option	Verdict
Give the console a BYPASSRLS role	Rejected. The process could read every record of every customer
Loop over organizations as erp_app, SET LOCAL per org	Rejected. erp_app can read every record once the setting is made. Same exposure, just slower
ops schema of SECURITY DEFINER functions that return aggregates only	Chosen

How the chosen option works:

• A new role, erp_ops_owner, owns the ops schema and its functions. It is NOLOGIN and BYPASSRLS: forced RLS applies to a table’s owner too, so without BYPASSRLS the functions would see nothing. Nobody can sign in as it, so the bypass is reachable only through the functions it owns.
• A new LOGIN role, erp_operator, is what cmd/operator connects as. It has USAGE on ops and EXECUTE on its functions, and no privilege on any table in public.
• Each function returns counts, booleans, timestamps, UUIDs and the organization name, and nothing else. Example: ops.org_overview() returns (org_id, name, status, created_at, member_count, last_activity_at).
• Every function sets search_path explicitly (a SECURITY DEFINER requirement, or the caller can hijack it).

The owner role reading across tenants is a real exception to “RLS confines every read”. It is confined to reviewed SQL, it is enforced by the drift tests in §6, and it is named here so nobody discovers it by surprise.

3.4 Operator identity and the operator audit log

Operators are not organization members and have no org_id. They need their own tables:

• ops.operator_account: id, email, password hash, TOTP secret (sealed by the existing internal/platform/identity/totp with TOTP_ENCRYPTION_KEY, as members’ secrets are), active.
• ops.operator_session: same single-use rotating refresh design as session_family.
• ops.operator_audit: append-only, with a prev_hash / row_hash chain the same shape as audit_log (migration 00007), and verified the same way.

These tables are not business tables, so they are declared exceptions to the “every table has org_id” drift test, with the reason written beside the exception. That test must still pass for everything else.

Access rules:

• Reachable only on the private network (VPN / Tailscale). Not published through the public reverse proxy.
• Password plus TOTP. No password reset by email; another operator reactivates.
• The first operator account is created from the CLI, like orgadmin create-org.

3.5 Checks run in the worker, not on page load

The reconciliation checks (V4) and VerifyChain are expensive on a large ledger. The console enqueues them as River jobs. cmd/worker runs each one inside that organization’s own tenant transaction (the path that already exists), and writes only the outcome (check, passed, detail_code, ran_at) to ops.check_result, on which erp_app holds INSERT only. The console reads that table. A nightly periodic job keeps it fresh; this is also the nightly reconciliation the roadmap leaves open (Plan 9, Ruling R30).

The nightly job has to list every organization before it can check each one, and erp_app cannot (forced RLS). So it depends on the §3.3 exception (an ops function that returns org ids only) and waits for the §5 question 1 review (decided 2026-09-28: not before that review).

This also means the console never runs business queries itself.

3.6 Metrics

• cmd/api and cmd/worker expose /metrics (Prometheus) on the private interface only: request count, error count and latency per route template, pool stats, job counts. Labels never carry an org id, user id or any record value, only route templates and status codes.
• Prometheus and Grafana run self-hosted on the Malaysian VPS. Grafana covers V2 and the time-series part of V3. The console links to it rather than redrawing it.

3.7 Support access (A7)

When an operator needs to see an organization’s data to fix a problem:

1. The operator raises a request in the console, giving a reason and a duration (max 72h).
2. The organization’s Owner, or a holder of a new org.support.grant permission, approves it in the ERP. The request and the decision are written to their audit_log.
3. For the approved window, a named operator can sign in to that one organization with a read-only support role. Every request made in that session is audited under the operator’s identity, not a customer member’s.
4. It expires on its own. The organization can revoke it early.

The operator cannot approve their own request, which is the same rule as “a claimant never picks their own approver”. Build this last. It touches authentication, and it needs its own design review.

3.8 Where things live

Piece	Location
Binary	cmd/operator
Handlers, templates	internal/operator/ (a top-level app package like internal/app, not a module; it owns no business tables)
Roles, ops schema, functions, operator tables	db/migrations, range 00230–00249 (Plan 11 holds 00190–00209, Plan 12 holds 00210–00229)
Metrics	internal/kernel/httpx middleware (route template only)
New env vars	OPERATOR_DATABASE_URL, OPERATOR_LISTEN_ADDR, both in .env.example (the drift test requires it). No separate TOTP key: identity/totp and TOTP_ENCRYPTION_KEY are reused

---

4. Roadmap

Each stage ships on its own and leaves the system safe if work stops after it. Stages 1–3 are read-only. Nothing that writes arrives before operator identity and the operator audit log exist.

The console does not depend on any Phase 2 item, so it can run alongside Phase 2.

Stage 0 — Decisions (before code)

• Review §3.3 (the SECURITY DEFINER exception) and accept or change it
• Decide suspend semantics: block sign-in only, or also block River jobs for the org
• Decide the network path: VPN vs Tailscale vs SSH tunnel (D3; the test server uses an SSH tunnel)
• Add the console to docs/ROADMAP.md, and reserve migrations 00230–00249

Stage 1 — Metrics, with no console yet

• /metrics on cmd/api and cmd/worker, private interface, route-template labels only
• Prometheus and Grafana on the VPS; one dashboard for V2 (the dashboard is deploy/grafana; the scrape jobs are deploy/vps/README.md step 5)
• Drift test: no metric label is named org, org_id, user, member or email

Verify: the Grafana dashboard shows request rate and errors; the drift test fails when a label named org_id is added.

Stage 2 — Skeleton: identity and audit, read-only

• Roles erp_ops_owner (NOLOGIN) and erp_operator (LOGIN, EXECUTE only)
• ops.operator_account, ops.operator_session, hash-chained ops.operator_audit
• cmd/operator: sign-in with password and TOTP, private listen address, CLI to create the first operator
• V8 releases page (no business data at all, which proves the plumbing)
• Drift tests from §6

Verify: sign-in fails without TOTP; a sign-in appears in ops.operator_audit; erp_operator gets permission denied on SELECT * FROM organization.

Stage 3 — Read-only views

• ops.org_overview() → V1
• ops.job_summary() → V3
• ops.check_result and the nightly worker job → V4
• ops.compliance_flags() → V5 (open periods, missing TIN / e-Invoice counts)
• ops.security_summary() → V7
• V6 backups: read the backup tool’s status file, plus a manual “restore tested on” entry until restore tests are automated

Verify: for each view, the presence-then-absence shape (§6): a fixture org with a failing check shows red, then the same org after the fix shows green; no function’s result contains a column outside the allowed types.

Stage 4 — Actions

• A1 create organization (wraps orgadmin create-org)
• A2 re-run provisioning
• A3 verify audit chain on demand (enqueued, not inline)
• A4 retry job
• A5 suspend / reactivate: organization.status (vocabulary string: active, suspended), enforced in Authenticate and on writes
• Each action writes ops.operator_audit, and the organization’s audit_log where it affects them

Verify: a suspended organization’s member cannot sign in and no row is deleted; the org’s own audit trail shows the suspension.

Stage 5 — Support access

• Its own design spec first (touches authentication)
• org.support.grant permission, request / approve / revoke / expire
• Read-only support role, per-request audit under the operator’s identity
• Test: an operator cannot approve their own request

Later, tied to other roadmap items

• A6 tenant export, with §17.3
• MyInvois submission failures per organization, with Phase 2 MyInvois
• Usage and billing, when there is pricing

---

5. Open questions

1. §3.3 exception. Is a SECURITY DEFINER owner role that reads across tenants acceptable to the auditor, given that it only returns aggregates? The alternative is no cross-tenant view at all, only per-org checks run by the worker.
2. Suspend. Does suspension also stop scheduled jobs (reminders, reconciliation)? The retention clock keeps running either way.
3. Organization name. V1 shows it. It is the customer’s business name, not personal data, unless the customer is a sole proprietor trading under their own name. Should V1 show the org id only, with the name one click away and audited?
4. Tax agent review. The roadmap still has it open. §1 and §3.7 should be part of it.

---

6. Drift tests this adds

Following AGENTS.md: establish the presence before asserting the absence, and watch each one fail once.

Test	Presence first	Then absence
erp_operator reads no table	as the owner, SELECT from organization returns rows	as erp_operator, the same SELECT is permission denied
ops functions return aggregates only	the walker finds a non-zero number of ops functions	no result column is outside {uuid, bigint, integer, boolean, timestamptz, date, the org name}; adding a text column of an email fails it
every ops function pins search_path	the walker finds a non-zero number of SECURITY DEFINER functions	none lacks SET search_path
every console route is authorized	the walker reads a non-zero number of routes	none lacks the operator auth middleware, apart from sign-in
metrics carry no tenant labels	a scrape returns a non-zero number of series	no label is named org, org_id, user, member or email
operator tables are the only non-org_id exception	the tenant-isolation test still finds its usual table count	its exception list gains exactly the ops.operator_* tables, each with a reason