Skip to content

Operator Console Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: The provider’s operator signs in to a private, server-rendered console with a password and TOTP. The console shows, for every organization, its status, active member count, last activity, the latest pass/fail of each integrity check, compliance counts, security event counts and job counts. It also shows service health, backups and the release. From it, the operator can re-run provisioning, run the checks now, retry a discarded job, and suspend or reactivate an organization. The console shows counts and pass/fail only, never record contents. Every sign-in and action is written to a hash-chained operator audit trail. Every action that affects an organization is also written to that organization’s own audit_log.

Architecture: The console is a fifth binary, cmd/operator, with its handlers and templates in internal/operator (html/template and stdlib net/http). It listens on loopback and connects as the new role erp_operator, which holds EXECUTE and no table privilege. It reads across tenants only through ops.* SECURITY DEFINER functions. Those functions are owned by erp_ops_owner, a NOLOGIN BYPASSRLS role that holds column-level SELECT on the named columns and nothing else. The operator’s own identity, sessions, audit chain and action log live in schema ops_auth. Actions reach the worker as one River job kind, operator_action, and internal/platform/ops runs each one in the target organization’s tenant transaction. The same package runs the nightly integrity checks into ops.check_result, which closes R30’s nightly reconciliation. Suspension is read-only: liveness reads organization.status, and httpx.Authenticate refuses writes. cmd/api and cmd/worker expose Prometheus /metrics on a private listener, with route-template labels only.

Tech Stack: Go 1.26 · Gin (api) · stdlib net/http + html/template + http.CrossOriginProtection (console) · pgx v5 · goose · River v0.47 · prometheus/client_golang (new) · testify · testcontainers (Postgres 17) · React 19 + i18next (one refusal string) · Playwright

Spec: docs/superpowers/specs/2026-09-25-operator-console-design.md. The Stage 0 decisions S1–S10 are binding and are cited below as Sn.

Every task’s requirements implicitly include this section.

  • Migrations 00230–00249 only. This plan uses 00230–00235. Task 2 registers {230, 249, "Operator console"} in internal/kernel/db/migrationranges_test.go in the same commit as 00230. goose runs with allow-missing (cmd/migrate/main.go runGoose), so every migration here must commute with every higher-numbered one: add, never reorder or rename, anything 00250+ created.
  • No money here. The console computes no amounts. Counts are bigint / int. No float64 anywhere in internal/operator or internal/platform/ops, and a latency histogram’s seconds (float64 inside client_golang) never carries an amount.
  • Tables. ops.check_result is the one business table. It has org_id UUID NOT NULL, ENABLE and FORCE ROW LEVEL SECURITY, and a policy named org_isolation with USING and WITH CHECK on current_setting('app.current_org', true)::uuid. The ops_auth.* tables are the operator’s own data, carry no org_id, and are declared in db/rls_exempt.txt with a reason. Every CHECK / UNIQUE is CONSTRAINT <table>_<rule> (TestEveryConstraintIsNamed).
  • Functions in ops and ops_auth are owned by erp_ops_owner (ALTER FUNCTION … OWNER TO erp_ops_owner) and declared SET search_path = pg_catalog, public, pg_temp. They schema-qualify every table (public.audit_log, ops_auth.operator_session). Each migration runs REVOKE ALL ON FUNCTION … FROM PUBLIC, then GRANT EXECUTE to exactly the role that calls the function (erp_operator for the console, erp_app for the worker, none for the CLI, which runs as the migration role). TestEveryOpsFunctionIsOwnedPinnedAndNotPublic (Task 2) enforces this.
  • ops returns aggregates only (S1c). A column returned by an ops function is uuid, bigint, integer, boolean, timestamptz or date, or text whose name is one of name, status, kind, state, queue, check_key, entity_kind, action, outcome, detail_code. There is no jsonb. TestOpsFunctionsReturnAggregatesOnly (Task 2) enforces this. ops_auth functions may return operator data (emails, hashes), and they read no public table except river_job (the ops action functions do that).
  • erp_ops_owner grants are column-level and exact (S1a). A task that makes an ops function read a new column grants that column in its migration and adds it to opsOwnerGrants in internal/kernel/db/opsschema_test.go in the same commit. Nothing is granted table-level SELECT, and nothing gets UPDATE, DELETE or TRUNCATE on public.
  • Counts and pass/fail only (PDPA, design §1). No console page renders a member’s name, email or payout detail, a record’s text, an amount, or a document. Task 7’s TestNoConsolePageShowsRecordContents shows the seeded values exist, then shows they are absent from every page’s HTML.
  • The console is not in api/openapi.yaml (design §3.2). It is covered by its own route test (Task 6). The one customer-API change, the org_suspended refusal code (Task 3), is documented in api/openapi.yaml first, then make generate, then a clean git status on generated files.
  • i18n. The console is English only (S10). Every ERP web string added (Task 3’s refusal) is in web/src/lib/i18n/locales/{en,ms,zh}/common.json, in genuine Malay. web/src/drift/i18nparity.test.ts must pass.
  • Ledger untouched. Nothing here writes a journal line or registers a posting rule. accounting/ is read only through reconcile.Default().Run.
  • One org per transaction. Worker code runs org work in db.InTx(tenancy.WithOrg(ctx, orgID), …). Cross-org work happens only inside ops.* functions, or in a pool transaction with no tenant that calls only ops.* (the notifications.PruneWorker shape). No nested InTx.
  • Lock order in a worker action’s transaction: the action’s own work (provisioning.Run, or the organization row UPDATE) comes first, and its audit.Write comes last (the audit-chain class-1 rule, internal/kernel/audit/audit.go). The operator audit chain uses LOCK TABLE ops_auth.operator_audit, not an advisory lock, so the advisory lock class registry is unchanged.
  • Presence before absence (AGENTS.md). Every refusal and absence test shows the accepted or present case first. Every guard gets a Sabotage step: break it, run the test, read the red message, restore, and record in the commit message what was broken.
  • Acting as the operator role in tests: tx.Exec(ctx, "SET LOCAL ROLE erp_operator") inside an h.Admin transaction. Do not add a password to the role in a migration (S3, Task 2). Use testdb.Start. Roles are created by the migrations once per shared container, and nothing here needs StartIsolated.
  • Tests: run only the touched packages while iterating. Run one full go test -p 2 -count=1 ./... in Task 10 only.
  • E2E lane 6: Postgres 56472, MinIO 59160, api 3160, vite 5197, console 3161. Never the user’s dev stack (55432 / 59000 / 3000 / 5173).
  • Never bare git stash. Use a WIP commit.
  • Git commits end with: Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Rulings on where the design is silent or cannot be built as written. Format: Ruling: <decision> — <why>.

  • O1 Ruling: Schemas ops (cross-tenant aggregates, actions, check_result) and ops_auth (operator identity, sessions, audit, action log) — S1 item 2. The aggregates-only type test then applies cleanly to ops.
  • O2 Ruling: The existing public-only guards stay as they are. A new file, internal/kernel/db/opsschema_test.go, holds four guards for the two new schemas. TestEveryTableIsTenantIsolated and TestExemptionFilesNameRealTables are widened to scan ops and ops_auth, with schema-qualified names (ops_auth.operator_account). A separate file keeps each test’s review local.
  • O3 Ruling: The erp_operator database credential is equivalent to operator power. ops_auth.signin_result trusts the console’s password and TOTP verdict, because argon2id verification is in Go (identity/password). That is why the credential is held only by the cmd/operator process (I15). Action attribution does not trust the console: ops.request_action takes the session token hash and resolves the operator itself, so a console bug cannot act as the wrong operator.
  • O4 Ruling: The operator audit chain has no recorded head. Rows cut off the end of the chain leave a shorter chain that still verifies, the same limit audit.VerifyChain has without a period close. The console shows the row count and head seq. Record a head externally only if an auditor asks. — // ponytail: comment in the migration.
  • O5 Ruling: ops.request_action inserts the River row with raw SQL (kind, args, queue, max_attempts, the rest defaulted). Two tests guard it: River’s worker client works a row inserted this way (Task 8), and TestRiverAgreesTheSchemaIsCurrent already fails on a River schema change. The alternative, giving erp_operator River’s insert privileges, would expose river_job.errors to the console.
  • O6 Ruling: The nightly fan-out enqueues one ops_org_checks job per organization and does not dedupe. A restart re-runs a night’s checks, which is harmless extra rows. max_attempts is 3: a check that errors leaves the organization stale on V4, which is visible, rather than retrying 25 times.
  • O7 Ruling: Writes allowed while suspended are suspendedWritable in internal/kernel/httpx/auth.go: POST /v1/auth/switch-org, DELETE /v1/auth/sessions/:id, POST /v1/reports/exports, POST /v1/me/notifications/:id/read, POST /v1/me/notifications/read-all (S2). TestSuspendedWritableNamesRealRoutes in internal/app shows each entry is a route the real router registers.
  • O8 Ruling: One console page per concern. / shows V1 with V4, V5 and V7 summarised per organization. /orgs/{id} shows the detail and the actions. /jobs shows V3 and A4. /system shows V2, V6 and V8 and the operator chain. /actions is the action log. Charts are Grafana’s (design §3.6), linked from /system.
  • O9 Ruling: A sign-in within 90 seconds of the account’s previous successful one is refused (S6). That makes a TOTP code single-use without storing steps. Local e2e reruns inside 90 seconds fail by design. Wait, or use a fresh stack.
  • O10 Ruling: METRICS_LISTEN_ADDR and OPERATOR_LISTEN_ADDR go through config.PrivateListenAddr (S3). It accepts loopback, private and unspecified IP literals, and refuses public addresses, hostnames and an empty host (:9100). METRICS_LISTEN_ADDR empty means metrics are off. OPERATOR_LISTEN_ADDR is required.
  • O11 Ruling: Build version via -ldflags "-X main.version=$(VERSION)" for api, worker and operator in the Makefile’s build target. The worker and the console declare var version = "dev" like cmd/api. erp_build_info{version} is the one gauge that carries a version label.

Path Task Responsibility
go.mod, go.sum 1 github.com/prometheus/client_golang
internal/kernel/httpx/metrics.go (+ _test) 1 Metrics, Middleware, ServeMetrics; label drift test
internal/kernel/httpx/server.go 1 Deps.Metrics wired in NewRouter
internal/kernel/db/poolmetrics.go 1 RegisterPoolMetrics(reg, pool)
internal/kernel/config/listen.go (+ _test) 1 PrivateListenAddr
cmd/api/main.go, cmd/worker/main.go, Makefile, .env.example 1 metrics listener, version stamp, METRICS_LISTEN_ADDR
db/migrations/00230_ops_roles.sql 2 roles, schemas, default-privilege revoke
db/migrations/00231_ops_auth.sql 2 operator tables, audit chain, sign-in/session functions, ops.release_info
internal/kernel/db/opsschema_test.go 2, 4, 5, 8 the four ops guards; opsOwnerGrants
internal/kernel/db/rlscoverage_test.go, db/rls_exempt.txt 2, 8 scan ops/ops_auth; exempt ops_auth.*
internal/kernel/db/migrationranges_test.go 2 {230, 249, "Operator console"}
db/migrations/00232_organization_status.sql 3 organization.status
internal/kernel/authn/authn.go 3 ErrOrgSuspended
internal/platform/identity/liveness/liveness.go (+ _test) 3 reads status, returns ErrOrgSuspended
internal/kernel/httpx/auth.go (+ auth_test.go) 3 read-only enforcement, suspendedWritable
internal/app/routes_test.go 3 TestSuspendedWritableNamesRealRoutes
api/openapi.yaml, web/src/lib/api/schema.d.ts, web/src/lib/i18n/locales/{en,ms,zh}/common.json 3 org_suspended
db/migrations/00233_ops_check_result.sql 4 ops.check_result, ops.org_ids()
internal/platform/ops/checks.go (+ _test) 4 RunChecks, fan-out and per-org workers, ChecksPeriodic
cmd/worker/exports.go, cmd/worker/main.go 4, 8 register the ops workers and periodic job
db/migrations/00234_ops_views.sql 5 org_overview, job_summary, discarded_jobs, check_latest, compliance_flags, security_summary
internal/kernel/db/opsviews_test.go 5 each view: presence then absence
cmd/operator/main.go (+ wiring_test.go) 6 serve, add, deactivate
internal/operator/{server.go,auth.go,system.go} + templates/{layout,signin,system}.html 6 mux, route table, session middleware, sign-in, V8
internal/operator/server_test.go 6 TestEveryConsoleRouteRequiresASignedInOperator, sign-in tests
internal/kernel/layering/layering_test.go, internal/app/binaries_test.go 6, 8 operator layer; cmd/operator links no module; worker links all
internal/operator/{orgs.go,jobs.go} + templates/{orgs,org,jobs}.html, system.go (V2, V6) 7 pages
internal/operator/pages_test.go 7 TestNoConsolePageShowsRecordContents
db/migrations/00235_ops_actions.sql 8 ops_auth.operator_action, request_action, claim_action, finish_action, ops_auth.action_log
internal/platform/ops/actions.go (+ _test) 8 ActionWorker
internal/operator/actions.go + templates/{confirm,actions}.html (+ _test) 9 action forms and log
scripts/dev-stack.sh, web/playwright.config.ts, web/e2e/{totp.ts,operator-console.spec.ts} 10 lane-parameterised stack with the console; e2e
docs/RUNBOOK.md, docs/DEPLOY_NOTES.md, docs/ROADMAP.md 10 infra checklist, deploy notes, tick

Task 1: Build version, /metrics on api and worker, no tenant labels

Section titled “Task 1: Build version, /metrics on api and worker, no tenant labels”

Stage 1 in code. The Prometheus and Grafana servers themselves are infrastructure (Task 10 checklist, I9–I11).

Files:

  • Modify: go.mod, go.sum, Makefile, .env.example, cmd/api/main.go, cmd/worker/main.go, internal/kernel/httpx/server.go
  • Create: internal/kernel/httpx/metrics.go, internal/kernel/httpx/metrics_test.go, internal/kernel/db/poolmetrics.go, internal/kernel/config/listen.go, internal/kernel/config/listen_test.go

Interfaces:

  • Produces: httpx.NewMetrics(reg prometheus.Registerer) *httpx.Metrics; (*Metrics).Middleware() gin.HandlerFunc; httpx.ServeMetrics(addr string, g prometheus.Gatherer) *http.Server; httpx.Deps.Metrics *httpx.Metrics; db.RegisterPoolMetrics(reg prometheus.Registerer, pool *pgxpool.Pool); config.PrivateListenAddr(name, raw string) (string, error); main.version in cmd/worker.

  • Step 1: Write the failing label test — internal/kernel/httpx/metrics_test.go:

// TestMetricsCarryNoTenantLabels: a metric label is exported to Prometheus
// and Grafana, outside the tenant boundary. Only the route TEMPLATE may
// appear, never a path value, an org, a user or an email (design §3.6).
func TestMetricsCarryNoTenantLabels(t *testing.T) {
reg := prometheus.NewRegistry()
m := httpx.NewMetrics(reg)
r := gin.New()
r.Use(m.Middleware())
r.GET("/v1/claims/:id", func(c *gin.Context) { c.Status(http.StatusOK) })
id := uuid.NewString()
r.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest(http.MethodGet, "/v1/claims/"+id, nil))
fams, err := reg.Gather()
require.NoError(t, err)
allowed := map[string]bool{"method": true, "route": true, "status": true}
series, sawTemplate := 0, false
for _, f := range fams {
for _, mt := range f.GetMetric() {
series++
for _, l := range mt.GetLabel() {
require.True(t, allowed[l.GetName()], "metric %s has label %q; only %v may be exported",
f.GetName(), l.GetName(), allowed)
require.NotContains(t, l.GetValue(), id, "metric %s carries a path value", f.GetName())
sawTemplate = sawTemplate || l.GetValue() == "/v1/claims/:id"
}
}
}
// PRESENCE: the request was counted, so the loop above judged something.
require.Positive(t, series, "no series gathered; the middleware recorded nothing")
require.True(t, sawTemplate, "the route template is not a label value; the middleware is not labelling by route")
}

Also TestPrivateListenAddr in listen_test.go, table-driven. Accepted: 127.0.0.1:9100, [::1]:9100, 10.0.0.5:9100, 172.18.0.2:9100, 0.0.0.0:9100. Refused: 203.0.113.7:9100 (public), metrics.local:9100 (hostname), :9100 (empty host), 127.0.0.1 (no port), 127.0.0.1:99999.

  • Step 2: Run, and watch it fail to compile. go test ./internal/kernel/httpx/ ./internal/kernel/config/ -run 'Metrics|PrivateListen' -count=1

  • Step 3: Implement. go get github.com/prometheus/client_golang@latest. metrics.go:

// Metrics is the api's request telemetry (design §3.6). Labels are the
// method, the route TEMPLATE (gin's FullPath) and the status code — never a
// path value, so no org id, member id or record id can reach Prometheus.
type Metrics struct {
requests *prometheus.CounterVec
duration *prometheus.HistogramVec
}
func NewMetrics(reg prometheus.Registerer) *Metrics {
m := &Metrics{
requests: prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "erp_http_requests_total", Help: "HTTP requests by method, route template and status."},
[]string{"method", "route", "status"}),
duration: prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "erp_http_request_duration_seconds", Help: "HTTP request latency by route template.",
Buckets: prometheus.DefBuckets}, []string{"method", "route"}),
}
reg.MustRegister(m.requests, m.duration)
return m
}
func (m *Metrics) Middleware() gin.HandlerFunc {
return func(c *gin.Context) {
start := time.Now()
c.Next()
route := c.FullPath()
if route == "" {
route = "unmatched" // a 404's raw path is caller-chosen text
}
m.requests.WithLabelValues(c.Request.Method, route, strconv.Itoa(c.Writer.Status())).Inc()
m.duration.WithLabelValues(c.Request.Method, route).Observe(time.Since(start).Seconds())
}
}
// ServeMetrics serves reg on its own listener, never on the public router.
func ServeMetrics(addr string, g prometheus.Gatherer) *http.Server {
mux := http.NewServeMux()
mux.Handle("GET /metrics", promhttp.HandlerFor(g, promhttp.HandlerOpts{}))
return &http.Server{Addr: addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second}
}

In NewRouter, if deps.Metrics != nil, it is r.Used first, before AccessLog, so every request, including a panic’s 500, is counted. poolmetrics.go registers three prometheus.NewGaugeFuncs over pool.Stat(): erp_db_pool_acquired_conns, erp_db_pool_idle_conns, erp_db_pool_total_conns. listen.go:

// PrivateListenAddr validates a listen address that must not face the
// internet (design S3). name is the env var, for the message. An IP literal
// is required, so what is validated is what is bound: a hostname could
// resolve to a public address later.
func PrivateListenAddr(name, raw string) (string, error) {
host, port, err := net.SplitHostPort(raw)
if err != nil {
return "", fmt.Errorf("%s: %w", name, err)
}
if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 {
return "", fmt.Errorf("%s: port %q is not 1-65535", name, port)
}
ip, err := netip.ParseAddr(host)
if err != nil {
return "", fmt.Errorf("%s: %q is not an IP literal; write 127.0.0.1 or a private address", name, host)
}
if !ip.IsLoopback() && !ip.IsPrivate() && !ip.IsUnspecified() {
return "", fmt.Errorf("%s: %s is a public address; this listener must stay private", name, ip)
}
return raw, nil
}

cmd/api/main.go and cmd/worker/main.go: read raw := os.Getenv("METRICS_LISTEN_ADDR") (a string literal, for the env drift test). If it is non-empty, validate it and build reg := prometheus.NewRegistry(), then register collectors.NewGoCollector(), collectors.NewProcessCollector(…), db.RegisterPoolMetrics, and a erp_build_info{version} gauge set to 1. The api passes httpx.NewMetrics(reg) in Deps. Both run ServeMetrics(addr, reg).ListenAndServe() in a goroutine, log the error, and shut it down with the process. Empty means off, with one log line. The worker gains var version = "dev" and logs it at start. Makefile:

VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
LDFLAGS := -X main.version=$(VERSION)
build:
go build -ldflags "$(LDFLAGS)" -o bin/api ./cmd/api
go build -ldflags "$(LDFLAGS)" -o bin/worker ./cmd/worker
go build -o bin/migrate ./cmd/migrate

.env.example gains METRICS_LISTEN_ADDR= with a comment: it is optional and empty means off; it must be an IP literal, loopback or private (127.0.0.1:9100 on a bare VPS, 0.0.0.0:9100 only inside a container whose port is not published); scraped by the self-hosted Prometheus (RUNBOOK §7).

  • Step 4: Run. go test ./internal/kernel/httpx/ ./internal/kernel/config/ ./internal/kernel/db/ -count=1 → PASS. go build ./cmd/...
  • Step 5: Sabotage. Add "org_id" to the CounterVec labels, passing c.GetString("org"). The test goes red naming label "org_id". Restore it. Then record c.Request.URL.Path instead of FullPath(). The test goes red on “carries a path value”. Restore it.
  • Step 6: Runtime check. Run METRICS_LISTEN_ADDR=127.0.0.1:9109 with the api on the dev stack, then curl -s 127.0.0.1:9109/metrics | grep erp_http_requests_total after one curl localhost:3000/v1/health. METRICS_LISTEN_ADDR=8.8.8.8:9109 refuses to boot.
  • Step 7: Commit Operator console T1: build version, private /metrics with route-template labels only

Task 2: Roles, ops / ops_auth, operator identity and audit chain, the four ops guards

Section titled “Task 2: Roles, ops / ops_auth, operator identity and audit chain, the four ops guards”

Files:

  • Create: db/migrations/00230_ops_roles.sql, db/migrations/00231_ops_auth.sql, internal/kernel/db/opsschema_test.go
  • Modify: internal/kernel/db/migrationranges_test.go, internal/kernel/db/rlscoverage_test.go, db/rls_exempt.txt

Interfaces:

  • Produces: roles erp_ops_owner (NOLOGIN BYPASSRLS) and erp_operator (LOGIN, no password). Tables ops_auth.operator_account, ops_auth.operator_session and ops_auth.operator_audit. Functions:

    • ops_auth.signin_lookup(email text) → (operator_id uuid, password_hash text, totp_sealed bytea) (erp_operator)
    • ops_auth.signin_result(email text, ok boolean, token_hash bytea) → uuid (erp_operator)
    • ops_auth.session_operator(token_hash bytea) → uuid (no grant; used inside)
    • ops_auth.session_check(token_hash bytea) → (operator_id uuid, email text) (erp_operator)
    • ops_auth.session_close(token_hash bytea) → void (erp_operator)
    • ops_auth.chain_status() → (row_count bigint, head_seq bigint, first_break bigint) (erp_operator)
    • ops_auth.operator_add(email text, password_hash text, totp_sealed bytea, reset boolean) → uuid (no grant)
    • ops_auth.operator_deactivate(email text) → void (no grant)
    • ops_auth.audit_append(operator uuid, action text, org uuid, ref uuid) → void (no grant)
    • ops_auth.audit_hash(...) → bytea (no grant)
    • ops.release_info() → (version bigint, applied_at timestamptz, applied_count bigint) (erp_operator)
    • opsOwnerGrants in opsschema_test.go.
  • Step 1: Write the four failing guards — internal/kernel/db/opsschema_test.go:

// opsOwnerGrants is every privilege erp_ops_owner holds on a table in
// public, column by column (design S1a). It is a BYPASSRLS role, so this list
// IS the set of customer data an ops function could ever read. A new column
// is a reviewed line here, in the same commit as its GRANT.
var opsOwnerGrants = map[string]string{
"goose_db_version.version_id": "SELECT",
"goose_db_version.is_applied": "SELECT",
"goose_db_version.tstamp": "SELECT",
}
// aggregateText is the only names a text column of an ops function may have
// (design S1c): vocabulary values and the organization's name.
var aggregateText = map[string]bool{"name": true, "status": true, "kind": true, "state": true,
"queue": true, "check_key": true, "entity_kind": true, "action": true, "outcome": true, "detail_code": true}
var aggregateTypes = map[string]bool{"uuid": true, "bigint": true, "integer": true, "boolean": true,
"timestamp with time zone": true, "date": true, "void": true}
func TestEveryOpsFunctionIsOwnedPinnedAndNotPublic(t *testing.T) {
h := testdb.Start(t)
rows, err := h.Admin.Query(context.Background(), `
SELECT n.nspname || '.' || p.proname, pg_get_userbyid(p.proowner),
EXISTS (SELECT 1 FROM aclexplode(coalesce(p.proacl, acldefault('f', p.proowner))) a
WHERE a.grantee = 0 AND a.privilege_type = 'EXECUTE'),
coalesce((SELECT c FROM unnest(p.proconfig) c WHERE c LIKE 'search_path=%'), '')
FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname IN ('ops', 'ops_auth')`)
require.NoError(t, err)
seen := 0
for rows.Next() {
var name, owner, setting string
var public bool
require.NoError(t, rows.Scan(&name, &owner, &public, &setting))
seen++
require.Equal(t, "erp_ops_owner", owner, "%s must be owned by erp_ops_owner (ALTER FUNCTION ... OWNER TO)", name)
require.False(t, public, "%s is executable by PUBLIC; REVOKE ALL ON FUNCTION ... FROM PUBLIC", name)
require.True(t, pinsPgTempLast(setting), "%s must SET search_path = pg_catalog, public, pg_temp (got %q)", name, setting)
}
require.NoError(t, rows.Err())
// PRESENCE: the scan reached the schemas.
require.GreaterOrEqual(t, seen, 5, "found %d ops/ops_auth functions; the query is not reaching them", seen)
}
func TestOpsFunctionsReturnAggregatesOnly(t *testing.T) {
h := testdb.Start(t)
// Every result column: OUT/TABLE arguments, or the plain return type.
rows, err := h.Admin.Query(context.Background(), `
SELECT p.proname, coalesce(a.name, ''), format_type(coalesce(a.typ, p.prorettype), NULL)
FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
LEFT JOIN LATERAL (
SELECT u.name, u.typ FROM unnest(p.proallargtypes, p.proargmodes, p.proargnames)
AS u(typ, mode, name) WHERE u.mode IN ('o', 't')) a ON true
WHERE n.nspname = 'ops'`)
require.NoError(t, err)
cols := 0
for rows.Next() {
var fn, col, typ string
require.NoError(t, rows.Scan(&fn, &col, &typ))
cols++
if typ == "text" {
require.True(t, aggregateText[col], "ops.%s returns text column %q; text is allowed only for %v "+
"(design S1c). A name, email or record text must never leave ops", fn, col, aggregateText)
continue
}
require.True(t, aggregateTypes[typ], "ops.%s returns %q (%s); ops returns counts, flags, ids and times only", fn, col, typ)
}
require.NoError(t, rows.Err())
require.Positive(t, cols, "no ops result columns found; the guard judged nothing")
}

setof uuid (Task 4’s org_ids) reads as uuid through prorettype. A TABLE function has no plain return type to check, and its record type never reaches the fallback because its columns are t mode.

Two more tests:

  • TestTheOpsOwnerHoldsExactlyItsGrants:
    • Presence: the table-level ACL query over public finds rows for erp_app.
    • Absence: the same query finds none for erp_ops_owner.
    • The column-level query (aclexplode(att.attacl) over pg_attribute in public, grantee 'erp_ops_owner'::regrole) returns exactly opsOwnerGrants.
    • pg_roles says erp_ops_owner is NOT rolcanlogin, rolbypassrls and NOT rolsuper.
  • TestTheOperatorRoleReadsNoTable:
    1. Insert an organization as h.Admin and show SELECT count(*) FROM organization > 0 (presence).
    2. In an h.Admin transaction, SET LOCAL ROLE erp_operator and run SELECT * FROM ops.release_info(), which succeeds. The role is usable.
    3. As the last statement, SELECT * FROM organization fails with SQLSTATE 42501.
    4. A catalog query finds no table-level or column-level ACL entry for erp_operator in public, ops or ops_auth.

Widen TestEveryTableIsTenantIsolated and TestExemptionFilesNameRealTables. Each selects CASE WHEN n.nspname = 'public' THEN c.relname ELSE n.nspname || '.' || c.relname END and filters n.nspname IN ('public', 'ops', 'ops_auth'). Every message keeps naming the table. Add {230, 249, "Operator console"} to migrationRanges.

  • Step 2: Run, and watch them fail. go test ./internal/kernel/db/ -run 'Ops|OperatorRole|TenantIsolated|ExemptionFiles|MigrationIsNumbered' -count=1 → fails: no schemas, fewer than 5 functions.

  • Step 3: Write 00230_ops_roles.sql:

-- +goose Up
-- The operator console's two roles (design §3.3, S1). erp_ops_owner owns the
-- ops functions and bypasses row-level security, so it can count across
-- organizations. It cannot sign in, and it holds SELECT only on the columns
-- the functions need (opsschema_test.go, opsOwnerGrants). erp_operator is what
-- cmd/operator connects as: EXECUTE on named functions, no table privilege.
-- It has no password here. Each environment sets one (DEPLOY_NOTES), so no
-- known password ships in a migration.
-- +goose StatementBegin
DO $$
BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'erp_ops_owner') THEN
CREATE ROLE erp_ops_owner NOLOGIN BYPASSRLS;
END IF;
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'erp_operator') THEN
CREATE ROLE erp_operator LOGIN;
END IF;
END
$$;
-- +goose StatementEnd
CREATE SCHEMA ops AUTHORIZATION erp_ops_owner;
CREATE SCHEMA ops_auth AUTHORIZATION erp_ops_owner;
REVOKE ALL ON SCHEMA ops, ops_auth FROM PUBLIC;
GRANT USAGE ON SCHEMA ops, ops_auth TO erp_operator;
-- Functions the migration role creates here must not be executable by
-- PUBLIC, which is Postgres's default. Each migration still revokes
-- explicitly; this is the backstop.
ALTER DEFAULT PRIVILEGES IN SCHEMA ops, ops_auth REVOKE EXECUTE ON FUNCTIONS FROM PUBLIC;
-- +goose Down
ALTER DEFAULT PRIVILEGES IN SCHEMA ops, ops_auth GRANT EXECUTE ON FUNCTIONS TO PUBLIC;
DROP SCHEMA ops_auth CASCADE;
DROP SCHEMA ops CASCADE;
-- +goose StatementBegin
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'erp_operator') THEN
EXECUTE 'DROP OWNED BY erp_operator';
END IF;
IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'erp_ops_owner') THEN
EXECUTE 'DROP OWNED BY erp_ops_owner';
END IF;
END
$$;
-- +goose StatementEnd
DROP ROLE IF EXISTS erp_operator;
DROP ROLE IF EXISTS erp_ops_owner;
  • Step 4: Write 00231_ops_auth.sql. The tables, and the functions that are not obvious:
-- +goose Up
CREATE TABLE ops_auth.operator_account (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
email TEXT NOT NULL CONSTRAINT operator_account_email_once UNIQUE,
password_hash TEXT NOT NULL,
totp_sealed BYTEA NOT NULL,
active BOOLEAN NOT NULL DEFAULT true,
failed_count INT NOT NULL DEFAULT 0,
locked_until TIMESTAMPTZ,
last_signin_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT operator_account_email_normal CHECK (email = lower(btrim(email)) AND email LIKE '%_@_%')
);
CREATE TABLE ops_auth.operator_session (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
operator_id UUID NOT NULL REFERENCES ops_auth.operator_account (id),
token_hash BYTEA NOT NULL CONSTRAINT operator_session_token_once UNIQUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(),
revoked_at TIMESTAMPTZ
);
-- Append-only and hash-chained, the audit_log shape (00007). Kept seven
-- years (S10). No role holds UPDATE or DELETE, and nobody signs in as its
-- owner.
-- ponytail: no recorded head, so a truncated tail still verifies (Ruling O4).
-- Record the head externally if an auditor asks.
CREATE TABLE ops_auth.operator_audit (
seq BIGSERIAL PRIMARY KEY,
operator_id UUID,
action TEXT NOT NULL CONSTRAINT operator_audit_action_known CHECK (action IN
('operator_added','operator_reset','operator_deactivated',
'signin','signin_failed','signout','action_requested')),
org_id UUID,
ref UUID,
occurred_at TIMESTAMPTZ NOT NULL,
prev_hash BYTEA,
row_hash BYTEA NOT NULL
);
ALTER TABLE ops_auth.operator_account OWNER TO erp_ops_owner;
ALTER TABLE ops_auth.operator_session OWNER TO erp_ops_owner;
ALTER TABLE ops_auth.operator_audit OWNER TO erp_ops_owner;
-- One canonical rendering, used by append and verify alike: two renderings
-- are how audit_log's chain once broke (kernel/audit rowHashSQL).
-- +goose StatementBegin
CREATE FUNCTION ops_auth.audit_hash(p_seq bigint, p_operator uuid, p_action text, p_org uuid,
p_ref uuid, p_at timestamptz, p_prev bytea) RETURNS bytea
LANGUAGE sql STABLE SET search_path = pg_catalog, public, pg_temp AS $$
SELECT public.digest(concat_ws(chr(31), p_seq::text, coalesce(p_operator::text, ''), p_action,
coalesce(p_org::text, ''), coalesce(p_ref::text, ''),
to_char(p_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US'),
coalesce(encode(p_prev, 'hex'), '')), 'sha256')
$$;
-- +goose StatementEnd
-- +goose StatementBegin
CREATE FUNCTION ops_auth.audit_append(p_operator uuid, p_action text, p_org uuid, p_ref uuid) RETURNS void
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$
DECLARE
v_prev bytea;
v_seq bigint;
v_at timestamptz := clock_timestamp();
BEGIN
-- Serializes appenders so two cannot chain onto one prev_hash. A table
-- lock, not an advisory lock: the owner may lock its own table, and the
-- advisory class registry (kernel/audit) stays as it is.
LOCK TABLE ops_auth.operator_audit IN SHARE ROW EXCLUSIVE MODE;
SELECT row_hash INTO v_prev FROM ops_auth.operator_audit ORDER BY seq DESC LIMIT 1;
v_seq := nextval(pg_get_serial_sequence('ops_auth.operator_audit', 'seq'));
INSERT INTO ops_auth.operator_audit (seq, operator_id, action, org_id, ref, occurred_at, prev_hash, row_hash)
VALUES (v_seq, p_operator, p_action, p_org, p_ref, v_at, v_prev,
ops_auth.audit_hash(v_seq, p_operator, p_action, p_org, p_ref, v_at, v_prev));
END
$$;
-- +goose StatementEnd
-- The session rule, in one place: live, unrevoked, 8h absolute, 30m idle,
-- and the account still active (S6).
-- +goose StatementBegin
CREATE FUNCTION ops_auth.session_operator(p_token_hash bytea) RETURNS uuid
LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$
SELECT s.operator_id FROM ops_auth.operator_session s
JOIN ops_auth.operator_account a ON a.id = s.operator_id
WHERE s.token_hash = p_token_hash AND s.revoked_at IS NULL AND a.active
AND s.created_at > now() - interval '8 hours' AND s.last_seen_at > now() - interval '30 minutes'
$$;
-- +goose StatementEnd
-- The console verifies the password (argon2id, Go) and the TOTP code, then
-- reports the verdict here. The lock-out, the 90-second rule (S6) and the
-- audit row are decided here, so the console cannot skip them (Ruling O3).
-- +goose StatementBegin
CREATE FUNCTION ops_auth.signin_result(p_email text, p_ok boolean, p_token_hash bytea) RETURNS uuid
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$
DECLARE
v_id uuid;
v_allowed boolean;
v_session uuid;
BEGIN
SELECT id, active AND (locked_until IS NULL OR locked_until < now())
AND (last_signin_at IS NULL OR last_signin_at < now() - interval '90 seconds')
INTO v_id, v_allowed
FROM ops_auth.operator_account WHERE email = p_email FOR UPDATE;
IF p_ok AND v_allowed THEN
UPDATE ops_auth.operator_account SET failed_count = 0, locked_until = NULL, last_signin_at = now()
WHERE id = v_id;
INSERT INTO ops_auth.operator_session (operator_id, token_hash) VALUES (v_id, p_token_hash)
RETURNING id INTO v_session;
PERFORM ops_auth.audit_append(v_id, 'signin', NULL, v_session);
RETURN v_session;
END IF;
IF v_id IS NOT NULL THEN
UPDATE ops_auth.operator_account
SET failed_count = CASE WHEN failed_count >= 4 THEN 0 ELSE failed_count + 1 END,
locked_until = CASE WHEN failed_count >= 4 THEN now() + interval '15 minutes' ELSE locked_until END
WHERE id = v_id;
END IF;
PERFORM ops_auth.audit_append(v_id, 'signin_failed', NULL, NULL);
RETURN NULL;
END
$$;
-- +goose StatementEnd

The remaining functions are plain:

  • signin_lookup returns the row for an active account only.
  • session_check runs UPDATE … SET last_seen_at = now() … WHERE s.operator_id = ops_auth.session_operator(p_token_hash) RETURNING operator_id, email.
  • session_close sets revoked_at and appends signout.
  • chain_status walks ORDER BY seq, recomputes each row with audit_hash, and returns the first seq whose prev_hash or row_hash disagrees, or NULL.
  • operator_add inserts, or, with reset, replaces password_hash and totp_sealed, sets active, and revokes live sessions. It appends operator_added or operator_reset, and refuses a duplicate without reset, or a reset of an unknown email.
  • operator_deactivate sets active = false, revokes sessions and appends operator_deactivated.
  • ops.release_info: SELECT max(version_id), max(tstamp)::timestamptz, count(*) FROM public.goose_db_version WHERE is_applied AND version_id > 0.

Every function gets ALTER FUNCTION … OWNER TO erp_ops_owner and REVOKE ALL … FROM PUBLIC. Then run GRANT EXECUTE ON FUNCTION ops_auth.signin_lookup(text), ops_auth.signin_result(text, boolean, bytea), ops_auth.session_check(bytea), ops_auth.session_close(bytea), ops_auth.chain_status(), ops.release_info() TO erp_operator; and GRANT SELECT (version_id, is_applied, tstamp) ON public.goose_db_version TO erp_ops_owner;. The Down migration drops the functions and tables in reverse.

db/rls_exempt.txt gains the three ops_auth.* tables under a comment. They are the provider’s own operator identity and trail, not tenant data. No org_id is possible, because an operator is not a member of any organization (design §3.4). The condition matches the identity tables’ exemption: no package under internal/modules names them, and only ops_auth functions and internal/operator reach them.

  • Step 5: Add behaviour tests in opsschema_test.go, running as erp_operator via SET LOCAL ROLE:

    • TestOperatorSigninLocksAfterFiveFailures: add an operator as admin via operator_add. Presence: signin_result(email, true, h1) returns a session. Wait out the 90s rule by UPDATE … SET last_signin_at = now() - interval '2 minutes' as admin. Record five false results. Then signin_result(email, true, h2) returns NULL. As admin, locked_until is in the future.
    • TestOperatorSigninIsSingleUseWithin90Seconds.
    • TestOperatorSessionExpiresIdle: set last_seen_at 31 minutes back as admin, and session_check returns no row.
    • TestOperatorAuditChainVerifiesAndDetectsAnEdit: presence: after three events, chain_status() gives first_break NULL and row_count 3. Then, as admin, UPDATE ops_auth.operator_audit SET action = 'signout' WHERE seq = 2, and first_break = 2.
  • Step 6: Run. go test ./internal/kernel/db/ -count=1 → PASS (including every pre-existing test). go test ./internal/kernel/layering/ -count=1: the exempt-table grep must still pass with the dotted names.

  • Step 7: Sabotage, one at a time, recording each red message:

    1. Drop REVOKE ALL … FROM PUBLIC for chain_status, and also drop the default-privilege line.
    2. Change one SET search_path to pg_catalog, pg_temp, public.
    3. Add email text to ops.release_info’s RETURNS TABLE.
    4. Add GRANT SELECT ON public.goose_db_version TO erp_ops_owner (table-level).
    5. Add GRANT SELECT ON public.organization TO erp_operator.

    Restore after each.

  • Step 8: Commit Operator console T2: erp_ops_owner/erp_operator, ops_auth identity and audit chain, ops guards


Task 3: organization.status and read-only suspension

Section titled “Task 3: organization.status and read-only suspension”

Files:

  • Create: db/migrations/00232_organization_status.sql
  • Modify: internal/kernel/authn/authn.go, internal/platform/identity/liveness/liveness.go (+ its test), internal/kernel/httpx/auth.go, internal/kernel/httpx/auth_test.go, internal/app/routes_test.go, api/openapi.yaml, generated internal/gen/api, web/src/lib/api/schema.d.ts, web/src/lib/i18n/locales/{en,ms,zh}/common.json, web/src/lib/api/problem.test.ts

Interfaces:

  • Produces: column organization.status TEXT NOT NULL DEFAULT 'active' ('active' | 'suspended'). authn.ErrOrgSuspended. The 403 Problem with code: "org_suspended". httpx.SuspendedWritable() []string, for the route test.

  • Consumes: liveness.Cached.Check, httpx.Authenticate.

  • Step 1: Failing tests.

    • In liveness_test.go: presence: an active organization’s claims Check → nil. Then, as admin, UPDATE organization SET status = 'suspended', and Check → errors.Is(err, authn.ErrOrgSuspended). A revoked session in a suspended organization still returns ErrSessionDead, not ErrOrgSuspended, so the dead-session refusal wins.
    • In httpx/auth_test.go, with a stub LivenessChecker returning authn.ErrOrgSuspended:
      • Presence: the same stub returning nil lets POST /v1/x reach the handler.
      • Suspended: GET /v1/x reaches it.
      • Suspended: POST /v1/x → 403, and the body’s code is org_suspended.
      • Suspended: POST /v1/reports/exports, registered in the test router at that path, reaches it.
    • In internal/app/routes_test.go, add TestSuspendedWritableNamesRealRoutes. Build the real router the way TestEveryRouteIsAuthorized does. Presence: it registers more than 50 routes. Every httpx.SuspendedWritable() entry is among METHOD + " " + route.Path.
  • Step 2: Run, and watch them fail. go test ./internal/platform/identity/liveness/ ./internal/kernel/httpx/ ./internal/app/ -run 'Suspend|Liveness' -count=1

  • Step 3: Implement.

-- 00232_organization_status.sql
-- +goose Up
-- A5 suspension (design S2): read-only, never a deletion. A vocabulary string.
ALTER TABLE organization ADD COLUMN status TEXT NOT NULL DEFAULT 'active'
CONSTRAINT organization_status_known CHECK (status IN ('active', 'suspended'));
-- +goose Down
ALTER TABLE organization DROP COLUMN status;

authn.go:

// ErrOrgSuspended is a liveness answer, not a refusal: the session is live,
// and the organization it acts in is read-only (design S2). httpx.Authenticate
// admits reads and refuses writes.
var ErrOrgSuspended = errors.New("authn: organization is suspended")

In liveness.go, state gains Suspended bool json:“x”``. fromPostgres selects a fifth value, COALESCE((SELECT status = 'suspended' FROM organization WHERE id = $3 AND org_id = $3), false). Check returns authn.ErrOrgSuspended after the perm-version case. Every other refusal comes first. A cached entry written before this change has no x and reads as not suspended for at most livenessTTL (5s). In Authenticate:

err := l.Check(c.Request.Context(), claims)
suspended := errors.Is(err, authn.ErrOrgSuspended)
if err != nil && !suspended {
_ = c.Error(apperr.New(apperr.KindUnauthorized, "the session is no longer valid"))
c.Abort()
return
}
if suspended && !readOnlyMethod(c.Request.Method) && !suspendedWritable[c.Request.Method+" "+c.FullPath()] {
_ = c.Error(apperr.New(apperr.KindForbidden,
"this organization is suspended: its records can be read and exported, not changed").
With("code", "org_suspended"))
c.Abort()
return
}

readOnlyMethod is GET, HEAD or OPTIONS. suspendedWritable holds the five routes of Ruling O7, with a comment saying why each one is allowed.

Document the code in api/openapi.yaml’s Problem.code description: add “org_suspended, a 403 on any write while the organization is suspended (read-only; reads and exports still work)”. Run make generate.

Locales, common:refusals.org_suspended:

  • en: "This organization is suspended. You can still view and export its records, but you cannot change anything."
  • ms: "Organisasi ini digantung. Anda masih boleh melihat dan mengeksport rekodnya, tetapi tidak boleh mengubah apa-apa."
  • zh: "该组织已被暂停。您仍可查看和导出其记录,但不能进行任何更改。"

problem.test.ts gains the case: a 403 with code: 'org_suspended' → problemMessage returns the translation in ms.

  • Step 4: Run. go test ./internal/platform/identity/... ./internal/kernel/httpx/ ./internal/app/ ./internal/kernel/db/ -count=1, then pnpm -C web test -- problem i18nparity, then git status. Generated files are committed, and nothing else is dirty.
  • Step 5: Sabotage. Remove the !readOnlyMethod clause, so every request is refused. The GET case goes red. Then make Check return nil when suspended. The suspended-POST case goes red. Then add "POST /v1/nope" to suspendedWritable. The route test names it. Restore each.
  • Step 6: Commit Operator console T3: organization.status, suspension is read-only (S2)

Task 4: ops.check_result and the nightly integrity checks (V4; closes R30’s nightly reconciliation)

Section titled “Task 4: ops.check_result and the nightly integrity checks (V4; closes R30’s nightly reconciliation)”

Files:

  • Create: db/migrations/00233_ops_check_result.sql, internal/platform/ops/checks.go, internal/platform/ops/checks_test.go
  • Modify: cmd/worker/exports.go, cmd/worker/main.go, cmd/worker/wiring_test.go, internal/kernel/db/opsschema_test.go (organization.id)

Interfaces:

  • Produces:

    • ops.check_result, and ops.org_ids() RETURNS SETOF uuid (EXECUTE: erp_app).
    • package ops (internal/platform/ops), with RunChecks(ctx, d *db.DB, c clock.Clock, orgID uuid.UUID) error.
    • Job types ChecksFanoutArgs{} (kind ops_checks_fanout) and OrgChecksArgs{OrgID uuid.UUID} (kind ops_org_checks).
    • Workers ChecksFanoutWorker{DB} and OrgChecksWorker{DB, Clock}, and ChecksPeriodic() *river.PeriodicJob (24h, RunOnStart: true).
  • Consumes: reconcile.Default().Run, river.ClientFromContext[pgx.Tx], jobs.QueueMaintenance.

  • Step 1: Failing tests in internal/platform/ops/checks_test.go, using expensestest.New(t) for a provisioned organization:

    • TestRunChecksRecordsOneRowPerCheck: after RunChecks, as admin, SELECT check_key, passed, finding_count FROM ops.check_result WHERE org_id = $1 has one row per reconcile.Default() check (more than 5), all passed.
    • TestRunChecksRecordsAFailure, presence then absence. First, all checks pass. Then, as admin, append an unbalanced journal line the way reconcile’s own tests break entry_balance (read internal/platform/accounting/reconcile/*_test.go and use the same fixture helper). RunChecks then records entry_balance with passed = false and finding_count >= 1.
    • TestTheAppRoleCanOnlyAppendCheckResults: presence: as admin, both organizations’ rows are visible. As erp_app in org A’s tenant transaction, an INSERT for org A succeeds, and an INSERT naming org B fails the policy’s WITH CHECK (42501). In a fresh transaction, SELECT from ops.check_result is refused (42501), because erp_app holds INSERT only.
    • TestTheNightlyFanOutEnqueuesEveryOrganization: with 2 organizations, working one ChecksFanoutArgs through jobs.NewWorkerClient (the TestTheWorkerClientWorksAJob shape) leaves both organizations with results.
  • Step 2: Run, and watch them fail. go test ./internal/platform/ops/ -count=1

  • Step 3: Implement.

-- 00233_ops_check_result.sql
-- +goose Up
-- V4 (design §3.5). The worker writes one row per check per run, inside the
-- organization's own tenant transaction: erp_app holds INSERT only, and the
-- policy's WITH CHECK pins the row to that tenant. The console reads it only
-- through ops.check_latest (00234). Operational data; not pruned in v1 (S10).
CREATE TABLE ops.check_result (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
org_id UUID NOT NULL REFERENCES public.organization (id),
check_key TEXT NOT NULL CONSTRAINT check_result_key_shape CHECK (check_key ~ '^[a-z][a-z0-9_]{0,62}$'),
passed BOOLEAN NOT NULL,
finding_count INT NOT NULL CONSTRAINT check_result_findings_nonnegative CHECK (finding_count >= 0),
ran_at TIMESTAMPTZ NOT NULL,
CONSTRAINT check_result_passed_means_no_findings CHECK (passed = (finding_count = 0))
);
CREATE INDEX check_result_latest_idx ON ops.check_result (org_id, check_key, ran_at DESC);
ALTER TABLE ops.check_result ENABLE ROW LEVEL SECURITY;
ALTER TABLE ops.check_result FORCE ROW LEVEL SECURITY;
CREATE POLICY org_isolation ON ops.check_result
USING (org_id = current_setting('app.current_org', true)::uuid)
WITH CHECK (org_id = current_setting('app.current_org', true)::uuid);
ALTER TABLE ops.check_result OWNER TO erp_ops_owner;
GRANT USAGE ON SCHEMA ops TO erp_app;
GRANT INSERT ON ops.check_result TO erp_app;
-- The nightly fan-out's list of organizations. erp_app cannot list them
-- under forced RLS, by design. Ids only.
-- +goose StatementBegin
CREATE FUNCTION ops.org_ids() RETURNS SETOF uuid
LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$
SELECT id FROM public.organization ORDER BY id
$$;
-- +goose StatementEnd
ALTER FUNCTION ops.org_ids() OWNER TO erp_ops_owner;
REVOKE ALL ON FUNCTION ops.org_ids() FROM PUBLIC;
GRANT EXECUTE ON FUNCTION ops.org_ids() TO erp_app;
GRANT SELECT (id) ON public.organization TO erp_ops_owner;
-- +goose Down
REVOKE SELECT (id) ON public.organization FROM erp_ops_owner;
DROP FUNCTION ops.org_ids();
REVOKE USAGE ON SCHEMA ops FROM erp_app;
DROP TABLE ops.check_result;

The foreign key’s RI check runs as the referenced table’s owner, and RI checks skip RLS. TestRunChecksRecordsOneRowPerCheck inserting as erp_app is what proves the grant set is enough. If it fails on a permission, stop and report before adding any grant.

checks.go:

// RunChecks runs every registered reconciliation check for one organization
// in its own tenant transaction and records the outcome: key, pass/fail and
// a finding COUNT. Findings name rows (reconcile.Finding.Ref) and carry
// English detail, and neither leaves the tenant, so the console only ever
// sees counts (design §1).
func RunChecks(ctx context.Context, d *db.DB, c clock.Clock, orgID uuid.UUID) error {
return d.InTx(tenancy.WithOrg(ctx, orgID), func(ctx context.Context, tx pgx.Tx) error {
res, err := reconcile.Default().Run(ctx, tx)
if err != nil {
return fmt.Errorf("ops: checks %s: %w", orgID, err)
}
now := c.Now()
for _, r := range res.Checks {
if _, err := tx.Exec(ctx, `
INSERT INTO ops.check_result (org_id, check_key, passed, finding_count, ran_at)
VALUES ($1, $2, $3, $4, $5)`, orgID, r.Key, r.Passed, len(r.Findings), now); err != nil {
return fmt.Errorf("ops: record %s: %w", r.Key, err)
}
}
return nil
})
}
func (w *ChecksFanoutWorker) Work(ctx context.Context, _ *river.Job[ChecksFanoutArgs]) error {
rows, err := w.DB.Pool().Query(ctx, `SELECT org_id FROM ops.org_ids() AS org_id`)
if err != nil {
return fmt.Errorf("ops: org ids: %w", err)
}
ids, err := pgx.CollectRows(rows, pgx.RowTo[uuid.UUID])
if err != nil {
return fmt.Errorf("ops: org ids: %w", err)
}
params := make([]river.InsertManyParams, 0, len(ids))
for _, id := range ids {
params = append(params, river.InsertManyParams{Args: OrgChecksArgs{OrgID: id},
InsertOpts: &river.InsertOpts{Queue: jobs.QueueMaintenance, MaxAttempts: 3}}) // Ruling O6
}
_, err = river.ClientFromContext[pgx.Tx](ctx).InsertMany(ctx, params)
return err
}

OrgChecksWorker.Work calls RunChecks. Register both workers in startExportWorker, and add ops.ChecksPeriodic() to PeriodicJobs in cmd/worker/main.go. Update the doc comment above main to name the new scheduled work, and state that it deletes nothing. Add "organization.id": "SELECT" to opsOwnerGrants. The wiring_test.go for the worker asserts both kinds are registered. Follow the file’s existing shape for the export and prune workers.

  • Step 4: Run. go test ./internal/platform/ops/ ./internal/kernel/db/ ./cmd/worker/ -count=1
  • Step 5: Sabotage. Drop WITH CHECK from the policy. TestEveryTableIsTenantIsolated (now scanning ops) goes red. Then remove the organization.id line from opsOwnerGrants. The grants test goes red. Restore both.
  • Step 6: Commit Operator console T4: ops.check_result and nightly integrity checks per organization (R30)

Task 5: The read-only view functions (V1, V3, V5, V7, V4 latest)

Section titled “Task 5: The read-only view functions (V1, V3, V5, V7, V4 latest)”

Files:

  • Create: db/migrations/00234_ops_views.sql, internal/kernel/db/opsviews_test.go
  • Modify: internal/kernel/db/opsschema_test.go (opsOwnerGrants)

Interfaces (all EXECUTE: erp_operator):

  • ops.org_overview() → (org_id uuid, name text, status text, created_at timestamptz, active_members bigint, last_activity_at timestamptz)

  • ops.check_latest() → (org_id uuid, check_key text, passed boolean, finding_count integer, ran_at timestamptz)

  • ops.compliance_flags() → (org_id uuid, periods_open_past_end bigint, receipt_lines_without_basis bigint)

  • ops.security_summary(p_days integer) → (org_id uuid, entity_kind text, action text, events bigint)

  • ops.job_summary() → (org_id uuid, queue text, kind text, state text, jobs bigint, stuck bigint, oldest_at timestamptz)

  • ops.discarded_jobs() → (job_id bigint, org_id uuid, kind text, finalized_at timestamptz)

  • Step 1: Failing tests — opsviews_test.go. One test per function. Each calls the function as erp_operator (SET LOCAL ROLE), and each has the presence-then-absence shape:

    • org_overview: organizations A and B, seeded as admin, with 2 active members and 1 inactive in A and one audit row. A’s row shows active_members = 2, last_activity_at equal to that row’s occurred_at, and status active. After the admin suspends B, B shows suspended.
    • check_latest: two runs for A. Only the newer run’s rows come back, one per key.
    • compliance_flags:
      • Presence: A has an open period ending yesterday and one receipt line with document_basis IS NULL on a live claim, and the counts are 1 and 1.
      • Absence: closing the period and withdrawing the line gives 0 and 0.
      • Use expensestest fixtures for the claim.
    • security_summary(7): two session/login_failed rows for A, and one from 10 days ago. The count is 2. A non-allowlisted pair (claim/update) is absent while it is present in audit_log (presence, as admin).
    • job_summary: insert river rows as admin with args = {"org_id": A} and states available / running (attempted_at 40 minutes ago). The counts group by org, and stuck = 1. A row with args without org_id groups under NULL.
    • discarded_jobs: a discarded row is listed. A completed one is not.
    • TestOpsViewsReadNoRecordText, one test for S1a in practice:
      1. As admin, show audit_log.after holds a known string (presence).
      2. Concatenate row_to_json of every view function’s output, called as erp_operator.
      3. Assert that string is absent.
  • Step 2: Run, and watch them fail. go test ./internal/kernel/db/ -run 'Ops' -count=1

  • Step 3: Implement 00234_ops_views.sql. Each function is LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp, and is followed by OWNER TO erp_ops_owner, REVOKE ALL … FROM PUBLIC and GRANT EXECUTE … TO erp_operator. The two that are not obvious:

CREATE FUNCTION ops.org_overview()
RETURNS TABLE (org_id uuid, name text, status text, created_at timestamptz,
active_members bigint, last_activity_at timestamptz)
LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$
SELECT o.id, o.name, o.status, o.created_at,
(SELECT count(*) FROM public.organization_member m WHERE m.org_id = o.id AND m.active),
(SELECT a.occurred_at FROM public.audit_log a WHERE a.org_id = o.id ORDER BY a.seq DESC LIMIT 1)
FROM public.organization o
ORDER BY o.created_at
$$;
-- Job args are ids only (db/rls_exempt.txt, River). Only the org id is read,
-- and only when it is a well-formed uuid.
CREATE FUNCTION ops.job_summary()
RETURNS TABLE (org_id uuid, queue text, kind text, state text, jobs bigint, stuck bigint, oldest_at timestamptz)
LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$
SELECT CASE WHEN j.args->>'org_id' ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
THEN (j.args->>'org_id')::uuid END,
j.queue, j.kind, j.state::text, count(*),
count(*) FILTER (WHERE j.state = 'running' AND j.attempted_at < now() - interval '30 minutes'),
min(j.created_at)
FROM public.river_job j
GROUP BY 1, 2, 3, 4
$$;
  • compliance_flags:
    • periods_open_past_end counts fiscal_period rows with status = 'open' AND ends_on < current_date.
    • receipt_lines_without_basis counts expense_claim_line rows with line_type = 'receipt' AND withdrawn_at IS NULL AND document_basis IS NULL.
    • Both are grouped per organization from organization, so a zero shows as 0, not a missing row.
  • security_summary counts audit_log rows by (org_id, entity_kind, action) where occurred_at > now() - make_interval(days => p_days) and the pair is in ('session','login_failed'), ('session','replay_detected'), ('role','permissions_set') or ('organization_member','roles_set'). p_days is clamped to 1..90.
  • check_latest is DISTINCT ON (org_id, check_key) … ORDER BY org_id, check_key, ran_at DESC.

Grants:

GRANT SELECT (name, status, created_at) ON public.organization TO erp_ops_owner;
GRANT SELECT (org_id, active) ON public.organization_member TO erp_ops_owner;
GRANT SELECT (org_id, seq, occurred_at, entity_kind, action) ON public.audit_log TO erp_ops_owner;
GRANT SELECT (org_id, status, ends_on) ON public.fiscal_period TO erp_ops_owner;
GRANT SELECT (org_id, line_type, document_basis, withdrawn_at) ON public.expense_claim_line TO erp_ops_owner;
GRANT SELECT (id, kind, state, queue, args, attempted_at, created_at, finalized_at) ON public.river_job TO erp_ops_owner;

Add every column to opsOwnerGrants. ops.check_result is the owner’s own table and needs no grant.

  • Step 4: Run. go test ./internal/kernel/db/ -count=1
  • Step 5: Sabotage. Change security_summary to return a.after::text AS detail_code. The type test stays green (the name is allowed), so TestOpsViewsReadNoRecordText must go red, and the missing column grant makes the call itself fail with 42501. Read which one fires. That is the point of S1a. Restore.
  • Step 6: Commit Operator console T5: ops view functions over granted columns only

Task 6: cmd/operator and internal/operator: sign-in, sessions, route guard, V8

Section titled “Task 6: cmd/operator and internal/operator: sign-in, sessions, route guard, V8”

Files:

  • Create: cmd/operator/main.go, cmd/operator/wiring_test.go, internal/operator/server.go, internal/operator/auth.go, internal/operator/system.go, internal/operator/templates/{layout,signin,system}.html, internal/operator/server_test.go
  • Modify: internal/kernel/layering/layering_test.go, internal/app/binaries_test.go, .env.example, Makefile (build bin/operator with LDFLAGS)

Interfaces:

  • Produces:

    • operator.Server{DB *pgxpool.Pool; TOTPKey []byte; Clock clock.Clock; Version, APIURL, GrafanaURL, BackupStatusFile string; HTTP *http.Client} and (*Server).Handler() http.Handler.
    • operator.OperatorFromContext(ctx) (Operator, bool), where Operator{ID uuid.UUID; Email string; TokenHash []byte}.
    • (*Server).page(w, name string, data any).
    • (*Server).readTx(ctx, fn func(pgx.Tx) error) error, which runs BEGIN READ ONLY.
    • CLI: operator serve, operator add -email <e> [-reset] (password on stdin, prints the otpauth URI), operator deactivate -email <e>.
  • Consumes: ops_auth.* (Task 2), password.Hash/Verify, totp.Enrol/Seal/Open/Validate, config.PrivateListenAddr.

  • Step 1: Failing tests in server_test.go. The helper newConsole(t) runs testdb.Start. It adds an operator through ops_auth.operator_add as admin, with a known password and a TOTP secret sealed with a test key. It opens an erp_operator pool on h.AdminDSN with AfterConnect: SET ROLE erp_operator and returns httptest.NewServer(s.Handler()).

    • TestEveryConsoleRouteRequiresASignedInOperator:
func TestEveryConsoleRouteRequiresASignedInOperator(t *testing.T) {
c := newConsole(t)
routes := c.srv.Routes() // the table Handler() registers from
require.GreaterOrEqual(t, len(routes), 4, "route table is nearly empty; the guard would judge nothing")
var public []string
for _, r := range routes {
if r.Public {
public = append(public, r.Pattern)
}
}
require.ElementsMatch(t, []string{"GET /signin", "POST /signin"}, public,
"only the sign-in form may be reachable without a session")
// PRESENCE: with a session, a guarded page answers 200.
cookie := c.signIn(t)
require.Equal(t, http.StatusOK, c.get(t, "/system", cookie).StatusCode)
// ABSENCE: without one, every guarded route redirects to sign-in and runs nothing.
for _, r := range routes {
if r.Public {
continue
}
method, path := r.Sample() // pattern with {id} filled by a fresh uuid
res := c.do(t, method, path, nil)
require.Equal(t, http.StatusSeeOther, res.StatusCode, "%s reached its handler without a session", r.Pattern)
require.Equal(t, "/signin", res.Header.Get("Location"))
}
}
  • TestSigninNeedsPasswordAndTOTP:
    • Presence: the right password and a current code → 303 to /, and the cookie is HttpOnly, Secure and SameSite=Strict.
    • The right password with a wrong code, a wrong password with the right code, and an unknown email → 200, the same generic message, and no cookie.
    • As admin, ops_auth.operator_audit has one signin and three signin_failed rows.
  • TestSignoutEndsTheSession: after POST /signout, the same cookie gets 303 to /signin.
  • TestCrossSitePostIsRefused: POST /signout with Origin: https://evil.example and Sec-Fetch-Site: cross-site → 403.
  • TestSystemPageShowsReleaseAndChain: /system shows the applied migration version (00235 or later once Task 8 lands, so assert >= 231), the console version and “chain intact”.
  • cmd/operator/wiring_test.go: serve refuses to boot when OPERATOR_LISTEN_ADDR is 8.8.8.8:8081 or empty, and when TOTP_ENCRYPTION_KEY is missing. 0.0.0.0:8081 is accepted (S3). add refuses a password shorter than 12 characters.
  • internal/app/binaries_test.go: add require.Empty(t, linkedModules(t, repoRoot, "operator"), "cmd/operator must link no module: it reads through ops functions only"), then watch it pass for the right reason. Sabotage: add _ "github.com/mcom/erp/internal/modules/reports" to cmd/operator/main.go, and it goes red.
  • layering_test.go: add {"operator", "internal/operator", "github.com/mcom/erp/internal/operator"} after app.
  • Step 2: Run, and watch them fail. go test ./internal/operator/ ./cmd/operator/ ./internal/app/ ./internal/kernel/layering/ -count=1
  • Step 3: Implement. The route table and guard:
type Route struct {
Pattern string // net/http 1.22 pattern: "GET /orgs/{id}"
Public bool
h http.HandlerFunc
}
func (s *Server) Routes() []Route {
return []Route{
{Pattern: "GET /signin", Public: true, h: s.signinForm},
{Pattern: "POST /signin", Public: true, h: s.signin},
{Pattern: "POST /signout", h: s.signout},
{Pattern: "GET /{$}", h: s.home}, // redirects to /system until Task 7 makes it the overview
{Pattern: "GET /system", h: s.system},
// Tasks 7 and 9 append here; nothing registers outside this table.
}
}
func (s *Server) Handler() http.Handler {
mux := http.NewServeMux()
for _, r := range s.Routes() {
h := http.Handler(r.h)
if !r.Public {
h = s.requireOperator(h)
}
mux.Handle(r.Pattern, h)
}
return http.NewCrossOriginProtection().Handler(securityHeaders(mux))
}
// securityHeaders: nothing here may be cached, framed, or load another origin.
func securityHeaders(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h := w.Header()
h.Set("Cache-Control", "no-store")
h.Set("X-Frame-Options", "DENY")
h.Set("Content-Security-Policy", "default-src 'self'; form-action 'self'; frame-ancestors 'none'")
h.Set("Referrer-Policy", "no-referrer")
next.ServeHTTP(w, r)
})
}

requireOperator reads the op_session cookie and base64url-decodes it into 32 bytes. It calls ops_auth.session_check(sha256) in a short transaction and puts the Operator in the context. Otherwise it redirects 303 to /signin. Sign-in:

func (s *Server) signin(w http.ResponseWriter, r *http.Request) {
email := strings.ToLower(strings.TrimSpace(r.FormValue("email")))
var id uuid.UUID
var hash string
var sealed []byte
err := s.DB.QueryRow(r.Context(),
`SELECT operator_id, password_hash, totp_sealed FROM ops_auth.signin_lookup($1)`, email).Scan(&id, &hash, &sealed)
found := err == nil
if !found {
hash = dummyHash // same argon2 cost for an unknown email, so timing does not enumerate accounts
}
pwOK, _ := password.Verify(r.FormValue("password"), hash)
ok := found && pwOK && s.totpOK(sealed, r.FormValue("code"))
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
th := sha256.Sum256(raw)
var session *uuid.UUID
if err := s.DB.QueryRow(r.Context(), `SELECT ops_auth.signin_result($1, $2, $3)`, email, ok, th[:]).Scan(&session); err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
if session == nil { // wrong credentials, locked, inactive, or inside 90s: one message for all
s.page(w, "signin", map[string]any{"Failed": true, "Email": email})
return
}
http.SetCookie(w, &http.Cookie{Name: "op_session", Value: base64.RawURLEncoding.EncodeToString(raw),
Path: "/", HttpOnly: true, Secure: true, SameSite: http.SameSiteStrictMode})
http.Redirect(w, r, "/", http.StatusSeeOther)
}

dummyHash is a package-level password.Hash of a random string, computed once in init. /system (V8) renders:

  • version
  • ops.release_info()
  • ops_auth.chain_status()
  • the api’s version from GET {OPERATOR_API_URL}/v1/health, with a 2s timeout and “unreachable” on error

cmd/operator/main.go serve reads, each as an os.Getenv literal:

  • OPERATOR_DATABASE_URL (required)
  • OPERATOR_LISTEN_ADDR (required, through config.PrivateListenAddr)
  • TOTP_ENCRYPTION_KEY (required, 32 bytes, through the same decoder config.Load uses; export a config.TOTPKeyFromEnv() if needed rather than duplicating it)
  • OPERATOR_API_URL, OPERATOR_GRAFANA_URL, OPERATOR_BACKUP_STATUS_FILE (optional)

add and deactivate use DATABASE_MIGRATE_URL, because ops_auth.operator_add is granted to no role. add reads the password from stdin, password.Hashes it, runs totp.Enrol("ERP operator console", email) and Seals the secret. It prints the otpauth URI once, on stdout, and writes nothing else there.

.env.example gains OPERATOR_DATABASE_URL (postgres://erp_operator:…), OPERATOR_LISTEN_ADDR=127.0.0.1:8081, OPERATOR_API_URL, OPERATOR_GRAFANA_URL and OPERATOR_BACKUP_STATUS_FILE. Each has a comment: the console is reached by SSH port-forward (S3), and TOTP_ENCRYPTION_KEY is required by the console.

  • Step 4: Run. go test ./internal/operator/ ./cmd/operator/ ./internal/app/ ./internal/kernel/layering/ ./internal/kernel/config/ -count=1
  • Step 5: Sabotage. Mark GET /system Public: true. The guard goes red naming it, and the public-list assertion fires too. Remove s.totpOK from ok. The wrong-code case goes red. Restore both.
  • Step 6: Runtime check. On the dev stack, ALTER ROLE erp_operator PASSWORD 'erp' as postgres. Then echo 'correct horse battery' | go run ./cmd/operator add -email ops@demo.test, then go run ./cmd/operator serve. Sign in in a browser at http://127.0.0.1:8081 with a TOTP app, and /system shows the migration version.
  • Step 7: Commit Operator console T6: cmd/operator with password+TOTP sign-in, guarded routes, releases page

Files:

  • Create: internal/operator/orgs.go, internal/operator/jobs.go, internal/operator/templates/{orgs,org,jobs}.html, internal/operator/pages_test.go
  • Modify: internal/operator/server.go (routes), internal/operator/system.go and templates/system.html (V2, V6)

Interfaces:

  • Produces: routes GET /{$}, GET /orgs/{id} and GET /jobs. /system gains V2 and V6.

  • Consumes: Task 5’s functions, ops.check_latest, and the backup status file format {"last_backup_at": RFC3339, "last_restore_test_at": RFC3339, "wal_archive_lag_seconds": int}.

  • Step 1: Failing tests — pages_test.go:

    • TestOrgsPageShowsEveryOrganizationWithItsFlags: two organizations. A has a failing entry_balance check result, inserted as admin, and B’s checks all pass. / lists both names. A’s row is marked failing (data-state="fail" on the checks cell) and B’s is marked pass. A check run older than 48 hours is marked stale.
    • TestOrgPageShowsChecksSecurityAndJobs: /orgs/{A} lists each check key with pass/fail and ran_at, the security counts and the job counts for A only. Presence: B’s job row exists in ops.job_summary(). Absence: B’s job row is not on A’s page.
    • TestSystemPageReadsTheBackupStatusFile: with a file 2 days old, it shows the date. With no file, it shows “unknown” marked fail. With OPERATOR_API_URL pointing at an httptest server that answers /v1/ready 503, it shows “not ready”.
    • TestNoConsolePageShowsRecordContents (design §1, PDPA). Seed through expensestest a member with a distinctive legal name and email, and a claim line with a distinctive description. Presence: as admin, each value is in the database. Absence: GET /, /orgs/{id}, /jobs and /system, and none of the bodies contains any of the three values or the claim amount.
  • Step 2: Run, and watch them fail. go test ./internal/operator/ -count=1

  • Step 3: Implement. orgs.go calls org_overview, check_latest, compliance_flags and security_summary(7) in one readTx. It merges them by org_id into:

type orgRow struct {
ID uuid.UUID
Name, Status string
CreatedAt time.Time
ActiveMembers int64
LastActivityAt *time.Time
ChecksPassed, Checks int
ChecksState string // "pass", "fail", "stale", "none"
OpenPeriodsPastEnd int64
ReceiptsWithoutBasis int64
FailedLogins7d int64
}

ChecksState is stale when the newest ran_at is older than 48 hours, even if every check passed. The templates are plain HTML tables with data-state attributes and a small inline stylesheet in layout.html (red, amber, green). There is no JavaScript. /jobs shows job_summary grouped by queue and kind, then the discarded_jobs list. The retry button is added in Task 9. /system adds:

  • GET {OPERATOR_API_URL}/v1/ready → ready / not ready (V2)
  • a link to OPERATOR_GRAFANA_URL when set
  • the backup file’s three values, with “unknown” when the file is missing or unreadable, and fail when the last backup is older than 26 hours or the last restore test older than 92 days (RUNBOOK §3.1, quarterly)
  • Step 4: Run. go test ./internal/operator/ -count=1
  • Step 5: Sabotage. Add {{.Email}} of a member to org.html, fed by a temporary SELECT email FROM users as admin. TestNoConsolePageShowsRecordContents goes red. Restore.
  • Step 6: Commit Operator console T7: organizations, jobs and system pages (V1-V7), counts and pass/fail only

Task 8: Operator actions in the worker (A2, A3, A4, A5)

Section titled “Task 8: Operator actions in the worker (A2, A3, A4, A5)”

Files:

  • Create: db/migrations/00235_ops_actions.sql, internal/platform/ops/actions.go, internal/platform/ops/actions_test.go
  • Modify: cmd/worker/exports.go, cmd/worker/main.go (blank-import every module), cmd/worker/wiring_test.go, internal/app/binaries_test.go (worker joins migrate, orgadmin), db/rls_exempt.txt (ops_auth.operator_action), internal/kernel/db/opsschema_test.go

Interfaces:

  • Produces:

    • ops_auth.operator_action.
    • ops.request_action(p_session bytea, p_kind text, p_org uuid, p_owner_member uuid, p_job bigint) → uuid (erp_operator)
    • ops.claim_action(p_id uuid) → (kind text, org_id uuid, owner_member_id uuid, job_id bigint) (erp_app)
    • ops.finish_action(p_id uuid, p_ok boolean, p_detail text) → void (erp_app)
    • ops_auth.action_log(p_limit integer) → (action_id uuid, operator_email text, kind text, org_id uuid, job_id bigint, state text, detail_code text, requested_at timestamptz, finished_at timestamptz) (erp_operator)
    • Job kind operator_action (ops.ActionArgs{ActionID uuid.UUID}) and ops.ActionWorker{DB *db.DB; Clock clock.Clock}.
    • Detail codes: owner_member_required, company_profile_required, owner_member_invalid, already_suspended, already_active, job_not_retryable, error.
  • Step 1: Failing tests — actions_test.go. Each creates a signed-in operator session as in Task 6, via SQL as admin, and requests through ops.request_action as erp_operator. It then works the queue with a real jobs.NewWorkerClient over h.App, registering ActionWorker, with a 50ms poll.

    • TestARequestedActionIsWorkedByRiver: requesting run_checks for A leaves one river_job row of kind operator_action. After the client runs, the action is succeeded and A has ops.check_result rows. This test is what proves Ruling O5’s raw insert.
    • TestSuspendAndReactivate:
      1. Presence: A is active, and an erp_app tenant transaction can update A’s company profile.
      2. After suspend is worked, organization.status is suspended.
      3. A’s audit_log has entity_kind = 'organization', action = 'suspended', actor_id IS NULL, and after containing the action id.
      4. SELECT count(*) of A’s claims is unchanged, so nothing was deleted.
      5. reactivate restores active and writes reactivated.
      6. Suspending twice ends failed with already_suspended and writes no second audit row.
    • TestProvisionRunsTheRegistry: on a fully provisioned organization, provision without an owner ends succeeded and writes audit_log action = 'provisioned'. On one missing the org_structure step (expensestest has a helper, or delete that step’s marker the way internal/platform/provisioning tests do), it ends failed with owner_member_required.
    • TestRetryJobOnlyRetriesDiscarded:
      • Presence: a discarded export job, inserted as admin, can be requested. After working, River shows it available or retryable.
      • Refusals: request_action for a completed job, and for a job of kind operator_action, is refused with the constraint name operator_action_job_retryable.
    • TestRequestActionAttributesTheSessionsOperator:
      • Presence: request_action with a live session succeeds, and action_log names that operator’s email.
      • With a revoked session hash it raises 42501.
      • A second operator’s session is recorded as the second operator.
    • TestTheWorkerLinksEveryModule: binaries_test.go’s loop becomes {"migrate", "orgadmin", "worker"}.
  • Step 2: Run, and watch them fail. go test ./internal/platform/ops/ ./internal/app/ -run 'Action|Suspend|Provision|Retry|Binary' -count=1

  • Step 3: Implement 00235_ops_actions.sql:

-- +goose Up
CREATE TABLE ops_auth.operator_action (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
operator_id UUID NOT NULL REFERENCES ops_auth.operator_account (id),
kind TEXT NOT NULL CONSTRAINT operator_action_kind_known
CHECK (kind IN ('provision','run_checks','retry_job','suspend','reactivate')),
org_id UUID,
owner_member_id UUID,
job_id BIGINT,
state TEXT NOT NULL DEFAULT 'requested' CONSTRAINT operator_action_state_known
CHECK (state IN ('requested','running','succeeded','failed')),
detail_code TEXT CONSTRAINT operator_action_detail_shape CHECK (detail_code ~ '^[a-z_]{1,40}$'),
requested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
finished_at TIMESTAMPTZ,
CONSTRAINT operator_action_target_shape CHECK (
CASE WHEN kind = 'retry_job' THEN job_id IS NOT NULL AND org_id IS NULL
ELSE org_id IS NOT NULL AND job_id IS NULL END),
CONSTRAINT operator_action_owner_only_for_provision CHECK (owner_member_id IS NULL OR kind = 'provision')
);
ALTER TABLE ops_auth.operator_action OWNER TO erp_ops_owner;
-- +goose StatementBegin
CREATE FUNCTION ops.request_action(p_session bytea, p_kind text, p_org uuid, p_owner_member uuid, p_job bigint)
RETURNS uuid LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$
DECLARE
v_operator uuid := ops_auth.session_operator(p_session);
v_id uuid;
BEGIN
-- Attribution comes from the session, never from a parameter (Ruling O3).
IF v_operator IS NULL THEN
RAISE EXCEPTION 'ops.request_action: no live operator session' USING ERRCODE = '42501';
END IF;
IF p_kind = 'retry_job' AND NOT EXISTS (
SELECT 1 FROM public.river_job WHERE id = p_job AND state = 'discarded' AND kind <> 'operator_action') THEN
RAISE EXCEPTION 'job % is not a discarded job', p_job
USING ERRCODE = '23514', CONSTRAINT = 'operator_action_job_retryable';
END IF;
IF p_kind <> 'retry_job' AND NOT EXISTS (SELECT 1 FROM public.organization WHERE id = p_org) THEN
RAISE EXCEPTION 'organization % does not exist', p_org
USING ERRCODE = '23514', CONSTRAINT = 'operator_action_org_exists';
END IF;
INSERT INTO ops_auth.operator_action (operator_id, kind, org_id, owner_member_id, job_id)
VALUES (v_operator, p_kind, p_org, p_owner_member, p_job) RETURNING id INTO v_id;
PERFORM ops_auth.audit_append(v_operator, 'action_requested', p_org, v_id);
-- River's columns: kind, args, queue, max_attempts; the rest default
-- (00111-00116). The worker is poll-only (kernel/jobs), so no NOTIFY.
-- Ruling O5.
INSERT INTO public.river_job (kind, args, queue, max_attempts)
VALUES ('operator_action', jsonb_build_object('action_id', v_id), 'maintenance', 1);
RETURN v_id;
END
$$;
-- +goose StatementEnd
  • claim_action is UPDATE ops_auth.operator_action SET state = 'running' WHERE id = p_id AND state = 'requested' RETURNING kind, org_id, owner_member_id, job_id, so a replayed job claims nothing.
  • finish_action sets state to succeeded or failed, and sets detail_code and finished_at, WHERE id = p_id AND state = 'running'.
  • action_log joins operator_account for the email, newest first, and clamps the limit to 1..200.

Grants: GRANT INSERT (kind, args, queue, max_attempts) ON public.river_job TO erp_ops_owner; and GRANT USAGE ON SEQUENCE public.river_job_id_seq TO erp_ops_owner;. Add both to opsOwnerGrants. The test’s column query covers INSERT column ACLs. Extend it with a pg_class sequence check if the sequence grant is not visible in attacl, and give it its own line in the map, keyed river_job_id_seq.

actions.go:

func (w *ActionWorker) Work(ctx context.Context, j *river.Job[ActionArgs]) error {
var a struct {
Kind string
OrgID *uuid.UUID
OwnerMemberID *uuid.UUID
JobID *int64
}
err := w.DB.Pool().QueryRow(ctx, `SELECT kind, org_id, owner_member_id, job_id FROM ops.claim_action($1)`,
j.Args.ActionID).Scan(&a.Kind, &a.OrgID, &a.OwnerMemberID, &a.JobID)
if errors.Is(err, pgx.ErrNoRows) {
return nil // already claimed: a replayed job does nothing
}
if err != nil {
return fmt.Errorf("ops: claim %s: %w", j.Args.ActionID, err)
}
detail, runErr := w.run(ctx, j.Args.ActionID, a.Kind, a.OrgID, a.OwnerMemberID, a.JobID)
if runErr != nil && detail == "" {
detail = "error"
}
if _, err := w.DB.Pool().Exec(ctx, `SELECT ops.finish_action($1, $2, $3)`,
j.Args.ActionID, runErr == nil, nullIfEmpty(detail)); err != nil {
return fmt.Errorf("ops: finish %s: %w", j.Args.ActionID, err)
}
return nil // the outcome is recorded; River must not retry an operator action
}

run dispatches on the kind:

  • run_checks → RunChecks.
  • retry_job → river.ClientFromContext[pgx.Tx](ctx).JobRetry(ctx, *jobID).
  • suspend / reactivate → in db.InTx(tenancy.WithOrg(ctx, org)), run UPDATE organization SET status = $2 WHERE id = $1 AND org_id = $1 AND status <> $2. If 0 rows, return already_suspended or already_active with an error. Otherwise audit.Write {EntityKind: "organization", EntityID: org, Action: "suspended"|"reactivated", After: map[string]any{"by": "service_provider", "operator_action_id": id}}, last.
  • provision → in db.InTx, run provisioning.CheckOwnerMember when an owner is named, then provisioning.Run(ctx, tx, w.Clock, provisioning.Input{OwnerPermissions: member.RegisteredPermissionSet(), OwnerMemberID: owner}), then audit.Write action: "provisioned", with the same After plus the done step keys. Map ErrOwnerMemberRequired, ErrCompanyProfileRequired and ErrOwnerMemberNotFound to their codes with errors.Is, as internal/modules/identity/handlers.go provision does.

Register ActionWorker in startExportWorker. cmd/worker/main.go blank-imports the modules it does not yet import: activity, identity, orgstructure, payout, reports, search, workflow. Add the same comment cmd/orgadmin/main.go gives: the permission registry is filled by init, and a worker-run provision must seed the same Owner. db/rls_exempt.txt gains ops_auth.operator_action, under the Task 2 comment.

  • Step 4: Run. go test ./internal/platform/ops/ ./internal/kernel/db/ ./internal/app/ ./cmd/worker/ -count=1
  • Step 5: Sabotage. First, make request_action skip the v_operator IS NULL refusal, and the revoked-session case goes red. Second, drop the kind <> 'operator_action' clause, and the retry refusal case goes red. Third, make the suspend branch return nil when the UPDATE matched 0 rows, and the second-suspend case goes red. Restore each.
  • Step 6: Commit Operator console T8: operator actions run in the worker, org audit_log mirrored (A2-A5)

Files:

  • Create: internal/operator/actions.go, internal/operator/actions_test.go, internal/operator/templates/{confirm,actions}.html
  • Modify: internal/operator/server.go (routes), templates/org.html (buttons), templates/jobs.html (retry buttons)

Interfaces:

  • Produces: routes GET /orgs/{id}/actions/{kind} (confirmation page), POST /orgs/{id}/actions/{kind}, POST /jobs/{id}/retry and GET /actions.

  • Consumes: ops.request_action and ops_auth.action_log.

  • Step 1: Failing tests:

    • TestSuspendNeedsConfirmationAndIsRecorded: the GET confirmation page names the organization and the action, and has no side effect. As admin, operator_action stays empty (presence of the table is shown by the next step). The POST leads with 303 to /actions, one requested row and one action_requested audit row.
    • TestUnknownActionKindIs404: POST /orgs/{id}/actions/delete → 404, and no row.
    • TestRetryRefusalIsShownNotCrashed: retrying a completed job → the confirmation page re-renders with “This job cannot be retried: only discarded jobs can.” and returns 422, not 500. The mapping keys on the constraint name operator_action_job_retryable from *pgconn.PgError.
    • TestActionsPageShowsOutcome: after Task 8’s worker finishes a run_checks, /actions shows succeeded.
    • TestEveryConsoleRouteRequiresASignedInOperator (Task 6) now covers the new routes. Confirm its route count grew.
  • Step 2: Run, and watch them fail. go test ./internal/operator/ -count=1

  • Step 3: Implement. The allowed kinds from the path are provision, run_checks, suspend and reactivate. Anything else is 404. provision accepts an optional owner_member_id form field, parsed as a uuid; if it is malformed, the page re-renders with 422. The POST calls SELECT ops.request_action($1, $2, $3, $4, $5) with the session’s TokenHash from the context. Confirmation pages state what the action does in one sentence, in English:

    • suspend: “Members keep read and export access; every change is refused until reactivated. Nothing is deleted.”
    • provision: “Runs the provisioning steps that are not done yet; steps already done are left alone.”

    /actions renders ops_auth.action_log(100). The org page shows Suspend or Reactivate by status, plus Run checks now and Re-run provisioning. The jobs page shows Retry on each discarded row.

  • Step 4: Run. go test ./internal/operator/ ./internal/platform/ops/ -count=1

  • Step 5: Sabotage. Make the GET confirmation call request_action. The no-side-effect assertion goes red. Restore.

  • Step 6: Commit Operator console T9: action confirmation, forms and action log


Task 10: Dev stack, end to end, runbook and deploy notes, roadmap, full suite

Section titled “Task 10: Dev stack, end to end, runbook and deploy notes, roadmap, full suite”

Files:

  • Modify: scripts/dev-stack.sh, web/playwright.config.ts, docs/RUNBOOK.md, docs/DEPLOY_NOTES.md, docs/ROADMAP.md, docs/superpowers/specs/2026-09-25-operator-console-progress.md
  • Create: web/e2e/totp.ts, web/e2e/operator-console.spec.ts

Interfaces:

  • Consumes: everything above; as, api and runTag from web/e2e/support.ts.

  • Produces: <STATE>/operator.json = {"email","password","otpauth"} (mode 600), written by dev-stack.sh.

  • Step 1: Parameterise the stack for lane 6 (skip whatever another Phase 2 item has already landed). In dev-stack.sh, PG_PORT, MINIO_PORT, API_PORT, WEB_PORT and OPERATOR_PORT become ${VAR:-default}, with the defaults unchanged: 55432, 59000, 3000, 5173 and 8081. The container names and STATE take a DEV_STACK_NAME suffix. PORT, PUBLIC_BASE_URL and the health URL use the variables. After migrate up, run docker exec "$PG" psql -U postgres -d erp -c "ALTER ROLE erp_operator PASSWORD 'erp'". If TOTP_ENCRYPTION_KEY is empty in $STATE/env, append a generated one. Build and start cmd/operator with OPERATOR_LISTEN_ADDR=127.0.0.1:$OPERATOR_PORT and OPERATOR_API_URL=http://127.0.0.1:$API_PORT. On first seed, create ops@demo.test with a generated password, and write $STATE/operator.json. playwright.config.ts reads E2E_WEB_PORT and E2E_API_PORT, defaulting to today’s values, and passes the variables to the webServer command.

  • Step 2: Write the e2e — web/e2e/totp.ts:

import { createHmac } from 'node:crypto';
/** RFC 6238 TOTP (SHA-1, 6 digits, 30 s), what identity/totp validates. */
export function totp(secretBase32: string, at = Date.now()): string {
const alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567';
let bits = '';
for (const c of secretBase32.replace(/=+$/, '').toUpperCase())
bits += alphabet.indexOf(c).toString(2).padStart(5, '0');
const key = Buffer.from((bits.match(/.{8}/g) ?? []).map((b) => Number.parseInt(b, 2)));
const counter = Buffer.alloc(8);
counter.writeBigUInt64BE(BigInt(Math.floor(at / 30_000)));
const h = createHmac('sha1', key).update(counter).digest();
const o = h[h.length - 1] & 0xf;
return String((h.readUInt32BE(o) & 0x7fffffff) % 1_000_000).padStart(6, '0');
}

operator-console.spec.ts covers the main path:

  1. Read operator.json. Open http://127.0.0.1:${E2E_OPERATOR_PORT}/signin, fill the email, password and totp(secret), and expect “Demo Sdn Bhd” in the organizations table.
  2. Presence for the PDPA check: as the seeded owner in the web app, the profile page shows owner@demo.test. Absence: the console’s / and /orgs/{id} page text contains neither owner@demo.test nor Aisyah binti Rahman.
  3. Open the org page, then Suspend, then Confirm. Reload /actions until the row shows succeeded (poll up to 20 seconds).
  4. In the web app as the owner, a read page (claims list) loads. Then the first write action from claim-flow.spec.ts’s opening step fails, and the toast shows the en org_suspended sentence.
  5. In the console, Reactivate. Once succeeded, the same write succeeds.
  6. Run checks now, then / shows the organization’s checks as pass.

Run on lane 6: DEV_STACK_NAME=lane6 PG_PORT=56472 MINIO_PORT=59160 API_PORT=3160 WEB_PORT=5197 OPERATOR_PORT=3161 E2E_WEB_PORT=5197 E2E_API_PORT=3160 E2E_OPERATOR_PORT=3161 pnpm -C web exec playwright test operator-console, then DEV_STACK_NAME=lane6 scripts/dev-stack.sh down.

  • Step 3: RUNBOOK. Add §7 Operator console:

    • Access: the SSH port-forward (ssh -N -L 8081:127.0.0.1:8081 ops@vps) and the authorized_keys restriction line (S3).
    • Operators: operator add / -reset / deactivate, run with DATABASE_MIGRATE_URL. The otpauth URI is a credential shown once. Another operator’s reset is the only recovery path.
    • The backup status file format and who writes it (I14).
    • What suspension does (S2).
    • The operator audit chain: kept seven years, never pruned, and verified on /system.

    Then write the infrastructure checklist I1–I15, copied from the progress file §4.1, one - [ ] line each. Each line names the Stage 0 decision it follows (I2 → S3, I7 → S4). Add to §1.1’s object-lock line: “the bypass permission (s3:BypassGovernanceRetention) is held only by a break-glass MinIO identity that no service uses. Record every use (date, person, reason) (S4)”. Extend the §2 configuration table with an operator column, the METRICS_LISTEN_ADDR row and the OPERATOR_* rows.

  • Step 4: DEPLOY_NOTES. Add a newest-first entry, “Operator console”:

    • Migrations 00230–00235 create the roles erp_ops_owner / erp_operator and the schemas ops / ops_auth, and add organization.status (every existing organization becomes active).
    • Set a password: ALTER ROLE erp_operator PASSWORD '<secret>'. The role ships without one and cannot connect until it has one.
    • The migration role must be able to create a BYPASSRLS role. That needs a superuser, which the .env.example DATABASE_MIGRATE_URL already is. Say what breaks if it is not: 00230 fails.
    • The new operator binary and its env vars (TOTP_ENCRYPTION_KEY is now required for it). Create the first operator with operator add.
    • METRICS_LISTEN_ADDR is optional.
    • The worker now runs nightly checks and operator actions on the maintenance queue, and now links every module.
    • Nothing to re-provision.
  • Step 5: ROADMAP. Under “Operator console”, Stage 0 decisions is already ticked. Tick Plan and build, and add the e2e path (web/e2e/operator-console.spec.ts) and the drift tests (internal/kernel/db/opsschema_test.go). Add open sub-items: “Stage 5 support access — own design (S9)” and “A1 create organization in the console (S8)”. Where R30 is listed under Plan 9, mark “nightly reconciliation” done by this item. The BNM sync and idempotency pruner items stay open. Update the progress file’s status line to “Stages 1–4 built”.

  • Step 6: Full suite, once. Run go test -p 2 -count=1 ./..., make lint, and make web (typecheck, lint, test, build). Then make generate, followed by a clean git status.

  • Step 7: Commit Operator console T10: dev stack and e2e on lane 6, runbook infra checklist, deploy notes, roadmap


Tasks in the same wave touch disjoint files and can run in parallel. A later wave starts when every task it depends on has merged.

Wave Tasks Depends on Why this order
1 T1, T2 — T1 is kernel, cmd and Makefile. T2 is migrations 00230–00231 and internal/kernel/db tests. Disjoint.
2 T3, T4, T6 T2 (range registered, schemas, opsschema_test.go), and T1 for T6’s Makefile, .env.example and PrivateListenAddr T3 is migration 00232, liveness, httpx auth, openapi and web. T4 is 00233, platform/ops, cmd/worker and opsOwnerGrants. T6 is cmd/operator, internal/operator, layering and binaries tests. Only T4 edits opsschema_test.go.
3 T5 T3 (organization.status), T4 (check_result) Alone: it edits opsOwnerGrants.
4 T7, T8 T5, T6 (T7); T3, T4, T5 (T8) T7 is internal/operator pages only. T8 is 00235, platform/ops/actions.go, cmd/worker, binaries_test.go, rls_exempt.txt and opsschema_test.go. Disjoint.
5 T9 T7, T8 Console action UI.
6 T10 all Stack, e2e, docs, full suite.

Dependencies on other Phase 2 items: none for interfaces. Two things touch shared ground:

  • If another item parameterises scripts/dev-stack.sh for lanes first, T10 Step 1 reuses it.
  • The payment-batches range (00330–00339) overlaps AP’s registered {300, 339} in migrationranges_test.go. That is not this plan’s to fix, but T2’s edit to that file must not reformat the table.

  • Spec coverage:

    Design item Task
    Stage 1 T1
    Stage 2 T2, T6
    Stage 3 V1, V3, V5, V7 in T5 and T7; V4 in T4, T5 and T7; V6 and V2 in T7; V8 in T6
    Stage 4 A2–A5 in T8 and T9; A5 enforcement in T3
    §6 drift tests Each is in T1, T2 or T6, plus the owner-grants and no-record-text tests

    A1, A6, A7 and Stage 5 are deferred (S8, S9), and the roadmap records them.

  • Invariants:

    • No float touches money. There is none.
    • The one business table is RLS-forced.
    • Every request runs in a transaction. Console pages use readTx, and API writes are unchanged.
    • Vocabulary values are strings: status, kind and state.
    • The ledger is untouched.
    • Accounting is not referenced by state name.
    • Translation rows: the only customer-facing string is translated.
    • “Nobody grants themselves authority”: no operator path grants access to an organization. Support access is out.
  • Unverified here, verified by the named test:

    • River accepting a raw-inserted row (T8 TestARequestedActionIsWorkedByRiver).
    • The FK on ops.check_result under the owner’s grants (T4 Step 3 note).
    • Column ACLs covering INSERT column grants and the sequence (T8 Step 3 note).
    • Chrome and Firefox accepting a Secure cookie on http://127.0.0.1 (T6 Step 6 runtime check). They do on localhost. Check the literal IP in the runtime step, and use http://localhost:<port> in the RUNBOOK if not.