Operator Console Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: The provider’s operator signs in to a private, server-rendered console with a password and TOTP. The console shows, for every organization, its status, active member count, last activity, the latest pass/fail of each integrity check, compliance counts, security event counts and job counts. It also shows service health, backups and the release. From it, the operator can re-run provisioning, run the checks now, retry a discarded job, and suspend or reactivate an organization. The console shows counts and pass/fail only, never record contents. Every sign-in and action is written to a hash-chained operator audit trail. Every action that affects an organization is also written to that organization’s own audit_log.
Architecture: The console is a fifth binary, cmd/operator, with its handlers and templates in internal/operator (html/template and stdlib net/http). It listens on loopback and connects as the new role erp_operator, which holds EXECUTE and no table privilege. It reads across tenants only through ops.* SECURITY DEFINER functions. Those functions are owned by erp_ops_owner, a NOLOGIN BYPASSRLS role that holds column-level SELECT on the named columns and nothing else. The operator’s own identity, sessions, audit chain and action log live in schema ops_auth. Actions reach the worker as one River job kind, operator_action, and internal/platform/ops runs each one in the target organization’s tenant transaction. The same package runs the nightly integrity checks into ops.check_result, which closes R30’s nightly reconciliation. Suspension is read-only: liveness reads organization.status, and httpx.Authenticate refuses writes. cmd/api and cmd/worker expose Prometheus /metrics on a private listener, with route-template labels only.
Tech Stack: Go 1.26 · Gin (api) · stdlib net/http + html/template + http.CrossOriginProtection (console) · pgx v5 · goose · River v0.47 · prometheus/client_golang (new) · testify · testcontainers (Postgres 17) · React 19 + i18next (one refusal string) · Playwright
Spec: docs/superpowers/specs/2026-09-25-operator-console-design.md. The Stage 0 decisions S1–S10 are binding and are cited below as Sn.
Global Constraints
Section titled “Global Constraints”Every task’s requirements implicitly include this section.
- Migrations
00230–00249only. This plan uses00230–00235. Task 2 registers{230, 249, "Operator console"}ininternal/kernel/db/migrationranges_test.goin the same commit as00230. goose runs with allow-missing (cmd/migrate/main.gorunGoose), so every migration here must commute with every higher-numbered one: add, never reorder or rename, anything00250+ created. - No money here. The console computes no amounts. Counts are
bigint/int. Nofloat64anywhere ininternal/operatororinternal/platform/ops, and a latency histogram’s seconds (float64inside client_golang) never carries an amount. - Tables.
ops.check_resultis the one business table. It hasorg_id UUID NOT NULL,ENABLEandFORCE ROW LEVEL SECURITY, and a policy namedorg_isolationwithUSINGandWITH CHECKoncurrent_setting('app.current_org', true)::uuid. Theops_auth.*tables are the operator’s own data, carry noorg_id, and are declared indb/rls_exempt.txtwith a reason. EveryCHECK/UNIQUEisCONSTRAINT <table>_<rule>(TestEveryConstraintIsNamed). - Functions in
opsandops_authare owned byerp_ops_owner(ALTER FUNCTION … OWNER TO erp_ops_owner) and declaredSET search_path = pg_catalog, public, pg_temp. They schema-qualify every table (public.audit_log,ops_auth.operator_session). Each migration runsREVOKE ALL ON FUNCTION … FROM PUBLIC, thenGRANT EXECUTEto exactly the role that calls the function (erp_operatorfor the console,erp_appfor the worker, none for the CLI, which runs as the migration role).TestEveryOpsFunctionIsOwnedPinnedAndNotPublic(Task 2) enforces this. opsreturns aggregates only (S1c). A column returned by anopsfunction isuuid,bigint,integer,boolean,timestamptzordate, ortextwhose name is one ofname,status,kind,state,queue,check_key,entity_kind,action,outcome,detail_code. There is nojsonb.TestOpsFunctionsReturnAggregatesOnly(Task 2) enforces this.ops_authfunctions may return operator data (emails, hashes), and they read nopublictable exceptriver_job(theopsaction functions do that).erp_ops_ownergrants are column-level and exact (S1a). A task that makes anopsfunction read a new column grants that column in its migration and adds it toopsOwnerGrantsininternal/kernel/db/opsschema_test.goin the same commit. Nothing is granted table-levelSELECT, and nothing getsUPDATE,DELETEorTRUNCATEonpublic.- Counts and pass/fail only (PDPA, design §1). No console page renders a member’s name, email or payout detail, a record’s text, an amount, or a document. Task 7’s
TestNoConsolePageShowsRecordContentsshows the seeded values exist, then shows they are absent from every page’s HTML. - The console is not in
api/openapi.yaml(design §3.2). It is covered by its own route test (Task 6). The one customer-API change, theorg_suspendedrefusal code (Task 3), is documented inapi/openapi.yamlfirst, thenmake generate, then a cleangit statuson generated files. - i18n. The console is English only (S10). Every ERP web string added (Task 3’s refusal) is in
web/src/lib/i18n/locales/{en,ms,zh}/common.json, in genuine Malay.web/src/drift/i18nparity.test.tsmust pass. - Ledger untouched. Nothing here writes a journal line or registers a posting rule.
accounting/is read only throughreconcile.Default().Run. - One org per transaction. Worker code runs org work in
db.InTx(tenancy.WithOrg(ctx, orgID), …). Cross-org work happens only insideops.*functions, or in a pool transaction with no tenant that calls onlyops.*(thenotifications.PruneWorkershape). No nestedInTx. - Lock order in a worker action’s transaction: the action’s own work (
provisioning.Run, or theorganizationrowUPDATE) comes first, and itsaudit.Writecomes last (the audit-chain class-1 rule,internal/kernel/audit/audit.go). The operator audit chain usesLOCK TABLE ops_auth.operator_audit, not an advisory lock, so the advisory lock class registry is unchanged. - Presence before absence (
AGENTS.md). Every refusal and absence test shows the accepted or present case first. Every guard gets a Sabotage step: break it, run the test, read the red message, restore, and record in the commit message what was broken. - Acting as the operator role in tests:
tx.Exec(ctx, "SET LOCAL ROLE erp_operator")inside anh.Admintransaction. Do not add a password to the role in a migration (S3, Task 2). Usetestdb.Start. Roles are created by the migrations once per shared container, and nothing here needsStartIsolated. - Tests: run only the touched packages while iterating. Run one full
go test -p 2 -count=1 ./...in Task 10 only. - E2E lane 6: Postgres
56472, MinIO59160, api3160, vite5197, console3161. Never the user’s dev stack (55432/59000/3000/5173). - Never bare
git stash. Use a WIP commit. - Git commits end with:
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rulings
Section titled “Rulings”Rulings on where the design is silent or cannot be built as written. Format:
Ruling: <decision> — <why>.
- O1 Ruling: Schemas
ops(cross-tenant aggregates, actions,check_result) andops_auth(operator identity, sessions, audit, action log) — S1 item 2. The aggregates-only type test then applies cleanly toops. - O2 Ruling: The existing
public-only guards stay as they are. A new file,internal/kernel/db/opsschema_test.go, holds four guards for the two new schemas.TestEveryTableIsTenantIsolatedandTestExemptionFilesNameRealTablesare widened to scanopsandops_auth, with schema-qualified names (ops_auth.operator_account). A separate file keeps each test’s review local. - O3 Ruling: The
erp_operatordatabase credential is equivalent to operator power.ops_auth.signin_resulttrusts the console’s password and TOTP verdict, because argon2id verification is in Go (identity/password). That is why the credential is held only by thecmd/operatorprocess (I15). Action attribution does not trust the console:ops.request_actiontakes the session token hash and resolves the operator itself, so a console bug cannot act as the wrong operator. - O4 Ruling: The operator audit chain has no recorded head. Rows cut off the end of the chain leave a shorter chain that still verifies, the same limit
audit.VerifyChainhas without a period close. The console shows the row count and headseq. Record a head externally only if an auditor asks. —// ponytail:comment in the migration. - O5 Ruling:
ops.request_actioninserts the River row with raw SQL (kind, args, queue, max_attempts, the rest defaulted). Two tests guard it: River’s worker client works a row inserted this way (Task 8), andTestRiverAgreesTheSchemaIsCurrentalready fails on a River schema change. The alternative, givingerp_operatorRiver’s insert privileges, would exposeriver_job.errorsto the console. - O6 Ruling: The nightly fan-out enqueues one
ops_org_checksjob per organization and does not dedupe. A restart re-runs a night’s checks, which is harmless extra rows.max_attemptsis 3: a check that errors leaves the organization stale on V4, which is visible, rather than retrying 25 times. - O7 Ruling: Writes allowed while suspended are
suspendedWritableininternal/kernel/httpx/auth.go:POST /v1/auth/switch-org,DELETE /v1/auth/sessions/:id,POST /v1/reports/exports,POST /v1/me/notifications/:id/read,POST /v1/me/notifications/read-all(S2).TestSuspendedWritableNamesRealRoutesininternal/appshows each entry is a route the real router registers. - O8 Ruling: One console page per concern.
/shows V1 with V4, V5 and V7 summarised per organization./orgs/{id}shows the detail and the actions./jobsshows V3 and A4./systemshows V2, V6 and V8 and the operator chain./actionsis the action log. Charts are Grafana’s (design §3.6), linked from/system. - O9 Ruling: A sign-in within 90 seconds of the account’s previous successful one is refused (S6). That makes a TOTP code single-use without storing steps. Local e2e reruns inside 90 seconds fail by design. Wait, or use a fresh stack.
- O10 Ruling:
METRICS_LISTEN_ADDRandOPERATOR_LISTEN_ADDRgo throughconfig.PrivateListenAddr(S3). It accepts loopback, private and unspecified IP literals, and refuses public addresses, hostnames and an empty host (:9100).METRICS_LISTEN_ADDRempty means metrics are off.OPERATOR_LISTEN_ADDRis required. - O11 Ruling: Build version via
-ldflags "-X main.version=$(VERSION)"forapi,workerandoperatorin the Makefile’sbuildtarget. The worker and the console declarevar version = "dev"likecmd/api.erp_build_info{version}is the one gauge that carries aversionlabel.
File Structure
Section titled “File Structure”| Path | Task | Responsibility |
|---|---|---|
go.mod, go.sum |
1 | github.com/prometheus/client_golang |
internal/kernel/httpx/metrics.go (+ _test) |
1 | Metrics, Middleware, ServeMetrics; label drift test |
internal/kernel/httpx/server.go |
1 | Deps.Metrics wired in NewRouter |
internal/kernel/db/poolmetrics.go |
1 | RegisterPoolMetrics(reg, pool) |
internal/kernel/config/listen.go (+ _test) |
1 | PrivateListenAddr |
cmd/api/main.go, cmd/worker/main.go, Makefile, .env.example |
1 | metrics listener, version stamp, METRICS_LISTEN_ADDR |
db/migrations/00230_ops_roles.sql |
2 | roles, schemas, default-privilege revoke |
db/migrations/00231_ops_auth.sql |
2 | operator tables, audit chain, sign-in/session functions, ops.release_info |
internal/kernel/db/opsschema_test.go |
2, 4, 5, 8 | the four ops guards; opsOwnerGrants |
internal/kernel/db/rlscoverage_test.go, db/rls_exempt.txt |
2, 8 | scan ops/ops_auth; exempt ops_auth.* |
internal/kernel/db/migrationranges_test.go |
2 | {230, 249, "Operator console"} |
db/migrations/00232_organization_status.sql |
3 | organization.status |
internal/kernel/authn/authn.go |
3 | ErrOrgSuspended |
internal/platform/identity/liveness/liveness.go (+ _test) |
3 | reads status, returns ErrOrgSuspended |
internal/kernel/httpx/auth.go (+ auth_test.go) |
3 | read-only enforcement, suspendedWritable |
internal/app/routes_test.go |
3 | TestSuspendedWritableNamesRealRoutes |
api/openapi.yaml, web/src/lib/api/schema.d.ts, web/src/lib/i18n/locales/{en,ms,zh}/common.json |
3 | org_suspended |
db/migrations/00233_ops_check_result.sql |
4 | ops.check_result, ops.org_ids() |
internal/platform/ops/checks.go (+ _test) |
4 | RunChecks, fan-out and per-org workers, ChecksPeriodic |
cmd/worker/exports.go, cmd/worker/main.go |
4, 8 | register the ops workers and periodic job |
db/migrations/00234_ops_views.sql |
5 | org_overview, job_summary, discarded_jobs, check_latest, compliance_flags, security_summary |
internal/kernel/db/opsviews_test.go |
5 | each view: presence then absence |
cmd/operator/main.go (+ wiring_test.go) |
6 | serve, add, deactivate |
internal/operator/{server.go,auth.go,system.go} + templates/{layout,signin,system}.html |
6 | mux, route table, session middleware, sign-in, V8 |
internal/operator/server_test.go |
6 | TestEveryConsoleRouteRequiresASignedInOperator, sign-in tests |
internal/kernel/layering/layering_test.go, internal/app/binaries_test.go |
6, 8 | operator layer; cmd/operator links no module; worker links all |
internal/operator/{orgs.go,jobs.go} + templates/{orgs,org,jobs}.html, system.go (V2, V6) |
7 | pages |
internal/operator/pages_test.go |
7 | TestNoConsolePageShowsRecordContents |
db/migrations/00235_ops_actions.sql |
8 | ops_auth.operator_action, request_action, claim_action, finish_action, ops_auth.action_log |
internal/platform/ops/actions.go (+ _test) |
8 | ActionWorker |
internal/operator/actions.go + templates/{confirm,actions}.html (+ _test) |
9 | action forms and log |
scripts/dev-stack.sh, web/playwright.config.ts, web/e2e/{totp.ts,operator-console.spec.ts} |
10 | lane-parameterised stack with the console; e2e |
docs/RUNBOOK.md, docs/DEPLOY_NOTES.md, docs/ROADMAP.md |
10 | infra checklist, deploy notes, tick |
Task 1: Build version, /metrics on api and worker, no tenant labels
Section titled “Task 1: Build version, /metrics on api and worker, no tenant labels”Stage 1 in code. The Prometheus and Grafana servers themselves are infrastructure (Task 10 checklist, I9–I11).
Files:
- Modify:
go.mod,go.sum,Makefile,.env.example,cmd/api/main.go,cmd/worker/main.go,internal/kernel/httpx/server.go - Create:
internal/kernel/httpx/metrics.go,internal/kernel/httpx/metrics_test.go,internal/kernel/db/poolmetrics.go,internal/kernel/config/listen.go,internal/kernel/config/listen_test.go
Interfaces:
-
Produces:
httpx.NewMetrics(reg prometheus.Registerer) *httpx.Metrics;(*Metrics).Middleware() gin.HandlerFunc;httpx.ServeMetrics(addr string, g prometheus.Gatherer) *http.Server;httpx.Deps.Metrics *httpx.Metrics;db.RegisterPoolMetrics(reg prometheus.Registerer, pool *pgxpool.Pool);config.PrivateListenAddr(name, raw string) (string, error);main.versionincmd/worker. -
Step 1: Write the failing label test —
internal/kernel/httpx/metrics_test.go:
// TestMetricsCarryNoTenantLabels: a metric label is exported to Prometheus// and Grafana, outside the tenant boundary. Only the route TEMPLATE may// appear, never a path value, an org, a user or an email (design §3.6).func TestMetricsCarryNoTenantLabels(t *testing.T) { reg := prometheus.NewRegistry() m := httpx.NewMetrics(reg) r := gin.New() r.Use(m.Middleware()) r.GET("/v1/claims/:id", func(c *gin.Context) { c.Status(http.StatusOK) }) id := uuid.NewString() r.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest(http.MethodGet, "/v1/claims/"+id, nil))
fams, err := reg.Gather() require.NoError(t, err) allowed := map[string]bool{"method": true, "route": true, "status": true} series, sawTemplate := 0, false for _, f := range fams { for _, mt := range f.GetMetric() { series++ for _, l := range mt.GetLabel() { require.True(t, allowed[l.GetName()], "metric %s has label %q; only %v may be exported", f.GetName(), l.GetName(), allowed) require.NotContains(t, l.GetValue(), id, "metric %s carries a path value", f.GetName()) sawTemplate = sawTemplate || l.GetValue() == "/v1/claims/:id" } } } // PRESENCE: the request was counted, so the loop above judged something. require.Positive(t, series, "no series gathered; the middleware recorded nothing") require.True(t, sawTemplate, "the route template is not a label value; the middleware is not labelling by route")}Also TestPrivateListenAddr in listen_test.go, table-driven. Accepted: 127.0.0.1:9100, [::1]:9100, 10.0.0.5:9100, 172.18.0.2:9100, 0.0.0.0:9100. Refused: 203.0.113.7:9100 (public), metrics.local:9100 (hostname), :9100 (empty host), 127.0.0.1 (no port), 127.0.0.1:99999.
-
Step 2: Run, and watch it fail to compile.
go test ./internal/kernel/httpx/ ./internal/kernel/config/ -run 'Metrics|PrivateListen' -count=1 -
Step 3: Implement.
go get github.com/prometheus/client_golang@latest.metrics.go:
// Metrics is the api's request telemetry (design §3.6). Labels are the// method, the route TEMPLATE (gin's FullPath) and the status code — never a// path value, so no org id, member id or record id can reach Prometheus.type Metrics struct { requests *prometheus.CounterVec duration *prometheus.HistogramVec}
func NewMetrics(reg prometheus.Registerer) *Metrics { m := &Metrics{ requests: prometheus.NewCounterVec(prometheus.CounterOpts{ Name: "erp_http_requests_total", Help: "HTTP requests by method, route template and status."}, []string{"method", "route", "status"}), duration: prometheus.NewHistogramVec(prometheus.HistogramOpts{ Name: "erp_http_request_duration_seconds", Help: "HTTP request latency by route template.", Buckets: prometheus.DefBuckets}, []string{"method", "route"}), } reg.MustRegister(m.requests, m.duration) return m}
func (m *Metrics) Middleware() gin.HandlerFunc { return func(c *gin.Context) { start := time.Now() c.Next() route := c.FullPath() if route == "" { route = "unmatched" // a 404's raw path is caller-chosen text } m.requests.WithLabelValues(c.Request.Method, route, strconv.Itoa(c.Writer.Status())).Inc() m.duration.WithLabelValues(c.Request.Method, route).Observe(time.Since(start).Seconds()) }}
// ServeMetrics serves reg on its own listener, never on the public router.func ServeMetrics(addr string, g prometheus.Gatherer) *http.Server { mux := http.NewServeMux() mux.Handle("GET /metrics", promhttp.HandlerFor(g, promhttp.HandlerOpts{})) return &http.Server{Addr: addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second}}In NewRouter, if deps.Metrics != nil, it is r.Used first, before AccessLog, so every request, including a panic’s 500, is counted. poolmetrics.go registers three prometheus.NewGaugeFuncs over pool.Stat(): erp_db_pool_acquired_conns, erp_db_pool_idle_conns, erp_db_pool_total_conns. listen.go:
// PrivateListenAddr validates a listen address that must not face the// internet (design S3). name is the env var, for the message. An IP literal// is required, so what is validated is what is bound: a hostname could// resolve to a public address later.func PrivateListenAddr(name, raw string) (string, error) { host, port, err := net.SplitHostPort(raw) if err != nil { return "", fmt.Errorf("%s: %w", name, err) } if n, err := strconv.Atoi(port); err != nil || n < 1 || n > 65535 { return "", fmt.Errorf("%s: port %q is not 1-65535", name, port) } ip, err := netip.ParseAddr(host) if err != nil { return "", fmt.Errorf("%s: %q is not an IP literal; write 127.0.0.1 or a private address", name, host) } if !ip.IsLoopback() && !ip.IsPrivate() && !ip.IsUnspecified() { return "", fmt.Errorf("%s: %s is a public address; this listener must stay private", name, ip) } return raw, nil}cmd/api/main.go and cmd/worker/main.go: read raw := os.Getenv("METRICS_LISTEN_ADDR") (a string literal, for the env drift test). If it is non-empty, validate it and build reg := prometheus.NewRegistry(), then register collectors.NewGoCollector(), collectors.NewProcessCollector(…), db.RegisterPoolMetrics, and a erp_build_info{version} gauge set to 1. The api passes httpx.NewMetrics(reg) in Deps. Both run ServeMetrics(addr, reg).ListenAndServe() in a goroutine, log the error, and shut it down with the process. Empty means off, with one log line. The worker gains var version = "dev" and logs it at start. Makefile:
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)LDFLAGS := -X main.version=$(VERSION)
build: go build -ldflags "$(LDFLAGS)" -o bin/api ./cmd/api go build -ldflags "$(LDFLAGS)" -o bin/worker ./cmd/worker go build -o bin/migrate ./cmd/migrate.env.example gains METRICS_LISTEN_ADDR= with a comment: it is optional and empty means off; it must be an IP literal, loopback or private (127.0.0.1:9100 on a bare VPS, 0.0.0.0:9100 only inside a container whose port is not published); scraped by the self-hosted Prometheus (RUNBOOK §7).
- Step 4: Run.
go test ./internal/kernel/httpx/ ./internal/kernel/config/ ./internal/kernel/db/ -count=1→ PASS.go build ./cmd/... - Step 5: Sabotage. Add
"org_id"to theCounterVeclabels, passingc.GetString("org"). The test goes red naming label"org_id". Restore it. Then recordc.Request.URL.Pathinstead ofFullPath(). The test goes red on “carries a path value”. Restore it. - Step 6: Runtime check. Run
METRICS_LISTEN_ADDR=127.0.0.1:9109with the api on the dev stack, thencurl -s 127.0.0.1:9109/metrics | grep erp_http_requests_totalafter onecurl localhost:3000/v1/health.METRICS_LISTEN_ADDR=8.8.8.8:9109refuses to boot. - Step 7: Commit
Operator console T1: build version, private /metrics with route-template labels only
Task 2: Roles, ops / ops_auth, operator identity and audit chain, the four ops guards
Section titled “Task 2: Roles, ops / ops_auth, operator identity and audit chain, the four ops guards”Files:
- Create:
db/migrations/00230_ops_roles.sql,db/migrations/00231_ops_auth.sql,internal/kernel/db/opsschema_test.go - Modify:
internal/kernel/db/migrationranges_test.go,internal/kernel/db/rlscoverage_test.go,db/rls_exempt.txt
Interfaces:
-
Produces: roles
erp_ops_owner(NOLOGIN BYPASSRLS) anderp_operator(LOGIN, no password). Tablesops_auth.operator_account,ops_auth.operator_sessionandops_auth.operator_audit. Functions:ops_auth.signin_lookup(email text) → (operator_id uuid, password_hash text, totp_sealed bytea)(erp_operator)ops_auth.signin_result(email text, ok boolean, token_hash bytea) → uuid(erp_operator)ops_auth.session_operator(token_hash bytea) → uuid(no grant; used inside)ops_auth.session_check(token_hash bytea) → (operator_id uuid, email text)(erp_operator)ops_auth.session_close(token_hash bytea) → void(erp_operator)ops_auth.chain_status() → (row_count bigint, head_seq bigint, first_break bigint)(erp_operator)ops_auth.operator_add(email text, password_hash text, totp_sealed bytea, reset boolean) → uuid(no grant)ops_auth.operator_deactivate(email text) → void(no grant)ops_auth.audit_append(operator uuid, action text, org uuid, ref uuid) → void(no grant)ops_auth.audit_hash(...) → bytea(no grant)ops.release_info() → (version bigint, applied_at timestamptz, applied_count bigint)(erp_operator)opsOwnerGrantsinopsschema_test.go.
-
Step 1: Write the four failing guards —
internal/kernel/db/opsschema_test.go:
// opsOwnerGrants is every privilege erp_ops_owner holds on a table in// public, column by column (design S1a). It is a BYPASSRLS role, so this list// IS the set of customer data an ops function could ever read. A new column// is a reviewed line here, in the same commit as its GRANT.var opsOwnerGrants = map[string]string{ "goose_db_version.version_id": "SELECT", "goose_db_version.is_applied": "SELECT", "goose_db_version.tstamp": "SELECT",}
// aggregateText is the only names a text column of an ops function may have// (design S1c): vocabulary values and the organization's name.var aggregateText = map[string]bool{"name": true, "status": true, "kind": true, "state": true, "queue": true, "check_key": true, "entity_kind": true, "action": true, "outcome": true, "detail_code": true}
var aggregateTypes = map[string]bool{"uuid": true, "bigint": true, "integer": true, "boolean": true, "timestamp with time zone": true, "date": true, "void": true}
func TestEveryOpsFunctionIsOwnedPinnedAndNotPublic(t *testing.T) { h := testdb.Start(t) rows, err := h.Admin.Query(context.Background(), ` SELECT n.nspname || '.' || p.proname, pg_get_userbyid(p.proowner), EXISTS (SELECT 1 FROM aclexplode(coalesce(p.proacl, acldefault('f', p.proowner))) a WHERE a.grantee = 0 AND a.privilege_type = 'EXECUTE'), coalesce((SELECT c FROM unnest(p.proconfig) c WHERE c LIKE 'search_path=%'), '') FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace WHERE n.nspname IN ('ops', 'ops_auth')`) require.NoError(t, err) seen := 0 for rows.Next() { var name, owner, setting string var public bool require.NoError(t, rows.Scan(&name, &owner, &public, &setting)) seen++ require.Equal(t, "erp_ops_owner", owner, "%s must be owned by erp_ops_owner (ALTER FUNCTION ... OWNER TO)", name) require.False(t, public, "%s is executable by PUBLIC; REVOKE ALL ON FUNCTION ... FROM PUBLIC", name) require.True(t, pinsPgTempLast(setting), "%s must SET search_path = pg_catalog, public, pg_temp (got %q)", name, setting) } require.NoError(t, rows.Err()) // PRESENCE: the scan reached the schemas. require.GreaterOrEqual(t, seen, 5, "found %d ops/ops_auth functions; the query is not reaching them", seen)}
func TestOpsFunctionsReturnAggregatesOnly(t *testing.T) { h := testdb.Start(t) // Every result column: OUT/TABLE arguments, or the plain return type. rows, err := h.Admin.Query(context.Background(), ` SELECT p.proname, coalesce(a.name, ''), format_type(coalesce(a.typ, p.prorettype), NULL) FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace LEFT JOIN LATERAL ( SELECT u.name, u.typ FROM unnest(p.proallargtypes, p.proargmodes, p.proargnames) AS u(typ, mode, name) WHERE u.mode IN ('o', 't')) a ON true WHERE n.nspname = 'ops'`) require.NoError(t, err) cols := 0 for rows.Next() { var fn, col, typ string require.NoError(t, rows.Scan(&fn, &col, &typ)) cols++ if typ == "text" { require.True(t, aggregateText[col], "ops.%s returns text column %q; text is allowed only for %v "+ "(design S1c). A name, email or record text must never leave ops", fn, col, aggregateText) continue } require.True(t, aggregateTypes[typ], "ops.%s returns %q (%s); ops returns counts, flags, ids and times only", fn, col, typ) } require.NoError(t, rows.Err()) require.Positive(t, cols, "no ops result columns found; the guard judged nothing")}setof uuid (Task 4’s org_ids) reads as uuid through prorettype. A TABLE function has no plain return type to check, and its record type never reaches the fallback because its columns are t mode.
Two more tests:
TestTheOpsOwnerHoldsExactlyItsGrants:- Presence: the table-level ACL query over
publicfinds rows forerp_app. - Absence: the same query finds none for
erp_ops_owner. - The column-level query (
aclexplode(att.attacl)overpg_attributeinpublic, grantee'erp_ops_owner'::regrole) returns exactlyopsOwnerGrants. pg_rolessayserp_ops_ownerisNOT rolcanlogin,rolbypassrlsandNOT rolsuper.
- Presence: the table-level ACL query over
TestTheOperatorRoleReadsNoTable:- Insert an organization as
h.Adminand showSELECT count(*) FROM organization> 0 (presence). - In an
h.Admintransaction,SET LOCAL ROLE erp_operatorand runSELECT * FROM ops.release_info(), which succeeds. The role is usable. - As the last statement,
SELECT * FROM organizationfails with SQLSTATE42501. - A catalog query finds no table-level or column-level ACL entry for
erp_operatorinpublic,opsorops_auth.
- Insert an organization as
Widen TestEveryTableIsTenantIsolated and TestExemptionFilesNameRealTables. Each selects CASE WHEN n.nspname = 'public' THEN c.relname ELSE n.nspname || '.' || c.relname END and filters n.nspname IN ('public', 'ops', 'ops_auth'). Every message keeps naming the table. Add {230, 249, "Operator console"} to migrationRanges.
-
Step 2: Run, and watch them fail.
go test ./internal/kernel/db/ -run 'Ops|OperatorRole|TenantIsolated|ExemptionFiles|MigrationIsNumbered' -count=1→ fails: no schemas, fewer than 5 functions. -
Step 3: Write
00230_ops_roles.sql:
-- +goose Up
-- The operator console's two roles (design §3.3, S1). erp_ops_owner owns the-- ops functions and bypasses row-level security, so it can count across-- organizations. It cannot sign in, and it holds SELECT only on the columns-- the functions need (opsschema_test.go, opsOwnerGrants). erp_operator is what-- cmd/operator connects as: EXECUTE on named functions, no table privilege.-- It has no password here. Each environment sets one (DEPLOY_NOTES), so no-- known password ships in a migration.-- +goose StatementBeginDO $$BEGIN IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'erp_ops_owner') THEN CREATE ROLE erp_ops_owner NOLOGIN BYPASSRLS; END IF; IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'erp_operator') THEN CREATE ROLE erp_operator LOGIN; END IF;END$$;-- +goose StatementEnd
CREATE SCHEMA ops AUTHORIZATION erp_ops_owner;CREATE SCHEMA ops_auth AUTHORIZATION erp_ops_owner;REVOKE ALL ON SCHEMA ops, ops_auth FROM PUBLIC;GRANT USAGE ON SCHEMA ops, ops_auth TO erp_operator;
-- Functions the migration role creates here must not be executable by-- PUBLIC, which is Postgres's default. Each migration still revokes-- explicitly; this is the backstop.ALTER DEFAULT PRIVILEGES IN SCHEMA ops, ops_auth REVOKE EXECUTE ON FUNCTIONS FROM PUBLIC;
-- +goose DownALTER DEFAULT PRIVILEGES IN SCHEMA ops, ops_auth GRANT EXECUTE ON FUNCTIONS TO PUBLIC;DROP SCHEMA ops_auth CASCADE;DROP SCHEMA ops CASCADE;-- +goose StatementBeginDO $$BEGIN IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'erp_operator') THEN EXECUTE 'DROP OWNED BY erp_operator'; END IF; IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'erp_ops_owner') THEN EXECUTE 'DROP OWNED BY erp_ops_owner'; END IF;END$$;-- +goose StatementEndDROP ROLE IF EXISTS erp_operator;DROP ROLE IF EXISTS erp_ops_owner;- Step 4: Write
00231_ops_auth.sql. The tables, and the functions that are not obvious:
-- +goose UpCREATE TABLE ops_auth.operator_account ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), email TEXT NOT NULL CONSTRAINT operator_account_email_once UNIQUE, password_hash TEXT NOT NULL, totp_sealed BYTEA NOT NULL, active BOOLEAN NOT NULL DEFAULT true, failed_count INT NOT NULL DEFAULT 0, locked_until TIMESTAMPTZ, last_signin_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), CONSTRAINT operator_account_email_normal CHECK (email = lower(btrim(email)) AND email LIKE '%_@_%'));CREATE TABLE ops_auth.operator_session ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), operator_id UUID NOT NULL REFERENCES ops_auth.operator_account (id), token_hash BYTEA NOT NULL CONSTRAINT operator_session_token_once UNIQUE, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), last_seen_at TIMESTAMPTZ NOT NULL DEFAULT now(), revoked_at TIMESTAMPTZ);-- Append-only and hash-chained, the audit_log shape (00007). Kept seven-- years (S10). No role holds UPDATE or DELETE, and nobody signs in as its-- owner.-- ponytail: no recorded head, so a truncated tail still verifies (Ruling O4).-- Record the head externally if an auditor asks.CREATE TABLE ops_auth.operator_audit ( seq BIGSERIAL PRIMARY KEY, operator_id UUID, action TEXT NOT NULL CONSTRAINT operator_audit_action_known CHECK (action IN ('operator_added','operator_reset','operator_deactivated', 'signin','signin_failed','signout','action_requested')), org_id UUID, ref UUID, occurred_at TIMESTAMPTZ NOT NULL, prev_hash BYTEA, row_hash BYTEA NOT NULL);ALTER TABLE ops_auth.operator_account OWNER TO erp_ops_owner;ALTER TABLE ops_auth.operator_session OWNER TO erp_ops_owner;ALTER TABLE ops_auth.operator_audit OWNER TO erp_ops_owner;
-- One canonical rendering, used by append and verify alike: two renderings-- are how audit_log's chain once broke (kernel/audit rowHashSQL).-- +goose StatementBeginCREATE FUNCTION ops_auth.audit_hash(p_seq bigint, p_operator uuid, p_action text, p_org uuid, p_ref uuid, p_at timestamptz, p_prev bytea) RETURNS bytea LANGUAGE sql STABLE SET search_path = pg_catalog, public, pg_temp AS $$ SELECT public.digest(concat_ws(chr(31), p_seq::text, coalesce(p_operator::text, ''), p_action, coalesce(p_org::text, ''), coalesce(p_ref::text, ''), to_char(p_at AT TIME ZONE 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US'), coalesce(encode(p_prev, 'hex'), '')), 'sha256')$$;-- +goose StatementEnd
-- +goose StatementBeginCREATE FUNCTION ops_auth.audit_append(p_operator uuid, p_action text, p_org uuid, p_ref uuid) RETURNS void LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$DECLARE v_prev bytea; v_seq bigint; v_at timestamptz := clock_timestamp();BEGIN -- Serializes appenders so two cannot chain onto one prev_hash. A table -- lock, not an advisory lock: the owner may lock its own table, and the -- advisory class registry (kernel/audit) stays as it is. LOCK TABLE ops_auth.operator_audit IN SHARE ROW EXCLUSIVE MODE; SELECT row_hash INTO v_prev FROM ops_auth.operator_audit ORDER BY seq DESC LIMIT 1; v_seq := nextval(pg_get_serial_sequence('ops_auth.operator_audit', 'seq')); INSERT INTO ops_auth.operator_audit (seq, operator_id, action, org_id, ref, occurred_at, prev_hash, row_hash) VALUES (v_seq, p_operator, p_action, p_org, p_ref, v_at, v_prev, ops_auth.audit_hash(v_seq, p_operator, p_action, p_org, p_ref, v_at, v_prev));END$$;-- +goose StatementEnd
-- The session rule, in one place: live, unrevoked, 8h absolute, 30m idle,-- and the account still active (S6).-- +goose StatementBeginCREATE FUNCTION ops_auth.session_operator(p_token_hash bytea) RETURNS uuid LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$ SELECT s.operator_id FROM ops_auth.operator_session s JOIN ops_auth.operator_account a ON a.id = s.operator_id WHERE s.token_hash = p_token_hash AND s.revoked_at IS NULL AND a.active AND s.created_at > now() - interval '8 hours' AND s.last_seen_at > now() - interval '30 minutes'$$;-- +goose StatementEnd
-- The console verifies the password (argon2id, Go) and the TOTP code, then-- reports the verdict here. The lock-out, the 90-second rule (S6) and the-- audit row are decided here, so the console cannot skip them (Ruling O3).-- +goose StatementBeginCREATE FUNCTION ops_auth.signin_result(p_email text, p_ok boolean, p_token_hash bytea) RETURNS uuid LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$DECLARE v_id uuid; v_allowed boolean; v_session uuid;BEGIN SELECT id, active AND (locked_until IS NULL OR locked_until < now()) AND (last_signin_at IS NULL OR last_signin_at < now() - interval '90 seconds') INTO v_id, v_allowed FROM ops_auth.operator_account WHERE email = p_email FOR UPDATE; IF p_ok AND v_allowed THEN UPDATE ops_auth.operator_account SET failed_count = 0, locked_until = NULL, last_signin_at = now() WHERE id = v_id; INSERT INTO ops_auth.operator_session (operator_id, token_hash) VALUES (v_id, p_token_hash) RETURNING id INTO v_session; PERFORM ops_auth.audit_append(v_id, 'signin', NULL, v_session); RETURN v_session; END IF; IF v_id IS NOT NULL THEN UPDATE ops_auth.operator_account SET failed_count = CASE WHEN failed_count >= 4 THEN 0 ELSE failed_count + 1 END, locked_until = CASE WHEN failed_count >= 4 THEN now() + interval '15 minutes' ELSE locked_until END WHERE id = v_id; END IF; PERFORM ops_auth.audit_append(v_id, 'signin_failed', NULL, NULL); RETURN NULL;END$$;-- +goose StatementEndThe remaining functions are plain:
signin_lookupreturns the row for an active account only.session_checkrunsUPDATE … SET last_seen_at = now() … WHERE s.operator_id = ops_auth.session_operator(p_token_hash) RETURNING operator_id, email.session_closesetsrevoked_atand appendssignout.chain_statuswalksORDER BY seq, recomputes each row withaudit_hash, and returns the firstseqwhoseprev_hashorrow_hashdisagrees, or NULL.operator_addinserts, or, withreset, replacespassword_hashandtotp_sealed, setsactive, and revokes live sessions. It appendsoperator_addedoroperator_reset, and refuses a duplicate withoutreset, or aresetof an unknown email.operator_deactivatesetsactive = false, revokes sessions and appendsoperator_deactivated.ops.release_info:SELECT max(version_id), max(tstamp)::timestamptz, count(*) FROM public.goose_db_version WHERE is_applied AND version_id > 0.
Every function gets ALTER FUNCTION … OWNER TO erp_ops_owner and REVOKE ALL … FROM PUBLIC. Then run GRANT EXECUTE ON FUNCTION ops_auth.signin_lookup(text), ops_auth.signin_result(text, boolean, bytea), ops_auth.session_check(bytea), ops_auth.session_close(bytea), ops_auth.chain_status(), ops.release_info() TO erp_operator; and GRANT SELECT (version_id, is_applied, tstamp) ON public.goose_db_version TO erp_ops_owner;. The Down migration drops the functions and tables in reverse.
db/rls_exempt.txt gains the three ops_auth.* tables under a comment. They are the provider’s own operator identity and trail, not tenant data. No org_id is possible, because an operator is not a member of any organization (design §3.4). The condition matches the identity tables’ exemption: no package under internal/modules names them, and only ops_auth functions and internal/operator reach them.
-
Step 5: Add behaviour tests in
opsschema_test.go, running aserp_operatorviaSET LOCAL ROLE:TestOperatorSigninLocksAfterFiveFailures: add an operator as admin viaoperator_add. Presence:signin_result(email, true, h1)returns a session. Wait out the 90s rule byUPDATE … SET last_signin_at = now() - interval '2 minutes'as admin. Record fivefalseresults. Thensignin_result(email, true, h2)returns NULL. As admin,locked_untilis in the future.TestOperatorSigninIsSingleUseWithin90Seconds.TestOperatorSessionExpiresIdle: setlast_seen_at31 minutes back as admin, andsession_checkreturns no row.TestOperatorAuditChainVerifiesAndDetectsAnEdit: presence: after three events,chain_status()givesfirst_breakNULL androw_count3. Then, as admin,UPDATE ops_auth.operator_audit SET action = 'signout' WHERE seq = 2, andfirst_break= 2.
-
Step 6: Run.
go test ./internal/kernel/db/ -count=1→ PASS (including every pre-existing test).go test ./internal/kernel/layering/ -count=1: the exempt-table grep must still pass with the dotted names. -
Step 7: Sabotage, one at a time, recording each red message:
- Drop
REVOKE ALL … FROM PUBLICforchain_status, and also drop the default-privilege line. - Change one
SET search_pathtopg_catalog, pg_temp, public. - Add
email texttoops.release_info’sRETURNS TABLE. - Add
GRANT SELECT ON public.goose_db_version TO erp_ops_owner(table-level). - Add
GRANT SELECT ON public.organization TO erp_operator.
Restore after each.
- Drop
-
Step 8: Commit
Operator console T2: erp_ops_owner/erp_operator, ops_auth identity and audit chain, ops guards
Task 3: organization.status and read-only suspension
Section titled “Task 3: organization.status and read-only suspension”Files:
- Create:
db/migrations/00232_organization_status.sql - Modify:
internal/kernel/authn/authn.go,internal/platform/identity/liveness/liveness.go(+ its test),internal/kernel/httpx/auth.go,internal/kernel/httpx/auth_test.go,internal/app/routes_test.go,api/openapi.yaml, generatedinternal/gen/api,web/src/lib/api/schema.d.ts,web/src/lib/i18n/locales/{en,ms,zh}/common.json,web/src/lib/api/problem.test.ts
Interfaces:
-
Produces: column
organization.status TEXT NOT NULL DEFAULT 'active'('active' | 'suspended').authn.ErrOrgSuspended. The 403 Problem withcode: "org_suspended".httpx.SuspendedWritable() []string, for the route test. -
Consumes:
liveness.Cached.Check,httpx.Authenticate. -
Step 1: Failing tests.
- In
liveness_test.go: presence: an active organization’s claimsCheck→ nil. Then, as admin,UPDATE organization SET status = 'suspended', andCheck→errors.Is(err, authn.ErrOrgSuspended). A revoked session in a suspended organization still returnsErrSessionDead, notErrOrgSuspended, so the dead-session refusal wins. - In
httpx/auth_test.go, with a stubLivenessCheckerreturningauthn.ErrOrgSuspended:- Presence: the same stub returning nil lets
POST /v1/xreach the handler. - Suspended:
GET /v1/xreaches it. - Suspended:
POST /v1/x→ 403, and the body’scodeisorg_suspended. - Suspended:
POST /v1/reports/exports, registered in the test router at that path, reaches it.
- Presence: the same stub returning nil lets
- In
internal/app/routes_test.go, addTestSuspendedWritableNamesRealRoutes. Build the real router the wayTestEveryRouteIsAuthorizeddoes. Presence: it registers more than 50 routes. Everyhttpx.SuspendedWritable()entry is amongMETHOD + " " + route.Path.
- In
-
Step 2: Run, and watch them fail.
go test ./internal/platform/identity/liveness/ ./internal/kernel/httpx/ ./internal/app/ -run 'Suspend|Liveness' -count=1 -
Step 3: Implement.
-- 00232_organization_status.sql-- +goose Up-- A5 suspension (design S2): read-only, never a deletion. A vocabulary string.ALTER TABLE organization ADD COLUMN status TEXT NOT NULL DEFAULT 'active' CONSTRAINT organization_status_known CHECK (status IN ('active', 'suspended'));-- +goose DownALTER TABLE organization DROP COLUMN status;authn.go:
// ErrOrgSuspended is a liveness answer, not a refusal: the session is live,// and the organization it acts in is read-only (design S2). httpx.Authenticate// admits reads and refuses writes.var ErrOrgSuspended = errors.New("authn: organization is suspended")In liveness.go, state gains Suspended bool json:“x”``. fromPostgres selects a fifth value, COALESCE((SELECT status = 'suspended' FROM organization WHERE id = $3 AND org_id = $3), false). Check returns authn.ErrOrgSuspended after the perm-version case. Every other refusal comes first. A cached entry written before this change has no x and reads as not suspended for at most livenessTTL (5s). In Authenticate:
err := l.Check(c.Request.Context(), claims)suspended := errors.Is(err, authn.ErrOrgSuspended)if err != nil && !suspended { _ = c.Error(apperr.New(apperr.KindUnauthorized, "the session is no longer valid")) c.Abort() return}if suspended && !readOnlyMethod(c.Request.Method) && !suspendedWritable[c.Request.Method+" "+c.FullPath()] { _ = c.Error(apperr.New(apperr.KindForbidden, "this organization is suspended: its records can be read and exported, not changed"). With("code", "org_suspended")) c.Abort() return}readOnlyMethod is GET, HEAD or OPTIONS. suspendedWritable holds the five routes of Ruling O7, with a comment saying why each one is allowed.
Document the code in api/openapi.yaml’s Problem.code description: add “org_suspended, a 403 on any write while the organization is suspended (read-only; reads and exports still work)”. Run make generate.
Locales, common:refusals.org_suspended:
- en:
"This organization is suspended. You can still view and export its records, but you cannot change anything." - ms:
"Organisasi ini digantung. Anda masih boleh melihat dan mengeksport rekodnya, tetapi tidak boleh mengubah apa-apa." - zh:
"该组织已被暂停。您仍可查看和导出其记录,但不能进行任何更改。"
problem.test.ts gains the case: a 403 with code: 'org_suspended' → problemMessage returns the translation in ms.
- Step 4: Run.
go test ./internal/platform/identity/... ./internal/kernel/httpx/ ./internal/app/ ./internal/kernel/db/ -count=1, thenpnpm -C web test -- problem i18nparity, thengit status. Generated files are committed, and nothing else is dirty. - Step 5: Sabotage. Remove the
!readOnlyMethodclause, so every request is refused. The GET case goes red. Then makeCheckreturn nil when suspended. The suspended-POST case goes red. Then add"POST /v1/nope"tosuspendedWritable. The route test names it. Restore each. - Step 6: Commit
Operator console T3: organization.status, suspension is read-only (S2)
Task 4: ops.check_result and the nightly integrity checks (V4; closes R30’s nightly reconciliation)
Section titled “Task 4: ops.check_result and the nightly integrity checks (V4; closes R30’s nightly reconciliation)”Files:
- Create:
db/migrations/00233_ops_check_result.sql,internal/platform/ops/checks.go,internal/platform/ops/checks_test.go - Modify:
cmd/worker/exports.go,cmd/worker/main.go,cmd/worker/wiring_test.go,internal/kernel/db/opsschema_test.go(organization.id)
Interfaces:
-
Produces:
ops.check_result, andops.org_ids() RETURNS SETOF uuid(EXECUTE:erp_app).package ops(internal/platform/ops), withRunChecks(ctx, d *db.DB, c clock.Clock, orgID uuid.UUID) error.- Job types
ChecksFanoutArgs{}(kindops_checks_fanout) andOrgChecksArgs{OrgID uuid.UUID}(kindops_org_checks). - Workers
ChecksFanoutWorker{DB}andOrgChecksWorker{DB, Clock}, andChecksPeriodic() *river.PeriodicJob(24h,RunOnStart: true).
-
Consumes:
reconcile.Default().Run,river.ClientFromContext[pgx.Tx],jobs.QueueMaintenance. -
Step 1: Failing tests in
internal/platform/ops/checks_test.go, usingexpensestest.New(t)for a provisioned organization:TestRunChecksRecordsOneRowPerCheck: afterRunChecks, as admin,SELECT check_key, passed, finding_count FROM ops.check_result WHERE org_id = $1has one row perreconcile.Default()check (more than 5), all passed.TestRunChecksRecordsAFailure, presence then absence. First, all checks pass. Then, as admin, append an unbalanced journal line the wayreconcile’s own tests breakentry_balance(readinternal/platform/accounting/reconcile/*_test.goand use the same fixture helper).RunChecksthen recordsentry_balancewithpassed = falseandfinding_count >= 1.TestTheAppRoleCanOnlyAppendCheckResults: presence: as admin, both organizations’ rows are visible. Aserp_appin org A’s tenant transaction, anINSERTfor org A succeeds, and anINSERTnaming org B fails the policy’sWITH CHECK(42501). In a fresh transaction,SELECTfromops.check_resultis refused (42501), becauseerp_appholdsINSERTonly.TestTheNightlyFanOutEnqueuesEveryOrganization: with 2 organizations, working oneChecksFanoutArgsthroughjobs.NewWorkerClient(theTestTheWorkerClientWorksAJobshape) leaves both organizations with results.
-
Step 2: Run, and watch them fail.
go test ./internal/platform/ops/ -count=1 -
Step 3: Implement.
-- 00233_ops_check_result.sql-- +goose Up-- V4 (design §3.5). The worker writes one row per check per run, inside the-- organization's own tenant transaction: erp_app holds INSERT only, and the-- policy's WITH CHECK pins the row to that tenant. The console reads it only-- through ops.check_latest (00234). Operational data; not pruned in v1 (S10).CREATE TABLE ops.check_result ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), org_id UUID NOT NULL REFERENCES public.organization (id), check_key TEXT NOT NULL CONSTRAINT check_result_key_shape CHECK (check_key ~ '^[a-z][a-z0-9_]{0,62}$'), passed BOOLEAN NOT NULL, finding_count INT NOT NULL CONSTRAINT check_result_findings_nonnegative CHECK (finding_count >= 0), ran_at TIMESTAMPTZ NOT NULL, CONSTRAINT check_result_passed_means_no_findings CHECK (passed = (finding_count = 0)));CREATE INDEX check_result_latest_idx ON ops.check_result (org_id, check_key, ran_at DESC);ALTER TABLE ops.check_result ENABLE ROW LEVEL SECURITY;ALTER TABLE ops.check_result FORCE ROW LEVEL SECURITY;CREATE POLICY org_isolation ON ops.check_result USING (org_id = current_setting('app.current_org', true)::uuid) WITH CHECK (org_id = current_setting('app.current_org', true)::uuid);ALTER TABLE ops.check_result OWNER TO erp_ops_owner;GRANT USAGE ON SCHEMA ops TO erp_app;GRANT INSERT ON ops.check_result TO erp_app;
-- The nightly fan-out's list of organizations. erp_app cannot list them-- under forced RLS, by design. Ids only.-- +goose StatementBeginCREATE FUNCTION ops.org_ids() RETURNS SETOF uuid LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$ SELECT id FROM public.organization ORDER BY id$$;-- +goose StatementEndALTER FUNCTION ops.org_ids() OWNER TO erp_ops_owner;REVOKE ALL ON FUNCTION ops.org_ids() FROM PUBLIC;GRANT EXECUTE ON FUNCTION ops.org_ids() TO erp_app;GRANT SELECT (id) ON public.organization TO erp_ops_owner;-- +goose DownREVOKE SELECT (id) ON public.organization FROM erp_ops_owner;DROP FUNCTION ops.org_ids();REVOKE USAGE ON SCHEMA ops FROM erp_app;DROP TABLE ops.check_result;The foreign key’s RI check runs as the referenced table’s owner, and RI checks skip RLS. TestRunChecksRecordsOneRowPerCheck inserting as erp_app is what proves the grant set is enough. If it fails on a permission, stop and report before adding any grant.
checks.go:
// RunChecks runs every registered reconciliation check for one organization// in its own tenant transaction and records the outcome: key, pass/fail and// a finding COUNT. Findings name rows (reconcile.Finding.Ref) and carry// English detail, and neither leaves the tenant, so the console only ever// sees counts (design §1).func RunChecks(ctx context.Context, d *db.DB, c clock.Clock, orgID uuid.UUID) error { return d.InTx(tenancy.WithOrg(ctx, orgID), func(ctx context.Context, tx pgx.Tx) error { res, err := reconcile.Default().Run(ctx, tx) if err != nil { return fmt.Errorf("ops: checks %s: %w", orgID, err) } now := c.Now() for _, r := range res.Checks { if _, err := tx.Exec(ctx, ` INSERT INTO ops.check_result (org_id, check_key, passed, finding_count, ran_at) VALUES ($1, $2, $3, $4, $5)`, orgID, r.Key, r.Passed, len(r.Findings), now); err != nil { return fmt.Errorf("ops: record %s: %w", r.Key, err) } } return nil })}
func (w *ChecksFanoutWorker) Work(ctx context.Context, _ *river.Job[ChecksFanoutArgs]) error { rows, err := w.DB.Pool().Query(ctx, `SELECT org_id FROM ops.org_ids() AS org_id`) if err != nil { return fmt.Errorf("ops: org ids: %w", err) } ids, err := pgx.CollectRows(rows, pgx.RowTo[uuid.UUID]) if err != nil { return fmt.Errorf("ops: org ids: %w", err) } params := make([]river.InsertManyParams, 0, len(ids)) for _, id := range ids { params = append(params, river.InsertManyParams{Args: OrgChecksArgs{OrgID: id}, InsertOpts: &river.InsertOpts{Queue: jobs.QueueMaintenance, MaxAttempts: 3}}) // Ruling O6 } _, err = river.ClientFromContext[pgx.Tx](ctx).InsertMany(ctx, params) return err}OrgChecksWorker.Work calls RunChecks. Register both workers in startExportWorker, and add ops.ChecksPeriodic() to PeriodicJobs in cmd/worker/main.go. Update the doc comment above main to name the new scheduled work, and state that it deletes nothing. Add "organization.id": "SELECT" to opsOwnerGrants. The wiring_test.go for the worker asserts both kinds are registered. Follow the file’s existing shape for the export and prune workers.
- Step 4: Run.
go test ./internal/platform/ops/ ./internal/kernel/db/ ./cmd/worker/ -count=1 - Step 5: Sabotage. Drop
WITH CHECKfrom the policy.TestEveryTableIsTenantIsolated(now scanningops) goes red. Then remove theorganization.idline fromopsOwnerGrants. The grants test goes red. Restore both. - Step 6: Commit
Operator console T4: ops.check_result and nightly integrity checks per organization (R30)
Task 5: The read-only view functions (V1, V3, V5, V7, V4 latest)
Section titled “Task 5: The read-only view functions (V1, V3, V5, V7, V4 latest)”Files:
- Create:
db/migrations/00234_ops_views.sql,internal/kernel/db/opsviews_test.go - Modify:
internal/kernel/db/opsschema_test.go(opsOwnerGrants)
Interfaces (all EXECUTE: erp_operator):
-
ops.org_overview() → (org_id uuid, name text, status text, created_at timestamptz, active_members bigint, last_activity_at timestamptz) -
ops.check_latest() → (org_id uuid, check_key text, passed boolean, finding_count integer, ran_at timestamptz) -
ops.compliance_flags() → (org_id uuid, periods_open_past_end bigint, receipt_lines_without_basis bigint) -
ops.security_summary(p_days integer) → (org_id uuid, entity_kind text, action text, events bigint) -
ops.job_summary() → (org_id uuid, queue text, kind text, state text, jobs bigint, stuck bigint, oldest_at timestamptz) -
ops.discarded_jobs() → (job_id bigint, org_id uuid, kind text, finalized_at timestamptz) -
Step 1: Failing tests —
opsviews_test.go. One test per function. Each calls the function aserp_operator(SET LOCAL ROLE), and each has the presence-then-absence shape:org_overview: organizations A and B, seeded as admin, with 2 active members and 1 inactive in A and one audit row. A’s row showsactive_members = 2,last_activity_atequal to that row’soccurred_at, andstatusactive. After the admin suspends B, B showssuspended.check_latest: two runs for A. Only the newer run’s rows come back, one per key.compliance_flags:- Presence: A has an open period ending yesterday and one receipt line with
document_basis IS NULLon a live claim, and the counts are 1 and 1. - Absence: closing the period and withdrawing the line gives 0 and 0.
- Use
expensestestfixtures for the claim.
- Presence: A has an open period ending yesterday and one receipt line with
security_summary(7): twosession/login_failedrows for A, and one from 10 days ago. The count is 2. A non-allowlisted pair (claim/update) is absent while it is present inaudit_log(presence, as admin).job_summary: insert river rows as admin withargs = {"org_id": A}and statesavailable/running(attempted_at40 minutes ago). The counts group by org, andstuck= 1. A row withargswithoutorg_idgroups under NULL.discarded_jobs: adiscardedrow is listed. Acompletedone is not.TestOpsViewsReadNoRecordText, one test for S1a in practice:- As admin, show
audit_log.afterholds a known string (presence). - Concatenate
row_to_jsonof every view function’s output, called aserp_operator. - Assert that string is absent.
- As admin, show
-
Step 2: Run, and watch them fail.
go test ./internal/kernel/db/ -run 'Ops' -count=1 -
Step 3: Implement
00234_ops_views.sql. Each function isLANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp, and is followed byOWNER TO erp_ops_owner,REVOKE ALL … FROM PUBLICandGRANT EXECUTE … TO erp_operator. The two that are not obvious:
CREATE FUNCTION ops.org_overview() RETURNS TABLE (org_id uuid, name text, status text, created_at timestamptz, active_members bigint, last_activity_at timestamptz) LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$ SELECT o.id, o.name, o.status, o.created_at, (SELECT count(*) FROM public.organization_member m WHERE m.org_id = o.id AND m.active), (SELECT a.occurred_at FROM public.audit_log a WHERE a.org_id = o.id ORDER BY a.seq DESC LIMIT 1) FROM public.organization o ORDER BY o.created_at$$;
-- Job args are ids only (db/rls_exempt.txt, River). Only the org id is read,-- and only when it is a well-formed uuid.CREATE FUNCTION ops.job_summary() RETURNS TABLE (org_id uuid, queue text, kind text, state text, jobs bigint, stuck bigint, oldest_at timestamptz) LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$ SELECT CASE WHEN j.args->>'org_id' ~ '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$' THEN (j.args->>'org_id')::uuid END, j.queue, j.kind, j.state::text, count(*), count(*) FILTER (WHERE j.state = 'running' AND j.attempted_at < now() - interval '30 minutes'), min(j.created_at) FROM public.river_job j GROUP BY 1, 2, 3, 4$$;compliance_flags:periods_open_past_endcountsfiscal_periodrows withstatus = 'open' AND ends_on < current_date.receipt_lines_without_basiscountsexpense_claim_linerows withline_type = 'receipt' AND withdrawn_at IS NULL AND document_basis IS NULL.- Both are grouped per organization from
organization, so a zero shows as 0, not a missing row.
security_summarycountsaudit_logrows by(org_id, entity_kind, action)whereoccurred_at > now() - make_interval(days => p_days)and the pair is in('session','login_failed'),('session','replay_detected'),('role','permissions_set')or('organization_member','roles_set').p_daysis clamped to 1..90.check_latestisDISTINCT ON (org_id, check_key) … ORDER BY org_id, check_key, ran_at DESC.
Grants:
GRANT SELECT (name, status, created_at) ON public.organization TO erp_ops_owner;GRANT SELECT (org_id, active) ON public.organization_member TO erp_ops_owner;GRANT SELECT (org_id, seq, occurred_at, entity_kind, action) ON public.audit_log TO erp_ops_owner;GRANT SELECT (org_id, status, ends_on) ON public.fiscal_period TO erp_ops_owner;GRANT SELECT (org_id, line_type, document_basis, withdrawn_at) ON public.expense_claim_line TO erp_ops_owner;GRANT SELECT (id, kind, state, queue, args, attempted_at, created_at, finalized_at) ON public.river_job TO erp_ops_owner;Add every column to opsOwnerGrants. ops.check_result is the owner’s own table and needs no grant.
- Step 4: Run.
go test ./internal/kernel/db/ -count=1 - Step 5: Sabotage. Change
security_summaryto returna.after::text AS detail_code. The type test stays green (the name is allowed), soTestOpsViewsReadNoRecordTextmust go red, and the missing column grant makes the call itself fail with42501. Read which one fires. That is the point of S1a. Restore. - Step 6: Commit
Operator console T5: ops view functions over granted columns only
Task 6: cmd/operator and internal/operator: sign-in, sessions, route guard, V8
Section titled “Task 6: cmd/operator and internal/operator: sign-in, sessions, route guard, V8”Files:
- Create:
cmd/operator/main.go,cmd/operator/wiring_test.go,internal/operator/server.go,internal/operator/auth.go,internal/operator/system.go,internal/operator/templates/{layout,signin,system}.html,internal/operator/server_test.go - Modify:
internal/kernel/layering/layering_test.go,internal/app/binaries_test.go,.env.example,Makefile(buildbin/operatorwithLDFLAGS)
Interfaces:
-
Produces:
operator.Server{DB *pgxpool.Pool; TOTPKey []byte; Clock clock.Clock; Version, APIURL, GrafanaURL, BackupStatusFile string; HTTP *http.Client}and(*Server).Handler() http.Handler.operator.OperatorFromContext(ctx) (Operator, bool), whereOperator{ID uuid.UUID; Email string; TokenHash []byte}.(*Server).page(w, name string, data any).(*Server).readTx(ctx, fn func(pgx.Tx) error) error, which runsBEGIN READ ONLY.- CLI:
operator serve,operator add -email <e> [-reset](password on stdin, prints the otpauth URI),operator deactivate -email <e>.
-
Consumes:
ops_auth.*(Task 2),password.Hash/Verify,totp.Enrol/Seal/Open/Validate,config.PrivateListenAddr. -
Step 1: Failing tests in
server_test.go. The helpernewConsole(t)runstestdb.Start. It adds an operator throughops_auth.operator_addas admin, with a known password and a TOTP secret sealed with a test key. It opens anerp_operatorpool onh.AdminDSNwithAfterConnect: SET ROLE erp_operatorand returnshttptest.NewServer(s.Handler()).TestEveryConsoleRouteRequiresASignedInOperator:
func TestEveryConsoleRouteRequiresASignedInOperator(t *testing.T) { c := newConsole(t) routes := c.srv.Routes() // the table Handler() registers from require.GreaterOrEqual(t, len(routes), 4, "route table is nearly empty; the guard would judge nothing")
var public []string for _, r := range routes { if r.Public { public = append(public, r.Pattern) } } require.ElementsMatch(t, []string{"GET /signin", "POST /signin"}, public, "only the sign-in form may be reachable without a session")
// PRESENCE: with a session, a guarded page answers 200. cookie := c.signIn(t) require.Equal(t, http.StatusOK, c.get(t, "/system", cookie).StatusCode)
// ABSENCE: without one, every guarded route redirects to sign-in and runs nothing. for _, r := range routes { if r.Public { continue } method, path := r.Sample() // pattern with {id} filled by a fresh uuid res := c.do(t, method, path, nil) require.Equal(t, http.StatusSeeOther, res.StatusCode, "%s reached its handler without a session", r.Pattern) require.Equal(t, "/signin", res.Header.Get("Location")) }}TestSigninNeedsPasswordAndTOTP:- Presence: the right password and a current code → 303 to
/, and the cookie isHttpOnly,SecureandSameSite=Strict. - The right password with a wrong code, a wrong password with the right code, and an unknown email → 200, the same generic message, and no cookie.
- As admin,
ops_auth.operator_audithas onesigninand threesignin_failedrows.
- Presence: the right password and a current code → 303 to
TestSignoutEndsTheSession: afterPOST /signout, the same cookie gets 303 to/signin.TestCrossSitePostIsRefused:POST /signoutwithOrigin: https://evil.exampleandSec-Fetch-Site: cross-site→ 403.TestSystemPageShowsReleaseAndChain:/systemshows the applied migration version (00235or later once Task 8 lands, so assert>= 231), the console version and “chain intact”.cmd/operator/wiring_test.go:serverefuses to boot whenOPERATOR_LISTEN_ADDRis8.8.8.8:8081or empty, and whenTOTP_ENCRYPTION_KEYis missing.0.0.0.0:8081is accepted (S3).addrefuses a password shorter than 12 characters.internal/app/binaries_test.go: addrequire.Empty(t, linkedModules(t, repoRoot, "operator"), "cmd/operator must link no module: it reads through ops functions only"), then watch it pass for the right reason. Sabotage: add_ "github.com/mcom/erp/internal/modules/reports"tocmd/operator/main.go, and it goes red.layering_test.go: add{"operator", "internal/operator", "github.com/mcom/erp/internal/operator"}afterapp.- Step 2: Run, and watch them fail.
go test ./internal/operator/ ./cmd/operator/ ./internal/app/ ./internal/kernel/layering/ -count=1 - Step 3: Implement. The route table and guard:
type Route struct { Pattern string // net/http 1.22 pattern: "GET /orgs/{id}" Public bool h http.HandlerFunc}
func (s *Server) Routes() []Route { return []Route{ {Pattern: "GET /signin", Public: true, h: s.signinForm}, {Pattern: "POST /signin", Public: true, h: s.signin}, {Pattern: "POST /signout", h: s.signout}, {Pattern: "GET /{$}", h: s.home}, // redirects to /system until Task 7 makes it the overview {Pattern: "GET /system", h: s.system}, // Tasks 7 and 9 append here; nothing registers outside this table. }}
func (s *Server) Handler() http.Handler { mux := http.NewServeMux() for _, r := range s.Routes() { h := http.Handler(r.h) if !r.Public { h = s.requireOperator(h) } mux.Handle(r.Pattern, h) } return http.NewCrossOriginProtection().Handler(securityHeaders(mux))}
// securityHeaders: nothing here may be cached, framed, or load another origin.func securityHeaders(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { h := w.Header() h.Set("Cache-Control", "no-store") h.Set("X-Frame-Options", "DENY") h.Set("Content-Security-Policy", "default-src 'self'; form-action 'self'; frame-ancestors 'none'") h.Set("Referrer-Policy", "no-referrer") next.ServeHTTP(w, r) })}requireOperator reads the op_session cookie and base64url-decodes it into 32 bytes. It calls ops_auth.session_check(sha256) in a short transaction and puts the Operator in the context. Otherwise it redirects 303 to /signin. Sign-in:
func (s *Server) signin(w http.ResponseWriter, r *http.Request) { email := strings.ToLower(strings.TrimSpace(r.FormValue("email"))) var id uuid.UUID var hash string var sealed []byte err := s.DB.QueryRow(r.Context(), `SELECT operator_id, password_hash, totp_sealed FROM ops_auth.signin_lookup($1)`, email).Scan(&id, &hash, &sealed) found := err == nil if !found { hash = dummyHash // same argon2 cost for an unknown email, so timing does not enumerate accounts } pwOK, _ := password.Verify(r.FormValue("password"), hash) ok := found && pwOK && s.totpOK(sealed, r.FormValue("code"))
raw := make([]byte, 32) if _, err := rand.Read(raw); err != nil { http.Error(w, "internal error", http.StatusInternalServerError) return } th := sha256.Sum256(raw) var session *uuid.UUID if err := s.DB.QueryRow(r.Context(), `SELECT ops_auth.signin_result($1, $2, $3)`, email, ok, th[:]).Scan(&session); err != nil { http.Error(w, "internal error", http.StatusInternalServerError) return } if session == nil { // wrong credentials, locked, inactive, or inside 90s: one message for all s.page(w, "signin", map[string]any{"Failed": true, "Email": email}) return } http.SetCookie(w, &http.Cookie{Name: "op_session", Value: base64.RawURLEncoding.EncodeToString(raw), Path: "/", HttpOnly: true, Secure: true, SameSite: http.SameSiteStrictMode}) http.Redirect(w, r, "/", http.StatusSeeOther)}dummyHash is a package-level password.Hash of a random string, computed once in init. /system (V8) renders:
versionops.release_info()ops_auth.chain_status()- the api’s version from
GET {OPERATOR_API_URL}/v1/health, with a 2s timeout and “unreachable” on error
cmd/operator/main.go serve reads, each as an os.Getenv literal:
OPERATOR_DATABASE_URL(required)OPERATOR_LISTEN_ADDR(required, throughconfig.PrivateListenAddr)TOTP_ENCRYPTION_KEY(required, 32 bytes, through the same decoderconfig.Loaduses; export aconfig.TOTPKeyFromEnv()if needed rather than duplicating it)OPERATOR_API_URL,OPERATOR_GRAFANA_URL,OPERATOR_BACKUP_STATUS_FILE(optional)
add and deactivate use DATABASE_MIGRATE_URL, because ops_auth.operator_add is granted to no role. add reads the password from stdin, password.Hashes it, runs totp.Enrol("ERP operator console", email) and Seals the secret. It prints the otpauth URI once, on stdout, and writes nothing else there.
.env.example gains OPERATOR_DATABASE_URL (postgres://erp_operator:…), OPERATOR_LISTEN_ADDR=127.0.0.1:8081, OPERATOR_API_URL, OPERATOR_GRAFANA_URL and OPERATOR_BACKUP_STATUS_FILE. Each has a comment: the console is reached by SSH port-forward (S3), and TOTP_ENCRYPTION_KEY is required by the console.
- Step 4: Run.
go test ./internal/operator/ ./cmd/operator/ ./internal/app/ ./internal/kernel/layering/ ./internal/kernel/config/ -count=1 - Step 5: Sabotage. Mark
GET /systemPublic: true. The guard goes red naming it, and the public-list assertion fires too. Removes.totpOKfromok. The wrong-code case goes red. Restore both. - Step 6: Runtime check. On the dev stack,
ALTER ROLE erp_operator PASSWORD 'erp'as postgres. Thenecho 'correct horse battery' | go run ./cmd/operator add -email ops@demo.test, thengo run ./cmd/operator serve. Sign in in a browser athttp://127.0.0.1:8081with a TOTP app, and/systemshows the migration version. - Step 7: Commit
Operator console T6: cmd/operator with password+TOTP sign-in, guarded routes, releases page
Task 7: Console pages V1–V7
Section titled “Task 7: Console pages V1–V7”Files:
- Create:
internal/operator/orgs.go,internal/operator/jobs.go,internal/operator/templates/{orgs,org,jobs}.html,internal/operator/pages_test.go - Modify:
internal/operator/server.go(routes),internal/operator/system.goandtemplates/system.html(V2, V6)
Interfaces:
-
Produces: routes
GET /{$},GET /orgs/{id}andGET /jobs./systemgains V2 and V6. -
Consumes: Task 5’s functions,
ops.check_latest, and the backup status file format{"last_backup_at": RFC3339, "last_restore_test_at": RFC3339, "wal_archive_lag_seconds": int}. -
Step 1: Failing tests —
pages_test.go:TestOrgsPageShowsEveryOrganizationWithItsFlags: two organizations. A has a failingentry_balancecheck result, inserted as admin, and B’s checks all pass./lists both names. A’s row is marked failing (data-state="fail"on the checks cell) and B’s is markedpass. A check run older than 48 hours is markedstale.TestOrgPageShowsChecksSecurityAndJobs:/orgs/{A}lists each check key with pass/fail andran_at, the security counts and the job counts for A only. Presence: B’s job row exists inops.job_summary(). Absence: B’s job row is not on A’s page.TestSystemPageReadsTheBackupStatusFile: with a file 2 days old, it shows the date. With no file, it shows “unknown” markedfail. WithOPERATOR_API_URLpointing at anhttptestserver that answers/v1/ready503, it shows “not ready”.TestNoConsolePageShowsRecordContents(design §1, PDPA). Seed throughexpensestesta member with a distinctive legal name and email, and a claim line with a distinctive description. Presence: as admin, each value is in the database. Absence: GET/,/orgs/{id},/jobsand/system, and none of the bodies contains any of the three values or the claim amount.
-
Step 2: Run, and watch them fail.
go test ./internal/operator/ -count=1 -
Step 3: Implement.
orgs.gocallsorg_overview,check_latest,compliance_flagsandsecurity_summary(7)in onereadTx. It merges them byorg_idinto:
type orgRow struct { ID uuid.UUID Name, Status string CreatedAt time.Time ActiveMembers int64 LastActivityAt *time.Time ChecksPassed, Checks int ChecksState string // "pass", "fail", "stale", "none" OpenPeriodsPastEnd int64 ReceiptsWithoutBasis int64 FailedLogins7d int64}ChecksState is stale when the newest ran_at is older than 48 hours, even if every check passed. The templates are plain HTML tables with data-state attributes and a small inline stylesheet in layout.html (red, amber, green). There is no JavaScript. /jobs shows job_summary grouped by queue and kind, then the discarded_jobs list. The retry button is added in Task 9. /system adds:
GET {OPERATOR_API_URL}/v1/ready→ ready / not ready (V2)- a link to
OPERATOR_GRAFANA_URLwhen set - the backup file’s three values, with “unknown” when the file is missing or unreadable, and
failwhen the last backup is older than 26 hours or the last restore test older than 92 days (RUNBOOK §3.1, quarterly) - Step 4: Run.
go test ./internal/operator/ -count=1 - Step 5: Sabotage. Add
{{.Email}}of a member toorg.html, fed by a temporarySELECT email FROM usersas admin.TestNoConsolePageShowsRecordContentsgoes red. Restore. - Step 6: Commit
Operator console T7: organizations, jobs and system pages (V1-V7), counts and pass/fail only
Task 8: Operator actions in the worker (A2, A3, A4, A5)
Section titled “Task 8: Operator actions in the worker (A2, A3, A4, A5)”Files:
- Create:
db/migrations/00235_ops_actions.sql,internal/platform/ops/actions.go,internal/platform/ops/actions_test.go - Modify:
cmd/worker/exports.go,cmd/worker/main.go(blank-import every module),cmd/worker/wiring_test.go,internal/app/binaries_test.go(worker joinsmigrate,orgadmin),db/rls_exempt.txt(ops_auth.operator_action),internal/kernel/db/opsschema_test.go
Interfaces:
-
Produces:
ops_auth.operator_action.ops.request_action(p_session bytea, p_kind text, p_org uuid, p_owner_member uuid, p_job bigint) → uuid(erp_operator)ops.claim_action(p_id uuid) → (kind text, org_id uuid, owner_member_id uuid, job_id bigint)(erp_app)ops.finish_action(p_id uuid, p_ok boolean, p_detail text) → void(erp_app)ops_auth.action_log(p_limit integer) → (action_id uuid, operator_email text, kind text, org_id uuid, job_id bigint, state text, detail_code text, requested_at timestamptz, finished_at timestamptz)(erp_operator)- Job kind
operator_action(ops.ActionArgs{ActionID uuid.UUID}) andops.ActionWorker{DB *db.DB; Clock clock.Clock}. - Detail codes:
owner_member_required,company_profile_required,owner_member_invalid,already_suspended,already_active,job_not_retryable,error.
-
Step 1: Failing tests —
actions_test.go. Each creates a signed-in operator session as in Task 6, via SQL as admin, and requests throughops.request_actionaserp_operator. It then works the queue with a realjobs.NewWorkerClientoverh.App, registeringActionWorker, with a 50ms poll.TestARequestedActionIsWorkedByRiver: requestingrun_checksfor A leaves oneriver_jobrow of kindoperator_action. After the client runs, the action issucceededand A hasops.check_resultrows. This test is what proves Ruling O5’s raw insert.TestSuspendAndReactivate:- Presence: A is
active, and anerp_apptenant transaction can update A’s company profile. - After
suspendis worked,organization.statusissuspended. - A’s
audit_loghasentity_kind = 'organization',action = 'suspended',actor_id IS NULL, andaftercontaining the action id. SELECT count(*)of A’s claims is unchanged, so nothing was deleted.reactivaterestoresactiveand writesreactivated.- Suspending twice ends
failedwithalready_suspendedand writes no second audit row.
- Presence: A is
TestProvisionRunsTheRegistry: on a fully provisioned organization,provisionwithout an owner endssucceededand writesaudit_logaction = 'provisioned'. On one missing theorg_structurestep (expensestesthas a helper, or delete that step’s marker the wayinternal/platform/provisioningtests do), it endsfailedwithowner_member_required.TestRetryJobOnlyRetriesDiscarded:- Presence: a
discardedexport job, inserted as admin, can be requested. After working, River shows itavailableorretryable. - Refusals:
request_actionfor acompletedjob, and for a job of kindoperator_action, is refused with the constraint nameoperator_action_job_retryable.
- Presence: a
TestRequestActionAttributesTheSessionsOperator:- Presence:
request_actionwith a live session succeeds, andaction_lognames that operator’s email. - With a revoked session hash it raises
42501. - A second operator’s session is recorded as the second operator.
- Presence:
TestTheWorkerLinksEveryModule:binaries_test.go’s loop becomes{"migrate", "orgadmin", "worker"}.
-
Step 2: Run, and watch them fail.
go test ./internal/platform/ops/ ./internal/app/ -run 'Action|Suspend|Provision|Retry|Binary' -count=1 -
Step 3: Implement
00235_ops_actions.sql:
-- +goose UpCREATE TABLE ops_auth.operator_action ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), operator_id UUID NOT NULL REFERENCES ops_auth.operator_account (id), kind TEXT NOT NULL CONSTRAINT operator_action_kind_known CHECK (kind IN ('provision','run_checks','retry_job','suspend','reactivate')), org_id UUID, owner_member_id UUID, job_id BIGINT, state TEXT NOT NULL DEFAULT 'requested' CONSTRAINT operator_action_state_known CHECK (state IN ('requested','running','succeeded','failed')), detail_code TEXT CONSTRAINT operator_action_detail_shape CHECK (detail_code ~ '^[a-z_]{1,40}$'), requested_at TIMESTAMPTZ NOT NULL DEFAULT now(), finished_at TIMESTAMPTZ, CONSTRAINT operator_action_target_shape CHECK ( CASE WHEN kind = 'retry_job' THEN job_id IS NOT NULL AND org_id IS NULL ELSE org_id IS NOT NULL AND job_id IS NULL END), CONSTRAINT operator_action_owner_only_for_provision CHECK (owner_member_id IS NULL OR kind = 'provision'));ALTER TABLE ops_auth.operator_action OWNER TO erp_ops_owner;
-- +goose StatementBeginCREATE FUNCTION ops.request_action(p_session bytea, p_kind text, p_org uuid, p_owner_member uuid, p_job bigint) RETURNS uuid LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, public, pg_temp AS $$DECLARE v_operator uuid := ops_auth.session_operator(p_session); v_id uuid;BEGIN -- Attribution comes from the session, never from a parameter (Ruling O3). IF v_operator IS NULL THEN RAISE EXCEPTION 'ops.request_action: no live operator session' USING ERRCODE = '42501'; END IF; IF p_kind = 'retry_job' AND NOT EXISTS ( SELECT 1 FROM public.river_job WHERE id = p_job AND state = 'discarded' AND kind <> 'operator_action') THEN RAISE EXCEPTION 'job % is not a discarded job', p_job USING ERRCODE = '23514', CONSTRAINT = 'operator_action_job_retryable'; END IF; IF p_kind <> 'retry_job' AND NOT EXISTS (SELECT 1 FROM public.organization WHERE id = p_org) THEN RAISE EXCEPTION 'organization % does not exist', p_org USING ERRCODE = '23514', CONSTRAINT = 'operator_action_org_exists'; END IF; INSERT INTO ops_auth.operator_action (operator_id, kind, org_id, owner_member_id, job_id) VALUES (v_operator, p_kind, p_org, p_owner_member, p_job) RETURNING id INTO v_id; PERFORM ops_auth.audit_append(v_operator, 'action_requested', p_org, v_id); -- River's columns: kind, args, queue, max_attempts; the rest default -- (00111-00116). The worker is poll-only (kernel/jobs), so no NOTIFY. -- Ruling O5. INSERT INTO public.river_job (kind, args, queue, max_attempts) VALUES ('operator_action', jsonb_build_object('action_id', v_id), 'maintenance', 1); RETURN v_id;END$$;-- +goose StatementEndclaim_actionisUPDATE ops_auth.operator_action SET state = 'running' WHERE id = p_id AND state = 'requested' RETURNING kind, org_id, owner_member_id, job_id, so a replayed job claims nothing.finish_actionsetsstatetosucceededorfailed, and setsdetail_codeandfinished_at,WHERE id = p_id AND state = 'running'.action_logjoinsoperator_accountfor the email, newest first, and clamps the limit to 1..200.
Grants: GRANT INSERT (kind, args, queue, max_attempts) ON public.river_job TO erp_ops_owner; and GRANT USAGE ON SEQUENCE public.river_job_id_seq TO erp_ops_owner;. Add both to opsOwnerGrants. The test’s column query covers INSERT column ACLs. Extend it with a pg_class sequence check if the sequence grant is not visible in attacl, and give it its own line in the map, keyed river_job_id_seq.
actions.go:
func (w *ActionWorker) Work(ctx context.Context, j *river.Job[ActionArgs]) error { var a struct { Kind string OrgID *uuid.UUID OwnerMemberID *uuid.UUID JobID *int64 } err := w.DB.Pool().QueryRow(ctx, `SELECT kind, org_id, owner_member_id, job_id FROM ops.claim_action($1)`, j.Args.ActionID).Scan(&a.Kind, &a.OrgID, &a.OwnerMemberID, &a.JobID) if errors.Is(err, pgx.ErrNoRows) { return nil // already claimed: a replayed job does nothing } if err != nil { return fmt.Errorf("ops: claim %s: %w", j.Args.ActionID, err) } detail, runErr := w.run(ctx, j.Args.ActionID, a.Kind, a.OrgID, a.OwnerMemberID, a.JobID) if runErr != nil && detail == "" { detail = "error" } if _, err := w.DB.Pool().Exec(ctx, `SELECT ops.finish_action($1, $2, $3)`, j.Args.ActionID, runErr == nil, nullIfEmpty(detail)); err != nil { return fmt.Errorf("ops: finish %s: %w", j.Args.ActionID, err) } return nil // the outcome is recorded; River must not retry an operator action}run dispatches on the kind:
run_checks→RunChecks.retry_job→river.ClientFromContext[pgx.Tx](ctx).JobRetry(ctx, *jobID).suspend/reactivate→ indb.InTx(tenancy.WithOrg(ctx, org)), runUPDATE organization SET status = $2 WHERE id = $1 AND org_id = $1 AND status <> $2. If 0 rows, returnalready_suspendedoralready_activewith an error. Otherwiseaudit.Write{EntityKind: "organization", EntityID: org, Action: "suspended"|"reactivated", After: map[string]any{"by": "service_provider", "operator_action_id": id}}, last.provision→ indb.InTx, runprovisioning.CheckOwnerMemberwhen an owner is named, thenprovisioning.Run(ctx, tx, w.Clock, provisioning.Input{OwnerPermissions: member.RegisteredPermissionSet(), OwnerMemberID: owner}), thenaudit.Writeaction: "provisioned", with the sameAfterplus the done step keys. MapErrOwnerMemberRequired,ErrCompanyProfileRequiredandErrOwnerMemberNotFoundto their codes witherrors.Is, asinternal/modules/identity/handlers.goprovisiondoes.
Register ActionWorker in startExportWorker. cmd/worker/main.go blank-imports the modules it does not yet import: activity, identity, orgstructure, payout, reports, search, workflow. Add the same comment cmd/orgadmin/main.go gives: the permission registry is filled by init, and a worker-run provision must seed the same Owner. db/rls_exempt.txt gains ops_auth.operator_action, under the Task 2 comment.
- Step 4: Run.
go test ./internal/platform/ops/ ./internal/kernel/db/ ./internal/app/ ./cmd/worker/ -count=1 - Step 5: Sabotage. First, make
request_actionskip thev_operator IS NULLrefusal, and the revoked-session case goes red. Second, drop thekind <> 'operator_action'clause, and the retry refusal case goes red. Third, make the suspend branch return nil when theUPDATEmatched 0 rows, and the second-suspend case goes red. Restore each. - Step 6: Commit
Operator console T8: operator actions run in the worker, org audit_log mirrored (A2-A5)
Task 9: Console action UI and action log
Section titled “Task 9: Console action UI and action log”Files:
- Create:
internal/operator/actions.go,internal/operator/actions_test.go,internal/operator/templates/{confirm,actions}.html - Modify:
internal/operator/server.go(routes),templates/org.html(buttons),templates/jobs.html(retry buttons)
Interfaces:
-
Produces: routes
GET /orgs/{id}/actions/{kind}(confirmation page),POST /orgs/{id}/actions/{kind},POST /jobs/{id}/retryandGET /actions. -
Consumes:
ops.request_actionandops_auth.action_log. -
Step 1: Failing tests:
TestSuspendNeedsConfirmationAndIsRecorded: the GET confirmation page names the organization and the action, and has no side effect. As admin,operator_actionstays empty (presence of the table is shown by the next step). The POST leads with 303 to/actions, onerequestedrow and oneaction_requestedaudit row.TestUnknownActionKindIs404:POST /orgs/{id}/actions/delete→ 404, and no row.TestRetryRefusalIsShownNotCrashed: retrying acompletedjob → the confirmation page re-renders with “This job cannot be retried: only discarded jobs can.” and returns 422, not 500. The mapping keys on the constraint nameoperator_action_job_retryablefrom*pgconn.PgError.TestActionsPageShowsOutcome: after Task 8’s worker finishes arun_checks,/actionsshowssucceeded.TestEveryConsoleRouteRequiresASignedInOperator(Task 6) now covers the new routes. Confirm its route count grew.
-
Step 2: Run, and watch them fail.
go test ./internal/operator/ -count=1 -
Step 3: Implement. The allowed kinds from the path are
provision,run_checks,suspendandreactivate. Anything else is 404.provisionaccepts an optionalowner_member_idform field, parsed as a uuid; if it is malformed, the page re-renders with 422. The POST callsSELECT ops.request_action($1, $2, $3, $4, $5)with the session’sTokenHashfrom the context. Confirmation pages state what the action does in one sentence, in English:- suspend: “Members keep read and export access; every change is refused until reactivated. Nothing is deleted.”
- provision: “Runs the provisioning steps that are not done yet; steps already done are left alone.”
/actionsrendersops_auth.action_log(100). The org page shows Suspend or Reactivate by status, plus Run checks now and Re-run provisioning. The jobs page shows Retry on each discarded row. -
Step 4: Run.
go test ./internal/operator/ ./internal/platform/ops/ -count=1 -
Step 5: Sabotage. Make the GET confirmation call
request_action. The no-side-effect assertion goes red. Restore. -
Step 6: Commit
Operator console T9: action confirmation, forms and action log
Task 10: Dev stack, end to end, runbook and deploy notes, roadmap, full suite
Section titled “Task 10: Dev stack, end to end, runbook and deploy notes, roadmap, full suite”Files:
- Modify:
scripts/dev-stack.sh,web/playwright.config.ts,docs/RUNBOOK.md,docs/DEPLOY_NOTES.md,docs/ROADMAP.md,docs/superpowers/specs/2026-09-25-operator-console-progress.md - Create:
web/e2e/totp.ts,web/e2e/operator-console.spec.ts
Interfaces:
-
Consumes: everything above;
as,apiandrunTagfromweb/e2e/support.ts. -
Produces:
<STATE>/operator.json={"email","password","otpauth"}(mode 600), written bydev-stack.sh. -
Step 1: Parameterise the stack for lane 6 (skip whatever another Phase 2 item has already landed). In
dev-stack.sh,PG_PORT,MINIO_PORT,API_PORT,WEB_PORTandOPERATOR_PORTbecome${VAR:-default}, with the defaults unchanged: 55432, 59000, 3000, 5173 and 8081. The container names andSTATEtake aDEV_STACK_NAMEsuffix.PORT,PUBLIC_BASE_URLand the health URL use the variables. Aftermigrate up, rundocker exec "$PG" psql -U postgres -d erp -c "ALTER ROLE erp_operator PASSWORD 'erp'". IfTOTP_ENCRYPTION_KEYis empty in$STATE/env, append a generated one. Build and startcmd/operatorwithOPERATOR_LISTEN_ADDR=127.0.0.1:$OPERATOR_PORTandOPERATOR_API_URL=http://127.0.0.1:$API_PORT. On first seed, createops@demo.testwith a generated password, and write$STATE/operator.json.playwright.config.tsreadsE2E_WEB_PORTandE2E_API_PORT, defaulting to today’s values, and passes the variables to thewebServercommand. -
Step 2: Write the e2e —
web/e2e/totp.ts:
import { createHmac } from 'node:crypto';
/** RFC 6238 TOTP (SHA-1, 6 digits, 30 s), what identity/totp validates. */export function totp(secretBase32: string, at = Date.now()): string { const alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ234567'; let bits = ''; for (const c of secretBase32.replace(/=+$/, '').toUpperCase()) bits += alphabet.indexOf(c).toString(2).padStart(5, '0'); const key = Buffer.from((bits.match(/.{8}/g) ?? []).map((b) => Number.parseInt(b, 2))); const counter = Buffer.alloc(8); counter.writeBigUInt64BE(BigInt(Math.floor(at / 30_000))); const h = createHmac('sha1', key).update(counter).digest(); const o = h[h.length - 1] & 0xf; return String((h.readUInt32BE(o) & 0x7fffffff) % 1_000_000).padStart(6, '0');}operator-console.spec.ts covers the main path:
- Read
operator.json. Openhttp://127.0.0.1:${E2E_OPERATOR_PORT}/signin, fill the email, password andtotp(secret), and expect “Demo Sdn Bhd” in the organizations table. - Presence for the PDPA check: as the seeded owner in the web app, the profile page shows
owner@demo.test. Absence: the console’s/and/orgs/{id}page text contains neitherowner@demo.testnorAisyah binti Rahman. - Open the org page, then Suspend, then Confirm. Reload
/actionsuntil the row showssucceeded(poll up to 20 seconds). - In the web app as the owner, a read page (claims list) loads. Then the first write action from
claim-flow.spec.ts’s opening step fails, and the toast shows the enorg_suspendedsentence. - In the console, Reactivate. Once
succeeded, the same write succeeds. - Run checks now, then
/shows the organization’s checks aspass.
Run on lane 6: DEV_STACK_NAME=lane6 PG_PORT=56472 MINIO_PORT=59160 API_PORT=3160 WEB_PORT=5197 OPERATOR_PORT=3161 E2E_WEB_PORT=5197 E2E_API_PORT=3160 E2E_OPERATOR_PORT=3161 pnpm -C web exec playwright test operator-console, then DEV_STACK_NAME=lane6 scripts/dev-stack.sh down.
-
Step 3: RUNBOOK. Add
§7 Operator console:- Access: the SSH port-forward (
ssh -N -L 8081:127.0.0.1:8081 ops@vps) and theauthorized_keysrestriction line (S3). - Operators:
operator add/-reset/deactivate, run withDATABASE_MIGRATE_URL. The otpauth URI is a credential shown once. Another operator’s reset is the only recovery path. - The backup status file format and who writes it (I14).
- What suspension does (S2).
- The operator audit chain: kept seven years, never pruned, and verified on
/system.
Then write the infrastructure checklist I1–I15, copied from the progress file §4.1, one
- [ ]line each. Each line names the Stage 0 decision it follows (I2 → S3, I7 → S4). Add to §1.1’s object-lock line: “the bypass permission (s3:BypassGovernanceRetention) is held only by a break-glass MinIO identity that no service uses. Record every use (date, person, reason) (S4)”. Extend the §2 configuration table with anoperatorcolumn, theMETRICS_LISTEN_ADDRrow and theOPERATOR_*rows. - Access: the SSH port-forward (
-
Step 4: DEPLOY_NOTES. Add a newest-first entry, “Operator console”:
- Migrations
00230–00235create the roleserp_ops_owner/erp_operatorand the schemasops/ops_auth, and addorganization.status(every existing organization becomesactive). - Set a password:
ALTER ROLE erp_operator PASSWORD '<secret>'. The role ships without one and cannot connect until it has one. - The migration role must be able to create a
BYPASSRLSrole. That needs a superuser, which the.env.exampleDATABASE_MIGRATE_URLalready is. Say what breaks if it is not:00230fails. - The new
operatorbinary and its env vars (TOTP_ENCRYPTION_KEYis now required for it). Create the first operator withoperator add. METRICS_LISTEN_ADDRis optional.- The worker now runs nightly checks and operator actions on the
maintenancequeue, and now links every module. - Nothing to re-provision.
- Migrations
-
Step 5: ROADMAP. Under “Operator console”,
Stage 0 decisionsis already ticked. TickPlan and build, and add the e2e path (web/e2e/operator-console.spec.ts) and the drift tests (internal/kernel/db/opsschema_test.go). Add open sub-items: “Stage 5 support access — own design (S9)” and “A1 create organization in the console (S8)”. Where R30 is listed under Plan 9, mark “nightly reconciliation” done by this item. The BNM sync and idempotency pruner items stay open. Update the progress file’s status line to “Stages 1–4 built”. -
Step 6: Full suite, once. Run
go test -p 2 -count=1 ./...,make lint, andmake web(typecheck, lint, test, build). Thenmake generate, followed by a cleangit status. -
Step 7: Commit
Operator console T10: dev stack and e2e on lane 6, runbook infra checklist, deploy notes, roadmap
Tasks in the same wave touch disjoint files and can run in parallel. A later wave starts when every task it depends on has merged.
| Wave | Tasks | Depends on | Why this order |
|---|---|---|---|
| 1 | T1, T2 | — | T1 is kernel, cmd and Makefile. T2 is migrations 00230–00231 and internal/kernel/db tests. Disjoint. |
| 2 | T3, T4, T6 | T2 (range registered, schemas, opsschema_test.go), and T1 for T6’s Makefile, .env.example and PrivateListenAddr |
T3 is migration 00232, liveness, httpx auth, openapi and web. T4 is 00233, platform/ops, cmd/worker and opsOwnerGrants. T6 is cmd/operator, internal/operator, layering and binaries tests. Only T4 edits opsschema_test.go. |
| 3 | T5 | T3 (organization.status), T4 (check_result) |
Alone: it edits opsOwnerGrants. |
| 4 | T7, T8 | T5, T6 (T7); T3, T4, T5 (T8) | T7 is internal/operator pages only. T8 is 00235, platform/ops/actions.go, cmd/worker, binaries_test.go, rls_exempt.txt and opsschema_test.go. Disjoint. |
| 5 | T9 | T7, T8 | Console action UI. |
| 6 | T10 | all | Stack, e2e, docs, full suite. |
Dependencies on other Phase 2 items: none for interfaces. Two things touch shared ground:
- If another item parameterises
scripts/dev-stack.shfor lanes first, T10 Step 1 reuses it. - The payment-batches range (
00330–00339) overlaps AP’s registered{300, 339}inmigrationranges_test.go. That is not this plan’s to fix, but T2’s edit to that file must not reformat the table.
Self-Review
Section titled “Self-Review”-
Spec coverage:
Design item Task Stage 1 T1 Stage 2 T2, T6 Stage 3 V1, V3, V5, V7 in T5 and T7; V4 in T4, T5 and T7; V6 and V2 in T7; V8 in T6 Stage 4 A2–A5 in T8 and T9; A5 enforcement in T3 §6 drift tests Each is in T1, T2 or T6, plus the owner-grants and no-record-text tests A1, A6, A7 and Stage 5 are deferred (S8, S9), and the roadmap records them.
-
Invariants:
- No float touches money. There is none.
- The one business table is RLS-forced.
- Every request runs in a transaction. Console pages use
readTx, and API writes are unchanged. - Vocabulary values are strings:
status,kindandstate. - The ledger is untouched.
- Accounting is not referenced by state name.
- Translation rows: the only customer-facing string is translated.
- “Nobody grants themselves authority”: no operator path grants access to an organization. Support access is out.
-
Unverified here, verified by the named test:
- River accepting a raw-inserted row (T8
TestARequestedActionIsWorkedByRiver). - The FK on
ops.check_resultunder the owner’s grants (T4 Step 3 note). - Column ACLs covering
INSERTcolumn grants and the sequence (T8 Step 3 note). - Chrome and Firefox accepting a
Securecookie onhttp://127.0.0.1(T6 Step 6 runtime check). They do onlocalhost. Check the literal IP in the runtime step, and usehttp://localhost:<port>in the RUNBOOK if not.
- River accepting a raw-inserted row (T8