Public API and webhooks — design

Phase 6 item: “Public API and webhooks on the same OpenAPI contract” (ROADMAP.md; architecture spec section 16 and 19).

Status: design only, nothing built. No code, no migrations. Migration range proposed: 00960-00979 (see Task plan). Every statement about the current code was read on 2026-10-03; anything not read or not confirmed is collected under “Unverified” at the end and marked (U) inline.

1. What exists today, and what that forces

Read before designing, because three findings change the shape of the work.

There is one credential type. httpx.Authenticate accepts only an HS256 JWT bearer (15 minutes), then calls liveness.Check (session live, member active, org not suspended, perm_version current; Redis-cached 5 s, falls through to Postgres, never fails open). Identity{UserID, OrgID, MemberID, SessionID} is built from signed claims only. httpx.Tenant() copies OrgID into the context; db.InTx sets app.current_org. httpx.Protected(group, loader, method, path, permission, handler) registers the route, records "METHOD /path" -> permission and attaches Authorize in one call; TestEveryRouteIsAuthorized compares the assembled router with that record and with api/public_routes.txt. Machine callers must enter through this same chain, not beside it.
The event bus is not durable. internal/kernel/eventbus buffers events per transaction and dispatches them in-process after commit; a handler error stops the flush; a process crash between commit and flush loses the event; Publish from a post-commit handler panics. That is correct for notifications and wrong for webhooks, which must survive a crash. Only four kinds are published today (workflow.transitioned, receivables.document_issued, expenses.comment_added, documents.uploaded), and their payloads are in-process Go values, not versioned contracts. A transactional outbox is therefore part of this item, not an optimisation.
There is no outbound HTTP client with any policy, and no rate limiter. The architecture spec lists Redis rate limiting “per user and per IP” (section 15); grep found no limiter in internal/kernel/httpx and Redis is used only by liveness (U: confirmed only by search; the task plan starts with a re-check). The e-Invoice worker talks to a fixed authority host; nothing here takes a customer-supplied URL.

Also relevant and reusable: per-module Idempotency-Key tables (pettycash_idempotency_key, claimed first in the handler’s transaction, same-key-different-hash refused, a failed request returns its key); keyset cursors (next_cursor) already in the contract; Problem.code and the x-codes extension for stable refusal codes; payout.Keyring (AES-256-GCM, key id stored beside the ciphertext, PAYOUT_DETAILS_KEY and _PREVIOUS) for sealing a secret that must be recoverable; the ops.org_ids() and einvoice_due_orgs pattern for a worker that needs to visit every organization; River with MaxAttempts: 1 plus table-driven backoff (einvoice/worker.go); the erp_hr_leavings_failing gauge for the monitoring shape.

2. Principles

One contract, one chain. The public API is a subset of api/openapi.yaml, served by the same handlers, behind the same Authenticate -> Tenant -> Authorize -> db.InTx. No second router, no parallel handler set, no hand-maintained inventory.
A machine is never more powerful than a person. Effective permission = (key scopes) intersect (its owner’s current permissions).
Fail closed, and prove the guard exists before proving it blocks. Every guard below has a drift test written presence-first and watched failing once (section 12).
Data leaving the system is minimal by default (section 10).
At-least-once, unordered, replayable. Receivers dedupe on webhook-id.

3. Machine credentials

3.1 Decision: API keys now, OAuth client-credentials later if asked

	API key	OAuth client-credentials
Fit to the existing chain	One new branch in `Authenticate`	Needs a token endpoint, client registry, token store or a second JWT audience
Revocation latency	Row lookup, same 5 s cache as liveness	Bounded by token lifetime unless introspected
Integrator effort	Header	Token fetch and refresh
Standards-ecosystem expectation	Normal for ERPs of this class	Expected by large enterprise integrators

Recommendation: API keys for v1. The key is a bearer credential in the same Authorization: Bearer header, recognised by a literal prefix, so the OpenAPI securitySchemes gains one more scheme (apiKeyAuth, type: http, scheme: bearer) and nothing else about the document’s shape changes. Client-credentials is a compatible later addition: it would mint a short-lived token that resolves to the same Credential principal (section 3.5), so nothing in 3.2-3.8 is thrown away.

3.2 Shape of a key

mck_<key_id>_<secret> where key_id is a short non-secret identifier (lookup handle, also shown in the admin list and in audit rows) and secret is 32 random bytes, base64url. The mck_ prefix lets Authenticate choose the branch without parsing, lets secret scanners (GitHub, trufflehog) recognise a leaked key, and keeps keys visibly distinct from JWTs.

3.3 Storage and display

Hashed at rest. The secret is 256-bit random, so a single SHA-256 is sufficient (a slow password hash adds latency on every API request and protects nothing here: there is no weak secret to brute-force). Store secret_hash; compare with subtle.ConstantTimeCompare. Never store, log or audit the secret. Only the key_id appears anywhere after creation.
Show once. The create (and rotate) response is the only time the plaintext exists. The response carries Cache-Control: no-store. There is no “reveal” endpoint and no list field that could contain it; a drift test (section 12) asserts no read response contains secret or secret_hash.
Key custody ties to the runbook: a leaked key is revoked, not rotated in place.

3.4 Lifecycle

Event	Behaviour
Create	Admin only (`integrations.apikey.manage`). Name (required), scopes, expiry required (default 90 days, maximum 365). Owner = the creating member.
Scopes at creation	Chosen from permissions flagged `api_grantable` (3.6) and held by the creator right now. A creator cannot grant what they lack.
Effective permission at request time	scopes intersect the owner’s current permissions (the existing `member.Permissions` loader). Demote the owner and the key shrinks the same second the role cache turns over.
Owner deactivated or leaves	Key stops authenticating (the liveness check includes “owner active”). It is not silently re-homed. (Owner decision D2.)
Rotate	Creates a successor and sets the predecessor’s `expires_at = least(existing, now + overlap)`; overlap default 24 h, maximum 7 days, so an integrator can deploy the new key without downtime. Both are audited and linked (`rotated_from`).
Revoke	Immediate. `revoked_at` is checked in the same lookup; the Redis-cached positive answer is bounded at 5 s like liveness, so the worst case is 5 s and never fail-open.
Expiry	Checked per request against the injected `clock.Clock`. An expired key gets the same generic 401 as a wrong key. A job notifies the owner and org admins 14 days and 2 days before expiry (notifications module, translation rows).
Last used	`last_used_at` and `last_used_ip` are updated at most once per hour per key (`UPDATE ... WHERE last_used_at < now() - interval '1 hour'`) inside the request’s own tenant transaction, so there is no hot row and no extra connection. A key unused for 90 days is surfaced in the admin screen as stale.
Audit	`audit.Write` for create, rotate, revoke, scope change, expiry-warning sent, and for the first use of a key and any use after a long idle. Every audited write performed through a key records the key id alongside the actor (3.5).

Per-request authentication failures are not audited per request (that is a log-flood vector); they are counted (section 11) and rate-limited per source IP.

3.5 How a key authenticates without breaking the invariants

Authenticate sees the mck_ prefix and calls the key resolver instead of TokenVerifier/LivenessChecker. The kernel declares the interface (KeyResolver); platform/identity/apikey implements it and cmd/api adapts it, exactly as the three existing interfaces are wired. Kernel does not import platform.
The lookup has to run before a tenant exists (the org is derived from the key), but every business table is FORCE ROW LEVEL SECURITY. Resolve through one SECURITY DEFINER function, identity.api_key_lookup(key_id), returning only (org_id, member_id, secret_hash, scopes, expires_at, revoked_at, owner_active, org_suspended). It is added to the SECURITY DEFINER allowlist and is not executable by PUBLIC, which the existing drift test already enforces. The organization is never read from a header or body; the no handler reads an organization from a request header test keeps passing unchanged.
On success the middleware builds the same Identity the JWT path builds, plus Credential{KeyID, Scopes}. audit.WithActor(ctx, orgID, memberID) is set exactly as today, with the credential id carried alongside so the audit row can say “member M via key K”. SessionID is the zero UUID; nothing may treat that as a session. (U: audit_log columns and the hash-chain input; adding a nullable credential_id must be checked against chain_test.go before choosing between a column and a field inside after.)
httpx.Tenant() runs after, unchanged. The handler runs in db.InTx, unchanged. A read through a key is still a transaction with SET LOCAL app.current_org.

3.6 Which permissions a key may hold

Permissions are opt-in to the API, not opt-out. The permission registry gains a boolean api_grantable (default false). Rules:

Never grantable: anything in identity.* (members, roles, sessions), integrations.* (a key cannot mint keys or webhooks: no self-service escalation), organization provisioning, approval/decision permissions (*.approve, payroll.run.approve, …). This preserves “a claimant never picks their own approver” and segregation of duties: a key can submit and read, a person approves.
Payroll, salary and HR-person data are not grantable in v1 (section 10).
A drift test pairs the registry with the contract: every public operation’s permission is api_grantable, and every api_grantable permission is used by at least one public operation. A grantable permission with no public operation, or a public operation whose permission is not grantable, fails the build.

3.7 Authorize stays the single gate

Authorize(loader, key) is extended, not bypassed: after the existing member-permission check it additionally requires, when Identity.Credential != nil, that key is in the credential’s scopes. httpx.Protected is unchanged, so every public operation is still recorded in AuthorizedRoutes() and TestEveryRouteIsAuthorized still binds it to the router.

A second middleware, PublicSurface(), sits in the /v1 chain after Authenticate and answers a different question: is this method and path part of the published API at all? If Credential != nil and the matched route is not in the generated public set, it answers 403 not_in_public_api before any handler runs. This is what stops a key reaching /v1/members, /v1/auth/* or the key-management endpoints even if someone later makes a permission grantable by mistake. The two guards are deliberately redundant (permission and surface), and each has its own test.

3.8 Org-level switch

New org setting public_api_enabled, default off. An administrator turns it on (audited). The platform operator console gets one kill action, “suspend public API for this org” (distinct from suspending the whole org), recorded in the operator audit. The key resolver returns org_suspended or public_api_disabled as the same generic 401. (Owner decision D3.)

4. The public part of the contract

4.1 How an operation becomes public

An explicit marker on the operation in api/openapi.yaml, not an inferred rule:

x-public-api:
  since: "1"

plus the tag Public API. Allowlist, not blocklist: nothing is public until someone writes the marker in the same change that reviews its payload for personal data and its permission for grantability. A new internal endpoint is private by default.

tools/ gains a filter that emits api/public-openapi.yaml: only marked operations, only the schemas they transitively reference, only the apiKeyAuth and bearerAuth schemes, servers: /v1 unchanged. The file is committed and a drift test asserts it equals the filter’s output from the source (the same “generated code matches its source of truth” family as the existing test). It is the document customers receive. Whether the document is also served is an owner decision (D9); serving it would be a new entry in api/public_routes.txt under [no-permission], which is exactly the review that file is for.

4.2 What the public API is not

Routes outside x-public-api still serve the web app and sessions as today and are free to change with it. The public set is the only set with a compatibility promise.

4.3 Versioning and deprecation

The URL major (/v1, already the servers: value) is the API major. Within a major, only additive change: new optional fields, new endpoints, new enum values on response fields documented as open, new webhook event types. Clients must ignore unknown fields (stated in the published document).
A breaking change ships as /v2 side by side with /v1 operations marked deprecated: true, x-sunset: <date>; responses carry Deprecation and Sunset headers (RFC 8594 and the Deprecation header draft (U: whether it has been finalised as an RFC; the header name is what matters)). Minimum notice before removal: 12 months (D6).
Because public and web share handlers, a handler change is a public API change. The protection is a contract-diff drift test: the committed public document is compared with a committed baseline of the last released version, and the only differences allowed are the additive ones above. Removing an operation, a response field, an enum value, a documented error code, or making an optional request field required fails the build. Releasing a new baseline is a deliberate, reviewed commit. (U: whether oasdiff may be added as a tool dependency; the fallback is a small purpose-written comparison over the generated Go structures, covering exactly the rules listed. The task plan decides after a spike.)
Money stays a string, vocabulary values stay strings, ids stay UUID strings: these are already invariants and are tested over the public document specifically (no number type on a property whose name or x- annotation marks money; no integer on an enum).

4.4 Stable error codes

Errors are application/problem+json with the existing code field. For the public set:

Every non-2xx response of a public operation declares its possible codes with the existing x-codes extension. A drift test fails a public operation whose non-2xx response has none.
A code, once published, is never removed or repurposed (covered by the contract-diff test).
New public-layer codes: unauthenticated (generic 401, never says why), not_in_public_api, insufficient_scope, rate_limited, idempotency_key_required, idempotency_key_reused, idempotency_in_progress.
401 never distinguishes wrong, expired, revoked or org-suspended keys (the same discipline Authenticate already applies to tokens).

4.5 Pagination

Public list operations use keyset cursors only: limit (default 50, maximum 200, over-limit is a 422 not a silent clamp) and an opaque cursor; the response carries next_cursor (absent on the last page). Stable order is (created_at, id) or the resource’s documented sort plus id. offset is not part of the public contract (one internal ledger operation documents limit+offset; it is not marked public). A drift test over the public document requires every array-returning GET to declare limit, cursor and next_cursor.

5. Idempotency for writes

Every public non-GET operation requires Idempotency-Key (1-255 chars), already the convention for money-moving POSTs. Missing -> 422 idempotency_key_required.
A generic table api_idempotency_key(org_id, credential_id, key, request_hash, status_code, response, created_at) keyed on (org_id, credential_id, key). Scoped to the credential, not the member: an integration and the person who owns it must not collide, which the per-module member-scoped tables cannot promise. The hash covers method, matched route pattern, path parameters and canonicalised body.
Mechanics copy the established ones: claimed first in the request’s own transaction (a concurrent duplicate blocks on the row and replays the stored answer when the first commits, or proceeds if it rolled back); same key plus different hash -> 409 idempotency_key_reused; a failed request returns its key; a replay returns the stored status and body unchanged with Idempotent-Replayed: true.
Modules that already own an idempotency table keep it (it is inner; the generic layer is the outer one and short-circuits true replays before the module sees them). The same client header value feeds both and the two never conflict inside one transaction. (U: this interplay is reasoned from reading pettycash/idempotency.go; the first task builds a test with a module-keyed endpoint before anything depends on it.)
Retention: 24 hours, pruned by a maintenance job; the table is operational and joins the deletable list in retention_test.go.
Advisory locks, if any are needed for the claim, take a lock class registered in docs/LOCK_ORDER.md, per the existing convention. The claim is the first lock the transaction takes; audit.Write remains last.

6. Rate limits

Per credential (key id) token bucket, plus a per-organization ceiling so an org with many keys cannot multiply its share, plus a per-source-IP limiter for unauthenticated failures (wrong-key guessing) so key-id enumeration is expensive.

Limit	Default	Notes
Per key	600 requests/min, burst 100	configurable per key by an operator, never by the org
Per org	3 000 requests/min	across all keys
Failed key auth per IP	20/min	then 429
Webhook endpoint management	30/min per org	cheap abuse control

Mechanics: Redis-backed (as the architecture specifies); on Redis failure the limiter degrades to a per-process in-memory bucket at a fraction of the limit and logs once per minute, so Redis being down neither disables limiting (fail open) nor takes the API down (fail closed on a cache). Responses carry RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset (the IETF draft names; D-level detail) and 429 carries Retry-After and code: rate_limited. The limiter runs after Authenticate (it needs the credential) and the failure limiter runs before any database work. Limits apply to session callers on public routes too, keyed on the session id, so the contract’s behaviour is the same for both caller types. (U: no existing limiter was found; if one exists on another branch, reuse it.)

7. Webhooks

7.1 Source of events: a transactional outbox over published domain events

Principle: webhooks only ever announce a catalogued event, and a catalogued event is a published domain event with a versioned payload. Not “every transition”, not “every row change”.

Capture. eventbus today dispatches after commit. Add a second, in-transaction hook: db.InTx calls a registered TxRecorder(ctx, tx, kind, payload) for each buffered event before commit, so the outbox rows commit or roll back with the business change. The interface lives in kernel (eventbus), is implemented by platform/webhooks, and records only kinds present in the catalogue. Existing in-process subscribers are untouched. Alternative considered and rejected: subscribing a normal handler. It runs post-commit, so a crash drops the event, which is exactly the failure an integrator cannot detect.

Catalogue. A Go registry in platform/webhooks (event type, payload type, the in-process event kind it maps from) and, in the contract, a top-level x-webhooks: extension mapping each event type to a schema in components.schemas (the document is OpenAPI 3.0.3, which has no webhooks keyword; 3.1 has one, but upgrading the whole document is out of scope). (U: that oapi-codegen, as configured in oapi-codegen.yaml, generates the schemas referenced only from an extension; if it prunes them, they are referenced from a dummy x- path or the codegen is configured to include them. A spike in task A1 settles it.) Payload types are the generated types, so Go payload and published schema cannot differ.

Semantic tags, not state names. Workflow-driven events (an expense claim being approved) fire on transition into a state carrying the relevant semantic tag, never on a state name, for the same reason accounting/ does: a customer may rename and rearrange states. (U: whether workflow.transitioned carries the destination state’s tags; if not, the publisher resolves them inside the transaction, in the workflow package.)

Candidate v1 catalogue (each needs its publisher to exist; today four kinds do):

Event type	Source	Payload (thin, section 7.2)
`sales_document.issued`	`receivables.document_issued` (exists)	id, kind, number, currency, total (string), revision
`sales_document.voided`	receivables (U: publisher not found)	same
`expense_claim.approved` / `.paid`	workflow tag transition	id, number, currency, total (string), revision
`payment_batch.paid`	payables batches (U)	id, number, currency, total, revision
`einvoice.validated` / `einvoice.rejected`	e-Invoice worker (U)	id, source document id, status code, revision
`bank_statement.imported`	banking (U)	id, account id, line count

Not in v1: anything from hr, payroll, member or role data (section 10). hr currently publishes no in-process event at all (U: a search for leave_rerouted found no match in internal or docs; the leave routing is a table, leave_routing_move), so there is nothing to catalogue there yet and that is the right default.

7.2 Thin events by default

An event is an envelope plus identifiers, not the record:

{ "id": "evt_...", "type": "sales_document.issued", "api_version": "1",
  "occurred_at": "...", "org_id": "...",
  "data": { "resource": {"kind": "...", "id": "..."}, "revision": 7, ...few scalar fields } }

The receiver fetches the full record through the public API, where the key’s scopes decide what it may see. This keeps authorization at read time (a webhook cannot leak what the subscribing key could not read), keeps personal data out of the push channel (section 10), and makes payload schemas small enough to keep stable. Payload schemas must set additionalProperties: false; a drift test forbids property names on a denylist (name, email, phone, address, nric, ic_number, passport, bank_account, account_number, salary, wage, dob, …) in any x-webhooks payload schema, and demonstrates the check is live by also running it against a schema known to contain such properties (presence).

org_id is present because a receiver with several organizations needs it; it is not personal data. Money is a string.

7.3 Subscriptions per organization

webhook_endpoint(org_id, id, url, description, event_types text[], status, created_by, created_at, disabled_at, disabled_reason, consecutive_failures); status is the string vocabulary active | paused | disabled. Managed by administrators only (integrations.webhook.manage; not api-grantable). Event types are validated against the catalogue at write time. An endpoint belongs to exactly one org (org_id plus RLS); a (org_id, url) uniqueness stops accidental duplicates. A per-org cap (default 20 endpoints) bounds fan-out.

7.4 Signing

Adopt the Standard Webhooks shape rather than invent one, so receivers have libraries (U: the header names and signing string are recalled from that specification, not re-read today; check against the current text before the contract is written):

Headers: webhook-id (the event id, stable across retries and replays), webhook-timestamp (Unix seconds, freshly set on every attempt), webhook-signature (v1,<base64 HMAC-SHA256>; several space-separated values during a rotation overlap).
Signed content: {webhook-id}.{webhook-timestamp}.{raw body}.
Documented receiver rules, repeated in the published document: verify with a constant-time compare; reject a timestamp more than 5 minutes from receiver time (the replay window); dedupe on webhook-id; respond 2xx quickly and process asynchronously.
Secret: whsec_ plus 32 random bytes. HMAC needs the plaintext, so it cannot be hashed at rest. It is sealed with the existing payout.Keyring mechanism (AES-256-GCM, key id stored beside the ciphertext, previous key honoured for rotation); a new WEBHOOK_SECRET_KEY and WEBHOOK_SECRET_KEY_PREVIOUS in .env.example (the env-var drift test requires it), custody in the runbook alongside PAYOUT_DETAILS_KEY. Shown once at creation and at rotation; no read endpoint returns it. Moving the keyring out of payout into a shared platform package is a surgical prerequisite refactor, called out in the task plan, not smuggled in.
Rotation with overlap. Rotating adds a new secret version and keeps the previous one signing in parallel for the overlap (default 24 h, maximum 7 days); both signatures appear in webhook-signature. A job removes the expired version. Rotation is audited and may be forced earlier by an administrator (“revoke old now”) for a suspected leak.
Body is JSON, UTF-8, at most 64 KiB; the contract promises thin payloads precisely so the limit is never approached.

7.5 Delivery

Tables: webhook_event (the outbox: org_id, id, type, aggregate_kind, aggregate_id, payload, occurred_at) and webhook_delivery (one row per event and endpoint: status = pending | succeeded | failed | dead, attempts, next_attempt_at, last_attempt_at, last_status_code, last_error_class). Neither stores a response body (7.7).

Flow:

In the producing transaction: the TxRecorder inserts the webhook_event, inserts a webhook_delivery row for each active endpoint of that org subscribed to the type, and InsertTxs a River job per delivery (QueueWebhook, MaxAttempts: 1, the repository’s convention for table-driven retry). Atomic with the business change: rolled back means no event, no delivery, no job.
The attempt job opens a tenant transaction, reads the delivery, the endpoint and the sealed secret, closes the transaction, performs the HTTP call with no transaction and no row lock held (the repository’s lock-order rule: never hold a lock across a network call), then opens a second tenant transaction to record the outcome and, on failure, inserts the next attempt’s job with ScheduledAt.
A reconciler (periodic job, ops.org_ids() shape) re-enqueues pending rows whose next_attempt_at passed with no live job (a lost job, a crash between commit and enqueue is impossible by 1, but a River rescue discard is not), and sets the gauges in section 11. Belt and braces: table state is the truth, jobs are the trigger.
A dedicated queue QueueWebhook with its own concurrency, so a slow receiver cannot starve the e-Invoice or operator queues; a per-endpoint concurrency cap (default 2) stops one slow endpoint occupying every worker. (U: how River would express a per-endpoint cap; likely a FOR UPDATE SKIP LOCKED claim on an in_flight counter, decided in task W2.)

Success = any 2xx within 10 seconds. Redirects are not followed (a 3xx is a failed attempt). 410 Gone pauses the endpoint at once; 429 and 503 honour Retry-After, capped at 1 hour. Anything else retries.

Backoff: attempts at +10 s, +1 min, +5 min, +30 min, +2 h, +6 h, +12 h, +24 h, each with +/-20 % jitter (about three days in total), then the delivery becomes dead. (Owner may tune; D7.)

Dead-letter. A dead delivery stays visible, with its event, until retention. Manual replay (integrations.delivery.replay, admin only, audited) sets a dead or failed delivery back to pending with attempts reset, same webhook-id, fresh timestamp and signature, so a receiver that already processed the event still dedupes. Replay is available per delivery and as “replay all dead for this endpoint since T”, capped at 1 000 per action. A replay is only possible while the event row exists, so the replay horizon equals event retention (7.8).

Endpoint health. consecutive_failures counts dead deliveries with no success between; at 50, or seven days since the last success with at least one failure, the endpoint becomes disabled, an audit row and a notification go to the org’s integration administrators, and new events stop fanning out to it (events missed while disabled are not queued; re-enabling is manual and offers a replay of events since the disable time from the outbox, within retention).

7.6 Ordering

None guaranteed (stated plainly in the published document), because per-aggregate ordering needs a serialised lane per aggregate, which turns one stuck delivery into a blocked stream, and the thin-event design makes ordering unnecessary: the receiver fetches current state. To let receivers detect staleness, each event carries occurred_at and, where the resource has one, its revision (the same revision If-Match already uses). Rule for receivers, documented: ignore an event whose revision is older than what you hold. Per-aggregate ordering is an explicit “Not in scope” item (D8).

7.7 SSRF protection

The endpoint URL is attacker-influenced input that makes our servers open connections. The policy applies twice: when the endpoint is saved (to give the administrator an immediate error) and, authoritatively, at every delivery attempt, because DNS answers change.

https only, port 443 only, host must be a DNS name (IP literals refused, including decimal, octal and hex forms and bracketed IPv6), no userinfo, length-capped. A development-only escape hatch (WEBHOOK_ALLOW_INSECURE_DEV, honoured only when the environment is development, in .env.example) lets local tests deliver to an httptest server; the test suite asserts it is inert in any other environment.
Resolve-then-connect. Resolve the name once, validate every returned address, then dial the validated IP directly with ServerName set to the original host for TLS. Enforce again in the dialer’s Control hook against the socket’s real remote address, so a resolver that answers differently the second time (DNS rebinding) or a connection that lands somewhere else is refused at the last possible moment.
Blocked ranges (after unwrapping IPv4-mapped and, for NAT64 and 6to4, the embedded IPv4): 0.0.0.0/8, 10/8, 100.64/10, 127/8, 169.254/16 (cloud metadata), 172.16/12, 192.0.0.0/24, 192.0.2/24, 192.168/16, 198.18/15, 198.51.100/24, 203.0.113/24, 224/4, 240/4, 255.255.255.255, ::, ::1, fc00::/7, fe80::/10, ff00::/8, 64:ff9b::/96, 2002::/16. Plus a configurable WEBHOOK_BLOCKED_CIDRS for our own public addresses: the VPS’s public IP is a perfectly ordinary routable address that fronts Grafana, Postgres and other services on the same host, so the generic private-range rule does not protect them (U: the current exposure of those ports is a known deferred issue per the project notes; the allowlist is defence in depth either way).
No redirects, no proxy (Transport.Proxy = nil; an HTTP_PROXY in the environment must not change behaviour), TLS verification always on (minimum 1.2), no HTTP/2 connection reuse across endpoints belonging to different orgs.
The response body is read to a small cap and discarded, never stored or logged, so an internal service reached by a bypass cannot be exfiltrated through the delivery log. Only the status code and a coarse error class (timeout, tls, dns, blocked_address, refused, non_2xx) are recorded.
Network-level defence in depth: the worker container’s egress policy should deny the same ranges. That is a deployment change and is listed in the task plan, not assumed.

7.8 Retention of the delivery log

Operational data, not a statutory record; it joins the deletable list in retention_test.go. Recommended: webhook_delivery rows 30 days when succeeded, 90 days when failed or dead; webhook_event rows 30 days (that is the replay horizon); api_idempotency_key 24 hours. Pruned by the maintenance queue. Configuration changes (keys, endpoints, rotations, replays) live in audit_log and follow the audit retention, not these. (D5.)

8. Tenant isolation, in full

Every new table has org_id, ENABLE and FORCE ROW LEVEL SECURITY, policy org_isolation, a composite FK to organization_member (org_id, id) where it names a member (the existing tenantfk_test style). Tables: api_key, webhook_endpoint, webhook_endpoint_secret, webhook_event, webhook_delivery, api_idempotency_key.
Every public request: Authenticate (key -> org) -> Tenant() -> db.InTx. The one place that runs without a tenant is the definer lookup in 3.5, which returns a single row selected by the key id and exposes no table.
The outbox recorder runs inside the producing request’s tenant transaction; fan-out reads webhook_endpoint under that same org setting, so it cannot see another org’s endpoints.
Delivery workers receive (org_id, delivery_id) in the job args. That is a system-issued value, not a request header, and it is applied with tenancy.WithOrg before InTx, the existing worker pattern. A delivery whose id belongs to a different org than the args names is invisible under RLS and the job fails closed.
A secret leaked into a log, audit row, metric label, delivery row or error string is a defect: tested (section 12). Metric labels never include org id (cardinality and leakage).

9. Admin surface

New web area “Integrations” for administrators: API keys (create with show-once dialog, rotate, revoke, last used, expiry), webhook endpoints (create with show-once secret, pause, rotate, test ping to a verified-safe endpoint, delivery log with filters, replay), the org switch (3.8). Every label is a translation row; new permission keys carry labels (the existing permissionlabels_test). Default grant: Owner only for all four integrations.* keys (migration for existing orgs, registry for new ones), the same shape as payroll.run.approve.

10. Malaysian and PDPA angle

Stated as design intent, not legal advice; the section 129 reading below is unverified and needs counsel (U).

We are, for tenant data, a processor; the organization is the data user. A webhook or API read is the organization directing its own data to a destination it chose. We must still not make that easy to do carelessly.
Minimisation is the main control. Thin events (7.2), a catalogue that excludes hr, payroll and member data entirely in v1, api-grantable permissions that exclude personal-data scopes (3.6), and a contract test that bans personal-data property names from payloads. The push channel carries identifiers and amounts; anything personal requires an authenticated read the key’s scopes control.
Cross-border transfer. The PDPA 2010 restricts transfer of personal data outside Malaysia (section 129), and the 2024 amendments revised the basis for it (U: the exact current test and in-force dates were recalled, not re-read today). We cannot know where a customer’s receiver runs. Therefore: v1 payloads contain no personal data by design, so the question mostly does not arise for webhooks; the read API can return personal data to a receiver abroad, which is the customer’s decision under their own key and scopes. When an administrator creates a key or endpoint, the screen records an acknowledgement (audited, with the wording version) that the destination is theirs or their processor’s, and states that data sent there leaves our Malaysian hosting. Whether that wording is sufficient is a question for counsel (D10).
Records stay in Malaysia (architecture 11A.1, ITA s.82(8)). The outbox and delivery log live in the same Malaysian database; retention is short (7.8) so we do not build a second long-lived copy of statutory records; no response bodies are stored.
Breach readiness. A leaked key is a potential personal-data incident for the customer. last_used_at/last_used_ip, per-key audit and immediate revocation are the evidence and the remedy; the runbook gains a “leaked API key or webhook secret” procedure.
Retention outranks erasure (architecture 11A, existing invariant): the webhook outbox is not a statutory record, so it is not exempt from a tenant’s erasure request, but with thin payloads there is nothing personal in it to erase. A drift test asserts the catalogue contains no event whose payload is annotated personal.
last_used_ip is itself personal data of whoever runs the integration; it is retained only as the last value (not a history) and is cleared when the key is deleted.

11. Monitoring

Prometheus, following erp_hr_leavings_failing: set by the reconciler each run, scraped, with a Grafana alert rule recorded in docs/DEPLOY_NOTES.md (the hr rule is documented but, per the roadmap, not yet created on the VPS; this item does not repeat that gap: the rule is part of the deliverable’s checklist).

Metric	Type	Alert
`erp_webhook_deliveries_dead`	gauge (dead and un-replayed, all orgs)	rising for 1 h, or > 0 for 24 h
`erp_webhook_oldest_pending_seconds`	gauge	> 900 for 10 min (the queue is stuck, or a worker is down)
`erp_webhook_endpoints_disabled`	gauge	any increase (also notified to the org)
`erp_webhook_attempts_total{outcome}`	counter (`success`, `timeout`, `tls`, `dns`, `blocked_address`, `non_2xx`)	`blocked_address` > 0 is a security signal: someone registered a hostile URL
`erp_webhook_attempt_seconds`	histogram	informational
`erp_api_requests_total{status_class}` and `erp_api_rate_limited_total`	counters	429 ratio
`erp_api_key_auth_failures_total`	counter	spike = guessing

No org id, endpoint id or key id as a label. Per-org detail lives in the delivery log and the structured access log (which already carries org_id; add credential_id).

12. Drift tests

Each is written presence-first (AGENTS.md: establish the presence before asserting the absence) and watched failing once: the mechanism is broken, the message read, the mechanism restored. The “watch it fail” step is a named check in the task plan, not a hope.

#	Test	Presence established first	Then asserts	Break it by
T1	Public surface is closed	`> 0` operations marked `x-public-api`, and a session token reaches a non-public route (e.g. members) with 200	the same route with a key answers 403 `not_in_public_api`; table-driven over every non-public route in the assembled router	making `PublicSurface()` a no-op
T2	Public operations are grantable and vice versa	registry has `> 0` grantable permissions and `> 0` public operations	the two sets agree both ways	marking an `identity.*` permission grantable
T3	Key cannot exceed owner	owner holds P; a key scoped P succeeds	after the owner loses P (and `perm_version` bumps) the same key gets 403	computing effective permission from scopes alone
T4	Key lifecycle	a fresh key authenticates	expired, revoked, owner-deactivated, org-suspended, `public_api_enabled=false` each get the generic 401, indistinguishable from a wrong key	skipping the `revoked_at` predicate
T5	Secrets never read back	create returns the plaintext once; the plaintext is a known string	no list/get/audit/log/metric/problem body contains the plaintext or `secret_hash`; `secret_hash` is not equal to the plaintext	adding the field to a read DTO
T6	Public document equals the source filter	filter output has `> 0` operations	committed `api/public-openapi.yaml` is byte-equal	editing the committed file
T7	Contract compatibility	baseline loads with `> 0` operations, codes, fields	removing a field, operation, enum value or code, or making a field required, fails; additions pass	deleting a response field from a public schema
T8	Public operations are well-formed	`> 0` public operations	every non-2xx declares `x-codes`; every array GET declares `limit/cursor/next_cursor`; no `number` on money-named properties; no `integer` enums; every non-GET requires `Idempotency-Key`	removing `x-codes` from one response
T9	Every route stays behind Authorize	existing `TestEveryRouteIsAuthorized`	unchanged: new routes register via `httpx.Protected`; no new line in `public_routes.txt` in v1 unless the spec is served (D9)	registering a new route with `g.GET`
T10	Idempotency	first POST creates; replay returns the same body with `Idempotent-Replayed`	same key with a different body is 409; two concurrent identical requests create one record; a failed request returns its key; credential A’s key does not collide with credential B’s	scoping the key to the member instead of the credential
T11	Rate limit	requests under the limit succeed	the N+1th gets 429 with `Retry-After`; with Redis down the in-memory fallback still limits; limit is per credential, not shared	removing the limiter middleware
T12	Outbox atomicity	a committed change yields one event, one delivery per matching endpoint, one job	a rolled-back change yields none of the three; an event whose kind is not catalogued yields none	recording after commit instead of before
T13	Tenant isolation of webhooks	org A and org B each have an active endpoint subscribed to the type, each with a visible delivery row under its own org	an A event creates deliveries only for A’s endpoint; B’s key and B’s admin cannot list or replay A’s deliveries or endpoints; `app.current_org` unset returns zero rows; cross-org job args fail closed	disabling RLS on `webhook_delivery`
T14	Catalogue = contract	registry has `> 0` events and `x-webhooks` has `> 0` entries	identical key sets; each payload type is the generated type of its schema	adding a registry entry without a schema
T15	No personal data in payloads	a schema containing `email` / `name` exists in components and the checker flags it	no `x-webhooks` schema has a denylisted property or lacks `additionalProperties: false`	adding `employee_name` to a payload
T16	Signature and replay window	an independent reference verifier (written in the test from the published rules, not from our signer) accepts a valid delivery	rejects a tampered body, a wrong secret, and a timestamp older than 5 min; during rotation overlap both signatures verify; after overlap the old one is gone; every attempt has a fresh timestamp and the same `webhook-id`	signing the timestamp out of the string
T17	SSRF policy	a table of 40+ addresses (every range in 7.7, encoded as decimal/octal/hex/mapped/6to4/NAT64) and a public control address that is allowed	every blocked form refused; a fake resolver answering public then private (rebinding) dials the first validated IP and refuses the second; a 302 to a private address is refused; port 8443, `http`, userinfo and IP literals refused at save time	removing the `Control` hook
T18	Dev escape hatch is inert elsewhere	with `ENV=development` a local `httptest` endpoint is deliverable	with any other value it is refused	ignoring the environment check
T19	Backoff and dead-letter	a failing endpoint is retried on the schedule using the injected clock	after the last attempt the row is `dead`, visible, and replay returns it to `pending` with the same `webhook-id` and `attempts = 0`; replay is audited	replaying with a new event id
T20	Auto-disable	an endpoint with failures below the threshold stays `active`	at the threshold it is `disabled`, fan-out stops, a notification and audit row exist	removing the counter reset on success (flapping endpoint)
T21	Retention	rows younger than the window survive a prune	older rows are removed; the new tables are in the deletable list of `retention_test.go` (that test already fails for an unlisted table)	dropping a table from the list
T22	Migration hygiene	existing tests	`{960, 979, ...}` added to `migrationRanges` in the same change as the first migration; existing RLS, tenant-FK, constraint-name, SECURITY DEFINER allowlist tests cover the new tables and `identity.api_key_lookup` without edits beyond the allowlist entry	omitting the allowlist entry (the existing test must fail)
T23	Env and lock hygiene	existing	`WEBHOOK_SECRET_KEY`, `WEBHOOK_SECRET_KEY_PREVIOUS`, `WEBHOOK_BLOCKED_CIDRS`, `WEBHOOK_ALLOW_INSECURE_DEV`, rate-limit variables are in `.env.example`; any advisory lock uses a registered class	adding an unlisted `os.Getenv`

Guards that walk the repository (T1’s route walk, T6, T8, T14, T15) also assert they read a non-zero number of routes, operations and schemas before asserting none violate, and must not report violations that are not in the tree (the false-positive direction named in AGENTS.md).

13. Owner decisions

Recommended option first in each row.

#	Decision	Recommended	Alternatives
D1	Machine credential type	API keys now (bearer `mck_...`), client-credentials later on the same principal	OAuth client-credentials first; both at once
D2	Whose authority does a key carry?	The creating member’s, intersected with chosen scopes; key dies when the member is deactivated	A dedicated “service member” that outlives staff (more continuity, more audit and approval-rule design: it is a new kind of member)
D3	Org-level enablement	Off by default; admin enables; operator can suspend public API per org	On by default for all; operator switch only
D4	Key expiry	Mandatory, default 90 days, max 365, 24 h default rotation overlap (max 7 d)	No expiry; shorter maximum
D5	Log retention	Deliveries 30 d (success) / 90 d (failed, dead); events 30 d; idempotency 24 h	Longer for compliance comfort; shorter for minimisation
D6	Deprecation notice	12 months with `Deprecation`/`Sunset` headers	6 months
D7	Retry schedule and auto-disable	8 attempts over about 3 days; disable at 50 consecutive dead or 7 days without success	Fewer attempts; never auto-disable
D8	Ordering	None guaranteed; `occurred_at` and `revision` on every event	Per-aggregate ordering (blocked-stream risk, more state)
D9	Serve the public OpenAPI file from the API?	No in v1: commit it, publish with the docs (keeps `public_routes.txt` unchanged)	Serve `GET /v1/openapi.json` unauthenticated (one reviewed `[no-permission]` line)
D10	PDPA / cross-border	Thin events, no HR or payroll data in v1, audited acknowledgement on key/endpoint creation; counsel reviews the wording before launch	Allow personal fields per endpoint with explicit opt-in (needs a transfer basis per section 129)
D11	Public write surface in v1	Read-heavy: read endpoints plus a handful of create endpoints for sales documents and expense claims; no approvals, no posting, no payroll	Read-only v1; broader writes
D12	Which events	The seven in the 7.1 table that have publishers, adding the missing publishers in the same task as each event	Only the four existing kinds
D13	Secret sealing keyring	Reuse `payout.Keyring`, moved to a shared package, new `WEBHOOK_SECRET_KEY`	Reuse `PAYOUT_DETAILS_KEY` (couples two rotation schedules; not recommended)

14. Task plan outline

Proposed migration range 00960-00979, to be added to migrationRanges in internal/kernel/db/migrationranges_test.go in the same change as the first migration (another spec uses 940+; this spec stays at 960-979 and avoids 00940-00959). Names are indicative.

Task	Content	Migrations	Verifies
A0	Spike: re-check for an existing rate limiter and Redis use; oapi-codegen behaviour with `x-webhooks`; `oasdiff` vs hand-rolled compatibility check; audit chain input. Resolve the (U) items and amend this spec	none	the spec’s unknowns are answered
A1	Contract tooling: `x-public-api` marker, filter tool, committed `api/public-openapi.yaml`, baseline, T6, T7, T8	none	T6-T8
A2	Keys: `api_key` table, `identity.api_key_lookup` definer function, resolver in `platform/identity/apikey`, `Authenticate` branch, `Credential`, `Authorize` scope step, `PublicSurface()`, `api_grantable` registry flag, T1-T5	00960-00963 (table, lookup function, grants, org switch setting)	T1-T5, T22
A3	Admin endpoints and web screens for keys; permission keys and labels; Owner-only default grant; audit; expiry-warning job	00964-00965	T9
A4	Idempotency (`api_idempotency_key`, middleware, prune) and rate limiter (Redis plus in-memory fallback)	00966-00967	T10, T11, T21
A5	First public operations (D11), each with `x-codes`, keyset pagination, reviewed payloads	none	T8, T9
W1	Outbox: `TxRecorder` hook in `eventbus`/`db.InTx`, `webhook_event`, `webhook_endpoint(_secret)`, catalogue registry, `x-webhooks`, first publishers (D12), T12, T13, T14, T15	00968-00972	T12-T15
W2	Delivery worker: `QueueWebhook`, attempt job, SSRF-safe client, signing, keyring move and `WEBHOOK_SECRET_KEY`, backoff, dead-letter, replay, auto-disable, reconciler, T16-T20	00973-00975	T16-T20
W3	Admin endpoints and screens for endpoints, delivery log, replay, rotation; test ping; acknowledgement text (after counsel, D10)	00976	T13
W4	Metrics, Grafana rules recorded in `DEPLOY_NOTES.md` and created, runbook entries (leaked key, hostile webhook URL, replaying a dead letter), worker egress policy, retention prune, T21, T23	00977 (prune support if needed)	T21, T23
X	Integration: end-to-end with a local receiver; review pass against section 12 with every “break it” step actually performed and recorded	00978-00979 reserved	all

Order: A0 -> A1 -> A2 -> (A3 | A4) -> A5; W1 can start after A0 and in parallel with A2-A4; W2 needs W1 and the keyring move; W3 needs W2; W4 last. A5 and W-series publish nothing externally until X.

15. Not in scope

OAuth client-credentials, authorization-code OAuth for third-party apps and an app marketplace.
Per-aggregate ordered delivery; exactly-once delivery (the guarantee is at-least-once with dedupe on webhook-id).
Fat payloads, per-endpoint personal-data opt-in, field-level payload filtering.
HR, payroll, member, role and salary data over the public API or webhooks.
Approvals, posting and anything that touches the ledger through a key. The ledger stays append-only and modules still never write journal lines; a public write goes through the same module functions the web uses and nothing else.
Inbound webhooks (third parties calling us), polling “changes since” feeds, GraphQL, bulk/batch endpoints, SDK generation beyond what the existing generator already produces.
Mutual TLS or IP allowlisting for keys; customer-managed signing keys.
A developer portal, sandbox organizations and billing or metering per call.
Serving the OpenAPI document (D9, unless decided otherwise).

16. Unverified (collected)

No rate limiter exists; Redis is used only by liveness (search only).
Whether audit_log’s hash-chain input can take a credential id (column vs field in after).
Whether workflow.transitioned exposes destination-state semantic tags.
Whether oapi-codegen as configured generates schemas referenced only from x-webhooks.
Whether oasdiff is acceptable as a dependency; fallback described.
Interplay of the generic idempotency layer with per-module key tables (reasoned, untested).
Which candidate catalogue publishers exist beyond the four found (sales_document.voided, payment batch, e-Invoice status, bank statement import were not found as published events).
Standard Webhooks header names and signing string, and the current status of the Deprecation header specification, were recalled, not re-read.
The PDPA section 129 reading and the in-force dates of the 2024 amendments; counsel to confirm.
How River should express a per-endpoint concurrency cap.
Exposure of the VPS’s other services (known deferred issue) and the worker’s egress policy.
Grafana alert rules: the erp_hr_leavings_failing rule is documented as not yet created on the VPS, so the mechanism for creating alerts there was not exercised.

17. Corrections found while planning (2026-10-04)

See “Spike results” in docs/superpowers/plans/2026-10-03-public-api-webhooks.md. In short: no limiter exists (U1 confirmed); the audit credential goes in a side table, the hash chain is untouched (U2); workflow.transitioned carries semantic tags (U3); oapi-codegen needs skip-prune: true for x-webhooks schemas (U4); oasdiff is not available, a hand-written comparison is used (U5); the Standard Webhooks and Deprecation (RFC 9745) details are confirmed (U8). Errors in this document: x-codes is not an existing extension (it is introduced here); the catalogue lists eight event types, not “seven”; job code lives in internal/kernel/jobs.

14. Owner decisions recorded (2026-10-03)

D1-D13: the recommended option in each row of Section 13 (D13 per the answer given in chat).