Prometheus monitoring: research

As of: 2026-09-28. Every claim is sourced. “Not verified” items are listed at the end. For: operator console Stage 1 and items P1, I9–I12 in 2026-09-25-operator-console-progress.md, and the console views V2 (service health) and V3 (background jobs). Follows: 2026-09-28-malaysia-hosting-research.md (one VPS in AWS ap-southeast-5). Status: research and a proposed setup. The code part (§2) is built on ops-groundwork; the infrastructure is not. Sequencing is unchanged: this comes after the ERP’s own infrastructure and the console skeleton.

---

1. Proposed setup

Everything runs on the one Malaysian VPS and binds to 127.0.0.1.

Component	Version (2026-09)	Bind	Notes
api /metrics	client_golang v1.24.1	127.0.0.1:9464	A second http.Server, never on the public gin router
worker /metrics	client_golang v1.24.1	127.0.0.1:9465	Pool gauges and the River job-count collector
Prometheus	3.15.0 (LTS 3.13)	127.0.0.1:9090	15 s scrape, --storage.tsdb.retention.time=90d --storage.tsdb.retention.size=5GB
Alertmanager	0.34.1	127.0.0.1:9093	Outbound only
node_exporter	1.12.1	127.0.0.1:9100	Textfile collector for the backup “last success” metric (I14)
postgres_exporter	0.20.1	127.0.0.1:9187	Own role with pg_monitor only
blackbox_exporter	0.28.0	127.0.0.1:9115	Public HTTPS up/down and certificate expiry
Grafana	13.2.2	127.0.0.1:3000	Reached over the private network only (design §3.4, I2), never through the public proxy

Sources: client_golang, Prometheus, node_exporter, postgres_exporter, blackbox_exporter.

Install as release binaries under systemd, not Docker.

• Docker publishes ports on 0.0.0.0 unless told otherwise, and those ports bypass ufw (port publishing, firewalls).
• Containers also can’t reach the host’s 127.0.0.1 listeners without host networking.
• This affects progress doc §5 item 4, which proposed compose for Prometheus and Grafana. Decide there.

Disk: about 2,000 series × one sample every 15 s × about 1.5 bytes per sample × 90 days comes to about 1.6 GB (formula from storage). The 5 GB cap is a safety net.

2. Code (P1)

Use client_golang directly, about 20 lines of our own gin middleware, no new middleware library.

• Existing gin libraries label by raw path by default (for example go-gin-prometheus). That puts IDs into labels, which is exactly what the drift test forbids.
• Our own middleware uses c.FullPath(), which httpx already logs. It returns "" for unmatched routes (gin), so those requests go into one route="unmatched" bucket. The method label must be fixed too: Go accepts any method token, so a client could otherwise invent a new series with every request.
• The OpenTelemetry Prometheus exporter is still pre-1.0 (v0.68.0), while client_golang is v1.
• Histograms: classic DefBuckets. Native histograms are stable since Prometheus 3.8, but they need scrape_native_histograms turned on, and nothing at this size needs them (spec).

Database pool: pgx has no official collector. Expose pool.Stat() through GaugeFunc/CounterFunc: acquired, idle, max, EmptyAcquireCount and EmptyAcquireWaitTime. The last two show when requests are waiting for a connection (Stat).

Jobs (V3): River has no Prometheus metrics of its own.

• rivercontrib/otelriver v0.12.0 gives counts and durations labelled by kind, queue, status, attempt, priority, but no queue depth by state (docs, source).
• So V3 needs a small collector that runs SELECT kind, state, count(*) FROM river_job … GROUP BY 1,2 at scrape time, with a short timeout. Job kinds are fixed in code, not tenant data, so kind is a safe label.
• Add otelriver only when per-kind duration or error rate is wanted.

Sketch (untested):

var reqDur = prometheus.NewHistogramVec(prometheus.HistogramOpts{
  Name: "http_request_duration_seconds", Help: "HTTP latency by route template.",
  Buckets: prometheus.DefBuckets,
}, []string{"method", "route", "status"})

func Gin() gin.HandlerFunc {
  return func(c *gin.Context) {
    start := time.Now()
    c.Next()
    route, method := c.FullPath(), c.Request.Method
    if route == "" { // 404/405: never the raw path; the method is client-chosen
      route, method = "unmatched", "other"
    }
    reqDur.WithLabelValues(method, route, strconv.Itoa(c.Writer.Status())).
      Observe(time.Since(start).Seconds())
  }
}

// main.go: private listener. Built as API_METRICS_ADDR / WORKER_METRICS_ADDR (two, since api
// and worker share an env file), refusing ":port" and 0.0.0.0
srv := &http.Server{Addr: cfg.MetricsAddr, ReadHeaderTimeout: 5 * time.Second,
  Handler: promhttp.HandlerFor(reg, promhttp.HandlerOpts{})}

Drift test (design §6): it first shows that it found the route label (presence), then asserts that no label is named org, org_id, user, member or email. Watch it fail once. A second test checks that an unknown path produces route="unmatched".

3. Alerts (I12)

Alert	Rule
Disk nearly full	`node_filesystem_avail_bytes{fstype!~“tmpfs
Scrape target down	up == 0 for 2m
App down (public)	probe_success{job="blackbox_https"} == 0 for 2m
Certificate expiring	probe_ssl_earliest_cert_expiry - time() < 14*86400
Backup failed	time() - erp_backup_last_success_timestamp_seconds > 26*3600, or the metric is absent

• For the backup metric, the backup script writes a .prom file to a temporary path, then mvs it into the textfile directory. That is the atomic write the docs recommend (textfile collector).
• Gap: the server can’t report its own death. Monitoring on the same VPS goes silent when the VPS does. It needs an outside signal: an always-firing “watchdog” alert sent to an external heartbeat service, which alerts when the heartbeat stops (it carries no record data), or an AWS EC2 status alarm.

Where alerts go:

• Alertmanager supports email, Telegram, Slack, SNS, webhooks and more (config).
• Telegram and Slack deliver outside Malaysia. SNS has an ap-southeast-5 endpoint (SNS), but the final email or SMS still leaves.
• An alert only carries its labels and our summary text. If labels are clean and templates never include query results, no record data leaves. Keep alert templates under the same label rule.

4. Managed alternatives: not in Malaysia as a pair

• Amazon Managed Service for Prometheus is in ap-southeast-5 (endpoints).
• Amazon Managed Grafana is not. The nearest region is Singapore (regions). Dashboards would leave Malaysia, so self-hosting stays the plan.
• VictoriaMetrics (Apache-2.0) and Grafana Alloy are good tools, but they don’t help at this size.
• Licences: Prometheus Apache-2.0; Grafana AGPL-3.0-only (LICENSING.md). Running Grafana unmodified, internally, is the ordinary case. This is not legal advice.

5. Security

• Prometheus and the exporters have no authentication by default. --web.config.file adds bcrypt basic auth and TLS (https).
• Loopback binds plus the security group keep them off the internet. Never open 9090, 9093, 9100, 9115, 9187, 9464, 9465 or 3000.
• A publicly reachable blackbox_exporter would probe any URL it is given, making it an open prober. This is an inference; its docs don’t warn about it.
• pg_monitor can see query text in pg_stat_activity. Leave postgres_exporter’s query-text options off, so literal values (possible record data) never become labels.

6. Effort (rough)

• Code, about 1–1.5 days: middleware and registry, private listeners in api and worker, pool metrics, River count collector, METRICS_ADDR in .env.example, the label drift test, the unmatched-route test.
• Infrastructure, about 2–3 days: five systemd units and web configs, scrape config and alert rules, Alertmanager with one receiver, backup textfile hook, Grafana dashboards for V2 and V3, private access to Grafana, the external heartbeat, and a test-fire of each alert.

7. Not verified

• Idle CPU and RAM of Prometheus plus the exporters at this size
• postgres_exporter’s default bind address and exact query-text flags
• River’s full list of job states and its cleaner’s retention, which decide what a “failed” count means
• Whether gin-metrics labels by template or raw path
• Whether CloudWatch EC2 status alarms and SES are available in ap-southeast-5