fix(protocol): restrict metrics endpoints to admin keys and add operational Prometheus metrics by jsam · Pull Request #365 · inputlayer/inputlayer

jsam · 2026-10-05T17:27:03Z

Intent

I want you to really hyper focus on making the symbolic engine, you know, crisp and really truly production ready. ... just ensure that both machines are propelry utilized now (Context: implement #300 exactly as written.)

Issue #300, "Observability gaps: metrics blind spots, a cross-tenant metrics leak, undocumented Prometheus endpoint", says: from the engine production-readiness audit, see the audit report section P1-4; fix the cross-tenant leak first. The substance of what it refers to:

Cross-tenant leak (fix first): /metrics and /metrics/prometheus authorize any valid API key. The auth middleware only checks that the key authenticates, with no role or scope check, and the handlers scan all knowledge graphs, so a single-KG scoped viewer key can read global KG/relation/tuple/memory/session counts for every tenant. Fix: require global admin role for the metrics handlers (or a separate metrics listener).

P1-4, observability is blind to what an operator needs most, and the Prometheus endpoint is undocumented. /metrics/prometheus exports uptime, queries_total, gauges for KG/relation/view/tuple/session counts and query-latency histograms. Missing, each operationally needed: HTTP request/error rate by status; active-connection and active-WebSocket gauges (the connection semaphores are never surfaced); rejection/throttle counters (503 "Too many connections", 429 rate-limit, 401 auth-fail and slow-consumer disconnects increment nothing); WAL size, flush lag and dirty-shard count; subscription metrics (they exist in the subscription module but prometheus_metrics never calls handler.subscription_metrics()); per-KG labels. memory_usage_bytes is a crude estimate total_tuples*64, not RSS, misleading for capacity planning. The ops guide (deployment.mdx) documents only /health, /live and /ready and never mentions /metrics/prometheus (that it exists, needs a Bearer key, or what to alert on). Fix: (a) add counters at the reject/error sites and export them; wire subscription_metrics() in; add WAL/flush gauges; label key gauges by KG; replace the fake memory gauge with RSS. (b) A "Monitoring" doc section: endpoint, auth, scrape config, metric list and recommended alert thresholds.

What Changed

Closes the cross-tenant leak: /metrics and /metrics/prometheus (and their /v1 versions) now sit behind a new require_admin route middleware. Only an unscoped API key belonging to an admin user can read them. Viewer keys, editor keys and KG-scoped keys get a 403. The auth middleware now attaches the authenticated Principal to each request so this check can be made. A non-admin .status now shows only how many knowledge graphs the caller can see, and leaves out server-wide totals.
New ServerMetrics (src/protocol/metrics.rs), exported on /metrics/prometheus:
- HTTP responses by status code, plus HTTP requests in flight and WebSocket connection gauges with their limits.
- Rejection counters by reason: connection limit (503), IP rate limit (429), 401, 403, login throttled or busy, WS pre-auth limit, WS connection limit, WS rate limit.
- Auth failures by method, WS send timeouts (slow consumers) and WS error notices.
- Subscription metrics, which are now wired in.
- WAL and persistence gauges: WAL size and limit, dirty shards, buffered updates, oldest unflushed age, flush and flush-failure counts, read-only flag.
- Query memory and compute-permit usage.
- Per-KG gauges labelled by kg: relations, tuples, views, loaded, sessions.
- inputlayer_memory_usage_bytes now reports process RSS instead of the old tuples*64 estimate.
Docs and tests: deployment.mdx has a new Monitoring section covering the endpoint, admin-key auth, scrape config, the metric list and suggested alerts. The REST API, authentication, persistence, commands and OpenAPI docs (and the bundled GUI docs) are updated to match. New tests: tests/metrics_tests.rs, plus router tests that check non-admin keys are refused and the new counters are exported, and a scoped-role test for .status.

Risk Assessment

✅ Low: The cross-tenant leak is closed at the route layer. require_admin uses Principal::role(), which already downgrades scoped admin keys to editor and re-reads the role on each request, so demoted admins lose access. The user-decided .status scoping goes through the single run_execute_program choke point that every caller uses. The new metrics are lock-free counters or read-only snapshots, and their tests exercise real routers and servers.

Testing

I drove the real server end to end over HTTP and WebSocket in disposable instances (no prior baseline commands had run). The first instance covered the access matrix for the four metrics routes, the per-caller .status output, and the contents of a scrape during live load. Two more runs triggered rate-limit (429) and connection-limit (503) rejections and a persist flush. Every observed value matched expectations: memory RSS equal to /proc VmRSS, per-graph tuple counts, counters going up on writes, subscriptions and failed auth, and 403 for every non-admin key while the same key still worked for normal queries. The new focused integration and unit tests also pass, but they are automated tests rather than live runs, so that scenario is recorded as untested under the live-validation contract. Docs are the only other part of the change; I checked that the Monitoring section's metric names match the live output, but did not render them, since there is no UI change to screenshot. All temporary servers, data and the build output were removed.

Live validation: ✅ go - 10 of 11 scenarios driven live against the product

Scenario	Result	Live	Evidence
Viewer, editor and tenant_a-scoped admin keys get 403 on /metrics, /metrics/prometheus and their /v1 versions; the same viewer key still runs normal queries on tenant_a	✅ pass	live	access-matrix.txt
No key or an invalid key gets 401 on every metrics route	✅ pass	live	access-matrix.txt
Unscoped admin and bootstrap admin keys get 200 with JSON and Prometheus text on all four routes	✅ pass	live	access-matrix.txt
.status over WebSocket: scoped admin, viewer and editor keys see 'Knowledge graphs: 1' and no 'Total queries'; the unscoped admin sees 4 graphs and the query total	✅ pass	live	status-scoping.txt
Admin scrape shows per-graph gauges (inputlayer_kg_tuples{kg=tenant_a}=2, tenant_b=3, kg_sessions) and the counts follow writes (tenant_b 3 to 5)	✅ pass	live	prometheus-during-load.txt, wal-subscription-gauges.txt
inputlayer_memory_usage_bytes and /metrics memory_usage_bytes equal the process RSS (94519296 = VmRSS), not tuples*64	✅ pass	live	memory-rss.txt
Subscription metrics: subscriptions_active is 1 while subscribed and 0 after disconnect; evaluations_total goes up after a write that sends a delta	✅ pass	live	wal-subscription-gauges.txt, prometheus-during-load.txt
WAL and flush gauges: wal_size_bytes and buffered_updates go up on a write; persist_flushes_total goes up when buffer_size is reached	✅ pass	live	wal-subscription-gauges.txt, ratelimit-connlimit-flush.txt
Rejections are counted: 401 (http_unauthorized=8), 403 (http_forbidden=12), WebSocket over the limit (503, ws_connection_limit=1), rate limit (15 × 429, http_rate_limit=15), connection limit (49 × 503…	✅ pass	live	prometheus-during-load.txt, ratelimit-connlimit-flush.txt, http-connection-limit.txt
Connection and auth-failure gauges: ws_connections=3 with 1 unauthenticated under load; auth_failures_total{api_key}=4 for 4 bad Bearer keys; {password} goes up after a failed login	✅ pass	live	prometheus-during-load.txt, wal-subscription-gauges.txt
The change's focused automated tests (metrics_tests, test_metrics_refuse_non_admin_keys, test_metrics_follow_a_demoted_admin, status_shows_only_the_knowledge_graphs_a_caller_can_see) pass	⏸️ untested	no	The earlier payload backed this scenario only with `cargo test` runs, which are automated tests and not a live run of the product, so it did not establish a live result. The live behaviour these tests…

Evidence: Metrics endpoint access matrix (401/403/200 by caller and route)

# Access matrix for metrics endpoints (status code + first 120 bytes of body)
none                       /metrics                 401  Invalid or missing API key. Send `Authorization: Bearer <key>`. On first boot the server issues an admin API key: the va
none                       /metrics/prometheus      401  Invalid or missing API key. Send `Authorization: Bearer <key>`. On first boot the server issues an admin API key: the va
none                       /v1/metrics              401  Invalid or missing API key. Send `Authorization: Bearer <key>`. On first boot the server issues an admin API key: the va
none                       /v1/metrics/prometheus   401  Invalid or missing API key. Send `Authorization: Bearer <key>`. On first boot the server issues an admin API key: the va
bad                        /metrics                 401  Invalid or missing API key. Send `Authorization: Bearer <key>`. On first boot the server issues an admin API key: the va
bad                        /metrics/prometheus      401  Invalid or missing API key. Send `Authorization: Bearer <key>`. On first boot the server issues an admin API key: the va
bad                        /v1/metrics              401  Invalid or missing API key. Send `Authorization: Bearer <key>`. On first boot the server issues an admin API key: the va
bad                        /v1/metrics/prometheus   401  Invalid or missing API key. Send `Authorization: Bearer <key>`. On first boot the server issues an admin API key: the va
viewer(alice)              /metrics                 403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
viewer(alice)              /metrics/prometheus      403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
viewer(alice)              /v1/metrics              403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
viewer(alice)              /v1/metrics/prometheus   403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
editor(bob)                /metrics                 403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
editor(bob)                /metrics/prometheus      403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
editor(bob)                /v1/metrics              403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
editor(bob)                /v1/metrics/prometheus   403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
admin-scoped-to-tenant_a   /metrics                 403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
admin-scoped-to-tenant_a   /metrics/prometheus      403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
admin-scoped-to-tenant_a   /v1/metrics              403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
admin-scoped-to-tenant_a   /v1/metrics/prometheus   403  This endpoint needs an admin API key: an unscoped key of a user with the admin role. Keys of editors and viewers, and ke
admin-full                 /metrics                 200  {"success":true,"data":{"knowledge_graphs":4,"relations":9,"views":0,"memory_usage_bytes":90324992,"query_count":6,"upti
admin-full                 /metrics/prometheus      200  # HELP inputlayer_uptime_seconds Server uptime in seconds. # TYPE inputlayer_uptime_seconds gauge inputlayer_uptime_seco
admin-full                 /v1/metrics              200  {"success":true,"data":{"knowledge_graphs":4,"relations":9,"views":0,"memory_usage_bytes":90324992,"query_count":6,"upti
admin-full                 /v1/metrics/prometheus   200  # HELP inputlayer_uptime_seconds Server uptime in seconds. # TYPE inputlayer_uptime_seconds gauge inputlayer_uptime_seco
bootstrap-admin            /metrics                 200  {"success":true,"data":{"knowledge_graphs":4,"relations":9,"views":0,"memory_usage_bytes":90324992,"query_count":6,"upti
bootstrap-admin            /metrics/prometheus      200  # HELP inputlayer_uptime_seconds Server uptime in seconds. # TYPE inputlayer_uptime_seconds gauge inputlayer_uptime_seco
bootstrap-admin            /v1/metrics              200  {"success":true,"data":{"knowledge_graphs":4,"relations":9,"views":0,"memory_usage_bytes":90324992,"query_count":6,"upti
bootstrap-admin            /v1/metrics/prometheus   200  # HELP inputlayer_uptime_seconds Server uptime in seconds. # TYPE inputlayer_uptime_seconds gauge inputlayer_uptime_seco

# Same viewer/editor keys still authenticate elsewhere (proves 403 is role-based, not a dead key):
{"type":"authenticated","session_id":"09d230d0-28d0-4899-a726-f35983779848","knowledge_graph":"tenant_a","version":"0.1.0","role":"viewer","protocol_version":5,"stream_epoch":"7d0be207a22245c6"}
{"type":"result","id":"r1","columns":["X","Y"],"rows":[[1,"alpha"],[2,"beta"]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":2,"row_provenance":["unknown","unknown"],"timing_brea
closed 1006

Evidence: .status output per caller (scoped and non-admin keys see 1 graph and no query total; admin sees 4 graphs and the total)

## admin-scoped-to-tenant_a
"rows":[["Server Status"],["  Health: healthy"],["  Version: 0.1.0"],["  Uptime: 58 seconds"],["  Knowledge graphs: 1"]],
## viewer(alice, ACL on tenant_a only)
"rows":[["Server Status"],["  Health: healthy"],["  Version: 0.1.0"],["  Uptime: 58 seconds"],["  Knowledge graphs: 1"]],
## editor(bob, ACL on tenant_b only)
"rows":[["Server Status"],["  Health: healthy"],["  Version: 0.1.0"],["  Uptime: 59 seconds"],["  Knowledge graphs: 1"]],
## admin-full (unscoped)
"rows":[["Server Status"],["  Health: healthy"],["  Version: 0.1.0"],["  Uptime: 59 seconds"],["  Total queries: 11"],["  Knowledge graphs: 4"]],

Evidence: Full Prometheus scrape under load (subscription, WebSocket limit hit, rejections)

# HELP inputlayer_uptime_seconds Server uptime in seconds.
# TYPE inputlayer_uptime_seconds gauge
inputlayer_uptime_seconds 71
# HELP inputlayer_queries_total Total queries executed.
# TYPE inputlayer_queries_total counter
inputlayer_queries_total 12
# HELP inputlayer_knowledge_graphs Number of knowledge graphs.
# TYPE inputlayer_knowledge_graphs gauge
inputlayer_knowledge_graphs 4
# HELP inputlayer_knowledge_graphs_loaded Knowledge graphs held in memory.
# TYPE inputlayer_knowledge_graphs_loaded gauge
inputlayer_knowledge_graphs_loaded 4
# HELP inputlayer_knowledge_graph_loads_total Knowledge graphs loaded from disk on use.
# TYPE inputlayer_knowledge_graph_loads_total counter
inputlayer_knowledge_graph_loads_total 0
# HELP inputlayer_knowledge_graph_unloads_total Knowledge graphs unloaded from memory.
# TYPE inputlayer_knowledge_graph_unloads_total counter
inputlayer_knowledge_graph_unloads_total 0
# HELP inputlayer_relations_total Total base relations.
# TYPE inputlayer_relations_total gauge
inputlayer_relations_total 9
# HELP inputlayer_views_total Total derived views (rules).
# TYPE inputlayer_views_total gauge
inputlayer_views_total 0
# HELP inputlayer_tuples_total Total stored tuples.
# TYPE inputlayer_tuples_total gauge
inputlayer_tuples_total 31
# HELP inputlayer_kg_relations Base relations holding facts, per knowledge graph.
# TYPE inputlayer_kg_relations gauge
inputlayer_kg_relations{kg="_internal"} 7
inputlayer_kg_relations{kg="default"} 0
inputlayer_kg_relations{kg="tenant_a"} 1
inputlayer_kg_relations{kg="tenant_b"} 1
# HELP inputlayer_kg_tuples Stored tuples, per knowledge graph.
# TYPE inputlayer_kg_tuples gauge
inputlayer_kg_tuples{kg="_internal"} 26
inputlayer_kg_tuples{kg="default"} 0
inputlayer_kg_tuples{kg="tenant_a"} 2
inputlayer_kg_tuples{kg="tenant_b"} 3
# HELP inputlayer_kg_views Derived views (rules), per knowledge graph.
# TYPE inputlayer_kg_views gauge
inputlayer_kg_views{kg="_internal"} 0
inputlayer_kg_views{kg="default"} 0
inputlayer_kg_views{kg="tenant_a"} 0
inputlayer_kg_views{kg="tenant_b"} 0
# HELP inputlayer_kg_loaded 1 while the knowledge graph is held in memory, else 0.
# TYPE inputlayer_kg_loaded gauge
inputlayer_kg_loaded{kg="_internal"} 1
inputlayer_kg_loaded{kg="default"} 1
inputlayer_kg_loaded{kg="tenant_a"} 1
inputlayer_kg_loaded{kg="tenant_b"} 1
# HELP inputlayer_kg_sessions Open sessions, per knowledge graph they are bound to.
# TYPE inputlayer_kg_sessions gauge
inputlayer_kg_sessions{kg="tenant_b"} 2
# HELP inputlayer_memory_usage_bytes Resident set size of the server process in bytes.
# TYPE inputlayer_memory_usage_bytes gauge
inputlayer_memory_usage_bytes 94519296
# HELP inputlayer_memory_limit_bytes The memory limit of the server's cgroup (container or systemd unit).
# TYPE inputlayer_memory_limit_bytes gauge
inputlayer_memory_limit_bytes 126701535232
# HELP inputlayer_query_memory_bytes Bytes held by the computations of the requests in flight.
# TYPE inputlayer_query_memory_bytes gauge
inputlayer_query_memory_bytes 0
# HELP inputlayer_query_memory_budget_bytes Most bytes the computations in flight may hold together (storage.performance.max_total_query_memory_bytes; 0: no limit).
# TYPE inputlayer_query_memory_budget_bytes gauge
inputlayer_query_memory_budget_bytes 76020921138
# HELP inputlayer_compute_permits Queries that can compute at once; more wait for a permit.
# TYPE inputlayer_compute_permits gauge
inputlayer_compute_permits 24
# HELP inputlayer_compute_permits_in_use Compute permits held by queries running now.
# TYPE inputlayer_compute_permits_in_use gauge
inputlayer_compute_permits_in_use 0
# HELP inputlayer_sessions_total Active sessions.
# TYPE inputlayer_sessions_total gauge
inputlayer_sessions_total 2
# HELP inputlayer_sessions_clean Sessions with no ephemeral facts or rules.
# TYPE inputlayer_sessions_clean gauge
inputlayer_sessions_clean 2
# HELP inputlayer_sessions_dirty Sessions holding ephemeral facts or rules.
# TYPE inputlayer_sessions_dirty gauge
inputlayer_sessions_dirty 0
# HELP inputlayer_ephemeral_facts Total ephemeral facts across sessions.
# TYPE inputlayer_ephemeral_facts gauge
inputlayer_ephemeral_facts 0
# HELP inputlayer_ephemeral_rules Total ephemeral rules across sessions.
# TYPE inputlayer_ephemeral_rules gauge
inputlayer_ephemeral_rules 0
# HELP inputlayer_subscriptions_active Subscriptions registered across all connections.
# TYPE inputlayer_subscriptions_active gauge
inputlayer_subscriptions_active 1
# HELP inputlayer_subscription_views Shared standing-query views being evaluated.
# TYPE inputlayer_subscription_views gauge
inputlayer_subscription_views 1
# HELP inputlayer_subscription_evaluations_total Standing-query evaluations (initial snapshots and re-evaluations), one per shared view.
# TYPE inputlayer_subscription_evaluations_total counter
inputlayer_subscription_evaluations_total 1
# HELP inputlayer_subscription_shared_evaluations_total Evaluations of lifted queries, each serving every view of its family.
# TYPE inputlayer_subscription_shared_evaluations_total counter
inputlayer_subscription_shared_evaluations_total 0
# HELP inputlayer_wal_size_bytes Bytes in the write-ahead log.
# TYPE inputlayer_wal_size_bytes gauge
inputlayer_wal_size_bytes 6174
# HELP inputlayer_wal_size_limit_bytes WAL size that makes the next commit flush every shard (storage.persist.max_wal_size_bytes; 0: no limit).
# TYPE inputlayer_wal_size_limit_bytes gauge
inputlayer_wal_size_limit_bytes 67108864
# HELP inputlayer_persist_dirty_shards Shards holding committed updates not yet flushed to a batch file.
# TYPE inputlayer_persist_dirty_shards gauge
inputlayer_persist_dirty_shards 9
# HELP inputlayer_persist_buffered_updates Committed updates not yet flushed to a batch file (they are in the WAL).
# TYPE inputlayer_persist_buffered_updates gauge
inputlayer_persist_buffered_updates 35
# HELP inputlayer_persist_oldest_unflushed_seconds How long the longest-waiting dirty shard has held unflushed updates (0: none).
# TYPE inputlayer_persist_oldest_unflushed_seconds gauge
inputlayer_persist_oldest_unflushed_seconds 71.159255661
# HELP inputlayer_persist_flushes_total Shard buffers flushed to batch files.
# TYPE inputlayer_persist_flushes_total counter
inputlayer_persist_flushes_total 0
# HELP inputlayer_persist_flush_failures_total Shard flushes that failed (the updates stay in the WAL and are retried).
# TYPE inputlayer_persist_flush_failures_total counter
inputlayer_persist_flush_failures_total 0
# HELP inputlayer_store_read_only 1 once the store refuses writes until restart after a failed WAL write or fsync, else 0.
# TYPE inputlayer_store_read_only gauge
inputlayer_store_read_only 0
# HELP inputlayer_http_requests_in_flight_limit rate_limit.max_connections (0: no limit).
# TYPE inputlayer_http_requests_in_flight_limit gauge
inputlayer_http_requests_in_flight_limit 1024
# HELP inputlayer_ws_connections_limit rate_limit.max_ws_connections (0: no limit).
# TYPE inputlayer_ws_connections_limit gauge
inputlayer_ws_connections_limit 3
# HELP inputlayer_http_responses_total HTTP responses by status code (0: outside 100-599).
# TYPE inputlayer_http_responses_total counter
inputlayer_http_responses_total{code="101"} 18
inputlayer_http_responses_total{code="200"} 9
inputlayer_http_responses_total{code="401"} 8
inputlayer_http_responses_total{code="403"} 12
inputlayer_http_responses_total{code="503"} 1
# HELP inputlayer_rejections_total Requests and connections refused before they ran, by reason.
# TYPE inputlayer_rejections_total counter
inputlayer_rejections_total{reason="http_connection_limit"} 0
inputlayer_rejections_total{reason="http_rate_limit"} 0
inputlayer_rejections_total{reason="http_unauthorized"} 8
inputlayer_rejections_total{reason="http_forbidden"} 12
inputlayer_rejections_total{reason="ws_connection_limit"} 1
inputlayer_rejections_total{reason="ws_preauth_limit"} 0
inputlayer_rejections_total{reason="ws_rate_limit"} 0
inputlayer_rejections_total{reason="login_throttled"} 0
inputlayer_rejections_total{reason="login_busy"} 0
# HELP inputlayer_auth_failures_total Failed authentications (wrong password, unknown, revoked or expired API key), by method.
# TYPE inputlayer_auth_failures_total counter
inputlayer_auth_failures_total{method="password"} 0
inputlayer_auth_failures_total{method="api_key"} 4
# HELP inputlayer_ws_notices_total WebSocket notices sent, by code. All but notifications_missed and replay_gap close the connection.
# TYPE inputlayer_ws_notices_total counter
inputlayer_ws_notices_total{code="notifications_missed"} 0
inputlayer_ws_notices_total{code="replay_gap"} 0
inputlayer_ws_notices_total{code="slow_consumer"} 0
inputlayer_ws_notices_total{code="idle_timeout"} 0
inputlayer_ws_notices_total{code="lifetime_exceeded"} 0
inputlayer_ws_notices_total{code="auth_timeout"} 0
inputlayer_ws_notices_total{code="credential_revoked"} 0
inputlayer_ws_notices_total{code="credential_expired"} 0
inputlayer_ws_notices_total{code="server_shutdown"} 0
# HELP inputlayer_ws_errors_total WebSocket requests answered with an error frame, by error code (none: no code).
# TYPE inputlayer_ws_errors_total counter
inputlayer_ws_errors_total{code="store_read_only"} 0
inputlayer_ws_errors_total{code="validation"} 0
inputlayer_ws_errors_total{code="not_found"} 0
inputlayer_ws_errors_total{code="conflict"} 0
inputlayer_ws_errors_total{code="unsupported"} 0
inputlayer_ws_errors_total{code="internal"} 0
inputlayer_ws_errors_total{code="invalid_request"} 0
inputlayer_ws_errors_total{code="rate_limited"} 0
inputlayer_ws_errors_total{code="deadline_exceeded"} 0
inputlayer_ws_errors_total{code="cancelled"} 0
inputlayer_ws_errors_total{code="precondition_failed"} 0
inputlayer_ws_errors_total{code="outcome_unknown"} 0
inputlayer_ws_errors_total{code="resource_exhausted"} 0
inputlayer_ws_errors_total{code="none"} 2
# HELP inputlayer_ws_send_timeouts_total WebSocket connections closed because the client left a frame unread for http.ws_send_timeout_ms.
# TYPE inputlayer_ws_send_timeouts_total counter
inputlayer_ws_send_timeouts_total 0
# HELP inputlayer_http_requests_in_flight HTTP requests being served (limit: rate_limit.max_connections).
# TYPE inputlayer_http_requests_in_flight gauge
inputlayer_http_requests_in_flight 1
# HELP inputlayer_ws_connections Open WebSocket connections (limit: rate_limit.max_ws_connections).
# TYPE inputlayer_ws_connections gauge
inputlayer_ws_connections 3
# HELP inputlayer_ws_connections_unauthenticated Open WebSocket connections that have not authenticated yet.
# TYPE inputlayer_ws_connections_unauthenticated gauge
inputlayer_ws_connections_unauthenticated 1
# HELP inputlayer_query_parse_seconds Time spent parsing query source.
# TYPE inputlayer_query_parse_seconds histogram
inputlayer_query_parse_seconds_bucket{le="0.001"} 2
inputlayer_query_parse_seconds_bucket{le="0.005"} 2
inputlayer_query_parse_seconds_bucket{le="0.01"} 2
inputlayer_query_parse_seconds_bucket{le="0.05"} 2
inputlayer_query_parse_seconds_bucket{le="0.1"} 2
inputlayer_query_parse_seconds_bucket{le="0.5"} 2
inputlayer_query_parse_seconds_bucket{le="1"} 2
inputlayer_query_parse_seconds_bucket{le="5"} 2
inputlayer_query_parse_seconds_bucket{le="+Inf"} 2
inputlayer_query_parse_seconds_sum 0.000379
inputlayer_query_parse_seconds_count 2
# HELP inputlayer_query_optimize_seconds Time spent optimizing query plan.
# TYPE inputlayer_query_optimize_seconds histogram
inputlayer_query_optimize_seconds_bucket{le="0.001"} 2
inputlayer_query_optimize_seconds_bucket{le="0.005"} 2
inputlayer_query_optimize_seconds_bucket{le="0.01"} 2
inputlayer_query_optimize_seconds_bucket{le="0.05"} 2
inputlayer_query_optimize_seconds_bucket{le="0.1"} 2
inputlayer_query_optimize_seconds_bucket{le="0.5"} 2
inputlayer_query_optimize_seconds_bucket{le="1"} 2
inputlayer_query_optimize_seconds_bucket{le="5"} 2
inputlayer_query_optimize_seconds_bucket{le="+Inf"} 2
inputlayer_query_optimize_seconds_sum 0.000155
inputlayer_query_optimize_seconds_count 2
# HELP inputlayer_query_execute_seconds Time spent executing query (DD computation).
# TYPE inputlayer_query_execute_seconds histogram
inputlayer_query_execute_seconds_bucket{le="0.001"} 0
inputlayer_query_execute_seconds_bucket{le="0.005"} 2
inputlayer_query_execute_seconds_bucket{le="0.01"} 2
inputlayer_query_execute_seconds_bucket{le="0.05"} 2
inputlayer_query_execute_seconds_bucket{le="0.1"} 2
inputlayer_query_execute_seconds_bucket{le="0.5"} 2
inputlayer_query_execute_seconds_bucket{le="1"} 2
inputlayer_query_execute_seconds_bucket{le="5"} 2
inputlayer_query_execute_seconds_bucket{le="+Inf"} 2
inputlayer_query_execute_seconds_sum 0.003531
inputlayer_query_execute_seconds_count 2
# HELP inputlayer_query_total_seconds Total end-to-end query time.
# TYPE inputlayer_query_total_seconds histogram
inputlayer_query_total_seconds_bucket{le="0.001"} 0
inputlayer_query_total_seconds_bucket{le="0.005"} 2
inputlayer_query_total_seconds_bucket{le="0.01"} 2
inputlayer_query_total_seconds_bucket{le="0.05"} 2
inputlayer_query_total_seconds_bucket{le="0.1"} 2
inputlayer_query_total_seconds_bucket{le="0.5"} 2
inputlayer_query_total_seconds_bucket{le="1"} 2
inputlayer_query_total_seconds_bucket{le="5"} 2
inputlayer_query_total_seconds_bucket{le="+Inf"} 2
inputlayer_query_total_seconds_sum 0.004362
inputlayer_query_total_seconds_count 2

Evidence: Memory gauge equals /proc VmRSS

server pid 3294508 (~/.no-mistakes/worktrees/9eea230c3aae/01M46PSRBZRTKV3NC471W6MYJT/target/debug/inputlayer-server -c cfg.toml )
inputlayer_memory_usage_bytes 94519296
VmRSS bytes from /proc: 94519296
old estimate (total_tuples*64) = 1984
/metrics JSON:
{"success":true,"data":{"knowledge_graphs":4,"relations":9,"views":0,"memory_usage_bytes":94519296,"query_count":12,"uptime_secs":94,"sessions":{"total":0,"clean":0,"dirty":0,"total_ephemeral_facts":0,"total_ephemeral_rules":0}}}

Evidence: WAL, subscription and auth-failure counters before and after a write

## before
inputlayer_kg_tuples{kg="tenant_b"} 3
inputlayer_subscriptions_active 0
inputlayer_subscription_evaluations_total 1
inputlayer_wal_size_bytes 6174
inputlayer_persist_dirty_shards 9
inputlayer_persist_buffered_updates 35
inputlayer_auth_failures_total{method="password"} 0
inputlayer_ws_connections 0
## after a subscribed write of 2 facts into tenant_b and one failed password login (subscription still open)
inputlayer_kg_tuples{kg="tenant_b"} 5
inputlayer_subscriptions_active 1
inputlayer_subscription_evaluations_total 3
inputlayer_wal_size_bytes 6324
inputlayer_persist_dirty_shards 9
inputlayer_persist_buffered_updates 37
inputlayer_auth_failures_total{method="password"} 1
inputlayer_ws_connections 1
## subscriber received delta:
"type":"subscription_delta","subscription":"watch_b","generation":1,"knowledge_graph":"tenant_b","seq":1,"revision":24,"columns":["X","Y"],"inserted":[[13,"w"],[14,"v"]],"retracted":[]
## after subscriber disconnected
inputlayer_kg_tuples{kg="tenant_b"} 5
inputlayer_subscriptions_active 0
inputlayer_subscription_evaluations_total 3
inputlayer_wal_size_bytes 6324
inputlayer_persist_dirty_shards 9
inputlayer_persist_buffered_updates 37
inputlayer_auth_failures_total{method="password"} 1
inputlayer_ws_connections 0

Evidence: 429 rate limit counted, flush counter

# server: max_connections=1, per_ip_max_rps=5, persist buffer_size=2
## burst of 20 GET /health:
200 200 200 200 200 429 429 429 429 429 429 429 429 429 429 429 429 429 429 429 
## WS connection held open, then GET /health concurrently:
{"success":true,"data":{"status":"healthy","version":"0.1.0","uptime_secs":12}} -> 200
## scrape:
inputlayer_wal_size_bytes 758
inputlayer_persist_dirty_shards 4
inputlayer_persist_buffered_updates 4
inputlayer_persist_flushes_total 1
inputlayer_http_requests_in_flight_limit 1
inputlayer_http_responses_total{code="101"} 1
inputlayer_http_responses_total{code="200"} 7
inputlayer_http_responses_total{code="429"} 15
inputlayer_rejections_total{reason="http_connection_limit"} 0
inputlayer_rejections_total{reason="http_rate_limit"} 15
inputlayer_http_requests_in_flight 1

Evidence: 503 HTTP connection limit counted

# server: max_connections=1, per_ip_max_rps=0; 60 concurrent admin scrapes of /metrics/prometheus
     11 200
     49 503
## scrape after:
inputlayer_http_requests_in_flight_limit 1
inputlayer_http_responses_total{code="200"} 11
inputlayer_http_responses_total{code="503"} 49
inputlayer_rejections_total{reason="http_connection_limit"} 49
inputlayer_http_requests_in_flight 1

Evidence: WebSocket upgrade over the limit refused (503)

4th WS upgrade over limit: 503

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

🔧 **Rebase** - 2 issues found → auto-fixed ✅

⚠️ docs/content/docs/guides/rest-api.mdx - merge conflict rebasing onto origin/main
⚠️ gui/lib/docs-bundle.ts - merge conflict rebasing onto origin/main

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

Live validation: ✅ go - 10 of 11 scenarios driven live against the product

Scenario	Result	Live	Evidence
Viewer, editor and tenant_a-scoped admin keys get 403 on /metrics, /metrics/prometheus and their /v1 versions; the same viewer key still runs normal queries on tenant_a	✅ pass	live	access-matrix.txt
No key or an invalid key gets 401 on every metrics route	✅ pass	live	access-matrix.txt
Unscoped admin and bootstrap admin keys get 200 with JSON and Prometheus text on all four routes	✅ pass	live	access-matrix.txt
.status over WebSocket: scoped admin, viewer and editor keys see 'Knowledge graphs: 1' and no 'Total queries'; the unscoped admin sees 4 graphs and the query total	✅ pass	live	status-scoping.txt
Admin scrape shows per-graph gauges (inputlayer_kg_tuples{kg=tenant_a}=2, tenant_b=3, kg_sessions) and the counts follow writes (tenant_b 3 to 5)	✅ pass	live	prometheus-during-load.txt, wal-subscription-gauges.txt
inputlayer_memory_usage_bytes and /metrics memory_usage_bytes equal the process RSS (94519296 = VmRSS), not tuples*64	✅ pass	live	memory-rss.txt
Subscription metrics: subscriptions_active is 1 while subscribed and 0 after disconnect; evaluations_total goes up after a write that sends a delta	✅ pass	live	wal-subscription-gauges.txt, prometheus-during-load.txt
WAL and flush gauges: wal_size_bytes and buffered_updates go up on a write; persist_flushes_total goes up when buffer_size is reached	✅ pass	live	wal-subscription-gauges.txt, ratelimit-connlimit-flush.txt
Rejections are counted: 401 (http_unauthorized=8), 403 (http_forbidden=12), WebSocket over the limit (503, ws_connection_limit=1), rate limit (15 × 429, http_rate_limit=15), connection limit (49 × 503…	✅ pass	live	prometheus-during-load.txt, ratelimit-connlimit-flush.txt, http-connection-limit.txt
Connection and auth-failure gauges: ws_connections=3 with 1 unauthenticated under load; auth_failures_total{api_key}=4 for 4 bad Bearer keys; {password} goes up after a failed login	✅ pass	live	prometheus-during-load.txt, wal-subscription-gauges.txt
The change's focused automated tests (metrics_tests, test_metrics_refuse_non_admin_keys, test_metrics_follow_a_demoted_admin, status_shows_only_the_knowledge_graphs_a_caller_can_see) pass	⏸️ untested	no	The earlier payload backed this scenario only with `cargo test` runs, which are automated tests and not a live run of the product, so it did not establish a live result. The live behaviour these tests…

Built cargo build --bin inputlayer-server and ran it with an isolated config and data dir under /tmp (ports 18931 and 18932), bootstrap admin key from an env var
Over WebSocket (Node built-in WebSocket driver ws.mjs): created graphs tenant_a and tenant_b with facts, viewer and editor users with per-graph access (keys issued while the users were admins, then demoted to viewer/editor), an admin key scoped to tenant_a, and an unscoped admin key
curl access matrix: 7 callers (no key, bad key, viewer, editor, scoped admin, unscoped admin, bootstrap admin) × /metrics, /metrics/prometheus, /v1/metrics, /v1/metrics/prometheus
.status over WebSocket as the scoped admin key, the viewer key, the editor key and the unscoped admin key
Scraped /metrics/prometheus with a subscription open, 3 WebSocket connections (limit 3) and a 4th upgrade refused with 503; checked the kg-labelled gauges, subscription, WebSocket, rejection, auth-failure, WAL and persist metrics
Compared inputlayer_memory_usage_bytes and the /metrics JSON memory_usage_bytes with /proc/<pid>/status VmRSS
Scraped before and after a subscribed write and a failed password login: the subscriber got its delta; tuples, WAL bytes, subscription evaluations and password auth failures went up
Second server (per_ip_max_rps=5, buffer_size=2): sent a burst of 20 GET /health, checked the 429 and flush counters
Third run (max_connections=1, no per-IP limit): 60 concurrent scrapes, checked the 503 and http_connection_limit counters
cargo test --test metrics_tests (3 passed)
cargo test --lib -- status_shows_only_the_knowledge_graphs test_metrics (3 passed)

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

…a Monitoring guide /metrics and /metrics/prometheus aggregate over every knowledge graph but authorized any valid API key, so a viewer key or a key scoped to one KG could read every tenant's KG, relation, tuple and session counts. Both now need an unscoped admin key and answer others with 403. The Prometheus export gains what an operator needs to run the engine: HTTP responses by status, in-flight requests and open WebSocket connections against their limits, rejections by reason (connection and rate limits, 401/403, WS preauth and message rate, login throttle), failed authentications by method, WS error replies by code, WS notices (slow consumers, timeouts, revocations) and send timeouts, subscription counts and evaluations, WAL size, dirty shards, unflushed updates and their age, flushes and flush failures, the store's read-only state, per-KG relation/tuple/view/loaded/session gauges, query memory and compute permits in use. memory_usage_bytes is now the process RSS instead of tuples x 64, with the cgroup limit beside it. The deployment guide documents the endpoint, its auth, a scrape config, every metric and recommended alerts. Closes #300

…graphs

… of timing

…nitoring

jsam force-pushed the fm/fix-metrics-leak branch from 34a7a15 to 93a53d9 Compare October 5, 2026 17:38

jsam changed the title ~~fix(metrics): admin-only metrics endpoints, operational metrics, and a Monitoring guide~~ fix(protocol): admin-only metrics endpoints, operational metrics, and a Monitoring guide Oct 5, 2026

jsam force-pushed the fm/fix-metrics-leak branch from 305858a to 57f5983 Compare October 5, 2026 18:38

jsam changed the title ~~fix(protocol): admin-only metrics endpoints, operational metrics, and a Monitoring guide~~ fix(protocol): restrict metrics endpoints to admin keys and add operational metrics Oct 5, 2026

jsam added 4 commits October 5, 2026 18:57

no-mistakes(review): Limit .status to the caller's visible knowledge …

0c4639d

…graphs

no-mistakes(test): Make WebSocket rate-limit metrics test independent…

f018ad2

… of timing

no-mistakes(document): Document scoped .status output and point to Mo…

532071a

…nitoring

jsam force-pushed the fm/fix-metrics-leak branch from 57f5983 to 532071a Compare October 5, 2026 19:08

jsam changed the title ~~fix(protocol): restrict metrics endpoints to admin keys and add operational metrics~~ fix(protocol): restrict metrics endpoints to admin keys and add operational Prometheus metrics Oct 5, 2026

jsam marked this pull request as ready for review October 5, 2026 19:17

jsam merged commit 41f38ef into main Oct 5, 2026
23 checks passed

jsam deleted the fm/fix-metrics-leak branch October 5, 2026 19:19

This was referenced Oct 5, 2026

feat(perf-gate): add views benchmark and view/rule-evaluation counters on /metrics/prometheus #381

Merged

feat(perf-gate): views benchmark and view work counters on /metrics/prometheus #369

Closed

Navigation Menu

Repository navigation

fix(protocol): restrict metrics endpoints to admin keys and add operational Prometheus metrics - #365

fix(protocol): restrict metrics endpoints to admin keys and add operational Prometheus metrics#365
jsam merged 4 commits into
mainfrom
fm/fix-metrics-leak

jsam commented Oct 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jsam commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jsam commented Oct 5, 2026 •

edited

Loading