Skip to content

Monitoring

Probes

ProbeEndpointBehaviour
LivenessGET /v0/livez{"status":"ok"}. Takes no lock and reads no namespace, so it stays green while a long write or compaction runs. The official image’s HEALTHCHECK uses it.
ReadinessGET /v0/health200 with namespace state, or 503 with status: "degraded" when the writer is degraded or the memory ceiling is reached.
Terminal window
curl -s localhost:8080/v0/health
# {"status":"ok","namespace":"prod","manifest_version":8,"epoch":0,"writer":"ok",
# "memtable_bytes":0,"memory_resident_bytes":26001408,"memory_max_bytes":0,"memory_pressure":false}

Field meanings are in the HTTP API reference. In multi-tenant mode, /:namespace/v0/health opens that namespace; point node-level probes at /v0/livez.

A Kubernetes example:

livenessProbe:
httpGet: { path: /v0/livez, port: 8080 }
periodSeconds: 15
readinessProbe:
httpGet: { path: /v0/health, port: 8080 }
periodSeconds: 10
failureThreshold: 3

Metrics

GET /v0/metrics serves the Prometheus text format. It needs no token (like the probes); restrict it at the network level if required. With TLS on, it is served over HTTPS on the same listener.

scrape_configs:
- job_name: namidb
metrics_path: /v0/metrics
static_configs:
- targets: ["namidb:8080"]

Queries

MetricTypeLabelsMeaning
namidb_queries_totalcounterprotocol (http, bolt), status (ok, error)Statements executed
namidb_query_duration_secondshistogramprotocol, kind (read, write)Execution wall clock (backpressure sleeps excluded)
namidb_queries_in_flightgaugeStatements executing now
namidb_slow_queries_totalcounterStatements over --slow-query-threshold

Bolt schema-introspection calls made by GUIs are not counted as queries.

Writer and compaction

MetricTypeLabelsMeaning
namidb_writer_lock_wait_secondshistogramTime waiting for a namespace’s writer lock
namidb_writer_lock_timeouts_totalcounterWaits that hit --writer-lock-timeout (each one a 503)
namidb_compactions_totalcountertrigger (periodic, reactive, ddl, admin), statusCompaction attempts
namidb_compaction_phase_duration_secondshistogramtrigger, phase (prepare, install_wait, install_hold)Compaction time split by phase
namidb_compaction_l0_backlog_sstsgaugetrigger, stage (before, after)Deepest L0 bucket seen at a compaction boundary
namidb_compaction_ssts_totalcountertrigger, actionFiles removed or written by compaction

Memory and caches

MetricTypeMeaning
namidb_memory_resident_bytesgaugeLast sampled process RSS / working set
namidb_memory_max_bytesgaugeConfigured ceiling (0 = off)
namidb_memory_reclaims_totalcounterCache pressure-relief passes
namidb_memory_rejected_queries_totalcounterStatements refused at the ceiling
namidb_query_memory_limit_bytesgaugePer-query working-memory budget (0 = off)
namidb_query_memory_pool_bytesgaugeShared query pool by kind: capacity, reserved (now), peak
namidb_query_memory_stops_totalcounterRunning statements stopped with 503 memory_pressure, by reason: pool (the shared pool was full) or process (the process reached its ceiling). A 413 query_memory_limit is not counted here.
namidb_cache_max_bytesgaugeNAMIDB_CACHE_MAX_BYTES
namidb_cache_capacity_bytesgaugeCapacity assigned to enabled cache tiers
namidb_cache_resident_bytesgaugeBytes currently cached
namidb_range_cache_enabledgaugeWhether the object-range cache is on
namidb_range_cache_bytesgaugeRange cache by tier (ram, nvme) and kind (budget, capacity, usage, …)
namidb_range_cache_lookups_totalcounterBy outcome: memory_hit, disk_hit, miss
namidb_range_cache_remote_fetches_totalcounterObject-store reads after a cache miss
namidb_range_cache_events_totalcounterInserts, rejections and corruption events
namidb_range_cache_nvme_io_bytes_totalcounterBytes read from / written to the NVMe tier

Search and lookups

MetricTypeMeaning
namidb_search_index_cache_capacity_bytesgaugeShared decoded vector / full-text pool
namidb_search_index_cache_admission_rejections_totalcounterIndex bodies too large for the pool, by kind (vector, text)
namidb_search_route_totalcounterSearch served by index or by fallback, by kind and route
namidb_vector_filter_bitmap_searches_totalcounterVector searches that applied in-index metadata postings
namidb_property_lookup_route_totalcounterProperty lookups by route (native, fallback)
namidb_tuple_lookup_route_totalcounterComposite-index lookups by route
namidb_search_workspace_enabled, namidb_search_workspace_bytes, namidb_search_workspace_reservations_totalgauge / counterTransient search-memory pool

Process

MetricTypeMeaning
namidb_build_info{version="3.3.0"}gaugeAlways 1
namidb_uptime_secondsgaugeSeconds since start

Some families appear only after the first event that feeds them (the first compaction, the first search).

What to alert on

SignalSuggests
/v0/health 503 for more than a minuteWriter degraded (fenced, or failing commits) or memory ceiling reached; check writer_error and logs
rate(namidb_queries_total{status="error"}[5m]) risingClient errors or server faults; split by protocol, then look at logs
p99 of namidb_query_duration_secondsLatency regressions
namidb_compaction_l0_backlog_ssts staying highCompaction falling behind writes; run admin/compact, check spool disk
increase(namidb_writer_lock_timeouts_total[5m]) > 0Long write transactions or a writer stuck behind maintenance
increase(namidb_memory_rejected_queries_total[5m]) > 0The memory ceiling is shedding load; resize caches or the container
increase(namidb_query_memory_stops_total[5m]) > 0Running queries stopped for memory: concurrent large queries fill the pool (reason="pool"), or the process is at its ceiling (reason="process")
namidb_search_index_cache_admission_rejections_total risingSearch index bodies do not fit their pool; raise NAMIDB_SEARCH_INDEX_CACHE_MAX_BYTES

Slow-query log

Statements at or above --slow-query-threshold (default 1s; 0s disables) are logged at WARN, whether they succeeded or failed:

WARN namidb_server::metrics: slow query protocol="http" kind="read" status="error" elapsed_ms=109 query=MATCH (a:N), (b:N), (c:N) RETURN count(*)

The statement text is logged truncated; parameters are never logged. A value written inline as a literal is part of the text, so pass sensitive values as parameters.

In-flight queries

Terminal window
curl -s localhost:8080/v0/admin/queries -H "Authorization: Bearer $ADMIN_TOKEN"
# {"queries":[{"elapsed_ms":2077,"id":8,"namespace":"prod","protocol":"http",
# "statement":"MATCH (a:N), (b:N), (c:N) RETURN count(*)"}]}
curl -s -X POST localhost:8080/v0/admin/queries/8/cancel -H "Authorization: Bearer $ADMIN_TOKEN"
# {"cancelled":8}

Both need a read-write token. Statements are listed longest-running first, with whitespace collapsed and text truncated to 300 characters. A cancelled query stops at its next cancellation check, within milliseconds, and its client gets 409 cancelled; a cancelled write is rolled back. In multi-tenant mode /:namespace/v0/admin/queries shows only that namespace.

Logs

The server logs to stdout via tracing. Set the level with RUST_LOG (default info), per module if needed:

Terminal window
RUST_LOG=info,namidb_storage=debug

At startup the info log records the resolved request deadlines, the auth mode, the cache capacities, the memory ceiling and the listeners, which is the quickest way to confirm a configuration took effect.