Monitoring
Probes
| Probe | Endpoint | Behaviour |
|---|---|---|
| Liveness | GET /v0/livez | {"status":"ok"}. Takes no lock and reads no namespace, so it stays green while a long write or compaction runs. The official image’s HEALTHCHECK uses it. |
| Readiness | GET /v0/health | 200 with namespace state, or 503 with status: "degraded" when the writer is degraded or the memory ceiling is reached. |
curl -s localhost:8080/v0/health# {"status":"ok","namespace":"prod","manifest_version":8,"epoch":0,"writer":"ok",# "memtable_bytes":0,"memory_resident_bytes":26001408,"memory_max_bytes":0,"memory_pressure":false}Field meanings are in the HTTP API reference.
In multi-tenant mode, /:namespace/v0/health opens that namespace;
point node-level probes at /v0/livez.
A Kubernetes example:
livenessProbe: httpGet: { path: /v0/livez, port: 8080 } periodSeconds: 15readinessProbe: httpGet: { path: /v0/health, port: 8080 } periodSeconds: 10 failureThreshold: 3Metrics
GET /v0/metrics serves the Prometheus text format. It needs no token (like
the probes); restrict it at the network level if required. With TLS on, it is
served over HTTPS on the same listener.
scrape_configs: - job_name: namidb metrics_path: /v0/metrics static_configs: - targets: ["namidb:8080"]Queries
| Metric | Type | Labels | Meaning |
|---|---|---|---|
namidb_queries_total | counter | protocol (http, bolt), status (ok, error) | Statements executed |
namidb_query_duration_seconds | histogram | protocol, kind (read, write) | Execution wall clock (backpressure sleeps excluded) |
namidb_queries_in_flight | gauge | Statements executing now | |
namidb_slow_queries_total | counter | Statements over --slow-query-threshold |
Bolt schema-introspection calls made by GUIs are not counted as queries.
Writer and compaction
| Metric | Type | Labels | Meaning |
|---|---|---|---|
namidb_writer_lock_wait_seconds | histogram | Time waiting for a namespace’s writer lock | |
namidb_writer_lock_timeouts_total | counter | Waits that hit --writer-lock-timeout (each one a 503) | |
namidb_compactions_total | counter | trigger (periodic, reactive, ddl, admin), status | Compaction attempts |
namidb_compaction_phase_duration_seconds | histogram | trigger, phase (prepare, install_wait, install_hold) | Compaction time split by phase |
namidb_compaction_l0_backlog_ssts | gauge | trigger, stage (before, after) | Deepest L0 bucket seen at a compaction boundary |
namidb_compaction_ssts_total | counter | trigger, action | Files removed or written by compaction |
Memory and caches
| Metric | Type | Meaning |
|---|---|---|
namidb_memory_resident_bytes | gauge | Last sampled process RSS / working set |
namidb_memory_max_bytes | gauge | Configured ceiling (0 = off) |
namidb_memory_reclaims_total | counter | Cache pressure-relief passes |
namidb_memory_rejected_queries_total | counter | Statements refused at the ceiling |
namidb_query_memory_limit_bytes | gauge | Per-query working-memory budget (0 = off) |
namidb_query_memory_pool_bytes | gauge | Shared query pool by kind: capacity, reserved (now), peak |
namidb_query_memory_stops_total | counter | Running statements stopped with 503 memory_pressure, by reason: pool (the shared pool was full) or process (the process reached its ceiling). A 413 query_memory_limit is not counted here. |
namidb_cache_max_bytes | gauge | NAMIDB_CACHE_MAX_BYTES |
namidb_cache_capacity_bytes | gauge | Capacity assigned to enabled cache tiers |
namidb_cache_resident_bytes | gauge | Bytes currently cached |
namidb_range_cache_enabled | gauge | Whether the object-range cache is on |
namidb_range_cache_bytes | gauge | Range cache by tier (ram, nvme) and kind (budget, capacity, usage, …) |
namidb_range_cache_lookups_total | counter | By outcome: memory_hit, disk_hit, miss |
namidb_range_cache_remote_fetches_total | counter | Object-store reads after a cache miss |
namidb_range_cache_events_total | counter | Inserts, rejections and corruption events |
namidb_range_cache_nvme_io_bytes_total | counter | Bytes read from / written to the NVMe tier |
Search and lookups
| Metric | Type | Meaning |
|---|---|---|
namidb_search_index_cache_capacity_bytes | gauge | Shared decoded vector / full-text pool |
namidb_search_index_cache_admission_rejections_total | counter | Index bodies too large for the pool, by kind (vector, text) |
namidb_search_route_total | counter | Search served by index or by fallback, by kind and route |
namidb_vector_filter_bitmap_searches_total | counter | Vector searches that applied in-index metadata postings |
namidb_property_lookup_route_total | counter | Property lookups by route (native, fallback) |
namidb_tuple_lookup_route_total | counter | Composite-index lookups by route |
namidb_search_workspace_enabled, namidb_search_workspace_bytes, namidb_search_workspace_reservations_total | gauge / counter | Transient search-memory pool |
Process
| Metric | Type | Meaning |
|---|---|---|
namidb_build_info{version="3.3.0"} | gauge | Always 1 |
namidb_uptime_seconds | gauge | Seconds since start |
Some families appear only after the first event that feeds them (the first compaction, the first search).
What to alert on
| Signal | Suggests |
|---|---|
/v0/health 503 for more than a minute | Writer degraded (fenced, or failing commits) or memory ceiling reached; check writer_error and logs |
rate(namidb_queries_total{status="error"}[5m]) rising | Client errors or server faults; split by protocol, then look at logs |
p99 of namidb_query_duration_seconds | Latency regressions |
namidb_compaction_l0_backlog_ssts staying high | Compaction falling behind writes; run admin/compact, check spool disk |
increase(namidb_writer_lock_timeouts_total[5m]) > 0 | Long write transactions or a writer stuck behind maintenance |
increase(namidb_memory_rejected_queries_total[5m]) > 0 | The memory ceiling is shedding load; resize caches or the container |
increase(namidb_query_memory_stops_total[5m]) > 0 | Running queries stopped for memory: concurrent large queries fill the pool (reason="pool"), or the process is at its ceiling (reason="process") |
namidb_search_index_cache_admission_rejections_total rising | Search index bodies do not fit their pool; raise NAMIDB_SEARCH_INDEX_CACHE_MAX_BYTES |
Slow-query log
Statements at or above --slow-query-threshold (default 1s; 0s
disables) are logged at WARN, whether they succeeded or failed:
WARN namidb_server::metrics: slow query protocol="http" kind="read" status="error" elapsed_ms=109 query=MATCH (a:N), (b:N), (c:N) RETURN count(*)The statement text is logged truncated; parameters are never logged. A value written inline as a literal is part of the text, so pass sensitive values as parameters.
In-flight queries
curl -s localhost:8080/v0/admin/queries -H "Authorization: Bearer $ADMIN_TOKEN"# {"queries":[{"elapsed_ms":2077,"id":8,"namespace":"prod","protocol":"http",# "statement":"MATCH (a:N), (b:N), (c:N) RETURN count(*)"}]}
curl -s -X POST localhost:8080/v0/admin/queries/8/cancel -H "Authorization: Bearer $ADMIN_TOKEN"# {"cancelled":8}Both need a read-write token. Statements are listed longest-running first,
with whitespace collapsed and text truncated to 300 characters. A cancelled
query stops at its next cancellation check, within milliseconds, and its
client gets 409 cancelled; a cancelled write is rolled back. In multi-tenant mode /:namespace/v0/admin/queries shows only that
namespace.
Logs
The server logs to stdout via tracing. Set the level with RUST_LOG
(default info), per module if needed:
RUST_LOG=info,namidb_storage=debugAt startup the info log records the resolved request deadlines, the auth
mode, the cache capacities, the memory ceiling and the listeners, which is
the quickest way to confirm a configuration took effect.