Skip to content

Troubleshooting

Start with the server log (startup lines record the effective configuration) and GET /v0/health. Error responses carry a code; see the error taxonomy.

The server does not start

MessageFix
no auth configured: set --auth-token / NAMIDB_AUTH_TOKEN or --auth-tokens-file, or pass --no-auth …Configure a token or tokens file. --no-auth only for local development.
invalid --store: failed to open backend for 'file:///var/lib/namidb?ns=prod': … Permission denied (os error 13)The container user (uid 65532) cannot write the directory. Use a named volume, or chown -R 65532:65532 the bind-mounted host directory.
URI '…' is missing required parameter: nsAdd ?ns=<namespace> (not needed with --multi-tenant).
bad namespace '…': … must start with [a-z0-9]Use a valid name: lowercase letters, digits, -, _.
NAMIDB_MEMORY_MAX_BYTES=auto requires a finite cgroup memory limit …Give the container a memory limit, or set an exact byte count.
NAMIDB_CACHE_MAX_BYTES must be an exact non-negative byte count …Byte variables take plain integers (1073741824), not 1GB.
An error about NAMIDB_RANGE_CACHE_NAMESPACEWith NAMIDB_NVME_CACHE_PATH, also set NAMIDB_NVME_CACHE_MAX_BYTES and a stable NAMIDB_RANGE_CACHE_NAMESPACE.
NAMIDB_RESTORE_POINT_RETENTION needs the background maintenance loop: set NAMIDB_COMPACTION_INTERVAL above zero, or the retention to 0Restore points need maintenance: keep --compaction-interval above zero, or turn retention off.
the HTTP request ceiling is at or below a query budget … (warning)Raise --http-request-timeout above --query-timeout and --write-timeout, or leave it unset.

Authentication

SymptomCause
401 missing or invalid bearer tokenNo Authorization: Bearer … header, or an unknown token.
401 … or token not scoped to this namespaceMulti-tenant: the token’s namespaces list does not include the requested namespace.
403 this token is read-only; write queries are forbiddenUse a read-write token for writes, DDL and admin endpoints.
A revoked token still works over BoltSingle-tenant Bolt sessions keep their login principal; close the connections.
A new token is rejected for a few secondsThe tokens file reloads every --auth-tokens-reload-interval (10 s).

/v0/health returns 503

The body says why:

  • writer: "degraded" with a writer_error. Usually another process opened the namespace for writing and fenced this one: a second server, a CLI namidb run --store …, an MCP server or a writable Python client on the same store. The epoch in /v0/health increases each time the namespace is claimed. Make sure exactly one process writes each namespace; open extra Python readers with read_only=True. With the default --fence-policy steal the server reclaims the namespace on its own; with yield it stays degraded until restarted.
  • memory_pressure: true. The process reached --memory-max-bytes. See Memory and caches.

Requests fail with 503

Body or logCauseFix
Writer lock timeoutA write waited more than --writer-lock-timeout (30 s), typically behind a long Bolt write transaction or a long bulk statementKeep write transactions short; check /v0/admin/queries.
Too many concurrent queriesThe read admission gate was full for NAMIDB_QUERY_ADMISSION_WAIT_MSRetry with backoff, or raise NAMIDB_MAX_CONCURRENT_QUERIES if the machine has headroom.
memory_pressureThe process is at --memory-max-bytes when the statement arrives, or, while it runs, the shared query pool (75% of the ceiling) is full or the process reaches the ceilingRetry with backoff. If it keeps happening: run fewer large queries at once, make sure Bolt clients consume or close their results, lower cache sizes, raise the container limit together with the ceiling, or flush (POST /v0/admin/flush). See the per-query budget.
search_index_cache_capacity, search_workspace_capacitySearch memory pools are too small for the requestRaise NAMIDB_SEARCH_INDEX_CACHE_MAX_BYTES / NAMIDB_SEARCH_WORKSPACE_MAX_BYTES.
another backup or restore is already runningOne backup or restore runs per processRetry later.

A backup with as_of can also fail with 404 restore_point_unavailable (the instant is older than the restore point window, whose bounds the body carries), 400 as_of_in_future (later than the server’s clock) or 400 invalid_as_of.

Neo4j drivers retry these automatically (they arrive as Neo.TransientError.*).

Timeouts, row caps and large writes

  • 504 timeout on a read. The statement ran past --query-timeout. Check its plan with EXPLAIN, add an index or constraint on the filtered key, or raise the budget. Since 3.3.0 every operator checks the deadline, so a cross product or a large sort that used to run long past the budget now stops at it.
  • 504 on a bulk write. Writes have --write-timeout (60 s). Split the load into UNWIND $rows batches of a few thousand rows, or set NAMIDB_WRITE_TIMEOUT=0s for the duration of the load. A timed-out write commits nothing.
  • 413 row_cap. An operator would materialise more than --query-row-cap rows (1,000,000). Usually a missing join condition or an unbounded variable-length pattern; otherwise page the result.
  • 413 query_memory_limit. The statement needs more working memory than --query-memory-max-bytes allows (by default half of --memory-max-bytes). It fails the same way on every run, so retrying does not help, and a write that hits it commits nothing. Return fewer rows (LIMIT, pagination with ORDER BY … SKIP … LIMIT or a key range), return fewer or narrower columns, and aggregate in the query instead of returning or collect()ing raw rows. Raise the budget only if the query is legitimate and the ceiling and container limit have room; see Memory.
  • 413 on the request itself. The HTTP body exceeds 2 MiB; send smaller batches.
  • 408. The whole request exceeded --http-request-timeout.

Slow queries

  1. Run EXPLAIN on it. A NodeScan with predicates=[…] on a large label means no index serves the filter; declare one (Schema). An indexed lookup shows as NodeByPropertyValue with a # route: line.
  2. Anchor patterns on a selective node (a unique key or _id) and bound variable-length paths (*1..4, not *).
  3. After a bulk load, run POST /v0/admin/compact: a deep level 0 makes every read touch many files. namidb_compaction_l0_backlog_ssts shows the depth.
  4. Look at the slow-query log (--slow-query-threshold, default 1 s) and the namidb_query_duration_seconds histogram.

Search surprises

SymptomCause
CREATE VECTOR INDEX fails with “built without the vector-index feature”The binary lacks the feature: the CLI and MCP archives, or a server built from source without --features vector-index,text-index. Use the image, the release server, the Python wheel, or rebuild.
A write fails with embedding has dim N but vector index … declares MThe vector index enforces its dimension. Fix the embedding or the index.
A node never appears in cosine KNNIts vector has zero magnitude; cosine_similarity is NULL for it.
A vector search returns fewer than kFewer than k nodes match the label and filter.
does not recognise the option …A typo in a procedure option; the message lists the valid ones.
CALL must be the first clause in a queryProcedure calls (algo.*, search.*, db.*) must start the query; pass inputs as parameters.

Bolt connections

ErrorFix
Failed to obtain connection towards 'WRITE' serverUse bolt:// (or bolt+s://), not neo4j://: routing tables are not served.
UnsupportedServerProduct / “does not identify as a genuine Neo4j”A plaintext bolt:// client against a TLS listener, or NAMIDB_BOLT_SERVER_AGENT set to a non-Neo4j value for a strict client.
Neo.ClientError.Security.UnauthorizedWrong token. The user name is ignored; the token is the password.
Neo.ClientError.Security.ForbiddenMulti-tenant: the token is not scoped to the session’s database.
Neo.ClientError.Request.Invalid and the connection closesA message exceeded --bolt-max-message-bytes; send smaller batches.

Data looks wrong

SymptomExplanation
Dates come back as numbers like "20758" over HTTPHTTP serialises DATE as days since 1970-01-01. Return toString(d).
A second CREATE (a)-[:T]->(b) did not add a relationshipRelationships are unique per (type, start, end); the second replaced the first.
keys(n) lists a property that is NULLSET n.p = null stores an explicit null; use REMOVE n.p.
Values returned after a FOREACH look unchangedRETURN after FOREACH shows the row as it was before; re-read in a new query.
A new process sees no data in memory://Each memory:// store is private to the Client (or process) that opened it.

Disk keeps growing

  • Recent history is normal. Every commit writes a small manifest and pointer; the sweep reclaims them only after --sweep-min-age (24 h).
  • Maintenance off? With --compaction-interval 0s nothing compacts or sweeps; with --sweep-delete false the sweep only logs.
  • Restore point retention keeps what the namespace writes during the window on purpose. The sweep logs restore_window_retained and restore_window_bytes; shorten --restore-point-retention or set it to 0 if the history costs more than it is worth.
  • Embedded Python sweeps after applied compactions only; call sweep() after large loads, and sweep(min_age_secs=0) if no other process reads the namespace.
  • Incomplete multipart uploads on S3-compatible stores are kept by the bucket; add a lifecycle rule that aborts them.
  • The spool (NAMIDB_SPOOL_DIR) needs about three times the compacted node data during a large compaction, then empties.

Getting help

Collect the server version (/v0/version), the startup log, the failing statement with EXPLAIN, and the /v0/health body. Report engine bugs at github.com/namidb/namidb/issues; write to info@namidb.com for commercial support.