Flush, compaction and sweep
The write path
- Commit. A write is appended to the write-ahead log (WAL) and published with a conditional manifest-pointer write. Once the call returns, the write is durable.
- In-memory table. Committed data also sits in an in-memory table (the memtable) that reads consult.
- Flush. The memtable is written out as immutable level-0 (L0) data files plus side-cars (indexes, filters, exact-record lookups). The WAL segments it covered become reclaimable.
- Compaction. L0 files are merged into larger, non-overlapping files in deeper levels, and vector and full-text index bodies are rebuilt or extended. Fewer, larger files mean fewer reads per query.
- Sweep. Objects that no live manifest version references any more (merged-away files, retired WAL segments, old manifest versions) are deleted once they are old enough.
The server runs steps 3 to 5 in the background. The embedded Python client
runs them inline from flush(); see
Embedded Python.
Flush
| Setting | Default | Effect |
|---|---|---|
--flush-interval | 30s | Flush on a timer. 0s disables the timer. |
--memtable-flush-bytes | 64 MiB | Flush as soon as a commit pushes the memtable past this size. 0 disables. |
--memtable-stall-bytes | 256 MiB | Above this size, commits are slowed until the flush catches up (backpressure against bulk loaders). 0 disables. |
Flush on demand:
curl -s -X POST localhost:8080/v0/admin/flush -H "Authorization: Bearer $ADMIN_TOKEN"# {"ssts_written":1,"bloom_sidecars_written":0,"manifest_version":2}The admin flush is serialised process-wide, ignores the HTTP request timeout, completes even if the client disconnects, and stays available when the memory ceiling is refusing new queries, so it doubles as a pressure-relief valve.
Compaction
| Setting | Default | Effect |
|---|---|---|
--compaction-interval | 300s | Background compaction and sweep timer. 0s disables all background maintenance. |
--compaction-l0-trigger | 8 | Compact as soon as a flush leaves a bucket with this many L0 files. 0 disables the reactive trigger. |
--write-stall-l0 | 24 | Above this many L0 files in a bucket, each commit is delayed by --write-stall-delay so writers cannot outrun compaction. 0 disables. |
--write-stall-delay | 50ms | The delay applied while stalled. |
NAMIDB_COMPACTION_BASE_BYTES | 8 MiB | Size budget of level 1; deeper levels grow by the ratio below. |
NAMIDB_COMPACTION_LEVEL_RATIO | 10 | Size ratio between levels (minimum 2). |
Background compaction is incremental: one bucket per tick, which is right for steady state. After a bulk load, drain L0 explicitly, otherwise the next load re-reads a tall L0 stack and runs much slower:
curl -s -X POST localhost:8080/v0/admin/compact -H "Authorization: Bearer $ADMIN_TOKEN"# {"passes":1,"l0_before":5,"l0_after":0,"source_ssts_removed":5,"new_ssts_written":1,# "manifest_version":11,"truncated":false}It blocks until the drain has landed (it can take minutes on a large backlog
and ignores the HTTP request timeout), shares one process-wide permit (a
second caller gets 503), and needs a read-write token. truncated: true
means it stopped at its 128-pass bound; call it again. In multi-tenant mode
use /:namespace/v0/admin/compact.
Compaction prepares its work without holding the writer lock, so reads and writes continue while it runs; it takes the lock only briefly to install the result.
Search index segments
Vector and full-text indexes are maintained as a base plus delta segments: each flush adds at most one delta, and compaction consolidates them.
| Variable | Default | Effect |
|---|---|---|
NAMIDB_SEARCH_LSM_MAX_SEGMENTS | 8 (2–32) | Live segments per index. A flush that needs another delta waits for compaction. |
NAMIDB_SEARCH_LSM_COMPACT_SEGMENTS | max(MAX − 2, 2) | Deltas merged in one routine pass. |
NAMIDB_SEARCH_LSM_BASE_COMPACT_BYTES | 8 GiB | Rebuild the base once accumulated delta bytes reach this. |
NAMIDB_SEARCH_LSM_BASE_STALE_PERCENT | 800 | Rebuild the base once delta mutations reach this percentage of the live corpus. |
NAMIDB_SEARCH_LSM_FORCE_BASE_COMPACTION | false | One-shot: rebuild the base if there is any delta debt. |
Sweep
| Setting | Default | Effect |
|---|---|---|
--sweep-min-age | 24h | Only delete unreferenced objects at least this old. |
--sweep-delete | true | false makes the sweep a dry run that only logs what it would free. |
Readers are protected exactly: nothing a live reader, a backup lease, a read-only client’s lease or the restore point window still needs is deleted. The age window is a second guard for an object uploaded just before its commit landed; lower it only if you understand that race on your object store.
One immutable manifest body and one pointer are written per commit, so a
long load accumulates hundreds of small versions. That is normal temporary
history, reclaimed by the sweep once it is older than --sweep-min-age
(as long as maintenance is enabled and --sweep-delete is true). With
--restore-point-retention the sweep keeps that history on purpose for the
retention window, so any instant in it can be backed up; see
Restore points.
Recommended routines
| Situation | Do |
|---|---|
| Steady traffic | Nothing; the defaults keep up. |
| Bulk load through Cypher or Bolt | Batch with UNWIND $rows, raise or remove --write-timeout for the load, then POST /v0/admin/compact. |
| Bulk load from Python | put_nodes / merge_edges / load_parquet_*, commit(), flush(), compact(), then sweep(). |
| Disk filling up on the spool | Give NAMIDB_SPOOL_DIR more room; see Memory and caches. |
| You run compaction elsewhere | Set --compaction-interval 0s and schedule admin/compact yourself. Without maintenance nothing is swept. |
Writer health
A writer can become degraded: fenced by another process claiming the
namespace, or poisoned by a failed commit or flush. Reads keep working from
the published snapshot, writes fail, and /v0/health returns 503 with
writer: "degraded" and a writer_error.
- With
--fence-policy steal(default) the writer reopens itself automatically and takes the namespace back. - With
--fence-policy yielda fenced writer stays degraded until the process (or, in multi-tenant mode, the namespace) is opened again; use it when an external controller decides which server owns each namespace. --writer-lock-timeout(30 s) bounds how long a write, DDL statement, admin flush or BoltBEGINwaits for the writer before returning 503.