Skip to content

Embedded (Python)

The namidb wheel runs the whole engine in your process: no server, no network hop. This page covers how to run it well. For every method and parameter, see the Python API reference.

Open a namespace

import namidb
db = namidb.Client("file:///var/lib/namidb?ns=prod")

The URI picks the backend and the namespace:

URIUse it for
memory://demoTests and demos. Gone when the process exits.
file:///var/lib/namidb?ns=prodOne machine, durable on local disk.
s3://bucket/prefix?ns=prod&region=us-east-1AWS S3 and S3-compatible services (R2, MinIO, Tigris…).
gs://bucket?ns=prodGoogle Cloud Storage.
az://account/container?ns=prodAzure Blob Storage.

The full grammar and credential handling are in Storage URIs.

Opening a namespace that does not exist yet creates it. A plain Client is the namespace’s writer: opening it claims the namespace and fences any other process that was writing to it. To look at a namespace another process is writing, open it read-only.

When is data durable?

There are two write paths, with different commit rules.

Cypher writes commit on return. CREATE, MERGE, SET, DELETE, REMOVE and DDL are appended to the write-ahead log and committed with a conditional manifest write before cypher() returns.

db.cypher("CREATE (:Person {name: 'Ada'})") # durable now

The bulk API stages until commit(). upsert_node, upsert_edge, put_nodes, merge_nodes, merge_edges, tombstone_* and the Parquet loaders add to a pending batch. Nothing is durable until you call commit().

import uuid
alice, bob = str(uuid.uuid4()), str(uuid.uuid4())
db.put_nodes("Person", [
{"_id": alice, "name": "Alice"},
{"_id": bob, "name": "Bob"},
])
db.merge_edges("KNOWS", [{"src": alice, "dst": bob, "since": 2020}])
db.commit() # WAL append + manifest commit

Committed data lives in the WAL and an in-memory table until it is flushed into immutable data files. Call flush() periodically and at the end of a load; the embedded client has no background flush timer.

db.flush()

Disk space: flush, compaction and sweep

There is no background maintenance in embedded mode, so the client does it inline:

  • flush() also runs a compaction pass once any bucket holds compaction_l0_trigger level-0 files (default 8). Pass compaction_l0_trigger=0 to turn that off and call compact() yourself.
  • After a compaction that applies, the client sweeps: it deletes files no manifest version references any more (merged-away files, WAL segments already folded into data files, superseded manifests), provided they were written more than auto_sweep_min_age_secs ago (default 60). Pass auto_sweep_min_age_secs=None to disable the automatic sweep.
  • sweep(min_age_secs=60, dry_run=False) runs the same sweep on demand and returns what it removed (or would remove).
report = db.compact()
print(report["applied"], report["l0_before"], report["l0_after"])
print(db.sweep(min_age_secs=0, dry_run=True))

Compaction writes scratch files while it merges. For a file:// URI they go to <root>/.namidb-spool next to the graph, unless you pass spool_dir= or set NAMIDB_SPOOL_DIR. Other backends use NAMIDB_SPOOL_DIR, else /var/tmp. Plan for roughly three times the compacted bytes on that volume during a large pass. The spool directory is process-wide: the last client opened decides it.

db = namidb.Client(
"file:///data/graph?ns=case",
spool_dir="/data/scratch",
compaction_l0_trigger=8,
auto_sweep_min_age_secs=60,
)

Read from another process

A namespace has exactly one writer. A second process that only reads should open it read-only, which never claims the namespace:

reader = namidb.Client("file:///var/lib/namidb?ns=prod", read_only=True)
print(reader.cypher("MATCH (p:Person) RETURN count(p) AS n").rows())
# Pick up whatever the writer committed since:
if reader.refresh():
print("new data")
  • A reader sees every commit up to the moment it was opened or last refreshed, including commits the writer has not flushed yet (it replays the unflushed WAL).
  • It stays on that version until refresh(), or for at most an hour: a read on an older version refreshes first.
  • Every mutating call (writes, DDL, commit, flush, compact, sweep, loaders) raises PermissionError.
  • The namespace must already exist.
  • The reader keeps a small retention lease under manifest/pins/ so the writer’s sweep does not delete files it still needs, and renews it in the background. close(), or leaving a with block, releases it immediately; otherwise it expires about 15 minutes after its last renewal.
with namidb.Client("file:///var/lib/namidb?ns=prod", read_only=True) as r:
print(r.cypher("MATCH (n) RETURN count(n) AS n").first())

Threads and asyncio

Every blocking call releases the GIL. Reads (cypher reads, lookup_node, scan_*, edge lookups) run on the latest committed snapshot without waiting for the writer lock, so several threads can query one Client at once and a long read never blocks writes. Writes go one at a time.

For asyncio, use the coroutine twins acypher() and acompact(); they run on the same runtime as the synchronous calls, so mixing both on one client is fine.

import asyncio
import namidb
async def main():
db = namidb.Client("memory://acme")
await db.acypher("CREATE (:Person {name: 'Alice'})")
result = await db.acypher(
"MATCH (p:Person {name: $name}) RETURN p.name AS name",
params={"name": "Alice"},
)
print(result.rows()) # [{'name': 'Alice'}]
asyncio.run(main())

Limit a query

cypher() and acypher() accept keyword-only timeout_ms and max_rows:

from namidb import QueryRowCapError, QueryTimeoutError
try:
db.cypher("MATCH (a)-[]->(b) RETURN a, b", timeout_ms=5_000, max_rows=100_000)
except QueryTimeoutError: # subclass of TimeoutError
...
except QueryRowCapError: # subclass of RuntimeError
...

timeout_ms counts from the call, including time a write waits for the client’s lock. A write that times out is discarded, not committed. max_rows fails a read as soon as any step of its plan produces more rows than that; writes ignore it.

Differences from the server

The embedded client shares the engine with namidb-server, but a few server features do not apply:

  • No background flush or compaction timers (see above).
  • No process memory governor (NAMIDB_MEMORY_MAX_BYTES is server-only). The shared cache limits such as NAMIDB_CACHE_MAX_BYTES do apply.
  • No auth, TLS, metrics endpoint or admin routes.

See Configuration for which environment variables the embedded engine reads.