Embedded (Python)
The namidb wheel runs the whole engine in your process: no server, no
network hop. This page covers how to run it well. For every method and
parameter, see the Python API reference.
Open a namespace
import namidb
db = namidb.Client("file:///var/lib/namidb?ns=prod")The URI picks the backend and the namespace:
| URI | Use it for |
|---|---|
memory://demo | Tests and demos. Gone when the process exits. |
file:///var/lib/namidb?ns=prod | One machine, durable on local disk. |
s3://bucket/prefix?ns=prod®ion=us-east-1 | AWS S3 and S3-compatible services (R2, MinIO, Tigris…). |
gs://bucket?ns=prod | Google Cloud Storage. |
az://account/container?ns=prod | Azure Blob Storage. |
The full grammar and credential handling are in Storage URIs.
Opening a namespace that does not exist yet creates it. A plain Client
is the namespace’s writer: opening it claims the namespace and fences
any other process that was writing to it. To look at a namespace another
process is writing, open it read-only.
When is data durable?
There are two write paths, with different commit rules.
Cypher writes commit on return. CREATE, MERGE, SET, DELETE,
REMOVE and DDL are appended to the write-ahead log and committed with a
conditional manifest write before cypher() returns.
db.cypher("CREATE (:Person {name: 'Ada'})") # durable nowThe bulk API stages until commit(). upsert_node, upsert_edge,
put_nodes, merge_nodes, merge_edges, tombstone_* and the Parquet
loaders add to a pending batch. Nothing is durable until you call
commit().
import uuid
alice, bob = str(uuid.uuid4()), str(uuid.uuid4())db.put_nodes("Person", [ {"_id": alice, "name": "Alice"}, {"_id": bob, "name": "Bob"},])db.merge_edges("KNOWS", [{"src": alice, "dst": bob, "since": 2020}])db.commit() # WAL append + manifest commitCommitted data lives in the WAL and an in-memory table until it is
flushed into immutable data files. Call flush() periodically and at the
end of a load; the embedded client has no background flush timer.
db.flush()Disk space: flush, compaction and sweep
There is no background maintenance in embedded mode, so the client does it inline:
flush()also runs a compaction pass once any bucket holdscompaction_l0_triggerlevel-0 files (default8). Passcompaction_l0_trigger=0to turn that off and callcompact()yourself.- After a compaction that applies, the client sweeps: it deletes files
no manifest version references any more (merged-away files, WAL segments
already folded into data files, superseded manifests), provided they were
written more than
auto_sweep_min_age_secsago (default60). Passauto_sweep_min_age_secs=Noneto disable the automatic sweep. sweep(min_age_secs=60, dry_run=False)runs the same sweep on demand and returns what it removed (or would remove).
report = db.compact()print(report["applied"], report["l0_before"], report["l0_after"])
print(db.sweep(min_age_secs=0, dry_run=True))Compaction writes scratch files while it merges. For a file:// URI they go
to <root>/.namidb-spool next to the graph, unless you pass spool_dir= or
set NAMIDB_SPOOL_DIR. Other backends use NAMIDB_SPOOL_DIR, else
/var/tmp. Plan for roughly three times the compacted bytes on that volume
during a large pass. The spool directory is process-wide: the last client
opened decides it.
db = namidb.Client( "file:///data/graph?ns=case", spool_dir="/data/scratch", compaction_l0_trigger=8, auto_sweep_min_age_secs=60,)Read from another process
A namespace has exactly one writer. A second process that only reads should open it read-only, which never claims the namespace:
reader = namidb.Client("file:///var/lib/namidb?ns=prod", read_only=True)print(reader.cypher("MATCH (p:Person) RETURN count(p) AS n").rows())
# Pick up whatever the writer committed since:if reader.refresh(): print("new data")- A reader sees every commit up to the moment it was opened or last refreshed, including commits the writer has not flushed yet (it replays the unflushed WAL).
- It stays on that version until
refresh(), or for at most an hour: a read on an older version refreshes first. - Every mutating call (writes, DDL,
commit,flush,compact,sweep, loaders) raisesPermissionError. - The namespace must already exist.
- The reader keeps a small retention lease under
manifest/pins/so the writer’s sweep does not delete files it still needs, and renews it in the background.close(), or leaving awithblock, releases it immediately; otherwise it expires about 15 minutes after its last renewal.
with namidb.Client("file:///var/lib/namidb?ns=prod", read_only=True) as r: print(r.cypher("MATCH (n) RETURN count(n) AS n").first())Threads and asyncio
Every blocking call releases the GIL. Reads (cypher reads, lookup_node,
scan_*, edge lookups) run on the latest committed snapshot without waiting
for the writer lock, so several threads can query one Client at once and a
long read never blocks writes. Writes go one at a time.
For asyncio, use the coroutine twins acypher() and acompact(); they run
on the same runtime as the synchronous calls, so mixing both on one client is
fine.
import asyncioimport namidb
async def main(): db = namidb.Client("memory://acme") await db.acypher("CREATE (:Person {name: 'Alice'})") result = await db.acypher( "MATCH (p:Person {name: $name}) RETURN p.name AS name", params={"name": "Alice"}, ) print(result.rows()) # [{'name': 'Alice'}]
asyncio.run(main())Limit a query
cypher() and acypher() accept keyword-only timeout_ms and max_rows:
from namidb import QueryRowCapError, QueryTimeoutError
try: db.cypher("MATCH (a)-[]->(b) RETURN a, b", timeout_ms=5_000, max_rows=100_000)except QueryTimeoutError: # subclass of TimeoutError ...except QueryRowCapError: # subclass of RuntimeError ...timeout_ms counts from the call, including time a write waits for the
client’s lock. A write that times out is discarded, not committed.
max_rows fails a read as soon as any step of its plan produces more rows
than that; writes ignore it.
Differences from the server
The embedded client shares the engine with namidb-server, but a few
server features do not apply:
- No background flush or compaction timers (see above).
- No process memory governor (
NAMIDB_MEMORY_MAX_BYTESis server-only). The shared cache limits such asNAMIDB_CACHE_MAX_BYTESdo apply. - No auth, TLS, metrics endpoint or admin routes.
See Configuration for which environment variables the embedded engine reads.