Skip to content

Object stores

On an object store the bucket is the database: kill the process, start one on another machine with the same URI, and the graph is there.

Requirement: PUT-if-absent

NamiDB needs one conditional-write capability from the store: create-if-absent (If-None-Match: * on S3). It is the compare-and-swap behind every manifest pointer and WAL commit. Conditional overwrite (If-Match) is not required for writes, and an eventually consistent LIST is tolerated.

AWS S3, Google Cloud Storage, Azure Blob Storage, Cloudflare R2, Tigris and current MinIO releases support it. An “S3-compatible” service that ignores If-None-Match: * cannot host NamiDB safely: two writers could both believe they committed. Check a new service before trusting it:

import boto3
s3 = boto3.client("s3", endpoint_url="https://<endpoint>")
s3.put_object(Bucket="my-bucket", Key="probe", Body=b"1", IfNoneMatch="*")
try:
s3.put_object(Bucket="my-bucket", Key="probe", Body=b"2", IfNoneMatch="*")
print("UNSAFE: the second create succeeded")
except s3.exceptions.ClientError as e:
print("ok:", e.response["Error"]["Code"]) # ok: PreconditionFailed
s3.delete_object(Bucket="my-bucket", Key="probe")

AWS S3

db = namidb.Client("s3://my-bucket/graphs?ns=prod&region=us-west-2")
Terminal window
export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=… # or use an instance/task/pod role
namidb-server --store "s3://my-bucket/graphs?ns=prod&region=us-west-2" \
--auth-token "$NAMIDB_AUTH_TOKEN"

Without a /prefix, objects go under tenants/<ns>/ in the bucket.

Cloudflare R2

db = namidb.Client(
"s3://my-bucket?ns=prod"
"&endpoint=https://<ACCOUNT_ID>.r2.cloudflarestorage.com&region=auto"
)

AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY hold the R2 API token’s key id and secret.

Tigris and other S3-compatible services

Point endpoint= at the service and pass its keys as the AWS variables:

s3://my-bucket?ns=prod&endpoint=https://<service-endpoint>&region=auto

Add allow_http=true only for plain-HTTP endpoints on a trusted network (LocalStack, a MinIO in the same Compose project).

Google Cloud Storage

Terminal window
export GOOGLE_APPLICATION_CREDENTIALS=/etc/namidb/gcs-key.json
db = namidb.Client("gs://my-bucket/graphs?ns=prod")
# or name the key file in the URI:
db = namidb.Client("gs://my-bucket?ns=prod&service_account=/etc/namidb/gcs-key.json")

Azure Blob Storage

Terminal window
export AZURE_STORAGE_ACCOUNT_NAME=myaccount
export AZURE_STORAGE_ACCOUNT_KEY=…
db = namidb.Client("az://myaccount/mycontainer?ns=prod")
# Azurite emulator:
db = namidb.Client("az://devstoreaccount1/mycontainer?ns=dev&use_emulator=true")

LocalStack

Terminal window
docker run --rm -p 127.0.0.1:4566:4566 -e SERVICES=s3 localstack/localstack
aws --endpoint-url=http://localhost:4566 s3 mb s3://namidb-dev
export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test
db = namidb.Client(
"s3://namidb-dev?ns=local&endpoint=http://localhost:4566&allow_http=true&region=us-east-1"
)

Bucket settings

  • Abort incomplete multipart uploads. Add a lifecycle rule that aborts incomplete multipart uploads after a short window (a day is typical). NamiDB aborts the uploads it can, but a process killed mid-upload leaves parts the bucket keeps billing for.
  • Do not expire or transition NamiDB objects with lifecycle rules. The engine deletes what it no longer needs (see Maintenance); an external rule that removes or archives live objects corrupts the namespace.
  • Versioning is optional. NamiDB never overwrites its data files; bucket versioning only adds a safety net (and cost) for deletes.
  • Use a separate bucket or prefix for backups, with its own credentials if possible. See Backup and restore.
  • Keep the server close to the bucket (same region). Each commit is a few round trips to the store, so commit latency follows the store’s latency; group commit amortises it under concurrent writes.

Read cache

On an object store, reads fetch byte ranges of immutable files and keep them in a bounded RAM cache, optionally backed by a persistent local NVMe cache that survives restarts. Both are only caches; see Memory and caches.

Timeouts and retries

The object-store client retries transient errors. Tune it with NAMIDB_STORE_REQUEST_TIMEOUT, NAMIDB_STORE_RETRY_TIMEOUT and NAMIDB_STORE_MAX_RETRIES; see Storage URIs.