Skip to content

DDR-004 — Portable storage (SQLite as the default backend)

Version: 0.1.0 | Date: 2026-05-09 | Status: Accepted (merged 2026-05-10, PR #5)

Context

Through DDR-001 → DDR-003, Engrama settled on a clean four-layer architecture: tools → skills → engine → protocols → backends. Protocols existed (GraphStore, VectorStore, EmbeddingProvider) but only one implementation of each was shipped: a Neo4j stack.

That meant every "give Engrama a try" path went through:

  1. Install Docker Desktop.
  2. Pull a ~500 MB Neo4j image.
  3. Create credentials, populate .env.
  4. Run docker compose up, wait for the JVM to boot.
  5. Run engrama init to apply Cypher constraints.

For someone evaluating an agent-memory framework on a laptop, that is a multi-step prerequisite. For someone shipping a CLI or library that embeds Engrama as a memory layer, requiring users to also run Docker is a non-starter.

In parallel, a wider problem appeared: the EmbeddingProvider layer only had one production implementation (Ollama). Anyone wanting to use OpenAI, Jina, LM Studio, vLLM or llama.cpp had to write a new provider, even though all of those services already speak the same HTTP protocol (the OpenAI embeddings API).

Decision

Make Engrama installable and useful with zero external services, by shipping a SQLite-based backend as the default and consolidating embedding providers behind a single OpenAI-compatible client.

Concretely

  1. Default backend = SQLite + sqlite-vec.
  2. GRAPH_BACKEND defaults to sqlite (was neo4j).
  3. Storage lives in ~/.engrama/engrama.db (override via ENGRAMA_DB_PATH).

  4. Graph data uses SQLite's relational tables; full-text search uses FTS5; vector search uses the sqlite-vec extension's vec0 virtual table. All three live in the same file.

  5. Neo4j moves to an opt-in extra.

  6. Install with uv sync --extra neo4j (or pip install engrama[neo4j] once Engrama ships on PyPI).
  7. The neo4j driver is no longer a base dependency.

  8. Behaviour is unchanged when GRAPH_BACKEND=neo4j and credentials are in .env.

  9. Single OpenAI-compatible embedding provider.

  10. engrama/embeddings/openai_compat.py covers OpenAI, Ollama (/v1/embeddings), LM Studio, vLLM, llama.cpp, Jina, and any future service that speaks the same shape.

  11. The legacy OllamaProvider stays as a thin convenience wrapper for users who prefer the older env vars.

  12. Factory unifies wiring across the stack.

  13. engrama/backends/__init__.py exposes create_stores() and create_async_stores().
  14. The CLI, SDK, and MCP server all dispatch through the factory.

  15. Switching backends is a one-variable change.

  16. The data model is shared.

  17. Same labels, same relationships, same faceted classification.
  18. SQLite encodes labels in a label column and relations in an edges table; Neo4j uses native node labels and edges.
  19. The contract test suites (tests/contracts/) parameterise over both backends and over both sync and async stores, so the wire-level contract is enforced.

Consequences

Positive

  • Onboarding time drops to seconds. git clone + uv sync → uv run engrama verify → uv run engrama search foo. No Docker, no JVM. (Engrama is not yet on PyPI; once published the same path collapses to pip install engrama && engrama verify.)
  • CI / tests run anywhere. No service to start. The full SQLite test suite (~76 tests) passes against an empty checkout with no .env.
  • Embedded use becomes practical. Engrama can ship inside another CLI, a desktop app, or an edge runtime without dragging Docker into the dependency chain.
  • More embedder choice for free. The OpenAI-compatible client unlocks LM Studio, vLLM, llama.cpp, Jina, OpenAI proper, and Ollama through the same code path.
  • Vault portability story strengthens. Combined with DDR-002 (every relation is persisted to vault frontmatter), a fresh SQLite install pointed at an existing Obsidian vault rebuilds the full graph by running engrama_sync_vault.

Negative

  • Single-writer ceiling on SQLite. WAL mode handles concurrent readers fine, but only one process can write at a time. Multi-agent setups should pick Neo4j.
  • No ad-hoc Cypher on SQLite. Pattern detection in reflect required translating each Cypher query into SQL; the run_pattern / run_cypher entry points raise NotImplementedError on SQLite.
  • Vector search scales differently. sqlite-vec is brute-force at this stage; comfortable up to ~100k vectors. Beyond that, Neo4j's HNSW index is the right tool.
  • Two backends to maintain. The contract suite is the mitigation: any divergence between the backends is caught at CI time. Three bugs that slipped through pre-merge testing are now permanently guarded against by the parameterised tests.

Risks (mitigated during implementation)

The PR that landed this change found and fixed three bugs that are worth recording, both as a track record and as guardrails for future work:

  1. Async store contract drift. SqliteAsyncStore was forwarding sync results unchanged via __getattr__, which leaked the legacy [{"n": ...}] shape to the MCP server (which expected the rich {"node": ..., "created": ...} shape that Neo4jAsyncStore returns). Fix: explicit method-by-method delegation with shape translation; tests/contracts/test_async_graphstore_contract.py parameterises over both async backends.
  2. Reflect overwriting approved Insights. engrama_reflect called merge_node with status="pending" on every detected pattern, silently undoing user approvals. Fix: a get_approved_titles method on every store layer plus a dismissed | approved filter in the reflect tool.
  3. Search dropping enrichment on pure-semantic hits. The hybrid scorer only copied summary / tags from fulltext results; nodes ranked solely by vector similarity surfaced empty. Fix: search_similar now projects summary, tags, confidence, and updated_at on both backends, and the scorer copies them on the vector path too.

Supersedes

This DDR supersedes the implicit decision in DDR-003 that Neo4j was the sole production backend. The protocol layer described in DDR-003 Phase A is unchanged — DDR-004 is what makes a non-Neo4j implementation a first-class member of that layer.

Open follow-ups (not blocking)

  • FTS5 query sanitisation on SQLite. The default tokenizer treats - as an operator, so queries like engrama-mcp-server miss the fulltext path. Wrapping the query in "…" or escaping hyphens would close the gap. The shape of results is correct; only the ranking differs. Tracked separately from this DDR.
  • First-class export / import tool. Cross-backend migration today is a hand-rolled SDK script. A engrama export / engrama import command would make the migration story symmetric.
  • README embedder matrix. The provider list (OpenAI / Ollama / LM Studio / vLLM / llama.cpp / Jina) deserves a worked example for each.

References

  • PR #5 — feat: portable storage — SQLite + sqlite-vec default backend
  • docs/portable-storage-spec.md — pre-implementation spec (local, gitignored).
  • backends.md — public-facing decision guide for newcomers.
  • architecture.md — updated layer diagram.