Skip to content

DDR-008 — Reflect curates the graph, and Insights live inside it

Version: 0.1.0 | Date: 2026-09-24 | Status: Accepted

Context

engrama_reflect detects cross-entity patterns and writes Insight nodes for human review. It has three problems.

  • Insights are orphans by construction. Reflect creates each Insight with merge_node and never links it to the nodes it talks about. In a graph that runs reflect regularly, Insights make up most of the degree-0 nodes. They inflate every "orphan" count, and they can't be reached from the entities they describe.
  • The detectors are noisy where they fire and blind where it matters.
  • shared_technology emits one Insight per pair of entities sharing a technology (N² per technology), without excluding superseded or archived nodes. The review queue fills with near-identical Insights that end up dismissed by hand.
  • under_connected filters with n.status <> 'archived', which in Cypher's three-valued logic drops every node that has no status. It also returns only the 15 newest nodes, so the backlog of degree-0 nodes is never surfaced. The SQLite backend uses COALESCE and behaves differently.
  • There is no detector for the problems that actually degrade the graph: duplicate entities, hub stubs, tags without edges, archived bridges, label conflicts.

Decision

Reflect becomes the graph's curator. Its detectors are correct and aggregated, it finds structural problems as well as domain patterns, every Insight is linked to its evidence, and an approved curation Insight can carry the non-destructive action that fixes the problem it reports. Merging nodes is deliberately left out of this DDR.

Concretely

  1. Detector correctness.
  2. A single live_filter(var) helper with null-safe coalesce(n.status, 'active') NOT IN ['archived', 'superseded'] is used by every detector on both backends.
  3. under_connected reports the full count plus a sample instead of LIMIT 15.
  4. A parity test runs every detector on SQLite and Neo4j with the same fixture.
  5. Aggregation. shared_technology emits one Insight per technology with ≥ 3 live users, listing the members, instead of one per pair. Every detector has a per-run budget, so one run can't flood the queue.
  6. Curation detectors.
  7. orphans: degree 0, excluding system Insights (DDR-006 item 7). Hand-written Insights count as domain nodes here and in every other detector, which today skip :Insight altogether.
  8. hub_stubs: stub with degree ≥ 3.
  9. merge_candidates: resolver pairs above the threshold (DDR-006). Report only: resolving a candidate stays a human decision.
  10. tag_without_edge: a tag that resolves to an anchor that isn't linked.
  11. archived_bridges: archived nodes whose neighbours are live.
  12. label_conflicts.
  13. archive_candidates: low read-time recency, low degree, no live neighbours (DDR-007).
  14. Insights are linked. Each Insight gets (:Insight)-[:ABOUT]->(n) for every entity its detector returned. ABOUT is a core relation type, independent of the profile. Orphan metrics exclude system Insights by definition, and engrama_context on an entity shows the Insights about it.
  15. Actionable approvals. A curation Insight stores a machine-readable proposed_action: link, enrich or archive. None of them deletes data (archive is reversible through revival). engrama_approve_insight(..., apply=true) executes it through the same tools a human would use, inside the caller's scope. Without apply, approval behaves as today.
  16. Insight lifecycle. A regenerated Insight refreshes its body, confidence and ABOUT edges unless a human already judged it (DDR-007 item 5). The review queue (engrama_surface_insights, engrama_approve_insight) only handles system Insights, so hand-written Insights never show up as pending.

Consequences

  • Positive:
  • Orphan counts measure the domain graph again.
  • Reviewing Insights becomes a curation loop that fixes the problems it reports.
  • Noise drops by an order of magnitude for the technology detector.
  • Negative:
  • Duplicates are surfaced but not resolved: until a merge operation exists, fixing one means re-pointing edges by hand.
  • More detectors means a slower reflect. Reflect runs on demand, and each detector keeps a budget.
  • Neutral:
  • Existing Insights keep working. They gain ABOUT edges the next time reflect regenerates them.

Open follow-ups (not blocking)

  • A merge operation (engrama_merge) that folds node B into node A: it redirects edges, takes the union of properties, keeps B's key as an alias and records B's engrama_id. It is destructive, so it gets its own DDR, covering undo, auditing and vault notes, before any implementation.
  • Letting an EntityDecider (DDR-006) score merge candidates instead of a fixed threshold.
  • A engrama health report (read-only, scoped) that exposes the same metrics as the curation detectors, for dashboards and evaluation harnesses.

References

  • DDR-006: entity resolver, stub lifecycle, system and hand-written Insights.
  • DDR-007: read-time recency and explicit archiving.