Graph Engineering: Why Structure Beats a Bigger Model

I keep hitting the same wall with RAG. The question that actually matters is never "what documents mention March." It is "why did sales drop in March." Vector search finds fragments that look like the question. It cannot return a chain of causes that live in five documents sharing no keywords with each other.

That is the ceiling. You hit it on the questions that change a decision. Semantic similarity finds text that looks alike. It does not find facts that connect.

I wrote about this from the agent side in loop engineering: the system around the model matters more than the prompt. Graph engineering is the same shift applied to retrieval. You stop stuffing similar chunks into context and start walking relationships.

Similarity Is the Wrong Index for "Why"

Standard RAG is a search engine that writes prose. User asks, embedder finds nearby chunks, model summarizes them. Fine for "what is our refund policy." Useless for "what caused the refund spike."

The path you actually needed looks like this:

sales dropped
  ← release delay
    ← supplier problem
      ← warehouse failure
        ← negative reviews
          ← conversion down 23%

No embedding upgrade gets you there. The warehouse failure document does not mention sales. The review dump does not mention the supplier. Similarity cannot hop. A graph can.

Microsoft's GraphRAG research is the clearest split I have seen: local questions (facts about this entity) versus global questions (patterns across the corpus). Chunk RAG handles the first okay and the second almost never. Graph retrieval is built for both.

Store Triples, Query Paths

Graph engineering is not a fancier vector store. You store facts as triples, then query the relationships:

Subject → Relation → Object

A vector database stores "this paragraph is about supply chains." A knowledge graph stores "this warehouse failure caused that supplier delay." When you ask a question, you are not asking for similar text. You are asking: walk the path from A to B and show every link.

flowchart LR
    A[Warehouse failure] -->|triggered| B[Supplier delay]
    B -->|caused| C[Release slip]
    C -->|led_to| D[Negative reviews]
    D -->|cut| E[Conversion -23%]

That structural difference is the whole product. Everything else — Neo4j versus Postgres, Cypher versus SPARQL, which model you call — is implementation.

The Graph Beats the Model Size

There is a paper comparing 26 open-source models on knowledge graph engineering tasks. The result is the one I wish I had believed two years earlier: a smaller model with a good graph beats a larger model with a bad one. Consistently.

This is the same conclusion GraphRAG reached, and the same one I keep seeing in agent graphs and LangGraph-style systems. Structure beats scale. Most teams still respond to a bad answer by upgrading the model. The cheaper fix is the retrieval graph.

You will see "85% lower cost, 18% better accuracy" attached to this idea everywhere. Treat those as directional, not a promise. They come from specific document sets and specific baselines — including "cheaper than loading structured files straight into context," which is not the same as "cheaper than your current RAG." Measure on five of your own questions before you rewrite the stack.

Three Integration Modes — Build the Third

The research literature (Unifying Large Language Models and Knowledge Graphs) describes three ways to combine a model and a graph:

1. KG-enhanced LLM. The graph feeds facts. The model answers better. One direction.
2. LLM-augmented KG. The model extracts, cleans, and expands the graph. Also one direction.
3. Synergized. Both. The model writes new facts into the graph. The graph gives structured context for the next question.

Mode 3 is the one worth building. Modes 1 and 2 are parts of it. The practical consequence is the same compounding I want from loops: each answer adds structure the next answer can use. The system does not just reply. It gets denser.

Eight Layers, One Closed Loop

I would not start with a framework. I would start with eight jobs, each narrow enough to verify.

flowchart TB
    I[1. Ingestion] --> X[2. Extraction]
    X --> R[3. Resolution]
    R --> S[4. Storage]
    S --> T[5. Retrieval]
    T --> A[6. Agent]
    A --> V[7. Verification]
    V --> U[8. Update]
    U --> X

Ingestion is raw material: PDFs, pages, databases, Slack, Notion. No cleverness yet.

Extraction is the model pulling entities and relationships as JSON — canonical name, type, relation, evidence quote, confidence. If it cannot quote the source, the relation does not exist.

Resolution is the layer everyone skips. "Moonshot AI," "Moonshot," "Beijing Moonshot," and "月之暗面" are the same node or your graph is a pile of near-duplicates and every query comes back half-empty. Resolve before insert. Retrofitting this onto a polluted graph is miserable.

Storage can be Neo4j, Memgraph, Neptune, or Postgres with a graph extension. Neo4j is still the least painful first week: docs, visualization, Cypher you can read.

Retrieval is not one method. Five, together: vector search for fuzzy match, entity lookup for exact nodes, path search for connections, community search for patterns, temporal filters for "what was true when."

Agent plans, writes Cypher against the real schema, reads the subgraph, and searches again when a hop is missing.

Verification checks that the conclusion is supported by retrieved paths, flags contradictions, and states confidence. Skip this and you have built a very expensive hallucination machine.

Update writes new facts, timestamps superseded ones, and flags conflicts instead of silently overwriting. Then you go back to extraction. That close is what makes the system compound.

Five Prompts, Each With One Job

Graph engineering does not replace prompting. It gives each prompt a job you can check.

• Extract with a quote, or skip the edge.
• Resolve same vs related vs unrelated. Never merge without evidence.
• Translate the question into Cypher using only labels that exist. Pass the literal schema every time. Invented labels are why queries return everything or nothing.
• Answer from retrieved paths only. Do not infer causation from co-occurrence. caused_by is a claim. mentioned_alongside is not.
• Maintain by classifying each fact as new, duplicate, contradiction, update, or uncertain. Entropy is guaranteed. Treat this like linting.

Why Long Context Matters Here

A graph query does not return a paragraph. It returns a subgraph, evidence chains, and the paths between them. Truncate that to fit 128K and you destroyed the structure you just built.

This is why long-context models are the right engine for this architecture, not because they win every benchmark. Kimi K3 is the example I keep seeing: a 1,048,576-token window, plus hybrid attention aimed at making million-token decode affordable. Moonshot's own write-up is honest that overall scores still trail the top general models. That is fine. Graph work rewards context and long-sequence cost more than a couple of leaderboard points.

Even a million tokens is session memory. After the call, it is gone. The graph is what persists.

Long context  = working surface for this query
Knowledge graph = structured memory between queries

You still cap traversal. Most real questions need two or three hops, not six. If token cost climbs, retrieval is greedy. The model is not the leak.

A Week I Would Actually Run

• Day 1: Neo4j locally. Five Cypher queries by hand until the syntax stops feeling foreign.
• Day 2: One document set you care about. Run extraction. Stare at the JSON. This is where you learn the prompt is too loose.
• Day 3: Load triples. Entity lookup plus one hop. Test against a question your current RAG already fails.
• Day 4: Path search. Ask a three-hop "why." This is the wow, if there is going to be one.
• Day 5: Connect a coding agent over MCP. Let it query, find a gap, write a fact back. First closed loop.
• Days 6–7: Measure accuracy, tokens per query, latency against the old RAG. Your five questions beat anyone's published percentage.

What Breaks First

Duplicate entities. OpenAI / Open AI / OpenAI Inc. Fix with resolution before insert.

Invented relations. Co-mention is not a partnership. Demand the quote.

Cypher against a fantasy schema. Pass real labels. Validate.

Confident wrong answers. Adjacent in the graph is not causal. Verification layer plus the causation rule.

Trust decay. Old facts and new facts with no timestamps. Maintenance pass on a schedule.

I already treat this as a sibling of loop engineering. Loops persist work. Graphs persist facts and how they connect. Both replace the habit of hoping the next, larger model will paper over a missing system.

Build the Graph

Standard RAG gives you similar documents in nicer prose. Graph engineering answers "why" and "how are these connected," shows you the gaps because missing edges are visible, and gets denser every time you feed it a source.

The tradeoff is real. You are building extraction, resolution, storage, retrieval, verification, and maintenance instead of calling an embedding API. That is a week. It is also the week that separates a demo from a system.

When the answers are weak, the instinct is still to buy a bigger model. The evidence I trust says the opposite. Fix the retrieval structure. The model is the easy part.