In Part 1, I argued that every RAG variant is really a page replacement policy in disguise, and that the real architectural question isn't "which RAG" but "what's your memory hierarchy." That post ended with a tease: knowledge graphs might give you an approximation of Belady's optimal eviction policy, almost for free.
This is the post where I make that case.
The Eviction Problem Nobody Talks About
Most RAG discussions focus on retrieval. What chunks do I pull in? How do I rank them? How do I make sure they're relevant?
Almost nobody talks about eviction. What leaves the context window when new information needs to come in? When the model is three turns deep in a reasoning chain and the context is filling up, who decides what to drop?
In practice, most systems don't decide at all. They truncate the oldest turns. They drop the lowest-scoring chunks. They shove everything in and pray the model pays attention to the right parts. These are all versions of FIFO, the simplest and least effective eviction strategy in the OS literature. It was understood to be inadequate in 1966, and we're still using it in production AI systems sixty years later.
Belady proved that the optimal approach is to evict the item that won't be needed for the longest time in the future. You can't implement this directly because it requires an oracle. But you can approximate it. And the quality of that approximation determines how well your system performs under pressure, when the context window is full and every token counts.
LRU is a decent approximation in operating systems because temporal locality holds: pages accessed recently are likely to be accessed again soon. But temporal locality is a weak signal for knowledge retrieval. The fact that a chunk was loaded three turns ago says nothing about whether it's relevant to the current reasoning thread. A financial report from Q2 2024 might be critical for answering a question about year-over-year trends even though the model hasn't "touched" it in several turns.
What you actually want is a relevance signal that tracks semantic proximity to the current reasoning thread, not recency.
Knowledge graphs give you exactly that.
Graph Distance as Semantic Proximity
A knowledge graph organizes information as entities (nodes) connected by relationships (edges). "Company X" connects to "Q2 Revenue" via a "reported" edge. "Q2 Revenue" connects to "Gross Margin" via a "derived from" edge. "Gross Margin" connects to "Cost of Goods Sold" via a "component of" edge. The structure encodes how concepts relate to each other.
Graph distance, the number of hops between two nodes, is a natural semantic proximity metric. If the model is currently reasoning about Company X's revenue, then "Gross Margin" (two hops away) is likely more relevant than "CEO's alma mater" (five hops away, through a different branch of the graph entirely).
This is the key insight: graph distance gives you an approximation of Belady's without training a separate model.
Instead of asking "when was this chunk last accessed?" (LRU) or "how similar is this embedding to the query?" (vector RAG), you ask "how many hops is this concept from the node the model is currently working on?" Nodes close in graph space are likely to be needed soon. Nodes far away are safe to evict.
It's not a perfect oracle. Nothing is. But it's a structurally informed approximation that uses the actual relationships between concepts rather than embedding geometry or access timestamps. And it comes for free if you've already built the graph.
Why This Beats Vector Similarity
Vector similarity (cosine distance between embeddings) is the default relevance signal in vanilla RAG. It works for simple retrieval: "find me chunks that are about the same topic." But it breaks down for eviction decisions in at least two ways.
First, embeddings collapse structure. Two chunks about "revenue" and "cost of goods sold" might have moderate cosine similarity because they share financial vocabulary, but the relationship between them (one is a component of the other) is invisible in embedding space. In a knowledge graph, that relationship is an explicit edge. The graph knows these concepts are directly connected. Embeddings just know they're vaguely related.
Second, embeddings don't encode directionality. If you're reasoning about revenue, cost of goods sold is one hop away but the CEO's university is also "related" in embedding space because it appears in the same document. The graph distinguishes these cleanly. Embedding similarity doesn't.
Graph distance isn't always better. For a corpus with no meaningful relational structure (a collection of independent blog posts, say) there's nothing for the graph to encode. Vector similarity is fine there. The graph approach pays off when your knowledge has internal structure, when concepts build on each other, when the path from A to B matters, not just their surface similarity.
Financial data, medical records, legal corpora, codebases, engineering documentation. These are all domains where relationships between concepts carry real information. And these are exactly the domains where RAG pipelines struggle most, because the retrieval is returning topically adjacent chunks instead of structurally relevant ones.
The Compilation Step
You can't query a knowledge graph that doesn't exist. Building it is the hard part, and being honest about that difficulty is important.
Andrej Karpathy proposed one approach: compile raw sources into a structured wiki using an LLM, with index files that serve as navigation layers. The LLM reads your raw documents, extracts the key concepts, writes structured articles about them, and produces an index. Future queries consult the index first, then pull in only the articles they need. It's a compiler: raw sources go in, structured knowledge comes out.
The Graphify project takes a different path. Instead of wiki articles, it builds a knowledge graph directly. Nodes are entities, edges are relationships, each with confidence scores. Different file types (code, markdown, PDFs, images) follow different extraction paths but converge on the same node-and-edge schema. The output is a graph report that can be loaded into context.
Both approaches share the same architectural insight: you pay the compilation cost once, upfront, and amortize it across every future query. The raw documents are the source material. The compiled output (wiki or graph) is the storage layer your retrieval system actually operates on.
The quality of this compilation step determines everything downstream. Noisy entity extraction produces a noisy graph. Ambiguous relationship labeling produces unreliable distances. Graphs go stale when the underlying data changes. These are real engineering problems, not theoretical ones. If I told you building a good knowledge graph was easy, I'd be selling you something.
But the alternative is running retrieval directly on flat chunk stores, which is what most teams do, and the reason context pollution is the number one failure mode in production RAG systems.
Wiring It Into the Memory Hierarchy
In Part 1, I described four tiers: L1 cache (system prompt), L2 cache (tool results and recent turns), main memory (retrieved chunks), and disk (full corpus). The knowledge graph changes what's possible at the boundary between main memory and disk.
Without a graph, the page fault handler is simple and dumb. A query comes in, you embed it, you find the nearest vectors, you load those chunks. The eviction policy is equally dumb: when the context fills up, drop the oldest or lowest-scored chunks. There's no structural awareness of how concepts relate to each other.
With a graph, the page fault handler gets smarter. When a query comes in, you identify the relevant node (or nodes) in the graph and traverse outward to pull in structurally connected concepts. The traversal depth becomes a tuning parameter: one hop gives you directly related concepts, two hops gives you the neighborhood, three hops and you're probably pulling in too much.
Eviction becomes graph-informed too. When the context fills up, you measure graph distance from the current focus node. Concepts that are far away in graph space get evicted first. Concepts that are close stay, even if they haven't been explicitly referenced in recent turns. That's semantic Belady's: evict the thing that's least likely to be needed, estimated by structural distance rather than recency.
The focus node itself shifts as the conversation progresses. If the user starts asking about revenue and then pivots to supply chain issues, the focus moves in graph space, and the eviction boundary moves with it. Concepts that were close to "revenue" but far from "supply chain" get evicted. Concepts near "supply chain" get loaded. The working set in context tracks the actual reasoning thread.
Domain-Specificity Matters
One thing I've learned from working with clients across different verticals: what counts as "semantically hot" is completely domain-dependent.
In a financial knowledge graph, recency dominates. Q3 2025 earnings data is hot. Q3 2023 data is cold, unless the user is explicitly asking about historical trends. The graph distance metric needs a temporal decay factor layered on top.
In a medical knowledge graph, relationships are more stable. The connection between a drug and its mechanism of action doesn't go stale the way earnings data does. But edge weights matter more: a well-established causal relationship should be prioritized over a preliminary correlation from a single study.
In a codebase graph, structural proximity maps to functional coupling. If you're editing a function, the other functions it calls (one hop), the modules they belong to (two hops), and the tests that cover them (one hop via a different edge type) are all likely relevant. A utility function four modules away probably isn't.
There's no universal traversal policy. The page fault handler needs to be tuned to the domain, which means the memory management layer can't be a generic library. It has to understand the semantics of the graph it's operating on.
This is, incidentally, the kind of problem that a consultant who understands both the infrastructure and the domain can help with. Generic RAG frameworks won't get you here.
What's Still Missing
I want to be upfront about the gaps. Graph-distance-as-Belady's is a strong conceptual framework, but there are open engineering problems.
Incremental compilation is one. If your underlying data changes frequently, recompiling the entire graph is expensive. You need a way to update affected subgraphs without rebuilding from scratch. Some graph databases handle this natively. Most knowledge graph construction pipelines don't.
Multi-hop traversal cost is another. Walking three hops in a large graph can touch thousands of nodes. You need pruning heuristics or pre-computed subgraph summaries (which is what GraphRAG's community summaries are for, and credit where it's due, that part works well).
Hybrid retrieval is probably the practical answer for most production systems. Use vector similarity for the initial coarse retrieval (find the right neighborhood), then use graph distance for fine-grained ranking and eviction decisions within that neighborhood. You don't need to choose one or the other.
The Takeaway
The RAG taxonomy from Part 1 showed that every retrieval variant is a page replacement policy. This post argued that knowledge graphs give you something most of those policies lack: a structurally informed signal for what to keep in context and what to evict.
Graph distance approximates Belady's optimal because it encodes how concepts actually relate to each other, not just how similar their embeddings are. That approximation is imperfect, domain-specific, and requires real engineering to build. But it beats LRU, it beats FIFO, and it beats "stuff everything in and hope for the best."
The teams I work with that are getting RAG right aren't picking a retrieval framework off a comparison chart. They're building a memory hierarchy with a storage layer that understands the structure of their data.
That's the hard part. It's also the part that matters.