A while back I was able to build a working knowledge base in less than an hour. Fifty-five minutes to be accurate. I started by opening VS Code with Claude Code in the side panel. I had documentation scattered across roughly ten training manuals, a video archive nobody watched, support ticket histories, enhancement docs, and meeting transcripts. The knowledge existed. It was just spread across systems with nothing connecting them.
I described the islands to Claude the way I'd describe them to a new engineer. Then I asked for a chat interface backed by a vector database with RAG-style retrieval from plain-language questions. At around minute 45, it was running and returning accurate answers from the actual docs. I know it was not yet production software, but it was a real prototype on real data.
But I had slipped one extra requirement into that prompt, almost without thinking about it: client-specific information should be flagged and surfaced only when a query came from that client or was about that client. We had training recordings and support history tied to individual accounts. That requirement is the whole article. The moment you say "surface this when the query is about that client," you have stepped outside what retrieval by similarity can do. Similarity answers one question: which stored text resembles the wording of the query. But a support ticket and a training recording tied to the same client may share no vocabulary with each other, or with the question being asked. What links them is not wording; it is the client they both belong to. That link is a relationship, and a relationship is something a similarity search has no way to represent.
My prototype got away with it in a demo. In production, where real users ask cross-source questions every day, that shortcut stops holding.
What embeddings actually do, and where they stop
Embed-and-retrieve works like this: you chunk the documents, run each chunk through an embedding model, store the vectors, and at query time you embed the question and pull back the nearest neighbors by cosine similarity, a measure of how close two texts sit in meaning. It works. For a large class of questions — "what does our refund policy say," "how do I configure SSO" — it works well, and you can have it running in an afternoon.
The ceiling shows up the moment an answer lives across sources that don't look alike.
Take a question we actually get internally: "Why is this client's month-end close behaving differently from the others?" The answer is not in one chunk. It's in a custom configuration noted in a support ticket from two years ago, a behavior described in a training recording for that specific account, and a feature flag mentioned in an enhancement doc. None of those three documents are textually similar to each other, and none of them are similar to the question. An embedding search returns whatever happens to share vocabulary with "month-end close." That points straight to the generic manual, which is the one document in the set that does not contain the answer.
Similarity is a proxy for relevance. It's a decent proxy when relevant things tend to use the same words. It collapses when the relevant things are connected by something other than vocabulary: a shared client, a shared subsystem, a cause-and-effect chain that nobody wrote down in a single place.
This is the part that gets skipped in the market. A team sets up RAG, declares "the knowledge base is done," and stops exactly there. You end up with a search engine that talks. Useful, but it answers "what does the document say," and the questions that actually justify the project are the ones that ask "what does the situation mean."
The relationships are the knowledge
I was reminded of all this at Milano AI Week, where someone framed it cleanly: the real intelligence in a knowledge base isn't in the chunks; it's in how the segments relate, the graph rather than the index.
Concretely, that means a second representation alongside your vectors. You run entity extraction over the same corpus — clients, features, configurations, people, error conditions, dates — and you model the edges between them. Ticket 4471 references client Aurora. Aurora runs the custom close configuration. That configuration was introduced by enhancement request 882. The training recording from March covers that workflow. Those are explicit edges in a graph, and they are traversable regardless of whether any two of those nodes share a single word.
Now the same question (why is this client's month-end close behaving differently) gets answered along a different path. The system maps "this client" to the Aurora entity, walks the edges out to its configurations, tickets, and recordings, and assembles a candidate set that no similarity search would ever produce, because the documents have almost nothing lexically in common. The graph answers what is actually related; the embeddings answer what looks similar. You want both: similarity finds the entry points and the unstructured passages, and the graph traversal gathers everything connected. Either one alone leaves capability on the table.
I'm not arguing embeddings are wrong. I'm arguing that "chunks plus embeddings" is a floor people keep mistaking for a ceiling.
The cost is real, and it is not catastrophic
Here's the honest trade-off, because the graph approach is not free.
Entity extraction has to be tuned. The first pass over-extracts, treats every capitalized string as an entity, and cannot tell "Aurora the client" apart from "Aurora the internal name of a feature." Entity resolution — deciding that three different spellings refer to the same client — is where most of the real effort goes. You need a schema for what entities and relationships matter to your business, and that schema is opinionated, which means it requires someone who understands the domain to define it. You'll iterate it several times before it behaves. This is the same tuning loop I've hit building constraint surfaces for AI tools internally: the gap between how you described the rule and how the system interpreted it is always wider than you expect at the start, and you only close it by running real queries and watching what breaks.
Operationally you're now maintaining two stores and a pipeline that keeps the graph in sync as documents change. That's more moving parts than a single vector index. Budget for it.
But the cost difference between the two architectures is incremental, not an order of magnitude. The embed step is shared. The extra work is the extraction pipeline, a graph store, and the retrieval logic that combines the two. The capability difference, by contrast, is enormous. A chunk-only system can answer questions confined to a single document. The graph version can answer questions whose answer requires reasoning across sources that were never connected by an author. For an enterprise knowledge base, that second class of question is usually the entire reason the project was funded.
If your knowledge base is meant to serve support engineers, account managers, or anyone diagnosing a situation that spans history, the chunk-only version will demo beautifully and disappoint in week three, when the first cross-source question comes in and it confidently returns the generic manual. This is exactly the kind of representation decision we work through when we design a knowledge base meant to survive contact with real questions.
What to check before you sign off on an architecture
There's a simple test. Take five questions your team actually asks, the hard ones, the ones that make someone sigh, and for each one, ask yourself whether the full answer lives in a single document or has to be stitched together from several. If most of them need the stitching, similarity retrieval will not get you there, and no amount of better embeddings or bigger context windows fixes a representation problem.
That test is also the dividing line between a search engine and a knowledge base. A search engine retrieves text. A knowledge base models the entities your business cares about and the relationships between them, then reasons over that structure. The prototype I built in fifty-five minutes was the former with one relationship bolted on by hand. The production version of that same system is the latter, and the difference is entirely in how seriously you model the graph.
Before you approve any internal knowledge base design, ask the team building it one question: when the answer lives in three unrelated documents, what mechanism finds the other two? If the only answer is "similarity," that tells you two things: the team hasn't seen the ceiling yet, and the budget you're about to approve will buy a system that stops exactly where your hardest questions begin. Decide whether that's enough before the cost moves downstream into the maintenance no one can see.