Building a Knowledge Graph from Unstructured Text

There's a class of questions that RAG just can't answer well and it took me embarrassingly long to recognize the pattern.

"Which suppliers provide components used in products that failed testing in Q3?" The information exists, scattered across procurement records, test reports and product specs. No single chunk contains the answer, though. To get there you have to follow relationships from one record to the next: supplier → component → product → test result → date. Vector search can't do that. It finds chunks that look similar to the question and stops.

That's what finally pushed me toward knowledge graphs. I had users asking relational questions and a retrieval pipeline that kept returning adjacent but useless chunks no matter how much I tuned it.

Start with the schema, not the extraction

When you start building a knowledge graph the temptation is to extract everything you can find and figure out what's useful later.

I did this. I ended up with a graph that had 47 entity types, relationships pointing in contradictory directions and so much noise that querying it was slower and less reliable than just reading the original documents. Three weeks of work, essentially useless.

What actually worked was a sticky note. I wrote down the five queries my users cared about most, then worked backwards to the entity types and relationships those queries required. It all fit on the note: three entity types and four relationship types. That was the schema. My rule since then is that if a schema doesn't fit on a sticky note, it's too complex for a first pass. Start with 3 to 5 entity types and 3 to 5 relationship types and expand later. Adding a type is cheap. Cleaning junk out of a graph you've already filled is not.

Extraction is easy now, resolution isn't

LLMs made the extraction step almost trivial. Give the model a chunk of text, your schema and a JSON output format and it'll pull entities and relationships with maybe 80% accuracy out of the box. Two years ago this required custom NER models and weeks of annotation. Now it's a prompt.

The part that still makes me want to close my laptop is entity resolution. The same person shows up as "J. Smith", "John Smith", "Dr. John Smith" and "Smith, J. (2024)" across different documents. Are they the same person? Are there two John Smiths? Which publications belong to which one?

I use a layered approach. Fuzzy string matching catches the obvious cases. Embedding similarity catches the subtler ones, as long as I embed the entity name together with its surrounding context and not the name alone. The genuinely ambiguous leftovers go to an LLM with the question "same entity or different?" It's slow and it isn't pretty, but it gets me to about 90% accuracy and I've learned to stop there. A graph with some duplicate nodes is far more useful than no graph because you spent three months on deduplication.

Wiring it in

Once the graph exists, I wire it into the LLM pipeline with graph traversal for structural queries and vector search for semantic ones, both feeding context to the model. The multi-hop questions that used to fail now get answered. "Which suppliers provide components used in products that failed testing in Q3?" becomes a Cypher query that returns in milliseconds and hands the LLM exactly the structured context it needs.

Getting from "I have documents" to "I have queryable knowledge" takes less work than it used to, since LLMs handle extraction. Where I'd spend the effort is on keeping the schema small and on entity resolution. Pick fifty documents, build the pipeline and get it working on the sticky note schema before you add a single new entity type.