Evaluating Embedding Strategies for Structured Knowledge Extraction
The first embedding model I ever used in production was whatever sat at the top of the MTEB leaderboard that week. I sorted by average score, picked the winner, deployed it and genuinely believed I'd done my due diligence.
Retrieval quality came out mediocre, consistently underwhelming without ever being terrible. Queries that seemed obvious to a human would return chunks adjacent to the right answer but never quite it. I spent two weeks blaming my chunking strategy before I finally looked at what MTEB actually benchmarks: clean, self-contained passages of natural language prose such as Wikipedia paragraphs, news articles and textbook sections.
My data was procurement records with nested tables, specification documents full of part numbers and technical reports where "Section 4.2.1" carried meaning that no embedding model on that leaderboard had ever been trained to understand.
What structure does to embeddings
When a knowledge base has structure (tables, hierarchies, key-value pairs, identifiers), embedding models fail in specific, predictable ways.
Flatten a table to text and "Revenue: 3.8M, Q1 2023" produce nearly identical embeddings. The model sees similar tokens. The fact that they represent different time periods, which was the entire point, gets compressed away.
Hierarchical context disappears too. A chunk that says "Minimum: 8GB" means something completely different under the RAM section than under the storage section. The embedding captures the text but has no idea where in the document that text lived.
Then there are identifiers. Part numbers, postal codes and ISO standards carry enormous meaning for the humans searching for them and are essentially random noise to the embedding model.
Testing on your own data
What I do now is build a small evaluation set that reflects my own data. It's maybe two days of work. Pull 50 queries from your real (or realistic) query distribution and map each one to the chunks that should be retrieved, then test 3-4 embedding models and measure recall@5.
The result that surprised me most was that the embedding model matters less than how you chunk. I tested four models across three chunking strategies and the variance from chunking was larger than the variance from model choice every single time. A mediocre model on well-structured chunks, where tables stay intact and section headers travel with their content, outperformed the top-ranked model on naively split text.
It's also worth testing how short queries interact with long chunks. Structured knowledge queries tend to be terse, like "ISO 27001 requirements" or "Component X thermal specs." Some models handle the asymmetry between a five-token query and a two-hundred-token chunk gracefully while others collapse. You won't know which kind you have until you test with your actual query lengths.
There's no universal answer to which embedding model to use for structured data. The teams I've seen get this right all did the same thing: they spent two days building a test set that reflected their actual data, tested a handful of models and let the numbers decide. They used their own queries, their own documents and recall@5 rather than a leaderboard or someone's blog post recommendation. They also came away with a baseline they can re-run every time their data changes, their queries shift or a shiny new model drops on Hugging Face.