Ground Truth and LLM Judges Measure Different Things

For my thesis work I built multi-step LangGraph agents that answer questions over large, fragmented, structured geospatial datasets, plus an automated evaluation pipeline to score them. I wanted one number per run. I ended up with two kinds of scoring because each one misses what the other catches.

The first is a deterministic ground-truth benchmark. When a question has a verifiable answer, it checks whether the system returned the correct one. That's the only check I fully trust. Confident prose doesn't move it.

Plenty of real queries don't reduce to one checkable fact though. Is the answer grounded in the records it retrieved? Does the explanation hold together? Does it claim more than the data supports? No exact-match check can score that, so I used an LLM-as-judge for it. The judge has its own blind spot. It will give a fluent, well-cited answer a high score even when the underlying fact is wrong.

So when the two scores disagree I treat it as a finding. A correct answer with weak grounding and a well-grounded answer that's factually wrong are different failures with different fixes. For every claim I make about the system, I know which score is backing it.

Four architectures on one harness

That setup let me compare four retrieval architectures end to end: vector RAG, schema-routed, graph-indexed and text-to-SQL. They fail in very different ways, so a single metric would have misled me.

The result I care most about is that the quality of the routing index was the primary driver of answer accuracy. The same work cut inference token cost by 3x across paradigms.

Neither result means much unless the runs are actually comparable. Once you're running many configurations across four paradigms, small inconsistencies pile up. Someone edits a prompt. A dependency version moves. A metric gets computed slightly differently in one branch. Every run went into MLflow so parameters and scores were logged the same way each time. Every run also executed inside Docker so I could rerun it in the same environment. Without that I couldn't tell whether a score gap between two architectures was real or just drift in the setup.

Lineage

I also built Neo4j / Cypher knowledge graphs for data lineage, so every generated answer can be traced back to the source records it came from. In enterprise settings that's an auditability requirement. I count it as part of evaluation. Neither score lets a person follow a specific answer back to its records and check it by hand. The lineage graph does.

If you're building something similar, build the evaluation pipeline before you pick an architecture. Until it exists, "approach A beats approach B" is a feeling with no number behind it.