How to measure LLM hallucination rate rigorously

What model hallucination is

Hallucination is when a model produces content that reads plausibly but doesn't match reality — inventing papers, people or figures that don't exist. In knowledge Q&A, legal and medical settings, hallucination leads directly to bad decisions.

How to design the cases

An effective hallucination case has to be falsifiable. We usually work from three angles: fictional entities, asking about books or people that don't exist; beyond the knowledge boundary, asking for real-time information; and leading premises, where the question presupposes something false.

Scoring rules

The core judgement isn't how detailed the answer is — it's whether the model says so honestly when it lacks the information. We use two metrics: fabrication detection rate, for whether invented content appears, and uncertainty expression, for whether the model volunteers that it cannot confirm something.

Calculating the metric

Hallucination rate is an inverse metric, so lower is better. A case that passes scores full marks; one containing fabricated content scores zero; the dimension score is the weighted average across cases. At Deep intensity and above the run repeats several times and takes the mean, which cancels out the noise of any single run.