Why Geometric Features Fail at LLM Hallucination Detection

We investigate hallucination detection in Large Language Model outputs using geometric and semantic features from embedding spaces. Through rigorous evaluation with proper train-test separation (400 training samples, zero overlap with test sets), we find that simple question-response cosine similarity is the dominant signal (ΔF1=-0.261 when removed), while geometric features (centroid distance, local density, k-nearest neighbors, local outlier factor) provide minimal additional value and may actually be redundant (ΔF1=+0.042 when removed, indicating performance improvement). On in-distribution data, our approach achieves F1=0.600 with perfect precision (1.000) but modest recall (0.429). However, we document severe distribution shift challenges: performance drops dramatically on out-of-distribution benchmarks (TruthfulQA F1=0.094, HaluEval F1=0.327 with domain adaptation). Through comprehensive error analysis, we identify three primary failure modes: (1) training-test distribution mismatch (14.2% vs 40-60% hallucination rates), (2) systematic misconceptions that differ from training patterns, and (3) high-dimensional distance measures losing discriminative power. We compare against five baseline methods including SelfCheckGPT, which substantially outperforms our approach on out-of-distribution data (F1=0.750 vs 0.094 on TruthfulQA). Our work provides valuable negative results showing that geometric features from OOD detection literature do not transfer effectively to hallucination detection, and highlights the critical importance of distribution matching between training and deployment.