Burrows’ Delta as a Convergent Validator: Stylometric Analysis for Complementary Machine Translation Evaluation

Machine translation (MT) systems are typically evaluated by comparing outputs to human references using metrics that approximate adequacy and fluency, but these metrics are not designed to measure stylistic fidelity, i.e. how closely an output matches the target-language stylistic profile of a high-quality human literary translation. We test whether stylometric distance, operationalized with Burrows’ Delta over the 500 most frequent words, can serve as a convergent validator of adequacy signals while providing interpretable, reference-free diagnostics. Using nine contemporary Greek short stories with author-produced English self-translations and MT outputs, segmented into non-overlapping five-sentence windows, we compare an inverted, min–max normalized Burrows’ Delta score (invΔB) against standard reference-based MT metrics (BLEU, chrF2, TER, BLEURT, COMET, BERTScore) and against an adequacy composite (TQI_win). We find strong convergence between stylometric proximity and adequacy signals, particularly at decision-relevant extremes, but stylometry underperforms adequacy metrics when used alone and provides no incremental predictive benefit beyond semantic-embedding baselines. We conclude that stylometry is best used as a complementary, explainable diagnostic and as a constrained reference-free monitor and not as a substitute for adequacy-oriented MT evaluation.