MetaEval: Token-level Self-Evaluation and Selective Decision Training for Autoregressive Language Models

We propose MetaEval, a training paradigm that equips autoregressive language models with token-level self-evaluation and decision-making abilities. The method trains a compact meta head to predict, at generation time, the likelihood that the model's token prediction is correct while ensuring the meta head does not trivially access the ground-truth token. The meta prediction can be used either as a calibrated confidence score or as an input to a downstream policy (accept/reject/verify) trained with reinforcement learning to optimize utility under cost constraints. We present principled masking strategies that guarantee the meta head's input excludes token information leaked by teacher-forcing, formalize combined supervised and RL objectives, provide theoretical analyses for calibration and convergence under standard assumptions, and give detailed pseudocode and implementation recipes compatible with Transformer architectures. We also analyze computational cost, propose practical sampling schedules to minimize overhead, and design an evaluation protocol that measures both calibration and downstream improvement in selective prediction tasks. Experiments and ablations (proposed) show that MetaEval significantly improves calibration, reduces hallucination when paired with a verifier policy, and yields strong cost–benefit trade-offs for real-time generation systems.