Evidence-Budgeted Benchmarking: Assessing Robustness and Efficiency for Trustworthy Rumour Analysis

Rumour detection and verification are security, and dependability-critical functions in cyber-physical-social-thinking (CPST) systems, where highstakes decisions must be made under scarce evidence, evolving narratives, and adversarial influence. However, most existing evaluations emphasize in-domain accuracy and rarely quantify deployment-critical factors such as evidence availability, computational cost, robustness under distribution shift, and confidence reliability. We propose evidence-budgeted benchmarking for trustworthy rumour analysis: an assessmentfirst, cost-aware evaluation framework that (i) standardizes heterogeneous datasets using a unified JSONL schema, (ii) explicitly controls conversational evidence via reply budgets (K0/K3/K10/KALL), and (iii) jointly measures predictive performance, efficiency (latency/memory), robustness to dataset shift and lightweight perturbations, and calibration. Experiments on rumour verification (PHEME, RumourEval17) and supplementary detection (Twitter15/16, Weibo2021) show that evidence utility is non-monotonic: low-to-moderate budgets can match or exceed full-thread ingestion while reducing inference cost, whereas evidence overload degrades both performance and confidence reliability. Distribution shift emerges as the dominant failure mode, causing substantially larger degradation than lightweight perturbations, while inexpensive, human-plausible manipulations still measurably erode performance. These results indicate that dependable rumour assessment cannot be justified by accuracy alone, and motivate deployment-oriented evaluation with budget-aware evidence use, explicit cost reporting, shift-first robustness testing, and calibrated confidence to support security-and trust-critical decision policies.