Apples, Oranges, and Signatures

Cryptographic migration, particularly in the post-quantum setting, poses significant practical challenges and requires reliable performance data to support sound engineering decisions. For ML-DSA , however, existing benchmarking practices often produce misleading or non-comparable results, complicating migration and cryptographic agility efforts. This paper analyzes common pitfalls in benchmarking ML-DSA signature operations, including subtle inconsistencies when comparing security levels. We show that execution-time variability of the ML-DSA signing algorithm—an inherent property due to rejection sampling and other data-dependent components—makes commonly used straightforward metrics, e.g., min / average / max , unsuitable for migration planning. To address this gap, we propose a robust benchmarking methodology based on standardized input data sets and clearly qualified reporting metrics. The proposed approach enables fair comparison across hardware and software implementations and supports designers of real-time systems to assess the worst-case execution time.