Post-training quantization of CNN accelerator with mixed precision floating-point arithmetic selection based on a genetic algorithm

The increasing complexity of Deep Neural Network (DNN) models, coupled with rising demands for energy efficiency and computational speed in edge devices, requires reconsidering traditional numerical formats and computation methods. Conventional DNN quantization, which often uses low-bit integers due to fixed hardware implementations, applies uniform precision across all values. This approach can be inefficient when certain data points require finer granularity to maintain accuracy. In contrast, floating-point (FP) quantization provides greater flexibility in bit allocation, enabling more precise granularity where needed. This paper proposes mixed-precision FP arithmetic based on a genetic algorithm (GA), an evolutionary optimization method inspired by natural selection, to determine the optimal precision for each layer. To minimize rounding errors, stochastic rounding is incorporated, and layer-specific exponent bias adjustments are implemented to enhance data representation precision. Experimental results from implementing the proposed mixed-precision FP quantization on the YOLOv2-tiny model reveal a 2 . 9 × reduction in energy consumption per image and a 5 0 % reduction in memory requirements, with only a negligible loss of 0 . 1 3 % in mean Average Precision (mAP@0.5) as evaluated on the VOC dataset, compared to Bfloat16, a known low-cost floating-point method. Additional experiments with the three-format combination [FP8, FP7, FP6] are reported to demonstrate that the proposed search procedure generalizes naturally to diversified bit distributions.