A BEV Perception Transformer Accelerator With Saliency-Driven Image/Point Cloud Fusion and Phase-Linked Dataflow in 28 nm CMOS

Deploying advanced Transformer-based models for real-time, high-accuracy multimodal bird’s-eye-view (BEV) perception in autonomous driving imposes substantial hardware demands. To address this, we propose a low-cost, low-power image/point-cloud fusion Transformer accelerator that supports two modes: high-performance driving and ultra-low-power sentry operation. We first propose a cross-modal saliency evaluation mechanism that prunes redundant computation. We then propose a column-wise computing array with a three-level adaptive-precision strategy to enable saliency-driven fusion. Furthermore, we propose a phase-linked dataflow that optimizes on-chip data reuse and an on-chip data-forwarding method that reduces external memory access (EMA). Fabricated in 28nm CMOS, this chip achieves the best perception accuracy on the nuScenes dataset, surpassing prior state-of-the-art (SOTA) accelerators. This is the first BEV perception Transformer accelerator to support a dual-mode architecture of high-performance driving and low-power sentry mode, sustaining an active power of 26.6mW and a long-term average power of only 1.68mW in sparse-activity scenarios. In driving mode, it delivers a peak frame rate of 15.89FPS and a peak energy efficiency of 6.11TOPS/W. Overall, this chip outperforms SOTA accelerators in accuracy, frame rate, and energy efficiency.