FFAFormer: A Feature Fusion Attention Former of Point-Image for 3D Object Detection

In current multimodal feature fusion algorithms of 3D object detection for autonomous driving, there is a lack of accurate allocation of weights between features from different fields, which leads to mismatch and subsequent data loss. Ultimately, the accuracy of 3D object detection is reduced. Therefore, a Feature Fusion Attention Former of Point-Image (FFAFormer) is proposed for 3D object detection. Specifically, the features are firstly sampled and encoded through a Feature Encoding Mechanism of Dual Domain Space (FEM-DDS). Meanwhile, precise weight allocation is performed between different domain features, which maximizes the utilization of feature information from different modalities by the Transformer attention. Then, the interaction depth of features at different scale points is enhanced by Multi-Scale Depth Prediction (MSDP), which further refines features and reduces the loss of information. Finally, more accurate fusion of features from different domains are achieved by Dynamic Attention Feature Aggregation (DAFA). Cross-modal features are produced by aligning different features across modalities at a fine-grained level, which are achieved by a learnable dynamic offset and cross-attention mechanism. The effectiveness of FFAFormer is demonstrated by extensive experiments on the KITTI, NuScenes, and Waymo datasets. A real autonomous vehicle platform is also used to verify the proposed algorithm, which shows that the proposed method has enough generalization in complex real-world environments.