An RGB-Event Hierarchical Fusion Enhancement Network for Object Detection

Object detection in frame-based vision systems often faces the problem of significant performance degradation under challenging conditions due to the inherent sensing limitations of conventional cameras. Event cameras, with their advantages of high temporal resolution and sparse data output, offer a promising solution for addressing the performance degradation problem. However, due to the asynchronous and sparse nature of event data, event camera–based detection methods generally exhibit lower accuracy than their image-based counterparts. Therefore, effectively fusing data from both event and frame-based cameras represents the main challenge. To this end, this study introduces a hierarchical fusion enhancement network (FENET) designed to achieve more robust object detection through complementary multimodal fusion. First, to address the problem of underutilization of event timescales in conventional dense representations, this study designs an innovative, learnable multi-timescale event preprocessing module. This module generates temporal aggregation frames (TAFs) by leveraging event data from a spectrum of intervals, ranging from standard RGB exposure times down to finer timescales. This approach ensures the comprehensive capture and clear delineation of the target's structure, thereby enhancing the integrity of the visual information. Furthermore, a two-branch adaptive enhanced cross-modal aggregation module (ECAM), which guides the adaptive fusion of features from both modalities by first self-enhancing the features of each, is employed. The main goal of the ECAM is to achieve efficient feature aggregation by combining global context with local spatial information, subsequently using the aggregated features to steer the adaptive fusion process. Finally, by recognizing the scarcity of large-scale, high-quality annotated datasets for event-based object detection, the DSEC-D dataset comprising 60 sequences, with a total of 70,739 images and over 250,000 labels, is manually constructed. Extensive experimental results demonstrate that the proposed FENET can successfully overcome the perceptual limitations of conventional cameras. Notably, the proposed FENET surpasses current RGB-event fusion schemes, achieving an 8.6% improvement in the mAP50 metric and a 3.7% improvement in the mAP50:95 metric over the FRN baseline. We have released the code and the dataset at https://github.com/wkydawang/FENET.