GLCV: A Generalized Learnable Cost Volume for Per-Pixel Visual Correspondence

Establishing per-pixel correspondences between images is a fundamental technique in computer vision. Recent state-of-the-art methods leverage cost volumes to represent per-pixel similarity; however, their construction varies depending on the task. Stereo matching simplifies the task by constraining the search region to a 1D horizontal space, while tasks such as optical flow and video frame interpolation operate in a 2D search space. Leading deep stereo matching methods use a learnable 4D cost volume aggregated with 3D convolutions, enabling robust disparity map generation. In contrast, tasks with a 2D search space use a 4D all-pairs correlation volume rather than a learnable cost volume due to computational cost---which limits geometric information and necessitates iterative optimization. In this work, we propose the Generalized Learnable Cost Volume (GLCV), a novel module that enables the use of learnable cost volumes across all vision tasks entailing 2D search space. Based on the attention mechanism, GLCV splits the 2D search space into two 1D search spaces for horizontal and vertical motions---effectively transforming the 5D cost volume into more manageable 4D volumes. This method leverages the strength of the learnable matching cost aggregation while maintaining computational efficiency. We validate our method through extensive experiments on optical flow estimation and video frame interpolation tasks, demonstrating that replacing conventional all-pairs correlation volumes with GLCV significantly improves performance. Despite a slight increase in parameters due to a lightweight cost aggregation layer, our model achieves higher accuracy with fewer optimization iterations---reducing the overall run-time and enhancing real-time applicability. Our results establish GLCV as an effective and efficient alternative for cost volume construction across various computer vision tasks.