SASRO: Hardware--Software Codesign of Real-Time Self-Adaptive Deep Stereo Toward Robotic Applications

With the development of AI, robotics is gradually becoming one of the most popular research topics. Current robotic applications, such as Unmanned Aerial Vehicles (UAVs), robotic manipulators, and humanoid robots, all require stereo vision to perceive the accurate depth map of the environment in real time for subsequent decision-making. In the field of autonomous driving, learning-based stereo depth estimation methods achieve significantly higher accuracy compared to traditional methods. However, for the diverse environments in robots, self-adaptive deep stereo is considered as a promising approach to improve the poor generalization of learning-based methods. Although existing neural network accelerators are available for real-time stereo depth estimation, they all present limitations when it comes to self-adaptive scenarios. Some accelerators focus solely on optimizing the computational flow of forward or backward propagation, failing to support the computation of the self-supervised loss function and exploit the specific workload characteristics commonly found in self-adaptive algorithms. On the other hand, accelerator designers typically rely on domain knowledge from hardware rather than robotic applications to optimize compute-system metrics such as accuracy or latency, leading to suboptimal outcomes for application-level performance. To overcome these limitations, we propose a hardware-software co-design approach of real-time Self-Adaptive deep Stereo towards RObotic applications named SASRO. It includes a heterogeneous computing architecture consisting of an off-the-shelf inference accelerator and our designed Self-Adaptive Processing Unit (SAPU). Based on this architecture, a quantization-friendly network architecture is proposed. We believe that this design can fully leverage its efficiency in self-adaptive deep stereo, achieving a 20% improvement in accuracy and a 1.95× reduction in latency. Different from traditional design methods towards a single workload, SASRO additionally contains an automatic design framework which optimizes the end-to-end robotic metrics instead of isolated compute-system metrics. Taking robot navigation as an example, SASRO achieves a 1.3∼2.2× increase in navigation speed compared to the low-latency and high-accuracy designs.