Birch: A Real-Time Multi-Domain Multi-Task Extended Reality Perception Accelerator

Birch is a system-on-chip that efficiently and accurately accelerates the multi-task multi-domain extended reality (XR) perception pipeline, with workloads such as visual inertial odometry (VIO), eye gaze tracking, and scene understanding. Birch features vision modules with cascaded line buffers, in-step feature sorting, and double-buffered optical flow to extract and track quality features for VIO, a configurable matrix unit and vector unit for acceleration of deep neural network (DNN)-based tasks like eye gaze tracking and scene understanding, and reuse of the DNN vector unit for acceleration of matrix multiplications in VIO to save area in small-form-factor XR devices. Birch extracts and tracks features in 7.14 ms and 2.29 mJ per frame. For DNNs, Birch runs ResNet-18 at 109 frames-per-second (FPS) and eye gaze extraction at 985 FPS, 3.5× and 1.6× faster than prior work, respectively. ResNet-50 also runs in real time at 46.5 FPS.