Behavioral Cloning of Rover Control Policy with Pretrained Vision Encoders

In recent years, the use of large foundation models as starting points for training models on more specialized tasks has proven effective across a wide range of settings. This has been attributed to their ability to understand and reason about a broad domain of data, which can assist in the learning process for downstream tasks. There is broad interest in evaluating these new technologies for off-world vehicle navigation, which demands a high-level of autonomy in a relatively static environment. In this paper we investigate the effectiveness of the vision language model CLIP (Contrastive Language–Image Pretraining) in a rover control scheme, with the motivation that multimodal foundation models may be integral to robust out-of-distribution performance of future vehicle assets. In particular, we study a behavioral cloning pipeline in which single-camera input is used to predict steering values for an on-site test vehicle. As an initial ablation, we investigate the navigation-relevant feature retention of CLIP image embeddings. We train two models — a standard convolutional neural network (CNN) that accepts raw images and a variant of CLIP with an additional head to learn control values (ClipNN) — and compare their performance on a rudimentary navigation task: keeping to the center of a path. Our initial results indicate that ClipNN performs better than a simple linear probe of the CLIP embeddings, but the conventional CNN approach performs better than ClipNN across the board. Contrary to initial expectations, ClipNN does not demonstrate stronger generalization than the CNN. We hypothesize that this limitation stems from the foundation model's pretraining objective and dataset, which emphasize semantic abstraction over the low-level spatial features critical for visuomotor control.