Enhancing Opaque-Box VLMs Attack With Prior Information Guide

Large vision-language models (VLMs) have made significant progress in the field of response generation. However, these models also face more potential security risks due to diverse media inputs. In realistic settings, adversaries often only have access to black-box models, which makes query-based attacks on black-box models an important attack method. Although this method has high accuracy, it also suffers from the issue of low attack efficiency. To address this challenge, we propose an optimization method based on prior information. This method uses the gradient generated in the surrogate white-box model as prior information to guide the query-based attack in the black-box model, reducing the number of unnecessary queries and thus improving the attack efficiency. We conduct comprehensive experiments using various models to demonstrate the availability and universality of the optimization method based on prior gradient information.