Let Human Sketches Help: Empowering the Challenging Image Segmentation Task With Freehand Sketches

Sketches, with their expressive potential, enable humans to convey the essence of an object through a rough contour. This work leverages expressive power for the first time to improve segmentation performance in challenging tasks such as camouflaged object detection (COD). We propose a sketch guided interactive segmentation framework that allows users to intuitively annotate objects with freehand sketches rather than relying on traditional bounding boxes or points commonly used in models such as the SAM. Our method introduces dedicated network architectural enhancements and a novel sketch augmentation strategy to fully exploit sketch input, leading to significant accuracy gains compared with text- or box-based annotations. Furthermore, our model's output can directly train other neural networks, achieving performance comparable to that of pixel-level annotations while reducing the annotation time by up to 120× and thereby lowering the barrier for large-scale dataset creation and model training. To support future research, werelease KOSCamo+, the first freehand sketch dataset for COD, along with code and a labeling tool. These contributions open promising avenues for expanding sketch-based interaction to broader segmentation tasks and exploring multimodal annotation strategies that combine sketches, text, and other lightweight user inputs.