Sketchformer++: A Hierarchical Transformer Architecture for Vector Sketch Representation

With the rising ubiquity of digital touch devices and sketch-based interfaces, freehand sketching has become an essential mode of visual communication. Nevertheless, interpreting these often ambiguous and sparse sketches poses challenges for computers. This paper presents Sketchformer++, a hierarchical transformer architecture for the neural representation of vector sketches. It treats a vector sketch as a three-level structure, at sketch level, stroke level, and segment level. Three self-attention modules are adopted in the network architecture, corresponding to the sketch hierarchy. The semantics of sketches are aggregated from local to global levels, resulting in neural representations of sketches. Extensive experiments show that Sketchformer++ helps to achieve superior performance in various downstream tasks, including sketch reconstruction, sketch recog-nition, sketch semantic segmentation, and sketch retrieval, demonstrating its robustness and effectiveness as a means of sketch representation. Code is available at https://github.com/BHR7/SketchformerPlus.