SAIL: Seeking Attribute in Language to Improve Chinese Pretraining

Imbalanced exposure of Chinese characters during pretraining often leads to inconsistent performance across downstream tasks. This problem stems primarily from the randomness of the masking strategy and is further aggravated by the semantic granularity mismatch between Chinese and English. To address these challenges, we propose the seeking attribute in language (SAIL) scheme to enhance Chinese language modeling. SAIL constructs a hierarchical attribute pyramid grounded in the rich semantics of Chinese and comprises three core tasks: masked language modeling (MLM), masked character restructuring (MCR), and masked quality evaluation (MQE). MQE employs a Chinese association graph to dynamically assess the quality of randomly masked samples and uses this assessment as a supervisory signal during pretraining. MCR integrates character-level structural information to mitigate the imbalance caused by varying word frequencies. To facilitate multilevel semantic interaction, MLM not only models word-level semantics but also bridges the character-level structural semantics from MCR and the sentence-level semantics from MQE. In turn, MCR and MQE reevaluate and enhance the performance of MLM from the lower and upper layers of the hierarchical pyramid, respectively. Experimental results across a range of Chinese language understanding tasks, including multidomain comprehension, fine-grained entity recognition, and diverse text classification, demonstrate that our model consistently achieves superior average performance. These findings show the generalization capability and transferability of the SAIL framework in modeling complex Chinese semantics.