Adaptive Masking Over Masking for Sequence Generation

Transformer-based autoregressive (AR) methods have achieved appealing performance for varied sequence-to-sequence generation tasks, but suffer from low inference efficiency. In recent years, many non-autoregressive (NAR) strategies have been proposed to speed up the inference process. Among them, the Conditional Masked Language Model (CMLM) is one of the most versatile frameworks. It can support various sequence generation scenarios and achieve competitive performance on these tasks. In this paper, we introduce a simple yet effective adaptive masking over masking strategy to further enhance the ability of CMLM. Experiments on6different tasks with18datasets in total confirm that our proposed simple method achieves significant performance improvement over the strong CMLM model. Surprisingly, our proposed method yields state-of-the-art (SOTA) performance on 3 neural machine translation datasets when applied to the vanilla CMLM. Further applied to the Conditional Masked Language ModeL with Correction (CMLMC) which is an enhanced version of CMLM, our proposed method can achieve SOTA performance on all WMT datasets, even significantly outperforming the strong AR model (the vanilla Transformer). Furthermore, our proposed method can be simply applied in the fine-tuning stage with pre-trained language models, results on the GLGE benchmark (3 different tasks) outperform all other NAR pre-training methods.