MATdiff: Mask-aware transformer with diffusion model for large-mask image inpainting

Diffusion models are widely used in image inpainting, but their inherent randomness may cause semantic inconsistencies between inpainted regions and known parts of the image. To address this, we propose Mask-Aware Transformer with Diffusion Model (MATdiff), a latent diffusion model that first employs a transformer-based MAT as a conditional encoder and subsequently embeds the semantic features extracted from MAT into the reverse diffusion process. MATdiff exploits the long-range dependencies modeled by transformers while mitigating the randomness of diffusion models, resulting in inpainted regions with semantic coherence and visual fidelity to the visible content in large-mask image inpainting. During the training process of MATdiff, we modify the input layer of the pre-trained unconditional latent diffusion model to accommodate MATdiff, then freeze the conditional encoder. Finally, we conduct end-to-end optimization on the entire MATdiff. Our training method achieves remarkable results in generating plausible overall structures for image inpainting. Extensive experiments on the CelebA-HQ face dataset validate the state-of-the-art image inpainting performance of MATdiff. For large-mask image inpainting, our model notably improves on MAT, with gains of 5.94% (FID), 3.13% (LPIPS), 18.75% (P-IDS), and 5.14% (U-IDS).