Enhancing makeup transfer robustness under varied lighting conditions with lighting transfer GAN

Abstract Most existing makeup transfer methods ignore the effect of lighting conditions, resulting in imprecise and unnatural results of makeup transfer. To address this issue, in the paper, we propose a Lighting Transfer Generative Adversarial Networks (LTGAN) for makeup transfer, which includes Lighting Transfer Module (LTM), Facial Feature Encoder (FEEnc), Detail Refining Encoder (DREnc), Makeup Transfer Module (MTM) and Makeup Apply Decoder (MADec). First, lighting transfer is proposed to transfer the lighting of the source image to the reference image, reducing the impact of reference image lighting on makeup transfer. Second, DREnc is introduced to capture both global/local and multi-level features by combining CLIP (Contrastive Language-Image Pretraining) image encoder and attention model. Third, Hybrid Loss Function is proposed to achieve fine local makeup transfer by introducing a local loss function based on facial masks. In particular, ConvNeXt V2 block network is introduced to both FEEnc and MADec to achieve more accurate feature map extraction. Finally, we also consider the neck as a part of makeup to obtain a new facial mask by fusing the mask of the neck with the original mask of the face, thereby making the overall effect of makeup transfer more natural. Compared to other existing approaches, the experimental results show that the proposed fine makeup transfer model based on lighting transfer retains more information from the source image, and avoids interference from irrelevant reference image information on makeup transfer to attain more natural and state-of-the-art makeup.