Neural One-Shot Face Puppetry: Conditional GANs for Dynamic Expression Texture Synthesis

Realistic facial reenactment from a single image demands models that can hallucinate expression-dependent texture details which are never explicitly observed, such as transient wrinkles, teeth, and tongue appearance. This paper presents a conditional generative adversarial framework that learns to synthesize high-resolution dynamic facial textures in UV space by conditioning on a target identity texture and per-frame source expression textures. A convolutional encoder-decoder generator and PatchGAN-style discriminator are jointly trained with a hybrid objective combining adversarial and reconstruction losses, with spatial weighting that emphasizes eyes and mouth regions where fine-grained deformations are most salient. To handle extreme underconstrained regions like the inner mouth when the target image shows closed lips, the network infers plausible appearance which is further refined using flow-based upsampling from the source video. Trained on thousands of aligned expression sequences with extensive data augmentation over identity and skin tone, the model generalizes to unseen subjects and produces temporally coherent, expression-aware textures suitable for reenactment and video face replacement. Quantitative comparisons against direct texture-transfer baselines show lower reconstruction error and higher structural similarity, while qualitative results demonstrate sharper wrinkles and more convincing mouth interiors in full video composites.