Delayed-Gratification Deep Reinforcement Learning for Sample-Efficient Photonic Inverse Design

Inverse design in photonics is limited by expensive simulations, making sample efficiency crucial. Although adjoint topology optimization has dominated the field as the mainstream paradigm, it fundamentally struggles with realistic discrete design parameters (such as dispersive multi-material selection or minimum-feature-size constraints) and non-differentiable objectives. While DRL is naturally suited to these realistic regimes, the traditional stepwise "instant-reward" DRL paradigm (T-DRL) is prohibitively sample-inefficient. Here we propose delayed-gratification DRL (DG-DRL), which achieves adjoint-like scaling sample efficiency by enforcing exactly one simulation per episode (independent of design dimensionality) while retaining DRL's generality. A stylized analysis and three benchmarks validate its effectiveness: (i) for structural-color solar films involving dispersive multi-material selection, DG-DRL cuts design time from 1 month to 60 hours and achieves a 9.6× larger color gamut than T-DRL; (ii) for metagrating beam shaping involving 200-nm minimum-feature-size constraints, DG-DRL cuts design time from 1 month to 36 hours and achieves higher beam-shaping accuracy than T-DRL and hybrid methods; (iii) for a 2⁶⁴ freeform grating benchmark, DG-DRL outperforms the adjoint baseline (higher efficiency, lower variance, and constraint-satisfying designs), while matching other mainstream high-performance algorithms with 5–50× higher sample efficiency. These results position DG-DRL as a general, sample-efficient engine for fabrication-constrained photonic design.