True-to-physics modeling enables sensing with a handful of photons detected per inference in total

End-to-end optimization has become a powerful tool in optics, enabling co-optimization of physical front ends and digital back ends. In some realistic sensing tasks, a critical challenge is the restriction of physical resources— such as photons in optical sensing—that carry information. Under such constraints, the detection process acts as a lossy channel, discarding information that cannot be recovered later. Here we propose photon-aware neuromorphic sensing (PANS), targeting scenarios where the photon budget is extremely constrained at the detection bottleneck. Our approach models the stochastic photon-detection process exactly as it physically occurs—rather than using conventional approximations—and achieves effective end-to-end optimization despite the inherent stochasticity of few-photon detection. We experimentally demonstrated PANS on classification benchmarks with Ndet total detected photons per inference, achieving 73% (82%) accuracy on FashionMNIST with Ndet = 4.9 (17) and 86% (97%) on MNIST with Ndet = 8.6 (29) total photons—orders of magnitude more photon-efficient than conventional approaches. While demonstrated with classical resources to highlight algorithmic advances, our framework can be applied to other physical front ends including quantum states and emerging sensing hardware, offering a complementary dimension for optimizing sensing under extreme resource constraints.