Reinforcement Learning and Exact Optimization for Traffic-Weighted Hybrid Truck-Drone Last-Mile Delivery With Deadlines

We propose a Reinforcement Learning (RL) framework for hybrid truck-drone Last-Mile Delivery (LMD), where drones launch from a truck and either rendezvous with it or return to the depot after serving customers, subject to vehicle range limits and customer-defined deadlines. Deliveries completed after the latest allowable service time are treated as deadline violations (lateness) and are penalized in the objective. To provide an exact reference, a Mixed Integer Linear Programming (MILP) model is formulated. It serves both as a benchmark for evaluating the RL approach and as a practical method when the delivery set is small or when minimizing lateness is the primary objective, since the exact MILP solution is guaranteed to be optimal but become computationally expensive to solve as problem size grows due to the combinatorial increase in routing and coordination possibilities. The RL method is introduced as a scalable alternative for larger hybrid truck-drone deployments because it maintains practical computation times through iterative learning and avoids the exponential growth in solving effort associated with exact optimization. The RL framework employs an$\epsilon $-greedy policy with decay and a reward structure that mirrors the MILP objective. Traffic congestion is incorporated through a scaling matrix that adjusts truck travel times on road segments, while drones remain unaffected due to aerial operation. RL is further benchmarked against Genetic Algorithm (GA) and Particle Swarm Optimiztion (PSO), which are also formulated to align with the same MILP objective to ensure a consistent basis for comparison. Results show that the RL component scales more efficiently with increasing delivery size while maintaining competitive solution quality.