Batch sample-wise stochastic optimal control via stochastic maximum principle

In this work, we study the stochastic optimal control (SOC) problem mainly from the probabilistic viewpoint, i.e., via the stochastic maximum principle (SMP) [30]. We adopt the sample-wise backpropagation scheme proposed in [1] to solve the SOC problem under the strong convexity assumption. Importantly, in the stochastic gradient descent (SGD) procedure, we use batch samples with a higher-order scheme in the forward SDE to improve the convergence rate in [1] from $ \sim \mathcal{O}(\sqrt{\frac{N}{K} + \frac{1}{N}}) $ to $ \sim \mathcal{O}(\sqrt{\frac{1}{K} + \frac{1}{N^2}}) $, and note that the main source of uncertainty originates from the scheme for the simulation of the $ Z $ term in the BSDE. In the meantime, we note the SGD procedure uses only the necessary condition of the SMP, while the batch simulation of the approximating solution of BSDEs allows one to obtain a more accurate estimate of the control $ u $ that minimizes the Hamiltonian. We then propose a damped contraction algorithm to solve the SOC problem, and a proof of convergence for a special case is attained under appropriate assumptions. We then show numerical results to check the first-order convergence rate of the projection algorithm and analyze the convergence behavior of the damped contraction algorithm. We note that comparisons between the numerical results from these two algorithms (projection and contraction) show the projection approach is still favored in terms of stability and efficiency. Lastly, we briefly discuss how to incorporate the proposed scheme in solving practical problems, especially when randomized neural networks are used. We note that in this special case, the backward propagation error can be avoided, and parameter updates can be achieved via purely algebraic computation (vector algebra), which will potentially improve the efficiency of the whole training procedure. Such ideas will require further exploration, and we will leave this as our future work.