Research Seminar by LOW Siow Meng | Iterative Lower Bound Optimization for Planning in Continuous MDPs

Recent advances in deep learning have enabled optimization of deep reactive policies (DRPs) for continuous Markov Decision Process (MDP) planning by encoding a parametric policy as a deep neural network and exploiting automatic differentiation in an end-to-end model-based gradient descent framework. This approach has proven effective for optimizing DRPs in nonlinear continuous MDPs, but it requires a large number of sampled trajectories to learn effectively and can suffer from high variance in solution quality. In this talk, we revisit the overall model-based DRP objective and instead take a minorization-maximization perspective to iteratively optimize the DRP with respect to a locally tight lower-bounded objective. This reformulation has various advantages, such as ability to reuse samples between iterations and theoretical guarantee of monotonically improving objective. We present three empirical experiments and demonstrate the superior sample-efficiency and learning stability of our proposed method.