Convergence of policy gradient methods for finite-horizon stochastic linear-quadratic control problems

Journal article

We study the global linear convergence of policy gradient (PG) methods for finite-horizon continuous-time exploratory linear-quadratic control (LQC) problems. The setting includes stochastic LQC problems with indefinite costs and allows additional entropy regularisers in the objective. We consider a continuous-time Gaussian policy whose mean is linear in the state variable and whose covariance is stateindependent. Contrary to discrete-time problems, the cost is noncoercive in the policy and not all descent directions lead to bounded iterates. We propose geometry-aware gradient descents for the mean and covariance of the policy using the Fisher geometry and the Bures-Wasserstein geometry, respectively. The policy iterates are shown to satisfy an a-priori bound, and converge globally to the optimal policy with a linear rate. We further propose a novel PG method with discrete-time policies. The algorithm leverages the continuous-time analysis, and achieves a robust linear convergence across different action frequencies. A numerical experiment confirms the convergence and robustness of the proposed algorithm.

Authors: Giegrich, M; Reisinger, C; Zhang, Y

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Society for Industrial and Applied Mathematics
• Journal: SIAM Journal on Control and Optimization
• Volume: 62
• Issue: 2
• Pages: 1060 - 1092
• Publication date: 2024-03-22
• Acceptance date: 2024-01-05
• DOI: 10.1137/22M1533517
• EISSN: 1095-7138
• ISSN: 0363-0129
• Language: English
• Keywords: FFR; mesh-independent convergence; relative entropy; geometry-aware gradient; global linear convergence; continuous-time linear-quadratic control; policy optimization