Objective acceleration for unconstrained optimization

Journal article

Acceleration schemes can dramatically improve existing optimization procedures. In most of the work on these schemes, such as nonlinear Generalized Minimal Residual (N-GMRES), acceleration is based on minimizing the $\ell_2$ norm of some target on subspaces of $\mathbb{R}^n$. There are many numerical examples that show how accelerating general purpose and domain-specific optimizers with N-GMRES results in large improvements. We propose a natural modification to N-GMRES, which significantly improves the performance in a testing environment originally used to advocate N-GMRES. Our proposed approach, which we refer to as O-ACCEL (Objective Acceleration), is novel in that it minimizes an approximation to the \emph{objective function} on subspaces of $\mathbb{R}^n$. We prove that O-ACCEL reduces to the Full Orthogonalization Method for linear systems when the objective is quadratic, which differentiates our proposed approach from existing acceleration methods. Comparisons with L-BFGS and N-CG indicate the competitiveness of O-ACCEL. As it can be combined with domain-specific optimizers, it may also be beneficial in areas where L-BFGS or N-CG are not suitable.

Authors: Riseth, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Wiley
• Journal: Numerical Linear Algebra with Applications
• Volume: 26
• Issue: 1
• Pages: e2216
• Publication date: 2018-10-02
• Acceptance date: 2018-09-07
• DOI: 10.1002/nla.2216
• EISSN: 1099-1506
• ISSN: 1070-5325
• Keywords: nonlinear GMRES; acceleration; full orthogonalization method; optimization