AdaGeo: Adaptive geometric learning for optimization and sampling

Conference item

Gradient-based optimization and Markov Chain Monte Carlo sampling can be found at the heart of a multitude of machine learning methods. In high-dimensional settings, well-known issues such as slow-mixing, non-convexity and correlations can hinder the algorithms’ efficiency. In order to overcome these difficulties, we propose AdaGeo, a preconditioning framework for adaptively learning the geometry of parameter space during optimization or sampling. We use the Gaussian Process latent variable model (GP-LVM) to represent a lower-dimensional embedding of the parameters, identifying the underlying Riemannian manifold on which the optimization or sampling are taking place. Samples or optimization steps are consequently proposed based on the geometry of the manifold. We apply our framework to stochastic gradient descent and stochastic gradient Langevin dynamics and show performance improvements for both optimization and sampling.

Authors: Abbati, G; Tosi, A; Osborne, M; Flaxman, S

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Proceedings of Machine Learning Research
• Host title: Twenty-First International Conference on Artificial Intelligence and Statistics, Apr 9, 2018 - Apr 11, 2018, Playa Blanca, Spain
• Journal: International Conference on Artificial Intelligence and Statistics
• Volume: 84
• Pages: 226-234
• Publication date: 2018-04-09
• Acceptance date: 2017-12-22
• ISSN: 1938-7228