Conference item icon

Conference item

Stochastic expectation maximization with variance reduction

Abstract:
Expectation-Maximization (EM) is a popular tool for learning latent variable models, but the vanilla batch EM does not scale to large data sets because the whole data set is needed at every E-step. Stochastic Expectation Maximization (sEM) reduces the cost of E-step by stochastic approximation. However, sEM has a slower asymptotic convergence rate than batch EM, and requires a decreasing sequence of step sizes, which is difficult to tune. In this paper, we propose a variance reduced stochastic EM (sEM-vr) algorithm inspired by variance reduced stochastic gradient descent algorithms. We show that sEM-vr has the same exponential asymptotic convergence rate as batch EM. Moreover, sEM-vr only requires a constant step size to achieve this rate, which alleviates the burden of parameter tuning. We compare sEM-vr with batch EM, sEM and other algorithms on Gaussian mixture models and probabilistic latent semantic analysis, and sEM-vr converges significantly faster than these baselines.
Publication status:
Published
Peer review status:
Peer reviewed

Actions

Access Document

Authors

More by this author
Institution:
University of Oxford
Division:
MPLS
Department:
Statistics
Oxford college:
University College
Role:
Author
ORCID:
0000-0001-5365-6933


Publisher:
Massachusetts Institute of Technology Press
Host title:
Neural Information Processing Systems
Journal:
Neural Information Processing Systems More from this journal
Publication date:
2018-12-02
Acceptance date:
2018-12-02
Event location:
Montréal, Canada
ISSN:
1049-5258


Pubs id:
pubs:969052
UUID:
uuid:e1b7d53c-8ac5-42cc-a12d-47abdd363c8c
Local pid:
pubs:969052
Source identifiers:
969052
Deposit date:
2019-02-06
ARK identifier:

Terms of use


Views and Downloads

Views and downloads will return soon






If you are the owner of this record, you can report an update to it here: Report update to this record

TO TOP