Journal article
Automatic differentiation in machine learning: A survey
- Abstract:
- Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply “autodiff”, is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of numeric functions expressed as computer programs. AD is a small but established field with applications in areas including computational fluid dynamics, atmospheric sciences, and engineering design optimization. Until very recently, the fields of machine learning and AD have largely been unaware of each other and, in some cases, have independently discovered each other’s results. Despite its relevance, general-purpose AD has been missing from the machine learning toolbox, a situation slowly changing with its ongoing adoption under the names “dynamic computational graphs” and “differentiable programming”. We survey the intersection of AD and machine learning, cover applications where AD has direct relevance, and address the main implementation techniques. By precisely defining the main differentiation techniques and their interrelationships, we aim to bring clarity to the usage of the terms “autodiff”, “automatic differentiation”, and “symbolic differentiation” as these are encountered more and more in machine learning settings.
- Publication status:
- Published
- Peer review status:
- Peer reviewed
Actions
Access Document
- Files:
-
-
(Preview, Version of record, pdf, 591.6KB, Terms of use)
-
- Publication website:
- http://www.jmlr.org/papers/volume18/17-468/17-468.pdf
Authors
- Publisher:
- Journal of Machine Learning Research
- Journal:
- Journal of Machine Learning Research More from this journal
- Volume:
- 18
- Article number:
- 153
- Publication date:
- 2018-04-01
- Acceptance date:
- 2017-08-01
- EISSN:
-
1533-7928
- ISSN:
-
1532-4435
- Language:
-
English
- Keywords:
- Pubs id:
-
853503
- Local pid:
-
pubs:853503
- Deposit date:
-
2020-06-25
- ARK identifier:
Terms of use
- Copyright holder:
- Baydin et al.
- Copyright date:
- 2018
- Rights statement:
- © 2018 Atılım Güneş Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided at http://jmlr.org/papers/v18/17-468.html.
- Licence:
- CC Attribution (CC BY)
If you are the owner of this record, you can report an update to it here: Report update to this record