Journal article icon

Journal article

Universal characteristics of deep neural network loss surfaces from random matrix theory

Abstract:

This paper considers several aspects of random matrix universality in deep neural networks (DNNs). Motivated by recent experimental work, we use universal properties of random matrices related to local statistics to derive practical implications for DNNs based on a realistic model of their Hessians. In particular we derive universal aspects of outliers in the spectra of deep neural networks and demonstrate the important role of random matrix local laws in popular pre-conditioning gradient des...

Expand abstract
Publication status:
Published
Peer review status:
Peer reviewed

Actions


Access Document


Files:
Publisher copy:
10.1088/1751-8121/aca7f5

Authors


More by this author
Institution:
University of Oxford
Division:
MPLS
Department:
Mathematical Institute
Role:
Author
Publisher:
IOP Publishing
Journal:
Journal of Physics A: Mathematical and Theoretical More from this journal
Volume:
55
Issue:
49
Article number:
494002
Publication date:
2022-12-16
Acceptance date:
2022-12-02
DOI:
EISSN:
1751-8121
ISSN:
1751-8113
Language:
English
Keywords:
Pubs id:
1311204
Local pid:
pubs:1311204
Deposit date:
2022-12-04

Terms of use


Views and Downloads






If you are the owner of this record, you can report an update to it here: Report update to this record

TO TOP