Global convergence of deep networks with one wide layer followed by pyramidal topology
Nguyen Q, Mondelli M. 2020. Global convergence of deep networks with one wide layer followed by pyramidal topology. 34th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems vol. 33, 11961–11972.
Download (ext.)
Conference Paper
| Published
| English
Author
Nguyen, Quynh;
Mondelli, MarcoISTA 
Department
Abstract
Recent works have shown that gradient descent can find a global minimum for over-parameterized neural networks where the widths of all the hidden layers scale polynomially with N (N being the number of training samples). In this paper, we prove that, for deep networks, a single layer of width N following the input layer suffices to ensure a similar guarantee. In particular, all the remaining layers are allowed to have constant widths, and form a pyramidal topology. We show an application of our result to the widely used LeCun’s initialization and obtain an over-parameterization requirement for the single wide layer of order N2.
Publishing Year
Date Published
2020-07-07
Proceedings Title
34th Conference on Neural Information Processing Systems
Acknowledgement
The authors would like to thank Jan Maas, Mahdi Soltanolkotabi, and Daniel Soudry for the helpful discussions, Marius Kloft, Matthias Hein and Quoc Dinh Tran for proofreading portions of a prior version of this paper, and James Martens for a clarification concerning LeCun’s initialization. M. Mondelli was partially supported by the 2019 Lopez-Loreta Prize. Q. Nguyen was partially supported by the German Research Foundation (DFG) award KL 2698/2-1.
Volume
33
Page
11961–11972
Conference
NeurIPS: Neural Information Processing Systems
Conference Location
Vancouver, Canada
Conference Date
2020-12-06 – 2020-12-12
IST-REx-ID
Cite this
Nguyen Q, Mondelli M. Global convergence of deep networks with one wide layer followed by pyramidal topology. In: 34th Conference on Neural Information Processing Systems. Vol 33. Curran Associates; 2020:11961–11972.
Nguyen, Q., & Mondelli, M. (2020). Global convergence of deep networks with one wide layer followed by pyramidal topology. In 34th Conference on Neural Information Processing Systems (Vol. 33, pp. 11961–11972). Vancouver, Canada: Curran Associates.
Nguyen, Quynh, and Marco Mondelli. “Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology.” In 34th Conference on Neural Information Processing Systems, 33:11961–11972. Curran Associates, 2020.
Q. Nguyen and M. Mondelli, “Global convergence of deep networks with one wide layer followed by pyramidal topology,” in 34th Conference on Neural Information Processing Systems, Vancouver, Canada, 2020, vol. 33, pp. 11961–11972.
Nguyen Q, Mondelli M. 2020. Global convergence of deep networks with one wide layer followed by pyramidal topology. 34th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems vol. 33, 11961–11972.
Nguyen, Quynh, and Marco Mondelli. “Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology.” 34th Conference on Neural Information Processing Systems, vol. 33, Curran Associates, 2020, pp. 11961–11972.
All files available under the following license(s):
Copyright Statement:
This Item is protected by copyright and/or related rights. [...]
Link(s) to Main File(s)
Access Level
Open Access
Export
Marked PublicationsOpen Data ISTA Research Explorer
Sources
arXiv 2002.07867
Google Scholar