Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization

Francis Bach 1 Lénaïc Chizat 2
1 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, CNRS - Centre National de la Recherche Scientifique, Inria de Paris
Abstract : Many supervised machine learning methods are naturally cast as optimization problems. For prediction models which are linear in their parameters, this often leads to convex problems for which many mathematical guarantees exist. Models which are non-linear in their parameters such as neural networks lead to non-convex optimization problems for which guarantees are harder to obtain. In this review paper, we consider two-layer neural networks with homogeneous activation functions where the number of hidden neurons tends to infinity, and show how qualitative convergence guarantees may be derived.
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03379011
Contributor : Francis Bach Connect in order to contact the contributor
Submitted on : Friday, October 15, 2021 - 2:27:00 PM
Last modification on : Wednesday, November 17, 2021 - 12:33:30 PM

Files

ICM-Bach-Chizat-HAL.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03379011, version 1
  • ARXIV : 2110.08084

Collections

Citation

Francis Bach, Lénaïc Chizat. Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization. 2021. ⟨hal-03379011⟩

Share

Metrics

Record views

51

Files downloads

114