Abstract:  The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. In this talk I will discuss some general mathematical principles allowing for efficient optimization in over-parameterized non-linear systems, a setting that includes deep neural networks. I will discuss that optimization problems corresponding to these systems are not convex, even locally, but instead satisfy the Polyak-Lojasiewicz (PL) condition on most of the parameter space, allowing for efficient optimization by gradient descent or SGD. I will connect the PL condition of these systems to the condition number associated to the tangent kernel and show how a non-linear theory for those systems parallels  classical analyses of over-parameterized linear equations. As a separate related developement, I will discuss a  perspective on the remarkable recently discovered phenomenon of transition  to linearity (constancy of NTK) in certain classes of large neural networks. I will show how  this transition to linearity results from the scaling of the Hessian with the size of the network controlled by certain fuctional norms.  Combining these ideas, I will show how the transition to linearity can be used to demonstrate the PL condition and convergence for a general class of wide neural networks. Finally I will comment systems which are ”almost” over-parameterized, which appears to be common in practice.  

Based on joint work with Chaoyue Liu and Libin Zhu

The Putnam seminar meets every Wednesday from 4 to 5:30 in LOM 214.  As always, everyone is warmly welcomed to come to hang out, learn more cool math, and meet folks.  The seminar is casual, and folks can come and go as they like.  See Pat Devlin’s webpage (and/or contact him) for more information.  Folks can sign up for the mailing list here: https://forms.gle/nYPx72KVJxJcgLha8

See Algebra and Geometry lecture series (yale.edu)

Abstract:

The topology of three-manifolds with positive scalar curvature has been (mostly) known since the solution of the Poincare conjecture by Perelman. Indeed, they consist of connected sums of spherical space forms and S^2 x S^1’s. In spite of this, their “shape” remains unknown and mysterious. Since a lower bound of scalar curvature can be preserved by a codimension two surgery, one may wonder about a description of the shape of such manifolds based on a codimension two data (in this case, 1-dimensional manifolds).   In this talk, I will show results from a recent collaboration with Y. Liokumovich elucidating this question for closed three-manifolds.