Reporter: Li Jian
Reporting location: Tencent Conference 256-907-699
Bet365 Online Live Dealer time:2022Year11month3Thursday afternoon4:00
Bet365 Online Live Dealer title:Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability
Bet365 Online Live Dealer Introduction:
Recent findings demonstrate that modern neural networks trained by full-batch gradient descent typically enter a regime called Edge of Stability (EOS). In this regime, the sharpness, i.e., the maximum Hessian eigenvalue, first increases to the value 2/(step size) (the progressive sharpening phase) and then oscillates around this value (the EOS phase). Such behavior is inconsistent with several widespread presumptions and traditional wisdom in the field of optimization and raises several intriguing questions about neural network training. This paper aims to analyze the GD dynamics and the sharpness along the optimization trajectory. Our analysis naturally divides the GD trajectory into four phases depending on the change in the sharpness value. We empirically identify the norm of output layer weight as an interesting indicator of the sharpness dynamics. Based on this empirical observation, we attempt to theoretically and empirically explain the dynamics of various key quantities that lead to the change of the sharpness in each phase of EOS.
About the speaker:
Li Jian is currently an associate professor at the Institute of Interdisciplinary Information at Tsinghua Bet365 Online Live Dealer，Doctoral Supervisor。He obtained his bachelor’s degree from Sun Yat-sen Bet365 Online Live Dealer and his master’s degree from Fudan Bet365 Online Live Dealer，Ph.D. graduate from Bet365 Online Live Dealer of Maryland。His research interests mainly include algorithm design and analysis，Machine Learning，Database，Fintech。He has published in mainstream international conferences and magazines100More papers, etc., and obtainedVLDB 2009andESA 2010's Best Paper Award,ICDT 2017Best Newcomer Award, Tsinghua221Basic Research Young Talent Support Program，Ministry of Education New Century Talent Support Plan，National Natural Science Foundation of China Outstanding Youth Fund。He presided over and participated in many scientific research projects，Including Natural Science Foundation of China projects，China-Israel International Cooperation Project，Youth973Plans, etc.，And multiple enterprise-level cooperation projects，Including Ant Financial、Huatai Securities、Yifangda、Microsoft、Baidu、Didi, etc.。
