报告人：    吴方向 教授 

报告地点：永利官网长安校区博物馆附楼三层报告厅S314

报告时间：2024年4月7日 15:30

主办单位：永利yl23411

报告题目：From Numerical Integration to Determination of Learning Rates

摘要: The learning rate is a critical hyperparameter for deep learning tasks since it determines the extent to which the model parameters are updated during the learning course. However, the determination of learning rates typically depends on empirical judgment, which may not result in satisfactory outcomes without intensive try-and-error experiments. In this presentation, I first talk about numerical integration which stimulated our research idea. Then, after reviewing the existing popular learning rates, I present a novel learning rate adaptation scheme called QLABGrad. Without any user-specified hyperparameter, QLABGrad automatically determines the learning rate by optimizing the quadratic loss approximation-based (QLAB) function for a given gradient descent direction, where only one extra forward propagation is required. We theoretically proved the convergence of QLABGrad with a smooth Lipschitz condition on the loss function. Experiment results on multiple architectures, including MLP, CNN, and ResNet, on MNIST, CI FAR10, and ImageNet datasets, demonstrate that QLABGrad outperforms various competing schemes for deep learning.

报告人简介: Dr FangXiang Wu is currently a full professor in the Departments of Computer Science, Biomedical Engineering, and Mechanical Engineering at the University of Saskatchewan. His research interests include Artificial Intelligence, Machine/Deep Learning, Computational Biology, Health Informatics, Medical Image Analytics, and Complex Network Analytics. He has published more than 350 journal papers and more than 130 conference papers. His total google scholar citations are about 13500, h-index is 62 (dated in early April, 2024. He is among top 2% world’s scientists ranked by Stanford University. Dr Wu is serving as the editorial board member of several international journals (including IEEE TCBB, Neurocomputing, etc.) and as the guest editor of numerous international journals, and as the program committee chair or member of many international conferences.