Student: Majid Jahani
Project: Sampled Quasi-Newton Methods for Deep Learning
View: Research Poster (PDF) | Presentation (YouTube)
Department: Industrial and Systems Engineering
Advisor: Martin Takac
Abstract
We present two sampled quasi-Newton methods: sampled LBFGS and sampled LSR1. Contrary to the classical variants of these methods that sequentially build (inverse) Hessian approximations as the optimization progresses, our proposed methods sample points randomly around the current iterate to produce these approximations. As a result, the approximations constructed make use of more reliable (recent and local) information, and do not depend on past information that could be significantly stale. Our proposed algorithms are efficient in terms of accessed data points (epochs) and have enough concurrency to take advantage of distributed computing environments. We provide convergence guarantees for our proposed methods. Numerical tests on a toy classification problem and on popular benchmarking neural network training tasks reveal that the methods outperform their classical variants and are competitive with first-order methods such as ADAM.
About Majid Jahani
Majid Jahani received his B.S. degree in Applied Mathematics from Shahid Beheshti University, Tehran, Iran. He also received his M.S. degree in Applied Mathematics from Amirkabir University of Technology, Tehran, Iran. Majid is a Ph.D. Candidate in the Department of Industrial and Systems Engineering at Lehigh University. He is working with Professor Martin Takáč on the areas of large scale convex and nonconvex optimization, design and analysis of algorithms for machine learning.