Training more effective learned optimizers, and using them to train themselves (Paper Explained)

#ai #research #optimization Optimization is still the domain of hand-crafted, simple algorithms. An ML engineer not only has to pick a suitable one for their problem but also often do grid-search over various hyper-parameters. This paper proposes to learn a single, unified optimization algorithm, given not by an equation, but by an LSTM-based neural network, to act as an optimizer for any deep learning problem, and ultimately to optimize itself. OUTLINE: 0:00 - Intro & Outline 2:20 - From Hand-Crafted to