Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
Much as replacing hand-designed features with learned functions has
revolutionized how we solve perceptual tasks, we believe learned algorithms
will transform how we train models. In this work we focus on general-purpose
learned optimizers capable...
Jascha
67d ago
Modern deep learning is a story of learned features outperforming (then replacing!) hand-designed algorithms. But we still use hand designed loss functions and optimizers. Here is a big step towards learned optimizers outperforming existing optimizers: arxiv.org/abs/2009.11243 pic.twitter.com/Pg4ehwoEEg
199
We have a new paper on learned optimizers! We used thousands of tasks (and a lot of compute 😬) to train general purpose learned optimizers that perform well on never-before-seen tasks, and can even train new versions of themselves.
arxiv.org/abs/2009.11243
1/8
Neat to see a mention of AGI in the 'broader impacts' section of @GoogleAI 's paper on Learned Optimizers.
Writing up paper for Import AI - learning to learn has become learning how to learn tools that learn how to learn efficient training. arxiv.org/abs/2009.11243 pic.twitter.com/l0lcRbYXMH
3. Foundational research seems to be progressing very well and not slowing down (see e.g. 'A new backpropagation-free deep learning algorithm' arxiv.org/abs/2006.05964… or 'Learning to learn' arxiv.org/abs/2009.11243).