Sander Dieleman
Research Scientist at DeepMind. I tweet about deep learning (research + software), music, generative models, Kaggle, Lasagne ()
Last synced from Twitter 91d ago
Learning Invariances in Neural Networks
arXiv.org 96d 1 tweets
Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present in the data, or to what extent a model should be invariant to a given...
Caffe | Local Response Normalization (LRN)
Caffe.berkeleyvision.org 107d 1 tweets
Sander Dieleman I believe the original AlexNet model also used "local response normalisation" in the channel direction: Not quite the same thing though, as this was done to induce lateral inhibition between units, not to improve optimisation. 107d
WaveGrad: Estimating Gradients for Waveform Generation
OpenReview 114d 1 tweets
This paper introduces WaveGrad, a conditional model for waveform generation which estimates gradients of the data density. The model is built on prior work on score matching and diffusion...
Sander Dieleman Oh, and two more submissions that apply these ideas to waveform generation 🗣️🎶 WaveGrad: Estimating Gradients for Waveform Generation DiffWave: A Versatile Diffusion Model for Audio Synthesis (3/3) 114d
DiffWave: A Versatile Diffusion Model for Audio Synthesis
OpenReview 114d 1 tweets
In this work, we propose DiffWave, a versatile Diffusion probabilistic model for conditional and unconditional Waveform generation. The model is non-autoregressive, and converts the white noise...
Sander Dieleman Oh, and two more submissions that apply these ideas to waveform generation 🗣️🎶 WaveGrad: Estimating Gradients for Waveform Generation DiffWave: A Versatile Diffusion Model for Audio Synthesis (3/3) 114d
Improved Autoregressive Modeling with Distribution Smoothing
OpenReview 114d 1 tweets
While autoregressive models excel at image compression, their sample quality is often lacking. Inspired by randomized smoothing for adversarial defense, we incorporate randomized smoothing...
Sander Dieleman Improved AR Modeling with Distribution Smoothing Improved DDPMs Score-Based Generative Modeling through SDEs Learning Energy-Based Models by Diffusion Recovery Likelihood (2/3) 114d
Improved Denoising Diffusion Probabilistic Models
OpenReview 114d 1 tweets
We explore denoising diffusion probabilistic models, a class of generative models which have recently been shown to produce excellent samples in the image and audio domains. While these models...
Sander Dieleman Improved AR Modeling with Distribution Smoothing Improved DDPMs Score-Based Generative Modeling through SDEs Learning Energy-Based Models by Diffusion Recovery Likelihood (2/3) 114d
Score-Based Generative Modeling through Stochastic Differential...
OpenReview 114d 1 tweets
Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a...
Sander Dieleman Improved AR Modeling with Distribution Smoothing Improved DDPMs Score-Based Generative Modeling through SDEs Learning Energy-Based Models by Diffusion Recovery Likelihood (2/3) 114d
Learning Energy-Based Models by Diffusion Recovery Likelihood
OpenReview 114d 1 tweets
While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging. Inspired by recent progress on diffusion...
Sander Dieleman Improved AR Modeling with Distribution Smoothing Improved DDPMs Score-Based Generative Modeling through SDEs Learning Energy-Based Models by Diffusion Recovery Likelihood (2/3) 114d
ermongroup/ncsn
GitHub 114d 1 tweets
Noise Conditional Score Networks. Contribute to ermongroup/ncsn development by creating an account on GitHub.
4
Sander Dieleman Very excited about the renewed focus on iterative refinement as a powerful tool for generative modelling! Here are a few relevant ICLR 2021 submissions: (image credit: ) (1/3) 114d
Job detail
Deepmind 526d 22 tweets
Careers at DeepMind
Rotation-invariant convolutional neural networks for galaxy morphology prediction
arXiv.org 146d 1 tweets
Measuring the morphological parameters of galaxies is a key requirement for studying their formation and evolution. Surveys such as the Sloan Digital Sky Survey (SDSS) have resulted in the availability of very large collections of images, which have...
Sander Dieleman I wish I'd been more familiar with these terms when I wrote my first paper on the topic (). In fact, that model was neither invariant nor equivariant to rotation 😬 although it did *learn* to be more invariant to rotation than vanilla CNNs. 146d
Musings on typicality
Sander Dieleman - Sander Dieleman 148d 1 tweets
A summary of my current thoughts on typicality, and its relevance to likelihood-based generative models.
6
Sander Dieleman New blog post, in which I wax lyrical about typicality and the curse of dimensionality: I tweeted about this concept a while back, but it turns out I have more to say on the topic. It's a bit more speculative than what I usually write, hope you like it! 148d
34th Deep Learning Meetup in Vienna (virtual edition)
Meetup 162d 2 tweets
Thu, 20 Aug 2020, 7:00 pm: Dear Deep Learners,Welcome back to the Vienna Deep Learning Meetup talk series after a long break! We will start with a stellar contribution by Sander Dieleman from Google D
Sander Dieleman I tried to use the "schedule tweet" feature for this one, but it looks like there are still some issues with it! The preview was supposed to be for this link 162d
Sander Dieleman On Thursday Aug 20, I'm speaking about generating music in the waveform domain at the Vienna deep learning meetup! Virtually of course, from my desk in London 🙂 Sign up to attend: I'll cover most of plus some recent developments! 162d
Machine Learning of Sets
Akosiorek.github.io 168d 2 tweets
In machine learning, we typically work with input pairs (x, y), and we try to figure out how x and y depend on each other.To do so, we gather many such pairs...
A Spectral Energy Distance for Parallel Speech Synthesis
arXiv.org 175d 2 tweets
Speech synthesis is an important practical generative modeling problem that has seen great progress over the last few years, with likelihood-based autoregressive neural models now outperforming traditional concatenative systems. A downside of such...
deepmind/dm-haiku
GitHub 176d 1 tweets
JAX-based neural network library. Contribute to deepmind/dm-haiku development by creating an account on GitHub.
Sander Dieleman JAX users might appreciate this particularly poetic bit of code from the Haiku library: a function transformation transformation 🤯 𝚓𝚒𝚝 and 𝚛𝚎𝚖𝚊𝚝 transform Python functions -- this transforms 𝚓𝚒𝚝 and 𝚛𝚎𝚖𝚊𝚝 to be aware of Haiku state. 176d
BPE-Dropout: Simple and Effective Subword Regularization
arXiv.org 456d 3 tweets
Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which keeps the most frequent words intact while splitting the rare ones...
Sander Dieleman I quite like the idea of BPE-dropout ( randomly skip some merge steps in BPE during training, so the model is forced to learn to understand the compositionality of tokens. Although I suppose it feels a bit like a stop-gap solution. 183d
Jan Van Balen
Jvbalen.github.io 201d 1 tweets
Jan Van Balen : Personal website
HiFiC - High-Fidelity Generative Image Compression
Hific.github.io 218d 3 tweets
Awesome image compression at lower bitrates than ever before!
A Recipe for Training Neural Networks
Karpathy.github.io 643d 13 tweets
Musings of a Computer Scientist.