All Segments
Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU)
46 segments available
The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below. Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days: 1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk 2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M 3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw 4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w 5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8 6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I 7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo 8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI 9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs 10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc 11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4 12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc Full Day Live Streams: Day 1: https://youtu.be/eyovmAtoUx0 Day 2: https://youtu.be/9dXiAecyJrY Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.
Segments Timeline
Introduction to Unsupervised Learning
Ruslan Salakhutdinov introduces the concept of unsupervised learning, highlighting its significance in the context of the growing volume of unlabeled data across various domains such as images, speech, and social networks. He emphasizes the need for statistical models that can discover structures in data without supervision, setting the stage for a deeper exploration of deep learning frameworks.
"Sound is good. Okay, great. So, I wanted to talk to you about unsupervised learning. And that's the area where there's been a lot of research. Um, but compared to supervised learning that you've heard..."
Exploring Unsupervised Techniques
In this segment, Salakhutdinov discusses various unsupervised learning techniques, including autoencoders and clustering methods. He categorizes models into non-probistic and probabilistic frameworks, explaining their applications and the importance of learning hierarchical representations of data. This foundational knowledge is crucial for understanding the advancements in deep learning.
"interested in um as well as how can we sort of apply these models across multiple different uh multiple different domains and one particular framework of doing that is is is the framework of deep lear..."
Generative Models and Their Importance
Salakhutdinov delves into generative models, discussing their role in unsupervised learning. He introduces concepts like generative adversarial networks and variational autoencoders, explaining how these models differ from traditional approaches by not requiring explicit density specifications. This segment highlights the evolution of generative modeling in deep learning.
"little bit, you know, it's it's not a full set. uh but the way that I typically think about these models is that there's a class of uh what I would call non-proistic models uh you know models like spa..."
Building Blocks of Unsupervised Learning
In this part of the talk, Salakhutdinov outlines the basic building blocks of unsupervised learning models, focusing on sparse coding and autoencoders. He explains how these models can be applied in industry and their significance in learning representations from unlabeled data, providing a clear understanding of their functionality and applications.
"these are generative adversarial networks, uh moment matching networks and this is sort of a slightly different class of models where you don't really have to specify what the density is. You just nee..."
Representation Learning in Deep Learning
Salakhutdinov discusses the concept of representation learning in deep learning, emphasizing the importance of finding the right features for data classification. He contrasts traditional feature engineering with the potential of unsupervised learning to automatically derive representations from large datasets, setting the stage for advancements in image and audio classification.
"first part but before I do that um let me just sort of give you a little bit of motivation. I know Andre's done a great job and and Richard sort of alluded to that as well. uh but the idea is you know..."
Understanding Sparse Coding
This segment focuses on sparse coding, a foundational model in unsupervised learning. Salakhutdinov explains its origins, objectives, and how it can be applied to various types of data, including images and speech signals. He illustrates how sparse coding can effectively represent data points as linear combinations of learned bases, enhancing feature extraction.
"work done in that space, but we're not there yet. So, so, so I wanted to sort of lower your expectations as as I show you some uh some of the results. Okay, sparse coding. Um this is one of the models..."
Optimization in Sparse Coding
Salakhutdinov elaborates on the optimization processes involved in sparse coding. He describes the reconstruction error and sparse penalty terms that guide the learning of dictionary bases, explaining how these concepts relate to practical applications in unsupervised learning. This segment provides insights into the mathematical foundations of sparse coding.
"And in general, oops. Uh um how do we how do we fit these models? Um well, if I give you uh a whole bunch of image patches, but these don't necessarily have to be image patches. This could be you know..."
Applications of Sparse Coding
In this segment, Salakhutdinov discusses the practical applications of sparse coding in various fields, including medical imaging and neuroscience. He highlights its effectiveness in feature representation and classification tasks, showcasing its relevance in both deep learning and traditional machine learning contexts.
"um so back in 2006 there was uh work done uh again at Stanford um uh that basically showed a very interesting result so if I give you an input like this and these are my learned bases remember these l..."
The Encoder-Decoder Framework
Salakhutdinov introduces the encoder-decoder framework, a key concept in autoencoders. He explains how this framework allows for the encoding of input data into a lower-dimensional representation and the subsequent decoding back to the original input. This segment emphasizes the importance of constraints in preventing trivial solutions in representation learning.
"with so uh what's the interpretation of the sparse coding Well, look, let's look at this equation again. And we can think of sparse coding as finding an overcomplete representation of your data. Right..."
Optimizing Autoencoders
In the final segment, Salakhutdinov discusses the optimization of autoencoders, detailing the processes involved in training these models. He highlights the significance of using non-linearities and constraints to enhance the learning of representations, providing a comprehensive overview of how autoencoders function within the broader context of unsupervised learning.
"obviously you need some form of constraints you need some of constraints to avoid learning an identity right because if you don't With these constraints, what you could do is just take your input, cop..."
Binary Features and Sparsity
Salakhutdinov discusses the extraction of binary features using autoencoders and introduces the concept of sparsity in latent representations. He explains how models like predictive sparse decomposition can enhance feature learning by imposing sparsity constraints.
"interesting uh sort of relationship between autoenccoders and pro and principal component analysis. Many of you have probably heard about PCA as a practitioner. You know, if you're dealing with large ..."
Stacking Autoencoders for Feature Learning
This segment covers the stacking of autoencoders to learn hierarchical features. Salakhutdinov explains how this approach can be beneficial when working with large amounts of unlabeled data, allowing for effective representation learning without extensive labeled datasets.
"model. If you're dealing with binary input, uh sometimes we're dealing with uh like amnest for example, again your encoder and decoder could use sigmoid nonlinearities. So given an input, you extract ..."
Early Examples of Nonlinear Autoencoders
Salakhutdinov presents early examples of nonlinear autoencoders from 2006, demonstrating how these models can be pre-trained and combined into deep architectures. He discusses the reconstruction of images and the advantages of using autoencoders for dimensionality reduction.
"perhaps at the top level if you're trying to solve a classification problem you can do that um or and this is sometimes known as a greedy uh greedy layer wise uh learning and this is sometimes useful ..."
Compression and Regularization in Autoencoders
In this segment, Salakhutdinov illustrates how autoencoders can serve as compression mechanisms. He discusses the regularization effects of autoencoders, such as filtering out noise in data, and provides examples of their application in image and text data.
"autoenccoder and back propagate through uh reconstruction loss. Right? One thing I want to point out is that uh here's one particular example. You know the top row I show you real faces. The second ro..."
Semantic Hashing for Efficient Data Retrieval
Salakhutdinov introduces semantic hashing, a technique for compressing data into binary representations for efficient searching. He explains the advantages of binary codes in memory storage and retrieval, particularly in the context of large datasets.
"model might think that that's just a noise. Um, and you know, if you're dealing with uh text type of data, uh this was done using a Reuters data set. You have about 800,000 uh stories. You take bag of..."
Generative Models and Their Applications
This segment focuses on generative models and their ability to learn distributions from data. Salakhutdinov provides examples of generating handwritten characters and discusses the challenges of generalizing to new images within high-dimensional spaces.
"volumes of data using binary representation. So you can you know takes a fraction of a millisecond to retrieve uh images from you know a set of millions and millions of images. Uh and and again this i..."
Fully Observed Models and Their Properties
Salakhutdinov discusses fully observed models in generative learning, emphasizing their tractable properties and the ability to compute probabilities. He introduces neural autoregressive density estimators and their applications in generating complex data.
"This is one of the older models but this is you know what the model believes Sanskrit should look like and I think that I've asked couple of people to say that is that does that really look like Sansk..."
Introduction to Restricted Boltzmann Machines
In this segment, Salakhutdinov introduces Restricted Boltzmann Machines (RBMs) as a class of models for learning latent structures. He explains the graphical model framework and how RBMs can represent dependencies between random variables.
"in general to um to answer one class of models uh is so-called fully observed models right there sort of been a stream of uh learning generative models that are tractable and they have very nice prope..."
Understanding Restricted Boltzmann Machines
In this segment, Salakhutdinov explains restricted Boltzmann machines (RBMs) and their role in learning latent structures in data. He describes the graphical model framework and how RBMs can generate data based on learned features. The discussion includes the significance of conditional probabilities and how these models relate to sparse coding, emphasizing their utility in information retrieval and image classification.
"restricted boss machines. So this is the class of models where we actually trying to learn some latent structure some latent representation. uh these models belong to the class of so-called graphical ..."
Learning Models through Maximum Likelihood
Salakhutdinov elaborates on the process of learning models using maximum likelihood estimation. He explains the objective of maximizing the probability of observing training images and the mathematical principles behind it. The segment highlights the challenges of computing certain terms in the likelihood function and introduces Monte Carlo sampling as a solution for approximating these computations.
"classifying images because you need to do it uh you need to do it fast. How do we learn these models? Let me just give you an intuition maybe a little bit of math behind uh how we learn these models. ..."
Monte Carlo Methods in Deep Learning
This segment focuses on the application of Monte Carlo methods in deep learning, particularly in approximating exponential sums. Salakhutdinov discusses the GIP sampling method and its role in inferring latent variables. He emphasizes the importance of efficient sampling techniques and introduces the contrastive divergence algorithm as a practical approach to training models without requiring infinite iterations.
"right and we can actually uh do it uh do it explicitly the second term is the difficult difficult one to compute right so it sort of requires you know summing over all possible configurations, all pos..."
Extending Models to Real-Valued Data
Salakhutdinov discusses how restricted Boltzmann machines can be adapted to handle real-valued data, such as images. He explains the modifications needed for the model's conditional probabilities and how these adaptations allow for the generation of realistic images. The segment also touches on the application of these models to count data, such as documents, and the insights gained from analyzing learned topics.
"obviously you know you will never run them you know infinite uh you don't have time to do that. So there's a very clever algorithm that uh a contrastive divergence algorithm that was developed by Hint..."
Applications of Latent Variable Models
In this segment, Salakhutdinov explores various applications of latent variable models, including collaborative filtering and topic modeling. He illustrates how these models can capture user preferences and identify underlying themes in documents. The discussion highlights the effectiveness of these models in revealing correlations and structures within data, showcasing their versatility across different domains.
"document. Right? So if you you know it's it's a bag of words uh representation and the conditional here is given by so-called softmax distribution much like what you've seen in in in uh in the previou..."
Product Models and Their Intuition
Salakhutdinov concludes with an explanation of product models in the context of unsupervised learning. He describes how these models can be viewed as products of distributions over hidden variables and their implications for modeling complex data. The segment emphasizes the advantages of product models over traditional mixture models, particularly in capturing local versus global representations in data.
"different kinds of modality, real value data, you can model count data, multinnomials and it's very easy to infer the states of the hidden variables. So that's given just the product of of logistic fu..."
Modeling Dependencies in Deep Learning
Salakhutdinov delves into the importance of modeling dependencies between hidden variables in deep learning. He introduces the concept of bottom-up and top-down influences in hierarchical models, explaining how these dependencies enhance the model's ability to capture complex data structures.
"that's the differences uh uh between these these two classes of models. Important to know about them. Now let me jump uh and quickly tell you a little bit of inspiration behind what what can we build ..."
Learning with Maximum Likelihood Estimation
This segment focuses on the maximum likelihood estimation in unsupervised learning models. Salakhutdinov explains how to differentiate models with respect to parameters and the challenges of approximating these models, emphasizing the importance of matching model predictions with observed data.
"there has to be some consensus in the model to say ah yes what I'm seeing in the image and what my model believes the overall structure should be should be in agreement. Um right and so in this case o..."
Real vs. Simulated Data: A Challenge
Salakhutdinov presents a challenge to the audience by showing real and simulated handwritten characters. He discusses the differences in quality and diversity between real data and data generated by models, highlighting the difficulties in accurately simulating complex data distributions.
"particular equation. Uh right, but the first term is no longer factorial. So it's you know you have to do some approximation with these models. But let me give you notation what what each term is doin..."
Applications of Unsupervised Learning Models
In this segment, Salakhutdinov explores various applications of unsupervised learning models, including pattern completion and generating 3D objects. He discusses the challenges and successes of these models in different contexts, emphasizing their potential in real-world scenarios.
"techniques for doing that there's a class of uh algorithms called variational inference something that's called stocastic approximation which is multicolor based inference I'm not going to go into the..."
Multimodal Learning: Bridging Images and Text
Salakhutdinov discusses the challenges of multimodal learning, where images and text are combined. He explains the difficulties in learning cross-modal features and the importance of hierarchical models in achieving richer representations of data from different modalities.
"know they're much better than some of the existing techniques. This is another example you know trying to generate 3D objects. This is sort of a to data sets and later on I'll show you some you know b..."
Leveraging Unlabeled Data for Improved Performance
In this segment, Salakhutdinov highlights the benefits of using unlabeled data to enhance model performance. He shares insights from experiments showing how pre-training with unlabeled data can lead to significant improvements in tasks involving labeled images and noisy tags.
"Perhaps a bigger challenge is that uh a lot of times we see data that's very noisy, right? Sometimes it's just non-existent given an image there is no text or if you look at the first image you know a..."
Exploring Image Representations
In this segment, Salakhutdinov illustrates the concept of manipulating image representations using vector arithmetic. He provides examples of how subtracting and adding representations can yield interesting results, such as generating images of kittens in boxes or balls. This playful exploration showcases the creative potential of unsupervised learning in generating new visual concepts.
"us it was helping us a lot. Uh and then once you get into these sort of representations dealing with text and images this is one particular thing you can do and I think Richard pointed out you know wh..."
Introduction to Helmholtz Machines
Salakhutdinov introduces Helmholtz machines and variational autoencoders, explaining their architectures and the generative processes involved. He discusses the historical context of these models, their development by notable researchers, and the challenges faced in their implementation. This segment sets the stage for understanding how these models can effectively learn from data.
"they don't really mean much because we're not specifically optimizing for those things. Right now, let me u um uh spend some time also talking about another class of models. These are known as Helmho ..."
Variational Learning Explained
This segment delves into the principles of variational learning, focusing on maximizing the probability of observed data through variational bounds. Salakhutdinov explains the significance of Jensen's inequality in optimizing these models and the challenges associated with traditional likelihood objectives. The discussion highlights the innovative approaches that have emerged in the field of unsupervised learning.
"uh the semantics is a little bit different between these two models. So what is a variational autoenccoder? Variational encoder is is a Helmhol machine. It defines a generative process in terms of sam..."
The Reparameterization Trick
Salakhutdinov elaborates on the reparameterization trick, a breakthrough in variational autoencoders that allows for efficient gradient computation. He explains how this technique transforms stochastic systems into deterministic mappings, facilitating backpropagation and improving model training. This segment is crucial for understanding modern advancements in deep learning methodologies.
"uh for those of you who haven't seen variational learning and how it's done you know, but this one little trick this one little so-called Jensen's inequality actually allows you to solve a lot of prob..."
Generating Images from Captions
In this segment, Salakhutdinov presents a motivating example of using generative models to create images based on textual descriptions. He discusses the importance of stochasticity in generating diverse outputs and the challenges of accurately representing complex scenes. This exploration underscores the potential applications of unsupervised learning in bridging language and visual content.
"turns out that you can collapse these complicated models effectively into autoenccoders, right? And we know how to deal with autoenccoders. We can back propagate through the entire uh through the enti..."
Romantic Text Generation with Neural Networks
Salakhutdinov introduces a generative model trained on romance novels that generates text based on image inputs. He describes how the model attempts to create romantic narratives, highlighting the current limitations in semantic understanding. This segment illustrates the potential and challenges of using neural networks for creative text generation.
"toilet seat sits open in the bathroom. Right? I don't know if you can see toilet seats here, maybe. Uh, but you can say toilet seats, it's open in the grass field. That was a little bit better. Uh, at..."
One-Shot Learning in Generative Models
This segment focuses on one-shot learning in generative models, where the model learns to generate characters based on a single example. Salakhutdinov discusses experiments comparing human and machine-generated drawings, revealing the competitive capabilities of generative models in a defined domain. The segment highlights the advancements in generative modeling and its implications for creativity.
"much better representation for that sentence than than this one. Um here's another here's another sort of interesting model which is a u a model where you're trying to build a recurrent neural network..."
Introduction to Generative Adversarial Networks (GANs)
Salakhutdinov provides an overview of Generative Adversarial Networks (GANs), explaining the game-theoretic framework that underpins their operation. He describes the roles of the generator and discriminator in the GAN setup, emphasizing the innovative approach of learning through competition. This segment sets the stage for understanding the mechanics and applications of GANs in deep learning.
"structure. Here's another last example that I want to show you. Um this was done uh in the case of oneshot learning which is can you build generative model of characters. Right? That's a very defined ..."
Training GANs: The Minimax Objective
In this segment, Salakhutdinov delves into the training process of GANs, explaining the minimax objective function that guides the learning of both the generator and discriminator. He discusses the balance between generating realistic samples and accurately classifying them, highlighting the challenges and strategies involved in training effective GANs.
"Uh so in sort of this kind of small carved domain we can actually compete with people uh you know trying to generate these uh the uh these characters. Now let me step back a little bit and and and tel..."
Deconvolutional GANs and Their Applications
Salakhutdinov showcases the architecture of deconvolutional GANs, which generate high-dimensional images from random Gaussian codes. He presents examples of generated images from various datasets, illustrating the impressive capabilities of these models in producing realistic visuals. This segment emphasizes the advancements in image generation through GANs and their potential applications.
"goal as a generator would be to generate really nice looking digits so that the discriminator wouldn't be able to tell the difference between you know simulated and the real. Right? That's the key ide..."
Evaluating Generative Models
In this concluding segment, Salakhutdinov discusses the challenges of evaluating generative models, particularly in assessing their ability to learn and represent the distribution of images. He raises important questions about the coherence and realism of generated outputs, emphasizing the need for robust evaluation metrics in the field of generative modeling.
"uh negative and I really want to maximize it. And then discriminator says well anytime I generate a sample whatever samples comes out from my generator um you know I want to classify it as being fake ..."
The Future of Unsupervised Learning
Salakhutdinov summarizes the advancements in deep unsupervised learning models, highlighting their improvements over traditional methods. He discusses the excitement and potential of these models in various application domains, reinforcing the importance of ongoing research and development in the field of deep learning.
"follow-up work. Uh, these are samples um from the CFAR data set. So, here you're seeing training samples and here you're seeing samples uh generated from the model which is again very impressive. If y..."
GANs vs. Variational Autoencoders
Salakhutdinov contrasts Generative Adversarial Networks (GANs) with Variational Autoencoders (VAEs), discussing why GANs often produce clearer images. He explains the implications of loss functions in image generation and how GANs prioritize realism over precision, leading to different outcomes in generated images. This segment provides insights into the strengths and weaknesses of both approaches.
"you questions to the mics. Uh so oh sorry uh so uh as a basian guy I'm pretty depressed by the fact that gang can generate clearer image than the variational autoenccoder. So my question is do you thi..."
PCA Pre-processing in Variational Autoencoders
In this segment, Salakhutdinov addresses the use of PCA for pre-processing data in training Variational Autoencoders, questioning its necessity. He shares his experiences with pre-processing techniques and discusses the variability in practices within the field, providing a nuanced view of data preparation for deep learning models.
"there's been some work on you know trying to sharpen the images which is you're using variational quarters to generate you the globally coherent in C and then you're using uh generative adversarial ne..."
Challenges with Restricted Boltzmann Machines
Salakhutdinov explores the literature surrounding Restricted Boltzmann Machines (RBMs) and contrasts it with that of easing models. He discusses the difficulties in estimating the partition function for RBMs and the limitations of variational methods in this context, shedding light on the complexities of learning in probabilistic models.
"encoders any more question yes there's one queries regarding uh binary RBMs uh so if you look at u uh the literature for uh let us say uh estimation of the partition function for easing models right y..."
Using Autoencoders for Semantic Hashing
In this segment, Salakhutdinov discusses the application of autoencoders for semantic hashing in text data. He emphasizes the importance of representation techniques, such as word embeddings and GRUs, for effective learning. This segment highlights the flexibility of input representations and their impact on the efficiency of searching in semantic spaces.
"well. But it's a good question. Hello. Uh my question is about uh using auto coder with uh to get semantic hash uh especially in text. Do we need to any uh special representation text representation l..."