All Segments
Sequence to Sequence Deep Learning (Quoc Le, Google)
54 segments available
The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below. Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days: 1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk 2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M 3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw 4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w 5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8 6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I 7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo 8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI 9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs 10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc 11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4 12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc Full Day Live Streams: Day 1: https://youtu.be/eyovmAtoUx0 Day 2: https://youtu.be/9dXiAecyJrY Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.
Segments Timeline
Introduction to Sequence to Sequence Learning
Quoc Le introduces the concept of sequence to sequence learning, explaining its relevance in automating email responses. He shares a personal anecdote about returning from vacation to a flooded inbox and discusses the potential for a system to automatically reply to simple yes/no questions in emails.
"eating that were divided in two parts so number one and we work with you and develop the sequence to sequence learning and then that's the second part I would I will place sequin to sequence in a broa..."
Tokenization and Normalization
In this segment, Quoc Le discusses the importance of tokenization and normalization in processing email content. He explains how to prepare input data by cleaning and structuring it into a format suitable for machine learning models, emphasizing the significance of feature representation.
"some inputs content so less for now let's ignore the the the on the author of the email and the subject but let's focus on the content so let's suppose that you gather some email and some input would ..."
Feature Representation and Logistic Regression
Quoc Le elaborates on feature representation by constructing a 2000-dimensional vector to represent email content. He introduces the concept of logistic regression as a method to classify emails into yes/no responses, detailing the process of updating weights based on email replies.
"step I would do would be to do feature representation so in this step what I'm going to do is the following I'm going to construct a 2,000 dimensional vector 2,000 represent the size of English vocabu..."
Stochastic Gradient Descent Explained
This segment covers the stochastic gradient descent algorithm used to optimize the weights in the logistic regression model. Quoc Le explains how to adjust probabilities based on the responses to improve the model's accuracy over time.
"categories you can think of it as a logistic regression problem now if anybody follow the gray cs2 10:29 class by andrew probably can formulate this very quickly but in a very short you the album come..."
Limitations of Bag-of-Words Representation
Quoc Le discusses the limitations of the bag-of-words representation in capturing the order of words in emails. He introduces recurrent networks as a solution to preserve ordering information, setting the stage for more advanced sequence modeling.
"to be answered as you know so the second probability okay so let's call those a p1 and p2 now so because to update I said to update the increase what does that mean what that means is that you find th..."
Understanding Recurrent Networks
In this segment, Quoc Le explains the architecture of recurrent networks and how they maintain information over sequences. He discusses the hidden state and how it evolves with each input, emphasizing the importance of this structure in sequence to sequence tasks.
"now now this there's a problem with this representation is that there's some information loss so somebody in the audience just said that the order of the words don't matter and that's that's true now ..."
Auto Differentiation in Neural Networks
Quoc Le introduces the concept of auto differentiation, which simplifies the process of computing gradients for training neural networks. He highlights tools like TensorFlow and PyTorch that facilitate this process, making it easier to implement complex models.
"said earlier now but you also have to update all the relevant matrices which is the matrix you the matrix a and some work vectors right so this is basically you have to compute the partial derivative ..."
The Big Picture of Sequence Learning
Quoc Le summarizes the key concepts discussed so far, including bag-of-words, recurrent networks, and auto differentiation. He emphasizes the importance of these elements in building effective sequence to sequence models for various applications.
"smaller model the matrix you yeah so the the work vectors the world vectors the number of work vectors that you use are the size of vocabulary right which is so you gonna tend to end up with 20,000 wo..."
Challenges in Email Response Generation
In this segment, Quoc Le addresses the challenges of generating meaningful responses to emails. He discusses the need for more nuanced replies beyond simple yes/no answers, highlighting the importance of context in conversational AI.
"okay you have a question oh that's also depends on how big your your training set and how big is your computer and so on right but usually if you use an N and if you used like a hidden state of a hund..."
Auto-Regressive Models for Sequence Generation
Quoc Le concludes by introducing auto-regressive models as a solution for generating variable-length outputs. He explains how these models can predict sequences token by token, using previous outputs as inputs for subsequent predictions, enhancing the flexibility of the model.
"auto reply which is what we've been working on so far but we can also work on user to do translation just like between English French you can do image captioning so input would be an a fixed like vect..."
Greedy Decoding Explained
This segment covers the greedy decoding algorithm used in sequence generation. Quoc Le explains how the model predicts the most likely next word based on the previous context and continues until a stopping condition is met.
"far or whenever it produced end then just stop there's a special token end yeah now okay so the so relevant architecture here would be the end code people also call the encoder as the what the recurre..."
Beam Search for Better Predictions
Quoc Le introduces beam search as an enhancement over greedy decoding. He explains how predicting multiple candidates at each step can lead to more accurate and diverse outputs, and discusses the computational implications of this approach.
"update the iron and encoder and decoder parameters so that you can increase the probability that Y of T is correct given all what you seen before which is your YT minus 1 YT minus 2 etc and also all t..."
Handling Vocabulary Limitations
In this segment, Quoc Le addresses the challenge of vocabulary limitations in sequence models. He discusses strategies for dealing with unknown words by mapping them to a special token and the implications for model training.
"number of steps and are different I have a question there okay yeah I'm gonna get to that in the next slide yeah okay all right so the question is a in practice how long would I go to for the RN I wou..."
Cost Function and Probability Maximization
Quoc Le explains the cost function used in training sequence models, focusing on maximizing the probability of correct predictions. He illustrates how the model conditions its predictions based on previous outputs.
"then you can ask questions so okay so how do you do prediction so this the first algorithm that can we can you can do is go greedy decoding okay in greedy decoding is for any incoming email X I'm goin..."
Scheduled Sampling for Robust Training
This segment introduces scheduled sampling as a technique to improve model robustness. Quoc Le explains how feeding the model its own predictions during training can help it learn to recover from mistakes.
"what is your reply any question to see the most complicated slide in my talk oh yeah yes so the question is what do you do with our vocabulary works now it turns out in this algorithm what you do is t..."
Embedding Users for Personalization
Quoc Le discusses methods for personalizing sequence models by embedding user information as vectors. He explores different strategies for integrating user embeddings into the model architecture.
"that you sample a random were YT here let's suppose that here I this is my input sofa or an input and I'm sample YT let's say T is equal to 2 so which means the work fine okay I'm at the work fine I w..."
Managing Beam Search Complexity
In this segment, Quoc Le addresses the complexity of beam search in sequence generation. He discusses strategies for truncating low-probability sequences to manage computational resources effectively.
"question is how do you personalize so well one way to do it is basically embed a user as a vector so let's suppose that you have a lot of users and you embed a user as a vector that's one way to do it..."
Batch Processing for Efficiency
Quoc Le explains the importance of batch processing in training sequence models. He discusses how batching can optimize GPU utilization and improve training speed.
"to be fast sometimes we have to Pat the input because we want to make use make sure that batch processing what's very well so you'd be bad but we paired with only like zero tokens okay yeah so let's s..."
Addressing Model Degradation
Quoc Le highlights the issue of model degradation during predictions. He introduces techniques to mitigate this problem, ensuring that the model can recover from poor predictions.
"the beginning or you can insert a tab at every prediction steps but my proposal is just predict put it at the beginning the simpler okay I have a question there yeah you yeah that's a very good questi..."
Learning Synonyms and Contextual Understanding
In this segment, Quoc Le discusses how models can learn synonyms and contextual relationships between words. He explores the effectiveness of different embedding techniques in capturing these relationships.
"yeah any question there's a question here okay yeah yeah yeah so in this algorithm yeah the question is how large is the the size of the Dakota well my answer is that try to be as large as possible bu..."
Real-World Applications of Sequence Models
Quoc Le shares insights into the practical applications of sequence-to-sequence models, particularly in email response generation. He discusses how these models are used in production systems to enhance user experience.
"what else to do the other approach is basically to train the world embeddings using water vac and then try to ask the model to regress to the world imbalance right so that's one way to address this is..."
Ensuring Model Termination
Quoc Le addresses the challenge of ensuring that sequence models terminate correctly. He discusses strategies for preventing infinite loops in predictions and the importance of training accuracy.
"you okay any question yeah I have a question here yeah there's no guarantees so the question is how do I guarantee that the the beam would terminate an end now there's no guarantee it can go on foreve..."
The Big Picture of Sequence Models
Quoc Le provides an overview of the architecture of sequence-to-sequence models, detailing the roles of encoders and decoders. He emphasizes the use of stochastic training methods and beam search decoding to enhance the model's predictive capabilities.
"sure that the model doesn't do that doesn't go on crazy right I have a question here that's very interesting yeah it just comes out because there's a lot of emails and if you invite someone there's mo..."
Attention Mechanism in Sequence Models
In this segment, Quoc Le introduces the attention mechanism, explaining how it allows models to focus on relevant parts of the input during prediction. He describes the mathematical process behind attention, including the computation of coefficients that guide the model's focus on specific input elements.
"question is do I do any post processing to correct the grammar of the beams in this algorithm we did not have to do it yeah okay I have another question so okay so the question how contextual so I wou..."
Challenges with Language Order
Quoc Le addresses the complexities of translating languages with different word orders, such as English to Chinese or Japanese. He explains how the attention mechanism adapts to these variations, allowing the model to learn appropriate weights for different input sequences.
"did I answer your question okay yeah we you can catch me up after the talk yeah oh yeah it ran down too so yeah slow question oh oh I see so the question is there's some some emails are not relevant f..."
Normalization and Vocabulary Issues
Quoc Le discusses the importance of normalization in training sequence-to-sequence models, particularly when dealing with small datasets. He emphasizes the need to standardize text inputs and addresses vocabulary challenges, including handling unknown words effectively.
"so in fact we have two algorithms so one hour with them this is to say yes or no to reply right and then after it passes the threshold there's an algorithm to run to produce the threshold so it's a co..."
Long Short-Term Memory (LSTM) Networks
In this segment, Quoc Le explains the architecture and functionality of Long Short-Term Memory (LSTM) networks. He highlights how LSTMs maintain information over time and their advantages in training sequence models, particularly in handling long-range dependencies.
"you you do beam search decoding usually you do app in search of up 3 and then you should be able to find good food good beam with the highest probability now someone in the audience brought up the iss..."
Applications of Sequence Models
Quoc Le outlines various applications of sequence-to-sequence models, including summarization, image captioning, and speech recognition. He discusses how these models can be utilized across different domains, showcasing their versatility and effectiveness.
"before you make a prediction let's say you predict the world am you kind of won a loop again at all the hidden state so far you want to look at all what you see in the input software okay now say when..."
Challenges in Speech Recognition
Quoc Le delves into the challenges faced in speech recognition tasks using sequence models. He explains the complexities of processing large amounts of input data and the strategies employed to manage these challenges effectively.
"predict the war and and then you keep going with that right so in the next step you also predict another C and then you take that C to compute the dot product you compute the B the a and then you can ..."
Implicit Language Models in Output
In this segment, Quoc Le discusses the implicit language models that emerge in the output of sequence-to-sequence models. He explains how the model's predictions are conditioned on previous inputs, enhancing the coherence and relevance of generated responses.
"everything okay and this autumn is implemented in tensorflow okay so how how into table what is going on here so let's suppose that you want to use this for translation so in translation you wanna for..."
Decoding Challenges in Real-Time Applications
Quoc Le highlights the decoding challenges in real-time applications like voice search. He explains the need for immediate output and the limitations of current models that require complete input before generating responses, emphasizing the importance of improving real-time processing capabilities.
"want Ola to correspond to the world hi okay because there's an one-to-one mapping between the word high and Ola so if you use the attention model the beta's that you learn will put a strong wait for t..."
Image Captioning with Deep Learning
Quoc Le discusses the application of sequence-to-sequence deep learning for image captioning. He explains how the model takes image representations from networks like VGG or GoogleNet as input and generates descriptive captions as output, showcasing the versatility of deep learning in interpreting visual data.
"summarization so I've seen I started seeing work in some radiation pretty exciting you can do image captioning so and the input in that case would just be a representation of an image coming out from ..."
Challenges in Speech Recognition
In this segment, Quoc Le delves into the complexities of speech recognition using sequence-to-sequence models. He describes the process of converting audio waveforms into text, highlighting the difficulties posed by long input sequences and the need for efficient backpropagation techniques.
"okay so well in speech recognition the input could be maybe waveforms right and then an output could be some words you know hi how's it well one thing that you can do is you drop your input into Windo..."
Decoding Challenges in Speech Models
Quoc Le explains the limitations of current speech recognition algorithms, particularly the need for offline decoding. He discusses the implications for real-time applications like voice search and introduces the concept of online decoding as a potential solution.
"now the problem with this algorithm is that in turn when it comes to speech you end up with a lot of input right you can end up with thousands and thousand steps so back propagating in time even with ..."
Comparing Speech Recognition Algorithms
This segment focuses on the performance of sequence-to-sequence models versus CTC (Connectionist Temporal Classification) in speech recognition. Quoc Le shares insights on the strengths and weaknesses of these approaches, referencing published results and industry standards.
"model already but the problem with this is that actually you have to wait until the end of the input to do the coding so the decoding has to be done offline okay so if you use this for voice search it..."
Translation with Sequence-to-Sequence Models
Quoc Le discusses the mechanics of translation using sequence-to-sequence models with attention. He explains how the model utilizes pairs of sentences to improve predictions, emphasizing the importance of attention mechanisms in enhancing translation accuracy.
"some of the hmm DNN hybrid which is which is the most Wylie speech system currently so I want to pause there and then I can take questions any questions I have a question at the back yeah yeah yeah ye..."
User Input in Translation Models
In this segment, Quoc Le addresses the integration of user input in translation models. He highlights the flexibility of the model to adapt to different languages and user behaviors, stressing the importance of a comprehensive vocabulary for effective predictions.
"well in translation what we do is basically we have pairs of sentences so for example hi how are you and then hola como estas right and then we have pairs of sentences like this and then we just feed ..."
Online Decoding in Speech Recognition
Quoc Le elaborates on the potential for online decoding in speech recognition, allowing for real-time predictions. He discusses the challenges and ongoing research in this area, emphasizing the need for efficient algorithms to enhance user experience.
"is that okay yeah yeah and then we can do some paper okay together I have a question yeah yeah yeah okay so the model I did the inbox thing that I presented it was on in English but there's no limitat..."
Sequence-Level Training in NLP
This segment introduces the concept of sequence-level training in natural language processing. Quoc Le references recent advancements in the field, including techniques that optimize predictions based on global context rather than step-by-step outputs.
"bank vocabulary is large enough so that it covers not only the English words but also the Spanish word and etc like Vietnamese and so on so your vocabulary gonna be not going to be 20,000 it's going t..."
Incorporating User Behavior in Models
Quoc Le discusses how user behavior can be integrated into translation models to improve predictions. He emphasizes the importance of adapting models to accommodate multilingual inputs and personalizing outputs based on user preferences.
"that okay I have a question there yeah over here so we have some input email and then some output email where export written emails reply and then you can just strain it that way yeah yeah okay I have..."
Memory Networks in NLP
In this segment, Quoc Le explores the concept of memory networks in natural language processing. He discusses how these networks can enhance the ability of models to retain and utilize information over time, improving overall performance in tasks like question answering.
"recognition the CDC seems to be a very nice framework because it match it laser like a monotonic increase Minh in the output and the input but let CTC make this independent assumption it doesn't have ..."
Augmented Memory in Neural Networks
Quoc Le introduces the idea of augmented memory in neural networks, explaining how it allows for dynamic reading and writing of information. He outlines the potential benefits of this approach for improving model efficiency and accuracy in various NLP tasks.
"output and maybe use some kind of reinforcement learning to adjust the output and the answer is yes so there's a recently a recent paper at Facebook who I think sequence level training or something li..."
Global Context in Predictions
This segment discusses the importance of considering global context in predictions made by sequence-to-sequence models. Quoc Le highlights recent research that aims to optimize outputs based on overall context rather than individual steps, enhancing the quality of translations.
"so the question is can we add the GaN loss like it again lost yeah I think that's a great idea yeah I have a question here yeah yeah change yeah yeah so let's suppose that you type the first ha hola t..."
User Input and Beam Search
Quoc Le explains how user input can influence the beam search process in translation models. He discusses the potential for improving predictions by conditioning the model on user-provided context, enhancing the relevance of outputs.
"did we use so in translation for example we use the several several WMT coppices Cobra and the W empty copper I usually have tens of millions of seven pairs of tendencies something like that and every..."
Data Requirements for Translation Models
In this segment, Quoc Le addresses the data requirements for training effective translation models. He shares insights on the scale of datasets needed for successful training and the implications for model performance.
"lots of interesting stuff coming along so okay okay so what's a big picture so the big picture is so far I talked about sequin to sequence learning and yesterday Andrew was talking about most of the b..."
Emoji and Language Models
Quoc Le discusses the integration of emojis into language models, emphasizing their role as text tokens. He explains how expanding vocabulary can accommodate diverse inputs, including emojis, to enhance model versatility.
"about sequence sequence and then the next part I'm going to apply sequence to sequence in a big picture of the active on ongoing work in neural nets for NLP so if you have any questions you you can as..."
Continuous Learning in NLP
This segment focuses on the concept of continuous learning in natural language processing. Quoc Le explains how models can adapt to new data by adjusting learning rates, ensuring they remain effective as they encounter new information.
"I have a question yeah also the question is does the modem handle emoji I don't know but it's emoji is like a piece of text to write so you can just like feed it into as another extra token if you mak..."
Automatic Question Answering with Deep Learning
Quoc Le explores the potential of using sequence-to-sequence models for automatic question answering. He discusses the framework's ability to read and comprehend text, enabling it to answer questions based on the information it processes.
"so this is an active area that actually is a very exciting which is in the area of automatic unite so you can think that maybe the set up would be can you read a Wikipedia page and then answer a quest..."
Attention Mechanisms in Memory Networks
In this segment, Quoc Le elaborates on the role of attention mechanisms in memory networks. He explains how these mechanisms facilitate the reading and writing of information, enhancing the model's ability to retain context and improve predictions.
"I'm not a definite expert but I'm very aware so I can place you in the right context here so work in this area would be memory networks by Western and folks at Facebook there will be new rotating mach..."
Soft Attention and Memory Reading
Quoc Le discusses the challenges of implementing soft attention in memory networks. He highlights the difficulties in differentiability and training, suggesting potential solutions through reinforcement learning techniques.
"now now let's let's try to be a little bit more general and the general would be at any point in time you can read and write right you have a controller and you can read and write read and write all t..."
Augmented Operations in Neural Networks
Quoc Le concludes the discussion by introducing the concept of augmented operations in neural networks. He emphasizes the potential for enhancing model capabilities through innovative architectural designs.
"right because this the thing that you don't read don't contribute to the gradient so it's going to be hard to train but you can use to reinforce and so on to train it so there's a reason our paper rei..."
Augmenting Neural Networks with Operations
Quoc Le introduces the concept of augmenting neural networks with operations like addition and subtraction. He illustrates this with an example involving historical context and discusses how neural networks can learn to perform arithmetic operations, emphasizing the need for extensive training data.
"so the another extension that a lot of people talk about is using an N with augmented operations so you want to augment the neural network with some kind of operations like addition subtraction multip..."
The Future of Sequence-to-Sequence Learning
Quoc Le summarizes the significance of sequence-to-sequence learning in natural language processing. He discusses the potential for joint training with autoencoders and the importance of memory operation augmentation as a promising area for future research.
"there are two papers last year from Google brain and nygma was talking about this so so that's that's some of the related work in the area of augmenting recurrent networks with with operations with me..."
Practical Insights for Training Models
In this segment, Quoc Le shares practical advice on training sequence-to-sequence models, including the importance of dataset size and strategies for reducing data requirements. He emphasizes the use of pre-trained word vectors and dropout techniques to enhance model performance.
"area in the near future so so you if you want to know more you can take a look at pre-solar block you talk about attention and of my augmented recurrent networks I also wrote some tutorials pretty sim..."