
30 segments available
The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below. Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days: 1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk 2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M 3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw 4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w 5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8 6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I 7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo 8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI 9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs 10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc 11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4 12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc Full Day Live Streams: Day 1: https://youtu.be/eyovmAtoUx0 Day 2: https://youtu.be/9dXiAecyJrY Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.
Pascal Lamblin introduces Theano, explaining its purpose and basic principles. He highlights the familiarity of concepts for those who attended previous presentations on TensorFlow and introduces the structure of the tutorial, which includes a hands-on example of logistic regression.
"okay so today I'm going to briefly introduce you tno how to use it and go over the basic principles behind the libraries and if you paid attention during yesterday's presentation of tensor flow some c..."
Lamblin elaborates on Theano as a mathematical symbolic expression compiler. He discusses its capabilities, including defining mathematical expressions using NumPy syntax, automatic differentiation, and optimizations for numerical stability. The segment emphasizes Theano's role in research and industry applications.
"the same time then we're going to have a more hands-on example basically applying logistic regression on the Emnes digits data set and then if we have time we'll go quickly over to more examples conce..."
This segment covers Theano's ecosystem, mentioning various libraries built on top of it, such as Blocks and Keras. Lamblin explains how these libraries provide higher-level interfaces for deep learning, making it easier for users to implement complex models and algorithms.
"it's possible to use that optimized graph and the Endo's runtime to actually compute some values some output values even inputs we also have a couple of tools that help debug both pianos code and the ..."
Lamblin explains how to define symbolic variables in Theano, which form the basis of a computation graph. He discusses the importance of specifying variable types and dimensions while allowing for flexibility in shape during function calls.
"to use TN well first of all we are working with symbolic expression symbolic variables so that will make up a computation graph so let's see how how to do that so to define the symbolic expression so ..."
In this segment, Lamblin demonstrates how to build computation graphs in Theano. He illustrates the process of defining expressions, creating intermediate variables, and visualizing the graph structure, emphasizing the connections between input variables and operations.
"so x and y are purely symbolic variables here we will give them values later but for now that's just that's just empty there's another kind of input variables that is share variables and they they're ..."
Lamblin introduces the concept of backpropagation in Theano, explaining how to compute gradients using the chain rule. He discusses the symbolic representation of gradients and the importance of efficient computation without explicitly forming the Jacobian matrix.
"structure like that by using for instance by dot print which is a helper function so variables are those square boxes and we have other nodes here we call apply nodes that represent the mathematical o..."
This segment focuses on extending the computation graph to include gradient updates for training algorithms. Lamblin explains how to define update expressions and the significance of incorporating these updates into the overall graph structure.
"a function that returns that and it actually returns not numbers not a numerical expression for that but it returns a symbolic expression that represents that computation again usually without having ..."
Lamblin discusses the process of compiling functions in Theano, detailing how to create functions that compute outputs based on input variables. He highlights the flexibility of defining functions that only compute subsets of the graph.
"and we can continue extending the graph from these variables for instance to compute update expressions corresponding to gradient descent something like that like we do here so for instance this is wh..."
In this segment, Lamblin explains how to update model parameters during training in Theano. He describes the process of defining updates for shared variables and how these updates are computed alongside other outputs.
"function out of it so so that's it you can first compile it get value and call it so you have to provide values for all the input variables that that you define you don't have to provide values for sh..."
Lamblin concludes with a discussion on graph optimization techniques in Theano. He explains how the graph is optimized during compilation, including eliminating redundant computations and enhancing numerical stability, ultimately improving performance.
"updated so here if we print the value of B before and after having calling after having called the same function then we see the value has changed what happens also during graph compilation is that th..."
Pascal Lamblin discusses the compilation of Theano functions and the generation of optimized C++ or CUDA code for improved runtime efficiency. He emphasizes the importance of on-the-fly code generation to adapt to user-defined operations, ensuring optimal performance during execution.
"except for a couple of checks and assertions but if you're iterating and want fast feedback and don't care that much about Timothy about the runtime speed then you have a couple of ways of enabling an..."
In this segment, Lamblin explains advanced optimization techniques in Theano, including loop fusion and the handling of element-wise operations. He illustrates how these techniques contribute to reducing computational overhead and improving the overall efficiency of deep learning models.
"the compile function that goes from these input variables to out which was called predicts this is what we have I won't go into details about what's happening in there but here you have a gem G operat..."
Lamblin introduces the backprint tool in Theano, which provides a text-based visualization of the graph structure. This tool helps users understand the relationships between variables and operations within the graph, aiding in debugging and optimization efforts.
"happen often when you have automatically generated gradient expressions oh and here you see the update for the shared eyeballs which are inputs so you see the cost and the implicit outputs for the upd..."
This segment focuses on handling dynamic loops in Theano using the 'scan' node. Lamblin explains how this node encapsulates a function to compute operations at each time step, allowing for flexible sequence generation and gradient computation in recurrent models.
"here you can see more in detail like what the structure is and you see the inputs of gmv and the scaling parameters and so on so when the function is compiled then we can actually run it so T no funct..."
Lamblin discusses the integration of GPU support in Theano, detailing how users can easily configure their environment to leverage GPU capabilities. He highlights the importance of selecting appropriate data types and optimizing memory usage for enhanced performance.
"the desired results we have a couple of different ones and in particular there's one which was written in C++ which avoids having to switch contacts between the Python interpreter and the C++ executio..."
In this segment, Lamblin explains how to set up configuration flags for Theano to optimize performance. He discusses the importance of environment variables and configuration files in managing device selection and memory allocation for deep learning tasks.
"the device that you want to use the primary device that you want to use with just the configuration flag for instance you could to get the first GPU that's available or one specific one and if you spe..."
Lamblin outlines upcoming features and improvements in Theano, including enhanced GPU support, performance optimizations, and new diagnostic tools. He emphasizes the ongoing development efforts aimed at making Theano more efficient and user-friendly for deep learning practitioners.
"you can also set things directly from Python but some flags have to be known in advance before you know is is imported so for instance the device itself you have to set it either in the configuration ..."
This segment covers the debugging and visualization tools available in Theano. Lamblin discusses the importance of error messages and completion modes that help users identify issues during execution, enhancing the overall development experience.
"going to represent the computation that has to be done at each time step so you have at the end of function that performs the competition for one time step and you have the scan node that calls it in ..."
Lamblin explains how users can extend Theano by adding custom operations and optimizations. He discusses the potential for increased numerical stability and efficiency through user-defined functions, encouraging developers to contribute to Theano's capabilities.
"some completion modes that enable to for instance check for not a number fall out values you can assign test values to the symbolic variables so that each time you create a new symbolic intermediate v..."
In this concluding segment, Lamblin introduces practical examples to demonstrate Theano's capabilities. He provides instructions for setting up the repository and launching Jupyter notebooks, preparing attendees for hands-on experience with logistic regression and other models.
"diagnostic tools that are quite useful and interactive visualization an interactive graphical ization tool and pdb breakpoints that enables you to monitor a couple of eyeballs and only break if some c..."
In this segment, Lamblin walks through the implementation of logistic regression in Theano. He explains the model definition, loss function, and the process of compiling a training function, emphasizing the importance of monitoring misclassification rates during validation.
"it's time to start the practical examples so for those who have not clone the repository yet then this is the command line you want to two nouns for those who had cloned it you might want to do a git ..."
Lamblin details the training loop for the logistic regression model, discussing early stopping techniques and validation processes. He highlights the importance of tracking the best model during training and saving model parameters for future use.
"to have one croissants for P or log likelihood operation by itself you can just build it from the basic building blocks so we take the log of the probability you take the index of the actual target an..."
This segment focuses on visualizing the learned filters from the logistic regression model. Lamblin demonstrates how to access and plot the internal values of the model, providing insights into the features that the model has learned to recognize.
"just save the values of all parameters which is more robust than trying to pick all the whole Python object and it also enables more easily transferred to other frameworks to visualization frameworks ..."
Lamblin introduces convolutional networks, explaining how to define layers and initialize weights in Theano. He discusses the architecture of convolutional layers and their role in processing image data, setting the groundwork for more complex models.
"too simple and we should use something more advanced so to use something more advanced if you go back to the home of the Jupiter notebook can have a look at the continent and run Lynnette so this new ..."
In this segment, Lamblin outlines the process of building a complete neural network using Theano. He covers the integration of various layers, including convolutional and fully connected layers, and discusses the training loop and validation strategies for the network.
"on that they find useful and intuitive so this logistic regression layer basically holds well parameters weight and bias and compute the well the conditional probability of classes prediction holds th..."
Lamblin concludes with an introduction to Long Short-Term Memory (LSTM) networks, explaining their structure and purpose in predicting sequences. He sets the stage for further exploration of LSTM applications in deep learning.
"you're welcome to to play with it and after that yeah you can visualize the the the round filters as well here you you have them for the first layer and for the and here you have the an example of the..."
Lamblin introduces Long Short-Term Memory (LSTM) networks, explaining their structure and purpose in predicting the next character in a sequence. He details the parameters involved in LSTM layers and how to implement them using Theano. This segment is essential for grasping the complexities of recurrent neural networks and their applications in sequence prediction tasks.
"then so this model is an SEM network that tries to predict the next character of our sentence given the previous ones so not going to go into details but here you can see that the LSM layer is defined..."
This segment covers the definition of the cost function for LSTM networks, focusing on categorical cross-entropy and the use of masks for sequence predictions. Lamblin discusses the importance of data preprocessing using the Fuel library, which enhances efficiency in handling mini-batches of sequences. Understanding these concepts is vital for optimizing model training and ensuring accurate predictions.
"step function is going to be passed to Tiano dot scan where the sequences are the masks and input so the mask is is useful because we're using mini batches of sequences and not all the sequences in th..."
Lamblin explains how to generate sequences using the trained LSTM model, showcasing the process of normalizing predictions and monitoring output quality. He shares examples of generated text, illustrating the model's learning progression. This segment highlights the practical applications of LSTMs in generating coherent sequences and the challenges involved in fine-tuning model outputs.
"yeah here we build our final key on a graph we have symbolic inputs for well the input and masks we create lsdm layered a lot correct layer define our cost parameters are the concatenation of the para..."
In this final segment, Lamblin addresses debugging techniques in Theano, demonstrating how to identify and resolve shape mismatches in symbolic graphs. He discusses the importance of understanding error messages and optimizing functions for better performance. This part is crucial for developers working with Theano, as it provides practical insights into troubleshooting and improving model implementations.
"bit and here are some suggestions of things you might want to do like better training algorithms different nonlinearities inside the lsdm sell different initialization of weights try to generate somet..."