All Segments
Complete Statistical Theory of Learning (Vladimir Vapnik) | MIT Deep Learning Series
12 segments available
Lecture by Vladimir Vapnik in January 2020, part of the MIT Deep Learning Lecture Series. Slides: http://bit.ly/2ORVofC Associated podcast conversation: https://www.youtube.com/watch?v=bQa7hpUpMzM Series website: https://deeplearning.mit.edu Playlist: http://bit.ly/deep-learning-playlist OUTLINE: 0:00 - Introduction 0:46 - Overview: Complete Statistical Theory of Learning 3:47 - Part 1: VC Theory of Generalization 11:04 - Part 2: Target Functional for Minimization 27:13 - Part 3: Selection of Admissible Set of Functions 37:26 - Part 4: Complete Solution in Reproducing Kernel Hilbert Space (RKHS) 53:16 - Part 5: LUSI Approach in Neural Networks 59:28 - Part 6: Examples of Predicates 1:10:39 - Conclusion 1:16:10 - Q&A: Overfitting 1:17:18 - Q&A: Language CONNECT: - If you enjoyed this video, please subscribe to this channel. - Twitter: https://twitter.com/lexfridman - LinkedIn: https://www.linkedin.com/in/lexfridman - Facebook: https://www.facebook.com/lexfridman - Instagram: https://www.instagram.com/lexfridman
Segments Timeline
Introducing Vladimir Vapnik
In this segment, the host introduces Vladimir Vapnik, a pioneer in statistical learning theory and co-inventor of support vector machines. The introduction highlights Vapnik's significant contributions to the field and sets the stage for his lecture on the complete statistical theory of learning.
"- Today, we're happy and honored to have Vladimir Vapnik with us, co-inventor of supported vector machines, support vector clustering, VC theory of statistical learning, and author of "Statistical Lea..."
Foundations of Statistical Learning Theory
Vladimir Vapnik discusses the origins of statistical learning theory, co-developed with Professor Chervonenkis. He explains the fundamental question of generalization from training data to test data and introduces the concept of empirical error, emphasizing the need for a robust theoretical framework beyond the law of large numbers.
"- Thank you. About 50 years ago, Professor Chervonenkis and me started statistical learning theory. The problem was to answer the question when if we will do well with training data if you will have s..."
The Two Principles of Generalization
Vapnik introduces two principles of generalization in statistical learning: the brute force principle, which suggests that more data leads to better answers, and an intelligent principle that he will elaborate on. He emphasizes the importance of understanding intelligence in the context of learning theory.
"because there are more, another way to do something. There is no short way for generalization. You should use both of them, so that is complete theory. But it is not so bad because you will see that l..."
VC Theory of Generalization
In this segment, Vapnik delves into the VC (Vapnik-Chervonenkis) theory of generalization. He explains the concept of minimizing a functional within a given set of functions and introduces the VC-dimension, which measures the capacity of a set of functions to shatter data points. This foundational theory is crucial for understanding model performance.
"So let me start. The first part is, this is theory of generalization, and that is the question, when in set of given set of function, you can minimize functional. This is pretty general functional. In..."
Target Functional for Minimization
Vapnik discusses the target functional for minimization in pattern recognition problems. He outlines the general setting of these problems, focusing on how to select functions from a set based on observed data. This segment emphasizes the importance of choosing the right loss function and constructing an appropriate set of functions.
"Target functional for minimization. And this is important slide, God plays dice. What is setting of pattern recognition problem? I will consider in this talk, just pattern recognition problem for two ..."
Selection of Admissible Functions
In this segment, Vapnik addresses the critical question of how to select an admissible set of functions for minimization. He discusses the challenges of constructing effective function sets and the importance of understanding the structure of these sets in the context of statistical learning.
"But it is not major stuff. Because, OK, I am prove rate of convergence, but is still this square method good? But now, the most important part, selection of admissible set of functions. What it means?..."
Complete Solution in RKHS
Vapnik introduces the concept of Reproducing Kernel Hilbert Space (RKHS) as a complete solution for statistical learning problems. He explains how RKHS provides a framework for understanding function spaces and their properties, which is essential for effective learning and generalization.
"So that is concept, what we have to do. We have to solve our problem using both big and strong, strong convergence that means, using invariance and minimizing functionals, and we can do it in exact wa..."
LUSI Approach in Neural Networks
In this segment, Vapnik discusses the LUSI (Learning Using Statistical Information) approach in the context of neural networks. He highlights how this approach integrates statistical learning principles with neural network architectures to enhance learning efficiency and model performance.
"Let me show something about neural net. I am not fond of neural nets, but we can use our theory from neural net as well. What is neural net? That is neural net, you're minimizing least square error. I..."
Examples of Predicates
Vapnik provides practical examples of predicates in statistical learning, illustrating how these concepts apply to real-world scenarios. This segment emphasizes the application of theoretical principles to concrete problems in machine learning.
"what is predicate. I don't know, exactly, I think this is for many hundred years theory, I will show you that it is continuation of major philosophy from Plato to Hegel to Wigner and so on, I will sho..."
Conclusion and Key Takeaways
In the conclusion of his lecture, Vapnik summarizes the key points discussed throughout the presentation. He reinforces the importance of understanding statistical learning theory and its implications for future research and applications in machine learning.
"I have conclusion remarks. What we did, is that we can minimize this functional which is slightly better than, say, least square subject to this constraint which is serious, because this constraint me..."
Q&A: Overfitting
During the Q&A session, Vapnik addresses questions related to overfitting in machine learning models. He discusses strategies for mitigating overfitting and emphasizes the importance of generalization in achieving robust model performance.
"- [Host] I think we have time for a few questions. - [Man in Audience] Hello, thank you, I have two questions. First one is, do you know of any predicates that you recommend for language classificatio..."
Q&A: Language
In this final Q&A segment, Vapnik responds to questions about the language and terminology used in statistical learning theory. He clarifies concepts and provides insights into the nuances of discussing complex ideas in the field.
"- [Host] He also asked about natural language. Recommendations for predicates for language, natural language processing, the Turing test, any good predicates. - You know it is very complicated story, ..."