All Segments
Nuts and Bolts of Applying Deep Learning (Andrew Ng)
42 segments available
The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below. Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days: 1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk 2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M 3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw 4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w 5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8 6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I 7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo 8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI 9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs 10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc 11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4 12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc Full Day Live Streams: Day 1: https://youtu.be/eyovmAtoUx0 Day 2: https://youtu.be/9dXiAecyJrY Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen. CONNECT: - If you enjoyed this video, please subscribe to this channel. - AI Podcast: https://lexfridman.com/ai/ - Show your support: https://www.patreon.com/lexfridman - LinkedIn: https://www.linkedin.com/in/lexfridman - Twitter: https://twitter.com/lexfridman - Facebook: https://www.facebook.com/lexfridman - Instagram: https://www.instagram.com/lexfridman - Slack: https://deep-mit-slack.herokuapp.com
Segments Timeline
From Visionary to Practical Insights
Andrew Ng discusses the shift from a visionary talk to a practical presentation aimed at providing actionable insights for attendees. He emphasizes the importance of sharing common lessons learned from his experience leading BYU's AI team, focusing on how these insights can be applied across various machine learning projects.
"so you know when we're uh organizing this Workshop My My co-organizers initially asked me hey Andrew end of the first day go give a Visionary talk so until uh several hours ago my talk was advertised ..."
The Changing Landscape of Machine Learning
Ng highlights the evolving workflow of organizing machine learning projects in the era of deep learning. He introduces the concept of bias-variance tradeoff and discusses how traditional understandings are shifting as deep learning becomes more prevalent in various applications.
"several weeks or months um so one common theme that will appear in in this presentation today is that the workflow of organizing machine learning projects feels like parts of it are changing in in the..."
The Power of Scale in Deep Learning
Andrew Ng explains the significance of scale in deep learning, focusing on how the amount of data and computational power drives progress. He illustrates this with a graph showing the relationship between data volume and model performance, emphasizing the need for larger neural networks to leverage vast datasets effectively.
"um the one biggest Trend in deep learning the the is is is scale that scale drives deep learning progress um and uh I think Andrea mentioned scale of data and scale of computation um and I'm just draw..."
Building Effective AI Teams
Ng shares insights on the importance of organizing AI teams alongside computer systems teams. He discusses the challenges of expertise in high-performance computing (HPC) and machine learning, advocating for collaboration between these domains to achieve optimal performance in AI projects.
"this is a trend that we' seen in many verticals in many application areas um couple comments one is that um this uh you know actually when I draw this picture some people ask me well does this mean a ..."
Categorizing Deep Learning Work
In this segment, Ng categorizes deep learning work into four buckets: General DL, sequence models, image models, and other innovative approaches. He emphasizes that most current industry value is derived from the first three categories, while also highlighting the potential of unsupervised learning as a future direction.
"I say that is um I guess uh so let's see what so when when we started you know bu research we said our team that way other teams are also organized this way I think Peter mentioned to me that open AI ..."
The Rise of End-to-End Deep Learning
Ng discusses the trend of end-to-end deep learning, where models directly map inputs to complex outputs without intermediate representations. He provides examples from image captioning and speech recognition, illustrating how this approach has transformed traditional machine learning tasks.
"right so um I said the trend one the the major Trends one of deep learning is scale um this is what I would say is maybe major Trend two of of two of two Trends this is not going to go on forever righ..."
When to Use End-to-End Learning
Ng outlines the conditions under which end-to-end deep learning is effective, using speech recognition as a recurring example. He explains the concept of phonemes and contrasts traditional methods with end-to-end approaches, emphasizing that while promising, end-to-end learning is not a universal solution.
"ago has this trend toward not just learning algorithms an output not just a number that can output very complicated things like a sentence or caption or French sentence or image or or or or or let the..."
The Controversy of Phonemes
In this segment, Ng shares a personal anecdote about the backlash he faced from linguists when advocating for an end-to-end approach in speech recognition that disregards phonemes. He reflects on the tension between traditional linguistic methods and modern deep learning techniques, highlighting the evolving nature of the field.
"the output so the phrase end to end deep learning refers to uh just having a neuronet or you know like a learning algorithm directly go from input output that's that's what n to end means um this ENT ..."
Challenges of End-to-End Learning
Ng outlines the limitations of end-to-end deep learning, particularly the need for vast amounts of labeled data. He presents a case study involving X-ray images to predict a child's age, contrasting traditional methods with end-to-end approaches and discussing the difficulties in obtaining sufficient annotated data.
"all right um so let's see um but the the the Ares heel of a lot of deep learning is that you need tons of label data right so if this is your X and that's your y then for endtoend deep learning to wor..."
Self-Driving Cars and Data Requirements
Andrew Ng examines the architecture used in self-driving cars, emphasizing the complexity of the task and the necessity for extensive data. He discusses the balance between using deep learning for perception tasks and the importance of traditional algorithms for planning, cautioning against over-reliance on end-to-end methods without adequate data.
"right which which you can kind of get tables about um there are a lot of examples like this and I think one of the unfortunate things about deep learning is that um let's see uh you know you can for f..."
Navigating Machine Learning Challenges
Ng addresses common challenges faced by machine learning practitioners when models underperform. He emphasizes the importance of understanding bias and variance, and how these concepts guide decisions on model improvement, including data collection and architecture adjustments.
"value right so this is one thing to watch out for um I have seen project teams say I can get X I can get y I'm G to train deep learning um but unless you actually have the data you know some of these ..."
Bias and Variance in Deep Learning
In this segment, Ng elaborates on the bias-variance trade-off in deep learning, providing a framework for diagnosing model performance. He discusses how to interpret training and validation errors to identify whether a model suffers from high bias or high variance, and the actionable steps to take in each scenario.
"problem so um I want to talk a bit about bias and variance since that's one of the most basic you know Concepts in machine learning and I feel like it's evolving slightly in the era of of of deep lear..."
The Power of Bigger Models
Andrew Ng discusses the advantages of using larger models in deep learning. He explains that the ability to train bigger neural networks is a significant benefit of deep learning compared to traditional methods like logistic regression. This segment underscores the importance of model size in achieving better performance across various tasks.
"always train a bigger model bigger neuron Network in a way that was harder to do when you're training logistic regression is to come up with more and more features right that was just harder to do um ..."
Innovations in Data Synthesis
Ng introduces the concept of automatic data synthesis as a growing area in deep learning. He provides examples of how synthetic data can be generated for tasks like optical character recognition and speech recognition, illustrating the potential of this approach to enhance training datasets and improve model performance.
"lot of problems so um let's see uh so bigger model puts pressure on you know systems which is why we we have high performance Computing team um more data has has led to another interesting um set of I..."
Real-World Applications of Data Synthesis
In this segment, Ng shares practical examples of data synthesis in action, including OCR and speech recognition. He explains how generating synthetic data can significantly amplify training sets, leading to improved model accuracy. Ng emphasizes the importance of careful engineering in creating effective synthetic datasets.
"to feed into the giant mole of your neuron Network right so let me best Illustrated a couple examples um one of the easy ones is OCR so so let's say you want to train a um optical character recognitio..."
Challenges of Synthetic Data
Ng discusses the limitations of synthetic data, particularly in scenarios like recognizing cars in video games. He highlights the potential pitfalls of relying solely on synthetic datasets and stresses the need for diverse and realistic training data to ensure effective model learning.
"have a we found a huge effect on um performance um and then also NLP you know here here's here's one example actually done by some Stanford students which is um using entend deep learning to do gramma..."
Building a Unified Data Warehouse
Andrew Ng advocates for the establishment of a unified data warehouse within organizations. He explains how centralizing data can streamline access and improve collaboration among teams, ultimately enhancing the performance of machine learning projects. Ng emphasizes the importance of treating company data as a shared resource.
"synthesis has a lot of limits um I'll tell you one other story um you know let's say you want to recognize cars right uh there are a lot of video games um I need to play more video games what's a vide..."
Effective Data Splitting Strategies
In this segment, Ng addresses the importance of proper data splitting in machine learning. He shares insights from his experience at BYU, emphasizing that training, development, and test sets should come from the same distribution to avoid performance issues. Ng provides a practical example of how to effectively manage data for a speech recognition system.
"to to drive performance so really if if if your boss ask you tell them that I said like build a unified data warehouse right so um I want to take the uh train test you know bias variance picture and r..."
Best Practices for Data Splitting
In this segment, Ng outlines best practices for splitting datasets into training, development, and test sets. He stresses the importance of ensuring that development and test sets come from the same distribution to avoid wasting effort during model tuning. Ng uses an analogy to illustrate the pitfalls of mismatched datasets and highlights how proper data splitting can optimize machine learning team efficiency.
"but the question is given this data set you know how do you split this into train dep test right so here's one thing you could do which is call this your training set call this your dep set and call t..."
Understanding Bias and Variance
Ng delves into the concepts of bias and variance in machine learning, explaining how they affect model performance. He emphasizes the significance of measuring human-level performance as a benchmark and discusses strategies for addressing bias and variance issues. This segment provides insights into how to analyze errors and improve model accuracy effectively.
"efficiency because it you know the development set which is what your team is going to be tuning as algorithms to that is really the problem specification right and you problem specification tells the..."
Analyzing Error Metrics for Improvement
In this part, Ng discusses the importance of analyzing error metrics across training, development, and test sets. He explains how to identify gaps in performance and the implications of these gaps for model tuning. Ng provides a framework for understanding where to focus efforts in reducing bias and variance, ultimately leading to better model performance.
"variance concept um actually let me use this board and and but has say the the um the fact that training and test sets don't match is one of the problems that um Academia doesn't study much there's so..."
The Challenge of Surpassing Human Performance
Ng explores the challenges of surpassing human-level performance in machine learning tasks. He discusses the theoretical limits of performance and the reasons why progress often slows after reaching human benchmarks. This segment highlights the importance of understanding human capabilities as a reference point for improving machine learning systems.
"bias reduction techniques uh this Gap here right this is kind of um really the variance this Gap here is due to your train test distribution mismatch um and this is overfitting of Dev okay right um so..."
Strategies for Error Analysis
In this segment, Ng emphasizes the value of error analysis in machine learning. He explains how examining errors can provide insights into model weaknesses and guide improvements. Ng discusses various strategies for conducting error analysis, including leveraging human insights to refine models and enhance performance.
"that much more uh finally uh uh there's this yeah well and then hopefully if you're done uh uh hopefully your test set error will be will be good and if if you're doing well your death set but not you..."
The Role of Human-Level Benchmarks
Ng discusses the significance of human-level benchmarks in evaluating machine learning models. He explains how these benchmarks can inform development strategies and help teams identify areas for improvement. This segment highlights the evolving nature of performance metrics in deep learning and the importance of continuous assessment against human capabilities.
"progress on things um so you know there's one thing I I I kind of snuck in here without making it explicit which is that in this whole analysis we were benchmarking against human level performance rig..."
Theoretical Limits and Practical Implications
In this concluding segment, Ng addresses the theoretical limits of machine learning performance. He discusses how noise in data can set upper bounds on accuracy and the implications for model development. Ng emphasizes the need for realistic expectations and strategic planning when working towards improving machine learning systems.
"there multiple reasons why this is the case I'm I'm curious like why why why why do you think this is the case any any guesses yeah cool labels are coming from humans the labs are oh cool yep labels c..."
The Challenge of Surpassing Human Accuracy
Andrew Ng discusses the complexities of surpassing human-level accuracy in deep learning applications. He explains that while being worse than humans allows for clearer paths to improvement, once human-level performance is exceeded, it becomes challenging to identify areas for further enhancement. This segment emphasizes the importance of understanding human performance as a benchmark for machine learning progress.
"there might not be a lot of room right to go to go further up so that's kind of one reason as just humans are pretty good um other reasons I think a couple people said right um and and it turns out th..."
Error Analysis in Deep Learning
In this segment, Ng elaborates on the significance of error analysis in deep learning. He explains how examining the mistakes made by AI systems can provide insights into human reasoning and help improve model performance. By understanding where AI fails compared to human judgment, practitioners can better target their efforts for improvement.
"ways uh to make progress and so some of those ways are right a couple of you mentioned this you can get labels from humans right um You can also carry out error analysis and error analysis just means ..."
Estimating Bias and Variance
Andrew Ng highlights the importance of estimating bias and variance in machine learning models. He discusses how knowing human error rates can guide developers in deciding whether to focus on reducing bias or variance. This segment underscores the role of human performance metrics in optimizing deep learning systems.
"effects right and here's what I mean um so let's see to take another confute example let's say that uh you let's say that you're working on some image recognition task right if I tell you that um uh y..."
Finding Subsets for Improvement
Ng shares strategies for improving deep learning models even after surpassing human-level performance. He suggests identifying specific subsets of data where the AI still underperforms compared to humans. This targeted approach allows teams to apply effective improvement techniques to areas where they can still make significant gains.
"humans are and and I think of humans as a proxy as an approximation for the Bas error rate for the optimal error rate this piece of information really tells you where you should focus your effort and ..."
Defining Human-Level Performance
In this interactive segment, Ng poses a question to the audience about how to define human-level performance in the context of medical imaging. He presents various choices, including typical humans, doctors, and expert teams, and discusses the implications of each definition for driving machine learning progress. This highlights the importance of clear benchmarks in AI development.
"where you segment the training center and analyze sub separate subsets of training Cent yeah yeah I see actually you know that's a wonderful question I want to ask a related Qui question to everyone i..."
Using Human Performance as a Proxy
Ng explains how human performance metrics serve as a proxy for optimal error rates in machine learning. He emphasizes that understanding the best possible human performance can guide AI development and improve decision-making processes. This segment illustrates the critical role of human benchmarks in advancing AI capabilities.
"sure uh uh uh yeah don't worry about ease of obtaining this data yeah right so which is the most useful definition Choice a who anyone just a couple people choice B who thinks uses Cool like a fifth C..."
The Role of Human-Level Performance in AI
In this segment, Ng discusses the increasing relevance of human-level performance metrics as deep learning systems approach human accuracy. He advises teams to invest time in determining these metrics, as they provide valuable insights for improving AI systems. This highlights the evolving landscape of AI and the need for effective performance measurement.
"different than um the goal of publishing papers is different than the goal of actually you know building the best possible product right so for the purpose of publishing papers people like to say oh w..."
Designing AI Products
Ng addresses the challenges of designing AI products in the current landscape. He notes the lack of established workflows for AI product development compared to traditional software. This segment emphasizes the need for new processes and frameworks to effectively integrate AI into product design.
"question the back oh is it possible that team of expert doctors does worse than a single doctor I don't know I I I had to ask the doctors in the audience know all right um all right just just I just t..."
Understanding AI Capabilities
In this segment, Ng shares insights on what AI can realistically achieve. He provides rules of thumb for product managers to consider when designing AI applications, emphasizing tasks that can be automated based on human performance. This guidance is crucial for aligning AI capabilities with business objectives.
"with them a very common piece of advice is he's going to figure out what is human level performance and and they didn't spend some time to have humans labor and get that number because that number is ..."
Short-Term Applications of Deep Learning
Ng discusses the short-term applications of deep learning, particularly in automating tasks that humans can perform quickly. He highlights the potential for deep learning to enhance efficiency in various domains, including computer vision and speech recognition. This segment showcases the immediate value of AI in practical applications.
"often you know with the rise of AI I feel like um uh maybe this is again a company thing um in silic Valley we've developed pretty good workflows for Designing products in the desktop era and in the m..."
Predicting Outcomes with AI
In this segment, Ng explores the predictive capabilities of AI, particularly in understanding user behavior and outcomes. He provides examples of how deep learning can be applied to predict future events based on historical data, illustrating the transformative potential of AI in various industries.
"um I find us inventing new processes in order to design AI product right processes that really didn't exist before but one of the questions I often get asked partially sometimes at product people some..."
Building a Career in Machine Learning
Ng concludes with advice on building a career in machine learning. He reflects on common questions he receives about entering the field and shares insights on the skills and knowledge necessary for success. This segment serves as a valuable resource for aspiring machine learning professionals.
"a piece of given a picture tell me if the face in this picture is smiling or frowning you don't need to think for more than a second so yes we can build deep loading systems and do that really well ri..."
Building a Career in Machine Learning
Ng addresses the common question of how to build a career in machine learning. He shares his personal journey and emphasizes the importance of continuous learning and attending events like deep learning school to enhance one's skills and knowledge in the field.
"these are clearly highly FAA and there are many many counter examples right um so I think let's see um say it's exciting find for deep learning because I think it's letting us do a lot of interesting ..."
The Path to Becoming a Machine Learning Researcher
Andrew Ng shares insights on transitioning from non-machine learning roles to becoming proficient in machine learning. He emphasizes the value of working on personal projects and engaging with platforms like Kaggle to gain practical experience and improve skills.
"which is the kind of personal advice um you know I think that um I was asking myself this same question uh uh uh like a couple months ago right which is you know after you've taken a machine learning ..."
The Importance of Reading Research Papers
Ng outlines a reliable process for aspiring machine learning researchers: reading and replicating research papers. He explains how this practice can lead to new ideas and innovations in the field, encouraging attendees to immerse themselves in academic literature.
"shared oh by the way almost everything I talked about today is is is new content that I've never presented before right so so I I don't know as I hope it worked okay thank you thank you so I want to s..."
Embracing the 'Dirty Work' in AI
Andrew Ng discusses the often-overlooked 'dirty work' involved in AI development, such as data cleaning and debugging. He stresses the importance of balancing this practical work with theoretical learning to foster creativity and innovation in machine learning.
"this daily art right I I I don't know how the I don't really I don't know how the human brain works but I've seen this be an incredibly reliable process read enough papers and you know between 20 and ..."
The Long-Term Commitment to Learning
Ng encourages attendees to commit to continuous learning in machine learning, emphasizing that consistent effort over time leads to significant skill development. He draws parallels between this dedication and the transformative potential of AI in various industries.
"researches right so um I want to close with uh uh uh uh uh uh just one more story about this and I guess some of you may have heard me talk about the the the the Saturday story right but um for those ..."