All Segments
Efficient Computing for Deep Learning, Robotics, and AI (Vivienne Sze) | MIT Deep Learning Series
46 segments available
Lecture by Vivienne Sze in January 2020, part of the MIT Deep Learning Lecture Series. Website: https://deeplearning.mit.edu Slides: http://bit.ly/2Rm7Gi1 Playlist: http://bit.ly/deep-learning-playlist LECTURE LINKS: Twitter: https://twitter.com/eems_mit YouTube: https://www.youtube.com/channel/UC8cviSAQrtD8IpzXdE6dyug MIT professional course: http://bit.ly/36ncGam NeurIPS 2019 tutorial: http://bit.ly/2RhVleO Tutorial and survey paper: https://arxiv.org/abs/1703.09039 Book coming out in Spring 2020! OUTLINE: 0:00 - Introduction 0:43 - Talk overview 1:18 - Compute for deep learning 5:48 - Power consumption for deep learning, robotics, and AI 9:23 - Deep learning in the context of resource use 12:29 - Deep learning basics 20:28 - Hardware acceleration for deep learning 57:54 - Looking beyond the DNN accelerator for acceleration 1:03:45 - Beyond deep neural networks CONNECT: - If you enjoyed this video, please subscribe to this channel. - Twitter: https://twitter.com/lexfridman - LinkedIn: https://www.linkedin.com/in/lexfridman - Facebook: https://www.facebook.com/lexfridman - Instagram: https://www.instagram.com/lexfridman
Segments Timeline
Introduction to Efficient Computing
Vivienne Sze introduces her research on energy-efficient and high-performance systems for machine learning and AI. She emphasizes the importance of optimizing algorithms and architectures to balance power, speed, and quality, highlighting the energy efficiency gap between the human brain and AI systems.
"we'd have Viviane see here with us she's a professor here at MIT working in the very important and exciting space of developing energy efficient and high-performance systems for machine learning compu..."
Overview of the Talk
Sze outlines the structure of her lecture, which will cover deep learning, robotics, and the significance of efficient computing in enabling advanced AI applications. She notes that her work is collaborative and invites the audience to access her slides online.
"the research and an overview of this area efficient computing so actually what I'm going to be talking about today is gonna be a little bit broader than just deep learning will start with deep learnin..."
The Compute Demand in Deep Learning
Vivienne discusses the exponential growth in computational requirements for deep learning applications, noting a staggering increase of over 300,000 times in compute needs to enhance accuracy. She also addresses the environmental impact of this increased compute demand.
"it's the deep learning lecture series I want to first start out talking up a little bit about deep neural nets so we know that deep neural Nets has you know generate a lot of a lot of interest has a v..."
Cloud vs. Edge Computing
Sze explains the necessity of moving computation from the cloud to edge devices. She highlights the benefits of reduced communication reliance, enhanced privacy for sensitive data, and the need for low-latency processing in applications like autonomous vehicles.
"start looking at basically the environmental implications of all of this processing can be quite severe so if we look at for example the carbon footprint of you know training neural nets if you think ..."
Challenges of Power Consumption
The lecture addresses the significant power consumption challenges in devices like self-driving cars, which require over 2000 watts for computation. Sze emphasizes the logistical issues posed by heat generation and space requirements in these systems.
"order to do a lot of these applications so again you know removing your tethering from the cloud is important another reason is a lot of the times that we you know apply deep learning on a lot of appl..."
Energy Constraints in Portable Devices
Vivienne discusses the limitations of energy capacity in portable devices, such as smartphones and small robots. She explains how current embedded platforms consume too much power, necessitating a focus on energy efficiency for handheld applications.
"hopefully this is establishing why we want to move the compute into the edge but one of the big challenges of doing processing in the robot or in the device actually has to do with power consumption i..."
The Slowdown of Moore's Law
Sze highlights the challenges posed by the slowing down of Moore's Law and Dennard scaling, which traditionally allowed for smaller, faster, and more efficient transistors. This slowdown complicates the demand for increased compute power in deep learning.
"you know decade or so or decades what we would do to address this challenge is that we would wait for transistors become smaller faster and more efficient however this has become a challenge over the ..."
Specialized Hardware for AI
To meet the growing computational demands, Sze advocates for the development of specialized hardware tailored for AI and deep learning tasks. She notes the rise of startups focusing on this area, indicating a shift towards more efficient computing solutions.
"turn to in order to address this is we need to turn towards specialized hardware to achieve the significant speed and energy throughputs that we require for our particular application and we talked ab..."
Power Consumption in Data Movement
Vivienne explains that the majority of power consumption in AI applications is due to data movement rather than computation itself. She illustrates the energy costs associated with data transfer, emphasizing the need to minimize data movement to enhance efficiency.
"first question you should ask is where is the power actually going for these applications and so as it turns out power is dominated by data movement so it's actually not the computations themselves th..."
Data Challenges in Autonomous Navigation
Sze discusses the data-intensive nature of autonomous navigation applications, which require semantic and geometric understanding. She highlights the challenges posed by the vast amounts of data that need to be processed in real-time.
"movement right again this is delivering the input data to do the multiplication and then you know moving the output of the multiplication somewhere into memory it can be very expensive so for example ..."
Deep Learning's Role in AI Applications
Vivienne emphasizes the critical role of deep learning in various AI applications, including computer vision and speech recognition. She outlines the computational complexity involved in these tasks and the need for efficient processing to make deep learning more accessible.
"itself right so this is a key takeaway here so if we really want to address the energy consumption of these particular types of processing we really want to look at reducing data movement okay but wha..."
Understanding Deep Neural Networks
Sze provides an overview of deep neural networks, explaining their structure and function. She discusses how these networks learn features from data rather than relying on predefined rules, highlighting the importance of the multiply-and-accumulate operation in neural network computations.
"this is a significant challenge for autonomous navigation in terms of mounted data other aspects of Thomas navigations also other applications like AR VR and so on is understanding your environment ri..."
Types of Neural Network Layers
Vivienne describes various types of layers used in deep neural networks, including feed-forward, convolutional, and recurrent layers. She explains how these layers connect and function, providing insight into their applications in processing different types of data.
"briefly I'm just gonna give a quick overview of some of the key components and deep neural nets not because you know I'm sure all of you understand it but because since this area is very popular the t..."
Understanding Convolutional Neural Networks
In this segment, Sze delves into the complexities of convolutional neural networks (CNNs), explaining how they perform convolutions on multi-dimensional data. She illustrates the challenges posed by increased dimensionality and the necessity of processing multiple channels and filters, emphasizing the computational demands of CNNs.
"like what we talked about all inputs and all outputs are connected there tend to be feed-forward and when you put them together they're typically referred to as a multi-layer perceptron you have convo..."
Trends in Deep Neural Network Architectures
Vivienne Sze discusses the evolving trends in deep neural network architectures, noting the increasing depth and complexity of models over time. She highlights the growing number of weights and the computational challenges that arise, underscoring the need for advanced hardware solutions to manage these demands effectively.
"as to why you know convolutional or deep learning is much more computationally more complex than other types of processing so we'll focus on you know convolutional neural Nets is an example although m..."
Hardware Acceleration for Neural Networks
This segment focuses on the importance of hardware acceleration in processing deep neural networks. Sze explains how the parallelism inherent in neural network computations can be leveraged to enhance throughput, while also addressing the bottlenecks associated with memory access and data movement.
"couple of interesting trends one is that the networks tend to become deeper so you can see in the convolutional layers they're getting deeper and deeper and then also the number of weights that they'r..."
Exploiting Data Reuse in Neural Networks
Vivienne Sze introduces the concept of data reuse opportunities in neural networks, explaining how certain data can be utilized multiple times across different computations. She discusses convolutional reuse and the implications for memory hierarchy design, emphasizing strategies to minimize data movement and energy consumption.
"these multiplies and accumulates you can actually do them all in parallel right so that's great so that what that means is high throughput or high speed is actually possible cuz I can do a lot of thes..."
Optimizing Data Movement in Neural Network Processing
In this segment, Sze explores various strategies for optimizing data movement within neural network architectures. She presents different data flow approaches, such as weight stationary and output stationary, and introduces her group's row stationary data flow method, which aims to balance the movement of all data types for improved efficiency.
"that's what we called filter eaters okay so there's a lot of these great filter reuse opportunities in the neural network itself and so what what can we do to exploit this reuse opportunities well all..."
Row Stationary Data Flow Explained
In this segment, Sze introduces the row stationary data flow approach, which balances the data movement of various data types in neural networks. She emphasizes the importance of optimizing all types of data movement rather than focusing on a single type, and discusses how this method can be expanded to higher dimensions in neural network architectures.
"performing a one row but you just talked about the fact that the neural network is much more than a 1d convolution so you can imagine expanding this to higher dimensions so this is just showing how yo..."
Exploiting Sparsity for Power Savings
Sze explains how exploiting sparsity in neural networks can lead to significant power savings. She discusses techniques such as skipping multiplications when inputs are zero and using data compression methods like run-length encoding to reduce data movement, ultimately enhancing energy efficiency in deep learning systems.
"when you're trying to optimize you know given piece of hardware you don't want to just optimize one you know for one particular type of data you want to optimize overall for all the movement in the ha..."
The Iris Chip: A Case Study
Vivienne Sze presents the Iris chip, a custom hardware designed to implement the row stationary data flow and exploit sparsity in activation data. She shares insights on its architecture, including the number of processing elements and the significant reductions in energy consumption achieved compared to traditional mobile GPUs.
"okay so you know what is the impact of all these types of things so we actually looked at building hardware I'm in particular a customized chip that we called iris to demonstrate these particular proa..."
Energy vs. Accuracy in Neural Networks
In this segment, Sze discusses the critical trade-off between energy consumption and accuracy in neural network systems. She illustrates this with a plot showing the relationship between energy usage and accuracy for various object detection tasks, emphasizing the need for efficient designs that do not compromise battery life.
"this is great we can build custom hardware but what does this actually mean in terms of you know building a system that can efficiently compute neural nets so let's say we take a step back let's say w..."
Challenges in Neural Network Efficiency
Sze highlights the challenges faced in making neural networks more energy-efficient. She compares the energy consumption of traditional methods like histogram of oriented gradients (HOG) with modern neural networks, stressing the importance of reducing energy usage without sacrificing performance.
"really took off is because they really improve the accuracy so you can imagine Alex said here almost doubled the accuracy and then vgg you know further increase the accuracy so it's super exciting the..."
Research Directions for Energy Efficiency
Vivienne Sze outlines various research directions aimed at improving energy efficiency in deep learning. She discusses algorithmic approaches such as pruning, efficient network architectures, and reduced precision, emphasizing the need to focus on energy savings and latency rather than just the number of operations.
"significant I'll give you an example so if I told you on your cell phone I'm gonna double the accuracy of its recognition but your phone would die three hundred times faster who here would be interest..."
Incorporating Energy Metrics into Neural Network Design
In this segment, Sze discusses the importance of incorporating energy consumption metrics into the design of neural network algorithms. She explains how understanding where energy is consumed can guide pruning decisions and improve overall efficiency, ensuring that the most energy-intensive weights are prioritized for removal.
"that we learned about as we were doing this kind of survey of the area is that we actually identified various limitations in terms of how people are approaching or how the research is approaching this..."
Rethinking Energy Metrics in Neural Networks
Vivienne Sze discusses the importance of understanding energy consumption in neural networks beyond just the weights. She emphasizes that energy usage is significantly influenced by the movement of input and output feature maps, and proposes a more holistic approach to energy-efficient neural network design through energy-aware pruning.
"consumption okay and so by doing this when we take a look it should be no surprise what are the key observations for this exercise is that the weights alone are not a good metric for energy consumptio..."
Energy-Aware Pruning Techniques
In this segment, Sze introduces energy-aware pruning as a method to enhance neural network efficiency. By focusing on removing weights that consume the most energy rather than just small weights, she explains how this approach can lead to a significant reduction in energy consumption while maintaining accuracy.
"actually know where the energy go is going how can we factor that into this of the neural networks to make them more efficient so we talked about the concept of pruning right so again pruning was sett..."
Latency vs. Computational Complexity
Sze highlights the complexities of latency in neural networks, explaining that latency is not directly correlated with the number of computations. She presents data from Google's mobile vision team to illustrate the variability in latency across different neural network architectures, emphasizing the need for a better understanding of latency in design.
"consumption into the design of the neural network itself or and the prune models are all available in the iris website another important thing that we care about from a performance point of view is la..."
Net Adapt: Tailoring Neural Networks for Performance
Vivienne Sze introduces the Net Adapt algorithm, which customizes neural networks for specific mobile platforms based on latency and energy budgets. This empirical approach allows for real-time tuning of neural networks, ensuring optimal performance on various hardware without the need for extensive modeling.
"actually much more challenging than that and so what we want to ask is is there a way that we can take latency and use that again to design the neural net correctly so rather than looking at max use l..."
Improving Latency and Accuracy Trade-offs
In this segment, Sze discusses the impact of the Net Adapt approach on the trade-off between latency and accuracy in neural networks. She presents data showing how this method can achieve faster performance while maintaining accuracy, illustrating its effectiveness in real-world applications.
"this is on the net adapt website I'm just to give you a quick example of how this might work so let's say you start off with it as your input a neural network you know that has the accuracy that you w..."
Depth Estimation with Efficient Networks
Sze explores the application of efficient neural networks for depth estimation tasks. She explains how techniques like Net Adapt can be applied to improve frame rates and accuracy in depth estimation, showcasing the potential of these methods in practical scenarios.
"network this particular you know example here is shown for an image classification task meaning I give you an image and you can classify it to the right you can say what's in the image itself you can ..."
Challenges in Hardware Design for Neural Networks
Vivienne Sze addresses the challenges faced by hardware designers in creating efficient neural network processors. She discusses the need for flexible hardware that can accommodate various algorithmic approaches and the importance of optimizing for energy efficiency and latency.
"correct depth so higher the better and so you can see you know the various different proaches out out there this star red star is the approach using fast a fast step using all the different efficient ..."
Optimizing Data Flow in Neural Network Hardware
In this segment, Sze explains the significance of optimizing data flow in neural network hardware. She discusses the trade-offs between high bandwidth and spatial reuse, emphasizing the need for innovative network architectures to support efficient computation in neural networks.
"so a very typical structure that you might see is that you might have an array of multiply and accumulate units so Mac array and it's going to reduce memory access by amortize amides across erase what..."
Flexible Data Flow in Neural Networks
Vivienne Sze discusses the importance of flexible data flow in neural networks, emphasizing the need for efficient data delivery mechanisms. She explains how different configurations, such as row stationary and multicast, can optimize data reuse across processing elements, enhancing performance in deep learning applications.
"reused by all of these is from cores and also be difficult to exploit sparsity so what you need here are two things one is a very flexible data flow meaning that there's many different ways for the da..."
Hierarchical Mesh Networks for Scalability
In this segment, Sze introduces the concept of hierarchical mesh networks as a solution for efficient data delivery in neural network accelerators. She highlights how this architecture balances the need for high bandwidth and low spatial reuse, enabling scalable and flexible processing for deep learning tasks.
"high bandwidth there's a lot of unique information going out and low spatial reuse they're not sharing data now it's very challenging to go across this entire spectrum one solution would be what we ca..."
Accelerating Super Resolution Techniques
Sze explores the challenges of super resolution in deep learning, particularly its computational demands. She presents a framework called 'fast' that aims to significantly accelerate super-resolution algorithms by leveraging compressed video data, thus improving efficiency without compromising quality.
"key things that enable the second version of iris to be both flexible and efficient right so this is some results from the second version of iris it supports a wide rater to filter state suppose the v..."
Leveraging Compressed Video for Efficiency
This segment focuses on how compressed video can be utilized to enhance super resolution processes. Sze explains the structured representation of compressed video and how it can be used to generate high-resolution frames efficiently, reducing the computational load typically associated with super resolution.
"also important to take a step back and look beyond just you know building specialized Hardware the accelerator itself both in terms of algorithms and the hardware so can we look beyond the DNA on acce..."
Quality vs. Efficiency in Super Resolution
Vivienne Sze compares traditional super resolution methods with her proposed approach using the 'fast' framework. She discusses the balance between processing speed and output quality, demonstrating how her method achieves high-quality results while maintaining efficiency.
"be very low so for example if I'm gonna transfer to n frames I'm gonna get an end frame an X speed-up so to evaluate this we showcase this on a range of videos this range of videos is the data set tha..."
Beyond Deep Neural Networks
In this concluding segment, Sze emphasizes the importance of exploring computing solutions beyond deep neural networks. She introduces visual inertial odometry as a critical application in robotics and discusses the need for efficient computing in various domains, highlighting the potential for specialized hardware.
"worlds and so the key takeaway for this is that if you want to accelerate dnns for a given process it's good to look beyond you know the hardware for the acceleration we can look at things like the st..."
Innovations in Visual Inertial Odometry
Sze presents a novel chip designed for visual inertial odometry, showcasing its capabilities in processing camera and IMU data efficiently. She explains how this specialized hardware achieves significant energy savings while performing complex localization tasks in robotics.
"is a very key task and navigation and the key thing is can you do in a fair energy efficient way um so we've looked at kind of building specialized hardware to do localization this is actually the fir..."
Planning and Mapping in Robotics
This segment delves into the planning and mapping challenges faced by autonomous robots. Sze discusses the concept of mutual information in navigation, explaining how robots can optimize their exploration strategies to gather the most information about their environment efficiently.
"below 25 milliwatts another thing that really matters for autonomous navigation is once you know where you are where are you gonna go next so this is kind of a planning and mapping problem and so in t..."
Parallel Processing for Efficient Navigation
Vivienne Sze concludes her lecture by addressing the challenges of data movement in parallel processing for robotic navigation. She proposes solutions for optimizing memory access patterns, enabling multiple cores to efficiently process data and enhance the performance of autonomous systems.
"the course themselves if we take a closer look at you know the memory access pattern you can see here that as you scan it out the numbers indicate which cycle you would use to read you know each of th..."
Revolutionizing Neurodegenerative Disease Monitoring
Sze explores the application of efficient computing in monitoring neurodegenerative diseases like Alzheimer's and Parkinson's. She discusses the challenges of traditional assessments and introduces innovative methods using eye movement analysis for more accessible and quantitative evaluations.
"compared to a CPU at 1/10 of the power right on an FPGA so that's another important example of how data movements really critical in order to allow you to process things very very quickly and having h..."
Affordable Eye Movement Measurement
Vivienne Sze presents a groundbreaking approach to measuring eye movements using consumer-grade devices like smartphones. She explains how this technology can provide low-cost, in-home assessments for patients, enhancing the quality of data collected for neurological evaluations.
"actually a quantitative way of measuring or quantitative evaluating these types of diseases potentially using eye movements right so I mean this can be used by quantitative way to evaluate the severit..."
Reducing Power Consumption in Depth Estimation
In this segment, Sze discusses the power challenges associated with depth estimation sensors and proposes methods to reduce their usage. She explains how efficient computing can allow for depth estimation using RGB cameras, significantly lowering power consumption while maintaining accuracy.
"camera so shown here in the red are basically eye reaction times that are measured on a subject on an iPhone 6 which is obviously under $1,000 way cheaper now compared to a phantom camera shown here i..."
Key Takeaways on Efficient Computing
Vivienne Sze summarizes the importance of efficient computing in extending AI applications beyond the cloud. She emphasizes the need for cross-layer design in hardware and algorithms to enable energy-efficient solutions for robotics and healthcare, and shares resources for further learning.
"okay so you know a high level what are the key takeaways I want you guys to get from today's lecture first is the efficient computing is really important it can extend the reach of AI beyond the cloud..."