All Segments
Sacha Arnoud, Director of Engineering, Waymo - MIT Self-Driving Cars
45 segments available
This is a talk by Sacha Arnoud for course 6.S094: Deep Learning for Self-Driving Cars (2018 version). Sacha is the Director of Engineering at Waymo and his talk is titled "The rise of machine learning in self-driving cars." This class is free and open to everyone. It is an introduction to the practice of deep learning through the applied theme of building a self-driving car. INFO: Course website: https://selfdrivingcars.mit.edu Contact: deepcars@mit.edu CONNECT: - If you enjoyed this video, please subscribe to this channel. - AI Podcast: https://lexfridman.com/ai/ - Show your support: https://www.patreon.com/lexfridman - LinkedIn: https://www.linkedin.com/in/lexfridman - Twitter: https://twitter.com/lexfridman - Facebook: https://www.facebook.com/lexfridman - Instagram: https://www.instagram.com/lexfridman - Slack: https://deep-mit-slack.herokuapp.com LINKS: Playlist: https://goo.gl/SLCb1y Lecture 1: Deep Learning - https://youtu.be/-6INDaLcuJY Lecture 2: Self-Driving Cars - https://youtu.be/_OCjqIgxwHw Lecture 3: Deep Reinforcement Learning - https://youtu.be/MQ6pP65o7OM Lecture 4: Computer Vision - https://youtu.be/CLOAswsxudo Lecture 5: Deep Learning for Human Sensing - https://youtu.be/Z2GfE8pLyxc Guest talk: Sacha Arnoud, Waymo - https://youtu.be/LSX3qdy0dFg Guest talk: Emilio Frazolli, nuTonomy - https://youtu.be/dWSbItd0HEA Guest talk: Sterling Anderson, Aurora - https://youtu.be/HKBhP9JISF0 2017: Guest talk: Sertac Karaman, MIT - https://youtu.be/0fLSf3NO0-s Guest talk: Chris Gerdes, Stanford - https://youtu.be/LDprUza7yT4
Segments Timeline
Introduction to Sacha Arnoud
Sacha Arnoud, Director of Engineering at Waymo, introduces himself and expresses excitement about sharing insights on self-driving cars. He highlights Waymo's achievements, including driving over four million miles autonomously, and sets the stage for discussing the advancements in machine learning and engineering in the self-driving space.
"today we have the director of engineering head of perception at way mo a company that's recently driven over four million miles autonomously and in so doing inspired the world in what artificial intel..."
Objectives of the Talk
Sacha outlines the three main objectives of his presentation: providing background on the self-driving industry, sharing technical insights on current machine learning techniques used in self-driving cars, and discussing the complexities of building such sophisticated systems. He emphasizes the importance of understanding both the technical and contextual aspects of self-driving technology.
"welcome to Sasha our new [Applause] thanks a lot Lex for the introduction well it's it's a pretty packed house thanks a lot I'm really excited thanks a lot for giving me the opportunity to to be able ..."
The Evolution of Self-Driving Cars
Sacha discusses the rapid evolution of self-driving cars, particularly in 2017 when Waymo became its own company. He describes the latest generation of self-driving vehicles, including the Chrysler Pacifica, and emphasizes the potential of self-driving technology to transform mobility, improve safety, and enhance efficiency in urban environments.
"that that I'd like to convey today so keep that in mind as we go through the through the presentation my first one is is to give you some background around the self-driving space and what's happening ..."
Safety and Accessibility in Self-Driving Technology
Sacha highlights the motivations behind self-driving technology, focusing on safety, accessibility, and efficiency. He notes that 94% of crashes involve human error and discusses how self-driving cars can reduce accidents, improve mobility for disabled individuals, and optimize traffic flow in urban areas.
"a very hot topic and for very good reasons I can tell you for sure that 2017 has been a great year for whammo actually only a year ago in January 2017 when Moe became its own company so that was a maj..."
The Journey of Waymo's Development
Sacha shares the history of Waymo's development, starting from its inception in 2009 as a Google project. He recounts the initial challenges faced by the team in assembling a self-driving vehicle and the ambitious goal of completing 1,000 miles of autonomous driving in complex environments.
"last but not least is efficiency a collective efficiency so not only we spend a lot of time in our cars in in long commute hours I personally spend a lot of time in on commit hours and that time we sp..."
Challenges in Autonomous Driving
Sacha elaborates on the challenges encountered during the early testing phases of self-driving cars, including navigating through the Santa Cruz Mountains and dense urban areas. He emphasizes the importance of rigorous testing in diverse conditions to ensure the reliability of self-driving technology.
"those days so remember we were before the deep learning days at least in the industry and so really back in those days the the first the first objective of the project was to try and assemble first pr..."
Milestones Achieved in Self-Driving
Sacha discusses significant milestones achieved by Waymo, including the successful completion of autonomous driving loops and the decision to remove safety drivers from vehicles. He shares a video showcasing the first instances of fully autonomous driving, highlighting the progress made in the technology.
"conditions those routes were going around bridges and the Bay Area has quite a few bridges to go through though some of them were even going through a dense urban area so you can see San Francisco bei..."
The Road to Full Autonomy
Sacha reflects on the journey from initial prototypes to achieving full autonomy in self-driving cars. He emphasizes the extensive work required to transition from a working demo to a safe, reliable system ready for public roads, highlighting the need for continuous improvement and testing.
"eight years ago so on that on the heels of that success the team decided and Google decided that self-driving was worth worth pursuing and moved and moved forward with the development of the technolog..."
The Rise of Deep Learning
Sacha provides insights into the rise of deep learning and its impact on self-driving technology. He discusses the breakthroughs in machine learning that have enabled advancements in the field and shares his experiences working on projects that utilized deep learning for practical applications.
"quick capture of that event so that the video is from one of the first times we did that since then we've been continuously operating drug arrest cars self-driving cars in the Phoenix area in Arizona ..."
Deep Learning Applications in Mapping
Sacha explains how deep learning has been applied to improve mapping technologies at Google. He shares examples of using machine learning to analyze street imagery and enhance the accuracy of Google Maps, demonstrating the broader implications of deep learning beyond self-driving cars.
"are in 2018 and were getting there but what it took it took quite a bit of time so I think one of the one of the key ideas that I'd like to convey here today and that I will I will go back to during r..."
Conclusion and Future Directions
Sacha concludes by summarizing the key points discussed in his talk, emphasizing the transformative potential of self-driving technology and deep learning. He encourages continued exploration and innovation in the field, highlighting the importance of collaboration and research in advancing autonomous driving.
"actually it took a lot of breakthroughs to to be able to reach that stage and one of them was the Agora algorithm breakthrough that deep learning gave us and I'll give you a little bit of of backstage..."
Mapping Addresses with Deep Learning
Sacha Arnoud discusses the importance of accurately mapping street numbers and names for self-driving cars. He explains how combining data from various sources can enhance the quality of address book applications and improve navigation systems. This segment highlights the breakthrough in extracting street numbers from Street View imagery, which significantly aids in localizing business listings and traffic patterns.
"and and properly localized would drastically help you build better maps so street numbers obviously that are really useful to map addresses street names that when combined event on similar techniques ..."
Real-World Applications of Mapping
In this segment, Arnoud showcases a video demonstrating the successful detection and transcription of house numbers in various cities, including Sao Paulo and Paris. He emphasizes the impact of deep learning on improving map accuracy and the challenges faced in real-time processing for autonomous vehicles. The discussion highlights the necessity of high-quality data for effective navigation and mapping.
"mentioned one of the hot piece is to do is to map addresses at Cal and so you can imagine that we had a breakthrough when we first were able to properly find those street numbers out of the Street Vie..."
Challenges of Real-Time Processing
Arnoud elaborates on the complexities of processing data in real-time for self-driving cars. He explains the need for autonomous vehicles to operate without relying on external data centers, emphasizing the importance of low-latency processing. This segment discusses the evolution of deep learning applications in pedestrian detection and the necessity of handling various driving conditions.
"video that kind of sums it up so look every one of those segments is actually a view from starting from the car going to the physical number of all those house numbers that we've been able to detect a..."
Perception Systems in Self-Driving Cars
This segment focuses on the perception systems that enable self-driving cars to understand their environment. Arnoud explains how these systems utilize prior knowledge of the scene and real-time sensor data to build a comprehensive representation of the surroundings. He discusses the importance of mapping and sensor integration for effective navigation and safety.
"doing that on the car is even harder is even harder because you need to do that rigor time and and very quickly with low latency and you also need to do that in in an embedded system right so the cars..."
The Role of Sensors in Navigation
Arnoud details the various sensors used in self-driving cars, including vision systems, radar, and LiDAR. He explains how these sensors complement each other to provide a comprehensive understanding of the environment. The segment emphasizes the significance of high-quality sensors in enhancing the perception system's effectiveness.
"real-time and if you saw the cyclist going through so you have air stuff happening on the scene that you need to detect and properly understand interpret and predict and at the same time he expressed ..."
Understanding Complex Driving Scenarios
In this segment, Arnoud discusses the complexities of driving scenarios that self-driving cars must navigate. He highlights the need for advanced perception systems to anticipate the behavior of other road users, such as cyclists and pedestrians. The segment underscores the importance of deep learning in improving the decision-making capabilities of autonomous vehicles.
"looked a little bit at how things happened I want to spend more time and and go into more of the details of what's going on in the cars today and how deep learning is actually impacting our current sy..."
Behavior Prediction in Autonomous Driving
Arnoud explains the critical role of behavior prediction in autonomous driving. He illustrates how understanding the intentions of other road users, such as cars and pedestrians, is essential for safe navigation. This segment emphasizes the need for deep learning to enhance the predictive capabilities of self-driving systems.
"piece which is which is a core element of what the self-driving car needs to do so what is what is perception so fundamentally set perception is assist in a system in the car that needs to build an un..."
Filtering Sensor Data for Safety
This segment focuses on the importance of filtering sensor data to ensure safe operation of self-driving cars. Arnoud discusses the challenges posed by irrelevant data, such as reflections and environmental noise, and how deep learning can help in accurately interpreting sensor inputs. The segment highlights the necessity of robust data processing for effective decision-making.
"so as we saw on the initial picture we have quite a set of sensors on our self-driving cars so they go from vision systems radar and later how the other three big families of sensors we have one point..."
Enhancing Sensor Capabilities
Arnoud discusses the development of in-house sensors at Waymo to improve the perception system of self-driving cars. He explains how enhancing sensor capabilities is crucial for building a reliable autonomous driving system. This segment emphasizes the integration of advanced technologies to achieve better performance in real-world driving conditions.
"so all those sensors are designed to be complimentary in terms of their capabilities it goes without saying that the better your sensors are the better your perception system is gonna be right so that..."
Deep Learning's Impact on Driving Safety
In this segment, Arnoud elaborates on how deep learning is transforming the safety of self-driving cars. He discusses the need for a deep understanding of complex driving scenarios, including interactions with police cars and cyclists. The segment highlights the importance of advanced perception systems in making informed decisions to ensure safe navigation.
"hit them you're good enough in most cases when you don't have a driver on the dragon seat obviously the challenge totally changes scale so to give you an example for instance if you're if you're on th..."
The Future of Autonomous Driving
Arnoud concludes by discussing the ongoing challenges and future directions for autonomous driving technology. He emphasizes the need for continuous improvement in perception systems and deep learning applications to enhance safety and reliability. This segment encapsulates the vision for the future of self-driving cars and the role of technology in achieving it.
"scene if you understand that this car is parked and we see this a variable piece of information that's going to tell you whether you can pass it or not something you may have not noticed is that there..."
The Challenge of Sensor Data
Sacha Arnoud discusses the complexities of using sensor data in self-driving cars, emphasizing that real-world data is often imperfect. He highlights the importance of filtering irrelevant data, such as exhaust smoke from a pickup truck, to improve behavior prediction and driving safety. This segment underscores the challenges faced in robotics and the necessity of effective data processing.
"for all those agents in the in on the scene so that you can come up with a proper strategy for your planning control so how is a deep learning playing into that whole space and how he is a deep learni..."
Understanding Reflections in Driving
In this segment, Arnoud explains the complications that arise from reflections in sensor data, such as a car's reflection in a bus window. He illustrates how naive detection can lead to misinterpretations, potentially causing dangerous driving decisions. This highlights the need for advanced algorithms to discern real objects from reflections in the context of autonomous vehicle navigation.
"ignore in terms of sin understanding right so filtering the whole whole bunch of data coming off your sensors is is a very important task because that reduces the computation you're gonna have to do w..."
Convolutional Layers in Computer Vision
Arnoud introduces convolutional layers as a fundamental technique in computer vision for self-driving cars. He explains how these layers help in feature extraction from images, allowing the system to recognize patterns such as lines and contours. This segment emphasizes the efficiency of convolutional networks over traditional fully connected layers in processing visual data.
"per hour trajectory so that's a big that's a big complicated challenge but assume we are able to get to a proper sensor data that we can start the process with our machine running so by the way a lot ..."
Projecting Sensor Data for Processing
This segment covers the projection of sensor data into 2D planes for better processing in self-driving applications. Arnoud discusses the advantages of top-down and driver-view projections, explaining how they facilitate the understanding of the driving environment. He emphasizes the importance of accurately registering sensor data to enhance scene comprehension.
"image that's a very common technique and much more efficient we slid and fully connected layers for instance that wouldn't work but unfortunately a lot of a lot of the state of the art is actually in ..."
Segmentation Techniques for Object Detection
Arnoud elaborates on segmentation techniques used to group sensor data into recognizable objects. He discusses the challenges of detecting irregularly shaped objects, such as snow or trash bags, and introduces the sliding window approach for efficient processing. This segment highlights the computational demands of object detection in autonomous driving.
"better understand the scene so the first the first kind of processing you can do is is is what is called their segmentation so once you have pixels or laser points you need to group them together into..."
Using Shape Priors for Efficient Detection
In this segment, Arnoud explains how predefined shape priors can enhance the efficiency of object detection in self-driving cars. He discusses the concept of single-shot multi-box detectors, which allow for quick identification of objects like cars and road signs. This approach reduces computational load while maintaining accuracy in detecting essential elements on the road.
"need to care about have predefined priors so Francis if you take a car from the bird from the top down view from the birds view it's gonna be a rectangle you can you can take that that shape prior int..."
Understanding Emergency Vehicle Semantics
Arnoud addresses the importance of recognizing emergency vehicles and their specific semantics in driving scenarios. He discusses the need for classifiers that can identify various states of emergency vehicles, such as school buses with activated lights. This segment emphasizes the complexity of interpreting visual data in real-time driving situations.
"efficient way to get to get that data so we talked about the member the flashing lights on top of the police car so even if you if you properly detect and segment cars let's say on the road many cars ..."
The Complexity of Pedestrian Detection
This segment focuses on the challenges of detecting pedestrians in various poses and situations. Arnoud highlights the need for high recall rates in pedestrian detection due to their unpredictable behavior. He discusses the importance of understanding the context, such as a pedestrian's movement relative to vehicles, to ensure safe navigation.
"further further processing so those embeddings have been actually historically they've been more closely associated with word embeddings so in a typical text if you were able to build those vectors wi..."
Tracking Objects Over Time with RNNs
Arnoud concludes by discussing the use of recurrent neural networks (RNNs) for tracking objects over time in self-driving cars. He explains how RNNs can improve understanding of object behavior by analyzing sequential observations. This segment highlights the significance of temporal data in enhancing the decision-making capabilities of autonomous vehicles.
"move in any direction that car moving that direction you can safely bet connect it's gonna it's not a drastic keychain angle in in a moment's notice right but if you take children for instance it's a ..."
The Importance of Labeling Data
Arnoud highlights the critical role of high-quality labeled data in training machine learning models for self-driving cars. He discusses the challenges of supervised learning and the need for extensive labeling efforts to create representative datasets. This segment underscores the significance of data quality and quantity in developing effective autonomous driving systems.
"is those vector representation combined with recurrent neural networks is a common technique that that can help you figure that out back to the point when you're 90% done you still have 90% to go and ..."
Advancements in Data Collection
This segment focuses on the advancements in data collection techniques for self-driving cars. Arnoud explains how modern methods allow for more efficient generation of labeled datasets, including the use of machine learning to assist in labeling. He emphasizes the evolution of data collection from earlier methods to current practices that enhance the training of autonomous systems.
"it still is revised so to give you orders of magnitude so here represented in a logarithmic scale the size of a couple data sets so you may be familiar with image net which i think is the 15 million o..."
Computational Power and Infrastructure
Arnoud discusses the necessity of robust computational power and infrastructure for training and deploying self-driving car models. He explains how Google's advanced hardware accelerators, like TPUs, facilitate efficient deep learning processes. This segment highlights the technological backbone required to support the complex algorithms driving autonomous vehicles.
"that combining those those techniques together obviously you can get you can get to completion faster it's very common to still need so that in the minions minions range kind of same pose to train a r..."
Testing and Validation Strategies
In this segment, Arnoud outlines the three-pronged approach to testing self-driving systems: real-world driving, simulation, and structured testing. He emphasizes the importance of extensive testing to ensure safety and reliability in autonomous vehicles. This discussion provides insight into the rigorous validation processes that underpin the development of self-driving technology.
"teams to collaborate and work together that's a data representation in which you can represent your your label data sets for instance or your training batches that's a runtime that that that you can d..."
Driving Data Accumulation
Arnoud shares statistics on the accumulation of driving data by Waymo, illustrating the scale of their testing efforts. He highlights the significance of diverse driving conditions and environments in building a robust dataset for training self-driving cars. This segment emphasizes the vast amount of experience that machine learning systems can leverage from extensive real-world driving data.
"and and with a high safety bar is around your testing program so we have three legs that that we that we use to make sure that we our machine running is ready for production one is around we are what ..."
Simulation as a Testing Tool
This segment focuses on the role of simulation in testing self-driving systems. Arnoud explains how simulation allows for efficient testing of software updates without the need for extensive real-world driving. He discusses the capabilities of internal tools that enable the modification of driving scenarios to test various situations, enhancing the robustness of the autonomous driving system.
"years of experience that your machine learning can tap into to learn to learn what to do even more importantly is your ability to simulate obviously the software changes regularly so if for each new r..."
Future Directions in Self-Driving Technology
Arnoud concludes with insights into the future directions of self-driving technology, including expanding operational domains and enhancing semantic understanding. He shares personal experiences and challenges faced in complex driving environments, emphasizing the need for deeper comprehension of human driving behaviors. This segment provides a forward-looking perspective on the ongoing development of autonomous vehicles.
"that you need to cover so a couple orders of magnitude here so using Google's infrastructure we have the ability to run a vehicle fleet of 25,000 cars 24/7 in data centers so those those are those are..."
Technical Solutions for Production Systems
Arnoud discusses the technical and algorithmic solutions necessary for implementing self-driving technology in production systems. He stresses the importance of engineering infrastructure and the scale of work required to transition from theoretical concepts to real-world applications in self-driving cars.
"that's an example of a direction so back to my objectives I hope I covered many of those at least you have you have directions to for further reading and investigations on those those three objectives..."
Perception and Planning Challenges
This segment addresses the intersection of perception and planning in self-driving cars. Arnoud explains how planners may assume ideal conditions that perception systems cannot always deliver, and he discusses the use of simulation environments to test and validate perception capabilities.
"tweet that's a scene with a kids on jumping on bags and as Frogger of course the scene and I think we have time for a few questions [Applause] since tend to fail at this intersection between perceptio..."
Testing Perception Systems
Arnoud elaborates on how Waymo tests its perception systems by measuring performance across real-world scenarios. He explains the importance of understanding and reproducing mistakes made by the perception system to enhance the planning capabilities of self-driving cars.
"between perception and planning so your planner might assume something about a perfect world that perception cannot deliver so what's wondering if you use the simulation environment also to induce the..."
Choosing the Right Architecture
In this segment, Arnoud discusses the complexities of selecting the architecture for embedded systems in self-driving cars. He highlights the importance of research collaboration and the need for a robust team to explore various solutions and ensure effective production systems.
"produced at scale do you have a systematic way of creating the architectures of the embedded system you have so many choices for sensors algorithms each problem you showed has many different solutions..."
Generalizing Decision-Making
Arnoud addresses the decision-making process of self-driving cars, emphasizing the need for algorithms to learn general principles rather than memorizing specific scenarios. He discusses the importance of diversity in training data to prepare the system for a wide range of driving situations.
"our development environment our testing is really key to be able to grow that that team has that the biggest scale and essentially explore all those paths and come up with the best one right and at th..."
Semantic Understanding of Environmental Factors
This segment focuses on the challenges of identifying environmental factors like snow in self-driving scenarios. Arnoud discusses the need for a wide array of object embeddings and the balance between computational feasibility and effective scene understanding for safe driving.
"system new situation occurs okay okay fantastic talk one of the questions I had was you mentioned the difficulty of identifying snow because they could come in many different shapes one things that I ..."
Addressing Perception Errors
Arnoud concludes by discussing the potential for perception errors in self-driving systems, including adversarial examples. He emphasizes the importance of sensor redundancy and semantic understanding to mitigate these errors and ensure safe navigation in complex environments.
"you need to drive essentially right so yeah it's an in-between last question make it a good one thanks for the talk so if you're using perception for your scene understanding are you worried about lik..."