0:00
So, I'm going to tell you a very not super deep into anyone topic, but very broad brush sense of the kinds of things we've been using deep learning for, the kinds of systems we've built around making deep learning faster.
Search this video
0:16
And this is joint work with many, many, many people at Google, so this is not purely my work.
0:21
But most of it is from the Google Brain team, which I lead.
0:24
And uh So, the Brain team's mission is basically make machines intelligent and then use that new capability to improve people's lives in a number of different ways.
0:36
And the way we do this is we conduct long-term research kind of independent of any particular application in Oh, sorry.
0:43
I'm probably supposed to stand in one place.
0:46
Uh independent of any particular uh uh application, we build an open-source uh systems that help us with our research and deploying of machine learning models like TensorFlow.
1:02
We collaborate across Google and all of Alphabet in getting machine learning systems and research that we've done into real Google products.
1:10
So, we've done a lot of work in like Google Search, Gmail, Photos, uh speech recognition, Translate, and you know, many other places.
1:18
Um we also bring in a lot of people into our group uh through internships and a new residency program that we started last year for people who want to learn how to do uh deep learning research.
1:29
And that's been a pretty successful program as well.
1:33
Um So, the main research areas that our group is working in are these.
1:39
I'm going to focus mostly on these today.
1:42
Actually, a little bit of perception, too.
1:45
Uh but um So, uh in January I put out a blog post that kind of just highlighted some of the work our group has done over the last over 2016.
1:54
And in putting that together, I kind of realized we were doing a lot of different things.
2:00
So the nice thing about this is each one of these blue links is a link to something kind of interesting and substantial like a research paper or a product launch using machine learning or some new TensorFlow features we've we've added.
2:17
So I won't go through that all now but you can go find that blog post and learn more about some of the stuff we've been up to.
2:25
Okay, so why are we here?
2:25
You probably already all know this given that you're working on AI related companies as I understand it.
2:32
But um the field of deep learning and neural networks in particular are really causing a shift in how we think about approaching a lot of problems.
2:43
And I think it's really changing the kinds of machine learning approaches that we use.
2:46
In the 80s and 90s it was the case that neural nets seemed interesting and appealing but they weren't the best solution at the time for a lot of problems that we cared about cuz they just didn't quite have enough training data, enough computational capabilities.
3:05
And so people used other methods or developed kind of shallower machine learning methods with much more hand engineering features.
3:13
And if you fast forward to now what's happened is we've got much much more compute.
3:16
I actually did an undergrad thesis in 1990 on parallel training of neural nets cuz I liked the appealing attraction of the neural net model and I thought if we could just get like you know a bunch more compute by parallelizing over a you know a 64 processor hypercube machine that it would all be even better.
3:36
It turned out what we needed was like a 100,000 times as much compute not 60 times.
3:41
But if you fast forward to today we actually have that and so what's happened is we now actually have you know the case where neural nets are the best solution for an awful lot of problems and a growing set of problems where we either previously didn't really know how to solve the problem or where we could solve it, but now we can solve it better with neural nets.
4:02
And so, the talk is really meant to uh orient you across a whole bunch of different problems where this is the case.
4:11
Uh so, growing use of deep learning.
4:11
So, really our group started in order to investigate the hypothesis that large amounts of compute could actually solve interesting uh problems using neural nets.
4:22
And so, when we first started, you know, there weren't we were sort of the vanguard of people using neural nets at Google.
4:29
Uh we did a bunch of work on unsupervised learning uh at a really large scale using at that time we didn't even have GPUs in our data centers, so we just used 16,000 CPU cores and we did kind of interesting things with unsupervised learning there.
4:44
Uh but gradually we kind of built tools that enable people to apply machine learning and deep deep learning in particular to a lot of problems and you can see the growth rate of uh you know, this is directories containing model description files either from our first generation system and starting in about uh 2015, our second generation system, TensorFlow.
5:07
Uh and we've deployed machine learning in in collaboration with lots of teams.
5:11
Other teams have also been independently just picking up this idea of deep learning and using it in lots and lots of places in Google products.
5:19
And that's why you see that growth rate and it's continuing to go up.
5:23
Um one of the things we focus on a lot is how can we reduce experimental turnaround time for our uh machine learning experiments and because there's a very different qualitative feel to doing science and research in a domain where an experiment takes a month versus doing it in a domain where, you know, minutes or hours you get uh you know, an answer and then you can figure out what the next set of experiments are that you want to run.
5:50
So, a lot of our focus is on scaling machine learning models and scaling the underlying infrastructure and systems so that we can actually uh for some problems approach minutes or hours rather than uh weeks or months.
6:05
Uh so, part of that has been building the right tools.
6:06
So, TensorFlow is kind of our second generation uh system that we built for tackling deep learning problems and machine learning problems.
6:14
Um the first one we didn't open source.
6:16
The second one we said we should really uh fix some of the design problems we saw in our first system, keep the good features about it, and then design this from the start to be an open source platform so that people all over the world, not just at Google, can benefit from from this and can help build a community that can all contribute to and improve uh the system.
6:39
Uh Zack Stone here uh is our TensorFlow uh product manager extraordinaire and is uh um uh been doing a great job of building the community both in inside Google and outside.
6:53
Uh So, the goals of TensorFlow that we want to establish this common platform for expressing all kinds of machine learning ideas.
7:00
So, it's something that can be used for deep learning, can be used for other kinds of machine learning, that can be used for tackling perception problems and uh you know, language understanding problems.
7:09
Um and what if you have a crazy new machine learning research idea that doesn't really fit into what people have done before, we want it to be at least expressible relatively easily in TensorFlow.
7:22
Um and then we want to make that platform really great for research, but we also want to be able to take uh that something you've developed in TensorFlow maybe experimentally and then if you now want to deploy that in a production setting, run it at a data center, run it at scale, uh run it on a phone, all these kinds of things, we want that it to be uh another something you can do in this the TensorFlow framework.
7:47
And by open sourcing it, we make it available to everyone.
7:51
Um so, how's this been going?
7:56
Well, uh so this is a comparison of GitHub stars of which is one metric of popularity or interest in different uh source code repositories on GitHub.
8:07
And I show you a comparison of TensorFlow with a bunch of other open source machine learning packages.
8:12
Uh many of which have been around for for uh you know, many more years than TensorFlow.
8:17
TensorFlow is this brown line up uh going up fairly steeply.
8:22
Um so, this has been pretty good.
8:22
I think the the reception for what TensorFlow does, which is enable flexible research, but also this kind of production readiness and uh being able to run in lots of places uh is pretty appealing.
8:36
Um And if you look at the other open source packages which we did when we were starting to work on TensorFlow, you know, many of them have two of the three attributes that we care about, being able to be really flexible, uh scalable, and uh sort of run on any platform.
8:50
Uh and they all have different emphases, uh but we wanted something that satisfied all three of those.
8:57
Uh so, that's kind of cool.
8:59
Uh we've been focusing a fair amount on speed.
9:01
I think when we first released TensorFlow, we released a bunch of really nice tutorials uh that showed how to do different things with TensorFlow, but one of the mistakes we made was we released code that was meant to be expository and clear and not necessarily the little highest performance way you would write that.
9:19
But often then people would take that as the way you should write a high performance TensorFlow model, and that wasn't necessarily the case.
9:27
So, we're now adapting and trying to put out things that are both the best way from a clarity standpoint, but also are high performance.
9:35
Uh that's been a bit of a uh, TensorFlow got I think a bit of a bad rap, but actually our performance is quite good.
9:43
So, we've been uh, doing a bunch of benchmarking and producing reproducible benchmark results to show that our scaling is quite good.
9:51
So, this is single machine scaling, nearly linear speed speed up for a bunch of different uh, image models on up to eight GPU cards.
10:01
Uh, pretty close to linear speed up for 64 GPU cards for a bunch of different kinds of problems.
10:08
Uh, so don't don't if you hear TensorFlow is slow, don't don't believe it.
10:15
Um, we also support lots of different platforms and I think this is important because often you want to train a model on a large data set in a data center, but then deploy that on a phone.
10:24
And so, we run on, you know, iOS and Android and Raspberry Pis, but also CPUs and uh, if you have a GPU card or eight GPU cards, we're happy to use that.
10:34
Uh, we also run on our custom machine learning accelerators that I'll talk about in a minute.
10:41
Um, but really we want to run on everything.
10:42
So, there's a bunch of other device manufacturers that are developing kind of funky mobile uh, ML accelerators or Qualcomm has a DSP and they're all working to make sure that TensorFlow runs well on those devices.
10:57
Uh, we also kind of want to be agnostic of language that people want because you want to be able to run machine learning where makes sense and different people have different uh, sort of language environments.
11:08
Um, most the most fully developed system is obviously Python, uh, but the C++ front end works pretty well for production use and then a bunch of other people uh, external community members have added a variety of other kind of not fully fleshed out, but reasonable support for some of these other languages.
11:29
Uh, we have a pretty broad uh, usage base.
11:31
So, uh, like a year ago, we had a almost a year ago, we had a meeting at Google of people using, uh, TensorFlow and it was pretty impressive.
11:41
We had people from most of these companies in the room, which I think normally they don't all get in a room together.
11:47
Places like Apple was actually there as well, Nvidia, Qualcomm, Uber, Google, Snapchat, you know, many many Intel.
11:56
Uh, many many other places.
11:58
Uh, so, in terms of stars, you know, I showed you the graph related to machine learning platforms.
12:03
This is the top, uh, repositories on GitHub overall and we're up to number six, which is pretty good.
12:12
Um, and all the other ones are either JavaScript or a list of programming books.
12:20
Uh, this is a visualization of where people are interested in different GitHub repositories, uh, which is kind of cool.
12:27
Uh, machine learning is done all over the world.
12:31
Uh, so, one of the things that's happened as that growth in interest has happened is is there's been a a pretty broad set of external to Google contributors and so there's really, you know, I think we're up to almost a thousand non-Google contributors across the world doing all kinds of different things for uh, adding features or fixing bugs or improving the system in various ways, which has been really nice.
12:58
Oh, and I think it's kind of nice that there's growing use in machine learning classes of TensorFlow as a way of illustrating machine learning concepts.
13:06
So, at really good machine learning universities like Toronto, Berkeley, Stanford, and other places, they're starting to use that as the core of their curriculum. Uh, okay.
13:18
So, now I'm going to switch gears a bit and talk about some sort of more product-oriented applications of deep learning at Google.
13:22
Uh, Google Photos is a good example.
13:25
Obviously, computer vision now works and one thing you could do is, you know, make a photos this uh uh product around the idea that you can actually understand what's in people's photos and that's been going really well.
13:38
Um as a lesson for for you uh who are starting companies often applied domains, I think it's really important to be able to look at the machine learning work that is happening in the world and realize that often you can reuse many of the same ideas from one domain and just by pointing it at kind of different data sets get completely different interesting product features.
14:04
So, if you um for example, use the same basic model structure, train it on different data and you get something different.
14:09
One general model trend is given an image, predict interesting pixels.
14:13
So, there's a bunch of, you know, uh ways you could do that, but if you have a model structure that does that, uh my uh summer intern from a few years ago, Matt Zeiler, who actually went off to found Clarifai, which is a computer vision company, um uh we were working in collaboration with the Street View team on uh identifying text in Street View images.
14:36
And so, to do that, you can have training data where people have circled or drawn boxes around text and you just try to predict a heat map of which pixels contain text in a Street View image.
14:48
And so, uh this works reasonably well and then you can run an OCR model on those pixels and actually read the text.
14:55
Uh and it works, you know, across lots of different font sizes and colors and whether it's close or far from the from the camera.
15:05
Um and so then some people in the Maps team decided they would build this thing that would help you uh identify whether your rooftop has solar energy potential and how much energy you could generate by installing solar panels.
15:19
Uh and so, obviously, one of the first things you have to do is find rooftops.
15:22
And that's exactly the same model, but just with different training data, where you now have circles around rooftops.
15:28
And then there's a bunch of other work to estimate the angle of the rooftop from the the imagery or multiple views of the same same house.
15:36
Uh and then some stuff to predict, you know, what is the solar energy potential for that.
15:41
Um and another area where we've applied this is in the medical domain.
15:44
So, the same basic model, we want to be able to say take a medical imaging problem.
15:50
Uh one of the first ones we've been tackling is uh uh ophthalmology problems, and in particular, taking a retinal image like this and tackling deciding whether or not this has symptoms of a degenerative disease called diabetic retinopathy.
16:07
And so, this is a again the same kind of problem.
16:09
You want to identify parts of the eye that are related to uh you know, that seem to be diseased in some way.
16:17
Um and then you also have a whole image classification problem of does this eye show symptoms at the level of 1 2 3 4 or 5?
16:26
And turns out you can do this.
16:26
So, some people in our group uh have done a really nice uh sort of uh medical study uh showing that if you collect 150,000 ophthalmology images and then you get each one labeled by seven ophthalmologists because if you ask two ophthalmologists to grade the same image 1 2 3 4 or 5, they agree 60% of the time.
16:51
Just slightly terrifying.
16:51
If you ask the same ophthalmologist to grade the same image a few hours later, they agree with themselves 65% of the time.
17:00
And and that's mildly terrifying.
17:00
So, we had to get every image labeled by seven ophthalmologists to reduce the variance in the score and say, "Oh, it five people think it's a two, so it's probably more like a two than a three."
17:11
Um but in any case, the the punchline of this paper is we now have a model that performs on par slightly better than the median of eight US board-certified ophthalmologists.
17:22
Uh which is cool because there's a bunch of places in the world, especially in India and other countries where there are, you know, many people at risk and there just aren't enough ophthalmologists.
17:33
Um so we're actually doing clinical trials in India.
17:34
We've licensed this to our Verily subsidiary who's licensed it to a camera and ophthalmology camera manufacturer who's going to be uh integrating this into the actual ophthalmology camera.
17:46
Um another area where being able to see is pretty useful is uh robotics.
17:52
If you're trying to build robots, just being able to perceive the world around you it clearly makes things a lot better.
17:58
Um so we've been doing a bunch of experiments both with real real robots and also with uh simulated uh robotic environments and also with trying to do imitation learning from people performing actions and then trying to get robots to do it.
18:13
So we set up a uh what we call an arm farm. Oops. Let's see. Why is that not playing?
18:20
Oh, maybe I'm not on the internet. Well, anyway.
18:25
Uh it's not that exciting except that we have a bunch of robots trying to grasp things and they can essentially learn to gra- learn on their own whether they're grasping something successfully by just having a bin of things in front of them and they just try to pick something up.
18:39
And if they fail, their gripper closes all the way.
18:42
If they succeed, then uh they don't close their gripper all the way and they can actually see from the camera that they've managed to pick something up.
18:51
And so they can practice picking things up and we can pool all the sensor data from all the robots that are doing this to uh retrain a model every night uh for grasping so that the next day's grasping attempts are better and better.
19:04
And by having lots of robots do this, you actually get a lot of parallel experience, much more than you can get on a single robotic arm.
19:11
And so, we have a data set of that we've actually released publicly of about 800,000 grasp attempts uh versus about 30,000 grasp attempts, which was kind of the big public data set in the past.
19:24
And surprisingly, 800,000 grasp attempts gives you a much better grasping mechanism and model than 30,000. Not surprising.
19:34
Uh we've also been trying to This is me awkwardly looking at a robot on a screen which you can't see doing some actions.
19:42
I'm trying to like mimic the robotic nature of it.
19:45
And then we have a video of me doing that.
19:46
And then we're just trying to learn from the videos to transfer that action to the real robots.
19:57
And that's working reasonably well.
20:00
Uh here's another example.
20:00
And we're doing that first you via simulator, and then we're taking that simulator and then trying to transfer those activities to a real real robot.
20:10
Um and that works reasonably well as well.
20:15
Um another place that I'm pretty excited about deep learning is in lots and lots of scientific domains, you often have the case where you have a uh uh simulator of some really complex phenomena.
20:31
And that's often a sort of HPC style application and very computationally expensive, but it kind of gives you insight into whatever scientific processes are going on, and that allows you to kind of iterate in a computational science uh methodology.
20:47
Um but often those computations are pretty expensive.
20:51
And so, one of the things we've been working on, and this is just one example.
20:54
We have a lot of different fields of science where we've seen this to be true, is you can use those simulators as training data for a neural net.
21:01
So, uh quantum chemists have a sim They have a problem where they take in a configuration of molecules and they run a bunch of time steps and then at the end they get some information about how the ultimate configuration of those molecules uh turns out and from that they get a few properties about those molecules. Like is it toxic?
21:21
Did it bind with something else?
21:23
Um you know, a handful of these things.
21:25
So, it turns out that's the data that if you use that as input, you run this really expensive uh simulator for an hour and then you get these 13 numbers out.
21:35
Uh that turns out to be great training data for a neural net.
21:40
And so, you can train a neural net to do exactly that same task or to approximate that task, approximate the entire simulator.
21:46
Uh and you can essentially the punchline is the bottom there.
21:51
You essentially get indistinguishable accuracy from using the real simulator, but it's 300,000 times faster.
21:58
And that has a lot of implications for how you might do quantum chemistry if you suddenly have something that's 300,000 times faster.
22:06
Uh you might like run 100 million things through your through your uh simulator uh neural net-based simulator and figure out what's going on.
22:16
Uh look you know, to identify a bunch of candidates that you might want to look into in more detail. Uh so, that's exciting.
22:22
Uh another place where these kinds of pixel-to-pixel models come in uh is uh some people in Google have done a model that tries to predict depth from an input image and we have some training data where we have the true depth given uh the camera viewpoint and where things are in the room or in the world.
22:43
And then we try to train a model to do predicted depth from just the raw pixels of that image.
22:50
So, that's a pixel-to-pixel learning problem and you can imagine a lot of pixel-to-pixel learning problems.
22:57
And indeed, you know, one one application in in uh cameras is you want to predict depth in a portrait and then you can do kind of funky cool effects like identify the person in the foreground and turn the background black and white or like make it all fuzzy and artsy in the background um which is kind of cool.
23:15
Uh but it turns out you can also take microscope microscope images as uh the raw microscope images input and then the uh chemically stained uh microscope image as the target for your model.
23:32
And so, for example, that's often how people see, you know, cell bodies and cell boundaries is you apply different kinds of stains to the cells and then you can make them show up on a microscope better and you can see what's going on.
23:45
Uh Well, it turns out uh So, this animation, that's the input.
23:53
Uh that's the ground truth and that's the predicted output of a neural net that's trained to sort of virtually stain something without actually staining it.
24:02
Um and this is important because uh it turns out when you actually stain something that kills the cells, so you don't get any temporal information about what's going on in the the cells.
24:12
They essentially die when you apply the stain, but here you can virtually stain something, but then follow them longitudinally in time and see how cell processes kind of continue to to happen without actually staining them.
24:28
Uh you can also stain for things that you can't actually really necessarily develop a true chemical stain.
24:33
So, if you have someone label which things are axons and which things are dendrites in neural tissue, you can have a microscope viewer that highlights axons and dendrites in different colors and cell bodies um even if that's kind of not something that you can chemically do with a real stain.
24:53
Um one of the areas we've been doing a lot of work is in uh language understanding models and so this started out as research in our group to do essentially sequence to sequence learning.
25:10
So you have some input sequence and conditioned on that input sequence you want to predict an output sequence.
25:16
So this turns out to be useful for actually a whole bunch of different problems.
25:19
But one of them is translation.
25:22
So if you have a bunch of sentence pairs one in French and and the corresponding meaning sentence in English then you can use a sequence to sequence model to take the input sentence one word at a time or even like one character at a time and then when you hit a special end of French token then you essentially start spitting out the corresponding English meaning English translation of that of that French sentence.
25:48
And so that works like this.
25:51
And you have training data that that is like that and you just try to predict the next word from that training data using recurrent neural net.
26:00
And that turns out to work reasonably well.
26:03
And then you're actually trying to find the most probable sequence not the sequence with the most probable individual individual terms and so you do a little beam search where you kind of keep a window of candidates and you you sort of search over possible vocabulary items until you are happy and found a likely output sequence and that's how you do translation.
26:29
Um so one application of this is in Gmail we had we added a feature called smart reply where essentially we get an incoming email so this is one sent to my colleague Greg Corrado from his brother.
26:41
Hi, we wanted to invite you to join us for Thanksgiving dinner.
26:44
Please bring your favorite dish. RSVP by next week.
26:48
So to reduce the computational cost we have a small feed forward neural net that says is this the kind of thing where a small short reply would make sense?"
26:55
And if yes, then we are going to activate a sequence-to-sequence model um and we're going to do a much more computationally expensive thing with that in uh message as input, and then we're going to try to predict plausible replies.
27:09
And so, this system produces three. It says, "Count us in. We'll be there."
27:13
or "Sorry, we won't be able to make it."
27:16
Um and so, this is a nice application of sequence-to-sequence models uh and if you squint in the world, you'll find lots of applications of these.
27:27
And so, turns out that Smart Reply in April 2009, there was an April Fools' joke that Google put out saying, "Haha, we're going to reply to your email automatically. Ha."
27:37
Uh but then in November 2015, we launched this as a real product.
27:39
And in just 3 months, 10% of mobile inbox replies are generated by these Smart Replies.
27:50
Um so, that's kind of cool.
27:52
Uh but obviously, one of the real potential applications of this was Translate, which is what we were doing demonstrating that this research was effective on on a large by academic standards, but smallish public data set of translation data called WMT.
28:10
So, when we looked to work on applying this to the real Google Translate product, we actually had a 100x to 1,000x as much training data.
28:20
And so, scaling this up was actually pretty challenging.
28:22
Um and we wanted to make the the model a lot higher quality, but we did a nice uh a fairly detailed write-up of the engineering behind that uh in this many, many author paper.
28:35
Um and uh so, this is kind of the structure of the model that we came up with.
28:40
It has a very deep LSTM stack, each of which runs on a different GPU.
28:45
Uh there's an attention module so that rather than just having a single state that's updated by the recurrent model, you keep track of all the states and then you learn to pay attention to different parts of the input data when you're generating different parts of the output sequence.
29:00
So you're about to generate, you know, the next word and you look back at the word hello in the input sentence. And so on.
29:10
And so this thing runs on one replica of this model runs on a machine with eight GPU cards with different pieces of it in different places.
29:19
And then we run a lot of copies of this model to do data parallelism across the large training data and we share the parameters.
29:27
So this is a technique we've been using for quite a while that we originally published in 2012 about what we call at that time a parameter server.
29:38
And then using many parallel data data parallel copies to process different input data all trying to update those shared parameters by applying gradients to those parameters.
29:49
And this allows you to scale training quite quickly.
29:53
So you can have, you know, 50 replicas of this kind of setup or 20.
29:59
I think in this case we were using about 16.
30:00
So we're using 100 GPU cards to train a model.
30:05
And the really good news is the blue line here is the old phrase-based machine translation system that didn't really have much machine learning in it any machine learning in it.
30:16
It had large statistical models for lots of different sub-pieces of the problem.
30:20
So it had a a target language model that told you how often every five-word sequence in English occurred.
30:26
It had an alignment model that says how words in English and French sentences align, had a phrase table and a dictionary of plausible English and French phrases and sentences.
30:39
Um and it was like 500,000 lines of code to glue this whole thing together.
30:44
And that's the blue line.
30:44
Uh and what we're showing is the quality of translations generated by that system as judged by humans.
30:52
Um and the green line uh has a substantial jump in quality uh for basically nearly every language pair jumps up very substantially.
31:01
It doesn't look like much, but those are really big jumps in quality.
31:06
Um and the other nice thing is that system is 500 lines of TensorFlow code instead of 500,000 lines of gooey code with like lots of uh handwritten logic.
31:17
Um And the green the yellow line on top is human bilingual human, not professional translator, but someone who speaks both of those languages, translations as judged by other humans.
31:31
And so you can see that for some language pairs we're actually getting quite close to uh that human-level quality for translation.
31:38
Which is pretty exciting.
31:38
And when we we were trying to kind of roll this out slowly across lots of different language pairs.
31:45
And so we launched it in the dead of night in Japan.
31:49
And all of a sudden all of Japan kind of or many people in Japan noticed that suddenly uh English to Japanese translation was actually usable in quality as opposed to before.
31:59
Uh but when it was kind of supported but not usable as one of the people on our translate team uh referred to it.
32:06
Uh and so this uh professor at a Japanese university um decided he would do this experiment translating uh the first paragraph of Hemingway's The Snows of Kilimanjaro to Japanese and then back and see what the quality looked like.
32:21
And so if we focus on the last sentence, the old phrase-based system says, "Whether the leopard had what the demand at that altitude, there is no that nobody explained."
32:32
So I think there's a leopard involved, but other than that I really can't understand that.
32:35
Um and the neural machine translation uh just generates much more natural sounding translation.
32:40
So, "No one can explain what leopard was seeking at that altitude."
32:43
And the only mistake it made was it left out the word "the".
32:47
Um, so you can see how this transforms it from like really not usable to like actually pretty good.
32:55
Uh, another area we're doing a lot of research in is this notion of automating solution of machine learning problems.
33:02
Uh, what we call learn to learn.
33:04
Um, and the idea here is that, uh, the current way you solve a machine learning problem, probably many of you in your companies are solving machine learning problems.
33:13
You have data, you have some way of doing lots of compute, bunch of GPU cards or something.
33:18
And then you have a human machine learning expert saying, "Okay, I'm going to try this kind of model.
33:23
I'm going to use this learning rate, and I'm going to do transfer learning from this data set."
33:28
And and then you hopefully get a solution.
33:31
Uh, what we'd like to turn that into is, um, you have data, and maybe you use 100 times as much compute, but you don't need a human machine learning expert.
33:41
And if we could do that, that would be really, really good.
33:42
Because if you think about what's happening in the world, you know, there's probably 10 million organizations in the world that should be using machine learning and actually have probably data in electronic form that would be suitable for machine learning, but there's, you know, order a thousand organizations that have really hired machine learning experts in the world to actually tackle some of these problems.
34:05
So, uh, we're trying lots of different, uh, efforts in this area.
34:10
And I'll talk about two of them.
34:10
One is a way of designing neural architectures automatically, and the other is a way of learning optimizers automatically. So, architecture search.
34:23
Um, the idea is we want to have a model generating model.
34:28
So, the same way a human machine learning expert says, "I'm going to try this kind of model."
34:31
We're going to have a model generating model that's going to spit out models for this problem.
34:36
Uh to solve to tackle a particular problem.
34:39
And so the way this will work is we're going to generate 10 model architectures.
34:41
We're going to train each of them for a few hours.
34:45
And then we're going to use the loss of the generated models as a reinforcement learning signal for the model generating model.
34:51
And this is sort of just on the realm of feasible for small problems today.
34:58
Uh but it actually works for small problems.
35:00
So here is an example of a model architecture I came up with.
35:04
And you'll see it looks sort of not like something a human would have designed.
35:10
Uh the wiring is kind of crazy.
35:13
Uh and this is CIFAR-10, which is a very small uh color image problem with 10 different classes.
35:18
Got pictures of horses and planes and cars.
35:20
Uh not that many classes, but it's been pretty well studied in the machine learning literature.
35:26
And the error rate, like all machine learning image problems, has been dropping over the years.
35:30
But everything above these last four lines is a human generated machine learning expert model uh that someone came up with a new thing and published and beat the previous state of the art.
35:44
Um and so this is the current state of the art.
35:46
And this neural architecture search, basically with that architecture, got very very close to that state of the art uh without any human sort of knowledge of the underlying architecture.
35:58
Uh we also tried it on a uh a language modeling task.
36:01
And the traditional way you do this for recurrent models is use an LSTM cell, whose structure is shown there.
36:10
Uh that's kind of the the default thing you're going to do if you're going to use any sequence data.
36:15
Um and we just gave the architecture search the sort of underlying primitives of an LSTM cell and said, "Go to it.
36:21
Find us some way of dealing with sequential data."
36:26
Uh and that's the cell that came up with.
36:28
It looks somewhat different.
36:32
But in this case, it actually beat the state of the art by a pretty substantial margin for this language modeling task.
36:39
And the other interesting thing is we then took that cell and used it on a completely different sequential task in medical records future prediction task.
36:50
And it performed better than an LSTM cell in that domain as well.
36:56
So learning the optimizer rule is similar.
36:59
We're going to have symbolic expressions with and give it the model the the optimizer learn expression learning model access to the raw primitives that you might consider using in a neural optimizer update rule.
37:16
Things like here's the gradient, here's the running average of the recent gradients, here's the momentum term.
37:22
And so the top four lines here are human-designed update rules that people traditionally use.
37:28
And they're you know, they've been designed over the last decade or few decades in case of SGD.
37:37
And are generally what people use.
37:39
Adam is a pretty good choice these days.
37:44
And often SGD with momentum, which is the second line, is the the best choice.
37:50
And what you see is that this thing came up with 15 or something completely different expressions than what we've explored, and they're almost all better than all of the human-designed ones.
38:01
And so that's kind of encouraging.
38:05
That's going to appear in ICML in a month or two.
38:08
Uh And we also took one of the most promising ones of those and we then transferred it to a different problem where we hadn't the problem we didn't design am optimizer on.
38:21
And we use this other optimizer and found that it gave, you know, better training perplexity, lower is better for perplexity, and better blue score, which higher is better for that metric, than Adam, which was the best optimizer we'd found before.
38:36
Uh so, I think this whole notion of learning to learn is going to be pretty powerful because a lot of what machine learning experts do when they sit down to solve a problem is actually they run lots of experiments.
38:47
And right now, a human can't run that many experiments, right?
38:52
It's just a lot of cognitive load to run 50 experiments or 100 experiments.
38:55
And this thing can run, you know, 12,000 experiments in a in a weekend, and many of them suck, but many of them don't.
39:06
Uh so, the other thing that's interesting is that a lot of what's happened is we've been able to solve lots of problems because we have a lot of data and because we've been able to scale the amount of compute we throw at different problems.
39:19
And so, really, uh one of the nice properties of deep learning have is really transforming how we think about designing computers these days.
39:29
So, deep learning has two really nice properties.
39:31
So, one is that it's perfectly tolerant of very reduced precision arithmetic, so, you know, one significant digit kind of thing.
39:37
You don't need uh double precision.
39:39
You certain You certainly you don't don't need single precision floating point.
39:43
Um and the other property it has is it's generally made up all the algorithms I showed you are made up of a handful of specific operations kind of cobbled together in different ways.
39:56
And so, that really leads to an opportunity where if you can build custom machine learning hardware targeted at doing very reduced precision uh linear algebra, then you can all of a sudden uh unlock huge amounts of compute relative to CPUs or GPUs, which are not really targeted at doing uh these kinds of things.
40:18
And so this is we've been doing uh custom machine learning accelerators for a while.
40:22
Uh we've had a first generation one that was targeted at speeding up inference, so not training uh but inference when you're actually running a trained model uh in in the context of a product.
40:33
We had our first version uh deployed in our data center for 2 and 1/2 years or something, and we just revealed this system uh which is designed for both training and inference uh at Google I/O.
40:47
And uh this is uh a board.
40:50
Uh one of the things we felt was important was to design not just a chip for training, but also an entire system.
40:58
So because you're unlikely to get enough compute for large problems on a single chip.
41:04
Um so we designed a really high-performance chip, and we also designed them to be uh hooked together.
41:08
So this is what we call a pod, which is uh 64 of these boards, each of which has four chips. So 256 chips.
41:19
Uh and that's 11 and 1/2 petaflops of compute.
41:21
And we're going to have lots and lots of these in our data centers, uh which is pretty exciting cuz I think we'll be able to tackle much bigger problems.
41:29
It's going to bring a lot more compute for some of the learn-to-learn approaches.
41:34
Uh and normally programming a supercomputer is kind of annoying.
41:38
Uh so we decided we would make our these programmable via TensorFlow.
41:41
So you essentially can express a model uh with a new interface that we're adding to TensorFlow 1. 2 called estimators.
41:50
And then the same program will run with minor modifications on CPUs, GPUs, or on TPUs.
41:56
Uh and that's going to be available through Google Cloud.
41:57
You can get a thing uh called a Cloud TPU later this year, which is going to be a virtual machine with 180 teraflop uh TPU version 2 device attached, and it'll run TensorFlow programs super fast, we hope.
42:13
Uh we're also uh making a thousand of these devices available for free to researchers around the world uh who are doing interesting work and want more compute and are committed to actually publishing the results of that work openly.
42:29
Uh and also hopefully giving us feedback about what's working well on these TPU devices and what's not.
42:33
And ideally open-sourcing code associated with those models, but not we're not sure that's going to be a hard requirement, but it's uh desire on our part uh to help sort of speed up the whole uh science and machine learning research ecosystem.
42:50
Uh and so you can sign up there if you're interested in any of these things.
42:55
Uh Cloud Google Cloud uh is also producing higher-level APIs that are more managed services or pre-trained models that you can just use um without necessarily being a machine learning expert.
43:07
So if you have like photographs, you can run them through the Vision API and it will read all the text in it and find all the faces and tell you what kinds of objects are in it and do all kinds of good stuff.
43:18
Uh and the translation API has really nice high-quality translations that might be useful lots of things.
43:25
Uh One final closing thing, uh we've also been experimenting with machine learning for doing higher-performance machine learning models.
43:35
Uh and so in this case, what we've been doing is a similar kind of reinforcement learning where we're going to take an abstract TensorFlow graph and we have a bunch of computational devices that we want to run that on, say four GPU cards.
43:46
And we say to the RL algorithm, we want you to find the placement of TensorFlow operations onto devices that makes that model run as fast as possible.
43:57
And the the current way people do this is they okay, I have four GPU cards, I'm going to run this part of my graph on GPU card one, this part on GPU card two.
44:08
And that's okay, but it's kind of annoying because it's not something that humans really want to think about.
44:11
And so we're actually able to come up with pretty exotic placements.
44:14
So each color there is a different GPU card.
44:16
And this is a On the left you see a um a uh sequence prediction model unrolled in time.
44:25
So different time steps are on uh different GPU cards, which is not kind of counterintuitive to what a human expert would do.
44:32
And this is a image model and but the punchline is they're basically 20% faster than the human expert uh placement uh that uh people came up with. Okay.
44:45
So now we're here and we think there's a big opportunity with more compute to actually accelerate a lot of the use of machine learning and the sort of different applications and societal benefits that uh you can actually get from it.
44:59
So uh I'm pretty excited about that.
45:01
And you know, example queries of the future, you know, uh actually the upper left one we can already answer.
45:09
Describe this video in Spanish.
45:09
I didn't show you, but we can actually caption and generate sentences about images.
45:13
Um it's probably not that long before we'll be able to describe a uh a human uh video.
45:21
Uh find me documents related to reinforcement learning for robotics and summarize them in German.
45:26
Um you know, that's a pretty complicated request, but imagine how That's the kind of thing you would give to an undergraduate as like a semester project and then please come back with the report for me.
45:35
But imagine what if we could actually do that, uh how much more productive everyone would be. It'd be pretty amazing.
45:41
And then robotics I think is at a inflection point where through machine learning for control, we're going to have robots that can actually operate in messy environments like this one or the kitchen over there uh and actually know how to manipulate things in a safe way interacting with humans.
45:58
So that's going to be exciting, too.
46:00
So uh you already know this, but deep neural nets are making big changes and you should pay attention.
46:07
Uh you can find more info about our work at g. co/brain. And that's all I have.
46:12
You could join our team, but you're already starting companies.
46:22
Before we get to questions, I have a poll that Zach requested me to do and I I'm curious, too.
46:28
How many of you are using deep learning models in what you're doing? Okay.
46:35
How many of you are using Caffe?
46:38
How many of you are using PyTorch? Okay. Half-hearted PyTorch.
46:44
How many How many are using Theano? Keras? Okay. And TensorFlow? Okay. Cool. That's good to know. Uh Excellent. Yes. Yes.
46:58
It's roughly in proportion, in fact. Cool. Anything to add, Zach? Okay. Any questions? Yeah. Oh, well. Should I describe it? Sure.
47:16
Uh I Um when you talk about sort of the the learning to learn stuff and sort of the the neural net models designing other neural net models, uh for like for example, when the neural net model designed a model that performed better on Cfar10 than than other models, do you look at those models and say, "Oh, I understand why that performed better?"
47:33
Or is it kind of the case that it did something wacky and you don't understand really why it works better?
47:41
I mean, I think it depends.
47:41
Like some sometimes you just want the most accurate end model for the problem you care about, and that's fine.
47:48
Sometimes you're trying to uh come up with a model and you want to understand why it's more accurate so that you can then drive further human-oriented machine learning research. So, I think it depends.
48:04
Like the symbolic expressions for the optimizer update rule, those are actually pretty interpretable.
48:09
So, like if you if I go back to the um it's actually pretty interesting, right?
48:14
If you look here, there's this subexpression e to the sign of the gradient times the sign of the momentum.
48:25
Uh that seems to reoccur in a lot of these different optimizers that it's learned, and that sort of makes sense.
48:32
Basically, if the sign is the same as the direction you've been going, then speed up.
48:37
And if it's different, then slowly down. Right?
48:40
And that's kind of a good intuition to have, and you can see that the reinforcement learning wanted to do that in like five of these things.
48:50
Uh so, in the some sense, depending on what problem you set up in the learn-to-learn framework, you can actually come up with human insights about, "Oh, well, that makes sense."
49:00
from the experiments it's run.
49:00
Uh but, you know, here that that you can kind of investigate that cell and understand if you actually look, it's doing a bunch of adds at the bottom, but it's also doing an element-wise multiply for the input data in for one of the path through the cell, which is kind of different than the LSTM cell is doing, and so that might be sort of insight into why it's doing that.
49:24
If you look here, um you know, I think that architecture's kind of crazy, but we do know from ResNet work that these skip connections make a lot of sense.
49:34
And so, this is just kind of like crazy skip connections in lots of places.
49:39
Well, yeah, then I guess maybe a follow-up question is, do you think this is going to be a tool for humans to build better nets, or this is going to be how nets are built in the future is with other nets? Uh could be both.
49:50
But, uh I will say that this system can run 12,000 experiments in a weekend, and humans are not that good at that.
50:01
So, with all that compute you were showing, it strikes me that you might run out of human trainable data.
50:05
Is that stuff really for the reinforcement learning where you can run 12,000 experiments in a weekend, or do you have enough human labeled data to Oh, so for satisfy that amount of computation?
50:23
When we were training our translation models for one language pair, we were using hundreds of GPUs for a week.
50:30
And for that problem, we actually have enough training data that we could only get through 1/6 of that data once. Wow.
50:39
Uh, so we know that if we could get through all of it, the quality would be way better, right?
50:42
Cuz that's just a general rule of machine learning.
50:44
If you could get through all your data, probably it'd be better than not.
50:48
Um, and if you could even go through it a few times, that would probably be even better.
50:51
So, we think there are plenty of problems where there's enough labeled data in the world that you want to tackle a single problem and uh train a single model on something like that.
51:00
But, it's also going to be pretty good for small model exploration where you try to, you know, 10,000 different things, and they each take an hour to run on some subset of the chips.
51:07
It just depends on the problem.
51:11
You know, the the architecture search is kind of tenable with not current generation, but one generation ago GPUs for things like CIFAR-10, cuz you run that for an hour and you get an answer for one of the experiments, and you run 12,000 of those.
51:27
So, 700 GPUs over a weekend.
51:27
We know there's a bunch of algorithmic improvements we could do that would drop that by a factor of 10, but it's kind of just on the boundary of practical for tiny problems.
51:38
And making it practical for real problems at scale, I think it's going to be really, really cool.
51:45
Maybe maybe a follow-up question to that.
51:46
So, you had that slide on there where you have person, data, compute. Person's gone.
51:51
Um with with data, do you see anything in the near term in which you could have um really powerful models on very very small mu- much smaller data sets than a company like Google would would have access to?
52:05
Yeah, I mean, I think the the right way to tackle that is right now the way we as a community tackle machine learning problems is we say, "Okay, we're going to train a model to do this."
52:18
And we might say, "Gee, we don't have much data for this problem.
52:20
We're going to do transfer learning from ImageNet, and then I have my 5,000 flower images, and I'm going to do transfer learning and fine-tuning on that."
52:30
But that's really kind of lame, right?
52:32
Like, if we want to build real systems, real intelligent systems, we want a model that knows how to do a thousand things, 10,000 things.
52:38
And then when the 10,000 and first thing comes along, we want it to build on its knowledge for how to solve those 10,000 things so that it can solve the 10,000 and first thing with much less data, with many fewer examples, with building on the representations it's already learned.
52:54
So, if we can build a single giant model that can do thousands of things, that's going to improve the data efficiency problem a lot, and also the time to the wall time to actually being able to master a new task problem as well.
53:08
So, I think that's the way we're going to get to uh you know more data efficient, more flexible things.
53:14
Cuz the problem with the current approach is we train a model to do one thing, and then it can't do anything else. Right? Which is pretty lame. Yeah.
53:25
What do your best engineers do while they're waiting for models to learn?
53:31
Uh well, they often start up other experiments uh and hit reload on the the visualizer. Uh they write code.
53:39
They think of ideas uh at a whiteboard.
53:42
They do lots of things, but you know, getting that that cycle iteration time down from you know, days or weeks to hours really just qualitatively changes your workflow.
53:57
And so I think we're really shooting for making that time to result as low as possible.
54:04
And then they won't have, you know, people will not have these these week-long things where they're you know, gosh, I hope my experiment works.
54:13
So what would you attribute the gap in translation quality to between languages?
54:17
Is it just amount of data behind each one?
54:20
I think some language pairs the translations are more natural cuz they're more related kinds of language families and the alignment is maybe the similar as opposed to a very different word order and very different character sets for example.
54:34
But I think ultimately we will get higher accuracy models by you know, using a pod to train a really big model and get through all the data once.
54:42
I I suspect we could probably exceed human quality translations for some language pairs, you know, if we could get through all the data once maybe the maybe a slightly bigger model.
54:55
And the analogy is you know, even a the best human translator is only going to see so many words in their life.
55:02
And if your translation system can train on a lot more data and see more of them even though it's probably not as intelligent and flexible at getting its maximum information from each word that it sees, it's maybe at some point it'll do better.
55:19
Um do you find that you can kind of brute force the network architecture better on certain types of data versus others and what's the variance between like social tasks?
55:29
So to be honest, we haven't experimented with a broad enough set of tasks to really make conclusions here.
55:32
Um I suspect that there may be tasks.
55:36
I think probably for any supervised task that's like where you have a crisply defined input and output and you have enough training data, you know, it'll probably work.
55:48
It's a question of how much compute you need to apply.
55:53
Uh we have a lot of ideas around uh you know, making the algorithmic search more efficient by cutting off experiments that are obviously deads early rather than running them to conclusion and doing lots of things like that.
56:06
Yeah, I think the architecture search itself right now we train a bespoke model generating model for each problem we're trying to solve.
56:12
And so obviously you would want to train a model generating model that solves many problems.
56:17
And then you'll be able to get in a better state of good architectures for a new problem because of seeing similar problems and you're like, oh yeah, like lots of convolutions and 12 layers is a good place to start or something.
56:35
Uh how does the uh kind of the internal the bonus cycle look like for optimizing the the machine learning models?
56:41
Like how often do you retrain?
56:45
How often do you play with the different set of hyper parameters?
56:49
For the the learn to learn models in particular?
56:50
Uh no, but for example for all of pre-trained APIs you provide like for the vision API, how frequently do you kind of do the retraining and Right.
57:00
Uh it it varies depending on the domain.
57:05
Like some domains like vision are pretty stable.
57:07
Like you don't need to retrain every every hour.
57:10
Uh but other domains like for some of our internal problems like predicting, you know, uh um you know, maybe you're trying to predict what ads are relevant.
57:20
That set changes fairly rapidly.
57:24
Like there's a new chocolate festival on Long Island tomorrow that wasn't there yesterday.
57:30
And now you actually want to know that that's important.
57:31
Um so some things have a very stable distribution, some don't.
57:36
Uh it really just varies a lot depending on the problem.
57:38
Certainly it's easier for things like speech or vision where just the basic perception is what you're trying to do and the distribution is pretty stable.
57:47
Um and if you have a changing distribution, that just introduces lots of annoying production issues cuz now you have to retrain and you need to sort of somehow integrate new data so that you can learn new concepts and new new correlations relatively quickly so that you can then produce good correlation good outputs. Yep.
58:09
Um you mentioned sort of fast iteration being really important for developing this stuff.
58:12
How much of the process sort of now and like the cutting edge, you know, neural net development is still trial and error and how much of it is like I'm going to insert this and I know what's going to happen?
58:26
Um I mean I think a lot of machine learning research is empirical these days, right?
58:34
You have an idea, you think it'll work, but you need to try it implement it, uh try it on interesting problems, uh explore the set of hyper parameters or whatever that will make the idea go from not working to hopefully working.
58:50
And so it's often the case that you need to do this kind of empirical stuff.
58:55
I mean there's some ideas that you could have a lot of intuition like oh yeah, that's definitely going to work.
58:58
Uh even beforehand cuz it's sort of putting together two things that did work with a third thing that also did work and it seemed pretty obvious that combining them is going to going to work as well.
59:10
But other things it's hard to build the intuition.
59:16
Um I know a while ago you guys had done some really great work on helping visualize what a convolutional neural network doing image classification was doing.
59:25
So like the interpretability of models seemed to be like a focus for a bit.
59:29
And then there's a point where you kind of cross that and it's like the learn-to-learn stuff you just can't interpret.
59:34
And maybe this is kind of what you were asking, but how important is that for delivering production models to humans who maybe are not machine learning experts that need to work alongside a robot classifier?
59:44
It's really important in some domains and not important in others.
59:48
And we actually have a pretty big focus.
59:50
I I have a much longer talk or a set of slides I select a subset among uh we we have a we've doing a bunch of work in sort of understanding and visualizing and and building interpretability for models uh that I didn't talk about, but it is an important area.
1:00:07
Uh the main areas where I think it's really important are in healthcare.
1:00:13
Like, if you tell someone if you're providing advice to a physician, you say, "Patient needs a heart valve replacement."
1:00:18
You know, they're going to want to know why are you saying this, right?
1:00:22
And so, if you can go back and highlight a part of a medical note that says, "You know, a year and a half ago patient was complaining about like their heart felt like it skipped a beat every so often or something."
1:00:32
Um that's going to give much more uh uh smooth interactions between a machine learning system and a human and let them kind of each do play to their strengths.
1:00:44
Whereas, if you just give a black box prediction, that's often not as useful in some domains.
1:00:49
But, some things like image classification, I just want the most accurate image possible uh classification possible. Yeah. Okay. Yeah.
1:01:01
What caused you to originally start focusing on machine learning 6 years ago?
1:01:06
And is there anything to be learned from that in terms of finding other areas that may have kind of explosive growth in the future?
1:01:14
Um so, the thing that caused me to start doing this is I heard I kind of like to keep a pulse on different areas of computer science and I started to see neural nets being successful in some domains just by like reading abstracts of things and I had this bug in the back of my head from my undergrad thesis that neural nets were actually the right abstraction and so I kind of heard inklings.
1:01:38
I chatted with Andrew Eng who was consulting at Google one day a week uh and he said, "Oh yeah, neural nets are I'm I'm like, "What are you working on at Stanford?"
1:01:48
And he's like, "Oh, neural nets are kind of interesting again." And I'm like, "Oh, cool. Really?
1:01:51
I used to do work on that."
1:01:53
And I kind of he and I started talking and I just felt like if the problem for neural nets was scale like back from my experience where you know, more compute seemed like the right answer but wasn't then but now we actually have a lot more data.
1:02:10
We have much more compute in a single processor but if we could throw lots of processors at this problem then perhaps scale would let us solve problems that we couldn't before.
1:02:20
And so I kind of had this inkling that neural nets were great abstraction from 20 years before and felt like it would be fun to go see if we could make them scale. Yeah.
1:02:36
Yeah, can you talk a little bit about what you think the development of the larger field of AI will be in 10 years especially in terms of like developing particularly like real reasoning systems because the neural nets of today are not necessarily reasoning most of the time today. Right.
1:02:57
I I mean I think there's probably a lot of algorithmic things that we're going to need but I do think one of the major problems and why we don't have systems that appear to reason is because of this problem of training neural nets to do one thing. Right?
1:03:13
If you had a lot more compute and you had a model that could do tens of thousands of things, and you had some algorithmic constructs where you sat there and cogitated for a while and built sort of plausible scenarios and explored them with compute computation, and then eventually came back with an answer for this new thing.
1:03:30
That might appear to be more like reasoning because you're building on all this other groundwork of knowledge that you've learned from solving 10,000 other things.
1:03:42
And I think that's what humans do, right?
1:03:44
We We We learn to do a new task or to reason through something by building on our experience that we've already accumulated from, you know, perception and building up that kind of low-level thing around the world, but also from, you know, bringing concepts together from math and science through our education and being able to reason through something.
1:04:05
So, I think, you know, a lot of it is that we don't have these massively multitask models being trained today.
1:04:11
Do Do you feel like there's like a lot of some sort of persistent knowledge store that missing from the current Cuz it it it feels like a human brain kind of has that, right? Right.
1:04:23
Uh I didn't put in this talk, but I think one of the real problems that we have is we kind of have a model and we densely activate the entire model for everything we do.
1:04:34
I think what we actually want is a model that's very, very big, like think, you know, uh 100 billion, trillion parameters, uh but where for any given thing, you activate only a tiny fraction of it, 1% of it, 5% of it.
1:04:49
Um you know, your brain works this way.
1:04:52
Uh and that seems like And you have to be able to the the way you store stuff is just by having a lot of parameters. Yeah.
1:05:05
So, on that note of having um some sort of network that can do like 10,000 things rather than just one.
1:05:11
Like, I know you haven't like brought up uh like the mind like differential neural neural computing or like memory augmented networks.
1:05:18
So, like can you talk a little bit about that? Sure.
1:05:21
I mean, I think memory networks are kind of an interesting emerging area where you have this kind of local state that you can update and mutate in the process of accomplishing some sort of task.
1:05:34
Um so far, I think those have been applied to relatively modest-sized problems.
1:05:39
They may be part of the solution, part of something that we would want like short-term working memory as you're like looking through a set of possible solutions to some problem.
1:05:50
Uh I think, you know, they're definitely an interesting area.
1:05:53
Um I think uh combining that with, you know, a model that does 10,000 things or a million things um might get us pretty far.
1:06:06
Uh as we automate more and more types of hyperparameter search and architecture search, where do you see the role of data scientists or ML experts going?
1:06:18
Well, I think one of the nice things about architecture search is it actually combines really well with machine learning researchers.
1:06:25
So, if someone comes up with a new interesting thing, you can put that in the search space of these automated learning-to-learn systems pretty easily.
1:06:36
And then all of a sudden, you now have the access to this hybrid best-of-both-worlds like very primitive things, but also these hand-designed things that humans have come up with that seem effective.
1:06:47
And um use that as the the search space.
1:06:50
So, it's not like machine learning researchers will not have uh anything to do.
1:06:57
Um and I think you know, there's a ton of work in figuring out what are interesting problems where machine learning can actually uh make a difference and which ones are worth solving and how do we solve them.
1:07:13
Maybe Maybe a good time for this question.
1:07:14
What What's in your opinion what's like the coolest thing neural nets are being applied to right now?
1:07:21
Um I'm really excited about health care.
1:07:24
I think the ability of neural nets to uh ingest a lot of data uh and then make sort of interesting predictions uh in a smooth way so that you can take a patient in a particular state and say, "Okay, here are the five most likely diagnoses for this patient because, you know, I've seen, you know, a million other patients and I've I know these 17 seem to have similar conditions."
1:07:51
Um you know, I think that's that's one that will have a really big societal impact.
1:07:54
It's, you know, fraught with lots of rollout issues cuz it's a heavily regulated environment.
1:07:59
There's all kinds of privacy issues, but ultimately I think um making better health care decisions is going to be pretty big.
1:08:08
Um the coolest things, you know, I I really like all the art generation kind of things. Those are fun.
1:08:15
Uh the ability of neural nets to write a sentence about an image, you know, I I was kind of surprised that happened that early.
1:08:23
I would have said, you know, before that work, uh some of which was done in our group, I would have said, "Uh I don't you know, we're good at saying that's the lion.
1:08:29
I don't think we can say that's a lion sleeping on a rock uh with a pretty yellow mane or whatever it is."
1:08:36
Um and that that's pretty cool. Okay. Okay. Thank you very much. Sure. Thank you. Mhm.