"Today I have the honor of chatting with Jeff  Dean and Noam Shazeer. Jeff is Google's Chief   Scientist, and through his 25 years at the  company, he has worked on basically the most   transformative ..."

"How did Google recruit you, by the way? I kind of reached out to them, actually.   And Noam, how did you get recruited? I actually saw Google at a job fair in 1999,   and I assumed that it was already..."

"decades changed the kinds of considerations you  have to take on board when you design new systems,   when you figure out what projects  are feasible? What are still the   limitations? What are things..."

"considering changing for future versions of TPU to  integrate how you're thinking about algorithms?   I think one general trend is we're getting  better at quantizing or having much more   reduced pre..."

"Let me start with the undergrad thesis. I got  introduced to neural nets in one section of one   class on parallel computing that I was taking  in my senior year. I needed to do a thesis to   graduate..."

"me about how the 2007 paper came together. Oh yeah, so that, we had a machine translation   research team at Google led by Franz Och,  who had joined Google maybe a year before,   and a bunch of other..."

"you're looking at a research area, you come up  with this idea, and you have this feeling of,   "Holy shit, I can't believe that worked?" One thing I remember was in the early days of   the Brain tea..."

"These examples illustrate how these AI systems  fit into what you were just mentioning:   that Google is fundamentally a company that  organizes information. AI, in this context,   is finding relation..."

"I know one thing you're working on right now is  longer context. If you think of Google Search,   it's got the entire index of the internet in  its context, but it's a very shallow search.   And then ..."

"I want to talk more about the thing you mentioned  about: look, Google is a company with lots of   code and lots of examples. If you just think  about that one use case and what that implies,   so you..."

"your own personal work? What will it be like to  be a researcher at Google? You have a new idea   or something. With the way in which you're  interacting with these models in a year,   what does that ..."

"idea. Suppose in the world today there are  on the order of 10,000 AI researchers in this   community coming up with a breakthrough- Probably more than that. There were   15,000 at NeurIPS last week. ..."

"we could dramatically speed up the chip design  process. As we were talking earlier, the current   way in which you design a chip takes you roughly  18 months to go from "we should build a chip" to   ..."

"applying more compute at inference time. I  guess the way I like to describe it is that   even a giant language model, even if you’re doing  a trillion operations per token, which is more   than most ..."

"already tapping out nuclear power plants in  terms of delivering power into one single   campus. Do we have to have just two gigawatts  in one place, five gigawatts in one place,   or can it be more d..."

"decode? You've got these things, some of which are  making the model better, some of which are making   it worse. When you go into work tomorrow, how do  you figure out what the most salient inputs ar..."

"and the models get better and better over time”,  even if you take the hardware part out of it.   Should the world be thinking more about, and  should you guys be thinking more about this?   There's o..."

"of nuclear war or something. Just think about  it, like a million evil Jeff Deans or something.   Where do we get the training data? But, to the extent that you think that's   a plausible output of so..."

"All right, let's talk about a few more fun topics.  Make it a little lighter. Over the last 25 years,   what was the most fun time? What period of  time do you have the most nostalgia over?   I think ..."

"What I find remarkable about some  of the calls you guys have made   is you're anticipating a level of demand for  compute, which at the time wasn't obvious or   evident. TPUs being a famous example o..."

"Yeah, I've been thinking about this more and  more. I've been a big fan of models that are   sparse because I think you want different parts  of the model to be good at different things. We   have our..."

"Man, there are so many interesting implications  of this that I could just keep asking you about   this- I would regret not asking you more about  this, so I'll keep going. One implication is,   curre..."

"Right now, language models,  obviously, you put in language,   you get language out. Obviously, it's multimodal. But the Pathways blog post talks about so many   different use cases that are not obvio..."

"How would you characterize what's missing  from current distillation techniques?   Well, I just want it to work faster. A related thing is I feel like we   need interesting learning techniques during ..."

"So here's the question I have. What you've just  laid out over the last hour is potentially just   like the big next paradigm shift in AI.  That's a tremendously valuable insight,   potentially. Noam,..."

"So we've discussed some of the things you guys  have worked on over the last 25 years, and there  "

"I'd say probably a big thing is humility, like  I’d say I’m the most humble. But seriously,   to say what I just did is nothing compared to what  I can do or what can be done. And to be able to   drop..."