"Today I have the pleasure to speak with John  Schulman, who is one of the co-founders of OpenAI   and leads the post-training team here. He also  led the creation of ChatGPT and is the author   of man..."

"Maybe I should take a step back and ask this.  Right now we have these models that are pretty   good at acting as chatbots. Taking a step  back from how these processes work currently,   what kinds of..."

"thing. I would also expect that as models get  better, they get better at recovering from   errors or dealing with edge cases. When things  go wrong, they’ll know how to recover from it. 
  The models..."

"is it going to be the case that each one takes  10X more compute, analogous to the current   scaling laws for pre-training? Or is it going  to be a much more streamlined process of just   getting to t..."

"how fast progress will be. That's still uncertain.  I wouldn't expect everything to be immediately   solved by doing any training like this. There'll  be other miscellaneous deficits that the models  ..."

"That's an interesting question. I expect that  models will be able to use websites that are   designed for humans just by using vision, after  the vision capabilities get a bit better. So there   woul..."

"We've seen some version of this with multimodal  data where if you do text-only fine-tuning,   you also get reasonable behavior with images.  Early on in ChatGPT we were trying to fix some   issues wi..."

"It seems like then, you should be planning for  the possibility you would have AGI very soon.   I think that would be reasonable. So what's the plan if there's no   other bottlenecks. In the next ..."

"things down. That's what I would hope for. If there's more of a discontinuous jump,   there’s a question of “how do you know if the  thing you've got is safe to release”. I can't   give a generic answ..."

"What are you keeping track of while  you're doing long-horizon RL or when   you eventually start doing it? How could  you notice this sort of discontinuous jump   before you deployed these systems bro..."

"make the model turn against you. That doesn't  seem like the hardest thing to do. The way we   train them with RLHF, that does feel very  safe even though the models are very smart.   The model is jus..."

"world. Of course if you assigned a task such as  “make money,” then maybe that would lead to some   nefarious behavior as an instrumental goal. Before we get back to that, let's step back   and talk..."

"There are probably some analogies though I don't  know exactly how close it is. To some extent,   the models do have drives and goals in  some meaningful way. In the case of RLHF   where you're trying..."

"time. So I think that you’d get the best  results by combining these two things. 
   Right now, you have these two ways the  model learns. One is in training, whether   it's pre-training or post-trai..."

"memory? Too much to fit in context but  much smaller scale than pre-training?   It might be memory. I don't have context.  Certainly when I'm trying to prepare for   this conversation, I think of wha..."

"systems that do some online learning and also  have some cognitive skills, like introspecting   on their own knowledge and seeking out  new knowledge that fills in the holes.   Is this all happening ..."

"There are all these complicated RL procedures,  many of which you've pioneered. How many of them   will be relevant when you get to the point where  the model itself is smart enough to act as its   ow..."

"of thing that gets used in a particular task. Interesting. I want to step back and ask about   your own history, at least at OpenAI. You  led the creation of ChatGPT.
At what point   did you reali..."

"At the same time there were definitely a lot  of people thinking about chat. Google had some   papers like LaMDA and earlier, Meena. They  had these chatbots. It was more like a base   model that was ..."

"the most interesting thing about it. We had it  out to friends and family for a while and we   were thinking about doing a public release. Actually, GPT-4 finished training in August   that year. The ..."

"That was actually one of the things that I  got excited about as we were developing it.   I realized a lot of the things that people  thought were flaws in language models, like   blatant hallucinatio..."

"done that, you could have gotten pretty  close but it would have been non-trivial.  We also had another instruction-following  model trained with RL, released a little   before ChatGPT. If you put a c..."

"We found a lot of gains through post-training.  So I would expect us to keep pushing this   methodology and probably increasing  the amount of compute we put into it.   The current GPT-4 has an Elo s..."

"Do you think that hypothesis is wrong? We've  talked about some examples of generalization,   like Spanish to English. One example I think of is  the transfer from code to reasoning in language.   If ..."

"I can give you a sketchy explanation. You could  say that the model is an ensemble of different   circuits that do the computation. You could  imagine that it's doing computations in parallel   and th..."

"useful to you. Everyone would have all these  AIs helping them do more and get more done.   Obviously at some point they're going to be  better than everyone at whatever they want to   do. What would..."

"what happens if China doesn't decide to do that? You would either have to have every country agree   to this regulatory regime, or you would need  all of the model infrastructure or the model   prov..."

"with RLHF. You have to aggregate preferences  across a lot of different humans. It'll be maybe   more marked with future, more powerful systems.  But when you say we want these eventual AI systems   t..."

"not impose our opinions on people. We mostly want  to let people do what they want with the models.   I got a chance to read the Spec beforehand.  This is a question of how well that transfers   over..."

"methods. There's been a decent amount of that  recently. We could use more of that. I think   that's a good thing for academics to work on. On a slightly different note, I'd be really   excited to see..."

"However, there is a sense in which all of these  models, once they're put in a chatbot form,   have a very similar way of speaking. They really  want to “delve” into things. They want to turn   things..."

"that we tend to train for one message at a  time rather than the full interaction. If you   only see one message, then something  that just has a clarifying question,   or maybe a short response with ..."

"How much of a moat is better post-training?  Companies distinguish themselves currently by   how big their model is and so forth. Will it  be a big moat for who has figured out all the   finickiness t..."

"I guess it helps clear the moat. What is the  median rater like? Where are they based? What are   their politics? What is their knowledge level? It varies a lot. We've definitely hired raters   with..."

"we have now are quite skilled and conscientious. With regards to the plateau narrative,   one of the things I've heard is that a lot of  the abilities these models have to help you   with specific t..."

"reasonable behavior in the programming domain. Maybe a final question. We've touched on this   in different ways but let’s put it together. You  said you're training on much more multimodal data.   ..."