"OpenAI recently dropped GPT OSS, its first open weights model since GPT2 in 2019. It's one of the highest profile open source model launches since DeepSeek R1 made waves back in January. But how does ..."

"share the same key value pairs to reduce memory use and speed up inference. It also includes swiggloo activations in the feed forward network layers which allow for more nuance transformations than si..."

"builds on the O200K tokenizer used in models like GPT40. As for the data set GPT OSS was trained on, OpenAI has only disclosed the broad strokes. The model was trained on a textonly corpus in the tril..."

"of open source AI, GPToss arrives as a fully equipped long context model ready for immediate use. As impressive as it is, however, it's just one of several models in a rapidly expanding field of open ..."

"3 incorporates features like group query attention, swiggloo, rope, and RMS norm. Quinn 3's sparse models share the same fundamental architecture as its dense models, but add a mixture of experts laye..."

"3 was trained on 36 trillion pre-training tokens, twice as many as the Quen 2.5 models. In addition to pulling data from multilingual texts, STEM and coding sources, and reasoning tasks, Quen 3 also u..."

"Together, all of these optimizations allow the model to reason over much longer inputs at inference. Finally, Quen uses a four-step post-training pipeline with two goals. Giving users more control ove..."

"what developers did in this step was fine-tune the model on a mix of thinking data, which includes intermediate reasoning steps, and non-thinking data, which omits them, and then build a chat interfac..."

">> The chatbot developed in China called Deep Seek. >> Deepseek is such a fundamental change to the economics of what's going on. >> The most downloaded free app in the US. This is an update in what p..."

"on the original V3based checkpoint, extending it with a two-phase long context training approach and adding a hybrid thinking mode that lets the same model switch between reasoning heavy and lightweig..."

"let's take a step back from V3 to Quen to GPDoss. How should we think about at a high level the differences between these models? One big difference is size. The Quen 3 model family is the only one of..."

"rotary positional embeddings so that it can handle far longer sequences than it was originally trained on. Normally rope starts to break down when you feed it more tokens than its base frequency was s..."

"train model can do without more long context training. Personally, I think one of the most interesting things about these papers and the state-of-the-art in deep learning more generally is that a lot ..."

"training efforts. And it's fascinating and pretty surprising how some of these RL efforts require very little amounts of data. just 4,000 data pairs in the case of Quinn. Another point here is that it..."

"are tons of high performing open source models that we didn't discuss in this video, like Kim K2 or Google Gemma 3. But when you peek under the hood of many of these, you'll find nuance differences th..."