"After helping people optimize models for about a decade and a half, I came to a really interesting realization that I was solving the wrong problem. And foundationally, it was that the thing that's ho..."

"more interactive and like the types of applications, the types of ways that you can use these systems becoming so much larger because I mean gen is in the the generative aspect of them is in the name,..."

"robustness do I now have because I've overfit this system?" And we're seeing people do the exact same thing again today with LLMs where they're focusing on these highlevel metrics, these end outputs, ..."

"of those past learnings all those past mistakes made a ton of mistakes as just a naive PhD student trying to build a company I won't force you to go through the mistakes that you made maybe maybe that..."

"original machine learning systems. I think one of the main distinctions is how atomic some of these units maybe are. And so with a lot of traditional ML and AI, it was all about can I make a specific ..."

"to change dramatically without me knowing or it could be that I explicitly don't want it to exhibit a specific bias or a specific type of response. And I think the main thing that's shifted though is ..."

"create massive behavioral changes by the time it actually starts to affect an end user. And so it's really important and what I think a lot of firms are running into right now is if you're only lookin..."

"And and this is what you're doing at uh distributional, right? Definitely. So distributional is an enterprise platform to allow teams to test these applications in production to make sure that they ar..."

"possible. But by detecting that you have bias, maybe you want to factor that into your performance metric over time. And this is about adapting what that desired behavior over time as you learn more a..."

"really leverage all of its resources you want to have more centralized tooling. And so much in the same way that we saw the the rise of ML and AI platforms over the last decade, we're now starting to ..."

"code base is now public or whatever it may be. So if I'm a technology executive at at a big company right now experiencing this phenomenon of little AI projects popping up everywhere, some of them doi..."

"incentives are misaligned. So, OpenAI obviously wants to create the best general purpose foundational models, but an individual business may want a model that solves a very specific problem a very spe..."

"be, or turn it on to real enterprise value. And so I think the shift in mindset needs to be not just does it work the way that I want to to look at it, but does it work more holistically? Do you have ..."

"little bit from maybe traditional approaches to LLM eval because instead of trying to come up with a small number of strong estimators for performance where we want to be able to conclusively say A is..."

"harder problems to be completely honest. We see some firms attacking the lowhanging fruit internal chat bots to like ask questions about HR because they're afraid to take that leap to develop the the ..."

"So, it's an interesting thing you say like the the system and even the prompt sent to a language model kind of reflects the the the organization it's coming from and kind of like the behavior they're ..."