"To build great AI products, you need to be really good at building evals. It's the highest ROI activity you can engage in. This process is a lot of fun. Everyone that does this immediately gets addict..."

"It's to actionably improve your product. >> I did not realize how much controversy and drama there is around eval. There's a lot of people with very strong opinions. People have been burned by evals i..."

"The top one is we live in the age of AI. Can't the AI just eval it? But it doesn't work. A term that you used in your post that I love is this idea of a benevolent dictator. When you're doing this ope..."

"Oftentimes it is the product manager. Today my guests are Hamill Hussein and Shrea Shankar. One of the most trending topics on this podcast over the past year has been the rise of evals. Both the chie..."

"from being an obscure mysterious subject to one of the most necessary skills for AI product builders. They teach the definitive online course on evals, which happens to be the number one course on Mav..."

"This episode is the deepest yet most understandable primer you will find on the world of evals and honestly got me excited to write evals. Even though I have nothing to write eels for, I think you'll ..."

"number one AI agent for customer service. If your customer support tickets are piling up, then you need Finn. Finn is the highest performing AI agent on the market with a 65% average resolution rate. ..."

"by the Finn AI engine, which is a continuously improving system that allows you to analyze, train, test, and deploy with ease. Finn can continuously improve your results, too. So, if you're ready to t..."

"of events. It's been a the concept of a trace has been around for a really long time but it's especially really important when it comes to AI applications. So we have all the different components and ..."

">> That's amazing because that's really it's rare you see actual company products system prompt. That's like their crown jewels a lot of times. So this is actually very cool on its own. >> Yeah. Yeah...."

"AI responds, hey, we we have several one-bedroom apartments available, but none specifically listed with a study. Here are a few options. Uh, and then it says, "Can you let me know when one with a stu..."

"is completely manageable and it's not something that we invented. It's been around in machine learning and data science for a really long time and it's called error analysis. And what you do is the fi..."

"conversation. Okay. A user asked about availability. The AI said, "Oh, we don't really have that. Have a nice day." Now, for a product that is helping you with lead management, is that good? Like, do ..."

"quick note um should you know should have handed off to a human >> and as we watch this happening it's like you mentioned this and you'll explain more you're doing this this feels very ma manual and u..."

"So, this is the next trace. I just pushed a hot key on my keyboard. Let me go back to uh looking at it. >> And these tools make it easy to go through a bunch and add these notes quickly. >> Yes. And s..."

">> This is more of hey we're not handling this interaction correctly. This is more of a technical problem. um rather than hey the AI is not doing exactly what we want. So we would write down too like ..."

"first two or three can be very painful, but you know, it doesn't we can, you know, do a bunch of them really fast. So, here's another one. And um let's skip the system prompt again. And the user asks,..."

"different kinds of errors that we're seeing and we're actually learning a lot about your application um in a very short amount of time. >> One common question that we get from people at this stage is ..."

"are like, let me automate this with an LLM. >> Do you think they'll we'll get to a place where where an agent can do this? >> Oh, no, no, no. Sorry. There are parts of error analysis that an LLM is su..."

"hey, you need to simplify this across as many dimensions as you can. Another thing that we'll talk about later is when you goes to building an LLM as a judge, you need a binary score. You don't want t..."

"become infinitely expensive if you're not careful. >> Yeah. Okay, cool. Let's go back to your examples. Yeah, no problem. So this is another example where we have someone saying, "Okay, do you have an..."

"should talk about >> yeah so there's actually a term in data analysis and quant qualitative analysis called theoretical saturation So what this means is when you do all of these processes of looking a..."

"you're like not going to discover new types of problems. >> Yeah. Awesome. So let's say you did a 100 of these. What's the next step? >> Yeah. Okay. So you did 100 of these. Now you have all these not..."

"different uh coding agents or you know uh AI tools and had it categorize these notes. So one is okay, I uploaded into a cloud project. I uploaded a CSV of these notes and I just exported them directly..."

"what is the most prevalent. So then you can go and run and attack that problem. >> That is really helpful. Basically, you're just synthesizing all these categories and themes. >> Super cool. And we'll..."

">> But, you know, now we can wade through this mess of open codes a lot easier. Another thing that's interesting here in this prompt to generate the axial codes is you can be very detailed if you want..."

"demystify some of this magic. >> And just to double click on this point, like this is not a thing everyone does or knows. This is something you two developed based on your experience doing data analys..."

"social science. >> Amazing. Okay. Like what's funny about you guys doing this is I just want to go do this somewhere. I don't have I don't have an AI product to do this on, but it's just like oh this ..."

"collect these codes into a list. So now we have a commaepparated list of these codes. And then what you can simply do is you could take your notes that you have those open codes and you can tell an AI..."

"somewhat detailed in your open code. >> Okay. So avoid the word janky is a good rule of thumb >> or other words. >> Okay. >> I was being funny. >> Yeah. Okay. What are some of those other words just t..."

">> Awesome. And what's cool about this is you don't need to do this many many times. Like no, >> for most products, you do this process once and then you build on it, I imagine, and you just tweak it ..."

"gone from chaos to some kind of thinking around oh you know what these are my biggest problems I need to fix conversational issues you know maybe these human handoff issues it's not necessarily the co..."

"want to get carried away with it. Um, but you want to start, you want to usually ground yourself in your actual errors. You don't want to skip this step. And so, the reason I'm kind of spending so muc..."

"judge, for example. So, there's different kinds of evals. One is codebased which you should try to do if you can because they're cheaper. You don't have to, you know, LM as a judge is something it's l..."

"describing, you want to test what your say agent or AI product is doing. You ask it a question, it gets back with something. One way to test if it's giving you the right answer is if it's consistently..."

"thing about LLM judge is you can use them in unit test or CI sure but you could also use it online for monitoring right like I can sample like thousand traces every day run my LLM judge real productio..."

"decision. Like no, you need to make a decision. Is this good enough or not? Yes or no? can be painful to think about what that is, but you should absolutely do it. Otherwise, this thing becomes very u..."

"create it, but again, put yourself in the loop. Don't just blindly accept what the LLM does. And in all of these cases, that's what we did. Like with the axial codes, we kind of iterated on this. You ..."

"so we can take this prompt and I'm going to use a spreadsheet again. So the first step is okay when I'm doing this judge I wrote the prompt. Now a lot of people stop there and they say okay I have my ..."

"as a judge, you want to make sure it's aligned to the human. So how do you do that is you actually you have those axial codes and you want to like measure your judge against the axial code and say lik..."

"have some misalignment that's okay we talk about in our course also how to code correct that misalignment but in this stage if you're a product manager and the person who's building the LLM judge eval..."

">> I love that. And Haml's pulling up some cool research report. What's this about? >> Oh, this is one of the coolest research reports you can possibly read if you want to know about evals. So, it was..."

">> and probably the ones that are most risky to your business if they say something like Mecca Hitler Grock and >> cool okay so that's that's very uh relieving that this because this is this prompt is..."

"Um and I think what's very empowering now is that product managers are doing this and can do this and can really build very very profitable products with this skill set. >> Okay, great segue to a deba..."

"then unfortunately X or Twitter is like a medium where you know people are misinterpreting what everybody is saying all the time and you just get all these strong opinions of like don't do eval it's b..."

"probably very systematic about the error analysis to some extent. I bet you that they are monitoring who is using Claude, how many people are using Claude, how many chats are being created, how long t..."

">> because you're seeing the code like you see the code it's generating you can tell this is great this is terrible >> yeah yeah and so and so I think a lot of people had generalized coding agents bec..."

"now. >> Do fooding is is a dangerous one only because a lot of people will say they're dog fooding. They're like, "Yeah, we dog fooded." But are they really? And a lot of people aren't really dog food..."

"thought the errors might be. They were these like weird handoff issues or like I don't know like the text message thing was strange. Um, so I would say that like if you're going to do AB tests and the..."

"up until recently that some of the big labs have they don't have error analysis. They have gener a suite of generic tools cosign similarity hallucination score whatever and that doesn't work. It's a g..."

">> Well, the fact that your course on Maven is the number one highest grossing course on Maven, clearly there's demand and interest and there's more people I think on your side. Interestingly, uh, jus..."

"I was like, "Oh my god, >> so true. >> This is the world we live in." Oh my god. >> Okay. So, I want to ask about just top misconceptions people have with Evals and top tips and tricks for being succe..."

">> The second one that you know I see a lot is hey um just not looking at the data you know. So in my consulting people come to me with problems all the time and the first thing I'll say is let's go l..."

">> Amazing. Okay. What are a couple just tips and tricks you want to leave people with as they start on their eval journey or just try to get better at something they're already doing? >> So tip numbe..."

"would say is we're very pro- AI. Use LLM to help you organize any thoughts that you have throughout this entire process. So this could be everything ranging from like initial product requirements, rig..."

"the nurture boss use case, we wanted to remove all the friction of looking at data. And so what you see here is just some screenshots of uh what the application that they created looks like. It's just..."

"AI products better because the experience is how users interact with your AI. Absolutely. If any, you know, we teach our students, hey, when you're doing these evals, if you see something that's wrong..."

">> Yeah, it's really not that much time. I think people just get overwhelmed by how much time they spend up front and then thinking that they have to keep doing this all the time. >> Amazing. Is there..."

"Education shouldn't be this thing where you you're only watching lectures and doing homework assignments. So students should have access to an AI that also helps them. So what we've done is we've uh y..."

"students 10 months free unlimited access to that alongside the course. >> Amazing. And then you'll charge for that later down the road is the idea. I just take one month at a time. I don't know what w..."

">> Amazing. Okay. We also have a discord >> of all the students who have ever taken the class and that discord is so active >> I I can't go on vacation without getting notified on the plane or >> bitt..."

"reading Apple in China which the name of the author is slipping my mind but this is kind of more of an exposition written by a journalist on how Apple did a lot of manufacturing processes in Asia over..."

"simplest and also engineering. So like a lot of times the simpler approach generalizes better. And so that's the thing I kind of internalized deeply from that book. And um also really like this one. S..."

"Okay, next question. Favorite recent movie or TV show? I'll jump to Haml first. Okay. So, I'm a dad of two parents. I have two parents. >> I don't get to Oh, sorry. Uh, two kids. So, yeah, I'm a dad o..."

">> Worth it. Okay, next question. Do you have a favorite product you've recently discovered that you really love? And we'll start with Shrea. >> Yeah, I really like using cursor. Honestly, now Cloud C..."