Hamel Husain

3 weeks ago | news.aakashg.com | Aakash Gupta |Hamel Husain

Just like you can't be a PM without using analytics, you can't be a PM on AI products without evals. Unlike traditional software, LLM pipelines do not produce deterministic outputs. A response may be factually accurate but inappropriate (i.e., the “vibes are off”). They may sound persuasive while conveying incorrect information. The core challenge is: How do we assess whether an LLM pipeline is performing adequately?And how do we diagnose where it is failing?

2 months ago | productcompass.pm | Paweł Huryn |Hamel Husain

Hey, Paweł here. Welcome to the free edition of The Product Compass Newsletter. With 107,800+ PMs from companies like Meta, Amazon, Google, and Apple, this newsletter is the #1 source for learning and growth as an AI PM. Consider subscribing and upgrading your account for the full experience:Recently, subscribers kept asking me about AI Evals. It’s arguably the most critical element of any AI initiative. But it’s hard to find reliable information.

Oct 30, 2024 | hamel.dev | Hamel Husain

Earlier this year, I wrote Your AI product needs evals. Many of you asked, “How do I get started with LLM-as-a-judge?” This guide shares what I’ve learned after helping over 30 companies set up their evaluation systems. The Problem: AI Teams Are Drowning in DataEver spend weeks building an AI system, only to realize you have no idea if it’s actually working? You’re not alone.

Aug 26, 2024 | hamel.dev | Hamel Husain

What is Dokku? Dokku is an open-source Platform as a Service (PaaS) that runs on a single server of your choice. It’s like Heroku, but you own it. It is a great way to get the benefits of Heroku without the costs (Heroku can get quite expensive!). I need to deploy many applications for my LLM consulting work. Having a cost-effective, easy-to-use serverless platform is essential for me. I run a Dokku server on a $7/month VPS on OVHcloud for non-gpu workloads.

Jul 29, 2024 | hamel.dev | Hamel Husain

Today, we are releasing Mastering LLMs, a set of workshops and talks from practitioners on topics like evals, retrieval-augmented-generation (RAG), fine-tuning and more. This course is unique because it is:Taught by 25+ industry veterans who are experts in information retrieval, machine learning, recommendation systems, MLOps and data science. We discuss how this prior art can be applied to LLMs to give you a meaningful advantage.