Hamel Husain

Oct 30, 2024 | hamel.dev | Hamel Husain

Earlier this year, I wrote Your AI product needs evals. Many of you asked, “How do I get started with LLM-as-a-judge?” This guide shares what I’ve learned after helping over 30 companies set up their evaluation systems. The Problem: AI Teams Are Drowning in DataEver spend weeks building an AI system, only to realize you have no idea if it’s actually working? You’re not alone.

Aug 26, 2024 | hamel.dev | Hamel Husain

What is Dokku? Dokku is an open-source Platform as a Service (PaaS) that runs on a single server of your choice. It’s like Heroku, but you own it. It is a great way to get the benefits of Heroku without the costs (Heroku can get quite expensive!). I need to deploy many applications for my LLM consulting work. Having a cost-effective, easy-to-use serverless platform is essential for me. I run a Dokku server on a $7/month VPS on OVHcloud for non-gpu workloads.

Jul 29, 2024 | hamel.dev | Hamel Husain

Today, we are releasing Mastering LLMs, a set of workshops and talks from practitioners on topics like evals, retrieval-augmented-generation (RAG), fine-tuning and more. This course is unique because it is:Taught by 25+ industry veterans who are experts in information retrieval, machine learning, recommendation systems, MLOps and data science. We discuss how this prior art can be applied to LLMs to give you a meaningful advantage.

May 11, 2024 | maven.com | Dan Becker |Hamel Husain

New·Cohort-based CourseAn online conference for everything LLMs.New·Cohort-based CourseAn online conference for everything LLMs.This course is popular130 people enrolled last week. Course overviewNote: Registration is paused and will reopen on June 2. The original course registration included compute credits on several platforms. Registrations after reopening will not include compute credits, but they will include access to all videos, materials and the Discord community.

Apr 12, 2024 | hamel.dev | Hamel Husain

Minimal Example: ft_driftI work with lots of folks who are fine-tuning models using the OpenAI API. I’ve created a small CLI tool, ft_drift, that detects drift between two multi-turn chat formatted jsonl files. Currently, ft_drift only detects drift in prompt templates, schemas and other token-based drift (as opposed to semantic drift). However, this is a good starting point to understand the general concept of adversarial validation.