The Chatbot That Invented a Refund Policy: How AI Products Know They Work
12m | Sep 24, 2026Part of the Dev Tools for PMs series.
Air Canada's chatbot invented a bereavement refund policy that didn't exist, and a tribunal made the airline honor it. That's what shipping an AI product without evals looks like. This episode covers what evals actually are and how practitioners build them, with Madhu Guru's ten-part series as the spine: start from real traces, write the rubric before you measure, name failures precisely, run a ladder of evals, and never collapse everything into one average. Plus Hamel Husain on why failed AI products share one root cause. Then the counter-case: evals are a craft with open problems - judge reliability, trajectory traps, and real cost.

Learning Loop
Loading...