Eugene Yan lays out a straightforward approach to evaluating LLM products: label some data, align LLM evaluators, and run the eval harness with each change. The emphasis is on starting small and iterating, rather than building an elaborate evaluation system upfront.
The first step is to label a small set of examples that reflect real product use. These labels become the reference point for what good output looks like. Next, LLM-as-judge evaluators are aligned to those labels, so they can score outputs consistently with human expectations.
Finally, the eval harness is run every time a change is made to the system. That makes regressions visible early and keeps product quality measurable over time. The method is deliberately simple, treating evaluation as a routine part of development rather than a one-off research exercise.