Most recommendation systems are evaluated on historical user interactions, but those logs are observational—they reflect what users were shown, not what they might have clicked if given different options. Eugene Yan's recent essay argues that this confounds cause and effect, making standard offline metrics misleading. He frames the problem as a choice between interventional evaluation (like A/B tests) and observational evaluation, and proposes a middle ground.
The key technique is inverse propensity scoring (IPS). In a logged dataset, each interaction has a probability of being observed given the system's prior policy. By weighting each logged event by the inverse of that probability, IPS creates a pseudo-randomized sample that approximates what an A/B test would have shown. This corrects for the selection bias inherent in production logs.
Yan's approach is not a silver bullet—it requires accurate propensity estimation and can be unstable with extreme weights. But it offers a principled way to squeeze more reliable signal from existing data, which is especially valuable when running live experiments is costly or slow. For teams building recommenders, this shifts the question from "what did users click?" to "what would they have clicked if we'd shown them something else?"—a subtle but crucial reframing.