A new technique for studying large language models turns the tables on them: instead of asking models to answer test questions, researchers ask them to write biographies. This simple prompt acts as a window into what the model has absorbed and how it reproduces that information. Because biography generation is open-ended, the model must draw on its internalized knowledge of people and events, making it easier to notice when it is relying on training data rather than reasoning from context.
The method, described by Eugene Yan, treats biographies as a way to gauge memorization and regurgitation. It does not measure performance on a fixed benchmark but instead lets the model freely generate, offering a direct view of what it remembers. This can reveal not only that a model knows certain facts, but also how it arranges and expresses them in a natural narrative.
This technique could help researchers better understand the boundary between memorization and genuine generalization. It also provides a practical way to inspect what models have learned, though it does not by itself adjust or correct that behavior.