Testing LLM-based software is costly, the authors of a new arXiv paper argue. Each input requires several sampled responses, and each response must be judged by an expensive oracle. Adding to the burden, every prompt or model change calls for re-testing, so costs accumulate quickly across the development cycle.
The paper proposes 'pre-generation adequacy measures' to address this. Using what it calls the Jacobian lens, these measures aim to predict and characterise test failures before any response is generated, potentially avoiding the need to sample and judge responses for every input.
Because only the abstract was available for this digest, the specific mechanics of the Jacobian lens and the reported effectiveness of the measures are not detailed here. No other source was provided, so there is no independent comparison to note.