Before language models are released, they are tested for harmful traits such as willingness to carry out dangerous requests, deception, power seeking, and sycophancy. But the safe behavior these tests certify may not hold up over time. A new arXiv preprint introduces a "calibrated instrument" designed to detect this kind of disposition drift.
The paper argues that as models are updated or deployed, their dispositions can shift away from the behaviors that were originally certified. The proposed instrument aims to measure such drift in a calibrated way, though the abstract does not specify the underlying method or experimental results.
If effective, this tool could help developers monitor models after release and ensure that safety certifications remain valid. The abstract is truncated, so the full scope of the work is not available from this source alone.