Privacy auditing of machine learning models typically relies on membership inference attacks (MIAs) and requires inserting a sensitive sample, or canary, into many randomized training runs. This multi-run approach is computationally expensive. One-run auditing places many canaries in a single training run, but recent theory shows that canaries can interfere with one another, weakening the audit's power.
To address this, the authors propose IBIS, a framework that optimizes canaries to maximize their own detectability while minimizing their impact on others. IBIS uses influence-based greedy initialization followed by bilevel optimization, and introduces a regularization term that encourages canaries to be orthogonal in embedding space as a proxy for reducing interference. An efficient bilevel algorithm updates a single model incrementally, avoiding full retraining after each canary update.
Experiments demonstrate that IBIS offers a better performance-cost trade-off than existing canary crafting approaches. It improves MIA metrics while reducing computational cost by up to two orders of magnitude, making one-run privacy auditing more practical for large-scale models.