Influence functions offer a principled way to identify which training samples most affect a model's predictions, enabling tasks like data selection, debugging, and efficient fine-tuning. However, scaling these methods to large language models has remained challenging due to the computational overhead involved.
The new work introduces ZOIF, a zeroth-order approach that estimates influence functions using only forward passes, avoiding the need for gradient computations or backward passes. This design could make influence estimation feasible for LLMs, where backpropagation is often prohibitively expensive.
The abstract does not include experimental results or comparisons, so the practical effectiveness of ZOIF is not yet demonstrated. Full details are expected in the paper.