Large language models are being used more often to annotate data, but their success varies widely across tasks and models. A new preprint, "Can This LLM Annotate My Data? Estimating Zero-Shot LLM Performance," takes up the question of how to estimate whether a particular LLM will be effective for a given annotation task without having to run it first.
The abstract notes that while several studies have examined both the opportunities and shortcomings of LLMs for annotation, a critical question remains unanswered. The paper appears to propose a way to estimate zero-shot performance, though the provided abstract excerpt does not detail the method or results.
For practitioners, such an estimation could save time and resources when choosing a model for annotation. The work is a preprint and has not yet undergone peer review.