A new arXiv preprint takes up a practical question: when should teams fine-tune a small BERT-style model instead of prompting a large language model? The abstract notes that BERT-based models typically need thousands of labeled samples to train, while larger models like GPT-X can work with just 1–5 examples per class. That gap in data efficiency is well known, but the authors say the broader tradeoffs in performance, cost, and time are still under-explored.
The paper's title — "Best of Both Worlds: Fine-Tuned Small Models Complement LLMs for Cost and Performance Benefits" — suggests a nuanced answer: small fine-tuned models are not a replacement for LLMs but a complement. However, the abstract as provided is truncated at the point where the authors begin describing their work, so the specific experiments and results are not available in this summary.
Readers should treat the headline claim as a hypothesis or framing rather than a confirmed finding. The value of the paper, based on the abstract alone, lies in drawing attention to a decision that many teams face: choosing between data-hungry small models and few-shot-capable LLMs under real-world constraints.