A new preprint on arXiv examines how small, on-premises LLMs behave in legal settings. The authors note that confidentiality requirements are pushing legal professionals toward on-device models that keep case data in-house, using them for routine tasks like summarization, translation, and reformulation.
The study, titled "LLMs Prompted for Legal Context Refuse More," reports that these small models tend to overrefuse when prompted with criminal legal context. That is, they decline requests more often than they should, likely due to heightened caution around legal language.
The authors suggest this overrefusal could be a practical problem: if a model refuses too frequently, it becomes less useful for the very tasks it was adopted for. The preprint highlights a trade-off between safety and utility in domain-specific deployments of small LLMs.