Split and distributed large language models are designed to protect privacy by keeping raw input text on the client device, sending only intermediate representations to a server. However, the paper notes that attacks can reconstruct text from these messages, undermining that protection. The challenge is that privacy evaluations are difficult to compare because different attackers may have access to different information or capabilities.
To address this, the paper introduces LLM-Attacker, a unified red-team framework for privacy evaluation of such systems. By standardizing the attack setup and evaluation process, the framework aims to make results more consistent and reproducible across studies. The abstract emphasizes the need for a common benchmark rather than ad-hoc tests.
The paper is a single source, so there are no conflicting findings to reconcile. The details of the framework's implementation and experimental results are not fully described in the abstract, but the motivation is clear: without a unified approach, it is hard to know whether one split-LLM system is genuinely more private than another.