Split large language models are designed to keep raw input text on the client, sending only intermediate representations to a server. But those representations are not necessarily private: existing attacks can reconstruct text from the messages. The problem for researchers is that evaluating such risks is inconsistent when different attackers are assumed to see different parts of the system.
The paper introduces LLM-Attack Suite, a unified red-team framework for privacy evaluation of split and distributed LLM systems. By standardizing the evaluation setting, the authors aim to make attack results more comparable across different models and deployment patterns.
This version is a replacement of the original preprint, though the abstract does not describe specific benchmark results or defenses. The framework itself is the contribution highlighted here.