ConCI benchmark tests LLMs' contextual integrity with counterfactuals

A new arXiv preprint introduces ConCI, a benchmark for evaluating whether large language models respect contextual integrity (CI). The core idea of CI is that the appropriateness of sharing personal information depends on the context of the information flow, not on the information alone.

The authors argue that existing CI benchmarks often use independent scenarios, which may not adequately capture how context changes the judgment. ConCI is proposed as a controlled counterfactual benchmark, presumably varying contextual factors to test LLMs more rigorously.

The abstract is brief, so details on the benchmark's construction or results are not yet available from this source. The paper is listed as arXiv:2610.24494v1.