Mock politeness—saying something agreeable while meaning the opposite—is a subtle form of hostility that language models often miss. The Chinese idiom 'kou mi fu jian' ('honeyed words, hidden daggers') captures this phenomenon. A new paper introduces ZhenhuanBench, a benchmark designed to test how well LLMs can see through such language.
The benchmark consists of 1,592 items built from expert commentary on the Chinese drama 'Empresses in the…' (the title is truncated in the abstract). These items presumably pair polite surface forms with hostile underlying meanings, requiring models to go beyond literal interpretation.
The authors also investigate training approaches to improve LLM performance on this task. While the abstract does not provide results, the work highlights an important gap in current models: understanding pragmatic intent, especially when it is deliberately masked by politeness.