Automatic prompt optimization has shown strong results for text-based large language models, but extending that success to audio-visual systems is not straightforward. A new arXiv paper argues that the key bottleneck is the textual interface through which multimodal information is passed to the LLM.
The authors focus on audio-visual question answering, where the model must reason over both sound and vision. According to the abstract, applying APO in this setting requires designing textual interfaces that can adequately represent multimodal content for the LLM to reason over.
Because only the abstract is available, the specific interface designs and experimental outcomes are not detailed here. The paper's contribution, as stated, is to identify this requirement and propose a path for bringing prompt optimization into multimodal reasoning.