Vision-language-action (VLA) models perform well on standard robotic manipulation benchmarks, but their reliability drops when task conditions deviate from the nominal setting—for example, when object positions or task instructions are perturbed. The authors of a new arXiv paper argue that existing VLA policies lack an explicit runtime mechanism to distinguish failure modes and select an appropriate recovery strategy.
To address this, they propose FIT-MAC, a failure-aware, closed-loop multi-agent control framework that operates at inference time. A Planner Agent parses the language instruction into a structured task specification, and a Grounder Agent resolves each instruction-level reference to a visual target. A Verifier Agent then tracks phase-specific execution progress, distinguishing target-binding errors from local execution failures and determining whether intervention is warranted. If so, a Correction Agent routes the response to either target rebinding or bounded grasping/placement recovery; otherwise, the pretrained VLA remains the default action generator.
In experiments on the LIBERO-PRO benchmark, FIT-MAC improved robustness across multiple VLA backbones, consistently increasing success rates and achieving state-of-the-art performance under diverse perturbations. The results suggest that adding an inference-time control layer could be a practical way to make VLA-based manipulation more reliable without retraining the underlying policy.