A team of researchers from multiple institutions proposes a structured approach to recursive self-improvement for video-understanding harnesses. The work, posted on arXiv, argues that an LLM system's capabilities depend not only on model parameters but also on the executable harness that organizes information acquisition, tool use, reasoning, and memory. Recursive evolution can automate harness design, but reliably discovering generalizable systems from limited, noisy, and repeatedly reused feedback remains an open problem.
The authors first introduce VideoHarness-BRSI, a recursive framework that evolves video harnesses around a frozen vision-language model. Analyzing over 4,000 evolved candidates, they find it competitive with hand-engineered systems but identify three interconnected failure modes: validation overfitting, search-budget dilution, and the early-experience trap. All three, they argue, share a common root: the same imperfect validation evidence drives both harness revisions and the proposer's search strategy.
To address this, they propose VideoHarness-SRSI, an evidence-governed framework in which every search decision rests on evidence that is transferable, sufficient, and gathered before the search commits. It restricts the proposer to content-free feedback, informs exploration with instrumented warm-up experiments, and develops mechanisms in isolated probe-race branches before factorial composition and joint refinement. Across four video-understanding benchmarks, SRSI improves the equally weighted mean held-out score by 6.2% relative to BRSI and reduces the mean validation-test gap by 37.9%. The authors conclude that reliable harness evolution depends on how search is organized and informed, not only on how well it optimizes validation performance.