IBM Research has released ScarfBench, a benchmark designed to measure the ability of AI agents to migrate enterprise Java applications from one framework to another. The benchmark aims to fill a gap in evaluating agents on realistic, large-scale coding tasks that go beyond simple code generation.
ScarfBench includes tasks that mimic the kinds of migrations enterprises routinely undertake, such as shifting between frameworks with different architectural conventions and dependency management. By standardising these scenarios, the benchmark provides a common ground for comparing AI agents' performance across both accuracy and efficiency.
The initial findings indicate that while agents can handle straightforward components, they often stumble on multi-step migrations that require understanding legacy code and adapting to framework-specific idioms. This suggests that current AI tools are not yet ready to fully automate enterprise migration work without human oversight.