T2J-BENCH, introduced in a new arXiv paper, is a benchmark for measuring how well coding agents handle migrating machine-learning codebases from PyTorch to JAX. The authors describe this as a routine but costly engineering task, and one that is fundamentally cross-paradigm: PyTorch modules tend to be eager, stateful, and object-oriented, while JAX code follows a different programming model.
The benchmark is execution-grounded, meaning that agent outputs are judged on whether the migrated code runs correctly rather than on surface-level similarity to a reference. This makes the task more demanding than typical code-generation benchmarks, which often reward syntactically plausible but non-functional code.
By focusing on repository migration, T2J-BENCH targets a real-world maintenance scenario that is common in ML engineering. It provides a way to assess whether coding agents can perform the kind of careful, behavior-preserving rewriting that framework migrations require.