IBM Research's ALTK-Evolve has taken the top spot on the τ²-bench retail leaderboard without relying on a frontier model. The system pairs a stock ReAct agent with GPT-4.1, an older non-reasoning model, and reaches 67.5% Pass^4 — meaning it completes the task on all four independent attempts for 67.5% of tasks. That score beats every retail entry in the October 12, 2026 leaderboard snapshot, according to the team's blog post.
Rather than fine-tuning weights, ALTK-Evolve builds a 'playbook' of learned guidelines. It analyzes the agent's attempts on training tasks and uses GPT-4.1 to distill successes and failures into short behavioral rules, such as authenticating a customer before taking order-related actions or confirming changes with a structured summary. These guidelines are appended to the system prompt for subsequent conversations. The same model handles both extraction and inference, and no human annotation is involved.
The approach was previously demonstrated on coding agents; this work applies it to customer service, where agents must follow domain policies and use tools correctly. On τ²-bench, tasks pair the agent with an LLM-simulated customer and a written policy, with a strict binary reward based on the final database state. The team compared configurations with no guidelines, a high-support core (support ≥ 3), and a lossless set of all consolidated guidelines, retaining 31 vs 292 guidelines in retail and 14 vs 101 in airline. The blog reports that the guidelines improve both capability and reliability, with Pass^4 as the leaderboard's key metric.