ServiceNow CoreAI researchers describe AutoSynthData, a pipeline for generating agent training data tailored to a particular enterprise environment. The motivation is that a broadly capable model can still struggle with environment-specific workflows, tool combinations, or constraints. AutoSynthData starts by running the target model and a stronger teacher on diagnostic tasks, then distills failure patterns into capability specification cards. The generator creates new tasks from those cards without seeing the original prompts, entities, trajectories, or verifier details.
Each task is represented as a triple: system specification, user prompt, and verifier. The system specification defines constraints and environment state; the user prompt must be feasible, realistic, and difficult for the current model; and the verifier must be consistent, sound, and complete. The authors warn that a lax verifier can reward incorrect behavior, while an overly restrictive verifier can penalize valid solutions.
Generated tasks are executed in the environment, and accepted samples are used for post-training. The updated model is then evaluated again, so the next generation round targets gaps that remain. The authors illustrate the approach with EnterpriseOps Gym and a released dataset.