A new arXiv preprint reports a mixed-methods pilot in which a human researcher coordinated multiple LLM agents to build and evaluate SAT Math multiple-choice questions. The pipeline extracted questions from PDFs, converted open textbooks into structured representations, aligned each question with relevant content, and generated new questions under specified difficulty and cognitive levels. All original and generated questions were then scored against a 24-criterion quality framework.
Average quality was high across all evaluations, but criterion-level analysis and equivalence testing showed that generated questions were not fully comparable to expert-vetted baselines. Strict similarity—meaning equivalence on all 24 criteria—was never achieved. Persistent gaps appeared in skill depth, cognitive engagement, difficulty calibration, and metadata alignment, while surface-level qualities such as grammar fluency, clarity of options, and absence of duplicates were consistently strong.
The study also documents a shift in labor: the researcher's work moved from authoring items toward specification, orchestration, verification, and governance. Formalizing constraints, designing rubrics, building validation loops, recovering from tool failures, and auditing provenance became the primary activities. The author argues this points to emerging "AI research operations" skills needed for AI-empowered research pipelines.