A new arXiv paper reports on a system that generates feedback with large language models, deployed in a live web development course. The study spans three iterations, each using a different underlying model and a different feedback format, allowing the researchers to compare conditions in a real educational setting.
The authors ask how the presence of feedback and its characteristics—such as length—affect effectiveness. The paper's title, "It's Complicated," suggests that no single factor dominates, and that outcomes likely depend on interactions between model choice, format, and context.
Because the available source is the abstract, specific results are not detailed here. The significance lies in the study's ecological validity: it examines LLM feedback as actually used by students in a live course, rather than in a controlled lab task.