Large language models are increasingly used in first-principles computational workflows, but their ability to handle the configuration of scientific calculations has not been systematically tested. A new arXiv preprint introduces INCARBench, a benchmark designed to evaluate LLMs on the task of writing VASP INCAR files.
INCARBench focuses on VASP, a widely used software package for atomic-scale materials simulation. The INCAR file controls the parameters of a calculation, so getting it right is essential. The authors argue that this configuration step is an area where LLM performance remains poorly understood, and they propose INCARBench as a way to measure it.
Because this is a single preprint, the article does not compare it with other benchmarks or report independent results. The paper's contribution is the benchmark itself, not a set of findings about model performance.