A new benchmark called ABACUS aims to measure how well instruction-based image editing models handle object category and counting. The benchmark is designed around scene conservation: the model must add, remove, or change objects while keeping the overall scene unchanged.
According to the arXiv abstract, current models can perform these operations, but they frequently produce the wrong number of objects. The abstract also states that these failures grow, though the announcement is truncated before giving the full trend.
ABACUS appears intended as a diagnostic tool for a specific weakness in image editing. By separating counting and category accuracy from scene preservation, it may help researchers identify where models fall short.