Generative models for discrete data typically follow one of two paradigms: autoregressive models like GPT, which generate tokens in a fixed left-to-right order, or masked diffusion models, which can generate in any order. Each choice imposes assumptions about how sequences should be produced, but neither treats the order itself as something the model can optimize.
OrderGPT, described in a new arXiv preprint, reframes generation order as a learnable modeling choice. Rather than hard-coding a left-to-right or any-order strategy, the model can discover the ordering that best suits the data and task. The abstract suggests this offers a way to bridge the two popular paradigms, though full experimental details are not included in the announcement.
Because only the abstract was available for this digest, the specific architecture and results are not yet detailed. The significance lies in the conceptual shift: instead of treating order as a fixed prior, it becomes part of what the model learns.