Tabular foundation models are designed to perform in-context learning: given a labeled support table, they predict unseen observations without needing task-specific parameter updates. This makes them flexible for a wide range of tabular tasks, but most existing approaches rely on dense architectures, which may limit scalability or efficiency.
OrionMoE-v1.0, described in a new arXiv preprint, instead adopts a mixture-of-experts (MoE) architecture for tabular in-context learning. By routing inputs to specialized experts, the model aims to capture diverse patterns in tabular data more effectively than a single dense network. The abstract explicitly frames this as a departure from the dense-model norm, though it does not detail the specific advantages.
The preprint's abstract is truncated, so no experimental results or comparisons are available in the source. The contribution is therefore presented as a architectural proposal, with the promise of improved in-context learning for tabular data. Further details would require the full paper.