Most tabular machine learning models are built on the assumption that training data faithfully represent the population on which they will be deployed. In practice, that assumption is often violated: datasets are assembled from whatever is easiest to collect, a practice known as convenience sampling. A new arXiv preprint introduces INCONVENIENT, a benchmark designed to assess how well large language models handle this mismatch.

INCONVENIENT focuses on tabular prediction tasks, where the benchmark likely exposes models to training sets that are skewed or otherwise non-representative. The goal is to measure whether LLMs can generalize beyond the convenience sample to the broader population, a capability that standard evaluation setups may not capture.

Because the abstract is brief, details on the benchmark's construction and results are not yet available from the source. The paper's contribution at this stage is to formalize a problem that is common in applied machine learning but often overlooked in model evaluation.