Tabular foundation models based on prior-data fitted networks (PFNs) can predict on a new dataset in one forward pass, but they do not say which inputs their predictions depend on. The new paper introduces SelectPFN, a PFN that jointly performs Bayesian variable selection and prediction for sparse nonlinear regression. In a single pass it outputs posterior inclusion probabilities for all variables, and its predictions are computed from sampled subsets only, so excluded variables provably cannot affect them.
The network is pretrained once on synthetic tasks drawn from a meta-prior built around a group spike-and-slab sparse-input neural network. Column-permutation equivariance, combined with random interleaving of original and knockoff columns, yields knockoff statistics with exact finite-sample false discovery rate control, even when the meta-prior is misspecified. The theory also includes a Bayes-regret bound for the amortized median probability model and a characterization of how the information needed to detect a variable grows with the prior weight of structures that can use it.
On out-of-prior nonlinear designs with up to 500 predictors, SelectPFN improves the area under the precision-recall curve by 0.23–0.28 over lasso, BART, DART, LassoNet, and a group-penalized sparse neural network, at a false discovery proportion of about 0.07 and 0.04 seconds per dataset. The advantage persists on real covariate matrices, and detection of interaction-only variables rises monotonically with their prior weight, as the theory predicts. The paper also reports a negative result: average-case training does not transfer to frequentist guarantees at a fixed truth.