Prior-data fitted networks (PFNs) are a popular end-to-end approach to amortized Bayesian inference, but their behavior for semi-parametric targets—such as means under missingness or causal effects—has been unclear. In a new paper, Melnychuk and colleagues develop a theoretical foundation for debiased PFNs in these settings. They show that an idealized PFN that is Bayes-optimal for the target can still have a posterior concentration rate that implicitly depends on latent nuisance functions and slows down as the prior becomes more diverse.
To fix this, the authors propose two generic debiasing strategies based on efficient influence functions: a label correction and a prior correction. Both are designed to speed up the target posterior concentration rate. The key result is that the debiased PFNs achieve concentration rates that asymptotically match traditional frequentist bias-corrected estimators, establishing a novel connection between amortized Bayesian inference and semi-parametric efficiency theory.
Under additional assumptions, the debiased PFNs also guarantee the Bernstein-von Mises property and minimax optimality, meaning they are locally and globally optimal. The work provides practical guidelines for designing PFNs for semi-parametric inference with provable asymptotic guarantees.