Four recent arXiv preprints illustrate the range of open problems in federated learning. One pair of papers concentrates on efficiency and scale: one describes a way to make federated learning itself more easily distributable, while another applies federated ideas to accelerate sharded data-parallel training of foundation models. The other two papers take different angles, with one addressing distributional robustness when client data comes from unknown mixtures and the other proposing capacity-constrained submodeling for clients with unequal compute.
The papers converge on a shared premise: practical federated learning cannot rely on idealized assumptions. Existing frameworks often omit key operational concerns, foundation-model training at scale creates convergence challenges, client data distributions can vary in ways the server cannot observe, and clients may have very different computational budgets. Together, these works point to a field that is maturing from prototype federations toward real-world deployment.
Where the papers differ is in their proposed remedies. One emphasizes efficient distribution of the federated workflow itself, another targets the scaling bottleneck in high-performance computing systems, a third uses distributionally robust optimization to handle unknown group mixtures, and the fourth selects submodel parameters based on Fisher importance to match client capacities. These approaches are complementary rather than competing, each addressing a distinct failure mode in federated learning at scale.