Dynamic Tensor Rematerialization (DTR) is a technique for training neural networks under a fixed memory budget. Instead of keeping all intermediate tensors in memory, DTR evicts some and recomputes them when needed. This trades compute for memory, and the hope is that a budget large enough to finish training will stay sufficient as training proceeds.
A new preprint on arXiv challenges that expectation. The authors describe "hidden feasibility holes" in DTR: situations where a memory budget that should be sufficient for training nonetheless leads to failure. The abstract does not specify the exact mechanisms, but it suggests that the relationship between budget size and feasibility is not monotonic in the way one might assume.
Because this is a single preprint, the findings have not yet been peer-reviewed or independently verified. The abstract is also truncated, so the full scope of the claimed failure modes is not clear from the announcement alone. Still, the result is a useful caution for practitioners relying on rematerialization to meet memory constraints.