HPC users routinely download datasets onto cluster storage for analysis and processing. When a dataset consists of hundreds of thousands of small files—or more—parallel access to those files can place a tremendous burden on the underlying file system, slowing jobs and complicating data management.
The preprint "Creating SquashFS Images to Support HPC and AI Datasets" (arXiv:2610.20578) addresses this problem by proposing the creation of SquashFS images. SquashFS is a compressed, read-only filesystem format that packages many files into a single image, which can make large collections of small files easier to store and access in high-performance computing and AI workflows.
The paper is positioned as a practical approach to supporting HPC and AI datasets, but the abstract provided is truncated, so specific implementation details and evaluation results are not available in this source. The core idea, however, is clear: convert unwieldy collections of small files into compact SquashFS images to reduce the access burden.