Pipeline parallelism is a critical technique for training large language models, but existing sequence-level methods schedule subsequences in a single direction, which can limit throughput. A new arXiv preprint, BSIPipe, proposes a bidirectional approach to address this bottleneck.
The method, described in the abstract as "bidirectional sequence-level interleaved pipeline parallelism," is designed specifically for long-sequence training. By scheduling subsequences in both directions, it aims to overcome the throughput limitation that unidirectional scheduling imposes. The abstract does not yet provide experimental details or quantitative results.
As a preprint, the work has not been peer-reviewed. Its significance lies in the proposed direction: improving efficiency for the increasingly long sequences used in modern LLM training, where pipeline parallelism is a key scaling tool.