Transformer models dominate modern AI, but their computational and memory demands come with a steep energy cost. Analog compute-in-memory (CIM) is a promising way to cut that cost by performing calculations directly in memory using analog circuits. However, analog hardware is noisy, and that noise can degrade the accuracy of a trained transformer.
The paper introduces ABNAT (Attention-Based Noise-Aware Training), a method designed to make transformers robust to such noise. Instead of treating the hardware as ideal, ABNAT incorporates noise into the training process, specifically targeting the attention mechanism that is central to transformers. This allows the model to learn representations that survive the imperfections of analog CIM.
The authors report that ABNAT improves robustness compared to standard training, making it a practical step toward deploying energy-efficient analog accelerators for large-scale transformer workloads. The work highlights the importance of co-designing algorithms and hardware to bridge the gap between AI performance and energy efficiency.