Spiking neural networks are a candidate architecture for energy-efficient language modeling, replacing dense continuous activations with sparse, binary spikes. But according to a new arXiv preprint, the efficiency-accuracy tradeoff of these models has not been characterized at scale. The authors say they present the first compute-optimal scaling laws for spiking language models.
However, the abstract provided is cut off after "We present the first…", so the specific scaling exponents, model sizes, and comparisons to conventional transformers are not available in the source. The claim of being first is the only concrete result we can report.
Because there is only one source, there is no independent agreement or disagreement to note. Readers interested in the actual scaling-law coefficients will need to consult the full paper.