Large language models hold substantial factual knowledge in their parameters, yet that knowledge is difficult to control, update, and reliably elicit at inference time. A new arXiv paper, "Mixture-of-Memories: Low-Overhead Language Modeling with Modular Knowledge Injection," addresses this problem by proposing a memory-augmented approach.
The paper introduces Mixture-of-Memories, a method designed for modular knowledge injection with low overhead. The abstract notes that existing memory-augmented approaches typically rely on a mechanism that is cut off in the available text, so the full technical details are not yet clear from the announcement.
What is clear is the motivation: making factual knowledge in LLMs more manageable and updatable. The proposed method appears to target this by injecting knowledge in a modular way, rather than relying solely on parameters. Further details would require reading the full paper.