Most multimodal LLMs rely on dedicated encoders to convert images, audio, or other modalities into token embeddings that the language model can process. A new arXiv preprint examines two recent models—Gemma-4 12B and Inkling-Small—that do away with this component entirely. Instead of using modality-specific encoders, these models take raw inputs directly, and the authors empirically analyze how they handle multimodal data.
Despite lacking explicit encoders, the models appear to learn implicit encoders as part of their internal representations. The paper positions this as a departure from the standard architecture, suggesting that explicit encoder modules may not be strictly necessary for multimodal understanding. The analysis is limited to the two named models, and the abstract does not claim broader applicability beyond these cases.