Transformer-based language models such as RoBERTa represent text using contextual word embeddings, which adjust each token's embedding based on the surrounding context. A new arXiv paper (2610.00840) builds on this property to investigate how meaning is constructed dynamically within a single utterance.
The authors propose tracking token-wise "contextual trajectories" — the path an embedding takes as context is incorporated — and measuring "incremental contextual displacement," or how much each added piece of context shifts the representation. Together, these tools offer a way to observe meaning construction step by step rather than treating the final embedding as a static output.
The paper's abstract is truncated in the source, so full methodological details are not yet available. However, the stated goal is clear: using LLMs themselves to better understand the dynamic, utterance-specific nature of meaning, rather than only using them as black-box predictors.