Digital note-taking stores handwriting as vectorized "digital ink" that can be edited, searched, and replayed, but most people still write on paper. Bridging the two formats requires converting a photo of handwriting back into stroke data — a task the authors call derendering. Previous attempts focused on geometric image properties and generalized poorly beyond their training domains.

InkSight takes a different approach. It trains a vision-language model using both reading and writing priors, so the model learns what text means and how it is produced. This reduces the need for large paired datasets of photos and corresponding digital ink, which are difficult to obtain.

The model was tested on the challenging HierText dataset. In a human evaluation, 87% of generated samples were judged a valid tracing of the input image, and 67% looked like a pen trajectory a human would produce. The authors also report that the approach generalizes beyond handwriting to simple sketches. The paper has been accepted by Transactions on Machine Learning Research.