Manual segmentation of tumors and organs-at-risk is a bottleneck in oncology, and most existing AI tools require manual spatial prompts or task-specific retraining. The new model SyRe instead takes text descriptions as input, strengthening the interaction between visual and linguistic representations through a method the authors call synergistic vision-language reinforcement.
To support this approach, the team built SyReData, a dataset of 20 million image-mask-description triplets covering 9 modalities and 229 segmentation tasks. A key component is Color Region Description (CRD), which generates region-level semantic supervision. Training with varied prompt formats lets SyRe handle single-target, multi-target, and open-ended segmentation, while also rejecting prompts that do not correspond to valid targets.
In evaluation on 28 unseen external datasets—including 17 cancer types, 3 non-cancer diseases, and multinational in-house cohorts—SyRe showed strong generalization under distribution shift. Beyond spatial overlap, the generated masks preserved quantitative pathology information: across five retrospective tumor imaging cohorts using CT and MRI, radiomics derived from SyRe masks stratified survival and improved prognostic models over clinical baselines. The authors also describe a clinician-in-the-loop refinement step for case-level corrections, positioning SyRe as a generalizable framework for scalable quantitative oncology analysis.