RGB-infrared multi-object tracking combines visible and thermal cameras to keep identities consistent across frames. Most current methods compress both modalities into a single descriptor. If one modality is temporarily degraded—for example, by poor lighting or thermal noise—the tracker cannot reduce its influence, and unreliable evidence can distort association and contaminate appearance memory.
The authors propose RCA-MOT, which keeps RGB-specific, infrared-specific, and fused identity representations as separate, jointly accessible states. A simple training-free estimator computes branch-wise reliability for each detection. Relative reliability weighs the identity branches, absolute reliability calibrates borderline candidates near the ReID acceptance boundary, and the same reliability values control how strongly matched descriptors update historical memory.
The design deliberately avoids complex fusion or learned reliability modules, focusing instead on reliability-controllable appearance organization. On the VT-MOT and UniRTL benchmarks, RCA-MOT improves the previous best HOTA by 5.402 and 6.075 points, and IDF1 by 11.549 and 12.789 points, respectively. Post-hoc analyses indicate that the estimated reliability is an informative control signal. Code is planned for release after acceptance.