Estimating a person's 3D pose from a head-mounted camera is useful for AR and VR, but the setup creates unique difficulties. The stereo fisheye lenses introduce strong distortion, the body is often only partially visible, and the short baseline between the two cameras limits depth accuracy. According to the paper, existing refinement decoders that sample image features do not fully address these issues.

The authors propose EgoRayFormer, a refinement decoder that operates in ray space. Instead of sampling features from the distorted image plane, it samples along rays in 3D space, which better matches the geometry of fisheye stereo. This allows the model to combine information from both cameras more effectively and refine an initial 3D pose estimate.

The abstract positions this as a solution to the specific problems of egocentric pose estimation, though the full details of the architecture and experiments are in the paper.