Ego-Exo Object Correspondence by SAM2 and Cross-View Prompting

The Ego-Exo Object Correspondence task is a recently introduced problem in computer vision that focuses on transferring object segmentation between synchronized egocentric (first-person) and exocentric (third-person) video streams. While the Segment Anything Model 2 (SAM2) has shown strong performance in single-view video object segmentation (VOS), it is not inherently equipped to handle cross-view video segmentation tasks. This limitation leaves open the question of how to fully exploit SAM2’s capabilities in multi-view scenarios. In this work, we propose a method that automatically generates cross-view prompts for SAM2 using an ego-exo object co-segmentation algorithm. By doing so, we extend SAM2’s effectiveness to more accurately segment object instances that appear across both perspectives during the same human-object interaction. Our approach demonstrates promising performance compared to existing state-of-the-art methods.