Demonstrator Self-Preparation
Demonstrator-Learner Guidance
Learner Self-Practice
AI personal assistants, deployed through robots or wearables, require embodied understanding to collaborate effectively with humans. However, current Multimodal Large Language Models (MLLMs) primarily focus on third-person (exocentric) vision, overlooking the unique challenges of first-person (egocentric) videos. Additionally, high acquisition costs limit data size, impairing MLLM performance. To address these challenges, we propose learning the mapping between exocentric and egocentric domains, leveraging the extensive exocentric knowledge within existing MLLMs to enhance egocentric video understanding. To this end, we introduce Ego-ExoClip, a pre-training dataset comprising 1.1M synchronized ego-exo clip-text pairs derived from Ego-Exo4D, together with the instruction-tuning dataset EgoIT, which is collected from multiple sources to enhance the model's instruction-following capabilities. Building upon the datasets, we propose a migration strategy and further design a progressive mapping learning pipeline with three stages: Demonstrator Self-Preparation, Demonstrator-Learner Guidance, and Learner Self-Practice. Extensive experiments across diverse egocentric tasks reveal that existing MLLMs perform inadequately in egocentric video understanding, while our model significantly outperforms these leading models.
Illustration of our progressive training process, including the stages of Demonstrator Self-Preparation, Demonstrator-Learner Guidance, and Learner Self-Practice. "VTG", "KL", and "CCL" denote vision-grounded text generation, Kullback-Leibler divergence, and cycle consistency loss, respectively. "F" is a mapping function.
Our Exo2Ego model achieves optimal results across various egocentric video understanding tasks, with the results on EgoSchema, EgoTaskQA, and EgoMCQ coming from VideoChat2, and the rest from VideoLLaMA2.
In this paper, we propose Exo2Ego, a unified framework that significantly advances egocentric video understanding by cost-effectively transferring rich exocentric knowledge from MLLMs. To enable robust cross-view representation learning, we introduce Ego-ExoClip, a large-scale dataset comprising 1.1 million synchronized egocentric-exocentric clip-text pairs. Additionally, we curate EgoIT, an egocentric instruction-tuning dataset designed to further enhance instruction-following capabilities. By leveraging a novel exocentric-to-egocentric migration strategy and a progressive three-stage training pipeline, Exo2Ego effectively bridges the domain gap and promotes egocentric self-consistency. Extensive experiments on multiple benchmarks demonstrate that Exo2Ego consistently achieves substantial performance improvements.