Real-Time Dual-Modal 6D Pose Estimation for Virtual Reality Interaction Using Deep Learning

Abstract

Accurate and low-latency 6D pose estimation is a key requirement for natural interaction in virtual reality systems, yet existing methods often struggle to simultaneously satisfy precision, real-time performance, and robustness in multi-user or interference-prone environments. To address these challenges, this study proposes a deep learning-based dual-modal 6D pose estimation framework that integrates RGB and depth sensing for real-time virtual reality interaction. The proposed method employs a two-stream feature extraction architecture, where two-dimensional texture features are extracted from RGB images using a ResNet-18 backbone enhanced with efficient channel attention, while three-dimensional geometric features are obtained from depth data through an SO(3)-equivariant encoder combined with vector neuron linear layers and multilayer perceptrons. A cross-modal fusion strategy is introduced to achieve complementary enhancement between texture and geometric representations, followed by pose regression based on point pair feature matching. Experimental results on public benchmarks and multi-user virtual interaction scenarios demonstrate that the proposed approach achieves a median rotation error of 0.42◦ and a translation error of 0.38 cm, representing an error reduction of over 67% compared to models without SO(3) encoding. The system maintains a success rate of 95-98% under normal illumination and achieves a processing latency of 4.81 ms with 208 FPS on high-end hardware. Furthermore, the framework exhibits strong robustness to lighting variation, occlusion, and cross-user interference. These results indicate that the proposed dual-modal, SO(3)-aware pose estimation framework provides an effective computational solution for real-time 6D pose tracking and control in virtual reality interaction systems.