Monocular Biomechanical Tracking of Fingers with Inverse Kinematics to Foundation Models
Researchers have developed a novel method for accurate hand and finger tracking from single-view video, addressing a gap in monocular biomechanical analysis. The approach integrates the SAM 3D Body foundation model with inverse kinematics optimization within a full-body biomechanical framework to extract anatomically constrained finger joint angles. To enable GPU-accelerated optimization, the team ported SAM 3D Body from PyTorch to JAX for seamless integration with MuJoCo-MJX. Additionally, they created a new mapping between Momentum Human Rig outputs and biomechanical model markers. Validation against an 8-camera multiview reconstruction system, using data from seven participants performing various hand poses and object manipulation tasks, demonstrated robust performance. The method achieved finger joint angle errors of approximately 10 degrees and hand position errors of around 6 mm after Procrustes alignment. These results remained consistent across different camera viewpoints and reference value methods. This advancement significantly expands access to quantitative characterization of hand movement using readily available video, offering potential clinical applications for monitoring daily living activities and measuring range of motion without requiring complex multi-camera setups.
Wire timeline
Monocular Biomechanical Tracking of Fingers with Inverse Kinematics to Foundation Models
Researchers have developed a novel method for accurate hand and finger tracking from single-view video, addressing a gap in monocular biomechanical analysis. The approach integrates the SAM 3D Body foundation model with inverse kinematics optimization within a full-body biomechanical framework to extract anatomically constrained finger joint angles. To enable GPU-accelerated optimization, the team ported SAM 3D Body from PyTorch to JAX for seamless integration with MuJoCo-MJX. Additionally, they created a new mapping between Momentum Human Rig outputs and biomechanical model markers. Validation against an 8-camera multiview reconstruction system, using data from seven participants performing various hand poses and object manipulation tasks, demonstrated robust performance. The method achieved finger joint angle errors of approximately 10 degrees and hand position errors of around 6 mm after Procrustes alignment. These results remained consistent across different camera viewpoints and reference value methods. This advancement significantly expands access to quantitative characterization of hand movement using readily available video, offering potential clinical applications for monitoring daily living activities and measuring range of motion without requiring complex multi-camera setups.
cs.AI updates on arXiv.org