OdoTest: An Automated Testing Approach for Odometry Systems

Full abstract

Read the full abstract

Reliable pose estimation is critical for intelligent systems, including autonomous vehicles, unmanned aerial vehicles, and virtual reality applications. While visual-inertial odometry (VIO) has made significant advancements in estimating pose, its performance can still be affected by sensor noise, environmental variations, and calibration errors. To evaluate the performance of VIO, existing testing methods rely on real-world datasets or manually degraded data, where sensor measurements or images are artificially modified according to predefined, fixed perturbation patterns. These approaches are costly, require time-consuming annotation, and rely on ad hoc modifications that limit scenario diversity and hinder the exploration of challenging cases. In this paper, we design and implement OdoTest, the automated testing framework for VIO systems. OdoTest equips sensor-specific transformation operators, including IMU perturbations, camera degradations, and sensor calibration errors, that systematically generate various and realistic test scenarios. Moreover, OdoTest adopts an odometry fitness-guided testing strategy to prioritize scenario generation, improving testing efficiency. By leveraging odometry-specific metamorphic relations (MRs), OdoTest can automatically detect odometry errors without requiring manual ground-truth labels for test scenarios. We evaluate OdoTest on twelve state-of-the-art VIO systems to answer three main research questions: 1) the effectiveness of transformation operators in revealing odometry errors; 2) the ability of OdoTest to generate error-triggering scenarios; and 3) whether retraining on these scenarios can improve odometry performance. Results show that OdoTest’s transformation operators effectively expose odometry errors, while its fitness-guided scenario generation efficiently creates challenging test cases. Moreover, retraining with these scenarios significantly enhances both estimation accuracy and system robustness, which demonstrates the usefulness of OdoTest.