The increasing complexity of automated and AI-based vehicle systems necessitates innovative testing methodologies to ensure safety, reliability, and performance in diverse and dynamic scenarios. This paper introduces AV Test Operating System (ATOS), an open-source platform that seamlessly integrates simulated and physical testing environments for Advanced Driver-Assistance Systems and Automated Driving systems. ATOS supports comprehensive scenario evaluations under varied conditions such as weather, traffic, and connectivity environments by automating and orchestrating tests involving multiple simultaneous virtual and physical objects. Key contributions of this work include: 1) The introduction and design of ATOS, an open-source platform for automating and orchestrating complex vehicle testing scenarios. 2) A method for evaluating the repeatability of automated test orchestration systems. 3) Data analysis of two scenarios where ATOS was used in real-life AV testing. Results demonstrate ATOS’s effectiveness in executing repeatable and reliable tests across diverse configurations, highlighting its utility for research institutions, vehicle manufacturers, and testing facilities. Using ATOS in repeated runs, the time delay when triggering dynamic events was within 1 millisecond with virtual objects and 7 milliseconds when using physical objects. The positional variation between runs using virtual and real-life objects amounted to 6 and 20 cm, respectively. Future enhancements will focus on integrating real-world data and expanding ATOS’s capabilities to support evolving vehicle testing needs, contributing to safer, more robust transportation systems.