This paper proposes a reference model for defining security benchmarks for the safety assessment of Cooperative Driving Automation (CDA) applications. Our reference model provides a systematic approach to benchmark the resilience of CDA applications against malicious attacks through extensive system simulations. It enables the test repeatability and comparison of results across different implementations of CDA applications. In our approach, a benchmark is defined as a series of tests that expose the target system to specific attacks while recording its response. Using this model, we define a benchmark for evaluating the resilience of Cooperative Adaptive Cruise Control (CACC) algorithms against barrage jamming attacks targeting the physical layer of the IEEE 802.11p communication standard. We apply this benchmark to assess and compare the performance of four CACC algorithms: P1, Flatbed, Ploeg, and Consensus. The benchmark measures reveal that the Consensus algorithm demonstrates the highest resilience against jamming attacks, primarily due to its heavy reliance on onboard sensors and the use of sensor data from all other vehicles for decision-making. In contrast, the P1 algorithm, which depends mainly on vehicle-to-vehicle (V2V) communication, proves to be the most vulnerable. Furthermore, the results indicate that vehicles are most susceptible to jamming attacks during acceleration phases, making these periods critical for security evaluation. These findings validate the effectiveness of our benchmarking framework in identifying strengths and vulnerabilities of CACC algorithms under cyberattacks.

This paper presents our initial contributions toward defining security benchmarks for simulation-based assessment of Cooperative Driving Automation (CDA) applications. A security benchmark is a process or procedure for assessing and validating a system’s ability to achieve its operational objectives in the presence of specific security attacks. This work lays the groundwork for developing security benchmarks that assess the robustness of CDA applications against jamming attacks. The driving scenario and the attack model are the core components of our proposed security benchmark. We used two scenarios braking and sinusoidal as a stimulus for evaluating the robustness of a platooning application modeled in a simulation framework called Plexe. The platooning application is equipped with a Cooperative Adaptive Cruise Control (CACC) controller. We injected barrage jamming attacks into the physical layer of the wireless communication system modeled by the IEEE 802.11p protocol. We demonstrate that jamming attacks can compromise safety, leading to emergency braking and collision incidents among platooning vehicles. Our findings also indicate that the severity of jamming attacks varies with the driving scenario, with the most severe impacts (i.e., collisions) occurring when the attack is injected during vehicle acceleration. 

Fault- and attack injection are techniques usedto measure dependability attributes of  omputer systems. Animportant property of such techniques is their efficiency inexploring the target system’s fault- or attack space. As this spaceis generally very large, pre-injection analysis techniques may beused to effectively explore the space. In this paper, we studytwo such techniques proposed in the past, namely inject-on-readand inject-on-write. Furthermore, we propose two new techniquescalled error space pruning of signals and error space pruning ofsignals and ports and evaluate their efficiency in reducing thespace needed to be explored by injection experiments. Thesetechniques were integrated into MODIFI, a fault- and attackinjector targeting Simulink models. To the best of our knowledge,we are the first to evaluate these pre-injection techniques for thiskind of injector.The results of our evaluation of 11 Simulink models from theautomotive domain and one from the avionics domain, show thatthe new proposed techniques reduce the fault- and attack spaceneeded to be explored by about 27–49%. Using MODIFI, we thenperformed injection experiments on two automotive models,  swell as an aero engine control model, while elaborating on theresults obtained.

Recent advances in high-performance graphics and physics engines (e.g., Unreal Engine) have popularized simulators for safety-critical system testing, yet credible validation is essential for reliable outcomes. This paper introduces a novel methodology for validating simulation toolchains, combining principles from SAE and UNECE frameworks with validation cycles to accommodate evolving safety-critical requirements. We demonstrate this approach through a case study evaluating the color fidelity of an Unreal Engine-based perception toolchain for safety-critical applications such as human and obstacle detection. Comparative tests of real and simulated camera outputs show that Unreal Engine’s camera model achieves "Delta E" < 4 under controlled lighting, closely matching the reference colors, but complex real-world lighting and seasonal variations can introduce perceivable color discrepancies. Our iterative methodology enables progressive refinements (reducing "Delta E" variations) and establishes critical traceability links for assessors related to evolving system requirements, toolchain modifications, as well as validation evidence. The resulting framework provides assessors with a verifiable chain of evidence from initial discrepancies to compliance, bridging the gap between adaptive development and certification needs.

This paper presents our initial contributions toward defining security benchmarks for simulation-based assessment of Cooperative Driving Automation (CDA) applications. A security benchmark is a process or procedure for assessing and validating a system’s ability to achieve its operational objectives in the presence of specific security attacks. This work lays the groundwork for developing security benchmarks that assess the robustness of CDA applications against jamming attacks. The driving scenario and the attack model are the core components of our proposed security benchmark. We used two scenarios braking and sinusoidal as a stimulus for evaluating the robustness of a platooning application modeled in a simulation framework called Plexe. The platooning application is equipped with a Cooperative Adaptive Cruise Control (CACC) controller. We injected barrage jamming attacks into the physical layer of the wireless communication system modeled by the IEEE 802.11p protocol. We demonstrate that jamming attacks can compromise safety, leading to emergency braking and collision incidents among platooning vehicles. Our findings also indicate that the severity of jamming attacks varies with the driving scenario, with the most severe impacts (i.e., collisions) occurring when the attack is injected during vehicle acceleration.

This paper presents CarFASE, an open-source carla-based fault and attack simulation engine that is used to test and evaluate the behavior of autonomous driving stacks in the presence of faults and attacks. Carla is a highly customizable and adaptable simulator for autonomous driving research. In this paper, we demonstrate the application of CarFASE by running fault injection experiments on OpenPilot, an open-source advanced driver assistance system designed to provide a suite of features such as lane keeping, adaptive cruise control, and forward collision warning to enhance the driving experience. A braking scenario is used to study the behavior of OpenPilot in the presence of brightness and salt&pepper faults. The results demonstrate the usefulness of the tool in evaluating the safety attributes of autonomous driving systems in a safe and controlled environment.

A remotely operated road vehicle (RORV) refers to a vehicle operated wirelessly from a remote location. In this paper, we report results from an evaluation of two safety mechanisms: safe braking and disconnection. These safety mechanisms are included in the control software for RORV developed by Roboauto, an intelligent mobility solutions provider. The safety mechanisms monitor the communication system to detect packet transmission delays, lost messages, and outages caused by naturally occurring interference as well as denial-of-service (DoS) attacks. When the delay in the communication channel exceeds certain threshold values, the safety mechanisms are to initiate control actions to reduce the vehicle speed or stop the affected vehicle safely as soon as possible. To evaluate the effectiveness of the safety mechanisms, we exposed the vehicle control software to various communication failures using a software-in-the-loop (SIL) testing environment developed specifically for this study. Our results show that the safety mechanisms behaved correctly for a vast majority of the simulated communication failures. However, in a few cases, we noted that the safety mechanisms were triggered incorrectly, either too early or too late, according to the system specification. 

A remotely operated road vehicle (RORV) refers to a vehicle operated wirelessly from a remote location. In this paper, we report results from an evaluation of two safety mechanisms: safe braking and disconnection. These safety mechanisms are included in the control software for RORV developed by Roboauto, an intelligent mobility solutions provider. The safety mechanisms monitor the communication system to detect packet transmission delays, lost messages, and outages caused by naturally occurring interference as well as denial-of-service (DoS) attacks. When the delay in the communication channel exceeds certain threshold values, the safety mechanisms are to initiate control actions to reduce the vehicle speed or stop the affected vehicle safely as soon as possible. To evaluate the effectiveness of the safety mechanisms, we exposed the vehicle control software to various communication failures using a software-in-the-loop (SIL) testing environment developed specifically for this study. Our results show that the safety mechanisms behaved correctly for a vast majority of the simulated communication failures. However, in a few cases, we noted that the safety mechanisms were triggered incorrectly, either too early or too late, according to the system specification.

Verification and validation (V&V) are complex processes combining different approaches and incorporating many different methods including many activities. System engineers regularly face the question if their V&V activities lead to better products, and having appropriate criteria at hand for evaluation of safety and cybersecurity of the systems would help to answer such a question. Additionally, when there is a demand to improve the quality of an already managed V&V process, there is a struggle over what criteria to use in order to measure the improvement. This paper presents an extensive set of criteria suitable for safety and cybersecurity evaluation of cyberphysical systems. The evaluation criteria are agreed upon by 60 researchers from 32 academic and industrial organizations jointly working in a large-scale European research project on 13 real-world use cases from the domains of automotive, railway, aerospace, agriculture, healthcare, and industrial robotics.

Reasoning about safety, security, and other dependability attributes of autonomous systems is a challenge that needs to be addressed before the adoption of such systems in day-to-day life. Formal methods is a class of methods that mathematically reason about a system’s behavior. Thus, a correctness proof is sufficient to conclude the system’s dependability. However, these methods are usually applied to abstract models of the system, which might not fully represent the actual system. Fault injection, on the other hand, is a testing method to evaluate the dependability of systems. However, the amount of testing required to evaluate the system is rather large and often a problem. This vision paper introduces formal fault injection, a fusion of these two techniques throughout the development lifecycle to enhance the dependability of autonomous systems. We advocate for a more cohesive approach by identifying five areas of mutual support between formal methods and fault injection. By forging stronger ties between the two fields, we pave the way for developing safe and dependable autonomous systems. This paper delves into the integration’s potential and outlines future research avenues, addressing open challenges along the way.