Verification and validation of vehicles is a complex yet critical process, particularly for ensuring safety and coverage through simulations. However, achieving realistic and useful simulations comes with significant challenges. To explore these challenges, we conducted a workshop with experts in the field, allowing them to brainstorm key obstacles. Following this, we distributed a survey to consolidate findings and gain further insights into potential solutions. The experts identified 17 key challenges, along with proposed solutions, an assessment of whether they represent next steps for research, and the roadblocks to their implementation. While a lack of resources was not initially highlighted as a major challenge, utilizing more resources emerged as a critical necessity when experts discussed solutions. Interestingly, we expected some of these challenges to have already been addressed or to have systematic solutions readily available, given the collective expertise in the field. Many of the identified problems already have known solutions, allowing us to shift focus towards unresolved challenges and share the next steps with the broader community

As software applications get increasingly connected and complex, cybersecurity becomes more and more important to consider during development and evaluation. Software engineers need to be aware of various security threats and the countermeasures that can be taken to mitigate them. Currently, there is a lack of guidance for software engineers aiming to develop secure web applications. We conducted a design science research study, resulting in a set of guidelines to aid software engineers in developing secure web applications. The set of guidelines was constructed based on interview data with 10 industry practitioners. These guidelines were then evaluated using a survey with 28 respondents. Additionally, we conducted experiments in which we provided a large language model with our guidelines and vulnerability reports as input. The large language model should extend the given vulnerability reports by recommending which of our guidelines can help prevent the given vulnerability in the future. The extended reports were evaluated by two external researchers experienced in cyber security and one author. Our results indicate that developers consider using these proposed guidelines for the development and assessment of secure web applications in different stages of the software development lifecycle. Our results also show that it is possible to automatically enhance vulnerability reports to support developers meaningfully and that the guidelines recommended by the large language model are useful to prevent the respective vulnerabilities in the future

Software defined vehicles mark a shift to software-driven mobility, posing complex engineering and organizational challenges. To stay competitive, automotive industry players must adapt the architecture, development process, and organization to effectively manage complexity, updates, regulatory compliance, and rapid technological evolution

Security Chaos Engineering (SCE) is a proactive approach to identify vulnerabilities and enhance security of systems. It embraces continuous security experimentation to build confidence in the capability of systems to withstand malicious conditions. Different SCE techniques are proposed for enhancing the resilience of software systems. The diversity of SCE techniques indicates the need for their collective analysis to uncover valuable practices and potential research opportunities. To fulfill this need, we consolidate and unify the knowledge on SCE practices through a systematic literature review. The results show that there has been limited and unsystematic investigation of SCE by the community, highlighting the importance of creating and promoting guidelines for SCE practices. Therefore, we create SCENE, a comprehensive set of guidelines for systematically reporting SCE. The goal is to support the clarity, consistency, and reproducibility of SCE practices. SCENE guidelines are evaluated by cybersecurity practitioners and active researchers in the field, and is mapped to established methodological guidelines. The results indicates that SCENE is perceived positive in terms of usefulness, understandability, practicality, and completeness. SCENE is also found to complement established experimental reporting guidelines and bridge the gap between academic studies and industrial use

The EVIDENT project aims to address challenges in the automotive industry's validation and verification (V&V) processes for advanced driver assistance systems (ADAS) and autonomous driving (AD) features. Traditional V&V methods struggle to keep up with the increasing frequency of software updates. The project explores virtual validation strategies to complement or replace physical testing, thereby enhancing efficiency and safety assurance.

Automotive innovations are increasingly software-driven, necessitating frequent updates. Current validation processes heavily rely on physical testing, which is time-consuming and costly. The project focuses on how vehicle functionalities could be tested and validated in simulation models and what fidelity level that could be reached. By utilizing virtual environments, the project aims to proactively test software functions before deployment, ensuring accurate assessments of system performance in diverse scenarios.

The primary goal is to develop strategies that balance the realism of virtual test environments with practical implementation. Key research questions include:

• What level of realism is required for simulations to be credible for testing edge cases?
• How can virtual testing be integrated with real-world data to discover new edge cases?
• How can virtual testing ensure functional safety to satisfy regulatory bodies?

The project also seeks to establish metrics for comparing physical and virtual test results and to utilize open-source tools for broader industry use.

The project follows a structured approach:

1. Gap Analysis: Semi-structured interviews with industry experts were conducted to identify current best practices and challenges.
2. Simulation Toolchain Assessment: Each partner's simulation tools, and maturity levels were evaluated.
3. Scenario Development: Road network representations and test scenarios were developed using ASAM OpenDRIVE and OpenSCENARIO formats.
4. Physical Testing: Various scenarios were tested on the AstaZero proving ground using vehicles equipped with advanced sensors and emergency braking systems.
5. Simulations: Partners conducted virtual tests using the respective tool chains. The simulations aimed to replicate physical test conditions and gather comparable data.
6. Data Comparison: Physical and simulated test data were compared to evaluate fidelity levels and trustworthiness. Metrics such as time to collision (TTC), braking distances, and object detection errors were analysed.

Five key case studies were tested:

1. Automated Lane Keeping System (ALKS)
2. Car-to-Car Front Turn-Across-Path (CCFTap)
3. Car in Curve
4. S-Curve
5. Occluded Child

Each scenario focused on different aspects of vehicle dynamics, sensor performance, and emergency braking responses. For instance, the Occluded Child scenario tested automatic emergency braking when a child runs out from behind parked cars.

The project identified gaps between physical and simulated test results, such as differences in braking activations between physical test and simulation. It also highlighted the need for improving simulation tools' ability to replicate real-world vehicle behaviour accurately.

Key findings include:

• Virtual tests can be reliable but require tuning to achieve higher fidelity.
• Physical tests remain crucial for validating simulation models.
• Establishing standardized KPIs for virtual testing is essential to enhance credibility.

The project faced several challenges such as:

• Variability in sensor models across partners.
• Human factors introducing inconsistencies in physical tests.
• Limitations of existing simulation tools to accurately replicate real-world scenarios.

A comprehensive list of challenges was compiled to guide future research and development efforts.

EVIDENT successfully demonstrated the potential of virtual validation for ADAS and AD features. The project contributed to developing methodologies for comparing physical and virtual tests and provided insights into the requirements for credible virtual toolchains.

Future research is recommended to focus on refining simulation validation methods, improving data synchronization methods, and addressing identified challenges to make virtual validation a practical and reliable component of automotive software development.

As software applications get increasingly connected and complex, cybersecurity becomes more and more important to consider during development and evaluation. Software engineers need to be aware of various security threats and the countermeasures that can be taken to mitigate them. Currently, there is a lack of guidance for software engineers aiming to develop secure web applications. We conducted a design science research study, resulting in a set of guidelines to aid software engineers in developing secure web applications. The set of guidelines was constructed based on interview data with 10 industry practitioners. These guidelines were then evaluated using a survey with 28 respondents. Our results indicate that these proposed guidelines can be applied by software engineers to support the development and assessment of secure web applications in different stages of the software development lifecycle. 

In safety-critical software systems, cybersecurity activities become essential, with risk assessment being one of the most critical. In many software teams, cybersecurity experts are either entirely absent or represented by only a small number of specialists. As a result, the workload for these experts becomes high, and software engineers would need to conduct cybersecurity activities themselves. This creates a need for a tool to support cybersecurity experts and engineers in evaluating vulnerabilities and threats during the risk assessment process. This paper explores the potential of leveraging locally hosted large language models (LLMs) with retrieval-augmented generation to support cybersecurity risk assessment in the forestry domain while complying with data protection and privacy requirements that limit external data sharing. We performed a design science study involving 12 experts in interviews, interactive sessions, and a survey within a large-scale project. The results demonstrate that LLMs can assist cybersecurity experts by generating initial risk assessments, identifying threats, and providing redundancy checks. The results also highlight the necessity for human oversight to ensure accuracy and compliance. Despite trust concerns, experts were willing to utilize LLMs in specific evaluation and assistance roles, rather than solely relying on their generative capabilities. This study provides insights that encourage the use of LLMbased agents to support the risk assessment process of cyber-physical systems in safety-critical domains

The risks associated with adopting large language model (LLM) chatbots in software organizations highlight the need for clear policies. We examine how 11 companies create these policies and the factors that influence them, aiming to help managers safely integrate chatbots into development workflows.

Color fidelity is often overlooked in simulation-based validation for autonomous vehicles, yet even minor color mismatches can undermine the reliability of AI-driven perception systems. In this paper, we systematically examine how controlled deviations in color reproduction—quantified by \DeltaE{}—affect object detection accuracy across 32 variants of YOLO. Using a Macbeth ColorChecker, we derive calibrations for key color transforms (brightness, contrast, hue, gamma, saturation and color bias) and apply these to the COCO validation set. Our evaluations demonstrate that increasing \DeltaE{} yields significant drops in detection metrics, especially for safety-critical categories such as pedestrians and cyclists. Based on these findings, we propose \DeltaE{} thresholds that define acceptable color fidelity in camera simulations (e.g., \DeltaE{} $\leq 3$ for $\Delta$mAP $\leq 1\%$). Furthermore, we contribute these transformed datasets and scripts as a publicly available benchmark, enabling reproducible comparisons and guiding future research on color-based vulnerabilities in automated driving and other safety-critical domains.

Large Language Models (LLMs) are increasingly used by software engineers for code generation. However, limitations of LLMs such as irrelevant or incorrect code have highlighted the need for prompt programming (or prompt engineering) where engineers apply specific prompt techniques (e.g., chain-of-thought or input-output examples) to improve the generated code. While some prompt techniques have been studied, the impact of different techniques — and their interactions — on code generation is still not fully understood. In this study, we introduce CodePromptEval, a dataset of 7072 prompts designed to evaluate five prompt techniques (few-shot, persona, chain-of-thought, function signature, list of packages) and their effect on the correctness, similarity, and quality of complete functions generated by three LLMs (GPT-4o, Llama3, and Mistral). Our findings show that while certain prompt techniques significantly influence the generated code, combining multiple techniques does not necessarily improve the outcome. Additionally, we observed a trade-off between correctness and quality when using prompt techniques. Our dataset and replication package enable future research on improving LLM-generated code and evaluating new prompt techniques.