This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (Formula presented.) and (Formula presented.) evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific crossover and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints. 

Over the past few decades, Industrial Control Systems (ICS) have been targeted by cyberattacks and are becoming increasingly vulnerable as more ICSs are connected to the internet. Using Machine Learning (ML) for Intrusion Detection Systems (IDS) is a promising approach for ICS cyber protection, but the lack of suitable datasets for evaluating ML algorithms is a challenge. Although a few commonly used datasets may not reflect realistic ICS network data, lack necessary features for effective anomaly detection, or be outdated. This paper introduces the ’ICS-Flow’ dataset, which offers network data and process state variables logs for supervised and unsupervised ML-based IDS assessment. The network data includes normal and anomalous network packets and flows captured from simulated ICS components and emulated networks, where the anomalies were applied to the system through various cyberattacks. We also proposed an open-source tool, ’ICSFlowGenerator,’ for generating network flow parameters from Raw network packets. The final dataset comprises over 25,000,000 raw network packets, network flow records, and process variable logs. The paper describes the methodology used to collect and label the dataset and provides a detailed data analysis. Finally, we implement several ML models, including the decision tree, random forest, and artificial neural network to detect anomalies and attacks, demonstrating that our dataset can be used effectively for training intrusion detection ML models.

Integration of machine learning (ML) components in critical applications introduces novel challenges for software certification and verification. New safety standards and technical guidelines are under development to support the safety of ML-based systems, e.g., ISO 21448 SOTIF for the automotive domain and the Assurance of Machine Learning for use in Autonomous Systems (AMLAS) framework. SOTIF and AMLAS provide high-level guidance but the details must be chiseled out for each specific case. We initiated a research project with the goal to demonstrate a complete safety case for an ML component in an open automotive system. This paper reports results from an industry-academia collaboration on safety assurance of SMIRK, an ML-based pedestrian automatic emergency braking demonstrator running in an industry-grade simulator. We demonstrate an application of AMLAS on SMIRK for a minimalistic operational design domain, i.e., we share a complete safety case for its integrated ML-based component. Finally, we report lessons learned and provide both SMIRK and the safety case under an open-source license for the research community to reuse. © 2023, The Author(s).

Multi-level image thresholding is a common approach to image segmentation where an image is divided into several regions based on its histogram. Otsu's method is the most popular method for this purpose, and is based on seeking for threshold values that maximise the between-class variance. This requires an exhaustive search to find the optimal set of threshold values, making image thresholding a time-consuming process. This is especially the case with increasing numbers of thresholds since, due to the curse of dimensionality, the search space enlarges exponentially with the number of thresholds. Population-based metaheuristic algorithms are efficient and effective problem-independent methods to tackle hard optimisation problems. Over the years, a variety of such algorithms, often based on bio-inspired paradigms, have been proposed. In this paper, we formulate multi-level image thresholding as an optimisation problem and perform an extensive evaluation of 23 population-based metaheuristics, including both state-of-the-art and recently introduced algorithms, for this purpose. We benchmark the algorithms on a set of commonly used images and based on various measures, including objective function value, peak signal-to-noise ratio, feature similarity index, and structural similarity index. In addition, we carry out a stability analysis as well as a statistical analysis to judge if there are significant differences between algorithms. Our experimental results indicate that recently introduced algorithms do not necessarily achieve acceptable performance in multi-level image thresholding, while some established algorithms are demonstrated to work better. 

With the advent of the smart industry, Industrial Control Systems (ICS) moved from isolated environments to connected platforms to meet Industry 4.0 targets. The inherent connectivity in these services exposes such systems to increased cybersecurity risks. To protect ICSs against cyberattacks, intrusion detection systems (IDS) empowered by machine learning are used to detect abnormal behavior of the systems. Operational ICSs are not safe environments to research IDSs due to the possibility of catastrophic risks. Therefore, realistic ICS testbeds enable researchers to analyze and validate their IDSs in a controlled environment. Although various ICS testbeds have been developed, researchers’ access to a low-cost, extendable, and customizable testbed that can accurately simulate ICSs and suits security research is still an important issue. In this paper, we present ICSSIM, a framework for building customized virtual ICS security testbeds in which various cyber threats and network attacks can be effectively and efficiently investigated. This framework contains base classes to simulate control system components and communications. Simulated components are deployable on actual hardware such as Raspberry Pis, containerized environments like Docker, and simulation environments such as GNS-3. ICSSIM also offers physical process modeling using software and hardware in the loop simulation. This framework reduces the time for developing ICS components and aims to produce extendable, versatile, reproducible, low-cost, and comprehensive ICS testbeds with realistic details and high fidelity. We demonstrate ICSSIM by creating a testbed and validating its functionality by showing how different cyberattacks can be applied. © 2023 The Authors

Processing and reviewing nightly test execution failure logs for large industrial systems is a tedious activity. Furthermore, multiple failures might share one root/common cause during test execution sessions, and the review might therefore require redundant efforts. This paper presents the LogGrouper approach for automated grouping of failure logs to aid root/common cause analysis and for enabling the processing of each log group as a batch. LogGrouper uses state-of-art natural language processing and clustering approaches to achieve meaningful log grouping. The approach is evaluated in an industrial setting in both a qualitative and quantitative manner. Results show that LogGrouper produces good quality groupings in terms of our two evaluation metrics (Silhouette Coefficient and Calinski-Harabasz Index) for clustering quality. The qualitative evaluation shows that experts perceive the groups as useful, and the groups are seen as an initial pointer for root cause analysis and failure assignment.

There is a growing body of knowledge on network intrusion detection, and several open data sets with network traffic and cyber-security threats have been released in the past decades. However, many data sets have aged, were not collected in a contemporary industrial communication system, or do not easily support research focusing on distributed anomaly detection. This paper presents the Westermo network traffic data set, 1.8 million network packets recorded in over 90 minutes in a network built up of twelve hardware devices. In addition to the raw data in PCAP format, the data set also contains pre-processed data in the form of network flows in CSV files. This data set can support the research community for topics such as intrusion detection, anomaly detection, misconfiguration detection, distributed or federated artificial intelligence, and attack classification. In particular, we aim to use the data set to continue work on resource-constrained distributed artificial intelligence in edge devices. The data set contains six types of events: harmless SSH, bad SSH, misconfigured IP address, duplicated IP address, port scan, and man in the middle attack.

Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case-based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments in a simulated performance testing setup, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process and performs adaptively without access to source code and performance models. © 2021, The Author(s).

Differential evolution (DE) is widely used for global optimisation problems due to its simplicity and efficiency. L-SHADE is a state-of-the-art variant of DE algorithm that incorporates external archive, success-history-based parameter adaptation, and linear population size reduction. L-SHADE uses a current-to-pbest/1/bin strategy for mutation operator, while all individuals have the same probability to be selected. In this paper, we propose a novel L-SHADE algorithm, RWS-L-SHADE, based on a roulette wheel selection strategy so that better individuals have a higher priority and worse individuals are less likely to be selected. Our extensive experiments on the CEC-2017 benchmark functions and dimensionalities of 30, 50 and 100 indicate that RWS-L-SHADE outperforms L-SHADE.

Clustering is one of the prominent approaches for image segmentation. Conventional algorithms such as k-means, while extensively used for image segmentation, suffer from problems such as sensitivity to initialisation and getting stuck in local optima. To overcome these, population-based metaheuristic algorithms can be employed. This paper proposes a novel clustering algorithm for image segmentation based on the human mental search (HMS) algorithm, a powerful population-based algorithm to tackle optimisation problems. One of the advantages of our proposed algorithm is that it does not require any information about the number of clusters. To verify the effectiveness of our proposed algorithm, we present a set of experiments based on objective function evaluation and image segmentation criteria to show that our proposed algorithm outperforms existing approaches.