Change search
Refine search result
12 1 - 50 of 77
CiteExportLink to result list
Permanent link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Rows per page
  • 5
  • 10
  • 20
  • 50
  • 100
  • 250
Sort
  • Standard (Relevance)
  • Author A-Ö
  • Author Ö-A
  • Title A-Ö
  • Title Ö-A
  • Publication type A-Ö
  • Publication type Ö-A
  • Issued (Oldest first)
  • Issued (Newest first)
  • Created (Oldest first)
  • Created (Newest first)
  • Last updated (Oldest first)
  • Last updated (Newest first)
  • Disputation date (earliest first)
  • Disputation date (latest first)
  • Standard (Relevance)
  • Author A-Ö
  • Author Ö-A
  • Title A-Ö
  • Title Ö-A
  • Publication type A-Ö
  • Publication type Ö-A
  • Issued (Oldest first)
  • Issued (Newest first)
  • Created (Oldest first)
  • Created (Newest first)
  • Last updated (Oldest first)
  • Last updated (Newest first)
  • Disputation date (earliest first)
  • Disputation date (latest first)
Select
The maximal number of hits you can export is 250. When you want to export more records please use the Create feeds function.
  • 1.
    Bjarnason, Elizabeth
    et al.
    Lund University, Sweden.
    Borg, Markus
    RISE, Swedish ICT, SICS, Security Lab.
    Aligning Requirements and Testing - Working Together Toward the Same Goal2017In: IEEE Software, ISSN 0740-7459, E-ISSN 1937-4194, Vol. 34, no 1, p. 20-23, article id 7819382Article in journal (Other academic)
    Abstract [en]

    The proper alignment of requirements engineering and testing (RET) can be key to software's success. Three practices can provide effective RET alignment: using test cases as requirements, harvesting trace links, and reducing distances between requirements engineers and testers. The Web extra https://youtu.be/M65ZKxfxqME is an audio podcast of author Elizabeth Bjarnason reading the the Requirements column she cowrote with Markus Borg.

    Download full text (pdf)
    fulltext
  • 2.
    Bjarnason, Elizabeth
    et al.
    Lund University, Sweden.
    Borg, Markus
    RISE, Swedish ICT, SICS, Security Lab. Lund University, Sweden.
    Lindvall, Bertil
    Lund University, Sweden.
    Supervising for Independence – A Case Study of Master Science Projects in Higher Education2015In: LU:s femte högskolepedagogiska utvecklingskonferens, 2015, 12Conference paper (Refereed)
    Abstract [en]

    Students completing a Swedish Master's degree in engineering should have knowledge and skills to independently solve engineering issues. This autonomy should be developed and demonstrated within the M.Sc. project course. But, how can supervisors encourage independence? We have explored this in a case study through semi-structured interviews with students, supervisors and examiners of two M.Sc. projects. We investigated their view of independence, and how supervision correlates to independence. The results identify areas relevant to independence, namely supervision roles and relationships, student characteristics, M.Sc. process, and view on independence. The results confirm previous findings that students' knowledge of and motivation for the topic support independence. The supervisor's role is to guide and support through frequent peer-level discussions and to act as a discussion partner, while the student should have the main responsibility for the project. We conclude that it is important for supervisors to encourage students to take ownership of their M.Sc. projects and to design their own solutions, while providing the overall process and timelines.

    Download full text (pdf)
    FULLTEXT01
  • 3. Bjarnason, Elizabeth
    et al.
    Borg, Markus
    RISE, Swedish ICT, SICS, Security Lab.
    Lindvall, Bertil
    Supervising Towards Independence2016Other (Other academic)
    Abstract [en]

    Supervising a student can be compared to teaching someone to drive a car. The student is in the driver's seat while the supervisor provides structure and guidance, and can intervene in risky and unsafe situations. It is a learning process in which the student gradually gains experience and sufficient skill to obtain a driving license, and to drive without an instructor. Similarly, a student attending the MSc project course at the technical faculty of Lund University is to "develop and demonstrate knowledge and ability required to autonomously work as an engineer" (from MSc course plan). But what factors affect a MSc project, and how can we as supervisors support students in their learning process towards independence? We performed a case study of two completed MSc projects where we interviewed the student, the supervisor and the examiner for each case. In this article we present the main conclusions drawn from the cross-case analysis of this study. Details on the studied cases and the results on which these conclusions are based can be found in our previous publication of this study.

  • 4.
    Bjarnason, Elizabeth
    et al.
    Lund University, Sweden.
    Unterkalmsteiner, Michael
    Blekinge Institute of Technology, Sweden.
    Borg, Markus
    RISE, Swedish ICT, SICS, Security Lab.
    Engström, Emelie
    Lund University, Sweden.
    A Multi-Case Study of Agile Requirements Engineering and the Use of Test Cases as Requirements2016In: Information and Software Technology, ISSN 0950-5849, E-ISSN 1873-6025, Vol. 77, p. 61-79Article in journal (Refereed)
    Abstract [en]

    [Context] It is an enigma that agile projects can succeed "without requirements" when weak requirements engineering is a known cause for project failures. While agile development projects often manage well without extensive requirements test cases are commonly viewed as requirements and detailed requirements are documented as test cases. [Objective] We have investigated this agile practice of using test cases as requirements to understand how test cases can support the main requirements activities, and how this practice varies. [Method] We performed an iterative case study at three companies and collected data through 14 interviews and 2 focus groups. [Results] The use of test cases as requirements poses both benefits and challenges when eliciting, validating, verifying, and managing requirements, and when used as a documented agreement. We have identified five variants of the test-cases-as-requirements practice, namely de facto, behaviour-driven, story-test driven, stand-alone strict and stand-alone manual for which the application of the practice varies concerning the time frame of requirements documentation, the requirements format, the extent to which the test cases are a machine executable specification and the use of tools which provide specific support for the practice of using test cases as requirements. [Conclusions] The findings provide empirical insight into how agile development projects manage and communicate requirements. The identified variants of the practice of using test cases as requirements can be used to perform in-depth investigations into agile requirements engineering. Practitioners can use the provided recommendations as a guide in designing and improving their agile requirements practices based on project characteristics such as number of stakeholders and rate of change.

    Download full text (pdf)
    FULLTEXT01
  • 5.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems. Lund University, sweden.
    Agility in Software 2.0 – Notebook Interfaces and MLOps with Buttresses and Rebars2022In: International Conference on Lean and Agile Software DevelopmentLASD 2022: Lean and Agile Software Development pp 3-16, Springer Science and Business Media Deutschland GmbH , 2022, p. 3-16Conference paper (Refereed)
    Abstract [en]

    Artificial intelligence through machine learning is increasingly used in the digital society. Solutions based on machine learning bring both great opportunities, thus coined “Software 2.0,” but also great challenges for the engineering community to tackle. Due to the experimental approach used by data scientists when developing machine learning models, agility is an essential characteristic. In this keynote address, we discuss two contemporary development phenomena that are fundamental in machine learning development, i.e., notebook interfaces and MLOps. First, we present a solution that can remedy some of the intrinsic weaknesses of working in notebooks by supporting easy transitions to integrated development environments. Second, we propose reinforced engineering of AI systems by introducing metaphorical buttresses and rebars in the MLOps context. Machine learning-based solutions are dynamic in nature, and we argue that reinforced continuous engineering is required to quality assure the trustworthy AI systems of tomorrow.

  • 6.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems. Lund University, Sweden.
    The AIQ Meta-Testbed: Pragmatically Bridging Academic AI Testing and Industrial Q Needs2021In: Lecture Notes in Business Information Processing, ISSN 1865-1348, E-ISSN 1865-1356, Vol. 404, p. 66-77Article in journal (Refereed)
    Abstract [en]

    AI solutions seem to appear in any and all application domains. As AI becomes more pervasive, the importance of quality assurance increases. Unfortunately, there is no consensus on what artificial intelligence means and interpretations range from simple statistical analysis to sentient humanoid robots. On top of that, quality is a notoriously hard concept to pinpoint. What does this mean for AI quality? In this paper, we share our working definition and a pragmatic approach to address the corresponding quality assurance with a focus on testing. Finally, we present our ongoing work on establishing the AIQ Meta-Testbed. 

  • 7.
    Borg, Markus
    RISE, Swedish ICT, SICS, Security Lab. Lund University, Sweden.
    TuneR: A Framework for Tuning Software Engineering Tools with Hands-on Instructions in R2016In: Journal of Software: Evolution and Process, ISSN 2047-7473, E-ISSN 2047-7481, Vol. 28, no 6, p. 427-459Article in journal (Refereed)
    Abstract [en]

    Numerous tools automating various aspects of software engineering have been developed, and many of the tools are highly configurable through parameters. Understanding the parameters of advanced tools often requires deep understanding of complex algorithms. Unfortunately, suboptimal parameter settings limit the performance of tools and hinder industrial adaptation, but still few studies address the challenge of tuning software engineering tools. We present TuneR, an experiment framework that supports finding feasible parameter settings using empirical methods. The framework is accompanied by practical guidelines of how to use R to analyze the experimental outcome. As a proof-of-concept, we apply TuneR to tune ImpRec, a recommendation system for change impact analysis in a software system that has evolved for more than two decades. Compared with the output from the default setting, we report a 20.9% improvement in the response variable reflecting recommendation accuracy. Moreover, TuneR reveals insights into the interaction among parameters, as well as nonlinear effects. TuneR is easy to use, thus the framework has potential to support tuning of software engineering tools in both academia and industry.

    Download full text (pdf)
    FULLTEXT01
  • 8.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Using Search-Based Software Testing to Guide the Strive for Robust Machine Learning Components: Lessons Learned Across Systems and Simulators in the Mobility Domain2022In: Proceedings - 2022 IEEE 14th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2022, Institute of Electrical and Electronics Engineers Inc. , 2022Conference paper (Refereed)
  • 9.
    Borg, Markus
    et al.
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Abdessalem, Raja
    University of Luxembourg, Luxembourg.
    Nejati, Shiva
    University of Luxembourg, Luxembourg; University of Ottawa, Canada.
    Jegeden, Francois
    ESI Group, France.
    Shin, Donghwan
    University of Luxembourg, Luxembourg.
    Digital Twins Are Not Monozygotic – Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators2021In: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), 2021, p. 383-393Conference paper (Refereed)
    Abstract [en]

    The increasing levels of software- and data-intensive driving automation call for an evolution of automotive soft-ware testing. As a recommended practice of the Verification and Validation (V&V) process of ISO/PAS 21448, a candidate standard for safety of the intended functionality for road vehicles, simulation-based testing has the potential to reduce both risks and costs. There is a growing body of research on devising test automation techniques using simulators for Advanced Driver-Assistance Systems (ADAS). However, how similar are the results if the same test scenarios are executed in different simulators? We conduct a replication study of applying a Search-Based Software Testing (SBST) solution to a real-world ADAS (PeVi, a pedestrian vision detection system) using two different commercial simulators, namely, TASS/Siemens PreScan and ESI Pro-SiVIC. Based on a minimalistic scene, we compare critical test scenarios generated using our SBST solution in these two simulators. We show that SBST can be used to effectively generate critical test scenarios in both simulators, and the test results obtained from the two simulators can reveal several weaknesses of the ADAS under test. However, executing the same test scenarios in the two simulators leads to notable differences in the details of the test outputs, in particular, related to (1) safety violations revealed by tests, and (2) dynamics of cars and pedestrians. Based on our findings, we recommend future V&V plans to include multiple simulators to support robust simulation-based testing and to base test objectives on measures that are less dependant on the internals of the simulators.

  • 10.
    Borg, Markus
    et al.
    RISE, Swedish ICT, SICS, Security Lab.
    Alégroth, Emil
    Blekinge Institute of Technology, Sweden.
    Runeson, Per
    Lund University, Sweden.
    Software Engineers' Information Seeking Behavior in Change Impact Analysis: An Interview Study2017In: ICPC '17 Proceedings of the 25th International Conference on Program Comprehension, IEEE Press, 2017, p. 12-22Conference paper (Refereed)
    Abstract [en]

    Software engineers working in large projects must navigate complex information landscapes. Change Impact Analysis (CIA) is a task that relies on engineers' successful information seeking in databases storing, e.g., source code, requirements, design descriptions, and test case specifications. Several previous approaches to support information seeking are task-specific, thus understanding engineers' seeking behavior in specific tasks is fundamental. We present an industrial case study on how engineers seek information in CIA, with a particular focus on traceability and development artifacts that are not source code. We show that engineers have different information seeking behavior, and that some do not consider traceability particularly useful when conducting CIA. Furthermore, we observe a tendency for engineers to prefer less rigid types of support rather than formal approaches, i.e., engineers value support that allows flexibility in how to practically conduct CIA. Finally, due to diverse information seeking behavior, we argue that future CIA support should embrace individual preferences to identify change impact by empowering several seeking alternatives, including searching, browsing, and tracing.

    Download full text (pdf)
    fulltext
  • 11.
    Borg, Markus
    et al.
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Bengtsson, J.
    Lund University, Sweden.
    Osterling, H.
    Lund University, Sweden.
    Hagelborn, A.
    NordAxon AB, Sweden.
    Gagner, Isabella
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Tomaszewski, Piotr
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Quality Assurance of Generative Dialog Models in an Evolving Conversational Agent Used for Swedish Language Practice2022In: Proceedings - 1st International Conference on AI Engineering - Software Engineering for AI, CAIN 2022, Institute of Electrical and Electronics Engineers Inc. , 2022, p. 22-32Conference paper (Refereed)
    Abstract [en]

    Due to the migration megatrend, efficient and effective second-language acquisition is vital. One proposed solution involves AI-enabled conversational agents for person-centered interactive language practice. We present results from ongoing action research targeting quality assurance of proprietary generative dialog models trained for virtual job interviews. The action team elicited a set of 38 requirements for which we designed corresponding automated test cases for 15 of particular interest to the evolving solution. Our results show that six of the test case designs can detect meaningful differences between candidate models. While quality assurance of natural language processing applications is complex, we provide initial steps toward an automated framework for machine learning model selection in the context of an evolving conversational agent. Future work will focus on model selection in an MLOps setting.

  • 12.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden, ICT, SICS.
    Bjarnason, Elizabeth
    Lund University, Sweden.
    Unterkalmsteiner, Michael
    Blekinge Institute of Technology, Sweden.
    Yu, Tingting
    University of Kentucky, USA.
    Gay, Gregory
    University of South Carolina, USA.
    Felderer, Michael
    University of Innsbruck, Austria.
    Summary of the 4th International Workshop on Requirements Engineering and Testing (RET 2017)2018In: Software Engineering Notes: an Informal Newsletter of The Specia, ISSN 0163-5948, E-ISSN 1943-5843, Vol. 42, no 4, p. 28-31Article in journal (Other academic)
  • 13.
    Borg, Markus
    et al.
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Bronson, Joshua
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Christensson, Linus
    Lund University, Sweden.
    Olsson, Fredrik
    Lund University, Sweden.
    Lennartsson, Olof
    Infotiv AB, Sweden.
    Sonnsjö, Elias
    Infotiv AB, Sweden.
    Ebabi, Hamid
    Infotiv AB, Sweden.
    Karsberg, Martin
    Infotiv AB, Sweden.
    Exploring the Assessment List for Trustworthy AI in the Context of Advanced Driver-Assistance Systems2021In: 2021 IEEE/ACM 2nd International Workshop on Ethics in Software Engineering Research and Practice (SEthics), 2021, p. 5-12Conference paper (Refereed)
    Abstract [en]

    Artificial Intelligence (AI) is increasingly used in critical applications. Thus, the need for dependable AI systems is rapidly growing. In 2018, the European Commission appointed experts to a High-Level Expert Group on AI (AI-HLEG). AI- HLEG defined Trustworthy AI as 1) lawful, 2) ethical, and 3) robust and specified seven corresponding key requirements. To help development organizations, AI-HLEG recently published the Assessment List for Trustworthy AI (ALTAI). We present an illustrative case study from applying ALTAI to an ongoing development project of an Advanced Driver-Assistance System (ADAS) that relies on Machine Learning (ML). Our experience shows that ALTAI is largely applicable to ADAS development, but specific parts related to human agency and transparency can be disregarded. Moreover, bigger questions related to societal and environmental impact cannot be tackled by an ADAS supplier in isolation. We present how we plan to develop the ADAS to ensure ALTAI-compliance. Finally, we provide three recommendations for the next revision of ALTAI, i.e., life-cycle variants, domainspecific adaptations, and removed redundancy.

  • 14.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden, ICT, SICS.
    Brytting, Andreas
    KTH Royal Institute of Technology, Sweden.
    Hansson, Daniel
    Verifyter AB, Sweden.
    An Analytical View ofTest Results Using CityScapes2018Conference paper (Other academic)
    Abstract [en]

    t— In this paper we map test results from a real ASIC project on to the file structure of the design under test andpresent it as a cityscape. In the cityscape each house is a file where its height reflects the number of commits to that file. Thecolor reflects the fraction of bad commits.We identify error prone areas (red "bad" neighborhoods) as well as the most active areas (tall "downtown" areas). Thecityscape also allows us to identify potential test coverage holes (tall green buildings) where there are a lot of activities but nofailures.

  • 15.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden, ICT, SICS.
    Brytting, Andreas
    KTH Royal Institute of Technology, Sweden.
    Hansson, Daniel
    Verifyter AB, Sweden.
    Enabling Visual Design Verification Analytics– From Prototype Visualizations to anAnalytics Tool using the Unity Game Engine2018Conference paper (Other academic)
    Abstract [en]

    The ever-increasing architectural complexity in contemporary ASIC projects turns Design Verification (DV)into a highly advanced endeavor. Pressing needs for short time-to-market has made automation a key solution in DV.However, recurring execution of large regression suites inevitably leads to challenging amounts of test results. Following thedesign science paradigm, we present an action research study to introduce visual analytics in a commercial ASIC project. Wedevelop a cityscape visualization tool using the game engine Unity. Initial evaluations are promising, suggesting that the tooloffers a novel approach to identify error-prone parts of the design, as well as coverage holes.

  • 16.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden, ICT, SICS.
    Chatzipetrou, Panagiota
    Blekinge Institute of Technology, Sweden; Örebro University, Sweden.
    Wnuk, Krzysztof
    Blekinge Institute of Technology, Sweden.
    Alégroth, Emil
    Blekinge Institute of Technology, Sweden.
    Gorschek, Tony
    Blekinge Institute of Technology, Sweden.
    Papatheocharous, Efi
    RISE - Research Institutes of Sweden, ICT, SICS.
    Shah, Syed
    iZettle, Sweden.
    Axelsson, Jakob
    RISE - Research Institutes of Sweden, ICT, SICS.
    Selecting component sourcing options: A survey of software engineering's broader make-or-buy decisions2019In: Information and Software Technology, ISSN 0950-5849, E-ISSN 1873-6025, Vol. 112, p. 18-34Article in journal (Refereed)
    Abstract [en]

    Context: Component-based software engineering (CBSE) is a common approach to develop and evolve contemporary software systems. When evolving a system based on components, make-or-buy decisions are frequent, i.e., whether to develop components internally or to acquire them from external sources. In CBSE, several different sourcing options are available: (1) developing software in-house, (2) outsourcing development, (3) buying commercial-off-the-shelf software, and (4) integrating open source software components. Objective: Unfortunately, there is little available research on how organizations select component sourcing options (CSO) in industry practice. In this work, we seek to contribute empirical evidence to CSO selection. Method: We conduct a cross-domain survey on CSO selection in industry, implemented as an online questionnaire. Results: Based on 188 responses, we find that most organizations consider multiple CSOs during software evolution, and that the CSO decisions in industry are dominated by expert judgment. When choosing between candidate components, functional suitability acts as an initial filter, then reliability is the most important quality. Conclusion: We stress that future solution-oriented work on decision support has to account for the dominance of expert judgment in industry. Moreover, we identify considerable variation in CSO decision processes in industry. Finally, we encourage software development organizations to reflect on their decision processes when choosing whether to make or buy components, and we recommend using our survey for a first benchmarking.

  • 17.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden, ICT, SICS.
    Chatzipetrou, Panagiota
    Blekinge Institute of Technology, Sweden; Örebro University, Sweden.
    Wnuk, Krzysztof
    Blekinge Institute of Technology, Sweden.
    Alégroth, Emil
    Blekinge Institute of Technology, Sweden.
    Gorschek, Tony
    Blekinge Institute of Technology, Sweden.
    Papatheocharous, Efi
    RISE - Research Institutes of Sweden, ICT, SICS.
    Shah, Syed
    iZettle, Sweden.
    Axelsson, Jakob
    RISE - Research Institutes of Sweden, ICT, SICS.
    Selecting Software Component Sourcing Options: Detailed Survey Description and Analysis2018Report (Other academic)
    Abstract [en]

    Component-based software engineering (CBSE) is a common approach to develop and evolve contemporary software systems. When evolving a system based on components, make-or-buy decisions are frequent, i.e., whether to develop components internally or to acquire them fromexternal sources. In CBSE, several different sourcing options are available: 1) developing software in-house, 2) outsourcing development, 3) buying commercial-off-the-shelf software, and 4) integrating open source software components. Unfortunately, there is little available research on howorganizations select component sourcing options (CSO) in industry practice. In this work, we seek to contribute empirical evidence to CSO selection. Method: We conduct a cross-domain survey on CSO selection in industry, implemented as an online questionnaire. Based on 188 responses, we find that most organizations consider multiple CSOs during software evolution, and that the CSO decisions in industry are dominated by expert judgment. When choosing between candidate components, functional suitability acts as an initial filter, then reliability is the most important quality. We stress that future solution-oriented work on decision support has to account for the dominance of expert judgment in industry. Moreover, we identify considerable variation in CSO decision processes in industry. Finally, we encourage software development organizations to reflect on their decision processes when choosing whether to make or buy components, and we recommend using our survey for a first benchmarking.

    Download full text (pdf)
    fulltext
  • 18.
    Borg, Markus
    et al.
    RISE, Swedish ICT, SICS, Security Lab.
    de la Vara, José Luis
    Carlos III University of Madrid, Spain.
    Wnuk, Krzysztof
    Blekinge Institute of Technology, Sweden.
    Practitioners' Perspectives on Change Impact Analysis for Safety-Critical Software - A Preliminary Analysis2016In: Computer Safety, Reliability, and Security: SAFECOMP 2016 Workshops / [ed] Amund Skavhaug, Jérémie Guiochet, Erwin Schoitsch, Friedemann Bitsch, 2016, 11, Vol. 9923, p. 346-358Conference paper (Refereed)
    Abstract [en]

    Safety standards prescribe change impact analysis (CIA) during evolution of safety-critical software systems. Although CIA is a fundamental activity, there is a lack of empirical studies about how it is performed in practice. We present a case study on CIA in the context of an evolving automation system, based on 14 interviews in Sweden and India. Our analysis suggests that engineers on average spend 50-100 hours on CIA per year, but the effort varies considerably with the phases of projects. Also, the respondents presented different connotations to CIA and perceived the importance of CIA differently. We report the most pressing CIA challenges, and several ideas on how to support future CIA. However, we show that measuring the effect of such improvement solutions is non-trivial, as CIA is intertwined with other development activities. While this paper only reports preliminary results, our work contributes empirical insights into practical CIA.

    Download full text (pdf)
    FULLTEXT01
  • 19.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden, ICT, SICS.
    Englund, Cristofer
    RISE - Research Institutes of Sweden, ICT, Viktoria.
    Duran, Boris
    RISE - Research Institutes of Sweden, ICT, Viktoria.
    Traceability and Deep Learning: Safety-critical Systems with Traces Ending in Deep Neural Networks2017Conference paper (Other academic)
  • 20. Borg, Markus
    et al.
    Englund, Cristofer
    RISE - Research Institutes of Sweden, ICT, Viktoria.
    Wnuk, Krzysztof
    Blekinge Institute of Technology, Sweden.
    Duran, Boris
    RISE - Research Institutes of Sweden, ICT, Viktoria.
    Levandowski, Christoffer
    QRTECH AB, Sweden.
    Gao, Shenjian
    Blekinge Institute of Technology, Sweden.
    Tan, Yanwen
    Blekinge Institute of Technology, Sweden.
    Kaijser, Henrik
    AB Volvo, Sweden.
    Lönn, Henrik
    AB Volvo, Sweden.
    Törnqvist, Jonas
    QRTECH AB, Sweden.
    Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry2019In: Journal of Automotive Software Engineering, Vol. 1, no 1, p. 1-13Article in journal (Refereed)
    Abstract [en]

    Deep neural networks (DNNs) will emerge as a cornerstone in automotive software engineering. However, developing systems with DNNs introduces novel challenges for safety assessments. This paper reviews the state-of-the-art in verification and validation of safety-critical systems that rely on machine learning. Furthermore, we report from a workshop series on DNNs for perception with automotive experts in Sweden, confirming that ISO 26262 largely contravenes the nature of DNNs. We recommend aerospace-to-automotive knowledge transfer and systems-based safety approaches, for example, safety cage architectures and simulated system test cases.

  • 21.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Garousi, Vahid
    Wageningen University, Netherlands.
    Mahmoud, Anas
    Louisiana State University, US.
    Olsson, Thomas
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Stalberg, Oskar
    Plausible Concept, Sweden.
    Video Game Development in a Rush: A Survey of the Global Game Jam Participants2020In: IEEE Transactions on Games, ISSN 2475-1502, Vol. 12, no 3, p. 246-259Article in journal (Refereed)
    Abstract [en]

    Video game development is a complex endeavor, often involving complex software, large organizations, and aggressive release deadlines. Several studies have reported that periods of “crunch time” are prevalent in the video game industry, but there are few studies on the effects of time pressure. We conducted a survey with participants of the Global Game Jam (GGJ), a 48-hour hackathon. Based on 198 responses, the results suggest that: (1) iterative brainstorming is the most popular method for conceptualizing initial requirements; (2) continuous integration, minimum viable product, scope management, version control, and stand-up meetings are frequently applied development practices; (3) regular communication, internal playtesting, and dynamic and proactive planning are the most common quality assurance activities; and (4) familiarity with agile development has a weak correlation with perception of success in GGJ. We conclude that GGJ teams rely on ad hoc approaches to development and face-to-face communication, and recommend some complementary practices with limited overhead. Furthermore, as our findings are similar to recommendations for software startups, we posit that game jams and the startup scene share contextual similarities. Finally, we discuss the drawbacks of systemic “crunch time” and argue that game jam organizers are in a good position to problematize the phenomenon.

  • 22.
    Borg, Markus
    et al.
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Groen, Eduard
    Preface: REFSQ 2020 posters and tools track2020In: CEUR Workshop Proceedings, ISSN 1613-0073, E-ISSN 1613-0073, Vol. 2584Article in journal (Other academic)
  • 23.
    Borg, Markus
    et al.
    RISE Research Institutes of Sweden. Lund University, Sweden.
    Henriksson, Jens
    Semcon AB, Sweden.
    Socha, Kasper
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems. Lund University, Sweden.
    Lennartsson, Olof
    Infotiv AB, Sweden.
    Sonnsjö Lönegren, Elias
    Infotiv AB, Sweden.
    Bui, Thanh
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Tomaszewski, Piotr
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Sathyamoorthy, S R
    QRTECH AB, Sweden.
    Brink, Sebastian
    Combitech AB, Sweden.
    Helali Moghadam, Mahshid
    RISE Research Institutes of Sweden.
    Ergo, SMIRK is safe: a safety case for a machine learning component in a pedestrian automatic emergency brake system2023In: Software quality journal, ISSN 0963-9314, E-ISSN 1573-1367Article in journal (Refereed)
    Abstract [en]

    Integration of machine learning (ML) components in critical applications introduces novel challenges for software certification and verification. New safety standards and technical guidelines are under development to support the safety of ML-based systems, e.g., ISO 21448 SOTIF for the automotive domain and the Assurance of Machine Learning for use in Autonomous Systems (AMLAS) framework. SOTIF and AMLAS provide high-level guidance but the details must be chiseled out for each specific case. We initiated a research project with the goal to demonstrate a complete safety case for an ML component in an open automotive system. This paper reports results from an industry-academia collaboration on safety assurance of SMIRK, an ML-based pedestrian automatic emergency braking demonstrator running in an industry-grade simulator. We demonstrate an application of AMLAS on SMIRK for a minimalistic operational design domain, i.e., we share a complete safety case for its integrated ML-based component. Finally, we report lessons learned and provide both SMIRK and the safety case under an open-source license for the research community to reuse. © 2023, The Author(s).

  • 24.
    Borg, Markus
    et al.
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems. Lund University, Sweden.
    Jabangwe, Ronald
    RISE Research Institutes of Sweden.
    Aberg, Simon
    Lund University, Sweden.
    Ekblom, Arvid
    Lund University, Sweden.
    Hedlund, Ludvig
    Lund University, Sweden.
    Lidfeldt, August
    Lund University, Sweden.
    Test automation with grad-CAM Heatmaps - A future pipe segment in MLOps for Vision AI?2021In: Proceedings - 2021 IEEE 14th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2021, Institute of Electrical and Electronics Engineers Inc. , 2021, p. 175-181Conference paper (Refereed)
    Abstract [en]

    Machine Learning (ML) is a fundamental part of modern perception systems. In the last decade, the performance of computer vision using trained deep neural networks has outperformed previous approaches based on careful feature engineering. However, the opaqueness of large ML models is a substantial impediment for critical applications such as in the automotive context. As a remedy, Gradient-weighted Class Activation Mapping (Grad-CAM) has been proposed to provide visual explanations of model internals. In this paper, we demonstrate how Grad-CAM heatmaps can be used to increase the explainability of an image recognition model trained for a pedestrian underpass. We argue how the heatmaps support compliance to the EU's seven key requirements for Trustworthy AI. Finally, we propose adding automated heatmap analysis as a pipe segment in an MLOps pipeline. We believe that such a building block can be used to automatically detect if a trained ML-model is activated based on invalid pixels in test images, suggesting biased models.

  • 25.
    Borg, Markus
    et al.
    RISE, Swedish ICT, SICS, Security Lab.
    Lennerstad, Iben
    Lund University, Sweden.
    Ros, Rasmus
    Lund University, Sweden.
    Bjarnason, Elizabeth
    Lund University, Sweden.
    On using active learning and self-training when mining performance discussions on stack overflow2017In: EASE'17 Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering. ACM International Conference Proceeding Series, 2017, p. 308-313Conference paper (Refereed)
    Abstract [en]

    Abundant data is the key to successful machine learning. However, supervised learning requires annotated data that are often hard to obtain. In a classification task with limited resources, Active Learning (AL) promises to guide annotators to examples that bring the most value for a classifier. AL can be successfully combined with self-training, i.e., extending a training set with the unlabelled examples for which a classifier is the most certain. We report our experiences on using AL in a systematic manner to train an SVM classifier for Stack Overflow posts discussing performance of software components. We show that the training examples deemed as the most valuable to the classifier are also the most difficult for humans to annotate. Despite carefully evolved annotation criteria, we report low inter-rater agreement, but we also propose mitigation strategies. Finally, based on one annotator's work, we show that self-training can improve the classification accuracy. We conclude the paper by discussing implication for future text miners aspiring to use AL and self-training.

  • 26.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden, ICT, SICS.
    Olsson, Thomas
    RISE - Research Institutes of Sweden, ICT, SICS.
    Franke, Ulrik
    RISE - Research Institutes of Sweden, ICT, SICS.
    Assar, Saïd
    IMT Business School, France.
    Digitalization of Swedish Government Agencies: A Perspective Through the Lens of a Software Development Census2018In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Society, 2018, p. 37-46Conference paper (Refereed)
    Abstract [en]

    Software engineering is at the core of the digitalization of society. Ill-informed decisions can have major consequences, as made evident in the 2017 government crisis in Sweden, originating in a data breach caused by an outsourcing deal made by the Swedish Transport Agency. Many Government Agencies (GovAgs) in Sweden are rapidly undergoing a digital transition, thus it is important to overview how widespread, and mature, software development is in this part of the public sector. We present a software development census of Swedish GovAgs, complemented by document analysis and a survey. We show that 39.2% of the GovAgs develop software internally, some matching the number of developers in large companies. Our findings suggest that the development largely resembles private sector counterparts, and that established best practices are implemented. Still, we identify improvement potential in the areas of strategic sourcing, openness, collaboration across GovAgs, and quality requirements. The Swedish Government has announced the establishment of a new digitalization agency next year, and our hope is that the software engineering community will contribute its expertise with a clear voice.

    Download full text (pdf)
    fulltext
  • 27.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden, ICT, SICS.
    Olsson, Thomas
    RISE - Research Institutes of Sweden, ICT, SICS.
    Franke, Ulrik
    RISE - Research Institutes of Sweden, ICT, SICS.
    Assar, Saïd
    IMT Business School, Sweden.
    Digitalization of Swedish Government Agencies: Detailed Census Description and Analysis2018Report (Other academic)
    Abstract [en]

    Software engineering is at the core of the digitalization of society. Ill-informed decisions can have major consequences, as made evident in the 2017 government crisis in Sweden, originating in a data breach caused by an outsourcing deal made by the Swedish Transport Agency. Many Government Agencies (GovAgs) in Sweden are rapidly undergoing a digital transition, thus it is important to overview how widespread, and mature, software development is in this part of the public sector. We present a software development census of Swedish GovAgs, complemented by document analysis and a survey. We show that 39.2% of the GovAgs develop software internally, some matching the number of developers in large companies. Our findings suggest that the development largely resembles private sector counterparts, and that established best practices are implemented. Still, we identify improvement potential in the areas of strategic sourcing, openness, collaboration across GovAgs, and quality requirements. The Swedish Government has announced the establishment of a new digitalization agency next year, and our hope is that the software engineering community will contribute its expertise with a clear voice.

    Download full text (pdf)
    fulltext
  • 28.
    Borg, Markus
    et al.
    RISE, Swedish ICT, SICS, Security Lab.
    Olsson, Thomas
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Svensson, John
    Boliden, Sweden.
    From LiDAR to Underground Maps via 5G - Business Models Enabling a System-of-Systems Approach to Mapping the Kankberg Mine2017Report (Other academic)
    Abstract [en]

    With ever-increasing productivity targets in mining operations, there is a growing interest in mining automation. The PIMM project addresses the fundamental challenge of network communication by constructing a pilot 5G network in the underground mine Kankberg. In this report, we discuss how such a 5G network could constitute the essential infrastructure to organize existing systems in Kankberg into a system-of-systems (SoS). In this report, we analyze a scenario in which LiDAR equipped vehicles operating in the mine are connected to existing mine mapping and positioning solutions. The approach is motivated by the approaching era of remote controlled, or even autonomous, vehicles in mining operations. The proposed SoS could ensure continuously updated maps of Kankberg, rendered in unprecedented detail, supporting both productivity and safety in the underground mine. We present four different SoS solutions from an organizational point of view, discussing how development and operations of the constituent systems could be distributed among Boliden and external stakeholders, e.g., the vehicle suppliers, the hauling company, and the developers of the mapping software. The four scenarios are compared from both technical and business perspectives, and based on trade-off discussions and SWOT analyses. We conclude our report by recommending continued research along two future paths, namely a closer cooperation with the vehicle suppliers, and further feasibility studies regarding establishing a Kankberg software ecosystem.

    Download full text (pdf)
    fulltext
  • 29.
    Borg, Markus
    et al.
    RISE, Swedish ICT, SICS, Security Lab.
    Olsson, Thomas
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Svensson, John
    Boliden, Sweden.
    Piggybacking on an Autonomous Hauler: Business Models Enabling a System-of-Systems Approach to Mapping an Underground Mine2017Conference paper (Refereed)
    Abstract [en]

    With ever-increasing productivity targets in mining operations, there is a growing interest in mining automation. In future mines, remote-controlled and autonomous haulers will operate underground guided by LiDAR sensors. We envision reusing LiDAR measurements to maintain accurate mine maps that would contribute to both safety and productivity. Extrapolating from a pilot project on reliable wireless communication in Boliden's Kankberg mine, we propose establishing a system-of-systems (SoS) with LIDAR-equipped haulers and existing mapping solutions as constituent systems. SoS requirements engineering inevitably adds a political layer, as independent actors are stakeholders both on the system and SoS levels. We present four SoS scenarios representing different business models, discussing how development and operations could be distributed among Boliden and external stakeholders, e.g., the vehicle suppliers, the hauling company, and the developers of the mapping software. Based on eight key variation points, we compare the four scenarios from both technical and business perspectives. Finally, we validate our findings in a seminar with participants from the relevant stakeholders. We conclude that to determine which scenario is the most promising for Boliden, trade-offs regarding control, costs, risks, and innovation must be carefully evaluated. 

  • 30.
    Borg, Markus
    et al.
    RISE, Swedish ICT, SICS, Security Lab.
    Petter, Gulin
    Lund University, Sweden.
    Linus, Olofsson
    Lund University, Sweden.
    Do Take it Personal: It's Not What You Say, It's Who (and Where) You Are!2016In: Tiny Transactions on Computer Science, Vol. 4Article in journal (Refereed)
    Abstract [en]

    Issue management in market-driven software projects is constantly under time pressure. A limited set of developers must share their time between developing features for the next release and resolving reported issues. Project managers need to find the appropriate balance between a high quality product and fast time to market. We study a telecom company in Sweden developing embedded systems for a consumer market. The project managers report that developers resolve approximately 10% of the issues reported during a project. Consequently, it is critical to properly prioritize the issues to receive the best possible return on investment, and above all to remove all bugs that might impact the market's reception of the product. We use machine learning to investigate what features of an issue report are the best predictors of changes to production code during its corresponding resolution. After removing all features jeopardizing the confidentiality of individual engineers, the issue reports are characterized by 19 features (apart from text). We extract 80,000 issue reports, an equal mix of positive and negative examples, and train a Bayesian Network classifier [2], obtaining 73% classification accuracy. Moreover, it reveals that the feature with the highest predictive value is from which physical site the issue was submitted. The general priority feature however, is only ranked 17 out of 19, whereas the submitting team is ranked 12. Our findings confirm a suspicion in the company: the priority set by the issue submitter is indeed a poor predictor of a future code change.

    Download full text (pdf)
    FULLTEXT01
  • 31.
    Borg, Markus
    et al.
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Svensson, Oscar
    Lund University, Sweden.
    Berg, Kristian
    Lund University, Sweden.
    SZZ unleashed: an open implementation of the SZZ algorithm: featuring example usage in a study of just-in-time bug prediction for the Jenkins project2019Conference paper (Refereed)
    Abstract [en]

    Machine learning applications in software engineering often rely on detailed information about bugs. While issue trackers often contain information about when bugs were fixed, details about when they were introduced to the system are often absent. As a remedy, researchers often rely on the SZZ algorithm as a heuristic approach to identify bug-introducing software changes. Unfortunately, as reported in a recent systematic literature review, few researchers have made their SZZ implementations publicly available. Consequently, there is a risk that research effort is wasted as new projects based on SZZ output need to initially reimplement the approach. Furthermore, there is a risk that newly developed (closed source) SZZ implementations have not been properly tested, thus conducting research based on their output might introduce threats to validity. We present SZZ Unleashed, an open implementation of the SZZ algorithm for git repositories. This paper describes our implementation along with a usage example for the Jenkins project, and conclude with an illustrative study on just-in-time bug prediction. We hope to continue evolving SZZ Unleashed on GitHub, and warmly invite the community to contribute.

  • 32.
    Borg, Markus
    et al.
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Wernberg, Joakim
    Swedish Entrepreneurship Forum, Sweden.
    Olsson, Thomas
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Franke, Ulrik
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Andersson, Martin
    Blekinge Institute of Technology, Sweden.
    Illuminating a Blind Spot in Digitalization - Software Development in Sweden’s Private and Public Sector2020In: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, Association for Computing Machinery , 2020, p. 299-302Conference paper (Refereed)
    Abstract [en]

    As Netscape co-founder Marc Andreessen famously remarked in 2011, software is eating the world – becoming a pervasive invisible critical infrastructure. Data on the distribution of software use and development in society is scarce, but we compile results from two novel surveys to provide a fuller picture of the role software plays in the public and private sectors in Sweden, respectively. Three out of ten Swedish firms, across industry sectors, develop software in-house. The corresponding figure for Sweden’s government agencies is four out of ten, i.e., the public sector should not be underestimated. The digitalization of society will continue, thus the demand for software developers will further increase. Many private firms report that the limited supply of software developers in Sweden is directly affecting their expansion plans. Based on our findings, we outline directions that need additional research to allow evidence-informed policy-making. We argue that such work should ideally be conducted by academic researchers and national statistics agencies in collaboration.

  • 33.
    Chatzipetrou, Panagiota
    et al.
    Örebro University, Sweden.
    Alégroth, Emil
    Blekinge Institute of Technology, Sweden.
    Papatheocharous, Eli
    University of Cyprus, Cyprus.
    Borg, Markus
    RISE - Research Institutes of Sweden, ICT, SICS.
    Gorschek, Tony
    Blekinge Institute of Technology, Sweden.
    Wnuk, Krzysztof
    Blekinge Institute of Technology, Sweden.
    Component selection in software engineering - Which attributes are the most important in the decision process?2018In: Proceedings - 44th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2018, 2018, p. 198-205Conference paper (Refereed)
    Abstract [en]

    Component-based software engineering is a common approach to develop and evolve contemporary software systems where different component sourcing options are available: 1)Software developed internally (in-house), 2)Software developed outsourced, 3)Commercial of the shelf software, and 4) Open Source Software. However, there is little available research on what attributes of a component are the most important ones when selecting new components. The object of the present study is to investigate what matters the most to industry practitioners during component selection. We conducted a cross-domain anonymous survey with industry practitioners involved in component selection. First, the practitioners selected the most important attributes from a list. Next, they prioritized their selection using the Hundred-Dollar ($100) test. We analyzed the results using Compositional Data Analysis. The descriptive results showed that Cost was clearly considered the most important attribute during the component selection. Other important attributes for the practitioners were: Support of the component, Longevity prediction, and Level of off-the-shelf fit to product. Next, an exploratory analysis was conducted based on the practitioners' inherent characteristics. Nonparametric tests and biplots were used. It seems that smaller organizations and more immature products focus on different attributes than bigger organizations and mature products which focus more on Cost. .

  • 34.
    Chatzipetrou, Panagiota
    et al.
    Örebro University, Sweden.
    Papatheocharous, Efi
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Wnuk, Krysztof
    Blekinge Institute of Technology, Sweden.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Alegroth, Emil
    Blekinge Institute of Technology, Sweden.
    Gorschek, Tony
    Blekinge Institute of Technology, Sweden.
    Component attributes and their importance in decisions and component selection2020In: Software quality journal, ISSN 0963-9314, E-ISSN 1573-1367, Vol. 28, p. 567-593Article in journal (Refereed)
    Abstract [en]

    Component-based software engineering is a common approach in the development and evolution of contemporary software systems. Different component sourcing options are available, such as: (1) Software developed internally (in-house), (2) Software developed outsourced, (3) Commercial off-the-shelf software, and (4) Open-Source Software. However, there is little available research on what attributes of a component are the most important ones when selecting new components. The objective of this study is to investigate what matters the most to industry practitioners when they decide to select a component. We conducted a cross-domain anonymous survey with industry practitioners involved in component selection. First, the practitioners selected the most important attributes from a list. Next, they prioritized their selection using the Hundred-Dollar ($100) test. We analyzed the results using compositional data analysis. The results of this exploratory analysis showed that cost was clearly considered to be the most important attribute for component selection. Other important attributes for the practitioners were: support of the componentlongevity prediction, and level of off-the-shelf fit to product. Moreover, several practitioners still consider in-house software development to be the sole option when adding or replacing a component. On the other hand, there is a trend to complement it with other component sourcing options and, apart from cost, different attributes factor into their decision. Furthermore, in our analysis, nonparametric tests and biplots were used to further investigate the practitioners’ inherent characteristics. It seems that smaller and larger organizations have different views on what attributes are the most important, and the most surprising finding is their contrasting views on the cost attribute: larger organizations with mature products are considerably more cost aware.

    Download full text (pdf)
    fulltext
  • 35.
    Cicchetti, Antonio
    et al.
    Mälardalen University, Sweden.
    Borg, Markus
    RISE, Swedish ICT, SICS, Security Lab.
    Sentilles, Severine
    Mälardalen University, Sweden.
    Wnuk, Krzysztof
    Blekinge Institute of Technology, Sweden.
    Carlsson, Jan
    Mälardalen University, Sweden.
    Papatheocharous, Efi
    RISE, Swedish ICT, SICS, Software and Systems Engineering Laboratory.
    Towards Software Assets Origin Selection Supported by a Knowledge Repository2016In: 2016 1st International Workshop on Decision Making in Software ARCHitecture (MARCH), 2016, 10, p. 22-29Conference paper (Refereed)
    Abstract [en]

    Software architecture is no more a mere system specification as resulting from the design phase, but it includes the process by which its specification was carried out. In this respect, design decisions in component-based software engineering play an important role: they are used to enhance the quality of the system, keep the current market level, keep partnership relationships, reduce costs, and so forth. For non trivial systems, a recurring situation is the selection of an asset origin, that is if going for in-house, outsourcing, open-source, or COTS, when in the need of a certain missing functionality. Usually, the decision making process follows a case-by-case approach, in which historical information is largely neglected. This solution avoids the overhead of keeping detailed documentation about past decisions, but hampers consistency among multiple, possibly related, decisions. The ORION project aims at developing a decision support framework in which historical decision information plays a pivotal role: it is used to analyse current decision scenarios, take well-founded decisions, and store the collected data for future exploitation. In this paper, we outline the potentials of such a knowledge repository, including the information it is intended to be stored in it, and when and how to retrieve it within a decision case.

    Download full text (pdf)
    FULLTEXT01
  • 36.
    Cito, Jurgen
    et al.
    University of Zurich, Switzerland.
    Wettinger, Johannes
    University of Stuttgart, Germany.
    Lwakatare, Lucy
    University of Oulu, Finland.
    Borg, Markus
    RISE - Research Institutes of Sweden, ICT, SICS.
    Li, Fei
    Siemens AG, Austria.
    Feedback from operations to software development—a devops perspective on runtime metrics and logs2019In: Lect. Notes Comput. Sci., 2019, p. 184-195Conference paper (Refereed)
    Abstract [en]

    DevOps achieve synergy between software development and operations engineers. This synergy can only happen if the right culture is in place to foster communication between these roles. We investigate the relationship between runtime data generated during production and how this data can be used as feedback in the software development process. For that, we want to discuss case study organizations that have different needs on their operations-to-development feedback pipeline, from which we abstract and propose a more general, higher-level feedback process. Given such a process, we discuss a technical environment required to support this process. We sketch out different scenarios in which feedback is useful in different phases of the software development life-cycle.

  • 37.
    de la Vara, José Luis
    et al.
    Carlos III University of Madrid, Spain.
    Borg, Markus
    RISE, Swedish ICT, SICS, Security Lab.
    Wnuk, Krzysztof
    Blekinge Institute of Technology, Sweden.
    Moonen, Leon
    Certus Centre for Software V&V, Norway.
    An Industrial Survey of Safety Evidence Change Impact Analysis Practice2016In: IEEE Transactions on Software Engineering, ISSN 0098-5589, E-ISSN 1939-3520, Vol. 42, no 12, p. 1095-1117Article in journal (Refereed)
    Abstract [en]

    In many application domains, critical systems must comply with safety standards. This involves gathering safety evidence in the form of artefacts such as safety analyses, system specifications, and testing results. These artefacts can evolve during a system's lifecycle, creating a need for impact analysis to guarantee that system safety and compliance are not jeopardised. Although extensive research has been conducted on change impact analysis and on safety evidence management, the knowledge about how safety evidence change impact analysis is addressed in practice is limited. This paper reports on a survey targeted at filling this gap by analysing the circumstances under which safety evidence change impact analysis is addressed, the tool support used, and the challenges faced. We obtained 97 valid responses representing 16 application domains, 28 countries, and 47 safety standards. The results suggest that most practitioners deal with safety evidence change impact analysis during system development and mainly from system specifications. Furthermore, the level of automation in the process is low and insufficient tool support is the most frequent challenge. Other notable findings include that the different artefact types used as safety evidence seem to co-evolve, the evolution of safety case should probably be better managed, and no commercial impact analysis tool has been reported as used for all artefact types. Finally, we identified over 20 areas where the state of the practice in safety evidence change impact analysis can be improved.

    Download full text (pdf)
    FULLTEXT01
  • 38.
    Ebadi, Hamid
    et al.
    Infotiv AB, Sweden.
    Helali Moghadam, Mahshid
    RISE Research Institutes of Sweden, Digital Systems, Industrial Systems.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Gay, Gregory
    Chalmers and the University of Gothenburg, Sweden.
    Fontes, Afonso
    Chalmers and the University of Gothenburg, Sweden.
    Socha, Kasper
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Efficient and Effective Generation of Test Cases for Pedestrian Detection - Search-based Software Testing of Baidu Apollo in SVL2021In: 2021 IEEE International Conference on Artificial Intelligence Testing (AITest), 2021, p. 103-110Conference paper (Refereed)
    Abstract [en]

    With the growing capabilities of autonomous vehicles, there is a higher demand for sophisticated and pragmatic quality assurance approaches for machine learning-enabled systems in the automotive AI context. The use of simulation-based prototyping platforms provides the possibility for early-stage testing, enabling inexpensive testing and the ability to capture critical corner-case test scenarios. Simulation-based testing properly complements conventional on-road testing. However, due to the large space of test input parameters in these systems, the efficient generation of effective test scenarios leading to the unveiling of failures is a challenge. This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator. We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment. Our approach models the input space using a generic and flexible data structure and benefits a multi-criteria safety-based heuristic for the objective function targeted for optimization. This paper presents the results of our proposed test generation technique in the 2021 IEEE Autonomous Driving AI Test Challenge. In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique. Our evaluation shows that the proposed evolutionary test case generator is more effective at generating failure-revealing test cases and provides higher diversity between the generated failures than the random baseline.

  • 39.
    Garousi, Vahid
    et al.
    Queen’s University Belfast, UK.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Oivo, Markku
    University of Oulu, Finland.
    Practical relevance of software engineering research: synthesizing the community’s voice2020In: Empirical Software Engineering, ISSN 1382-3256, E-ISSN 1573-7616, Vol. 25, no 3, p. 1687-1754Article in journal (Refereed)
    Abstract [en]

    Software engineering (SE) research should be relevant to industrial practice. There have been regular discussions in the SE community on this issue since the 1980’s, led by pioneers such as Robert Glass. As we recently passed the milestone of “50 years of software engineering”, some recent positive efforts have been made in this direction, e.g., establishing “industrial” tracks in several SE conferences. However, many researchers and practitioners believe that we, as a community, are still struggling with research relevance and utility. The goal of this paper is to synthesize the evidence and experience-based opinions shared on this topic so far in the SE community, and to encourage the community to further reflect and act on the research relevance. For this purpose, we have conducted a Multi-vocal Literature Review (MLR) of 54 systematically-selected sources (papers and non peer-reviewed articles). Instead of relying on and considering the individual opinions on research relevance, mentioned in each of the sources, the MLR aims to synthesize and provide the “holistic” view on the topic. The highlights of our MLR findings are as follows. The top three root causes of low relevance, discussed in the community, are: (1) Researchers having simplistic views (or wrong assumptions) about SE in practice; (2) Lack of connection with industry; and (3) Wrong identification of research problems. The top three suggestions for improving research relevance are: (1) Using appropriate research approaches such as action-research; (2) Choosing relevant (practical) research problems; and (3) Collaborating with industry. By synthesizing all the discussions on this important topic so far, this paper aims to encourage further discussions and actions in the community to increase our collective efforts to improve the research relevance. Furthermore, we raise the need for empirically-grounded and rigorous studies on the relevance problem in SE research, as carried out in other fields such as management science.

  • 40.
    Gregory, Sarah
    et al.
    Intel Internet of Things Group, USA.
    Borg, Markus
    RISE Research Institutes of Sweden. Lund University, Sweden.
    Looking Back, Moving Forward: A Handover2022In: IEEE Software, ISSN 0740-7459, E-ISSN 1937-4194, Vol. 39, no 5, p. 17-20Article in journal (Refereed)
    Abstract [en]

    A coworker teased me a few months ago, describing requirements engineering (RE) as "plumbing,"something he absolutely considered functional, even necessary, but otherwise best not discussed in polite conversation. I responded with a graphically descriptive suggestion of what one might experience with a plumbing failure. I then drew parallels to a situation where a team's excellent execution of their standard requirements practice proved insufficient for the complex circumstances of their newest program. Although they'd laid the pipes as intended with the tools and practices with which they were proficient, this "new construction" had different needs.

  • 41.
    Helali Moghadam, Mahshid
    et al.
    Mälardalen University, Sweden.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Mousavirad, Seyed
    Hakim Sabzevari University, Iran.
    Deeper at the SBST 2021 Tool Competition: ADAS Testing Using Multi-Objective Search2021In: Proceedings - 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing, SBST 2021, Institute of Electrical and Electronics Engineers Inc. , 2021, p. 40-41Conference paper (Refereed)
    Abstract [en]

    Deeper is a simulation-based test generator that uses an evolutionary process, i.e., an archive-based NSGA-II augmented with a quality population seed, for generating test cases to test a deep neural network-based lane-keeping system. This paper presents Deeper briefly and summarizes the results of Deeper's participation in the Cyber-physical systems (CPS) testing competition at SBST 2021. 

  • 42.
    Helali Moghadam, Mahshid
    et al.
    RISE Research Institutes of Sweden. Mälardalen University, Sweden.
    Borg, Markus
    RISE Research Institutes of Sweden.
    Saadatmand, Mehrdad
    RISE Research Institutes of Sweden.
    Mousavirad, Seyed
    Universidade da Beira Interior, Portugal.
    Bohlin, Markus
    RISE Research Institutes of Sweden. Mälardalen University, Sweden.
    Lisper, Björn
    Mälardalen University, Sweden.
    Machine learning testing in an ADAS case study using simulation-integrated bio-inspired search-based testing2023In: Journal of Software: Evolution and Process, ISSN 2047-7473, E-ISSN 2047-7481Article in journal (Refereed)
    Abstract [en]

    This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (Formula presented.) and (Formula presented.) evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific crossover and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints. © 2023 The Authors. 

  • 43.
    Helali Moghadam, Mahshid
    et al.
    RISE Research Institutes of Sweden, Digital Systems, Industrial Systems.
    Hamidi, Golrokh
    Mälardalen University, Sweden.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Saadatmand, Mehrdad
    RISE Research Institutes of Sweden, Digital Systems, Industrial Systems.
    Bohlin, Markus
    Mälardalen University, Sweden.
    Lisper, Björn
    Mälardalen University, Sweden.
    Potena, Pasqualina
    RISE Research Institutes of Sweden, Digital Systems, Industrial Systems.
    Performance Testing Using a Smart Reinforcement Learning-Driven Test Agent2021In: 2021 IEEE Congress on Evolutionary Computation (CEC), 2021, p. 2385-2394Conference paper (Refereed)
    Abstract [en]

    Performance testing with the aim of generating an efficient and effective workload to identify performance issues is challenging. Many of the automated approaches mainly rely on analyzing system models, source code, or extracting the usage pattern of the system during the execution. However, such information and artifacts are not always available. Moreover, all the transactions within a generated workload do not impact the performance of the system the same way, a finely tuned workload could accomplish the test objective in an efficient way. Model-free reinforcement learning is widely used for finding the optimal behavior to accomplish an objective in many decision-making problems without relying on a model of the system. This paper proposes that if the optimal policy (way) for generating test workload to meet a test objective can be learned by a test agent, then efficient test automation would be possible without relying on system models or source code. We present a self-adaptive reinforcement learning-driven load testing agent, RELOAD, that learns the optimal policy for test workload generation and generates an effective workload efficiently to meet the test objective. Once the agent learns the optimal policy, it can reuse the learned policy in subsequent testing activities. Our experiments show that the proposed intelligent load test agent can accomplish the test objective with lower test cost compared to common load testing procedures, and results in higher test efficiency.

  • 44.
    Helali Moghadam, Mahshid
    et al.
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Saadatmand, Mehrdad
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Borg, Markus
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Bohlin, Markus
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Lisper, Björn
    Mälardalen University, Sweden.
    Adaptive Runtime Response Time Control in PLC-based Real-Time Systems using Reinforcement Learning2018Conference paper (Refereed)
    Abstract [en]

    Timing requirements such as constraints on response time are key characteristics of real-time systems and violations of these requirements might cause a total failure, particularly in hard real-time systems. Runtime monitoring of the system properties is of great importance to check the system status and mitigate such failures. Thus, a runtime control to preserve the system properties could improve the robustness of the system with respect to timing violations. Common control approaches may require a precise analytical model of the system which is difficult to be provided at design time. Reinforcement learning is a promising technique to provide adaptive model-free control when the environment is stochastic, and the control problem could be formulated as a Markov Decision Process. In this paper, we propose an adaptive runtime control using reinforcement learning for real-time programs based on Programmable Logic Controllers (PLCs), to meet the response time requirements. We demonstrate through multiple experiments that our approach could control the response time efficiently to satisfy the timing requirements.

  • 45.
    Helali Moghadam, Mahshid
    et al.
    RISE Research Institutes of Sweden, Digital Systems, Industrial Systems. Mälardalen University, Sweden.
    Saadatmand, Mehrdad
    RISE Research Institutes of Sweden, Digital Systems, Industrial Systems.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Bohlin, Markus
    Mälardalen University, Sweden.
    Lisper, Björn
    Mälardalen University, Sweden.
    An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning2022In: Software quality journal, ISSN 0963-9314, E-ISSN 1573-1367, p. 127-159Article in journal (Refereed)
    Abstract [en]

    Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case-based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments in a simulated performance testing setup, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process and performs adaptively without access to source code and performance models. © 2021, The Author(s).

  • 46.
    Helali Moghadam, Mahshid
    et al.
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Saadatmand, Mehrdad
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Borg, Markus
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Bohlin, Markus
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Lisper, Björn
    Mälardalen University, Sweden.
    Learning-based Response Time Analysis in Real-Time Embedded Systems: A Simulation-based Approach2018Conference paper (Refereed)
    Abstract [en]

    Response time analysis is an essential task to verify the behavior of real-time systems. Several response time analysis methods have been proposed to address this challenge, particularly for real-time systems with different levels of complexity. Static analysis is a popular approach in this context, but its practical applicability is limited due to the high complexity of the industrial real-time systems, as well as many unpredictable run-time events in these systems. In this work-in-progress paper, we propose a simulation-based response time analysis approach using reinforcement learning to find the execution scenarios leading to the worst-case response time. The approach learns how to provide a practical estimation of the worst-case response time through simulating the program without performing static analysis. Our initial study suggests that the proposed approach could be applicable in the simulation environments of the industrial real-time control systems to provide a practical estimation of the execution scenarios leading to the worst-case response time.

  • 47.
    Helali Moghadam, Mahshid
    et al.
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Saadatmand, Mehrdad
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Borg, Markus
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Bohlin, Markus
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Lisper, Björn
    Mälardalen University, Sweden.
    Learning-Based Self-Adaptive Assurance of Timing Properties in a Real-Time Embedded System2018Conference paper (Refereed)
    Abstract [en]

    Providing an adaptive runtime assurance technique to meet the performance requirements of a real-time system without the need for a precise model could be a challenge. Adaptive performance assurance based on monitoring the status of timing properties can bring more robustness to the underlying platform. At the same time, the results or the achieved policy of this adaptive procedure could be used as feedback to update the initial model, and consequently for producing proper test cases. Reinforcement-learning has been considered as a promising adaptive technique for assuring the satisfaction of the performance properties of software-intensive systems in recent years. In this work-in-progress paper, we propose an adaptive runtime timing assurance procedure based on reinforcement learning to satisfy the performance requirements in terms of response time. The timing control problem is formulated as a Markov Decision Process and the details of applying the proposed learning-based timing assurance technique are described.

  • 48.
    Helali Moghadam, Mahshid
    et al.
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS. Mälardalen University, Sweden.
    Saadatmand, Mehrdad
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Borg, Markus
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Bohlin, Markus
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Lisper, Björn
    Mälardalen University, Sweden.
    Machine Learning to Guide Performance Testing: An Autonomous Test Framework2019In: ICST Workshop on Testing Extra-Functional Properties and Quality Characteristics of Software Systems ITEQS'19, 2019, 2019Conference paper (Refereed)
    Abstract [en]

    Satisfying performance requirements is of great importance for performance-critical software systems. Performance analysis to provide an estimation of performance indices and ascertain whether the requirements are met is essential for achieving this target. Model-based analysis as a common approach might provide useful information but inferring a precise performance model is challenging, especially for complex systems. Performance testing is considered as a dynamic approach for doing performance analysis. In this work-in-progress paper, we propose a self-adaptive learning-based test framework which learns how to apply stress testing as one aspect of performance testing on various software systems to find the performance breaking point. It learns the optimal policy of generating stress test cases for different types of software systems, then replays the learned policy to generate the test cases with less required effort. Our study indicates that the proposed learning-based framework could be applied to different types of software systems and guides towards autonomous performance testing.

  • 49.
    Helali Moghadam, Mahshid
    et al.
    RISE Research Institutes of Sweden, Digital Systems, Industrial Systems. Malardalen University, Sweden.
    Saadatmand, Mehrdad
    RISE Research Institutes of Sweden, Digital Systems, Industrial Systems.
    Borg, Markus
    RISE Research Institutes of Sweden, Digital Systems, Mobility and Systems.
    Bohlin, Markus
    RISE Research Institutes of Sweden.
    Lisper, Björn
    Malardalen University, Sweden.
    Poster: Performance Testing Driven by Reinforcement Learning2020In: 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020, p. 402-405Conference paper (Refereed)
    Abstract [en]

    Performance testing remains a challenge, particularly for complex systems. Different application-, platform- and workload-based factors can influence the performance of software under test. Common approaches for generating platform- and workload-based test conditions are often based on system model or source code analysis, real usage modeling and use-case based design techniques. Nonetheless, creating a detailed performance model is often difficult, and also those artifacts might not be always available during the testing. On the other hand, test automation solutions such as automated test case generation can enable effort and cost reduction with the potential to improve the intended test criteria coverage. Furthermore, if the optimal way (policy) to generate test cases can be learnt by testing system, then the learnt policy can be reused in further testing situations such as testing variants, evolved versions of software, and different testing scenarios. This capability can lead to additional cost and computation time saving in the testing process. In this research, we present an autonomous performance testing framework which uses a model-free reinforcement learning augmented by fuzzy logic and self-adaptive strategies. It is able to learn the optimal policy to generate platform- and workload-based test conditions which result in meeting the intended testing objective without access to system model and source code. The use of fuzzy logic and self-adaptive strategy helps to tackle the issue of uncertainty and improve the accuracy and adaptivity of the proposed learning. Our evaluation experiments show that the proposed autonomous performance testing framework is able to generate the test conditions efficiently and in a way adaptive to varying testing situations.

  • 50.
    Henriksson, J.
    et al.
    Semcon, Sweden; Chalmers University of Technology, Sweden.
    Berger, C.
    University of Gothenburg, Sweden; Chalmers University of Technology, Sweden.
    Borg, Markus
    RISE - Research Institutes of Sweden (2017-2019), ICT, SICS.
    Tornberg, L.
    Volvo Cars, Sweden.
    Sathyamoorthy, S. R.
    QRTech AB, Sweden.
    Englund, Cristofer
    RISE - Research Institutes of Sweden (2017-2019), ICT, Viktoria.
    Performance Analysis of Out-of-Distribution Detection on Various Trained Neural Networks2019In: 2019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2019, p. 113-120Conference paper (Refereed)
    Abstract [en]

    Several areas have been improved with Deep Learning during the past years. For non-safety related products adoption of AI and ML is not an issue, whereas in safety critical applications, robustness of such approaches is still an issue. A common challenge for Deep Neural Networks (DNN) occur when exposed to out-of-distribution samples that are previously unseen, where DNNs can yield high confidence predictions despite no prior knowledge of the input. In this paper we analyse two supervisors on two well-known DNNs with varied setups of training and find that the outlier detection performance improves with the quality of the training procedure. We analyse the performance of the supervisor after each epoch during the training cycle, to investigate supervisor performance as the accuracy converges. Understanding the relationship between training results and supervisor performance is valuable to improve robustness of the model and indicates where more work has to be done to create generalized models for safety critical applications.

12 1 - 50 of 77
CiteExportLink to result list
Permanent link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf