Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice
2024 (English)In: Proc. ACM Softw. Eng., Vol. 1, no FSE, p. 1819-1841, article id 81Article in journal (Refereed) Published
Abstract [en]
Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently, there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how the (i) purpose of the interaction, (ii) internal factors (e.g., the user’s personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.
Place, publisher, year, edition, pages
Association for Computing Machinery , 2024. Vol. 1, no FSE, p. 1819-1841, article id 81
Keywords [en]
Chatbots, Large Language Models (LLMs), Software Development Bots
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:ri:diva-76000DOI: 10.1145/3660788OAI: oai:DiVA.org:ri-76000DiVA, id: diva2:1908564
Conference
Proc. ACM Softw. Eng.
Note
This work was partially supported by the Wallenberg AI, Autonomous Systems and SoftwareProgram (WASP) funded by the Knut and Alice Wallenberg Foundation. It was also partiallysupported by AGRARSENSE, a project funded by the Chips JU and its members, including thetop-up funding by Sweden, Czechia, Finland, Ireland, Italy, Latvia, Netherlands, Norway, Polandand Spain (Grant Agreement No.101095835).
2024-10-282024-10-282025-09-23Bibliographically approved