Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice
Chalmers University of Technology, Sweden; University of Gothenburg, Swede.
RISE Research Institutes of Sweden, Safety and Transport, Electrification and Reliability. Chalmers University of Technology, Sweden.ORCID iD: 0000-0002-3446-1265
Chalmers University of Technology, Sweden; University of Gothenburg, Swede.
Chalmers University of Technology, Sweden; University of Gothenburg, Sweden.
2024 (English)In: Proc. ACM Softw. Eng., Vol. 1, no FSE, p. 1819-1841, article id 81Article in journal (Refereed) Published
Abstract [en]

Large Language Models (LLMs) are frequently discussed in academia and the general public as support tools for virtually any use case that relies on the production of text, including software engineering. Currently, there is much debate, but little empirical evidence, regarding the practical usefulness of LLM-based tools such as ChatGPT for engineers in industry. We conduct an observational study of 24 professional software engineers who have been using ChatGPT over a period of one week in their jobs, and qualitatively analyse their dialogues with the chatbot as well as their overall experience (as captured by an exit survey). We find that rather than expecting ChatGPT to generate ready-to-use software artifacts (e.g., code), practitioners more often use ChatGPT to receive guidance on how to solve their tasks or learn about a topic in more abstract terms. We also propose a theoretical framework for how the (i) purpose of the interaction, (ii) internal factors (e.g., the user’s personality), and (iii) external factors (e.g., company policy) together shape the experience (in terms of perceived usefulness and trust). We envision that our framework can be used by future research to further the academic discussion on LLM usage by software engineering practitioners, and to serve as a reference point for the design of future empirical LLM research in this domain.

Place, publisher, year, edition, pages
Association for Computing Machinery , 2024. Vol. 1, no FSE, p. 1819-1841, article id 81
Keywords [en]
Chatbots, Large Language Models (LLMs), Software Development Bots
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:ri:diva-76000DOI: 10.1145/3660788OAI: oai:DiVA.org:ri-76000DiVA, id: diva2:1908564
Conference
Proc. ACM Softw. Eng.
Note

This work was partially supported by the Wallenberg AI, Autonomous Systems and SoftwareProgram (WASP) funded by the Knut and Alice Wallenberg Foundation. It was also partiallysupported by AGRARSENSE, a project funded by the Chips JU and its members, including thetop-up funding by Sweden, Czechia, Finland, Ireland, Italy, Latvia, Netherlands, Norway, Polandand Spain (Grant Agreement No.101095835).

Available from: 2024-10-28 Created: 2024-10-28 Last updated: 2025-09-23Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full texthttps://doi.org/10.1145/3660788

Authority records

Mohamad, Mazen

Search in DiVA

By author/editor
Mohamad, Mazen
By organisation
Electrification and Reliability
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 134 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf