Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Overview of the “Voight-Kampff” Generative AI Authorship Verification Task at PAN and ELOQUENT 2024
Leipzig University, Germany; Bauhaus-Universität Weimar, Germany.
Bauhaus-Universität Weimar, Germany.
Silo AI, Finland.
RISE Research Institutes of Sweden, Digital Systems, Data Science.ORCID iD: 0000-0003-3246-1664
Show others and affiliations
2024 (English)In: CEUR Workshop Proceedings, CEUR-WS , 2024, Vol. 3740, p. 2486-2506Conference paper, Published paper (Refereed)
Abstract [en]

The “Voight-Kampff” Generative AI Authorship Verification task aims to determine whether a text was generated by an AI or written by a human. As in its fictional inspiration,1 the Voight-Kampff task structures AI detection as a builder-breaker challenge: The builders, participants in the PAN lab, submit software to detect AI-written text and the breakers, participants in the ELOQUENT lab, submit AI-written text with the goal of fooling the builders. We formulate the task in a way that is reminiscent of a traditional authorship verification problem, where given a pair of texts, their human or machine authorship is to be inferred. For this first task installment, we further restrict the problem so that each pair is guaranteed to contain one human and one machine text. Hence the task description reads: Given two texts, one authored by a human, one by a machine: pick out the human. In total, we evaluated 43 detection systems (30 participant submissions and 13 baselines), ranging from linear classifiers to perplexity-based zero-shot systems. We tested them on 70 individual test set variants organized in 14 base collections, each designed on different constraints such as short texts, Unicode obfuscations, or language switching. The top systems achieve very high scores, proving themselves not perfect but sufficiently robust across a wide range of specialized testing regimes. Code used for creating the datasets and evaluating the systems, baselines, and data are available on GitHub.

Place, publisher, year, edition, pages
CEUR-WS , 2024. Vol. 3740, p. 2486-2506
Keywords [en]
Problem oriented languages; AI detection; Authorship verification; Detection system; Human-machine; One-machine; Task description; Task structure; Verification problems; Verification task; Written texts
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:ri:diva-75037Scopus ID: 2-s2.0-85201598034OAI: oai:DiVA.org:ri-75037DiVA, id: diva2:1895667
Conference
25th Working Notes of the Conference and Labs of the Evaluation Forum, CLEF 2024. Grenoble, France 9 September 2024 through 12 September 2024
Note

The “Voight-Kampff” Generative AI Authorship Detection Task @ PAN 2024 has been funded as part ofthe OpenWebSearch project by the European Commission (OpenWebSearch.eu, GA 101070014).

Available from: 2024-09-06 Created: 2024-09-06 Last updated: 2025-09-23Bibliographically approved

Open Access in DiVA

fulltext(1588 kB)241 downloads
File information
File name FULLTEXT01.pdfFile size 1588 kBChecksum SHA-512
e52cfba73b6cedc5b35e96cb8cfa9e92f2598b3cfa14a95ed7d7562a3b3b2abcf1cb9bd8f5c2ebd15b0e40293e647d97b6f75cdcb8920cf2d98a5a97bf27edf4
Type fulltextMimetype application/pdf

Other links

ScopusFull text

Authority records

Dürlich, LuiseGogoulou, Evangelia

Search in DiVA

By author/editor
Dürlich, LuiseGogoulou, Evangelia
By organisation
Data Science
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 244 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 888 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf