Planned maintenance
A system upgrade is planned for 17/11-2026, at 12:00-13:00. During this time DiVA will be unavailable.
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
THE HYPERFITTING PHENOMENON: SHARPENING AND STABILIZING LLMS FOR OPEN-ENDED TEXT GENERATION
RISE Research Institutes of Sweden, Safety and Transport, Electrification and Reliability.ORCID iD: 0000-0003-2811-7481
Google DeepMind, United Kingdom.
PwC Sweden, Sweden.
RISE Research Institutes of Sweden, Digital Systems.ORCID iD: 0000-0002-7020-8275
Show others and affiliations
2025 (English)In: Int. Conf. Learn. Represent., ICLR, International Conference on Learning Representations, ICLR , 2025, p. 45526-45551Conference paper, Published paper (Refereed)
Abstract [en]

This paper introduces the counter-intuitive generalization results of overfitting pre-trained large language models (LLMs) on very small datasets. In the setting of open-ended text generation, it is well-documented that LLMs tend to generate repetitive and dull sequences, a phenomenon that is especially apparent when generating using greedy decoding. This issue persists even with state-of-the-art LLMs containing billions of parameters, trained via next-token prediction on large datasets. We find that by further fine-tuning these models to achieve a near-zero training loss on a small set of samples - a process we refer to as hyperfitting - the long-sequence generative capabilities are greatly enhanced. Greedy decoding with these Hyperfitted models even outperforms Top-P sampling over long-sequences, both in terms of diversity and human preferences. This phenomenon extends to LLMs of various sizes, different domains, and even autoregressive image generation. We further find this phenomena to be distinctly different from that of Grokking and double descent. Surprisingly, our experiments indicate that hyperfitted models rarely fall into repeating sequences they were trained on, and even explicitly blocking these sequences results in high-quality output. All hyperfitted models produce extremely low-entropy predictions, often allocating nearly all probability to a single token.

Place, publisher, year, edition, pages
International Conference on Learning Representations, ICLR , 2025. p. 45526-45551
National Category
Natural Language Processing
Identifiers
URN: urn:nbn:se:ri:diva-79932Scopus ID: 2-s2.0-105010262423OAI: oai:DiVA.org:ri-79932DiVA, id: diva2:2018806
Conference
13th International Conference on Learning Representations, ICLR 2025
Note

Conference paper; Granskad

Available from: 2025-12-04 Created: 2025-12-04 Last updated: 2025-12-08Bibliographically approved

Open Access in DiVA

No full text in DiVA

Scopus

Authority records

Carlsson, FredrikKurfalı, Murathan

Search in DiVA

By author/editor
Carlsson, FredrikKurfalı, Murathan
By organisation
Electrification and ReliabilityDigital Systems
Natural Language Processing

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 63 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf