HellaSwag — Can a Machine Really Finish Your Sentence?

Abstract

Recent work by Zellers et al. (2018) introduced SWAG: a dataset of 113k multiple choice questions about grounded situations. A model trained on BERT achieved near-human performance, prompting the question: has the task been solved?

HellaSwag answers with a resounding no. By applying a stronger version of Adversarial Filtering (AF), the authors construct a dataset where state-of-the-art models score below 48% — while humans achieve ~95%. The construction sheds light on the inner workings of deep pretrained models and suggests a new path forward for NLP research.

HellaSwag has since become a canonical evaluation benchmark, appearing in the technical reports of GPT-4, LLaMA, Mistral, Gemini, and hundreds of other large language models.

Key Contribution

Adversarial Filtering

01

Oversample counterfactuals

Language models generate a large pool of plausible-sounding wrong answer candidates for each context.

02

Train stylistic classifiers

An ensemble of discriminators learns to distinguish machine-generated endings from human-written ones.

03

Filter iteratively

Only the hardest, least-detectable distractors survive — those that fool the classifiers most consistently.

04

Benchmark that stays hard

The result is a dataset that resists simple stylistic shortcuts, forcing models to reason about world knowledge.

Model Performance at Release (2019)

The Gap Was Stark

Human

95.6%

BERT-Large (fine-tuned)

47.3%

GPT (zero-shot)

41.8%

Random baseline

25.0%

State-of-the-art models in 2019 barely exceeded random chance on HellaSwag, despite near-human performance on SWAG.

Authors

The Team

RZ

Rowan Zellers

First Author

PhD student at UW (advised by Choi & Farhadi). Graduated from Harvey Mudd College (CS & Math, 2016). Later joined OpenAI, where he led video perception for GPT-4o. Now researcher at Thinking Machines.

Harvey Mudd '16 UW PhD OpenAI

AH

Ari Holtzman

Second Author

PhD student at UW (advised by Zettlemoyer). Won the inaugural Amazon Alexa Prize (2017). Known for nucleus sampling ("The Curious Case of Neural Text Degeneration," ICLR 2020). Postdoc at Meta FAIR; now Assistant Professor at University of Chicago.

NYU BA UW PhD Meta FAIR UChicago Prof

YB

Yonatan Bisk

Third Author

PhD from UIUC. Held visiting researcher positions at Microsoft Research (vision+language) and Meta AI (Embodied AI). Founded Origami Robotics, an embodied-AI startup. Now Associate Professor at CMU's Language Technologies Institute and Robotics Institute.

UIUC PhD Microsoft Research Meta AI CMU Prof Origami Robotics

AF

Ali Farhadi

Fourth Author

Professor at UW and researcher at AI2. Co-founded Xnor.ai, the first on-device deep learning startup — acquired by Apple in 2020 for ~$200M. Led Apple's next-generation ML efforts, then returned to AI2 as CEO (2023–2026). Forbes Top 5 AI Entrepreneurs, 2018.

UW Prof Xnor.ai founder Apple ML AI2 CEO

YC

Yejin Choi

Senior / Corresponding Author

Brett Helsel Professor at UW Paul G. Allen School; Senior Research Director at AI2. PhD from Cornell (2010). MacArthur Fellow (2022). One of the most cited NLP researchers globally. Extensive collaborations with Microsoft, Google, and Meta researchers.

Cornell PhD UW Prof AI2 MacArthur Fellow

Legacy & Impact

A Benchmark That Defined an Era

📊

4,697+ citations

One of the most-cited NLP benchmark papers of the past decade, per Semantic Scholar (August 2026).

🤖

Universal LLM eval

HellaSwag scores appear in the technical reports of GPT-4, LLaMA 2/3, Mistral, Gemini, Claude, and hundreds of other models.

🔬

Methodological contribution

Adversarial Filtering became a widely adopted technique for constructing robust NLP benchmarks that resist annotation artifacts.

🏆

Staying power

Seven years after publication, HellaSwag remains a standard evaluation task — a rare achievement for any benchmark dataset.

Cite This Work

@inproceedings{zellers2019hellaswag,
  title     = {HellaSwag: Can a Machine Really Finish Your Sentence?},
  author    = {Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan
               and Farhadi, Ali and Choi, Yejin},
  booktitle = {Proceedings of the 57th Annual Meeting of the
               Association for Computational Linguistics},
  pages     = {4791--4800},
  year      = {2019},
  doi       = {10.18653/v1/P19-1472}
}