ROUGE Metric: How to Score Summarization and When Lexical Overlap Fails

GPT-4o scores a ROUGE-1 F1 of 0.250 on CNN/DailyMail, while fine-tuned models like Llama-3.1 with TRAC reach 44.92 on the same benchmark. Human raters, meanwhile, judge LLM summaries on par with human-written ones. If your evaluation pipeline treats a ROUGE score as an absolute quality signal, that gap will mislead you.

This guide covers what the ROUGE metric measures, how ROUGE-N, ROUGE-L, and ROUGE-S differ, current benchmark scores, the metric's documented failure modes, and how to implement it correctly inside a multi-metric evaluation framework.

TLDR:

• ROUGE measures n-gram overlap between generated summaries and human references, reporting recall, precision, and F1

• 76% of ROUGE software citations reference packages with scoring errors, per ACL 2023 research

• Fine-tuned models score ROUGE-1 of roughly 38–45 on CNN/DailyMail; frontier LLMs score ~24–25 despite better human ratings

• Reference choice alone can swing ROUGE scores by roughly 35 points, so multi-reference evaluation matters more than variant choice

• Production systems should pair ROUGE with semantic metrics like BERTScore and validated LLM-as-a-judge evaluation

What is the ROUGE metric?

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) evaluates overlapping text elements between machine-generated summaries and human-written references. Introduced by Lin in 2004, it remains the most widely reported summarization metric more than two decades later.

ROUGE relies on n-gram matching. An n-gram is a contiguous sequence of n words: "the cat" is a bigram. The more overlapping words or phrases between candidate and reference, the higher the score.

It reports three values. Recall measures how much of the reference appears in the generated summary. Precision measures how much of the generated summary matches the reference. F1 combines both.

How well does that overlap track human judgment? It depends heavily on domain. Some narrow-domain studies report stronger alignment between ROUGE and human ratings, especially where terminology is stable and references are extractive. On the broader SummEval benchmark, the picture is far weaker: the G-EVAL study measured ROUGE-L at an average Spearman correlation of 0.165 with human quality judgments, versus 0.514 for GPT-4-based evaluation. Treat high correlations as domain-specific, not general.

ROUGE vs. BLEU, METEOR, and BERTScore

ROUGE's recall orientation fits summarization, where covering reference content matters. BERTScore compares contextual embeddings rather than exact tokens, which makes it better suited to paraphrased or abstractive output. One caveat spans both: semantic similarity and lexical overlap still do not guarantee factual consistency.

ROUGE variants: ROUGE-N, ROUGE-L, and ROUGE-S

Each ROUGE variant captures a different notion of overlap, from strict n-gram matching to word-order-aware subsequences and flexible skip-bigrams.

ROUGE-N: fixed n-gram overlap

ROUGE-N counts overlapping n-grams. ROUGE-1 uses unigrams, ROUGE-2 bigrams:

• Recall = overlapping n-grams / total n-grams in reference

• Precision = overlapping n-grams / total n-grams in candidate

• F1 = 2 × (Precision × Recall) / (Precision + Recall)

Example. Reference: "The cat sits on the mat." Candidate: "The cat sits on the floor." Five of six words overlap, so ROUGE-1 recall, precision, and F1 all equal 0.833.

ROUGE-L: Longest Common Subsequence

ROUGE-L measures the Longest Common Subsequence (LCS) between candidate and reference, rewarding correct word order even when matched words aren't adjacent. Recall is LCS length divided by reference length; precision divides by candidate length.

ROUGE-L earns its keep when content is similar but arranged differently. If the candidate reads "The cat on the floor sits" while the reference reads "The cat sits on the floor," ROUGE-L captures the ordering difference that ROUGE-1 misses. This structural sensitivity is why the ROUGE-Lsum variant anchors summarization shared tasks.

ROUGE-S: skip-bigrams

ROUGE-S allows gaps between matched word pairs, counting bigram matches even when words are separated. For the example sentences above, 10 of 15 skip-bigrams match, giving F1 = 0.667. The lower score reflects how the single "mat" vs. "floor" substitution affects multiple skip-bigram pairs. ROUGE-S can inflate scores for loosely related text, so use it when flexible phrasing genuinely deserves credit.

ROUGE score benchmarks: what "good" looks like in practice

Benchmark expectations split sharply by model type, and conflating the two is the most common interpretation error.

Fine-tuned models on CNN/DailyMail: BRIO scores ROUGE-1 38.49 / ROUGE-2 17.08 / ROUGE-L 31.44, and Llama-3.1 with TRAC reaches ROUGE-1 44.92 / ROUGE-2 22.47 / ROUGE-L 35.92. On XSum, fine-tuned BRIO hits ROUGE-1 49.66 / ROUGE-2 25.97 / ROUGE-L 41.04.

Frontier LLMs used zero-shot score far lower on the same data. An April 2025 comparison measured ROUGE-1 F1 on CNN/DailyMail at 0.250 for GPT-4o, 0.241 for Claude-3.5-Sonnet, and 0.245 for Gemini-1.5-Pro. On XSum the dataset average fell to 0.160; the paper attributes this to XSum's "highly abstractive single-sentence summary style, which diverges lexically from references while maintaining factual alignment."

The lesson: a ROUGE score has meaning only relative to a specific reference set, model family, and task. A ROUGE-1 of 0.25 from a zero-shot LLM can accompany a summary humans prefer over a fine-tuned model's 0.44.

Reference variability can outweigh model differences

Which human reference you score against can matter more than which model you evaluate. Casola et al. (INLG 2025) found ROUGE-1max varies by roughly 35 points on average across human-written references, and that "model rankings vary depending on the reference sets, undermining the reliability of model comparisons." Other domain studies show the same basic pattern: reference selection alone can move ROUGE enough to change conclusions.

Practical mitigations:

• Use multiple references. Casola et al. found ROUGE needs 5–10 references to match the stability BERTScore achieves with one; historical summarization evaluations also used multiple references per topic.

• Prefer average pooling (ROUGEavg) over max pooling when reference sets are large, per the same INLG 2025 study.

• Document which references you used. Rankings don't transfer across reference sets.

A practical guide to understanding ROUGE scores, their limitations, and how to use them effectively alongside modern evaluation metrics.

A reproducibility problem in the tooling

Grusky's "Rogue Scores" (ACL 2023) audited 2,834 ROUGE papers and 831 codebases. Among papers citing a ROUGE software package, 76% cited packages with scoring errors. Only 5% of papers listed configuration parameters, and only 6% performed significance testing. The study estimated over 2,000 papers used incorrect ROUGE packages. If you deploy ROUGE in production, verify your implementation against a reference and version-pin it.

Semantic blindness and paraphrasing penalties

ROUGE measures token overlap, not meaning. Ng and Abrecht showed it is "biased towards surface lexical similarities" and "unsuitable for the evaluation of abstractive summarization, or summaries with substantial paraphrasing." This bites hardest with modern LLMs: Zhang et al. found that "despite major stylistic differences such as the amount of paraphrasing, LLM summaries are judged to be on par with human written summaries." Humans don't penalize paraphrasing; ROUGE does.

The inverse failure is worse. A hallucinated summary can score well on lexical overlap while fabricating facts, a failure mode Galileo tracks through its hallucination research. Even a single negation can reverse meaning while leaving most tokens unchanged: "pneumonia is seen" and "pneumonia is not seen" share nearly all words but make opposite clinical claims.

Degrading correlation on LLM outputs

As output quality improves, ROUGE's usefulness for ranking shrinks. Casola et al. observed that correlations with human judgment are "generally higher on SummEval (where we consider outputs from the pre-LLM era) than on GUMSum (where we consider LLMs)." The practical result is clear: traditional lexical overlap metrics become less reliable as summaries become more fluent, paraphrased, and abstractive.

Domain-specific failure modes

• Healthcare: ROUGE can miss factual and clinical errors because near-identical wording can carry opposite meaning. Expand medical abbreviations before scoring and add entity-level checks.

• Legal: ROUGE can understate or overstate quality when legal meaning depends on citations, section structure, or jargon. Preserve section numbering and citations as atomic tokens, and evaluate segment-wise.

• Financial: ROUGE is weak when summaries require exact numerical accuracy across tables, footnotes, and financial figures. Supplement lexical metrics with exact-match scoring on figures.

• Cross-lingual: ROUGE was designed around English tokenization assumptions, so multilingual and non-Latin-script evaluation needs language-specific preprocessing and tokenizers.

ROUGE in RAG pipelines and LLM evaluation stacks

For retrieval-augmented generation, ROUGE covers less ground than teams assume. The 2025 RAG evaluation survey maps ROUGE exclusively to correctness (response vs. reference answer). It does not measure faithfulness (whether the response is supported by retrieved documents) or relevance (whether it answers the query). Grounding-focused RAG evaluations instead use metrics such as context relevance, faithfulness, completeness, and citation coverage. When building RAG systems, use ROUGE only where a ground-truth answer exists, and pair it with faithfulness metrics for hallucination coverage.

ROUGE also hasn't been displaced by LLM judges; it works alongside them. LLM-as-a-judge evaluation has become common, but many implementations still lack validation against human evaluation. Lexical metrics persist because they are cheap, deterministic, and easy to compare across runs, while LLM judges carry their own documented biases, including position, verbosity, and concreteness bias.

That's why the field has converged on hybrid stacks rather than a single successor metric. Clinical summarization shared tasks often pair ROUGE-L-sum with BERTScore, and follow-up analyses find that different metric families answer different questions: semantic metrics better capture correctness, while syntactic metrics can better track completeness. That is the core argument for multi-metric evaluation frameworks.

ROUGE's remaining strength in these stacks is speed and determinism. It requires no model calls, which suits regression checks and CI gates where you need a fast signal that nothing broke. Simple lexical metrics can still be useful out of domain, but they should be treated as one signal rather than the final verdict.

How to implement the ROUGE metric correctly

Getting ROUGE right in practice means pinning a trusted library, preprocessing consistently, scoring against multiple references, and wiring it into CI as a fast regression gate alongside complementary metrics.

1. Use current library versions

The rouge-score package remains at version 0.1.2 with an unchanged API:

from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)
scores = scorer.score('reference text', 'candidate text')

Check tokenizer dependencies before upgrading evaluation pipelines. Older NLTK-based code may expect one tokenizer resource name while newer environments require another, so pin dependencies and test ROUGE output before comparing new scores with old baselines.

2. Preprocess consistently

Lowercase, tokenize, and stem so "running" and "runs" match. Set use_stemmer=True. For domain text, handle special tokens before scoring: expand clinical abbreviations, and treat currency values like "$1,000" as single tokens in financial text.

3. Score against multiple references

Given the up-to-40-point reference variance documented above, single-reference scoring can be unreliable for model comparison:

def calculate_rouge_with_multiple_references(candidate, references):
    scorer = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)
    scores_list = [scorer.score(ref, candidate) for ref in references]
    max_rouge1 = max(score['rouge1'].fmeasure for score in scores_list)
    max_rougeL = max(score['rougeL'].fmeasure for score in scores_list)
    return {'rouge1': max_rouge1, 'rougeL': max_rougeL}

4. Wire ROUGE into CI/CD as a regression gate

Because ROUGE is deterministic, it fits pre-merge checks that run before slower LLM-judge evaluations. CI systems can fail checks on non-zero exit codes, and required status checks can block merges:

- name: Check ROUGE Threshold
  run: |
    python -c "scores = evaluate_model(); assert scores['rougeL'] >= 0.35"

Set the threshold from your own baseline, not from published benchmarks, since scores don't transfer across reference sets. Version-control baselines alongside model artifacts as part of your MLOps workflow, and document ROUGE configuration parameters; only 5% of published papers do, per the ACL 2023 audit.

5. Follow these practices in production

• Choose the variant for the job: ROUGE-N for exact token matching, ROUGE-L when sentence structure matters, ROUGE-S when reordering deserves credit.

• Never gate a release on ROUGE alone. Pair it with a semantic metric and, where hallucination risk exists, a faithfulness check.

• Run significance testing before declaring one model better than another; score differences smaller than reference variance are noise.

• Watch for systematic underestimation. If ROUGE consistently underpredicts human ratings, your model is likely paraphrasing well, which signals a need for semantic metrics rather than a model problem.

How Galileo extends ROUGE-based evaluation

ROUGE gives you a fast lexical baseline; production AI systems need the semantic and faithfulness dimensions it cannot see. The Galileo platform covers both:

• ROUGE and BLEU in Experiments: Per Galileo's documentation, "BLEU and ROUGE are only supported in experiments, and require a Ground Truth to be set in the output column of your experiment's dataset." That matches how the research says ROUGE should be used: reference-based, in controlled comparisons.

• Context Adherence: Detects closed-domain hallucinations, "cases where your model said things that were not provided in the context," covering the precision dimension of RAG generation quality that ROUGE misses entirely.

• Completeness: "Measures how thoroughly a model's response covers the relevant information provided in the context," the recall dimension, without depending on a single reference summary.

• Luna-2 evaluation models: Purpose-built small language models that run evaluation at 98% lower cost than LLM-based judges with sub-200ms latency, making 100% traffic evaluation feasible instead of sampling.

• Continuous Learning via Human Feedback: Tune LLM-powered metrics with your own judgments; feedback becomes few-shot examples that raise metric accuracy by 20–30% with as few as one or two examples.

Get started with Galileo to combine lexical baselines with semantic and faithfulness evaluation in one pipeline.

Frequently asked questions

What is the ROUGE metric and how does it work?

ROUGE measures overlap between machine-generated text and human reference summaries using n-gram matching (ROUGE-N), longest common subsequence (ROUGE-L), or skip-bigrams (ROUGE-S). It reports recall, precision, and F1 for each variant.

What is a good ROUGE score for summarization?

It depends on model type and dataset. Fine-tuned models reach ROUGE-1 of roughly 38–45 on CNN/DailyMail, while zero-shot frontier LLMs score around 24–25 on the same benchmark despite producing summaries humans rate highly. Compare against your own baseline on your own reference set rather than published numbers.

What is the difference between ROUGE-1, ROUGE-2, and ROUGE-L?

ROUGE-1 counts unigram overlap, ROUGE-2 counts bigram overlap, and ROUGE-L measures the longest common subsequence, which rewards correct word ordering even when matched words aren't adjacent.

Should I use ROUGE or BERTScore?

Both. ROUGE gives a fast, deterministic lexical signal suited to regression gates; BERTScore captures semantic similarity for paraphrased output. Clinical summarization tasks often pair ROUGE-L-sum with BERTScore for exactly this reason, and neither reliably detects factual errors on its own.

How does Galileo work with ROUGE?

Galileo supports ROUGE and BLEU in Experiments with a ground truth set, then adds what lexical overlap can't measure: Context Adherence for hallucination detection, Completeness for context coverage, and Luna-2 models that evaluate full production traffic at 98% lower cost than LLM judges.