Understanding ROUGE Scores: How to Calculate and Interpret Them for AI Evaluation

Your summarization model scores well on internal tests, but users keep complaining that summaries miss the point. You check the ROUGE scores and see numbers: 0.38, 0.21, 0.42. You have no idea whether those are good, bad, or meaningless for your use case. ROUGE is the most widely used family of metrics for evaluating text summarization, yet most teams misuse it. 

They pick the wrong variant for the task, compare scores across datasets where comparison is invalid, or treat ROUGE as a sole quality signal when it only measures surface-level word overlap. A study of ROUGE reporting practices found that only 5% of papers list the ROUGE parameters they used, which makes most published scores hard to interpret or reproduce. This article explains each ROUGE variant, when to use which, what a good score looks like by task type, how to calculate scores in Python, and where ROUGE falls short, so you can use it as one informed signal in a broader evaluation strategy.

TL;DR

• ROUGE measures n-gram overlap between AI-generated and reference texts across four variants: ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S

• ROUGE-1 captures word-level coverage, ROUGE-2 captures two-word phrase overlap, ROUGE-L captures sentence-level word order

• On CNN/DailyMail abstractive summarization, strong systems score roughly 0.41–0.45 ROUGE-L; scores are dataset-dependent and never comparable across benchmarks

• ROUGE cannot assess factual accuracy, semantic meaning, or coherence; it only counts word matches

• Pair ROUGE with semantic metrics or LLM-as-judge evaluation for production-grade assessment

What Is ROUGE in AI?

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a family of metrics that measure the overlap between AI-generated text and human-written reference text. Chin-Yew Lin of USC's Information Sciences Institute published it in 2004 for summarization evaluation, and three of its four measures were adopted in the NIST-sponsored DUC 2004 evaluation. In Lin's words, ROUGE "count[s] the number of overlapping units such as n-gram, word sequences, and word pairs between the computer-generated summary to be evaluated and the ideal summaries created by humans."

That word-counting design separates ROUGE from semantic metrics like BERTScore, which compare contextual embeddings, and from LLM-based evaluation like G-Eval, which prompts a model to judge quality directly. ROUGE tells you whether your model used similar words to the reference, not whether it captured the meaning. The trade-off is that ROUGE is fast, deterministic, and nearly free to compute, which keeps it useful as a baseline even though it is a weak sole signal.

How ROUGE Variants Work

ROUGE is not one metric but a family. Each variant captures a different dimension of text similarity, and choosing the right one depends on whether you care about word coverage, phrase quality, sentence structure, or flexible word ordering. All four definitions come from Lin's original paper.

ROUGE-N: Unigram, Bigram, and Trigram Overlap

ROUGE-N counts contiguous n-gram matches at a fixed length. ROUGE-1 counts matching single words, ROUGE-2 counts matching two-word phrases, and ROUGE-3 counts trigrams (rarely reported in practice). Most papers report ROUGE-1 and ROUGE-2 together. 

Each score decomposes into three components: precision is the share of the generated text's n-grams that appear in the reference, recall is the share of the reference's n-grams that the generated text captured, and F1 is their harmonic mean. Lin's own example shows the blind spot: given the reference "police killed the gunman," the candidates "police kill the gunman" and "the gunman kill police" receive identical ROUGE-2 scores because each shares exactly one bigram, "the gunman," despite the second candidate reversing who did what.

Reading Precision, Recall, and F1

The three components answer different questions, and reading them together tells you more than any single number. Recall answers "did we capture what the reference said?" It measures how much of the reference's content the generated summary covered. Precision answers "is what we generated actually in the reference?" It measures how much of the generated text was relevant rather than padding.

The two failure modes fall out of that split. High recall with low precision usually means the summary captured most of the reference's content but buried it in extra words. The output is verbose or over-long, and the surplus text drags precision down even though coverage is good. High precision with low recall points the other way: the summary is tight and accurate word-for-word but too short, so it leaves large portions of the reference uncovered.

F1, the harmonic mean of the two, rewards summaries that cover the reference and stay concise at the same time. A model can't game F1 by padding for coverage or trimming for precision, since either move pulls the other component down. Report F1 as the headline figure when neither verbosity nor brevity should be rewarded. Weight recall more heavily when your main worry is a model that drops key information, and weight precision when a model tends to ramble.

ROUGE-L: Longest Common Subsequence

ROUGE-L measures the longest sequence of words appearing in both texts in the same order, gaps allowed. As Lin puts it, LCS "does not require consecutive matches but in-sequence matches that reflect sentence level word order," and it needs no predefined n-gram length. "In-sequence but not consecutive" means the matched words have to appear in the same relative order in both texts, but other words can sit between them. 

Take the reference "the cat sat quietly on the mat" and an output "the cat rested on the mat." The words the cat on the mat line up in order in both, with "rested" filling the gap where "sat quietly" was, so all five of those words count toward the longest common subsequence even though they are not contiguous. 

That flexibility resolves the earlier example: with β = 1, "police kill the gunman" scores 0.75 on ROUGE-L while "the gunman kill police" scores 0.5, separating the two candidates that ROUGE-2 could not. ROUGE-L is part of the standard reporting trio (ROUGE-1, ROUGE-2, ROUGE-L) and is a strong choice when you want to know whether the output preserves the reference's logical flow rather than its exact phrasing.

ROUGE-W: Weighted Longest Common Subsequence

ROUGE-W extends ROUGE-L by weighting consecutive matches more heavily than scattered ones, using a weighting function such as f(k) = k². Lin's example: against reference [A B C D E F G], a candidate with four consecutive matches and a candidate with four spread-out matches earn identical ROUGE-L scores, but ROUGE-W scores them 0.571 and 0.286 respectively. When fluency matters as much as coverage, ROUGE-W helps, though it sees far less use than ROUGE-L in practice.

ROUGE-S: Skip-Bigram Overlap

ROUGE-S counts skip-bigrams: "any pair of words in their sentence order, allowing for arbitrary gaps." A four-word sentence yields C(4,2) = 6 skip-bigrams. Because matches survive intervening words, ROUGE-S is more flexible about word order than ROUGE-N. A variant, ROUGE-SU, adds unigram credit for sentences with no skip-bigram matches at all.

One practical caveat: ROUGE-W and ROUGE-S are defined in the original paper but absent from the standard Hugging Face Evaluate and Google Research rouge-score implementations. In day-to-day work you effectively have ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum.

ROUGE-1 vs ROUGE-2 vs ROUGE-L: Which Variant to Use When

The variants disagree often enough that picking the wrong one changes your model rankings. A meta-evaluation of summarization metrics at EMNLP 2020 found ROUGE-1 most reliable for evaluating extractive summarization and ROUGE-2 most reliable for abstractive summarization. Task-by-task guidance from the research:

• Extractive summarization: ROUGE-1 for evaluation. ROUGE-2 is the standard for oracle construction, where a greedy algorithm selects sentences that maximize the ROUGE-2 score.

• Abstractive summarization: ROUGE-2, paired with a semantic metric, because correlation between overlap metrics decreases as outputs become more abstractive.

• Single-document and headline summarization: ROUGE-L performs well, per Lin's original experiments. Avoid ROUGE-L for multi-document summarization, where the same experiments found it did not perform well.

• Machine translation: BLEU is the primary metric; ROUGE is not standard here.

• Dialogue: for open-domain chatbots, ROUGE shows weak or no correlation with human judgments, so avoid it. For task-oriented dialogue, ROUGE-L shows only moderate correlation (Spearman 0.294–0.346) and improves with multiple references.

• Multilingual summarization: for cross-lingual evaluation, BERTScore is a better choice than ROUGE for most languages.

Benchmarks conventionally report ROUGE-1, ROUGE-2, and ROUGE-L together as F1 scores. Whichever you report, document your configuration: stemming, tokenization, rougeL versus rougeLsum, and whether scores are on a 0–1 or 0–100 scale. Skipping those details is the reproducibility gap the ACL 2023 study identified.

How to Calculate ROUGE Scores in Python

Use the rouge-score package on PyPI, maintained by Google under Apache 2.0 and designed to replicate the original Perl package natively in Python (current release v0.1.2, Python ≥ 3.7):

pip install rouge-score

from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
scores = scorer.score(
    'The quick brown fox jumps over the lazy dog',
    'The quick brown dog jumps on the log.'
)
print(scores['rouge1'])             # Score(precision=..., recall=..., fmeasure=...)
print(scores['rouge1'].precision)
print(scores['rouge1'].recall)
print(scores['rouge1'].fmeasure)

scorer.score() returns a dictionary mapping each rouge type to a Score namedtuple with precision, recall, and fmeasure fields. Valid types are rouge1, rouge2, rougeL (sentence-level LCS), and rougeLsum, which treats newlines as sentence boundaries and computes a union-LCS. To aggregate over a corpus, the package's BootstrapAggregator returns low, mid, and high confidence-interval scores.

Avoid the older py-rouge package. Its last release was in 2018 and shows zero commit or issue activity. For production-scale evaluation across thousands of traces, most teams run metrics through an evaluation platform rather than scripting ROUGE by hand.

What Is a Good ROUGE Score? Benchmarks by Task

There is no universal "good" ROUGE score. Scores depend entirely on the dataset, task type, and tokenization, and model rankings can flip depending on which reference sets are used. With that caveat, published leaderboards give useful anchors.

Abstractive summarization (CNN/DailyMail), scored on the 0–100 scale per the CodeSOTA leaderboard: strong systems land at ROUGE-1 ≈ 44–48, ROUGE-2 ≈ 21–24, ROUGE-L ≈ 41–45. BRIO holds the top spot (47.78 / 23.55 / 44.57), with GPT-4o at 46.30 / 22.10 / 43.40 and PEGASUS-Large at 44.17 / 21.47 / 41.11. The leaderboard notes ROUGE-based evaluation on this benchmark has been saturated since mid-2022.

Extractive summarization (CNN/DailyMail): MatchSum reaches 44.41 / 20.86 / 40.55, a similar band to abstractive systems on the same data. Dataset difficulty matters more than task type: the same class of extractive models scores lower on PubMed, where MemSum leads at 43.08 / 16.71 / 38.30.

The PubMed gap is instructive. PubMed articles are long scientific documents, and a good summary has to compress far more source text into a short abstract than a CNN news story requires. More compression means the reference and the generated summary share fewer exact word sequences, so even a strong extractive model like MemSum lands several points below what the same architecture manages on CNN/DailyMail. A 43 ROUGE-1 on PubMed can represent a harder problem solved well, while a 44 on CNN/DailyMail may reflect an easier one. Reading the raw number without the dataset behind it tells you almost nothing.

Benchmark saturation compounds the problem. Since mid-2022 the CNN/DailyMail leaderboard has barely moved, with BRIO at 47.78 ROUGE-1, GPT-4o at 46.30, and PEGASUS-Large at 44.17 clustered within a few points of each other. When the top systems sit that close, a fractional ROUGE gain no longer reliably signals a better summary. It can also reflect tokenization choices, stemming, or reference-set quirks. Teams chasing another tenth of a point on this benchmark are usually measuring noise, not progress.

Machine translation: ROUGE appears mainly in low-resource shared tasks, where scores swing widely by language pair. In the WMT 2025 low-resource Indic MT task, the top Assamese-to-English system hit ROUGE-L 0.699, while Kokborok-to-English systems scored 0.18–0.22. Note those are proportions on a 0–1 scale, not the 0–100 percentages used for summarization leaderboards; the two are not comparable.

Never compare ROUGE scores across datasets or task types. A 0.38 ROUGE-L on a hard abstractive task can reflect a stronger model than a higher score on an easy extractive one.

ROUGE vs BLEU vs BERTScore: Key Differences

BLEU is ROUGE's precision-oriented counterpart. It clips candidate n-gram counts against the references and applies a brevity penalty, which suits translation where word-for-word fidelity matters. Where BLEU and ROUGE stop at exact matching, BERTScore substitutes cosine similarity between contextual embeddings, so it credits paraphrases that the overlap metrics penalize, at the cost of running a transformer model. 

G-Eval takes a different route again, using an LLM with chain-of-thought reasoning to score quality dimensions directly. On the SummEval benchmark it averaged 0.514 Spearman correlation with human judgments versus 0.165–0.205 for the ROUGE variants.

Treat these as layers, not competitors. In practice a team sequences them by cost. ROUGE and BLEU run first as a fast, deterministic, reproducible baseline over every candidate, cheap enough to gate a CI pipeline and catch regressions before anything else runs. BERTScore comes next on a sampled subset, adding paraphrase awareness where exact overlap would flag a correct rewording as a failure. 

LLM-as-judge sits on top, run on a smaller sample still because inference is the most expensive step, and it supplies the criterion-level assessment of coherence, faithfulness, and relevance that neither overlap nor embeddings capture. No source in the current research advocates dropping reference-based metrics entirely, and Galileo's documentation states that "Best practices dictate that LLM-as-a-judge should be used in combination with traditional evaluation metrics, as each approach complements the other's strengths."

Limitations of ROUGE

ROUGE has three limitations that no variant fixes.

First, it cannot detect factual errors or hallucinations. The FRANK benchmark measured ROUGE-1 at a partial Pearson correlation of just 0.14 with human factuality judgments, and ROUGE-L at 0.13. A summary can invert who did what to whom and still score well. For a practitioner, that means a high ROUGE score offers no protection against a confidently wrong summary. If factual consistency matters, you have to test for it separately, because the overlap number will look fine either way.

Second, it penalizes valid paraphrasing. N-gram overlap metrics "cannot appropriately reward semantic or syntactic variations of a given reference," so a correct summary phrased differently from the reference gets punished. In practice this hurts exactly the abstractive systems you most want to reward: a model that restates an idea in fresh, fluent language can score below a clumsier one that parrots the reference wording. Ranking models by ROUGE alone can quietly favor copying over comprehension.

Third, it says nothing about coherence, tone, or usefulness. The SummEval study found ROUGE-L's system-level Kendall's tau with human coherence judgments was 0.0735, near zero. A summary can hit its overlap targets and still read as a disjointed list of fragments that no reader would accept. If coherence is part of what you ship, ROUGE will not warn you when it degrades.

These gaps are exactly what LLM-based evaluation covers, which is why production teams layer ROUGE under judge-based metrics rather than choosing between them. At production scale, purpose-built evaluation models make that layer affordable: Galileo's Luna-2 small language models benchmark at $0.02 per 1M tokens with a 0.95 F1 score and 152ms average latency, against $2.50 per 1M tokens and 3,200ms for GPT-4o.

Using ROUGE as One Layer in a Production Evaluation Stack

ROUGE remains valuable for what it actually measures: fast, deterministic, nearly free n-gram overlap that catches regressions before anything else runs. But treating it as a sole quality signal is where teams get burned. It cannot detect factual errors or hallucinations, it penalizes valid paraphrasing, and its correlation with human coherence judgments sits near zero. 

The research is clear that production-grade summarization evaluation requires layering ROUGE under semantic metrics and LLM-based judgment, sequenced by cost so every candidate gets a baseline check and the most expensive assessments run only where they add signal. That layered approach needs infrastructure that makes full-coverage evaluation affordable, not just technically possible. Galileo provides the evaluation platform that turns ROUGE from a standalone number into one signal inside a reliable quality stack:

• Luna-2 evaluation models: Purpose-built SLMs that run LLM-as-judge evaluation at $0.02 per million tokens and 152ms latency, making full-traffic assessment practical alongside ROUGE baselines.

• Metrics Engine: Over 20 out-of-the-box metrics spanning response quality, safety, and agentic performance, so teams evaluate coherence, factual consistency, and completeness where ROUGE cannot.

• CLHF: Continuous Learning via Human Feedback improves any LLM-powered metric with as few as one to two examples, increasing accuracy by 20-30% without retraining.

• CI/CD evaluation gates: Experiments run as unit tests in your pipeline, so ROUGE regressions and quality-threshold failures block the build instead of reaching production.

• Signals: Automatic failure detection that surfaces quality degradation, hallucination patterns, and drift across production traces without manual search.

• Eval-to-guardrail lifecycle: Offline evaluation criteria become production-enforced standards automatically, closing the gap between benchmark testing and live quality control.

Book a demo to see how Galileo layers LLM-as-judge evaluation on top of your existing metrics for production-scale text quality assessment.

FAQ

What is a ROUGE score in AI?

A ROUGE score quantifies how much an AI-generated text overlaps with a human-written reference, counting shared words, phrases, or word sequences. Chin-Yew Lin introduced the metric family in 2004 for automatic summarization evaluation, and it remains a standard baseline for summarization benchmarks.

What is a good ROUGE score for text summarization?

It depends on the dataset. On CNN/DailyMail, top abstractive systems reach ROUGE-1 in the mid-to-high 40s and ROUGE-L in the low 40s (0–100 scale), while the same architectures score several points lower on PubMed. Judge your model against published results on your specific dataset, not against a universal threshold.

How do I calculate ROUGE scores in Python?

Install the rouge-score package with pip, create a RougeScorer with your chosen variants, and call .score(reference, candidate). You get back precision, recall, and F1 for each variant. Prefer this Google-maintained package over the unmaintained py-rouge, and record your configuration (stemming, variant, scale) so results are reproducible.

What is the difference between ROUGE-1, ROUGE-2, and ROUGE-L?

ROUGE-1 matches individual words, ROUGE-2 matches two-word phrases, and ROUGE-L finds the longest common subsequence of words in order, gaps allowed. ROUGE-1 rewards content coverage, ROUGE-2 rewards matching two-word phrases, and ROUGE-L rewards preserved word order. Benchmarks conventionally report all three.

Should I use ROUGE or BLEU for evaluating my AI model?

Use ROUGE for summarization, where capturing the reference's content (recall) is the priority, and BLEU for machine translation, where precision against the reference matters more. For dialogue, creative generation, or any task where paraphrasing is legitimate, neither is enough on its own; add a semantic metric or LLM-as-judge layer.