Document Length Normalization: Why Long Pages Need Context

A user searches documentation for:

rotate api key

Two pages match.

The first page is a focused, 200-word guide named Rotate an API key. It contains the phrase two times.

The second page is a 2,000-word security guide. It contains the phrase four times and covers many other security topics.

A simple match count makes the long guide look better. It has four matches instead of two.

However, the long guide has ten times more text. Its length gives it more opportunities to contain the query terms.

The higher match count does not prove that the long guide is better. Its extra text gives it more opportunities to match.

What is document length normalization?

Document length normalization adds context to term counts. It compares term frequency with the length of the scored text.

A short scoring unit can focus on one topic. However, BM25 sees only the term counts and field length.

BM25 is the main example in this article. Other ranking methods can handle length differently.

The calculations in this article apply to BM25.

What counts as a document?

The word document can mean a complete page. However, one indexed record can represent a page, section, product, or support article.

A query can also score one field in that record, such as the title or body.

In this article, the scoring unit is the record and field used in the term-frequency calculation.

Search systems analyze text before ranking. Thus, length does not always mean the number of visible words.

An indexed token is one searchable unit after analysis. The analysis rules depend on the search engine.

We will use visible words in the first example. This makes the difference easy to see.

For the calculation, assume the body fields contain 200 and 2,000 indexed tokens.

The query has three terms. A complete BM25 score usually adds one contribution for each term.

To isolate document length, we will calculate only the contribution from rotate. We will keep all other evidence equal.

Raw repetition needs length context

Term repetition can be useful evidence. Several uses of rotate can show that a page is about rotation.

However, repetition can have two causes:

1. The page has a strong focus on the term.
2. The page has enough text to contain the term several times.

The first example shows this difference:

Focused page2 mentions / 200 words= 1 mention per 100 wordsLong guide4 mentions / 2,000 words= 1 mention per 500 words

Term density makes the difference easy to see. BM25 does not calculate this ratio.

Simple density can give too much value to very small pages. Each added nonmatching word also reduces the ratio directly.

BM25 uses two controls instead. Term-frequency saturation reduces the value of each additional match. Length normalization adjusts for relative length.

First, give both scoring units the same term frequency:

Short unit: 2 matches in 200 indexed tokensLong unit:  2 matches in 2,000 indexed tokens

Now only the length changes. The field, term rarity, and other query evidence stay equal.

Length is relative to the collection

A 2,000-token guide is long in a collection of short help pages. It can be normal in a collection of technical references.

BM25 compares the field length with the average field length. The formula usually calls this average avgdl.

Assume an average of 800 indexed tokens:

200 / 800   = 0.25  -> shorter than average800 / 800   = 1.00  -> average length2,000 / 800 = 2.50  -> longer than average

avgdl uses every record in the selected statistics scope. It does not use only the current results.

Index design determines whether avgdl compares similar records. Title fields and body fields can also have separate statistics.

Thus, document length is relative. No universal word count makes a page too long.

Equal term frequency does not mean equal context. The same two matches occur in scoring units with very different lengths.

How BM25 uses relative length

Our BM25 guide Search / Jun 24, 2026 What is BM25? A Plain English Guide BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF. explains the complete ranking method. This section isolates the length component.

Start with the length factor:

length factor = 1 - b + b * (dl / avgdl)

dl is the indexed field length. avgdl is the average length in the same statistics scope.

The parameter b controls the strength of the length adjustment.

When b is greater than zero:

• A short field has a smaller length factor.
• An average field has a length factor of 1.
• A long field has a larger length factor.

BM25 puts this factor in the denominator of the term-frequency fraction:

term contribution before IDF =(tf * (k1 + 1)) / (tf + k1 * length factor)

A larger length factor makes the denominator larger. The same numerator divided by a larger denominator gives a smaller contribution.

A common form of one BM25 term contribution is:

term contribution =IDF(t) * ((tf * (k1 + 1)) /          (tf + k1 * (1 - b + b * dl / avgdl)))

The other symbols have these meanings:

tf  = frequency of the query term in the scored fieldk1  = control for term-frequency saturationIDF = rarity of the term in the same statistics scope

Our two example fields use the same statistics scope. Therefore, the tracked term has the same IDF in both fields.

The TF-IDF guide Search / Jul 1, 2026 TF-IDF: The Search Ranking Idea Behind BM25 TF-IDF is a classic lexical ranking method that scores documents using repeated query words and rare query words. Learn how it works, where it helps, and how BM25 addresses some of its limits. explains term rarity.

Some implementations omit or absorb rank-equivalent constants such as k1 + 1. This change does not alter the order for one BM25 configuration.

A small calculation

Use these illustrative values:

k1    = 1.2b     = 0.75avgdl = 800 tokenstf    = 2

For the 200-token page:

length factor= 1 - 0.75 + 0.75 * (200 / 800)= 0.4375term contribution before IDF= (2 * 2.2) / (2 + 1.2 * 0.4375)≈ 1.74

For the 2,000-token guide:

length factor= 1 - 0.75 + 0.75 * (2,000 / 800)= 2.125term contribution before IDF= (2 * 2.2) / (2 + 1.2 * 2.125)≈ 0.97

The query term and its frequency are equal. The long scoring unit gets a smaller contribution because it contains much more indexed text.

These values show only the length effect for one term. They are not complete document scores.

Search engines can also report BM25 scores on different numeric scales.

The b parameter controls sensitivity

The parameter b controls the sensitivity to relative length.

At b = 0:

length factor = 1

This setting removes length normalization. It does not remove IDF or the k1 saturation behavior.

At b = 1:

length factor = dl / avgdl

At b = 1, BM25 uses the complete length ratio. The BM25 score still does not prove that a page is relevant.

A value between zero and one gives a smaller adjustment. Lucene 10.4.0 uses 0.75 by default.

Lucene’s default is not the best value for every search system.

When dl equals avgdl, the factor is always 1. The selected b value does not change this result.

The b parameter controls sensitivity to relative length. The average-length unit stays fixed while shorter and longer units change.

Return to the original term counts

The first calculation used equal term frequencies. Now restore the original counts:

Focused page: 2 matches in 200 tokensLong guide:   4 matches in 2,000 tokens

The focused page contribution stays near 1.74. The long guide contribution increases from about 0.97 to about 1.34.

The extra two matches help the long guide. They do not remove the length context.

The long guide can still rank first when it has stronger evidence. It can match more terms, rarer terms, or other ranking signals.

Long pages are not bad results. Length normalization changes how a lexical ranker interprets repetition. It does not make shortness equal to relevance.

The scoring unit changes the calculation

The index defines the scoring unit. A change to the record or field boundary changes the length comparison.

Assume the 2,000-word guide is one indexed page. Its four uses of rotate occur in one long body field.

Now split the same guide into six records of approximately 300 tokens. The Rotate an API key section gets its own body field.

The source page did not become shorter. The scoring unit became shorter.

The chunk has its own dl. Its avgdl comes from all chunks in the selected statistics scope.

Poor chunk boundaries can separate an answer from necessary context. Thus, smaller scoring units are not always better.

Fields can work in the same way. A title can use title-length statistics, while the body uses body-length statistics.

The exact behavior depends on the index and search engine.

Limits of length normalization

Length normalization uses a small set of numbers. It does not know why a page is long.

A page can be long because it is repetitive. Another page can be long because it gives a complete explanation.

Both pages can have the same dl.

One average can also hide different content types. An index can contain 30-token products and 3,000-token tutorials.

Separate fields, indexes, or content types can make the comparison clearer.

Boilerplate can also increase length. Navigation, legal text, and generated metadata add tokens when extraction puts them in the scored field.

Finally, BM25 is lexical. It does not directly understand intent, freshness, authority, or different phrases with the same meaning.

Search systems can add field weights, filters, semantic retrieval, or reranking.

Length normalization reduces the extra matching opportunity that comes from longer text. It does not rank documents by itself.

Questions for your search index

Before you change b, answer these questions:

1. What is the scoring unit? Is it a page, chunk, record, title, or body?
2. Which analyzed tokens count? Does the scored field contain boilerplate?
3. Which units produce avgdl? Does the scope mix very different content types?
4. What do labeled relevance judgments show? Do long pages win because of useful coverage or extra matching opportunities?

The last question is the most important. Test the selected b value with real queries and labeled results.

Frequently asked questions

Does BM25 always prefer shorter documents?

No. A result still needs matching query terms. Length changes only the contribution from those matches.

Is there an ideal document length for search?

No. BM25 compares a field with its statistics scope. It does not use a universal word-count target.

How does chunking affect length normalization?

Chunking changes the indexed records and field lengths. It can change both dl and the records that produce avgdl.

Conclusion

Do not use length normalization as a reason to make all pages short. Use it to understand what the ranker compares.

Define the scoring unit. Examine the indexed tokens and the statistics scope. Then tune the system with representative searches.

Sources

• Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze, Introduction to Information Retrieval: Pivoted normalized document length opens in a new tab
• Stephen Robertson and Hugo Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond opens in a new tab
• Apache Lucene, BM25Similarity documentation opens in a new tab
• Apache Lucene, BM25Similarity source opens in a new tab
• Elasticsearch, Practical BM25 — Part 2 opens in a new tab