--- title: "What Is Document Length Normalization in Search? | Ceisium" description: "Document length normalization helps search ranking compare focused pages with long guides fairly. Learn how average length and the BM25 b parameter work." canonical: https://ceisium.com/blog/document-length-normalization/ --- > For the complete documentation and public content index, see [llms.txt](/llms.txt). This page is the Markdown alternative to the canonical HTML page. ![Equal query-term matches sit in short and long document lanes against one average-length reference.](https://ceisium.com/blog/document-length-normalization-cover.webp) Equal query-term matches sit in short and long document lanes against one average-length reference. ![Equal query-term matches sit in short and long document lanes against one average-length reference.](https://ceisium.com/blog/document-length-normalization-cover.webp) Equal query-term matches sit in short and long document lanes against one average-length reference. 1. [Home](https://ceisium.com/) 2. / 3. [Blog](https://ceisium.com/blog/) 4. / 5. Document Length Normalization: Why Long Pages Need Context Published July 21, 2026 Updated August 9, 2026 # Document Length Normalization: Why Long Pages Need Context Learn why long pages get more chances to match a search query, how BM25 adjusts for relative document length, and why shorter is not automatically better. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=Document%20Length%20Normalization%3A%20Why%20Long%20Pages%20Need%20Context&url=https%3A%2F%2Fceisium.com%2Fblog%2Fdocument-length-normalization%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fblog%2Fdocument-length-normalization%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fblog%2Fdocument-length-normalization%2F&t=Document%20Length%20Normalization%3A%20Why%20Long%20Pages%20Need%20Context) [Search](https://ceisium.com/blog/?tag=Search) [Learn](https://ceisium.com/blog/?tag=Learn) A user searches documentation for: TEXT ``` rotate api key ``` Two pages match. The first page is a focused, 200-word guide named **Rotate an API key**. It contains the phrase two times. The second page is a 2,000-word security guide. It contains the phrase four times and covers many other security topics. A simple match count makes the long guide look better. It has four matches instead of two. However, the long guide has ten times more text. Its length gives it more opportunities to contain the query terms. The higher match count does not prove that the long guide is better. Its extra text gives it more opportunities to match. ## What is document length normalization? Document length normalization adds context to term counts. It compares term frequency with the length of the scored text. A short scoring unit can focus on one topic. However, BM25 sees only the term counts and field length. BM25 is the main example in this article. Other ranking methods can handle length differently. The calculations in this article apply to BM25. ## What counts as a document? The word _document_ can mean a complete page. However, one indexed record can represent a page, section, product, or support article. A query can also score one field in that record, such as the title or body. In this article, the **scoring unit** is the record and field used in the term-frequency calculation. Search systems analyze text before ranking. Thus, length does not always mean the number of visible words. An **indexed token** is one searchable unit after analysis. The analysis rules depend on the search engine. We will use visible words in the first example. This makes the difference easy to see. For the calculation, assume the body fields contain 200 and 2,000 indexed tokens. The query has three terms. A complete BM25 score usually adds one contribution for each term. To isolate document length, we will calculate only the contribution from `rotate`. We will keep all other evidence equal. ## Raw repetition needs length context Term repetition can be useful evidence. Several uses of `rotate` can show that a page is about rotation. However, repetition can have two causes: 1. The page has a strong focus on the term. 2. The page has enough text to contain the term several times. The first example shows this difference: TEXT ``` Focused page2 mentions / 200 words= 1 mention per 100 wordsLong guide4 mentions / 2,000 words= 1 mention per 500 words ``` Term density makes the difference easy to see. BM25 does not calculate this ratio. Simple density can give too much value to very small pages. Each added nonmatching word also reduces the ratio directly. BM25 uses two controls instead. Term-frequency saturation reduces the value of each additional match. Length normalization adjusts for relative length. First, give both scoring units the same term frequency: TEXT ``` Short unit: 2 matches in 200 indexed tokensLong unit: 2 matches in 2,000 indexed tokens ``` Now only the length changes. The field, term rarity, and other query evidence stay equal. ## Length is relative to the collection A 2,000-token guide is long in a collection of short help pages. It can be normal in a collection of technical references. BM25 compares the field length with the average field length. The formula usually calls this average `avgdl`. Assume an average of 800 indexed tokens: TEXT ``` 200 / 800 = 0.25 -> shorter than average800 / 800 = 1.00 -> average length2,000 / 800 = 2.50 -> longer than average ``` `avgdl` uses every record in the selected statistics scope. It does not use only the current results. Index design determines whether `avgdl` compares similar records. Title fields and body fields can also have separate statistics. Thus, document length is relative. No universal word count makes a page too long. ![Black llama-head blob measures a short document and a long document with the same two query matches against one average-length line.](https://ceisium.com/blog/document-length-normalization/01-relative-length-context.webp) Equal term frequency does not mean equal context. The same two matches occur in scoring units with very different lengths. ![Black llama-head blob measures a short document and a long document with the same two query matches against one average-length line.](https://ceisium.com/blog/document-length-normalization/01-relative-length-context.webp) Equal term frequency does not mean equal context. The same two matches occur in scoring units with very different lengths. ## How BM25 uses relative length Our [BM25 guide ![](https://ceisium.com/blog/bm25-cover-ceisium-bottom-right-aligned.webp)Search / Jun 24, 2026 What is BM25? A Plain English Guide BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF.](https://ceisium.com/blog/what-is-bm25/) explains the complete ranking method. This section isolates the length component. Start with the length factor: TEXT ``` length factor = 1 - b + b * (dl / avgdl) ``` `dl` is the indexed field length. `avgdl` is the average length in the same statistics scope. The parameter `b` controls the strength of the length adjustment. When `b` is greater than zero: - A short field has a smaller length factor. - An average field has a length factor of `1`. - A long field has a larger length factor. BM25 puts this factor in the denominator of the term-frequency fraction: TEXT ``` term contribution before IDF =(tf * (k1 + 1)) / (tf + k1 * length factor) ``` A larger length factor makes the denominator larger. The same numerator divided by a larger denominator gives a smaller contribution. A common form of one BM25 term contribution is: TEXT ``` term contribution =IDF(t) * ((tf * (k1 + 1)) / (tf + k1 * (1 - b + b * dl / avgdl))) ``` The other symbols have these meanings: TEXT ``` tf = frequency of the query term in the scored fieldk1 = control for term-frequency saturationIDF = rarity of the term in the same statistics scope ``` Our two example fields use the same statistics scope. Therefore, the tracked term has the same IDF in both fields. The [TF-IDF guide ![](https://ceisium.com/blog/tf-idf-cover.webp)Search / Jul 1, 2026 TF-IDF: The Search Ranking Idea Behind BM25 TF-IDF is a classic lexical ranking method that scores documents using repeated query words and rare query words. Learn how it works, where it helps, and how BM25 addresses some of its limits.](https://ceisium.com/blog/what-is-tf-idf/) explains term rarity. Some implementations omit or absorb rank-equivalent constants such as `k1 + 1`. This change does not alter the order for one BM25 configuration. ## A small calculation Use these illustrative values: TEXT ``` k1 = 1.2b = 0.75avgdl = 800 tokenstf = 2 ``` For the 200-token page: TEXT ``` length factor= 1 - 0.75 + 0.75 * (200 / 800)= 0.4375term contribution before IDF= (2 * 2.2) / (2 + 1.2 * 0.4375)≈ 1.74 ``` For the 2,000-token guide: TEXT ``` length factor= 1 - 0.75 + 0.75 * (2,000 / 800)= 2.125term contribution before IDF= (2 * 2.2) / (2 + 1.2 * 2.125)≈ 0.97 ``` The query term and its frequency are equal. The long scoring unit gets a smaller contribution because it contains much more indexed text. These values show only the length effect for one term. They are not complete document scores. Search engines can also report BM25 scores on different numeric scales. ## The b parameter controls sensitivity The parameter `b` controls the sensitivity to relative length. At `b = 0`: TEXT ``` length factor = 1 ``` This setting removes length normalization. It does not remove IDF or the `k1` saturation behavior. At `b = 1`: TEXT ``` length factor = dl / avgdl ``` At `b = 1`, BM25 uses the complete length ratio. The BM25 score still does not prove that a page is relevant. A value between zero and one gives a smaller adjustment. Lucene 10.4.0 uses `0.75` by default. Lucene’s default is not the best value for every search system. When `dl` equals `avgdl`, the factor is always `1`. The selected `b` value does not change this result. The average-length unit stays fixed. At the average, dl divided by avgdl is 1. Every b value then gives a length factor of 1. ![Black llama-head blob moves a control while short, average, and long document strips change around a fixed center reference.](https://ceisium.com/blog/document-length-normalization/02-b-sensitivity.webp) The b parameter controls sensitivity to relative length. The average-length unit stays fixed while shorter and longer units change. ![Black llama-head blob moves a control while short, average, and long document strips change around a fixed center reference.](https://ceisium.com/blog/document-length-normalization/02-b-sensitivity.webp) The b parameter controls sensitivity to relative length. The average-length unit stays fixed while shorter and longer units change. ## Return to the original term counts The first calculation used equal term frequencies. Now restore the original counts: TEXT ``` Focused page: 2 matches in 200 tokensLong guide: 4 matches in 2,000 tokens ``` The focused page contribution stays near `1.74`. The long guide contribution increases from about `0.97` to about `1.34`. The extra two matches help the long guide. They do not remove the length context. The long guide can still rank first when it has stronger evidence. It can match more terms, rarer terms, or other ranking signals. Long pages are not bad results. Length normalization changes how a lexical ranker interprets repetition. It does not make shortness equal to relevance. ## The scoring unit changes the calculation The index defines the scoring unit. A change to the record or field boundary changes the length comparison. Assume the 2,000-word guide is one indexed page. Its four uses of `rotate` occur in one long body field. Now split the same guide into six records of approximately 300 tokens. The **Rotate an API key** section gets its own body field. The source page did not become shorter. The scoring unit became shorter. The chunk has its own `dl`. Its `avgdl` comes from all chunks in the selected statistics scope. Poor chunk boundaries can separate an answer from necessary context. Thus, smaller scoring units are not always better. Fields can work in the same way. A title can use title-length statistics, while the body uses body-length statistics. The exact behavior depends on the index and search engine. ## Limits of length normalization Length normalization uses a small set of numbers. It does not know why a page is long. A page can be long because it is repetitive. Another page can be long because it gives a complete explanation. Both pages can have the same `dl`. One average can also hide different content types. An index can contain 30-token products and 3,000-token tutorials. Separate fields, indexes, or content types can make the comparison clearer. Boilerplate can also increase length. Navigation, legal text, and generated metadata add tokens when extraction puts them in the scored field. Finally, BM25 is lexical. It does not directly understand intent, freshness, authority, or different phrases with the same meaning. Search systems can add field weights, filters, semantic retrieval, or reranking. Length normalization reduces the extra matching opportunity that comes from longer text. It does not rank documents by itself. ## Questions for your search index Before you change `b`, answer these questions: 1. **What is the scoring unit?** Is it a page, chunk, record, title, or body? 2. **Which analyzed tokens count?** Does the scored field contain boilerplate? 3. **Which units produce `avgdl`?** Does the scope mix very different content types? 4. **What do labeled relevance judgments show?** Do long pages win because of useful coverage or extra matching opportunities? The last question is the most important. Test the selected `b` value with real queries and labeled results. ## Frequently asked questions ### Does BM25 always prefer shorter documents? No. A result still needs matching query terms. Length changes only the contribution from those matches. ### Is there an ideal document length for search? No. BM25 compares a field with its statistics scope. It does not use a universal word-count target. ### How does chunking affect length normalization? Chunking changes the indexed records and field lengths. It can change both `dl` and the records that produce `avgdl`. ## Conclusion Do not use length normalization as a reason to make all pages short. Use it to understand what the ranker compares. Define the scoring unit. Examine the indexed tokens and the statistics scope. Then tune the system with representative searches. ## Sources - Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze, _Introduction to Information Retrieval_: [Pivoted normalized document length opens in a new tab](https://nlp.stanford.edu/IR-book/html/htmledition/pivoted-normalized-document-length-1.html) - Stephen Robertson and Hugo Zaragoza, [_The Probabilistic Relevance Framework: BM25 and Beyond_ opens in a new tab](https://doi.org/10.1561/1500000019) - Apache Lucene, [`BM25Similarity` documentation opens in a new tab](https://lucene.apache.org/core/10_4_0/core/org/apache/lucene/search/similarities/BM25Similarity.html) - Apache Lucene, [`BM25Similarity` source opens in a new tab](https://github.com/apache/lucene/blob/releases/lucene/10.4.0/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java) - Elasticsearch, [_Practical BM25 — Part 2_ opens in a new tab](https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables) [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=Document%20Length%20Normalization%3A%20Why%20Long%20Pages%20Need%20Context&url=https%3A%2F%2Fceisium.com%2Fblog%2Fdocument-length-normalization%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fblog%2Fdocument-length-normalization%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fblog%2Fdocument-length-normalization%2F&t=Document%20Length%20Normalization%3A%20Why%20Long%20Pages%20Need%20Context) Copy image[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fblog%2Fdocument-length-normalization%2F)[](https://twitter.com/intent/tweet?text=%E2%80%9CDocument%20Length%20Normalization%3A%20Why%20Long%20Pages%20Need%20Context%E2%80%9D&url=https%3A%2F%2Fceisium.com%2Fblog%2Fdocument-length-normalization%2F) **Yuvraj** _Building Ceisium and working on growth @ SigNoz_ ceisium.com ## Next reads [ ![BM25 scores query terms using rarity, repetition, and document length signals.](https://ceisium.com/blog/bm25-cover-ceisium-bottom-right-aligned.webp) What is BM25? A Plain English Guide BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF. ](https://ceisium.com/blog/what-is-bm25/)[ ![TF-IDF combines term frequency and inverse document frequency to rank matching documents.](https://ceisium.com/blog/tf-idf-cover.webp) TF-IDF: The Search Ranking Idea Behind BM25 TF-IDF is a classic lexical ranking method that scores documents using repeated query words and rare query words. Learn how it works, where it helps, and how BM25 addresses some of its limits. ](https://ceisium.com/blog/what-is-tf-idf/)