--- title: "What Is Inverse Document Frequency? IDF Explained Simply | Ceisium" description: "Inverse document frequency, or IDF, helps search engines give more weight to rare query terms. Learn why rare words matter in keyword search." canonical: https://ceisium.com/blog/what-is-inverse-document-frequency/ --- > For the complete documentation and public content index, see [llms.txt](/llms.txt). This page is the Markdown alternative to the canonical HTML page. ![Rare query terms separating a small set of documents from common matches in an IDF ranking diagram.](https://ceisium.com/blog/idf-cover.webp) Rare query terms separating a small set of documents from common matches in an IDF ranking diagram. ![Rare query terms separating a small set of documents from common matches in an IDF ranking diagram.](https://ceisium.com/blog/idf-cover.webp) Rare query terms separating a small set of documents from common matches in an IDF ranking diagram. 1. [Home](https://ceisium.com/) 2. / 3. [Blog](https://ceisium.com/blog/) 4. / 5. What is Inverse Document Frequency? Published July 2, 2026 Updated July 2, 2026 # What is Inverse Document Frequency? Inverse document frequency helps search rank rare query words above words that appear everywhere. Learn the idea with a simple docs-search example. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=What%20is%20Inverse%20Document%20Frequency%3F&url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-inverse-document-frequency%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-inverse-document-frequency%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-inverse-document-frequency%2F&t=What%20is%20Inverse%20Document%20Frequency%3F) [Search](https://ceisium.com/blog/?tag=Search) [Learn](https://ceisium.com/blog/?tag=Learn) Inverse document frequency sounds like a formula term, but the idea starts with a very normal search problem. A user searches your docs for: TEXT ``` hmac webhook setup ``` The search engine can find pages that contain those words. That is matching. But after matching, it still has to rank the pages. This is where the problem appears. `setup` may show up across hundreds of docs pages. `webhook` narrows the field. `hmac` probably appears on only a few pages, and those pages are much more likely to answer the query. Inverse document frequency, or IDF, is the small ranking idea that handles this difference. It asks the search system to treat words that appear everywhere as weaker signals, and words that appear in only a few documents as stronger signals. ## What is inverse document frequency? IDF is a way to measure how much a term helps separate documents inside one searchable collection. Here, a document can be a docs page, product page, support article, API page, or any other page the search engine can return. The collection is the full set of those searchable documents. For each term, IDF asks: TEXT ``` How many documents contain this term? ``` That one count changes how much the term should count during ranking. Rare terms separate. Common terms blur. IDF is not judging whether a word is useful in normal language. It is judging whether the word separates documents inside this collection. In our example collection, the counts may look like this: Term Pages containing the term IDF intuition `setup` 700 pages weak signal `webhook` 90 pages useful signal `hmac` 8 pages strong signal The table does not say `setup` is useless. It says `setup` does not narrow the search very much. The term is still part of the query, but it should not dominate the score. ![Black llama-head blob using a lamp to separate rare hmac pages from many setup pages.](https://ceisium.com/blog/what-is-inverse-document-frequency/rare-words-separate.webp) IDF starts from the collection: a word that appears everywhere does less separating work than a word that appears on only a few pages. ![Black llama-head blob using a lamp to separate rare hmac pages from many setup pages.](https://ceisium.com/blog/what-is-inverse-document-frequency/rare-words-separate.webp) IDF starts from the collection: a word that appears everywhere does less separating work than a word that appears on only a few pages. ## Why not just count matching words? Imagine the query is: TEXT ``` hmac webhook setup ``` Now compare two pages. Page A: TEXT ``` Webhook setup guideConfigure webhook endpoints and delivery settings. ``` Page B: TEXT ``` Verify HMAC webhook signaturesUse HMAC to verify webhook payloads before accepting events. ``` Both pages match the query. Page A matches `webhook` and `setup`. Page B matches `hmac` and `webhook`. A simple word-counting system may only ask: TEXT ``` How many query words did this page match? ``` IDF adds a better question: TEXT ``` Which matched words are actually good at narrowing the result set? ``` That is the mechanism. IDF does not replace matching. It adds weight after matching, so a focused page with the specific term can beat a broader page with a more common term. ## The simple IDF formula The common textbook shape is: TEXT ``` idf(term) = log(total documents / documents containing term) ``` So if a collection has 1,000 pages: TEXT ``` idf("setup") = log(1000 / 700)idf("webhook") = log(1000 / 90)idf("hmac") = log(1000 / 8) ``` You do not need the exact decimal values here. The direction is the useful part: - `setup` appears on many pages, so its weight is low. - `webhook` appears on fewer pages, so its weight is higher. - `hmac` appears on very few pages, so its weight is highest. Different search systems use slightly different versions of the formula. Some add smoothing so the math behaves nicely when a term appears everywhere or almost nowhere. The mental model stays the same: TEXT ``` more documents contain the term -> lower IDFfewer documents contain the term -> higher IDF ``` ## IDF depends on the collection A word is not rare by itself. It is rare inside a specific collection. On a general marketing website, `api` may be rare. On a developer docs site, `api` may appear everywhere. So the search engine has to learn IDF from the actual pages being searched. When it builds the index, it counts how many documents contain each term. Rarity is local to the corpus. This is why IDF is a collection-level signal. The same word can get a different weight on two different websites. This also means the weights can change over time. If you add many new pages about webhooks, `webhook` becomes less rare in that collection than it was before. ## How IDF affects ranking Now switch to a second query: TEXT ``` api token reset ``` On many developer docs sites, `api` appears everywhere. `token` narrows the field. `reset` may point to an even smaller group of account or security pages. IDF turns those collection counts into weights: TEXT ``` api -> common, lower weighttoken -> more specific, medium weightreset -> more specific, higher weight ``` Now a page about resetting API tokens has a better chance of beating a generic API overview, even if the overview repeats `api` many times. ![Black llama-head blob placing api, token, and reset weights on an IDF scale.](https://ceisium.com/blog/what-is-inverse-document-frequency/idf-weight-scale.webp) IDF gives each query term a collection-level weight before the final ranking score is built. ![Black llama-head blob placing api, token, and reset weights on an IDF scale.](https://ceisium.com/blog/what-is-inverse-document-frequency/idf-weight-scale.webp) IDF gives each query term a collection-level weight before the final ranking score is built. This is why IDF is useful in technical search. Error codes, config keys, API fields, package names, CLI flags, and protocol names often carry the real intent. ## What IDF does not do It does not understand synonyms. If the user searches `sign in` and the document says `login`, IDF alone does not know those words mean similar things. It also does not prove that every rare word is important. A typo can be rare too. IDF only says the term is unusual in this collection; the rest of the ranking system still has to decide whether the document is actually useful. That is why real search systems combine IDF with other signals: - whether the query words appear in the document - how often they appear - where they appear, such as title or body - how long or focused the document is - other lexical or semantic retrieval steps IDF is one part of the ranking story. ## How IDF connects to TF-IDF and BM25 IDF is the second half of [TF-IDF ![](https://ceisium.com/blog/tf-idf-cover.webp)Search / Jul 1, 2026 TF-IDF: The Search Ranking Idea Behind BM25 TF-IDF is a classic lexical ranking method that scores documents using repeated query words and rare query words. Learn how it works, where it helps, and how BM25 addresses some of its limits.](https://ceisium.com/blog/what-is-tf-idf/) . TF-IDF combines: - term frequency: how often the query term appears in this document - inverse document frequency: how rare the query term is across all documents BM25 also uses an IDF-style rarity signal. But [BM25 ![](https://ceisium.com/blog/bm25-cover-ceisium-bottom-right-aligned.webp)Search / Jun 24, 2026 What is BM25? A Plain English Guide BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF.](https://ceisium.com/blog/what-is-bm25/) adds more ranking controls, especially term frequency saturation and document length normalization. So if you are learning keyword ranking, the path is: TEXT ``` word matching -> IDF -> TF-IDF -> BM25 ``` IDF is the point where search stops treating every matching word as equally useful. ## Summary Inverse document frequency helps search engines answer one ranking question: TEXT ``` Which matching words actually separate the useful pages from the noisy pages? ``` Words like `setup`, `guide`, or `page` can still matter, but they appear in too many places to carry the whole score. Terms like `hmac`, `retry-after`, `sso`, or a specific API field often narrow the result set faster. That is why IDF is one of the basic ideas behind TF-IDF, BM25, and many keyword search systems. Thanks for reading. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=What%20is%20Inverse%20Document%20Frequency%3F&url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-inverse-document-frequency%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-inverse-document-frequency%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-inverse-document-frequency%2F&t=What%20is%20Inverse%20Document%20Frequency%3F) Copy image[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-inverse-document-frequency%2F)[](https://twitter.com/intent/tweet?text=%E2%80%9CWhat%20is%20Inverse%20Document%20Frequency%3F%E2%80%9D&url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-inverse-document-frequency%2F) **Yuvraj** _Building Ceisium and working on growth @ SigNoz_ ceisium.com ## Next reads [ ![TF-IDF combines term frequency and inverse document frequency to rank matching documents.](https://ceisium.com/blog/tf-idf-cover.webp) TF-IDF: The Search Ranking Idea Behind BM25 TF-IDF is a classic lexical ranking method that scores documents using repeated query words and rare query words. Learn how it works, where it helps, and how BM25 addresses some of its limits. ](https://ceisium.com/blog/what-is-tf-idf/)[ ![BM25 scores query terms using rarity, repetition, and document length signals.](https://ceisium.com/blog/bm25-cover-ceisium-bottom-right-aligned.webp) What is BM25? A Plain English Guide BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF. ](https://ceisium.com/blog/what-is-bm25/)