--- title: "What Is TF-IDF? Term Frequency-Inverse Document Frequency Explained | Ceisium" description: "TF-IDF ranks search results by combining term frequency with inverse document frequency. Learn the plain-English model, formula, examples, and how BM25 builds on similar lexical signals." canonical: https://ceisium.com/blog/what-is-tf-idf/ --- > For the complete documentation and public content index, see [llms.txt](/llms.txt). This page is the Markdown alternative to the canonical HTML page. ![TF-IDF combines term frequency and inverse document frequency to rank matching documents.](https://ceisium.com/blog/tf-idf-cover.webp) TF-IDF combines term frequency and inverse document frequency to rank matching documents. ![TF-IDF combines term frequency and inverse document frequency to rank matching documents.](https://ceisium.com/blog/tf-idf-cover.webp) TF-IDF combines term frequency and inverse document frequency to rank matching documents. 1. [Home](https://ceisium.com/) 2. / 3. [Blog](https://ceisium.com/blog/) 4. / 5. TF-IDF: The Search Ranking Idea Behind BM25 Published July 1, 2026 Updated July 1, 2026 # TF-IDF: The Search Ranking Idea Behind BM25 TF-IDF is a classic lexical ranking method that scores documents using repeated query words and rare query words. Learn how it works, where it helps, and how BM25 addresses some of its limits. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=TF-IDF%3A%20The%20Search%20Ranking%20Idea%20Behind%20BM25&url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-tf-idf%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-tf-idf%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-tf-idf%2F&t=TF-IDF%3A%20The%20Search%20Ranking%20Idea%20Behind%20BM25) [Search](https://ceisium.com/blog/?tag=Search) [Learn](https://ceisium.com/blog/?tag=Learn) TF-IDF is a simple way to answer a search-ranking question: which page uses the user’s words in the most useful way? Search ranking starts with a small practical problem. A user types: TEXT ``` reset api token ``` The search system may have hundreds or thousands of docs, support articles, API pages, changelog entries, and product pages. Many of them contain one of those words, but only a few are probably the right result. Here, a document means one searchable page, the collection means all searchable pages, and a query term means one word from the user’s search. TF-IDF is one way to sort those pages. It is not semantic search, and it does not understand the query like a person. It asks a simpler question: TEXT ``` Which pages use the query words, and which of those words are specific enough to matter? ``` That small question is enough to make many exact-word searches work. ## What is TF-IDF? TF-IDF means: TEXT ``` term frequency x inverse document frequency ``` It is a ranking method for lexical search. Lexical search works with matching words. If the user searches for `webhook signature`, the system looks for pages that contain `webhook`, `signature`, or both. TF-IDF gives each matching document a score for this query. A higher score means the document looks more relevant for that search. The same page can get different TF-IDF scores for different searches. The two parts do different jobs: - Term frequency asks how often the query word appears in this document. - Inverse document frequency asks how rare that word is across the whole collection. Term frequency is local. Inverse document frequency is global. TF-IDF is not one magic score. It is two simple signals multiplied together. That is the core idea. A word matters more when it appears in the current document and does not appear everywhere else. ![Black llama-head blob pulling term count and word rarity levers into one TF-IDF score.](https://ceisium.com/blog/what-is-tf-idf/two-score-levers.webp) TF-IDF has two levers: how often the word appears in this page, and how rare that word is across the whole site. ![Black llama-head blob pulling term count and word rarity levers into one TF-IDF score.](https://ceisium.com/blog/what-is-tf-idf/two-score-levers.webp) TF-IDF has two levers: how often the word appears in this page, and how rare that word is across the whole site. ## Why not just count words? Let us start with a very simple search system. The user searches: TEXT ``` api token ``` We have two pages. Page A: TEXT ``` API tokensCreate, rotate, and revoke API tokens. ``` Page B: TEXT ``` Authentication guideThis guide explains API access, token scopes, token rotation,token permissions, API clients, and API examples. ``` If we only count words, Page B can look stronger because it says `api` and `token` many times. Page B wins raw count because it repeats both words. But Page A may be the better result because it is short and focused. Raw word counting has another problem. Common words appear everywhere: TEXT ``` docspageguideusesetup ``` If a word appears on almost every page, matching it does not tell us much. A page that contains `docs` is not special. A page that contains `hmac` or `retry-after` is much more specific. TF-IDF fixes part of this by making common words quieter and specific words louder. ## Term frequency Term frequency is the easy half. It asks: TEXT ``` How often does this query term appear in this document? ``` If someone searches: TEXT ``` webhook signature ``` Then a page that talks about webhook signatures several times is probably more relevant than a page that mentions the phrase once in a footer. For example: Page `webhook` mentions `signature` mentions Verify webhook signatures 8 6 Webhook delivery retries 10 1 Integrations overview 1 0 For now, ignore page length. We are only looking at whether both query words appear strongly. In this tiny example, the first page looks strongest because both query terms appear strongly there. This is the useful part of term frequency. Repeated words can be evidence. But we should be careful. Repetition is evidence, not proof. A page can repeat a word because it is long, noisy, or covering many topics. ## Inverse document frequency Inverse document frequency is the more interesting half. It asks: TEXT ``` How rare is this term across all documents? ``` Imagine a docs site with 1,000 pages. Rarity does not prove relevance. It changes how much a match should count. Term Pages containing the term What it means `docs` 950 tiny signal `webhook` 120 useful signal `hmac` 8 strong signal If the query is: TEXT ``` webhook hmac signature ``` then `hmac` should matter a lot. It appears on very few pages, so it helps separate the specific pages from the general webhook pages. This is why IDF is useful for product and technical search. Product names, API fields, error codes, config keys, CLI flags, and protocol names often carry the real intent. IDF is about separation. A rare word is not automatically important. It is important only when it appears in the user's query and helps separate relevant pages. ## The TF-IDF formula At a high level, TF-IDF looks like this: TEXT ``` tf-idf(term, document) = tf(term, document) x idf(term) ``` If the query has more than one term, the search system calculates a score for each query term and adds the scores together. TEXT ``` score(document, query) = tfidf("webhook", document) + tfidf("signature", document) ``` Different systems use different versions of term frequency and IDF. Some use raw counts, some use logarithms, and some normalize vectors. You do not need every variant to understand the ranking idea. The plain-English version is enough: TEXT ``` score = query words used in this document x how rare those query words are across the collection ``` Common words whisper. Specific words speak loudly. ## A small TF-IDF example Let us use a tiny docs search example. The user searches: TEXT ``` reset api token ``` The collection has three pages. Page What it says A: Reset API tokens Short API page about rotating and resetting API tokens B: Complete authentication guide Long guide with many mentions of API clients and tokens C: Account settings overview General page that mentions reset once Now look at the query terms: - `api` may appear on many pages. - `token` appears on fewer pages. - `reset` may appear on a smaller set of settings and recovery pages. The ranking might look like this: Rank Page Why 1 A: Reset API tokens Matches all three query terms on a focused page 2 B: Complete authentication guide Matches `api` and `token`, but the page is broader 3 C: Account settings overview Mentions `reset`, but misses the stronger API token context TF-IDF is meant to prefer Page A because the query terms are concentrated there, and the more specific terms help separate it from general pages. This is why TF-IDF is a useful baseline. It can rank exact terminology before the search system understands anything about meaning. ## Where TF-IDF works well TF-IDF works well when the user and the document use the same words. That happens a lot in technical search: - `api token` - `webhook signature` - `429 retry-after` - `rate limit` - `sso setup` - `npm install` - `connection refused` - `delete workspace` These are good lexical queries. The exact words matter. In docs and product search, exact words are not a small detail. They are often the whole point. If a user searches for an error code, an API field, a product setting, or a command, they expect those words to match. TF-IDF is good when the query vocabulary and document vocabulary overlap. ## Where TF-IDF struggles TF-IDF struggles when the meaning matches but the words do not. For example: User searches Useful page says `sign in problem` `login troubleshooting` `single sign on` `SSO` `payment failed` `invoice collection error` `make search faster` `latency optimization` A human can see the connection. TF-IDF may not. It mostly sees word overlap. If the words are different, the score can stay low. ![Black llama-head blob holding a small bridge between phrases that mean similar things but use different words.](https://ceisium.com/blog/what-is-tf-idf/lexical-miss.webp) Lexical ranking is strong when words match. It needs help when the useful page says the same idea with different words. ![Black llama-head blob holding a small bridge between phrases that mean similar things but use different words.](https://ceisium.com/blog/what-is-tf-idf/lexical-miss.webp) Lexical ranking is strong when words match. It needs help when the useful page says the same idea with different words. Search systems can work around this with synonyms, stemming, query rewriting, semantic retrieval, or hybrid search. You do not need all of those pieces yet; they are different ways to add help around word matching. TF-IDF itself is still a word-based method. ## Do titles and headings matter? The core TF-IDF idea does not automatically know that a title is more important than the body. But real search systems usually have fields: - Title - URL - Headings - Body - Meta description - Anchor text A match in the title often deserves more weight than a match deep in the body. For example: TEXT ``` title score x title weight+ heading score x heading weight+ body score x body weight ``` This is a layer around the lexical scoring method. The scoring method handles term frequency and rarity. Field weighting tells the system where the match happened. ## How BM25 addresses TF-IDF’s limits BM25 keeps a useful TF-IDF-style idea: TEXT ``` important pages use the query terms, especially specific terms ``` But BM25 adds better controls for two problems: - Repetition should not grow forever. - Long documents should not win just because they have more words. You can read the [plain-English BM25 guide ![](https://ceisium.com/blog/bm25-cover-ceisium-bottom-right-aligned.webp)Search / Jun 24, 2026 What is BM25? A Plain English Guide BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF.](https://ceisium.com/blog/what-is-bm25/) for the full walkthrough. The short version is that BM25 is still lexical ranking, but it is more careful about the repetition and length problems simple TF-IDF leaves unfinished. ## Repetition should flatten out Suppose the user searches: TEXT ``` token ``` A page that says `token` once gives us useful evidence. A page that says it five times gives stronger evidence. But a page that says `token` fifty times should not be ten times better than the page that says it five times. At some point, the search system should say: TEXT ``` Okay, this page is clearly about tokens. More repetition is not adding much. ``` BM25 handles this with term frequency saturation. ![Black llama-head blob operating a spring gauge where one token mention helps, five are strong, and fifty stop growing the score.](https://ceisium.com/blog/what-is-tf-idf/repetition-without-saturation.webp) BM25 keeps the useful repetition signal, but it flattens the gain after the page has already proved the term matters. ![Black llama-head blob operating a spring gauge where one token mention helps, five are strong, and fifty stop growing the score.](https://ceisium.com/blog/what-is-tf-idf/repetition-without-saturation.webp) BM25 keeps the useful repetition signal, but it flattens the gain after the page has already proved the term matters. Many real TF-IDF implementations soften repetition too, but BM25 is the common named model built around this saturation behavior. ## Long documents need fair treatment Long pages have more chances to match a query. A 10,000-word guide might mention `billing` once, `invoice` once, and `retry` once because it covers everything. A short support page might focus only on failed invoice retries. If we do not adjust for length, the long guide can get too much credit. BM25 includes document length normalization. A match inside a short focused page can be stronger than the same match buried inside a long page. This does not mean long pages are bad. A long guide can still be the best answer. The goal is only to avoid rewarding length by accident. ## Is TF-IDF obsolete? No. It is old, but old does not mean useless. TF-IDF is still useful as a teaching model because it explains the two signals behind many lexical rankers: - How much does this document use the query term? - How rare is that term across the collection? In production search, BM25 is often a stronger default lexical ranker because it handles repetition and length better. Modern systems may also add vector search, reranking, synonyms, filters, personalization, freshness, or learning-to-rank signals. But TF-IDF is still the clean bridge between simple word counting and stronger lexical ranking. ## Summary TF-IDF is a simple ranking method for lexical search. It combines: - Term frequency: how often a query word appears in a document. - Inverse document frequency: how rare that word is across the collection. It works well when exact words matter. That makes it useful for docs, API references, support articles, product search, and technical search. It struggles when the same idea is written with different words. It also does not handle repetition and document length as carefully as BM25. TF-IDF scores a document for a query, not for all time. It matches words rather than understanding meaning. Once you understand TF-IDF, BM25 becomes easier to understand. BM25 is not a totally different idea. It keeps the useful word-counting and rarity model, then makes it behave better for real search results. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=TF-IDF%3A%20The%20Search%20Ranking%20Idea%20Behind%20BM25&url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-tf-idf%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-tf-idf%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-tf-idf%2F&t=TF-IDF%3A%20The%20Search%20Ranking%20Idea%20Behind%20BM25) Copy image[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-tf-idf%2F)[](https://twitter.com/intent/tweet?text=%E2%80%9CTF-IDF%3A%20The%20Search%20Ranking%20Idea%20Behind%20BM25%E2%80%9D&url=https%3A%2F%2Fceisium.com%2Fblog%2Fwhat-is-tf-idf%2F) **Yuvraj** _Building Ceisium and working on growth @ SigNoz_ ceisium.com ## Next reads [ ![BM25 scores query terms using rarity, repetition, and document length signals.](https://ceisium.com/blog/bm25-cover-ceisium-bottom-right-aligned.webp) What is BM25? A Plain English Guide BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF. ](https://ceisium.com/blog/what-is-bm25/)