TF-IDF: The Search Ranking Idea Behind BM25

TF-IDF is a simple way to answer a search-ranking question: which page uses the user’s words in the most useful way?

Search ranking starts with a small practical problem. A user types:

reset api token

The search system may have hundreds or thousands of docs, support articles, API pages, changelog entries, and product pages. Many of them contain one of those words, but only a few are probably the right result.

Here, a document means one searchable page, the collection means all searchable pages, and a query term means one word from the user’s search.

TF-IDF is one way to sort those pages. It is not semantic search, and it does not understand the query like a person. It asks a simpler question:

Which pages use the query words, and which of those words are specific enough to matter?

That small question is enough to make many exact-word searches work.

What is TF-IDF?

TF-IDF means:

term frequency x inverse document frequency

It is a ranking method for lexical search. Lexical search works with matching words. If the user searches for webhook signature, the system looks for pages that contain webhook, signature, or both.

TF-IDF gives each matching document a score for this query. A higher score means the document looks more relevant for that search. The same page can get different TF-IDF scores for different searches.

The two parts do different jobs:

• Term frequency asks how often the query word appears in this document.
• Inverse document frequency asks how rare that word is across the whole collection.

That is the core idea. A word matters more when it appears in the current document and does not appear everywhere else.

TF-IDF has two levers: how often the word appears in this page, and how rare that word is across the whole site.

Why not just count words?

Let us start with a very simple search system.

The user searches:

api token

We have two pages.

Page A:

API tokensCreate, rotate, and revoke API tokens.

Page B:

Authentication guideThis guide explains API access, token scopes, token rotation,token permissions, API clients, and API examples.

If we only count words, Page B can look stronger because it says api and token many times. Page B wins raw count because it repeats both words. But Page A may be the better result because it is short and focused.

Raw word counting has another problem. Common words appear everywhere:

docspageguideusesetup

If a word appears on almost every page, matching it does not tell us much. A page that contains docs is not special. A page that contains hmac or retry-after is much more specific.

TF-IDF fixes part of this by making common words quieter and specific words louder.

Term frequency

Term frequency is the easy half.

It asks:

How often does this query term appear in this document?

If someone searches:

webhook signature

Then a page that talks about webhook signatures several times is probably more relevant than a page that mentions the phrase once in a footer.

For example:

For now, ignore page length. We are only looking at whether both query words appear strongly.

In this tiny example, the first page looks strongest because both query terms appear strongly there.

This is the useful part of term frequency. Repeated words can be evidence.

But we should be careful. Repetition is evidence, not proof. A page can repeat a word because it is long, noisy, or covering many topics.

Inverse document frequency

Inverse document frequency is the more interesting half.

It asks:

How rare is this term across all documents?

Imagine a docs site with 1,000 pages.

Rarity does not prove relevance. It changes how much a match should count.

If the query is:

webhook hmac signature

then hmac should matter a lot. It appears on very few pages, so it helps separate the specific pages from the general webhook pages.

This is why IDF is useful for product and technical search. Product names, API fields, error codes, config keys, CLI flags, and protocol names often carry the real intent.

The TF-IDF formula

At a high level, TF-IDF looks like this:

tf-idf(term, document) = tf(term, document) x idf(term)

If the query has more than one term, the search system calculates a score for each query term and adds the scores together.

score(document, query)  = tfidf("webhook", document)  + tfidf("signature", document)

Different systems use different versions of term frequency and IDF. Some use raw counts, some use logarithms, and some normalize vectors. You do not need every variant to understand the ranking idea.

The plain-English version is enough:

score = query words used in this document x how rare those query words are across the collection

Common words whisper. Specific words speak loudly.

A small TF-IDF example

Let us use a tiny docs search example.

The user searches:

reset api token

The collection has three pages.

Now look at the query terms:

• api may appear on many pages.
• token appears on fewer pages.
• reset may appear on a smaller set of settings and recovery pages.

The ranking might look like this:

TF-IDF is meant to prefer Page A because the query terms are concentrated there, and the more specific terms help separate it from general pages.

This is why TF-IDF is a useful baseline. It can rank exact terminology before the search system understands anything about meaning.

Where TF-IDF works well

TF-IDF works well when the user and the document use the same words.

That happens a lot in technical search:

• api token
• webhook signature
• 429 retry-after
• rate limit
• sso setup
• npm install
• connection refused
• delete workspace

These are good lexical queries. The exact words matter.

In docs and product search, exact words are not a small detail. They are often the whole point. If a user searches for an error code, an API field, a product setting, or a command, they expect those words to match.

Where TF-IDF struggles

TF-IDF struggles when the meaning matches but the words do not.

For example:

A human can see the connection. TF-IDF may not.

It mostly sees word overlap. If the words are different, the score can stay low.

Lexical ranking is strong when words match. It needs help when the useful page says the same idea with different words.

Search systems can work around this with synonyms, stemming, query rewriting, semantic retrieval, or hybrid search. You do not need all of those pieces yet; they are different ways to add help around word matching. TF-IDF itself is still a word-based method.

Do titles and headings matter?

The core TF-IDF idea does not automatically know that a title is more important than the body.

But real search systems usually have fields:

• Title
• URL
• Headings
• Body
• Meta description
• Anchor text

A match in the title often deserves more weight than a match deep in the body.

For example:

title score x title weight+ heading score x heading weight+ body score x body weight

This is a layer around the lexical scoring method. The scoring method handles term frequency and rarity. Field weighting tells the system where the match happened.

How BM25 addresses TF-IDF’s limits

BM25 keeps a useful TF-IDF-style idea:

important pages use the query terms, especially specific terms

But BM25 adds better controls for two problems:

• Repetition should not grow forever.
• Long documents should not win just because they have more words.

You can read the plain-English BM25 guide Search / Jun 24, 2026 What is BM25? A Plain English Guide BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF. for the full walkthrough. The short version is that BM25 is still lexical ranking, but it is more careful about the repetition and length problems simple TF-IDF leaves unfinished.

Repetition should flatten out

Suppose the user searches:

token

A page that says token once gives us useful evidence. A page that says it five times gives stronger evidence.

But a page that says token fifty times should not be ten times better than the page that says it five times.

At some point, the search system should say:

Okay, this page is clearly about tokens. More repetition is not adding much.

BM25 handles this with term frequency saturation.

BM25 keeps the useful repetition signal, but it flattens the gain after the page has already proved the term matters.

Many real TF-IDF implementations soften repetition too, but BM25 is the common named model built around this saturation behavior.

Long documents need fair treatment

Long pages have more chances to match a query.

A 10,000-word guide might mention billing once, invoice once, and retry once because it covers everything. A short support page might focus only on failed invoice retries.

If we do not adjust for length, the long guide can get too much credit.

BM25 includes document length normalization. A match inside a short focused page can be stronger than the same match buried inside a long page.

This does not mean long pages are bad. A long guide can still be the best answer. The goal is only to avoid rewarding length by accident.

Is TF-IDF obsolete?

No. It is old, but old does not mean useless.

TF-IDF is still useful as a teaching model because it explains the two signals behind many lexical rankers:

• How much does this document use the query term?
• How rare is that term across the collection?

In production search, BM25 is often a stronger default lexical ranker because it handles repetition and length better. Modern systems may also add vector search, reranking, synonyms, filters, personalization, freshness, or learning-to-rank signals.

But TF-IDF is still the clean bridge between simple word counting and stronger lexical ranking.

Summary

TF-IDF is a simple ranking method for lexical search.

It combines:

• Term frequency: how often a query word appears in a document.
• Inverse document frequency: how rare that word is across the collection.

It works well when exact words matter. That makes it useful for docs, API references, support articles, product search, and technical search.

It struggles when the same idea is written with different words. It also does not handle repetition and document length as carefully as BM25.

TF-IDF scores a document for a query, not for all time. It matches words rather than understanding meaning.

Once you understand TF-IDF, BM25 becomes easier to understand. BM25 is not a totally different idea. It keeps the useful word-counting and rarity model, then makes it behave better for real search results.