For agents: Read llms.txt for the complete documentation and public content index. Request this page as Markdown.

Skip to main content
Ceisium logo Ceisium
Website routes pass through crawl boundaries into a search corpus before a query selects candidates.
Website routes pass through crawl boundaries into a search corpus before a query selects candidates.

Website routes pass through crawl boundaries into a search corpus before a query selects candidates.

How Website Crawling Shapes Search Quality

Learn how crawl scope, sitemaps, rendering, URL identity, and recrawl freshness shape the pages a site-search ranker can choose from.

A documentation team publishes a new authentication guide at /docs/v2/authentication. The page is public, and a direct link opens it.

A user then searches for “authentication setup.” Search shows the old /docs/auth page.

Do not start by changing the ranking settings or semantic model. First, check whether the search corpus contains the new guide.

Website crawling discovers eligible URLs and gets their content for an index. Site search and search-engine optimization can use different crawl rules.

Verify which directives and signals your site-search crawler supports.

Ranking starts after crawling and indexing. It can order known pages, but it cannot use content that the index never received.

Search needs a correct corpus before ranking

A site-search system usually has this input path:

  1. Scope defines where the crawler can go.
  2. Discovery gives URLs to the crawler.
  3. Fetch, render, and extraction produce useful content.
  4. Indexing stores the content in an inverted index or another search structure.
  5. Retrieval selects candidates for one query.
  6. Ranking or reranking puts those candidates in order.

The first four stages build the search corpus. Retrieval then selects a smaller candidate set from that corpus.

A page outside the corpus cannot become a candidate. Ranking starts too late to restore a page that did not enter the corpus.

Four checks help you examine the corpus:

  • Coverage: Did the crawl include the correct pages?
  • Content fidelity: Does each indexed document contain the useful page content?
  • Identity: Does one stable document represent each real page?
  • Freshness: Does the document contain the current page content?

Coverage, content fidelity, identity, and freshness describe the corpus that the first four stages produce.

Crawling builds the corpus first. A query can select candidates only from pages in that corpus.
Black llama-head blob admits eligible page cards into a search corpus before a query selects a smaller candidate set.

Crawling builds the corpus first. A query can select candidates only from pages in that corpus.

Scope defines where the crawler can go

A crawler must not follow every URL that it finds.

A documentation crawl can include pages under /docs/. It can exclude account settings, site-search URLs, calendars, and unlimited filter combinations.

The crawl can also include one host and exclude other company hosts.

Scope defines this boundary. A search source can use a base URL with included and excluded paths.

In our example, /docs/v2/authentication is inside the /docs/ scope. A guide at help.example.com/authentication is not automatically in the same corpus.

Scope makes a URL eligible, but it does not always reveal that URL to the crawler. A start URL can reveal the first page.

Other pages need a discovery path through a link, redirect, or sitemap.

Discovery gives URLs to the crawler

A page can be public but still be unknown to a crawler.

Use these terms carefully:

  • A public page does not require authentication.
  • A reachable page gives a usable response at its exact URL.
  • A discoverable page has a path that gives its URL to the crawler.

Opening a URL manually only proves that the page is reachable for you. It does not prove that the crawler can discover it.

Crawlers can find URLs through start pages, eligible links, sitemaps, and redirect destinations.

The new authentication guide can be public and reachable. However, the navigation can still point to the old page.

If the sitemap also omits the new route, the URL is in scope but unknown.

A sitemap gives the crawler a list of URLs. It is especially useful for new pages without internal links.

A sitemap entry does not guarantee indexing. The URL can fail during scope checks, fetching, rendering, extraction, or indexing.

The opposite is also true. Removal from a sitemap does not delete a page. Other links can still lead to the old URL.

A successful fetch can produce a poor document

After discovery, the crawler requests the URL.

A server-rendered response can already contain the title, headings, paragraphs, and code examples. A client-rendered response can initially contain only a container and a script.

HTML
<div id="app"></div><script src="/docs-app.js"></script>

If the first HTML response lacks content, a renderer can run the page scripts. Extraction then selects content from the rendered page.

Rendering and extraction are different operations. The crawler can render a complete guide but store a poor document.

A useful document can contain the title, headings, steps, code, description, and URL. A poor document can contain repeated navigation or duplicate hidden text.

The extracted document can also omit the code sample or separate a heading from its section.

More indexed text does not always improve search. The indexed document must keep the text that gives meaning to the page.

Content fidelity measures how well the indexed document preserves useful page content. A noisy or empty document can still miss the query.

URL identity controls duplicate documents

The old /docs/auth route now has a replacement at /docs/v2/authentication.

Other URL forms can also exist:

TEXT
/docs/auth/docs/auth//docs/auth?source=nav

Treat two URL forms as one page only when the site shows that they are equal. A permanent redirect provides useful evidence.

Do not assume that every query parameter or path difference has no meaning.

Redirect status codes also give different information. 301 and 308 state that a move is permanent. 302 and 307 state that it is temporary.

A supported rel="canonical" tag is another publisher hint. It does not redirect a visitor, and each crawler can handle it differently.

Identity rules need limits. Supported v1 and v2 guides can share text but contain different instructions.

Do not merge them only because their text is similar.

Identity rules should produce one stable search document for each real piece of content.

Freshness controls the stored version

Identity selects the document that represents a page. Freshness shows whether that document contains the current content.

An index is a stored copy of the website. The website can change before the search document changes.

When the team changes a code sample, search does not get that change immediately. The crawler must revisit the source, and indexing must replace the old document.

Two failures can make search stale:

  1. The crawler did not revisit the page.
  2. The crawler got the new page, but indexing did not replace the old document.

Readers see stale search results in both cases.

Frequent recrawls can reduce delay. They also increase crawler work and load on the source site.

A dynamic documentation site can need a different schedule from a static marketing site.

Removal also needs a clear policy:

  • A permanent 301 or 308 states that a page moved.
  • A 404 states that the origin has no current representation.
  • A 410 states more clearly that the removal is intentional.
  • A soft 404 returns 200 with a “page not found” message.
  • A timeout does not show that a page was removed.

Do not delete the last good document after one timeout. Also, do not keep a confirmed obsolete page forever.

A successful recrawl replaces a stored copy. An uncertain failure needs another check. A confirmed move or removal can retire the document.
Black llama-head blob replaces an old page snapshot while an uncertain failure waits for another check and a confirmed move exits.

A successful recrawl replaces a stored copy. An uncertain failure needs another check. A confirmed move or removal can retire the document.

Ranking cannot repair a broken corpus

Lexical retrieval matches query terms. Semantic retrieval can match related meaning. A ranker then orders the candidates.

Every search stage still needs indexed evidence.

An undiscovered authentication page has no document for retrieval. An empty extracted document has no authentication steps to match.

The old page can still appear if the index keeps it. Duplicate URL forms can also occupy several candidate positions with the same answer.

A ranker can reorder known candidates. It cannot accurately rebuild a missing page or confirm a deletion.

Check crawling and indexing before you change ranking weights or models.

Diagnose search quality in pipeline order

Use the corpus and crawl telemetry as evidence. Outcomes such as discovered, indexed, skipped, and failed show where a page stopped.

Ask these questions in order:

  1. Scope: Is the URL in the permitted host and paths?
  2. Discovery: Can a start URL, link, redirect, or sitemap supply the URL?
  3. Fetch: Which status and final URL did the request return?
  4. Render: Did meaningful content appear after required scripts ran?
  5. Extract: Does the document contain the correct title, headings, body, and code?
  6. Identity: Did proven aliases resolve to the intended document?
  7. Freshness: Does the indexed document match the current page?
  8. Retrieval: Does the page enter the candidate set for the query?
  9. Ranking: Does the system put the best candidate too low?

Search products can implement these stages differently. The sequence helps you test the corpus before you tune the result order.

Make the input layer easy to verify

Use these checks to prevent common failures:

  • Compare important pages with the sitemap and discovered-URL list.
  • Examine the extracted document, not only the page in a browser.
  • During URL changes, verify permanent redirects and preserve real page versions.
  • After a large change, start a supported fresh crawl and examine the new corpus.
  • Test queries only after the current page and intended URL are in the index.

Frequently asked questions

Does a sitemap guarantee that a page appears in site search?

No. A sitemap supplies a URL to try. The page must still pass all crawl and indexing checks.

Can a crawler index a JavaScript-rendered website?

Sometimes. A rendering step can expose content that is absent from the first HTML response. Support and failure modes depend on the crawler.

What is the difference between a redirect and a canonical tag?

A redirect sends a client to another URL. A canonical tag names a preferred URL for crawlers that support the hint.

How often should a website be recrawled?

Match the schedule to the change rate, freshness target, and permitted load on the source site.

Fix the input layer before the ranker

When search shows the wrong page, first verify coverage, content fidelity, identity, and freshness.

After you verify the corpus, you can measure retrieval and ranking changes with more confidence.

Next reads

A term dictionary sends one highlighted term through posting lists to matching document nodes.

What Is an Inverted Index?

An inverted index maps terms to the documents that contain them, helping search engines find text matches without scanning every page.

BM25 scores query terms using rarity, repetition, and document length signals.

What is BM25? A Plain English Guide

BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF.

A query vector follows a highlighted path toward a nearby cluster of related content.

What Is Vector Search?

Vector search finds related content by comparing embeddings. Learn how vectors, similarity, approximate nearest-neighbor search, and hybrid retrieval work together.