# Ceisium > Ceisium is building infrastructure for growth and marketing teams. Ceisium Search is its first product: YouTube-like search for any website. ## Ceisium Search Ceisium Search is the first product from Ceisium. It is YouTube-like search for any website. Visitors can use vague words or describe what they want, and search works to put the right page first. YouTube search is the reference for the result, not the implementation. This position does not claim a YouTube affiliation or the use of YouTube technology. ## When to use Ceisium Search Use Ceisium Search when a company has valuable public content but does not want to operate its own crawler, indexing jobs, search engine, ranking pipeline, and monitoring. It is a good fit for documentation, guides, pricing, changelogs, blogs, support content, and public product pages. Use the Ceisium Search API when a website, application, or trusted backend needs ranked results from one project. Use a Browser/public Search key for browser code and restrict that key to exact allowed origins. Use a Server Search key for trusted backend code and never expose it to a browser. Do not use Ceisium Search for secrets or private content without a separate approved security design. Do not claim that a source is searchable until a crawl has completed and the pages are indexed. ## Product model A workspace contains members and projects. A project is the isolation boundary for sources, Search keys, indexes, usage, and results. A source defines a public website URL plus crawl scope and schedule. A crawl discovers allowed pages, extracts useful content, records page outcomes, and writes searchable chunks to the project's index. Ceisium Search uses more than one retrieval method and can reorder a limited result set before returning compact results. Search settings are project scoped. A Search key cannot search another project. ## Setup workflow 1. Sign in with an approved work email. 2. Create or open a workspace project. 3. Add a public website source and define its crawl scope. 4. Start the crawl and wait for a terminal outcome. 5. Review indexed pages and failures. 6. Create a Browser/public or Server Search key for the intended caller. 7. Store the key in an environment variable or secret store. 8. Call the Search API with the project ID and a representative query. 9. Render the title, snippet, and URL. Keep the score as an internal signal. 10. Review search analytics and improve source content or scope. ## Search API Send `POST https://api.ceisium.com/v1/search` with a project-scoped bearer Search key. The JSON body requires `project_id` and `query`. It can include `top_k` from 1 to 50 and an absolute `path_prefix` that limits returned URLs. ```bash curl https://api.ceisium.com/v1/search \ -H "Authorization: Bearer $CEISIUM_SEARCH_KEY" \ -H "Content-Type: application/json" \ -d '{"project_id":"prj_example","query":"search setup","top_k":5}' ``` A successful response contains ranked `results`. Each result has `url`, `title`, `snippet`, and `score`. Treat `score` as an internal ranking signal. Public discovery is available at `GET https://api.ceisium.com/healthz` and `GET https://api.ceisium.com/v1/capabilities`. The complete machine contract is at `https://ceisium.com/openapi.json`. ## Authentication and safety Browser/public Search keys require an exact allowed browser origin. Browsers add the `Origin` header. Server Search keys belong only in trusted backend code. Never put a Server key in a client bundle, repository, log, screenshot, ticket, or public document. Correct authentication, billing, authorization, and validation errors before retrying. Use bounded backoff only for transient server errors. Do not treat an empty result list as an API failure. ## Trust and contact Ceisium provides product, support, billing, security, and privacy contact at yuvraj@ceisium.com. Do not send passwords, API keys, private connection strings, or full payment details by email. Ceisium receives account and workspace information, customer source configuration, extracted public page content, search queries and result metadata, crawl outcomes, billing state, support messages, and bounded technical and usage data. Read https://ceisium.com/privacy/ for current providers, purposes, retention, and privacy choices. Read https://ceisium.com/terms/ for account duties, acceptable use, billing, service changes, and limits. ## Current documentation ### Quickstart Canonical URL: https://ceisium.com/docs/ Create a project, crawl it, query the API, and embed search. This guide uses the [Next.js App Router docs](https://nextjs.org/docs/app) as the example site. The same steps work for any public docs, changelog, blog, pricing, or product pages. ## 1. Create a project Create a Search project and give it a name. In Setup, add the source URL, path rules, crawl frequency, and page limit. Save the source to start its first crawl. For the Next.js App Router docs, use: ```txt Project name: Next.js App Router Website URL: https://nextjs.org Include path rule: Starts with /docs/app Exclude paths: none Crawl frequency: Daily Max pages: 1,000 ``` When the crawl finishes, the source should show indexed pages. Check for the [installation guide](https://nextjs.org/docs/app/getting-started/installation), [JSON-LD guide](https://nextjs.org/docs/app/guides/json-ld), and [`useParams` reference](https://nextjs.org/docs/app/api-reference/functions/use-params). ## 2. Query the API Create a **Browser/public key** for a website search box. Restrict it to the website origins where it may run, then call `/v1/search` directly from the browser. The browser supplies the `Origin` header automatically. ```ts const response = await fetch("https://api.ceisium.com/v1/search", { method: "POST", headers: { Authorization: `Bearer ${CEISIUM_PUBLIC_SEARCH_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ project_id: CEISIUM_PROJECT_ID, query: "How do I add typed JSON-LD to a page component?", top_k: 8, path_prefix: "https://nextjs.org/docs/app", }), }); const { results } = await response.json(); ``` If you configured allowed result URL prefixes on the key, send a matching `path_prefix` with every request. ## 3. Embed search Use a UI component when visitors should search inside your product site. Start with one of these patterns: - Command palette for keyboard-first docs search. - Inline search for a visible search field on docs pages. - Search bar when the whole bar should open a modal. - Search modal for an overlay with a search input and results. Use `CEISIUM_PUBLIC_SEARCH_KEY` only for a Browser/public key with its website and result restrictions configured. If you prefer to proxy searches through your backend, create a Server key, store it as `CEISIUM_SEARCH_KEY`, and never expose that key in browser code. ### Search benchmark Canonical URL: https://ceisium.com/docs/benchmark/ We tested four search products with 1,000 Next.js documentation queries. Compare relevance, response time, setup choices, and missed results. We tested Ceisium Search, Algolia, Typesense, and Meilisearch with 1,000 queries on the [Next.js App Router docs](https://nextjs.org/docs/app/getting-started/installation). The queries included searching for exact page names, long-tail questions, typos, error messages, and more. Note: The goal was to measure the performance using the default settings of each product rather than engineering different knobs. We left optional settings unchanged after completing the required setup, which differs for each product. ## Benchmark results per provider Ceisium Search returned an expected page in the first three results for **912 of the 1,000 queries**. That compares with 357 for Algolia, 361 for Typesense, and 423 for Meilisearch.
ProductHit@3Hit@10MRR@10nDCG@10
Ceisium Search91.2%96.6%0.84680.8761
Algolia35.7%38.6%0.31490.3326
Typesense36.1%40.3%0.32200.3420
Meilisearch42.3%49.5%0.36790.3988
### What do these metrics mean? **Hit@3** is the share of queries with an expected page in the first three results. **Hit@10** checks the first ten. **MRR@10** measures how high the primary expected page ranks: first place earns 1, second earns 0.5, and so on. Pages outside the first ten earn 0, and we average the score across all queries. **nDCG@10** also checks the order, but considers all acceptable pages. It gives more credit to pages near the top and compares the result with the best possible order. Higher is better for all four metrics. MRR and nDCG range from 0 to 1. ### The query type made a difference The results become clearer when we break them down by query type. These are the Hit@3 scores for each category:
Query typeWhat the category testsQueriesCeisium SearchAlgoliaTypesenseMeilisearch
ExactKnown page terms28093.6%80.0%72.5%82.5%
Long-tailMore specific descriptions28089.6%14.6%18.9%23.9%
How-toA task the visitor wants to complete12093.3%0.8%2.5%3.3%
ConfigurationOptions and settings8097.5%21.3%25.0%28.8%
ErrorA failure the visitor wants to resolve5589.1%14.6%20.0%20.0%
ConceptAn idea the visitor wants to understand5580.0%16.4%14.6%18.2%
VariantAlternative wording6088.3%20.0%38.3%48.3%
TypoMisspelled words5096.0%76.0%62.0%78.0%
NavigationA destination within the documentation2075.0%35.0%45.0%45.0%
The three external engines did best on exact queries. Their lowest scores were on how-to queries, where Ceisium Search found an expected page in the top three 93.3% of the time. **This is the problem we want Ceisium Search to solve**. You should be able to describe what you need, even when you do not know the page name. LLMs are getting people used to asking full, descriptive questions. So it only makes sense that website search handles that habit too, along with short keywords. ### A few examples To give you an idea of how different queries perform, here are a few examples from the benchmark. The rank labels show the benchmark results. The result cards and their timings come from later checks. For `Next.js useParmas hook`, the expected page was `useParams`. Ceisium returned it first, Algolia third, Typesense fourth, and Meilisearch fifth. All four found the page despite the typo; two put it in the top three.
Now take a full question: `How do I add typed JSON-LD to a page component?` Ceisium returned the expected JSON-LD guide first. None of the other three returned it in the top ten. The query describes a task, which is the kind of search we want Ceisium Search to support well.
Ceisium placed an expected page in the top three for 93.6% of exact queries. `Next.js App Router guides` was one of the exceptions. Ceisium put the expected page eighth; Algolia, Typesense, and Meilisearch all put it first.
We still want to improve cases like this. A strong overall score does not make an individual miss less frustrating. ## Latency vs quality Ceisium Search found expected pages more often, but all three other products were faster.
ProductMedian latencyP95 latency
Ceisium Search857.86 ms1,983.09 ms
Algolia178.91 ms776.95 ms
Typesense225.41 ms759.55 ms
Meilisearch240.58 ms395.78 ms
The median is the middle request time. At P95, 95% of requests finished within the reported time. These times include the network round trip from one client in Bengaluru. Ceisium's median was about 0.86 seconds, compared with 0.18–0.24 seconds for the others. Its P95 was almost two seconds. Meilisearch had the lowest P95 at about 0.40 seconds. That wait matters when someone is using a search box. We believe returning the right page matters more than getting irrelevant pages faster. We also plan to [improve Ceisium's latency](/changelog/faster-search-better-first-results/) in the coming weeks. We have since improved latency. A later Next.js run recorded **1.05 seconds client P95**, including the network round trip. The runs used different test conditions and network paths. The original results above remain unchanged. [Read the search update](/changelog/faster-search-better-first-results/). ## What this benchmark answers All four products could return better results with tuning, but getting those settings right takes time and effort. As your content changes, settings that once worked well can become stale and search quality can fall. That is why we care about the quality you get before manual tuning. With Ceisium Search, we aim to keep your website search accurate without constant manual tuning. Also, this run covers one documentation site and a query set we wrote. We did not measure cost, setup time, performance under production traffic, or the best results each product could reach after tuning. For a broader product comparison, read [Ceisium Search vs Algolia](/compare/algolia/), [Ceisium Search vs Typesense](/compare/typesense/), and [Ceisium Search vs Meilisearch](/compare/meilisearch/). ## Methodology and how to repeat it We ran the benchmark on September 3, 2026, using 280 Next.js App Router page URLs from an August 24 snapshot. ### What “default settings” means here Each product needs some setup before it can search these pages. We completed that setup and left optional relevance settings unchanged. **Ceisium Search** crawled the Next.js pages and used the search settings generated for that website. We made no manual relevance adjustments for the benchmark.
The other three products each received 390 records prepared from the same 280 page URLs. The records contained titles, headings, and body text. **Algolia** used a fresh index. We left searchable attributes, ranking, synonyms, rules, and AI settings unchanged after importing the records.
**Typesense Cloud 30.2** used a fresh collection. Typesense requires a list of fields to search; we chose `title,headings,body`, in that order, before the run. That order was our setup choice.
**Meilisearch Cloud 1.53.1** used a fresh index with default searchable attributes and ranking rules. We added no field priorities, synonyms, custom ranking, or embedding provider.
### Repeat the method on your own docs This is the method we used: 1. Save a fixed set of pages and check for missing or invalid content. 2. Keep the queries and expected pages fixed before the run. 3. Complete each product's setup and record its settings and indexed content. 4. Send 100 successful warmup searches before measurement to reduce first-request setup effects. Exclude these requests from the scores. 5. Run the same 1,000 queries in the same order, separately for each product. Send one request at a time, at no more than 60 starts per minute. 6. Ask for up to 50 results. Normalize page URLs, remove duplicates, and score the first ten unique pages. 7. Keep failed requests in the quality scores and client latency statistics. Check the settings and indexed content again after the run. The same method can help you compare search quality and response time on your own documentation. In our run, Ceisium Search found expected pages more often across every query category, including exact queries. It also took longer to respond. We plan to reduce that wait while improving the results we still get wrong. For your own test, include questions from people who do not know your page titles. Check whether they find the right page, then check how long they wait. ### Search API Canonical URL: https://ceisium.com/docs/search-api/ Auth, request fields, response shape, result limits, errors, and examples. Use the Search API directly from a website with a Browser/public key, or from a backend with a Server key. ## Endpoint ```txt POST https://api.ceisium.com/v1/search ``` ## Choose a key type | Key type | Use it from | Required guardrails | | --- | --- | --- | | Browser/public keys | Browser code | Exact allowed website origins, optional allowed result URL prefixes, per-IP and total-key rate limits. | | Server keys | A trusted backend | Total-key rate limit. Never expose these keys in browser code. | Use `CEISIUM_PUBLIC_SEARCH_KEY` for browser integrations and `CEISIUM_SEARCH_KEY` for server integrations. Both key types are search-only and belong to one project. ## Auth Pass a Search API key as a bearer token. ```txt Authorization: Bearer $CEISIUM_PUBLIC_SEARCH_KEY Content-Type: application/json ``` For a Browser/public key, Ceisium Search checks the browser-supplied `Origin` against the key's allowed origins. **Origin is required**: a missing or disallowed origin returns `403`. Enter complete origins such as `https://docs.example.com` or `http://localhost:3000`; Ceisium Search also adds the expected scheme when you enter a bare hostname or localhost during key creation. Origins cannot contain paths. If the key has allowed result URL prefixes, each request must also include a matching `path_prefix`. Origin and path restrictions limit where a public key can be used and what it can return. Keep Server keys private because they do not use browser-origin restrictions. ## Request body | Field | Type | Required | Notes | | --- | --- | --- | --- | | `project_id` | string | yes | The project to search. | | `query` | string | yes | The visitor search query. | | `top_k` | number | no | Requested result count. Defaults to `10`; accepted range is `1` to `50`. The response is capped to the key's configured maximum, which defaults to `20`. | | `path_prefix` | string | no | Limit results to a URL prefix, such as `https://nextjs.org/docs/app`. This includes the App Router root page and its child paths. | ## Browser example ```ts const response = await fetch("https://api.ceisium.com/v1/search", { method: "POST", headers: { Authorization: `Bearer ${CEISIUM_PUBLIC_SEARCH_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ project_id: CEISIUM_PROJECT_ID, query: "How do I add typed JSON-LD to a page component?", top_k: 8, path_prefix: "https://nextjs.org/docs/app", }), }); ``` Do not manually set `Origin`; browsers add it to cross-origin requests. ## Server example ```bash curl -X POST https://api.ceisium.com/v1/search \ -H "Authorization: Bearer $CEISIUM_SEARCH_KEY" \ -H "Content-Type: application/json" \ -d '{ "project_id": "'$CEISIUM_PROJECT_ID'", "query": "How do I add typed JSON-LD to a page component?", "top_k": 5, "path_prefix": "https://nextjs.org/docs/app" }' ``` ## Per-minute limits New keys start with plan-based defaults. You can edit these values when creating a key; they are defaults, not plan ceilings. | Plan | Browser searches per IP | Total searches per key | | --- | ---: | ---: | | Starter | 60 per IP per minute | 100 total per minute | | Growth | 100 per IP per minute | 1,000 total per minute | | Pro | 200 per IP per minute | 5,000 total per minute | Browser/public keys apply both limits. Server keys apply only the total-key limit because many visitors may share the server's outbound IP. Per-minute limits protect against short traffic bursts; the plan's monthly search allowance is counted separately. When a request exceeds either applicable per-minute limit, the API returns `429`. A request whose `top_k` is higher than the key's maximum is capped to the key's configured maximum rather than rejected. ## Response Render `title`, `snippet`, and `url`. Treat `score` as an internal ranking signal, not visitor-facing copy. ## Errors | Status | Meaning | | --- | --- | | `401` | Missing or invalid API key. | | `402` | Billing is inactive, usage data is stale, or the monthly allowance is exhausted. | | `403` | Project, origin, or result-path access is not allowed for this key. | | `404` | Project not found. | | `413` | Request body is too large. | | `422` | Invalid body, empty query, or `top_k` outside `1` to `50`. | | `429` | A per-IP or total-key per-minute limit was reached. | | `500` | Search backend error. Retry later and check observability. | ### UI Components Canonical URL: https://ceisium.com/docs/ui-components/ Install search UI patterns and wire them to a Ceisium Search project. UI components give visitors a search surface backed by either a restricted Browser/public key or your own server route. ## Component patterns | Pattern | Use it when | | --- | --- | | Command palette | Users expect keyboard search with `/` or a shortcut. | | Inline search | A docs page needs a visible input and result list. | | Search bar | A compact bar should open a modal when clicked. | | Search modal | You want an overlay with a search input and results. | ## Direct browser setup Create a Browser/public key, configure its allowed origins and result URL prefixes, and expose it as `CEISIUM_PUBLIC_SEARCH_KEY` in your client configuration. ```ts const response = await fetch("https://api.ceisium.com/v1/search", { method: "POST", headers: { Authorization: `Bearer ${CEISIUM_PUBLIC_SEARCH_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ project_id: CEISIUM_PROJECT_ID, query, top_k: 8, path_prefix: "https://nextjs.org/docs/app", }), }); ``` ## Server route alternative If you use a Server key, keep it on your backend and have the component call your route. ```ts export async function POST(request: Request) { const { query, top_k = 8 } = await request.json(); const response = await fetch("https://api.ceisium.com/v1/search", { method: "POST", headers: { Authorization: `Bearer ${process.env.CEISIUM_SEARCH_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ project_id: process.env.CEISIUM_PROJECT_ID, query, top_k, }), }); return Response.json(await response.json()); } ``` ## Client config Keep the config small: ```ts { placeholder: "Search docs...", shortcut: "/", limit: 8, debounceMs: 200, emptyMessage: "No results found." } ``` For the Next.js App Router docs, useful preview queries are: ```txt How do I add typed JSON-LD to a page component? Next.js useParmas hook install Next.js App Router next.config.js reference ``` ## Result behavior Each result should show: - Page title. - Short snippet. - Canonical URL or path. - Click handler that navigates to the result. Keep result lists scrollable so long docs sites do not make the modal grow past the viewport. ## Latest request wins During fast typing, an older request can finish after a newer request. Debouncing reduces request volume, but it does not guarantee response order. Cancel superseded work and only let the newest request update the UI. ```ts let activeController: AbortController | null = null; let requestVersion = 0; async function search(query: string) { const version = ++requestVersion; activeController?.abort(); const controller = new AbortController(); activeController = controller; try { const response = await fetch("https://api.ceisium.com/v1/search", { method: "POST", signal: controller.signal, headers: { Authorization: `Bearer ${CEISIUM_PUBLIC_SEARCH_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ project_id: CEISIUM_PROJECT_ID, query, top_k: 8, }), }); const data = await response.json(); if (version !== requestVersion) return; renderResults(data.results); } catch (error) { if (error instanceof DOMException && error.name === "AbortError") return; if (version === requestVersion) renderError(); } } ``` Ceisium Search's built-in component previews already apply this guard. Add it when you call the raw Search API from a custom typeahead client. ## Styling Use your app theme tokens for borders, background, text, and focus states. Avoid hardcoded colors in the component. ### Analytics Canonical URL: https://ceisium.com/docs/analytics/ Read search events, no-result queries, popular queries, and engagement. Analytics shows what visitors search for and where search needs better content.
## What to watch | Metric | What it means | | --- | --- | | Searches | Total Search API requests for the selected window. | | Average results | How many results visitors get per query. | | No-result searches | Queries that returned no usable result. | | Popular queries | Repeated searches worth checking in the playground. | ## Next.js examples For a Next.js App Router project, review queries like: ```txt How do I add typed JSON-LD to a page component? Next.js useParmas hook install Next.js App Router next.config.js reference ``` If the JSON-LD query returns the right page, leave it alone. If `install Next.js App Router` has no results, check whether the crawl included `https://nextjs.org/docs/app/getting-started/installation`. ## How to use the table - Toggle no-result queries first. - Run a query in the playground before changing crawl scope. - If the right page exists but ranks low, inspect the page title, headings, and snippet. - If the page is missing, add or widen the source path and crawl again. ## Export Use CSV export when you want to review repeated queries outside the app or share a short list with the team writing docs. ### Crawl Logs Canonical URL: https://ceisium.com/docs/crawl-logs/ Track crawl status, page outcomes, failures, retries, and coverage. Crawl Logs show what Ceisium Search found, indexed, skipped, or failed during a crawl.
## Run states | State | Meaning | | --- | --- | | Queued | Crawl is waiting to start. | | Running | Ceisium Search is fetching and indexing pages. | | Completed | Crawl finished and page counts are ready. | | Failed | Crawl stopped before completing. Check the error and source settings. | ## Page counts | Count | Meaning | | --- | --- | | Discovered | URLs found during crawl. | | Indexed | Pages added to search. | | Skipped | Pages ignored because of scope, duplicate URLs, or unsupported content. | | Failed | Pages Ceisium Search tried but could not fetch or index. |
## Next.js example For a source that includes `/docs/app` and its child paths on `https://nextjs.org`, check for pages such as: ```txt /docs/app /docs/app/getting-started/installation /docs/app/guides/json-ld /docs/app/api-reference/functions/use-params ``` If `/docs/app/guides/json-ld` is missing, check include and exclude paths before rerunning the crawl. ## Debug flow 1. Open the latest crawl. 2. Filter URL events to `Failed` and `Skipped`. 3. Check whether the missing page is outside the source path. 4. Adjust source scope. 5. Run crawl again. ### Usage Canonical URL: https://ceisium.com/docs/usage/ Review API requests, crawl volume, indexing usage, and plan limits. Usage shows how much search and crawl activity a project is using.
## Main counters | Counter | Meaning | | --- | --- | | Indexed pages | Pages currently available to search. | | Successful searches | Search API calls that completed successfully. | | Failed searches | Search API calls that errored. | | Total crawls | Manual and scheduled crawl runs. | | Crawl pages checked | Pages fetched or checked during crawls. | ## Source usage Use source usage to see which source consumes crawl volume. For a Next.js project, a source that includes the full `/docs/app` tree will usually check more pages than a source limited to the single `/docs/app/getting-started/installation` page. ## Monthly allowance Your plan's monthly allowance counts successful searches across the billing period. If a project reaches it, reduce unnecessary requests or move to a larger plan. The Search API returns `402` when the allowance is exhausted or billing is inactive. ## Per-minute key limits Per-minute key limits protect against short traffic bursts and accidental cost spikes. They are separate from the monthly allowance: - Browser/public keys have a per-IP limit and a total-key limit. - Server keys have a total-key limit only, because many visitors can share one server IP. - The API returns `429` when an applicable per-minute limit is reached. The defaults depend on the plan, but you can edit them when creating a key. See [Search API](/docs/search-api/) for the current defaults and key behavior. ## Quick checks - Search volume high, indexed pages stable: visitors are using search. - Crawl pages checked high, indexed pages flat: crawl scope may be too wide. - Failed searches rising: check API auth, project id, and backend logs. ## Current changelog ### Faster search, better first results Canonical URL: https://ceisium.com/changelog/faster-search-better-first-results/ Better first results and lower client response time, with no setup changes. The most accurate website search just got even better with reduced latency and better top 1 hit rate. Here are the latest results of Ceisium Search on the [NextJS docs benchmark.](/docs/benchmark/) ## The right page, higher in the results We tested the update with 1,000 questions about the [Next.js documentation](). The primary answer appeared first in **898 searches**, up from 819 with the earlier version. That is **79 more searches with the main answer in the first position**. | Search result | Earlier version | This update | | -- | -- | -- | | Primary answer first | 81.9% | **89.8%** | | An accepted answer in the first three results | 93.7% | **97.9%** | | An accepted answer in the first ten results | 97.4% | **98.9%** | Finding the correct results matters. Getting them quickly matters too. ## Less time waiting for results (47.2% lower client P95 latency) **Client P95 latency was 1,047 ms in the latest run, compared with 1,983 ms in our published benchmark.** Both figures include the network round trip. In the latest run, 95% of successful searches returned within about 1.05 seconds. | Client response time | Published benchmark, 3 September | Latest run, 27 September | | -- | -- | -- | | Median | 858 ms | **812 ms** | | P95 | 1,983 ms | **1,047 ms** | The quality table above compares the earlier version with this update on the same 1,000 questions. Note: Response time will vary with location, website content, and traffic. ## No extra setup Existing projects receive the update automatically. Your current search integration, indexed content, and project settings continue to work. You do not need to change your API calls, rebuild your index, or tune new settings. ### Introducing Ceisium Search Canonical URL: https://ceisium.com/changelog/first-release/ YouTube-like search for your website. A few months back, a friend/colleague of mine shared an example of incorrect search results by Algolia on our company website. That message led me to a series of tinkering, experiments, optimization and the results of which become "Ceisium Search" - The simplest out of box search experience for your website and docs. While working on it, I realized three things wrong with how current tools approach setting up the "Search box" for a website. 1. The setup needs you to define certain configuration to work properly. Things like setting up the searchable attribute, correct order, synonyms, and more. 2. Even after spending time and effort figuring out the right setup, it can go stale quickly as your website or docs evolve. 3. And finally, the search quality can be pretty spiky depending on the type of queries your user ask. As people get used to working with LLMs, the tend to ask for longer, detailed queries which are not well handled by the out of box experience of current website search providers (likes of algolia, typesense, miliesearch). The one thing the current website search providers were good at was the speed of showing the results (which was wrong at times). Ceisium Search solves all three problems while making a concious tradeoff of ~2 second P95 end to end latency. (Update: now we have made the P95 latency ~1 sec without compromising the quality gains or the out of the box setup experience). ## The Youtube Analogy YouTube search can take vague words, a half-remembered phrase, or a feeling and still put the right video first. Website search should do the same for the right page. Ceisium Search brings this goal to any public website. It also handles the maintenance loop that keeps results useful as the site changes. Your docs update. A new pricing page ships. A changelog entry goes live. And the search results visitors see still point to content from three months ago, or return nothing at all. Keeping search current means refreshing content, checking coverage, debugging missing pages, and starting over the next time something changes. Ceisium Search handles that loop. Give Ceisium Search a URL. It keeps your public pages searchable, refreshes as the site changes, and gives you the tools to see what visitors can find. Website search that keeps up with your website. ## Start from a URL Create a search project, add a source with the paths you want visitors to find, and Ceisium Search takes it from there. For a docs-heavy site, that might mean starting with `/docs/`, `/pricing/`, `/changelog/`, and `/blog/`, excluding noisy paths, setting a crawl frequency, and capping pages. Ceisium Search keeps those pages searchable and keeps crawl history available when you need to inspect coverage. ## Search through the API or UI Use the Search API with your existing search box, or start with one of our UI examples. The API returns ranked pages with titles, URLs, and short text previews. Choose a search layout for your website: * A command palette for keyboard shortcuts * Inline search with results below the input * A search bar that opens a modal * A search modal with an input and results Try each layout with your project's indexed pages. Copy the example code, or give the supplied setup instructions to your coding assistant. Connect from your website with a Browser/public search key restricted to your allowed sites. You can also connect through your server with a private Server key. ## Get Started with Ceisium Search Ceisium Search works best for teams with docs, pricing, changelog, guides, blog, and product pages that need to stay searchable as the site changes. [Get started with Ceisium Search](/app). ## Current technical articles ### BM25 in RAG: Why Exact Keywords Still Matter Canonical URL: https://ceisium.com/blog/bm25-in-rag/ Dense retrieval can miss error codes, identifiers, and exact phrases. BM25 gives RAG systems another path to the source evidence they need. An embedding model can connect “login problem” with “authentication failure.” That is useful until the answer depends on one exact string. ```text How do I fix ZQ7-AUTH-N9X4? ``` In a common retrieve-then-generate RAG pipeline, retrieval selects a small set of passages and supplies them to the generator as external evidence. If the page for `ZQ7-AUTH-N9X4` never enters that set, the generator cannot cite it or ground an answer in it. Consider an illustrative failure case. Dense retrieval returns general pages about expired sessions, while the exact troubleshooting page stays out of the candidate set. The answer can still sound plausible, but any claim that depends on the missing page cannot be grounded in the retrieved source evidence. I would not make embeddings the only retrieval path for a corpus that contains identifiers, error codes, proper nouns, niche terms, or quoted phrases. Those queries need a path that preserves literal evidence, and BM25 is one of the simplest ways to add it. ## RAG fails before generation The [original RAG paper](https://proceedings.neurips.cc/paper/2020/hash/6b493230-Abstract.html) combines parametric memory with retrieved non-parametric memory. RAG architectures have changed since 2020, but this dependency has not. A source passage must reach the generator before the answer can use it as retrieved evidence. For the `ZQ7-AUTH-N9X4` query, dense retrieval might return pages about invalid credentials, expired sessions, access-token failures, and general login troubleshooting. All four are related to the question, and some may help. None is a substitute for the page that documents this exact error code. Semantic similarity tells us what a passage is about. The identifier tells us which piece of evidence the user needs. Lexical retrieval can catch the gap between those two questions.
## RAG needs meaning and identity [Vector search](/blog/what-is-vector-search/) helps when a query and a document use different words. Someone can search for “recover account access” and still find a page called “Reset a forgotten password.” Dense retrieval crosses that vocabulary gap. Identifiers create the opposite problem. A query containing `ZQ7-AUTH-N9X4`, `POST /v1/search`, `client_max_body_size`, `Acme Ultra 14`, or a copied log line asks for more than a related topic. The literal text carries part of the meaning. Sometimes it carries nearly all of it. [Salient Phrase Aware Dense Retrieval](https://aclanthology.org/2022.findings-emnlp.19/) gives us evidence for this failure mode. Chen and colleagues report that the dense retrievers they studied lagged BM25 at reliably matching salient phrases and rare entities. Their result does not mean dense retrieval can never handle exact terms. The same paper trained a dense lexical model that recovered sparse-style matching capacity. BM25 is a lexical ranker. It rewards passages that share important analyzed terms with the query. Our [plain-English guide to BM25](/blog/what-is-bm25/) covers its scoring mechanics, so I will keep the RAG takeaway short. > Semantic retrieval protects related meaning. BM25 can raise evidence that shares the relevant analyzed terms. “Analyzed terms” is an important limit. Search engines tokenize text before ranking it. Depending on the tokenizer, `/v1/search` may remain one value or become `v1` and `search`. Hyphens, underscores, punctuation, and case can change what the index stores. [Weaviate's keyword-search documentation](https://docs.weaviate.io/weaviate/concepts/search/keyword-search) shows the practical consequence. Keyword search matches indexed tokens, so fields such as URLs and email addresses need tokenization that fits the way you plan to query them. BM25 therefore does not guarantee whole-string equality or phrase order. Use a keyword field, term lookup, or phrase query when byte-for-byte identity or ordered phrases are requirements. Those mechanisms can share the candidate path with BM25, but they are separate retrieval tools. ## Hybrid retrieval widens the candidate set Dense and lexical retrieval fail in different ways, so many RAG systems run both. A common pipeline for the error-code query works like this. 1. BM25 finds passages whose analyzed terms overlap `ZQ7-AUTH-N9X4`. 2. Dense retrieval finds passages about the authentication problem. 3. Fusion combines both result lists into one candidate set. 4. A reranker reorders that set. It cannot recover a passage that neither retriever found. 5. The selected passages become context for generation.
[NVIDIA's RAG Blueprint](https://docs.nvidia.com/rag/latest/hybrid_search.html) documents one deployable version of this design. It supports sparse-plus-dense retrieval and can combine results with reciprocal rank fusion or configured weights. The blueprint demonstrates the design. Whether it improves a different corpus remains an evaluation question. A second retrieval path adds storage and compute. Its wall-clock cost depends on whether both searches run in parallel and where the slower path sits in the latency budget. Fusion introduces another choice because the two lists need weights or a rank-based combination. It can also send duplicates and weak candidates into the reranker. The best balance changes with the query. The lexical result may deserve more influence for `ZQ7-AUTH-N9X4`, while the dense result may deserve more for “How can I recover account access?” One fixed weighting can be a poor fit for a query mix that spans both, so test the weights against each query class before considering per-query routing. Hybrid retrieval protects against two kinds of evidence loss. It still has to earn its extra cost in evaluation. ## Exact matching cannot judge source quality Now make the example harder. Two pages contain `ZQ7-AUTH-N9X4`. 1. An old guide has the code in its title and body. 2. A current guide mentions the code once. BM25 may rank the old guide first because it has stronger lexical evidence. The match is exact, yet the result is wrong for the user. Access control must remain a hard eligibility constraint before ranking, never a preference that a reranker can ignore. [Azure AI Search documents this distinction](https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview) for document-level access control. Among eligible documents, lexical overlap still says nothing about freshness, product version, or source authority. Metadata filters, version-aware indexing, authority signals, or a reranker may be needed to prefer the current guide.
This is why the slogan “BM25 is precise, vectors understand meaning” falls apart under real source constraints. Lexical retrieval can be precise about the terms it indexed. It cannot decide whether a passage is current, authorized, or true. Dense retrieval has the same boundary. A passage can be semantically close and still be stale or wrong. If an unauthorized passage reaches the candidate set, the eligibility filter has already failed. The retriever must preserve the right evidence for this user, at this time, from the right source. ## When dense retrieval is already enough The strongest counterargument is a good one. A well-engineered dense system may already retrieve product names, codes, and technical language reliably. [Dense Passage Retrieval](https://aclanthology.org/2020.emnlp-main.550/) beat its BM25 baseline on top-20 passage retrieval accuracy across the paper's evaluated open-domain QA benchmarks. The SPAR result above also shows that dense models can learn lexical matching behavior. In practice, a strong dense system may have a domain-tuned embedding model, chunks that keep identifiers beside their explanations, metadata filters, and enough surrounding context to make an identifier meaningful. What matters is the measured result. If top-k recall already passes for the query types you care about, adding BM25 may only add complexity. Dense-only retrieval is most plausible when the corpus is small and consistent, users ask mostly natural-language questions, and evaluation shows that required passages already appear reliably. Lexical retrieval has its own misses. Without stemming, synonym expansion, translation support, or query rewriting, it can lose passages that use different vocabulary. The important distinction is between model capability and system evidence. “The embedding model can retrieve this identifier” is a capability claim. “Our retriever reliably preserves this identifier class” is a measured system claim. Test the second claim against the identifiers your users actually send. ## Test lexical retrieval against real failures Do not add BM25 because hybrid retrieval sounds safer. Start with the failures in your query traffic. If exact-term misses are absent, keep the simpler system. If they are present, test whether lexical retrieval fixes them. A small RAG evaluation set should separate these query classes. 1. Error codes and IDs 2. Proper nouns and model names 3. Niche terminology and configuration keys 4. Exact phrases copied from logs or support messages 5. Paraphrased natural-language questions 6. Stale-content traps where the strongest exact match is not the source you want ### Candidate recall - Did the required passage enter the candidate set at Recall@k? - How did MRR or nDCG change for each query class? - Did fusion or reranking help one class while hurting another? ### Answer grounding - Did the answer use the right passage? - Did citation correctness improve? ### System cost - What happened to retrieval, fusion, and reranking P50/P95 latency? - What infrastructure and tuning did the gain require? - Did lexical candidates consume reranker or context budget without improving recall? A polished answer can hide weak retrieval. The candidate list cannot. I add lexical retrieval when it fixes an observed evidence gap and the gain is worth the cost. I keep dense-only retrieval when it already passes the query mix I care about. ## Choose the retriever that preserves the evidence BM25 is old and embeddings are newer. Their age tells you nothing about recall on your corpus. The useful question is whether your retriever preserves every kind of evidence the generator needs. Use both signals when evaluation shows that one keeps losing required evidence. Keep the simpler system when it does not. ### Boolean Retrieval: AND, OR, and NOT in Search Canonical URL: https://ceisium.com/blog/boolean-retrieval/ Learn how Boolean search uses AND, OR, and NOT to combine result sets, where exact control helps, and why website search still needs ranking. Search does not always need to infer a user’s meaning. Sometimes the user knows which words are necessary or not permitted. For example: ```text api AND authentication invoice OR receipt webhook AND NOT legacy ``` These are Boolean queries. Boolean retrieval uses a match-or-no-match model. We will use a small documentation site as an example. First, Boolean operators select the matching pages. Then, the ranking system orders those pages. ## What is Boolean retrieval? Boolean retrieval finds documents that satisfy logical conditions. Each query term has a set of matching documents. Boolean operators combine those sets: * `AND` keeps documents that occur in both sets. * `OR` keeps documents that occur in either set or both sets. * `NOT B` keeps documents outside set B in a defined search universe. * `A AND NOT B` removes set B from set A. A document satisfies the Boolean condition, or it does not. Boolean operators give users direct control when they know which documents are acceptable. Boolean retrieval defines the matching logic. The search engine defines the query syntax, supported operators, and scoring behavior. ## A five-page documentation site All examples use the following five pages. A full-text search system usually analyzes page text before it searches. It converts selected text into indexed terms. The system also converts the query into compatible terms. It then gets the document set for each query term. For example: ```text Raw page text: "Create API keys" Indexed terms: api, key Match set for key: {D1, D2, D5} ``` The **Complete indexed terms** column contains all indexed terms for each page. | ID | Page | Representative text | Complete indexed terms | | -- | -- | -- | -- | | D1 | Create API keys | Create API keys; authenticated request. | `create`, `api`, `key`, `authenticated`, `request` | | D2 | API authentication | API authentication; bearer token; key. | `api`, `authentication`, `bearer`, `token`, `key` | | D3 | Verify webhook signatures | Webhook authentication; signing secret. | `webhook`, `authentication`, `signing`, `secret` | | D4 | Retry failed webhooks | Retry failed webhook delivery. | `retry`, `failed`, `webhook`, `delivery` | | D5 | Legacy API key migration | Legacy API key migration; deprecated. | `legacy`, `api`, `key`, `migration`, `deprecated` | The example analyzer does these operations: * It converts uppercase letters to lowercase. * It removes punctuation. * It reduces simple plurals such as `keys` to `key`. * It does not stem other words or add synonyms. These are the match sets: ```text api -> {D1, D2, D5} key -> {D1, D2, D5} authentication -> {D2, D3} webhook -> {D3, D4} retry -> {D4} legacy -> {D5} ``` The search engine can get these sets from an [inverted index](/blog/what-is-an-inverted-index/). The index records the documents that contain each term. ## AND means intersection `AND` requires both conditions. It does not require adjacent words. Exact phrase matching is a separate feature. Use this query: ```text api AND authentication ``` The `api` set is `{D1, D2, D5}`. The `authentication` set is `{D2, D3}`. `AND` keeps their intersection: ```text {D1, D2, D5} ∩ {D2, D3} = {D2} ``` Only D2 matches. D1 does not contain `authentication`, and D3 does not contain `api`. Now use this query: ```text webhook AND authentication ``` The two sets are: ```text webhook -> {D3, D4} authentication -> {D2, D3} ``` Their intersection is `{D3}`. `AND` usually makes the result set smaller. Each added condition removes documents that do not satisfy it. An `AND` condition can make a broad query more precise. ## OR means union Boolean `OR` is usually inclusive. A document can satisfy the left clause, the right clause, or both clauses. Use this query: ```text api OR authentication ``` The sets overlap at D2: ```text api -> {D1, D2, D5} authentication -> {D2, D3} ``` `OR` combines the sets: ```text {D1, D2, D5} ∪ {D2, D3} = {D1, D2, D3, D5} ``` D2 occurs once in the result set. D1 and D5 match `api`, and D3 matches `authentication`. `OR` usually makes the set larger. It is useful when two terms are acceptable alternatives: ```text invoice OR receipt error OR failure remove OR delete ``` The query lists the acceptable alternatives. Do not assume that the engine knows they are related. ## NOT means exclusion `NOT` starts with a defined search universe. It keeps the documents outside the excluded set. Our complete universe is: ```text U = {D1, D2, D3, D4, D5} ``` This query removes the `legacy` set: ```text NOT legacy ``` The result is: ```text U - {D5} = {D1, D2, D3, D4} ``` Now use: ```text api AND NOT legacy ``` The `api` set is `{D1, D2, D5}`. Remove D5 because it contains `legacy`: ```text {D1, D2, D5} - {D5} = {D1, D2} ``` Some query languages permit `api NOT legacy`. Others require `api AND NOT legacy`. Some languages do not permit a standalone `NOT` query. The parser grammar controls the valid forms. The search universe needs a clear boundary. The universe can be one site, project, language, or collection. Without this boundary, `NOT legacy` has no complete set to search.
## Parentheses show the intended order A query can contain several operators: ```text api OR webhook AND authentication ``` The query has two possible meanings. The first meaning is: ```text (api OR webhook) AND authentication ``` `api OR webhook` matches all five pages. The `authentication` condition reduces this set to `{D2, D3}`. The second meaning is: ```text api OR (webhook AND authentication) ``` `webhook AND authentication` matches D3. The final union is `{D1, D2, D3, D5}`. The two groupings return different document sets. Many Boolean systems apply `NOT` before `AND` and `AND` before `OR`. Do not assume that every product uses this order. Parsers can use different precedence rules, case rules, or symbols. Parentheses show the intended grouping. Use explicit parentheses for mixed operators. Also, use the syntax in the product documentation. ## Results from the example corpus | Query | Matching pages | | -- | -- | | `api AND key` | D1, D2, D5 | | `webhook AND authentication` | D3 | | `api OR authentication` | D1, D2, D3, D5 | | `api AND NOT legacy` | D1, D2 | | `NOT legacy` | D1, D2, D3, D4 | | `(api OR webhook) AND authentication` | D2, D3 | | `api OR (webhook AND authentication)` | D1, D2, D3, D5 | ## Text analysis changes the input sets Boolean matching usually operates on analyzed terms, not direct character strings. An analyzer can do these operations: * Convert text to lowercase * Split text into tokens * Remove punctuation * Reduce words to a common root * Remove common words * Expand synonyms * Apply language rules Our analyzer changes `API keys` to `api` and `key`. Thus, `key` can match a page that contains `keys`. The analyzer does not connect `authenticated` to `authentication`. Therefore, D1 is absent from the `authentication` set. A different stemmer can produce this larger set: ```text authentication -> {D1, D2, D3} ``` The Boolean logic is unchanged. `AND` still calculates an intersection. The analyzer changed the input sets. ```text Analyzer: Which indexed terms match this query clause? Boolean logic: How should the match sets combine? ``` You must know both parts to explain a Boolean result. ## Text clauses and structured filters are different A Boolean text clause searches analyzed content: ```text api AND key ``` A structured filter checks exact field values or metadata. This example is pseudocode, not portable syntax: ```text section = "guides" version = "v2" language = "python" ``` Text clauses can search titles, headings, and body content. Filters can check product, version, language, category, date, or workspace. A request can use both: ```text Text query: (api AND key) AND NOT legacy Filters: version = "v2" section = "guides" ``` The text query checks analyzed terms. The filters restrict the candidates with structured data. Authorization is a mandatory system constraint. User query syntax must not weaken it. For one user, the search universe must contain only permitted documents. A user-controlled `NOT private` clause is not access control. ## Useful cases for Boolean retrieval Boolean retrieval works well when the user knows precise terms. ### Require an important term ```text webhook AND authentication ``` This query removes general webhook pages that do not discuss authentication. ### Accept two names ```text member AND (remove OR delete) ``` The alternatives can find pages that use either verb. ### Exclude marked old content ```text api AND key AND NOT legacy ``` The query works only when `legacy` is a reliable marker. Structured version data is safer when it is available. ### Search exact technical terms Boolean clauses can combine error codes, function names, product names, fields, and protocols. ```text ERR_AUTH_42 AND webhook ``` The analyzer controls whether `ERR_AUTH_42` stays one term. Use the engine’s exact field and syntax when the complete identifier is necessary. ### Build advanced search controls An interface can provide these controls: ```text Must contain May contain Must not contain ``` The interface can convert them to Boolean clauses. Users do not need to learn the raw syntax. ## Candidate selection is not ranking Boolean retrieval answers one question: Which documents can match? Boolean retrieval does not always answer a second question: Which matching document must appear first? The query `api AND key` matches D1, D2, and D5. Basic Boolean logic treats all three as matches. A ranker can then use term frequency, title matches, or other evidence to set the order. [BM25](/blog/what-is-bm25/) is one method for lexical ranking. A system can enforce Boolean conditions and then rank the permitted documents. Some engines also let scored Boolean clauses affect the order. Check the documented behavior of your engine.
Boolean conditions select documents. Ranking signals put them in order. ## Limits of Boolean retrieval Boolean retrieval gives control, but the user must select useful terms. This query can be too strict: ```text api AND authentication AND bearer AND rotation ``` The result is empty if no page contains every analyzed term. This query can be too broad: ```text api OR authentication OR request OR token ``` Common terms can admit many weak candidates. `NOT` also needs care. `python NOT legacy` removes a current page that only mentions a legacy comparison. Boolean retrieval does not solve all vocabulary differences. `remove teammate` can miss a page named `Delete workspace member`. An analyzer, synonym rule, or explicit alternative can connect those terms. Search engines can also support phrases, proximity, field rules, wildcards, and fuzzy matching. Boolean operators do not define those features. ## Summary Boolean retrieval combines document sets: * `AND` calculates an intersection. * `OR` calculates an inclusive union. * `NOT` excludes a set from a defined universe. * Parentheses show the intended order for mixed operators. * Text analysis determines which documents enter each term set. * Structured filters operate on fields and metadata. * Authorization defines a mandatory permitted universe. * Boolean logic selects candidates, and ranking puts them in order. In our five-page corpus, `api AND NOT legacy` returns D1 and D2. D5 contains `api`, but the `legacy` condition removes it. Boolean retrieval has two steps. First, build the match sets. Then, combine them with explicit logical rules. ### Document Length Normalization: Why Long Pages Need Context Canonical URL: https://ceisium.com/blog/document-length-normalization/ Learn why long pages get more chances to match a search query, how BM25 adjusts for relative document length, and why shorter is not automatically better. A user searches documentation for: ```text rotate api key ``` Two pages match. The first page is a focused, 200-word guide named **Rotate an API key**. It contains the phrase two times. The second page is a 2,000-word security guide. It contains the phrase four times and covers many other security topics. A simple match count makes the long guide look better. It has four matches instead of two. However, the long guide has ten times more text. Its length gives it more opportunities to contain the query terms. The higher match count does not prove that the long guide is better. Its extra text gives it more opportunities to match. ## What is document length normalization? Document length normalization adds context to term counts. It compares term frequency with the length of the scored text. A short scoring unit can focus on one topic. However, BM25 sees only the term counts and field length. BM25 is the main example in this article. Other ranking methods can handle length differently. The calculations in this article apply to BM25. ## What counts as a document? The word *document* can mean a complete page. However, one indexed record can represent a page, section, product, or support article. A query can also score one field in that record, such as the title or body. In this article, the **scoring unit** is the record and field used in the term-frequency calculation. Search systems analyze text before ranking. Thus, length does not always mean the number of visible words. An **indexed token** is one searchable unit after analysis. The analysis rules depend on the search engine. We will use visible words in the first example. This makes the difference easy to see. For the calculation, assume the body fields contain 200 and 2,000 indexed tokens. The query has three terms. A complete BM25 score usually adds one contribution for each term. To isolate document length, we will calculate only the contribution from `rotate`. We will keep all other evidence equal. ## Raw repetition needs length context Term repetition can be useful evidence. Several uses of `rotate` can show that a page is about rotation. However, repetition can have two causes: 1. The page has a strong focus on the term. 2. The page has enough text to contain the term several times. The first example shows this difference: ```text Focused page 2 mentions / 200 words = 1 mention per 100 words Long guide 4 mentions / 2,000 words = 1 mention per 500 words ``` Term density makes the difference easy to see. BM25 does not calculate this ratio. Simple density can give too much value to very small pages. Each added nonmatching word also reduces the ratio directly. BM25 uses two controls instead. Term-frequency saturation reduces the value of each additional match. Length normalization adjusts for relative length. First, give both scoring units the same term frequency: ```text Short unit: 2 matches in 200 indexed tokens Long unit: 2 matches in 2,000 indexed tokens ``` Now only the length changes. The field, term rarity, and other query evidence stay equal. ## Length is relative to the collection A 2,000-token guide is long in a collection of short help pages. It can be normal in a collection of technical references. BM25 compares the field length with the average field length. The formula usually calls this average `avgdl`. Assume an average of 800 indexed tokens: ```text 200 / 800 = 0.25 -> shorter than average 800 / 800 = 1.00 -> average length 2,000 / 800 = 2.50 -> longer than average ``` `avgdl` uses every record in the selected statistics scope. It does not use only the current results. Index design determines whether `avgdl` compares similar records. Title fields and body fields can also have separate statistics. Thus, document length is relative. No universal word count makes a page too long.
## How BM25 uses relative length Our [BM25 guide](/blog/what-is-bm25/) explains the complete ranking method. This section isolates the length component. Start with the length factor: ```text length factor = 1 - b + b * (dl / avgdl) ``` `dl` is the indexed field length. `avgdl` is the average length in the same statistics scope. The parameter `b` controls the strength of the length adjustment. When `b` is greater than zero: * A short field has a smaller length factor. * An average field has a length factor of `1`. * A long field has a larger length factor. BM25 puts this factor in the denominator of the term-frequency fraction: ```text term contribution before IDF = (tf * (k1 + 1)) / (tf + k1 * length factor) ``` A larger length factor makes the denominator larger. The same numerator divided by a larger denominator gives a smaller contribution. A common form of one BM25 term contribution is: ```text term contribution = IDF(t) * ((tf * (k1 + 1)) / (tf + k1 * (1 - b + b * dl / avgdl))) ``` The other symbols have these meanings: ```text tf = frequency of the query term in the scored field k1 = control for term-frequency saturation IDF = rarity of the term in the same statistics scope ``` Our two example fields use the same statistics scope. Therefore, the tracked term has the same IDF in both fields. The [TF-IDF guide](/blog/what-is-tf-idf/) explains term rarity. Some implementations omit or absorb rank-equivalent constants such as `k1 + 1`. This change does not alter the order for one BM25 configuration. ## A small calculation Use these illustrative values: ```text k1 = 1.2 b = 0.75 avgdl = 800 tokens tf = 2 ``` For the 200-token page: ```text length factor = 1 - 0.75 + 0.75 * (200 / 800) = 0.4375 term contribution before IDF = (2 * 2.2) / (2 + 1.2 * 0.4375) ≈ 1.74 ``` For the 2,000-token guide: ```text length factor = 1 - 0.75 + 0.75 * (2,000 / 800) = 2.125 term contribution before IDF = (2 * 2.2) / (2 + 1.2 * 2.125) ≈ 0.97 ``` The query term and its frequency are equal. The long scoring unit gets a smaller contribution because it contains much more indexed text. These values show only the length effect for one term. They are not complete document scores. Search engines can also report BM25 scores on different numeric scales. ## The b parameter controls sensitivity The parameter `b` controls the sensitivity to relative length. At `b = 0`: ```text length factor = 1 ``` This setting removes length normalization. It does not remove IDF or the `k1` saturation behavior. At `b = 1`: ```text length factor = dl / avgdl ``` At `b = 1`, BM25 uses the complete length ratio. The BM25 score still does not prove that a page is relevant. A value between zero and one gives a smaller adjustment. Lucene 10.4.0 uses `0.75` by default. Lucene's default is not the best value for every search system. When `dl` equals `avgdl`, the factor is always `1`. The selected `b` value does not change this result. The average-length unit stays fixed.
## Return to the original term counts The first calculation used equal term frequencies. Now restore the original counts: ```text Focused page: 2 matches in 200 tokens Long guide: 4 matches in 2,000 tokens ``` The focused page contribution stays near `1.74`. The long guide contribution increases from about `0.97` to about `1.34`. The extra two matches help the long guide. They do not remove the length context. The long guide can still rank first when it has stronger evidence. It can match more terms, rarer terms, or other ranking signals. Long pages are not bad results. Length normalization changes how a lexical ranker interprets repetition. It does not make shortness equal to relevance. ## The scoring unit changes the calculation The index defines the scoring unit. A change to the record or field boundary changes the length comparison. Assume the 2,000-word guide is one indexed page. Its four uses of `rotate` occur in one long body field. Now split the same guide into six records of approximately 300 tokens. The **Rotate an API key** section gets its own body field. The source page did not become shorter. The scoring unit became shorter. The chunk has its own `dl`. Its `avgdl` comes from all chunks in the selected statistics scope. Poor chunk boundaries can separate an answer from necessary context. Thus, smaller scoring units are not always better. Fields can work in the same way. A title can use title-length statistics, while the body uses body-length statistics. The exact behavior depends on the index and search engine. ## Limits of length normalization Length normalization uses a small set of numbers. It does not know why a page is long. A page can be long because it is repetitive. Another page can be long because it gives a complete explanation. Both pages can have the same `dl`. One average can also hide different content types. An index can contain 30-token products and 3,000-token tutorials. Separate fields, indexes, or content types can make the comparison clearer. Boilerplate can also increase length. Navigation, legal text, and generated metadata add tokens when extraction puts them in the scored field. Finally, BM25 is lexical. It does not directly understand intent, freshness, authority, or different phrases with the same meaning. Search systems can add field weights, filters, semantic retrieval, or reranking. Length normalization reduces the extra matching opportunity that comes from longer text. It does not rank documents by itself. ## Questions for your search index Before you change `b`, answer these questions: 1. **What is the scoring unit?** Is it a page, chunk, record, title, or body? 2. **Which analyzed tokens count?** Does the scored field contain boilerplate? 3. **Which units produce `avgdl`?** Does the scope mix very different content types? 4. **What do labeled relevance judgments show?** Do long pages win because of useful coverage or extra matching opportunities? The last question is the most important. Test the selected `b` value with real queries and labeled results. ## Frequently asked questions ### Does BM25 always prefer shorter documents? No. A result still needs matching query terms. Length changes only the contribution from those matches. ### Is there an ideal document length for search? No. BM25 compares a field with its statistics scope. It does not use a universal word-count target. ### How does chunking affect length normalization? Chunking changes the indexed records and field lengths. It can change both `dl` and the records that produce `avgdl`. ## Conclusion Do not use length normalization as a reason to make all pages short. Use it to understand what the ranker compares. Define the scoring unit. Examine the indexed tokens and the statistics scope. Then tune the system with representative searches. ## Sources - Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze, *Introduction to Information Retrieval*: [Pivoted normalized document length](https://nlp.stanford.edu/IR-book/html/htmledition/pivoted-normalized-document-length-1.html) - Stephen Robertson and Hugo Zaragoza, [*The Probabilistic Relevance Framework: BM25 and Beyond*](https://doi.org/10.1561/1500000019) - Apache Lucene, [`BM25Similarity` documentation](https://lucene.apache.org/core/10_4_0/core/org/apache/lucene/search/similarities/BM25Similarity.html) - Apache Lucene, [`BM25Similarity` source](https://github.com/apache/lucene/blob/releases/lucene/10.4.0/lucene/core/src/java/org/apache/lucene/search/similarities/BM25Similarity.java) - Elasticsearch, [*Practical BM25 — Part 2*](https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables) ### How Website Crawling Shapes Search Quality Canonical URL: https://ceisium.com/blog/how-website-crawling-shapes-search-quality/ Learn how crawl scope, sitemaps, rendering, URL identity, and recrawl freshness shape the pages a site-search ranker can choose from. A documentation team publishes a new authentication guide at `/docs/v2/authentication`. The page is public, and a direct link opens it. A user then searches for “authentication setup.” Search shows the old `/docs/auth` page. Do not start by changing the ranking settings or semantic model. First, check whether the search corpus contains the new guide. Website crawling discovers eligible URLs and gets their content for an index. Site search and search-engine optimization can use different crawl rules. Verify which directives and signals your site-search crawler supports. Ranking starts after crawling and indexing. It can order known pages, but it cannot use content that the index never received. ## Search needs a correct corpus before ranking A site-search system usually has this input path: 1. **Scope** defines where the crawler can go. 2. **Discovery** gives URLs to the crawler. 3. **Fetch, render, and extraction** produce useful content. 4. **Indexing** stores the content in an [inverted index](/blog/what-is-an-inverted-index/) or another search structure. 5. **Retrieval** selects candidates for one query. 6. **Ranking or reranking** puts those candidates in order. The first four stages build the search corpus. Retrieval then selects a smaller candidate set from that corpus. A page outside the corpus cannot become a candidate. Four checks help you examine the corpus: * **Coverage:** Did the crawl include the correct pages? * **Content fidelity:** Does each indexed document contain the useful page content? * **Identity:** Does one stable document represent each real page? * **Freshness:** Does the document contain the current page content? Coverage, content fidelity, identity, and freshness describe the corpus that the first four stages produce.
## Scope defines where the crawler can go A crawler must not follow every URL that it finds. A documentation crawl can include pages under `/docs/`. It can exclude account settings, site-search URLs, calendars, and unlimited filter combinations. The crawl can also include one host and exclude other company hosts. Scope defines this boundary. A search source can use a base URL with included and excluded paths. In our example, `/docs/v2/authentication` is inside the `/docs/` scope. A guide at `help.example.com/authentication` is not automatically in the same corpus. Scope makes a URL eligible, but it does not always reveal that URL to the crawler. A start URL can reveal the first page. Other pages need a discovery path through a link, redirect, or sitemap. ## Discovery gives URLs to the crawler A page can be public but still be unknown to a crawler. Use these terms carefully: * A **public** page does not require authentication. * A **reachable** page gives a usable response at its exact URL. * A **discoverable** page has a path that gives its URL to the crawler. Opening a URL manually only proves that the page is reachable for you. It does not prove that the crawler can discover it. Crawlers can find URLs through start pages, eligible links, sitemaps, and redirect destinations. The new authentication guide can be public and reachable. However, the navigation can still point to the old page. If the sitemap also omits the new route, the URL is in scope but unknown. A sitemap gives the crawler a list of URLs. It is especially useful for new pages without internal links. A sitemap entry does not guarantee indexing. The URL can fail during scope checks, fetching, rendering, extraction, or indexing. The opposite is also true. Removal from a sitemap does not delete a page. Other links can still lead to the old URL. ## A successful fetch can produce a poor document After discovery, the crawler requests the URL. A server-rendered response can already contain the title, headings, paragraphs, and code examples. A client-rendered response can initially contain only a container and a script. ```html
``` If the first HTML response lacks content, a renderer can run the page scripts. Extraction then selects content from the rendered page. Rendering and extraction are different operations. The crawler can render a complete guide but store a poor document. A useful document can contain the title, headings, steps, code, description, and URL. A poor document can contain repeated navigation or duplicate hidden text. The extracted document can also omit the code sample or separate a heading from its section. More indexed text does not always improve search. The indexed document must keep the text that gives meaning to the page. Content fidelity measures how well the indexed document preserves useful page content. A noisy or empty document can still miss the query. ## URL identity controls duplicate documents The old `/docs/auth` route now has a replacement at `/docs/v2/authentication`. Other URL forms can also exist: ```text /docs/auth /docs/auth/ /docs/auth?source=nav ``` Treat two URL forms as one page only when the site shows that they are equal. A permanent redirect provides useful evidence. Do not assume that every query parameter or path difference has no meaning. Redirect status codes also give different information. `301` and `308` state that a move is permanent. `302` and `307` state that it is temporary. A supported `rel="canonical"` tag is another publisher hint. It does not redirect a visitor, and each crawler can handle it differently. Identity rules need limits. Supported `v1` and `v2` guides can share text but contain different instructions. Do not merge them only because their text is similar. Identity rules should produce one stable search document for each real piece of content. ## Freshness controls the stored version Identity selects the document that represents a page. Freshness shows whether that document contains the current content. An index is a stored copy of the website. When the team changes a code sample, search does not get that change immediately. The crawler must revisit the source, and indexing must replace the old document. Two failures can make search stale: 1. The crawler did not revisit the page. 2. The crawler got the new page, but indexing did not replace the old document. Readers see stale search results in both cases. Frequent recrawls can reduce delay. They also increase crawler work and load on the source site. A dynamic documentation site can need a different schedule from a static marketing site. Removal also needs a clear policy: * A permanent `301` or `308` states that a page moved. * A `404` states that the origin has no current representation. * A `410` states more clearly that the removal is intentional. * A soft 404 returns `200` with a “page not found” message. * A timeout does not show that a page was removed. Do not delete the last good document after one timeout. Also, do not keep a confirmed obsolete page forever.
## Ranking cannot repair a broken corpus [Lexical retrieval](/blog/what-is-bm25/) matches query terms. [Semantic retrieval](/blog/what-is-vector-search/) can match related meaning. A ranker then orders the candidates. Every search stage still needs indexed evidence. An undiscovered authentication page has no document for retrieval. An empty extracted document has no authentication steps to match. The old page can still appear if the index keeps it. Duplicate URL forms can also occupy several candidate positions with the same answer. A ranker can reorder known candidates. It cannot accurately rebuild a missing page or confirm a deletion. Check crawling and indexing before you change ranking weights or models. ## Diagnose search quality in pipeline order Use the corpus and [crawl telemetry](/docs/crawl-logs/) as evidence. Outcomes such as discovered, indexed, skipped, and failed show where a page stopped. Ask these questions in order: 1. **Scope:** Is the URL in the permitted host and paths? 2. **Discovery:** Can a start URL, link, redirect, or sitemap supply the URL? 3. **Fetch:** Which status and final URL did the request return? 4. **Render:** Did meaningful content appear after required scripts ran? 5. **Extract:** Does the document contain the correct title, headings, body, and code? 6. **Identity:** Did proven aliases resolve to the intended document? 7. **Freshness:** Does the indexed document match the current page? 8. **Retrieval:** Does the page enter the candidate set for the query? 9. **Ranking:** Does the system put the best candidate too low? Search products can implement these stages differently. The sequence helps you test the corpus before you tune the result order. ## Make the input layer easy to verify Use these checks to prevent common failures: * Compare important pages with the sitemap and discovered-URL list. * Examine the extracted document, not only the page in a browser. * During URL changes, verify permanent redirects and preserve real page versions. * After a large change, start a supported fresh crawl and examine the new corpus. * Test queries only after the current page and intended URL are in the index. ## Frequently asked questions ### Does a sitemap guarantee that a page appears in site search? No. A sitemap supplies a URL to try. The page must still pass all crawl and indexing checks. ### Can a crawler index a JavaScript-rendered website? Sometimes. A rendering step can expose content that is absent from the first HTML response. Support and failure modes depend on the crawler. ### What is the difference between a redirect and a canonical tag? A redirect sends a client to another URL. A canonical tag names a preferred URL for crawlers that support the hint. ### How often should a website be recrawled? Match the schedule to the change rate, freshness target, and permitted load on the source site. ## Fix the input layer before the ranker When search shows the wrong page, first verify coverage, content fidelity, identity, and freshness. After you verify the corpus, you can measure retrieval and ranking changes with more confidence. ### Semantic Search vs Keyword Search: How to Choose and When to Combine Them Canonical URL: https://ceisium.com/blog/semantic-search-vs-keyword-search/ Keyword search uses literal evidence. Embedding-based semantic search bridges vocabulary gaps. Learn where each fails and when hybrid retrieval earns its added complexity. A user searches a help center for `stop paying every month`. The help center has three pages: 1. Cancel subscription 2. Pause subscription 3. Change billing cycle All three pages relate to the query, but they describe different actions. The query does not say which action the user wants. Keyword search uses shared terms as evidence. Semantic search can connect a query to pages that use different words. Both methods can help. Both methods can also return a plausible but incorrect page. ## How keyword search finds pages Use a reverse word map as the basic model for keyword search: ```text cancel -> Cancel subscription pause -> Pause subscription monthly -> Change billing cycle PLAN-4821 -> Change billing cycle ``` Keyword search is also called lexical search. A lexical search system builds this map before users search. When a query arrives, its terms point to pages with the same indexed terms. This map is an [inverted index](/blog/what-is-an-inverted-index/). Real lexical systems use more evidence than simple term presence. They can use the field, term rarity, and term frequency. They can also limit the value of repeated terms. [BM25](/blog/what-is-bm25/) is one method for ranking lexical evidence. Keyword ranking starts with text shared by the query and page. Keyword search is strong when literal text has important meaning: * Plan IDs such as `PLAN-4821` * Product and model names * Software versions * Error codes * Quoted phrases * Commands and configuration keys A search for `PLAN-4821` must preserve the identifier. It must not depend only on a model’s idea of a billing concept. The field and analyzer must keep the identifier intact. A general text analyzer can split or change it. ### Keyword search is not the browser Find function Keyword search does not always require identical character strings. An analyzer can convert text to lowercase, split punctuation, stem words, or expand synonyms. Other features can correct spelling or permit fuzzy matches. These features help, but they do not remove every vocabulary gap. A rule can connect `stop paying` to `cancel subscription`. Users can also write `end my membership` or `turn off renewal`. Known rules solve known differences. They become difficult to maintain for many forms of natural language. Context also changes meaning. `Cancel my trial` and `cancel my order` share a verb but need different pages. Semantic search addresses this vocabulary problem. ## How semantic search connects related language Semantic search uses meaning as search evidence. This article uses similarity between text embeddings as the semantic mechanism. An embedding model converts each page passage into a numeric representation. It also converts the query into the same type of representation. The search system compares the query embedding with the page embeddings. It retrieves the closest representations. One number in an embedding does not represent a label such as “billing.” The complete embedding represents the text in a model-specific comparison space. The model tries to put related uses near each other. This article uses text embeddings for semantic retrieval. For our query, semantic retrieval can show all three subscription pages. The pages do not need to contain `stop paying every month`. The system can match the query to these pages even though they use different words. However, semantic similarity cannot resolve the missing action. The query does not say whether the user wants to cancel, pause, or change the billing cycle. Semantic retrieval can supply related candidates. It cannot create missing user intent. The [vector search guide](/blog/what-is-vector-search/) explains embeddings and similarity in more detail. ### Chunking defines the retrieval unit The system must select the text unit before it creates embeddings. Many systems split long pages into passages called chunks. Assume the cancellation guide contains these sentences: ```text Open Billing and select Cancel subscription. Cancellation takes effect immediately and removes remaining access. ``` Different chunks can separate the action from its consequence. Search can retrieve the action without the warning. Small chunks can make retrieval more specific. They can also divide one complete idea. The system can attach nearby passages after retrieval. It cannot attach a passage that retrieval and context enrichment both omit. ## Keyword search and semantic search compared
| Question | Keyword search | Embedding-based semantic retrieval | | -- | -- | -- | | Main signal | Shared indexed terms and lexical evidence | Similarity between model-made text representations | | Strongest uses | Names, IDs, versions, codes, phrases, and known terms | Paraphrases, natural-language descriptions, and vocabulary gaps | | Possible misses | Useful pages that use different words | Literal details that have weak representation | | Possible false matches | Pages with shared words but incorrect context | Pages with a related topic but incorrect action | | Required system | Text analysis, lexical ranking, and usually an inverted index | An embedding model and similarity retrieval; often a vector index and chunks | | Does a high score prove correctness? | No | No | The two methods use different evidence. The better method depends on the words and intent in each query. ## Similar does not mean correct Related does not mean correct. Compare these queries: ```text cancel my subscription do not cancel my subscription ``` The queries share nearly all words, but their requested actions are opposite. Lexical overlap alone does not solve the difference. An embedding model can also place them near each other because both discuss cancellation. Missing answers create another problem. Assume the user searches for `transfer my subscription to another person`. The help center has no transfer page. Semantic retrieval can still return cancellation or billing pages because they are the nearest topics. The nearest page is not automatically an answer. A search experience can need a no-result response or a question for the user. It can also reject candidates below a tested threshold. Keyword search has a similar failure. An old page can contain more exact terms than the current page. Strong relevance evidence does not prove that a result is current or permitted. Filters and access controls must define the eligible documents. ## Hybrid search uses both signals Keyword and semantic retrieval fail in different ways. Hybrid search lets both methods supply candidates. The result lists for our query can look like this: | Rank | Keyword list | Semantic list | | -- | -- | -- | | 1 | Change billing cycle | Cancel subscription | | 2 | Cancel subscription | Pause subscription | | 3 | Pause subscription | Change billing cycle | These lists are examples. They do not describe one production system. Use this simple model for the merge: 1. Put the pages from both lists into one set. 2. Remove duplicate pages. 3. Give more value to pages near the top of both lists. 4. Produce one combined shortlist.
The rule that combines lists is called **fusion**. A later **reranker** can examine the merged candidates and change their order. Keyword and vector scores do not use one common scale. A keyword score of `8` and a vector score of `0.8` measure different things. A system can combine ranks or normalize scores under a tested configuration. It can also apply weights or rerank the merged set. Filters have a different purpose. They can define language, version, freshness, workspace, or permissions. Some filters can operate before retrieval, and others can operate later. Access control must always remain a hard rule. The system must enforce access control before it exposes a result to an unauthorized user or component. A reranker cannot restore a page that both retrieval methods missed. Hybrid search can improve coverage, but it adds work: * Another retrieval path to configure and observe * A fusion or weighting decision * More candidates and duplicate pages * Possible latency, storage, and compute costs * More query groups to test The second retrieval path must produce a measured improvement that justifies this extra work. Our [BM25 in RAG article](/blog/bm25-in-rag/) applies the same evidence problem to retrieve-then-generate systems. ## When to use keyword search Start with keyword search when literal text is the main evidence. It also fits collections with controlled terminology. Keyword search is often a good first choice when: * Users search for names, identifiers, versions, codes, or quoted text. * The collection uses consistent terms. * Synonyms and field rules cover important known gaps. * Low latency and simple operations are important. * Tests do not show a large vocabulary gap. Keyword search can use carefully designed analyzers, fields, filters, and ranking rules. ## When to use semantic search Semantic retrieval helps when users describe a need with different words from the content. It is useful when: * Queries are natural-language descriptions. * Writers and readers use different terms. * Many phrases can express the same idea. * A useful passage shares few terms with the query. * Tests show that lexical retrieval misses useful candidates. Do not assume that semantic-only retrieval is better than keyword retrieval. Test exact identifiers, negation, missing answers, chunk boundaries, language coverage, and model changes. ## When to use hybrid search Use [hybrid search](/blog/what-is-hybrid-search/) when the same product receives both types of query. A help center can receive `how do I stop the next charge?` and `PLAN-4821`. The first query benefits from semantic evidence. The second query needs a preserved identifier. Use hybrid search when each retrieval path finds useful evidence that the other path misses. Keep the simpler method when one path already meets the target for the important query mix. Use this decision rule: > Add another retrieval signal only when it fixes a measured evidence gap. The quality gain must justify the latency, cost, and tuning work. ## How to evaluate the methods Do not choose a search method from a few impressive examples. Build a test set from real query types. Use three query groups. ### 1. Paraphrase queries Use different words for the same need: * `stop paying every month` * `end my plan permanently` * `pause payments for two months` These queries test vocabulary gaps and different user actions. ### 2. Exact-evidence queries Use strings that must keep their literal form: * `PLAN-4821` * Product and model names * Versions * Error codes * Commands and configuration keys These queries test whether exact evidence survives retrieval and ranking. ### 3. Safety and constraint queries Include these cases: * Opposite instructions with similar words * Queries with no answer * Old-version traps * Language constraints * Permission-sensitive documents * Important warnings near chunk boundaries Label the useful results for each query. Then measure candidate inclusion and final rank. This separates two failures: * **Retrieval failure:** The useful page does not enter the candidate set. * **Ranking failure:** The useful page becomes a candidate but appears too low. Select metrics that fit the labels: * Hit@k asks whether at least one required result occurs in the first k positions. * Recall@k measures how many labeled results occur in those positions. * Mean Reciprocal Rank gives more value to the first relevant result near the top. * nDCG supports several results with different relevance grades. Also measure P95 latency. This is the response time that 95 percent of measured requests do not exceed. A relevance gain is not useful when the added response time exceeds the product's latency target. ## How to add semantic retrieval safely Do not replace keyword search only because a semantic demonstration looks better. Start with measured lexical failures. Use this sequence after tests show a candidate gap: 1. Collect queries where keyword retrieval missed useful pages. 2. Add semantic retrieval beside the existing lexical path. 3. Compare keyword, semantic, and merged candidates for all query groups. 4. Keep the second path only when its benefit justifies the measured cost. A feature flag and a compatible lexical index can make rollback easier. Do not move every query to semantic search. Preserve the exact and semantic evidence that real queries need. ## Frequently asked questions ### Is semantic search the same as vector search? No. Semantic search uses meaning as evidence. Similarity between text embeddings is one common method. Vector search can also compare images, audio, and other data. ### Is keyword search only exact matching? No. Keyword search can normalize text, use phrases, expand synonyms, and support fuzzy or spelling features. Exact identifiers still need a field and analyzer that preserve them. ### Is hybrid search always better? No. It can add weak candidates, latency, storage, compute, and tuning work. Use it when tests show that the second signal recovers enough useful evidence. ## Conclusion Keyword search and semantic search use different evidence. Keyword search is strongest when literal text matters. Semantic search helps when users and writers use different words for the same need. Hybrid search combines both signals when real queries need both. The query `stop paying every month` has no guaranteed single answer. Better retrieval can show the three useful choices. The search experience must still handle the ambiguity. Choose the simplest retrieval method that preserves the necessary evidence. Add another signal only after measured failures show why it is necessary. ### TF-IDF: The Search Ranking Idea Behind BM25 Canonical URL: https://ceisium.com/blog/what-is-tf-idf/ TF-IDF is a classic lexical ranking method that scores documents using repeated query words and rare query words. Learn how it works, where it helps, and how BM25 addresses some of its limits. TF-IDF is a simple way to answer a search-ranking question: which page uses the user's words in the most useful way? Search ranking starts with a small practical problem. A user types: ```text reset api token ``` The search system may have hundreds or thousands of docs, support articles, API pages, changelog entries, and product pages. Many of them contain one of those words, but only a few are probably the right result. Here, a document means one searchable page, the collection means all searchable pages, and a query term means one word from the user's search. TF-IDF is one way to sort those pages. It is not semantic search, and it does not understand the query like a person. It asks a simpler question: ```text Which pages use the query words, and which of those words are specific enough to matter? ``` That small question is enough to make many exact-word searches work. ## What is TF-IDF? TF-IDF means: ```text term frequency x inverse document frequency ``` It is a ranking method for lexical search. Lexical search works with matching words. If the user searches for `webhook signature`, the system looks for pages that contain `webhook`, `signature`, or both. TF-IDF gives each matching document a score for this query. A higher score means the document looks more relevant for that search. The same page can get different TF-IDF scores for different searches. The two parts do different jobs: - Term frequency asks how often the query word appears in this document. - Inverse document frequency asks how rare that word is across the whole collection. Term frequency is local. Inverse document frequency is global. That is the core idea. A word matters more when it appears in the current document and does not appear everywhere else.
## Why not just count words? Let us start with a very simple search system. The user searches: ```text api token ``` We have two pages. Page A: ```text API tokens Create, rotate, and revoke API tokens. ``` Page B: ```text Authentication guide This guide explains API access, token scopes, token rotation, token permissions, API clients, and API examples. ``` If we only count words, Page B can look stronger because it says `api` and `token` many times. Page B wins raw count because it repeats both words. But Page A may be the better result because it is short and focused. Raw word counting has another problem. Common words appear everywhere: ```text docs page guide use setup ``` If a word appears on almost every page, matching it does not tell us much. A page that contains `docs` is not special. A page that contains `hmac` or `retry-after` is much more specific. TF-IDF fixes part of this by making common words quieter and specific words louder. ## Term frequency Term frequency is the easy half. It asks: ```text How often does this query term appear in this document? ``` If someone searches: ```text webhook signature ``` Then a page that talks about webhook signatures several times is probably more relevant than a page that mentions the phrase once in a footer. For example: | Page | `webhook` mentions | `signature` mentions | | --- | ---: | ---: | | Verify webhook signatures | 8 | 6 | | Webhook delivery retries | 10 | 1 | | Integrations overview | 1 | 0 | For now, ignore page length. We are only looking at whether both query words appear strongly. In this tiny example, the first page looks strongest because both query terms appear strongly there. This is the useful part of term frequency. Repeated words can be evidence. But we should be careful. Repetition is evidence, not proof. A page can repeat a word because it is long, noisy, or covering many topics. ## Inverse document frequency Inverse document frequency is the more interesting half. It asks: ```text How rare is this term across all documents? ``` Imagine a docs site with 1,000 pages. Rarity does not prove relevance. It changes how much a match should count. | Term | Pages containing the term | What it means | | --- | ---: | --- | | `docs` | 950 | tiny signal | | `webhook` | 120 | useful signal | | `hmac` | 8 | strong signal | If the query is: ```text webhook hmac signature ``` then `hmac` should matter a lot. It appears on very few pages, so it helps separate the specific pages from the general webhook pages. This is why IDF is useful for product and technical search. Product names, API fields, error codes, config keys, CLI flags, and protocol names often carry the real intent. IDF is about separation. ## The TF-IDF formula At a high level, TF-IDF looks like this: ```text tf-idf(term, document) = tf(term, document) x idf(term) ``` If the query has more than one term, the search system calculates a score for each query term and adds the scores together. ```text score(document, query) = tfidf("webhook", document) + tfidf("signature", document) ``` Different systems use different versions of term frequency and IDF. Some use raw counts, some use logarithms, and some normalize vectors. You do not need every variant to understand the ranking idea. The plain-English version is enough: ```text score = query words used in this document x how rare those query words are across the collection ``` Common words whisper. Specific words speak loudly. ## A small TF-IDF example Let us use a tiny docs search example. The user searches: ```text reset api token ``` The collection has three pages. | Page | What it says | | --- | --- | | A: Reset API tokens | Short API page about rotating and resetting API tokens | | B: Complete authentication guide | Long guide with many mentions of API clients and tokens | | C: Account settings overview | General page that mentions reset once | Now look at the query terms: - `api` may appear on many pages. - `token` appears on fewer pages. - `reset` may appear on a smaller set of settings and recovery pages. The ranking might look like this: | Rank | Page | Why | | ---: | --- | --- | | 1 | A: Reset API tokens | Matches all three query terms on a focused page | | 2 | B: Complete authentication guide | Matches `api` and `token`, but the page is broader | | 3 | C: Account settings overview | Mentions `reset`, but misses the stronger API token context | TF-IDF is meant to prefer Page A because the query terms are concentrated there, and the more specific terms help separate it from general pages. This is why TF-IDF is a useful baseline. It can rank exact terminology before the search system understands anything about meaning. ## Where TF-IDF works well TF-IDF works well when the user and the document use the same words. That happens a lot in technical search: - `api token` - `webhook signature` - `429 retry-after` - `rate limit` - `sso setup` - `npm install` - `connection refused` - `delete workspace` These are good lexical queries. The exact words matter. In docs and product search, exact words are not a small detail. They are often the whole point. If a user searches for an error code, an API field, a product setting, or a command, they expect those words to match. TF-IDF is good when the query vocabulary and document vocabulary overlap. ## Where TF-IDF struggles TF-IDF struggles when the meaning matches but the words do not. For example: | User searches | Useful page says | | --- | --- | | `sign in problem` | `login troubleshooting` | | `single sign on` | `SSO` | | `payment failed` | `invoice collection error` | | `make search faster` | `latency optimization` | A human can see the connection. TF-IDF may not. It mostly sees word overlap. If the words are different, the score can stay low.
Search systems can work around this with synonyms, stemming, query rewriting, semantic retrieval, or hybrid search. You do not need all of those pieces yet; they are different ways to add help around word matching. TF-IDF itself is still a word-based method. ## Do titles and headings matter? The core TF-IDF idea does not automatically know that a title is more important than the body. But real search systems usually have fields: - Title - URL - Headings - Body - Meta description - Anchor text A match in the title often deserves more weight than a match deep in the body. For example: ```text title score x title weight + heading score x heading weight + body score x body weight ``` This is a layer around the lexical scoring method. The scoring method handles term frequency and rarity. Field weighting tells the system where the match happened. ## How BM25 addresses TF-IDF's limits BM25 keeps a useful TF-IDF-style idea: ```text important pages use the query terms, especially specific terms ``` But BM25 adds better controls for two problems: - Repetition should not grow forever. - Long documents should not win just because they have more words. You can read the [plain-English BM25 guide](/blog/what-is-bm25/) for the full walkthrough. The short version is that BM25 is still lexical ranking, but it is more careful about the repetition and length problems simple TF-IDF leaves unfinished. ## Repetition should flatten out Suppose the user searches: ```text token ``` A page that says `token` once gives us useful evidence. A page that says it five times gives stronger evidence. But a page that says `token` fifty times should not be ten times better than the page that says it five times. At some point, the search system should say: ```text Okay, this page is clearly about tokens. More repetition is not adding much. ``` BM25 handles this with term frequency saturation.
Many real TF-IDF implementations soften repetition too, but BM25 is the common named model built around this saturation behavior. ## Long documents need fair treatment Long pages have more chances to match a query. A 10,000-word guide might mention `billing` once, `invoice` once, and `retry` once because it covers everything. A short support page might focus only on failed invoice retries. If we do not adjust for length, the long guide can get too much credit. BM25 includes document length normalization. A match inside a short focused page can be stronger than the same match buried inside a long page. This does not mean long pages are bad. A long guide can still be the best answer. The goal is only to avoid rewarding length by accident. ## Is TF-IDF obsolete? No. It is old, but old does not mean useless. TF-IDF is still useful as a teaching model because it explains the two signals behind many lexical rankers: - How much does this document use the query term? - How rare is that term across the collection? In production search, BM25 is often a stronger default lexical ranker because it handles repetition and length better. Modern systems may also add vector search, reranking, synonyms, filters, personalization, freshness, or learning-to-rank signals. But TF-IDF is still the clean bridge between simple word counting and stronger lexical ranking. ## Summary TF-IDF is a simple ranking method for lexical search. It combines: - Term frequency: how often a query word appears in a document. - Inverse document frequency: how rare that word is across the collection. It works well when exact words matter. That makes it useful for docs, API references, support articles, product search, and technical search. It struggles when the same idea is written with different words. It also does not handle repetition and document length as carefully as BM25. TF-IDF scores a document for a query, not for all time. It matches words rather than understanding meaning. Once you understand TF-IDF, BM25 becomes easier to understand. BM25 is not a totally different idea. It keeps the useful word-counting and rarity model, then makes it behave better for real search results. ### What Is an Inverted Index? Canonical URL: https://ceisium.com/blog/what-is-an-inverted-index/ An inverted index maps terms to the documents that contain them, helping search engines find text matches without scanning every page. A user searches a help site for: ```text reset password ``` The site has four pages. In this example, the search system indexes both the page title and the text below it. | ID | Page title | Text | | -- | -- | -- | | P1 | Manage your account | Manage your account and password. | | P2 | Reset a forgotten password | Reset a forgotten password. | | P3 | Billing help | Download invoices and update payment details. | | P4 | Protect your account | Protect your account with two-factor authentication. | The system could open all four pages and check every word. That works here, but repeating the scan for every query becomes wasteful as the collection grows. Instead, it prepares a lookup before the user searches. That lookup is an inverted index. ## What is an inverted index? An inverted index maps each searchable term to the documents that contain it. For our four pages, part of the index looks like this: ```text account -> P1, P4 password -> P1, P2 reset -> P2 payment -> P3 authentication -> P4 ``` This is similar to the index at the back of a book. You start with a topic and get the pages where it appears. The mapping is called inverted because it reverses the direction we begin with. The original content starts as pages containing words. Search needs words pointing to pages. The set of searchable terms is often called the **vocabulary** or **lexicon**. A term dictionary organizes those terms and points to their postings lists. The list beside each term is a **postings list**. The postings list for `password` is `P1, P2`. Each document entry inside that list is one **posting**.
## How is an inverted index built? The system first extracts the text it wants to make searchable. It then breaks that text into smaller units called tokens. For example: ```text "Reset a Forgotten Password" ``` may become: ```text reset, a, forgotten, password ``` The system may also make text lowercase or handle punctuation. These choices make document terms and later query terms compatible. The exact processing varies by implementation. Each final term is added to the postings for the field being indexed. Some systems keep separate term dictionaries and postings for fields such as titles and bodies. ## How does search use the index? Now the user searches for: ```text reset password ``` The search system analyzes the query into terms compatible with those stored in the index: ```text [reset, password] ``` Then it reads the two postings lists: ```text reset -> P2 password -> P1, P2 ``` P2 appears in both lists, so it contains both query terms. P1 appears only in the `password` list, so it is a partial match. Whether P1 is retained depends on whether the search requires every term or allows partial matches. The shared part of the lists is their **intersection**. For this query, the intersection is just P2. The engine can start with these short term-led lists instead of reopening every page to discover possible matches. Those possible matches are the candidates that a ranking method can order.
## Does an inverted index rank the results? The index retrieves candidates and can store clues about them. A posting may include how often the term appeared and where it appeared among the other tokens. The system may also keep separate postings for the title and body. In our example, P2 contains both query terms, and both appear in its title. P1 contains only `password`. A ranking formula can use those differences to give P2 a stronger score. Ranking turns those clues into an order. Methods such as [BM25](/blog/what-is-bm25/) rank word-based matches using term evidence. The inverted index makes that evidence available; the ranking method decides how much each clue matters. Stored token positions can also support phrase or nearby-word search. P2 is not an exact-phrase match for `reset password` because another word sits between the two terms. A hypothetical page containing the exact phrase could be treated differently from one where the words appear paragraphs apart. Not every index stores positions, so this behavior depends on the system. ## What happens when a page changes? Suppose P1's title and text both change: ```text Title: Manage your account Text: Manage your account and password. ``` becomes: ```text Title: Profile and login settings Text: Update your profile and login settings. ``` Conceptually, selected affected postings change like this: ```text account -> remove P1 password -> remove P1 profile -> add P1 login -> add P1 settings -> add P1 ``` This does not mean every engine edits one list in place. An implementation may write new index segments, mark old entries as deleted, merge segments later, or rebuild on a schedule. The useful reader model is simpler: P1 must stop appearing under its old terms and start appearing under its new ones. A single edit does not always require rebuilding the complete index. Search can still lag behind the source content when updates are delayed or batched. The targeted lookup has a cost. The index takes storage and work to build and maintain, and new content must reach it before search can use it. The index is a maintained lookup over the source content, not the source content itself. ## What does an inverted index not solve? A basic inverted index is strong at finding exact or normalized word-based terms. It does not automatically understand every way a person can describe the same need. A user may search for: ```text forgot my login ``` while the useful page says: ```text Reset your password ``` Those phrases share little vocabulary. Synonym handling, spelling correction, query expansion, or [vector search](/blog/what-is-vector-search/) can help with differently worded meaning. An inverted index does not add those choices automatically. Word-based matching still matters for product names, API routes, error codes, and exact technical terms. A search system can combine more than one retrieval method when its query mix needs both literal and semantic evidence. ## Inverted index summary An inverted index keeps a postings list for each searchable term. The basic flow is: ```text indexed fields -> searchable terms -> postings lists query -> compatible terms -> matching postings ``` It makes candidate lookup targeted. Optional stored clues can support ranking and phrase behavior, while update work keeps the lookup fresh. Start with the term, then follow it to the documents. ### What is BM25? A Plain English Guide Canonical URL: https://ceisium.com/blog/what-is-bm25/ BM25 is a keyword search ranking function that scores documents using term frequency, term rarity, and document length. Learn how it works, where it helps, and how it differs from TF-IDF. Search looks simple from the outside. You type a few words into a documentation search box, a product search box, or a small site search, and the system returns a list of pages. Behind that list is a ranking problem. The search engine may know about hundreds or thousands of pages, but it still has to decide which matches should come first. For example, the user may search: ```text pricing page ``` Some pages are clearly about pricing, some mention pricing once in a footer or changelog entry, and many pages have nothing to do with pricing. BM25 is one way for the search engine to sort those matches. ## What is BM25? BM25 is a ranking function for keyword search. It gives each document a score based on how well that document matches the search query. Higher score, higher chance of appearing near the top. Keyword search is also called lexical search. It works with matching words. If the query contains `pricing`, the search engine first looks for documents that contain `pricing`. BM stands for Best Matching, and the `25` is a version number from a family of ranking functions. The name sounds more mysterious than the idea behind it. BM25 looks at four main things: - Which query words appear in the document - How many times those words appear - How rare those words are across all documents - How long or short the document is
BM25 is useful when exact words matter, which happens a lot in product docs, support articles, API docs, error messages, settings pages, and technical search. ## A simple matching example The user searches: ```text pricing page ``` We have two documents. Document A: ```text Pricing page for teams ``` Document B: ```text Billing and pricing ``` Document A has both words, `pricing` and `page`. Document B only has `pricing`. A very simple search system can say: ```text Document A = 2 matches Document B = 1 match ``` That is the first idea behind keyword search: if a document contains more words from the query, it is often a better match. But this simple method breaks quickly. ## Repeating a word should not win forever Search only for: ```text pricing ``` Document A: ```text Our pricing page explains all plans. ``` Document B: ```text Pricing pricing pricing pricing pricing pricing pricing. ``` If we only count word frequency, Document B looks much better because `pricing` appears many times. The problem is easy to see. Seven mentions of `pricing` are not seven times more useful than one mention, because after a point repetition stops giving new information. BM25 handles this using term frequency saturation: ```text 0 mentions = no evidence 1 mention = useful evidence 2-3 mentions = stronger evidence 20 mentions = not 20x stronger ``` Repetition still helps, but it does not help forever. A page should not win only because it repeats the same word again and again. ## Rare words should count more Take another query: ```text configure sso ``` The word `configure` may appear on many documentation pages, while `sso` may appear only on a few. If a document contains `sso`, that is a stronger signal than if it only contains `configure`. BM25 uses this idea with IDF, which means inverse document frequency. In simple words: - If a word appears in many documents, it is less special. - If a word appears in fewer documents, it is more special. A user searches: ```text stripe webhook retry ``` On the website: ```text stripe appears on 40 pages webhook appears on 15 pages retry appears on 120 pages ``` Here, `webhook` is the rarest word, so it gives a stronger clue. `stripe` is also useful. `retry` is still useful, but it appears in more places, so it is less specific. This depends on the website. On a developer website, `api` may appear everywhere; on another website, `api` may be rare. BM25 learns term importance from the corpus itself. ### How does the search engine know these counts? BM25 does not read every page from scratch when a user searches. The search engine prepares this information before the query comes in. When pages are crawled or indexed, the search engine builds an inverted index. An inverted index is a map from words to documents. It looks like this: ```text stripe -> doc1, doc7, doc9, doc20, ... webhook -> doc3, doc7, doc12, ... retry -> doc2, doc3, doc7, doc8, doc21, ... ``` With this index, the search engine can know: ```text stripe appears in 40 documents webhook appears in 15 documents retry appears in 120 documents ``` That count is called document frequency. The search engine also stores other information: - How many times each word appears in each document - How long each document is - What the average document length is At search time, the engine can use these stored numbers to calculate the BM25 score. The crawler reads pages ahead of time. [Website crawling](/blog/how-website-crawling-shapes-search-quality/) determines which pages enter the index. The index remembers where words appear. BM25 uses that stored memory when a query comes in.
## Long documents need adjustment Search for: ```text billing ``` Document A is short: ```text Billing Manage billing settings for your workspace. ``` Document B is very long: ```text ... 8,000 words ... billing appears once somewhere inside it ... ``` Both documents contain `billing` once, but they should not get the same score. In Document A, `billing` is probably the main topic; in Document B, it may only be a small mention. BM25 handles this using document length normalization. A match in a short, focused document can be stronger than the same match inside a very long document. This does not mean long documents are bad. A long guide can still be the best result. BM25 is only trying to avoid giving long documents an unfair advantage because they have more words and more chances to match something. So far, BM25 has made four corrections to simple word counting. - It gives more credit when more query words match. - It lets repetition help, but only up to a point. - It gives more weight to rare words. - It adjusts the score when the same match appears inside a very long document. BM25 is still counting words, but it is counting them with useful corrections. ## BM25 in real search systems In a real website, search usually does not treat the whole page as one big block of text. A page has many fields: - Title - URL - Headings - Meta description - Body text - Anchor text - Content chunks A match in the title is usually more important than a match deep inside the body. If the user searches: ```text api keys ``` A page titled `API Keys` should usually rank higher than a long guide that mentions API keys once in the middle. Many systems solve this by scoring each field separately: ```text final keyword score = title BM25 score x title weight + heading BM25 score x heading weight + body BM25 score x body weight + URL BM25 score x URL weight ``` Clear titles, headings, and URLs help both the reader and the search engine. Search systems also prepare text before ranking it. They may lowercase words, remove punctuation, handle plurals, remove stop words, or tolerate small typos: ```text API keys api-key api key ``` After tokenization and normalization, these become easier to compare. Modern search systems may also split long pages into chunks, because one long page can talk about many topics. A chunk can tell the search engine, "this section is about API keys," instead of only saying, "this whole page contains API keys somewhere." ## The BM25 formula The formula is less scary if we connect it back to the ideas above. For the query: ```text reset api key ``` BM25 scores each query word and adds the scores: ```text score("reset") + score("api") + score("key") ``` A common BM25 formula looks like this: Same idea, written as math. ```text score(D, Q) = sum over query terms: IDF(q) * ((f(q, D) * (k1 + 1)) / (f(q, D) + k1 * (1 - b + b * |D| / avgdl))) ``` The symbols mean: ```text D = the document Q = the query q = one query word f(q,D) = how many times q appears in D IDF(q) = how rare q is across the collection |D| = document length avgdl = average document length k1 = how much repetition helps b = how much document length matters ``` The two knobs to remember are `k1` and `b`. `k1` controls repetition. If `k1` is low, extra mentions stop helping quickly; if it is high, repeated words keep adding score for longer. `b` controls document length. If `b` is `0`, document length does not matter; if it is `1`, document length matters a lot. Many systems use a value somewhere in the middle. ## BM25 vs TF-IDF BM25 is related to TF-IDF. TF-IDF means: ```text term frequency x inverse document frequency ``` It uses two ideas: - Words that appear more in a document can matter more. - Words that appear in fewer documents can matter more. BM25 keeps those ideas, but adds two controls that matter in real search: - Repeated words do not keep increasing the score forever. - Long documents are adjusted so they do not win only because they contain more words. That is why BM25 is usually a better baseline than plain TF-IDF for website search, documentation search, and product search. ## Where BM25 works well BM25 works well when the user searches for words that actually appear in the content: ```text reset api key billing invoice download api_key_invalid SOC2 ``` These are good BM25 queries because the exact words matter. That makes BM25 useful for: - Product names - Feature names - API names - Error codes - SKUs - Settings pages - Technical terms - Support articles If someone searches for `api_key_invalid`, we want the result that contains `api_key_invalid`. Exact matching should have real weight here. ## Where BM25 struggles BM25 struggles when the user and the document use different words for the same idea. The user may search: ```text change password ``` while the page says: ```text Update your credentials ``` A human can understand that these are related, but BM25 may not, because `change` does not match `update`, and `password` does not match `credentials`. Another query: ```text remove teammate ``` may need this page: ```text Delete a workspace member ``` The meaning is close, but the words are different. BM25 can still do well if the page includes alternate wording, synonyms, headings, or anchor text; the core ranking function itself is lexical, so it rewards word overlap.
## Other ranking methods BM25 is one ranking method in the larger family of lexical and statistical search methods. TF-IDF is the closest comparison because it is simpler and helps explain why BM25 adds saturation and document length normalization. Boolean retrieval is older and more strict. It uses logic like: ```text api AND key billing OR invoice error NOT deprecated ``` With Boolean retrieval, a document either matches or it does not. That is useful for filters and exact control, but it is not always enough for ranking. Language models for information retrieval ask a different question: how likely is this document to produce this query? DFR, or Divergence From Randomness, asks whether a term appears more often than we would expect by chance. BM25F is a field-aware version of BM25 for documents with fields like title, body, anchor text, and metadata. BM25+ is another variant that changes how long documents are treated in some cases. Learning to Rank goes broader. Instead of using one fixed formula, it learns how to combine many signals: BM25 score, title match, freshness, click data, phrase match, document type, and more. Even there, BM25 is often still one of the useful signals. ## FAQ ### What does BM25 stand for? BM stands for Best Matching. BM25 is one version from the Best Matching family of ranking functions. ### What is 25 in BM25? The `25` is a version number. It does not mean BM25 has 25 steps or 25 inputs. ### Is BM25 just TF-IDF? No. BM25 is related to TF-IDF, but it is not the same. Both use term frequency and inverse document frequency. BM25 also adds term frequency saturation and document length normalization. ### What is BM25 used for? BM25 is used to rank keyword search results in full-text search, documentation search, product search, support search, and technical search. ### What is BM25 in RAG? In RAG systems, BM25 can retrieve documents or chunks that contain exact query terms before the results are sent to a language model. This helps with names, IDs, error codes, and technical terms. ### Does BM25 understand meaning? Not really. BM25 mostly works with word overlap and term statistics, so it may miss two phrases that mean the same thing but use different words. ### What are k1 and b? `k1` and `b` are tuning parameters. `k1` controls how much repeated words help, and `b` controls how much document length matters. ## Summary BM25 is a practical way to rank keyword search results. It looks at: - Which query words match - How often they appear - How rare they are - How long the document is BM25 is not trying to understand language like a person. It is trying to rank exact word matches in a smarter way than simple word counting. BM25 is still useful because exact words still matter. Product names, API names, error codes, settings pages, and technical concepts often need exact matching. Once you understand BM25, many search results start to look less mysterious: the engine is asking which words matched, how strong those matches are, and whether the document looks focused enough to deserve the top spot. ### What Is Hybrid Search? Canonical URL: https://ceisium.com/blog/what-is-hybrid-search/ Hybrid search combines keyword and vector retrieval. Learn how fusion, candidate windows, filters, and reranking turn two result lists into one. Assume a developer searches fictional documentation with an invented error code: ```text How do I keep users logged in longer after ERR_AUTH_004? ``` The best page is **Configure session lifetime**. It solves the task, but it does not use the words in the query. Another page is **ERR_AUTH_004: Session expired**. It contains the exact code but only explains why the session ended. The query has two useful clues: * A task in ordinary language * An exact error code Lexical retrieval can preserve the code when text analysis keeps it intact. Vector retrieval can connect `logged in longer` with `session lifetime`. Hybrid search uses both types of evidence. ## What is hybrid search? Hybrid search combines ranked results from more than one retrieval method. Our example uses lexical retrieval and vector retrieval. Each method returns one ranked list. A fusion step then combines the lists into one candidate list. Lexical retrieval uses indexed terms. Vector retrieval compares numeric representations, called embeddings, of the query and content. The basic process is: ```text query ├─ lexical retrieval → ranked list └─ vector retrieval → ranked list ↓ fusion ↓ fused candidate list ``` Products can combine the results in different ways. We will explain one common process that joins two result lists. All hybrid designs use different evidence sources. They do not require one method to solve every query. Hybrid search gives more evidence to the ranker. It does not guarantee that the evidence is better. ## What lexical retrieval contributes Lexical retrieval starts with the exact error code. An analyzer converts page text into a token stream. A tokenizer creates tokens, and token filters can change them. The engine then stores the analyzer output as indexed terms. [Lucene analyzer documentation](https://lucene.apache.org/core/10_2_0/core/org/apache/lucene/analysis/Analyzer.html) explains these stages. The analysis settings can convert text to lowercase or split punctuation. They can also preserve one technical identifier. An exact ID helps only when page analysis and query analysis produce compatible forms. Lexical rankers compare query terms with indexed terms. [BM25](/blog/what-is-bm25/) is one common lexical ranking method. Its inverse document frequency can give more value to a rare error code. [Lucene BM25 documentation](https://lucene.apache.org/core/10_3_0/core/org/apache/lucene/search/similarities/package-summary.html) describes this behavior. [TF-IDF](/blog/what-is-tf-idf/) uses a related term-ranking idea. The lexical list can look like this: | Lexical rank | Page | Illustrative score | | -- | -- | -- | | 1 | ERR_AUTH_004: Session expired | 8.4 | | 2 | Rotate an API key | 5.1 | | 3 | Authentication concepts | 3.7 | The first result preserves the exact code. The other pages share common authentication terms but do not solve the session task. **Configure session lifetime** can be absent from the first three results. The heading and query use different words. Lexical retrieval found the exact error code but did not find the task page. ## What vector retrieval contributes Vector retrieval starts with the task in ordinary language. An embedding model converts the query and passages into lists of numbers. These lists are embeddings. An embedding represents text as a point in a learned comparison space. The system finds related content by comparing these points. The [vector search guide](/blog/what-is-vector-search/) explains embeddings and similarity in more detail. The vector list can look like this: | Vector rank | Page | Illustrative score | | -- | -- | -- | | 1 | Configure session lifetime | 0.78 | | 2 | Authentication concepts | 0.71 | | 3 | ERR_AUTH_004: Session expired | 0.65 | The vector path finds the task page. It connects `keep users logged in longer` with `session lifetime`. Vector retrieval also finds the broad **Authentication concepts** page. This page is related but does not explain the setting. Vector similarity is not correctness. It does not prove that a page is useful, current, or permitted. Each list contains one useful page and one weak page. The lexical list favors the incomplete error page. The vector list includes a broad concepts page.
## Raw retrieval scores are not directly comparable The error page has a lexical score of `8.4`. The session-lifetime page has a vector score of `0.78`. Do not add these values without a defined calibration method. The scorers use different functions, ranges, and distributions. A lexical score of `8.4` is not automatically stronger than a vector score of `0.78`. The vector score does not mean 78 percent relevant. A retrieval score is usually comparable only within one scorer and one query. It is not a probability or confidence value. Different retrieval scores do not share one measurement unit. A hybrid system needs a defined fusion method. One simple method uses result positions instead of raw scores.
## Rank fusion combines positions Rank fusion uses the position of each document in each source list. A high position contributes more to the fused rank than a low position. A document can also receive contributions from both lists. A document can remain in the fused list even when only one retrieval method finds it. In our example: * **ERR_AUTH_004: Session expired** is first in the lexical list and third in the vector list. * **Authentication concepts** is third in the lexical list and second in the vector list. * **Configure session lifetime** occurs only in the vector list. * **Rotate an API key** occurs only in the lexical list. One rank-fusion result can be: 1. ERR_AUTH_004: Session expired 2. Authentication concepts 3. Configure session lifetime 4. Rotate an API key Fusion keeps the task page, but it does not put that page first. Appearances in two lists can outweigh one high position. Fusion combines ranks or scores. It does not determine which page correctly completes the user's task. [Reciprocal Rank Fusion](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion) is one common method. High positions contribute more, and repeated appearances can strengthen a document. Some systems normalize source scores and apply weights. [OpenSearch normalization](https://docs.opensearch.org/latest/search-plugins/search-pipelines/normalization-processor/) makes numeric ranges more comparable. Normalization makes score ranges more comparable, but it does not give the scores the same meaning. Min-max normalization also depends on the observed extremes. Outliers and candidate changes can therefore change the result. A weight controls the influence of one signal. It is not a measure of confidence, and no one weight works for all query sets. ## Candidate windows limit fusion Our examples use the first three results from each retrieval method. This limit is the **candidate window**. Fusion sees only the union of candidates inside both windows. Candidate windows define what fusion can see. If the session-lifetime page is fourth in both lists, fusion cannot include it. Score normalization cannot restore an absent document. A larger window gives more opportunities to keep the correct page. It can also increase retrieval, transfer, memory, and reranking work. The [Elasticsearch RRF reference](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion) describes this quality and performance tradeoff. First measure whether useful pages enter the candidate set. Then measure their final order. ## Filters, fusion, and reranking have different jobs Search products can put these stages in different positions. The stages still answer different questions. ### Filters define eligibility Authorization and tenant boundaries must restrict retrieval before exposure. Every retrieval path must fail closed. A private page must not become visible because one path omits an access rule. Other filters can use version, category, language, price, or date. Their exact placement depends on the system. Filtering defines permitted documents. Ranking puts permitted documents in order. A small ranking penalty is not a substitute for a required version filter. ### Fusion combines evidence Fusion combines the candidate lists. After this step, the search system already uses both retrieval methods. Hybrid search does not require another model or a generative language model. ### Reranking examines the shortlist A reranker is an optional later stage. It uses more computation on a smaller candidate set. For our query, a reranker can move **Configure session lifetime** above the error page. It can also move the broad concepts page lower. A reranker cannot restore a document outside both candidate windows. It can only reorder the supplied candidates. This table summarizes the jobs: | Stage | Question | | -- | -- | | Filter | Is this document permitted? | | Retrieval | Did a retrieval method include this document? | | Fusion | How must the lists combine? | | Reranking | Which remaining candidate best answers the query? | The stage definitions make diagnosis easier. If the correct page is absent, do not start with reranking. ## Common hybrid search designs The parallel two-list method is common. Hybrid search also includes other multi-retriever designs. ### Parallel retrieval and fusion The system runs lexical and vector retrieval for the same query. It then combines the two lists. The paths can run at the same time to reduce total response time. They still require a query embedding, two retrieval operations, and fusion. The system can also add reranking. ### Conditional retriever routing Some systems use one retrieval path first. They use a second path only when the first result set is weak or sparse. A request with one active path does not perform two-list fusion. It is outside the narrow definition that this article uses. Conditional routing still uses multiple retrieval methods across the system. It can reduce work for simple queries. The routing rule becomes another source of search errors. ### Candidate union and reranking The system can give the combined candidates directly to a reranker. A simple union can replace a scored fusion step. The reranker still depends on the candidate windows. A missing document stays missing. Each design has different quality, latency, and cost effects. Select a design that fits the query mix, filters, and quality target. ## When to use hybrid search Hybrid search is useful when queries contain exact and semantic clues. It also helps when different query groups need different retrieval methods. Do not add hybrid search when one method already meets the quality and latency targets. Use labeled tests to find lost exact terms or missed paraphrases. [BEIR](https://openreview.net/pdf?id=wCu6T5xFjeJ) also shows that one retrieval method does not win on all datasets. ### Developer documentation A user can know an error code but describe the required fix with different words. Lexical retrieval protects the code. Vector retrieval protects the task description. ### Product search A shopper can combine a natural-language category with an exact model code. Lexical and vector retrieval can both help. Structured requirements such as size, price, or availability usually belong in fields and filters. ### Support search A user can describe a symptom and include a feature name. Hybrid retrieval can preserve the name and find related explanations. ### Site search A user can search for `keep users signed in` when the page uses `session lifetime`. Semantic evidence connects the phrases. Lexical evidence still protects product names and navigation terms. Use lexical and semantic evidence when real queries contain both exact terms and natural-language descriptions. ## When hybrid search can reduce quality More evidence can make the final order worse. ### Broad semantic noise Vector retrieval can add pages that share the topic but do not solve the task. **Authentication concepts** is one example. ### Strong but stale exact matches Lexical retrieval can rank an old page high because it contains an exact code. A version filter can be more important than another rank adjustment. ### Missing candidates Fusion and reranking cannot restore a page outside both candidate windows. ### Incorrect fusion settings One normalized score can dominate because of poor calibration. Rank-fusion settings can also give too much influence to one window or position. A setting that helps paraphrases can damage exact-identifier queries. ### Added latency and cost Hybrid search can add query embedding, another index lookup, larger windows, and reranking. Parallel paths overlap in time. Do not add their individual durations to estimate the complete request. Measure end-to-end latency. These failure modes require careful tests. A measured quality gain can still justify hybrid search. ## How to evaluate hybrid search Start with one practical question: Does **Configure session lifetime** appear in the first visible results within the required response time? Run the same labeled query set through these configurations: 1. Lexical retrieval only 2. Vector retrieval only 3. Hybrid retrieval 4. Hybrid retrieval with reranking, when applicable Keep the corpus, filters, labels, and test period equal. Use separate names for these limits: * **Candidate N:** The number of documents from retrieval * **Reranker input size:** The number of fused candidates for the reranker * **Evaluation k:** The number of final results that the user sees Group queries by evidence type: * Exact identifiers and names * Paraphrases * Queries with tasks and identifiers * Version-sensitive or filter-sensitive queries * Queries with no correct answer Then measure candidate quality and final ranking quality. ### Candidate quality Candidate Recall@N measures the fraction of labeled relevant documents in the first N candidates. Hit@N asks only whether at least one relevant candidate is present. A page at rank 80 cannot reach a reranker that receives only 20 candidates. ### Final ranking quality * Hit@k asks whether at least one relevant result occurs in the first k positions. * MRR uses the reciprocal position of the first relevant result. * nDCG@k supports graded relevance and gives less value to low positions. Select metrics that match the user experience. A search box with three visible results must not celebrate a useful page at rank 80. ### Latency and hard constraints Report P50 and P95 response time with the quality results. P50 is the median response time. P95 is the response time that 95 percent of measured requests do not exceed. State the test period, request population, cache state, and load conditions. Also test authorization, tenant, version, and language constraints. Relevance cannot replace access control or correctness. A useful comparison shows the quality gain from each added stage and its measured time cost. ## Frequently asked questions ### Is hybrid search the same as semantic search? No. Semantic search uses meaning as evidence. Vector retrieval is one common semantic method. Hybrid search combines several retrieval signals. A hybrid system can include semantic retrieval, but the terms are not equal. ### Does hybrid search require BM25? No. BM25 is a common lexical ranker, but a system can use another lexical method. The combination of evidence defines hybrid search. ### Does hybrid search require a large language model? No. Embedding models often use machine learning, but hybrid search does not require a generative model. Rank fusion can be deterministic. A reranker is optional and can use different scoring methods. ## Conclusion Hybrid search combines different retrieval evidence. Lexical retrieval preserves words, identifiers, and exact clues when text analysis keeps them intact. Vector retrieval connects related content with different wording. Fusion combines the candidate lists. Filters enforce eligibility, and an optional reranker examines the shortlist again. The additional path can also add noise, cost, and failure modes. Candidate windows can hide the correct page. An exact match can be the wrong task. A close vector can be broad or stale. The final test is practical. The correct page must rank near the top for labeled queries within the required P95 latency. ### What is Inverse Document Frequency? Canonical URL: https://ceisium.com/blog/what-is-inverse-document-frequency/ Inverse document frequency helps search rank rare query words above words that appear everywhere. Learn the idea with a simple docs-search example. Inverse document frequency sounds like a formula term, but the idea starts with a very normal search problem. A user searches your docs for: ```text hmac webhook setup ``` The search engine can find pages that contain those words. That is matching. But after matching, it still has to rank the pages. This is where the problem appears. `setup` may show up across hundreds of docs pages. `webhook` narrows the field. `hmac` probably appears on only a few pages, and those pages are much more likely to answer the query. Inverse document frequency, or IDF, is the small ranking idea that handles this difference. It asks the search system to treat words that appear everywhere as weaker signals, and words that appear in only a few documents as stronger signals. ## What is inverse document frequency? IDF is a way to measure how much a term helps separate documents inside one searchable collection. Here, a document can be a docs page, product page, support article, API page, or any other page the search engine can return. The collection is the full set of those searchable documents. For each term, IDF asks: ```text How many documents contain this term? ``` That one count changes how much the term should count during ranking. Rare terms separate. Common terms blur. In our example collection, the counts may look like this: | Term | Pages containing the term | IDF intuition | | --- | ---: | --- | | `setup` | 700 pages | weak signal | | `webhook` | 90 pages | useful signal | | `hmac` | 8 pages | strong signal | The table does not say `setup` is useless. It says `setup` does not narrow the search very much. The term is still part of the query, but it should not dominate the score.
## Why not just count matching words? Imagine the query is: ```text hmac webhook setup ``` Now compare two pages. Page A: ```text Webhook setup guide Configure webhook endpoints and delivery settings. ``` Page B: ```text Verify HMAC webhook signatures Use HMAC to verify webhook payloads before accepting events. ``` Both pages match the query. Page A matches `webhook` and `setup`. Page B matches `hmac` and `webhook`. A simple word-counting system may only ask: ```text How many query words did this page match? ``` IDF adds a better question: ```text Which matched words are actually good at narrowing the result set? ``` That is the mechanism. IDF does not replace matching. It adds weight after matching, so a focused page with the specific term can beat a broader page with a more common term. ## The simple IDF formula The common textbook shape is: ```text idf(term) = log(total documents / documents containing term) ``` So if a collection has 1,000 pages: ```text idf("setup") = log(1000 / 700) idf("webhook") = log(1000 / 90) idf("hmac") = log(1000 / 8) ``` You do not need the exact decimal values here. The direction is the useful part: - `setup` appears on many pages, so its weight is low. - `webhook` appears on fewer pages, so its weight is higher. - `hmac` appears on very few pages, so its weight is highest. Different search systems use slightly different versions of the formula. Some add smoothing so the math behaves nicely when a term appears everywhere or almost nowhere. The mental model stays the same: ```text more documents contain the term -> lower IDF fewer documents contain the term -> higher IDF ``` ## IDF depends on the collection A word is not rare by itself. It is rare inside a specific collection. On a general marketing website, `api` may be rare. On a developer docs site, `api` may appear everywhere. So the search engine has to learn IDF from the actual pages being searched. When it builds the index, it counts how many documents contain each term. Rarity is local to the corpus. This also means the weights can change over time. If you add many new pages about webhooks, `webhook` becomes less rare in that collection than it was before. ## How IDF affects ranking Now switch to a second query: ```text api token reset ``` On many developer docs sites, `api` appears everywhere. `token` narrows the field. `reset` may point to an even smaller group of account or security pages. IDF turns those collection counts into weights: ```text api -> common, lower weight token -> more specific, medium weight reset -> more specific, higher weight ``` Now a page about resetting API tokens has a better chance of beating a generic API overview, even if the overview repeats `api` many times.
This is why IDF is useful in technical search. Error codes, config keys, API fields, package names, CLI flags, and protocol names often carry the real intent. ## What IDF does not do It does not understand synonyms. If the user searches `sign in` and the document says `login`, IDF alone does not know those words mean similar things. It also does not prove that every rare word is important. A typo can be rare too. IDF only says the term is unusual in this collection; the rest of the ranking system still has to decide whether the document is actually useful. That is why real search systems combine IDF with other signals: - whether the query words appear in the document - how often they appear - where they appear, such as title or body - how long or focused the document is - other lexical or semantic retrieval steps IDF is one part of the ranking story. ## How IDF connects to TF-IDF and BM25 IDF is the second half of [TF-IDF](/blog/what-is-tf-idf/). TF-IDF combines: - term frequency: how often the query term appears in this document - inverse document frequency: how rare the query term is across all documents BM25 also uses an IDF-style rarity signal. But [BM25](/blog/what-is-bm25/) adds more ranking controls, especially term frequency saturation and document length normalization. So if you are learning keyword ranking, the path is: ```text word matching -> IDF -> TF-IDF -> BM25 ``` IDF is the point where search stops treating every matching word as equally useful. ## Summary Inverse document frequency helps search engines answer one ranking question: ```text Which matching words actually separate the useful pages from the noisy pages? ``` Words like `setup`, `guide`, or `page` can still matter, but they appear in too many places to carry the whole score. Terms like `hmac`, `retry-after`, `sso`, or a specific API field often narrow the result set faster. That is why IDF is one of the basic ideas behind TF-IDF, BM25, and many keyword search systems. Thanks for reading. ### What Is Tokenization in Search? Canonical URL: https://ceisium.com/blog/what-is-tokenization-in-search/ Learn how search engines split and normalize page text and queries into searchable terms, and why those choices can decide which pages reach ranking. Your documentation contains a page named **Rotate an API-key**. The hyphen is intentional. A user searches for **API keys**. The page is relevant, but search does not show it. You can first suspect the ranking settings. However, the problem can occur before ranking. The search engine must convert the page text and query into comparable terms. Tokenization decides where one token ends and the next token starts. ## What is tokenization in search? Tokenization divides text into searchable pieces. Each piece is a **token**. Search tokens are intermediate pieces in a text-analysis process. They are different from the subword tokens that a language model uses. One configuration can produce these tokens: ```text Page: Rotate an API-key Tokens: [Rotate] [an] [API] [key] Query: API keys Tokens: [API] [keys] ``` The tokenizer uses spaces and the hyphen as boundaries. It does not make `API` and `api` equal. It also does not connect `keys` to `key`. Lowercase conversion can make `API` and `api` equal. Stemming can connect `keys` to `key`. A different tokenizer can keep `API-key` as one token. Another process can remove the hyphen first and produce `APIkey`. Tokenizer settings determine which terms are available for matching. Therefore, punctuation can affect search results.
## Tokenization is one part of text analysis People sometimes call the complete analysis process *tokenization*. This article separates the stages so that you can examine each operation: ```text Page: characters → optional character filters → tokens → optional token filters → indexed terms Query: characters → optional character filters → tokens → optional token filters → query terms ``` Character filters change text before the tokenizer selects boundaries. Token filters operate after tokenization. They can change, remove, or add tokens. This example shows the stages: ```text Page: Rotate an API-key Tokenize: [Rotate] [an] [API] [key] Lowercase: [rotate] [an] [api] [key] Optional stopwords: [rotate] [api] [key] Query: API keys Tokenize: [API] [keys] Lowercase: [api] [keys] Optional stemming: [api] [key] ``` A **token** is output from the tokenizer. An **indexed term** is the final page-side form that the index stores. A **query term** is the comparable query-side form. A token filter can remove a token or transform it into one or more terms. Each analysis step has one purpose: * Lowercase conversion can make `API` and `api` comparable. * A stop-word filter can remove a common word such as `an`. * A stemmer or lemmatizer can reduce `keys` to `key`. * A synonym filter can connect different expressions. These filters are optional. Stop-word removal can damage names and phrase searches. Stemming can also combine words that a site must keep separate. Select the analysis steps that fit your content and your users. ## The page and query need compatible terms An **analyzer** is the complete analysis process. It has optional character filters, one tokenizer, and optional token filters. The analyzer processes page fields when the search engine indexes a page. The engine stores the final terms in an [inverted index](/blog/what-is-an-inverted-index/). The analyzer also processes the query when the user searches. The page analyzer and query analyzer can be different. However, both analyzers must produce compatible final terms for a lexical match. Compatible terms must meet in the index. For example, the index can store `api-key` as one term. If the query produces `api` and `key`, a simple lexical lookup can miss the page. A **field** is one searchable part of a document. A title, body, or identifier can each be a separate field. Different fields can use different analysis settings: * A body field can split `api-key` so that `key` can find the page. * An identifier field can keep `api-key` because the punctuation has meaning. * The system can index both forms when both search behaviors are necessary. There is no universal rule that removes all punctuation. The correct boundary depends on the purpose of the text. ## Tokenization occurs before ranking A **candidate** is a page that a ranker can put in order. A lexical search path has this basic form: ```text Page text → analysis → indexed terms ┐ ├→ candidate pages → ranking Query text → analysis → query terms ┘ ``` Shared terms help the engine find candidates in the inverted index. A ranking method such as [BM25](/blog/what-is-bm25/) then puts those candidates in order. A ranker cannot promote a page outside its candidate set. Search engines use different candidate rules. An engine can accept any query term, all query terms, or a specified minimum. The candidate rules use the terms that text analysis produces.
A [hybrid system](/blog/bm25-in-rag/) can add another retrieval path. Tokenization still determines which terms the lexical path can match. ## Not every mismatch is a tokenization problem A missing page can have many causes. Find the first analysis stage that breaks the useful match. | Page and query forms | Mechanism to inspect | | -- | -- | | `API` and `api` | lowercase conversion | | `keys` and `key` | stemming or lemmatization | | `API-key` and `API key` | delimiter handling | | `apikey` and `api key` | split and join handling | | `kdy` and `key` | typo or fuzzy matching | | adjacent and separated `api key` | phrase or proximity matching | Split and join handling connects forms such as `apikey` and `api key`. Phrase matching checks whether terms occur in sequence. Proximity matching limits the distance between terms. Some implementations also permit the terms in either order. The word *exact* can describe different comparisons: * **Exact string:** The original characters are the same. * **Exact analyzed term:** The final analysis terms are the same. * **Exact phrase:** The analyzed terms occur in sequence with no permitted gap. * **Fuzzy match:** A separate rule permits a small spelling difference. After lowercase conversion, `API` and `api` can produce the same analyzed term. Their original strings are still different. ## Unicode also affects analysis Some characters look the same but use different internal representations. Two copies of `café` can use different character sequences for `é`. Unicode normalization can make those representations consistent. Accent folding is a different process. It changes `café` to `cafe` and removes information. Languages also use different word boundaries. Thus, an English whitespace rule is not a universal tokenizer. Character normalization and token boundaries are configuration choices. They are not laws of text. ## How to debug a tokenization miss Use this sequence when a correct result does not appear: 1. Record the exact page text and the exact query. 2. Examine the tokens and final terms from both analyzers. 3. Find the first stage that makes the useful terms incompatible. 4. Decide whether the field must split, preserve, or store more than one form. 5. Reindex the content after a change to page-side analysis. 6. Measure missed relevant pages and new unwanted pages. Fix the first stage that breaks the match. This sequence shows whether the failure occurs in tokenization, normalization, candidate retrieval, or ranking. ## Summary Tokenization selects the boundaries in text. Other steps control lowercase conversion, stop words, stemming, synonyms, fuzzy matching, and ranking. Before you change the rank of the `API-key` page, ask one question: **Did the page and the query produce compatible terms?** If the page becomes a candidate, ranking can do its work. Read about [inverted indexes](/blog/what-is-an-inverted-index/) to learn how search stores the terms. Then read about [BM25](/blog/what-is-bm25/) to learn how lexical search ranks the candidates. ### What Is Vector Search? Canonical URL: https://ceisium.com/blog/what-is-vector-search/ Vector search finds related content by comparing embeddings. Learn how vectors, similarity, approximate nearest-neighbor search, and hybrid retrieval work together. A user searches the docs for `keep users logged in`. The answer may live under a heading called “Configure session lifetime.” A keyword search can match `users` or `session`, but it may miss the answer when the writer talks about a timeout, an idle period, or a cookie that lasts for seven days. The reader and the docs are describing the same need with different words. Nearby does not mean correct. Vector search helps with that first gap: it turns the query and each searchable passage into lists of numbers, then compares the lists to find related ones. ## What is vector search? The word “vector” sounds mathematical, but the first idea is simple. A vector is a row of numbers that describes a position. On a map, a coordinate such as `(3, 5)` tells you where a point sits compared with other points. The numbers do not mean much by themselves; the position they make together is what we use. Here is a tiny vector: ```text [0.12, -0.44, 0.81, 0.09, ...] ``` The same is true here. `0.12` is not “the session number,” and `-0.44` is not “the timeout number.” Meaning comes from the pattern across the whole list. A search system compares one complete vector with another, looking for vectors that land near one another under the chosen comparison method. For text, a model creates one vector for the query and one for each passage. The vector made for a piece of text is called an embedding. So the next question is not “what does each number mean?” It is “how did the model learn to place related uses near one another?” ## What is an embedding? An embedding is the fixed-size list of numbers a model returns for a piece of content. The model turns text into numbers so a computer can compare that text with other text. To make the list, the model first breaks the text into small pieces called tokens. A token may be a whole word, part of a word, or punctuation. The model processes those pieces together through a trained neural network and returns one list with the same length every time the same model and setup are used. Developers do not choose those numbers by hand. During training, the model sees examples of how text is used and adjusts its internal settings. When two pieces are used in related ways, training pushes their output lists closer together. When a piece is unrelated, it should stay farther away. That repeated adjustment shapes the map. The model uses the same internal map for every passage. The map is useful even though it is not a literal picture. During training, related uses are encouraged to produce nearby vectors in that learned space, while unrelated uses stay farther apart. It is a useful mental model, not a literal picture with human-readable x/y meanings. That learned space gives us a bridge to stored search data. If an entire page covers many unrelated ideas, one vector has to blur them together. Search usually needs smaller pieces.
## What gets embedded? A search system could make one embedding for a whole page. That is often too coarse. A session guide might explain cookie lifetime, idle timeout, logout behavior, and account security. One page-level vector has to represent all of those ideas at once. Many systems split a page into smaller passages called chunks: - Set the session lifetime - Explain the idle timeout - Keep users signed in after a restart - End a session when the user logs out Each chunk gets its own embedding. Search can now retrieve the passage about lifetime instead of returning a broad page and asking the reader to hunt through it. Smaller chunks are not automatically better. If a warning lands in the next chunk, the result may lose the context that makes the setting safe. Chunking is a balance: narrow enough to be specific, but wide enough to keep the idea complete. ## How vector search works The full search path has five jobs. The first two happen while content is indexed. The other three happen when a user searches. The illustration shows the whole route before we walk through each handoff.
### 1. Prepare the searchable content The system reads the pages or records it wants to search. It may clean the text, split long pages into chunks, and attach details such as the URL, section heading, product version, language, or publication date. Those details are metadata. They let later stages remove an old version or prefer the right language. This work happens before a reader types a query, so the stored content is ready when the request arrives. ### 2. Create content embeddings The embedding model turns each chunk into a vector. The system stores that vector beside the original text and its metadata. When a reader later searches, the query must be embedded in a compatible way. Common model families you may encounter include: | Family | Examples documented by the provider or model card | What to check | | --- | --- | --- | | OpenAI | [text-embedding-3-small](https://developers.openai.com/api/docs/guides/embeddings) and `text-embedding-3-large` | Dimensions, cost, latency, and whether the model fits your language and domain | | Google | [gemini-embedding-2](https://ai.google.dev/gemini-api/docs/embeddings) and `gemini-embedding-001` | Query/document task instructions, modality, dimensions, and compatibility when upgrading | | Cohere | [embed-v4.0](https://docs.cohere.com/docs/cohere-embed) | Text or image inputs, dimensions, metric, and context length | | Voyage | [voyage-4-large](https://docs.voyageai.com/docs/embeddings) and the `voyage-4` family | Query/document input type, output dimensions, cost, and latency | | Open weights | [BAAI/bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) and [intfloat/e5-large-v2](https://huggingface.co/intfloat/e5-large-v2) | Hardware, language coverage, prompt format, dimensions, and serving cost | This is a list of examples, not a popularity ranking. Choose based on query/document support, language and domain fit, dimensions, cost, and latency. The same compatible model and setup must be used for the items you compare. Two vectors are not compatible just because their lists happen to have the same length. If you switch to an incompatible model or setup, the stored content must be embedded again. Otherwise the old content vectors and new query vectors come from different maps. ### 3. Embed the query When a user searches, the model turns the query into a vector using the matching query setup. For our example, `keep users logged in` becomes one more point in the same comparison map as the stored session-lifetime passages. ### 4. Retrieve nearby candidates The system compares the query vector with stored vectors. It gives each comparison a similarity score or distance. Closer vectors become the first candidates. For the session-lifetime query, the candidate list might start like this: 1. Configure session lifetime 2. Explain idle timeout behavior 3. Keep users signed in after a restart 4. Account security and logout This is candidate retrieval. It finds a small set worth examining more carefully. It does not settle the final order. ### 5. Filter and rerank The system can now use clues that vector distance missed: - A version filter removes an old session guide. - A product filter keeps the result in the right application. - A language signal chooses the reader's preferred language. - A reranker places the direct configuration steps above a general security overview. A reranker is a later model or scoring step that reads the candidate set more carefully. It can change the order. It cannot recover a useful passage that the retrieval step never found. The interface can then return the full docs page and open it near the matching session section. ## Vector search, semantic search, and vector databases These names overlap, and different products use them differently. A simple distinction is enough for this article. Vector search is the mechanism: turn items into vectors and find nearby vectors. Semantic search is the goal: match what the reader means, even when the words differ. Vector search is one way to do that. Synonyms, query rewriting, and reranking can also add semantic behavior. A vector database stores and searches vectors. It usually provides indexing and metadata filters too. But you do not need a product called a vector database to use vector search. General search engines and database extensions can also provide vector indexes. ### Is vector search AI? The embeddings usually come from a machine-learning model. The search step then compares the resulting vectors. That is not the same as asking a large language model to write an answer, and it does not turn the search system into a chatbot. ## How is similarity measured? The model documentation usually tells you which comparison method to use. - Cosine similarity compares vector direction. - Dot product compares alignment and may also reflect vector size, depending on the embeddings. - Euclidean distance measures the straight-line distance between points. Cosine similarity and dot product usually give higher values to closer matches. A distance gives lower values to closer matches. The resulting similarity score or distance is a ranking clue, not a probability. Scores from different models, metrics, or index settings should not be treated as if they share one scale. ## How does vector search stay fast? With a small collection, search can compare the query with every stored vector. This is an exact, brute-force search. That gets expensive as the collection grows. Many systems use approximate nearest-neighbor search, shortened to ANN. Instead of checking every vector, an ANN index guides the search toward promising parts of the collection. Imagine looking for a cafe in a city. You would start in an area likely to have cafes, then inspect nearby streets. You would not inspect every building in the city. Many ANN indexes follow the same broad idea, though their actual data structures differ. The shortcut is faster, but it can miss neighbors that a complete scan would find. Retrieval recall measures how often the search finds the neighbors or relevant results it was supposed to find. ANN settings trade some of that recall for lower latency or lower resource use. Approximate does not mean random. It means finding very good candidates without proving that every stored vector was checked. ## Vector search vs keyword search Keyword search and vector search notice different things. Keyword methods such as [TF-IDF](/blog/what-is-tf-idf/) and [BM25](/blog/what-is-bm25/) work with the words in the query and document. They are good at protecting exact strings and judging how distinctive those strings are. Vector search compares model-made representations. It is useful when two phrases express a related idea with different words. | Query | Typical keyword strength | Typical vector strength | | --- | --- | --- | | `keep users logged in` | Match shared words if present | Connect the query with session-lifetime documentation | | `session timeout expired` | Protect the exact setting name | Find passages that explain idle or maximum lifetime | | `remember my sign-in` | Match words such as `sign-in` | Connect the request with persistent-session guidance | Exact strings still matter for configuration keys, product names, versions, and identifiers. Vector search helps when the reader and author chose different vocabulary. Neither method wins every query. ## Why hybrid search is common Hybrid search uses keyword and vector evidence together. For `keep users logged in`, keyword retrieval protects a setting name when it appears. Vector retrieval can add passages about session lifetime, idle timeout, and persistent sign-in when the wording differs. Filters and later ranking decide what reaches the top. The two systems' raw scores are not naturally comparable. A hybrid system might normalize the scores, combine result ranks, or rerank one merged candidate set. The method varies. The reason stays simple: exact wording and related meaning answer different parts of the same search. ## Where vector search helps Documentation search is a natural fit because readers do not always know a product's preferred terms. A reader who asks to `keep users logged in` can find a section called “Configure session lifetime.” Product search has the same vocabulary gap, but the search system still needs product filters and exact attributes to avoid a plausible-looking wrong result. Technical search often mixes plain language with exact identifiers. A natural-language request can find the right concept, while keyword evidence protects the endpoint or configuration key once it appears. Text is only the easiest example to follow. With a suitable model and setup, a system can also turn images, products, audio, or other items into vectors and compare them. ## Where vector search can fail Vector search can find a nearby result that is still wrong. ### Similar topic, opposite instruction `keep sessions alive` and `end sessions quickly` are about the same subject. Their vectors may be close even though the instructions point in opposite directions. ### Wrong version or language A passage may explain the right idea for an old release or a different language while the reader needs current instructions. Similarity does not enforce those constraints by itself. ### Exact identifiers A configuration key, product name, or version may matter more than the broad topic. Keyword search often protects those exact values better. ### Weak chunking If an instruction and its warning are split apart, the result may lose the context that makes it safe. ### Domain vocabulary An embedding model may handle common language well but struggle with a new API, internal terminology, a niche field, or a language it saw less often during training. ### The nearest available result is still weak A top-k vector search returns the nearest available items unless the system applies a threshold or a no-result rule. If the collection has no useful answer, its closest passage can still be irrelevant. Vector search is a comparison mechanism, not a truth engine. ## How should vector search be evaluated? Do not judge a search system from three impressive demos. Build a test set that looks like real use: - Paraphrases that use different words - Exact configuration keys and versions - Queries that mix natural language and code - Version-sensitive questions - Answers buried inside long pages - Opposite instructions with similar vocabulary - Questions that should return no useful result People can label which results are useful. Then you can measure whether those results appear near the top. Calibrate thresholds and no-result rules for the actual model, metric, content collection, and any score changes made by the search system. Also measure latency, failed searches, harmful false matches, and what changes after an embedding model or index update. Search quality comes from the whole path: content preparation, chunking, embeddings, keyword matching, vector retrieval, filters, and ranking. ## Conclusion Vector search turns a query and searchable content into vectors, then looks for nearby vectors. An embedding model creates the numbers. A similarity function compares them. An ANN index can avoid checking every stored vector. Filters and reranking narrow the candidate list into a useful answer. This works well when a reader and an author use different words for the same need. It does not remove the need for exact keyword matches, version filters, good chunks, or honest evaluation. Treat vector similarity as one useful clue inside the full search path. Thanks for reading. ## Current public site pages ### About Ceisium | Company and Products Canonical URL: https://ceisium.com/about/ Learn why Ceisium is building a platform for growth and marketing teams and how Ceisium Search becomes its first product. About Ceisium # Infrastructure for growth and marketing teams. Ceisium is building a platform of practical products for growth and marketing teams. Each product handles one part of the work behind acquiring, understanding, converting, and retaining customers, with useful defaults and less infrastructure to maintain. ## Why Ceisium exists Growth and marketing teams run websites, content, search, analytics, experiments, automation, and distribution across tools that were not designed to work together. The gaps create manual work, fragmented data, and slow decisions. Ceisium is building the shared layer beneath that work. Each product solves one job well on its own, uses clear interfaces and common controls, and becomes more useful with the rest of the platform. ## The first product Ceisium Search is the first product. It gives public websites YouTube-like search. Visitors can use vague words or describe what they need, and Ceisium Search works to put the right page, guide, talk, pricing plan, or product answer first. Give it a URL. Ceisium Search handles crawling, indexing, ranking, freshness, and monitoring so the team does not have to operate its own search stack. ## Product principles - **Useful defaults:** a first setup should work without a long configuration project. - **Visible operations:** state, limits, and failures should be clear to the team that owns the work. - **Programmatic access:** teams and agents should be able to understand and use products through documented interfaces. - **Clear boundaries:** access, customer data, and product behavior should stay scoped to the intended customer and use case. ## Who builds Ceisium Ceisium is built by Yuvraj. He works on product development, developer experience, reliability, and the public technical writing behind the company. Follow the work on [X](https://x.com/yuvraajsj18) or [LinkedIn](https://www.linkedin.com/in/yuvraajsj18). ### Compare Ceisium Search | Technical Guides Canonical URL: https://ceisium.com/compare/ Compare Ceisium Search with Algolia, Typesense, and Meilisearch across setup, defaults, search quality, latency, cost, and maintenance. # Comparisons ## Comparison guides [ ![Algolia and Ceisium Search logos over a dark search-routing diagram.](https://ceisium.com/compare/algolia/card.webp) ### Ceisium Search vs Algolia Compare Ceisium Search and Algolia for website search, with setup screenshots, search quality, response times, pricing, and guidance on which to choose. Yuvraj · Aug 22, 2026 ](https://ceisium.com/compare/algolia/) [ ![Typesense and Ceisium Search logos over a dark indexing diagram.](https://ceisium.com/compare/typesense/card.webp) ### Ceisium Search vs Typesense Compare Ceisium Search and Typesense for website search, with setup screenshots, search quality, response times, pricing, and guidance on which to choose. Yuvraj · Aug 22, 2026 ](https://ceisium.com/compare/typesense/) [ ![Meilisearch and Ceisium Search logos over a dark ranking diagram.](https://ceisium.com/compare/meilisearch/card.webp) ### Ceisium Search vs Meilisearch Compare Ceisium Search and Meilisearch for website search, with setup screenshots, search quality, response times, pricing, and guidance on which to choose. Yuvraj · Aug 22, 2026 ](https://ceisium.com/compare/meilisearch/) ### Algolia Alternative: Ceisium Search vs Algolia Canonical URL: https://ceisium.com/compare/algolia/ Compare Ceisium Search and Algolia for website search: setup, Next.js search quality, response time, pricing, and controls for unexpected traffic. ![Algolia and Ceisium Search logos over a dark search-routing diagram.](https://ceisium.com/compare/algolia/card.webp) Algolia and Ceisium Search logos over a dark search-routing diagram. ![Algolia and Ceisium Search logos over a dark search-routing diagram.](https://ceisium.com/compare/algolia/card.webp) Algolia and Ceisium Search logos over a dark search-routing diagram. 1. [Home](https://ceisium.com/) 2. / 3. [Compare](https://ceisium.com/compare/) 4. / 5. Algolia Published August 22, 2026 Updated September 13, 2026 # Algolia Alternative: Ceisium Search vs Algolia Compare Ceisium Search and Algolia for website search: setup, Next.js search quality, response time, pricing, and controls for unexpected traffic. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=Algolia%20Alternative%3A%20Ceisium%20Search%20vs%20Algolia&url=https%3A%2F%2Fceisium.com%2Fcompare%2Falgolia%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fcompare%2Falgolia%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fcompare%2Falgolia%2F&t=Algolia%20Alternative%3A%20Ceisium%20Search%20vs%20Algolia) Verified September 13, 2026 Read 10 min Ceisium Search is the best Algolia alternative if your goal is to get the simplest out of the box experience for the most accurate website search without the burden of continuously maintaining different search settings. We built Ceisium Search for people who want visitors to find the right page without managing search relevance themselves. You give it a website URL, choose the pages to include, and let it handle crawling, ranking, and tuning as your website evolves. On the other hand, Algolia gives you more direct control over your records, ranking, filters, and promotions. That is useful when you know exactly what you want and can spend the time and effort to keep tuning it. In this comparison - we tried both Ceisium Search and Algolia on the [Next.js docs opens in a new tab](https://nextjs.org/docs/app/getting-started/installation) to give you an idea how both compare on different parameters like setup, search quality, and latency. ## TL;DR Area Algolia Ceisium Search Setup Create an index and supply records, or configure Algolia Crawler for your domain. Add a website URL and choose the pages to crawl. Configuration You are responsible to configure searchable fields, ranking, synonyms, rules, [typo tolerance](https://www.algolia.com/doc/api-reference/api-parameters/typoTolerance), facets, and AI options. Feature access depends on the plan. Choose crawl paths, schedule, and page limit. Ceisium Search generates the relevance settings for the website. Installation InstantSearch libraries and widgets, connected to your index. Directly use Search API in your existing UI or refer to UI examples available in the app. Search quality Expected page in the first three results for 35.7% of our benchmark queries. Needs tuning to improve results. Expected page in the first three results for 91.2% of our benchmark queries. Works great out of the box. Latency 179 ms median in our benchmark. 858 ms median in our benchmark. Ease of use Requires understanding of how to configure different settings but allow direct control over fields, ranking, synonyms, and rules; you maintain the choices you make. Managed crawling and ranking; you can review page coverage and visitor queries. Pricing Free plan, usage-based Grow and Grow Plus plans, and custom Elevate pricing. Plans from $29/month, with included usage and an optional hard stop. Main use case Product catalogs and ecommerce search with filters, promotions, and custom rules. Public website search across docs, blogs, marketing pages, and product pages. The quality and latency figures come from [our September 3, 2026 benchmark Search quality Benchmark Results, response times, and methods for our search comparison.](https://ceisium.com/docs/benchmark/) , with optional relevance settings unchanged. The sections below explain the setup and limits. ## Setting up search ### Algolia Algolia stores content as records. For a docs site, each record can contain a title, headings, body text, and page URL. An index holds the records we want to search. We started by creating an empty index in the dashboard. ![An empty Algolia index with the Upload records button.](https://ceisium.com/compare/algolia/setup-index.webp) The new index is ready for records. Captured on August 24, 2026. ![An empty Algolia index with the Upload records button.](https://ceisium.com/compare/algolia/setup-index.webp) The new index is ready for records. Captured on August 24, 2026. We then uploaded 390 records prepared from 280 pages in the Next.js App Router documentation. ![Algolia showing 390 indexed records after the Next.js import.](https://ceisium.com/compare/algolia/setup-import-complete.webp) The import completed with 390 records. Captured on August 24, 2026. ![Algolia showing 390 indexed records after the Next.js import.](https://ceisium.com/compare/algolia/setup-import-complete.webp) The import completed with 390 records. Captured on August 24, 2026. The dashboard accepts JSON, CSV, and TSV files. Uploads can replace existing records when their object IDs match. For ongoing updates, [connect a hosted data source or use an API integration opens in a new tab](https://www.algolia.com/doc/guides/sending-and-managing-data/send-and-update-your-data/how-to/importing-from-the-dashboard) . Algolia also has a [website crawler opens in a new tab](https://www.algolia.com/doc/tools/crawler/getting-started/create-crawler) . It requires domain ownership verification, then a start URL and crawler configuration. You can control which pages it visits, what it extracts, and when it runs. We used prepared records because we do not own `nextjs.org`. For the benchmark, we created a fresh index and left optional relevance settings unchanged. That includes searchable attributes, ranking, synonyms, rules, and AI settings. ![Algolia searchable attributes showing Not configured.](https://ceisium.com/compare/algolia/setup-fields.webp) Searchable attributes were left at the default. Captured on September 3, 2026. ![Algolia searchable attributes showing Not configured.](https://ceisium.com/compare/algolia/setup-fields.webp) Searchable attributes were left at the default. Captured on September 3, 2026. With [searchable attributes unset opens in a new tab](https://www.algolia.com/doc/api-reference/api-parameters/searchableAttributes) , Algolia searches all string attributes. You can set priorities, such as giving titles more weight than body text. We did not make that change for this benchmark. To add a search interface, Algolia provides [InstantSearch libraries opens in a new tab](https://www.algolia.com/doc/guides/building-search-ui/installation/js) . For a JavaScript project, the package installation is: SH ``` npm install algoliasearch instantsearch.js ``` You then connect the UI to an application ID, index name, and Search-only API key. The dashboard also lets you try queries before building that interface. ![Algolia dashboard results for Next.js optional catch-all dynamic routes.](https://ceisium.com/compare/algolia/setup-query-results.webp) A query from our August 24 setup check. This screenshot is separate from the September benchmark. ![Algolia dashboard results for Next.js optional catch-all dynamic routes.](https://ceisium.com/compare/algolia/setup-query-results.webp) A query from our August 24 setup check. This screenshot is separate from the September benchmark. ### Ceisium Search We create a project, add the website URL, and choose the paths, crawl schedule, and page limit. Once the crawl finishes, we can test queries in the playground. ![Ceisium source settings for the 280 Next.js App Router URLs.](https://ceisium.com/blog/ceisium-search-benchmark/ceisium-source.webp) Our source includes the Next.js App Router docs, with a limit of 280 pages. ![Ceisium source settings for the 280 Next.js App Router URLs.](https://ceisium.com/blog/ceisium-search-benchmark/ceisium-source.webp) Our source includes the Next.js App Router docs, with a limit of 280 pages. Ceisium Search turns those pages into searchable content and generates the search settings for the website. We made no manual relevance adjustments for the benchmark. For a website search box, we create a Browser/public key and restrict it to the allowed website origins. The search interface calls `POST /v1/search` with the project ID and query. The [quickstart ![](https://ceisium.com/blog/ceisium-cover-system.webp)Docs Docs | Ceisium Search Ceisium Search documentation for setup, crawl scope, indexing, the Search API, relevance tuning, observability, and security.](https://ceisium.com/docs/) includes the request and UI examples. With either product, use a key meant for browser search. Keep Algolia Admin keys and Ceisium Search Server keys out of browser code. ## Search quality We ran the same 1,000 labeled queries on September 3, 2026. They included page names, full questions, typos, error messages, and alternative wording. Ceisium Search found an expected page in the first three results for **912 queries**. Algolia did so for **357**. Product Hit@3 MRR@10 Ceisium Search 91.2% 0.8468 Algolia 35.7% 0.3149 **Hit@3** is the share of queries with an expected page in the first three results. **MRR@10** measures how high the primary expected page ranks. First place earns 1, second earns 0.5, and pages outside the first ten earn 0. We average that score across all queries. Here is Hit@3 for each query category: Query type What the category tests Queries Ceisium Search Algolia Exact Known page terms 280 93.6% 80.0% Long-tail More specific descriptions 280 89.6% 14.6% How-to A task the visitor wants to complete 120 93.3% 0.8% Configuration Options and settings 80 97.5% 21.3% Error A failure the visitor wants to resolve 55 89.1% 14.6% Concept An idea the visitor wants to understand 55 80.0% 16.4% Variant Alternative wording 60 88.3% 20.0% Typo Misspelled words 50 96.0% 76.0% Navigation A destination within the documentation 20 75.0% 35.0% We want people to describe what they need, even when they do not know the page name. This is an experience similar to how search works on youtube, you may have found yourself just describing the “vibe” and still getting the exact result you were looking for. This was one documentation site and a query set we wrote. It measures the results after required setup, with optional relevance settings unchanged. Algolia’s tuned or AI-assisted search may perform differently. The [full benchmark Search quality Benchmark Results, response times, and methods for our search comparison.](https://ceisium.com/docs/benchmark/) includes all query categories, example results, and the method. ## Response time Algolia was faster in this run. Product Median latency P95 latency Ceisium Search 857.86 ms 1,983.09 ms Algolia 178.91 ms 776.95 ms The median is the middle request time. At P95, 95% of requests finished within the reported time. These are client wall-clock measurements, including the network round trip from Bengaluru. Location affects the wait, so test from where your visitors are. Ceisium Search took about 0.86 seconds at the median, compared with Algolia’s 0.18 seconds. Its P95 was almost two seconds. We care about returning the right page, but that wait matters too. We plan to [improve Ceisium Search’s latency](https://ceisium.com/changelog/faster-search-better-first-results/) in the coming weeks. ## Keeping search useful ### Content and relevance For Algolia, you maintain the connection between your content and the index. That may mean an API integration, connector, or crawler configuration. If you add field priorities, synonyms, or rules, check them as your content changes. [Facets opens in a new tab](https://www.algolia.com/doc/guides/managing-results/refine-results/faceting) let visitors narrow results by categories such as brand or author. [Merchandising rules opens in a new tab](https://www.algolia.com/doc/guides/managing-results/rules/merchandising-and-promoting) can promote products or redirect a query to a chosen page. Feature access and limits depend on the plan. With Ceisium Search, the main checks are crawl coverage and the queries visitors run. Its [crawl logs Observability Crawl Logs Track crawl status, page outcomes, failures, retries, and coverage.](https://ceisium.com/docs/crawl-logs/) show indexed, skipped, and failed pages. If a page is missing, check the source paths and crawl result first. Managed ranking still needs useful content. If your docs do not answer a question, search cannot find a page that does. ### Analytics Both products show popular searches and queries with no results. [Algolia’s analytics opens in a new tab](https://www.algolia.com/doc/guides/search-analytics/overview) also supports click, conversion, and revenue metrics. These need events from your website, and access depends on the plan. [Ceisium Search’s analytics Start Analytics Read search events, no-result queries, popular queries, and engagement.](https://ceisium.com/docs/analytics/) shows search counts, average results, no-result queries, and popular queries, with CSV export. We use these to find gaps in content and check searches that need attention. ## Cost ### Algolia Algolia charges for search requests, stored records, crawling, and the features you choose. Its [current plans opens in a new tab](https://www.algolia.com/pricing) include: - **Free:** 10,000 search requests per month and 50,000 records, with no credit card required. - **Grow:** 10,000 search requests per month and 100,000 records included. Additional search requests cost $0.50 per 1,000. - **Grow Plus:** the same included search requests and records, with more AI features. Additional search requests cost $1.75 per 1,000. - **Elevate:** custom pricing for larger requirements. Grow and Grow Plus charge $0.40 per additional 1,000 records. Both include 10,000 crawls per month, then charge $0.80 per additional 1,000 crawls. Search-as-you-type can send a request on each keystroke, so count requests rather than visitors when estimating traffic. Algolia also offers [free DocSearch for eligible public technical docs and blogs opens in a new tab](https://docsearch.algolia.com/docs/who-can-apply/) . Check that option if your scope is documentation. Our benchmark used a separate Algolia index. ### Ceisium Search [Ceisium Search plans ![](https://ceisium.com/blog/ceisium-cover-system.webp)Pricing Pricing | Ceisium Search Ceisium Search starts at $29/mo with clear workspace quotas for projects, indexed pages, searches, crawl schedules, and crawl pages checked.](https://ceisium.com/pricing/) start with a monthly price and included usage: Plan Monthly price Indexed pages Searches/month Starter $29 5,000 20,000 Growth $99 50,000 150,000 Pro $299 250,000 750,000 Each plan also includes project, source, and crawl allowances. Scheduled crawls run weekly on Starter, daily or weekly on Growth, and as often as every six hours on Pro. Extra indexed pages cost $10 per 10,000, and extra searches cost $10 per 100,000. Extra crawl pages checked cost $20 per 100,000. These charges are prorated to actual usage. A page and a record are different units: our 280 Next.js pages became 390 Algolia records. Compare your expected content size, search traffic, and crawl frequency before choosing a plan. Prices above were checked on September 13, 2026. ### Unexpected traffic and bills For one small docs index we worked with, monthly Algolia bills rose from $6.50 to $49.50, then $114. Bots were spamming search with repeated queries such as `sw js` and `apple touch icon png`. We set Algolia’s hourly limit per IP to 1,000, which reduced that traffic. Those are the kinds of requests we would rather catch before the next bill. Algolia’s [per-IP key limit opens in a new tab](https://www.algolia.com/doc/guides/security/api-keys/in-depth/api-key-restrictions) defaults to zero, which means no limit for that setting. You can set a limit yourself. Algolia’s [bot-traffic guidance opens in a new tab](https://support.algolia.com/hc/en-us/articles/10154087058705-How-do-I-prevent-or-reduce-bot-traffic-in-Algolia-Search) also recommends blocking suspicious traffic before it reaches search. Ceisium Search sets [request limits on new API keys Start Search API Auth, request fields, response shape, pagination, errors, and examples.](https://ceisium.com/docs/search-api/) . The defaults depend on your plan and key type. For example, Starter Browser/public keys default to 60 requests per IP per minute and 100 requests per key per minute. You can edit those defaults. Rate limits reduce traffic bursts; a monthly usage stop serves a different purpose. Ceisium Search lets workspace admins select [hard-block mode ![](https://ceisium.com/blog/ceisium-cover-system.webp)Pricing Pricing | Ceisium Search Ceisium Search starts at $29/mo with clear workspace quotas for projects, indexed pages, searches, crawl schedules, and crawl pages checked.](https://ceisium.com/pricing/) , which stops metered actions when their included allowance is reached. **Paid plans allow overage charges by default**, so enable hard-block mode if you want that stop. Request limits alone do not guarantee a fixed bill. ## When to choose Algolia vs Ceisium Search? **Algolia** is a great option for many teams and we recommend choosing it for usecases like an ecommerce site or product catalog that needs filters, promotions, and direct control over ranking. Its UI libraries and search controls suit teams that can spend time shaping and maintaining that experience. We would choose **Ceisium Search** for public website search across docs, blogs, marketing pages, and product pages. We built it for accurate results from the first setup, with less manual relevance work as the website grows. Before deciding, try questions from people who do not know your page titles. Check which pages they find, how long they wait, and what you need to maintain. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=Algolia%20Alternative%3A%20Ceisium%20Search%20vs%20Algolia&url=https%3A%2F%2Fceisium.com%2Fcompare%2Falgolia%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fcompare%2Falgolia%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fcompare%2Falgolia%2F&t=Algolia%20Alternative%3A%20Ceisium%20Search%20vs%20Algolia) Copy image[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fcompare%2Falgolia%2F)[](https://twitter.com/intent/tweet?text=%E2%80%9CAlgolia%20Alternative%3A%20Ceisium%20Search%20vs%20Algolia%E2%80%9D&url=https%3A%2F%2Fceisium.com%2Fcompare%2Falgolia%2F) **Yuvraj** _Building Ceisium and working on growth @ SigNoz_ ceisium.com ### Meilisearch Alternative: Ceisium Search vs Meilisearch Canonical URL: https://ceisium.com/compare/meilisearch/ Compare Ceisium Search and Meilisearch for website search: setup, Next.js search quality, response time, pricing, and ongoing maintenance. ![Meilisearch and Ceisium Search logos over a dark ranking diagram.](https://ceisium.com/compare/meilisearch/card.webp) Meilisearch and Ceisium Search logos over a dark ranking diagram. ![Meilisearch and Ceisium Search logos over a dark ranking diagram.](https://ceisium.com/compare/meilisearch/card.webp) Meilisearch and Ceisium Search logos over a dark ranking diagram. 1. [Home](https://ceisium.com/) 2. / 3. [Compare](https://ceisium.com/compare/) 4. / 5. Meilisearch Published August 22, 2026 Updated September 13, 2026 # Meilisearch Alternative: Ceisium Search vs Meilisearch Compare Ceisium Search and Meilisearch for website search: setup, Next.js search quality, response time, pricing, and ongoing maintenance. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=Meilisearch%20Alternative%3A%20Ceisium%20Search%20vs%20Meilisearch&url=https%3A%2F%2Fceisium.com%2Fcompare%2Fmeilisearch%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fcompare%2Fmeilisearch%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fcompare%2Fmeilisearch%2F&t=Meilisearch%20Alternative%3A%20Ceisium%20Search%20vs%20Meilisearch) Verified September 13, 2026 Read 10 min Ceisium Search is a Meilisearch alternative for teams that want accurate website search without maintaining their own relevance settings. We built it for people who want visitors to find the right page even when they do not know its title. Add a website URL and choose the pages to include. Ceisium Search handles crawling, ranking, and tuning as the website changes. Meilisearch gives you more control over documents, searchable fields, ranking, and filters. You can use its managed Cloud service or host the engine yourself. That suits teams building search into an application or working with data beyond public web pages. We tried both products on the [Next.js docs opens in a new tab](https://nextjs.org/docs/app/getting-started/installation) . This comparison covers setup, search quality, response time, pricing, and the work needed to keep results useful. ## TL;DR Area Meilisearch Ceisium Search Setup Create an index and supply documents, or use the Meilisearch Cloud crawler. Add a website URL and choose the pages to crawl. Configuration Choose searchable fields, ranking rules, synonyms, filters, and optional vector or hybrid search. Choose crawl paths, schedule, and page limit. Ceisium Search generates the relevance settings for the website. Installation Use a client library or search UI integration, connected to your index. Use the Search API in your existing UI or start from the UI examples in the app. Search quality Expected page in the first three results for 42.3% of our benchmark queries. Expected page in the first three results for 91.2% of our benchmark queries. Latency 241 ms median in our benchmark. 858 ms median in our benchmark. Ease of use Simple document import with direct control over fields, ranking, and updates. Ceisium Search handles crawling and search settings; your team reviews page coverage and visitor queries. Pricing Cloud starts at $20/month, with usage or resource billing. Self-hosting has separate infrastructure and operating costs. Plans from $29/month, with included usage and an optional hard stop. Main use case Application or catalog search that needs filters, custom ranking, or a self-hosted engine. Public website search across docs, blogs, marketing pages, and product pages. The quality and latency figures come from [our September 3, 2026 benchmark Search quality Benchmark Results, response times, and methods for our search comparison.](https://ceisium.com/docs/benchmark/) , with optional relevance settings unchanged. The sections below explain the setup and limits. ## Setting up search ### Meilisearch You can run Meilisearch yourself or create a Meilisearch Cloud project. We used Cloud for this comparison. Meilisearch stores content as documents in an index. Its [schema-less document model opens in a new tab](https://www.meilisearch.com/docs/capabilities/indexing/overview) accepts documents with different fields. Each document still needs a primary key that identifies it. We created an index for the Next.js App Router documentation. ![Meilisearch index setup for the Next.js App Router documents.](https://ceisium.com/compare/meilisearch/setup-index.webp) We created a Meilisearch index before adding the prepared documents. Captured on August 24, 2026. ![Meilisearch index setup for the Next.js App Router documents.](https://ceisium.com/compare/meilisearch/setup-index.webp) We created a Meilisearch index before adding the prepared documents. Captured on August 24, 2026. We prepared 390 records from 280 documentation pages. The records contained titles, headings, body text, and page URLs. Document imports run as [asynchronous tasks opens in a new tab](https://www.meilisearch.com/docs/capabilities/indexing/tasks_and_batches/async_operations) . The initial response means the work is queued. Check that the task reaches `succeeded` before treating the import as complete. ![Meilisearch index with all 390 prepared Next.js records imported.](https://ceisium.com/compare/meilisearch/setup-import-complete.webp) Meilisearch indexed the 390 prepared records. Captured on August 24, 2026. ![Meilisearch index with all 390 prepared Next.js records imported.](https://ceisium.com/compare/meilisearch/setup-import-complete.webp) Meilisearch indexed the 390 prepared records. Captured on August 24, 2026. You can then choose searchable fields and their order. Our August setup walkthrough selected `title`, `headings`, and `body`. ![Meilisearch settings with title, headings, and body selected as searchable fields.](https://ceisium.com/compare/meilisearch/setup-fields.webp) We selected three searchable fields during the August 24, 2026 walkthrough. The September benchmark used a fresh index with default searchable attributes. ![Meilisearch settings with title, headings, and body selected as searchable fields.](https://ceisium.com/compare/meilisearch/setup-fields.webp) We selected three searchable fields during the August 24, 2026 walkthrough. The September benchmark used a fresh index with default searchable attributes. For the September benchmark, we kept the fresh index’s default `searchableAttributes: ["*"]`. We left optional relevance settings unchanged and did not enable vector or hybrid search. The dashboard lets you test a query before building the website interface. ![Meilisearch dashboard results for Next.js optional catch-all dynamic routes.](https://ceisium.com/compare/meilisearch/setup-query-results.webp) A Meilisearch query from our August 24, 2026 setup check. This screenshot is separate from the September benchmark. ![Meilisearch dashboard results for Next.js optional catch-all dynamic routes.](https://ceisium.com/compare/meilisearch/setup-query-results.webp) A Meilisearch query from our August 24, 2026 setup check. This screenshot is separate from the September benchmark. Meilisearch Cloud also includes a [website crawler opens in a new tab](https://www.meilisearch.com/docs/getting_started/features) , with JavaScript rendering, DocSearch mode, and schema extraction. This gives website teams another path to populate an index. We used prepared records for the benchmark. To add the search interface, use a [Meilisearch client or UI integration opens in a new tab](https://www.meilisearch.com/integrations) . Connect it to the project URL, index, and a search-only API key. Your team chooses how to display results and any filters. ### Ceisium Search We create a project, add the website URL, and choose the paths, crawl schedule, and page limit. Once the crawl finishes, we can test queries in the playground. ![Ceisium source settings for the 280 Next.js App Router URLs.](https://ceisium.com/blog/ceisium-search-benchmark/ceisium-source.webp) Our Ceisium Search source includes the Next.js App Router docs, with a limit of 280 pages. ![Ceisium source settings for the 280 Next.js App Router URLs.](https://ceisium.com/blog/ceisium-search-benchmark/ceisium-source.webp) Our Ceisium Search source includes the Next.js App Router docs, with a limit of 280 pages. Ceisium Search turns those pages into searchable content and generates the search settings for the website. We made no manual relevance adjustments for the benchmark. For a website search box, create a Browser/public key and restrict it to the allowed website origins. The interface calls `POST /v1/search` with the project ID and query. The [quickstart ![](https://ceisium.com/blog/ceisium-cover-system.webp)Docs Docs | Ceisium Search Ceisium Search documentation for setup, crawl scope, indexing, the Search API, relevance tuning, observability, and security.](https://ceisium.com/docs/) includes the request and UI examples. Keep Meilisearch admin keys and Ceisium Search Server keys out of browser code. Use keys intended for browser search and review which data they can return. ## Search quality We ran the same 1,000 labeled queries on September 3, 2026. They included page names, full questions, typos, error messages, and alternative wording. Ceisium Search found an expected page in the first three results for **912 queries**. Meilisearch did so for **423**. Product Hit@3 MRR@10 Ceisium Search 91.2% 0.8468 Meilisearch 42.3% 0.3679 **Hit@3** is the share of queries with an expected page in the first three results. **MRR@10** measures how high the primary expected page ranks. First place earns 1, second earns 0.5, and pages outside the first ten earn 0. We average that score across all queries. Here is Hit@3 for each query category: Query type What the category tests Queries Ceisium Search Meilisearch Exact Known page terms 280 93.6% 82.5% Long-tail More specific descriptions 280 89.6% 23.9% How-to A task the visitor wants to complete 120 93.3% 3.3% Configuration Options and settings 80 97.5% 28.8% Error A failure the visitor wants to resolve 55 89.1% 20.0% Concept An idea the visitor wants to understand 55 80.0% 18.2% Variant Alternative wording 60 88.3% 48.3% Typo Misspelled words 50 96.0% 78.0% Navigation A destination within the documentation 20 75.0% 45.0% Meilisearch’s strongest results were for known page terms and typos. Ceisium Search found the expected page more often in every category. We want people to describe what they need, even when they do not know the page name. This is similar to how search works on YouTube. You may have found yourself just describing the “vibe” and still getting the exact result you were looking for. This was one documentation site and a query set we wrote. Meilisearch received prepared records; Ceisium Search crawled the website. The test measures required setup with optional relevance settings unchanged. Tuned Meilisearch, vector search, or hybrid search may perform differently. The [full benchmark Search quality Benchmark Results, response times, and methods for our search comparison.](https://ceisium.com/docs/benchmark/) includes examples, source differences, and the method. ## Response time Meilisearch was faster in this run. Product Median latency P95 latency Ceisium Search 857.86 ms 1,983.09 ms Meilisearch 240.58 ms 395.78 ms The median is the middle request time. At P95, 95% of requests finished within the reported time. These are client wall-clock measurements, including the network round trip from Bengaluru. Location affects the wait, so test from where your visitors are. Meilisearch took about 0.24 seconds at the median, compared with Ceisium Search’s 0.86 seconds. Meilisearch’s P95 stayed below 0.4 seconds; Ceisium Search’s was almost two seconds. We care about finding the right page, but visitors also feel that wait. [Reducing Ceisium Search’s latency](https://ceisium.com/changelog/faster-search-better-first-results/) is a priority for us in the coming weeks. ## Keeping search useful ### Content and relevance With Meilisearch, keep the source and index in sync through your document integration or the Cloud crawler. Check import tasks, replace changed documents, and remove records for deleted pages. You can change [ranking rules opens in a new tab](https://www.meilisearch.com/docs/resources/internals/ranking) , searchable field order, synonyms, and filters. Those controls are useful when your team wants to shape a catalog or application search experience. Cloud runs the servers, while your team chooses the documents, search settings, and interface. With [resource billing opens in a new tab](https://www.meilisearch.com/blog/resource-based-pricing) , you also choose CPU and memory capacity. If you self-host, you also manage capacity, upgrades, backups, and availability. With Ceisium Search, choose a plan by pages and searches, then review source coverage and visitor queries. Its [crawl logs Observability Crawl Logs Track crawl status, page outcomes, failures, retries, and coverage.](https://ceisium.com/docs/crawl-logs/) show indexed, skipped, and failed pages. If a page is missing, check the source paths and crawl result first. Your website still needs useful content. If your docs do not answer a question, search cannot find a page that does. ### Analytics [Meilisearch Cloud analytics opens in a new tab](https://www.meilisearch.com/docs/capabilities/analytics/advanced/analytics_metrics) shows search activity, popular queries, and searches with no results. Click and conversion metrics need events from your website. They cannot appear from search requests alone. [Ceisium Search analytics Start Analytics Read search events, no-result queries, popular queries, and engagement.](https://ceisium.com/docs/analytics/) shows search counts, average results, no-result queries, and popular queries, with CSV export. We use these to find content gaps and review searches that need attention. ## Cost ### Meilisearch [Meilisearch Cloud opens in a new tab](https://www.meilisearch.com/pricing) lists a starting price of $20/month and a 14-day trial with no credit card required. It offers two billing models: - **Usage billing:** the plan is based on documents and searches. - **Resource billing:** you pay for the selected compute capacity and storage. On September 13, 2026, the page’s estimator used 100,000 documents, 10,000 searches per month, and an average document size of 1 KB. It showed: Billing model Monthly estimate What the example includes Usage $30 Base plan with 100,000 documents and 50,000 searches Resource $23 XS instance: 0.5 vCPU and 1 GB RAM for $18, plus $5 for 32 GiB of billed disk These are estimates for the displayed inputs. Larger documents, more traffic, or different features can change the required plan or resources. Use your own workload in the calculator. [Resource billing opens in a new tab](https://www.meilisearch.com/blog/resource-based-pricing) removes per-document and per-search caps from that billing model. Capacity still matters, and storage can grow as data increases. Check those meters when estimating future costs. You can also self-host [Meilisearch Community Edition under the MIT license opens in a new tab](https://www.meilisearch.com/docs/resources/self_hosting/enterprise_edition) . Include hosting, upgrades, backups, and the time needed to operate the engine. Enterprise Edition has separate licensing and features. ### Ceisium Search [Ceisium Search plans ![](https://ceisium.com/blog/ceisium-cover-system.webp)Pricing Pricing | Ceisium Search Ceisium Search starts at $29/mo with clear workspace quotas for projects, indexed pages, searches, crawl schedules, and crawl pages checked.](https://ceisium.com/pricing/) include website crawling and search. Ceisium Search handles the settings that decide which pages appear first. Plan Monthly price Indexed pages Searches/month Starter $29 5,000 20,000 Growth $99 50,000 150,000 Pro $299 250,000 750,000 Plans also include project, source, and crawl allowances. Starter supports weekly scheduled crawls, Growth adds daily crawls, and Pro supports crawls as often as every six hours. Extra indexed pages cost $10 per 10,000; extra searches cost $10 per 100,000. Extra crawl pages checked cost $20 per 100,000. These charges are prorated to actual usage. Prices were checked on September 13, 2026. A page and a document are different units: our 280 website pages became 390 Meilisearch records. Compare content size, search traffic, crawl frequency, and the work needed to maintain the index. ### Traffic and cost controls Meilisearch’s [custom API keys opens in a new tab](https://www.meilisearch.com/docs/capabilities/security/overview) can restrict actions, indexes, and expiry. Tenant tokens can apply user-specific filters. Review application request limits as well: permission to search does not set a monthly budget. With usage billing, unexpected requests can increase the usage you need to pay for. With resource billing, they can consume capacity. Check the billing model and controls for the project you create. Ceisium Search sets [request limits on new API keys Start Search API Auth, request fields, response shape, pagination, errors, and examples.](https://ceisium.com/docs/search-api/) , with defaults based on the plan and key type. Starter Browser/public keys default to 60 requests per IP per minute and 100 per key per minute. You can edit those defaults. **Paid Ceisium Search plans allow overage charges by default.** Workspace admins can enable [hard-block mode ![](https://ceisium.com/blog/ceisium-cover-system.webp)Pricing Pricing | Ceisium Search Ceisium Search starts at $29/mo with clear workspace quotas for projects, indexed pages, searches, crawl schedules, and crawl pages checked.](https://ceisium.com/pricing/) to stop metered actions at their included allowance. Request limits control bursts; this monthly usage stop controls whether metered actions continue beyond the plan allowance. ## When to choose Meilisearch vs Ceisium Search? Choose **Meilisearch** when you want direct control over documents, ranking, and filters, with a choice of Cloud or self-hosting. It fits application search, catalogs, and teams that already maintain a document pipeline. Its Cloud crawler can collect website content for the index. We would choose **Ceisium Search** for public website search across docs, blogs, marketing pages, and product pages. We built it for accurate results from the first setup, with less manual relevance work as the website grows. Before deciding, try questions from people who do not know your page titles. Check which pages they find, how long they wait, and what you need to maintain. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=Meilisearch%20Alternative%3A%20Ceisium%20Search%20vs%20Meilisearch&url=https%3A%2F%2Fceisium.com%2Fcompare%2Fmeilisearch%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fcompare%2Fmeilisearch%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fcompare%2Fmeilisearch%2F&t=Meilisearch%20Alternative%3A%20Ceisium%20Search%20vs%20Meilisearch) Copy image[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fcompare%2Fmeilisearch%2F)[](https://twitter.com/intent/tweet?text=%E2%80%9CMeilisearch%20Alternative%3A%20Ceisium%20Search%20vs%20Meilisearch%E2%80%9D&url=https%3A%2F%2Fceisium.com%2Fcompare%2Fmeilisearch%2F) **Yuvraj** _Building Ceisium and working on growth @ SigNoz_ ceisium.com ### Typesense Alternative: Ceisium Search vs Typesense Canonical URL: https://ceisium.com/compare/typesense/ Compare Ceisium Search and Typesense for website search: setup, Next.js search quality, response time, pricing, and ongoing maintenance. ![Typesense and Ceisium Search logos over a dark indexing diagram.](https://ceisium.com/compare/typesense/card.webp) Typesense and Ceisium Search logos over a dark indexing diagram. ![Typesense and Ceisium Search logos over a dark indexing diagram.](https://ceisium.com/compare/typesense/card.webp) Typesense and Ceisium Search logos over a dark indexing diagram. 1. [Home](https://ceisium.com/) 2. / 3. [Compare](https://ceisium.com/compare/) 4. / 5. Typesense Published August 22, 2026 Updated September 13, 2026 # Typesense Alternative: Ceisium Search vs Typesense Compare Ceisium Search and Typesense for website search: setup, Next.js search quality, response time, pricing, and ongoing maintenance. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=Typesense%20Alternative%3A%20Ceisium%20Search%20vs%20Typesense&url=https%3A%2F%2Fceisium.com%2Fcompare%2Ftypesense%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fcompare%2Ftypesense%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fcompare%2Ftypesense%2F&t=Typesense%20Alternative%3A%20Ceisium%20Search%20vs%20Typesense) Verified September 13, 2026 Read 10 min Ceisium Search is the best Typesense alternative for teams that want accurate website search with less setup and tuning. We built it for people who want visitors to find the right page without managing search relevance themselves. Add a website URL, choose the pages to include, and Ceisium Search handles crawling, ranking, and tuning as the website changes. Typesense gives you more control over the search engine. You choose the records, searchable fields, filters, ranking, and hosting. That is useful when search is part of an application and your team wants to build the experience around it. We tried both products on the [Next.js docs opens in a new tab](https://nextjs.org/docs/app/getting-started/installation) . Here is how they compare on setup, search quality, response time, cost, and the work needed to keep search useful. ## TL;DR Area Typesense Ceisium Search Setup Create a Cloud cluster or self-host the engine, then create a collection and supply records. Add a website URL and choose the pages to crawl. Configuration Choose a schema, searchable fields, field priorities, filters, synonyms, and optional vector or hybrid search. Choose crawl paths, schedule, and page limit. Ceisium Search generates the relevance settings for the website. Installation Use an API client or an InstantSearch adapter, connected to your collection. Use the Search API in your existing UI or start from the UI examples in the app. Search quality Expected page in the first three results for 36.1% of our benchmark queries. Expected page in the first three results for 91.2% of our benchmark queries. Latency 225 ms median in our benchmark. 858 ms median in our benchmark. Ease of use Your team manages records and search settings, and chooses memory and CPU capacity. Ceisium Search handles crawling and search settings; your team reviews page coverage and visitor queries. Pricing Cloud charges for cluster capacity and bandwidth. Self-hosting adds your own infrastructure and operating costs. Plans from $29/month, with included usage and an optional hard stop. Main use case Application or catalog search where engineers need data, ranking, and hosting control. Public website search across docs, blogs, marketing pages, and product pages. The quality and latency figures come from [our September 3, 2026 benchmark Search quality Benchmark Results, response times, and methods for our search comparison.](https://ceisium.com/docs/benchmark/) , with optional relevance settings unchanged. The sections below explain the setup and limits. ## Setting up search ### Typesense You can [install Typesense on your own infrastructure opens in a new tab](https://typesense.org/docs/guide/install-typesense.html) or use Typesense Cloud. We used a Cloud cluster in Frankfurt for this comparison. Typesense groups records into a collection. For our docs site, each record contained a title, headings, body text, and page URL. We created the collection before importing the content. ![Typesense Cloud collection setup for the Next.js App Router records.](https://ceisium.com/compare/typesense/setup-collection.webp) We created a Typesense collection for the prepared records. Captured on August 24, 2026. ![Typesense Cloud collection setup for the Next.js App Router records.](https://ceisium.com/compare/typesense/setup-collection.webp) We created a Typesense collection for the prepared records. Captured on August 24, 2026. A [collection schema opens in a new tab](https://typesense.org/docs/30.2/api/collections.html) defines the fields and their types. Typesense can infer these, use explicit definitions, or combine both. We defined our fields explicitly so we could inspect the setup. ![Typesense schema with title, headings, body, and URL fields.](https://ceisium.com/compare/typesense/setup-schema.webp) Our Typesense setup used an explicit schema. Automatic field detection is also available. Captured on August 24, 2026. ![Typesense schema with title, headings, body, and URL fields.](https://ceisium.com/compare/typesense/setup-schema.webp) Our Typesense setup used an explicit schema. Automatic field detection is also available. Captured on August 24, 2026. We prepared 390 records from 280 pages in the Next.js App Router documentation. The Cloud browser editor showed a 1 MB limit and directed larger imports to another import path. ![Typesense Cloud browser editor showing a one-megabyte limit and command-line import options.](https://ceisium.com/compare/typesense/setup-browser-import-limit.webp) The Typesense browser editor directed larger imports to its JSONL or CSV command paths. Captured on August 24, 2026. ![Typesense Cloud browser editor showing a one-megabyte limit and command-line import options.](https://ceisium.com/compare/typesense/setup-browser-import-limit.webp) The Typesense browser editor directed larger imports to its JSONL or CSV command paths. Captured on August 24, 2026. We used the [bulk import API opens in a new tab](https://typesense.org/docs/30.2/api/documents.html) . It accepts JSON Lines: one record on each line. An HTTP 200 response can still contain individual record failures, so we checked each returned line. ![Completed Typesense import with all 390 records indexed.](https://ceisium.com/compare/typesense/setup-import-complete.webp) All 390 Typesense records were imported. We checked the response for individual failures. Captured on August 24, 2026. ![Completed Typesense import with all 390 records indexed.](https://ceisium.com/compare/typesense/setup-import-complete.webp) All 390 Typesense records were imported. We checked the response for individual failures. Captured on August 24, 2026. For the September benchmark, we used a fresh collection. Typesense requires [`query_by` opens in a new tab](https://typesense.org/docs/30.2/api/search.html) , which names the fields to search. We set it to `title,headings,body`, in that order, and left optional relevance settings unchanged. That order matters: Typesense gives earlier fields more weight when explicit weights are absent. We did not add synonyms, custom ranking, vector search, or hybrid search for this run. To build the search box, use a client library or the [Typesense InstantSearch adapter opens in a new tab](https://github.com/typesense/typesense-instantsearch-adapter) . The interface connects to the cluster, collection, and a search-only key. For a docs crawler, the [Typesense DocSearch guide opens in a new tab](https://typesense.org/docs/guide/docsearch.html) provides a separate scraper and UI. Your team runs that content import and keeps it current. ### Ceisium Search We create a project, add the website URL, and choose the paths, crawl schedule, and page limit. Once the crawl finishes, we can test queries in the playground. ![Ceisium source settings for the 280 Next.js App Router URLs.](https://ceisium.com/blog/ceisium-search-benchmark/ceisium-source.webp) Our Ceisium Search source includes the Next.js App Router docs, with a limit of 280 pages. ![Ceisium source settings for the 280 Next.js App Router URLs.](https://ceisium.com/blog/ceisium-search-benchmark/ceisium-source.webp) Our Ceisium Search source includes the Next.js App Router docs, with a limit of 280 pages. Ceisium Search turns those pages into searchable content and generates the search settings for the website. We made no manual relevance adjustments for the benchmark. For a website search box, create a Browser/public key and restrict it to the allowed website origins. The interface calls `POST /v1/search` with the project ID and query. The [quickstart ![](https://ceisium.com/blog/ceisium-cover-system.webp)Docs Docs | Ceisium Search Ceisium Search documentation for setup, crawl scope, indexing, the Search API, relevance tuning, observability, and security.](https://ceisium.com/docs/) includes the request and UI examples. With either product, use a key meant for browser search. Keep the Typesense bootstrap or admin key and the Ceisium Search Server key out of browser code. ## Search quality We ran the same 1,000 labeled queries on September 3, 2026. They included page names, full questions, typos, error messages, and alternative wording. Ceisium Search found an expected page in the first three results for **912 queries**. Typesense did so for **361**. Product Hit@3 MRR@10 Ceisium Search 91.2% 0.8468 Typesense 36.1% 0.3220 **Hit@3** is the share of queries with an expected page in the first three results. **MRR@10** measures how high the primary expected page ranks. First place earns 1, second earns 0.5, and pages outside the first ten earn 0. We average that score across all queries. Here is Hit@3 for each query category: Query type What the category tests Queries Ceisium Search Typesense Exact Known page terms 280 93.6% 72.5% Long-tail More specific descriptions 280 89.6% 18.9% How-to A task the visitor wants to complete 120 93.3% 2.5% Configuration Options and settings 80 97.5% 25.0% Error A failure the visitor wants to resolve 55 89.1% 20.0% Concept An idea the visitor wants to understand 55 80.0% 14.6% Variant Alternative wording 60 88.3% 38.3% Typo Misspelled words 50 96.0% 62.0% Navigation A destination within the documentation 20 75.0% 45.0% Typesense did better with known terms than in its other query categories, while still behind Ceisium Search. That matters for docs search as a visitor may know the problem without knowing the page title. We want people to describe what they need, even when they do not know the page name. This is similar to how search works on YouTube. You may have found yourself just describing the “vibe” and still getting the exact result you were looking for. This was one documentation site and a query set we wrote. Typesense received prepared records; Ceisium Search crawled the website. The test measures required setup with optional relevance settings unchanged. Tuned Typesense, vector search, or hybrid search may perform differently. The [full benchmark Search quality Benchmark Results, response times, and methods for our search comparison.](https://ceisium.com/docs/benchmark/) includes examples, source differences, and the method. ## Response time Typesense was faster in this run. Product Median latency P95 latency Ceisium Search 857.86 ms 1,983.09 ms Typesense 225.41 ms 759.55 ms The median is the middle request time. At P95, 95% of requests finished within the reported time. These are client wall-clock measurements, including the network round trip from Bengaluru. Location affects the wait, so test from where your visitors are. Typesense took about 0.23 seconds at the median, compared with Ceisium Search’s 0.86 seconds. Ceisium Search’s P95 was almost two seconds. We care about finding the right page, but visitors also feel that wait. [Reducing Ceisium Search’s latency](https://ceisium.com/changelog/faster-search-better-first-results/) is a priority for us in the coming weeks. ## Keeping search useful ### Content and relevance With Typesense, you maintain the connection between your source content and the collection. Check failed imports, update changed records, and remove records for deleted pages. If you use a scraper, include its runs in those checks. You can tune [ranking and relevance opens in a new tab](https://typesense.org/docs/guide/ranking-and-relevance.html) through field priorities, matching settings, synonyms, and other controls. Filters and sorting also let you shape a catalog or application search experience. Typesense Cloud operates the cluster. Your team still owns the records, settings, and interface. If you self-host, you also maintain the servers, upgrades, backups, and availability. With either option, you [choose memory and CPU capacity opens in a new tab](https://typesense.org/docs/guide/system-requirements.html) for your data and traffic. Cloud offers [automatic capacity upgrades opens in a new tab](https://cloud.typesense.org/security) when usage grows. You choose the starting size and whether to enable those upgrades, which can increase your bill. With Ceisium Search, choose a plan by pages and searches, then review source coverage and visitor queries. Its [crawl logs Observability Crawl Logs Track crawl status, page outcomes, failures, retries, and coverage.](https://ceisium.com/docs/crawl-logs/) show indexed, skipped, and failed pages. If a page is missing, check the source paths and crawl result first. Your website still needs useful content. If your docs do not answer a question, search cannot find a page that does. ### Analytics [Typesense analytics opens in a new tab](https://typesense.org/docs/30.2/api/analytics-query-suggestions.html) supports popular queries, searches with no hits, and click or conversion events. You configure the analytics rules and send the required events. Cloud enables the analytics process. Self-hosted installations need to enable it explicitly. In both cases, your team decides which data to collect and how to use it. [Ceisium Search analytics Start Analytics Read search events, no-result queries, popular queries, and engagement.](https://ceisium.com/docs/analytics/) shows search counts, average results, no-result queries, and popular queries, with CSV export. We use these to find content gaps and review searches that need attention. ## Cost ### Typesense The [Typesense server opens in a new tab](https://github.com/typesense/typesense) is open source under GPL-3.0. Self-hosting has no Typesense Cloud bill, but you pay for servers and the work needed to run them. Typesense Cloud charges for dedicated cluster capacity and outgoing bandwidth. The [pricing calculator opens in a new tab](https://cloud.typesense.org/pricing/calculator) lets you choose memory, processors, region, high availability, and multi-region delivery. On September 13, 2026, its default Ohio configuration showed: Item Selected configuration or price Memory 0.5 GB Processors 2 vCPUs, with one hour of burst capacity per day High availability and multi-region delivery Off Cluster $0.03/hour, or $21.60 for 720 hours Outgoing bandwidth $0.09/GB This small example has no high availability. A production setup may need more memory, processors, or nodes. The calculator does not charge per record or search request, but cluster capacity still limits the work it can do. [Cluster charges continue while it exists, even when idle opens in a new tab](https://cloud-help.typesense.org/article/billing-process) . The free tier gives eligible configurations [720 free cluster hours and 10 GB of bandwidth once per account opens in a new tab](https://cloud-help.typesense.org/article/how-does-the-free-tier-work) . It is not a monthly allowance. For keyword search, Typesense suggests [RAM around two to three times the indexed data size opens in a new tab](https://typesense.org/docs/guide/system-requirements.html) . Include crawler hosting and ongoing record updates when you estimate the complete cost. ### Ceisium Search [Ceisium Search plans ![](https://ceisium.com/blog/ceisium-cover-system.webp)Pricing Pricing | Ceisium Search Ceisium Search starts at $29/mo with clear workspace quotas for projects, indexed pages, searches, crawl schedules, and crawl pages checked.](https://ceisium.com/pricing/) include website crawling and search. Ceisium Search handles the settings that decide which pages appear first. Plan Monthly price Indexed pages Searches/month Starter $29 5,000 20,000 Growth $99 50,000 150,000 Pro $299 250,000 750,000 Plans also include project, source, and crawl allowances. Starter supports weekly scheduled crawls, Growth adds daily crawls, and Pro supports crawls as often as every six hours. Extra indexed pages cost $10 per 10,000; extra searches cost $10 per 100,000. Extra crawl pages checked cost $20 per 100,000. These charges are prorated to actual usage. Prices were checked on September 13, 2026. The units differ. Our 280 website pages became 390 Typesense records, while Cloud capacity depends on the indexed data and traffic. Compare the service bill and the work needed to prepare, refresh, and operate search. ### Traffic and cost controls Typesense’s [scoped search keys opens in a new tab](https://typesense.org/docs/guide/data-access-control.html) can restrict collections, filters, returned fields, and result counts. Its guide also recommends a proxy with DDoS protection. Extra traffic can consume cluster capacity and outgoing bandwidth even without a per-search charge. Ceisium Search sets [request limits on new API keys Start Search API Auth, request fields, response shape, pagination, errors, and examples.](https://ceisium.com/docs/search-api/) , with defaults based on the plan and key type. Starter Browser/public keys default to 60 requests per IP per minute and 100 per key per minute. You can edit those defaults. **Paid Ceisium Search plans allow overage charges by default.** Workspace admins can enable [hard-block mode ![](https://ceisium.com/blog/ceisium-cover-system.webp)Pricing Pricing | Ceisium Search Ceisium Search starts at $29/mo with clear workspace quotas for projects, indexed pages, searches, crawl schedules, and crawl pages checked.](https://ceisium.com/pricing/) to stop metered actions at their included allowance. Request limits control bursts; this monthly usage stop controls whether metered actions continue beyond the plan allowance. ## When to choose Typesense vs Ceisium Search? Choose **Typesense** when your engineers want to control the records, ranking, filters, and hosting. It fits application search, catalogs, and teams with an existing data pipeline. Self-hosting is also available when you need to operate the engine yourself. We would choose **Ceisium Search** for public website search across docs, blogs, marketing pages, and product pages. We built it for accurate results from the first setup, with less manual relevance work as the website grows. Before deciding, try questions from people who do not know your page titles. Check which pages they find, how long they wait, and what you need to maintain. [![](https://ceisium.com/blog/share-icons/x.webp)](https://twitter.com/intent/tweet?text=Typesense%20Alternative%3A%20Ceisium%20Search%20vs%20Typesense&url=https%3A%2F%2Fceisium.com%2Fcompare%2Ftypesense%2F)[![](https://ceisium.com/blog/share-icons/linkedin.webp) ](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fcompare%2Ftypesense%2F)[![](https://ceisium.com/blog/share-icons/hacker-news.webp)](https://news.ycombinator.com/submitlink?u=https%3A%2F%2Fceisium.com%2Fcompare%2Ftypesense%2F&t=Typesense%20Alternative%3A%20Ceisium%20Search%20vs%20Typesense) Copy image[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fceisium.com%2Fcompare%2Ftypesense%2F)[](https://twitter.com/intent/tweet?text=%E2%80%9CTypesense%20Alternative%3A%20Ceisium%20Search%20vs%20Typesense%E2%80%9D&url=https%3A%2F%2Fceisium.com%2Fcompare%2Ftypesense%2F) **Yuvraj** _Building Ceisium and working on growth @ SigNoz_ ceisium.com ### Contact Ceisium | Product and Support Questions Canonical URL: https://ceisium.com/contact/ Contact Ceisium for product, support, billing, security, privacy, and partnership questions. Contact Ceisium # Tell us what you need from your website search. Email is the direct support path for Ceisium Search. Send enough context to reproduce the problem or understand the request. Do not send passwords, private API keys, payment-card details, or other secrets. [yuvraj@ceisium.com](mailto:yuvraj@ceisium.com) ## Product and support questions For a crawl or search problem, include the affected project, the public source URL, the search query, what you expected, and what happened. A timestamp and request ID can help find related logs. Remove private content before you attach screenshots or copied output. ## Billing and account questions For billing help, use the email address connected to your Ceisium Search workspace and name the plan or invoice that needs review. Do not email full payment details. Payment information is handled by the payment provider and should stay in its secure checkout and portal. ## Security and privacy Use the same contact for a security report, a personal-data request, or a question about the privacy policy. For a security report, describe the affected surface and safe steps to reproduce the issue. Do not test against another customer or access data that is not yours. ## Partnerships and technical discussions Ceisium also welcomes practical questions about Ceisium Search integrations, documentation tools, and agent-readable websites. Include the product, site, and use case so the response can be specific. You can also review the [documentation](https://ceisium.com/docs/) and [About page](https://ceisium.com/about/) before you write. ## Response expectations Ceisium Search is an early product with direct founder support. Response time can vary with the request and its impact. Account access, active service failures, and security reports take priority over general product feedback. ### Ceisium Search Developer Resources | API and OpenAPI Canonical URL: https://ceisium.com/developers/ Find the Ceisium Search API, OpenAPI contract, authentication guide, capability discovery, Agent Skill, and integration documentation. Ceisium Search developer resources # Build with the Ceisium Search API. Use these stable resources to discover the public REST surface, understand authentication, create typed calls, and integrate project-scoped website search. ## Machine contracts [OpenAPI 3.1 Typed operations, unique operation IDs, request and response schemas, errors, and bearer authentication.](https://ceisium.com/openapi.json) [API capabilities Public JSON discovery for the current Ceisium Search REST surface.](https://api.ceisium.com/v1/capabilities) [Agent content index A generated directory of current documentation, trust pages, changes, and articles.](https://ceisium.com/llms.txt) [Ceisium Search Agent Skill Portable setup, Search API, safety, and troubleshooting instructions for coding agents.](https://ceisium.com/skill.md) ## Documentation - [Quickstart](https://ceisium.com/docs/) Add a source, crawl it, create a Search key, and run the first query. - [Search API and authentication](https://ceisium.com/docs/search-api/) Choose a Browser/public or Server key and handle requests, results, and errors. - [UI components](https://ceisium.com/docs/ui-components/) Connect the Search API to a website interface. - [Usage and limits](https://ceisium.com/docs/usage/) Understand search volume, crawling, plans, and limits. ## First Search request Create a project-scoped Server Search key in the Ceisium Search application. Store it in an environment variable, then send the project ID and query. ``` curl https://api.ceisium.com/v1/search \ -H "Authorization: Bearer $CEISIUM_SEARCH_KEY" \ -H "Content-Type: application/json" \ -d '{ "project_id": "prj_example", "query": "search setup", "top_k": 5 }' ``` ## Authentication boundary Search is not anonymous. Browser/public keys need exact allowed origins. Server keys belong only in trusted backend code. Never put a Server key in a client bundle, public repository, log, screenshot, or support message. The public health and capabilities endpoints do not return tenant data or credentials. ### Pricing | Ceisium Search Canonical URL: https://ceisium.com/pricing/ Current Ceisium Search plans, quotas, and usage notes. > For the complete documentation and public content index, see [llms.txt](/llms.txt). This page is the Markdown alternative to the canonical HTML page. # Pricing | Ceisium Search Ceisium Search plans have workspace quotas for projects, sources, indexed pages, searches, crawl schedules, and crawl pages checked. ## Starter — $29 per month Small teams getting Search live without operational overhead. - 3 projects - 3 sources per project - 5,000 indexed pages - 20,000 searches / month - Weekly scheduled crawls - 25,000 crawl pages checked / month ## Growth — $99 per month Growing docs and product surfaces with daily freshness. - 10 projects - 10 sources per project - 50,000 indexed pages - 150,000 searches / month - Daily or weekly scheduled crawls - 250,000 crawl pages checked / month ## Pro — $299 per month Larger Search deployments with six-hour crawl options. - Unlimited projects - Unlimited sources per project - 250,000 indexed pages - 750,000 searches / month - Every 6 hours, daily, or weekly scheduled crawls - 1,000,000 crawl pages checked / month ## Usage beyond included amounts - $10 / extra 10k indexed pages, prorated - $10 / extra 100k searches, prorated - $20 / extra 100k crawl pages checked, prorated Extra projects and sources are plan-packaging limits. Checkout starts inside the app so the subscription attaches to the right workspace. ## Frequently asked questions ### What counts as an indexed page? An indexed page is a page stored in the searchable corpus for a workspace. It is the primary storage and value limit for Search. ### What happens when usage goes over the included amount? Indexed pages, searches, and crawl pages checked use prorated overages: $10 / extra 10k indexed pages, prorated; $10 / extra 100k searches, prorated; $20 / extra 100k crawl pages checked, prorated. ### Can usage hard-block instead of continuing with overages? Yes. Paid workspaces default to continuing through metered overages, and workspace admins can switch billing behavior to hard-block mode. ### What happens if my renewal payment fails? If a renewal payment fails, your search stays available for 72 hours while you update your payment details. Your plan limits still apply. Search pauses after that until payment succeeds. This grace period does not apply to cancellations or failed first payments. ### Where does checkout happen? Checkout starts from the Ceisium app after sign-in so the subscription can be attached to the correct workspace. ### Privacy Policy | Ceisium Canonical URL: https://ceisium.com/privacy/ Read how Ceisium collects, uses, shares, protects, and retains personal data and customer content. Privacy Policy # How Ceisium handles data. This policy explains what Ceisium collects, why it is used, when it is shared, how long it is kept, and the choices available to people who visit the website or use Ceisium Search. Last updated: August 22, 2026 ## 1\. Scope This policy applies to ceisium.com, the Ceisium Search web application, the Ceisium Search API, and related support and operational services. In this policy, “Ceisium,” “we,” and “us” refer to the company. “Ceisium Search” refers to the search product. A customer controls the websites and content that it asks Ceisium Search to crawl and index. The customer is responsible for having a lawful basis to make that content available and for giving notices to its own users when required. Ceisium processes that customer content only for the purposes described in this policy and the applicable service terms. ## 2\. Information we collect ### Account and workspace information When you sign in, Ceisium receives the account information that you allow the authentication provider to share. This can include your name, verified work email address, and basic profile details. We also store workspace membership, roles, project settings, credential metadata, and records needed to protect the account. Secret credentials are protected and are not shown again after creation where the product does not need to recover them. ### Customer content and search data Ceisium Search processes source URLs, crawl rules, extracted public page content, page metadata, search indexes, crawl outcomes, search queries, result metadata, and search-quality signals needed to provide and improve the requested search service. A customer should not add a private or sensitive source unless a separate approved product and security design supports that content. ### Billing information We store plan, subscription, invoice, usage, and payment-status records. A payment provider handles checkout and payment processing. Ceisium does not ask you to send full payment-card details through the website, application, or support email. ### Support and communications If you contact us, we receive the email address and information you include in the message. This can include product feedback, troubleshooting details, or a security or privacy request. Do not include passwords, private keys, or full payment details. ### Technical and usage information When you use the website or product, we can receive an IP address, device and browser type, user agent, language, referring page, requested URL, timestamps, response status, performance data, error data, and product-interaction data. We use this information for delivery, security, reliability, abuse prevention, support, and product analytics. ## 3\. How we use information Ceisium does not sell personal information. Ceisium does not operate an advertising network. Ceisium does not use customer content to train a Ceisium foundation model. - Provide accounts, workspaces, crawling, indexing, search, billing, and support. - Authenticate requests, enforce project boundaries, and prevent fraud or abuse. - Monitor availability, investigate errors, recover failed work, and protect the service. - Measure product use and improve setup, search quality, performance, and documentation. - Send service messages such as invitations, security notices, and billing information. - Meet legal duties, resolve disputes, and enforce the service terms. ## 4\. Service providers We use service providers only where they help us operate the website, Ceisium Search, or the business. Their processing is governed by their agreements and privacy terms. We disclose provider categories because they explain the relevant data flow without publishing internal system design. A provider can process data in a country other than the country where you live. We select providers that publish privacy and security terms. We can change providers as the product changes and will update this policy when a change materially affects the processing described here. - infrastructure and hosting providers for website delivery, security, crawling, storage, queues, and email delivery. - authentication providers when a user signs in. - payment providers for checkout, subscriptions, invoices, payment status, and usage billing. - analytics and reliability providers for product analytics, error investigation, metrics, traces, and bounded operational logs. - model-processing providers for defined product operations when a requested feature needs that processing. ## 5\. Model processing Some Ceisium Search features can send a limited part of public page content, a search query, candidate result text, or setup context to a model-processing provider when the requested feature needs that processing. The provider processes the information to return the requested result. Its business terms and data controls apply to that processing. Customers should avoid indexing data that is not suitable for this processing. Ceisium limits the information sent to the part needed for the requested operation. The exact product implementation can change without changing the purpose described in this policy. ## 6\. Cookies and analytics choices Ceisium uses first-party cookies for authentication, security, product operation, analytics consent, anonymous analytics identity, and session continuity. The website does not need third-party advertising cookies to provide the service. Use “Analytics preferences” in the site footer to review or change the optional analytics choice. Ceisium also respects an enabled Global Privacy Control signal by disabling optional browser analytics. Essential storage used for sign-in, security, or a requested service remains active because the website and product need it to work. ## 7\. Retention and deletion We keep account and customer data while an account or service remains active and for the limited period needed to provide recovery, billing, security, support, and legal records. Operational logs and detailed analytics normally have shorter retention than account and billing records. Aggregated information that no longer identifies a person or customer can remain for product measurement. When a customer archives or deletes a project or source, access can end before all copies are removed from recovery systems. Backups, fraud-prevention records, payment records, and records needed for a legal claim can remain for their normal protection or legal period. A provider can also retain limited records under its own lawful duties. You can ask about the retention that applies to a specific account or request deletion where applicable. We can require identity verification and can keep information when law, security, billing, or a valid legal claim requires it. ## 8\. Security Ceisium uses administrative, technical, and organizational measures intended to protect personal data and customer content. These measures include access controls, project boundaries, protected credentials, monitoring, and procedures for responding to security events. No online service can promise absolute security. Customers are responsible for keeping account access and Search keys secure, selecting the correct key type, restricting browser keys to approved origins, and rotating a credential that may have been exposed. Do not send a password, private key, private connection string, or full payment-card detail by email. ## 9\. Privacy rights Depending on your location and the applicable law, you can have a right to ask for access, correction, deletion, restriction, portability, or an objection to some processing. You can also have a right to withdraw consent where consent is the basis for processing and to complain to a local data-protection authority. We can ask for information needed to verify your identity and protect the account. A request can have legal limits. If Ceisium processes personal data only on behalf of a customer, we can direct the request to that customer because it controls the data. ## 10\. International processing Ceisium and its providers can process information in countries where they or their systems operate. Those countries can have data-protection rules that differ from the rules where you live. Where required, Ceisium relies on a lawful transfer mechanism and provider commitments intended to protect the transferred information. ## 11\. Children Ceisium Search is a business service and is not directed to children. If you believe a child has provided personal data to Ceisium, contact us so we can review and remove it where required. ## 12\. Changes to this policy We can update this policy when the product, provider categories, business, or legal requirements change. The date at the top shows the latest revision. If a change materially affects how active customers’ data is used, we will provide an appropriate notice through the product or email. ## 13\. Contact For a privacy question or request, email [yuvraj@ceisium.com](mailto:yuvraj@ceisium.com) . Include enough information to identify the relevant account or interaction, but do not include passwords, API keys, private connection strings, or full payment details. ### Terms of Service | Ceisium Canonical URL: https://ceisium.com/terms/ Read the terms for Ceisium accounts, public-source crawling, Search API use, plans, billing, acceptable use, and service limits. Terms of Service # The rules for using Ceisium Search. These terms explain the service, customer responsibilities, billing, acceptable use, service changes, and the limits that apply when you use Ceisium Search. Last updated: August 22, 2026 ## Terms of Service These Terms of Service govern access to ceisium.com, the Ceisium Search web application, the Ceisium Search API, and related services. In these terms, “Ceisium,” “we,” and “us” refer to the company. “Ceisium Search” refers to the search product. “You” means the person using the service and, when that person acts for a company or other organization, that organization. ## 1\. Agreement and authority By creating an account, selecting a plan, calling the Search API, or otherwise using the service, you agree to these terms and the [Privacy Policy](https://ceisium.com/privacy/) . If you do not agree, do not use the service. If you use Ceisium Search for a company or other organization, you confirm that you have authority to accept these terms for it. You must be legally able to enter into a binding agreement. Ceisium Search is a business service and is not directed to children. ## 2\. The service Ceisium Search provides managed public website crawling, content extraction, project-scoped indexing, ranked search, crawl and search records, integration guidance, and an authenticated Search API. A workspace can contain members and projects. Each project separates its sources, Search keys, index, usage, and results. Subject to these terms and the limits of your plan, Ceisium gives you a limited, non-exclusive, non-transferable right to use Ceisium Search for your own business or internal purposes and to show authorized Search API results in websites and applications that you operate. This right ends when your right to use the service ends. Documentation, pricing, plan limits, and product notices can describe additional operational rules for a feature. If a signed order or separate written agreement expressly conflicts with these terms, that signed agreement controls only for the conflict. ## 3\. Accounts, workspaces, and access You must provide accurate account and billing information and keep it current. You are responsible for activity under your account, for people you invite to a workspace, and for assigning suitable workspace roles. Keep sign-in methods, session access, Search keys, and other credentials confidential. Do not share one person’s account as a substitute for inviting an authorized workspace member. Tell Ceisium promptly if you believe an account, workspace, or credential has been compromised. You are responsible for configuring projects, sources, crawl scope, schedules, allowed origins, result URL restrictions, and Search key type for your intended use. Review crawl outcomes and test representative queries before you rely on an index in a public experience. ## 4\. Customer content and public sources You keep your rights in websites, text, configuration, queries, feedback, and other material that you provide or direct Ceisium to process. You give Ceisium a limited, worldwide, non-exclusive right to access, fetch, copy, extract, transform, index, store, transmit, and display that material only as needed to provide, secure, support, and maintain the service and to meet legal duties. You must have the rights and permissions needed for each source and for the processing you request. A URL being publicly reachable does not by itself prove that you own it or may use all of its content. You are responsible for applicable notices, permissions, copyright, privacy, data-protection, and other legal duties for your sources and your use of search results. Use Ceisium Search for content that is suitable for the approved product design. Do not use the current public-source workflow to submit passwords, private keys, private connection strings, payment-card data, health records, or other sensitive or regulated information without a separate written security and product design that expressly supports it. ## 5\. Acceptable use You may not use Ceisium Search, or help another person use it, in a way that harms the service, another customer, a source website, or any person. You also may not use the service in violation of applicable law or another person’s rights. - Do not crawl, index, search, or distribute content when you lack the required rights or authorization. - Do not upload or transmit malware, harmful code, unlawful content, deceptive material, spam, or content that infringes intellectual-property, privacy, publicity, or other rights. - Do not probe, scan, disrupt, overload, or attempt unauthorized access to Ceisium Search, its providers, another workspace, another project, or a source website. - Do not bypass authentication, allowed-origin checks, rate limits, billing controls, crawl scope, plan limits, or other technical restrictions. - Do not use credentials, accounts, or content obtained through fraud, impersonation, or unauthorized access. - Do not reverse engineer or copy protected parts of the service except where applicable law does not allow that restriction. - Do not resell or provide Ceisium Search itself as your own service unless a written agreement or plan expressly permits it. You may use authorized Search API results in your own website or application. ## 6\. Search keys and API use Search is not anonymous. A Search key belongs to one project and must be used only for that project. Browser/public keys must have exact allowed origins and can have result URL restrictions. Server keys must remain only in trusted backend systems. Do not put a Server Search key in browser code, a public repository, a public build artifact, a log, a screenshot, or a support message. Rotate a key if it can have been exposed. You are responsible for requests made with your active credentials until you revoke or rotate them, except to the extent caused by Ceisium’s breach of these terms or applicable law. Follow the published request schema, response schema, rate limits, and error guidance. Do not treat ranking scores as universal measures or use the API in a way that creates misleading claims about source content. ## 7\. Plans, usage, and billing Plan prices, included quotas, and current usage rates are shown on the pricing page and during checkout. Checkout starts after sign-in so a subscription can attach to the correct workspace. Dodo Payments processes checkout, subscriptions, invoices, payment status, and usage-based billing. Paid plans can include a recurring base fee and metered charges for indexed pages, searches, and crawl pages checked. The workspace administrator can use the available billing settings to choose whether supported metered usage continues through overages or hard-blocks at the included amount. Project and source counts are plan limits and can require an upgrade instead of an overage. You authorize the payment provider to charge the selected payment method for amounts shown at checkout and for applicable metered usage. You are responsible for taxes and accurate billing information. A failed or overdue payment can limit paid features after the provider and Ceisium record the related subscription state. You can manage or schedule cancellation through the available billing portal. When a cancellation is scheduled for the end of a current billing period, paid access normally continues until that period ends. Plan changes can require cancellation of the current usage-based subscription before a new tier starts. Refunds, credits, and payment disputes are handled under information shown at checkout, any applicable written plan terms, and applicable law. Contact Ceisium promptly if you believe a charge is incorrect. ## 8\. Third-party services Ceisium uses third-party services to provide infrastructure, authentication, search, model processing, analytics, observability, email, and billing. The current provider categories and data uses are described in the [Privacy Policy](https://ceisium.com/privacy/) . A third-party service can have its own terms and privacy notice. Ceisium does not control a customer’s source website, identity provider, payment provider, or other service that the customer chooses to use with Ceisium. Features that depend on a third party can change or stop if that third party changes its service or terms. ## 9\. Service changes and availability Ceisium can add, change, limit, or remove Ceisium Search features as the product develops. We will try to give reasonable notice when a material change affects an active paid service, unless a faster change is needed for security, law, provider failure, abuse prevention, or reliability. Crawling and search depend on source availability, network conditions, third-party systems, content quality, and configuration. Ceisium Search does not promise that every page will be discovered, that every query will return a result, that results will always have the order you expect, or that the service will be uninterrupted or error-free. No service-level commitment applies unless a separate signed agreement expressly provides one. Preview, experimental, or evaluation features can change or stop without notice and may have different reliability, retention, or support limits. Do not use them for a critical workload unless the related documentation and a separate approved design support that use. ## 10\. Suspension and termination You can stop using Ceisium Search at any time. A paid subscription must also be managed through the available billing flow. Stopping product use by itself does not cancel charges already authorized or incurred. Ceisium can limit, suspend, or terminate access when needed to prevent a security threat, protect another customer or source, respond to unlawful or abusive use, enforce plan or payment limits, comply with law, or address a material breach of these terms. When practical, we will give notice and a chance to correct the problem. After termination or archive, access to projects, indexes, and other content can end. Current recovery and deletion periods are described in the [Privacy Policy](https://ceisium.com/privacy/) and product documentation. Some account, billing, security, and dispute records can remain when law or a legitimate operational need requires them. Sections that by their nature must continue after termination will continue, including responsibility for fees already incurred, intellectual-property terms, disclaimers, liability limits, and general contract terms. ## 11\. Intellectual property and feedback Ceisium and its licensors keep all rights in the service, software, design, documentation, branding, and improvements, except for customer content and third-party material. These terms do not grant a right to use Ceisium names, logos, or marks except to identify an authorized integration accurately. If you provide a suggestion or feedback, you give Ceisium a non-exclusive, worldwide, perpetual, irrevocable, royalty-free right to use it to operate, improve, and promote the service. This does not transfer ownership of your customer content and does not permit Ceisium to publish your confidential information or identify you as a customer without permission. ## 12\. Disclaimers To the maximum extent permitted by applicable law, the service is provided “as is” and “as available.” Ceisium disclaims implied warranties of merchantability, fitness for a particular purpose, non-infringement, and any warranty arising from a course of dealing or usage of trade. Ceisium does not warrant source content, search results, third-party services, or decisions made from them. You must review results and use suitable human or technical checks for your context. Nothing in these terms excludes a warranty or consumer right that applicable law does not allow the parties to exclude. ## 13\. Limits of liability To the maximum extent permitted by applicable law, Ceisium will not be liable for indirect, incidental, special, exemplary, punitive, or consequential loss, or for lost profits, revenue, goodwill, business opportunity, or data, arising from or related to the service, even if Ceisium was told that the loss was possible. To the maximum extent permitted by applicable law, Ceisium’s total liability arising from or related to the service will not exceed the amount you paid to Ceisium for the service during the 12 months before the event that caused the claim. These limits do not apply where applicable law does not allow them, or to liability that cannot legally be limited. ## 14\. Changes to these terms Ceisium can update these terms when the product, providers, business model, or legal requirements change. The date at the top shows the latest revision. If a change materially affects active customers, we will provide an appropriate notice through the product or email before the change takes effect when practical. Your continued use after updated terms take effect means that you accept them. If you do not accept an update, stop using the service and manage any active subscription before the update takes effect. ## 15\. General terms These terms, the [Privacy Policy](https://ceisium.com/privacy/) , applicable pricing and checkout terms, and any separate signed agreement are the agreement about the service. If one provision is unenforceable, it will be limited to the minimum extent needed and the remaining provisions will continue. A failure to enforce a provision is not a waiver. You may not transfer these terms or an account without written permission, except as part of a lawful transfer of your business with notice to Ceisium. Ceisium can transfer these terms as part of a reorganization, financing, sale, or transfer of the service, subject to applicable law. Neither party is responsible for a delay or failure caused by events outside its reasonable control, but this does not remove payment duties for service already provided. These terms do not create a partnership, employment, agency, or joint venture between you and Ceisium. ## 16\. Contact Questions about these terms, billing, or the service can be sent to yuvraj@ceisium.com. Do not include passwords, Search keys, private connection strings, or full payment-card details in email. ## Related information Read the [Privacy Policy](https://ceisium.com/privacy/), [pricing](https://ceisium.com/pricing/), and [product documentation](https://ceisium.com/docs/).