Skip to main content
Ceisium logo Ceisium

Crawl Logs

Track crawl status, page outcomes, failures, retries, and coverage.

Browse docs

Crawl Logs show what Ceisium found, indexed, skipped, or failed during a crawl.

Crawl Logs list recent crawl runs with status, timing, source scope, and page outcome counts.
Ceisium Crawl logs page showing source selection, run filters, crawl statuses, page counts, and recent SigNoz crawl jobs.

Crawl Logs list recent crawl runs with status, timing, source scope, and page outcome counts.

Run states

StateMeaning
QueuedCrawl is waiting to start.
RunningCeisium is fetching and indexing pages.
CompletedCrawl finished and page counts are ready.
FailedCrawl stopped before completing. Check the error and source settings.

Page counts

CountMeaning
DiscoveredURLs found during crawl.
IndexedPages added to search.
SkippedPages ignored because of scope, duplicate URLs, or unsupported content.
FailedPages Ceisium tried but could not fetch or index.
Open URL details to inspect individual indexed, skipped, and failed URLs for a crawl.
Ceisium URL details drawer showing indexed, skipped, failed, and total URL counts with a folder tree of indexed SigNoz documentation pages.

Open URL details to inspect individual indexed, skipped, and failed URLs for a crawl.

SigNoz example

For https://signoz.io/docs/, a healthy crawl should discover docs paths such as:

TXT
/docs/install//docs/install/docker//docs/instrumentation//docs/instrumentation/python//docs/userguide/logs/

If /docs/instrumentation/python/ is missing, check include and exclude paths before rerunning the crawl.

Debug flow

  1. Open the latest crawl.
  2. Filter URL events to Failed and Skipped.
  3. Check whether the missing page is outside the source path.
  4. Adjust source scope.
  5. Run crawl again.