Skip to main content

Site Crawl

Open in QuickMonitoring→ Sites → [Your Site] → New Scan

QuickMonitoring crawls every page of your site using Playwright — a real headless browser. No code install required. It visits each page, captures a screenshot, and records all resources, links, and console output. The crawl is the foundation for every other check.

What you'll see​

After triggering a scan, go to Sites → your site → Scans (sidebar) → select the scan to see its crawled pages.

Each crawled page is listed with:

  • Full page URL
  • Screenshot thumbnail (click to expand)
  • HTTP status code
  • Page load time

This is your proof that the crawler actually visited the page — not just that it exists.

How to use it​

  1. Go to Sites → click your site
  2. Click "New Scan" or "Run Scan"
  3. Wait for the crawl to complete — progress shows in real time on the scan detail page
  4. Go to Sites → your site → Scans (sidebar) → select the scan to browse all crawled URLs

The crawler starts at your root URL and follows every internal link it finds. It won't crawl pages that require authentication unless you've configured bypass credentials (see Phase 4).

Pass condition​

✅ You've earned this achievement when your scan shows at least 1 crawled page with a screenshot and a 200 OK status.

Common issues​

  • "0 pages crawled" — Check that your site URL is correct and publicly accessible. Private or staging sites behind IP whitelists, VPNs, or HTTP basic auth will block the crawler. The URL must be reachable from the open internet.

  • Screenshots are blank or show only a white page — Some sites aggressively block headless browsers with bot detection. Try enabling JavaScript rendering delays in scan settings. If the site uses heavy client-side rendering, the crawler may need more time to let the page hydrate.

  • Missing pages you expected — The crawler follows links. If a page isn't linked from anywhere, it won't be crawled. Add it to the sitemap or link it from a crawled page. You can also manually specify additional start URLs in site settings.

  • Crawl stops early — Check the crawl depth setting. A low depth limit will stop the crawler before it reaches deeply nested pages. Increase it in scan configuration.