Most websites maintain an XML sitemap listing every page on the site. PageCrawl can monitor these sitemaps to detect new pages, removed URLs, and structural changes automatically.
Sitemap monitoring is one of the discovery methods available in PageCrawl's Page Discovery feature. This article covers how sitemap monitoring works specifically.
Why Monitor Sitemaps?
XML sitemaps are generated automatically by most CMS platforms (WordPress, Shopify, Squarespace, Wix). A single sitemap file can contain thousands of URLs, making it the fastest way to detect new content on a website.
Common use cases:
- Competitor tracking - Know the same day a competitor launches a new product or publishes content
- Job postings - Catch new listings on target company websites before they appear on job boards
- Regulatory monitoring - Track new filings, guidance documents, and regulations on government sites
- Documentation changes - Detect new API docs, changelogs, and deprecation notices
How It Works
- PageCrawl downloads the website's XML sitemap on your configured schedule
- New URLs are compared against the previous scan
- Newly discovered pages are matched against your filters
- You receive a notification listing the new pages
- Optionally, matched pages are auto-monitored for content changes
Getting Started
- Click Track New Page and select Scan a Website
- Enter the website URL (e.g.,
competitor.com) - PageCrawl automatically detects the sitemap
- Set your check frequency and add filters
- Enable notifications and optionally enable auto-monitoring
Filtering Discovered Pages
Large websites may add many pages between checks. Filters help you focus on what matters:
- URL filters - Match by path patterns (e.g.,
/products/,/blog/2026/*) - Exclude filters - Skip irrelevant sections (e.g.,
/products/accessories/) - Title/content filters - Match against page title or body text after fetching
Exclude filters always take priority over include filters. You can combine multiple filter types.
Auto-Monitoring
When auto-monitoring is enabled, pages matching your filters are automatically added to your monitoring workspace. For example:
- A competitor publishes a new product page on Monday
- Sitemap monitoring discovers the URL the same day
- From Tuesday onward, PageCrawl tracks that page for price and content changes
No manual setup required. Combined with templates, auto-monitored pages inherit your preferred check frequency, notification channels, and tracking settings.
Beyond Sitemaps
Not all websites have complete sitemaps. PageCrawl supplements sitemap monitoring with additional discovery methods:
- Base URL Link Discovery - Extracts all links from a specific page
- Deep Scan - Follows links multiple levels deep with JavaScript rendering
- Automatic Mode - Runs all discovery methods together and deduplicates results
See Page Discovery for full details on all discovery methods.
Plan Limits
Sitemap monitoring is available on all plans:
| Plan | Pages per Website |
|---|---|
| Free | Up to 2,000 |
| Standard | Up to 20,000 |
| Enterprise | Up to 100,000 |
All plans include filters, notifications, and auto-monitoring.