Most monitoring tools tell you when a page changes. But what about pages that didn't exist yesterday?
A competitor launches a new product. A job board posts a role you've been waiting for. A government agency publishes a new filing. A news site drops a story in your niche. You can't monitor what you don't know exists yet.
PageCrawl.io's Page Discovery scans websites on a schedule and lists the pages it finds. You can review those candidates first or let matching pages become monitors automatically, within your configured limits. Start with one website and a narrow business purpose before expanding the watchlist.
What is the difference between change monitoring and page discovery?
Change monitoring compares content on a page you already track. Page discovery looks for URLs that your watchlist does not yet contain, using the site's available links and sitemaps. A newly discovered URL is new to your discovery history; that alone does not prove the publisher created it that day.
If a competitor updates its existing pricing page, change monitoring can detect the edit. If it adds a product page that your discovery sources expose, a later scan can find that URL. Discovery coverage depends on those sources and the selected scan method, so it is not a guarantee that every page on a website will be found.
How does page discovery work?
Add a website from Discovered Pages, choose whether to review candidates or monitor them automatically, and let the first scan populate the list. PageCrawl creates a template for that website. Use the template to adjust discovery sources, monitoring rules, tracked content, and limits after you have inspected representative results.
- Automatic (recommended) combines sitemap and link discovery using the available sources for the website. It does not mean every possible scanning method always runs.
- Sitemap only checks the site's XML sitemap daily. See sitemap monitoring for what these published URL lists can and cannot reveal.
- Homepage links only follows links on the homepage, with daily or weekly options.
- Follow links 2 levels deep, Follow links 3 levels deep, and Full website crawl extend coverage beyond the homepage. These weekly options are available on Enterprise and Ultimate.
The scan schedule and the new monitors' Check Frequency are separate. A daily scan might find a new product URL and create a monitor that checks its price hourly afterward. Setting that monitor to hourly does not make a weekly discovery scan run hourly, and neither schedule guarantees detection on the publication day.
How do you filter candidates before expanding the watchlist?
A large site can expose many URLs that do not belong in your monitoring programme. Start with Let me review pages first, inspect relevant and irrelevant examples, then configure the template's automatic-tracking rules. These rules determine which discovered candidates become monitors; finding a URL and choosing to monitor it are separate steps.
URL rules are a useful starting point. Match the section you need, such as product URLs, using the rule types available in the editor. Test the rule against actual candidates rather than assuming that every URL containing “product” is a product detail page.
Title and text rules narrow the list by content. A job board might put all listings under /jobs/, but a title rule for "Software Engineer" can narrow job listing and career page monitoring to relevant roles. Inspect examples of both accepted and rejected candidates before enabling automatic tracking.
Exclude rules help keep login, search, archive, and other unwanted pages out of monitoring. Also check Track All Pages and Always include top-level pages, because these settings affect the scope alongside your rules. Set a Tracked Page Limit before letting the template add monitors.
| Candidate | Review question | Possible decision |
|---|---|---|
| Product detail page | Does it contain the product, price, or stock fields we need? | Include after checking the capture |
| Category or filtered search | Is the list itself useful, or would it duplicate product monitors? | Monitor deliberately or exclude |
| Localized version | Do we need a separate market or language view? | Assign a distinct scope or exclude |
| Login or account page | Is this an authorized source with a configured login? | Review access before adding |
| Old article discovered for the first time | Is it relevant despite its age? | Assess content rather than treating it as a new publication |
For a site with several regional versions, use the regional monitoring matrix to decide which views belong in the watchlist. Discovery should not silently multiply a team’s coverage across markets it never intended to track.
Use Cases
New product alerts. Find competitor product pages or new rebate offers when a scan encounters them, then review or monitor the relevant items. Use the capture and discovery dates accurately rather than assuming the scan ran when the product launched.
Documentation and changelog tracking. Keep up with software tools and platforms you rely on. Discover new help articles, API docs, or changelog entries as they're published.
News and media. Discover relevant pages from news outlets and academic journals, then use rules to focus on your company, industry, or competitors. Match the scan cadence to the needs of your news and media monitoring workflow.
Regulatory and government. Track agency websites for new filings, guidance documents, or regulations. The Federal Register alone publishes new US rules and notices every business day, and finding a new regulation on day one versus day seven makes a real difference for compliance teams.
Real estate. Watch listing sites for new properties in specific areas. URL filters target neighborhoods while text filters catch price ranges or features.
Competitor content. Track when competitors publish new blog posts or documentation. Over time, you'll see their publishing patterns and content strategy.
How do you get notified about new pages?
Review newly found URLs on Discovered Pages and check the discovery notification settings for the website. After a candidate becomes a monitor, its content changes follow the template's selected notification channels and any covering report. Verify both paths during setup: hearing about a new URL and hearing about its later edits are different outcomes.
When should discovered pages become monitors automatically?
Enable automatic monitoring after representative candidates have the right scope, tracked content, and notification route. Matching discoveries then become monitors with the template's settings, subject to limits. Leave the rules empty and choose review-first when a person needs to approve each candidate before it consumes ongoing checks or sends content alerts.
The page discovery guide explains the effect of rule changes and limits. The template's Tracked Page Limit includes pages added to that template manually. At the team's active-page allowance, additional monitors can be created paused; reaching the total-page limit can stop discovery. Inspect the actual result rather than assuming every discovered URL is now active.
Monitor a few approved candidates through a successful capture before expanding. The acceptance-test protocol provides positive, negative, and failed-capture checks for the settings the template will share. The site-wide tutorial covers tracking multiple pages within one website.
Choosing your PageCrawl plan
The Free plan includes a limited discovery test window for new accounts, not indefinite automatic discovery. Check the window shown in your account. Once it ends, discovery stops finding new pages automatically; existing monitors continue under their normal settings and plan limits. Choose a paid plan for an ongoing business discovery programme.
| Plan | Price | Pages | Checks / month | Frequency |
|---|---|---|---|---|
| Free | $0 | 6 | 220 | every 60 min |
| Standard | $8/mo or $80/yr | 100 | 15,000 | every 15 min |
| Enterprise | $30/mo or $300/yr | 500 | 100,000 | every 5 min |
| Ultimate | $99/mo or $999/yr | 1,000 | 100,000 | every 2 min |
Annual billing saves two months across every paid tier. Enterprise and Ultimate scale up to 100x if you need thousands of pages or multi-team access.
Discovered-page capacity, active monitors, and monthly checks are separate limits. Standard's base discovery limit is 20,000 pages per website, while its active-page and check allowances determine how much of that list you can monitor continuously. Use the check-budget worksheet before enabling broad automatic tracking.
All plans include the PageCrawl MCP Server, letting you ask Claude things like "what new pages did my competitor add this week?" and pull answers directly from your discovery history without leaving your workflow. AI assistants can create monitors through conversation on every plan, including Free.
How do you set up the first website?
Open Discovered Pages, click Add Website, and enter the site URL. Start with Let me review pages first if the scope has not been validated. Choose the check frequency and notification channels for pages that become monitors, then inspect the scan results and the website template before expanding automatic tracking.
- Enter the Website URL.
- Under What to track, choose Let me review pages first, Track top-level pages only, or Track all pages on this website.
- If tracking all pages, set an appropriate Max monitored pages.
- Choose Check Frequency and Notify me via for the resulting monitors.
- Click Add Website, then inspect relevant and irrelevant candidates in the results.
- Open the website's template to adjust discovery mode, automatic-tracking rules, tracked elements, and limits.
- Verify a few resulting captures and their routing before broadening the scope.
Which limits should a business team check?
Check the discovery window, per-website discovery allowance, template tracking cap, team page allowance, and remaining checks. These determine different parts of the workflow. A large discovery list does not authorize unlimited active monitors, and a fast interval for existing monitors does not increase the cadence of the discovery source.
The base per-website discovery allowances are 2,000 on Free during its test window, 20,000 on Standard, and 100,000 on Enterprise and Ultimate. Account quantities can change paid allowances. Deeper scanning methods require Enterprise or Ultimate. Use the limits displayed in your account when planning the rollout.
Start with one website, one business purpose, and a review of the first candidates. Once the rules find relevant pages and their monitors capture the right content, expand with a clear owner and a check budget.
