Automatically Discover New Pages To Track

PageCrawl is designed to make website change monitoring and management seamless. Page Discovery automatically finds new pages on a website and, if you choose, starts monitoring them for you, so your coverage stays up to date as the site grows. The fastest way to get started is to add a whole website from the Discovered Pages page, then fine-tune how it works from the template that gets created for you.

Add a Website to Discover

Start from the Discovered Pages page and click Add Website. This is the quickest way to begin: enter a website and PageCrawl handles the rest.

In the dialog, set:

1. Website URL - the site you want to monitor (for example, a competitor's store or your own marketing site).
2. What to track - Track all pages on this website (every discovered page becomes a monitor, up to the Max monitored pages you set), Track top-level pages only (like /pricing and /about), or Let me review pages first (nothing is monitored until you pick).
3. Check Frequency - how often discovered pages are checked for changes.
4. Notify me via - the channels that should receive change alerts.

Click Add Website. PageCrawl scans the site, lists everything it finds on the Discovered Pages page, and creates a template with sensible defaults that controls how this website is monitored.

Adjust Advanced Configuration via the Template

Adding a website automatically creates a configured template for it. The template is where you fine-tune scanning, filters, and what gets tracked. Open it from Templates settings (or the "Templates settings" link in the Add Website dialog) and edit the template for your website.

Choose a Scanning Method

The template's Discover New Pages section controls how PageCrawl looks for new pages. Under Source, the Mode setting defaults to Automatic (recommended), which combines methods to find pages using the best approach for each website:

• Automatic (recommended): Combines sitemap and link discovery to find pages using the best method for the website. This is the default and recommended setting.
• Homepage links only: Discover new links by following links on the homepage. Available as a daily or weekly check. Useful if you want to focus on pages directly linked from the main page.
• Sitemap only: Discover pages listed in the website's sitemap, checked daily. Most websites have a sitemap to help search engines find their pages, making this an efficient method for large sites.
• Follow links 2 levels deep: Follows links on the homepage, then follows links on those pages too. Available as a weekly check. Note: Only available on Enterprise and Ultimate plans.
• Follow links 3 levels deep: Follows links on the homepage, then follows links two more levels deep. Available as a weekly check. Note: Only available on Enterprise and Ultimate plans.
• Full website crawl: Crawls the entire website by following all internal links starting from the homepage, so new links are found even on deeply nested pages. Available as a weekly check. Note: Only available on Enterprise and Ultimate plans.

Turn on Capture once, then pause to have each new page checked once and then paused, which builds an archive without spending checks on later visits.

Apply Include and Exclude Filters

The template's Automatic tracking rules control which discovered pages become monitors, so you avoid tracking irrelevant pages.

• Include rules: Specify keywords or patterns that a page must match to be tracked. Useful for tracking specific types of content, such as /product/ pages only. Click Add Rule to match on a page's URL, title, text or an element.
• Exclude rules: Define keywords or patterns that should be skipped. Ideal for ignoring pages you do not care about, such as /tag/ or /archive/ URLs.
• Track All Pages: Track every discovered page without filtering.
• Always include top-level pages: Track pages like /pricing and /about whatever the other rules say.
• Tracked Page Limit: Cap how many pages the template holds, to keep usage under control. Pages you added to the template yourself count too.

How Discovered Pages Become Monitors

Once the template has rules (or Track All Pages), every discovery run adds each newly found matching page as a monitor with the template's settings and the Auto-discovered label (or the Labels set under the template's Organization settings). There is no review step, so leave the rules empty (or choose Let me review pages first when adding the website) if you would rather pick pages yourself.

• Pages added after your plan's active page allowance is used up are created paused. Once the total page limit (paused pages included) is reached, discovery switches itself off and you get a notification.
• Saving changed rules re-checks the pages found so far, while discovery is on, and adds the ones that now match.
• On the Discovered Pages page, Automatically monitor and Automatically track... create an include rule from a site or page, and Ignore creates an exclude rule. While the template has no title, text or element rules, a URL include rule added there turns the matching pages found so far into monitors at once, within the same limits and even with discovery off. A URL exclude rule added there pauses the template's monitors whose address matches it: the monitors made from the template and the ones its discovered pages link to.
• Without any rules, discovery only lists the pages it finds ("No pages will be tracked automatically"). Pick the pages you want on the Discovered Pages page to monitor them.

To remove pages you no longer want, use Pause to keep a monitor and its history, or Stop monitoring, which deletes the monitor and its check history.

Configure Tracked Elements

The template also defines what is tracked on each discovered page. You can monitor all pages, or only those with a specific structure (for example, only product pages).

1. To monitor whole pages, set the tracked element to Full-page Text.
2. To monitor pages with a specific layout, configure multiple tracked elements, such as product title, price, and description. If these elements do not exist on a discovered page, that page is simply skipped.

Save the template, and PageCrawl will keep discovering and monitoring matching pages automatically. If too many irrelevant pages are discovered, tighten the include/exclude filters and remove the pages you do not want.

What Happens When You Delete the Template?

Deleting the template keeps the monitors it created. Each one gets its own copy of the template's tracked elements and settings, keeps its history, and keeps being checked unless it is paused. The list of pages the template discovered is cleared, and this cannot be undone. A template that still has monitors disappears from your lists straight away and finishes deleting in the background.

Note: discovery can add many monitors on the same site, and they check on the same schedule. If the site starts refusing checks or rate limiting them, turn on Crawl Politeness so your checks to that site start one after another.