Pagecrawl.io is designed to make website monitoring and management seamless. The "Discover New Pages" feature takes your website monitoring to the next level by automatically identifying new links, tracking changes, and ensuring your online presence remains up-to-date. In this guide, we'll delve into the capabilities of this feature, including its scanning methods, automated monitoring, and filtering options.

Features of the "Discover New Pages" Functionality

Automated Link Discovery

This feature performs automated scans of your website, identifying new links that have been added. This proactive approach keeps you informed about any changes to your website's link structure and updates.

Choice of Scanning Methods

Pagecrawl.io provides multiple scanning methods to suit your needs. All available discovery options are enabled by default (Mode: Automatic):

• Base URL Link Discovery: Discover new links directly on your provided base URL. This method is particularly useful if you want to focus on specific sections of your website without going too deep.
• Deep Scan: Conduct a comprehensive analysis by visiting every accessible page on your website. This ensures that no new links go unnoticed, even on nested pages.
• Sitemap Utilization: Utilize existing sitemaps to uncover new links. This method is efficient for websites with extensive content structures. Since most websites want to be featured on search engines like Google and Bing, they usually have sitemaps.

Filtering Options

• Include Pages: Specify keywords or patterns that pages must contain to be included in monitoring. Useful for tracking specific types of content.
• Exclude Pages: Define keywords or patterns that pages must not contain to be included in monitoring. Ideal for excluding pages that you are not interested in.

Configuring Automated Monitoring and Tracking:

Create a Template

To start monitoring the website and automatically discover all new pages, configure a new Template which will serve as the basis for monitoring new pages.

1. Under "Sample URL address," enter an example page URL that you wish to track. The rest of the fields will be auto-filled for you.

Configure Tracked Elements

You may choose to monitor all pages on the website or only those with a specific structure (e.g., if you only want to track product pages and not other pages).

1. If you wish to monitor all pages, for Tracked Element configuration, select "Full-page Text."
2. To monitor pages with a specific layout, configure multiple Tracked Element configurations, such as product title, price, and description. If these elements do not exist on the page, the page will simply be skipped.

Enable "Discover New Pages" feature

1. Activate the "Discover New Pages" feature and customize any settings if needed.
2. Save the template and watch out for newly added pages when they become discovered
3. If there are too many irrelevant pages discovered, adjust filters and remove irrelevant pages.