PageCrawl is designed to make website change monitoring and management seamless. The "Discover New Pages" feature takes your change monitoring to the next level by automatically identifying new links, tracking changes, and ensuring your online presence remains up-to-date. In this guide, we'll delve into the capabilities of this feature, including its scanning methods, automated monitoring, and filtering options.
Automated Link Discovery
This feature performs automated scans of your website, identifying new links that have been added. This proactive approach keeps you informed about any changes to your website's link structure and updates.
Choice of Scanning Methods
PageCrawl provides multiple scanning methods to suit your needs. The default mode is Automatic (recommended), which combines methods to find new pages using the best approach for each website:
- Automatic (recommended): Combines sitemap and link discovery to find pages using the best method for the website. This is the default and recommended setting.
- Homepage Links Only: Discover new links by following links on the homepage. Available as a daily or weekly check. Useful if you want to focus on pages directly linked from the main page.
- Sitemap Only: Discover pages listed in the website's sitemap. Most websites have a sitemap to help search engines find their pages, making this an efficient method for large sites.
- Follow Links 2 Levels Deep: Follows links on the homepage, then follows links on those pages too. Available as a weekly check. Note: Only available on Enterprise and Ultimate plans.
- Follow Links 3 Levels Deep: Follows links on the homepage, then follows links two more levels deep. Available as a weekly check. Note: Only available on Enterprise and Ultimate plans.
- Deep Scan: Conduct a comprehensive analysis by visiting every accessible page on your website. This ensures that no new links go unnoticed, even on deeply nested pages. Note: Only available on Enterprise and Ultimate plans.
Filtering Options
- Include Pages: Specify keywords or patterns that pages must contain to be included in monitoring. Useful for tracking specific types of content.
- Exclude Pages: Define keywords or patterns that pages must not contain to be included in monitoring. Ideal for excluding pages that you are not interested in.
Configuring Automated Monitoring and Tracking
Create a Template
To start monitoring the website and automatically discover all new pages, configure a new Template which will serve as the basis for monitoring new pages.
- Under "Sample URL address," enter an example page URL that you wish to track. The rest of the fields will be auto-filled for you.
Configure Tracked Elements
You may choose to monitor all pages on the website or only those with a specific structure (e.g., if you only want to track product pages and not other pages).
- If you wish to monitor all pages, for Tracked Element configuration, select "Full-page Text."
- To monitor pages with a specific layout, configure multiple Tracked Element configurations, such as product title, price, and description. If these elements do not exist on the page, the page will simply be skipped.
Enable "Discover New Pages" feature
- Activate the "Discover New Pages" feature and customize any settings if needed.
- Save the template and watch out for newly added pages when they become discovered
- If there are too many irrelevant pages discovered, adjust filters and remove irrelevant pages.