There may be various reasons why a page fails to open. This guide describes the most common problems and suggests solutions to help you overcome these issues.

Timeout

A timeout occurs when the page takes too long to respond. This may be a temporary issue with the page, or the page may be loading very slowly. Timeout limits vary depending on your plan:

• Free plan: 45 seconds
• Standard plan: 90 seconds
• Enterprise plan: 180 seconds

To avoid timeouts please consider subscribing to a paid plan or upgrading your plan.

Selector not found

This error will be shown if the page has changed significantly and element with configured XPath/CSS selector could not be found. In this case, you should review the page and update selector if needed.

Page blocked

Some pages may use site protection features to block scrapers and website tracking tools like PageCrawl.io. Different pages may use different blocking mechanisms, but here are the most common ones:

• Access Restricted to Specific Countries Page may be configured to only allow visitors from a specific country.

  • Solution: Specify a proxy location from a country that is not blocked. If you cannot find an available proxy, consider purchasing a proxy service for a specific country and configuring custom proxy in PageCrawl.io.

• Proxy Location blocked The website may block the IP address of the proxy server PageCrawl.io is using.

  • Solution: Use "Residential proxy pool" to avoid being blocked. You will need to purchase a proxy service for a specific country and configuring custom proxy in PageCrawl.io.

401 or 403 Error

Most often indicates that PageCrawl.io Bot was not allowed to access the website. Use "Residential proxy pool" to avoid being blocked.

404 Page Not Found

In most cases this error indicates that page is no longer available to view. You should check and update the page URL.

500 Series error

500, 502, 503, 504 indicates that website server is not responsive, overloaded, currently in maintenance or experiencing server issues. If such error occurs, our bots will retry page check later.

Page Unreachable

The page can't be opened. In most cases website is down or the website in only reachable from a specific country

Site Protected with CAPTCHA

Pages may use CAPTCHA to protect the website from bots. To bypass this, you can use a service like 2Captcha which will use human workers to solve the captcha for you. PageCrawl.io has an integration with 2Captcha (you must be subscribed to Enterprise plan) you can sign up for and configure the API token generated from 2Captcha.

Unknown Error

In some cases there could be an unexpected error that causes pagecrawl.io bot to fail to check the page for changes. In case this error does not go away after a while, please contact support to notify us about the problem so we could prioritize the issue.