Over 30% of websites now use bot protection services like Cloudflare, Akamai, and similar tools that block automated access. This means your monitored pages can stop returning data without warning.

PageCrawl provides multiple layers of protection to keep your monitors working, most of which happen automatically.

What Happens Automatically

PageCrawl handles most bot protection automatically. When a check fails, PageCrawl detects the block and adjusts its approach on the next attempt. This includes automatic retries, switching to stealth mode, and rotating through different proxy locations.

For most pages, you do not need to configure anything. The steps below are only needed if automatic handling does not resolve the issue.

How Do I Know If My Page Is Blocked?

PageCrawl will show a warning on the page if it detects a block. You may also notice that the captured content is empty, shows an error code (403, 401), or looks different from what you see when you visit the page yourself.

Troubleshooting Guide

Note: The settings below require Advanced mode. To enable it, click Edit on any page and toggle Advanced at the bottom of the form.

Follow these steps in order. After each step, wait for the check to complete before moving on.

Step 1: Enable Stealth Mode

This is the first thing to try and resolves most blocking issues.

1. Open the blocked page in PageCrawl
2. Click Edit
3. Scroll down and enable Advanced mode
4. Change Engine from "Default" to Stealth
5. Click Save - a check will trigger automatically
6. Wait for the check to complete and review the result

If the content now loads correctly, you are done. Stealth mode will be used for all future checks on this page.

Step 2: Change Proxy Location

If Stealth mode alone does not work, the site may be blocking the specific IP address or region.

1. Open the page and click Edit
2. Under Proxy Location, select Random
3. Click Save - a check will trigger automatically

Random proxy rotation means each check comes from a different IP address, making IP-based blocking ineffective.

You can also try specific locations (London, New York, San Francisco, Toronto, Frankfurt) if you know the site serves content differently by region.

Step 3: Use Residential Proxies

For sites with the strictest protections, residential proxies are the most effective option. These route requests through real consumer internet connections, making them virtually indistinguishable from regular visitors.

1. Open the page and click Edit
2. Under Proxy Location, select Residential
3. Select a country for the residential proxy
4. Click Save - a check will trigger automatically

Residential proxy traffic is available as an add-on. You can purchase residential proxy traffic directly from your PageCrawl account.

Note: Residential proxies consume traffic from your purchased balance. Each check uses a small amount of traffic depending on the page size.

Step 4: Use a Custom Proxy

If none of the built-in options work, you can use your own proxy server from a third-party provider.

1. Open the page and click Edit
2. Enable Advanced mode
3. Enter your proxy details in the Custom Proxy field (format: http://user:password@host:port)
4. Click Save and trigger a manual check

This is useful when you need a proxy from a specific country or provider, or when you already have a proxy subscription. See Custom Proxies for more details.

Quick Reference

Solution	How to Enable	When to Use
Stealth mode	Edit > Advanced > Engine: Stealth	First thing to try for any blocked page
Proxy rotation	Edit > Proxy: Random	When a specific IP is blocked
Residential proxy	Edit > Proxy: Residential	For the strictest access controls
Custom proxy	Edit > Advanced > Custom Proxy	When you need a specific provider or location

Still Blocked?

If you have tried all the steps above and the page is still not loading:

• Double-check the URL - Make sure the URL is correct and the page is publicly accessible. Try opening it in a private/incognito browser window to confirm.
• Purchase residential proxy traffic directly from PageCrawl if you have not already. This is the most effective solution for heavily protected sites.
• Try a custom proxy from a third-party provider if you need a specific geographic location or a different proxy type.
• Contact support - Email support@pagecrawl.io with the page URL and a description of what you see. We can review the specific page and suggest the best configuration.

Related Articles

• Real Browser Mode - Engine selection including Stealth mode
• Custom Proxies - Configure proxy servers
• Residential Proxies - Purchase residential proxy traffic
• Page Loading Issues - Other common loading problems