Over 30% of websites now use bot protection services like Cloudflare, Akamai, and similar tools that block automated access. This means your monitored pages can stop returning data without warning.
PageCrawl provides multiple layers of protection to keep your monitors working, most of which happen automatically.
What Happens Automatically
PageCrawl handles most bot protection automatically. When a check fails, PageCrawl detects the block and adjusts its approach on the next attempt, including automatic retries.
For most pages, you do not need to configure anything. The steps below are only needed if automatic handling does not resolve the issue.
How Do I Know If My Page Is Blocked?
PageCrawl will show a warning on the page if it detects a block. You may also notice that the captured content is empty, shows an error code (403, 401), or looks different from what you see when you visit the page yourself.
Troubleshooting Guide
Note: The settings below require Advanced mode. To enable it, click Edit on any page and toggle Advanced at the bottom of the form.
Follow these steps in order. After each step, wait for the check to complete before moving on.
Step 1: Enable Stealth Mode
This is the first thing to try and resolves most blocking issues.
- Open the blocked page in PageCrawl
- Click Edit
- Scroll down and enable Advanced mode
- Change Engine from "Default" to Stealth
- Click Save - a check will trigger automatically
- Wait for the check to complete and review the result
If the content now loads correctly, you are done. Stealth mode will be used for all future checks on this page.
Step 2: Change Proxy Location
If Stealth mode alone does not work, the site may be blocking the specific IP address or region.
- Open the page and click Edit
- Under Proxy Location, select Random
- Click Save - a check will trigger automatically
Random proxy rotation means each check comes from a different IP address, making IP-based blocking ineffective.
You can also try specific locations (London, New York, San Francisco, Toronto, Frankfurt) if you know the site serves content differently by region.
Step 3: Use Residential Proxies
For sites with the strictest protections, residential proxies are the most effective option. These route requests through real consumer internet connections, making them virtually indistinguishable from regular visitors.
Residential proxies are included on the Enterprise and Ultimate plans, at no extra cost.
- Open the page and click Edit
- Under Proxy Location, select Residential
- Select a country for the residential proxy
- Click Save - a check will trigger automatically
Step 3b: Use Web Unblocker
Web Unblocker is typically not needed. For the small number of sites that block even the included residential proxies, select Web Unblocker as the Location instead. It is our managed option: geo-targeted residential IPs, charged pay-as-you-go at $10/GB from a bandwidth balance you buy up front. You can purchase Web Unblocker traffic directly from your PageCrawl account.
If you would rather handle proxies yourself, skip this and go to Step 4. Buying and configuring custom proxies from any provider works on every plan and is usually cheaper if you are comfortable managing them.
Note: Web Unblocker consumes traffic from your purchased balance. Each check uses a small amount of traffic depending on the page size. The included residential proxies do not consume any balance.
Step 4: Use Your Own Proxies
If none of the built-in options work, bring your own proxies as a Proxy Pool.
- Go to Settings → Proxy Pools → Add New and paste your proxy servers (format:
user:password@host:port, one per line) - Open the page, click Edit, and set the Location to your pool
- Click Save and trigger a manual check
This is useful when you need a proxy from a specific country or provider, or when you already have a proxy subscription. See Custom Proxies for more details.
Quick Reference
| Solution | How to Enable | When to Use |
|---|---|---|
| Stealth mode | Edit > Advanced > Engine: Stealth | First thing to try for any blocked page |
| Proxy rotation | Edit > Proxy: Random | When a specific IP is blocked |
| Residential proxy (included on Enterprise or Ultimate) | Edit > Proxy: Residential | For the strictest access controls |
| Web Unblocker ($10/GB, Enterprise or Ultimate) | Edit > Proxy: Web Unblocker | For the sites that block even residential proxies |
| Your own proxies | Settings > Proxy Pools, then set the page Location to your pool | When you need a specific provider or location |
Still Blocked?
If you have tried all the steps above and the page is still not loading:
- Double-check the URL - Make sure the URL is correct and the page is publicly accessible. Try opening it in a private/incognito browser window to confirm.
- Purchase Web Unblocker traffic directly from PageCrawl if you have not already. This is the most effective solution when the included residential proxies are not enough, and it requires an Enterprise or Ultimate plan.
- Try a custom proxy from a third-party provider if you need a specific geographic location or a different proxy type.
- Contact support - Email support@pagecrawl.io with the page URL and a description of what you see. We can review the specific page and suggest the best configuration.
Related Articles
- Real Browser Mode - Engine selection including Stealth mode
- Custom Proxies - Configure proxy servers
- Residential Proxies - Purchase residential proxy traffic
- Page Loading Issues - Other common loading problems