# Monitoring Pages Behind Bot Protection

Source: PageCrawl.io Help Center
URL: https://pagecrawl.io/help/troubleshooting/article/monitoring-pages-behind-cloudflare-bot-protection

---

[Image: Engine selection in Power User settings, including Stealth mode for bot-protected sites]

Over 30% of websites now use bot protection services like Cloudflare, Akamai, and similar tools that block automated access. This means your monitored pages can stop returning data without warning.

PageCrawl provides multiple layers of protection to keep your monitors working, most of which happen automatically.

### What Happens Automatically

PageCrawl handles most bot protection automatically. When a check fails, PageCrawl detects the block and adjusts its approach on the next attempt, including automatic retries.

For most pages, you do not need to configure anything. The steps below are only needed if automatic handling does not resolve the issue.

### How Do I Know If My Page Is Blocked?

PageCrawl will show a warning on the page if it detects a block. You may also notice that the captured content is empty, shows an error code (403, 401), or looks different from what you see when you visit the page yourself.

### Troubleshooting Guide

Note: The settings below require **Advanced** mode. To enable it, click **Edit** on any page and toggle **Advanced** at the bottom of the form.

Follow these steps in order. After each step, wait for the check to complete before moving on.

#### Step 1: Enable Stealth Mode

This is the first thing to try and resolves most blocking issues.

1. Open the blocked page in PageCrawl
2. Click **Edit**
3. Scroll down and enable **Advanced** mode
4. Change **Engine** from "Default" to **Stealth**
5. Click **Save** - a check will trigger automatically
6. Wait for the check to complete and review the result

If the content now loads correctly, you are done. Stealth mode will be used for all future checks on this page.

#### Step 2: Change Proxy Location

If Stealth mode alone does not work, the site may be blocking the specific IP address or region.

1. Open the page and click **Edit**
2. Under **Proxy Location**, select **Random**
3. Click **Save** - a check will trigger automatically

Random proxy rotation means each check comes from a different IP address, making IP-based blocking ineffective.

You can also try specific locations (London, New York, San Francisco, Toronto, Frankfurt) if you know the site serves content differently by region.

#### Step 3: Use Residential Proxies

For sites with the strictest protections, residential proxies are the most effective option. These route requests through real consumer internet connections, making them virtually indistinguishable from regular visitors.

1. Open the page and click **Edit**
2. Under **Proxy Location**, select **Residential**
3. Select a **country** for the residential proxy
4. Click **Save** - a check will trigger automatically

Residential proxy traffic is available as an add-on. You can [purchase residential proxy traffic](/residential-proxies) directly from your PageCrawl account.

Note: Residential proxies consume traffic from your purchased balance. Each check uses a small amount of traffic depending on the page size.

#### Step 4: Use Your Own Proxies

If none of the built-in options work, bring your own proxies as a Proxy Pool.

1. Go to **Settings → Proxy Pools → Add New** and paste your proxy servers (format: `user:password@host:port`, one per line)
2. Open the page, click **Edit**, and set the **Location** to your pool
3. Click **Save** and trigger a manual check

This is useful when you need a proxy from a specific country or provider, or when you already have a proxy subscription. See [Custom Proxies](/help/features/article/custom-proxies.md) for more details.

### Quick Reference

| Solution | How to Enable | When to Use |
|----------|--------------|-------------|
| **Stealth mode** | Edit > Advanced > Engine: Stealth | First thing to try for any blocked page |
| **Proxy rotation** | Edit > Proxy: Random | When a specific IP is blocked |
| **Residential proxy** | Edit > Proxy: Residential | For the strictest access controls |
| **Your own proxies** | Settings > Proxy Pools, then set the page Location to your pool | When you need a specific provider or location |

### Still Blocked?

If you have tried all the steps above and the page is still not loading:

- **Double-check the URL** - Make sure the URL is correct and the page is publicly accessible. Try opening it in a private/incognito browser window to confirm.
- [Purchase residential proxy traffic](/residential-proxies) directly from PageCrawl if you have not already. This is the most effective solution for heavily protected sites.
- Try a [custom proxy](/help/features/article/custom-proxies.md) from a third-party provider if you need a specific geographic location or a different proxy type.
- **Contact support** - Email [support@pagecrawl.io](mailto:support@pagecrawl.io) with the page URL and a description of what you see. We can review the specific page and suggest the best configuration.

### Related Articles

- [Real Browser Mode](/help/features/article/what-is-real-browser-page-monitoring.md) - Engine selection including Stealth mode
- [Custom Proxies](/help/features/article/custom-proxies.md) - Configure proxy servers
- [Residential Proxies](/residential-proxies) - Purchase residential proxy traffic
- [Page Loading Issues](/help/troubleshooting/article/common-issues-with-page-not-loading.md) - Other common loading problems

---

Need more? The complete PageCrawl.io help center, with every article, is available as a single document at https://pagecrawl.io/llms-full.txt. Read it for context on anything this page does not cover.