PageCrawl provides built-in proxy locations and supports custom proxy servers for pages that require specific geographic access or have IP-based restrictions.
Built-in Proxy Locations
Select a proxy location per page or apply one to multiple pages via Bulk Edit:
| Location | Description |
|---|---|
| London | UK-based proxy |
| New York | US East Coast |
| San Francisco | US West Coast |
| Toronto | Canada |
| Frankfurt | Germany |
| Tel Aviv | Israel |
| Random | Randomly selected from the pool |
| Residential | Residential IP address (less likely to be blocked) |
Custom Proxy Setup
Use your own proxy servers when the built-in locations do not work for your use case.
Supported formats:
host:port
username:password@host:portConfiguration options:
| Method | How |
|---|---|
| Single page | Edit the page > Power User settings > Custom Proxy |
| Multiple pages | Select pages > Bulk Edit > Custom Proxies |
| Template | Add proxy settings to a template for reuse |
You can paste multiple proxy servers (one per line). PageCrawl will randomly select one for each check. If a proxy fails, the system retries with a different proxy from the list, up to three attempts.
Automatic Engine Switching
When a page is blocked (Cloudflare timeout, 403, or 401), PageCrawl automatically switches to Stealth mode in addition to the proxy configuration. This combination of proxy and stealth mode resolves most access issues.
Residential Proxy Quote
For pages that block datacenter IPs, contact support@pagecrawl.io with the URLs you want to monitor. We can review the pages and provide a quote for residential proxy services.
Choosing a Proxy Provider
Most pages work fine without any proxy configuration. You only need a custom proxy if a website is actively blocking bots or restricting access by geographic location. Start without a proxy, and only set one up if you are seeing access errors (403, Cloudflare blocks, empty pages).
If the built-in proxy locations are not enough for your needs, you can use a third-party proxy provider. Here is what to look for and some popular options.
Understanding bandwidth usage:
Each page check downloads the full page without caching, so bandwidth adds up quickly. An average web page uses 2-3 MB per check. Heavier pages (news sites, e-commerce, image-heavy pages) can use 5-10 MB or more. For example, monitoring 50 pages every 30 minutes at 3 MB each would use roughly 7 GB per day, or around 216 GB per month. Because of this, avoid proxy providers that charge per GB of traffic. Those plans are designed for one-off scraping, not ongoing monitoring.
What to look for:
- Unlimited bandwidth - This is the most important factor. Look for plans priced per proxy/port or as a flat monthly rate, not per GB.
- Username/password authentication - PageCrawl connects to proxies from multiple processing servers, so IP-based allowlists will not work. Choose a provider that supports
username:password@host:portauthentication. - Rotating IPs - Providers that rotate IPs automatically reduce the chance of being blocked over time.
- Geographic coverage - Pick a provider with servers in the regions your monitored pages target.
- HTTP/HTTPS support - PageCrawl requires standard HTTP proxies. SOCKS proxies are not supported.
Datacenter vs. residential proxies:
Datacenter proxies with unlimited bandwidth are the most cost-effective option for monitoring. They work well for most websites. Residential proxies (using real ISP addresses) are only needed for sites with strict bot detection that blocks datacenter IPs. If you need residential proxies, look for providers that offer them with unlimited bandwidth or per-IP pricing rather than per-GB billing.
Popular proxy providers that work with PageCrawl:
| Provider | Type | Pricing Model | Notes |
|---|---|---|---|
| Webshare | Datacenter, Residential | Per proxy, unlimited bandwidth | Free tier available, good for testing. Paid datacenter plans include unlimited bandwidth. |
| IPRoyal | Datacenter, Static residential | Per proxy (datacenter) | Datacenter proxies with unlimited traffic. Static residential proxies available per IP. |
| Proxy-Cheap | Datacenter, Static residential | Per proxy, unlimited bandwidth | Budget-friendly static residential and datacenter proxies with no traffic limits. |
| ProxyRack | Datacenter, Residential | Flat monthly rate | Unlimited bandwidth on most plans. Rotating and geo-targeted options. |
These are independent providers and not affiliated with PageCrawl. Prices and features may change.
Not every provider works for every website. A proxy that works perfectly for one site may get blocked on another. This depends on the website's bot detection, the proxy provider's IP reputation, and the type of proxies used. Always test a provider against your specific pages before committing to a long-term plan. Most providers offer short trial periods or small starter plans for this purpose.
Country-specific access: Some websites restrict content to visitors from a specific country (geo-blocking). Government portals, local news sites, and region-locked services often require an IP address from that country to load correctly. If you are monitoring pages like these, make sure the proxy provider offers proxies in the required country. Check the provider's location list before purchasing, as coverage varies significantly between providers, especially for smaller countries.
Note: Most providers give you a proxy endpoint in the username:password@host:port format. Paste it directly into the Custom Proxy field in PageCrawl. If the provider offers rotating proxies through a single gateway endpoint, you only need to add one line.
Avoiding Free Proxies
Free proxy servers are unreliable, slow, and frequently stop working. They should not be used for monitoring pages where uptime matters. Use the built-in proxy locations, your own paid proxy service, or contact us for residential proxy options.
Related Articles
- Real Browser Mode - Engine selection including Stealth mode
- Monitoring Pages Behind Cloudflare - Handling anti-bot protections
- Bulk Edit - Apply proxy settings to multiple pages at once