Most blocked monitors can be unblocked by changing one thing: the connection the check arrives over. Neither the URL nor the way you set the monitor up is usually the problem. The site is refusing the route, not the request, and PageCrawl has several routes built in. Work down the list below, from the free options to the paid ones, and stop at the first one that returns the real page.
Why do some sites refuse our checks?
Some websites only serve visitors whose connection looks like an ordinary home or office one, and they turn away anything else before the page loads. Your monitor then records an access error such as 403 Forbidden instead of the content. Nothing about the URL, the tracked elements or the schedule causes this, so changing those settings will not change the result. What changes it is where the check arrives from.
What is a "connection" in PageCrawl?
A connection is the route a check takes to reach the website, set per page in the Location field. The site sees that route rather than PageCrawl. Options fall into four groups, cheapest first:
- Ours (Auto and the built-in locations). Included on every plan. Auto moves between our connections and retries elsewhere when one is refused.
- Yours, through a computer you own (PageCrawl Relay). Checks leave from your own internet connection. No PageCrawl fee on any plan, including Free.
- Residential, managed by us. Included on Enterprise and Ultimate at no extra cost.
- Yours, from a proxy provider (custom proxy pools). Proxies you buy elsewhere, grouped into a named pool and reused across pages.
Above all of them sits Web Unblocker, the premium option on Enterprise and Ultimate. It is the most capable route and the only one billed by the traffic each check uses, so it belongs at the end of the list rather than the start.
How do I unblock the page?
Work down this list and stop at the first one that returns the real page. Change one thing at a time, save, and look at the captured content rather than only at the status.
| Order | Try | Why it is here | Cost |
|---|---|---|---|
| 1 | Location: Auto | A page pinned to one location keeps retrying the same refused route. Auto moves between connections. | Included on every plan |
| 2 | Stealth mode | Opens the page more like a person would, which is enough for many protected sites. | Included; uses more credits per check |
| 3 | PageCrawl Relay | If the page opens normally in your own browser, your connection is one the site accepts. | No PageCrawl fee on any plan |
| 4 | Residential connections | The same kind of home internet an ordinary visitor uses, so it gets through on most sites of this kind. | Included on Enterprise and Ultimate |
| 5 | Custom proxy pools | Full control over which addresses are used, for teams who already pay a proxy provider. | Whatever your provider charges |
| 6 | Web Unblocker | The last resort for a page nothing above reaches. The most capable option and by far the most expensive. | Metered bandwidth you buy, on Enterprise and Ultimate |
Note: Web Unblocker sits at the bottom of the list on purpose. It is the premium option and the only one billed by the traffic it uses, so every single check spends part of a balance you paid for. That is worth it for a page you check a few times a day and cannot reach any other way, and it is a poor fit for a page checked every few minutes, where the same monthly spend buys far less coverage. Try everything above it first, and if you land on it, consider slowing that page's schedule at the same time.
Does any one connection work on every site?
No. None of the options above is a guarantee, including the paid ones. "Blocked" is not one problem either: a page can refuse the check because it only serves visitors in one country, because it does not accept the route the check arrived on, because it wants a sign-in, or because it shows a challenge instead of content. Only the first two are about the connection at all.
Sites also disagree with each other. A route one site accepts is refused by the next, and a site can accept a route for months and then stop. That is why PageCrawl offers a few genuinely different routes rather than one "best" one: when a page is refused, the useful move is to arrive from somewhere else, not to try harder from the same place. Work down the list in order, and contact us before buying traffic if you are unsure which step is worth trying for a particular page.
Why not use the strongest option on every page?
Because the strongest option is not the best result for most pages, and on some pages it is the wrong one.
- Where the check comes from changes what the page shows. Prices, currency, language, stock and shipping all vary by country. A connection in the wrong country returns a real page with the wrong content, so the monitor looks healthy while it tracks numbers you did not mean to track. Matching the country you care about matters more than picking the heaviest option.
- The heavier options cost money, and the top one costs the most. Web Unblocker bills for the traffic every check uses, and custom proxy pools cost whatever your provider charges. Putting a page that loads fine on Auto onto either buys nothing, and putting a frequently checked page onto Web Unblocker spends a balance quickly for no benefit.
- Relay depends on your computer. It is free, but the checks only run while that machine is online, which is a fair trade for a handful of pages and a poor default for all of them.
- Included does not mean unlimited. Residential connections come with Enterprise and Ultimate, and they are the right tool for a page that needs them, but routing everything through them uses more of your plan's resources for no gain on pages that were never blocked.
- Most pages need none of this. The large majority load correctly on Auto with the default engine, and PageCrawl already switches to Stealth by itself in several situations, including when a page comes back blocked. Escalating one blocked page at a time keeps checks simple, keeps costs down, and leaves you a next thing to try when a site changes its mind.
A reasonable rule: leave every monitor on Auto, pick a country only when the content depends on one, and move a page up the list only when it is actually refused.
How do I change the connection for one page?
Open the monitor and click Edit. In the simple editor, expand Page adjustments to find Location and Engine; in the advanced editor, Location is always visible and Engine appears once Power User Settings is on. Pick the connection, click Save Changes, then wait for the next check or use Run now where available.
When a monitor is blocked, PageCrawl also shows the options inline. The warning on the monitor page and the red status tag in the Tracked Pages list both offer one-click switches for the steps that apply to your plan, so you usually do not have to find these fields yourself. To apply a connection to many pages at once, use Bulk Edit, or set the Location on a template so pages created from it inherit the choice.
When is PageCrawl Relay the right choice?
Relay suits you when the page opens normally in your own browser and you can keep a computer online: a home server, a spare laptop or a Raspberry Pi. Checks then leave from your connection instead of ours, which is exactly what a site refusing our usual routes is asking for. It has no PageCrawl fee on any plan, and your normal monitoring limits still apply.
Relay uses your own bandwidth, works with public pages only, and does not reuse your browser's signed-in session, so pages that require a login still need one configured in PageCrawl. It also cannot reach private intranet or local-network pages. See PageCrawl Relay for setup.
When do residential connections help?
Residential connections are the same kind of home internet an ordinary visitor uses, which is why they get through on most sites that refuse other routes. They are included on Enterprise and Ultimate at no extra cost, and they are the better fit for business monitoring because they do not depend on a computer of yours staying online.
Web Unblocker is the premium step above them, and the most expensive thing you can put a monitor on. It spends a bandwidth balance you buy rather than anything included in the plan, and it spends it on every check, so the bill follows the schedule: a page checked twice a day costs a fraction of the same page checked every few minutes. Reach for it when a page matters, nothing else gets through, and you can live with checking it a few times a day. Larger pages use more traffic per check. See Residential Proxies and Web Unblocker.
Do I need custom proxies?
Only if you already pay a proxy provider, or you need control over exactly which addresses your checks use. A proxy is someone else's internet connection that your checks travel through; you buy proxies from a provider, and PageCrawl does not sell them. Group them into a named pool once under Settings → More → Proxies, then pick that pool as a page's Location.
Authenticate with a username and password rather than IP whitelisting: checks run from a large, changing set of servers, so an allowlist of our addresses will not hold. See Using Custom Proxies.
Is the page blocked, or is something else wrong?
Open the exact URL in a private browser window and compare it with the monitor's last result. A 401 or 403 does not always mean the site refused an automated visit; it can mean the page needs a sign-in or that the account lacks access, and no connection can supply a login.
- Needs a sign-in: Monitoring Password-Protected Websites.
- Shows a challenge instead of the page: Monitoring Pages That Show CAPTCHA.
- Times out, 404s or returns a 500-series error: Page Loading Issues, which covers causes that a different connection will not fix.
- Refused across several connections: Monitoring Pages Behind Bot Protection.
What if the page still does not load?
Send us the URL and we will look at it. Some pages need a specific combination, such as a residential connection in one country plus Stealth mode, and some are not reachable for automated monitoring at all. We would rather tell you that than have you buy bandwidth or a proxy subscription that does not help, so contact us before spending money on a page you are unsure about.