Some pages load perfectly in your browser and refuse PageCrawl. A 403, a challenge page, or a stripped-down version with none of the content you wanted to track. Today we are releasing PageCrawl Relay, which fixes that class of page by running the check through a computer you already own, so the site sees an ordinary connection rather than ours. It is free on every plan, including Free.
What is PageCrawl Relay?
PageCrawl Relay is a small program you run on a computer you own. When a monitor is set to use it, PageCrawl still opens and reads the page as usual, but the network traffic leaves from your machine instead of ours. The checking, the change detection and the alerts are unchanged; only the last mile moves.
It is not a proxy server. It opens no port, accepts no incoming connection, and carries only your own monitors.
Why not just run a proxy server at home?
Because a proxy server at home has to accept incoming connections, which means opening a port on your router and publishing your home address. That is the part people underestimate. Open proxies get found by internet-wide scanners within hours, and once found they are used by strangers, with the traffic looking like it came from you.
A relay inverts the direction. Your computer makes one outgoing connection to PageCrawl and holds it open, the same way a chat app or a smart plug does. That single change removes most of the work and all of the exposure:
| Proxy server at home | PageCrawl Relay | |
|---|---|---|
| Router setup | Port forward, and a firewall rule | None |
| Visible from the internet | Yes, and scanners will find it | No, it only dials out |
| Changing home IP | Breaks it, needs dynamic DNS | Does not matter |
| Credentials | A proxy password to set and rotate | A token you paste once |
| If someone else finds it | They can use your connection | There is nothing to find |
| When the machine sleeps | Checks fail | Checks carry on from PageCrawl |
| Reach into your own network | Whatever the proxy allows | Refused, by two independent checks |
That last row is the one worth dwelling on, because it is the risk people rarely think about when they set up a proxy at home. A general-purpose proxy on your LAN will happily connect to your router's admin page, your NAS, or a printer, because those are just addresses to it. The relay refuses them.
How does it keep my own network out of it?
Your machine never chooses where to connect. PageCrawl asks it to reach one site for one of your monitors, and before connecting, your machine checks that the address is a real public one. Anything on your own network is refused outright.
That list covers your router and any device on it, the private ranges described in
RFC 1918, carrier-grade NAT ranges,
the
link-local addresses
that cloud servers use for their own configuration, and names ending in .local or
.lan. The check happens after the address is looked up and before the connection is
made, so a web address that secretly points at your router is refused as well.
The relay also cannot read the pages it carries. Encryption runs between PageCrawl and the site, so your machine moves sealed traffic it cannot open, and so can anyone watching your network.
The program is open source, so none of this has to be taken on trust. You can read exactly what it does, build it yourself, and check that the binary you downloaded matches.
When is a relay actually useful?
Three cases, in the order we see them:
- A site that refuses our servers. Retail, ticketing and travel sites often serve your browser normally while turning away anything that looks like a datacenter. If a page loads fine for you but returns a 403 or a challenge in PageCrawl, that is the signal.
- A page only your network can reach. An intranet, a staging site, or a supplier portal that allows your office address and nothing else. No proxy you can buy will ever reach these, because the allow-list is the point.
- Content that varies by country in a way our locations do not cover, when you happen to be in the country in question.
When do you not need one?
Most of the time, and we would rather say so than have people set up a machine for nothing.
- The page already works. If your monitor is green, relaying it changes nothing except that your connection now pays for the traffic.
- You are on Enterprise or Ultimate. The Location dropdown already includes the residential option and per-country choices, which cover most of what a relay does and need no hardware.
- You need several countries. A relay gives you exactly one location: the one you are actually in. Checking a page from three countries is a job for the built-in locations.
- You have no machine that stays on. A laptop relays only while it is awake. If the page genuinely needs your connection, it needs a computer that is genuinely on.
Why not just buy residential proxy bandwidth?
Sometimes you should, and PageCrawl sells it: metered residential bandwidth at $10 per gigabyte on Ultimate and Enterprise. The deciding factor is usually whether you need somebody else's address or simply a normal one.
Buying bandwidth is the better answer when you need pages checked from several countries, when the page serves a market you are not in, when you would rather your own address stayed out of it, or when you do not want to keep a machine running.
A relay is the better answer when the page simply wants to see an ordinary connection and you already have a computer that stays on, for two reasons:
- It is metered, and pages are heavy. A standard page is around 2 MB once images and scripts are counted, so one monitor checking hourly uses about 1.4 GB a month, or roughly $14 for that one page. Twenty of them is a bill worth thinking about. The cost calculator prices your own pages.
- A bought address is a shared address. Residential pools are rented to many customers at once, so a given address has carried a great deal of other people's traffic before it reaches you, and some sites have already formed an opinion about it. Your own connection is one a site has no particular reason to distrust.
What does it cost to run?
Almost nothing in hardware terms. About 15 MB of memory and no measurable processor time, because the browser work stays on PageCrawl's side and your computer only carries the traffic. A Raspberry Pi, a NAS, a Home Assistant box or an old laptop left plugged in are all more than enough.
Bandwidth is the part worth sizing. Every relayed page is downloaded from the site by your machine and then sent on to PageCrawl, so it uses both your download and your upload allowance, roughly the page's weight in each direction. At 2 MB a page, one monitor checking hourly is about 3 GB a month; five are about 14 GB. You can set a monthly limit per machine, and once it is reached, checks go back to PageCrawl's own locations on their own.
How do I set it up?
- Go to Settings → Relays and choose Add machine. Copy the token it shows, which appears once.
- Download and open the program on the computer you want to use. A settings page opens in your browser. There is no command line involved.
- Paste the token and press Connect.
- On any monitor, set Location to that machine, or to Any of my machines if you run more than one.
For a computer that stays on, there are Docker, systemd and Home Assistant packages. A Home Assistant box is often the best machine for this, because it is already running all the time.
What happens when the machine is off?
By default the check runs from PageCrawl's own locations instead, so nothing stops being monitored, and your machine is used again the moment it returns. If a result from a different address would be wrong rather than merely different, you can tell PageCrawl to wait instead, or to flag the monitor so you hear about it. That choice is made once for your whole account, under Settings → Relays.
There is no interval to choose, because waiting ends on the event rather than on a clock: the moment one of your machines is usable again, its postponed checks are due.
Full details are in the PageCrawl Relay guide.
