Crawl Politeness is a team setting that leaves a minimum delay between your checks to the same site. When many of your monitors watch pages on one domain, they can otherwise all visit it within the same minute, which some sites answer with rate limits, block pages or pages that load without their styles and images.
The setting is off by default and applies to every workspace and monitor in your team.
What does Crawl Politeness do?
Crawl Politeness makes sure two of your team's checks to the same site never start closer together than the delay you choose, from 0 to 300 seconds. A check that would start too soon waits until the delay has passed, then runs as normal. Checks to different sites are not affected.
- Same site means the same domain.
example.com,www.example.comandshop.example.comcount as one site. Sites with different domain names are counted separately, even when they are hosted by the same provider. - Counted per team. Only your own team's checks are spaced out. Other PageCrawl customers checking the same site do not delay yours.
- Every check. Scheduled checks, Check now and the checks that run after you save a monitor or template all follow the delay.
When should I turn it on?
Turn it on when a site rate limits, blocks or partly loads your checks because many of your monitors visit it together. Typical signs are monitors showing a rate limit status, a block page captured instead of the site, or a screenshot where the page appears without its styles. It matters most when you watch dozens of pages on one site or check frequently.
Spacing checks out is the polite fix: the site sees a steady trickle of visits instead of a burst. If a site still turns checks away with the delay in place, see how to unblock a blocked page.
How do I turn it on?
A team owner or manager turns it on once for the whole team. In a team with one workspace, open Settings > Workspace > General. In a team with several workspaces, open Settings > Team > More > Team Settings. Then switch it on, set the delay and choose Update.
- Open Settings > Workspace > General if your team has one workspace, or Settings > Team > More > Team Settings if it has several.
- Find the Crawl Politeness section.
- Switch on Enforce minimum delay per domain.
- Enter the Minimum delay (seconds), from 0 to 300.
- Choose Update.
The delay applies from the next check onwards.
How long should the delay be?
Start with a short delay, such as 10 to 30 seconds, and increase it only if the site keeps refusing checks. A longer delay is gentler on the site but makes checks wait longer, because every check to that site queues behind the previous one.
As a rule of thumb, multiply the delay by the number of monitors you have on the site. That is roughly how long one round of checks takes to work through. For example, 40 monitors on one site with a 30-second delay take about 20 minutes to check in full, so some of them run up to 20 minutes after their scheduled time.
| Monitors on one site | Delay | Time to check them all |
|---|---|---|
| 10 | 10 seconds | about 2 minutes |
| 40 | 30 seconds | about 20 minutes |
| 60 | 60 seconds | about 1 hour |
Note: keep the full round shorter than your check frequency. If it takes longer than the time between checks, monitors on that site fall behind their schedule.
How does it work with robots.txt Crawl-delay?
When your team also has Respect robots.txt and bot protection turned on, PageCrawl honours the Crawl-delay a site sets in its robots.txt. If both apply, the longer of the two delays is used, up to 300 seconds. Crawl Politeness works on its own too, whether or not robots.txt is respected.
Does it make my checks run less often?
No. Your check frequency stays the same, and every check still runs. Crawl Politeness only changes when a check starts: a check to a busy site may start a little later than scheduled so it does not overlap with the previous one. Monitors on other sites run on time.