This guide shows three ways to connect your own application to PageCrawl and provides working code for each in Python, Node.js, and PHP. These are the same patterns the official Home Assistant integration uses, distilled into minimal examples you can adapt.
Pick the pattern that fits your needs:
- Polling is the simplest. You read the API on a timer. Best for dashboards and reports that can wait for the next poll.
- Webhooks (push) send each change to your server when PageCrawl saves it. Best for automation and alerting.
- Hybrid combines a webhook for current values with a slow reconcile poll that catches anything missed. This is the most robust option and what the Home Assistant integration runs.
Building with a coding agent such as Claude Code, Codex or Cursor? PageCrawl's pagecrawl-api skill teaches it these same patterns, including webhook signature verification, so the code it writes follows them.
Authentication
All API requests use a bearer token in the Authorization header:
Authorization: Bearer YOUR_TOKENYou can use an API token (Settings > API access) or an OAuth access token. The API is available on every plan. Treat the token like a password and keep it server-side.
Rate Limits
Limits apply per endpoint and per account, and depend on your plan. When you exceed one, the API responds with HTTP 429 and a JSON message. Honor the Retry-After response header (seconds to wait) before retrying. Choose a poll interval that stays well under your limit, especially if you also fetch history or diffs for many monitors.
Polling
Poll GET /api/pages?simple=1&folder=*&take=1 on an interval. Keep folder=*: without it, a workspace that uses folders returns only the pages outside folders. The response is one plain JSON array of every active page, not paged, and with take=1 each page carries its latest check.
Each page object has id, name, url, status, latest, elements (the page's tracked elements) and checks (newest first). latest.contents is a preview of the main tracked element, cut to 300 characters. Full values are in checks[0].elements, an object keyed by element_id (the keys are strings, and a check with no values comes back as an empty array). Those rows carry no label, so take labels from the page's elements list, where each tracked element's id matches an element_id. A monitor made from a template has its tracked elements on the template, so its elements list can be empty.
Each check in checks is a change PageCrawl saved, so a new check id means something moved. It is not always a change you want to act on: a monitor's first capture is a check too.
Python
import time
import requests
BASE = "https://pagecrawl.io"
TOKEN = "YOUR_TOKEN"
SESSION = requests.Session()
SESSION.headers["Authorization"] = f"Bearer {TOKEN}"
def fetch_pages():
"""Every active monitor with its latest check, honoring 429."""
url = f"{BASE}/api/pages?simple=1&folder=*&take=1"
while True:
resp = SESSION.get(url, timeout=60)
if resp.status_code == 429:
wait = int(resp.headers.get("Retry-After", "5"))
time.sleep(wait)
continue
resp.raise_for_status()
return resp.json() # one plain JSON array, not paged
def poll_once():
for page in fetch_pages():
latest = page.get("latest") or {}
print(page["id"], page.get("name") or page["url"], "->", latest.get("contents"))
labels = {str(el["id"]): el.get("label") for el in page.get("elements") or []}
checks = page.get("checks") or []
values = checks[0].get("elements") if checks else None
if not isinstance(values, dict):
continue # no values recorded on this check
for element_id, el in values.items():
# element_id is stable across every check; use it as your key.
print(" ", element_id, labels.get(element_id), el.get("contents"))
if __name__ == "__main__":
while True:
poll_once()
time.sleep(300) # stay well under the rate limitNode.js
const BASE = "https://pagecrawl.io";
const TOKEN = "YOUR_TOKEN";
const HEADERS = { Authorization: `Bearer ${TOKEN}` };
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function fetchPages() {
const url = `${BASE}/api/pages?simple=1&folder=*&take=1`;
while (true) {
const resp = await fetch(url, { headers: HEADERS });
if (resp.status === 429) {
const wait = parseInt(resp.headers.get("Retry-After") || "5", 10);
await sleep(wait * 1000);
continue;
}
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
return resp.json(); // one plain JSON array, not paged
}
}
async function pollOnce() {
for (const page of await fetchPages()) {
const latest = page.latest || {};
console.log(page.id, page.name || page.url, "->", latest.contents);
const labels = new Map((page.elements || []).map((el) => [String(el.id), el.label]));
const values = page.checks?.[0]?.elements || {};
for (const [elementId, el] of Object.entries(values)) {
// element_id is stable across every check; use it as your key.
console.log(" ", elementId, labels.get(elementId), el.contents);
}
}
}
async function main() {
while (true) {
await pollOnce();
await sleep(300_000); // stay well under the rate limit
}
}
main();Webhooks (Push)
Create a hook so PageCrawl POSTs to your server each time it saves a change, then verify every delivery.
1. Create the hook
POST /api/hooks
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json
{
"target_url": "https://your-server.example.com/pagecrawl",
"match_type": "all",
"events": ["change_detected"]
}events takes change_detected (the default) and error. The response includes a signing_secret. Store it securely: you will use it to verify deliveries. You can also create hooks in the app under Settings > Integrations > Webhooks. A hook created there starts with the Essential payload fields, so choose All fields (or add the fields you read under Custom) when your receiver needs contents, page_elements or previous_check. A hook created through the API without payload_fields sends every field.
2. Verify each delivery
Every webhook includes two headers:
X-PageCrawl-Signature: sha256=<hmac>X-PageCrawl-Timestamp: <unix>
The HMAC is HMAC_SHA256(signing_secret, "{timestamp}.{body}") where {body} is the exact raw request body. Compute the same value, compare it in constant time, and reject deliveries whose timestamp is too old (to prevent replay). Always verify against the raw bytes, not a re-serialized object.
Deliveries that carry whole-page text can run to several megabytes, so let your server accept large bodies: the Node.js example raises Express's 100 KB default, and in PHP post_max_size must be larger than the biggest delivery. Answer with a 2xx status within 60 seconds.
3. Decide what to act on
A change_detected delivery is sent for every change PageCrawl saves, including a monitor's first capture (previous_check is null) and changes that sent no alert. An error delivery repeats the monitor's latest saved check with a status naming the error. Retries, Test Webhook and Sync History can send the same check again, so the receivers below skip failed checks and first captures, and a real integration should also skip ids it has already processed. The Webhook Integration guide describes every case.
Python
import hashlib
import hmac
import time
MAX_AGE = 300 # seconds
def verify_signature(secret: str, timestamp: str, raw_body: bytes, header: str) -> bool:
if not secret or not timestamp or not header:
return False
# Plain digits only, so a malformed header is refused instead of raising.
if not (timestamp.isascii() and timestamp.isdigit() and len(timestamp) <= 12):
return False
if abs(time.time() - int(timestamp)) > MAX_AGE:
return False # stale, possible replay
expected = hmac.new(
secret.encode("utf-8"),
f"{timestamp}.".encode("utf-8") + raw_body,
hashlib.sha256,
).hexdigest()
provided = header[len("sha256="):] if header.startswith("sha256=") else header
# Compare bytes, so a header with non-ASCII characters fails instead of raising.
return hmac.compare_digest(expected.encode(), provided.encode("utf-8", "replace"))A minimal Flask receiver:
from flask import Flask, request, abort
app = Flask(__name__)
SIGNING_SECRET = "YOUR_SIGNING_SECRET"
@app.post("/pagecrawl")
def receive():
sig = request.headers.get("X-PageCrawl-Signature")
ts = request.headers.get("X-PageCrawl-Timestamp")
if not verify_signature(SIGNING_SECRET, ts, request.get_data(), sig):
abort(401)
payload = request.get_json(silent=True)
if not isinstance(payload, dict):
abort(400)
if payload.get("status") != "ok":
return "", 204 # a failed check: the values repeat the last saved check
if "previous_check" in payload and payload["previous_check"] is None:
return "", 204 # the monitor's first capture: a starting value, not a change
print("change on", payload.get("id"), payload.get("short_summary"))
return "", 204Node.js
const crypto = require("crypto");
const express = require("express");
const SIGNING_SECRET = "YOUR_SIGNING_SECRET";
const MAX_AGE = 300; // seconds
function verifySignature(secret, timestamp, rawBody, header) {
if (!secret || !timestamp || !header) return false;
const ts = parseInt(timestamp, 10);
if (Number.isNaN(ts)) return false;
if (Math.abs(Date.now() / 1000 - ts) > MAX_AGE) return false; // stale
const expected = crypto
.createHmac("sha256", secret)
.update(`${timestamp}.${rawBody}`)
.digest("hex");
const provided = header.startsWith("sha256=") ? header.slice(7) : header;
const a = Buffer.from(expected);
const b = Buffer.from(provided);
return a.length === b.length && crypto.timingSafeEqual(a, b);
}
const app = express();
// Capture the raw body exactly as received so the HMAC matches. Whole-page
// deliveries are larger than Express's 100 KB default, so raise the limit.
app.use(express.raw({ type: "application/json", limit: "25mb" }));
app.post("/pagecrawl", (req, res) => {
const sig = req.get("X-PageCrawl-Signature");
const ts = req.get("X-PageCrawl-Timestamp");
const raw = Buffer.isBuffer(req.body) ? req.body.toString("utf8") : "";
if (!verifySignature(SIGNING_SECRET, ts, raw, sig)) {
return res.sendStatus(401);
}
let payload;
try {
payload = JSON.parse(raw);
} catch {
return res.sendStatus(400);
}
if (payload === null || typeof payload !== "object" || Array.isArray(payload)) {
return res.sendStatus(400); // not a JSON object
}
res.sendStatus(204); // answer first, then process
if (payload.status !== "ok") return; // a failed check
if (payload.previous_check === null) return; // the monitor's first capture
console.log("change on", payload.id, payload.short_summary);
});
app.listen(8080);PHP
<?php
function verify_signature(string $secret, ?string $timestamp, string $rawBody, ?string $header): bool
{
$maxAge = 300; // seconds
if ($secret === '' || $timestamp === null || $header === null) {
return false;
}
if (! ctype_digit($timestamp)) {
return false;
}
if (abs(time() - (int) $timestamp) > $maxAge) {
return false; // stale, possible replay
}
$expected = hash_hmac('sha256', "{$timestamp}.{$rawBody}", $secret);
$provided = str_starts_with($header, 'sha256=') ? substr($header, 7) : $header;
return hash_equals($expected, $provided);
}
$signingSecret = 'YOUR_SIGNING_SECRET';
$rawBody = file_get_contents('php://input');
$sig = $_SERVER['HTTP_X_PAGECRAWL_SIGNATURE'] ?? null;
$ts = $_SERVER['HTTP_X_PAGECRAWL_TIMESTAMP'] ?? null;
if (! verify_signature($signingSecret, $ts, $rawBody, $sig)) {
http_response_code(401);
exit;
}
$payload = json_decode($rawBody, true);
if (! is_array($payload)) {
http_response_code(400);
exit;
}
http_response_code(204);
if (($payload['status'] ?? null) !== 'ok') {
exit; // a failed check: the values repeat the last saved check
}
if (array_key_exists('previous_check', $payload) && $payload['previous_check'] === null) {
exit; // the monitor's first capture: a starting value, not a change
}
error_log('change on '.$payload['id'].' '.($payload['short_summary'] ?? ''));Hybrid (Push Plus Reconcile)
The most robust integration uses a webhook to keep values current and a slow background poll that reconciles state. The reconcile poll catches anything a webhook might miss and refreshes monitors that did not change. Webhook retries stop once the monitor has a newer check, so after your server has been offline, the poll is what brings you back in step. This is the model the Home Assistant integration runs: push updates the in-memory snapshot, and a slow loop re-fetches the full list on a long interval.
The webhook side reads page_elements, which the Essential payload of a hook created in the app leaves out: choose All fields, or add page_elements under Custom.
Python (sketch)
import threading
import time
state = {} # element_id -> latest value, shared between push and poll
lock = threading.Lock()
def on_webhook(payload):
"""Called from your verified webhook receiver."""
# An error delivery repeats the latest saved values, so applying it is harmless.
with lock:
for el in payload.get("page_elements") or []:
state[el["element_id"]] = el.get("contents")
def reconcile_loop():
"""Slow safety net. Re-reads everything on a long interval."""
while True:
for page in fetch_pages(): # from the polling example above
checks = page.get("checks") or []
values = checks[0].get("elements") if checks else None
if not isinstance(values, dict):
continue
with lock:
for element_id, el in values.items():
state[int(element_id)] = el.get("contents")
time.sleep(3600) # reconcile hourly; the webhook keeps values current
threading.Thread(target=reconcile_loop, daemon=True).start()Node.js (sketch)
const state = new Map(); // element_id -> latest value
function onWebhook(payload) {
// Called from your verified webhook receiver.
for (const el of payload.page_elements || []) {
state.set(String(el.element_id), el.contents);
}
}
async function reconcileLoop() {
// Slow safety net. Re-reads everything on a long interval.
while (true) {
for (const page of await fetchPages()) {
// fetchPages from the polling example
for (const [elementId, el] of Object.entries(page.checks?.[0]?.elements || {})) {
state.set(elementId, el.contents);
}
}
await new Promise((r) => setTimeout(r, 3_600_000)); // reconcile hourly
}
}
reconcileLoop();Keep the reconcile interval long (hourly or slower) so the webhook does the everyday work and the poll stays comfortably within your rate limit.
Related Articles
- Webhook Integration - Full webhook payload reference, including which checks are delivered, the
page_elementsarray andelement_id - Full API Reference - Interactive OpenAPI reference for every endpoint
- PageCrawl Skills for AI Agents - Teach coding agents these patterns with the
pagecrawl-apiskill