Push API: Monitor Data You Send UsDeveloper

Most PageCrawl monitors watch a public web page. Data sources flip the direction: you push values into PageCrawl over a simple REST endpoint (or by email), and PageCrawl treats each value exactly like a check on a monitored page. You get the same change comparison, history, charts, multi-channel notifications, and AI summaries, but for data that only you can see.

When to Use a Data Source

Use a data source whenever the value you care about is not on a public page that a monitor can reach:

• Internal KPIs - Signup counts, error rates, queue depth, revenue figures from your own systems
• Values behind authentication - Numbers from an admin panel, a private dashboard, or an intranet page that your own script can read
• Partner or third-party API responses - Poll an API yourself on your own schedule and push the result, so PageCrawl tracks it over time
• Anything a crawler cannot reach - Values computed on a device, inside a private network, or output by a batch job

If the value is on a public web page, a regular monitor is usually simpler. Data sources are for everything else.

Creating a Data Source

In the app

Go to Settings > API > Data Sources and click Create Data Source. Give it a name, optionally define named fields (for example price and stock), and save. The secret ingest URL and the private inbound email address are shown right after creation, and again on the monitor's Send data panel whenever you need them later.

Via the API

All requests use a Bearer token from Settings > API (see the API and Webhooks guide for token setup).

curl -X POST "https://pagecrawl.io/api/data-sources" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Competitor API price",
    "fields": [
      {"label": "price", "type": "number"},
      {"label": "stock", "type": "number"}
    ]
  }'

• fields is optional. Leave it out to track a single value.
• email_sender_filter (optional) restricts who can email values in, either an exact address or an exact domain.

The response is 201 with the monitor object: id, slug, elements, ingest_url (the secret push URL for this source), and ingest_email (the private inbound email address).

Pushing Values

Every data source has its own secret ingest URL, shown when you create the data source and on the monitor's Send data panel. The URL is the whole credential: no API token, no headers, no client library. POST a value to it and the value is recorded.

Single value, one line:

curl -d value=42 https://pagecrawl.io/api/ingest/data-a1b2c3d4e5f6g7h8i9j0k1l2

The same call from code:

fetch('https://pagecrawl.io/api/ingest/data-a1b2c3d4e5f6g7h8i9j0k1l2', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ value: 42 }) });

file_get_contents('https://pagecrawl.io/api/ingest/data-a1b2c3d4e5f6g7h8i9j0k1l2', false, stream_context_create(['http' => ['method' => 'POST', 'header' => 'Content-Type: application/json', 'content' => json_encode(['value' => 42])]]));

import requests

requests.post('https://pagecrawl.io/api/ingest/data-a1b2c3d4e5f6g7h8i9j0k1l2', json={'value': 42})

Multiple named fields work with plain form encoding:

curl -d 'values[price]=9.99' -d 'values[stock]=12' https://pagecrawl.io/api/ingest/data-a1b2c3d4e5f6g7h8i9j0k1l2

or as JSON:

curl -H 'Content-Type: application/json' \
  -d '{"values": {"price": 9.99, "stock": 12}}' \
  https://pagecrawl.io/api/ingest/data-a1b2c3d4e5f6g7h8i9j0k1l2

A successful push returns 200:

{
  "changed": true,
  "check_id": 123,
  "monitor_id": 55,
  "values": {"price": "9.99", "stock": "12"},
  "created_fields": []
}

Each named field becomes its own tracked element with its own chart and history, so a data source with price and stock gives you two separate value timelines on one monitor. New field names in values are created automatically: numeric-looking values become number fields, everything else becomes a text field, and any fields created this way are listed in created_fields.

Pushing the same value again returns "changed": false and stores nothing new, so it is safe to push on a schedule even when the value rarely moves.

Authenticated endpoint

You can also push to /api/ingest/{slug} with a Bearer token from Settings > API. Prefer this for workspace-scoped scripts that already hold an API token, or when you do not want the secret URL stored in the script:

curl -X POST "https://pagecrawl.io/api/ingest/my-kpi-source" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"value": 42}'

Both endpoints accept the same bodies and return the same responses.

Sending Values by Email

Every data source has a private inbound address like data-a1b2c3d4e5f6g7h8i9j0k1l2@ingest.pagecrawl.io. It is shown in the app and returned as ingest_email when you create the source via the API. Recording a value is one line:

echo "42" | mail data-a1b2c3d4e5f6g7h8i9j0k1l2@ingest.pagecrawl.io

Email a value in the message body:

• Lines like price: 9.99 that match existing field names map to those fields.
• Otherwise the whole body is stored as the single value.
• If the body is empty, the subject line is used instead.

This is useful for cron jobs, no-code tools, and systems that can only send email: pipe a value into your mail command, point a form or automation at the address, and the value lands in PageCrawl like any other push.

How Changes and Notifications Work

A data source behaves like any other monitor once a value arrives:

1. Each changed push is stored as a check. The value appears in the monitor's history with a timestamp, and number fields plot on the value chart.
2. The new value is compared with the previous one. If nothing changed, the push is acknowledged with "changed": false and no new history entry is created.
3. When a value changes, your notification rules fire. Email, Slack, Discord, Teams, Telegram, and webhooks all work, and alerts arrive shortly after the value is received.
4. AI features apply the same way. When enabled, AI summaries and priority scoring run on data source changes just like on page changes.

Because pushes flow through the normal change pipeline, everything built on top of monitors (dashboards, reports, RSS feeds, webhook payloads, MCP read tools) works on data sources with no extra setup.

Errors and Limits

Value size: a single pushed value can be up to 500,000 characters. Data sources store the first 65,535 characters of each value (10,000 on the Free plan); longer values also keep a checksum of the full content, so a change anywhere in the value is still detected and recorded even when it falls beyond the stored portion. Truncated values end with a marker line that includes the checksum.

Every accepted push counts toward your plan's check allowance, the same as a scheduled check on a monitored page, whether or not the value changed. Pushes that return "changed": false deduplicate the stored history entry (no new history row is written), so frequent same-value pushes do not flood your history.

Using Data Sources with AI Assistants

The PageCrawl MCP server includes two data source tools, available on all plans:

Existing read tools (get latest values, get monitor history) work on data sources like any other monitor, so you can ask your assistant things like "record today's signup count in my KPI data source" or "chart the last 30 days of the price field". See the AI Assistants (MCP Server) guide for setup.

Related Articles

• API and Webhooks for Custom Integrations - Token setup, monitor endpoints, and webhook quick start
• Webhook Integration - Receive change payloads from data sources and monitors alike
• AI Assistants (MCP Server) - Connect Claude, ChatGPT, and other MCP clients
• Full API Reference - Interactive OpenAPI reference with every endpoint and schema