Transform your page monitoring with AI-powered features that intelligently summarize changes and prioritize notifications based on importance.

PageCrawl.io's AI features bring intelligent analysis to your page monitoring workflow. Using your own API keys (BYOK - Bring Your Own Key), you can enhance change detection with smart summarization and automated prioritization, helping you focus on what matters most.

Key AI Features

1. AI-Powered Summarization

Get clear, concise summaries of page changes instead of raw technical details. Our AI analyzes detected changes and provides human-readable explanations of what actually changed on your pages.

Benefits:

• Quickly understand what changed without analyzing technical diffs
• Get context-aware summaries that highlight the most important modifications
• Save time by reading summaries instead of reviewing raw HTML changes
• Perfect for non-technical team members who need to understand page updates

2. Signal Prioritization

Let AI determine which changes are most important to your business. Our prioritization system analyzes changes and assigns priority levels, ensuring critical updates get your immediate attention.

Benefits:

• Focus on high-priority changes first
• Reduce notification fatigue from minor updates
• Automatically categorize changes by business impact
• Customize prioritization criteria based on your specific needs

Supported AI Providers

PageCrawl.io supports multiple AI providers with your own API keys:

• OpenAI (GPT-3.5, GPT-4, GPT-4 Turbo)
• Google Gemini (Gemini Pro, Gemini Pro Vision) - Free tier available
• Anthropic Claude (Claude 3 Haiku, Claude 3 Sonnet, Claude 3 Opus)
• OpenRouter (Access to multiple AI models through a single API) - Free models available

Setting Up AI Features

Prerequisites

• Active PageCrawl.io paid plan
• API key from one of the supported AI providers
• Pages already configured for monitoring
• Important: AI features must be configured separately for each workspace. You can have some workspaces with AI enabled and others without.

Step 1: Get Your AI Provider API Key

For OpenAI:

1. Visit OpenAI Platform
2. Create a new API key
3. Ensure your account has sufficient credits

For Google Gemini:

1. Visit Google AI Studio
2. Create a new API key
3. Enable the Gemini API in your Google Cloud Console
4. Note: Free tier models may use your data for training unless you upgrade to paid tier

For Anthropic Claude:

1. Visit Anthropic Console
2. Create a new API key
3. Ensure your account has sufficient credits

For OpenRouter:

1. Visit OpenRouter
2. Create an account and generate an API key
3. Add credits to your account
4. Access 200+ AI models through a single API endpoint
5. Benefit from unified billing and automatic fallbacks
6. Free models available: Several models like Qwen 2.5 72B and Qwen QwQ 32B are completely free
7. Note: Free models may use your data for training. Check each model's data usage policy on OpenRouter.

Step 2: Configure AI in PageCrawl.io

1. Access AI Settings

   • Log in to your PageCrawl.io account
   • Navigate to Settings > Integrations
   • Find the AI Features section

2. Add Your API Key

   • Select your preferred AI provider
   • Enter your API key
   • Click Test Connection to verify the key works
   • Save your configuration

3. Choose Your Model

   • Select from available models based on your needs
   • Budget-friendly: Cost-effective models for high-volume monitoring
   • Balanced: Good quality-to-cost ratio for most use cases
   • Premium: Highest quality for critical pages
   • Models marked with 🆓 are completely free (OpenRouter only)
   • Context length is shown for each model (e.g., 128k tokens)

4. Enable AI Features

   • Toggle on AI-Powered Summarization to get intelligent change summaries
   • Toggle on Signal Prioritization to automatically prioritize important changes

Step 3: Configure Per-Page Settings

For granular control, you can enable/disable AI features for specific pages:

1. Go to your tracked pages list
2. Click on a specific page to edit its settings
3. In the AI section, choose which features to enable:
   • Summarization: Get AI summaries for changes on this page
   • Prioritization: Apply AI prioritization to changes on this page

How AI Features Work

AI Summarization Process

1. PageCrawl detects a change on your page
2. The raw change data is sent to your chosen AI provider
3. AI analyzes the changes and generates a human-readable summary
4. You receive the summary along with the original change notification

Signal Prioritization Process

1. PageCrawl detects a change on your page
2. AI analyzes the change content, context, and configured criteria
3. A priority level is assigned (High, Medium, Low)
4. Notifications are marked with priority levels
5. You can filter and sort notifications by priority

Advanced Configuration

Monthly Usage Limits

Set a maximum number of AI requests per month to prevent unexpected costs:

• Navigate to Settings > Integrations > AI Features
• Set AI Requests Per Month to your desired limit
• View current usage directly in the settings (e.g., "Used: 150 / 1000 this month")
• When the limit is reached, AI features automatically pause until next month
• Leave empty for unlimited requests

Token Limits

Control maximum tokens per AI request to manage costs:

• Set Max Tokens to limit the size of content sent to AI
• About 1 token = 4 characters of text
• If full content exceeds this limit, the system automatically retries with diff mode
• Diff mode uses ~70% fewer tokens than full content
• View Token Usage Estimates based on your workspace history:
  • Shortest, Average, and Longest token usage
  • Separate estimates for Full Content vs Diff mode
  • Helps you set appropriate limits

Content Analysis Modes

Full Content Mode (Default):

• Sends both previous and current page content to AI
• Provides better context for more accurate summaries
• Uses more tokens, which means higher costs
• Best for: Complex pages where full context matters

Diff Mode:

• Sends only the changed lines (added/removed text)
• Uses ~70% fewer tokens
• More cost-effective for large pages
• Best for: Pages with frequent small changes

Toggle Analyze Full Content in Advanced Settings to switch between modes.

Custom Instructions

Teach the AI about your specific monitoring needs:

• Add workspace-wide instructions that apply to all AI analyses
• Tell AI what types of changes matter most to you
• Define what to focus on and what to ignore
• Example: "I monitor e-commerce product pages. Price changes and stock availability are critical. Product descriptions are medium priority. Ignore CSS changes."
• Maximum 2,000 characters

Summary Language

Generate AI summaries in your preferred language:

• Choose from 35+ supported languages
• Summaries will be generated in the selected language regardless of source content
• Available languages include: English, Spanish, French, German, Italian, Portuguese, Japanese, Chinese, Arabic, and many more

Initial Check Summaries

Control whether AI analyzes the first monitoring check:

• Toggle Summarize Initial Check in Advanced Settings
• When enabled: AI runs even when no changes have been detected yet
• When disabled: AI only runs when changes are detected
• Useful for getting initial insights about tracked pages

Priority Threshold

When prioritization is enabled, set the minimum priority score for notifications:

• Adjust the slider from Low (0) to Critical (100)
• Changes below this threshold are still tracked but won't trigger notifications
• Example: Setting to 70 means only high-importance changes notify you
• Find the right balance between staying informed and avoiding notification fatigue

Usage Tracking and Costs

Monitoring Your Usage

• View real-time usage statistics in Settings > Statistics
• See current month's usage directly in AI settings
• Track API calls, tokens used, and costs by feature
• Monitor usage across different AI providers
• Get alerts when approaching your monthly limit

Cost Management

• AI features use your own API keys, so you pay providers directly
• Usage is optimized to minimize token consumption
• You have full control over when and where AI features are applied
• No markup or additional fees from PageCrawl.io
• Set monthly limits to cap your spending
• Use diff mode to reduce token usage by ~70%
• Choose cost-effective models for high-volume monitoring

Best Practices

1. Start Small

• Begin by enabling AI features on your most important pages
• Monitor usage and costs before expanding to all pages
• Test different AI models to find the best balance of quality and cost

2. Optimize for Your Needs

• Use prioritization for high-volume pages where you need to filter important changes
• Use summarization for technical pages where changes are hard to interpret
• Consider disabling AI for pages with frequent, minor changes

3. Cost Optimization

• Start with free options: Google Gemini offers 250 free requests daily (1,000 with Flash model)
• Use free OpenRouter models: Qwen 2.5 72B and Qwen QwQ 32B are completely free
• Choose appropriate AI models based on your accuracy needs
• Use GPT-3.5 or Gemini Flash for cost-effective summarization
• Use GPT-4 or Claude for more complex prioritization tasks
• Monitor token usage and adjust settings if costs are higher than expected
• Enable diff mode to use ~70% fewer tokens

Troubleshooting

Common Issues

AI Features Not Working:

• Verify your API key is correct and has sufficient credits
• Check that the AI provider's service is operational
• Ensure your API key has the necessary permissions

High Costs:

• Review which pages have AI features enabled
• Consider using more cost-effective models
• Check if you're processing very large page changes

Inaccurate Summaries:

• Try different AI models to find better accuracy
• Use Custom Instructions to provide context about what matters for your monitoring
• Enable Full Content mode for better context (uses more tokens)
• Report persistent issues to our support team

Missing Notifications:

• Verify AI features are enabled for the specific pages
• Check your API key status and credits
• Review your notification settings

Getting Help

If you encounter issues with AI features:

1. Check your API provider's status page
2. Review usage statistics in PageCrawl.io
3. Contact our support team with specific error messages
4. Join our community forum for tips and best practices

Security and Privacy

Data Protection

• Your page content is only sent to AI providers when AI features are enabled for that workspace
• If AI features are not enabled, your page content is never analyzed or sent to any AI provider
• When enabled, your page content is sent to your chosen AI provider for analysis
• AI-generated summaries are stored in PageCrawl.io for your reference
• All data transmission uses secure HTTPS connections
• Your API keys are encrypted and stored securely

Provider Policies

• Review your AI provider's data usage and retention policies
• Understand how your chosen provider handles and processes data
• Consider using providers that offer data processing agreements if required for compliance

AI-powered change detection transforms how you monitor and respond to page changes, making your workflow more intelligent and efficient while keeping you in control of costs and data.