How to Monitor Password-Protected Websites for Changes

9 min read
How to Monitor Password-Protected Websites for Changes

Some of the most valuable content on the web lives behind login walls. Client portals, partner dashboards, internal wikis, vendor pricing pages, regulatory databases, and SaaS admin panels all require authentication to access. Standard website monitoring tools cannot see this content because they do not know how to log in.

Monitoring password-protected websites requires a tool that can automate the login process before checking for changes. If you are new to website monitoring, start with our complete guide to monitoring website changes for the fundamentals. This guide covers the techniques, tools, and best practices for monitoring content behind authentication.

Why Monitor Behind Logins?

The pages that require authentication are often the ones that matter most:

  • Vendor portals: Track pricing changes, contract terms, and product availability in your supplier's portal
  • Client dashboards: Monitor data feeds, report updates, and system status in tools you depend on
  • Partner programs: Watch for changes to commission rates, partner tiers, program requirements, and marketing materials
  • Regulatory databases: Some regulatory filings and compliance databases require registration to access
  • Internal tools: Monitor your own staging environments, admin dashboards, or internal wikis for unauthorized changes
  • SaaS pricing: Some B2B tools only show actual pricing after you log into a demo or trial account
  • Academic databases: Track research publications, journal updates, and grant listings behind institutional logins
  • Government portals: Monitor procurement opportunities, permit statuses, and regulatory submissions

How Login-Based Monitoring Works

The concept is straightforward: before the monitoring tool checks the page for changes, it runs a series of automated actions that replicate the human login process.

The Login Sequence

A typical login sequence looks like this:

  1. Navigate to the login page
  2. Wait for the login form to load
  3. Fill the username/email field
  4. Fill the password field
  5. Click the login/submit button
  6. Wait for the redirect to complete
  7. Navigate to the target page you want to monitor
  8. Check the page content for changes

This sequence runs every time the tool checks for changes. The tool does not store a session between checks (sessions expire), so it logs in fresh each time.

Browser-Based Automation

This only works with tools that use a real browser engine. The login process involves rendering JavaScript, handling form validation, processing cookies, and following redirects. Simple HTTP-based monitoring tools cannot handle this.

PageCrawl uses a full browser engine to render pages, which means it can handle any login flow that works in a regular browser: JavaScript-heavy login forms, CAPTCHA-free authentication, OAuth redirects, multi-step wizards, and cookie-based sessions.

Setting Up Login Monitoring in PageCrawl

PageCrawl supports login sequences through its "Actions" system. Actions are automated steps that run before each page check.

Basic Username/Password Login

For a standard login form with username and password fields:

  1. Add the target URL (the page you want to monitor after logging in)
  2. Open the Actions panel
  3. Add a "Navigate" action to go to the login page URL
  4. Add a "Fill" action for the username field:
    • Selector: The CSS selector for the email/username input (e.g., input[name="email"], #username, input[type="email"])
    • Value: Your username or email
  5. Add a "Fill" action for the password field:
    • Selector: The CSS selector for the password input (e.g., input[name="password"], #password, input[type="password"])
    • Value: Your password
  6. Add a "Click" action for the submit button:
    • Selector: The CSS selector for the login button (e.g., button[type="submit"], .login-btn, #login-submit)
  7. Add a "Wait" action to wait for the page to load after login (2-5 seconds is usually sufficient)

The monitor will then check the original target URL for changes after completing the login sequence.

Finding CSS Selectors

To find the right CSS selectors for form fields:

  1. Open the login page in your browser
  2. Right-click on the username field and select "Inspect"
  3. Look for identifying attributes:
    • id attribute: Use #the-id (e.g., #email-input)
    • name attribute: Use input[name="the-name"] (e.g., input[name="email"])
    • type attribute: Use input[type="email"] or input[type="password"]
    • class attribute: Use .the-class (e.g., .login-email)
    • placeholder attribute: Use input[placeholder="Enter email"]

The most reliable selectors use id or name attributes, as these rarely change. Avoid selectors based on class names that look auto-generated (e.g., .css-1a2b3c), as these change when the site rebuilds. For a thorough overview of selector strategies, see our CSS selector guide for web monitoring.

Handling Multi-Step Logins

Some login flows use multiple pages or steps:

Step 1: Enter email, click "Next" Step 2: Enter password, click "Sign In"

For these, chain the actions:

  1. Navigate to login page
  2. Fill email field
  3. Click "Next" button
  4. Wait 2-3 seconds for the password step to load
  5. Fill password field
  6. Click "Sign In" button
  7. Wait for redirect

Many sites show a cookie consent banner that can interfere with the login form. Add a "Click" action to dismiss the cookie banner before attempting to fill the login form:

  1. Click cookie accept button (e.g., button.cookie-accept, #accept-cookies)
  2. Wait 1 second
  3. Then proceed with login steps

PageCrawl's built-in "Remove Cookies" and "Remove Overlays" actions can also handle this automatically without manual configuration.

Handling "Remember Me" Checkboxes

If the login form has a "Remember Me" checkbox and you want to check it:

  1. Add a "Click" action targeting the checkbox (e.g., input[name="remember"], #remember-me)
  2. Place this before the submit button click

This is optional and usually not necessary, since the monitoring tool logs in fresh each time regardless.

Common Login Patterns and How to Handle Them

Basic Form (Most Common)

Navigate to: https://app.example.com/login
Fill: input[name="email"] -> your@email.com
Fill: input[name="password"] -> yourpassword
Click: button[type="submit"]
Wait: 3 seconds

This covers the majority of login forms (WordPress, SaaS apps, custom web applications).

HTTP Basic Authentication

Some sites use HTTP Basic Auth (the browser popup asking for username/password). PageCrawl supports this through HTTP authentication settings rather than form-filling actions. Enter the username and password in the authentication section of the monitor settings.

Two-Factor Authentication (2FA)

Two-factor authentication presents a challenge for automated monitoring because it requires a time-based code or physical device.

Options:

  • Use an app-specific password: Some services offer app-specific passwords that bypass 2FA (Google, Microsoft)
  • Use an API key: If the service offers API access, monitor the API response instead of the web portal
  • Create a monitoring-specific account: Set up a dedicated account without 2FA enabled (if the security policy allows it)
  • Use TOTP programmatically: Some advanced setups can generate TOTP codes programmatically using a shared secret

For most use cases, creating a dedicated monitoring account without 2FA is the simplest approach.

OAuth / SSO Logins ("Sign in with Google")

OAuth flows redirect through a third-party provider (Google, Microsoft, GitHub). These are more complex to automate:

  1. Click "Sign in with Google" on the target site
  2. Google's login page loads
  3. Fill email, click Next
  4. Fill password, click Next
  5. Handle any consent screens
  6. Redirect back to the target site

This requires more actions in the sequence, and Google specifically may challenge automated logins with additional verification. Where possible, use a direct username/password login instead of OAuth.

Single Sign-On (Enterprise SSO)

Enterprise SSO systems (Okta, Azure AD, Auth0) redirect through a corporate identity provider. The action sequence needs to handle the redirect:

  1. Navigate to the target app's login page
  2. The app redirects to the SSO provider
  3. Fill credentials on the SSO page
  4. The SSO provider redirects back to the app
  5. Wait for the final page to load

Add appropriate wait times between steps to account for redirects.

Security Best Practices

Storing login credentials in a monitoring tool requires careful security consideration:

Use Dedicated Monitoring Accounts

Create a separate account specifically for monitoring, with minimal permissions (read-only if possible). Do not use your personal admin account.

Benefits:

  • You can track monitoring access separately in the target system's audit logs
  • You can revoke access without affecting your personal account
  • The account can have 2FA disabled safely if it only has read access

Use Strong, Unique Passwords

Generate a unique, strong password for the monitoring account. Do not reuse passwords from other services.

Review Permissions

The monitoring account should have the minimum permissions needed to view the content you want to track. It should not have write, delete, or admin access.

Monitor the Monitoring Account

Keep an eye on the activity of your monitoring account in the target system's audit logs. Unusual access patterns could indicate a problem.

Understand the Terms of Service

Some services prohibit automated access in their terms of service. Review the ToS of any service you plan to monitor, especially if you are monitoring a competitor's platform.

Real-World Use Cases

Monitoring Vendor Pricing Portals

A procurement team monitors their vendors' B2B pricing portals to catch price increases early. They set up login monitoring on 12 vendor portals, checking daily. When a vendor updates pricing, the team gets a Slack notification with the AI summary of what changed, giving them time to negotiate or find alternatives before the new prices take effect.

Tracking Regulatory Database Updates

A compliance team monitors state regulatory databases that require registration to access. New filings and rule changes are posted behind the login. Daily monitoring with AI summaries flags which updates are relevant to their industry, reducing the manual review time from hours to minutes.

Monitoring Partner Program Changes

A partnerships team tracks changes to three technology partner programs. Commission structures, tier requirements, and program benefits are all behind partner portal logins. Weekly monitoring catches changes before the quarterly business review, keeping the team prepared.

Getting Started

Set up your first authenticated monitor and stop manually logging in to check for updates.

Last updated: 8 March, 2026