MyURLMonitor

Fix Vercel Bot Protection Issues

Resolve issues when MyURLMonitor cannot access your Vercel-hosted websites due to bot protection settings.

The Problem

If your website is hosted on Vercel with bot protection enabled, MyURLMonitor may be unable to access your sitemaps and website content. This happens when Vercel's firewall has:

  • •Bot Protection: Set to "On"
  • •Managed Rulesets: Enabled
  • •AI Bots: Set to "On"

Solution 1: System Bypass Rules (Pro Plan Required)

Best Solution: If you have a Vercel Pro plan or higher, you can create System Bypass Rules to allow MyURLMonitor access while keeping bot protection active.

Step-by-Step Instructions:

  1. 1
    Access Vercel Dashboard: Go to your project settings in the Vercel dashboard
  2. 2
    Navigate to Firewall: Click on "Firewall" in the left sidebar
  3. 3
    Create System Bypass Rule: Click "Add Rule" → "System Bypass Rules"
  4. 4
    Configure Rule: Add MyURLMonitor's user agent or IP ranges to bypass bot protection
  5. 5
    Apply and Test: Save the rule and test MyURLMonitor access

Note: System Bypass Rules are only available on Vercel Pro plans and above. Regular custom rules will not work for this purpose.

Solution 2: Static Subdomain (Free Alternative)

Free Workaround: If you're on Vercel's Hobby (free) plan, you can create a static subdomain to serve your sitemaps without bot protection.

Implementation Steps:

  1. 1
    Create Static Subdomain: Set up static.yourdomain.com
  2. 2
    Deploy Static Files: Create a separate Vercel project for static content (sitemaps, robots.txt)
  3. 3
    Disable Bot Protection: Turn off bot protection only for the static subdomain
  4. 4
    Update Sitemap URLs: Point MyURLMonitor to static.yourdomain.com/sitemap.xml
  5. 5
    Sync Content: Set up automated deployment to keep sitemaps updated

Tip: This approach keeps your main site protected while allowing MyURLMonitor to access essential files for indexing.

Alternative Solutions

Temporary Disable Bot Protection

Temporarily turn off bot protection in Vercel's firewall settings during MyURLMonitor's crawling periods.

Not Recommended - Reduces Security

Whitelist MyURLMonitor IPs

Contact our support team for MyURLMonitor's IP ranges to whitelist in your firewall.

Requires Pro Plan

Testing Your Configuration

After implementing any solution, test MyURLMonitor's access:

  1. 1. Try accessing your sitemap URL directly in a browser
  2. 2. Run a test crawl in MyURLMonitor dashboard
  3. 3. Check for any error messages in the MyURLMonitor logs
  4. 4. Verify that new URLs are being discovered and indexed

Still Having Issues?

Our support team can help you configure the right solution for your Vercel setup.

Contact SupportBack to Troubleshooting