Staging Site Indexed Despite noindex? Causes & Fixes
Your staging, test or preview subdomain shows up in Google even though you set "noindex, nofollow". Here is how Google found it, why noindex did not stop it, and how to keep non-production sites out of search for good.
Prefer a done-for-you fix? We can clean it up for you →
How Google Discovered Your Staging Subdomain
Google does not need you to submit a host to find it. Any public URL that is referenced somewhere Googlebot can reach is a candidate for crawling. The usual sources:
- Links from your live site: a hardcoded staging URL in a navigation item, an image path, a blog post or an email template that ended up on the web.
- Sitemaps and canonical tags: a sitemap or
rel="canonical"generated from the wrong base URL points Google straight at the staging host. - Links in public places: issue trackers, public repositories, forum posts, client portals or documentation that mention the staging URL.
- Certificate transparency logs: every TLS certificate you issue (including free Let's Encrypt certificates) is published in public logs with its hostname. Crawlers and scanners read these logs, so a "secret" subdomain is not secret once it has HTTPS.
- Preview deployments: platforms like Vercel create public preview URLs for every branch. Their generated preview URLs are usually protected or marked noindex, but a custom domain you attach to a staging branch may not be. Check the headers instead of assuming.
Why Your noindex Didn't Work
A noindex only works if Google can crawl the page and actually read it. Almost every "noindex not working" case comes down to one of these:
1. robots.txt blocked the host, so Google never saw the noindex
This is the most common cause. If the staging robots.txt contains Disallow: /, Googlebot is not allowed to fetch the pages, so it never sees the noindex tag on them. Google can still index the bare URL based on links pointing to it. In Search Console this shows up as "Indexed, though blocked by robots.txt".
Rule of thumb: use robots.txt or noindex, never both on the same URL. See also What does 'Blocked by robots.txt' mean?
2. The noindex was added after Google had already indexed the pages
noindex is not retroactive. Google drops a page only after it recrawls it and sees the tag. Staging hosts get crawled rarely, so this can take weeks.
3. The noindex was injected with JavaScript
If the robots meta tag is added on the client (for example by a React component after hydration), it is not in the HTML Google fetches first. Google may or may not apply it after rendering. Put noindex in the server-rendered HTML or, better, in an HTTP header.
4. The noindex only covered some pages
A noindex set in one layout or template often misses other routes, PDFs, images and API responses. Non-HTML files cannot carry a meta tag at all; they need the X-Robots-Tag: noindex HTTP header.
5. nofollow does not prevent indexing
nofollow only tells Google not to follow the links on a page. It has no effect on whether the page itself is indexed. Only noindex does that.
How to Keep Staging and Preview Sites Out of Google
Best: put staging behind authentication
HTTP basic auth, an IP allowlist or VPN-only access means Google gets a 401 or 403 and has nothing to index. This also protects unreleased content from competitors and scrapers. Example for Caddy:
staging.example.com {
basic_auth {
# generate the hash with: caddy hash-password
team $2a$14$REPLACE_WITH_HASH
}
reverse_proxy app:3000
}On Vercel, enable Deployment Protection for preview deployments in your project settings.
Second best: send X-Robots-Tag on every response
If staging must stay public, set the header at the web server or proxy level so it covers every page, file and route, and keep crawling allowed so Google can read it.
Caddy
staging.example.com {
header X-Robots-Tag "noindex, nofollow"
reverse_proxy app:3000
}nginx
server {
server_name staging.example.com;
add_header X-Robots-Tag "noindex, nofollow" always;
# ...
}Cloudflare
Create a Response Header Transform Rule for the staging hostname that sets X-Robots-Tag to noindex, nofollow.
Don't: combine X-Robots-Tag or a meta noindex with Disallow: / in robots.txt. The Disallow hides the noindex from Google.
How to Check Your Staging Site
- 1. Search Google for
site:staging.example.comto see what is indexed. - 2. Check the response headers:
curl -sI https://staging.example.com/ | grep -i x-robots-tag - 3. Use URL Inspection in Google Search Console. A Domain property (
example.com) covers all subdomains, including staging. - 4. In the Pages report, look for "Indexed, though blocked by robots.txt" and "Excluded by 'noindex' tag" on the staging host.
Staging URLs Are Already Indexed. Now What?
Staging duplicates your live content
Don't just hide the URLs. Redirect them to the matching live pages with a 301 so Google consolidates them onto your main domain. How to 301 redirect an indexed subdomain →
Staging has no live equivalent
Remove any robots.txt Disallow, then either serve noindex or put the host behind authentication (401/403). Google drops the URLs after recrawling them. For faster results while you wait, use the Removals tool in Search Console. It hides URLs temporarily (about six months), so it is a stopgap, not the fix.
Frequently Asked Questions
Can I use robots.txt to noindex a page?
No. robots.txt has no supported noindex directive. Disallow blocks crawling, not indexing, and a blocked URL can still be indexed from links.
How long does noindex take to remove a page from Google?
Google has to recrawl the page first. Expect days for frequently crawled pages and several weeks for rarely visited hosts like staging.
Does nofollow stop Google from indexing?
No. nofollow only affects links on the page. Use noindex to control indexing.
How do I noindex a PDF or image?
Send the X-Robots-Tag: noindex HTTP header for those files, since they cannot contain a meta tag.
Fix It Yourself or Let Us Handle It
Track when Google drops your staging URLs with MyURLMonitor, or let our team audit the leak, set up the redirects and monitor the cleanup for you.
