Your staging, test or preview subdomain shows up in Google even though you set "noindex, nofollow". Here is how Google found it, why noindex did not stop it, and how to keep non-production sites out of search for good.
Prefer a done-for-you fix? We can clean it up for you →
Google does not need you to submit a host to find it. Any public URL that is referenced somewhere Googlebot can reach is a candidate for crawling. The usual sources:
rel="canonical" generated from the wrong base URL points Google straight at the staging host.A noindex only works if Google can crawl the page and actually read it. Almost every "noindex not working" case comes down to one of these:
This is the most common cause. If the staging robots.txt contains Disallow: /, Googlebot is not allowed to fetch the pages, so it never sees the noindex tag on them. Google can still index the bare URL based on links pointing to it. In Search Console this shows up as "Indexed, though blocked by robots.txt".
Rule of thumb: use robots.txt or noindex, never both on the same URL. See also What does 'Blocked by robots.txt' mean?
noindex is not retroactive. Google drops a page only after it recrawls it and sees the tag. Staging hosts get crawled rarely, so this can take weeks.
If the robots meta tag is added on the client (for example by a React component after hydration), it is not in the HTML Google fetches first. Google may or may not apply it after rendering. Put noindex in the server-rendered HTML or, better, in an HTTP header.
A noindex set in one layout or template often misses other routes, PDFs, images and API responses. Non-HTML files cannot carry a meta tag at all; they need the X-Robots-Tag: noindex HTTP header.
nofollow only tells Google not to follow the links on a page. It has no effect on whether the page itself is indexed. Only noindex does that.
HTTP basic auth, an IP allowlist or VPN-only access means Google gets a 401 or 403 and has nothing to index. This also protects unreleased content from competitors and scrapers. Example for Caddy:
staging.example.com {
basic_auth {
# generate the hash with: caddy hash-password
team $2a$14$REPLACE_WITH_HASH
}
reverse_proxy app:3000
}On Vercel, enable Deployment Protection for preview deployments in your project settings.
If staging must stay public, set the header at the web server or proxy level so it covers every page, file and route, and keep crawling allowed so Google can read it.
Caddy
staging.example.com {
header X-Robots-Tag "noindex, nofollow"
reverse_proxy app:3000
}nginx
server {
server_name staging.example.com;
add_header X-Robots-Tag "noindex, nofollow" always;
# ...
}Cloudflare
Create a Response Header Transform Rule for the staging hostname that sets X-Robots-Tag to noindex, nofollow.
Don't: combine X-Robots-Tag or a meta noindex with Disallow: / in robots.txt. The Disallow hides the noindex from Google.
site:staging.example.com to see what is indexed.curl -sI https://staging.example.com/ | grep -i x-robots-tagexample.com) covers all subdomains, including staging.Don't just hide the URLs. Redirect them to the matching live pages with a 301 so Google consolidates them onto your main domain. How to 301 redirect an indexed subdomain →
Remove any robots.txt Disallow, then either serve noindex or put the host behind authentication (401/403). Google drops the URLs after recrawling them. For faster results while you wait, use the Removals tool in Search Console. It hides URLs temporarily (about six months), so it is a stopgap, not the fix.
No. robots.txt has no supported noindex directive. Disallow blocks crawling, not indexing, and a blocked URL can still be indexed from links.
Google has to recrawl the page first. Expect days for frequently crawled pages and several weeks for rarely visited hosts like staging.
No. nofollow only affects links on the page. Use noindex to control indexing.
Send the X-Robots-Tag: noindex HTTP header for those files, since they cannot contain a meta tag.
Track when Google drops your staging URLs with MyURLMonitor, or let our team audit the leak, set up the redirects and monitor the cleanup for you.