A robots.txt disallow stops Googlebot from fetching a URL, so it never reads the page's noindex tag or header. Google can still index the URL from links and other signals, which is why blocking can prevent deindexing. To remove a page, keep it crawlable and serve noindex, or use password protection. Use robots.txt mainly to manage crawl traffic, not to hide pages.
Why blocking a URL can keep it in search results
Google's own guidance is direct: a robots.txt file tells search engine crawlers which URLs the crawler can access on your site, and it is used mainly to avoid overloading your site with requests. It is not a mechanism for keeping a web page out of Google. To keep a page out, the documentation says to block indexing with noindex or password-protect the page (Introduction to robots.txt).
The mechanism is a sequence, not a preference. A crawler must fetch a URL before it can read a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header. A disallow rule removes that fetch. The noindex instruction is never seen, so it is never applied. Google can still index the URL without visiting the page, for example when other pages link to it. The result is a search result that may show a URL and a title derived from links.
This is why the two tools are not interchangeable. Robots.txt controls crawling. Noindex controls indexing. They act at different stages, and using the crawl-stage tool to achieve an index-stage outcome creates the exact failure people are trying to avoid.
What does Google actually say about noindex and robots.txt?
The noindex documentation states the condition plainly: for the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results (Block Search indexing with noindex).
The same page notes that noindex can be implemented as a meta tag or an HTTP response header, with the same effect, and that specifying the noindex rule in the robots.txt file is not supported by Google. It also notes that noindex is a rule for search engines that support it, so a page might remain in another engine's results even after Google drops it.
For media files, the calculus changes. Google's robots.txt guidance says a disallow can prevent image, video and audio files from appearing in Google Search results, though it will not stop other pages from embedding or linking to them. So the blanket rule "never disallow" is too simple; the correct rule depends on the file type and the goal.
A decision table for common situations
| Goal | Correct control | Why |
|---|---|---|
| Remove an HTML page from Google | Keep it crawlable; serve noindex via meta tag or HTTP header | The crawler must read the rule |
| Stop a page being crawled to reduce server load | robots.txt disallow | Crawl management is the intended use |
| Keep a page out of all search engines | noindex plus password protection | noindex support varies by engine |
| Remove an image or video from results | robots.txt disallow can work | Media files behave differently from HTML |
| Remove a URL that is already indexed | Allow crawling, then serve noindex, then request removal if needed | Blocking first hides the rule |
Example: the sequence that fails, and the one that works
Consider a hypothetical UK retailer retiring a seasonal campaign page. The page still has inbound links from press coverage.
Failed sequence: the team adds Disallow: /campaign/spring-sale to robots.txt. Googlebot stops fetching the page. Because the page still carries no noindex tag, and other sites link to it, the URL can remain in results with a link-derived title. Months later the page is still visible, and the team concludes that "Google ignores removal requests." It does not; the instruction was never readable.
Working sequence: the team removes the disallow, confirms the URL returns HTTP 200 to Googlebot, adds <meta name="robots" content="noindex"> to the page head, and waits for the next crawl. Once Google processes the tag, the page is dropped from results regardless of inbound links. If speed matters, the team can also use the Removals tool for a temporary block while the noindex is processed.
This example is illustrative. It is not a client case and the timing is not a guarantee; crawl frequency varies with site size, authority and demand.
How to diagnose a page that will not deindex
Work through these checks in order. Each one is a verification step, not a guess.
- Check the live HTTP status. Fetch the URL as Googlebot would. A 404 or 410 removes the page through a different mechanism; a 200 means noindex can work.
- Check robots.txt for the exact path. Read the file at
/robots.txtand test the specific URL against the rules, including any wildcard or end-anchor patterns. A broad disallow can catch a page you thought was allowed. - Check the rendered head, not the source template. A noindex tag injected by JavaScript may not appear in the raw HTML. Use the URL Inspection tool to see what Google actually received.
- Check for conflicting directives. A page-level
indexcombined with a header-levelnoindexis ambiguous. Decide which layer wins and remove the conflict. - Check the header on non-HTML resources. PDFs and other non-HTML files cannot carry a meta tag, so they need an HTTP header or password protection.
If all five checks pass and the page is still indexed, the remaining variable is time. Indexing changes are not instant, and the documentation does not promise a fixed window.
When robots.txt is still the right tool
Use robots.txt when the problem is crawl behaviour rather than visibility: faceted navigation that generates thousands of near-duplicate URLs, internal search result pages, staging environments that should not be crawled at all, and media files you want excluded from results. The Google guidance frames the file as a way to manage crawler traffic and avoid crawling unimportant or similar pages.
One caution for CMS users: platforms such as Wix and Blogger may not expose robots.txt directly, offering a visibility setting instead. If you cannot edit the file, check the CMS search settings before assuming a rule exists.
For stores, the same logic applies to category and filter pages. If a filter URL should never rank, noindex is the cleaner control; if it merely wastes crawl budget, a disallow is reasonable.
What remains uncertain
Google's documentation describes the mechanism but not the timing. It does not state how many crawl cycles a noindex takes to be processed, and it does not quantify how often a blocked URL still appears in results. Those numbers depend on site-specific factors that no public source provides.
There is also a genuine edge case: a URL blocked by robots.txt can still be indexed, but Google may eventually drop it if nothing links to it and no other signal sustains it. That is not a removal method you can rely on or schedule. Treat it as an observation, not a plan.
Finally, noindex support is not universal. Google's own page notes that other engines may interpret the rule differently, so a multi-engine strategy should pair noindex with password protection where the content truly must not be public.
Follow-up questions
Does noindex work if the page is also blocked in robots.txt?
No. The crawler cannot fetch the page, so it cannot read the noindex tag or header. Remove the disallow for that URL first, confirm the page is fetchable, then let the noindex be processed.
Can I use robots.txt to remove a page that is already indexed?
Not reliably. A disallow prevents future crawling but does not remove an existing index entry, and it can hide the noindex rule that would. Allow crawling, serve noindex, and use the Removals tool for a temporary block if you need faster action.