Meta Robots Tag: Every Directive, Explained (Noindex, Nofollow)

The meta robots tag looks like a short, harmless line of HTML: <meta name="robots" content="noindex">. It controls whether a page shows up in Google at all, and it sits next to a dozen other directives most guides either skip or get wrong. Two values still floating around competitor articles, noarchive and noodp, stopped doing anything years ago, and one setup mistake, pairing a robots.txt block with a noindex tag, quietly defeats the thing it's supposed to do. Here's the current directive list, what changed and when, and the one conflict that trips up almost everyone who's never read the fine print.

Key takeaways

Google's current directive list has eleven active values, from noindex and nofollow to max-image-preview and unavailable_after. Full table below.
noarchive no longer does anything at Google, which retired the cached-page feature it controlled in early 2024. Google's own docs had already reclassified the rule as historical by the time Search Engine Journal reported it on October 2, 2024.
noodp isn't in Google's current documentation at all. If it's still in a template somewhere, it's inert leftover code.
A noindex tag only works if the crawler can reach the page. Block the URL in robots.txt first, and Google never sees the noindex tag: the page can keep appearing in results anyway, as a bare link with no snippet.
X-Robots-Tag carries the same directives as the meta tag but rides in the HTTP response header, the only way to apply these rules to a PDF, an image, or any file with no <head> to hold a tag.

What the meta robots tag actually controls

The meta robots tag governs indexing and snippet behavior, not crawling. That distinction is the root of most confusion here. <meta name="robots" content="..."> sits in a page's <head> and tells search engines what to do once they've already fetched it: show it in results or not, follow its links or not, offer a snippet or not. It has no say over whether the crawler may request the page at all; that job belongs to robots.txt, a separate file with a separate job, and the two only interact badly when someone assumes they stack.

There's also a targeting layer. Per Google's documentation on supported meta tags, name="robots" applies to every standard-respecting crawler, while name="googlebot" speaks to Google alone. Run both on one page and, if they conflict, Google applies the more restrictive one.

Every current directive, and the ones that quietly retired

Here's the complete list, current as of Google's Robots Meta Tags Specifications (last updated March 24, 2026), historical entries marked separately.

Directive	What it does	Status
`all`	No restrictions. The default; listing it explicitly changes nothing.	Current
`noindex`	Keep this page, media file, or resource out of search results.	Current
`nofollow`	Don't follow the links on this page.	Current
`none`	Shorthand for `noindex, nofollow` combined.	Current
`nosnippet`	No text snippet or video preview in the results listing.	Current
`indexifembedded`	Lets Google index content embedded via iframe even when the embedding page itself carries `noindex`. Added 2022.	Current
`max-snippet:[number]`	Caps the text snippet at this many characters.	Current
`max-image-preview:[setting]`	Caps image preview size: `none`, `standard`, or `large`.	Current
`max-video-preview:[number]`	Caps the video snippet at this many seconds.	Current
`notranslate`	Don't offer a translation of this page in results.	Current
`noimageindex`	Don't index the images on this page.	Current
`unavailable_after:[date/time]`	Stop showing this page in results after the given date.	Current
`noarchive`	Formerly hid the "cached" link. That link no longer exists at all.	Historical, unused
`nocache`	Alias for `noarchive` in older documentation.	Historical, unused
`nositelinkssearchbox`	Formerly suppressed a sitelinks search box in results.	Historical, unused
`noodp`	Tied to the long-defunct Open Directory Project. Not documented by Google today.	Not current, absent from Google's docs

The noarchive row deserves a closer look, because it's the one most likely to be wrong on a page you already trust. Google retired the cached-page feature entirely in early 2024, and by the time Search Engine Journal reported it on October 2, 2024, Google's own docs already said plainly that the rule "is no longer used by Google Search to control whether a cached link is shown in search results, as the cached link feature no longer exists." Several well-known SEO guides still list noarchive as live, nearly two years after that report. noodp is the older cousin of the same problem: not deprecated, just absent from Google's docs entirely.

The mistake that makes noindex do nothing

This is the one section worth reading twice on any site with inherited robots.txt rules. A robots.txt Disallow line and a noindex meta tag look like they cooperate. They don't. Google's own Robots Meta Tags Specifications state the mechanism directly: "If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored." The crawler never opens the page, so the noindex tag inside it might as well not exist.

The consequence isn't subtle. Google's guide to blocking indexing with noindex (updated December 10, 2025) states it plainly: for noindex to work, the page "must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler." Skip that, and the page "can still appear in search results" as a bare URL with no title and no description, indexed purely because something else links to it. Present enough to look sloppy, blocked enough to hide the reason why.

If a noindex page still shows up in Search, check robots.txt first

Per Google's own documentation, a robots.txt block means "the crawler will never see the noindex rule, and the page can still appear in search results." Fix order matters: let Google crawl the page and read the tag, confirm the drop on recrawl, and only block in robots.txt afterward if it's purely for crawl budget.

The diagram below shows that sequencing as two lanes: block first and the tag never gets read, or crawler first and it does.

Run your own robots.txt through a checker before blaming Google for a page that won't budge; our free Robots.txt Tester shows which rule matches a given URL. A related but separate mistake: putting noindex directly inside robots.txt never worked. Google stopped supporting that syntax in 2019, and per Bing's documentation, Bing never supported it there at all.

Meta tag, X-Robots-Tag, or robots.txt: three different jobs

These three get used interchangeably in conversation and they shouldn't be, because each one controls a different layer and lives in a different place.

Mechanism	Controls	Lives in	Can target
Meta robots tag	Index eligibility, snippet behavior	HTML `<head>`	HTML pages only
`X-Robots-Tag`	Same directives as the meta tag	HTTP response header	Any resource, including non-HTML files
`robots.txt`	Crawl access (whether the crawler may request the URL at all)	Server root file	Any resource, by URL pattern

The X-Robots-Tag uses the identical directive vocabulary described above, per the same Google specifications page, and exists specifically for files with no <head> to hold a tag: a PDF report, a scanned image, a video file. Send X-Robots-Tag: noindex as a response header, and that's the only way to keep a non-HTML file out of results. Of the three, it's the one most site owners never touch, since it needs server or CDN configuration rather than a line of HTML.

The same split shows up with hreflang: an annotation that lives either in the HTML or the HTTP header, depending on whether the resource even has an HTML head to hold it. We covered that duality, and its reciprocity requirement, in our guide to hreflang in sitemaps versus HTML tags.

Setting up robots.txt correctly avoids the whole problem upstream

Most noindex-versus-robots.txt conflicts trace back to a robots.txt file assembled by hand, months apart from whoever adds meta tags page by page. Our free Robots.txt Generator builds the file from explicit allow/disallow rules, which at least makes what's blocked obvious before a conflicting noindex tag lands on top.

Where crawl budget and indexing rules actually meet

Robots.txt is a crawl-budget lever, stopping the crawler from wasting time on URLs you don't want fetched. Noindex is an indexing lever: it lets the crawler in, then keeps the page out of results anyway, the right call when a page needs to stay crawlable for its internal links but shouldn't rank. Mixing up which lever fits which job is how a crawl-budget block silently overrides an indexing rule added later. The mirror-image version turns up in sitemaps too: our piece on sitemap validator errors that actually hurt indexing covers a URL disallowed in robots.txt yet still listed in the sitemap, the same contradiction in reverse.

Google ran an entire "Robots Refresher" blog series in early 2025 re-explaining exactly this ground. Nothing in the mechanics changed. A developer relations team doesn't write a multi-part refresher for a problem nobody has.

Frequently asked questions

Does adding noindex remove a page from Google immediately?

No. Google has to recrawl the page, see the tag, and process the removal, with no published fixed timeframe. Requesting indexing in Search Console can speed up the recrawl, but removal still depends on Google's own queue.

Can I combine multiple directives in one meta robots tag?

Yes, comma-separated in one content attribute, for example content="noindex, nofollow" or content="noindex, max-snippet:-1". none is shorthand for noindex plus nofollow together.

Is noarchive really dead, or does it still work at other search engines?

At Google, yes, tied directly to the cached-page feature it retired in early 2024. We found no current documentation from other major engines describing active noarchive support either, though we can't rule out some smaller crawler still reading it somewhere.

If name="robots" and name="googlebot" conflict, which one wins?

Google applies the more restrictive of the two. This mostly matters when a CMS-wide default gets overridden by a stricter per-page tag.

Does putting noindex inside robots.txt work as a shortcut?

No. Google stopped supporting that syntax in robots.txt in September 2019, and Bing never supported it there at all. Use the meta tag or the X-Robots-Tag header instead; robots.txt only ever controls crawl access, never indexing.

How do I check whether robots.txt is silently blocking a page I've set to noindex?

Run the URL through a robots.txt checker and see which rule matches it. If a Disallow line covers it, Google's crawler never reaches the page and never sees the noindex tag, no matter how correctly it's written.

Technical SEO