highTechnical SEO

Meta Robots and X-Robots-Tag Disagree on Indexability

Conflicting indexability directives in meta robots tag and X-Robots-Tag header cause search engines to deindex pages unintentionally.

By Seoxpert Editorial · Published May 4, 2026

Why it matters

When the HTML meta robots tag and the X-Robots-Tag HTTP header give different indexability instructions, Google will follow the most restrictive rule. This can result in pages being deindexed even if the meta tag suggests they should be indexed, leading to unexpected loss of search visibility.

Impact

Pages may be removed from search results despite appearing indexable in the HTML source.

How it's detected

A crawler compares the content of the meta robots tag and the X-Robots-Tag HTTP header for conflicting index/noindex directives.

Common causes

•Legacy server configuration setting X-Robots-Tag headers conflicting with updated HTML meta tags.
•Multiple teams managing SEO directives in different layers (server vs. frontend).
•Copy-paste errors or oversight during site migrations or redesigns.
•Automated tools or plugins adding conflicting directives.

How to fix it

Review both the HTML meta robots tag and the X-Robots-Tag HTTP header for each affected page. Ensure both sources use the same directive (e.g., both 'index, follow' or both 'noindex, nofollow'). For simplicity and clarity, use only one method—preferably the HTTP header for non-HTML resources, or the meta tag for HTML pages—and remove the other to avoid future conflicts.

Code examples

Conflicting directives (problem)

<!-- HTML meta tag -->
<meta name="robots" content="index, follow">

<!-- HTTP response header -->
X-Robots-Tag: noindex, nofollow

Aligned directives (fix)

<!-- HTML meta tag -->
<meta name="robots" content="index, follow">

<!-- HTTP response header -->
X-Robots-Tag: index, follow

Single source (fix)

<!-- HTML meta tag only -->
<meta name="robots" content="index, follow">
<!-- No X-Robots-Tag header -->

FAQ

Which directive does Google follow when meta robots and X-Robots-Tag conflict?

Google follows the most restrictive directive, typically 'noindex' if present in either location.

Should I use both meta robots and X-Robots-Tag?

It's best to use only one method per page to avoid conflicts—meta tag for HTML, X-Robots-Tag header for non-HTML resources.

How do I check for conflicting indexability directives?

Inspect both the HTML source for meta robots tags and the HTTP headers for X-Robots-Tag on each page.

Can this issue cause my pages to disappear from Google?

Yes, if 'noindex' is present in either directive, Google may remove the page from search results.

Found this issue on your site?

Run a scan to see if Meta Robots and X-Robots-Tag Disagree on Indexability affects your pages.

Scan my website →