mediumTechnical SEO

Conflicting Meta Robots and X-Robots-Tag Directives

Conflicting Meta Robots and X-Robots-Tag directives occur when a page's HTML meta robots tag and its HTTP X-Robots-Tag header provide different indexing instruc

By Seoxpert Editorial · Published Jan 1, 2024 · Updated Apr 27, 2026

Why it matters

Search engines rely on clear, unambiguous signals to determine how to index and display your pages. When the meta robots tag and the X-Robots-Tag header conflict, search engines may choose the most restrictive directive, which can result in pages being excluded from search results or indexed against your intentions. This undermines your SEO strategy and can harm site visibility.

Impact

Conflicting directives can cause important pages to be deindexed, block crawling of resources, or create inconsistent search engine behavior. This can reduce organic traffic, cause loss of ranking, and make it difficult to manage site indexing at scale.

How it's detected

Detection typically involves crawling the site and comparing the meta robots tag in the HTML with the X-Robots-Tag HTTP header for each page. SEO auditing tools, custom scripts, or manual inspection using browser developer tools or curl commands can reveal discrepancies.

Common causes

Server-level X-Robots-Tag overridden at page level
CDN injecting headers independently of CMS
Misconfigured server or CMS plugins
Legacy directives left after site migrations
Multiple teams managing headers and HTML independently

How to fix it

Ensure that only one mechanism is used per deployment layer (either meta robots in HTML or X-Robots-Tag in HTTP headers, not both). Audit your server, CDN, and CMS configurations to identify and remove redundant or conflicting directives. Align all indexing instructions to provide a single, consistent signal to search engines.

Code examples

Conflicting meta robots and X-Robots-Tag example

<!-- In the HTML -->
<meta name="robots" content="index,follow">

# In the HTTP response header
X-Robots-Tag: noindex, nofollow

Corrected: Consistent directives (HTML only)

<!-- In the HTML -->
<meta name="robots" content="noindex, nofollow">

# No X-Robots-Tag header sent

Corrected: Consistent directives (Header only)

# No meta robots tag in HTML

# In the HTTP response header
X-Robots-Tag: noindex, nofollow

FAQ

What happens if meta robots and X-Robots-Tag directives conflict?

Search engines typically honor the most restrictive directive, but behavior can be unpredictable. This may result in pages being excluded from search results or indexed in ways you did not intend.

How can I check if my site has conflicting robots directives?

You can use SEO auditing tools, browser developer tools (to inspect HTTP headers and HTML), or command line tools like curl to compare the meta robots tag and the X-Robots-Tag header for each page.

Should I use both meta robots and X-Robots-Tag on the same page?

It's best practice to use only one mechanism per deployment layer. Use meta robots in HTML for page-level control and X-Robots-Tag in HTTP headers for server or resource-level control, but avoid using both on the same page unless they are aligned.

Can CDNs or proxies introduce conflicting X-Robots-Tag headers?

Yes, CDNs or reverse proxies can inject or override X-Robots-Tag headers independently of your CMS or server configuration, leading to conflicts if not properly managed.

Which directive takes precedence: meta robots or X-Robots-Tag?

Generally, search engines follow the most restrictive directive, regardless of whether it comes from the meta tag or the HTTP header. However, implementation details may vary between search engines.

This issue occurs when a page is marked with both a noindex directive and a rel="canonical" pointing to a different URL. This creates conflicting signals for se

high

Found this issue on your site?

Run a scan to see if Conflicting Meta Robots and X-Robots-Tag Directives affects your pages.

Scan my website →