noindex and X-Robots-Tag on source pages

What the directive does

Google documents two ways to keep a page out of its results: a noindex value in a robots <meta> tag in the page head, or the same value sent in an X-Robots-Tag HTTP response header. When Googlebot next fetches the page and sees either one, Google drops the page from search results, regardless of how many other sites link to it. A <meta name="googlebot"> tag applies the same rule to Google only.

For a backlink owner this matters because the link lives on someone else's page. The anchor text and href can be unchanged while the page around it has quietly left the index. A check that only looks for the link in the HTML will report the placement as fine. That gap is the main reason a full backlink health check reads indexability signals, not just link presence.

Where noindex comes from on source pages

Publishers rarely add noindex to spite a buyer. The common causes are mundane:

• Template or plugin defaults. SEO plugins often noindex tag archives, author pages, attachment pages or paginated listings. A guest post that lands on one of those page types inherits the rule.
• Sponsored sections. Some sites move paid content into a section that is noindexed site-wide as a policy toward paid placements.
• Thin-content cleanups. A site owner pruning low-traffic pages may noindex old posts rather than delete them.
• Staging leftovers. A migration or redesign sometimes ships with a staging-wide noindex header still active.
• Header rules on a CDN or server. An X-Robots-Tag set in a server config can cover whole directories or file types without anyone editing the page.

The robots.txt interaction

Google states that noindex only works if the crawler can fetch the page. If robots.txt blocks the source page, Googlebot never sees the noindex rule, and the URL can still appear in results (usually without a description) if other pages link to it. Monitors therefore report the two conditions separately. A page blocked by robots.txt and a page carrying noindex are different problems with different fixes, which is why IndexChex keeps robots_blocked and not_indexable as distinct outcomes.

How a monitor detects it

A reliable check has to read three places on each fetch:

The header case is the one manual reviews miss most often. It is invisible in the browser, and it can apply to non-HTML resources such as PDFs, which some publishers use for whitepaper or press placements. Values like none also imply noindex, so a parser should treat none the same way.

The IndexChex backlink monitor records the robots meta values, the Googlebot meta values and the X-Robots-Tag values for each source page on each run, and reports a page with a noindex rule and a present link as not_indexable.

Reading the result

A noindex finding is a statement about the page, not about the link. The table below separates the cases a buyer usually has to explain:

A noindex directive also explains a mismatch you might see in source page indexation checks: the page was indexed when the link was bought, then fell out weeks later after a recrawl picked up the rule.

What to do

1. Confirm the directive on a fresh fetch, ideally with the header captured, so you are not acting on a one-off response.
2. Check whether the rule covers the whole section or only your page. A section-wide rule points to a site policy that the publisher is unlikely to reverse.
3. Contact the publisher with the URL, the directive and the detection date. Many fix template mistakes quickly once shown the header.
4. If the publisher refuses, record the placement as non-performing and follow the steps in recovering a lost backlink.

Monitoring cannot remove a noindex. Its job, as described in what backlink monitoring is, is to make sure you learn about it while the publisher relationship and any refund window are still open.