[‎]

A few months ago I played around with popular SEO plugins. Unbeknownst to me, one of them created a sitemap that included over 50,000(!) individual topic reply pages, and Googlebot indexed over 20,000 of them…making my site run out of crawler quota.

I didn’t notice the problem until Google went from instantly indexing my articles to ignoring them until manually submitted.

Blocking the community/reply/* URL in robots.txt has little effect because it only blocks googlebot from visiting the URL format again – it doesn’t drop the existing indexed pages.

So I added this to a custom plugin:

add_action("wp_head", function ()
{
	if (substr($_SERVER['REQUEST_URI'], 0, 17) == "/community/reply/")
	{
		echo '<meta name="robots" content="noindex">';
	}
} );

Self-explanatory – adds the noindex to the header of reply pages. If you’ve named the slugs differently, obviously edit them and the substr length count.

Let me know if you’ve experienced the same issue and how you went about fixing it. And.. what’s the point of the reply page any way? It’s just duplicate content because the reply is already listed in the thread.

[‎]

*If you can confirm the reply pages haven’t already been indexed, just a robots.txt block will suffice.