Control what page text web crawlers can see?
If using a theme where pages contains subjects and user names of other pages, using Google to find pages in the forum will give lots of false hits. Like if I use in Google
site:terapisnack.com social fobi knet99
this will not only give pages with my nick about the subject, but also lots of false hits on pages where I at the time the crawler was there, I was listed in the section “Senaste inläggen” (most recent posts) at the bottom of the page.
So how is this solved, and are there mutiple options how to solve this?
I assume one would be to have a separate “clean theme”, that only contains what we want the web crawler to see, and switch theme on user agent? Is this support built into bbPress, or is there a plugin that can do this?
Or is there some meta tag that can control what part of a page a crawler will ignore? I.e. just change the existing theme some?
Or some other solution?
Any advice is welcome!
You must be logged in to reply to this topic.