Skip to:

Control what page text web crawlers can see?

  • knet99


    If using a theme where pages contains subjects and user names of other pages, using Google to find pages in the forum will give lots of false hits. Like if I use in Google social fobi knet99

    this will not only give pages with my nick about the subject, but also lots of false hits on pages where I at the time the crawler was there, I was listed in the section “Senaste inläggen” (most recent posts) at the bottom of the page.

    So how is this solved, and are there mutiple options how to solve this?

    I assume one would be to have a separate “clean theme”, that only contains what we want the web crawler to see, and switch theme on user agent? Is this support built into bbPress, or is there a plugin that can do this?

    Or is there some meta tag that can control what part of a page a crawler will ignore? I.e. just change the existing theme some?

    Or some other solution?

    Any advice is welcome!


  • You must be logged in to reply to this topic.
Skip to toolbar