Skip to:
Content
Pages
Categories
Search
Top
Bottom

Control what page text web crawlers can see?

  • If using a theme where pages contains subjects and user names of other pages, using Google to find pages in the forum will give lots of false hits. Like if I use in Google

    site:terapisnack.com social fobi knet99

    this will not only give pages with my nick about the subject, but also lots of false hits on pages where I at the time the crawler was there, I was listed in the section “Senaste inläggen” (most recent posts) at the bottom of the page.

    So how is this solved, and are there mutiple options how to solve this?

    I assume one would be to have a separate “clean theme”, that only contains what we want the web crawler to see, and switch theme on user agent? Is this support built into bbPress, or is there a plugin that can do this?

    Or is there some meta tag that can control what part of a page a crawler will ignore? I.e. just change the existing theme some?

    Or some other solution?

    Any advice is welcome!

    knet99

You must be logged in to reply to this topic.