[icesar]

Hello, we integrated bbpress with our WP site at http://argentinastravel.com/forums/ The site ranks great for all kinds of search terms and gets indexed by Google daily, but for some reason the /forums/ are all being ignored entirely. Any ideas why this could be? Any other page / section that gets added is usually indexed within a day.

[chrishajer]

Do you have a sitemap that does not include the forums? Maybe Google is reading that and not looking any further?

This sitemap doesn’t contain anything about the forum at all:

http://argentinastravel.com/sitemap.xml

There was a sitemap plugin for bbPress at one time:

http://boakes.org/talk/topic/31

I have no idea if that works with current bbPress releases. I stopped using it a long time ago because generating the sitemap took too long.

I think that’s the problem though: Google knows only about the one site map and it doesn’t include anything about the forum. Other problems could be a restrictive robots.txt or a meta tag in the forum that restricts robots, but I didn’t see either of those.

I would add a sitemap for the forum and see if that helps.

[Graeme]

I noticed that your site has a sitemaps.xml. It may not be valid. You can use Google Webmaster Tools to submit and/or validate your google sitemap. I suggest you check it.

You will also find that there is a newer version of the google sitemaps plugin available (3.0.3) than the current one you are using (2.7.1). 2.7.1 may not generate correct sitemap for a wordpress 2.3.2 site. Downloading the updated plugin from the plugin’s site may help.

[chrishajer]

Will any of that help with indexing the forum though? My impression is that the WordPress sitemaps plugin generates a sitemap for WordPress content only, not bbPress.

[Graeme]

There is a facility in the plugin to enter additional URLs that will be generated into the sitemap. This is a way to point google at “/forums/” for a bbPress installation that falls under a wordpress main site. If the sitemap is not valid then I expect it is unlikely that the sitemap will be crawled.

There are also other tools around for validating sitemaps but the google webmaster tools would be the one to trust.

[howtogeek]

I use a sitemap for my main wordpress site only, and my forums are indexed in google… in fact Google is the main source of traffic to them.

I think it just takes a while for Google to get around to indexing the forum.

[icesar]

Hey all, thanks for all the feedback. I am using the sitemap plugin, and I did include the /forums/ in it.

I looked a little closer in Webmaster Central, and it turns out that the /forums/ section is sending a “404 not found” header!! That’s the actual problem.

What could be causing that?

[howtogeek]

The webmaster panel shows a 404 but your browser doesn’t? That really doesn’t make a heck of a lot of sense.

Is it possible that your forum has been down at any point? That would have caused the error.

curl -I http://argentinastravel.com/forums/

HTTP/1.1 200 OK

Date: Tue, 05 Feb 2008 20:46:10 GMT

Looks good to me.

[chrishajer]

A lot of times the 404 at Google Webmaster Tools is temporary. Have you tried it again?

[onecare]

Check your robots.txt, if you can, then exchange some links for http://argentinastravel.com/forums/

[chrishajer]

robots.txt looks OK.

http://argentinastravel.com/robots.txt

User-agent: *

Disallow:

[icesar]

Thanks for all the help so far guys. I think I know what the problem is now. Since the site is integrated with WP, I’m using the WP headers, which are responsible for throwing the 404 error even though the page generates fine.

The relevant discussion on this forum is here.

I still haven’t fixed the problem though. Should I give up on using a single header for both WP and BBPress? Is the plugin fix suggest the best bet? Other ideas?