[_ck_]

You want a better spider tool that can limit how many threads it uses, obeys robots.txt and uses a standard user agent.

Good luck getting all that into bbpress though. First you’ll need to write a custom parser that can standardize the posts and topics into a clean format. Some was working on a bbPress xml importer but if you get the data organized enough you could parse it directly into mysql. Not a trivial project by any means but I guess you realize that .

Hopefully you know PHP or some other language that can help you parse all the data?