[woklet]

Trying to import a very large bbpress 1.2 forum to the latest. As I understand it, the converter essentially reads through all topics, replies and forums and creates a new post of type “forum” in the main WP database.

My issue is that this forum has approx. 520k topics and many, many more replies. The furthest the import process has managed to get has been 67k posts imported before timing out or dying in some unshown way. This is running on a dedicated server with 4GB RAM which runs nothing else.

Short of me rewriting the converter so it’s pure scripting and doesn’t run in WP, is there some way for me to speed this up and make sure it doesn’t die? To recap, it’s died in the following ways:

1) got to 67k, ended in ———- : AFAIK from reading, this is due to login timeout. I’ve solved that by auto refreshing the dashboard page

2) Says “starting conversion” hangs there indefinitely : logging that I’ve put in place indicates that inserts are happening but nothing is coming through on the WP log window. This eventually dies as well.

Any advice would be great – I’m not sure if @netweb is still around but if yes, please help 😀 I’ve read all your posts to other people on this sorta topic.

[Stephen Edgar]

1. You should be sorted by doing what your doing and refreshing (in another browser tab)

2. This is also usually an authentication issue, though this time with the MySQL database, you’ll usually only get this error though if you haven’t refreshed this page recently before starting of an import.

You should also use the “development version” aka bbPress 2.6 alpha, theres some new bbPress 1.x to 2.x import improvements, you can get this from the following page or direct link:
• https://wordpress.org/plugins/bbpress/developers/
• https://downloads.wordpress.org/plugin/bbpress.zip

[woklet]

Thanks – I’ll take a look at those, perhaps run it in parallel on another machine to see which is going faster. I won’t fiddle with the current box just in case. Current projections show it’ll take around 5.5 days to complete which is a bit rough but the client is happy with it so I’ll let her ride 🙂

It’s quite a process from what I can see – basically converting each bbpress post to a wp post and then adding post metadata and working out all the relationships. I may write a script to run it for the hell of it while I’m waiting.

[Pascal Casier]

Hi Woklet, any improvement in the current way of working is greatly appreciated !
Pascal.

[Stephen Edgar]

Try also dropping the “delay”, I typically run it as 0.1 (you can change this “on the fly” if you’ve already kicked it off 😉

Also if your MySQL server can handle it, bump the “rows” to 1,000.

With or without the changes above it should be no where near 5.5 days, I’d expect somewhere it should do 500k on a “typical” machine in about 6-8 hours based on others historical imports.

[Stephen Edgar]

And, make sure you have no other plugins active when you do the import, a recent case of an import taking forever was because the plugin was hooking pre_get_posts and as such every single post as it was being imported was being passed through this plugin making the import ~20x slower.

[woklet]

Makes sense – I’ll have to check it. It’s currently shunting along at 5k “rows” and 0.2 delay

[woklet]

With all plugins disabled it’s much faster – still taking a long time though. Been running for approx. 24 hours and has processed 200k replies.

Machine is:

4gb RAM (2GB dedicated to MySQL, 1GB dedicated to apache)
60GB SSD

CPU graphs show it sitting at around 50% at all times so it’s not maxing out there and the disk isn’t stressing either. Very interesting to keep an eye on.

[woklet]

So I’m at this again for a different forum. Just as huge though 🙂

Any hints as to tweaking MySQL or anything to make this run faster? Currently sitting at about 3 post imports per second which means it’ll be about 84 hours before it’s done. Crazy.

@netweb – any insights? 😀

Machine has same spec as before and should be a monster.

[Stephen Edgar]

Make sure you are using Twenty Fifteen, or Twenty Sixteen and no other plugins activated, 3 a second sounds a little slow, like my reply here, sorry about not getting to it sooner, as said 84 hours have now passed did the import finish?

I’ve really got nothing else except what has already been discussed here in this topic.

[woklet]

Thanks! 🙂 I figured it’d been a while since I last posted so who knows? Maybe something would have popped up in your head 😀

I actually started it off in a totally clean wordpress this time around so I know that’s not the bottleneck. It really does just seem to take this long – I setup a new logging mechanism to watch the rate of insert statements and the highest I got it to with tweaking was ~6.7 p/s with 3500 posts and 0 seconds wait between.

It’s really quite odd because as far as I can see, it basically changes the posts to WP posts and populates the post metadata. It checks to see if this is a sticky etc and that’s about it. I’m wondering if a purely SQL based approach would work or if that’s too complex.

[Stephen Edgar]

Yeah, a pure MySQL will always be better, for bbPress 1.x imports both the topics and replies steps of the imports have a table join that is currently required for topics to determine the 1st post and ignore replies, then for replies to ignore the topic.

I’ve got some ideas on how we can eliminate this join which would speed things up significantly but that will have to wait a bit as it involves refactoring the entire process, that then means refactoring the other ~24 importers to also work. At the moment this process is all manual, try this, try that, and that makes it incredibly time consuming. Hopefully not to far down the road I can automate much of this with bbPress’ new testing setup much of the manual work can be eliminated and iterating the importers can happen alot faster.

[woklet]

Believe it or not, still whacking away at this one 🙂

I’ve managed to get the bloody thing to convert but now the issue is running the scripts to repair the forums. Is there some way I can do this programmatically or through wp-cli or something? I’m hitting memory limits even if I set PHP’s memory limit to 2048M and above.

[Stephen Edgar]

There is no wp-cli commands sadly, it’s a great idea though :+1

Running each tool one by one is the only way at this stage, run each of the “low” impact tools first.

Do not run the “Reply menu order” repair tool though, this is pretty broken at the moment

[woklet]

It’s the “Recalculate the position of each reply” tool at the moment that’s messing us around. I presume it’s the 500k or so replies giving issues.

[Stephen Edgar]

Yes, sorry, “Recalculate the position of each reply” is the one I meant, don’t worry about running it. 1) It will take forever and 2) bbPress 1.x never supported “threaded replies” which is what its trying to repair.

[woklet]

Hey again,

OK, we’re getting somewhere but now the task that fails is: “Remap existing users to default forum roles” and the various counts don’t appear to have updated. I still see last activity as being 4 years ago and forum post counts at zero

[woklet]

OK, got that task to run by fiddling around with the structure of the foreach.

Counts are still all screwy though. The tasks run fine when updating post counts, last activity etc but nothing changes.

[woklet]

Hmm, looks like a broken import. Dammit. That means I’ll have to run it again! Some topics don’t have replies in them.