[Rob Beschizza]

Hello. I’m trying to import a PHPFox forum into bbPress.

The PHPFox database has been installed locally, as has WordPress/BBpress, on my own desktop.

The conversion proceeds at a fair clip until it gets to importing the replies (i.e. the bulk of the posts), at which point it slows badly. I’ve tried it with various Rows Limit settings (between 10 and 1000) and there doesn’t seem to be an appreciable difference in overall speed. I’ve tried various mysql configuration settings, too, including the recommendations of mysqltuner.pl — no appreciable difference there, either. The tables have been optimized.

In about half an hour, it’s gotten through about 4500 rows. As it’s pulling from a 535,500-row database table, it’ll take about 60 hours!

Is this to be expected given the size of the database? Any hints to getting this done faster?

[Rob Beschizza]

The desktop is an i7 mac, with 32GB of RAM. So should be up to the job generally.

[Stephen Edgar]

Ahmmmm…. It looks like you have the majority of things taken care of.

You could drop the delay time down to 0.1 from 1 which overall is only a marginal performance increase.

The main bottleneck is typically hard disk drive performance, when doing imports like you are I try to have the MySQL database running on an SSD drive or as you have plenty of RAM setting up a RAM drive also works really well albeit a little trickier as you need to switch your database in and out of permanent storage.

I also haven’t had the opportunity to do any large PHPFox imports so I don’t have much of a benchmark to compare with.

[Rob Beschizza]

Everything’s on an SSD.

If you’d like to experiment on a large PHPFox import, you’re more than welcome to this 800MB SQL dump of one 🙂

[Rob Beschizza]

One thought: the notification window on the import is filling up with warnings like this, presumably one for every processed row.

Notice: Trying to get property of non-object in /Library/WebServer/Documents/eq/wp-includes/post-template.php on line 29

Notice: Trying to get property of non-object in /Library/WebServer/Documents/eq/wp-content/plugins/easy-add-thumbnail/easy-add-thumbnail.php on line 37

Notice: Trying to get property of non-object in /Library/WebServer/Documents/eq/wp-content/plugins/easy-add-thumbnail/easy-add-thumbnail.php on line 42

Notice: Trying to get property of non-object in /Library/WebServer/Documents/eq/wp-includes/post-template.php on line 29

Could these be slowing it down?

[Rob Beschizza]

Disabling plugins (other than bbpress) prevents those notices from appearing. Doesn’t seem to have an effect on the speed of the operation, alas.

[Stephen Edgar]

This thread here mentions a vBulletin conversion of ~600,000 posts took around 8-10 hours on an iMac so thats 1/6th of the time you’ve estimated.

vBulletin to bbPress questions

I’d love to test the DB you have, maybe some improvements can be made though I doubt they’d be done before your import completes 😉 I just followed you on Twitter so follow me back and drop me a DM with a ftp/dropbox link and I’ll give it a whirl.

[Rob Beschizza]

Done!

I can also report that switching the databases to a ramdisk is quite easy and has sped things up quite a bit. Though my earlier report was a little optimistic — it was going to take a couple of weeks. With the ramdisk, though, it really does seem to be headed for a few days work.

Overnight, it got to 30,000 of 500k rows, but then it started showing hyphens instead of “Converting replies” in the reporting windows, Like so:

------------------------------------------------------------------------
------------------------------------------------------------------------------------
------------------------------------------------------------------------------------
----------------------------------------------
Converting replies (26750 - 26999)
Converting replies (26500 - 26749)
Converting replies (26250 - 26499)
Converting replies (26000 - 26249)
Converting replies (25750 - 25999)

The hyphens are added a a rate of one per delay time, by the looks of it, at the top.

A peek at the database has it at _bpp_converter_step 10 still, and _bpp_converter_start at 27000

Has the conversion croaked or is this part of the process? Should I manually increment _bpp_converter_start a little and restart the convertion? (I had to do that a couple of times as it was converting topic titles)

[Rob Beschizza]

Update! The wp-posts table isn’t getting any larger, so my guess is that the conversion is unable to proceed and that the hyphens are the result of an error.