Skip to:
Content
Pages
Categories
Search
Top
Bottom

Rather slow PHPFox conversion


  • Rob Beschizza
    Participant

    @beschizza

    Hello. I’m trying to import a PHPFox forum into bbPress.

    The PHPFox database has been installed locally, as has WordPress/BBpress, on my own desktop.

    The conversion proceeds at a fair clip until it gets to importing the replies (i.e. the bulk of the posts), at which point it slows badly. I’ve tried it with various Rows Limit settings (between 10 and 1000) and there doesn’t seem to be an appreciable difference in overall speed. I’ve tried various mysql configuration settings, too, including the recommendations of mysqltuner.pl — no appreciable difference there, either. The tables have been optimized.

    In about half an hour, it’s gotten through about 4500 rows. As it’s pulling from a 535,500-row database table, it’ll take about 60 hours!

    Is this to be expected given the size of the database? Any hints to getting this done faster?

Viewing 8 replies - 1 through 8 (of 8 total)

  • Rob Beschizza
    Participant

    @beschizza

    The desktop is an i7 mac, with 32GB of RAM. So should be up to the job generally.


    Stephen Edgar
    Moderator

    @netweb

    Ahmmmm…. It looks like you have the majority of things taken care of.

    You could drop the delay time down to 0.1 from 1 which overall is only a marginal performance increase.

    The main bottleneck is typically hard disk drive performance, when doing imports like you are I try to have the MySQL database running on an SSD drive or as you have plenty of RAM setting up a RAM drive also works really well albeit a little trickier as you need to switch your database in and out of permanent storage.

    I also haven’t had the opportunity to do any large PHPFox imports so I don’t have much of a benchmark to compare with.


    Rob Beschizza
    Participant

    @beschizza

    Everything’s on an SSD.

    If you’d like to experiment on a large PHPFox import, you’re more than welcome to this 800MB SQL dump of one :)


    Rob Beschizza
    Participant

    @beschizza

    One thought: the notification window on the import is filling up with warnings like this, presumably one for every processed row.

    Notice: Trying to get property of non-object in /Library/WebServer/Documents/eq/wp-includes/post-template.php on line 29

    Notice: Trying to get property of non-object in /Library/WebServer/Documents/eq/wp-content/plugins/easy-add-thumbnail/easy-add-thumbnail.php on line 37

    Notice: Trying to get property of non-object in /Library/WebServer/Documents/eq/wp-content/plugins/easy-add-thumbnail/easy-add-thumbnail.php on line 42

    Notice: Trying to get property of non-object in /Library/WebServer/Documents/eq/wp-includes/post-template.php on line 29

    Could these be slowing it down?


    Rob Beschizza
    Participant

    @beschizza

    Disabling plugins (other than bbpress) prevents those notices from appearing. Doesn’t seem to have an effect on the speed of the operation, alas.


    Stephen Edgar
    Moderator

    @netweb

    This thread here mentions a vBulletin conversion of ~600,000 posts took around 8-10 hours on an iMac so thats 1/6th of the time you’ve estimated.

    http://bbpress.org/forums/topic/vbulletin-to-bbpress-questions/

    I’d love to test the DB you have, maybe some improvements can be made though I doubt they’d be done before your import completes ;) I just followed you on Twitter so follow me back and drop me a DM with a ftp/dropbox link and I’ll give it a whirl.


    Rob Beschizza
    Participant

    @beschizza

    Done!

    I can also report that switching the databases to a ramdisk is quite easy and has sped things up quite a bit. Though my earlier report was a little optimistic — it was going to take a couple of weeks. With the ramdisk, though, it really does seem to be headed for a few days work.

    Overnight, it got to 30,000 of 500k rows, but then it started showing hyphens instead of “Converting replies” in the reporting windows, Like so:

    ------------------------------------------------------------------------
    ------------------------------------------------------------------------------------
    ------------------------------------------------------------------------------------
    ----------------------------------------------
    Converting replies (26750 - 26999)
    Converting replies (26500 - 26749)
    Converting replies (26250 - 26499)
    Converting replies (26000 - 26249)
    Converting replies (25750 - 25999)
    

    The hyphens are added a a rate of one per delay time, by the looks of it, at the top.

    A peek at the database has it at _bpp_converter_step 10 still, and _bpp_converter_start at 27000

    Has the conversion croaked or is this part of the process? Should I manually increment _bpp_converter_start a little and restart the convertion? (I had to do that a couple of times as it was converting topic titles)


    Rob Beschizza
    Participant

    @beschizza

    Update! The wp-posts table isn’t getting any larger, so my guess is that the conversion is unable to proceed and that the hyphens are the result of an error.

Viewing 8 replies - 1 through 8 (of 8 total)

You must be logged in to reply to this topic.