Skip to:
Content
Pages
Categories
Search
Top
Bottom

Import to bbPress from custom forum


  • Indrekkor
    Participant

    @indrekhirsnik

    Hello,

    I am importing my forum to bbPress.

    It is a no-name software created by 4-5 persons during 10 years and it is seriously lacking in every aspect so switching the software makes a lot of sense.

    What are the most optimal and fast settings to use when importing my data ?

    Right now i am using Rows Limit 10 000 and Delay Time 1 second.

    It has taken 18 hours to convert ~1 650 000 rows and it is still going on.

    The forum is large, ~600 000, topics, ~2 400 000 replies in 18 forums.

    Best Wishes

Viewing 7 replies - 1 through 7 (of 7 total)

  • Robin W
    Moderator

    @robin-w

    @indrekhirsnik sorry I don’t know the answer to that one, and the bbPress import guru is I believe away at the moment.


    Indrekkor
    Participant

    @indrekhirsnik

    No problems.

    Just experimenting anyway in a test environment.

    Managed to speed up the process a little.

    Currently converting at the speed of 130 000 rows an hour vs previous 95 000 rows/hour.

    Do not know exactly but the slow speed might have to do with the fact that in the old DB post/reply content is stored in the DB in a serialized array and i added a function to unserialize all the data during the import.

    For some reason at 2 million post rows a big drop in speed, statistics show that MySQL command ratio dropped by three times compared to normal.

    Had to stop/start the converter to get back to the normal speed.


    Stephen Edgar
    Keymaster

    @netweb

    Yes, all of what you state above will be a factor and it simply comes down to resource management, allocate a ton of RAM for MySQL, put each database on separate hard disk drives, make these SSD’s if you can, if you have some serious RAM available put the MySQL databases into a RAM drive (only marginally quicker than SSD).

    I also set the MySQL delay between rows to 0.1.

    Another suggestion is to not worry so much about getting ‘everything’ imported, make a copy of your database and trim it down to 1,000 topics and 5,000 replies (Along with forums, categories, users etc) and verify all the data and get your custom importer working 100% and then do your full imoport. Waiting 24 hours for 3 million posts to be imported then finding out everything is mis matched tends not to bring much joy 😉


    Indrekkor
    Participant

    @indrekhirsnik

    The custom importer is already tested and everything is working properly.

    The odd thing is that after 2 million rows the converter slows down noticeably.

    Doing a stop/start for the conveter only helps for 10 minutes or so to get the speed up again.

    Also i can see, that memory/cpu usage went down noticably after 2m, so a lack of resources is not a problem here.


    Stephen Edgar
    Keymaster

    @netweb

    That is odd that it slows down circa the 2 million mark, I’ve only performed an import of just over a million posts so I’m not sure what could cause this.

    An option would be to split the database in half (with no doubt some very creative SQL queries) and perform the import in two passes…

    Either way no matter what watching and waiting for ~2 million records to be converted from one database to another is little fun to watch.


    Indrekkor
    Participant

    @indrekhirsnik

    Conversion is done.

    On the last one i used 10000 rows, 0.1 sec delay.

    Did a total of 3 conversions, the third one succeeded.

    The first two conversion failed because i started getting ———– lines.

    Split the conversion into two, 1.5m and 1.5m. Topics that haven’t been replied to since 2012 will get imported at a later date.
    Took about 9-10 hours.


    Stephen Edgar
    Keymaster

    @netweb

    Awesome…

    (The ———– issue is typically login credentials expiring in WordPress between WordPress and MySQL)

Viewing 7 replies - 1 through 7 (of 7 total)
  • You must be logged in to reply to this topic.
Skip to toolbar