@indrekhirsnik sorry I don’t know the answer to that one, and the bbPress import guru is I believe away at the moment.
No problems.
Just experimenting anyway in a test environment.
Managed to speed up the process a little.
Currently converting at the speed of 130 000 rows an hour vs previous 95 000 rows/hour.
Do not know exactly but the slow speed might have to do with the fact that in the old DB post/reply content is stored in the DB in a serialized array and i added a function to unserialize all the data during the import.
For some reason at 2 million post rows a big drop in speed, statistics show that MySQL command ratio dropped by three times compared to normal.
Had to stop/start the converter to get back to the normal speed.
Yes, all of what you state above will be a factor and it simply comes down to resource management, allocate a ton of RAM for MySQL, put each database on separate hard disk drives, make these SSD’s if you can, if you have some serious RAM available put the MySQL databases into a RAM drive (only marginally quicker than SSD).
I also set the MySQL delay between rows to 0.1.
Another suggestion is to not worry so much about getting ‘everything’ imported, make a copy of your database and trim it down to 1,000 topics and 5,000 replies (Along with forums, categories, users etc) and verify all the data and get your custom importer working 100% and then do your full imoport. Waiting 24 hours for 3 million posts to be imported then finding out everything is mis matched tends not to bring much joy 😉
The custom importer is already tested and everything is working properly.
The odd thing is that after 2 million rows the converter slows down noticeably.
Doing a stop/start for the conveter only helps for 10 minutes or so to get the speed up again.
Also i can see, that memory/cpu usage went down noticably after 2m, so a lack of resources is not a problem here.
That is odd that it slows down circa the 2 million mark, I’ve only performed an import of just over a million posts so I’m not sure what could cause this.
An option would be to split the database in half (with no doubt some very creative SQL queries) and perform the import in two passes…
Either way no matter what watching and waiting for ~2 million records to be converted from one database to another is little fun to watch.
Conversion is done.
On the last one i used 10000 rows, 0.1 sec delay.
Did a total of 3 conversions, the third one succeeded.
The first two conversion failed because i started getting ———– lines.
Split the conversion into two, 1.5m and 1.5m. Topics that haven’t been replied to since 2012 will get imported at a later date.
Took about 9-10 hours.
Awesome…
(The ———– issue is typically login credentials expiring in WordPress between WordPress and MySQL)
