Hi there,
First of all I am not a hardcore coder. Used to code 20 years ago, so I can read a lot of the syntax, but am quite new to the javascript, php etc. Can handle myself a bit in sql though.

I am trying to migrate a very big MyBB forum to bbPres (1,7 mlj posts and 63k discussions). Tried to do it directly on my VPS, but it took way too long. Ages…

After some trial and error I was able to make a complete dump of the MyBB as SQL and imported it on my Mac’s MAMP. I was one happy muppet and kicked of the conversion.

Whatever I do I keep getting the same messages. No error messages, just that there is nothing to convert. I have tried every scenario I could think of that could go wrong. Localhost port, userid/pass on database, prefix etc etc. Whatever I do, even leave blank fields to get some kind of error message, I still get the same result. Absolutely nothing to convert.

I’ve checked the code but could not find any error handling. Could be there, but I am not too familiar on linking php code files.

I get the feeling this is some kind of MAMP issue because I got it to run on my VPN.

Any help / suggestions are more than welcome.

Thanks in advance!

[@netweb Keymaster 10 years, 4 months ago]

As I don’t own a Mac, yet, I think this is the issue in that your MAMP setup is using port 8888:

https://bbpress.org/forums/topic/bbpress-2-3-now-available/page/3/#post-133305

The only difference between XAMPP and MAMP (free) is that XAMPP uses “localhost” host name, while MAMP (free) uses “localhost:8888″ when using MAMP ports by default (MAMP pro uses “localhost” that tell why it works OK with you). So I went to MAMP preferences and changed the ports settings to use default Apache and MySQL ports to use “localhost” instead of “localhost:8888″, and finally posting topics and replies is available again 🙂

Even better I setup a virtual hostname manually in MAMP (free) and the everything works fine.

I hope this will help others with the same issue with MAMP (free). Thanks for your time

[@ezyridah Participant 10 years, 4 months ago]

Hi Stephen,
Thank you for your tip! That was the problem. Turned on the conversion. Very curious on what the results will be 🙂 Thanks again!

Cheers

[@netweb Keymaster 10 years, 4 months ago]

Excellent, let me know how it goes, haven’t seen a MyBB that big be imported, yet 🙂

[@ezyridah Participant 10 years, 4 months ago]

A lot of trial and error. Have restarted it a couple of times. Running it on my Macbook Air with SSD and i7 and MAMP but it is taking days.

#3 test conversion is progressing slowly. Already running for almost 2 days. Now done more than 50%. Doing 10.000 records at a time, with 0.1 delay time. No issues so far. Not sure if I can boost it more. The restarts seem to work ok too.

Will keep you posted 😉 Will be really glad when we get rid of MyBB! Fingers crossed 😉

[@netweb Keymaster 10 years, 4 months ago]

Cool, it is quite a few and the only real way to boost the performance is more resources, back with my IT hat on, separate hard drive spindles, preferably SSD (or RAM drives) for OS, source DB, destination DB and lots of RAM…

[@ezyridah Participant 10 years, 4 months ago]

Hi Stephen / @netweb

I was finally able to run a complete conversion without any restarts. The MyBB SQL of the database was 435,9 Mb big. We cleaned a lot of the old forum. It contained 1 million+ messages, close to 40k topics and a 400+ members.

The conversion ran for roughly 48 hours on a MacBook Air with i7, 8Gb memory and a 500Gb Solid State drive under MAMP.

For people who want to do such a big conversion, I also had to tweak a lot of the PHP ini files to import such a big query and get maximum throughput. Below a summary. It took me a while to find all the necessary settings.

Copy /applications/mamp/library/support-files/my-huge.cnf and rename to /Applications/MAMP/conf/my.cnf
Change in my.cnf setting [mysqld]max_allowed_packet = 1000M
Change in /Applications/MAMP/bin/php/php5.5.10/conf/php.ini settings
max_execution_time = 259200,
max_input_time = 259200,
memory_limit = 6144M,
upload_max_filesize = 1000M,
post_max_size = 1000M,
mysql.connect_timeout = 1000.
Add or change in Applications/MAMP/bin/phpMyAdmin/config.inc.php setting $cfg['ExecTimeLimit'] = 0;
Don’t forget to restart MAMP.

I ran the conversion with the following settings: Rows Limit 10,000 and Delay Time 0. During the conversion my memory was around 50% and CPU was very low. Most likely it is possible to squeeze out more performance.

I did a lot of random checks and the outcome seems ok-ish. Not quite sure about some things and have some questions. Please bear with me 😉

Some topics are not combined correctly. All replies are separate topics. The ‘Recalculate the parent topic for each post’ does not fix this. I ran all repair queries btw. An option in the Topics interface to combine these, would be ace 😉 Is it possible to correct this with a query? My SQL skills are too limited unfortunately.

The Repair Forums option ‘Recalculate the position of each reply’ does not work. It creates a white screen. Load problem?

myBB accepts spaces in the login name. Unfortunately WordPress does not. The conversion leaves the spaces in the login name. Is it possible to correct this with an SQL? Is there another workaround? Correcting them one by one is not my preference 😉

Two forums are created double. The copies do not contain any topics or replies, so it is easy to remove them. Just thought you should know 😉

The original myBB forum contained 1mlj+ messages and 36k+ discussions. The converted forum contains 109k subjects. That can be explained because of the first issue. But it only contains 830k replies. It could be that myBB adds subjects and replies to messages, but the difference is a bit too big. Is it possible to do an SQL checksum to check the expected output?

Thanks in advance for your time!

[@netweb Keymaster 10 years, 4 months ago]

Sorry for the delay, life has been getting in the way.

Thanks for listing those tweaks you made, they make sense, we already bump the PHP timeout and up the memory limit to 256M but that is all we can really do safely before we cause grief in shared hosting environments.

I linked this post from our Import Troubleshooting docs so when I get the time we can expand that section (or a dedicated section) on tweaking ‘all the things’.

It doesn’t look I have asked this previously, what version of MyBB are you importing from?

All my tests have been with MyBB v1.6.10

Some topics are not combined correctly. All replies are separate topics. The ‘Recalculate the parent topic for each post’ does not fix this. I ran all repair queries btw. An option in the Topics interface to combine these, would be ace 😉 Is it possible to correct this with a query? My SQL skills are too limited unfortunately.

I just ran an MyBB import of my ‘standard’ MyBB database and without running any repair tools replies are correctly associated with their parent topics.

As to the possibility of correcting this with a MySQL query, it could be, it’s hard to say without knowing what data is mismatched.

Run the following query in phpMyAdmin:
– 83 is my topic ID
– 97,99,100 and 102 are the reply ID’s I know should be part of the topic ID 83
– A single topic and a couple of replies should be all I need to see what’s going on

SELECT * FROM wp_posts WHERE ID IN (83,97,99,100,102)

Once you’ve executed the query, click ‘Print View’ and the results should open in a new browser tab, select all the text and paste them into a GitHub Gist and select SQL from the language dropdown (You don’t need an account for this)

https://gist.github.com/

It should look similar to this (this is based on my above screenshot)
https://gist.github.com/ntwb/04ed1cf9cf5f67528fcb

The Repair Forums option ‘Recalculate the position of each reply’ does not work. It creates a white screen. Load problem?

Yes, sadly, this is because of the size of the forums, kind of a know issue so that is good, sortof. Anyway the best thing is is that you do not need to run this tool. This is for only repairing ‘threaded replies’ which MyBB did not have so you can just skip running this tool. I’ll update the verbiage for this text in the next release and make some improvements when used with ‘large’ sites and some error handling.

myBB accepts spaces in the login name. Unfortunately WordPress does not. The conversion leaves the spaces in the login name. Is it possible to correct this with an SQL? Is there another workaround? Correcting them one by one is not my preference

I’ll work out a fix for that shortly, second time I’ve hit this issue today, @Halo-DieHard also did on BuddyPress.org.

Two forums are created double. The copies do not contain any topics or replies, so it is easy to remove them. Just thought you should know

That shouldn’t be the case, though as you point out easily fixed, most likely one of two things, either those forums were still there from a previous import OR there is something funky with the forum categories versus the forum forums though checking my import I just did, everything is fine for me 😉

The original myBB forum contained 1mlj+ messages and 36k+ discussions. The converted forum contains 109k subjects. That can be explained because of the first issue. But it only contains 830k replies. It could be that myBB adds subjects and replies to messages, but the difference is a bit too big. Is it possible to do an SQL checksum to check the expected output?

It does sound a little off, some forum software combine the total topics and replies to create a total post count.

On my MyBB I have:

Our members have made a total of 22 posts in 10 threads.
We currently have 6 members registered.

After importing using the [bbp-stats] shortcode I have
Topics 10, Replies 12

So that works out correct in my case, total of 22 posts = 10 topics + 12 replies 🙂

It’s hard to match up yours because the issue above as you state, ball park 109k + 830k = 93k which is a few short of the 1m, this will be due to those issues above and we can take another look after they’re fixed.

[@ezyridah Participant 10 years, 3 months ago]

Hi Stephen / @netweb,
Many thanks for your replies! I am trying to convert MyBB 1.6.11. It is a very old forum which (unfortunately) was customized improperly by the previous owner. It was lagging many many versions. I paid a student to do an upgrade about a year ago, but there is some funky **** still in the PHP. With my limited knowledge of MyBB I even tried to a fresh install of MyBB and imported the complete forum in it. Pulled all my hairs out when I found out MyBB stores code in the database…

I’ve stressed my dear wife to start cleaning the old forum, which she’s (finally) diligently doing now.

Don’t worry too much about making any updates on the code 🙂 I also have some kind of life and job next to trying to convert a MyBB forum 😉 After all the cleaning of the old forum, I will do another test run to see if I can find any other issues. It is very likely that some errors are just because it is an ancient forum (10 years old). Read up on the conversion of @Halo-DieHard. Will make sure I do a repair/optimization of the old forum. To be honest, I am not sure if I did that with the last conversion, sorry.

I will report back on the forum on what I was able to squeeze out on performance with MAMP, with what settings and the results.

Another (most likely dumb) question I forgot to ask. This could be the answer to the missing(?) topics/replies. At the end of the conversion, the program shows in short intervals ‘-‘ on the screen. This takes forever and lasted almost as long as the initial part of the conversion which grabbed the Rows. I thought the conversion was finished when the ‘-‘ stopped appearing on screen ‘————‘ etc. Was I supposed to wait for some kind of other message which indicates the conversion ended?

In the meantime I am busy integrating the plugin USERPRO and GD bbPress Toolbox. Never coded in PHP and already hate the syntax 😉 I will get there and report back on the results!

Thanks again for your time!

[@netweb Keymaster 10 years, 3 months ago]

Another (most likely dumb) question I forgot to ask. This could be the answer to the missing(?) topics/replies. At the end of the conversion, the program shows in short intervals ‘-’ on the screen. This takes forever and lasted almost as long as the initial part of the conversion which grabbed the Rows. I thought the conversion was finished when the ‘-’ stopped appearing on screen ‘————’ etc. Was I supposed to wait for some kind of other message which indicates the conversion ended?

This is basically WordPress losing the connection with the MySQL database and is a right pain, once this occurs “nothing” is now happening, the import is essentially stopped.

So in answer to your question, most likely this is the result of the missing topic/replies.

So yes, at the end of the conversion, you’ll see “Conversion Complete” with a link to the repair tools.

With most of the databases I have had to test to get compatibility with the importers many of the sample databases I have been sent (I don’t have a license for most of the commercial forum packages) I make a copy of the database and then prune/delete all but ~10 topics, the topics replies, the topic and reply users. This makes testing a breeze as it completes in a couple of minutes, once I know all is good then I give it a whirl on the full database.

[@ezyridah Participant 10 years, 3 months ago]

Thanks for your reply! That explains a lot. Strange it happened, but I will check it again with the next trial run. Will keep you posted 😉

[@ezyridah Participant 10 years, 3 months ago]

Hi Stephen,

Been working on other stuff on the site and told the wife to start cleaning the old forum. Kill all the ancient history.

I was able to do a successful trial conversion last weekend. Took roughly 9 hours to complete 600k of records. I tried to process more records per tranche but that didn’t seem to work. The process seems to hang in the step where the forum hierarchy is determined. Going back to 10,000 records and delay time 0 worked out ok. When I do another test run, I will try to keep track of more details.

Another thing I found out the hard way, is to kill all addons before doing a conversion. Makes sense, but stupidly enough I didn’t think of it. Perhaps something to put into your manual for conversions.

I am now ironing out the iffy things to get various plugins running smoothly with bbPress: UserPro, bbPress toolbox, a little ‘like’ addon and a little addon for polls.

Will keep you posted 😉

[@bepyland Participant 10 years, 3 months ago]

I have a limit of 50000 query per hour, do you know how many query are required for each row? With 10 row every 2 secs i always going in overflow

[@netweb Keymaster 10 years, 2 months ago]

@bepyland If you set the bbPress import rows to 50,000 that would be one single query, it would not work though unless you had one amazing MySQL installation. 😉

What the “rows” value means is how many topics or replies for each single query should be imported at a time, the default is 100, so when the importer is running each single query will import 100 topics at a time.

WordPress itself, and then bbPress also make queries along the way so just counting only bbPress import is not the only calculation needed.

Your best bet is to create a test site to test the import before migrating this to your “live” site:
https://codex.bbpress.org/creating-a-test-site/