Importing Large Xenforo Forum
-
Hi,
I am currently converting over a fairly large Xenforo forum to bbPress.
Something like 14,000 users – 41,000 topics – 250,000 replies.I managed to move everything over without much effort or problem; let it run overnight..
The problem I’m having is running repair.
I managed to run ALL repair functions (one at a time) except:
[Recalculate the position of each reply]Is this really required? I’m 6 hours into my 3rd attempt.
I keep having to adjust this or that, increase this or that, and raise Execution time, etc.
Even got a white screen a few times which I assumed was a timeout.My server has virtually unlimited resources. It has 64gb of memory and 12 cores I believe. It is divided up into two seperate VMs. 8GB Ram for the database server and the remaining for the web server. Only using SSD drives. Only 4 wordpress sites running on server.
I’ve read that some imports don’t require the [Recalculate the position of each reply] to be run but I don’t know if Xenforo imports require it.
Really appreciate some insight.
Thanks,
Tim Ehrenkaufer – Tomorrows Retail, LLC.
-
why did you avoid the Recalculate the position of each reply option??
and you might also have to split our import file into pieces if you haven’t already.
other than that @netweb maybe would chime in to help out.
or you could hire a developer to do this for you.
post a job at http://jobs.wordpress.net/ , if you do this route comeback with some information to hopefully help others.
I have tried my huge Xenforo data to convert data using BBPress import tool, my script is running since last 2 days, and I have imported below data using BBPress import tool.
Users: 94,600 out of 1,59,679
Forums: 0 out of 41
Topic: 83867 out of 83,813
Replies: 31,279 out of 62,65,110Now, I am facing below issue to import the data using BBPress tools.
1) Sometimes scripts are automatically stopped and script not start automatically.
2) Currently, lots of data is still pending to import in BBPress but the script is not running and in tools bottom section I can see only “Starting Conversion” and loader.
3) I have noticed in the tool having the duplication issue you can see above in Xenforo only 83,813 topics but BBPress tool imported 83,867 records. So 54 records are duplicates. so this issue resolved using BBPress repair forum tabs?
4) BBPress import tool taking random data from the Xenforo database. I have noticed when I have started the script first the users come from the Xenforo. But unfortunately, the users are skipped and the tool has started to import the topic. So, I am a bit confused here BBpress import tool can complete import all the users, topic, forums, and replies successfully? you can see above my total Xenforo user is 1,59,679 and BBPress tool has imported only 94,600. and the now script has started to import replies.
Can you please suggest here what is the best solutions to import all the data properly?
@ dots Nice size forum, heres a few things that I hope will help:
Firstly, can you try by using bbPress 2.6-alpha, you can find it here https://bbpress.org/download/
1) Sometimes scripts are automatically stopped and script not start automatically.
Sometimes things timeout, a workaround for this is to open another tab in your browser to example.com/wp-admin and refresh it occasionally
2) Currently, lots of data is still pending to import in BBPress but the script is not running and in tools bottom section I can see only “Starting Conversion” and loader.
This is similar to 1), by refreshing you should in another browser tab this should keep that issue at bay, this particular issue is usually when PHP drops the MySQL connection and needs refreshing to update the connection between WordPress and the database.
3) I have noticed in the tool having the duplication issue you can see above in Xenforo only 83,813 topics but BBPress tool imported 83,867 records. So 54 records are duplicates. so this issue resolved using BBPress repair forum tabs?
It could be duplicates, though I’ve only ever tested 3 or 4 Xenforo forums with bbPress, potentially using the repair tools might fix this though without some more information here it will be hard to take a guess. If you can supply a copy of a couple of duplicates I can take a closer look as to the reason why.
4) BBPress import tool taking random data from the Xenforo database. I have noticed when I have started the script first the users come from the Xenforo. But unfortunately, the users are skipped and the tool has started to import the topic. So, I am a bit confused here BBpress import tool can complete import all the users, topic, forums, and replies successfully? you can see above my total Xenforo user is 1,59,679 and BBPress tool has imported only 94,600. and the now script has started to import replies.
Again, it should do all of this correctly, though testing with bbPress 2.6-alpha per above will hopefully fix many, or most of the issues you are having, can you try using 2.6-alpha and let me know how that goes for you please 🙂
Also a couple more tips in our docs:
@netweb
Hi Stephen,Thanks for the updates. I have updated my BBPress to bbPress 2.6-alpha.
Now, I have again started the import using BBPress tools and now my script is importing the data again, But it will taking the duplicate data from the Xenforo. I hope this version can import all my Xenforo data to BBPress. But still I have more questions about the import.
1) In my Xenforo having total 62,65,110 replies and tool has imported only 66,100 records only. so bbPress 2.6-alpha version are able to import all pending 61,99,010 data from the Xenforo?
2) Could you please share your thoughts for how to remove duplication data? Is there any good solutions? For duplicate data reports/ screenshot We will come back after my client permission.
@dots I’d suggest setting up a test site to test your imports before importing into your live site to iron out any issues you come across. The best process is to “reset” bbPress which will remove/delete all your bbPress data, you can then start the import with a fresh/clean slate for bbPress.
I cannot guarantee that it will work perfectly importing everything though I really hope it does, as I stated in my previous reply there hasn’t been much testing with Xenforo so any feedback would be great to hear, any issues and I’ll try to implement those fixes.
Unless people test the importers and give us feedback I cannot improve them 🙂
Thanks for the updates.
For now, we are giving up using BBPress due to import issues between Xenforo and BBPress. So once the corrections in import script are made, we will try again later.
So far we have encountered below issues in Xenforo -> BBPress import, to sum up:
1) Sometimes scripts are automatically stopped and script not start automatically.
2) Currently, lots of data is still pending to import in BBPress but the script is not running and in tools bottom section I can see only “Starting Conversion” and loader.
3) After script stopped it will taking the import the data from beginning so lots of data duplications issue come.I have tried all the solutions as you suggested but none of them working in 2.6 alpha version. Still I am facing the above issues related to import the data using BBPress tool.
I’m testing with a conversion from xenForo to bbpress as well. I’m using the latest bbpress available to the public, 2.5.8. My xenForo site has about 33,000 topics, 330,000 posts and 6,000 users. Running the bbpress import feature seems to work without a hitch. Then when I go to repair, all functions work great except for the “Recalculate the position of each reply”, as another user here has mentioned. After several seconds, I get a 500 error.
After the import and running the repair items (except for the one that doesn’t work), the forums *seem* to all be intact and working well until you look a bit closer. Many users aren’t associated with their posts — they show 0 posts in bbpress even though they have thousands of posts in xenForo.
I may go ahead and try bbpress 2.6 alpha and see how it goes.
FWIW, I’m running both forums on the same VPS, so the import is relatively quick. It takes a few hours.
I reset the forum, then upgraded to bbpress 2.6 alpha, checked the “purge previous import” checkbox (just to be sure), then started the import process again and got this lovely error message:
“WordPress database error: [Specified key was too long; max key length is 1000 bytes]
CREATE TABLE blp_bbp_converter_translator ( meta_id mediumint(8) unsigned not null auto_increment, value_type varchar(25) null, value_id bigint(20) unsigned not null default ‘0’, meta_key varchar(255) null, meta_value varchar(255) null, PRIMARY KEY (meta_id), KEY value_id (value_id), KEY meta_join (meta_key, meta_value) ) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci”So I unchecked the purge option and tried again. It’s currently importing the users.
@wendellh Thanks for the info on that error, not sure why that error occured, nothing really changed between 2.5.8 and 2.6 in regard to this table creation. I’ll try to duplicate the issue later today.
The import completed and I’m still having the problem with users being associated with their posts. For example:
Original thread on xenForo: http://surveyorconnect.com/threads/converting-from-lat-long-clarke-spheroid-to-spc.326148/
Imported thread on bbpress: http://surveyorconnect.com/portal/index.php/forums/topic/converting-from-lat-long-clarke-spheroid-to-spc/
It seems the topic is connected ok, but not the replies.
Yes, exactly.
I’ve just got my copy of Xenforo up and running locally, I don’t have an issue with reply authors here, see this screenshot https://cloudup.com/c_epfx4Q4z8
Can you open up phpMyAdmin and run this SQL query on your Xenfor database:
SELECT convert(post.post_id USING "utf8mb4") AS post_id, convert(thread.thread_id USING "utf8mb4") AS thread_id, convert(post.thread_id USING "utf8mb4") AS thread_id, convert(post.user_id USING "utf8mb4") AS user_id, convert(post.message USING "utf8mb4") AS message, convert(post.post_date USING "utf8mb4") AS post_date FROM xf_post AS post LEFT JOIN xf_thread AS thread USING (thread_id) WHERE thread.first_post_id != post.post_idYou might have to change the following two lines if your database does not use
xf_as the forum prefix:From:
FROM xf_post AS post LEFT JOIN xf_thread AS threadTo:
FROM myprefix_post AS post LEFT JOIN myprefix_thread AS threadI’ve just added support to the XenForo importer to support deleted users.
https://bbpress.trac.wordpress.org/ticket/2922
If you update your 2.6-alpha via this zip file you’ll have that included for your import.
What this does is if a user has been deleted from XenForo their topics and replies are not actually deleted (they are kept to maintain context) but during import because that user no longer exists they cannot be imported and the topic/reply would then be attributed to the
anonymoususer. Now the topic/reply is still attributed to theanonymoususer but includes the original author name, basically the same way XenForo handled this.Edit: You can see the difference from when the
Member Twouser still existed and was imported per https://cloudup.com/c_epfx4Q4z8/f, I then deletedMember Twoand the result looks like this https://cloudup.com/cNnYv-3eSyl https://cloudup.com/c5iQ-iIOoW4I’ve just bumped 2.6-alpha with the above changes and also a fix for deleting imported users that wasn’t playing nice 😉
TBH, I’m a bit nervous about changing anything in the live xenForo database…
First in test, always !
Pascal.
I’m a bit nervous about changing anything in the live xenForo database…
There is nothing changed in the XenForo database, the query only “reads” data it does not write anything to the database 🙂
p.s. It still pays to have a backup 😉
I ran the query but I’m not sure what information you are looking for from the result.
Tried the import again and saw some results I hadn’t seen before. The last several lines of the conversion said something about converting anonymous posts. That’s the first time it’s ever done that and it seems to have worked great. 🙂
Now I just have to find a way to convert all of the xenForo bbcode into something that will work in bbpress (such as image attachments).
Thanks for the help!
Awesome 🙂
If you look at the phpBB and SMF importers they have near the end of the file a bunch of regular expressions for converting custom BBCode into native HTML for WordPress.
Get me a list of what you want copied from the SMF/phpBB importers to Xenforo and I’ll add them.
If you need more, or tweaks to them happy to help out here also.
p.s. I wanted you to run that query to see if it failed, if it did fail that would have meant you and I have different database schemas for our Xenforo databases.
o are the reply authors all correct now or still having issues with them?
Hey Stephen,
I really appreciate your help! 🙂
Probably the most noticeable issue after the import are the attachments. They are basically a number with opening and closing ATTACH bbcodes. For example:
[ATTACH=full]872[/ATTACH]
In xenForo, this links to attachments/photo-jpg.872Then there are the quotes to other posts. These simply just come in as text and the post IDs don’t match those in bbpress. In xenForo, the quoted posts link back to the post that was quoted. For example, here is an imported post with a quote:
http://surveyorconnect.com/portal/index.php/forums/topic/july-2015-photo-submittal/#post-1647874
and here is the original post in xenForo:
http://surveyorconnect.com/threads/july-2015-photo-submittal.323039/#post-331560I’ve also noticed that soft deleted posts (only viewable by mods and admins) in xenForo are coming in as public posts.
I realize I may be asking for too much, but I figure it doesn’t hurt to ask. 😉 I do understand though if it’s just not possible. I’ll keep looking around for other bbcodes as well.
Thanks!
Probably the most noticeable issue after the import are the attachments. They are basically a number with opening and closing ATTACH bbcodes.
I’m not quite sure what to do with attachments as bbPress import system does not yet support attachments ¯\_(ツ)_/¯
I could add some code to “hide” the BBCode whilst still keeping it as a refernce?
Also I’ve added support for YouTube, Daily Motion and Vimeo BBCode conversion
Then there are the quotes to other posts. These simply just come in as text and the post IDs don’t match those in bbpress. In xenForo, the quoted posts link back to the post that was quoted.
I’ve added some styling to the quotes now, so it wraps that quotes in “blockquote” and cites the authors name, e.g “wendellh wrote:”
For now we can’t link to the quoted post, this is on my radar to do but requires some pretty big changes, maybe bbPress 2.8.
I’ve also noticed that soft deleted posts (only viewable by mods and admins) in xenForo are coming in as public posts.
Ah, I didn’t know about these types of posts, I’ve now done this, for topics/threads replies/posts that were “soft deleted” these are now imported with bbPress/WordPress “pending” status.
I’m not quite sure what to do with attachments as bbPress import system does not yet support attachments ¯\_(ツ)_/¯
Yeah, I’m actually thinking it might be best to require the user (me) to manually move the images directory from the old site to the new site. Then, maybe just use the xenForo URL to the image to display it in the post. It “should” be safe since xenForo reduced the images based on admin settings when they were originally uploaded. Just some random thoughts from a guy not looking at import code. 😉
I’m wondering if https://bbpress.dev4press.com would actually cure some of the bbcode issues automatically.
Also I’ve added support for YouTube, Daily Motion and Vimeo BBCode conversion
Sweet!
I’ve added some styling to the quotes now, so it wraps that quotes in “blockquote” and cites the authors name, e.g “wendellh wrote:”
That’s perfect. The post and member numbers are really just useless information at that point anyway. I don’t see any real importance in providing the links back to the quoted posts either. I mean, it would be really cool, but not a big deal.
Ah, I didn’t know about these types of posts, I’ve now done this, for topics/threads replies/posts that were “soft deleted” these are now imported with bbPress/WordPress “pending” status.
Thank you!
- You must be logged in to reply to this topic.
