[vogelsang]

I am working on getting a drupal forum migrated to bbpress. I am using the bbpress importer tool. The forum I am migrating has about 4000 topics and 40000 replies. The migration works fine for a few topics, but seems to get stuck at random places after about a 1000 records are migrated. This makes me think that it is not a data problem. More likely some sort of timeout.

Have any of you experienced something similar? I am running things locally with the most recent stable versions of WP and bbpress.

Here is my script. It is not well-tested. Any suggestions for improvements are welcome and feel free to use it as part of bbpress in whatever way you feel like.

<?php

/**
 * Implementation of Drupal Forum converter.
 */
class Drupal extends BBP_Converter_Base
{
	function __construct()
	{
		parent::__construct();
		$this->setup_globals();
	}

	public function setup_globals()
	{
		/** Forum Section ******************************************************/

		// Forum id. Stored in postmeta.
		$this->field_map[] = array(
			'from_tablename' => 'taxonomy_term_data', 'from_fieldname' => 'tid', //TODO correct ID??
			'to_type' => 'forum', 'to_fieldname' => '_bbp_forum_id',
		);
		
		// Forum parent id.  If no parent, than 0. Stored in postmeta.
		//$this->field_map[] = array(
		//	'from_tablename' => 'forum', 'from_fieldname' => 'parentid',
		//	'to_type' => 'forum', 'to_fieldname' => '_bbp_parent_id'
		//);
		
		// Forum title.
		$this->field_map[] = array(
			'from_tablename' => 'taxonomy_term_data', 'from_fieldname' => 'name',
			'to_type' => 'forum', 'to_fieldname' => 'post_title'
		);
		
		// Forum slug. Clean name.
		$this->field_map[] = array(
			'from_tablename' => 'taxonomy_term_data', 'from_fieldname' => 'name',
			'to_type' => 'forum', 'to_fieldname' => 'post_name',
			'callback_method' => 'callback_slug'
		);
		
		// Forum description.
		$this->field_map[] = array(
			'from_tablename' => 'taxonomy_term_data', 'from_fieldname' => 'description',
			'to_type' => 'forum', 'to_fieldname' => 'post_content',
			'callback_method' => 'callback_null'
		);
		
		// Forum display order.  Starts from 1.
		$this->field_map[] = array(
			'from_tablename' => 'taxonomy_term_data', 'from_fieldname' => 'weight',
			'to_type' => 'forum', 'to_fieldname' => 'menu_order'
		);
		
		// Forum date update.
		$this->field_map[] = array(
			'to_type' => 'forum', 'to_fieldname' => 'post_date',
			'default' => date('Y-m-d H:i:s')
		);
		$this->field_map[] = array(
			'to_type' => 'forum', 'to_fieldname' => 'post_date_gmt',
			'default' => date('Y-m-d H:i:s')
		);
		$this->field_map[] = array(
			'to_type' => 'forum', 'to_fieldname' => 'post_modified',
			'default' => date('Y-m-d H:i:s')
		);
		$this->field_map[] = array(
			'to_type' => 'forum', 'to_fieldname' => 'post_modified_gmt',
			'default' => date('Y-m-d H:i:s')
		);

		/** Topic Section ******************************************************/

		// Topic id. Stored in postmeta.
		$this->field_map[] = array(
			'from_tablename' => 'forum_index', 'from_fieldname' => 'nid',
			'to_type' => 'topic', 'to_fieldname' => '_bbp_topic_id'
		);
		
		// Forum id. Stored in postmeta.
		$this->field_map[] = array(
			'from_tablename' => 'forum_index', 'from_fieldname' => 'tid',
			'to_type' => 'topic', 'to_fieldname' => '_bbp_forum_id',
			'callback_method' => 'callback_forumid'
		);
				
		// Topic author.
		$this->field_map[] = array(
			'from_tablename' => 'node', 
			'from_fieldname' =>  'uid',
			'join_tablename'  => 'forum_index',
			'join_type'       => 'INNER',
			'join_expression' => 'ON node.nid = forum_index.nid',
			'to_type' => 'topic', 
			'to_fieldname' => 'post_author',
			'callback_method' => 'callback_userid'
		);	

		// Topic content.
		// Note: We join the posts table because topics do not have content.
		$this->field_map[] = array(
			'from_tablename'  => 'field_data_body',
			'from_fieldname'  => 'body_value',
			'join_tablename'  => 'node',
			'join_type'       => 'INNER',
			'join_expression' => 'ON field_data_body.revision_id = node.vid', 			
			'to_type'         => 'topic',
			'to_fieldname'    => 'post_content',
			'callback_method' => 'callback_html'
		);		
				
		// Topic title.
		$this->field_map[] = array(
			'from_tablename' => 'forum_index', 'from_fieldname' => 'title',
			'to_type' => 'topic', 'to_fieldname' => 'post_title'
		);
		
		// Topic slug. Clean name.
		$this->field_map[] = array(
			'from_tablename' => 'forum_index', 
			'from_fieldname' => 'title',
			'to_type' => 'topic', 
			'to_fieldname' => 'post_name',
			'callback_method' => 'callback_slug'
		);
		
		// Forum id.  If no parent, than 0.	
		$this->field_map[] = array(
			'from_tablename' => 'forum_index', 'from_fieldname' => 'tid',
			'to_type' => 'topic', 'to_fieldname' => 'post_parent',
			'callback_method' => 'callback_forumid'
		);

		// Topic reply count
		// $this->field_map[] = array(
			// 'from_tablename'  => 'forum_index',
			// 'from_fieldname'  => 'comment_count',
			// 'to_type'         => 'topic',
			// 'to_fieldname'    => '_bbp_reply_count'
			// //,
			// //'callback_method' => 'callback_topic_reply_count'
		// );
		
		// Topic date update.
		$this->field_map[] = array(
			'from_tablename' => 'forum_index', 'from_fieldname' => 'created',
			'to_type' => 'topic', 'to_fieldname' => 'post_date',
			'callback_method' => 'callback_datetime'
		);
		$this->field_map[] = array(
			'from_tablename' => 'forum_index', 'from_fieldname' => 'created',
			'to_type' => 'topic', 'to_fieldname' => 'post_date_gmt',
			'callback_method' => 'callback_datetime'
		);
		$this->field_map[] = array(
			'from_tablename' => 'forum_index', 'from_fieldname' => 'last_comment_timestamp',
			'to_type' => 'topic', 'to_fieldname' => 'post_modified',
			'callback_method' => 'callback_datetime'
		);
		$this->field_map[] = array(
			'from_tablename' => 'forum_index', 'from_fieldname' => 'last_comment_timestamp',
			'to_type' => 'topic', 'to_fieldname' => 'post_modified_gmt',
			'callback_method' => 'callback_datetime'
		);

		/** Tags Section ******************************************************/
		
		// Topic id.
		// $this->field_map[] = array(
			// 'from_tablename' => 'tagcontent', 'from_fieldname' => 'contentid',
			// 'to_type' => 'tags', 'to_fieldname' => 'objectid',
			// 'callback_method' => 'callback_topicid'
		// );
		
		//Tags text.
		// $this->field_map[] = array(
			// 'from_tablename' => 'tag', 'from_fieldname' => 'tagtext',
			// 'join_tablename' => 'tagcontent', 'join_type' => 'INNER', 'join_expression' => 'USING (tagid)',
			// 'to_type' => 'tags', 'to_fieldname' => 'name'
		// );		

		/** Post Section ******************************************************/

		// Post id. Stores in postmeta.
		$this->field_map[] = array(
			'from_tablename' => 'comment', 'from_fieldname' => 'cid',
			'to_type' => 'reply', 'to_fieldname' => '_bbp_post_id'
		);
		
		// Forum id. Stores in postmeta.
		$this->field_map[] = array(
			'from_tablename' => 'forum', 
			'from_fieldname' => 'tid',
			'join_tablename'  => 'comment',
			'join_type'       => 'INNER',
			'join_expression' => 'ON forum.nid = comment.nid',
			'to_type' => 'reply', 'to_fieldname' => '_bbp_forum_id',
			'callback_method' => 'callback_topicid_to_forumid'
		);
		
		// Topic id. Stores in postmeta.
		$this->field_map[] = array(
			'from_tablename' => 'comment', 'from_fieldname' => 'nid',
			'to_type' => 'reply', 'to_fieldname' => '_bbp_topic_id',
			'callback_method' => 'callback_topicid'
		);
		
		// Author ip.
		$this->field_map[] = array(
			'from_tablename' => 'comment', 'from_fieldname' => 'hostname',
			'to_type' => 'reply', 'to_fieldname' => '__bbp_author_ip'
		);	
			
		// Post author.
		$this->field_map[] = array(
			'from_tablename' => 'comment', 'from_fieldname' => 'uid',
			'to_type' => 'reply', 'to_fieldname' => 'post_author',
			'callback_method' => 'callback_userid'
		);
		
		// Topic title.
		$this->field_map[] = array(
			'from_tablename' => 'comment', 'from_fieldname' => 'subject',
			'to_type' => 'reply', 'to_fieldname' => 'post_title'
		);
		
		// Topic slug. Clean name.
		$this->field_map[] = array(
			'from_tablename' => 'comment', 
			'from_fieldname' => 'subject',
			'to_type' => 'reply', 
			'to_fieldname' => 'post_name',
			'callback_method' => 'callback_slug'
		);
		
		// Post content.
		// Note: We join the posts table because topics do not have content.
		$this->field_map[] = array(
			'from_tablename'  => 'field_data_comment_body',
			'from_fieldname'  => 'comment_body_value',
			'join_tablename'  => 'comment',
			// 'join_type'       => 'INNER',
			'join_expression' => 'ON field_data_comment_body.entity_id = comment.cid',
			'to_type'         => 'reply',
			'to_fieldname'    => 'post_content',
			'callback_method' => 'callback_html'
		);	
		
		// Post content.
		// BRUGES IKKE - SE OVENFOR
		// $this->field_map[] = array(
			// 'from_tablename' => 'post', 'from_fieldname' => 'pagetext',
			// 'to_type' => 'reply', 
			// 'to_fieldname' => 'post_content',
			// 'callback_method' => 'callback_html'
		// );
		
		// Topic id.  If no parent, than 0.
		$this->field_map[] = array(
			'from_tablename' => 'comment', 'from_fieldname' => 'nid',
			'to_type' => 'reply', 'to_fieldname' => 'post_parent',
			'callback_method' => 'callback_topicid'
		);

		// Topic date update.
		$this->field_map[] = array(
			'from_tablename' => 'comment', 
			'from_fieldname' => 'created',
			'to_type' => 'reply', 
			'to_fieldname' => 'post_date',
			'callback_method' => 'callback_datetime'
		);
		$this->field_map[] = array(
			'from_tablename' => 'comment', 
			'from_fieldname' => 'created',
			'to_type' => 'reply', 
			'to_fieldname' => 'post_date_gmt',
			'callback_method' => 'callback_datetime'
		);
		$this->field_map[] = array(
			'from_tablename' => 'comment', 
			'from_fieldname' => 'changed',
			'to_type' => 'reply', 
			'to_fieldname' => 'post_modified',
			'callback_method' => 'callback_datetime'
		);
		$this->field_map[] = array(
			'from_tablename' => 'comment', 
			'from_fieldname' => 'changed',
			'to_type' => 'reply', 
			'to_fieldname' => 'post_modified_gmt',
			'callback_method' => 'callback_datetime'
		);

		/** User Section ******************************************************/

		// Store old User id. Stores in usermeta.
		// $this->field_map[] = array(
			// 'from_tablename' => 'users', 'from_fieldname' => 'uid',
			// 'to_type' => 'user', 'to_fieldname' => '_bbp_user_id'
		// );
		
		// Store old User password. Stores in usermeta serialized with salt.
		// $this->field_map[] = array(
			// 'from_tablename' => 'users', 'from_fieldname' => 'pass',
			// 'to_type' => 'user', 'to_fieldname' => '_bbp_password',
			// 'callback_method' => 'callback_savepass'
		// );

		// Store old User Salt. This is only used for the SELECT row info for the above password save
		// todo ???
		//$this->field_map[] = array(
		//	'from_tablename' => 'users', 'from_fieldname' => 'salt',
		//	'to_type' => 'user', 'to_fieldname' => ''
		//);
				
		// // User password verify class. Stores in usermeta for verifying password.
		// $this->field_map[] = array(
			// 'to_type' => 'users', 'to_fieldname' => '_bbp_class',
			// 'default' => 'Vbulletin'
		// );
		
		// // User name.
		// $this->field_map[] = array(
			// 'from_tablename' => 'users', 'from_fieldname' => 'name',
			// 'to_type' => 'user', 'to_fieldname' => 'user_login'
		// );
				
		// // User email.
		// $this->field_map[] = array(
			// 'from_tablename' => 'users', 'from_fieldname' => 'mail',
			// 'to_type' => 'user', 'to_fieldname' => 'user_email'
		// );
		
		// User homepage.
		// $this->field_map[] = array(
			// 'from_tablename' => 'users', 'from_fieldname' => 'homepage',
			// 'to_type' => 'user', 'to_fieldname' => 'user_url'
		// );
		
		// User registered.
		// $this->field_map[] = array(
			// 'from_tablename' => 'users', 'from_fieldname' => 'created',
			// 'to_type' => 'user', 'to_fieldname' => 'user_registered',
			// 'callback_method' => 'callback_datetime'
		// );
		
		// User aim.
		//$this->field_map[] = array(
		//	'from_tablename' => 'users', 'from_fieldname' => 'aim',
		//	'to_type' => 'user', 'to_fieldname' => 'aim'
		//);
		
		// User yahoo.
		//$this->field_map[] = array(
		//	'from_tablename' => 'users', 'from_fieldname' => 'yahoo',
		//	'to_type' => 'user', 'to_fieldname' => 'yim'
		//);	
	}
	
	/**
	 * This method allows us to indicates what is or is not converted for each
	 * converter.
	 */
	public function info()
	{
		return '';
	}

	/**
	 * This method is to save the salt and password together.  That
	 * way when we authenticate it we can get it out of the database
	 * as one value. Array values are auto sanitized by WordPress.
	 */
	public function callback_savepass( $field, $row )
	{
		$pass_array = array( 'hash' => $field, 'salt' => $row['salt'] );
		return $pass_array;
	}

	/**
	 * This method is to take the pass out of the database and compare
	 * to a pass the user has typed in.
	 */
	public function authenticate_pass( $password, $serialized_pass )
	{
		$pass_array = unserialize( $serialized_pass );
		return ( $pass_array['hash'] == md5( md5( $password ). $pass_array['salt'] ) );
	}

}

[Trisha Salas]

Hi,

I did a large migration about a year ago (20,000 total topics and replies). You will definitely have timeout problems with an import that large, what I did was import items separately like so: users first then forum 1 topics, forum 1 replies, forum 2 topics, forum 2 replies and so on.

Of course back then there wasn’t an importer and we had to rely on custom scripts for all of it. The script was also run outside of WordPress/bbPress in an effort to stop timeouts. We also ran post-meta separately and did not attempt to transfer passwords (this was a migration from DNN/Microsoft SQL server so it wasn’t possible to retain passwords).

I am happy to help with specific question, I do agree it is a timeout issue.

[vogelsang]

Thanks for the answer.

My strategy will be to make the migration in chunks as you suggest. I will put the topics in different tables and have different scripts run each of them. It will take a bit of time to setup, but it should work fine.

[Trisha Salas]

Glad to hear it helped. Feel free to report back, I’d love to hear how it goes.

[Stephen Edgar]

I have just tweaked your Drupal Importer @vogelsang and updated it with a few tweaks for formatting and a couple of extra fields that you wern’t importing.

Details are in trac here https://bbpress.trac.wordpress.org/ticket/2375
(You can download my updated version directly from here)

There will be an FAQ over here https://codex.bbpress.org/import-forums/drupal

• Forum parents/hierarchy is now imported
• Topic/Reply counts are also imported to help out bbPress counts
• Open or Closed topic status is imported
• Drupal users are also imported BUT passwords need to be manually reset **

** See the FAQ and or Trac Ticket #2375 for details on the imported user passwords issue.

Lastly thanks a heap for doing 99% of the heavy lifting in matching the Drupal database fields, most excellent work on your behalf 🙂

As to the problems you are having with the importer stopping on you could you try to reduce the load on your server by changing the import values for ‘Rows Limit’ and/or ‘Delay Time’ to see if it is a server resource issue or if it stops at the same point there might be some funky data being imported that the importer is getting caught on during import.

Cheers,

Stephen

[vogelsang]

@netweb, I have run some tests on my site to-be-migrated and things seems to be working fine. Nice to have all the counting and stuff working up front 🙂

I think my problems with the importer is due to the data. It is some specific replies that stops the importer every time. Right now, I plan to just skip a reply when the importer stops. It’s possible by incrementing the options-table and then continue the importer from the web site afterwards. I am only missing 4-5 replies out of 40000 replies.

[pratip.ghosh]

Hi,

Is there a way so that bbPress can auto-migrate (as a scheduled job) using some sort of web service provided from Drupal? Is it possible?

Thanks,

[Stephen Edgar]

@pratipghosh No, the process cannot be automated, you need to set it up manually.