[Jonathan Stegall]

I’d love to hear if anyone has found a way to successfully convert the format of the passwords from phpBB to bbPress.

[paulhawke]

Where are we now?

Aside from the questions surrounding integration with WordPress, another of our big questions is “how do I import from _____” (insert a different forum product of your choosing)

Right now the approach is ad-hoc at best: We are handing a PHP script around hand-to-hand that is able to convert a phpBB v2.0.x forum over to bbPress v. 0.9, and if you want to convert a different forum (say, SMF) the accepted wisdom is to install phpBB, convert to that, then convert to bbPress 0.9 and then upgrade.

The official coding standards for WordPress state:

Avoid touching the database directly. If there is a defined function that can get the data you need, use it. Database abstraction (using functions instead of queries) helps keep your code forward-compatible and, in cases where results are cached in memory, it can be many times faster.

As it stands bbPress imports clearly violate that principle by using the script we are passing around.

Solving the problem

Importing data clearly is an example where a strategy pattern solves things in an elegant fashion.

The strategy pattern (also known as the policy pattern) is a particular software design pattern, whereby algorithms can be selected at runtime.

The strategy pattern is useful for situations where it is necessary to dynamically swap the algorithms used in an application. The strategy pattern is intended to provide a means to define a family of algorithms, encapsulate each one as an object, and make them interchangeable. The strategy pattern lets the algorithms vary independently from clients that use them.

More concretely, this means creating a bbPress Import plugin which provides an overall multi-step process, and delegates to an external sub-plugin to retrieve the external datasource specific steps. I would argue that this ought to be a plugin (and remain a plugin rather than being pulled into the core of bbPress) because once you’ve imported from a given external forum, you can deactivate and uninstall the “import” code as its no longer required.

Roadmap

1. We need to create a basic import plugin. — This I have done, it’s experimental and I’d like some more eyes on the code before calling it “done”
2. We need to create import strategies for external forums, starting with phpBB2 — This I have done, by extracting SQL from the script we all pass hand-to-hand. Again, it needs more eyes on the code as a QA step.
3. We need to create import strategies for other external forums, specifically SMF and phpBB3 — this is where the strategy pattern comes into its own – each can be developed independently and combined later. Its also a great place for community input.

Looking further out

The import plugin relies on calling the internal bb_new_post, bb_new_topic, bb_new_forum and to a certain degree the bb_new_user methods. As we move toward bbPress being a plugin to WordPress, these will become the only way to import data, as the script we pass around will become evermore obsolete.

Secondly, WordPress itself has an import framework. I suspect that any “import plugin” we write now will become somewhat obsolete in that timeframe. That said, if we do things right, the import strategies we create might migrate over. What I dont know is the timeframe. Is the bbPress-as-a-plugin-to-Wordpress change close enough on the horizon that we should simply target the WordPress import framework directly?

Thoughts / comments?

[David-]

I achieved this within 30mins or less. Here’s how to go about it:

The first thing you need to do is download the latest phpbb, and install on a subdomain (I name mine phpbb, like this: http://www.mysite.com/phpbb).

Then download the Smf 2.0 to Phpbb converter here: http://dicky.askmaggymae.com/files/SMF_20B3_to_phpBB3.zip

Now upload the zip file to the root directory where you installed phpbb (in my case: http://www.mysite.com/phpbb)

I suggest you upload the zip file directly, then extract from your cpanel file manager in order not to mess up with the folder structure.

Now point your browser to where you installed phpbb, in my case: http://www.yoursite.com/phpbb/install/index.php and then click on the convert tab, and a list of converters will come up, you’ll see smf 2.o listed there. Just continue from there following the instructions listed.

During the convert there’s a stage where phpbb would be asking for the path to your smf 2.0 board, simply enter it. In my case smf was on: http://www.mysite.com/forums, so I entered: ../forums. Initially my smf was installed at: http://forums.mysite.com, I kept getting errors, so I just moved it with my cpanel file manager to: http://www.mysite.com/forums. And the whole process went smoothly.

But if your smf is on the root, just enter this value there: ../

If you follow the above instructions, you should get your smf converted to phpbb without a glitch.

Now it’s time to convert to BBpress (the mighty bbpress).

Download the latest version, and install exactly where you want it to be, in my case I installed it on the root which is http://www.mysite.com. You must install bbpress using the same database with the phpbb you just converted.

Now download the phpbb to bbpress converter here: http://members.lycos.co.uk/wmnkhayal/files/phpbb3tobbpress%20converter.tar.gz

Extract the zip on your desktop, and you’ll end up with two files.

Upload the phpbb3tobbpress.php file to the root directory of your phpbb (in my case: http://www.mysite.com/phpbb)

copy the other file which is: _phpbb3_pass.php to your newly installed bbpress plugins directory. Copy this to bb-plugins folder and not to my plugins.

Now point your browser to where you installed phpbb, and add the link of the converter at the end, in my case this was: http://www.mysite.com/phpbb/phpbb3tobbpress.php and follow the instructions to convert your phpbb board to bbpress.

As simple as that. I’m no expert, so anybody should be able to do this. Anyways I’m available to questions.

Welcome to bbpress!

[David-]

I just converted from smf => Phpbb => bbpress, and the passwords aren’t working unless they request for a password reset.

Most members keep mailing to complain about this. I want to just send the whole thing to them once and for all.

[David-]

There was an error in my converted phpbb sql file, the error clearly pointed me to the right direction. So all I did was download the phpbb database, then I traced the error down to the line that was being pointed out, I deleted what I thought could be causing it, saved it and exported it back to the database.

As soon as I ran the converter script, it worked like a charm.

Are you having any problem? It didn’t take me more than 30mins to do all the conversions. I was initially on SMF, converted that to Phpbb, before finally converting to bbpress.

[kevinjohngallagher]

Kind of mate

In fact, I’m sure that a lot if not everything bbPress related could be done with plugins, but it comes to a point in time where a certain number of plugins are needed as standard for the forum to function properly.

I mean, “forum_is_category” and “topic_voices” are in the core now, but rather than being in the core tables, are still added and fiddled around with as meta data, so there’s always an INNER JOIN needed or additional SQL calls.

Call up a topic, in SQL, simple. How many Posts? there’s the data. How many tags? there’s the data. Heck, last poster’s name? sure! How many voices? oh hell no! I don’t keep that data here. It’s a secret!! Even though it’s a one to one relationship and it’s in the core, we keep that somewhere totally different, and you need to come up with a totally different method of getting that info. Ha, i mean, just cos it’s in the core, doesn’t mean it’s laid out prof… well we don’t use it on the WP support forums.

I’m not suggesting anything overtly Machiavellian at all, please don’t assume that and if I’ve given that impression I’m sorry (sometimes i jokes dont translate well), but if you give the data-structure to a DBA or heck, a basic developer they’ll scratch their heads. There is no really good reason for something like “forum_is_category” or “topic_views” to be held in a separate table as a data row, other than the one major forum where it’s not used. And that’s ok, he who pays the piper calls the tune, but that’s a core issue.

Realistically, thats not a coding issue, it’s a perspective issue. In order to get at the data, we need to call additional database information that we don’t need. Why do we need that overhead, even if it’s small.

Deep down, I can’t believe I need a plugin for my forum to be able to say it has posts underneath it though. That isn’t a bug and it isn’t intended, it just wasn’t thought of.

Same with Mass deleting of users / topics / posts. Or the ability to be able to add any form of ordering or filtering to any of them. Just wasn’t thought of.

I can understand a plugin with a specific purpose in this field (I’ve written one that marks any “.ru” and “.cn” user as bozo automatically), that’s cool. But I still think it’s daft that I have to manually search the user list every day for members with certain words in it – that stuff became bog standard on the internet a long time ago.

You know, once i search for (and find) a user, it then takes me 7 clicks and 4 different pages to delete the user. And there is no en-mass option. 7 clicks and 4 pages!!!! That’s not a ‘plugin or not plugin’ issue, that’s no PM/BA/IA/common-sense issue. I mean, really, having to leave the admin area to interact with a user? They got rid of that idea in wordpress almost 3 years ago.

I don’t want to flog a dead horse, but now that WordPress isn’t being ported to backPress (at least in the next version minimum), can you imagine the difference that rolling in a moderation suite or really any of _ck_’s non-fluff (smilies, bbcode) plugins into the core would have made in the 14 months between 0.9 and 1.0 being released? Hey has anyone used XML_RPC to post to their forums yet? it’s awesome…

=====================================================================

Is the concern that since the data design isn’t optimized for inherited forums, the query count will go up?

Yeah, kind of. It’s more that it’s already gone up, and as themes become more complex, it’ll go up again.

Let me give an example, and it’s a walkthrough – there could be code optimization.

To load the last forum poster/post (and btw this is for the singualar, not the child, which would multiply this by each child forum), you effectively are looking up the topic with the last date that’s attatched to that forum, and then look up the last post that’s attatched to that topic. Ok, so that’s 2 additional SQL calls for every forum thats displayed in the forum list.

(EDIT: actually, I’m confident it’s just 1 INNER JOIN SQL call now, but it’s still not needed really).

Now on a forum like this, or on WP support forums, thats only 6 forums. ok, so 6 additional calls total. Meh, not fussed.

But do you see how this doesn’t scale? On one of my forums, a football one, we’ve a “forum” for every football team in the UK. There’s no “latest” list, as we want people to talk in their own club’s section. Thing is that’s:

England

– Prem

– – 22 teams

– (Another 4 divisions of 24 teams)

Scotland

– SPL

– – 12 teams

– (Another 2 divisions of 16 teams)

It was facing meltdown every now and then, and I realised why, I’m placing an additional 150 SQL calls every time I want to display the list of forums, simply to find out the last post in a forum (and that’s NOT including the code needed if I was to respect genealogy).

Here’s the thing though, the TOPIC table has a column called: topic_last_post_id. Do you know why? Because it allows an inner join rather than an additional SQL call. Very sensible. Of course, it just wasn’t transposed to the bit that a certain support forums didn’t use.

If we applied the same logic to “forums” as to the topics, heck even put it in the meta data table, we’d add 1 SQL command in total when a new post was added, instead of 1 SQL per forum listed per pageload per user. Now, i could write a plugin to do that, but it’s not solving the issue, it’s just papering over the cracks in the datastructure.

I’m sure there will be some people reading this and they wont get that that i’m talking about data structure and optomization, and not specifically about “last forum poster”. I’m not advocaating category heavy forums or flat file forums, i’m saying that we’ve got some real data holes, and i think we should discuss them. How long until i’m told to leave bbPress because on some of my websites I use it differently to someone who has relatively few uesers and so isn’t hitting any of these issues? I bet not long.

I am not in any way having a go at any current/ Past developer/ development team/ Santa nor the Easter bunny. But the data is set up in a way in which it is ideally suited for 1 type of forum, and not for any of the others. Thats cool, but if bbPress is going to be a WPplugin, then this doesn’t matter. If there’s going to be a standalone version, then this sort of thing really needs to be looked at.

There’s countless examples of this; I’m just going with the one that people can replicate easily.

=========================================

I’d like to stress i’m not advocating a massive change to the core, honestly I’m just up for discussion. I’m really just wanting to know people’s feelings or thoughts on this. Far too much data is not readily/intrinsically/obviously available especially when said data is on a predefined 1-to-1 relationship.

Maybe if the data structure wasn’t so wishy-washy at points, there might be more plugin developers, or themes that looked a little more like the themes available on any of the other forums.

=========================================

John you’re also right that a “moderation suite” could be done via a plugin, my thoughts on that though would be “XML-RPC is a standard feature, but moderating my forum is a 3rd party plugin”?

I don’t think that making Forums read only on a selective basis is the work of a plugin, thats integral to how a forum works. Same with certain forums being visible to certain people.

The ability to move/sticky/resolved a topic/post from inside the admin area isn’t the job of a plugin, thats the job of the admin area. otherwise, what the point of the admin area

I’m sure there are things that I think should be standard that many other don’t; and vice versa. I don’t want bloatware, i’m not suggesting we copy phpBB or anything else, but i also think that just because we can have a plugin to do a job that we should always have a plugin to do a job. Also, just because something works with the current datastructure, doesn’t mean it’s even close to optimal.

=================================================================

Phew, longer than i wanted. Probably wrote many things which people will get annoyed at. Then again, i’m sure some idiot will be along in a minute demanding that we stop talking about this and include Facebook Connect, Polls, Smileys etc. I wonder how long that will take before my data structure conversation gets it’s first “ZOMG i cant code so ignore data structurz & make me XXX feature”.

[Navjot Singh]

Please share your solution David.

[Il Gatto]

I have many ehehe

from most important to least important:

1) Better BBCode (like PhpBB, VBulletin or any other forum) + Quoting (“user says”)

2) Better MULTILANGUAGE SUPPORT!!! (this would rock – like qtranslate in WP)

3) Advertising manager (a port of the popular Advertising Manager plugin in WP would do)

4) User profile pages (with many customizable profile fields, etc)

5) User photo galleries

6) Better moderation tools

7) Some social networking tool like Facebook connect, OpenID connect, Twitter feeds, etc.

Better Smilies

9) Better User Avatar System (Bavatars does a neat job though)

10) Better Integration with WordPress (sometimes sucks)

11) Customizable Forum icons

12) User can post Polls / Surveys

13) Event calendars…

[paulhawke]

I am currently working on a plugin that adds import capability to bbPress. The framework provides a 4-step wizard

1. Choose the input source
2. Configure the import
3. Verify what you’re about to import looks good
4. Perform the import

The “importers” for the first step are read dynamically, to allow for easy extension. The parameters names for step 2 come dynamically from the selected importer and the values entered are supplied to steps 3 and 4. The plugin provides methods for creating users, posts, topics and forums, the importer provides the methods needed to get that data from whatever the input source is.

My experiment was to import a fake phpBB 2 forum (3 users, 2 forums, and maybe a dozen or so posts). All looks good. I would value other sets of eyes on the project though. What would be awesome would be other people to write some importers for other platforms. The framework is there in the plugin, it just needs extension into the other platforms.

[David-]

I’ve fixed this error in the database file, and everything ran smoothly. Thanks a lot.

[David-]

Hello,

I’m getting an error during the convert.

General Error

SQL ERROR [ mysql4 ]

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘ve bought in the past or intend to buy.’, ’51’, ‘8’, ‘3’, ’26’)’ at line 1 [1064]

An SQL error occurred while fetching this page. Please contact the Board Administrator if this problem persists.

[tflight]

paulhawke, I’d like to give it a shot as I’m still several weeks away from a final cut-over. My two biggest pain points right now are preserving user passwords and preserving user email notifications… both of which are outside the scope of your project but issues I’m working on.

[bobtheman]

I would like to start discussion regarding a single sign on feature, and developing integration for other platforms other than just wordpress “joomla, drupal, etc” in order to open up the usability of bbpress and be less restrictive than just supporting WP.

conversion methods and tools for both CMS platforms and forum software conversion “preferably from “phpbb, smf, vb, vanilla” to bbpress of course ..

any ideas and suggestions are welcome

[Darfuria]

I think the only reason people are hanging on is because the vast majority of us are aware of the beauty of WordPress, and we’re convinced that bbPress can become a similar product – unfortunately there isn’t an existing piece of software that’s like the WordPress of forums. There isn’t a piece of software that has a simple core, that allows you to write an index.php file with ‘if has threads while has threads the thread’, and that is exactly what bbPress needs to become. I’m not bothered if it becomes a WordPress plugin, because WordPress isn’t really a lot of hassle to install, and I already know how to make a WordPress theme. I just want it to hurry up. I’m tired of working with phpBB3.

[yoyopop]

paul – I understand private messages aren’t part of the bbpress core, and I haven’t investigated all the possible plugins yet. If there could be an optional conversion that relates to a certain plugin, then of course that would be great. However, at the moment if I’m going to transfer my forum across, I was planning to inform my users before the move just to backup their PMs if they have anything important in them.

I just got a test conversion to work so will start to check out possible plugins etc next! Thanks for the help, hopefully bbpress can continue to grow!

[paulhawke]

@yoyopop – I have been looking at what is core to bbPress thus far. It hadnt even crossed my mind that private messages would want to be converted. But there’s a problem – if I convert them, what do I convert them to? Does the community feel that any one private messaging plugin should be endorsed over any other, and, does the private message plugin provide the code I need to create its raw data? If the answers are “yes” (or a ‘qualified yes’) then I’ve no problem working it in.

For those interested in beta testing, the plugin (at present) is grossly inefficient as far as memory goes and now that it works at a basic level, I’m looking at optimizing & reducing memory footprint.

[yoyopop]

tflight – I didn’t have rights to give php more memory on my server so I ran the php script locally on my machine using XAMPP, which successfully converted the database. Now just to replace the old db!

[yoyopop]

thanks tflight, I thought it could be something like that, I’ll give it another go!

paulhawke – this sounds great for the future, having a more user-friendly way to convert will open the door for a lot more people to use bbpress. Any plans to copy of private messages?

[paulhawke]

I have an experimental plugin for bbPress I’ve been working on this week that adds an “Import” wizard under the admin area “Tools” menu. As it stands I have been using a phpBB 2.0.23 forum with 3 users (“Anonymous”, “admin” and one other) and about a dozen posts for testing and things look pretty good.

The “Import” plugin needs beta testers who’re willing to give it a run through and provide feedback.

The plugin reads data from the phpBB database, and rather than generate an enormous SQL file that is version specific, the plugin calls the same internal methods inside bbPress as occur when you manually create forums/topics/posts/users. A mapping is maintained between old and new user/post/topic/forum ID values, and referential integrity is maintained throughout.

[tflight]

yoyopop, that happened to me the first few times as well. (I converted a database with about 120,000 posts this morning.) PHP was just running out of allocated memory. I bumped it up to 16 MB, then 64 MB, then 256 MB and finally 1 GB. After I gave it 1 GB the conversion went through.

So check your Apache error_log or your PHP error log to see if it is running out of memory… then give it as much as you can.

[yoyopop]

have been attempting this as well but getting the same problem as eslavin83

the script says it’s connected to the server and exporting the forums/users etc but nothing comes up after ‘Sending all your database e-mails to all major world-wide spam sending mafias…’ – I’m waiting for it to say its exported the new db, or give me an error message, but nothing else happens.

Any ideas?

thanks in adance!

[tflight]

I’d be interested to hear from anyone that has converted larger forums (100,000+ posts) from phpBB2 to bbPress. I’ve got 0.7 running in a dev environment, I’ve got the conversion script, but I haven’t fired it up yet. I’m going to take this slow and methodical. I also want to import user passwords and topic_watch (email subscriptions). Those I know I’ll end up having to do myself. I’m planning on temporarily storing phpBB passwords in a new table using two-way encryption as people login to the current site so that I can transfer those over when the time happens. I’ll also be working on my custom theme/templates before the conversion as well to make it as seamless as possible for users. Finally I might try to build a database of common words and do some auto tagging of old threads… that might be a bit too ambitious though.

Still… I’d be eager to hear from anyone that has converted a 100,000+ post site with the script for any info or tips.

[addotnet]

I’ve just converted my PHPBB3 forum to BBpress – everything seemed ok at first, the posts have converted and I was logged in and went to the admin panel. All of a sudden I am unable to log in, cannot reset a password and I apparently ‘do not exist’.

It appears that my user list hasn’t converted either as all my posts are labelled as made by anonymous users.

What can I do?

[kevinator]

We just imported nearly 20,000 posts from a phpBB forum to bbPress. The version of phpBB did not support tagging, so none of the posts/replies is tagged.

Is there a script or other plugin to search these posts/replies and assign a set of tags to them?

[mentalicgr]

@onthebeach96 I had the same problem as you with phpBB. I was getting bombed every hour by spam. When I installed BBPress I forgot to activate Akismet, and although spam users where registering they didn’t post anything (maybe because the bots weren’t ‘smart’ enough to think that the password was emailed at them). By activating Akismet and reCaptcha plugin no spam users have appeared.