I’ve spent the last week working with a client on a rather large WP/bbP setup. One of the real joys of this project is that it’s large enough for me to get some time in for more than asthetics and QA. Hence, a few more posts her this last week and some code, and some BackPress help.

Anyway, one of my junior developers came up with an interesting stat the other day that I’ve been looking into, namely Deep Integration and it’s hit/performance. I’m sure almost all of this will be obvious to a great many of you, but I thought I’d throw out some figures.

Single Loading of software after install

WordPress2.9.2		16 database queries

WordPress30			19 database queries

bbPress0.9			09 database queries

bbPress1.0.2			16 database quieres

Single Loading of software after setup

WordPress2.9.2		19 database queries

WordPress30			23 database queries ***1

bbPress0.9			10 database queries

bbPress1.0.2			18 database quieres

Now the first thing I noticed is that WP3.0 took a leap in queries. Once it takes you 20+ queries to load a simple homepage with no plugins my spider sense is tingling. Turns out that WP3.0 is somewhat sruggling with it’s new menu system. ***1 It takes 6 queries minimum, plus 1 query for each taxonomy type used in the new menu (apart from hardcoded links with don’t get an additional query). That’s a heck of a leap for something as simple as a menu. I mean, menu’s are rarely dynamic – they don’t change that often. For this particular website, as will probably be the way for the others I’ve moved to WP3.0, I’m looking at a minimum of 8 database queries simply to load the data needed to display the name and link in a menu. To be clear, that’s over 30% of database queries on a clean install of WordPress that are being called by this.

As cool as the drag/drop facility is, make every menu item a custom link and you’ll save a minimum of 2 db calls per page load.

Even better, hardcode the actual menu and save yourself 8 database calls per page load (you can hardcode an array for the walker class to iterate through and produce the same result)

Single loading of software after basic/core plugins:

Now, I believe that no WordPress (nor bbPress) install can run as expected these days without certain plugins. Your thoughts on this may vary ofcourse but I find that I need a minimum of WP-Super-cache, XML-sitemaps, WP-security-scan, wp-pagenavi, headspace/All-in-one, WP-stats, and probably a few others that I can’t remember off the top of my head. Oddly for bbPress I need more plugins.

WordPress2.9.2		23 database queries

WordPress30			27 database queries ***1

bbPress0.9			18 database queries

bbPress1.0.2			29 database quieres

That’s quite a leap. But I’m confident that I could bring down the bbPress Queries with some time spent considerably, and I could enable caching from WordPress. I’m not posting this data in itself as definative stats for everyone, so there could be optomization made for sure.

Deep Integration of WP3.0 and bbPress1.0.2:

WP3 & bbP1			56 database quieres

56! With virtually no caching availible (as a plugin, for the average user with no .ini or shell access).

56! On every page (+/- a few queries).

Deep Integration of WP3.0 and bbPress1.0.2 – after first pass:

WP3 & bbP1			43 database quieres

Ok some quick theme changes and delayed/circumvented loading of certain things and I’ve brought it down considerably.

But the fact still remains that this is a truly crazy amount of database queries per page load.

Anyway, after presenting this (in a better looking format) to the IT director of the client, I’ve managed to re-assign myself 2 weeks to come up with a planB. We want something that allows us the functionality of Deep Integration, but without the MASSIVE overhead. I have an idea down on paper, but right now I just wanted to post a few figures and give folks a heads up.

==================================================================================

Edit: If you want to see something fun, try this:

Add to your bb-config.php and your wp-config.php

In bb-config.php and wp-config.php add the following line:

define('SAVEQUERIES', true);

Then add the following code to your footer.php in your theme (right at the bottom)

wordpress footer

global $wpdb;

echo "[pre]";

print_r($wpdb->queries);

echo "[/pre]";

bbpress footer

global $bbdb;

echo "[pre]";

print_r($bbdb->queries);

echo "[/pre]";

*** replace the square brackets [] html brackets (they’re not showing up correctly)

[@gautamgupta Participant 14 years, 6 months ago]

Nice stats!

[@rebootnow Participant 14 years, 6 months ago]

[EDIT] Scratch the below. It seems I was only counting the bbpress-related queries. I now see 51 queries (for a logged in user), which is a lot to pay for the convenience of calling wordpress functions. I look forward to hearing more about what you come up with during those two weeks.

---

These stats are not consistent with my experience.

I have a site with deep integration between bbPress 1.0.2 and WordPress MU 2.9 (BuddyPress). The number of queries for the front page is:

18: logged out

23: logged in

I switched the theme back to Kakumei and this drops to:

14: logged out

19: logged in

The deep integration here is requiring ‘wp-load.php’ from bb-config. How did you do the deep integration to get the results above?

Or is WP3.0 doing something odd when integrated with bbP?

Also, although 56 queries is really heavy for a standard deep integration, I wouldn’t be overly concerned about that number for a sophisticated site. And you really want to know what those queries are. Not all queries are created equal.

[@rebootnow Participant 14 years, 6 months ago]

Ok, now I’m just confused. With our custom theme I am seeing 50+ queries. I was previously ignorant of this because “get_num_queries()” apparently isn’t returning a result for the site overall.

But on Kakumei, I still only see 19 queries overall.

I can see what optimization needs to be done on the custom theme thanks to the “SAVEQUERIES” magic incantation, but I’d still love to know why you’re getting 56.

Did you do the deep integration test on the standard themes?

[@rebootnow Participant 14 years, 5 months ago]

I have now (1) figured out how to measure the number of queries correctly for both sides and (2) removed some very silly overhead that was immediately revealed after I achieved (1).

The result is that both Kakumei and my custom theme do around 40 queries on the front page of the forum.

Apologies for all the noise in the two posts before this one — a great illustration of more haste less speed.

ps. One mod to Kevin’s extremely useful suggestion regarding the use of SAVEQUERIES to see where those queries are going… in bbPress you need to print the queries for BOTH the bbPress and WP databases. In other words, you need the following in your forum footer:

global $wpdb, $bbdb;

print_r($wpdb->queries);

print_r($bbdb->queries);

Or just to get the number of queries for both…

global $wpdb, $bbdb;

echo ‘WPDB Queries: ‘ . $wpdb->num_queries;

echo ‘BBDB Queries ‘ . $bbdb->num_queries;

Of course, the “>”s that appear in my code above should be greater than signs.

[@kevinjohngallagher Member 14 years, 5 months ago]

Nice one mate, and congrats on bringing down the overhead.

All data is good data!

[@_ck_ Participant 14 years, 5 months ago]

I’ve been talking about this for years.

The “performance” tag has a few gems in there I’ve written over time.

bb-benchmark will also show you all the queries via the same method and a few other details, unfortunately the newest version is not in the plugin section right now because of how bbpress.org is broken.

Part of the problem on the WP side is many times the plugins do not set their options to autoload (and have a whole bunch of options). I believe bbPress 1.x now has the same problem (it was avoided in 0.9 but re-introduced with backpress).

I have a mini-plugin somewhere for 1.x to load all options with one query.

Some setups can take hours to get the queries down, it’s exhausting

Next you should notice that WP 3.0 + bbPress 1.x takes nearly 1 megabytes of code per instance and in some cases won’t even run on shared hosts because it goes over the PHP ini default setting of 32MB during runtime.

If you are using a tag cloud, my “hot tags plus” will cache the complex query for the tag cloud and all rendering becomes static.

ps. the real answer is not to do deep integration – just make a clone theme for bbpress, it’s worth the effort – the good thing about that is if WP is down, bbpress can still run

[@rebootnow Participant 14 years, 5 months ago]

_ck_, I would agree if it were just a matter of a clone theme. I operated our site like that for a couple of years. But when your customization goes beyond just a theme, then the costs of “cloning” get high in their own way.

To repeat this thread, the duplication results in higher dev cost, more complexity to maintain and more bugs (to use a Rails-ism, the code isn’t very DRY).

No question that it gets more performance and uses less resources though.

[@_ck_ Participant 14 years, 5 months ago]

I know of at least one very complex, very high volume bbpress site that uses cloning very well. And stand-alone makes the site much more responsive. It even uses several installed copies of bbpress linked very well.

It’s the cost of the environment you chose. If you absolutely insist on using WordPress, you better be ready to throw a ton of hardware at it if you need to scale and insist on deep integration.

It’s perfectly possible to create cached portions of a site or common segments that can be included across WP and bbPress. It’s a matter of intelligent design.

When you start banking on 1mb of code loading per instance and a few dozen queries per user per page because of deep integration, you are guaranteed growth failure sooner or later – it just won’t scale.

[@mr_pelle Participant 14 years, 5 months ago]

Thank you Kevin for the SAVEQUERIES trick!

[@rebootnow Participant 14 years, 5 months ago]

I can only speak to my own experience, where during one stage of our growth deep integration saved me a lot of time at the cost of a more expensive hosting plan. Here “more expensive” means in the region of a hundred dollars a month, so when I trade this off against the cost of my own time or the cost of hiring a developer it is a good trade. Or, I would argue, “intelligent design”. ;-P

Now we are reaching another inflection point and I am making an investment in efficiency. I fully expect the site to a be completely custom at some point in the future.

_ck_, I’d like to know more about the “guaranteed growth failure”. I understand how vertical scaling (bigger server) eventually runs out of steam, but (aside from cost) what is to stop horizontal scaling (more servers) for WP?

Also, it sounds like you have experience of hitting the ceiling. I am interested in the rough traffic level where this happens. 1MB per instance is definitely high, but with 2GB or RAM on a server you need a pretty busy site to max that out.

I find that the bottleneck is more often on CPU. So high query counts and deep call stacks seem to be more of a culprit than the instance size. Is there some other reason that you’re concerned about the size of the RAM footprint?