Forum Software Benchmarks: http://bb-bench.com
-
I’ve built a site to collect software benchmarks of various bulletin board software, found at http://bb-bench.com/ . I’m trying to setup boards under the exact same conditions – forums, topics, posts, users, to show how they scale under various forum loads, to provide users fair, quantifiable claims regarding performance. All of the boards are setup with their stock configuration options (with the exception of phpBB with HipHop for PHP, which I’ll admit to having an unfair advantage)
I would love your comments. What other combinations of forums/topics/posts/etc you would be interested in seeing? Any additional pages?
Additionally, since there are obviously optimisations that can be performed on any forum software, perhaps you can suggest some that might improve the results of bbPress.
-
just do two benchmarks from here. bbPress 1.0+ release and the legacy one 0.9
What’s the point of benchmarking an old version? Unless you are actively promoting people using 0.9, it seems silly to benchmark a version no longer in use.
A fork of the legacy version is on its way. Its runs very fast as compared to the 1.0 v and supports a large number of plugins.
Trust me you won’t want to miss this in your benchmarks. I won’t be surprised if it topped the chart
Can you link me to this fork? This is the first I’ve heard of it.
Its a work under progress and its not live yet. Just see the difference in your results between 0.9v and 1.0v
I already know from my own benchmarks that 0.9 in many cases is 50% faster than 1.x, if only because it loads 50% less code, not to mention fewer queries out of the box.
But as nice of an ideas as it is, your benchmark really cannot compare apples to apples.
ALL the other forums you have listed already use a page cache, in part or whole, which is why they have such high numbers.
What your benchmark should do is attempt to DEFEAT any page caching the forum is doing by generating new posts BEFORE every page load.
ie. POST / READ / POST / READ / POST / READ / POST / READ
The next version of bbPress will very much need a page cache just like most buddypress sites desperately need as they are very sluggish (because of the WordPress core).
A few people have successfully deep integrated bbpress with wordpress and then used an exisiting wordpress page cache. That would give similar numbers to what you have posted for other forums.
Didn’t it dawn on you when you see all the boards performing exactly the same under load with static content that they are using a page cache?
I mean that leaps out at me when I look at this:
http://bb-bench.com/benchmark/1/graphs/p/forum-middle.png
That’s obviously dynamic vs static content.
Put a static HTML page on your server and run the same tests against it to show the baseline.
None of the software is serving data from a static page cache. Sure, they have various caching mechanisms, but all are retrieving their posts from the database on every page load, just as bbpress is.
I don’t have access to the server at the moment, but a based on my recollection (from accidental runs that resulted in 404 responses), it was able to handle several thousand requests per second to non-php based content, a magnitude more than any of the results I found. I’ll run a baseline tonight.
As for the previous post, I’d disagree. The majority of requests to a forum are going to be read requests, and they are the primary factor of site performance. If digg/slashdot/reddit links to your forum and the story becomes popular, the reads are going to kill your site, not the miniscule number of posts those guests will make.
I look forward to a static page for comparison on your graph.
It will prove what I am saying about content caching.
I have a test SMF install handy and I can see it uses less than a half-dozen queries on a 2nd page load which is a dead giveaway there is caching – I am sure every other 3rd gen forum package has it too. You have APC running and I know that SMF also takes advantage of that, so probably do others.
bbPress never got content caching because Matt keeps having the core re-invented every couple years (backpress and now as WP plugin) so we keep getting back to square one for advanced features. Don’t be confused by versions 0.9 is a different program than 1.0 and 1.2 is a completely different program than 1.0 or 0.9 (that’s Matt’s fault).
So I still insist this isn’t a fair comparison.
But bbPress 0.9 should be about halfway between the 1.x and other forum software (ie. 50% faster) 0.9 will be continued to developed independently, so it’s worth benchmarking.
If you are willing to put a couple of tweaks in the
bb-configwith 0.9 I think we can get a little closer to the rest of the pack at the bottom, ie.$bb->load_options = true;off the top of my head (note that does nothing in 1.0)
From my reading, we agree that 1.0 might not be a good choice for an admin when looking purely at performance, and based on your phrasing 0.9 may not be either (“bbPress 0.9 should be about halfway between the 1.x and other forum software”).
However, I will still post the apache bench output from a static file, and I will make that adjustment to the bb-config file.
Note the bb-config.php adjustment is for 0.9, it will have no effect on 1.x
When you test a static file, I’d like to see it done like this.
1. run it as php so it has to go through the php parser
2. call it with a unique query on the url for each page per thread
the purpose of the above is to (slightly) defeat pure caching on apache’s part
I will always disagree that forums are write-once/read-many-times
That behavior is found with blogs, not forums.
The popular forums I belong to have new posts every few seconds which is very different than even the most popular blogs I read.
I just ran some tests, with an html file and a php file, both containing “Hello World”. You can find the apache bench results here: http://pastebin.ca/1934137
Clearly they are much faster than any forum software, handling 5000+ and 9000+ requests per second each (I had to run apache bench with 100,000 hits to get a semi accurate reading)
As for a write once, read many times situation, blogs do exhibit such behavior more so than forums, I’ll admit. However, forums are certainly more read heavy than write heavy.
Take a look at phpBB.com’s support forum, http://www.phpbb.com/community/viewforum.php?f=46 . You will be hard pressed to find a topic without at least 5 times as many views as posts, and many of the sticky topics have over 100 times as many reads as posts. Show me any forum with more writes than reads and I will be impressed.
I’ll be running bbpress 0.9 later tonight, along with some changes to SMF that they suggested.
Actually the PHP version kinda proved my point about content caching.
It’s taking 3-4 ms.
The front page index on most of those forums is taking 20-25ms
That’s impossibly fast, it definitely means content is not being regenerated. If the cache was defeated it would have to re-render the whole thing and the page time would be significantly higher.
bbPress doesn’t even save the tag cloud between pages, it will re-render it each and every time, which is at least 1/4th of the page render time (that can be changed via a plugin).
So you aren’t testing renderer-vs-renderer, you’re testing content caching vs content caching. You aren’t even testing for logged-in users but only outside visitors.
If we are going to play that game, all someone has to do is port wp-super-cache to bbpress and it will beat every forum in existence, because static html will be served via htaccess directly and bypass PHP/MySQL entirely. So you’d be down to 4ms (or less) per page.
In any case, I am looking forward to seeing how 0.9 does against 1.0 under 20 concurrency.
quick ‘n’ dirty test:
ab -c 20 -n 1000bbpress 1.1
. min mean[+/-sd] median max
Total: 46 458 117.9 453 1250bbpress 0.9 (with
$bb->load_options=true;). min mean[+/-sd] median max
Total: 46 275 90.6 265 890453ms vs 265ms (or 458 vs 275)
So, I’d expect I’d see around a 50% improvement on your box on 0.9
now – vs static cache content simulation:
bbPress 0.9 front page html saved as
static.phpand
<?php $test=1; ?>put at top to force PHP parser to turn on. min mean[+/-sd] median max
Total: 15 75 14.3 78 109So 265ms vs 78ms
bbPress 0.9 is “only” 187ms slower than statically served PHP.
In a nutshell:
. min mean[+/-sd] median max
bbpress 1.0: 46 458 117.9 453 1250
bbpress 0.9: 46 275 90.6 265 890
static PHP 15 75 14.3 78 109@ckwalsh, Your benchmarks are incorrect this way. _ck_ is right about it
Well I don’t want to discourage his initiative, it’s a great idea.
I just think a better tool then apachebench needs to be written.
Basically I’d want to see a simulated load with logged in users being served different content – ie. unread posts for different users
and most importantly, I’d want to see new posts generated (or existing posts edited) that force content caching to be defeated.
If bbPress 0.9 had been finished instead of all that all wasted coding energy Sam was forced into porting it to backpress, we’d definitely have some content caching by now.
But plugins can still do content caching themselves. For example I realized awhile back that the Hot Tag cloud that bbPress renders is taking at least 1/4th of the total page render time, and it does it each and every time, regardless if there are new tags added or deleted. So my Hot Tags Plus plugin caches the tag cloud as static html (as well as add many other features to it at the same time).
Um, wow. I’ll admit I’m quite astounded and I’m not quite sure what to say, but I’ll try.
Actually the PHP version kinda proved my point about content caching.
It’s taking 3-4 ms.
The front page index on most of those forums is taking 20-25ms
Uh… no they aren’t. http://bb-bench.com/benchmark/1#section_p_index
That’s impossibly fast, it definitely means content is not being regenerated. If the cache was defeated it would have to re-render the whole thing and the page time would be significantly higher.
No, it’s not impossibly fast. I have not perused the code of most systems, but I can guarantee phpBB does not cache the output of its pages. They are retrieved every page load. Sure, there is a caching system in place, but it doesn’t catch the output of a page, only stores a few variables that don’t even improve performance particularly. As for caching at the apache level, you would see a much bigger difference in speed than you do see. Without special configuration, apache should never cache the output of a php page, since that would entirely defeat the purpose of a dynamically generated page.
Caching page output is near impossible for forums in general, due to forum permissions and session specific information. It would be kind of bad if my “One private message unread” were cached for you. The only other form of caching is the sql cache, which is contained entirely within MySQL, and is automatically activated for all queries, regardless of software.
bbPress doesn’t even save the tag cloud between pages, it will re-render it each and every time, which is at least 1/4th of the page render time (that can be changed via a plugin).
Sounds like that’s a big area for bbPress to improve. But it still doesn’t make the result invalid: bbPress is slower.
So 265ms vs 78ms
bbPress 0.9 is “only” 187ms slower than statically served PHP.
Do your math again. It’s 260ms slower than a static page. It’s still damn slow, and if someone installs bbpress and *pick another board* side by side, the bbpress one will not perform better. While technically correct, it does nothing to affect the overall results.
@ckwalsh, Your benchmarks are incorrect this way. _ck_ is right about it
Can you please paraphrase what his arguments are, and perhaps add a little bit more? I’m getting the feeling that you posted here to support _ck_ since the results I found were unfavorable to bbPress, and you don’t have anything to add, or even understand the discussion. Not to say I only want to talk to experts, but blind bandwagonning always frustrates me on the internet.
Basically I’d want to see a simulated load with logged in users being served different content – ie. unread posts for different users
Certainly apache bench isn’t perfect, however, it does show the relative performance of those 3 pages, which are almost guaranteed to be responsible for the majority of requests to a forum. I have been considering how to build a better tool, but have not gotten a chance to do so yet.
But plugins can still do content caching themselves. For example I realized awhile back that the Hot Tag cloud that bbPress renders is taking at least 1/4th of the total page render time, and it does it each and every time, regardless if there are new tags added or deleted. So my Hot Tags Plus plugin caches the tag cloud as static html (as well as add many other features to it at the same time).
If you think that will drop load time, it seems to be something anyone can add – I’d certainly be willing to install it for my tests.
Overall, you seem to be upset that bbPress performed so badly, leading you to say the benchmark is wrong. While it isn’t perfect, what it does show it shows rather precisely. In the current state of forum software, bbPress does appear to have a long way to go. It doesn’t matter if it was rewritten. It doesn’t matter if it doesn’t have caching (if so, that is a failure of bbPress, not an unfair advantage to other software). When comparing the same functionality between software, those are the results, like them or not.
All your current benchmark proves is that bbPress will be slower for 20 concurrent connections over 1000 passes, where no-one is logged in and the content never changes.
What I am saying is that’s not how people run forums, you need a more realistic benchmark, which I know from experience is going to be tricky to code.
bbPress 0.9 will be faster than what you have shown by about 50% and then with content caching it will compete fairly against the others.
Apachebench is not dynamic enough to prove real-world usefulness and barebones bbpress out-of-the-box is certainly not setup for a large site. But with plugins it certainly can be. There are some reasonably large sites on the Top 100.
@ckwalsh I am do supporting _ck_, bbPress as bbPress is generating each and everything again and again on each page load. It doesn’t matter if its a first load or second or third whereas in other forum scripts, there is some sort of caching involved in each of them which makes the difference in subsequent loads by caching atleast some part of it.
I hope I made my point. If I had nothing to say or I wouln’d have got a word about what’s being discussed here, I wouldn’t have left a comment.
Hey Everyone!
I am tracking opinions on bb-bench.com on internet from my curiosity (I am NOT associated with that initiative in any way).
I find this thread interesting in negative way. Hovewer responses on those results are not suprise to me. Every dev unsatissfied with results attacks test itself ignoring bad solutions (or lack of countersolutions) in his code.
You say “test is bad because we dont implement cache while others do”. That indeed has its effect on BBPress results hovewer as end-user I am interested in performace. Caching algorithms are necessary if you aim your software for bigger communities.
Ofcourse you can say “hey, if load is too big, you can always switch to more powerful hosting plan”. Thats true, however whats heaper for user? Change hosting solution or community software for faster one?
What I am especially disgusted with is how you claimed other solutions use full-page cache, while none of them does. Lying about other software to make your one appear “more fair” is really cheap shot.
If your code lacks features like caching, you implement those, dont argue over how cache is bad and gives others unfair advantage in tests. How hard it is to implement basic cache mechanism to code? Implementing it will make software better.
Of course it will be good to have content caching added, I already commented on how plugins can do that and bbPress never got content caching because it was being re-invented every couple of years. The reality is now it’s a moot point, bbPress 0.9 and 1.0-1.1 will never have it internally. I doubt 1.2/2.0 will ever either, they will simply rely on WP plugins to do page caching.
But I’m also saying the effects of content caching will be less visible if the test was more realistic to how forums are really used.
Apachebench is not complete enough and not real-world enough.
The problem is it’s going to take a week or two of coding to write something better and I can’t imagine someone doing that unless they were serious and skilled enough to do it.
I could write a plugin that defeats apachebench entirely, it’s very easy. AB does not load any content on the page, no images, no scripts. So basically all I have to do is send it one hard coded script tag, nothing else on the page, and the script loads the page. Then bbPress would rank as fast as an empty static html/php page.
That’s an extreme unrealistic example but it’s part of what I am saying. A real world test would load the page like a browser does, it would accept and return cookies so a user couple be logged in, and it would post new content between reads.
Writing a test like that is hard work, so I am not surprised that just using AB is the fallback.
You want bbPress to be cached? It can already do that, deep integrate with WP and then use wp-super-cache (or similar).
But then the page generation time is hidden in the first page render, let’s say it takes 800ms or more. Each next page will only take 20ms or less to serve. But Apachebench is “dumb” and doesn’t know that, it can only show best/worst and averages.
What if SMF or one of the others take 2 seconds for the first render on a new login and then 20ms for each successive? That 2 seconds is important if the content/user is constantly changing. What if a forum generates a list of topics the same for everyone but then uses javascript to change the list after the fact by using extra queries outside of the original page render? Apachebench cannot “catch” any of those scenarios or show their performance.
Can you quote me where on bb-bench it claims those tests aim to reproduce production enviroment?
Thats only comparision between systems. Everywhere I see people taking this as real-live scenario, but it doesnt. Its just to show people differences between softwares with same forum size (topics, posts and forums).
Oh come on, the average person who looks at a graph that says product B is slower than product A doesn’t know how to be critical about the testing methods. If they did, every other commercial you see on TV would be useless.
Making a benchmark is a learning process. I am certainly not critical of the effort, I think it’s great someone took the time to do it. All I am saying the concept can be improved and expanded.
Early on I wrote bb-benchmark to analyze how bbPress performs from the server-side. It can be very helpful. But as fancy as the analysis is, in the end something I wrote much later on that gives much simpler data is far more practical for real-world results (browser-timer).
AB could be considered the parallel of bb-benchmark. It has it’s usefulness.
But one day there will be something more real-world like browser-timer.
- You must be logged in to reply to this topic.
