[gbbgadmin]

Just to drop in here on the thread I originally started. We worked on this for several days. Probably put in way too many hours and way too much manpower. We never ever could resolve it. There are a few interesting things to note:

A. When we ran some of the “optimizing” and “updating” tools built into BBPress (like the tools to re-link posts with their authors, re-set user privileges, etc.), the site would function. Then as soon as someone made a post, everything broke again. So it was most definitely something in a relationship within the db that was breaking down. Running all the db cleaning tools in BBPress would fix the problem until a member made any post or replied to anything.

B. After looking deeply into the db, I found that new replies were somehow not being associated correctly across all tables with their author sometimes, but mostly the issue was they were not being correctly linked to their parent. What this did was to mess up WP’s (and BBPress’s) ability to render the pages in proper ordering. You know, who posted what, who replied, nested responses, etc. So, the page 500 errored out.

Now then, our company did not have the time or money for us to spend fixing this problem or rooting out what caused it. So, we had to make the financial decision to back up our data, set up clean db and WP app servers, and migrate to them. Then, destroy the problem db and app servers. On the clean install, moved data across and everything works absolutely perfectly.

This is good for us. But this is bad for people experiencing what we went through. Unfortunately, we never found what caused it, or how to solve it. Other than just manually “fixing” the db with BBPress tools every time someone made a post or reply. But that is just stupid.

[gbbgadmin]

UPDATE. NOT RESOLVED.

After repairing broken relationships the site was working as expected. Then, about 1 hour ago, a member reported that 503 errors are back on our bbPress forum. After looking at a narrower window of data, here is what we have found.

Replies are NOT being associated with their parent forum. They go into the database and even have their proper topic parent ID assigned in the post_parent column. The admin “replies” page, which becomes accessible after running “Recalculate the parent topic for each post” tool. Then, the forum for the post shows as “No Forum”.

Upon inspection of the form where users enter replies, we see this:

<input type=”hidden” name=”bbp_topic_id” id=”bbp_topic_id” value=”HIDDEN”/>
<input type=”hidden” name=”bbp_reply_to” id=”bbp_reply_to” value=”0″/>
<input type=”hidden” name=”action” id=”bbp_post_action” value=”bbp-new-reply”/>

NOTICE: we put “HIDDEN” into the value for bb_topic_id because it is not relevant here. It does show correctly. However, you will notice that the bb_reply_to is ALWAYS being set as value=”0″. There is some reason that bbPress is setting ALL replies to 0. We cannot seem to figure out why.

[gbbgadmin]

RESOLVED: before we had time to try the wireshark tools we ran a script on the database looking for broken relationships. Found that a post that had been removed using correct tools somehow had broken a relationship and the replies page would not load. Here is what happened:

1. A member joined our forum and spammed the crud out of it.
2. Akismet caught the spam and blocked it all, marking all topics and posts as “spam”.
3. One of our keymasters marked the user as a “spammer” according to the moderation tools provided by bbPress.
4. One of our admin staff moved the spam topics and posts to the trash, using the tools on the WordPress admin.
5. The same admin staff emptied the trash using the tools on the WordPress admin.

After this, they did not notice the replies page was not working or that people could no longer post. Later when people reported they could no longer post, our admin went to the WordPress admin and got a 503 error on the “replies” page. We fought this 503 error issue for nearly 1 week. After finding one broken relationship between a topic and the replies in that topic, we fixed the relationships and the 503 error went away. Now, all the forum is working perfectly.

So, somehow, using all the tools provided by bbPress, a user was marked as a spammer, their spam was blocked, trashed and emptied. And then the entire forum was broken due to a broken relationship. Fixed now, but really strange it could happen in the first place.

[gbbgadmin]

We use newrelic remotely and also some server-side tools, but wireshark looks interesting. Will see what it could reveal. The problem with the tools we have is that the break comes before they can seem to tell us why. It is almost like there is something appearing on the page that is causing the php to break before rendering the page. But we have combed through the database and can’t seem to find something a user could have posted or uploaded that would be doing that.

[gbbgadmin]

Update. Today we did the following:

1. Deactivated all BuddyPress and bbPress plugins. Uninstalled all BuddyPress and bbPress plugins. Removed all BuddyPress and bbPress data from the database, but kept a backup copy of the data and tables.
2. Installed fresh WordPress on fresh server. Installed new, fresh plugins.
3. Site worked.
4. Replaced data from backup.
5. Entire site works, EXCEPT there is a 503 error when people try to post a reply on a forum, and a 503 error when we try to go to the “replies” page from the wp-admin.

So, maybe something in the data? Could someone have posted a reply that has some content that is breaking the php code when trying to see the Admin panel “replies” page or when people try to post a reply?

We have learned another thing. When people post a reply, the reply DOES go into the db and it is visible to other members on the forum. HOWEVER, the person posting the reply gets a 503 error during the process and has to come back to the site fresh to see the forum, including their post.

Something is causing a 503 error ONLY when replying OR when we try to access the Admin “Replies” page as admin. It is very very strange.

[gbbgadmin]

If it were our host, it should happen to more than just one page out of hundreds of thousands of pages. We are only getting a 502 error on this page:

https://bitbillions.com/wp-admin/edit.php?post_type=reply

All other functions of BBPress work perfectly and all other pages of our site work perfectly. It is like something is breaking the php when that page is trying to load.

[gbbgadmin]

Maybe to help out a bit:

We run
PHP7.0-fpm (latest stable)
Nginx 1.10 (latest stable)
Ubuntu 16.04 (latest stable)
WordPress 4.6.1
bbPress 2.5.11
bbPress Advanced Statistics 1.4.02
bbPress NewTopics 1.0.1
bbPress No CAPTCHA reCAPTCHA 1.0
BuddyPress Version 2.7.2
BuddyPress Activity Plus 1.6.4
Buddypress Messages Spam Blocker 2.5
BuddyPress Profile Tabs 1.6.1