Ohh other plugins that was the culprit. I had installed bbPress Toolbox, which had checked an option to notify of new topics. Hopefully that was the setting causing this!
Anyone posts a comment
A comment is held for moderation
You mean those?
I have unchecked those but that didnæt change anything.