I’ve seen a plugin for a wysiwyg editor. Unortunately it’s not new and the toolbar may not work with the newest bbpress version.
TinyMCE can work in just about any forum so there should be no reason it wouldn’t work.
As for videoquicktags do you mean like bbcode? It can be done fairly simply I believe but I am not good enough at PHP to deconstruct the bbcode-lite script to make it for you since that script is so heavily optomized.
Edit:
http://urbangiraffe.com/plugins/html-purified/
Take a look at HTML purifier. It allows your user to use HTML but strips anything bad. That way you can use the regular YouTube embeds or whatever you like w/o need of a bbcode plugin.
http://test2.cell-systems.net/forums/topic/testing-forum-software/page/3
I’m running a custom Editor on my test site and will move it to my normal site this weekend works wonders