Hi there!
If you’re looking for an off-the-shelf solution, BuddyPress and the Private Messages component sounds like the best one. You’d still need to tune it a bit with some custom code to lock it down exactly how you’ve described, but it gives you isolated message threads with invitees, notifications, an inbox, etc…
Technically, in the next major version of bbPress (2.6) you could use the new engagements API to keep those conversations a forum topic, but limit them to specific invited users, but that’s probably more work and less obvious to end users what it’s for & doing.
