You don’t need to use a function: you could add the functionality you want just by hacking the plugin. You can always refactor later to move some things into functions.
Start simple. Find the place where the message is displayed to the user. Then, add some html there to display an image. Make sure you upload the image, then make sure you get the path to the image correct. I think that would be pretty simple.
I thought there was a “newmail.png” already included though? I don’t use the PM plugin, but if that icon is present, where is it displayed? If it’s not on the page you want, I would use the same method they use to display it, just on a different page.
Although the title of this topic is “Private Messaging Popup” you’re not looking to create a javascript popup when the user logs in that they have a PM, are you?
You could also contact the plugin author:
http://faq.rayd.org/bbpress_private_message/
From that page:
Visible representation of when a message hasn't been seen (in your inbox and sent box)
