Just got done taking off my rear passenger door panel. Was as easy as the OP stated, be careful taking off the power window piece, I broke a piece of a clip (the one towards the front of the vehicle), so it all goes back together, but a tad loose in the front (but that part isn't pulled on anyway to close the door, so shouldn't be an issue)
In the end, it was simply a learning experience for me, as the pair of speakers I got for free (nice Rockford Fosgates), one was blown lol