It's not too hard. Nearly the whole thing is just snapped on.
First, take the two screws out above the door release. Then carefully pry off the plastic surrounding the door release. I use a small screwdriver. It needs to come off so the handle will pass through the panel when you get it off.
Then, starting on the bottom of the panel, use a thin screwdriver or other pry tool to pry the panel from the door. The whole panel is snapped on, so it may help to get a flashlight and look behind the panel to see exactly where the snaps are. They come out much easier if you pry as close to the snap as possible.
Work your way all the way up both sides of the panel unsnapping as you go.
Once all the snaps are loose, then you need to lift the panel straight up to get it out of the window channel. Also, remember to work the door release out of the panel or it will get stuck on it. DO NOT lift from the door handle or it will break!!! I learned this the hard way. Lift from the sides or the bottom. If you have power locks and windows, be careful no to pull the wires to hard. You should be able to get the speaker out without unplugging everthing.
The speaker is held in with four screws which are easily removed.
Don't think I forgot anything. Hope this helps.