Here are the steps I went through to replace my crappy fabric arm rests with leather covered ones. Too many images so I will use two posts.
1. Using a small flathead screwdriver pry open the screw covers behind the inside door handle and the door latch.
2. With the same screwdriver push in on the center of the fastener that is on the door panel near the mirror. Then pull it out.
3. To remove the inner panel, you have to pry the bottom and both sides away from the door. I did this with a larger screwdriver. When this is done you should be able to remove the inner door panel by lifting it up. There are hooks underneath the window felt that hang the door panel on the door.
4. Once the panel is removed, turn it over and look at the back of the arm rest. The arm rest is fastened to the door panel with a series of small melt points. I used a 3/8" drill bit to drill down about 1/4" into the centre of the melt points. Do not drill too far because you will need enough plastic to put screws in later. The melt points are like plastic rivets.
5. Once the melt point are drilled out you should be able to pry the armrest off. Next remove the old fabric.