If you unplug it then you lose the night auto-dimming feature, better to just pull the wire responsible for the headlights and leave the rest of the wires in the connector, plugged in. There is a topic in this forum where I mentioned which wire that is, or it'd be in the wiring diagrams linked in my sig below.
Another thing you can try is to gently heat that slider area with mild heat like a hair dryer, then move the slider back and forth several times to push old, hardened grease out of the way to improve conduction. If it weren't a mirror then I'd suggest spraying contact cleaner in but I don't know if that would mess up the mirror. If you can get it apart non-destructively then there is also the option of more targeted manual cleaning and bending a little more spring in the potentiometer wipers.