ya, if you want to fully replace the mirrior you need to disconnect the wire inside the door card to take it off... but if the mirrior is just a small bit loose you could try tighting the 3 small screws on the bottom of the mirrior on the outside. most of the time they are the culprit.
Yes I done that to mine, cured the problem 99%. Bit awkward to get to those screws though, might be best to use new screws.