OK, there might be a few times that I have got up people's noses - but this morning I wake up to lots and lots of down votes across various different sites on SE. Everywhere where this had happened I had a note of the amount and then user removed beside it. I read the note about user removed notices, but is there any way to know who this user was, and why they took so vehemently against me? - I don't believe I have lost any rep though.
No, you can't find out. Votes are private, so revealing who was the source of your reputation changes from vote reversals would violate that. In addition, users are allowed to leave quietly if they want, or if SE removes them for violating the terms of service.
See this question on Meta.SE for more about this topic.