I just looked at my rep and I got hit with -70 because "User removed". Was someone banned? And I thought after about 5 mins up/down votes were cemented in and couldn't be changed unless an edit to the question/answer was made. Why do I get punished for someone else leaving?
When accounts are deleted, their votes go away with them. See What does “user was removed” mean and why did my reputation change because of it? on the main meta.
This isn't a user who was banned. There are only three reasons an account would be deleted:
- Spammers and the like — users who have never produced any useful content. These are very unlikely to have ever had enough reputation to vote.
- Sockpuppet accounts — when the same person had multiple accounts. The sockpuppet is merged into the main account; if both accounts had voted for you, you lose one of the votes (you can't have the same user voting twice on the same post), and I think that's the message you'd see.
- Other than that, accounts are only removed at users' request. Misbehaving users may be suspended, but that doesn't affect the votes they may have cast.
You are not punished for that. The reputation you lost is reputation that you would not have gained if the user hadn't voted for you. You don't take a penalty: since the user is no longer there to grant you these votes, you can no longer claim to have earned them.