I think there are a couple different kinds of downvotes. I think downvoting someone who is obviously abusing the system is pretty much a given. If I know someone stole someone else's content or is plagiarizing, I have zero issue nuking them. On the flip side, downvotes over rewards are a bit more tricky. I feel like you should at least give a reason why you did it in the comments. Whether they accept that or believe that is another story. Personally, if I do something that offends someone, I'd like to fix it as long as it doesn't compromise my morals or my style. If they just don't agree with the rewards, then I might not like that, but I can accept it.