Yea that’s a good point, like if you are downvoted for “over rewarded” content you cannot adjust anything.
There should be something the user can adjust to not be downvoted in most cases.
Otherwise, we are picking winners and losers when it comes to content quality and that just doesn’t make a lot of sense.