A recent question was put on hold as "off-topic". The initial question text (as it has changed):
Has any story, movie, TV show depicted a spaceport on either of Mars' moons, rather than a spaceport on the surface of Mars?
At least 5 users voted it off-topic, presumably because they considered it a list question.
Multiple answers which had already been provided to that question, including the Accepted answer, received multiple downvotes, with @Valorum explaining that he at least was downvoting the answer because he felt the question was off-topic and he wanted to discourage the user who posted the answer:
My question: Is it a reasonable practice to downvote answers, not on their intrinsic quality, but as a reflection of the voter's opinion of the question they are in response to?
Not asking: Was the question on-topic or not? Don't care. There's plenty of evidence that "Has any" translates to "Could someone provide at least one example of" rather than "I would like a list of" - consider multiple accepted and upvoted questions such as this, this, this, this, this, ...... In any case, the question has been edited, and revived, so this is moot. And the issue of whether we punish answer authors based on the question should be the same regardless of the quality of the question.
Not asking: Should the points be restored? Don't care. Mine was the accepted answer, I lost points to the downvotes, but I couldn't care less about the points.