The only time you need to strongly worry about what community thinks is answer accuracy.
If community downvoted an answer with an explanation that it's wrong (and why) it's probably a good idea to think hard before accepting it (still your right to do so if you wish). There are very few things that irk me more on SFF than seeing an accepted answer - AFTER I posted a canon quote in a comment proving it 100% wrong.
Otherwise, community votes should merely serve you as a rough (and not always accurate) hint as to the possible quality of the answers. In other words, if you are new to Stack Exchange, and not sure how to evaluate answer quality, you can give more credence to answers with more votes - but still more, to the comments explaining WHY the votes are there.
But the decision on acceptance is 100% yours - a hint is not a rule, merely one of the inputs into your decision making process.
As a side note, be aware that votes CAN be wildly misleading at times.
Sometimes people upvote the answer for fine (or not so fine) humor, and not actual accuracy or helpfulness (I had at least one of those).
Other times, simply because the poster sounded authoritative (I've seen +15 answers that looked OK but were either completely wrong or not really right to someone with deeper knowledge of facts).
Yet other times (granted, less likely on SFF vs SO), people may upvote "easier to understand" answer over a more in depth and more correct but harder to get answer.