Why is this issue so contentious?
Apparently, people have strong feelings over duplication. Some people feel that duping practice on SFF is far too restrictive, that many questions are getting closed which should really have remained open - it's natural that they'd be annoyed when a question is (in their view) unfairly closed. Other people are voting to close as duplicate much more, even questions which aren't asking the same thing at all but which have similar answers, or might have similar potential answers - it's natural that they'd be annoyed when a question is (in their view) left open to clutter up the site unnecessarily.
Some meta answers, like this one from Shog9, actually get cited a lot by both sides, but with differing interpretations.
I've noticed that too, but only in recent months. For a long time that answer was being cited only to support "leave open" arguments, while people arguing in favour of closure tended to rely more on the fact that (regardless of meta votes) long-standing practice on the site has been to dupe-close based on answers. It's only more recently that they've begun interpreting that meta answer to support dupe-closing based on answers.
Are we doing something wrong? Should we conduct this discourse differently?
I don't know. It feels like we never really come to any kind of clear agreement or consensus, but then I've thought that about other contentious issues which did end up with a good consensus.
One thing I'd suggest is that meta debates should be based on clear, solid lines of reasoning. If people post answers saying "close these questions" or "leave these questions open" without elaboration on why, that's only going to exacerbate the division between users and the 'faction' effect, as the voting on these answers will be based on people's pre-held beliefs about duplication rather than on the convincingness of their arguments.
If we are unable to reach an agreement, should the diamond moderators pick a side and enforce it by fiat?
I feel like we'd get a lot of pushback from the userbase if we tried to do that :-)
On SE as a whole, I believe this can sometimes be considered acceptable - if the community can't agree, then someone with the authority to do so has to step in and create a policy, so that the site can function at least up until the users can come to an agreement. (Cf beta site mods, chosen by CMs until the site is large enough to sustain elections.) But I know the SFF community too well to try doing that here. Moderators are here to enforce community consensus, and if there is no clear community consensus, then there's nothing to enforce (except, sometimes, common sense).
While we're here, I just found this answer to the very relevant discussion What is our position on closure when the community appears to be split?, which currently has +9/-0 voting for:
I suggest that our policy should be that in situations where the community has not been able to develop a clear consensus, we should err on the side of being welcoming to new users, and leave the questions open.
Of course, this policy is hard to enforce absolutely. Close votes are one of the main ways for users to express their opinions, and as long as they're not explicitly going against meta policy, it's hard to penalise them for doing so. On the other hand, if the review queues are mainly manned by a small subset of very active users, then closure/reopening practice on the site isn't as good an indicator of community opinion as it might seem; meta discussions might be more representative.