Why is there no Review audit on Scifi & Fantasy Stack Exchange?

Question

On Stackoverflow Review queues, there's always some hidden audit test post in the review. Sometimes they come up almost consecutively.

However, In Review queues of SFF Stack Exchange, I have never once run across a review audit. I read this Meta Stack Exchange post which states:

This is currently active only on Stack Overflow, Server Fault, Super User, Programmers, Ask Ubuntu, Mathematics, and Puzzling.

But we get our fair share of bad close votes, flags, reviews and edit suggestions.

For example this edit¹ where the editor just added a tag of Severus Snape to the post which I believed was unnecessary as Character tags, in my opinion, do not help in ease of search. Someone searching for that answer would still get Google find it for him with keywords Harry Potter and Severus Snape. Or on the contrary, perhaps I made the wrong decision in rejecting that edit and I should have some sort of sense that if my Reviews are not good, I might get my privilege suspended.

Or this suggestion Where an anonymous user just tried to make the post humorous which does not help in increasing the quality of the answer in any way yet it was approved by one user.

Or maybe this suggestion where I clearly made the wrong review by judging the suggestion on it's size rather than it's contents. I should be penalized for that.

Or maybe this comment I flagged as non-constructive² which was declined even though the comment said this on a story identification question:

I think you dreamt it. I've seen thousands of movies over 40 years but never that. You should write a story around it and publish it on a self publishing site. Good luck!

Do we have any reason for why the audit mechanism is not available for SFF Stack Exchange? Is it because of the low traffic in our review queues?

P.S. Please migrate this to Meta Stack Exchange if it is better served there. I thought because it is about specifically our site so it would be better posted here.

EDIT: I have also added a comment for the CM who wrote the linked Meta SE post here.

---

_{1: This in no way is implying that other reviewers performed a bad review. It's just a general example. As stated already, It may have been that I was the one doing the wrong review.}

_{2: That's a mod flag, as pointed out by Null and is therefore not a part of normal review queues}

Answer 1

I can't answer for the administrators who set up the review audit system, but I suspect it has to do with the fact that this site is small enough that review audits aren't really necessary.¹ There is a small group of reviewers on this site, and it's not too difficult for moderators to make sure those reviewers are doing a good job on reviews (either by noticing bad reviews directly, or by members of the community flagging to inform the moderators of a potential robo-reviewer). The sites that do have review audits are generally much larger than ours.²

Using the moderator tools, I counted the number of distinct users who reviewed at least two posts in each review queue in the last 30 days. I used a threshold of two since moderators aren't going to take action for just one bad review (not even the audit system does that). Here are the counts:

• 41 first post reviewers
• 27 late answers reviewers
• 23 low quality reviewers
• 40 suggested edit reviewers
• 27 close votes reviewers
• 17 reopen votes reviewers

Furthermore, as you can see from the publicly available review stats, there is considerable overlap between these reviewers in each of the queues -- there are probably only ~40 users on the site who regularly review posts. That's not too difficult to keep track of. By comparison, for example, there have been over 3,000 suggested edit reviews on Stack Overflow so far today.

In any case, the reviews you posted would probably make bad review audits anyway. Review audits should be unambiguously good or unambiguously bad. They are intended to catch robo-reviewers who consistently make obvious mistakes, not punish users for making an occasional and understandable mistake. I don't think any of the example reviews you posted would make particularly good audit posts.

• The Snape character tag edit should have been approved (and it was) since the community has accepted the tag, but the existence of character tags is controversial here.
• The anonymous editor's suggestion to remove an aside could go either way -- and indeed one reviewer approved it while two others rejected it.
• The one you erroneously rejected was a minor mistake and the existing review system (which requires multiple approvals/rejections) prevented your mistake from causing the wrong outcome (the suggested edit was ultimately approved).
• The comment flag falls outside of the review system since that was a moderator flag.

---

¹ That's not to say review audits on this site would be a bad thing. Review audits would slow down the rate of actual reviews completed, but that wouldn't really be a problem since our review queues are regularly empty anyway. A bigger problem might be wasted time discussing failed/disputed review audits here on meta -- just see Meta.SO's questions about disputed review audits.

² Puzzling is an outlier -- it's small but has review audits, though I'm not sure why.