So I'm not a SFF mod, nor have I modded for any of the stacks, but, as I've mentioned before (bear with me), I did mod for a very large Harry Potter site for five years. It was a forum-based site that required correct spelling, grammar, and punctuation (SPaG), and if a post had more than three SPaG errors, either the poster or the mod would make the necessary correction(s); if a post was rife with SPaG errors, the post was deleted. I'm sure you can imagine 1) how tedious this was, and 2) how quickly this engendered mass dislike of the site's mods.
Can a mod take of his/her hat? Well, in my experience, no, not really. Maybe I should say mods can't take their hats off very easily. It's hard to explain, but it's just difficult to make things work when a mod has one foot in the moderator pool, and one foot in the user base. If a mod becomes overly friendly with, say, three to five users, there's the risk that the mod might start treating that group of users with more leniency than other users, and that can't happen if the site is to maintain its integrity. I think that most mods are hyper-aware of this potential problem and strive to avoid it; sometimes this can translate into users perceiving mods as standoffish or unapproachable.
As a mod, if you take off your hat and let just one person slide on the site's rules or expectations, you will instantly open yourself up to a potential queue of additional users who might want to be afforded the same leniency. If you grant their request, you're not modding correctly. If you refuse, you, again, engender dislike.
I could list many more examples like the ones I've already given, but I think you probably get the gist. Perhaps the one place a mod can kind of hang up his/her hat is chat, which is a casual environment by default. Some mods feel comfortable just chatting along with the general flow; others only like to come into chat to discuss a site matter with a user or users. Otherwise, in my experience, once you put on a mod hat, a large site like SE or SFF changes from recreational to occupational. It just does.
Specifically, though, you asked, Is it okay for diamond mods to "take off their mod hats"? Yes, it's okay. But it can still be met with suspicion. Mods have opinions like any other user. The thing is, though, once a mod becomes a mod they are seen through a different lens; it is very difficult for many users to unsee that diamond, no matter what caveats a mod may give before expressing a personal opinion. If a mod is going to take off his/her hat for something that may be construed as controversial, the mod should definitely alert the community that they are speaking for themselves, as just a regular user of the site, and that they are not speaking on behalf of the site. This is a pretty typical thing for mods to do, and they should, but I question the efficacy of this practice, to be honest. I don't mean this disrespectfully toward any mod who does this -- I did it many times as a mod. But there always seemed to be a separation between mods and regular users.
ETA: Now that two (as if this writing) SE moderators have weighed in with answers, one effectively also saying "No, you can't take off your mod hat once it's on", although I have not modded at SE but at another large site, the community might find more credence in my answer. This is not a call for upvotes; I like comments just as well, and if I've provided any food for thought, I'd be glad. :)