The spoiler tag is just a bit of CSS. I believe all modern browsers allow use of a user style sheet. So all you need to do is add a rule that overrides the CSS in your user style sheet, and you're done.
(I don't think user style sheets can be site-specific, so this would change any site that uses the "spoiler" CSS class. If that does have an impact, then IMO a better feature request would be to have SE change the class to something like "stackexchange-spoiler").
For example, in Safari, go to Preferences, then Advanced, then select the stylesheet.
Your stylesheet only needs to contain something like this:
.spoiler { color: #444 !important; }
444 is the colour that you get when hovering (at least on this site) - you could make it something else if you wanted spoilers to stand out in some way (e.g. so you avoid referencing them in answers!).