Clear communication between flagger and handling moderator.
That's the optimal situation when you're flagging anything. You want the moderator to understand why you're flagging and to agree with your reasoning. The same applies in reverse: if we don't agree with it, ideally we'd like you to understand why we're declining the flag, so that you can learn better what sort of content really needs flags. (If we do agree, of course, there's no need for further communication.)
Custom reasons for flags is one way of achieving this. If you give us a personalised message, "I'm flagging this post because XYZ" or "this comment should be deleted because ABC", it's much easier for us to understand why you raised that flag, and we're probably more likely to agree with you if we understand your reasoning. Remember, there's a human on the other end of each flag you raise, and we do listen to your concerns!
For the same reason, each time we decline a flag, we have the option of giving you one of several canned reasons for declining (e.g. "a moderator reviewed your flag, but found no evidence to support it" or "flags should only be used to make moderators aware of content that requires their intervention") or of giving you a custom message, "this post shouldn't be deleted because PQR". I like to use these custom messages as much as possible1, because users are more likely to understand and accept the declining of their flag if they're given an explanation.
The point of these custom flags and responses is for clearer communication.
However, custom reasons certainly aren't always necessary. In very clear-cut cases (e.g. spam/offensive flags, obvious non-answers, many cases of obsolete comments), there's no point in raising a custom flag. Just flag with one of the canned reasons, and we'll deal with it if it's obvious. The point of custom flags is to explain your reason for flagging if it's not already clear.
In particular, spam and offensive posts should always be flagged as such, rather than using a custom flag reason. This is because there's an automated system for catching such posts, and the more user flags they get, the better trained this system will be. (See also this post.)
If your flag is declined ...
... you have a few possible options, depending on how much you care about the issue and/or how sure you are that the flag was handled incorrectly.
Forget it and move on. This is probably the best approach in most cases. We don't just decline flags for no reason; we do look at each one individually and decide whether or not to act on it. But if you're really confident it shouldn't have been declined, or if we haven't given you a custom decline-message and you'd like to understand why, you have other options.
Flag again and explain better. The only reason we would have declined a flag that you're certain should have been approved (assuming you're right) is because of poor communication: we didn't appreciate your reasoning for flagging the post. At this stage, you need to use a custom flag in order to explain yourself better. And if you used a custom flag the first time when it was declined, the best option is probably point 1 above.
Take it to chat or meta. As a last resort, either if you're really sure your flag was mishandled or if you really want to understand why it was declined, you can raise the issue publicly. Chat is relatively informal; you can either ping a mod there (we're all in Mos Eisley at least occasionally) or discuss it with other users - some flags, such as for deletion of crappy answers, don't need mod intervention. Meta is for a more serious discussion, and almost never worth the trouble.
Unfortunately, it's impossible for moderators to give a custom response to a comment