This is actually an incredibly hard question to answer; I'm not even sure if we'd be able to come up with a good general answer. After all, what counts as science fiction has been debated many times in the past, with Moorcock including stories in New Worlds that might not otherwise have been considered SF. (Indeed we've had that exact discussion.) Ultimately, I think that we largely can't have a useful discussion of the general case, and we'll need to consider contentious works individually.
Wikipedia defines fantasy as:
Fantasy is a genre of speculative fiction set in a fictional universe, often inspired by real world myth and folklore.
and, for comparison, defines science fiction as:
Science fiction (sometimes shortened to sci-fi or SF) is a genre of speculative fiction that typically deals with imaginative and futuristic concepts such as advanced science and technology, space exploration, time travel, parallel universes, and extraterrestrial life.
Both of which essentially hang a definition off of "speculative fiction," so let's look at that:
Speculative fiction is a broad category of fiction encompassing genres with certain elements that do not exist in terms of the recorded history and observed phenomena of the current universe, covering various themes in the context of the supernatural, futuristic, and many other imaginative topics. Under this umbrella category, the genres include, but are not limited to, science fiction, fantasy, horror, superhero fiction, alternate history, utopian and dystopian fiction, and supernatural fiction, as well as combinations thereof (e.g. science fantasy).
This list is actually fairly close to what we consider on topic, including superheroes, dystopias and alternate histories. Note, though, that we explicitly exclude certain sub-genres (such as spy-fi) which doesn't help matters.
I found an interesting blog post about how various genres relate to "speculative fiction," even discussing how some science fiction is not speculative fiction, and how what is science fiction today may not be science fiction tomorrow. (Which, I must note, is another discussion that we periodically have here!) Actually, the same might be said of fantasy, if we reach back far enough to a time where a story might include intervention from an angel or a saint - things which were part of the accepted worldview at the time, but which are now purely fantastic elements to us.
This post asserts that some stories that don't contain any speculative elements (well, other than the basic "what if?" used to construct the scenario) can still be science fiction. I'm not sure we can argue with that; for example, what if the crew of the ISS needed to escape a disaster and lost their re-entry vehicle? Is someone going to argue that Gravity isn't science fiction?
(Editorializing slightly, part of the problem is that we're trying to define the boundaries of a genre that exists, in large part, to challenge bounds.)
I think an absolutely excellent example of "fictional setting, but essentially historical" is The Lions of Al-Rassan by Guy Gavriel Kay, which is explicitly modeled on Moorish Spain and contains no overtly fantastical elements. Nevertheless, as it is by a known fantasy author most of whose modern work has very toned-down fantastical elements (basically just some ghosts in Under Heaven), it would be considered a work of fantasy. (One might argue that simply applying somewhat modern sensibilities to characters in the 13th century is ahistorical enough to count as fantastical.)
On the science fiction side, a comparable borderline example might be Spook Country, which is essentially a present-day thriller with no futuristic elements, invented places, etc., but is written by William Gibson in a way that nevertheless makes it feel like science fiction.
I don't believe it's possible to have a hard-and-fast rule about what is, and what is not, Fantasy. I think to some extent we have to rely on the author's intent to portray a world that is, or is not, ours - to whatever extent we can discern that. We can also use the author and/or the publisher as cues. In some cases it might simply be that we have to defer to a poster's belief that a work is fantasy, or we may need to discuss an individual work here.
(I've been told by people more widely read than I that there is a distinctive style or pattern to most SF that "feels" different to non-genre writings, but I don't think I'm aware enough of that to use it as a criterion.)