The other, which also has a short answer, is, "Why don't you read/write books set in the real world?" The answer to that, of course, is, "What real world?"
The fact is that no fiction, of any kind, is set in some objective world that we all inhabit. All authors invent a world to set their stories in; it's just that fantasy authors are more honest about it than most.
There are many ways of doing this, some more subtle than others, but perhaps the two main methods are the choice of what to include and exclude, and the presentation of moral beliefs as fact.
The inclusion issue, of course, can skew what we see as reality in any context. Some years ago, there was a TV advertisement for one of the "quality" newspapers (I forget which) that was based on the proposition that you have to see the whole picture in order to know what's happening (which, naturally, that paper provided).
It presented three short clips. In the first, a skinhead youth runs up to a respectable-looking man on the street and shoves him hard. In the second, the same skinhead is lounging on a street-corner when a police car draws up. An officer leans out and shouts, and the skinhead turns and runs away from the car.
The third clip shows the whole scenario. The man is standing under a cradle of bricks that's coming loose. The skinhead, alerted by the officer's shouted warning, runs over and, at some risk to himself, shoves him out of the way just as the bricks come crashing down. Without telling a single lie, the first two clips had the skinhead tried and convicted, whereas he's actually a hero.
Of course, even this isn't "the whole picture". That might include the reasons why the accident happened (personal carelessness or corporate corner-cutting), who the man was waiting for, why the skinhead was at a loose end, what in his background made him risk his life to save the man... and so on, right back to the Big Bang.
In a work of fiction, some aspects of the "real world" are included, and some aren't. This may be nothing more than what is or isn't relevant to the story, but even so, the author has to choose what story to tell. It's often been pointed out, for instance, that Jane Austen's novels include none of the poverty, vagrancy and social unrest that was widespread at the time. That doesn't mean she was unaware of it, or even unconcerned about it as a person, but she chose to write stories in a fictional reality in which those things didn't exist.
That doesn't make the stories "wrong" or even unrealistic. They're very realistic about what she chose to present, and no author can cover everything. It does mean, though, that they're set in a world of their own.
The presentation of moral belief as fact is a more complex issue. Consider, as an example, a familiar type of character: the maverick cop who ignores the rules, and sometimes the law, to get the bad guy. This can take place in either of two distinct realities (or a spectrum in between): the reality in which liberals are allowing criminals to rule the streets, and only ignoring the rules can safeguard decent people; or the reality in which the rules are there to protect the innocent, and ignoring them destroys lives.
This isn't just a matter of what happens in the story. The different versions are actually set in different fictional realities, in which the author's belief (or the belief s/he thinks will appeal to the target reader) is a universal fact. These are invented worlds, just as much as anything produced by fantasy.
Does it matter? Ideally not. When fantasy readers read of fictional realities, we understand them in the spirit they're meant: we learn broad lessons from the characters and situations or we just enjoy them (preferably both) but (apart from the very few who are always going to be out of touch) we don't believe in their objective reality.
Unfortunately, not all consumers of fiction have had our practice, and often seem to believe precisely that various kinds of popular fiction are objective reality. Soap operas are particularly adept at creating this illusion, since their stock-in-trade is that they're presenting some kind of slice of life, even though they're doing nothing of the kind. I've had arguments with otherwise intelligent and level-headed people who've based their arguments about the real world on what's happened in Eastenders or Coronation Street. If they're challenged, of course, they'll acknowledge that it's happened that way because of the author's choice, but it doesn't seem to occur to them without the challenge.
Fictional realities are essential for stories. No story could be written without them, and they only become a problem when the reader or viewer isn't aware of their presence. Perhaps those people should practice by reading fantasy.
So, the next time you're asked, "Why read fantasy?" perhaps you could simply answer, "To keep in touch with reality."