Those who dislike fantasy are very often equally bored or repelled by science. They don't like either hobbits, or quasars; they don't feel at home with them; they don't want complexities, remoteness. If there is any such connection, I'll bet that it is basically an aesthetic one
The Language of the Night: Essays on Fantasy and Science Fiction (ed. Ultramarine Publishing, 1980) - ISBN: 9780399504822