Some of Arthur C. Clarke's most well-remembered work belongs to this category of the inscrutably cosmic. In Rendezvous with Rama, we have an empty starship passing through our solar system. Exploring it really gains little more than before. While I haven't read the sequels, but it's fascinating that people are unsatisfied with leaving the central mystery dangling.
Likewise, Stanley Kubrick didn't want "The Sentinel" to end there. He and Clarke built the film 2001 around it. While 2001 remain enigmatic, it has a plumb-able mystery. "The Sentinel" is artifact found on the moon. When humans try to open it to find out what it is, why it is, who might have left it they break it and turn off the beacon. What is it? The narrator rattles through some possibilities but settles on one. In doing so, they've entered into the space-faring age of a species--the ability to travel space and to use atomic power. After discussing the human-aggrandizing leaps in progress and technology, the narrator states,
"[T]hose [aliens] whose duty it is will be turning their minds upon Earth. Perhaps they wish to help our infant civilization. But they must be very, very old, and the old are often insanely jealous of the young.... [W]e have set off the fire alarm and have nothing to do but wait."Implicit here are two attitudes: One, that humans are only newly civilized and remain childish, even mischievous and/or dangerous. Two, under discussion here, is that the mysterious other will not like us, even to the point of malevolence.
"The Nine Billion Names of God" takes this a step further. Two computer programmers design a program two help the lama count the nine billion names of God. They realize that the monks intend that the end result of the naming is God destroying the planet. The programmers escape because they fear being killed when God doesn't step in. That God does step in shows the hubris of the programmers' assumptions about the nature of the universe, of course. (Interestingly, the stars go out, one by one--due to their snuffing out or blocking their light or simple annihilation of all matter without explosions.) But again, there is that other that is so mysterious as to be malevolent.
H. P. Lovecraft often dabbled in this territory, but with the background music set to horror, played throughout, effectively, "I fear the universe. What's that? The indescribeable! The horror, the horror!" No doubt Clarke read Lovecraft but his effect varies in that he sets the tone to discovering the unknown but never quite reaching due to our limitations as a species. Somehow Clarke's is more surprising or horrifying in the sense that the ordered universe may not be ordered as we supposed. Certainly, science records the hubris of scientists who said, "That cannot be," and were proved wrong. But is a malevolent ineffable the only check for hubris?