I would like to respond to Virtus, risking repeating things already said because I haven't managed to read everything (what a discussion!), but trying to remain at a simple level.
Contemporary music is like jazz; it is vast, so the question isn't quite well-posed...
I think that in every listening and every observation, it is generally pleasant to recognize a sign, a synthesis, especially a coherence, which sometimes can also be subjective and could even be unconscious (on the part of the author). This doesn't mean you must necessarily find it, but in listening, you must be well-disposed and without prejudices or preconceptions.
I also think that knowledge increases the capacity to appreciate and therefore also the enjoyment. It also applies to Beethoven and the other greats of the past, if you think about it. The analysis of the piece and the technological background (knowledge of the violin and piano, like a filter or frequency modulation), as well as the cultural context in which the piece itself was born, I think gives an extra boost. I see nothing wrong with that.
Nevertheless, it is not mandatory: I remember "arguing" with a student at the conservatory because I said that listening to Maderna gave me a sense of peace, but it was the truth, beyond what the author wanted to convey and my knowledge on the subject (and then the sense of peace can also derive from the feeling of understanding and harmony with the author and with the listening experience, but he was too "intellectual")
I would advise not limiting oneself too much: our auditory and cognitive system is formidable, and setting it in motion can provide much satisfaction!
You make me think of when at school I tried to reveal to my classmates that listening to Genesis (just one example) was much more pleasant for me than most of the light music promoted on the radio, and that it would probably have become so for them too if they had had the patience to listen to it a couple more times... It's not just a matter of taste, but also of willingness to listen and trust in others.
I find various listenings, by very different authors, to be wonderful: Bach, Cage, Marcello, Chopin, Schoenberg, Webern, Orlando di Lasso, Roscoe Mitchell, Miles Davis, Terry Riley, and so on, and I cannot think of dividing and categorizing my enjoyment. The secret lies in listening, in the predisposition towards the work and towards others in general, and I find your observation sad and biased.
Regarding sound, I actually find it to be a boundless and extremely interesting world that can have an important weight in a work. It has always been this way, except that in modern times, thanks also to electronics and computer science, we have acquired an enormous capacity for investigation and control, and it is natural to remain fascinated by it and that, in certain cases, attention to "sound" is preponderant.
Happy listening!