I think you already gave a nice answer to that issue. Some fields in art (let's say all the "serious" art) requires a certain amount of efforts to be at least understood, and consequentially a certain amount of time and energy: now, most of people lack of both for various reasons (work, family, duties in general, etc.)
Let's say they cannot/don't want to pay the price in order to understand them, and that's okay. I think it has always been like this through history.
I don't believe any kind of "cultural experience" is required in order to be happy, despite the link knowledge-wisdom-happiness is very popular through all the history of philosophy. I'm not saying such things cannot help, but simply they are not the only path.
The real special and unique thing that the experience of culture (don't know if I'm using this word properly in english, with it I mean all cultural products like arts, philosophy, architecture etc. etc.) gives us is the ability to understand the world in a better way, and to organize our thoughts consequentially.
The real big loss of a life without intellectual experiences is the lack of social and political consciousness in the individual, imho.
Because listening to Bach, in our society, requires some preliminary efforts: you have to encounter his name somewhere, then be interested in him, then search his works and check if you like them. I know it sounds very easy to be made, but maybe for some people it is not so banal.
I'll never forget the first time I listened to Trout Mask Replica by Captain Beefheart: it was one of the most ugly and unlistenable album I've ever encountered in my life.
I red a lot about it, and I listened to it multiple times, and now I'm able to understand it conceptually and to appreciate it aesthetically. It required time and energy that I was willing to invest 'cause I love rock music, and lot of artist and critics that I respect loved that album.
Maybe if I was too busy working 12 hours a day I wouldn't even have found the time to find his name exploring the internet, you know what I mean?
I don't know if my point is clear, but I can summarize saying that art is something very elitist and most of the time talking about music/art tastes is almost like talking about the social class you are in.