I guess it depends on whether the music/artists you listen to are gearing the release towards the singles market. For much current electronic music, for example - which is not so geared - small collections of tracks might be released, with the collection taking its name from the first track, just as happens with a single, but other than the fact that the title track is the title track, there's nothing that really distinguishes it, in terms of significance, from the other tracks, so the a/b side distinction carries less weight. One of my favourite autechre tracks is technically a b-side, but that's certainly not how I think of it.
With dance music - house/techno/dnb - where artists tend to releases pairs of tracks on a single 12" , people talk more in terms of a/b side, but obviously it has a certain literal significance in that context, and again, there's not such a sense of the b-side being some special extra - it would certainly be perverse to not listen to it, or to disregard it, just because it isn't the title track. Perhaps the converse attitude occurs more when you think of the artist's body work as existing primarily on albums, with b-sides as non-album tracks, and so as a sort of bonus content. But this way of carving things up is not well suited to a lot of released music.
As for unreleased or bootlegged stuff, in my experience that tends to be more of a curiosity than something I'll return to for repeat listens. Sometimes it's interesting, particularly if you're interested in tracking the development of a particular artist, but often it isn't, and in cases where 'unreleased' stuff is commercially released, I often find myself thinking that there was no need to release it. An example here woud be alternative takes of jazz tracks - very rarely are they superior to the released take.