i see a lot of great ideas so far (that were totally shit on) but let me give you my two cents.
immersion is a great difficulty enhancer. a map that i can normally burn right through is exponentially harder when i feel sucked into it.
eternal was great at this...not really difficult maps...but really immersive maps that made you feel like you were in there and introduced monsters with a feeling of "holy shit what is that" even though youve seen them a thousand times.
that feeling of surprise is something a lot of people neglect either by making platforming "vanilla" maps or slaughtermaps....the former im okay with but i fuckin turn off slaughtermaps mad quick. if the first room opens four doors and you get jumped from both sides im out.
so you should look at getting into peoples heads and making them feel claustrophobic or boxed in, or like theyre being hunted...coupled with careful resource placement and youll see their play style change and theyll play with less "blow everything away" and more "oh shit its going down".
you can shoot that down all you want but thats definitely something ive noticed changes my playstyle and something my entire mapset is predicated on.
also spawning a chaingunner behind the player to chew down their health doesnt hurt.
but to be honest ive had a hard time making maps harder with the normal shit...so i have to play mind games.
one thing ive learned though is using enemies for their purpose.
want some shit to mow down? imps and zombies.
little hitscan action? Sgts and chaingunners (especially good for ranging you and making you shuck and jive to get to them)
wanna get the player moving? demons and lost souls are good for backing someone up...especially after some down town...they turn a corner and theyre getting rushed by some demons. pain elementals are great for this as well obviously. i try to avoid placing them in open areas or shit gets out of hand.
wanna set your player up against something heavy and play smart? burn their ammo with easy enemies and set them up against a bruiser. two hellknights and a baron is a great challenge.
spawning in a shitload of spiders is a good way to get them in an arena setting. they end up running around in circles...coupled with some other enemies and you have infighting and some good shit happening.
cacodemons, mancubus and turreted revenants can break up the tedium when fighting simpler enemies (revenants also a good way to kinda say "back up" after opening a door.
ETC.
sounds like you need to kinda figure out what you wanna do...and if doom doesnt have enough of that for you, play a different game. I still love all the mechanics and the enemies present enough of their own challenge i dont mind the "same ol same ol"
suck your player in with a good map...and then blow their fuckin face off with well placed enemies and scarce resources. straight up.