Turning off r_shadows seems like the only way to reduce those framerate "pits" and it's my suspicion that it's because it's one of the few commands that actually reduces the number of polygons on the screen (shadows are, I would assume, polygons themselves).
To me that paraphrase from iD explains a lot. By emphasizing dynamic shadows they are dramatically increasing the polygon count (or something to that effect). Many modern games go out of their way to have dynamic LOD to reduce the polygon count which keeps the framerate somewhat steady in almost any indoor setting, but in Doom 3 if you look down a corridor and see a second room with its own detail you can pretty much count on your framerate dipping way low. The only default polygon reduction seems to be the culling that occurs on objects the player simply cannot see. Someone enlighten me if I'm wrong.
Try this experiment if your processor is somewhere around 2Ghz or less: find a point of view in the game where your framerate is at or close to its lowest point, one where you can just sit and stare (I have one point where I can stare with the flashlight on and it consistently sits at a framerate that dips down to 15FPS). Now, quicksave the game at this point and start tweaking some settings (do a vid_restart after each change). What I noticed is that almost without exception, ALL non-polygon related commands had no effect whatsoever on that framerate. I even cranked the resolution up one notch and added 4xAA and 8xAF. No matter what I did it stayed at that same exact framerate EXCEPT for when I turned off shadows.
AA is a very videocard-dependent feature, AF is a very videocard-dependent feature, and resolution is a very videocard-dependent feature. So I'm guessing I am extremely bottlenecked by something OTHER than my videocard. Obviously the likely culprit is the CPU or something directly related to it like the FSB. My specs: I'm running an Athlon XP at 2GHz on a KT133 chipset (20x100) with a Radeon 9700pro.
So again, if anyone knows of a way to cut out more polygons without screwing up the game TOO much, be sure to let me know ;) I'll try messing with the shadows, but it'd be a shame to lose them.
One thing this means is that maybe it's not enough to simply say "Doom 3 is naturally going to have crappy performance because it's using a state-of-the art engine." The engine is only partly to blame. As you can see, all these fancy features like bumpmapping and specular highlighting amounted to a hill of beans in my case; my card can handle it, and my CPU can handle it. What really seems to kill me, in this case, is more of a design decision: those darn dynamic shadows, and all that that implies.
