From reading R_ProjectSprite I guessed that projection is something like "the distance from the viewer's eye to an imaginary plane onto which the game world is projected". Things nearer than that distance are magnified, and things further away are shrunk.
I was thinking something similar a while ago, and increasing the value of projection does seem to make things bigger.
I think projectiony exists to take Doom's graphics aspect ratio into account. Its value is projection scaled by 6/5, the ratio of pixel height to pixel width in DOS, which apparently had non-square pixels. I think if you remove the calculation and just set projectiony = projection, then running the game in a 4:3 aspect ratio resolution would make everything look short and fat.
Yep, I saw a few comments in one source release talking about projectiony taking the aspect ratio into account, although I thought it was more to do with handling more modern resolutions that aren't 4:3. Projectiony isn't present in Chocolate Doom, so I am guessing it also isn't present in the original source.
projection also seems to be related to focal length. I've found using its value in R_InitTextureMapping, instead of the focallength parameter the function calculates internally, gives the same results in 4:3 modes. Furthermore, this appears to maintain a 4:3 aspect ratio even if one runs the game in a widescreen resolution (no stretching of the player view horizontally, but increasing the field of view).
I discovered this by accident when trying to do non-4:3 aspect ratio screen/window sizes in my engine. My notes say: Honestly I don't know why this works. It was determined empirically by trial and error, and by comparing with PrBoom+. The focal length is half the view width when the field of view is 90° - "think of a right-angled isoceles triangle" - but is this actually relevant, or just coincidence?
Haha, yeah I know the feeling. Several times I thought I'd finally made some some progress on working out what R_ScaleFromGlobalAngle was doing, only to realise that my results were purely coincidental and meant nothing. It's an interesting idea, though. The comments just above the focallength assignment in R_InitTextureMapping state:
// Calc focallength
// so FIELDOFVIEW angles covers SCREENWIDTH.
Sounds like it could be related. Hopefully someone can confirm or deny someday but for me, based on the amount of time I've spent investigating this, it's not quite enough to justify anymore effort. Thanks for mentioning it, though, and I am also grateful for all the comments that have been posted so far, even if they don't directly relate to the subject of the thread.
Maybe I'll revisit all this at some point in the future, but for now, I give up!