Decided to dig around and see how GZDoom does things in this regard, and it seems my hunch was on track; both damage and GZDoom's kickback attribute influence how much an enemy is knocked back.
As you mentioned, Doom weapons have a default kickback of 100; this seems to be defined in ZScript as part of the DoomWeapon class (wiki, source). Among its other modifications and restructuring of P_DamageMobj, GZDoom actually handles kickback in ZScript as well, in the Actor class's ApplyKickback method (wiki, source). The relevant lines seem to be:
thrust = mod == 'MDK' ? 10 : 32;
if (Mass > 0)
{
thrust = clamp((damage * 0.125 * kickback) / Mass, 0., thrust);
}
Breaking this down, it sets thrust to an initial value of 10 if the attack has the "MDK" modifier, which I'm guessing is from the mdk debug command, or 32 otherwise. Next, if the enemy has mass, it calculates a thrust based on the attack damage, the weapon's kickback (which could be modified by mods, mind you), the victim's mass, and some coefficient of 0.125 that I'm guessing normalizes the result to something expected for thrust values. This result is clamped to always be between 0 and either 10 or 32 depending on that initial value.
Now, I haven't looked much farther than this, so I don't know how different GZDoom's thrust and speed values are, and if how they're calculated differs from vanilla Doom, but based on that calculation you could try setting the weapon kickback to 1 (not 0, as that won't move the target at all) and see if that feels closer to vanilla.
