I raelly hate robots.txt as it assumes that archiving is bad. There should be some exceptions for sites that used to not have it. Such attitude is why a lot of classic pages are gone.
There's nothing wrong with robots.txt that a bit of fine tuning can't fix. In any case, just like the lock on your front door it only keeps the honest robots out.