Just throwing in a few more of my pennies into this discussion, heh.
I agree that using ripped Doom 64 resources is probably not the way to go. Well it really depends on the actual goals of the project, which seem to be a bit wishy-washy at the moment. But I will say that Doom 64 resources mixed with vanilla Doom resources are going to create some awkward clashes.
First I think you guys should decide on how this project is going to reinterpret Doom 64. Are you going to go pure vanilla both in terms of port compatibility and resources? If you do that, there's going to have to be a lot of room for reinterpretations because the vanilla resources don't really match up to the same settings found in Doom 64. Personally I think it would be more interesting to fudge the strict interpretation mindset and allow for a few custom textures and stuff - mostly in regards to recreating certain key resources found in Doom 64, but in vanilla Doom's style. If your going to add in some of the extra enhanced monsters, it follows that adding some extra resources would make sense.
For example, someone can make a new sky texture that is the "space sky with mountains", but have it match the vanilla style. There may be some existing custom resources out there that already fill in this need. Or, alternatively, it might be possible to edit the Doom 64 resources to the point where they don't clash as bad.
Although to a point, this approach is a bit unfeasible. For example, I'm not sure it is going to be easy to find someone willing to remake the Motherdemon sprites to better match vanilla Doom's style. In that case, your probably better off ripping the sprite...it's not perfect, but it is definitely waaaay less work.
Also in regards to the dehacked, I think you should decide to either go one way or the other. Setting up the project to work both with and without a patch for enhanced/new monsters will be a really big pain to balance. Having a "special edition" version which adds things like the Unmaker in ZDoom is probably an ok thing to do so long as the maps aren't made to rely on the inclusion of said Unmaker. If you want to bump up to MBF, you could possibly get away with adding an Unmaker-ish weapon to the stock weapons using the beta BFG codepointers and projectiles. That probably goes outside the bounds of your goals though :P
Those are just my thoughts. If I were running this project, I would probably do this - recreate the enhanced monsters using the extra frames from the unused monsters, add in the new sky textures in the vanilla style, probably rip the Motherdemon sprites (unless someone is willing to come forward and make new sprites), and either make new textures/sprites as needed using stock resources or new resources in the vanilla style. From there the maps could just be vanilla-ized versions of the layouts from Doom 64, possibly with new bits and some reinterpreted versions of some areas (which is probably going to be mandatory since vanilla can't do some of the things that Doom 64 can). I'd probably also go Boom compatibility just for the easier dehacked editing and inclusion of extra map editing features...buuut that's more of a personal preference than anything. Most of you guys will probably prefer pure vanilla, which is probably better for creativity in the long run :)