Captmellow: awesome idea. I remember Fanatic was going to do something similar, but I don't think it ever took off.
When I did reviews for Team Insanity (I'll continue after Millennium is released), I began a rating system that went like this:
Item Placement: This counted for things like weapons, health and ammo, as well as decorations like torches, lights, candleabras, bodies, etc. I suppose this rating could be broken down into decorations and equipment if necessary.
Monster Placement: This goes somewhat hand-in-hand with the Item Placement category, but I figure it was worthy of it's own category. If there wasn't enough ammo or health to keep up with monsters, I usually deducted it from the Item Placement category under the idea that it was too little ammo, instead of too many monsters. Things I counted in this category besides the obvious were if the author used any tricks with monsters, such as pick up a red key and suddenly 6 hell knights teleport in, or walk down a hall and have monsters pop up out of the ground; surprises like that which go beyond just sticking guys in a room. Likewise, I'd take away points if they had a load of revenants 3,000 pixels across a giant open space which would be impossible to kill no matter how many bullets or rockets you fired over there.
Level Design/Layout: This is where all things like lighting, atmosphere, theme, texturing, etc. come together. Did the lighting make sense, did the level suddenly go from tech to hell without a reason, did the author create a totally real environment, totally fantastic, inconsistent, sloppy?
Gameplay: This could be seen as a mix of the first three, but I based it on how well they came together as a whole.
Innovativity: This is basically how original the level was. Is it a rehash of everything everybody's ever done, or is it a totally unique map? Do they use new concepts, new tricks, new architecture, new themes?
Misc: Sometimes I add categories based on addition things the author put in the map, like new sprites, new textures, new music, and how well each of those work with the overall map.
And then, I give it an overall rating.
Hope that helps a little.