On a STG (Shooting Game, like Gradius) forum I used to frequent, after the big ~15 or so games had world records that nobody could touch, they started making up weekly/monthly competitions.
Forum mods weren't involved, it was just users passing the torch around (there was apparently a sort of understanding that there shouldn't be 2 competitions going at once), and someone would also maintain a data table showing statistics across all competitions. Mostly lesser-known titles were chosen to compete on, and demos/recordings were only required for the top 3 or so (of course anyone was welcome to contribute their demos/recordings though). Once in a while someone would hold a competition on a big popular title, which lets the hardcore guys strut their stuff and keeps things fun.
Some of the more complicated competitions involved people joining teams, usually "led" by one or two of the more hardcore guys. It served a sort of social function, with people sharing tactics, techniques etc. in order to crush the other team. It would also get competitive enough for there to be a sort of metagame, where people would withhold stellar demos until the last day of submissions ;-P
But yeah, a few dedicated people maintaining a public scoreboard with links to winning demos, pruned from the individually run week by week/month by month threads. Only one game setting was allowed, so as to keep everyone competing against eachother (that is to say, having a competition that accepted UV, Nightmare, -fast, Tyson, etc. would have too much fragmentation of which category people would participate in). The person running the competition picks the settings, usually default settings to keep participation high.
Too many rules will kill it. Simplicity and freedom encourage participation, e.g. there was one competition where you had to "sign up" by emailing some guy, and that was enough to make me not bother participating. I think it'd also be important NOT to set an engine restriction, just a skill setting restriction IMO.