Public rating systems

[Brandon D. Lade]

As you probably know, I use the public rating system on /idgames to determine a wad's merits and if I should even try it out. But many of you have argued that either a public rating system is not a sufficient way to get the information, or even more so, that I ought to form my own opinion and keep to it no matter what the public thinks. What do you think?

By the way, when I say "rated well", I mean that it has at least 4 out of 5 stars overall.

[wildweasel]

It would be a good system if something could be done about the random, anonymous downvoters, i.e. the people who vote only to bring the score of a wad down, and for no reason whatsoever. Perhaps if the Doomworld accounts could be hooked into the /idgames frontend, like the /newstuff review system is?

[Brandon D. Lade]

WildWeasel said:

It would be a good system if something could be done about the random, anonymous downvoters, i.e. the people who vote only to bring the score of a wad down, and for no reason whatsoever. Perhaps if the Doomworld accounts could be hooked into the /idgames frontend, like the /newstuff review system is?

I don't understand you.

[40oz]

Sometimes people will play a wad for a minute, can't find a key or a switch, or will just be immediately turned off by the lack of detail in a wad, and ultimately give it a bad rating. It happens often. I wouldn't trust the rating system to tell me whats good unless theres well over a hundred voters.

[Jodwin]

Not to mention the many "acquired taste" wads, like puzzle wads, "ridiculous" slaughter wads, joke wads, etc. etc. With these wads in particular you should be very critical about the votes if they're being low and rather just form your own opinion. Because they're acquired taste and thusly most people probably won't like them, but some will, and you might be one of those. There's no shame in preferring redheads over blondes, and just the same way there's no shame in liking, for example, joke wads, no matter what some trolls would think.

[Brandon D. Lade]

Jodwin said:

Not to mention the many "acquired taste" wads, like puzzle wads, "ridiculous" slaughter wads, joke wads, etc. etc. With these wads in particular you should be very critical about the votes if they're being low and rather just form your own opinion. Because they're acquired taste and thusly most people probably won't like them, but some will, and you might be one of those. There's no shame in preferring redheads over blondes, and just the same way there's no shame in liking, for example, joke wads, no matter what some trolls would think.

I know about wads like that (Nuts, Mockery, etc.) and I do implement that exception and ignore ratings in such a case.

[Cupboard]

The rating system on idgames is, as it has been said before, anonymous. There is no accountability for what you write or how you vote. That alone makes it unreliable to a certain degree.

It does try to serve a purpose. People like to have a general idea of what they're getting into before they download it. That's why people read customer reviews on Amazon.com before they make a purchase. The idgames voting system should give you a general feel for what you are about to play but you shouldn't normally make a judgement call based on some stranger's review.

Can a small graphical display of red stars really describe all that a wad has to offer? I think you have to play something for yourself to form a proper opinion. You end up missing out on a lot if you don't think for yourself.

[myk]

WildWeasel said:
It would be a good system if something could be done about the random, anonymous downvoters,

There's also the upvoters, which are probably more people. Look in the comments and you'll see tons of 4s and 5s for WADs that aren't bad, but nothing extraordinary. It looks like they like it, so they give it many stars just in case someone else may give it less. Then there's the "balancers", which give many or few stars because of what others said or rated, as a reaction, and not according to the WAD itself.

Maybe the naming of the star system also impacts badly; "average" doesn't say much, except statistically, and 3 stars usually means "good" or "decent" in a 5-star rating system. Like, it's nice, and you can have some fun with it, but it won't blow you away and you'll probably forget it in sake of better stuff. It doesn't necessarily mean it's mediocre. Personally, I give out many 3s, some 2s and 4s (maybe more 2s than 4s), and very rarely the extremes. For the latter, the WAD either has to be plain awesome, genuine crap, or broken.

0: Broken
1: Decidedly lame
2: Mediocre or subpar
3: Decent to good
4: Exceptional
5: Top-notch

The problem with hooking the database to the forums is that it'd force people to sign up, and the database is independent of the forums. Also, people could still find ways to abuse it. In that case, also involving the forums in the process. You'd also be curtailing or complicating a degree of anonymity which is being used decently by many people.

[Brandon D. Lade]

myk said:

There's also the upvoters, which are probably more people. Look in the comments and you'll see tons of 4s and 5s for WADs that aren't bad, but nothing extraordinary. It looks like they like it, so they give it many stars just in case someone else may give it less. Then there's the "balancers", which give many or few stars because of what others said or rated, as a reaction, and not according to the WAD itself.

Maybe the naming of the star system also impacts badly; "average" doesn't say much, except statistically, and 3 stars usually means "good" or "decent" in a 5-star rating system. Like, it's nice, and you can have some fun with it, but it won't blow you away and you'll probably forget it in sake of better stuff. It doesn't necessarily mean it's mediocre. Personally, I give out many 3s, some 2s and 4s (maybe more 2s than 4s), and very rarely the extremes. For the latter, the WAD either has to be plain awesome, genuine crap, or broken.

0: Broken
1: Decidedly lame
2: Mediocre or subpar
3: Decent to good
4: Exceptional
5: Top-notch

The problem with hooking the database to the forums is that it'd force people to sign up, and the database is independent of the forums. Also, people could still find ways to abuse it. In that case, also involving the forums in the process. You'd also be curtailing or complicating a degree of anonymity which is being used decently by many people.

I understand the upvoters and balancers. People will vote higher than they necessarily think to lower the threat of the overall rating becoming lower, and people will magnify a positive or negitive vote if the overall consensus (average) is on the other side of the pendulum. You also talked about what the ratings SHOULD represent (i.e. 3=Decent to Good instead of just 3=Average). In your opinion, if I was to use the /idgames ratings as an indicator as to the quality of the wad, what should be the "break point" in general in deciding if a wad is worth playing or at least worth checking out? Should it perhaps be lower than the 4 stars I have it set at? And in regards to another question I asked at some point, can you provide a rough scale relating the total number of votes a wad has to how good of an actual indicator the overall rating might be? All that would be appreciated.

Maybe it would even be good to add options to the poll that talk about if my 4 star "break point" is too strict, and another option that uses the number of total votes as a variable. Just a suggestion.

Finally, sorry, but I don't understand your problem with hooking the database to the forums. Can you please explain it better?

[Brandon D. Lade]

TheCupboard said:

The rating system on idgames is, as it has been said before, anonymous. There is no accountability for what you write or how you vote. That alone makes it unreliable to a certain degree.

It does try to serve a purpose. People like to have a general idea of what they're getting into before they download it. That's why people read customer reviews on Amazon.com before they make a purchase. The idgames voting system should give you a general feel for what you are about to play but you shouldn't normally make a judgement call based on some stranger's review.

Can a small graphical display of red stars really describe all that a wad has to offer? I think you have to play something for yourself to form a proper opinion. You end up missing out on a lot if you don't think for yourself.

So you find it to be no problem if I enjoy something that the overall public does not? In other words, are you saying that that should NOT make me feel like an outsider or anything along those lines?

[Enjay]

TheCupboard said:

The rating system on idgames is, as it has been said before, anonymous. There is no accountability for what you write or how you vote. That alone makes it unreliable to a certain degree.

Conversely, not being able to be traced may make some reviewers less shy about saying what they feel, thereby increasing the accuracy.


My opinion? Try anything that takes your fancy and make up your own mind. I quite like to compare my opinions to the thoughts of others but I'm not usually swayed one way or another by the votes. Lets face it, most of time it doesn't take too long to DL the WADs released in a week and give them a whirl. So, what have you got to lose?

[Cupboard]

As myk and Enjay have noted, I agree that anonymity can be beneficial for expressing the author's true beliefs in a written review.

Brandon D. Lade said:

So you find it to be no problem if I enjoy something that the overall public does not? In other words, are you saying that that should NOT make me feel like an outsider or anything along those lines?

No, I would not have a problem if you enjoy something that I do not. Since we are dealing with a game (and not some sort of social issue that considerably influences society :P) it's not really serious business. Download some levels and give them a whirl. Everyone has different likes and dislikes, skills and weaknesses. In fact, if you like something that everyone else dislikes, leave a comment on the idgames page or current newstuff topic (you don't seem to have that problem). Bring your ideas to the table because no one likes stagnation.

As for your "outsider" concerns... There ought to be some sort of "outsider" who offers a dissenting opinion or else the majority won't think twice about their decisions. Someone should be there to argue for the other side. That's why there are lawyers. ;)

[Stilgar]

It's an okay tool to get a generaly idea but not always a good indicator, and not just in the sense of good stuff being underrated either, I've seen retarded wads have high ratings also.

I find that reading the comments that go with the review often gives a better idea than the star rating itself.

I also agree that the naming of the star ratings is bad, e.g. people marking the supposed first wad ever as "legendary" even though the map sucks balls.

[TheeXile]

Edit: -snip- (offtopic)

Not everyone likes slaughter maps, for example, so a review for a well-created slaughter map from someone without a taste for them could possibly not represent it well.

[Khorus]

But someone who likes slaughter maps would still play it anyway. That's what the screen shots of the map are for, I see what sort of map it is from the review. Then I look at the screens. If I'm still interested I'll play it.

[TheeXile]

Khorus said:

But someone who likes slaughter maps would still play it anyway. That's what the screen shots of the map are for, I see what sort of map it is from the review. Then I look at the screens. If I'm still interested I'll play it.

That's almost like judging a book by its cover. Never mind sometimes people suck at taking effective screenshots (like me :P).

[chexwarrior]

Its an okay tool, for most wads it gives a relatively good indication of quality. However, the assholes who for no reason down vote a wad can really suck. I tend to ignore the stars and read the actual comments. But then again you'd have to ignore the "THIS WAD SUUUUCKS LOL" comments too.

[Pirx]

brandon, i thought of asking exactly this question after you claimed that star ratings reflect a wad's quality. i didn't want to appear like a troll after 2 threads were locked because of this though.

in a nutshell, i don't think that votes reflect a wad's quality accurately enough to base my choice of what to play on them. people pimp their own creations, vote high / low just because a wad fits / doesn't fit their particular taste, and many still don't have a clue and give a bad rating only because their skills are not up to a wad's difficulty for example.

i prefer to read recommendations in forums, cacowards, review sites, news and the like. i rather trust someone who can write a decent review than some random guy who knows to use a vote function. then i play that wad and don't like it for whatever reason? delete it, period. but i don't refuse to test it.

[TheeXile]

Enjay said:

Conversely, not being able to be traced may make some reviewers less shy about saying what they feel, thereby increasing the accuracy.


My opinion? Try anything that takes your fancy and make up your own mind. I quite like to compare my opinions to the thoughts of others but I'm not usually swayed one way or another by the votes. Lets face it, most of time it doesn't take too long to DL the WADs released in a week and give them a whirl. So, what have you got to lose?

What's the average duration of a given WAD? 30 minutes? An hour?

That's a lot of time for someone who has a busy schedule, potentially. Never mind it can tax patience (an even more valuable resource, arguably), considering a dominant portion of released WADs are crap.


Plus that time doesn't factor in time (and ability!) in deciphering existing reviews to find good maps to play. Most people are idiots who will play anything with flashy screenshots or porn textures, for example. :P



Edit: Oh I know! Let's have a system to review peoples' reviews! Wahaha!

[Khorus]

TheeXile said:

That's almost like judging a book by its cover. Never mind sometimes people suck at taking effective screenshots (like me :P).

It gives a very quick and accurate indication of quality. If all the screens are of empty square rooms then I know it's made by a newbie. If it's full of bestiary monsters and more damn black imps I know that it's just another ZDoom shitfest. But if it shows some interesting architecture or originality then I will check it out. I'm not going to look at each review and the screenshots and think "We'll I can't be biased till I've played it." because I don't have hundreds of hours to do that with each wad I come across.

Oh, and mappers with ego need to fall off a cliff. :-)

[esselfortium]

TheeXile said:

Edit: Oh I know! Let's have a system to review peoples' reviews! Wahaha!

Can we have a system to review all these damn polls?

[esselfortium]

A brief summary of events leading up to the creation of this thread:

I base my opinions entirely off of arbitrary rankings made by strangers on the internet.
Hey Brandon, why are you basing your opinions on arbitrary polls and rankings? Think for yourself, bro!
Uhhhhh, shit! I better make a poll so they can decide for me whether they're right or not!

[Creaphis]

Oh god. The more I think about it the stranger it gets.

If Brandon learns from this poll that his opinions do matter, which is how the poll results are leaning, then that gives him the grounds to disregard the poll and continue following his own conviction that his opinions do not in fact matter which forces him to once again take the poll results into account, and so on into infinity.

Likewise, if Brandon had decided not to hold a poll at all, and independently decided that his opinions do not matter, then he would have no authority to decide that his opinions do not matter so they therefore may actually matter but in that case they wouldn't, and we devolve into a liar's paradox once again.

Brandon could avoid that snag by independently deciding, without a poll, that his opinions do matter. This, however, is circular reasoning, a logical fallacy which I suppose we all engage in every time we choose to share our two cents on any topic.

The final possibility is the only one with no clear logical difficulties - to decide, based on the opinions of others, that one's own opinion does not matter, which seems somewhat like what Brandon has been doing all along, making him superior to us in the strength of his reasoning.

What a magnificent intellectual mess.



(Yes Myk, I know this is resolvable. I'm just having some fun.)

[Reevys]

If I ever cared about the ratings people give things, I'd have half the games, half the movies, and half the CDs that I do.

I prefer to form my own opinion on everything, so I try to ignore what other people say most of the time. I do listen to people about certain things though, such as if they mention some totally game breaking feature. But if they mention something that simply annoyed them in a game, I'll ignore it, since I don't actually know if that particular feature will annoy me personally.

So to answer your question, I voted for "It's a good tool to determine the merits, but I should still form my own opinion and not let such ratings alter it."

[GreyGhost]

Myk said:

The problem with hooking the database to the forums is that it'd force people to sign up, and the database is independent of the forums. Also, people could still find ways to abuse it. In that case, also involving the forums in the process. You'd also be curtailing or complicating a degree of anonymity which is being used decently by many people.

I don't see why it shouldn't be possible to maintain some level of anonymity if the database and forums were linked - admittedly doing so may be more trouble than it's worth. While I'm no expert when it comes vBulletin or php there's surely some way of suppressing the display of member names in the ratings system (if the member wishes to remain anonymous) but not from the audit trail - so the admins could weed out system abusers. People who don't want to sign up could still leave star ratings but not comments.

@Brandon - while I've found the database ratings system to be handy guide, ultimately I have to form my own opinion.

[kristus]

What other people think is uninteresting to me. I've played many wads that I couldn't stand that people loved. And loved wads that people hated. Though right now I can't think of any specifics.

Comparing my opinion with the rest of humanity only makes me depressed cause I think I'm always right and everyone else is retarded. :p

[Gez]

I don't pay attention to ratings and don't vote myself. Word of mouth, a well-written readme and a few convincing screenshots is more useful and more reliable IMO.

[Kyka]

So let me see, Mr Creaphis, if I have understood you...

Unless I am mistaken, then you are saying that, as a matter of fact, your opinions on the subject of whether opinions actually matter do matter, as judged by the fact that you have shared them with us. As a matter of fact, this very sharing only ultimately matters if your opinions do not matter to you so much that you hold them too closely to yourself, thereby not allowing them to matter to the rest of us, in which case you gotta ask do they actually matter, so then obviously this matter must be worked out by a poll, in order to tell us whether or not our opinions matter, inasmuch as..."

TO THE POLLS!!!!!

(magnificent stuff, creaphis. I enjoyed reading your discourse a lot.)

Oh... and did I vote on this poll? well, as a matter of fact...

kristus said:

Comparing my opinion with the rest of humanity only makes me depressed cause I think I'm always right and everyone else is retarded. :p

Why does this depress you? Hell, I could live with being always right... I could probably even live with simply thinking I was always right. [edit] 'Course, it should have occurred to me earlier that you may have been being sarcastic.[/edit]

[esselfortium]

I'm not sure what I should vote in this poll. Maybe I should create another thread to help me decide. What do you guys think, should I make the thread????

*anxiously hovers over "Submit Reply" button for a few minutes before clicking*

[Jodwin]

esselfortium said:

I'm not sure what I should vote in this poll. Maybe I should create another thread to help me decide. What do you guys think, should I make the thread????

*anxiously hovers over "Submit Reply" button for a few minutes before clicking*

Based on the fact that the votes in this poll are relatively evenly distributed, I doubt you would get a definitive answer anyway!