Rating System Qualms



Lately I’ve been having a lot of doubts about the rating system here and how it is used. I’m having the urge to scrap it completely. Read on to find out why.

I originally switched the system from a 100/100 scale, to a 5/5 Cog based one with additional side awards. This worked fine for a while, but I started to see problems arising once again which are similar to the previous reasons.

Problem 1: Everything has been getting a 3 Cog and above, mostly fours and fives. The question I ask myself is, what is the point in having a 2 Cog and 1 Cog since I will probably never use them? Usually a lot of stuff I do review is good quality maps from decent to awesome. So generally, what is the point of a rating system on TAW?

Problem 2: What makes one 4 Cog the same as another 4 Cog, when I may have thought one of the maps is better? Change it to a 5 Cog, even though it doesn’t quite fit there? Have a 4.5 Cog? That would be silly since I may as well go back to the 100/100 scale, but nothing will get 100 or below 50. Something isn’t working here.

Problem 3: What makes one 4 Cog from Quake, similar to one 4 Cog from Thief? Two different games, two different boundaries in terms of level design, but if each game has it’s own scale, how does the reader know this? The problem actually started to occur to me back when I reviewed The Ashen Age. I originally gave it a 3 Cog. Design was great, gameplay was solid yet nothing special; but deffinitely not a bad map. But a 3 Cog makes the map seem average which it surely isn’t, so I upped it to 4 Cog. But I still don’t feel it quite belonged there either. Again, something isn’t working here.

Solution 1: Scrap the system and focus more on the review and have a more interesting conclusion to sum up my thoughts.

Solution 2: Spend the next 50 years trying to find a system that works perfectly with a 20billion word article explaining each of the different scales over the different games.

From this point on, I’ll probably refrain from using a rating and see how it goes. If I do adapt to this, what becomes of previous scored stuff? I can either leave them, but that can get really confusing with three seperate things going on, (theres still some 100 scale stuff about too). The other idea is to remove the scores, but archive what I originally gave them somewhere; this idea seems more appealing to me. We’ll see.

4 Comments

  1. Spirit
    January 22, 2010

    I’ve been thinking about this a lot lately for Quaddicted. Currently I fancy a simple(r) 3 way rating. “Should you play it”: “Yes”, “Maybe”, “Probably not”. It surely needs better words but should be much nicer and fair.

    Of course it takes away a bit of granularity but that actually turns out to be completely subjective anyways. If someone says a map is “nice” then there will be definitely some who say “excellent” instead. Less options = better in my opinion. After all ratings are meant as incentive rather than grades for me.

    Of course this might be much different for you, but I wanted to share my thoughts. Keep up the great work on this site. Your recent reviews made me long for time to play The Dark Project…

  2. quakis
    January 22, 2010

    Though the biggest difference between Quaddicted and TAW, is that; Quaddicted is an archive of Quake maps with a variety degree in terms of quality, while for TAW, I select the occasional map to review and will most likely have very little “don’t play” maps – possibly over long periods of time. It would probably be more likely if it were a recent release.

    It’s not an easy decision overall, but when I look at it this way it makes a little more sense to me.

  3. January 28, 2010

    as spirit you could go for a recommandation also on planetphillip
    or you give your opinion also for my site I go mostly for good ones and regarding game for wich it is (size of the release, content amount of time;;;) it sometimes unfair to give to extemity example the same rating but like for you it’s our choice and other will perhaps have another opinion (look at the difference at PP with the cooperative rating ) it’s certain not perfectly fitting but if you find it i will follow you ;)

  4. February 1, 2010

    Well I use the rating system to decide what is worth my time if I see something with a score of 1 or 2 I usually will check it out anyway but most of the time I’m disappointed.

Leave a Reply