Monday, September 27, 2010

Quality in MMORPGs

How do you measure the quality of a game? A common practice is to use the sales figures. As you can see below, that approach is contentious.

---Which objective measure of merit do you suppose to gauge quality on rather than supply and demand?---

Why must it be objective? Use the powers of persuasive argument to demonstrate why (for example) nonconsensual PVP is worse than Battlegrounds. Sales receipts aren't an "objective" measure because the feature may not be the cause of success. Maybe it's advertising, or circumstances, or a million other things.

Britney Spears isn't the greatest artist of all time, it's really not that hard to understand the discrepancy b/w sales and quality.

So what you are saying is that while no objective measure is able to tell us what a good game is, God Almighty has blessed YOU, and only you, with the gift of the one true subjective measure?

So, pray tell us, which one IS the greatest artist of all time?

I think it just happens that you like a smaller game more. That is okay. But trying to tell everybody that they are wrong, they are stupid, and only you can identify a good game is total bullshit.

MMORPGs are not songs. People pay for monthly subscriptions continuously, and often for years. If a large number of players does that, it is proof that the game is good.

Let’s first try to find out where we agree: We probably all agree that using sales figures is problematic. By sales figures Big Macs are the perfect meal. As are small and cheap cars the best cars around. The artist analogy has already been commented on by Ben. There are countless others. Of course, that does not mean that we should only use our own subjective taste to measure objective quality. It is a typical Tobold strawman (sorry) that can drive his commenters insane (if they care).

Now, let us do the next step. We agree that sales figures are a problematic way to measure quality. But what if it is the only one? Let’s assume for a moment that we could not come up with a better way to measure quality. Let’s assume that sales figures are the only way we know. Does that mean that we need to agree that a product that is sold often is very good?

We can certainly agree that the answer to that question is »No«.
Explanation: Assume you want to guess  the distance between sun and earth and your only tool are your eyes. Your guess is 1,000,000 km. Is it a good guess, just because you only had your eyes? No. Just because your only tool is a hammer, there is no reason to assume that all your problems are nails!

If sales figures are a problematic way to judge quality of a product, then using sales figures to judge the quality is just that: Problematic. It does not matter that you have no other tools at hand. In general, your conclusions do not magically become better, just because you lack the tools to achieve better conclusions.

Market Segmentation
Now, let’s have a look at how problematic, exactly, the sales figures approach is. I copy/paste from an old post.

Imagine five players and two ways to design your game: Option A and Option B. Option A could be “Introduce a feature” while option B could be “Do not introduce the feature”.

Imagine a scale of 1-10 to measure the subjective quality of the game, as rated by the individual players. 10 means a player loves your game; 1 means he hates it, considering the respective options A or B.

Let’s further assume that
1) All players buy a game if they give it at least a rating of 6/10.
2) All players pay the same price, there is no price differentiation.

Now consider this situation of possible ratings (=individual and subjective player benefit):

Player benefit of Option A
Player benefit of Option B
Player 1
Player 2
Player 3
Player 4
Player 5

To maximize aggregate player benefit you would have to choose option A, but with option B you sell the game 5 times. With option A you only sell it 3 times. Option B means 67% more revenue!

Thus, the game companies go for option B. It is better for them to make a game in a way that it is just good enough for every single player to play it. That means that the products that sell most are either products where consumer tastes do not differ, or option B like products.

There are a few ways out of that dilemma. For example, allowing players 1, 2 and 3 to pay more than players 4 and 5 for an option A game. This way they get what they pay for. Another way would be to make the same game twice. One version with option A and one with option B. This is called market segmentation by price differentiation or product differentiation, respectively.

Since the MMO industry does not do much price or product differentiation and since there are wildly different tastes in the MMO community about what is a (subjectively) good game, there is every reason to assume that the companies (just like Hollywood) always go for option B. It maximizes their profit. But does option B always make for a better game? Not for the majority for players. Three of five players, in the example, liked option A more. Also, the aggregate player benefit is higher for option A.

Even More Problems
There are even more problems about the sales figures approach. Firstly, there can be constrains, like money. A lot of people would like to buy better wine and better cars, but they do not have the means to do so. Secondly, as Ben already noted, some artists sell their products exorbitantly more often than other artists, not because they are so much better, but because of advertisement, network effects and the-winner-takes-it-all characteristics. A slightly worse product always runs danger to not be sold at all, because it is, well, slightly worse.

Other Tools
However, there are other ways to judge product quality. Consider good meals. There are experts who judge restaurants and chefs. They do not care about the costs of the meal, they do not care about advertisements and, hopefully, they are critical enough and do not care about network effects. They are able to rate a slightly worse meal just slightly worse. They know about some tricks, like too much fat and too much salt that can make a non-specialist always prefer McDonalds to Kathleen Daelemans.

This same tool, expert opinion, could be used in the MMORPG industry. Trust bloggers who seem to have the same taste you have (but still read the others!), trust independant game magazines (are there any?). Trust good arguments and your experience. And have a look at the sales figures if you want to. Just don’t think that they are a good indicator of product quality.


  1. I think Tobold often reacts to other parts of the blogosphere. There are the Syncaine and Wolfshead types who cite popularity as proof that WoW is terrible.

    While correlating popularity with quality is bad logic as you have shown, correlating popularity with lack of quality is even worse. Round wheels are popular, that doesn't make it automatically an error to design your wheels round.

    Both are fallacies but there is a grain of correlation between popularity and quality. If something is popular it may be good.

  2. There is a definite correlation of popularity and quality. In a market where

    1) consumer tastes differ a lot, like in MMORPGs,

    2) a game that owns as much as a percentage of the market as WoW,

    will usually be a type-B game.

    That means two things:

    1) It cannot be a game that many players rated worse than 6/10. That is: Almost all players like it at least a little bit.

    2) It cannot be a game that many people love. That is most players will not rate it much more than 6/10.

    So, Yes: Such a game will never be really bad and Yes: Such a game will never be really good, either, from a subjective point of view from almost any player.

    That is what people mean when they say that popularit often means less quality. It does not mean that the game is 1/10. It just means that it cannot be a 9/10 for most people.

    I do not remember Wolfshead ever argueing that WoW is bad, because it is popular. Can you quote that?
    I do not read Syncaine.

  3. This comment has been removed by the author.

  4. I was thinking of this article:

  5. Nils,

    You just made those numbers up though. You're not proving anything about quality vs. quantity by typing up that table. The way I see it, you're just proving that 9+10+9+3+3 is more than 6+6+6+6+6. And I don't think anyone has contested that.

    Or to put it in specific terms: I see a lot of people complaining that WoW is not a good game nowadays. But in your mini-spreadsheet format up there, WoW is a 10+9+9+10+10+10+10+9+8+10+10+10+10+9+9+6. In other words, everyone and her stepfather has loved WoW. In many cases, we have now done it to death. And, and this is important, very few games that get poor review scores sell well.

  6. @ Oscar:

    1) I do not consider WoW a bad game. I just think it does not rate 9/10 or more for many players. Most players can name a lot of things in Wow they dislike and that is a direct result of the type-B way of designing a game.

    Most pleople who like PvE dislike arena influence on the game. most people who like arena, dislike PvE influence.

    Both groups would be better of with a game that caters to them alone: A type-A game. But since Blizzard wants to make money, the players have to deal with a type-B game.

    They still like it enough to pay for it, of course. And there are also synergy effects that make the game better for all players. e.g. more expensive animations.

    2) What my numbers prove is that maximizing aggregate player benefit not necessarily means trying to sell the most copies of the game. It is a market failure due to missing/impractical(?) market segmentation.

    This is important, because a good quasi-objective definition of 'good game' could be: Maximum potential aggregate player benefit.

  7. @Stabs:

    Quotes from the Wolfshead article you linked:

    "I don’t fault Blizzard for their unparalleled success with WoW. They should be commended for taking a fringe demographic of the video game market and propelled it to new levels of popularity via innovation and polish. "

    "We’re all proud of what Blizzard has accomplished"

    "You’ve made your point Blizzard. We get it. And after all, they have an excellent product"

    Nowhere could I find a sentence like: "WoW is bad, because it is popular." It is a strawman.

  8. Don't get me wrong, I think that many long-time players are feeling that WoW is "slipping". And I have felt that too.

    But really, I think that we only feel this because we have already played the game *too long*. We're only disappointed with the dumbing down because we love it so much.

    As to that Type A/Type B thing: your numerical example is just that – numerical. Feature sets aren't binary (or decimal, if we're being picky ;)) like that.

  9. @Oscar:

    Have you read my Nostalgia post?

    I agree that nostalgia plays a role. I disagree that the changes Blizzard did over the years play no role at all.

    About the numerical example. Well, somewhere you need to start. You want such an abstract and even objective definition of 'good game'? This is a start1

    The alternative were be to deny that such a definition is possible.

  10. Nils,

    No, I'm afraid I haven't read your nostalgia post. But from reading your other comments elsewhere I am fairly confident that we agree entirely there. To be clear: I am convinced that several of the changes to WoW over the years have made the game less enjoyable (arena, DF, 360 Swipe etc...). For me personally and for many others.

    I see that my previous remark was unclear and unhelpful in that respect. I didn't really mean to say that nostalgia is why we dislike changes. I meant that we dislike changes mainly because we have played the game so long, that we love it and care for it so much that we get upset when something happens that makes it less good.

    And yes, I think I tried to imply that I prefer your alternative, to accept that a definition of "good" is simply not possible.

  11. Have a look at this post then Nils:

    He doesn't cite popularity as proof that WoW is terrible. He does cite the pursuit of popularity as the reason why WoW lacks quality.

  12. @Stabs:

    And to an extend, so do I.

    The pursuit of revenue alone leads to type-B games. And this type of game tries to cater to everyone at the expense of everyone.

    I really cannot describe it any better than I did in this post - with a numerical example.

    For a revenue-maximizing MMORPG developer, a game is optimal when everybody likes it just enough to buy it.

    If anybody likes it too much, there is always some feature he does not like. And by implementing that feature somebody else can be made to like the game just enough.

    It is the Hollywood curse. It is why all Hollywood movies have a love story. A love story makes nobody leave a movie, but some enter it.

    It is why the vast majority of actors need to be young. We all have been young before, but not all of us have been old before.
    Therefore we all can identify ourselves better with an young actor.

    Needless to say that especially older people would had an easier time to identify with older actors. But they have an easiert time identifying with a young actor than the younger people with an old actor, they all have been young before.

    Pursuit of revenue alone will make great games good and bad ones good, too. It reduces my pleasure at playing otherwise great games and increases my pleasure at playing otherwise bad games.

  13. @Oscar:

    Firstly, I tried a quasi-objective definition of 'good game' in my latest post. It is, of course, based on subjective valuations.

    Secondly, it is nice to read that you, too, complain, because you care. That is actually one of the most despicable things happening in such discussions: "If you do not like the game, do not play it".

    Now, Tobold, Wolfshead, me, you, even Syncaine like the game to some degree. That is why we care. That is why we spend so much free time discussing it.

    Since Wolfshead is so much under attack (by Tobold and his readers), let me say this: He does not consider WoW a bad game. He thinks that it is a rather good game and has/had the potential to be a great game. He write that very often.

    Unfortunately Blizzards game design constantly went in the wrong direction (from his PoV). That is why he complains. That is why he is searching for explanations and revenue-maximization is one of these explanations. The mechanism has been demonstrated in detail in this blog post.

    All these discussions would be much more fruitful and friendly if we first tried to understand the other, before we discard them as some maniac. (That is not directed at you, but at the whole community).

  14. Yeah, I agree with just about all you said there, Nils. Very few people in this world are the maniacs we think they are. :)

    I am not so sure about the motives part, though. I think Blizzard is a rare company in that in game design issues they very much act by doing. Blizzard is not the typical focus-group testing crowd. They think up something they like, then sit down and do it. Quite carefully.

    Applying this process to a single-player game is straightforward. Do it, release it. If people like it, it sells. If they don't... well, people have liked everything since Lost Vikings.

    WoW is different, because it has evolved not only in expansions but also through incremental patches. It's a living beast. But they have kept at it, applying their method, their "done when it's done" ethos.

    Game design is something they care a lot about. Sometimes they have made mistakes. Arena, for example, is something they have recognised publicly that they basically regret introducing at all since it led to balancing issues they hadn't appreciated in advance. It's equally obvious that they don't really like the AE tanking style that is prevalent nowadays, etc.

    This, however, is where their problems begin. They introduced these features/changes because they thought they'd be good, because people wanted them. But taking them back risks costing them customers. Ouch! They don't want that. So they try to renege on their previous mistakes by fudging it, slowly erasing them (and sometimes just designing around them). I don't think that's quite as cynical as the cash-grabbing accusations usually leveled at them. But perhaps I'm just splitting hairs.

    Anyway, I'm not saying that the organisation as such is a paragon of virtue, a solitary pillar of pure honesty in a harsh world of capitalism and greed. I'm just saying that's not how they approach game design. The cash-grabbing part comes with the shiny ponies, pandas and whatnot that don't really affect the design.

    But still, I definitely agree with basically all you just wrote there.

  15. Oscar,

    I hope you are right. I agree with Ghostcrawlr 90% of the time. Blizzard has indeed said that arena was a stupid idea. They are finally trying to get resilience independent from crit rating ..

    Cataclysm will in many regards be a much better game than WotLK/late TBC. There is a long list of things that I look forward to.

    Unfortunately, there are still a few big issues left, like the Tx,T(x+1) gear system and the dungeon finder. Remains of WotLK.

    But, just like Wolfshead, I and almost anybody else will buy Cataclysm and we will see for ourselves.

  16. The problem with your argument is games are not cars. They are not food/nutrition.

    If your car gets you reliably where you want to go with no issues, then it is a good car quality speaking. But if you go by ideas such as "fun to drive" then you will find people who say a Ford Mustang is a good car, even if it breaks down a lot!

    Big Macs?? If you are measuring on a taste scale and not a health scale and Big Macs sale more than any other fast food? Then you can say they are a better fast food hamburger. But lumping it in with gourmets is a false comparison. It would be like comparing WoW or Lotro to a console game. These are different platforms as are McDonalds and Gordon Ramsey.

    I do agree that sales figures for WoW are not the best measure, but mainly because I have a friend with 3 accounts and I know others with multiple accounts as well. If you go by this figure it inflates the population of players.

    And the reason people are jumping on Wolf is because he does complain about WoW and how it is getting dumber, but then he whines about not getting invited to the beta. Why does he care about a beta key for something he says this about: "creating dumb games for dumb people equals big profits."

  17. @Nils: Concerning the debate Wolfshead vs Tobold, which also has involved me, I think that Wolfshead with his very strong rhetorics and "bold letters" sets a certain standard. If he liked WoW it doesn't show very much. And if you start by making black-and-white statements, you're more likely to get replies in the same manner. Sometimes I think we misunderstand each other deliberately, because the debate and the reads will be way more fun to read as well as to write that way. And yes, I'm as guilty if this as anyone else.

  18. I have talked about a flaw with this logic in a previous post (which, being on a low-pop-blog, nobody really read ^^).

    My point is really that MMOs are a genre in which you play only one game at a time. Everyone plays the MMO he or she likes best (with exceptions).

    In such a market, a 666666 game makes absolutely no money if there are 999333 and 333999 games around. The popularity of WoW directly means that most MMO players consider it the best MMO to play. Sure, WoW could be better (I'm not playing myself anymore) but none of the contenders manages to appeal more to a sufficiently large subsection of players. (Nor does the sum of niche games appeal to players enough to take away from WoW's market share.)

    MMO designers can't design "just good enough for most people" or they will fail. What they need to design is "just a bit better than every other contender for most people". I don't see how being better than every contender for most people does not make yours the "best game". Not the best game possible, mind you, but the best game actually available.