Quality in MMORPGs

How do you measure the quality of a game? A common practice is to use the sales figures. As you can see below, that approach is contentious.

Ben:
---Which objective measure of merit do you suppose to gauge quality on rather than supply and demand?---

Why must it be objective? Use the powers of persuasive argument to demonstrate why (for example) nonconsensual PVP is worse than Battlegrounds. Sales receipts aren't an "objective" measure because the feature may not be the cause of success. Maybe it's advertising, or circumstances, or a million other things.

Britney Spears isn't the greatest artist of all time, it's really not that hard to understand the discrepancy b/w sales and quality.

Tobold:
So what you are saying is that while no objective measure is able to tell us what a good game is, God Almighty has blessed YOU, and only you, with the gift of the one true subjective measure?

So, pray tell us, which one IS the greatest artist of all time?

I think it just happens that you like a smaller game more. That is okay. But trying to tell everybody that they are wrong, they are stupid, and only you can identify a good game is total bullshit.

MMORPGs are not songs. People pay for monthly subscriptions continuously, and often for years. If a large number of players does that, it is proof that the game is good.

Let’s first try to find out where we agree: We probably all agree that using sales figures is problematic. By sales figures Big Macs are the perfect meal. As are small and cheap cars the best cars around. The artist analogy has already been commented on by Ben. There are countless others. Of course, that does not mean that we should only use our own subjective taste to measure objective quality. It is a typical Tobold strawman (sorry) that can drive his commenters insane (if they care).

Now, let us do the next step. We agree that sales figures are a problematic way to measure quality. But what if it is the only one? Let’s assume for a moment that we could not come up with a better way to measure quality. Let’s assume that sales figures are the only way we know. Does that mean that we need to agree that a product that is sold often is very good?

We can certainly agree that the answer to that question is »No«.
Explanation: Assume you want to guess  the distance between sun and earth and your only tool are your eyes. Your guess is 1,000,000 km. Is it a good guess, just because you only had your eyes? No. Just because your only tool is a hammer, there is no reason to assume that all your problems are nails!

If sales figures are a problematic way to judge quality of a product, then using sales figures to judge the quality is just that: Problematic. It does not matter that you have no other tools at hand. In general, your conclusions do not magically become better, just because you lack the tools to achieve better conclusions.

Market Segmentation

Now, let’s have a look at how problematic, exactly, the sales figures approach is. I copy/paste from an old post.

Imagine five players and two ways to design your game: Option A and Option B. Option A could be “Introduce a feature” while option B could be “Do not introduce the feature”.

Imagine a scale of 1-10 to measure the subjective quality of the game, as rated by the individual players. 10 means a player loves your game; 1 means he hates it, considering the respective options A or B.

Let’s further assume that
1) All players buy a game if they give it at least a rating of 6/10.
2) All players pay the same price, there is no price differentiation.

Now consider this situation of possible ratings (=individual and subjective player benefit):

	Player benefit of Option A	Player benefit of Option B
Player 1	9	6
Player 2	10	6
Player 3	9	6
Player 4	3	6
Player 5	3	6
Sum:	34	30

To maximize aggregate player benefit you would have to choose option A, but with option B you sell the game 5 times. With option A you only sell it 3 times. Option B means 67% more revenue!

Thus, the game companies go for option B. It is better for them to make a game in a way that it is just good enough for every single player to play it. That means that the products that sell most are either products where consumer tastes do not differ, or option B like products.

There are a few ways out of that dilemma. For example, allowing players 1, 2 and 3 to pay more than players 4 and 5 for an option A game. This way they get what they pay for. Another way would be to make the same game twice. One version with option A and one with option B. This is called market segmentation by price differentiation or product differentiation, respectively.

Since the MMO industry does not do much price or product differentiation and since there are wildly different tastes in the MMO community about what is a (subjectively) good game, there is every reason to assume that the companies (just like Hollywood) always go for option B. It maximizes their profit. But does option B always make for a better game? Not for the majority for players. Three of five players, in the example, liked option A more. Also, the aggregate player benefit is higher for option A.

Even More Problems

There are even more problems about the sales figures approach. Firstly, there can be constrains, like money. A lot of people would like to buy better wine and better cars, but they do not have the means to do so. Secondly, as Ben already noted, some artists sell their products exorbitantly more often than other artists, not because they are so much better, but because of advertisement, network effects and the-winner-takes-it-all characteristics. A slightly worse product always runs danger to not be sold at all, because it is, well, slightly worse.

Other Tools

However, there are other ways to judge product quality. Consider good meals. There are experts who judge restaurants and chefs. They do not care about the costs of the meal, they do not care about advertisements and, hopefully, they are critical enough and do not care about network effects. They are able to rate a slightly worse meal just slightly worse. They know about some tricks, like too much fat and too much salt that can make a non-specialist always prefer McDonalds to Kathleen Daelemans.

This same tool, expert opinion, could be used in the MMORPG industry. Trust bloggers who seem to have the same taste you have (but still read the others!), trust independant game magazines (are there any?). Trust good arguments and your experience. And have a look at the sales figures if you want to. Just don’t think that they are a good indicator of product quality.