Wednesday, September 29, 2010

Quality in MMORPGs, Part 2

Tobold wrote a reply to my post about Quality in MMORPGs. In this blog post I will reply to him. I will do so line by line. This way it is guaranteed that I do not just post my own opinion, but actually reply to his.

Nils and I have agreed that our lengthy exchanges on opinions are better handled blog-post to blog-post instead of totally overwhelming the comment section, and this already lead to a marked increase in the number of commenters here. Our current discussion is on the subject of what a good game is, sparked by a comment from Ben who said "Britney Spears isn't the greatest artist of all time, it's really not that hard to understand the discrepancy b/w sales and quality."
Nothing to add, really. For some reason those socials feel uncomfortable if somebody writes one long comment instead of several short ones *grin*.

Now it is easy to get 100 people to agree to the statement that Britney Spears isn't the greatest artist of all time. I'd sign that too. The problem is that if you ask those 100 people who they think *is* the greatest artist of all time, you will get 100 different answers. And the people making statements like the one above are usually those who think that their own subjective answer of what is good is more valid than the subjective answers of the other 99 people. They also usually think that Britney Spears is a *bad* artist, or that the Harry Potter books are bad books, *just because* they are popular. I don't agree with that.
No blogger I ever read thinks that something is of bad quality, *just because* it is popular. I do not and I am pretty sure no other blogger does, either. I’d appreciate if you had a quote that proved me wrong.

Any book, film, song, or game can be measured on two very different scales: The scale that measures their entertainment value, and the scale that measures their artistic value. Where Ben is totally right in saying is that the two are not correlated. But they aren't inversely correlated either. Something which has a high entertainment value will be very popular, but that doesn't tell you anything about the artistic value, neither that it is artistically good nor that it is artistically bad.
You introduce a beast here: A distinction between entertainment and artistic value. I will not delve into that. My post was about the individual, subjective quality that players assign to games and how an objective (or quasi-objective) measurement of quality should relate to it. This subjective quality, that is the benefit of a player when he uses a product, can be applied to art and pure entertainment, likewise. I see no need to differentiate and open another box.

I am a scientist. I do not like judgment on artistic value, because that is so highly subjective. I'd claim that for the example the Harry Potter books have an artistic value, because of the way the language of the books matures with the age of the hero, which is both very subtly done and used to great effect. But that is my subjective opinion of the art of writing, and I'm sure many people would disagree.
I studied Financial Mathematics and Physics so I guess I am a scientist, too. But I feel uneasy about the way you mention it here. It feels like those ads where people stress that their statements were tested with "scientifics tool" or by "scientists".

In my opinion, any objective measurement of quality needs to rely on subjective measurements. Otherwise, there were no connection. Now, that would be a strange definition "of good game".
Of course, also the definition by sales numbers indirectly relies on subjective valuations of quality. Do not mistake measurability for objectivity!

Furthermore I would say that games, especially massively multiplayer games are not like books, films, or songs, in that games very rarely qualify as art at all. Yes, there are a few borderline cases like Myst or Ico, but the kind of game I'm discussing on this blog is not art in my opinion. MMORPGs are huge projects created by hundreds of people, and even an "art director" or anyone else on the team can hardly claim the whole game as a work of *his* art, not like the author of a book can, or the director of a movie (and lots of movies aren't art either for pretty much the same reason). Games are most of the time not created with any artistic aspiration in mind, but *only* for entertainment value.
I agree with the last sentence. Otherwise, I already stated that I consider the distinction of art and entertainment, for the purpose of defining an ‘objective’ way to measure quality in games, redundant.

Therefore if you hear me speaking about a game as being "good" or "bad", please keep in mind my narrow definition of what a "good game" is: As I assume that the fundamental purpose of a game is to entertain, I judge a game on its ability to do exactly that. A good game for me is one that is entertaining to its players. If you personally think that to qualify for "good game" a game has to fulfill other criteria, be that some artistic value or something else, we simply risk to miscommunicate, because we are using so very different definitions.
I, too, think that a game should entertain. But I do not see any reason to assume that the entertainment value is determined well by the number of subscribers. The number of subscriber has a lot of problems if you use it as a measurement of quality. I have listed several in the last post. It would be great to read your response to that.

I'm not saying that my definition of "good" is the only one possible, or the best, or anything. But I'm saying that this is the definition I use, and have always used on this blog.
Tobold, you can use whatever definition you want. Hell, you can define your blog to be always right. (I know: You do not. That is the point). This only shifts the problem of finding a good definition to finding out whether your definition is good.
So, thanks for the clarification. We now know what you mean when you say “good game”. But I still consider your definition a bad one. And I listed many reasons for this opinion in my last blog post. What would really help would be you answering to these points and why you consider them unproblematic or non-significant.

And as my definition of "good" only judges a game by its entertainment value, and entertainment value is highly correlated with popularity and ultimately sales, I do like to use subscriber numbers.
Woa. Stop! Why is entertainment value highly correlated with popularity and ultimately sales? I agree that there is a correclation, but a high one? No.

Firstly: Careful with popularity: Second Life is arguably more popular than WoW and I am pretty sure you do not consider it better entertainment.

Secondly: Using entertainment value as a definition still has the same problem that sales numbers have. I.e. sales numbers can change without the game changing. Sales numbers are influenced by advertisement, prices, network effects, etc.
Sales numbers, as well as popularity are influenced by a lot of factors that are not inherent properties of the game. Now, in my opinion one property a good definition of “good game” should have is that it is not significantly influenced by factors that are outside of the game.

Although I of course agree with Craig Morrison that "1 million registered users" and "1 million subscribers" are not the same thing, and you need to look at all numbers closely to avoid being misled by some marketing trickery. MMORPGs with monthly subscriptions are relatively easy to compare, because the pricing tends to be similar. And unlike listening to a song, which is most often free, or reading a book, which usually just requires a single payment which you might end up regretting, a game with a monthly subscription requires a continued statement from its players, who are effectively saying: "Yes, this game still entertains me enough for me to be willing to pay $15 for another month". That constitutes a valid measure of the entertainment value of a game, and that is what I like about these numbers. But remember, that is *my* definition of what a "good game" is, to which not necessarily everybody agrees. (/wave Wyrm, Ben, Nils, etc.)
I think, you mistake measurability for objectivity here. It is the same mistake the guys on wall street did some time ago. Just because your way to measure risk produces hard numbers, does not mean that it is a good indicator of risk.

Besides, don't you think that revenue is a better measurement than sales numbers? (Not that revenue would be a good one, either, in my opinion).

My suggestion for a definition of “good game”
In my humble opinion a good measurement of quality of a product is this:
"Aggregate, potential consumer benefit"

What a monster :). Yeah. Sometimes there are no easy solutions.

You remember that table from the last post? If not, you should look it up. To find out “how good a game is”, you should ask all potential players to write down how much they like the game on a scale of 1-10. Or any other scale, of course. Then you add their ratings.

Now, why is this a superior definition?
Firstly, because all potential players are asked. Not only the ones that know about the game by watching ads. Secondly, because network effects, or any effects outside of the scope of the game itself, play no role. Thirdly, because this is a proven method in microeconomics to judge societal benefit of a product.


  1. I think you can still get skewed results in your theoretical sit all potential players down test. People may include things like percieved popularity in their subjective rating in the game. That being said I agree that it is still better then looking at just sales figures.

  2. You are devising an objective measure which is THEORETICALLY better, but practically impossible. Sales numbers might be less accurate, but they are available.

    I believe marketing can make somebody buy a game and try it. I do not believe marketing can make somebody stick with a game for thousands of hours and pay a monthly fee every month. Thus subscriber numbers are effectively asking every player to write down not a scale from 1-10 but "which game would you at the moment most like to play". Less perfect, yes. But still accurate enough.

  3. Tobold,

    You write
    . I do not believe marketing can make somebody stick with a game for thousands of hours and pay a monthly fee every month.

    and I agree. However, absence of marketing can keep potential players unaware of a game. And that is the bigger problem.

    More importantly, however, we have finally found where, exactly, we disagree.

    A definition is a passage that explains the meaning of a term (a word, phrase or other set of symbols), or a type of thing.

    A definition does not have to describe an experiment. That is, a definition may very well be useless for actually determining something. It just clarifies what something is.

    I, of course, agree that sales numbers are very easy to get in contrast to what I propose. So if you want to conduct an experiment to determine 'how good a game is', you are free to use sales numbers as a rough guess for quality as described in my post.

    But this were just a guess with the help of an experiment. It is not the definition.

    A theoretical experiment can very well be used as a definition.

    For example, nowadays we try to find out whether planets surrounding other stars are 'habitable'. We know how to theoretically do that if we were right on the planet. But since we are unable to get on it, we use experiments to guess 'habitability'.

    Be we would never define habitability according what our experiments can do. We still define it as "Try to survive on top of that rock for more than 1 hour" (or something similar :).

    PS: Of course, also my try at defining game quality is not perfect. It is just a better try better than sales numbers.

  4. "No blogger I ever read thinks that something is of bad quality, *just because* it is popular. I do not and I am pretty sure no other blogger does, either. I’d appreciate if you had a quote that proved me wrong."

    Every hipster I know thinks this way about ANYTHING (except starbucks for some reason).

  5. Pangoria Fallstar ,

    I was talking about bloggers I read. Not about "hipsters".

  6. As someone who knows a bit about the reliability of surveys, I have to point out: they aren't. I can go into more detail if you want, but some of the problems of surveys are:

    -People have different perceptions of what constitutes different ratings (what is a "7"?).
    -People surprisingly don't really know or understand what they like or how they feel themselves.
    -People lie. Even when the survey is anonymous, even when there would seem to be no reason to lie, they lie anyway.

    I'm sure you can see how especially the first one would be a problem. Most game sites use a scale where below 7/10 is reserved for only the worst of the worst games, whereas I would use the full spectrum 1-10. A 7 from me is a pretty good game, but an abomination from a game site.

    So I see Tobold's method as better, if incomplete. Voting with their dollar is honest, and constitutes a comparable threshold (this game is worth $15 for me to continue playing).

    Player retention would seem to be a bit better. Out of the people who have actually played the game, how many liked it enough to pay $15 to keep playing?

    But even this has problems. I would suspect that Darkfall has a very high retention rate. While there are a great many people who would hate Darkfall, they largely understand what kind of game it is and don't even try it.