WoW: Is Rolling the Dice a Good Way to Select Beta Testers?

Lately I’ve started wondering about the validity of the selection process for the beta test of Blizzard’s Wrath of the Lich King. In the Burning Crusade Blizzard admitted that players were selected randomly. In the new expansion players have been given the chance to opt in. Still, it seems to me that getting into a beta should be based on more then just relying on capricious Lady Luck to choose your testers. You really have to wonder how serious Blizzard is about their MMO if they are not basing the selection process on some form of transparent and reasonable criteria.

Here’s what we know so far about how Blizzard selects their beta testers:

We generally pull from many places so that we have a good variety of people in the beta. Some of it is random and some are winners from contests, or specific groups that we want to pull from. We don’t have a specific schedule for you on who we will pull in or when to share with you, but we’ll do what we can to keep people as informed as possible.

This seems purposely arbitrary. It’s designed to make Blizzard look like they have some kind of criteria but vague enough to cloud the expectations of a potential beta applicant. I’m sure that Blizzard has already contacted their “pet” guilds out there in the community and offered them beta slots. Most other MMO’s have devs that have ties with uber guilds that secretly test content for them. MMO companies love this as they don’t have to pay these semi-pro testers. One of the main reasons these guilds test content on test servers so they can go back and enjoy an intel advantage on their player servers — the very same edge that Blizzard employees have when they play WoW.

While beta tests may be somewhat dubious as to their value in MMO’s, as a game developer I can tell you that selection of candidates for focus testing in the games we developed was done with great care. Only gamers that fit a certain demographic that corresponded to our target audience were selected to play-test our games. These testers all had to meet certain requirements regarding how much time they spent playing video games each week. Evaluating the feedback gleaned from play-tests is an art and science all its own. Of course beta tests in MMO’s are a completely different animal but still…there should be some criteria that are met by the people you allow to test your game.

Here’s a few criteria that I would like to see Blizzard at least consider when granting beta slots:

Loyalty

One of Blizzard’s current methods of inviting beta testers is to reward attendee’s at various official events with guaranteed beta slots. I’m not opposed to that as people who make the effort to attend those functions are extremely loyal as they have spent a lot of money in traveling and lodging. People like that are very passionate about WoW and are likely to be better beta testers then the players who don’t attend. Why not take this idea of loyalty a bit further?

Another way of rewarding loyalty would be to give beta slots to people who have been faithfully subscribing to WoW since its release back in 2004. I’ve calculated that someone who’s been subscribing since the launch would have spent approximately $700 US funds when you combine the cost of both software packages and monthly subscriptions. That’s a substantial investment that deserves some form of acknowledgment and respect from Blizzard.

Then there are the people that have purchased both collectors editions. Add to that people with multiple accounts. Surely loyal subscribers should have preference for getting a chance to get in the beta then just dumb luck?

Character and Diligence

Another type of player that should get preferential treatment for beta slots are the players demonstrate good character by helping to police their community; specifically those vigilant players that go out of their way to report chat, naming violations and other infractions. These people have a positive impact on the quality of the play experience because as we all know Blizzard does not proactively enforce their own codes of conduct — players must do this and request a GM’s help. Without players that spend their precious leisure time filling out petitions offending players would be able to keep violating the rules with impunity.

How hard would it be for Blizzard to track these valuable members of the community by seeing how many petitions they have submitted and reward them with a beta slot for their unpaid service? Players that care enough about the WoW community probably would care enough to actually submit bug reports.

Proven Aptitude

Other beta testers should be admitted based on the strength of a well-written application. Being a good beta tester takes time and commitment. You need to be able to submit detailed bug reports. You need to be able to commit to testing for X amount of  hours per week. People that actually spend the time to thoughtfully articulate why they would be a good beta tester are more likely to be good beta testers. What’s the point of allowing people into the beta test that haven’t at least demonstrated they have the maturity or inclination or time to actually “test” the game? Giving someone a beta slot just because they “opted in” at a website seems like a very shallow criterion for determining someone’s fitness to actually test the game.

Experience and Expertise

Another good asset for a beta tester is experience. Players who’ve beta tested other MMO’s and games and previous releases of WoW have valuable experience that could benefit Blizzard. Also people who’ve raided before are obviously better suited to testing raid content then people who have not. This goes for every aspect of WoW including PVP, crafting, questing, etc.

Players that have taken the time and trouble to actually enter in bugs at the official WoW bug forum should be rewarded for their efforts. At least they have a demonstrated a past willingness to report bugs unlike the average drooling WoW fanboy.

Conclusion

For the most part, recent beta tests have been used by MMO companies as a way to promote interest in their game and I believe that Blizzard is no exception here. It seems they are using the beta test (and the leaks) as a way of generating just enough buzz to keep bored players subscribing. If they were truly serious about testing their game they would show more care about who they are selecting to be part of the beta.

Dangling the carrot of offering beta test slots to players who go above and beyond the call of duty to help other players would also be a good way to improve the community. In a MMO that is so achievement driven, it’s unfortunate that Blizzard hasn’t created volunteer program that rewards the players among us that are good Samaritans. Beta slots cost nothing and would be a great way to motivate players to do good by helping other players.

While I believe that there may be some benefit to inviting a certain small percentage of your beta testing pool based on a lottery, I think Blizzard would stand to gain much more from having a selection process based on some kind of sensible criteria. Why they are purposely missing a great opportunity to bring in qualified testers is beyond me. Just as a company wouldn’t hire unqualified people off the street to work at a company, you also shouldn’t bring in unqualified and undeserving people in to beta test your MMO.

-Wolfshead