Video Games That Watch Back
Few gamers are aware that when they are watching their monitors while playing video games online, those games may be watching back, keeping track of their play habits and their likes and dislikes. These so-called "player metrics" techniques are like focus groups on steroids.

The Hollywood Reporter | November 30, 2006

Indeed, media mavens who use standard focus groups to test TV pilots or soon-to-be-released movies would be more than a little impressed -- and jealous -- of what leading-edge techniques game developers have available to them to test their products.

Take the popular first-person shooter "Half-Life 2 Episode 1." The developer, Bellevue, Wa.-based Valve Software, uses an online "Gameplay Stats" tool to collect such real-time data as: length of average playing session, average completion time, total playing time, and what game options players enable or disable. The developers can determine -- as their game is being played online -- how far gamers are advancing into the game, how often and where they are being killed, and at which difficulty levels gamers are choosing to play, among other metrics. The results are available to anyone with just the click of a mouse on the site's stats page.

The cumulative results, says Erik Johnson, Valve's product manager, are used to not only make improvements in the current game, but to make the next episode -- "Half-Life 2 Episode 2," which is scheduled for a summer 2007 release -- even better.

"For example, there was a particular scenario in Episode 1 where you're fighting a bunch of zombies, ammo becomes difficult to find, it's getting pretty frenetic, your health tends to be pretty low and a timer is running," describes Johnson. "From the metrics, we learned that not many players were getting through that part of the game, even when they played on 'easy' mode; there was too high a rate of people quitting at that point. Clearly this moment had become too frustrating for them. So we went back, made it somewhat easier for them to survive, and the metrics proved that we'd done the right thing -- the data came back to levels that we consider acceptable."

Valve's peering at gameplay began in June, and is made possible because its games are sold via its online service, called Steam, which gives Valve the channel to talk to and hear back from customers. It also allows Valve to deliver updates; any changes made in the game by the developers immediately go into effect each time a player logs on.

"Previously, we'd bring in a hundred play testers and have them play the game prior to launch -- what we call "usability testing" -- while we watched," noted Johnson. "Now we can still do that, ship the product, and then get a million external play testers, receive their data, and it helps us make the game even better."

Meanwhile, at Austin, Tx.-based NCsoft, each of its popular massive multiplayer online games (MMOGs) -- like "Lineage II", "Auto Assault," "City of Heroes" and "Villains," and the recently released "Guild Wars Nightfall" -- has had its own extensive metrics-gathering system. But it's only been within the past six months that the company has been working on a centralized system that will gather data and standardize it in a set of tools that will be available to all its developers, both internal and external.

"Some of these metrics have been collected for years and then resided entirely at whatever product development team created the game," says Dallas Snell, director of business development. "Now we're trying to determine the best way to look at player behavior, and we intend to share the information we gather with all of our developers."

Two different types of data are collected. The first is "fun factor behavior" which measures how recently gamers logged in, how frequently they log in and how long they stick around after they start playing. This gives an indication of the overall health of the game and whether it's continuing to pique customers' interests. The second tracks gameplay behaviors -- like where players spend most of their time in the game, what missions they choose to accept and whether they complete them, and how often they are dying -- and this allows developers to make appropriate adjustments in the game to give it balance.

"We can actually collect every keystroke that a player makes in terms of how they're controlling their character," says Janus Anderson, NCsoft creative director. "Based on the data we collect, a game design team can go in and do further analysis on, say, why people are dying too much in a certain part of the game. Then we can patch the game if we think it will improve gameplay. MMO gamers are used to our fiddling with the game and constantly patching it. That's how we give them the game experience they're looking for."

Does NCsoft believe it's making better games because of its ability to track its customers?

"You know, I've been in the games industry for 25 years," remarks Snell, "and up until recently, when we shipped a product, we had no idea other than from our internal QA or from a focus group what the gamers' reactions were going to be. I'd have given my left leg to have this kind of data years ago, where we can actually go into beta and have thousands of people playing it, and essentially have a camera aimed at every one of them and be able to look at what they're doing and how they're reacting. No, it doesn't mean we're making better games because of it. But it certainly gives the industry the tools that they never had before to work towards making those better gaming experiences."

Snell believes that, with more and more games designed to function online, this sort of data collection will begin to be used by game teams at other companies that may not yet be taking advantage of the process. But, he observes, "mountains of data is of no use if the teams don't have the skills to interpret it or to know how to react to it."

That's where Nicole Lazzaro comes in. She is the founder and president of Oakland, Ca.-based XeoDesign, a company that evaluates gamers' playing experiences and determines how developers can increase a game's "fun factor." With a degree in psychology from Stanford University and 15 years of expertise in "player experience research," Lazzaro admits to being excited about the latest techniques online game companies are using to study player metrics.

"The part I really, really like about gathering behavior-based data is that it's far more accurate than, say, asking a player to answer questions about a gaming experience, as they do in typical focus groups," she says. "Instead of forcing the player to articulate something that they may not be aware of or may not remember, you're collecting information about what the player did without their feeling uncomfortable about not pleasing the examiner or telling the game developers that their product is mediocre."

But, she says, if data shows that players are quitting a game and not coming back, one doesn't necessarily know what to fix.

"Games are very complex," she says. "The best of all possible worlds is to collect the data and then use XeoDesign's methods to understand on a one-to-one basis what is influencing the gamer in a particular part of the game."

Clients like Sony Online, Electronic Arts, and Ubisoft have hired Lazzaro to come in and do one-on-one testing with gamers. "We photograph them and record moment-by-moment what is happening in their play experience," she explains. "We can measure seven different emotions in the face -- surprise, frustration, excitement, amusement, etc. -- and combine that with our observations about body language. And then we determine with the game developer whether the game is creating the sort of experience for the player that they had in mind when they built the game."

For example, game developers at Cyan observed that players were quitting "Uru Live," their MMOG inspired by the classic game Myst, but they didn't know why.

"We diagnosed exactly how and why players' expectations and emotions interfered with their properly learning the game's controls, which caused them to disengage from the game," says Lazzaro. "We made suggestions on what improvements to make, and Cyan is using our 126-page report to redesign the game before launching it."

Lazzaro predicts that this "next-generation of game testing" involving studying gamers while they are actually playing -- both online and in controlled situations -- will result in refreshing new experiences in terms of game quality.

At Valve, director of marketing Doug Lombardi believes there's much that other media sectors can learn from the game industry's ability to track customers' online play sessions.

"It behooves any business to go out and make sure they're satisfying their customers," he says. "If you're not, if there's a disconnect there, then you could be making both a big mistake in your product, and you could also not understand the things that work really well."

Currently online games have the advantage of being connected to gamers which enables developers to receive feedback.

"As soon as Hollywood moves from physical media to direct digital distribution, film companies should start thinking about how they can do the same," says Lombardi. "Think about it ... if people were able to download, say, the DVD of "Mission Impossible VI," directors would be able to measure whether customers actually cared about and watched the deleted scenes and other features included on the DVD. They could determine whether people actually sat through to the end of the film or, perhaps, it wasn't worth blowing an entire production budget on that big finale, and so on. There's much to be learned by studying your customers, and this is the best tool we've got for doing that."

CLICK ON THE BANNER TO BUY TERRORSTORM IN HARD COPY