When it comes to game design, we’ll find out one of these days if artificial intelligence can beat a human being at creativity. But today, the matter is still up for debate.
Some game designers believe that intuition is the best tool for being creative, while others think that data-driven design rules, particularly in an age of digital games — mobile or online — where you can collect so much data about what players want.
What’s the right balance between science and art when it comes to game design?
We discussed this topic at Casual Connect Europe in Berlin on a panel that I moderated. The panelists included Philipp Karstaedt, general manager for Europe at Gree International; Tammy Levy, director of product for Mobile at GameStop’s Kongregate; Adam Telfer, product lead at Wooga; and Christopher Kassulke, CEO and owner of HandyGames.
Here’s an edited transcript of our panel.
Image Credit: Dean Takahashi
GB: What should game developers, at a minimum, know about data-driven game-making?
Philipp Karstaedt: There’s a place for data driven and a place for design driven. The only thing that can go wrong is when you say “this is the only way.” It depends on what you want you want to do, what stage you’re at, whether you’re a publisher or developer. There’s no right path. Everyone is still figuring this out.
Tammy Levy: It’s fine to think about both sides, data and design. If you’re going at it from the data side, get informed developers who can make design decisions. If you’re coming from the other side, learn about data. Understand the basics so you can make informed decisions. It depends on context.
Adam Telfer: As game makers, we’re all more coming from design in the beginning. But then data takes over. We’ve seen that in the games and in the stores. A lot of games are becoming data driven. There’s some sense that what we’re missing is that the data guys aren’t really talking to the design guys, and the design guys don’t understand the data guys. They speak totally different languages. Most of the data guys aren’t gamers. Without any game experience, data is worthless.
Christopher Kassulke: My role is part game design and part data design. I’m at the meeting between the two. I see data as a tool, just like any other tool in game design’s toolbox. I go back to when I started in game design, when I worked in playtesting. Data, all it is is effectively playtesting your entire user base, using it as a high-level view of what’s going on.
If you’re just getting started, there are free tools available out there for collecting data, looking at things like retention and engagement numbers. That will help you take that first step toward making better decisions in game design.
GB: I sense a lot of emotion around the divide between data and design. As far back as I can remember, Zynga has been a lightning rod for this. They would talk about doing A/B tests on what color a button should be. That was a bit more of a controversy about how to design monetization into games that might be able to proceed without monetization, though. It’s not the whole topic.
Image Credit: Zynga
Kassulke: Perhaps it’s not, but at the end of the day, ever since we’ve had free-to-play we’ve had data-driven games. Before, we’d just say, “This is a great game,” and you’d buy it at the store for $60. We had a design in our head and we believed that was a good way of making a game. Maybe it’d be tested in small groups, but that’s all. Keep in mind that it’s always about what happens in testing.
Karstaedt: Zynga was really just the first company that used data for monetization. In the end, data is about way more than that. On the other hand, it’s also true that data has almost become too powerful in some cases. People have said, “This is my data hammer,” and suddenly they see data nails everywhere. The art of game-making goes far beyond that. Understanding game design and how people interact with it, that’s data as well. It’s about retention. It’s not always monetization. People are unhappy about the monetization side for various reasons. But it’s a very powerful tool. Without it, it wouldn’t be possible to emulate some of the successes we’ve seen. It’s self-defeating to ignore data just because you don’t like monetization.
Levy: It’s funny that you mention Zynga, because I was there at a point when it was overly optimized at times. It’s important to be aware of how to use data as a tool, but not let it dictate all the decisions you’re making. Understand that it has its place and time. Not everything can be optimized to the last percentage point, the last extra cent out of each player.
It’s also important to talk about scale when you talk about data optimization. A lot of times you don’t have enough players to optimize certain features based on data. Then, as a designer, you have to ask yourself what your instincts tell you as far as how to optimize the game and make it better at that point. If you only have 100 or maybe even 10 players getting to that point in the game, there’s no significance in your data. It’s important to understand at what levels you can use data and at what levels it can hurt you.
GB: Do you run into a lot of game designers who will not pay attention to data? I saw Warren Spector, who’s a pretty famous game developer in the U.S., put up a Facebook post after the U.S. presidential election. All the analysts were wrong about who would win, and so he said, “If anybody comes to me and says I have to do a data-driven game design, I’m just going to say ‘Trump.’” Have you ever had to convince someone to look at data that way?
Image Credit: Dean Takahashi
Kassulke: It’s not only Trump. It’s also Brexit, to give another example, here in Europe. Everyone said that Brexit couldn’t happen, and it did. Life is not that simple. You can’t quantify everything. Games are entertainment, and everyone is entertained in different ways. I always ask people, “What was your favorite Indiana Jones movie?” It’s always the first one. Nobody likes the last one, because the marketing and analytics took over. It was just boring. Games that are only data-driven will be boring.
Consumers are not stupid. You need to have some soul in your game. Data is good, but you need to make the first steps into a game as a game designer. “That’s my vision. That’s the way it will be.” Then you can check that against data and perhaps change it step by step. But that’s a different topic.
Levy: As a publisher, we see both ends. We see developers coming to us where they’re fully design driven, and when we show them data they say, “No, I’m not interested in listening to that.” And we see the other end of the spectrum, games without a soul. “It has good retention because it has this feature that these other games have as well.” It’s just a compilation of pieces of other games that are supposed to work well, jammed together into one Frankengame.
For developers that don’t want to listen to data, one thing that I’ve found is very powerful is benchmarking, talking through the impact of making changes based on things we see out in the market. We have a lot of games out there to use as examples. “You think you’re going to be the outlier, but of 10 games out there exactly like this, nine have behaved in a particular way in this area.” Talking through the impact is important – “If you make this change, this is what could potentially happen in terms of these metrics.”
Karstaedt: I don’t want to talk about the Trump stuff too much, but when you think about that situation, it wasn’t just the data that said Hillary was going to win. It was also a lot of intuition from a lot of experts. I don’t think you can just assume that the data created that issue. Also, there was a lot of soft data from polls that had potential bias, which points to a problem in the way that data is gathered. In the future we’ll hopefully have better models to gather and use polling data.
Going back to Warren Spector, he’s one of the few game designers right now who have that luxury of not being affected by data at all. Most of us don’t have that luxury, especially when you’re talking about free-to-play games, high-volume games, games that are trying to find a broad appeal. In those cases data is very important.
When I work with designers who are resistant toward using data, it usually comes down to the idea that as soon as we start using data, that means we’re going to entirely focus on quick wins and what you said about Frankenfeatures. That’s not really true. It’s two separate things. You can look at data and use it to follow an easy path, or you can use it to try and build a more engaging part of your game.
Let’s say you want to increase retention and engagement. One way is to add a daily reward system, something that gives players a little reward every time they come in. That’s the usual way. Another way to approach it is to look at our game design loops and try to make them more engaging, so it’s inherently fun to come back. You can start both of those paths from data that says players aren’t coming back and you need to build higher engagement.
Levy: Where we see that work best is when we point out a scenario or an improvement, and then let the designers come up with a creative way of approaching that problem, instead of dictating a way of doing it to them. It’s a hard thing to do. Say you’re testing a website and you have trouble finding a button. The obvious solution is to make the button bigger. But it might be the color or the way the layout leads you to it. You need to let the designer fix the real problem. A lot of data people make the mistake of simply saying, “We need to increase retention, so you should add a log-in bonus.” But that’s not the problem. You need to be able to say, “I’m seeing this gap. Can you come up with a solution to fix that?”
Image Credit: Dean Takahashi
GB: When Wooga started, I imagine your role didn’t exist. Along the way, when did data people become part of design teams?
Telfer: I wasn’t there from the founding, so I don’t exactly know. But Wooga was always a free-to-play company. We started on Facebook. If I were to speculate, I think data was embedded from the beginning. Over my own career, starting in indie games—as soon as you move into free-to-play, you see the value of data. My job as a game lead – part game design, part producer, part product manager – became part data as well. As I gained more knowledge of how to use data, it became a bigger part of my job.
Karstaedt: Going back to what I was saying about what’s hard data and what’s soft data, if you do exit polls, that gives you only so much customer insight. What people tell you they want is not necessarily what they really want, which is why it’s important to back that up with hard data. It doesn’t matter if everyone tells you they love a certain feature, or they’d play your game if it had this feature, when they don’t follow through on that. If the data shows that they’re not acting on this thing they say they love, clearly they don’t love it. That’s the value of hard data.
Sometimes you have to abandon your own idea of what you’d like to play. A lot of indies, I think, have that problem standing in their way on the path to mass-market success. If you start working on games with a publisher or with a license, at some point you may have to abandon some of what you personally think is good and go see what the market wants.
Even if you don’t want to design your game based on data or market research, at least—in case you want another company to come in and operate your game or have anything to do with your game, at least put in the data collection triggers. Make sure the game backend collects data, even if you don’t think you want to use it. Give others the opportunity to use it if you want anyone else to be involved with your game. Change things in a way you think is best, but see what your players actually want. With data you can determine that.
GB: Chris, you have all kinds of people at your company. You’re the CEO. You have finance people, marketing people, game designers, and now data people as well. You may be the boss of these people, but you have to get them to work together. How do you make sure data people mesh well with designers?
Kassulke: Communication. Drinking beer. [laughs] It’s about understanding where both parties come from. We have very small teams, compared to Zynga or GREE or Wooga and many others. The team decides if they want to use the data. We always say, “Are we asking the right questions?” Data is good, but if you do games like we do – globally, on many different platforms – they have to work on PC, on console, on smartphones, on tablets. Perhaps a title isn’t the best performer in the mobile space, so maybe we should concentrate on another platform. Or we might concentrate on a particular country. Why is a specific game not performing in this specific region? You can dive deeper and deeper into the analytics, but can you find the right answer?
With us, it’s easy enough. We just try it out and see. That’s what we’ve found is always the best option. If you look at which games are performing in the category—take Zynga for an example. Their latest titles were not huge successes. The problem is that they’re having to invest so much money and time into a new title to get it past a certain point where the analytics will work. When was the last time Wooga released a new game? Release more titles. Try new things.
Data, in my opinion, is currently holding back the game industry. I see a lot of guys who don’t care about data. They’re just trying new, cool stuff. The freshest titles I see are coming from indies who don’t care very much about data. They’ll try out something without data, and then they’ll analyze from there. That’s a valuable experience for the game designer and the studio.
Karstaedt: In some ways it’s a bit like building cars. There are cars that look amazing, that are unique, that are hand-built, and there are maybe 10 of them in the world. They’re made without data. That’s not what their customer wants. They couldn’t care less. That’s fine. And then there are the Toyota Camrys and VW Golfs that are maybe a bit more boring, but they sell to a mass market.
It always comes back to this question of video games as an art form versus video…