August 25, 2004
Pharaoh Speaks
by peterb
Andrew Tepper
Typically, interviews like this focus on the gameplay design or community development. I'm less interested in that (in part because I think you can often explore those aspects by simply playing the game). Instead, my aim for this interview was to focus specifically on the actual technical development challenges in creating a complex online game with a small team, something that few of us have done. If you're want details about the game itself, you might want to look at my recent review. I reached Andrew at home, just after he had finished up work for the evening.
peterb: A Tale in the Desert has been released on multiple platforms (Windows and Linux) with more on the way. How early did you decide to go multi-platform? Did it impact development?
Andrew Tepper: We did it immediately, so right from the start we did things like separating the rendering code so that in theory there was one file that needed to be changed to go to a different platform, or to a different graphics API (openGL to DirectX, for example) and then a single #ifdef to change to a system that was different byte order. In practice, it worked pretty well. There were a few bugs, but it's been pretty easy. For the most part there were just a half dozen or so bugs where we made assumptions about byte order...the whole idea was that stuff that hooks in to the operating system is isolated. It was pretty clean. The Linux port took about 3 weeks. And the Mac port has been underway for about 5 weeks.
How big is the team, and what development environment are you using?
We have 2 full time coders, and 3 full time graphics artists. We use Visual C (a slightly older version) on the Windows side and just gcc in Linux and, I don't even know what we use in Mac. gcc, I think?
How do you track bugs?
We just started using bugzilla for script code kinda bugs. With two programmers, tracking bugs isn't a big deal -- if there's a bug, you just scribble down a note and then fix the bug.
How about version control?
We use cvs. That's been helpful. Sometimes, i'll go off on a project kind of experimentally, or my partner will, and 20% of the time it won't work and we'll just throw it all away.
What's the single hardest bug you've encountered in developing A Tale in the Desert? How did you fix it?
The hardest bug was probably -- well, I have to explain this. Our execution model is that the client has synchronous and an asynchronous, or we sometimes say a predictive, model of the game world. The server has a master world model, distributed over many physical CPUs, and a predictive world model for each client. The synchronous model is the subset of the master model that a particular client sees. The invariant is that the server model for a given client, and the synchronous model on a client, always stay in lockstep. So for a given time, the synchronous models must match exactly. So we had one or probably a few bugs where for some reason the synchronous model on the client got out of sync with the model on the server. So that required changing the memory manager and rewriting a whole bunch of subsystems in the server so that we were able to log every byte going in to one of the cpus and every byte coming out. So we wrote a replay facility which we still use which simulates the bytes going in to a server. The solution to finding the bug was replaying on both the client and server kind of at the same time and figuring out where the two synchronous models diverged. It turned out to be something like uninitialized memory. It was not a very glorious bug. But because our execution model is pretty unique, it took some custom tools.
The replay tool is so handy that now when we get a server crash, 90-plus percent of the time, the solution is you take the replay log, rerun it in a debugger, the server crashes in the same place, and then you say "aha!" For something as complicated as a massively parallel system as this -- boy, I can't imagine a better way to do it. The server itself, by the way, is a single-threaded application. the server is designed to handle a million pseudo-threads at once -- every building in the game gets its own pseudo-thread. On the client we use three threads.
A pseudo-thread isn't actually sharing memory with other threads, so there aren't any tricky concurrency issues?
Yeah, it's not an OS-level construct. It's just an internal thing. That's just what we call it.
Is there any feature you implemented that you wish, from a timeline standpoint, that you wouldn't have implemented? Something that you look back on and say "Wow, that really blew the schedule for not a lot of benefit?"
Josh [Yelon] would have different answers than me. For instance, I bet Josh would say bump mapping was more effort than it was worth, but I would say just the opposite. For me, there were a bunch of kind of specialized things we added, not big architectural things that were a pain in the ass.
Are any of those still in the codebase?
That still remain in the code? No, because usually when something like that happens we rip it out. For instance we had a conversation system built in, so if you had NPCs in the game you could talk to them. We didn't actually start out to write A Tale in the Desert. We were writing a system for creating massively multiplayer RPGs, so you had a character in the middle of the screen, and there are other characters running around, and objects to interact with. We knew we were a small company, so we spent a long time writing sophisticated tools to let us quickly develop content, because we knew we'd never be able to dump millions of dollars into developing the content. We spent 2 and 1/2 years not writing a game, just developing a toolset. If you ask me what part of the toolset is the biggest thing, it's being able to change the code on the fly without kicking people off. That's as big a change to developing as going from punch cards to interactive editing. Instead of the big testing cycle, you can put your code out as soon as its done, and the really bleeding edge players will run into bugs, and you can fix them in minutes. It's a giant leap forward.
A lot of the players in A Tale in the Desert seem fairly sophisticated, and they're very vocal. So it's clear that a lot of the content has been developed with these more sophisticated players in mind. Certainly, most of them have played other MMORPGs before. So how do you gauge how the game is received by Joe Average who has never played an online game before? Do you capture those guys, or do they wander away confused?
It's very hard, because they're kinda shy. They don't talk as much as the MMO veterans. I've done things like kinda gone incognito and played the start of the game, to try to find out. But you don't know if someone's dropping out because their connection went down, or they don't like it and they're embarassed to say it, or they feel stupid. I don't know. It's very hard.
You're extremely accessible to your users; players and GMs have your cell phone number, and often lobby for features vocally (even outside of the in-game 'election' protocol). Is there a downside to giving such easy access to your users? Has pressure for a particular feature ever blown the schedule?
In 18 months I've only had 10 inappropriate calls. As for the questions on the schedule, players will always ask for features, but ultimately I'm in control.
I've tried to stick to questions about development, but do you have any comments on the game design in terms of the choice to go with economic model vs. monster killing?
I consider this an adventure game. I think Tale is kinda the MMO side of adventure games, where the puzzles are all social puzzles... Think about the Test of the Covered Cartouche, for example. [Ed: In this test, participate in successive rounds of creating a building in secret, all throughout Egypt. At the end of each round, the player whose building is the smallest is eliminated. Since creating larger buildings is quite expensive (which can spoil your chances for future rounds), the challenge is to spend as little as possible on your building while not spending the least.] The puzzles in A Tale in the Desert are all like the old-style adventure games, but with the smartest NPCs you ever came across. If you think about the game, about unlocking technologies, it's an adventure game. We add some stats so it's RPG-like, but I think Tale is much more an adventure game then everquest.
To me the boring stuff is where there's just a single plodding path -- find a sheep, grow onions, feed the sheep -- but, y'know, the whole game can't be a chess match. There has to be some mindless stuff, then some thinking stuff. But the coolest thing to me is where I make up a challenge where I don't know what the best solution is. A few times, I've screwed up and the players haven't stumbled on a solution. For example, the Test of the Seven Phoenix in Tale 1 [Ed: The Test of the Seven Phoenix involves building expensive statues of a specific type in named locations within a one hour period. "Waypoint" travel during the test is forbidden. Which statues need to be built in which locations is a secret before the test begins.] Either I just made it too hard, or there is no good solution, but it hasn't been a great success.
If you could work on any game next, and it couldn't be an MMO, what would it be?
If it couldn't be an MMO, I like physics games. But what those have in common with Tale is that they're both sorts of sandboxes where the designer ahead of time doesn't know what the best way to solve it is.
Thanks for your time. I hope Tale 2 is a great success.
Thank you. Nice talking to you.
Posted by peterb at August 25, 2004 07:33 AM | Bookmark ThisPlease help support Tea Leaves by visiting our sponsors.
