Tea Leaves

I admit it: I didn’t watch the candidate’s debate tonight. I had two reasons:

(1) My mind is already made up.

(2) Contemplating the sick feeling I will get in the pit of my stomach when I wake up the day after Election Day and read about who won makes me break out in hives. I get enough existential dread when I think about being forced to program in Tcl without adding politics to the mix.

So, I ask those of you who watched, who won? Did it change your mind? Do you think it changed anyone’s mind? What surprised you, if anything?

Add your comments below.

The number of people that know how to effectively debug and triage problems in a complex software product is upsettingly small.

I don’t know why this is. Debugging has always seemed to me a very simple, straightforward task. Start at the top: figure out if the problem is reliably reproducible. If it is, start eliminating codepaths. It’s basically the Holmes Principle applied — when you’ve eliminated every other explanation as impossible, whatever is left, however improbable, is the explanation. Of course there are some bugs that are harder than others — usually, the worst are the ones where you can’t actually describe the conditions under which it happens. But even if we forget about those, I still see people who should know better whose preferred approach seems to be simply flailing at the problem until it goes away.

Computers, and software, are complex things. No one can keep the entirety of a complex software product in their mind at once when debugging. The desire to simplify the problem solving regimen is understandable, and even correct. But there’s a right way and a wrong way to do it. The right way, as I described it, is to break the problem down into subcomponents and perform tests that either validate or eliminate a hypothesis about that subcomponent. Here are some of the wrong ways.

• Magical Thinking - This is something the Mac community had a lock on for many many years. Every time someone asks me “Did you zap your PRAM?” I want to punch them in the face. “Zapping the PRAM” was recommended as the solution to every problem from system lockups, to color profile mismatches, heartbreak, psoriasis, and as a substitute for Apple developing an operating system with protected memory and virtual addressing. You get magical thinking in the Windows world as well, of course — “Defrag your hard disk!” say people who are diagnosing a problem while using NTFS. “Run Norton!” “Scan for viruses!” Why do they say these things? Well, they worked once. Maybe that’ll work again!
• Denial/Blame the Victim - Yes, naive Slashdot addicts and new Linux users, I’m looking at you. “That never happens to me. You must be doing something wrong,” is not, in fact, a useful thing to say. Ever. Inevitably, if these people are or become developers, they’re the guys who write code that doesn’t check for errors because “that error is really unlikely.”
• Broken Logic - “Aristotle and Plato are both Men. A Man killed Sophocles. Aristotle did not kill Sophocles. Therefore, Plato killed Sophocles.” You’d think you wouldn’t see this sort of logic in the real world. But you’d be wrong. The most clear cases of this are when you see someone jump to a conclusion when they eliminate one possibility. What about the other possibilities? It’s one thing to pick a hypothesis to test next, and it’s quite another to declare “Through the process of elimination, the bug is here” — when you haven’t actually eliminated the other likely candidates.
• Punish Good Deeds - Figure out which person on your project is the best at triaging bugs. Declare that the bug is in their code. Then they have to debug the problem for you in order to prove that it’s not.
• Print statements everywhere! - Ok, really, we’ve all done this at some time or another, but it should be a last resort. Particularly in a multithreaded system, anything you do that might change the timing can make debugging harder, not easier.

If you ever go to the hospital for a serious condition, you’ll see proper problem solving in action. A succession of medical students, residents, and interns will march into your room and start asking you questions. Typically, they will ask the same questions, in the same order, until you are ready to scream from the monotony. This is because they are running down the diagnostic evaluation, and each answer is helping them (hopefully) eliminate some possibilities from the universe of things that might be wrong with you. As software developers, we don’t have the collective knowledge that the medical community has about the human body, but we can at least aspire to their commitment to proceed logically.

There’s an old saying “If architects built buildings the way programmers wrote programs, the first woodpecker that came along would destroy civilization.” I’d suggest a corollary: “if doctors diagnosed patients the way programmers debug programs, no one would ever risk going in to a hospital.”

I should also admit that I’m blurring the lines between “triage,” which is coming up with a rough idea of what component of a software system is failing, with “debugging,” or fixing the bugs. One is really a special case of, or a part of, the other. I’d argue that getting triage wrong is worse than getting a bug fix wrong, because of the inherently timewasting nature of misclassified bugs.

So I’ve given you some sarcastic ranting about how not to triage problems. How about some constructive suggestions? I’ve already talked about trying to ensure that you can reproduce problems, but that won’t be possible for every bug. What else can we do?

• Always have a hypothesis - Hypothesize what is going wrong early, and often. Don’t get attached to your hypotheses. Expect to be wrong most of the time, at least early on. A good hypothesis is one that you can construct a test to disprove.
• Triage bugs to your limit - There’s nothing wrong with talking to other people about a bug, or asking for help. But when a bug comes your way, you should not hand it off to someone else until you can give a narrative of what you think is happening, and why you can’t trace it any further. It’s all just code. Even if it’s code you’re not familiar with, most problems you encounter will be of a pattern you’ve seen before. And if you find a type of error you haven’t seen before, congratulations! You’re learning something new.
• Write everything down - Keep a running explanation in your bug tracking system of what tests you’re trying, what machines you’re using, what your results are, theories, conversations, etc. If the bug is handed off to someone else, they’ll appreciate it, and even if it isn’t you’ll be able to search that database later when the exact same bug crops up in a different place.
• Use the best tools - Learn how to use every tool at your disposal, and master it. Debuggers, profilers, and fault injection are your friends.

I hope in some small way this little rant has proven helpful. Good luck. Now go squash some bugs.

What time is it?

Coffee Time is a Toronto coffee and donuts chain, roughly equivalent to what Dunkin’ Donuts used to be in the US, before they made the conscious corporate decision to make all their donuts suck. They also have the best brewed coffee of any chain in Toronto (which, as I’ve alluded to before, isn’t really saying much), and better donuts than Tim Horton’s. They seem to be a thoroughly urban phenomenon — as soon as you get on the highway, they disappear, and you are left and bereft, Coffeetimeless.

venti skim no-foam vault

Meanwhile, in another sector, CIBC provides banking services to the eh-ing multitudes of Canada, along with convenient ATM service, from which we lucky American citizens can withdraw money from our American banks without paying any extra fees. (Have I mentioned how much it annoys me that it is cheaper for me to withdraw money from an ATM in Madrid than it is at the “wrong” bank in the US?) But the most notable feature of CIBC bank, to me, is that their logo looks disturbingly like Coffee Time’s. So I always experience a moment of cognitive dissonance when I encounter a CIBC bank and think “Cool! I can get coffee!” and then a rush of disappointment hits me as I realize that all they have there is a bunch of stupid money.

Every problem, however, is also an opportunity. I propose that CIBC Bank purchase Coffee Time, thus adding the latter’s roster of delicious and inexpensive treats to their bank’s portfolio. In these highly competitive times, the ability to draw in new customers through any means cannot be dismissed. And Coffee Time will benefit as well, as CIBC’s large customer base discovers the delights of a walnut cruller and a large black coffee while waiting on line to make a deposit.

Do not be dissuaded by the naysayers and small-minded ones who throw out trivial objections — “Oh, but banking and coffee shops are different,” they will whine. “What about your customers who don’t drink coffee?” they will mewl. They’ll tell you that basing a merger simply on the color of your logos is “crazy.” To which I say: “HA! Crazy like a fox!” Think of the most successful financiers you know. Think of stockbrokers, think of floor traders. Is there even one that doesn’t drink coffee? Of course not. Coffee is bold. Coffee is strong. Just as currency is the lifeblood of our economy, my friends, so too coffee is the…um, lymph. Coffee Time is one of the most productive lymph nodes. With one fell swoop, you can grab that lymph node and squeeze! Sure, they’ll laugh at you now, but when their banks are closing because you’ve got the best lymph coffee, well, they’ll be singing a different tune, I promise you.

Gentlemen of the Boards of Directors of CIBC Bank and Coffee Time — I thank you, capitalism thanks you, and the people of Canada thank you.

Additional Resources

Both Coffee Time and CIBC have web sites. Soon, I hope there will only be one.

I’ve heard about Retsina for years, but it was not until recently, at a Greek restaurant on Danforth in Toronto, that I actually got to taste it.

It tastes like rosemary wine.

That’s not a bad thing.

Rosemary, of course, is a member of the pine family, and Retsina is treated with pine tree resin; hence the taste.

It’s a light white wine, similar in body to a Frascati or a Pino Grigio. Behind the rosemary/pine nose and taste, there’s a vegetal, grassy flavor, and a slightly medicinal, bitter tang. I could see some people not liking it, but like the hoppy finish in a good beer, I found that the bitterness cleansed my palette and readied me for the next sip, or the next taste of food. This is definitely a wine to drink with food.

Not something I’d drink every day, but it makes a nice wine for a warm summer day.

Bonjour Brioche is a breakfast place in Toronto. It’s inconveniently situated on Queen Street East about midway between downtown and the Beaches, at the corner of Queen and Broadview. It’s small, cluttered, and there’s usually a wait to get in. The hours are annoying and too short. And it has some of the best bread you’ll ever have.

The croissant is superb. The baguette transcends belief. The coffee is good. Amusingly, the brioche they are named for is only just sort of OK — if you’re in the mood for a pastry, skip it and get the croissant instead.

Bonjour Brioche sports a full breakfast menu, so if you want an omelette, or some bacon, or some scrambled eggs, smoked salmon, and caviar, no problem. Their non-pastry breakfasts are good, but kind of miss the point. The highlight of any of the omelettes, inevitably, is the piece of baguette that comes with it.

One notable item on the menu is the “baked French toast.” This is somewhat of a misnomer. It’s really a slab of bread pudding about the size of a loaf of bread. It is loaded with cinnamon, and sugar, and butter. In a pinch, it could be used to feed a family of four for an entire night. It’s a bit overwhelming, but is definitely worth convincing someone else at your table to order.

My recommendation? Just get a whole baguette and some butter and homemade preserves. This is not intended as an insult to the rest of their breakfast food, which ranges from serviceable to good. Rather, it’s an indication of how excellent their baguettes are. There’s a belief I hear expressed that you can only get good baguettes in France. That’s false. The main issue with baguettes is that they are breads with minimal fat, and high surface area. This means that the amount of time in which a baguette is truly fresh can be measured in a small number of hours. Baguettes are consistently better in France not because of some unknown mysterious secret baguette-making technique, but simply because, being smarter about food, they eat more of them and the ones you get are, usually, fresher.

There are also a variety of fruit tarts and quiche. All are excellent. If you’re in Toronto over the weekend, and don’t mind waiting for your breakfast, it’s definitely worth a visit.

I considered making a short film of my last trip to Bonjour Brioche, but realized that they were too busy to talk to me, so basically all you would have had was a movie of me eating food that was better than whatever you’re eating right now. So I skipped it. My new plan is to contact them and ask them to let me spend a night in their kitchen and film them while they bake bread. That would be a good short film.

Bonjour Brioche is closed on Mondays. Other days, they open at 8 am and close at 5 on Tuesday through Friday, 4 on Saturday, and 3 on Sunday.

Don’t be fooled. This is easy. You just have to follow a few simple rules and you too can make a beef stir fry that will show up the dishes at P.F. Chang’s as the chewey carboard crap that they are.

First, the typical chinese stir fry starts with flank steak. I don’t know why, but it’s what my mom did, and she’s generally right.

Second, cut the meat as thin as you can across the grain. This is critical. If you cut with the grain, you get rubber biscuit. I’m assuming for the rest of this piece that you started with about 1/2 to 1 pound of meat.

Third, marinate for a while in a mix of soy, sherry and chopped ginger. A while is around a half hour.

Now, heat your pan. Any pan will do. I like the small non-stick omlette pans. My mom always used a cast iron frying pan. You can use a Wok if you must, but it’s not critical. When the pan is hot, add oil. I usually have olive oil, so that’s what I use even though it’s sort of wrong. Add about a teaspoon or two of corn starch to the meat and mix it around. Toss the coated meat in the pan and mix it around so the meat isn’t all stuck together. Toss in more ginger, some salt, and pepper.

The fourth secret is as critical as the second. Let the meat cook until it starts to make its own gravy in the pan. Do not under any circumstances add any liquid of any kind to the pan. If you do this, you will get rubber beef.

When the meat is done, it will expel some juices back into the pan and give you a nice sauce. At this point you can add oyster sauce or something if you like that. You can also toss in cut up green onion and wilt it with the meat.

Otherwise, just throw the meat on a plate. You are done.

The meat should be tender and coated evenly with a nice brown sauce that is not gloppy.

To make the classic variation on this dish, take the meat out of the pan just before it is done. Throw garlic, scallion and big chunks of tomatoes into the pan. Cook until the tomatoes start to fall apart. Add the meat back in. Mix and take out of the pan. Don’t cook the meat with the tomatoes too long or it will get tough (see point 4).

This is all you need to know about beef stir fry.

Everett Kaser had one great idea for a game.

He’s taken that one idea, and leveraged it into many, many games. Which of them you play is a matter of taste. But if you don’t own at least one Kaser game, you’re missing out on the most addictive puzzlers since Sokoban.

Rewind to 1990: my friend Nerak hands me a floppy with a DOS shareware version of Kaser’s great idea: the game Sherlock. Have you ever heard those logic puzzles that begin: “Doctor so-and-so needed to seat 12 people at the dinner table. The lawyer wants the vegetarian dish, and won’t sit next to the doctor. The man with the yellow hat wants roast beef. Red wine is being served to the person who has the chicken…” (and so on, and so on, and so on). Sherlock is a more abstract version of that puzzle.

The main playing board is a 6×6 matrix of cells. When the puzzle is solved, each cell will contain a single icon. When you start playing, each cell has 6 small icons representing the possible inhabitants of that cell. Cells are grouped into rows. So, for example, the 6 cells in the top rows are all faces; to “solve” this row, each of the 6 possible faces must be put in its proper cell. With the default icon set, from top to bottom, the rows were faces, houses, numbers, fruits, road signs, and letters. (More recent versions of Sherlock allow the player to customize the board size from as small as 3×3 to as large as 8×8.)

Arrayed around the main board are the clues. There are a few different sorts of clues, all in pictographs. I find that as I read them, I translate them into words in my head: “The ‘No Parking’ sign is in the same column as the face which looks like a big lemon head.” “The green house is between the strawberry and the number ‘3′.” “The letter ‘H’ must be to the left of the Egyptian Headdress Guy.” You’ll quickly learn there are corollaries to the different rules that let you make progress quickly (for example, if the green house is between the strawberry and the number 3, then logically it can’t be in columns 1 or 6, which are not ‘between’ anything).

To help you keep track of what possibilities are available, you can right click on a ’small’ icon (or a clue) to dismiss it from consideration. Left-clicking on an icon is an assertion that you believe you know where it belongs.

To call Sherlock an addiction is to underestimate it. Every game is different. Every game is fun. I have been playing Sherlock for more than 10 years. I’ll be playing it forever.

Kaser, knowing a good thing when he saw it, took the Sherlock idea and ran with it. The first sequel was called Dinner with Moriarty, and was a more elaborate (and slightly less abstract) version of the same game:

The nefarious Professor Moriarty is giving a dinner party. As luck would have it, YOU have been invited. Professor Moriarty and his guests are seated about the table, and each person is eating a different food on a different colored plate and drinking a different drink. It’s your task to determine WHERE Moriarty is seated (along with everyone else) and which person he’s poisoning.

The core mechanics of Moriarty are the same as Sherlock, although the board configuration differs and there are some slightly different types of clues.

There are other variations, as well. There’s Honeycomb Hotel (Sherlock tilted diagonally and with the addition of borders between cells that you must deduce.) There’s Latin Squares, a particularly vicious variation which adds affinity groups which span multiple rows and columns. There’s Occam’s Quilt, in which every cell has not only an icon, but also a background color and a background pattern, only one of which must match with its neighbors. These are generally harder games. Any given Sherlock puzzle is always solvable — somehow — given the provided clues. It’s about straightforward deduction. Honeycomb Hotel and Latin Squares, however, will often generate puzzles which require the player to make forward progress by making an assumption and then eliminating that assumption as invalid when they discover a contradiction (the games provide an in-game mechanism to make reverting to the move before you made your assumption simple).

Kaser has created some other great games in addition to his deductive masterpieces: Descarte’s Enigma, in particular, is a great implementation of the “Paint by Numbers” or “Nonogram” puzzle. But it is with Sherlock and its kin that my addiction rests. Until Cliff Johnston releases A Fool and His Money later this fall, I’ll be clicking my way through Kaserland, one small deduction after another.

Regrettably, Kaser’s games are only for Windows (and in some cases PocketPC). If you can track down a DOS version of Sherlock it runs fine on Mac OS X under DOSBox. Sherlock is available at Kaser’s web site, as are demos of all of his other fine games. You can purchase full versions on line, as well. Enjoy.

And remember: I warned you.

Back from Toronto, with new tales to tell. This week, expect to see some or all of the following items:

• The best baguette on Queen Street
• Why is coffee in Toronto so uniformly bad?
• A proposal for Canadian banking reform.
• Retsina: the other white wine

…and, when they’re done, book reviews of Persepolis 2: The Story of a Return, In the Shadow of No Towers, and the most recent of Jasper Fforde’s “Thursday Next” books, Something Rotten.

In an earlier article, I outlined my general thoughts on digital cameras. In that article, I noted that digital cameras fall into two basic categories: point and shoot cameras, and digital SLRs. At the time, my main camera had been a Nikon D100. So I thought now I’d write a more specific piece about that body and its more recent cousin, the D70.

The D100

The Nikon D100 handles and shoots basically like a high quality mid-end Nikon film body, except that it captures digital files instead of slides or negatives. Light goes in one end, hits a sensor, and out the other comes a digital representation of what the camera was looking at. I will forego the particulars and specifications of the body because you can google for that.

Instead, I’ll concentrate on what I like and dislike.

Likes:

• Shoots just like a normal film camera.
• Starting with the raw CCD data, this camera makes files that I’ve made into prints that are better than any color prints I have ever had made from 35mm slide or negative film. This is especially true at higher ISO values, like 400 or 800.
• Shoots just like a normal film camera.

There isn’t much more to say than that. The thing is awesome. Now here are my picky whiny quibbles:

• The Nikon RAW (NEF) files are ludicrously large. Unlike the Canon bodies, when you shoot RAW files with the D100, there is no compression. Somehow they left a hardware compression pipeline out of the image processing hardware. This means I have to put up with 10MB NEF files instead of the 5MB files that you’d get with a Canon. The big files also take a long time to write to storage, so when the camera’s memory buffer fills, the wait is long.

• The memory buffer in the camera only fits 4 NEF frames. And, when it fills, it takes a long time before you can shoot another frame.

• The big NEF files make you want to shoot JPEG, but in camera JPEGs don’t always come out right, so you really want the NEF file too. The D100 makes you choose.

• I’ve never been happy with the JPEG files that D100 makes. They seem to be soft and noisy at the same time, especially at higher ISO speeds. Also, the color always seemed off. Maybe I just never got it tuned right.

• Setting ISO, white balance, and image quality and other things on the D100 requires that you put the camera in “set this parameter” mode. In this mode, the shooting engine of the camera is locked out. You have to put the camera back into shooting mode to shoot. This blows.

• The interface for zooming in playback mode is a bit clunky.

• I don’t like the little knob that sets the metering mode. I like the little button that my old Nikon 8008s had.

• The flash sync is too slow to do fill flash out doors under most conditions.

• The viewfinder is not as nice as the better ones on Nikon’s film bodies. This turns out not to be a huge problem.

The D70

Summary: The D70 is the D100 with most of the above quibbles fixed.

The only thing that isn’t really improved in the D70 is the viewfinder and the interface for changing some autofocus modes that I never change. Here is a more detailed summary:

• NEF files are now compressed in hardware. So they are nice and small.

• The D70 has a NEF+JPEG mode that shoots a NEF file and a proof JPEG at the same time. Perfect! If a good picture requires some touchup, you have the NEF file. When the JPEG works out well, you can put the proof right on the web and even make 8×10 or bigger prints from it.

• The D70 writes to the card faster than the D100. So even though the memory buffer still only fits four frames, with a fast CF card you can shoot NEF files at almost 1fps, which is a lot better than what the D100 could do.

• In camera JPEG files seem somewhat better than what the D100 did. But this might just be the piece of mind that comes from knowing I have the NEF file around.

• In playback mode, hitting “Enter” now puts you right into a zoomed view. This is nice because most of the time it lets you quickly evaluate sharpness, which is all you do with this mode anyway.

• ISO, WB, and such are now set using a button+dial interface in shooting mode.

• They put the metering mode button back where it was on the 8008s. Yay!

• Flash sync is now 1/500th! Take that Hasselblad wankers. This is great for fill flash.

• The viewfinder is not better, but it’s not worse either.

All this, and they fixed the flash system too. Although I have to buy a new flash to get the new features.

Two Updates

First, the NEF+JPEG isn’t as useful as I’d hoped. The JPEG files are not tagged with the right color space, and they are full size. I like to make proof JPEG files that are half size. Also, the new NEF files preview very quickly at nearly full resolution (apparently the preview image embedded in the NEF is now a full sized JPEG instead of the teeny tiny one the D100 put in), so I don’t really need JPEG files for quick proofing. They are mostly just useful for my dual catalog workflow.

Second, the lens hood casts a shadow. I never noticed this before with my D100.

Appendix

This article is also at my main web site, so I’m gonna link to it here to up my google points:

here you go.

I’m still in my office, rather than on my way to Toronto, because Pittsburgh — inland though it is — has had its own close, personal, and somewhat intimate encounter with Hurricane Ivan. To call what has happened here “flooding” is somewhat of an understatement.

17 September 2004, is officially the rainiest day in Pittsburgh’s recorded history, with 5.08 inches falling in a single day.

Small Japanese sedans are floating through the streets. Highways are closed. Basements are flooded. And, most tragic of all, the traditional Friday Night Pizza has not been delivered to my place of employment. While realizing that you can’t go home does tend to focus the mind on fixing the bugs — since there’s nothing else to do — the gnawing hunger tends to distract.

It’s not just raining cats and dogs here; the cats and dogs themselves are actually spitting and drooling as they tumble from the sky. I still have plans to try to get to Toronto tomorrow — the room, alas, has been booked and paid for — but won’t be surprised if the highways are blocked. Somewhere on Yonge street there is a hot dog with my name on it, though. I shall persevere, and find a way.