It’s an ancient story: I did something dumb in haste, forgot about it, and then regretted it at leisure. Fortunately, ANTS Performance Profiler saved me. Hopefully this sordid tale will provide a moral lesson to young programmers everywhere.

The Problem

Does this scenario sound familiar? The customer needs an infinitely configurable software tool for “programming” manufacturing test equipment and analyzing data. They have infinite needs, but an infinitesimal budget.   Anything we do has to fit into a twelve-week development schedule and result in a complex tool that is production-line reliable. Oh, and if we so much as grimace, they’ll offshore the whole thing for 40-50% of our already discounted hourly rate.

The Solution

We take two weeks out of the development schedule, working late nights and early mornings. By the end of the two weeks, we have:

1. a generic XML persister tailored to the project needs,
2. a simple programming model – it’s not Turing-complete, but it is pretty powerful, and
3. a user interface that enables non-programmers to program with these tools without writing code, making syntax errors, or even realizing they’re programming.

The Result

We’re heroes! The customer is happy; within a short time, three separate manufacturing engineers with no programming experience have respectively become the Ada Lovelace, Grace Hopper and John Kemeny of our test-programming environment. Our stuff passes the critical test of any programming tool: the product engineers are inventing uses for the framework we had never anticipated – and the simple programming model accommodates them! Whew. We congratulate ourselves as the framework-development gods we truly are.

The “Uh-Oh” Moment

It’s amazing how quickly yesterday’s heroes can become today’s losers (in this case, er, um, us).  A few months later, one of our newly minted test-programming ninjas has created a bazillion programming objects… and it takes more than a minute to load them for editing.  We isolate the problem to the XML persister – clever solution component 1, from above. This tool makes heavy use of runtime reflection, everyone’s favorite performance punching-bag. But we see the problem only on loading, not saving, though both use lots of reflection. What’s the difference between them? Indirect construction!

An experiment establishes that ConstructorInfo.Invoke() takes 60-100x longer than calling a constructor directly. Certain that we have a culprit, we start designing a solution that will require big changes to the framework infrastructure. The seat of my pants tells me that this is a couple weeks’ effort, and should improve performance by 50%, maybe more. It’ll still be too slow, but it’ll be better, and maybe someday we can convince the customer to spring for a 12-week rewrite using System.Reflection.Emit. Sure.

Resolution (In Which I Air A Shameful Secret)

One day, while wondering how I’m going to broach this with the customer, I decide to download ANTS Profiler, so I can have some quantitative data to support my decision. The download and install are quick and painless, so I jump right in and run the executable inside the profiler.  The “Hot” tree pops up. I drill down a few times. This is cool! Then, I can’t believe my eyes.

ANTS is fixated on this one trivial line which, it says, accounts for 86% of the time used by Program.Main():

This can’t possibly be true. Granted, it’s a stupid mistake, but it can’t possibly be accounting for this abysmal performance. But, what the heck, let’s try it. In two minutes, I implement the obvious fix:

I rebuild and run and, holding my breath, I click Edit: The editor pops up with an imperceptible delay. I literally feel dizzy. (Here’s James Musson’s excellent article discussing the issue behind my mistake)

We have just achieved a 120x speedup in fifteen minutes — including ANTS download and installation! Best of all, we’ve avoided two weeks of difficult, risky development. The bad news: I’m so embarrassed; ANTS has caught me in the ultimate newbie mistake! I wish I could blame this on Andrew, our junior associate just out of college, but even he says “I’d never do that!”.

In my defense, I probably wrote that code while sleep deprived, and figured I’d go back and fix it later if we ever got this “proof of concept” working. Perhaps there’s a lesson in this story.

Denouement

The main lesson I choose to take away is this: use a profiler! I’ve been at the software game for a long time, but it’s easy to do something silly in Rapid Prototyping Mode, lose track of possible consequences when my attention moves elsewhere, and then end up with a hidden “performance time bomb”. I have always been hesitant to consider a commercial profiler – our customers have no extra money, our margins are tiny, our discretionary budget tinier, and the Red Gate tools are justifiably pricey. But, that said, now I’m sold. I can’t wait to start refining our stuff using these tools.

Next time, ANTS will help me catch mistakes like these before they cause my customers hours of aggravation.