Posts Tagged ‘change management’

My Biggest Test Mistake

October 13, 2013 Leave a comment

Lesson learned: A one character change can have a very significant impact on quality.

I love to hear and tell stories in illustrating lessons learned. They really convey the emotional impact of the lesson at the time it was learned. This is a story about a very costly mistake when attempting to make a small one character change in application resources that I would never want to repeat again.

In 1993, I was working at Custom Applications on the FreedomOfPress product line. This is a print driver application that provided a software PostScript RIP (raster image processor) for rendering pages to over 150 different printers.  We were releasing a new Mac version for mass distribution to the major software distributors (Ingram, MicroD, etc.). At the time, software was distributed on CD’s from a glass master and silk-screened. This included sending a CD image to the disc duplicator, who created a glass master image, and then stamped out tens of thousands of resin discs. The discs were silk-screened with images and print, and assembled into a printed box along with its printed manual and other material.

Macintosh-centris-660avA blockbuster release of the new Macintosh Centris 660 AV was was the hottest thing in desktop publishing and the CEO had it in his office. We were able to get Apple hardware ahead of distribution to develop and test with the hardware, so we only had one. We had a chance to test with it at times and didn’t have it available to us at all times. As we were preparing for the trade show that Apple was unveiling this new hardware we were received registered trademark ® certificate. Of course we had to have this symbol on our box and on the software! The marketing department was all over this.

We received a request from marketing to add one last change and test, on the day we had to send out the disc image for duplication. This was only a one character change, right? How hard could it be? Our application artwork was created on a Solaris work station and saved to the Mac as a resource. We were asked to add the ® symbol was the splash screen when you start up the application. The text was updated on the Solaris workstation and saved to the Mac (as we normally did, right?). We built and spot checked this new version with the ® symbol proudly displayed on the splash screen on machines we had in the lab. We then sent the new disc image to the duplicators by overnight courier.

The next morning the CEO comes into the office and installs the new version of FreedomOfPress onto the Centris 660AV. He launches the application and the splash screen appears first…the system restarts. What (?!) he says to himself…he thought he must have done something wrong. When the system comes back up he tries again with the same result. He calls me in and explains what he experienced and shows me what is happening. Wow! How did this happen and how did we miss this? Well that machine was nicely tucked into his office when we were testing permutation of machines and printers the day before.

The VP of Engineering was called in and we discussed what might be the issue. After all, we only made one character change from what was running properly on this machine only 24 hours earlier. This is bad…very bad. All the engineers involved in making this changed looked carefully at the code and resources and diff’ed the files. Ah, there is one difference other than the ® symbol. A control+D symbol appears at the end of the text file. This was a normal file termination character for the text file created by the editing application on the Solaris workstation. This control character means something very different on a Mac however, instructing the computer to reset. Ouch! We had to stop the disc duplication immediately and get a new image to the duplicators.

Marketing calls the disc duplicators and they find out that the duplicators had already created the glass masters and proceeded with high speed manufacturing of the disc copies. They had already created thousands of copies. Agh, they had to stop the line and these masters and copies had to be trashed. After removing this control+D character, we spent that day testing through this disc image again, including the Centris 660AV. The new image was sent out to the duplicators overnight again and they rushed through the order. We were able to get the new product boxes to the trade show just in the nick of time.

This was an expensive one character change. As it turns out, there was a corner cut when making this change and the text was not re-editing and saved on the Mac before adding the resource to the bundle. If this step was not skipped, the control-D would not have appeared in the final resource file and this problem would not have been introduced. Also, we should have included the new hardware in the list if things we had to verify.

The real lesson learned here is to pay very close attention to the proven process and not cut any corners when making even simple changes, especially when making a seemly benign change at the last moment.

‡ a.k.a. ColorAge, purchased by Splash in the late 1990’s

%d bloggers like this: