Monthly Archives: June 2006

Disaster relief effort

We are now in the third week of data patching and emergency code changes to salvage a live system gone very wrong, and morale is at a very low point.

Bosses call it a fire, I call it a disaster. And now we are working our asses off to save as many records as we can, no matter how tired we are. Such an incomplete software system should never have been allowed to go live, but it slipped through several layers of checks and did so anyway. The decision was made to "bite the bullet" and to cut over, crossing over "the point of no return", with nobody really aware of how big a mistake that was until a few days later when we discovered inherent flaws in the design. Worse, the project lead was released to go on a 3 week overseas leave just after cutting over.

My Hokkien-spewing boss liked to joke that working in the IT industry should not involve us acting as superman all the time, wearing red underwear outside and putting in huge amounts of effort. He advocated going home on time and spending time with the family, and went against the company culture of working at least half an hour overtime. He sure needed supermen to help stabilise the system and endlessly patch data. Not that I'm eager to put in heroic efforts to save his ass. I have already given 2 weeks of my youth, or what is left of it, patching data. Crimson Editor is my best friend during this time, its column mode making the very manual task of formulating SQL inserts and updates easier.

The HOD concluded that this product was only at Alpha stage and should not have been released. At this level of immaturity, we would not nornally even call users to come for an acceptance testing! Sigh. We did not have to work so hard even when working on our own projects! I wonder if heads will roll. Not mine; I was identified as the technical lead but taken out because there was some other less interesting thing they wanted me to do. I had no chance to contribute to the success, or failure, of this project. I can say with certainty that I could have averted the technical errors we encountered in the last 2 weeks. They were easy to catch! You know that they don't really believe in the complexity of doing systems when they insist on going ahead without adequately competent people on the team. Don't they know from their years of experience that doing it properly the first time round with the right mix of people can save much grief later on? And they do not lack reminders from us.

Morale dropped to an all-time low. Mistakes in data patching surfaced. Other pieces of work are put on hold. I was entertaining thoughts of quitting without a job. The only good thing to come out of this is that I finally started updating my resume. 6 years is far too long to stay in the same place. Surely there must be better things that I can learn elsewhere.

Bosses talk about how systems know how to wait until the key personnel are on leave to throw tantrums, and leave the rest of us scrambling to rectify the problem with our limited knowledge. The way I see it, a series of bad decisions led to a very bad mess. There is hell for them to pay before everything is over. A lot of questions for the bosses to answer to their superiors and end users.

Leave a comment

Filed under blog, tech