July 2010 saw the start of a few months of steady pace. We were continually improving our processes and team work. The product was taking shape and we had just acquired our second client, with many more in the pipeline. Things were looking up.

In Dec 2010, the roller coaster had reached the first twist in the rails. We realized there was a huge technical issue with the way our application was structured. We were using another internal application to facilitate our data import/export. It was currently being hosted as a custom copy of our main data management application. This as you can imagine caused all sorts of maintainability issues. With the introduction of a decent layer of web services around the original copy, we wanted to move our application away from our custom version of the data tool to the main line (maintained by a separate team. Yay!!). This wouldn't be an easy or small task though. Over the last year the two applications had become deeply integrated in all layers except the UI. I don't remember exactly how many of us were working on the separation process but, I do remember we were literally changing the wheels while the car was in motion. Oh did I mention we had little over a month to do it in. 

Troopers that we had become, we took the task on with enthusiasm. What followed is a blur to me, but what I learnt remains. How do you juggle the dependencies between 4 different teams, 3 different but connected code bases and the hopes of a management team? Get the major stakeholders on a phone call (or preferably in the same room) and get them talking! I can guarantee you one 2 hour phone call is better than a million emails, and one 30 min face to face meeting better than four or five 2 hour meetings. We got through the mess and came out with 2 beautifully separate yet, connected applications. I've kept that lesson with me since then, a strict preference for face to face over phone calls over emails for communication (which is funny considering I am talking about this in a blog post :)).

Now you'd think that one such rush of work at a time would be enough but, somebody wanted to really test our resolve so along came a crisis that could really hurt the product's image in the market unless we fixed it and fast. One of our clients had just put the application through the paces and discovered alarming performance issues and we had no clue why. Our tests had shown us that we should have been able to easily handle the load the system was under, but it wasn't. A lot of investigation, pulled out hair and frustration finally led us to why this had happened. Somewhere along the way we had only load tested the system under "read" conditions. The minute we introduced a small number of concurrent writes into the mix, all hell broke loose.  It was such an amateur mistake, none of us could believe we had not noticed this till now.

March 2011 saw us re-write a large portion of our data access layer and put the system through the paces like never before. Handling that amount of pressure was not easy for anyone, but at the end of it all, I came away with two lessons. I would never let  testing be an afterthought to development, it would front and center from the beginning. And the best way to deal with pressure is to take small periodic breaks to keep the stress and frustration under check. An over worked, fatigued mind was bound to compound the mistakes it was trying to fix. 

The lessons that I had learned would prove to be invaluable later that same year, even though I didn't know that at the time. Every failure I have ever faced has taught me an important lesson. Every setback has helped me do better in the future. This might seem cliche to many people as this is the first thing people tell you when you fail, but trust me if you can, they are absolutely right.

1 Comment