BlackBerry Outage Caused by Bad Practices

The good news behind the massive Blackberry outage is that it wasn’t due to hackers or a security breach.  The bad news is that the outage was caused by poor fundamental security practices.

They rolled out new software into a production environment without fully testing it in a development environment.  Then when the crash happened, they tried to fail over to a backup system.  But the backup system didn’t work because they don’t practice failover often enough.

From Reuters here:

A massive service outage of BlackBerry e-mail devices earlier this week resulted from a new storage feature that was not sufficiently tested, device maker Research In Motion said.
 
RIM said it was able to rule out concerns that the failure was caused by a security breach into its system or an inability to provide enough capacity to nearly 8 million subscribers, according to a statement issued late on Thursday.

The crash left politicians, lawyers, business executives and other “CrackBerry” addicts without wireless e-mail service on Tuesday night and into Wednesday morning.

RIM also defended the way it had handled the crisis, despite criticism that it was slow to communicate with subscribers about the extent and cause of the outage.

RiM must have sent an outage notification to its users over the Blackberry.  Seriously, this was negligence on the part of the CIO of RiM.  It is his responsibility to ensure that his staff properly tests patches and new software before putting them to use on a production network.  And failure to practice disaster recovery also lies squarely on his shoulders.

Had a similar outage happened with Hotmail, Yahoo Mail or GMail, people would probably just shrug their shoulders.  After all, its just webmail.  But fierce competition among these groups keeps these services to a near 0% downtime.  Maybe if RIM had tougher competition they wouldn’t be so lazy about fundamental security practices.

Now the question is-  Who will get fired over this?