The joys of disaster recovery

After moving from the CU to the CUSO, I’ve had the opportunity to interact with many more CU people across the nation.  One thing I’ve noticed is the massive amount of resources (time and money) spent on disaster recovery and business continuity.  Obviously, getting a credit union back up and running after a disaster is critical.  Surviving said disaster is equally important.  At our last all staff meeting the topic of bird flu came up and some of the preparations that our credit union is making from a business continuity perspective to keep the credit union operational in that period of time. There were a few chuckles around the room, but ultimately, we’ll be prepared if and/or when something of that nature happens.

All of this disaster stuff I see at work started making me change some of the stuff I do at home too.  Like Brad Garland, I too have started to remotely store all of our family pictures, videos, etc off-site, meaning not in our house.  The analogy I always use was if you house burned down and you could only take one thing, what would it be?  For me, it was my computer housing all of our pictures, but now that they are backed up, I don’t have to worry about it.

Continuity is defined as the “absence of interruption”.  This has also made us change some of the other things at home.  We try to keep good batteries in the flashlights, enough water for a few days for us, dog, and the cat, and a few basic non-perishable foods around.  A tremendous resource for at home business continuity and disaster recovery is  They have great info about creating a preparedness kit, how to deal with animals, and many other aspects of general preparedness.  If we all have to do this stuff everyday at work, it probably would be beneficial to do it at home as well.

How well are credit unions prepared for data center outages?

 Back in July, 365 Main, a major data center hosting companies such as c|net, Craigslist, e-surance, and the Oakland Raiders, lost power.  For most of the day.  And 365 Main has the time, money, and resources to make sure that (almost) never happens.  Here is the full story straight from 365 Main.  Jesse Robbins has a slightly more digestible version here at O’Reilly Radar.

 So if a major provider like 365 Main can drop clients for half the day, what protections are in place in our CU’s?  What are reasonable expectations from our members?  Do we need to stay up for 24 hours after a power outage, or just survive until the power company comes out to replace the broken transformer or power pole?  Obviously the most important thing is data loss and, for the most part, as long as the CU has the appropriate safeguards in place to protect against power surges, the servers and data should be safe from melting.  After basic data preservation then comes up time.  As a member, how would you feel if you went into a branch and the teller said that they couldn’t actually do anything on your account until tomorrow because their computer system was down?  You probably wouldn’t have online banking to check your balance and the CU would probably have to limit the amount of cash they could withdraw for members since they don’t actually know how much each member has.  So out comes the pen and paper to hand write which members took how much money.  Not an ideal situation.

Put the members’ hat on.  What would they expect?  I’d be miffed if I couldn’t get cash out, but I’d probably survive.  For a day.