Issue #6
February 2002
 |
 |
 |
Disaster Recovery Done Right by David Katz At the Managed Funds Association Network 2002 meeting this month in Miami Beach, Mark Fichtel, CEO of the New York Board of Trade (parent organization of the Coffee, Sugar and Cocoa Exchange and the Cotton Exchange), described the process that enabled these commodity exchanges to be able to resume trading within days after the trading floor and computer systems, housed at 4 World Trade Center, were totally destroyed on 9/11. In these uncertain times, the Board of Trade's experience provides a model for corporations and other organizations where business interruption and loss of data would be disastrous.
Until 1993, when terrorists blew up a truck in the basement of the World Trade Center, NYBOT's disaster recovery program consisted mainly of an arrangement with a third-party organization to provide a cold-backup computer facility; that is, an agreement to allow NYBOT to use the external company's computer center to load and run NYBOT's data and programs in the event that NYBOT's facility was down. The organization they contracted with had similar agreements to back up other companies and organizations.
In the wake of the 1993 explosion, NYBOT's administrative and IT management realized that this arrangement was inadequate. An area-wide disaster (or power-outage) involving many firms would overwhelm the backup capabilities of the company providing the facilities NYBOT relied upon. Furthermore, with a process that required traveling to the backup site, loading programs and data and then starting processing, even a relatively brief power outage would develop into an all-day shutdown, with possible loss of data.
The internal task force charged with improving NYBOT's disaster recovery capabilities created a report calling for a hot-backup site dedicated to NYBOT's needs. Under this scenario, a duplicate, off-site data processing and communications installation, located in another part of the power grid and telephone switching system, would run continuously, mirroring all the processes of the primary data processing and communications setup. In the event of disruption at 4 World Trade Center, an automatic changeover to the backup systems would take place.
With the assistance of recommendations and economic justification by Deloitte and Touche, the Board of Trade's membership approved the expenditures for the hot-backup site. It was at this point that the administrative and IT people on the in-house task force showed their true professionalism. Realizing that a disaster severe enough to knock out computer processing might also be severe enough to make the trading floor unusable, the task force altered their recommendations to include a modest trading floor at the backup site. As Mr. Fichtel put it, "That was the point at which a disaster recovery plan became a business continuity plan."
Following another Deloitte and Touche report endorsing the plan and laying out the enormous costs to the Board and its membership of each day of halted trading, the backup site was acquired and built. A business continuity manual was created laying out responsibilities and procedures in case of disaster. “Calling trees” and other communications procedures to allow information to be disseminated to staff if their offices became unavailable were put in place. Most importantly, quarterly disaster simulations were held, so that the procedures were thoroughly tested and staff knew what to do. When 9/11 came, they were ready. And it worked.
Of course, nothing is ever perfect. Mr. Fichtel offers some hints that would have made life easier for the NYBOT operation-in-exile. These include: have enough bathrooms and parking; consider where everyone will work and take steps to bring people back together; have enough security; don't neglect doors, staircases and elevators to allow adequate movement of people and deliveries; back up paper offsite (not all critical documents have electronic equivalents); and, especially important, convince your suppliers and customers to also have backup plans.
Larry Kalmis, Vice Chairman of the Business Continuity Institute and Principal of Larry Kalmis Associates, a business continuity consultancy, commented about the Board Of Trade experience, "The Board Of Trade is to be commended for the thoroughness of its business continuity process and in particular for the inclusion of the quarterly simulation exercises. Business continuity plans are implemented to protect critical assets and business processes from unplanned disruptive events. To support the protection of these assets, business continuity plans must be regularly exercised to identify needed plan enhancements, train critical response staff, and help ensure proper operation in a disaster.
“While many companies may have plans, few are truly prepared to execute their plan when it is needed. Creation of business continuity plans is just the first step. Resources must be identified and secured, and plans must be validated, refined and maintained. The plan is, at best, a guide and not a bible.
“Exercising the plan is a vital component of the business continuity management process. It's simply not enough to develop plans, document them and then set them on shelves until a disaster occurs."
Click on underlined by-lines for the author’s home page.
New York Stringer is published by NYStringer.com. For all communications, contact David Katz, Editor and Publisher, at david@nystringer.c om
All content copyright 2002 by nystringer.com