Lockbox Lessons Learned from the Front Lines of Katrina and Rita

David Frady, Capital One

Executive Summary

Hibernia Bank, being a 135-year resident of the below-sea level hurricane zone that is New Orleans, had what it thought was a dependable, oft-tested Disaster Recovery Plan in place as Hurricane Katrina approached last August. They laid in more cash, since power outages would halt credit and debit card transactions and make checks unverifiable. They were going to send key personnel to recovery sites where they could keep operations humming and hold twice-daily conference calls so that the Incident Management Team could give senior management status reports. They intended to keep employees abreast of work conditions through hotline and Website updates.

But the planning for the unprecedented catastrophe that was Katrina wasn’t enough, because nobody in New Orleans, including the banking community, could fathom the impact of a Katrina-sized phenomenon, which they had never experienced. But Hibernia emerged from this catastrophe having learned valuable lessons about what worked, what failed, what to correct or install for the inevitable Next Time, and about the human capacity for survival, adaptability and teamwork under the most terrifying circumstances.

The limits of preparedness

Hibernia had hot-site back-ups for its balance reporting, ACH systems and wire transfer systems “and we didn’t miss a beat from the first day of Katrina” after the bank switched over those systems, Frady reported. At the main branch in New Orleans, where the central money vault and operations center were located, there was enough food to last a month and enough fuel to last eight days. Treasury management, the bank’s most operationally-intensive activity, successfully relocated to Shreveport.

But 3,500 Hibernia employees—many of whom lost almost everything—evacuated in every direction—and it was the bank’s seemingly impossible task to find them, get them to check in, find places for them to work and, most important, get them the basic necessities they required for their own survival.

The cash in the central vault was insured, and could be replaced. But the Central Processing Unit data was indispensable, and the bank chose not to move it to the back-up operations center in Dallas. “That was our biggest mistake throughout the whole process,” admitted Frady. The retail lockbox was successfully transferred to Hibernia’s third-party software provider in New England, but the much bigger wholesale lockbox operation, unfortunately, was slated for back up to Houston. Hurricane Rita and the Postal Service’s incompetence kept that from happening. The Post Office simply lost the bank’s mail, and with it untold millions of customer dollars. “We had clients relying on us to process all these payments for them so they could fund their payrolls and payables,” recalled Frady. “But they weren’t getting them.”

Thirty-five bank associates and their families hunkered down at the New Orleans operations center to keep systems running. But the water system failed, so the CPU overheated. Downed cell phone towers and power lines knocked out all phone service, effectively cutting the bank off from its Incident Management team members. The bank was filling with water, while people in the streets fired guns at helicopters sent to evacuate the people inside. Ultimately, airdropped inflatable rafts took the bank residents to a medical center parking garage roof, where they spent an anxious night, clutching the bank data in 150-degree, mosquito-ridden heat, serenaded by gunfire, before the helicopters could return to ferry them to safety.

When the data tapes were finally jetted to Dallas the next day, the CPU back-up operation failed, making it two full days when the CPU was down. “In the scheme of things, it was a miracle it was only two days,” marveled Frady. “Other banks in the area had theirs down significantly longer than that.”

Sharing experience: suggested disaster plan steps

With phones down, Hibernia’s dispersed Incident Management Team members couldn’t function. They should have been sent to one place, where they could work together and communicate face to face during the disaster. Frady thinks other banks should do just that. “I can’t stress that enough, otherwise, how are you going to coordinate your efforts to get through this?” Now, all Hibernia IM Team members and key operations staff have phones with area codes outside the hurricane zone, while top managers have satellite phones.

Hibernia also moved its primary systems and servers out of New Orleans: the CPU is in a hardened facility in Little Rock and redundant servers are in Dallas.

There’s power in numbers when banks band together to consult with the Postal Service on disaster preparedness. Instead of taking the Post Office’s assurances that it has a DR plan, banks need to verify it. “Meet with your local Postal officials now and know exactly what their disaster recovery plans are,” Frady urged. “If you can’t quantify how their plan’s going to work, you’ll have some serious problems if they ever get hit.” Hibernia and other New Orleans banks have done this and know that mail will be moved out to Baton Rouge or Shreveport in future emergencies. “Because wholesale lockbox is fairly commoditized, we just got together and said, ‘we’ve got to put down our competitiveness and work together to get this thing better for southeastern Louisiana or else none of us are going to come back’,” Frady explained. “So that’s another recommendation.”

Not only was this practical, but ethical and humane.

Question: Did you have to limit cash withdrawals because of the possibility there would be a run on the bank from people who were desperate for money?

“No, we actually ended up helping other banks that wound up not having enough cash on hand,” answered Frady.” We actually let a lot of other banks come and set up shop in our branch to service their clients because their facilities were wiped out.”

Corporate values put people before profits

Before Hibernia could expect its employees to return to productive work, it had to account for the tremendous losses they had suffered—both material and emotional—and care for them as people in need first. It did that by guaranteeing employment for all employees for 90 days, paying them whether or not they showed up for work. The bank found them places to live, paid for their room and board, and gave them a weekly living stipend. Frady and his senior executives got down in the trenches, too, and joined in doing the return work. “They saw that and said ‘wow, everybody’s in this with us trying to make this thing work,’ “he said. “Our sales people were doing lockbox work, too. It’s amazing the morale boost that gave to our associates.”

A little humility helped, too. Frady conceded he was wrong to assign a low priority to CRM just because it was an internal function. “I didn’t think it through,” he admitted. “Our Customer Relationship Managers were angry because everything they knew was in that system and we were going to shut it down for a couple weeks.” Now, Hibernia has re-thought what systems should take priority if a disaster happens.

One more thing – Capital One Corporation was scheduled to purchase Hibernia just days after Katrina hit. The hurricane put that on hold until November, when the price went down by $500 million. But Hibernia began getting unpaid dividends from Capital One during the storm’s aftermath. Capital One staff manned Hibernia call centers and assisted the bank with technologies, like Web systems, that were giving Hibernia problems. Capital One even directed Hibernia officials to go the extra mile in helping their employees weather the after-effects of Katrina. “We wouldn’t have made it through this without them,” praised Frady.

About David Frady, Executive Vice President, Commercial Products and Services,
Capital One

David Frady’s responsibilities include Treasury management, commercial knowledge management, equipment finance, international banking, U.S. corporate correspondent banking, commercial loan services and merchant services.