|
[See the timeline for an hour-by-hour
account.]
How would your operation be affected if a major virus disabled your
network? That's what happened recently at one of the largest banks in the
Midwest, National City Corporation (NCC).
Here's how NCC worked with IBM to successfully stop the virus and keep
the bank's operations running, from a recent interview with NCC Executive
Vice President Jon Gorney by the editors of Application/ Enabling World.
National City Corporation is a $50 billion commercial bank holding company
headquartered in Cleveland, with about 900 retail offices in Ohio, Kentucky,
Indiana, and Pennsylvania. NCC's computing network of about 300 servers
and 8,000 workstations handles both mission-critical and supporting applications.
Like many organizations, NCC had taken prudent steps to prevent viruses.
Virus detection software was in place on workstations throughout the network.
All new equipment was checked for viruses by the PC dealer before delivery,
then rechecked by NCC before installation. Employees were prohibited from
bringing in software from home.
Despite these precautions, a virus infected NCC's network last spring.
With ingenuity and the help of IBM's Virus Emergency Response Team and
AntiVirus software, NCC was able to solve the crises with minimal disruption
to bank operations and absolutely no impact on customers.
|
|
The first signs of a problem
|
| Late one Tuesday afternoon, users trying to log on to the NCC network
were suddenly being denied access. NCC's IS staff soon realized this was
more than a set of isolated incidents. It was the start of a serious virus
infection and it was spreading through the network at an alarming rate.
Jon Gorney, NCC Executive Vice President, was at a hotel in Pittsburgh
when he was alerted to the problem. "I got a call at 2:00 a.m.,"
he recalls. "A virus of some kind was totally locking out the workstations
when you tried to sign on. It was clear that when people came in the next
morning and signed on, it would spread even more rapidly."
Gorney quickly assembled other NCC staff members traveling with him
and secured a hotel conference room for planning and communications. Meanwhile,
NCC executives in Cleveland called their IBM Client Executive Don Parker
at 2:00 a.m. to ask for help.
|
|
IBM's local team and virus
experts respond at once
|
| Parker went into action immediately, marshaling local IBM resources
and IBM's expert virus-fighters in New York. Before daybreak, the local
team had captured a sample of the virus and transmitted it to Ed Hahn,
manager of the IBM Virus Emergency Response Project. Hahn and his staff
went to work to identify the virus and find an antidote. |
| "By 5 or 6 in the morning, we were able to have our technicians
talking to IBM specialists and transmitting software with the virus embedded
so that some analysis could begin," Gorney explains. "By early
morning, IBM was able to come back and say, 'Yes, it's a virus and we've
identified it as MAJO-1644.' That was the good news. The bad news was that
a disinfectant did not exist." |
|
NCC executes its communications
plans
|
| Meanwhile, National City's staff was rushing to keep the virus from
spreading and trying to develop workarounds to accomplish the bank's day-to-day
functions without affecting customers. Effective communications was critical.
"As part of our disaster and recovery contingency planning process,
[we had] a very good understanding of the call matrix -- who to call and
how to contact them," Gorney comments. "That worked extremely
well. There were signs posted: 'Do not sign on to your PC.' There were
broadcasts over the internal PA system. Our support teams at the local
level were all aware of it so we had assigned a person per floor or department
to be the spokesman. It was a combination of talking to individuals who
would then communicate to our contingency planning teams [and] business
units."
|
|
The real challenge: Deploying
the disinfectant
|
| By 11:00 a.m., IBM technicians had developed an antidote and transmitted
it to NCC. While IBM was working on a solution, the supplier of NCC's installed
anti-virus software was also trying to find an antidote. NCC evaluated
both options and elected to try the installed vendor's solution.
But problems with the other disinfectant emerged almost immediately.
NCC decided to deploy the IBM solution and asked Ed Hahn to come to Cleveland
to help. The NCC corporate jet, which had already retrieved Gorney from
Pittsburgh, was dispatched to New York. By 6:30 p.m., Hahn was on his way.
|
| "Now the good new was, we had a cure," Gorney remembers.
"As it turns out, the disinfectant was the easy piece. Orchestrating
how you go about getting this rolled out...became the challenge.
"It took some senior people within my organization and within IBM
to get in one room and say, 'What do we need to do? We need a strategy
to roll this out.' It was clear there was not going to be a quick fix."
The team moved several high-powered workstations to a conference room
to create a data center as the central point for distributing the antidote
and cleansing servers. But inventory and asset management issues hampered
the cleansing process.
"To be perfectly blunt, we did not have a good inventory and asset
management system in place," Gorney states. "You couldn't easily
[say], 'Here's where every server is physically located.' Nor could you
say, 'Here's the exact configuration of every workstation.'...The process
was very complex and very difficult and required rotating teams in and
out about every five or six hours for 48 hours."
Ed Hahn also recalls the complexity of the deployment. "As we came
into contact with servers they had cleaned up with the other solution,
the virus was gone but the programs wouldn't execute and we couldn't get
the servers up," he explains. "The easiest way was to restore
the infected servers and let IBM AntiVirus clean those infected programs."
AntiVirus detects more than 7,500 viruses. During the deployment, other
viruses were discovered and resolved.
|
|
NCC staff works to keep
the bank running smoothly
|
| Meanwhile, another NCC team was ensuring the bank operations could
continue. "While all that was going on, we had another team dealing
with business disruption," Gorney recalls. "[They] went application
by application, city by city, business unit by business unit, and we had
contact points in all our various cities to try to define, 'Okay, what
can't we run? What kind of workarounds do we have?
"That team did an outstanding job. They were able to...develop
a workaround for every single application. There was no disruption to customers
that we're aware of. It didn't have any direct customer impact, through
a lot of hard work."
|
|
The final test: Bringing
the network up
|
| When all servers were cleansed, it was time to allow users back on
the network. The plan was to have IBM AntiVirus cleanse each workstation
during the logon process.
"Then the most difficult task was getting people comfortable with
a different sign-on process," Gorney recalls. "A good part of
Thursday night was [spent deciding] 'How do we communicate with 8,000 users
about what they're going to experience when they sign on?"
This was complicated by multiple PC configurations. "We pretty
much know how it would work for a 486...but we couldn't go out and hand
[users] a set procedure," explains Gorney. "We had to go back
to our whole call tree again and have people back out on each floor helping
with the sign-on process."
By 10 a.m., the network was up and running and the expected flood of
calls to the help desk had not materialized. NCC was back in business.
|
|
How the virus got in
|
| National City later determined that the virus came in on a new PC installed
at its distribution center the day before the outbreak began. Though the
distribution center went through multiple steps to detect viruses in the
new equipment, the virus got in anyway.
"It was a real wake-up call for us," Gorney acknowledges.
"We had a procedure that made sense, we were following that process,
yet it still got through. We've challenged our team: 'If 1645 shows up
tomorrow, given everything we've just done, what would our experience be?'
My belief is that based on what we've put in, we may not be able to resolve
it right away, but we'd recognize it, we'd isolate it, and we'd shut it
down."
|
|
How the crisis strengthened
NCC
|
| NCC had done many disaster drills in the past to prepare for unexpected
crises, but ironically, a virus infection had never been tested. "Every
year, we do a simulation of a disaster in our major cities. Generally,
it's a physical disaster, like a plane crashing into the data center,"
Gorney notes. "The interesting thing is that as much as we had tested
physical disaster, the likelihood of that is probably remote compared to
this."
Gorney believes many benefits resulted from the virus outbreak and calls
it "an absolute blessing. I'd never want to do it again and I wouldn't
wish it on anybody, but the reality is that when it was done, we're smarter
and we understand it better... We're more secure than we were.
|
| "There was a point when I wasn't sure we could get out of it,"
he acknowledges. "But when I look back, it was the best experience
we could have had.
"We learned a hell of a lot. It challenged us to have tighter standards.
The more we had out there, the more complex it became. It highlights the
need to have greater central management of this asset and a true asset
management system.
"We had been far too accommodating, giving people far too many
choices versus saying, 'If your job is a financial analysis, you need this
kind of PC.' We are migrating to a much more rigid and tighter standard
to try to eliminate the variability."
|
| Gorney lauds IBM for its rapid response and customer service. "Their
performance was outstanding," he states. "There were teams around
the clock made up of National City people and IBM people and they would
change from one shift to another and I honestly couldn't tell you, when
I went into the room, who was who. That may be the best way to describe
it."
Many colleagues in the banking industry who heard about the incident
have called Gorney for advice. He suggests organizations plan a human communications
strategy and evaluate the strength of their anti-virus software.
"You've really got to have a very strong communications plan already
built," he observes. "I've also suggested they really need to
take a look at what their anti-virus software can do. I was very open.
I said I don't know how we would have gotten out of this situation if IBM
had not been there."
|
