Computer crash leads to near disaster

By: Brian Coburn

Posted: 9/22/08

Washtenaw Community College averted disaster but still has a large and expensive mess to clean up when its computer system temporarily crashed perilously close to the kickoff of the fall semester.

The biggest problem was the permanent loss of all data in the Windows program SharePoint, which carried documents that will be submitted in a report to the North Central Association Higher Learning Commission - (NCA HLC) - a division of AdvancED that enables WCC to be an accredited institution - when the group visits the college in October 2009.

When pressed, several administrators were evasive about what kind of financial losses the school would suffer as a result of the lost data that took years to compile and rebuilding process which could take months of overtime.

"I have not heard anyone talk about this being an extravagant cost to repair," said Linda Blakey, associate vice president of Student Services. "The bulk of the cost is going to be for man hours."

Those man hours will be logged when faculty and administrators on committees to collect data for the report will have to make up an entire year's work in a month.

"If (the NCA HLC) were coming in October of this year, it would have been a disaster," said Bill Abernethy, dean of human/social sciences and the person responsible for coordinating the report. "They only pay a visit once every ten years and we probably would have had to postpone it."

To recover from the total wipeout of data, Abernethy has his committees and several subcommittees of faculty and administrators working overtime. Most of their work entails taking hard copy documents describing college procedures to support criteria of the NCA HCL (which is required if WCC is to remain an accredited school) and converting them in digital form.

"No one wants to do the same work all over again," Abernethy said, "but I'm quite proud of how hard everyone is working on this."
The loss of the Helpdesk is less serious, as it was used primarily to deal with complaints and request tickets by faculty having trouble with their computer equipment. The last two weeks of tickets were lost and the archive has been wiped out.

"Not to minimize what is happening with the (NCA CASI) report, but there are no hard files here to get recover the tickets," said Kim Tom, Manager of User Support Services.

"If you put in a ticket and we never got to it, you're going to have to put it in again."

According to Amin Ladha, WCC's chief information officer, each disk in the SAN has a likelihood of failing once every 200 days or so. When two disks in the same volume group failed at the same time, it caused a crash.

"We're always looking for ways we can do things better, but if you were to calculate the odds that two out of the five in the volume group would go out at the same time, you would have to say this was just really bad luck," Ladha said.

The failed disks were replaced on Sept. 1 and then a team of information technology employees worked day and night to make sure Blackboard, e-mail and other systems would be fully restored from tape backups.

"It was almost the worst possible time for everything because you want everything up and ready to go by the time class starts," Ladha said. "But (my staff) worked so hard to get it all back so soon. These guys are the best."