The States of Guernsey in the UK has found that a series of IT outages were caused by the failure of backup systems and reliance on out of date kit.
The British Island's government experienced four outages between November and January, rendering the States’ website offline and disrupting internal systems. Reportedly there was no data lost as a result of the incidents, but a number of schools, Guernsey Airport’s boarding card system, and benefit payments were all impacted.
PricewaterhouseCoopers (PwC) conducted a review of the outages, concluding that the data center was significantly outdated, and the States needed to make major changes for better future resiliency.
Some of the issues identified included that the ownership of the States’ data center infrastructure was unclear, as well as who was responsible for maintenance. As a result, when one of the two air conditioning systems broke down in June, it remained unfixed in November, when the second also failed leaving the IT equipment without cooling.
Equipment is generally old, and the review found delays in moving older systems to new data centers. Additionally, the maintenance contracts for the UPS and generator had expired over a year ago.
The States also experienced two near-miss incidents with the data centers housed at the hospital and Edward T Wheadon House. After which, apparently, no action was taken to reinforce the infrastructure.
Deputy Peter Ferbrache said: “There are failings we must acknowledge and actions that were missed that could have prevented these outages."
States of Guernsey had previously signed a 10-year £200 million ($254.4m) contract with IT provider Agilisys in 2019, but apparently, this did not include the maintenance responsibility.
While Guernsey is a part of the UK, it is operated as a crown dependency, meaning it has its own autonomous government. Thus, the island’s data centers cannot be located on the mainland. As a result, the government funded the 2019 deal with Agilisys which saw two facilities being developed on the island and using connectivity providers JT Global. Dell Technologies provided VxRail servers that incorporated AMD Epyc processors.
The States’ software partner was Microsoft Azure, a relationship handed by Agilisys.
Andrew Mindenhall, Agilisys chief executive said: "We know outages like these are not what islanders expect and we're committed to working with the States on delivering its plan to improve resilience and mitigate the risks."
PwC has recommended Agilisys and States of Guernsey review its contract, ensuring that all oversights are dealt with. In the meantime, the local government will have to bear the brunt of an unspecified bill to replace that equipment that has broken down.
According to Ferbrache, if there is a fault found on the side of Agilisys, the government will “look to them to recompense us in some way.”