The report notes that Guy’s and St Thomas’ has 371 legacy IT systems that support patient records, patient administration, clinical services and infrastructure. “These systems run on technical infrastructure housed in two data centres; the Guy’s data centre situated in Borough Wing which was constructed in 2007 and the St Thomas’ data centre located in a modular building which was constructed in 2012. The two data centres were designed to act as back-ups for each other in the event that one failed. The IT infrastructure was updated in 2015/16 as part of the Strategic Data Centre programme. Separate data centres support the IT systems at Royal Brompton and Harefield hospitals.”
As part of the review it highlighted a series of reflections on the response: “The trust initially under-estimated the probable duration of the IT incident, and this was reflected in the trust’s communications during the first few days, which was felt by many staff and stakeholders to under-play the severity of the situation.
“Whilst the operational response to move to a ‘paper hospital’ was managed with speed and determination, there was widespread frustration with how long it took to recover core clinical IT systems: several weeks rather than hours or days. This was not a reflection on the effort or professionalism of the trust’s IT team, but demonstrated the limited number of individuals who had a detailed understanding of the trust’s legacy IT systems which were too numerous, complex and inter-linked to be recovered quickly.”
The report adds: “This review has found no single, egregious failure in the root cause analysis which has been carried out, but rather a combination of the following factors led to the catastrophic failure of the IT systems: sub-optimal cooling systems; ageing technological infrastructure; overly complex and distributed roles and responsibilities for managing elements of the data centre.”
As part of the review it notes the costs incurred of “£1.4m out-of-plan spending on technology services to respond to the incident”. This included a cloud-hosted environment to provide resilience for data backups, and a third-party specialist recovery service to image and extract data from the corrupted disks damaged during the data centre failure.
The trust said it “must never again allow itself to be in a situation where the recovery of its core IT systems, whether as a result of infrastructure failure, cyber-attack or another cause, takes so long to complete”. The critical site incident was stood down on 21 September.
As a result, the report details the need for a “comprehensive strategic plan, backed by appropriate investment, to ensure future computer processing and data storage requirements are robust, able to meet growing demand and also resilient to foreseeable risks,” and for these plans to include periodic and thorough testing of systems recovery.