The Cyber Attack – From the POV of the CEO

I am writing this article to recount the events of the last few days in the hope that it will provide insights for our associates, members of our community, and other interested parties.  My hope is that this retelling of the events will help shed light into the extraordinary efforts our organization mounted in response to a potentially disastrous event.  Please know, this is not a moment-by-moment diary, nor is it intended to be a technical manual, it is my own summary of what happened.

At approximately 9:30 PM on Thursday, January 11, 2018, an attack on the information systems of Hancock Health was initiated by a sophisticated criminal group we now believe was located in Eastern Europe.  The group obtained the login credentials of a vendor that provides hardware for one of the critical information systems used by the hospital.  Utilizing these compromised account credentials, the hackers targeted a server located in the emergency IT backup facility utilized by the hospital and located many miles away from the Hancock Health main campus and made use of the electronic connection between the backup site and the server farm on the hospital main campus to deliver SamSam malware using remote execution techniques.  This malware was targeted to encrypt data files associated with the most critical information systems of the hospital. Fortunately, patient life support systems were not directly affected.

A few moments after initiation of the attack, IT staff began to notice negative changes in system performance and messages began to appear on PC screens in the hospital indicating that the system was being encrypted using SamSam ransomware.  These messages noted that decryption keys could be purchased with Bitcoin payable via the TOR web browser on the Dark Web.  The message included step by step instructions required to obtain the decryption keys and noted that lack of payment within seven days would result in permanent encryption of the data.  As an aside, most ransomware attacks are initiated by phishing emails sent to users.  These emails lure the reader to click a link, which then downloads the ransomware.  In some ways, that makes episodes like this somewhat random.  That said, the attack on Hancock Health was not random, it was a pre-planned event that used the hacked login ID and password of an outside vendor to gain entrance into the system.  The fact that this was a premeditated attack specifically targeted on a health care facility makes the attack indefensible in my estimation.

With the preliminary information available, the IT staff reacted quickly and began implementing disaster response procedures including the immediate shut down of all network and desktop systems in the organization.  The latter step was not straightforward as the more than 1,200 employees of Hancock Health utilize an equivalent number of computers.  Given that the attack came in the evening hours when many employees had left for the day, facilities management staff were called on to visit the location of every PC and physically turn them off.  In addition, signs were posted across the campus providing notification of what had occurred and noting that computers should remain powered down.

As part of the disaster response, Hancock Health executive leadership was immediately contacted and the incident command center was mobilized.  After ensuring downtime procedures were fully implemented and patient care processes were stabilized, the work of initializing event-response began.  During the early morning hours on Friday, organization leadership established contact with the hospital’s healthcare attorneys at Hall-Render in Indianapolis, and cybersecurity specialists from Pondurance, Inc., were quickly engaged.  At this point the FBI became embedded in the process as well.  This team promptly began the arduous task of identifying the cause and scope of the cyber-attack and determining potential responses.  When the source of the infection was identified on a server at the backup site, and it was learned that the electronic tunnel between the backup site and the hospital had been compromised, it became clear that there were no easy-to-implement means of purging the encrypted data and replacing it with clean data from backup systems. With this in mind, the decision was made to purchase the decryption keys.

Of interest, at the time the decision to pay the ransom was made it was believed that the backup files had not been directly affected. Several days later it was learned that, though the electronic medical record backup files had not been touched, the core components of the backup files from many other systems had been purposefully and permanently corrupted by the hackers. Thus, backup of the rest of the network systems would never have been a possibility and acquisition of the decryption keys was unavoidable.

At this point, two parallel paths of work began.  The first path was associated with turning network servers back on and employing software and hardware interventions on the network to encapsulate the encrypted files and protect the uninfected files from damage.  This group also began the work of identifying if patient information had been stolen from the electronic medical record system.  The second path was associated with obtaining the decryption keys needed to restore the system, which would require the purchase of Bitcoin, and notifying the hospital’s cybersecurity insurance company of the situation as well.

Because Pondurance is a firm specializing in cybersecurity threat mitigation and disaster response, they were able to provide Hancock Health IT staff with a supplemental state-of-the-art artificial intelligence software package that is deployed to network servers and desktop PCs in situations like this.  In addition, they provided a hardware appliance to connect with all network servers to monitor network traffic for additional threat detection and to further encapsulate encrypted files.  These were in addition to the already-robust software and hardware countermeasures in place on the network.  The team also began the process of looking through network server logs to answer the question of whether the hackers had merely injected encryption software, or if they had also stolen patient information files.  The latter project required the analysis of many thousands of files.

The work of this team continued through the day and night on Friday and into Saturday morning with software rollout completed mid-day on Saturday and protection-appliance installation completed by midafternoon that day.  For these efforts to be completely successful, each PC was required to be physically turned on, again no small task given that it was now a weekend.  With software and hardware mitigation techniques fully implemented, the team took the additional step of resetting all user passwords and implementing additional password requirements.

While these accomplishments were critical, of primary importance was the work associated with understanding the impact of the criminal intrusion on the security of patient information.  While early indications from a review of history files on Friday showed that this information appeared not to have been compromised, by Saturday the team could state that, based on the evidence, no patient information had been diverted from the Hancock Health network, it was also later identified that patient information had not been accessed by the hackers inside the network – major findings indeed!

The second team was also hard at work on Friday identifying potential sources for Bitcoin and learning about the techniques required to complete the exchange of the Bitcoin for the decryption files.  The recent volatility and huge trading volume for Bitcoin were complicating factors in this endeavor, but a source was identified and financial team members from Hancock arranged for money to be wired late Friday afternoon.  The acquisition of four Bitcoin (the price of the ransom) was completed mid-evening on Friday and the transaction with the hackers was initiated late Friday night.  In the wee hours of Saturday morning, a response was received and directions to download the decryption keys were uploaded from the other side of the world.  This information was quickly taken from the Dark Web and placed in a secure online vault for validation and threat checking, with all files approved by Saturday morning.

Of interest, at 2:00 AM on Saturday morning, a rotation of team members was initiated to ensure fresh minds and bodies would be in place for the duration of the event.  This was especially important considering the intensive efforts that had been going non-stop for nearly 30 hours.

By mid-day Saturday, network servers were back on line and in a protected state, WiFi was enabled, and file decryption was well underway. With more than 1,400 individual decryption files to handle, this portion of the process required the most time and was not completed until early evening on Sunday.  At this point the many signs that had been posted on the main campus early Friday morning were taken down and a “ceremonial trashing” was captured in a photo in the IT office.

The focus of effort on Sunday was the arduous task of validating that files were being safely recovered, encrypted files were being deleted, and the various discrete information systems were being brought back on line.  During the day, communication systems were restored, network file servers were brought back on line, and most importantly, the electronic medical record system became functional by Sunday evening.  This last element was exceptionally important because downtime procedures in a health care environment are associated with reverting to paper documentation.  While the hospital had been paper-based for the first 60 years of its existence, our team members have become very attuned to electronic record keeping over the last decade, thus it was a relief when the electronic system came back on line.

At this point, the reams of paper documents generated during the 70 hours that the system was offline were scanned into the system and discrete data elements were entered into the system by a small group of clinical staff.  By Monday morning, critical information systems were back online and the work of the disaster recovery team was beginning to shift to monitoring the network and ensuring remaining systems work was completed, tasks that will be ongoing for some time.

Amazingly, all of this happened in less than four days.  During that short time, babies were born at the hospital, surgeries were completed at the hospital, patients were treated in the emergency room and many were admitted to the hospital.  X-rays were taken, CT and MRI scans recorded, and laboratory tests were accomplished at the hospital.  Patients visited our physician clinics and wellness centers.  Food was served in our cafeteria and rooms were cleaned.  In short, life went on.  We even had a winter storm, and still, life went on.

Hancock was open for business during this time because our extraordinary people made sure we were open.  They came in on days off, they worked through the nights, they consumed many boxes of pizza and gallons of caffeinated beverages, all to make sure that what we do – caring for our patients – never missed a beat.

At Hancock Health, we have an unofficial motto – “What a blessing it is to love people for a living.”  Even a cyber-attack from the other side of the world could not keep us from living this out!