Cyber Harmony

Download TCIL Technical Note

Introduction

In 2015, Russian cyber operators launched a coordinated cyberattack on civilian energy infrastructure across Ukraine. The attack was unprecedented in scale. Over 100 cities and towns, comprising hundreds of thousands of residents, lost power for one to six hours.¹ The hackers first gained access through the information networks of several regional power companies by using spear-phishing emails to target individuals with administrative credentials. Then the intruders used remote-access tools to seize control of power distribution systems. Next, they used the communications lines and operator workstations they now controlled to send commands to field equipment, all the while locking the actual operators out of the system.²

At the request of the Ukrainian government, U.S. investigators helped pinpoint the cause of the disruption and identify vulnerabilities Kyiv could fix to prevent further attacks. The investigation concluded that a series of cascading problems caused the breach. Poor security hygiene in the power companies’ business networks made them easy prey for the hackers’ phishing campaign. Once inside, dilapidated operational infrastructure and a lack of monitoring for anomalous activity let the attackers run rampant through the system, destroying what they wished.

While the lights may have gone out across Ukraine, the attack (and the follow-on forensics) rang alarm bells throughout the U.S. government and the American energy sector. U.S. electricity providers were vulnerable to the same sort of devastating attack that struck their Ukrainian counterparts.

After reflecting on its own susceptibility, one U.S. energy company (hereafter referred to as “the Company”) concluded that although it was meeting all its industry’s cybersecurity requirements, it was still vulnerable. The Company was not centrally monitoring the operation and security of its systems. This rendered it unable to detect signs that malicious actors were poking around until after the launch of disruptive or destructive malware. Early detection could have kept the lights on in Ukraine. With that in mind, the Company determined it would need to do better if it hoped to thwart an attack before systems started shutting down.

The following report outlines how the Company set up a new security operations center (SOC), with supporting technologies and processes across all its business lines to better detect indicators of risk and emerging problems before hackers could launch devastating attacks. In particular, because attackers traverse business networks to degrade the operations of the physical equipment that makes up critical infrastructure, the Company determined it needed a unified view of all of its systems.

The implementation process was not easy. In fact, it was hard, costly, and time-consuming. But the Company knew it had to prepare for threats from bolder, more skilled attackers.

While the Company’s specific technical solutions may not be applicable to other organizations, the broader lessons are. Organizations of all stripes can benefit from a centralized security monitoring system that differentiates between normal and abnormal operation and alerts them to potential threats. Other organizations should also take note of how the Company’s cybersecurity team gained buy-in from decision-makers, engineers, operators, human resources, and many others. In other words, the Company did not just have the right hardware and software. It also devoted attention to what this report will call “peopleware.”

Starting the Process

The equipment, technology, and processes that make up information technology (IT) and operational technology (OT) are very different. Separate tools and expertise are necessary to monitor the two. Thus, organizations usually have distinct IT and OT SOCs managed by separate teams.

An IT SOC protects an organization’s information systems, such as computer networks, servers, and applications. It typically uses tools — like firewalls, intrusion detection systems, and antivirus software — to protect the systems detect anomalous activity and respond to security incidents. In addition, an IT SOC usually works on logging, correlation, incident detection, response, and continuous monitoring to identify, mitigate, and contain a security incident. IT SOCs typically handle incident response, vulnerability management, and compliance with IT-focused regulations.

OT SOCs, meanwhile, monitor and protect industrial processes, such as industrial control systems, supervisory control and data acquisition systems, programmable logic controllers, and distributed control systems. OT equipment is often highly interconnected, with specialized devices that communicate with each other using unique protocols. Like the IT SOC, the OT SOC handles incident response, vulnerability management, and compliance with regulations for operational technology. In addition, OT SOCs also monitor the performance and maintain the availability of the OT assets.

IT and OT systems typically connect at a demilitarized zone (DMZ) that allows limited communication between the two. But as the underlying technology of both OT and IT grows more complex, the DMZ has become an entirely new cyber threat vector, where an attack on one side could cross into the other. Indeed, this happened in Ukraine in 2015: The hackers compromised the business IT network and then traversed IT systems to cause operational issues within the OT environment. Given this reality, the Company realized that keeping the SOCs separate was no longer tenable.

Instead of merely meeting cybersecurity standards, the Company determined it needed to completely restructure its operations. The program lead for this new effort understood that the Company needed changes to its hardware, software, and processes. But he also learned that peopleware deserves equal attention. For this new type of cybersecurity system to function well, the program lead had to work closely with and address concerns from Company decision-makers, engineers, operators, and administrators as well as the personnel in finance, human resources, and procurement. Peopleware can be just as important as hardware and software.

The team began by developing four high-level objectives that the Company’s current technologies and processes were failing to achieve:

• “Know Good” – Enable automated, centralized collection, and monitoring of OT networks to provide visibility and establish a baseline for “normal” communication. At the beginning of the process, the Company had some automated IT monitoring but lacked automated OT monitoring, and neither was centralized.
• “See Bad” – Implement uniform monitoring of networks across all aspects of the Company to detect anomalies and possible intrusions in a timely manner.
• “Take Action” – Ensure coordinated action by the cybersecurity team, operators, and enterprise partners in response to alerts of anomalous activity.
• “Do No Harm” – Minimize operational impact across the organization while enabling a safe and injury-free workspace.

To achieve these objectives, the team determined it needed a combined IT-OT SOC with the necessary supporting technology and processes embedded throughout the Company to provide a single overarching, correlated view. Execution of this vision required a three-phase approach. First, the team would secure senior management support and buy-in. Second, it would establish the infrastructure capable of providing a centralized view of the monitored IT and OT. Finally, the team would hone and advance the operations.