Plant floor cyber security is among today’s most serious threats facing our individual manufacturing enterprises and our collective national security. Yet the potential of the Internet to radically and constructively transform our businesses is undeniable. The key will be to strike appropriate balances between security and productivity and between risk and revenue streams. The decisions to be made in this regard are C-suite decisions, to be overseen by boards of directors. As engineers and managers, if we are to be recognized in the C-suite of our company, it had best be as part of the solution and not as the cause of the problem. We cannot allow the hype over the industrial Internet of things (IIoT) to lure us into positions of vulnerability. We must be certain that plant floor cyber security has been adequately addressed, before we do anything that may expose our operations.
This two-part article is not intended as a how-to guide, but rather a why-should guide. This month we will dig deeply into why should a reader of Packaging World—whether a packager, an equipment supplier, or a material supplier—be actively engaged in cyber security discussions at the highest levels of their company? Why should educators, professional organizations, lobbyists and others who work with these industries be part of the discussion? Next month we will suggest some of the areas that should be considered, some strategies that could be employed, and some resources that can be drawn upon in the process of turning discussions into action.
The cyber landscape for CPGs
Consumer Packaged Goods manufacturers (CPGs) in particular, and hybrid manufacturers in general, are being largely overlooked in cyber security oversight. Major sections of the process industries, as part of our critical energy infrastructure, are required by law to address cyber security. Discrete manufacturers, especially those involved in manufacturing parts for small arms and major weapons systems, are being coached and prodded by the Departments of Defense and Homeland Security to close cyber security loop holes. The 19th annual ARC Industry Forum held this past February included a day of standing-room-only workshops on cyber security conducted by The Automation Federation and Department of Homeland Security (DHS), and the topic was on the agenda throughout the remaining 21/2 day conference with speakers from DHS, the FBI, NIST, Chevron, Shell, MIT, 3M, major utility providers, and a variety of technology providers. But scanning the attendee list shows that the conference was significantly under-represented by packagers, small process operators, and process and packaging machinery builders.
Hybrid manufacturers and their packaging and processing equipment suppliers are being left largely to their own devices to recognize and address the plant floor cyber threat. My research suggests that only the largest among them are actually taking adequate steps to address the problem. Best practices would include those whose boards have directed steps be taken to secure the shop floor, provided funding to do so, and set their internal audit departments about the task of testing and reporting on progress. To execute these directives, one CPG company has established an engineering function with the term “security” in its charter and name, and one has been working jointly with the nuclear industry to develop world-class protections and processes.
At PACK EXPO Las Vegas 2003, the OMAC Packaging Workgroup sponsored a paper on the topic of plant floor network security. That paper presented one leading CPG company’s plan for securing its plant control networks while allowing for remote access by employees and vendors. Twelve years later, most manufacturers have yet to achieve the levels of security described at that time. But given today’s threats, those levels are no longer adequate. The fundamental difference between then and now is that 10 years ago, we were still focusing on protecting our shop floors from the mistakes or oversights of our own well-meaning but perhaps uninformed employees and trusted vendors. We did not wish to risk the safety of our products, machines or workforce to some accidental intrusion across our networks that might cause our systems to temporarily go out of control. Fast forward now past the Stuxnet, Target, and Home Depot breaches; the state actors who have breached Sony and the White House; those who use cyber intrusion as a means of terrorism or war; and the 3 billion Internet users around the world, some of whom may simply choose to allay their boredom by trying to disrupt one of the world’s branded icons—and we find ourselves looking at “network security” in a whole new light.
There has been no more important time in history for CPGs to interact with power, water, wastewater, oil & gas, chemical, nuclear, and defense industries to share best practices; but unfortunately, CPGs seem to have leaned out their manufacturing technical staffs to the point that there are few left to do this, and the industry has largely stopped sponsoring the kind of multi-vendor and multi-sector events that historically provided developmental and informal benchmarking opportunities for engineers and managers. One exception may beThe Automation Conference(TAC), sponsored by the publishers of this magazine and growing in popularity among a variety of segments. I am convinced that the web does not adequately replace face-to-face opportunities to interact across disciplines, sectors, and levels of experience to help people understand that they don’t know what they don’t know.
In February of this year, President Obama signed an Executive Order entitled “Promoting Private Sector Cybersecurity Information Sharing.” Companies don’t like to share the fact that they are being targeted, and they certainly don’t want to talk about having been breached. They don’t want to share how they are protected, because knowing a target’s defenses can be a key to defeating them. And in a world where sharing the tiniest bit of information with the public can open you up for patent trolls to come knocking, such as occurred when CPGs were drawn into the well-known Solaia law suits a decade ago, maintaining total silence seems the least risky action. But is it? I can say that the president’s executive order did not make my research for this article any easier.
Why worry about shop floor systems?
A white paper published by the National Defense Industrial Association (NDIA) cites a number of reports and statistics about the persistence of cyber attacks on manufacturers, including this statement from McAfee’s2012 Threat Predictions: “Attackers tend to go after systems that can be successfully compromised, and industrial control systems have shown themselves to be a target-rich environment. The NDIA report cites three categories of concern for manufacturers; 1) Theft of confidential technical data 2) Alteration of data affecting process and product integrity and 3) Impairment or denial of process control, reducing manufacturing availability. These 3 make up the C-I-A concerns of plant floor cybersecurity.
In testimony before a Senate committee, a National Association of Manufacturers (NAM) spokesperson said, “As holders of the world’s leading intellectual property, including designs, patents, and trade secrets, manufacturers are consistently targeted by cyber thieves.” Cyber attacks have been documented to have blown up a pipeline and to have disabled a steel mill, preventing the blast furnace from being shut down. Over 500 breaches were recorded by Verizon against manufacturers in 2014, probably far fewer than actually occurred.
As our factories have transitioned from analog to digital, as our controllers have become self-documenting, as our process flows have become available at-line, and as our operator interfaces have become fully graphic, perhaps the most complete sets of product specifications and formulations actually reside within our shop floor control systems. While the information in the corporate product data management (PDM) system contains the master specifications, those specs and the real-life specifications about how the product is really made reside on the shop floor, in digital format, that can be transferred on to a USB drive, someone’s smart phone, or a message over the Internet. The same can be said for equipment suppliers’ intellectual property that resides in their machines, often IP beyond that which is actually being used for a particular application. Security experts have pointed out that there is no point in a criminal attacking the PDM system when the same information is available in much softer targets, where confidentiality may be breached. This scenario represents the C in the C-I-A concerns.
One individual I spoke with in preparing this article linked cyber security with the Food Safety Modernization Act (FSMA) and Food Safety Defense Plans. Processors must ensure the safety of their foods, requiring security of the facilities and supply chains, which must include cyber security. Someone bent on adulterating an ingredient or a finished product no longer need be physically present to do so. What about hacking the HVAC or refrigeration systems to cause spoilage over a weekend? Or perhaps that network needn’t be hacked at all, because an employee of the company monitoring your utilities needs a little extra income and lives in a culture that finds no issue with accepting a bribe. Since many factory floor cyber security plans ignore device networks, could someone easily gain access to yours to change the calibration of a sterilization loop or a canning process? One temperature transmitter may supply data to both the process control and the quality control system, and by recalibrating it for an hour every week, a factory might turn out a couple pallets of unsafe material every week without anyone ever taking notice. These examples point out loss of integrity, the I in C-I-A.
Machinery suppliers are increasingly offering to provide remote diagnostics for their machines. A production line may consist of adjoining machines supplied by competitors, all of which are connected to a single local area network. Each supplier has been given proper secured access to this LAN and has protected their individual machines with user names and passwords that have been entrusted to the field service technicians. Suppose that one of these techs leaves his company under unpleasant circumstances and joins a competitor. Knowing that a big project is coming up to be won by either his current or former employer, the service tech decides to take some revenge on his old employer and influence the outcome of the new project. Over a period of several weeks, using his new employer’s legitimate access to the customer’s LAN and his old employer’s username and passwords, he begins to slowly detune the servo drives on the machines, resulting in steadily decaying operational performance. This example points to loss of availability, the A in C-I-A.
These examples have been fabricated. Resources found in next month’s installment will point to real examples, often through recommended practices that have been developed based upon actual intrusions. These examples also point to the reality that entrusting cyber security to your IT department alone, or to the IT contractor that many small companies depend upon, may not be an adequate strategy. Do they even know what a temperature transmitter is?
Many sources have cited differences between plant floor and IT systems and how these differences define a different set of security circumstances. This is an important realization when developing a cyber security strategy. Plants have 10’s to 10’s of thousands of network-connected devices to be concerned with. The nature of these devices is that many of them work 24-7-365 for upwards of 25 years. Their operation has been painstakingly tested. Their software may have evolved over decades. They cannot be subject to weekly patching and bi-annual obsolescence. A security plan cannot possibly be put in place to make each of these devices individually secure. What must be secure is the information that flows to and from them, especially if the source or destination is outside of the physical boundaries of the plant.
Going even deeper, the NDIA report cites differences that applybetweenmanufacturing segments. For example, for discrete manufacturers, every new job (order) may bring new executable code into the manufacturing control system. This would rarely be the case for process industries where new orders, at worst, entail a recipe change; and this would be unusual for hybrid manufacturers. However, CPGs have their own new product programs, most of which will entail both the introduction of some new code and new network connections.
Is there a real threat to my company?
We all know in our gut that the threat is real, but it is easy to pass it off as someone else’s threat. It’s easy to reflect on Y2k being perceived as much fuss about nothing and thinking “here we go again.” But have we adequately assessed how real of a threat plant floor cyber is to us, no matter how small or how low tech or how off the grid we think that we may be? Have we thought about how large the potential consequences are should our systems be hacked, our products compromised, or our customers’ intellectual property be stolen from us? What is the potential that our customers’ or suppliers’ systems could be penetrated using our systems or our people as a gateway?
We have all heard that the Target breach came through an HVAC contractor. Two stories seem to float around: one that the attack came through a project management system connection and one that it came through an equipment monitoring connection. It really doesn’t matter which is true, because both vulnerabilities provide us with something more to ponder. If your HVAC systems or packaging machines are being monitored by a vendor, how far does their network extend? How secure is it? Where are the people that can access it? How are they vetted? Are they in a culture that would find nothing wrong with accepting a bribe to turn your HVAC or packaging information over to a competitor who might use it to calculate your production rates? How much are you actually saving by having that vendor monitor your equipment? The facilities department may have saved a few thousand dollars, but how big is the risk? And who gets to decide? And if you are the company doing the monitoring, ask yourself all the same questions. How big is your risk if someone on another continent, your employee or not, uses or hacks your network to steal information from your customer? Could one of your customers use your system to spy on or infiltrate another of your customers? Could you be accused of stealing proprietary information from a competitor if their machines are connected to machines that you are monitoring? Forget about the criminal aspect, what would be your civil liability in any of these situations?
The White House believes that the cyber threat to America is real enough that on April 1st, the president declared a cybersecurity national emergency. But evidence from an informal survey of machine builders that I conducted at Pack Expo East in Philadelphia convinced me that far too few have really thought about this problem. I asked a number of suppliers, who obviously had equipment capable of being on a factory floor network, if they had thought about cybersecurity, and if so, what have they done to address it. The far most common response was a “deer in the headlights” look.
Whether you work for a large or a small CPG company, a packaging or processing machinery supplier, a technology supplier, or some other manufacturing-related company, the risks of plant floor cyber security affect you BOTH as a provider and as a consumer of products and services. The security of your network is of the utmost concern to your customers as should the security of their network be of the utmost concern to you. This is an issue as you look both upward and downward in the capital equipment supply chain, the materials and products supply chain, and the services supply chain. And by network, we don’t just mean the enterprise networks, but also the plant networks that connect to the enterprise, the process control and automation networks within the plants, and the device networks within the automation systems. Cyber security has been called a multi-dimensional problem requiring customized solutions for conventional IT, automation & control systems, intelligent network-connected devices (sensors, cameras, point of sale terminals), mobile devices, and the cloud.
Preparing for exponential growth
The concerns about plant floor cyber security are juxtaposed against the predictions for the Industrial Internet of Things (IIoT). Pundits tell us that if we do not embrace IIoT as manufacturers, we will be putting ourselves out of business. Others tell us that there can be no implementation of IIoT until security is established. Peter Holicki of DOW Chemical made a clear statement at the ARC Industry Forum reporting that DOW believes that it owes it to the community to guarantee that the systems that ensure plant safety are completely disconnected from the Internet. But keeping systems separated may be easier said than done. Today computer technology is so inexpensive that it creeps into our plants sometimes with little if any conscious planning; so it takes conscious planning to keep it out. One supplier told me of a case where a European plant was shut down for a day because someone hacked into the WiFi link on a conference room projector that was also connected to the plant’s Ethernet. Who would have planned for that threat?
In 1995, fewer than 1% of the world’s population was connected to the Internet. It took 10 years for the first billion users, 5 years for the second billion, and 4 years to bring us to where we are today with 3 billion users, almost half of which are in Asia. Projections are to hit 5 billion in the next 10 years (growth is slowing). Cisco claims that there are currently over 15 billion things connected to the Internet, with a projection of 50 billion by 2020. Some say that 40 billion of those connections will be wireless, with much of the growth coming from sensors, many of which will be in our factories.
TheSymantic Internet Security Threat Report 2014included this headline within the executive summary: “Attackers are turning to the Internet of Things.” It went on to say, “Today the burden of preventing attacks against IoT devices falls on the user .... Manufacturers [IoT device makers] are not prioritizing security...”
Last year, in conjunction with our sister publicationAutomation World, ARC conducted a web survey to gauge industry perspective on the adoption of the Industrial Internet of Things (IIoT). This survey sampled a wider manufacturing audience than CPGs and packagers and resulted in over 200 responses. The survey targeted those who were current or potential users of IIoT solutions or providers of such solutions, a rather well-informed group. When asked about inhibitors to adoption of IIoT, the number one concern of respondents was security (Figure 1).
More connected users (2 billion more), more connected devices (35 billion more), and a lack of prioritization of security would seem to be the ingredients for a perfect storm, substantially increasing the means, motive, and opportunity for those who may wish to attack our plants. While my colleagues and I often quip that technological advances in manufacturing should be measured in “dog-years,” a dog-year mentality will not prepare manufacturers for this explosive growth! We had best plan accordingly.
Next month we will suggest some of the areas that should be considered in that plan, some strategies that could be employed, and some resources that can be drawn upon in the process of turning discussions into meaningful action.
To see a spin + zoom 360° photo, click here.
