Manufacturing IT: 4 ways to better manage, secure, and innovate in your environment
Modern factory design emphasizes security and connectivity. But a number of factors play into the success of an evolving manufacturing environment.
IT in manufacturing is unlike in any other industry. Here are four of the most important factors for success in this space.
Ninety-five percent of the endpoints in a manufacturing environment are non-user interactive devices that either feed or ask for data on temperatures, the speed of the line, and so on.
When vendors come in to fix, update, or otherwise maintain those devices, you need to strike a balance between security, the ability of the plant to function, and the ability of vendors to do their job without overly relying on you.
Vendors can be a security risk. Probably the most famous example is the HVAC vendor that caused Target’s data breach in 2013. More recently, Tesla accused an employee of changing parts of the company’s manufacturing operating system code to disrupt production. Obviously that’s the work of an employee, not a vendor, but the risk is the same. Everyone should only have access to what they absolutely need to do their job.
Thought out and deliberate segmentation, based on an understanding of the role and function of each vendor, provides an easy way to limit access. Devices on the packaging line might have their own virtual area, for example, and have nothing to do with the shipping area.
Most importantly, you should implement these rules in a way that doesn’t drastically increase vendors’ reliance on you to do their job. You also don’t want to make a vendor’s life so difficult that they find a way around the rules.
You can build in flexibility too, with general access to a particular segment for a particular user. If the same vendor sends Bob to your factory on Tuesday, then Joe on Thursday, you can give that vendor a single login if they’re performing the same function.
Connectivity in a manufacturing environment is different than most IT folks are used to. Instead of large files and a thirst for bandwidth, there’s a huge number of very small files and latency is the bigger concern.
There are new connections every few milliseconds, multiplied by hundreds or thousands of devices. Many of these connections are devices communicating to each other about what they’re doing. Some talk back and forth to ensure they stay in sync, using precision time where picoseconds make a huge difference in the pace the manufacturing can maintain.
Part of ensuring connectivity is understanding the manufacturing devices deployed, how they interact in terms of their software and hardware, and how they’ll work with your design and devices.
Manufacturing devices use a suite of protocols unique in the networking world. You may find yourself having to design support for three, four, or even more specialized protocols if the OT team requires devices that use them. You may be able to ignore the eccentricities of each one for a while but eventually they will come back to bite you.
One of the major challenges in an industry rushing into the IoT space is that manufacturing hardware vendors are just now becoming familiar with concepts that have been commonplace in the IT world for ages. As a result, you might find bugs, missing features, or sometimes completely absent features that you normally take for granted.
For example, a manufacturer was trying to utilize power over ethernet (PoE) on its device but messed up the negotiation steps so it would never actually ask for what it needed at max draw. That resulted in what seemed like random failures of the device. However, after a lot of troubleshooting and setting PoE allowances manually, it worked fine.
Another time, I had a device that simply would not maintain a connection when communication required routing (multiple VLANS). As it turns out, the manufacturer didn’t code its IP implementation correctly and particular types of packets would not use the gateway when other traffic would.
Simple things like these are not really issues in IT and highlight just how far behind the curve the OT can be. Combined with the handful of specialized protocols, it really pays off to take the time to know what’s there and design to support it correctly so you aren’t left pointing fingers back and forth with vendors to find the mistakes.
3. Design strategies
Invest at the core level and make sure it’s flexible enough to adapt to anything at the access layer. Look at how the main connectivity is set up from a core infrastructure perspective, down to the actual device, line, or energy substation. That’s where it gets tricky. The key is having a core infrastructure that will allow you to change out the access layer with a reasonable amount of connectivity without altering the core level.
A common philosophy, and not just in manufacturing, is “if it ain’t broke, don’t fix it.” That is the absolute worst philosophy you can have. Most importantly, it purposefully ignores security by delaying updates.
Design in an easily scalable, modular manner so that your network can adapt to support new technologies and business initiatives without compromising availability, performance, or security, or triggering costly redesigns before your equipment reaches end of life.
Invest in key areas that will support a lot of systems for several years. Use solid, tested resiliency practices and make security part of the design from day one.
When thinking about the connectivity of uplinks and downlinks, placement, power, and physical access are crucial. Avoid mounting devices high up, for example, which makes them a lot harder to troubleshoot.
4. IT/OT relationships
IT and OT don’t have an adversarial relationship, but it can come across that way because of misunderstandings about what they both do. Those misunderstandings create tension, but can also create bigger problems.
The OT side of the house makes the decisions that you need to support. So don’t tell them no. Work with them to support their initiatives.
If you tell them no, they’ll find a way to do it anyway and then you won’t know about it. Instead, when OT comes to you for help, try to engage them and understand what they’re trying to accomplish.
Work to understand what a particular device does, what function it plays in the plant, what its communication needs are, and help OT find ways to improve.
Empower OT to use the technology you put in place. Train them to understand the concepts and what’s important, and give them tools to decrease their reliance on you when they have to troubleshoot and fix issues.
For example, I once went to a manufacturing plant where they had three different switches for three different devices because the OT person who had deployed them didn’t understand the concept of VLAN. You might argue that having the three switches is more secure, but it’s also three times the cost. The person deploying them just didn’t know the concepts that a networking or IT person fundamentally understands.
In that sense, IT can help OT be more innovative. Don’t throw up roadblocks. Help them find their way.
Manufacturing better IT
Use these four factors as a guide to prioritizing security without compromising productivity, ensuring connectivity fits the unique needs of your manufacturing plant, designing thoughtfully to best use the space available, and building and maintaining relationships with OT.
With all that in place, you can better manage, secure, and innovate in any manufacturing environment.
Ron Grohman, Sr. Network Engineer for Bush Brothers, contributed to this post. Ron has more than 10 years of experience designing, managing, and supporting large networks in the manufacturing, financial, medical, and education spaces.