Technology refresh is nothing new in the IT (Information Technology) world. In the OT (Operations Technology) world, it is not new either: it just doesn’t happen as often.
In the past few years, we at EES have helped several of our clients “undergo” a technology refresh. For example, we performed a major motor protection upgrade for one of our clients in 2020, and a network switch upgrade in a large building for another.
Over the past three years, we also have been working on a major, staged, upgrade of an entire control system for a building that encompasses a data center, central plant, air handlers, VAV controllers, a substation, backup generators, and other ancillary systems. Due to the critical nature of the facility (a data center for a major company), the upgrade took a while to engineer. This project requires a sub-system by sub-system approach to minimize the interruptions, down time, and operator monitoring and intervention.
First, one of the two chilled-water pump VFDs (variable frequency drives) was replaced. Since they have redundancy, this was relatively easy. The old VFD was well over twenty years old and one of the cooling fans, embedded in the back of the drive, had failed. This caused the drive to overheat and shut down at full speed. Due to supply chain issues, once the first drive was installed we immediately ordered the second which was scheduled for replacement. By the time the equipment arrived, the second old drive was also exhibiting the same cooling issue, and a problem was narrowly dodged. Because we had already planned a vision of replacing the PLC5 that controlled the drives, we chose not to have the new drives communicate over legacy Remote I/O (blue hose). We did some research and found a device that was designed to transition from legacy comms to the newer IP-based technology. By using this, when we do replace the PLC5 later this year, the drives and flex I/O (which replaced older block I/O due to a board failure and inability to get parts) will be able to talk natively to the ControlLogix processor that was chosen for the project.
For the VAV equipment, when installed thirty years ago, there was no off-the-shelf solution to communicating with them. EES designed a custom solution that worked quite well. Now that simpler solutions exist; we engineered out the PLC-to-Modbus-to-Field Server-to-BACnet path. By taking advantage of EES’s in-depth knowledge of these newer components, we designed in new BACnet controllers. We connected these directly to the HMI, removing several hard to maintain parts of the communication path.
The air handler equipment also provided an opportunity to take advantage of the newer technology. The old technology was circa 1995 and used Rockwell SLC500-family processors and MicroLogix 1100 processors. Technology of that era didn’t have the kind of memory or I/O capacity of today’s equipment. As such, the programmer had to be creative and “pack” things in to the data and make programs with a lot of loops and indexed addressing. This was efficient but extremely difficult to read and support.
The old equipment also had made extensive use of the “Basic” module, with a “C” chipset. This was a very powerful combination, allowing the SLC500 to talk to many different devices before there were off-the-shelf solutions at that time. In the 1990s, EES’s engineers wrote custom drivers for Allerton, Triatek, MeterMate, Field Server, and others. While this worked extremely well for thirty years, the ability to debug and support it has waned. Because of the highly customized nature of our solution, updating the system was a critical consideration for us. A cornerstone of our company’s philosophy is that it is important for our clients to have more than a “one-deep support”, or a single key engineer, from their systems integrator. While we do have one engineer who can still navigate these legacy items, we jumped at the opportunity to make the system more supportable by our team going in to the future.
Once the drive upgrade was completed, we started the upgrade with our customer’s most significant concerns in mind. The first problem addressed was that the DH-485-based PLCs (about half of them in the building) would get “hung” every month or so, requiring a reboot of the DH-485-to-Ethernet gateway. A minor inconvenience to be sure, but it did make the data on the screen unreliable. As a temporary solution, we showed some of the site engineers how to reboot the gateway. This turned out to be a double-edged sword as they subsequently did that for ANY problem they encountered, whether it had anything to do with the freezing issue or not. It was a great example of the old adage, when the only tool in your toolbox is a hammer, everything starts to look like a nail. Ultimately, our moving the DH-485 equipment to Ethernet solved the reboot issue. We then moved on to the IP-based SLC units and then on to the MicroLogix.
Whenever you start a major upgrade, the unexpected should be expected even for the most experienced professionals. It is an interesting thing to go from 16-bit-based logic and data to 32-bit. The old SLC was 16-bit and had considerably less memory to work with than today’s processors. As I said above, the data got “packed” to minimize space. For example, the scheduler (the automation item that turns equipment on and off at predetermined times) used the sign bit (bit 15) to indicate whether the time listed was an “on” or “off” command. Going to 32 bits, bit 15 is NOT the sign bit, bit 31 is. So, there was a lot of debugging work to get the code to run in 32-bit hoops.
More issues appeared when looking at integer division changes. One would think that division works like it did when we were in grade school. In fact, the old SLC units did have a quotient and remainder. This remainder was in a special register. Much manipulation was needed to use these values beyond the simple quotient. Not anymore. The 32-bit system does not have a remainder: It is thrown away. This requires the extra steps using a MOD (modulo) instruction when needed. Rounding is different in 16-bit addresses in a 32-bit processor than the 32-bit, meaning more debugging and work.
Before going in to the process of upgrading the old MicroLogix 1100s, I want to talk briefly about retrofits in general. When commissioning a new plant, it is much easier than a retrofit in some ways. Yes, there are more things to check out in a new plant, but one has the “luxury” of not going on line until everything is stable. In a retrofit situation, the added challenge is to minimize downtimes and plan for intermediate/temporary controls during the upgrade. It is more time consuming and more stressful to all involved, including both customers and integrators, than a new construction.
That said, the MicroLogix WAS a retrofit when installed in the 90s to the EES-designed “TC1” controller. The TC1 was a single-loop controller that was released in the very early ‘80s. Even a decade later, EES’s design was far more precise at control than pneumatic PID controllers available at that time. When it came time to retrofit them TO the MicroLogix, it had to be done one loop at a time. This made for a hybrid of equipment and a shortage of space for wiring. The resulting system worked very well but, from a support perspective, looked like it had been done one loop at a time.
Taking advantage of the planning and capabilities of the new equipment, EES engineers redesigned the I/O and layout to make it more supportable and sustainable.
Of course there were software issues that were resolved through the process. The PID instruction of the 1990s didn’t work like the current one and many manual translations had to be made.
Overall, the conversion continues along with only minor interruptions while an entire system is upgraded. The effort is, in the writer’s opinion, very worthwhile, both in the maintainability and the ability to get current parts. The biggest lesson learned from this particular upgrade is that updates should be more often than thirty years and that the longer ago the last one was, the more surprises one is going to find.