The 2 AM Phone Call
February 2024. A stamping plant outside Greenville, South Carolina. The night supervisor calls me at 2:14 AM: "Our line's down. The PLC's showing a fault. We need a replacement S7-1500 by 8 AM or we miss a $180,000 shipment."
I've been doing emergency industrial repairs for 11 years. I've handled 300+ rush calls. So when he said "the PLC's showing a fault," I already had my answer ready: probably a bad CPU, or a firmware issue. Maybe someone spilled coolant on the rack. Replace the module, reload the program, done.
I was wrong. And not for the first time.
What You Think the Problem Is
When a PLC stops working, the instinct is to blame the PLC. It's the most expensive component in the cabinet, it has the most status lights, and it's what the HMI screen says is broken. Replace it, right?
That's exactly what this client thought. They'd already ordered a replacement CPU overnight—$3,200 for the part, plus $800 in rush shipping. By the time I arrived at 5:30 AM, they had the new module sitting on a bench, ready to swap it in.
Here's the thing: the CPU wasn't broken. It was doing exactly what it was supposed to do—reporting a fault condition. The fault wasn't in the PLC. It was in everything around it.
The Problem Behind the Problem
I pulled the diagnostic buffer. The S7-1500 had logged over 200 "undervoltage" events in the previous 48 hours. The 24V DC power supply was putting out 21.3V under load—technically within the S7-1500's specified tolerance of 19.2-28.8V, but on the low end. Combined with a 15% voltage ripple that the power supply's aging capacitors were injecting into the backplane, the CPU was periodically resetting its communication module.
Why was the power supply struggling? A month earlier, the plant had added a new servo press to the same 24V circuit. Nobody recalculated the load. The power supply was rated for 10A; the new total draw was 9.2A. Add inrush current from the servo starting 40 times an hour, and you get exactly what the diagnostic buffer showed.
I've seen this pattern so many times now that I can almost predict it. The PLC gets blamed because it's the "brain" of the system. But the brain can't work if the nervous system is misfiring.
This isn't just a PLC problem. I've seen the same dynamic in entirely different contexts. A client with a solar panel battery charger setup kept getting "communication fault" alarms between their charge controller and an S7-1200 monitoring it. The problem wasn't the controller or the PLC—it was a grounding mismatch between the two systems. Another site had an Aqualink RS8 control panel showing "normal" status while the actuators wouldn't move; the root cause was a shared neutral line picking up noise from a nearby variable frequency drive.
And—this one still makes me laugh—a maintenance tech once called me about their lithium battery charger blinking red. They assumed it was a charger fault. The charger was fine; it was signaling a communication timeout because the PLC rack's communication module had lost its configuration after a power event. The charger was just the messenger.
Why This Keeps Happening
Here's the uncomfortable part: most of these problems trace back to decisions that seemed reasonable at the time.
When the plant added that servo press, they did an electrical load calculation for the new equipment. They just didn't include the existing 24V control circuit in the calculation. Why? Because it had been "working fine for years." That's the trap. Everyone assumes the infrastructure that's already there is sized correctly.
In my early days doing this work, I made the same mistake. A food processing client had a recurring S7-300 fault that I "fixed" three times by resetting the CPU. The fourth time I finally traced it to a failing power supply—but by then the client had spent $1,800 in emergency callouts that a $200 power supply replacement would have prevented. I'd been treating the symptom because it was faster and easier than investigating the root cause.
I knew I should have flagged that power supply during my previous visit, but thought "what are the odds it fails in the next three months?" The odds caught up with me on a Thursday afternoon. The line went down mid-shift. Three hours of production lost that a five-minute voltage check would have caught.
The other issue is the quality perception gap. I've watched maintenance managers fight to get budget for a spare S7-1500 CPU (around $4,000-6,000) but refuse to spend $800 on a power quality analyzer—because the analyzer doesn't "show up" on the line. It doesn't look impressive in a budget review. But I can tell you from 300+ emergency calls: the vast majority of PLC failures I've seen weren't caused by the PLC. They were caused by what was feeding it.
What It Actually Costs
Let's talk about that February call. The stamping plant had the right instinct—they wanted to fix the problem fast. But their fix was the wrong fix. They'd ordered a replacement CPU they didn't need.
The real costs:
- 8 hours of production downtime: approximately $24,000 in lost output (their estimate, based on $3,000/hour line value)
- Emergency shipping for a CPU they returned: $800 (non-refundable)
- My callout and diagnostic time: $1,200
- Actual fix (new 10A power supply + load rebalancing): $450 in parts
Total: roughly $26,450 for a problem that a $300 thermal camera inspection during planned downtime would have caught.
The replacement CPU would have cost them $3,200. Actually, $4,000 with the overnight shipping and the restocking fee on the return. They ate that cost for nothing.
The bigger cost wasn't on the invoice. Two other departments in that plant heard about the delay. Next quarter, when the maintenance budget came up for review, they were asked to justify why they needed condition monitoring equipment. The answer—based on what had just happened—was pretty compelling. But it took a near-miss on a $180,000 shipment to get there.
And here's what still frustrates me: the data was there. The diagnostic buffer had been logging undervoltage events for weeks. Nobody was looking.
What to Do About It
I said this was a problem deep-dive, so I'm not going to give you a 12-step inspection checklist. Here's what actually moves the needle:
Treat the power supply as part of the PLC. When you budget for a Siemens S7-1200 starter kit or an S7-1500 CPU, include a quality power supply and a power quality monitor in the same line item. The CPU's price tag gets the attention. The power supply is where the failures hide.
Check the load after every change. Not just the new equipment—the entire circuit. It takes 20 minutes. The alternative takes 8 hours and costs $26,000.
Watch the security bulletins. Siemens publishes ProductCERT advisories for their PLC families. If you're running older firmware on an S7-300 or S7-1200, those advisories matter. Not because the PLC will "break"—because an unpatched vulnerability can cause the same kind of 2 AM phone call as a bad power supply. The IEC 62443 framework exists for a reason: it treats security and reliability as the same problem, because at 2 AM, they are.
Investing in power quality monitoring is almost always cheaper than the downtime it prevents—at least, that's been my experience with the clients who actually did it. The ones who didn't? They're the ones calling me at 2 AM.
And one more thing: when someone tells you the PLC "just stopped working," ask what changed in the last 30 days. The PLC probably didn't break. Something around it did.
