Easily the most critical project I’ve had in months, and it’s over, except for the paperwork. Several months ago we brought in a new cabinet with state of the art protective relays. We’re talking about whiz-bang stuff here. A big box. Programmable. Filled with microprocessors. Florescent display. Ethernet ports.

The generator we’re talking about is 22.5 megawatts. That’s enough for a thousand or more nice homes. This one provides all the power for a plant when they absolutely cannot afford to lose power from a notoriously unreliable utility company line. It’s also 25 years old. When it was installed, the protective functions were provided by a dozen or so little devices consisting of cleverly configured coils of wire and pieces of metal. The idea is that these little boxes called relays watched different chunks of current and voltage associated with the generator and would shut it down if it looked like the generator was operating in a manner that would damage itself. 22.5 megawatts is millions of dollars. You wanna take good care of that investment.

Although this technology was very reliable, having been the industry standard for the best part of a century, it was, in a word, dumb. Imagine this. You’re kicked back in your chair, enjoying life, and the lights in your plant go out. Your generator has just tripped off. After the emergency generator comes on and you can see what’s what, you go look at the protective relays. The one(s) that operted to take your generator down identify themselves by dropping a little colored flag. If you know your business, this is good information, because you’ll know what parameters caused that particular relay to operate. But if several operate, you don’t know which came first. You also don’t know if the situation came up suddenly, or deteriorated over a period of time. Troubleshooting with this little bit of data could be time-consuming, taking many runs up blind alleys to isolate and eliminate causes. And worse, the electro-mechanical devices could suffer from settings drift over a period of time, causing them not to react at all or to react prematurely…

Fast forward to today’s magic box. First, it’s all in the one box. Of course, instead of a dozen boxes with four to twelve wires each, you now have this monster with maybe a hundred or so. You just hope you’re not using ALL of them. Second, while the setting of the old stuff was meant for engineers and technicians using nothing more than a few tables and common math, the new one is **programmable** with many features you might not use or need. The darned thing is COMPLEX.

So there I was: the new cabinet, the new relay, thirty or forty wires to connect the new relay to the old equipment. Not bad, huh? Well, there arises a problem. At the same time I hook up the new one, I have to disconnect the old stuff in such a manner as just incase the new stuff doesn’t work, the old stuff can be reconnected, because the generator has to be ready to run for Thursday, one way or another, new relay or old.

Now, in case you’re wondering, the generator controls reside in seven cabinets about the size of large home refrigerators. That’s just controls. The generator itself is outside the building, and back on the inside of the building there are two line-ups of 13,200 volt switchgear. The stuff I’m installing controls two of the circuit breakers in this gear. The circuit breakers in your house fit in the palm of your hand and weigh a few ounces. My circuit breakers fit in steel cabinets the size of walk-in closets, weigh five hundred pounds, and require a special lifting device to install.

When this stuff is built, there are a bunch of drawings provided: some are laid out in a manner that shows devices in a logical manner which tells how they function. Other drawings show the physical layout of the various devices, and yet others show how the hundreds of feet of wire have to be connected to hundreds of various terminals to make things work. Now, these drawings might have been pretty accurate in 1979 when the stuff was built, but over the years modifications and repairs changed things. And in this case, I seriously question the initial accuracy. Yes, it worked right. But no, it wasn’t wired up like the drawings showed.

Now, along comes an engineering company who has to take these old drawings and show two things: first, what has to be removed, and second, what has to be installed. So what I get are the old drawings AND the new drawings, and Monday morning I get to tell the client that when I cut the first wire, he doesn’t have a generator any more for three days. And that’s what I do.

Then, in the course of disconnecting the old and tying in the new, I start finding out exactly how screwed up the drawings are. They’re nice drawings. General Electric does good work to this day. But in too many cases, what’s on the drawing isn’t what I’m looking at on the equipment. So I’m investigating and figuring out solutions on the fly. Connecting two wires, a five minute job in itself takes three hours as I have to figure out what the deal is witha circuit that doesn’t match the drawings I’ve been given, I do have the help of one of my particularly capable technicians, and I’ve got the design engineer from the engineering company on short leash, but for all day Monday and all day and well into the night on Tuesday, I’m under a lot of stress.

Which brings us to Wednesday morning. Time for the test run. First we put the breakers into the test position. This means that the external control circuits are hooked up so that the breaker will open and close, but it’s not connected to the bus, that portion of the aparatus which actually carries the 13,200 volt power. Like this, we can close and open the breakers and test all the control and protective functions without actually dealing with the high voltage. The generator didn’t have to be running for these tests.

And we uncovered various little glitches. And fixed them. That’s why we do this in “test mode.” It’s far better to find out you can’t open a breaker in test mode than it is when it’s carrying enough power for a small town. We hook up test equipment and inject currents and voltages to simulate the various inputs to the new equipment, verifying that the wiring we’ve tied in does indeed squirt electrons into the proper orifices. Everything looks good.

So now it’s time to really do it. We call for an operator to come start the turbine and put the generator on line. While we’re wating for him to show up, we crank the mechanisms to move the breaker from the test position into the operate position. It’s getting ready to get real.

The operator cranks up the gas turbine. This thing’s almost exactly the same as a jet aircraft engine, except GE takes the exhaust gases off the tail end and uses them to spin a generator. Starting is automated. After twenty minutes of unlocking various safety devices that were locked out when the generator was turned over to us.

Anyway, the turbine comes up flawlessly. The generator comes on line. It’s making volts, but the the power doesn’t have anywhere to go until the breakers close, tying the generator to the plant electrical system. So the control is activated to close the breakers. They close. The sound is, shall we say, un-subtle.

And they immediately trip. They make more noise tripping than they do closing. Especially when they’re NOT supposed to open at all right now. I will admit that several expletives were uttered.

We have the client’s electrical engineer, the consultant electrical engineer, the expert who sold us the new relay, and oh, I guess a couple other eyes looking to see what happened. Well, this is THE MAGIC BOX. Unlike the old relays, instead of a bit of colored metal indicating only that this relay had operated, this one captured huge amounts of data as to which of its parameters had been exceeded to cause the breakers to trip. A laptop computer gave us access to all that data. It is supposed to tell us what’s wrong.

And it does. Seems it thinks that when we closed the breakers to allow power to flow FROM the generator into the plant, which is a GOOD thing, it thought power was flowing TO the generator from the plant. This is a BAD thing. **WE** knew that didn’t happen, but our new little smart box held a different view. Opinions flew.

Me, I grabbed the manual on the new relay, an 8.5×11 inch spiral-bound book over an inch thick. I flipped to the the installation instructions, then compared that with the drawings I’d been given to complete the installation, and there I spotted the problem. A set of critical components were wired up incorrectly, sending their output to the new relay reversed from what it should be. My comment? “I built it just like you told me and it’s STILL wrong!” A screwdriver and ten minutes later, we were ready to try again.

And so we tried again. And it worked. Mostly. The breakers closed. We boosted the power output from the generator and cut the plant loose from the utility. Everything mostly looked good. So we transferred the load back to the utility, a matter of controlling the voltage put out by the generator. And as a final act of testing, we told the new relay to trip the generator. We wanted to make sure that when the generator tripped, the turbine would trip also. And it worked.

And you have no idea how big a weight lifted off my shoulders. We’d made the deadline with hours to spare.

So today was spent carefully going over all those drawings and making sure that they reflected all the stuff we’d found and changed, creating the “as-builts”, which would go back to the engineering company to be rendered correctly and neatly in AutoCAD. And tomorrow there are some loose ends as we tidy up the graphic display, a flatscreen monitor and computer installed with the equipment which allows the operator to access all the information provided by the relay.

But it’s over. And I think they should have a brass plaque with my name on that generator… Maybe I won’t dream about it again tonight…