I’ve written before about my years at the San Francisco Chronicle, my first job which was exclusively network engineering. It was an interesting environment, as this was back in the years before the Internet totally displaced newspapers. We had printing plants to support, active newsrooms with reporters and photographers, and a massive circulation operation.
When I first arrived, our Internet was provided by a T1. At the time, the 1.5 Mbps was considered pretty fast, but with over 1000 users depending on the T1, it was slowing to a crawl. The T1 was terminated on a good old-fashioned Cisco 2500-series router. Later I upgraded our service to a T3 and terminated it on a 7204VXR, which led to a dramatic speed improvement. The 2500 sat in an open, free-standing rack in the corner of our data center.
One day I noticed that the Internet was down. Even though this was the early 2000’s, Internet connectivity was already critical, especially at a newspaper. The problem quickly escalated to the CIO and I scrambled into action. I could reach as far as the firewall, but there was no connectivity beyond that.
Sometimes the best thing to do is to physically check on a problem. The networking team was in the basement and our data center was on the second floor. I sprinted up the stairs and badged in to the data center. One of our mainframe operators, who worked in the data center grabbed me and asked “hey, do you know the Internet is…?”
“Yeah, yeah,” I said and ran to the rack with our T1 router and DMZ switches.
I saw a gentleman in white painters clothes with a paint roller in the corner. He was painting the wall right along side the rack. There was only a few inches of clearance between the rack and the wall where he was rolling paint. On the side of the rack, held in place with plastic zip ties, was a power strip with all the rack’s hardware plugged in. He’d hit it with the roller and knocked out the power.
Network engineers love to solve complex problems, but when people are yelling at you it’s a relief when the problem is simple. I flipped the switch on the power strip and Internet was restored as soon as the router booted up. Someone had obviously placed the power strip on the wall-side of the rack figuring it would minimize the chance of it getting bumped. They had inadvertently created the problem they were trying to solve. I told the painter to stay away from my rack.
Networks are complex entities, but often the problems we face involve bad splices in holes in the ground, physical obstructions to WiFi signals, loose connections, and rogue paint rollers.