My first full-time networking job was at the San Francisco Chronicle. Now there isn’t much to the Chronicle anymore, but in the early 2000’s the newspaper was still going strong. It was the beginning of the decline, but most people still took their local newspaper as their primary source of news. Being a network engineer at a major metropolitan newspaper was fascinating. It is a massive operation to print and distribute a newspaper every single day, and you can never, ever, miss. There is no slippage of production deadlines. It has to be out every day, and every day you start all over, with a blank page.
As the lead network engineer, I touched everything from editorial (the news and photography content of the paper) to advertising, pre-press, production systems, and circulation. Every one of these was critical. If editorial content didn’t make it through, there was nothing to go into the paper. If advertising didn’t make it in, we didn’t earn revenue. If pre-press or production had problems, the paper wasn’t printed. If circulation wasn’t working, nobody could get their paper.
The Chronicle owned and operated three printing plants in the Bay Area. One was on Army Street in San Francisco, while the other two were in Union City and Richmond in the East Bay. The main office was on Fifth and Mission in downtown SF, so the paper was prepared in San Francisco and then sent to the plants via microwave. That’s where I came in.
Our microwave system used a dish on the clock tower of our building. From 5th and Mission we sent a signal up to Roundtop Mountain in the East Bay hills. At Roundtop we leased space in a little concrete bunker that was used for various kinds of radio communication including cellular. From Roundtop we bounced the signal back to the three printing plants.
Chronicle building with the microwave visible on the clock tower
The microwave presented itself to us as T1 lines. I had the T1 lines connected to dual routers at the main site and each of the plants. In addition to the microwave, we had two additional backup T1’s to each plant which were landlines from different carriers with diverse paths into the buildings. We kept the microwave and the first T1 plugged into the routers, with the third one on manual standby in case we needed it. You don’t take chances with production in a newspaper, and we had triple redundancy on everything. I used OSPF for redundancy between the microwave and #1 backup circuit on the routers, and HSRP for gateway redundancy. With only four sites it was a simple enough topology and it never gave me much trouble.
Until, that is, the day when I got a call from our operations center that the primary circuits were all down. We were running on backups. I immediately called up the production systems engineer who managed the microwave and told him his circuits were down. “Impossible!” he said, “that microwave is five-nines reliable. Check your router!” I tried a few of the usual: shut/no shut the interface, changing the line encoding, etc. No go. He wanted me to start swapping hardware, which was a big deal in a live newspaper environment, and seemed pointless. If it was hardware, why would all of the circuits be down?
We bickered a bit before I moved to have the tertiary backup circuits swapped in so we had automatic failover while we worked on the microwave. I got out our old T-berd tester to see if I could find any indication of the problem. Then the systems engineer called: “We need to meet at the clock tower, I’ve found the problem,” he said. It’s always a relief to hear that when finger pointing is going around.
T-berd T1 Tester
I showed up at the entrance to the tower and followed the systems guy up a rusty ladder mounted to the wall. Up in the tower there were bird droppings and as I climbed higher I fought the urge to look down. I’ve never much liked heights and being out of shape and relying on my own strength to keep from falling several stories onto concrete was not promising. Once I got to the top there was a large separation between the ladder and the floor, and I fought the urge to panic as I flung my leg way over to climb onto the concrete flooring. From there we went outside and I saw the problem right away.
If you’ve ever been to a convention in San Francisco, chances are it took place in the Moscone Center. In the early 2000’s, the city decided to expand Moscone by building a new Moscone Center West on 4th and Howard streets. And from up on the clock tower it was plain as day: they had built a cooling tower on the roof right in the path of our microwave beam. I looked at the systems guy and said, “Well, I guess you could make popcorn in that cooling tower. Anyways, there goes your five nines.”
We hastily called meetings together to decide what to do. Sue the city? Call the FCC? Find another building to bounce the microwave off of? Those were long term solutions but we had an immediate problem. Two circuits might seem like enough, but they were telco circuits and not as reliable as the microwave was, at least when its path wasn’t blocked.
Getting the city to cut the cooling tower off Moscone West was a non-starter, especially when it was the newspaper asking, a newspaper that made its money being critical of city officials. So, we decided to lease roof space from another building and add an additional repeater. However, this was a long process. We needed to negotiate with the landlord, replan the radio deployment, license it and obtain permits, add the new repeater, and re-point the old dish to the new building. That last item was not as simple as it sounded, since this wasn’t a DirecTV dish. It was welded to the tower, so we needed to hire ironworkers to cut it off and re-position it.
Meantime, we ordered T1’s from downtown SF up to Roundtop to bypass the segment that wasn’t working. We’d go hard wire to Roundtop, the microwave the rest of the way. This was not, by any means, an ideal solution, nor was it an overnight solution, but we could at least get some redundancy faster than it would take to add the repeater. I’m glad we did because shortly after the microwave went down we started having terrible problems with the landlines and needed the triple redundancy.
If you drive by Fifth and Mission now, the microwave dish is gone from the clock tower. The Chronicle, a shadow of its former self, no longer operates its own printing plants, and has a circulation far smaller than it did in 2004, when I left. As I said in my last post, it’s great to have a sense of purpose when you work in IT. It wasn’t about fixing a microwave but about getting that paper in the hands of our readers. I’m thankful I got to be a part of that for a few years, even if it cost me some vertigo and sleepless nights.