It was four o’clock in the early hours of one Sunday morning in 2001. I had been up all night sitting in our data center at the San Francisco Chronicle with our Unix guy. He was handing off responsibility for managing the firewalls to the network team, and he was walking me through the setup. He’d been trying all night to get failover to work between the two firewalls, and so far nothing was going right.
We were using Checkpoint which was running on Solaris. Despite my desire to be Cisco-only, I was interested in security and happy to be managing the firewalls. Still, looking at the setup our Unix guy had conceived, my enthusiasm was waning.
He drew a complex diagram on a piece of paper, showing the two Solaris servers. There was no automatic failover, so any failure required manual intervention. He has two levels of failover. First, he was using RAID to duplicate the main hard disk over to a secondary hard disk. If the main disk failed, we’d need to edit some text files with vi to somehow bring the Sparc Ultra 10 up on the second drive. If the Ultra 10 failed entirely, we would have to edit some text files on the second Ultra 10 to bring it up with the configuration of the first. With Unix guys, it’s always about editing text files in vi.
Aside from being cumbersome, it didn’t work. We’d been at it for hours, and whatever disk targets he changed in whatever files, failover wasn’t happening. At the newspaper, we had until 5am Sunday to do our work, after which everything had to be back on line. And we were getting concerned it wouldn’t come back at all.
Finally the Unix guy did manage to get the firewall booted up and running again. On Monday I called Checkpoint and asked how we could get off Solaris. They made a product called SecurePlatform, which installed a hardened Linux and Checkpoint all with one installer. I ordered it at once, along with two IBM servers.
The software worked as promised, and I brought up a new system, imported our rules, and did interface and box failover with no problem. I told the Unix guy to decommission his Ultra 10s. He was furious that there was a *nix system on the network his team wasn’t managing. I told him it was an appliance and there was no customization allowed. The new system worked flawlessly and I didn’t even have to touch vi.
Network engineers are used to relatively simple devices that just work. Routers and switches can be upgraded with a single image, and device and OS-level management is mostly under the hood. While a lot of network engineers like Linux or Unix and have to work with these operating systems, at the end of the day when we want to do our job, we want systems that install and upgrade quickly, and fail over seamlessly. As networking vendors move more into “software”, we need to keep that in mind.