I worked for two years at a Cisco Gold Partner. The first year was great. We were trying to start up a Cisco practice in San Francisco (they were primarily a Citrix partner before), so my buddy and I wined and dined Cisco channel account managers trying to impress them with our CCIE’s and get them to steer business our way. Eventually, the 2009 financial crisis hit and business started to dry up. The jobs became fewer and less interesting. I had two CCIE’s and at one point, I drove out to Mare Island near San Francisco to install a single switch for a customer whose entire network consisted of–a single switch. I always recommend people not to stay in jobs like this too long, as it hurts your prospects for future employment.
Potential Employer: “So what kind of jobs have you done lately?”
You: “Uh, I installed one switch at a customer.”
Anyhow, we had one other customer that managed to keep me surprisingly busy, considering their network was quite small as well. They were a local builder, and with three small offices connected together with ASAs and VPN tunnels. The owner was filthy rich and also paranoid about security, which meant I was out there a lot changing passwords, tightening up ACLs, and cleaning up the mess the last network engineer had left.
The owner had a ranch near Wilits, CA which was reputed to be the size of the city of Concord, CA. He also had two jets to take him to his private landing strip at his ranch. Being a pilot myself, the prospect of a trip in a small jet to his ranch made me wish for some sort of network problems up there. However, there wasn’t much up there for me to work on. He had a single ASA 5505 connected to satellite uplink which he primarily used to connect to the cameras (which he had everywhere) at the ranch.
One day, my contact at the builder told me the cameras weren’t reachable. Yes! Finally a trip in the jet. We set a date and I spent my time wondering whether I’d get the Lear or the Citation.
Unfortunately, when the day rolled around, the weather was hideous. A Lear jet can handle most any weather, but the little airstrip had no instrument approaches. Instead, my contact gave me an alternative: I was to drive up there with her in-house cabling contractor (I’ll call him “Tim”) to do the job. (I never understood why a business this small had an in-house cabling contractor. As far as I knew he didn’t work on the actual construction projects associated with the company.) Now from San Francisco, the drive to Willits is about 2.5 hours. However, the ranch was near Willits. After driving 2.5 hours to Willits, we had another hour drive over dirt roads to the middle of nowhere.
The cabling contractor was exactly the sort of person with whom I have nothing in common, and spending 3.5 hours in a car with him, in the era before smartphones are a handy distraction, was painful. Tim loved fishtailing his truck as we drove on dirt roads on the side of a mountain. I think he also liked just scaring the white collar guy. It worked.
We arrived at the ranch and Tim opened up the back of his pickup. “Can you give me a hand here?” he asked. In the bed of his truck were several large carpet rolls and piles of dry cleaning. I grabbed one end of a carpet roll and began the backbreaking work. My company was billing me out at $250/hour to haul some lady’s dry-cleaning into her ranch.
The ASA itself was located in a pole in the middle of the property, which had a satellite dish on top. I was amazed the ASA 5505 even functioned out there, given that the external temperature could reach over 100 degrees Fahrenheit. The metal box housing the ASA was like an oven. I consoled into it and immediately saw a problem. Latency on the link was over one second round-trip. There was no way he was going to get real-time video streaming with this slow satellite uplink. I reported my findings to Tim and, after eating lunch with the ranch hands, we hopped back in the truck. Tim put on a song called “You piss me off, f*cking jerk” while we drove. I guess he didn’t like me.
When I mentor people, I often tell them you have to know the right time to quit a job. There were several signs in this story that it was time for a change. With two CCIEs, installing a single switch or working on a single ASA 5505 was not really a good use of my skills. Neither was moving in carpet rolls and dresses for $250/hour. Luckily I had enough big jobs at the partner that I managed to get through my interviews at Juniper without trouble.
Meanwhile, a few years later I read about the FBI raiding the builder who was my customer. I guess he had good reasons for cameras.