I haven’t written anything for a while, because of the simple fact that I had nothing to say. The problem with being a writer is that sometimes you have nothing to write. I also have a day job, and sometimes it can keep me quite busy. Finally, an afternoon drive provided some inspiration.
There’s a funny thing about the buildings I work in–they all tend to be purchased by Google. When I started at Juniper, I worked in their Ariba campus in Mountain View, several buildings they rented from that software company. We were moved to Juniper’s (old) main campus, on Mathilda Drive, and the old Ariba buildings were bought and re-purposed by Google. Then the Mathilda campus was bought by Google.
When I worked in TAC, from 2005-2007, I worked in building K on Tasman Drive in San Jose. Back then, the meteoric growth of Cisco was measured by the size of its campus, which stretched all along Tasman, down Cisco way, and even extended into Milpitas.
Cisco’s campus has been going the opposite direction for a while now. The letter buildings (on Tasman, West of Zanker Street) started closing before the COVID lockdowns changed everything. Now a lot of buildings sit empty and will certainly be sold, including quite possibly the ones in Milpitas, where I work.
Building K closed, if not sometime during the lockdowns, shortly after. I hadn’t driven by it in months, and when I did yesterday-lo and behold!-it was now a Google building!
What used to be building K
It’s funny how our memories can be so strongly evoked by places. Building K was, for a long time, the home to Cisco TAC. I vividly remember parking on Champion Drive, reviewing all of my technological notes before going in to be panel-interviewed by four tough TAC engineers. I remember getting badged in the day I started, after passing the interview, and being told by my mentor that he wouldn’t be able to put me “on the queue” for three months, because I had so much to learn.
Two weeks later I was taking cases. Not because I was a quick study, but because they needed a body.
I worked in High Touch Techical Support, dealing with Cisco’s largest customers. The first team I was on was called ESO. Nobody knew what it stood for. The team specialzied in taking all route/switch cases for Cisco’s large financial customers like Goldman Sacks and JPMC. Most of the cases involved the Cat 6k, although we supported a handful of other enterprise platforms.
When a priority 1 case came in, the Advanced Services Hotline (ASH) call center agents would call a special number that would cause all of the phones on the ESO team to play a special ring tone. I grew to develop a visceral hatred of that ring tone. Hearing it today would probably trigger PTSD. I’d wait and wait for another TAC engineer (called CSEs) to answer it. If nobody did, I’d swallow hard and grab the phone.
The first time I did it was a massive multicast meltdown disrputing operations on the NYSE trading floor. I had just gotten my CCIE, but I had only worked previosly as a network engineer in a small environment. Now I was dealing with a major outage, and it was the first time I had to handle real-world multicast. Luckily, my mentor showed up after 20 minutes or so and helped me work the case.
My first boss in HTTS told me on the day I started, “at Cisco, if you don’t like your boss or your cubicle, wait three months.” Three months later I had a new boss and a new cubicle. The ESO team was broken up, and its engineers dispersed to other teams. I was given a choice: LAN Switch or Routing Protocols. I chose the latter.
I joined the RP-LSA team as a still new TAC engineer. The LSA stood for “Large Scale Architectures.” The team was focused on service provider routing and platform issues. Routing protocol cases were actually a minority of our workload. We spent a lot of time dealing with platform issues on the GSR 12000-series router, the broadband aggregation 10000-series, and the 7500. Many of the cases were crashes, others were ASIC issues. I’d never even heard of the 12k and 10k, now I was expected to take cases and speak with authority. I leaned on my team a lot in the early days.
Fortunately for me, these were service provider guys, and they knew little about enterprise networking or LAN switching. With the breakup of the ESO team, the large financials were now coming into the RP-LSA queue. And anyone who has worked in TAC can tell you, a routing protocols case is often not an RP case at all. When the customer opens a case for flapping OSPF adjacencies, it’s usually just a symptom of a layer 2 issue. The SP guys had no clue how to deal with these, but I did, so we ended up mutually educating each other.
In those days, most of the protocol cases were on Layer 3 MPLS. I had never even heard of MPLS before I started there, but I did a one week online course (with lab), and started taking cases like a champ. MPLS cases were often easily because it was new, but usually when a large service provider like AT&T, Orange, or Verizon opens a case on soemthing like BGP, it’s not because they misconfigured a route map. They’ve looked at everything before opening the case, and so the CSE becomes essentially a middleman to coordinate the customer talking to developers. In many cases the CSE is like a paramedic, stabilizing the patient before the doctor takes over to figure out what is wrong. We often knew we were facing a bug, but our job was to find workarounds to bring the network back up so developers could find a fix.
I had my share of angry customers in those days, some even lividly angry. But most customers were nice. These were professional network engineers who understood that the machines we build don’t always act as we expect them to. Nevertheless, TAC is a high-stress job. It’s also relentless. Close one case, and two more are waiting in the queue. There is no break, no downtime. The best thing about it was that when you went home, you were done. If a call came in on an open case in your backlog, it would be routed to another engineer. (Though sometimes they routed it back to you in the morning.) In HTTS, we had the distinct disadvantage of having to work cases to resolution. If the case came in at 5:55pm on Friday night, and your shift ended at 6pm, you might spend the next five hours in the office. Backbone TAC engineers “followed the sun” and re-assigned cases as soon as their shift ended.
I make no secret of the fact that I hated the job. My dream was to work at Cisco, but shortly after I started, I wanted out. And yet the two years I spent in TAC are two of the most memorable of my career. TAC was a crucible, a brutal environment dealing with nasty technical problems. The fluff produced by marketeers has no place there. There was no “actualize your business intent by optimizing and observing your network”-type nonsense. Our emails were indeciperhable jumbles of acronyms and code names. “The CEF adjacency table is not being programmed because the SNOOPY ASIC has a fault.” (OK, I made that up… but you get the point.) This was not a place for the weak-minded.
When things got too sticky for me, I could call in escalation engineers. I remember one case where four backbone TAC escalation engineers and one from HTTS took over my cube, peering at my screen and trying to figure out what was going on during a customer meltdown.
Building K was constructed with the brutalist-stlye of architecture so common in Silicon Valley. One look at the concrete and glass and the non-descript offices and conference rooms is enough to drain one’s soul. These buildings are pure function over form. They are cheap to put up and operate, but emotionally crushing to work in. There is no warmth, even on a winter day with the heat on.
Still, when I look at building K, or what’s become of it, I think of all the people I knew there. I think of the battles fought and won, the cases taken and closed, the confrontational customers and the worthless responses from engineering, leaving us unable to close a case. I think of the days I would approach that building in dread, not knowing what hell I would go through that day. I also think of the incredible rush of closing a complex case, of finding a workaround, and of getting an all-5’s “bingo” (score) from a customer. TAC is still here, but for those of us who worked in building K, its closure represents the end of an era.