Category Archives: Reflections

by

Legacy language

The spambots who regularly read my blog know that I have a fascination with language, and have a background in linguistics. So, I’m always fascinated by language and how we use it. This post is a bit pointless, but amusing (perhaps only to me).

In my last post I mentioned I might file a bug on VRRP. Apparently, I still have access to do this, although I cannot remember how to assign it.

Anyways, most Cisco customers know that a bug at Cisco is called a “DDTS”. We’ve called bugs DDTS’s as long as I’ve worked on Cisco products. But what does it actually stand for?

DDTS stands for Distributed Defect Tracking System. DDTS is actually the name of the bug-tracking software that Cisco used, a long time ago, to file and track bugs. DDTS was replaced long ago by CDETS, Cisco Defect and Enhancement Tracking System. Even when I was in TAC in 2005, we used CDETS, and not DDTS, to file and track bugs.

Not only is the term DDTS obsolete, it doesn’t even make sense if you expand the acronym. “There is a DDTS in that version of IOS XE” actually means “There is a distributed defect tracking system in that version of IOS XE”. Not likely!

Language evolves and words often lose meaning as they are used. I remember the Greek word “thumos” which meant something like “warrior spirit” in the days of Homer. A positive thing, something you’d want to have, a certain virility or militancy. By the time of New Testament Greek, 700 or so years later, it just means “anger”.

So, at Cisco my exec will have an “off-site”. This used to mean you’d go to Hawaii. Then, a Hilton in another city. Then, the local Hilton. Then, another building on the same campus. Now, at Cisco, we have “off-sites” on the same floor of the same building we work in every day. Nothing off-site about that!

Maybe we should have an off-site about DDTS’s…

1
by

My years in management: Looking forward

 

As of December, my role at Cisco has transitioned from a leadership role back to an individual contributor.  Gone are the constant approval emails;  gone is the stack ranking of employees;  gone are the performance reviews and the one-on-ones.  It’s both relieving and eerily quiet.  As I make this transition back, after nearly eight years, I thought it would be a good time to reflect on leadership and what it means to me.

I never wanted to be a “people manager”.  When I first started in networking, way back in the early 2000’s, I was explicitly clear about this to my boss.  I loved hands-on work.  I wanted to be in the trenches.  If I were a cop, I wouldn’t want to be a detective–I’d want to be in the patrol car.  I also lacked the self-confidence to take on a team.  Routers I could handle; people not so much.

I worked in Cisco TAC, then at a partner, and then at Juniper Networks.  In all those years, the idea of managing people never once came up.  I didn’t ask for it, and nobody asked me to do it.  I was happy being the hands-on guy.  When I got married in 2012, I told my wife to never expect me to be in a leadership role.

That all changed when I came to Cisco for the second time in 2015.  I had landed my dream job as a technical marketing engineer.  I loved having a lab, doing Cisco Live presentations, writing blogs and books, and working with customers.  I was quite fine with this when my boss, Carl Solder, one day stunned me in our 1:1 by asking me to lead a small team.  I objected lightly–I want to do hands-on work, not manage people.  Don’t worry, he said, you still can.  To my surprise, as well as my wife’s, I said yes.

My first team had 12 people on it, covering Software-Defined Access, Assurance, and Programmability.  A bit of a hodge-podge.  The day the team was announced I was pulled one-by-one by my new team members into a room, to listen to each of their demands.  What had I gotten myself into?  My boss later told me that two experienced managers who were sitting outside the small conference room rolled their eyes and simultaneously said:  “welcome to management!”

The management philosophy I brought to my team didn’t come from books or coursework.  It came from my own observations.  Simply put, there are two management styles:  negative and positive.  Negative leaders are the most common.  They lack empathy and are thus unable to work with people.  They manage by assigning tasks and tracking metrics.  They pile on their people.  They are hard on their teams, task masters, and are mainly interested in their own self-promotion.  They see their team as a tool to achieve their own personal goals.  I’d had many such leaders, and worked poorly under them.  I struggled to remain motivated, and would usually do just enough work to get by.

Positive leadership looks at employees as individuals.  Positive leaders try to understand their employees’ needs and help them grow in their careers.  They look for potential in employees and try to develop that potential.  They try to align assignments to employee’s skills rather than forcing work upon people.  They look for strengths and play to those strengths.  Positive leaders are “servant leaders”, as much as I hate the cliche.  They care more about their people than themselves.  They promote their people rather than themselves.  Under this style of leadership, employees work hard because they feel their leadership cares about them.  They usually want to make their boss look good, and feel personally disappointed when they let their boss down.

I learned a simple rule from my first boss, Henry Sandigo, who practiced this style of leadership.  When your team fails, you take the blame as the leader.  When your team succeeds, you give them the credit.  Negative leaders do the opposite.  When their team succeeds they take the credit, and when their team fails, they blame their team.

One time, my team held up a software release due to critical bugs.  This upset engineering, who pushed back.  One of the product management leaders was furious with me.   He came to me, stood an inch from my face, and with bloodshot eyes yelled at me:  “I want the name of everyone who was in that room making the decision.”  I said to him he could have one name, mine, because it was my team and I was responsible.  It turns out we were right, but I had to endure the fury of that leader for a long time.

If negative leadership is so ineffective, why is it so common?  The answer is simple.  Negative leaders tend to be ladder-climbers and self-promoters, whereas positive leaders are humble by nature.  Negative leaders are always out for themselves, and in the corporate world, that tends to advance you to higher positions.  Additionally, if negative leaders are in power, they have no respect for positive leaders.  They tend to promote people with their own leadership style, and view positivity as soft and weak.

As a leader, you become involved in people’s lives, often quite intimately.  I had two employees go through bitter divorces while on my team.  My philosophy was to give them the room they needed to recover and get back to work.  I had interpersonal conflicts that went to HR.  I had one employee drop from a cardiac arrest and who has been in a coma for two and a half years.  I’ve been in the room with him and his crying family, and dealt with his long-term leave of absence.  Configuring routers is easy in comparison to the harsh realities of life.

I’ve also had to lay people off more times than I can count.  One of the main reasons I didn’t become a manager for so long was that I never wanted to lay someone off.  It’s an unfortunate reality in the corporate world today.  When I’ve made that call, some have been angry, some have cried, most were just quiet.  I can only say I hated having to do it, and that I fought hard to not do it.  In fact, I’ve been criticized for fighting too hard to save jobs.  I cannot really complain as it’s much harder to receive the call than to make it.  But I tried to see my employees as humans and to help them as much as I could.  The unfortunate reality of the corporate world is that people are just seen as OpEx, as numbers on a spreadsheet, and not as human beings whose lives are horribly affected by losing their jobs.  I don’t know if I will ever return to leadership, but I’d be happy if I never had to make those calls again.  (For what it’s worth, I once was on the receiving end of the call.)

I also got to make several happy calls.  Promoting people is one of the great joys of management.  I helped several people get director and principal promotions, which are very hard to get at Cisco.  Although they did the work, I did the back-end negotiation, and I’m proud of each one of them. 

I tried to go the extra mile for my employees.  During the COVID lockdowns, I drove around to each of my direct’s houses and surprised them with a Christmas gift.  They were grateful to see a co-worker after so much time in lockdown.  When one employee, a gin lover, had a really bad day with a difficult VP, I sent him a nice bottle of gin, at my own expense.  I found little touches like this go a long way in building loyalty and positivity in a team.

We all learn from the great leaders we worked for.  I mentioned Carl and Henry.  Bask Iyer was another one. Bask came in as CIO of Juniper at a time when working in IT was like working in a morgue.  We were all depressed, beat up by the business, and unmotivated.  Bask would defend us in company meetings when we got attacked.  He went to bat for us.  He was a great technologist, but what really impressed me was his ability to stand up for us.  Gary Clark, who reported to Bask, exuded the same positivity.  When you met with Gary for the first time, he had a series of lego blocks with different personality traits.  He would arrange them in order while he was talking to you, building a model of your personality.  In other words, Gary wanted to know who you were, how you thought, and meet you where you were.  I always appreciated that.  

At my peak, I had 50 people reporting to me and multiple layers of management.  Then, through attrition, it dwindled to 30, 20, then 8, and now none.  A team of 50 seems like a distant memory to me now.  It’s hard to believe I did that.

Many technical people come to the same fork in the road that I did.  Do you stay technical, or do you take a management job to advance your career?  Notice I put these in opposition.  I can affirm that when you go into the management track, your technical skills atrophy.  As much as I tried to maintain and work in a lab, it became nearly impossible.

There was one day when I was invited, as a senior director, to a meeting on software-defined access architecture with a bunch of distinguished engineers.   They were discussing an idea around multicast.  As I listened, I decided to interject:  “If you do it that way, you’ll break PIM registration.”  One of the DEs rolled his eyes, but then another said “wait, Jeff’s right!”  It was nice to know I still had it.

Those moments, however, become rare.  I know that all of my employees respected that I had a technical background.  It’s important that leaders in technology companies know technology.  But if you go the management route, you will definitely find the technical side of things recedes as people become your main concern.

The hardest thing about being a team leader at a company like Cisco is pleasing all the factions that will ultimately provide feedback on you.  The second you step into leadership, there is a target on your back.  The corporate world is Machiavellian.  Nice guys finish last.  If you try to partner with and please one leader, another leader will get upset.  Pivot to him, and the first guy gets mad.  This was especially true for TME, which is seen as a service organization.

It’s important to remember that the corporate world is vicissitudinous.  Over the course of the years, you will see roles come and go.   I’ve seen executives who were flying high one day shown the door the next, and a whole new regime comes in.

As I said at the beginning, in some ways it’s a relief.  On the other hand, one of our product management VPs saw me with my team and said I was like a “proud papa.”  While it’s nice to do things myself again, I can say I was proud of the teams I lead, and I miss taking pride in my team instead of myself.

Will I ever lead a team again?  Who knows.  If I’m asked I will gladly do it.  If not, I’ll do my job as an individual contributor.  I don’t think there’s much room in the corporate world for leaders with my style, anyways.

The upside is, I can now spend time in the lab.  My routers won’t ask for raises.

22
0
by

Why I am not an FBI Agent

Some years back I wrote a post poking fun at the Federal Bureau of Investigation, based on an experience I had at a briefing in their office.  The funny thing is–and this did not color my article– there was a point in my life when I badly wanted to be an FBI agent.

Back in the early 2000’s, I was a run-of-the-mill network engineer.  I liked what I was doing, but I also felt a lack of purpose and direction in life.  I also didn’t want to sit around in an office all day.  I think a lot of people in their twenties go through this, at least in America.  It’s why my nephew is trying his hardest to become an Air Force parajumper.  It’s why my brother wanted to be a Navy SEAL.  (He had to settle for infantry major.)  This seems to be a “first world” problem.  Coworkers from other countries tell me all they want is a good job and home, and a safe place to raise their kids.  American affluence leads to boredom, and boredom to a desire to do something more important.

At that time, the FBI had three tracks for joining as a special agent.  You could enter if you had a law degree, or an accounting degree.  If you didn’t have one of these, you could enter under the “diversified” qualification, meaning you had some other experience that might be relevant to the Bureau.  At some point in the early 2000’s, the FBI added CCNP and CCIE certifications, specifically, as qualifications equivalent to a law or accounting degree!

I’d actually had some interest in an FBI job back in college.  I remember a female student who was also quite interested, and at graduation we looked at each other and said “see you in Quantico“, the FBI’s training center.  (I looked her up for this article, and it turns out neither of us got there.)  I had parked my interest as I didn’t wind up getting a law degree, but it was rekindled when I saw they added Cisco certifications to the list.

How cool would it be to investigate and bust criminals?  How cool would it be to have a badge and credentials to flash at people?  How cool to have arrest powers?  I was bullied as a kid, so I always had respect for law enforcement officers, and a strong belief in the importance of law and order.  One night, walking home by myself, I remember some kids vandalizing cars and houses.  “We’re f**ing s**t up!” they shouted at me.  I didn’t have a cell phone on me, and I couldn’t do anything.  But if I had a badge and a gun, I could have put a stop to it!  (My guess is that actually, a lone FBI agent would call the cops and not try to singlehandedly stop a group of thugs.)

I bought several books on the FBI and the application process, and started reading up on it.  The X-Files was popular at the time, but I had a realistic understanding of what the FBI did and was well aware that I would not be driving around the country chasing space aliens.  My hope was to be a cybercrime investigator.  I remember telling my brother and some of my friends that this was my plan.

So what happened?  Well, I never even applied, for a few reasons.

Not for me!

First, I hate exercise.  I was always the last kid picked for sports teams.  I was always behind on our school’s annual fitness testing.  PE class was a constant source of embarrassment.  The FBI requires a challenging fitness test just to get in, let alone to pass their academy.  I ended up signing up at a local Karate studio to try to get in shape.  I never got beyond a white belt.  (I’m proud to have a white belt in Karate, Tae Kwon Do, and Judo!)  While I liked learning the techniques, they had an intense fitness class required as part of their program and I hated it.  How could I survive six months of that?

Second, I didn’t want to live in a dorm for the academy, and I really didn’t want to live with a roommate, even for a few months.  There were two videos on the FBI Academy at the time, one from Joan Lunden and another from CNN.  I paused them when they showed the dorm rooms, and I clearly saw two beds.  I had a horrible time with my college roommate, and I didn’t want to experience that again.  Granted, an FBI trainee was not likely to show up at 3am stoned and wake me up to chat, but still, the lack of personal space, the snoring…  I didn’t want it.

Actual FBI dorm room-no thanks!

Third, if I went to the FBI, I’d take a pay cut.  Even a few years into the IT world, as a Cisco-certified engineer, I was doing quite well for myself.  I wanted to serve a greater purpose, did I really want to get paid a lot less?

Next, FBI agents are required to go anywhere in the country the bureau assigns them.  They also have to move several times in their career.  All I wanted was to live in San Francisco, my home town.  My friends and family were there.  How could I deal with it if I were assigned to Peoria?  Or, horror of horrors, New York City??

Finally, I did not want my job to be tied to the polygraph examination.  I’d been fascinated by “lie detectors” since I built a rudimentary one from an electronics kit when I was a kid.  I had read A Tremor in the Blood by David Lykken, Ph.D.  I knew that the lie detector is anything but.  Sociopathic liars often pass it, and legitimate and honest people often get called out as liars.  If I did do the work to get in shape, set my sites on this accomplishment, and then failed a polygraph not because I was lying, but because it’s a stupid test, it’d be miserable.  And FBI agents can be routinely tested with polygraphs.  Imaging getting the job and then losing it due to a faulty polygraph!

Read this and you’ll never want to take a polygraph

A couple years ago, a former FBI special agent named Vincent Sellers released a book called “Eyes Pried Open:  Rookie FBI Agent.”  Sellers had a similar background, an IT guy who wanted to do something bigger.  (Although he was actually a strong runner.)  His book affirmed everything I had thought.  Even for him, the exercise was brutal, including knuckle pushups that left permanent scarring.

Vincent Sellers

Agent Sellers left the FBI after only two years.  He didn’t like the job!  The hours were brutal, and while it had a few exciting moments, it was not that rewarding for him.  Even if I made it through, which I wouldn’t have, I’d probably have hated it too.  Sellers went back to being an IT project manager.

I’ve been blessed to have an amazing career as a network engineer.  I’ve been an in-house engineer, a pre and post sales engineer at a partner, a TAC engineer, and now I work in product management.  I’ve been in telco CO’s, I’ve gotten to play with all kinds of interesting gear, I’ve presented at Cisco Live.  I suppose having a shiny badge to flash would be cool, but the novelty would wear off eventually.  And I’ve never done a single knuckle pushup.

I remember a negative review of the movie La La Land by James Bowman.  I appreciate Bowman because he basically doesn’t like any movie he sees.  He described the characters in La La Land as being successful, materially satisfied, with a lot of friends, and able to drink champagne unendingly without getting drunk.  Of them he says:  “The only thing they don’t have–and the only thing they really want–is fame.”  In other words, they have everything in life except that feeling of importance that comes from the recognition of other people.  My craving for an FBI badge was, to some degree, the same impulse.  Sure there were motives higher than the characters in the movie, but it was largely driven by an inflated sense of self-importance.

Virtually all tech companies (and corporations in general) have in recent years been moved to “create a sense of purpose” for their employees.  In furtherance of this, they create purpose statements of varying degrees of vacuousness, and often with no relevance to the real purpose of the company at all.  The flip side of this is the implication that the real purpose of the company is not meaningful–if it were, there would be no need to concoct purpose statements.  I can tell you that I find meaning and satisfaction in being a network engineer, in itself, and I hope most of my colleagues do as well.  Agent Sellers had his FBI badge for only two years, but I’ve had my Cisco badge for an aggregate total of 11 years, so there must be something to it.

4
by

HPE Buys Juniper

In 2007, I left Cisco after two brutal years in high-touch TAC.  I honestly hated the job, but it was an amazing learning experience.  I draw on my TAC experience every single day.  A buddy of mine got a job at a Gold Partner, offered to bring me in, and I jumped on the opportunity.  Things didn’t go so well, and in 2009, I was laid off and looking for a job again.  That’s when another buddy (buddies help!) called me and told me of an opportunity at Juniper.

I knew little about Juniper.  We had a Juniper SSL box in the network I used to manage, but the routers were mostly for service provider networks.  When I was at TAC, I had one case where a major outage was caused by misconfiguration of a Juniper BGP peer.  But otherwise, I didn’t know a thing.

The opportunity was to be the “network architect” for Juniper’s corporate network.  In other words, to work in internal IT at a network vendor.  It seemed like a good career move, but little did I know I would be thrust the corporate politics at the director-level instead of technical challenges.  I ended up spending six tumultuous years there, with several highlights:

  • My boss disappeared on medical leave on my very first day.
  • I was re-assigned to a Sr. Director who was an applications person and not knowledgeable in networking.  He viewed the network a bit like Col. Kendrick, the Marine, viewed the Navy in the movie A Few Good Men:  “Every time we gotta go some place to fight, you fellas always give us a ride.”
  • I proposed and got buy-off for a program to ensure we actually ran our own gear internally and to ensure we built solid network architectures.
  • I subsequently had the program taken away from me.
  • I found out a job posting with the identical title and JD to mine was listed on Juniper’s public site without my knowledge.
  • My manager was changed to a person two pay grades below me in another country without even informing me.  (Someone noticed it in the directory and told me.)
  • I quit in disgust, without any other job.
  • I was talked into staying.
  • After another year or misery, I was demoted two pay grades myself.
  • I focused on doing the best job I could ended up getting re-promoted to director and left on good terms.

Some of the above was my own fault, much of it was dysfunctional management, some of it was the stupidity we all know lurks in every good size company.  I actually bear Juniper no resentment at all.

I worked at Juniper in the pre-Mist days, and in the midst of the fiscal crisis that began in 2008.  We went from CEO Kevin Johnson’s rah-rah “Mission10” pep rallies that we would be the “next $10B company” (uh, no), to draconian OpEx cuts when a pump-and-dump “activist investor” took over our board.

At the time I was there, Juniper made some mistakes.  NetScreen firewalls had done well for us, but then we made the decision to kill the NetScreen in favor of the JunOS-based SRX.  This is the classic mistake of product management–replace a successful, popular product with a made-from-scratch product with no feature parity.  There were some good arguments to do SRX, but it was done abruptly which signalled EOL to NetScreen customers, and SRX didn’t even have a WebUI.

We also did QFabric while I was there.  We installed one of these beasts in a data center on campus.  I have no idea if they improved it, but the initial versions took a full day to upgrade.  Imagine taking a day-long outage on your data center just to do an upgrade!

Another product that didn’t work out was Space.  JunOS Space came out at the time when the iPhone was still new.  Juniper borrowed the idea.  Instead of building an NMS product, we’d build a platform, and then software developers could build apps on top of it.  Cisco might be able to get away with that approach, but Juniper didn’t have enough of the networking market to attract developers.

In addition, a bunch of other acquisitions fizzled out, including Trapeze, our WAN accelerator, our load balancer.

All that said, Juniper had some fine products when I worked there.  (And believe me, my current employer has had many failures too.)  I got my JNCIE-SP, working on MX routers, which were a really good platform.  I thought the EX switches were decent.  And the operating system was nicely done.  Funnily enough, I worked a solid year on the JNCIE and promptly went to Cisco.  I never renewed it and now it’s expired.

I left after meeting with a strategy VP and explaining our mission to use Juniper’s corporate network to demonstrate how to build an enterprise network to our customers.  She looked at me (and the CIO) and said, “Juniper is done with enterprise networking.  I’m not interested.”  I left after that.  In her defense, Mist was years off and she couldn’t have seen it coming.

She was right, in that Juniper certainly had a core SP market.  Juniper came about at the time when Cisco was still selling 7500’s and 12000’s to its service provider customers, dated platforms running a dated OS.  Juniper did such a nice job with their platform that Cisco had to turn around and build the CRS-1 and IOS-XR, both of which had, ehm, similarities to Juniper’s products.  Juniper really couldn’t crack the enterprise market while I was there.  The lack of a credible wireless solution was always a problem.  Obviously Mist changed the game for them.

Juniper always felt like a scrappy anti-Cisco when I was there, but it was fast becoming corporatized and taken over by the MBAs.  Many old-schoolers would tell me how different things were in the startup days.  It still always had the attitude of an anti-Cisco.  One of our engineers ALWAYS referred to Cisco devices as “Crisco boxes”, and when I announced I was returning to Cisco, a long-time IT guy called me an “asshole”.  A couple funny stories around this:

A customer came in to our office for training and looked in the window of one the data centers nearby.  He saw it was packed with Cisco gear and subsequently published a video on social media captioned “Juniper uses Cisco.”  He didn’t realize that we leased the building from another company called Ariba, and the data center was theirs, not ours.  In fact, we worked very hard to not run Cisco in our internal network.  Juniper subsequently asked Ariba to block out the window.

One time we solicited a proposal from one of our largest service provider customers to host a data center for us.  The SP came back to us with an architecture which was 100% Cisco.  Cisco switches, Cisco routers, Cisco firewalls.  I told the SP I would never deploy our DC on Cisco gear.  What if a major bug hit Cisco devices causing outages and our data center went down too?  What if we got hacked due to a Cisco PSIRT and it became public?

The SP didn’t care.  We were their customer, but they were also ours.  They used Cisco in their data center, and had no desire to re-tool for another vendor.  I escalated all the way to the CEO, who agreed with me, and the deal was scuttled.  Ironically, I used this story in my Cisco interviews when asked for an example of a time when I had taken a strong stand on something.

I work at Cisco now, and even ran the competitive team for a while.  Competition is healthy and makes us all better.  I actually value our competition.  Obviously my job is to win deals against them, but I have friends who work at Juniper and I have friends who work at HPE.  We’re all engineers doing our jobs, and I wish them no ill will.  I always respected Juniper, their engineering, and their scrappy attitude.  While I know some of this will be retained as they get absorbed into a large corporation, it’s definitely the end of an era, for the industry and for me.

39
1
by

Goodbye, Building K

I haven’t written anything for a while, because of the simple fact that I had nothing to say.  The problem with being a writer is that sometimes you have nothing to write.  I also have a day job, and sometimes it can keep me quite busy.  Finally, an afternoon drive provided some inspiration.

There’s a funny thing about the buildings I work in–they all tend to be purchased by Google.  When I started at Juniper, I worked in their Ariba campus in Mountain View, several buildings they rented from that software company.  We were moved to Juniper’s (old) main campus, on Mathilda Drive, and the old Ariba buildings were bought and re-purposed by Google.  Then the Mathilda campus was bought by Google.

When I worked in TAC, from 2005-2007, I worked in building K on Tasman Drive in San Jose.  Back then, the meteoric growth of Cisco was measured by the size of its campus, which stretched all along Tasman, down Cisco way, and even extended into Milpitas.

Cisco’s campus has been going the opposite direction for a while now.  The letter buildings (on Tasman, West of Zanker Street) started closing before the COVID lockdowns changed everything.  Now a lot of buildings sit empty and will certainly be sold, including quite possibly the ones in Milpitas, where I work.

Building K closed, if not sometime during the lockdowns, shortly after.  I hadn’t driven by it in months, and when I did yesterday-lo and behold!-it was now a Google building!

What used to be building K

It’s funny how our memories can be so strongly evoked by places.  Building K was, for a long time, the home to Cisco TAC.  I vividly remember parking on Champion Drive, reviewing all of my technological notes before going in to be panel-interviewed by four tough TAC engineers.  I remember getting badged in the day I started, after passing the interview, and being told by my mentor that he wouldn’t be able to put me “on the queue” for three months, because I had so much to learn.

Two weeks later I was taking cases.  Not because I was a quick study, but because they needed a body.

I worked in High Touch Techical Support, dealing with Cisco’s largest customers.  The first team I was on was called ESO.  Nobody knew what it stood for.  The team specialzied in taking all route/switch cases for Cisco’s large financial customers like Goldman Sacks and JPMC.  Most of the cases involved the Cat 6k, although we supported a handful of other enterprise platforms.

When a priority 1 case came in, the Advanced Services Hotline (ASH) call center agents would call a special number that would cause all of the phones on the ESO team to play a special ring tone.  I grew to develop a visceral hatred of that ring tone.  Hearing it today would probably trigger PTSD.  I’d wait and wait for another TAC engineer (called CSEs) to answer it.  If nobody did, I’d swallow hard and grab the phone.

The first time I did it was a massive multicast meltdown disrputing operations on the NYSE trading floor.  I had just gotten my CCIE, but I had only worked previosly as a network engineer in a small environment.  Now I was dealing with a major outage, and it was the first time I had to handle real-world multicast.  Luckily, my mentor showed up after 20 minutes or so and helped me work the case.

My first boss in HTTS told me on the day I started, “at Cisco, if you don’t like your boss or your cubicle, wait three months.”  Three months later I had a new boss and a new cubicle.  The ESO team was broken up, and its engineers dispersed to other teams.  I was given a choice:  LAN Switch or Routing Protocols.  I chose the latter.

I joined the RP-LSA team as a still new TAC engineer.  The LSA stood for “Large Scale Architectures.”  The team was focused on service provider routing and platform issues.  Routing protocol cases were actually a minority of our workload.  We spent a lot of time dealing with platform issues on the GSR 12000-series router, the broadband aggregation 10000-series, and the 7500.  Many of the cases were crashes, others were ASIC issues.  I’d never even heard of the 12k and 10k, now I was expected to take cases and speak with authority.  I leaned on my team a lot in the early days.

Fortunately for me, these were service provider guys, and they knew little about enterprise networking or LAN switching.  With the breakup of the ESO team, the large financials were now coming into the RP-LSA queue.  And anyone who has worked in TAC can tell you, a routing protocols case is often not an RP case at all.  When the customer opens a case for flapping OSPF adjacencies, it’s usually just a symptom of a layer 2 issue.  The SP guys had no clue how to deal with these, but I did, so we ended up mutually educating each other.

In those days, most of the protocol cases were on Layer 3 MPLS.  I had never even heard of MPLS before I started there, but I did a one week online course (with lab), and started taking cases like a champ.  MPLS cases were often easily because it was new, but usually when a large service provider like AT&T, Orange, or Verizon opens a case on soemthing like BGP, it’s not because they misconfigured a route map.  They’ve looked at everything before opening the case, and so the CSE becomes essentially a middleman to coordinate the customer talking to developers.  In many cases the CSE is like a paramedic, stabilizing the patient before the doctor takes over to figure out what is wrong.  We often knew we were facing a bug, but our job was to find workarounds to bring the network back up so developers could find a fix.

I had my share of angry customers in those days, some even lividly angry.  But most customers were nice.  These were professional network engineers who understood that the machines we build don’t always act as we expect them to.  Nevertheless, TAC is a high-stress job.  It’s also relentless.  Close one case, and two more are waiting in the queue.  There is no break, no downtime.  The best thing about it was that when you went home, you were done.  If a call came in on an open case in your backlog, it would be routed to another engineer.  (Though sometimes they routed it back to you in the morning.)  In HTTS, we had the distinct disadvantage of having to work cases to resolution.  If the case came in at 5:55pm on Friday night, and your shift ended at 6pm, you might spend the next five hours in the office.  Backbone TAC engineers “followed the sun” and re-assigned cases as soon as their shift ended.

I make no secret of the fact that I hated the job.  My dream was to work at Cisco, but shortly after I started, I wanted out.  And yet the two years I spent in TAC are two of the most memorable of my career.  TAC was a crucible, a brutal environment dealing with nasty technical problems.  The fluff produced by marketeers has no place there.  There was no “actualize your business intent by optimizing and observing your network”-type nonsense.  Our emails were indeciperhable jumbles of acronyms and code names.  “The CEF adjacency table is not being programmed because the SNOOPY ASIC has a fault.”  (OK, I made that up…  but you get the point.)  This was not a place for the weak-minded.

When things got too sticky for me, I could call in escalation engineers.  I remember one case where four backbone TAC escalation engineers and one from HTTS took over my cube, peering at my screen and trying to figure out what was going on during a customer meltdown.

Building K was constructed with the brutalist-stlye of architecture so common in Silicon Valley.  One look at the concrete and glass and the non-descript offices and conference rooms is enough to drain one’s soul.  These buildings are pure function over form.  They are cheap to put up and operate, but emotionally crushing to work in.  There is no warmth, even on a winter day with the heat on.

Still, when I look at building K, or what’s become of it, I think of all the people I knew there.  I think of the battles fought and won, the cases taken and closed, the confrontational customers and the worthless responses from engineering, leaving us unable to close a case.  I think of the days I would approach that building in dread, not knowing what hell I would go through that day.  I also think of the incredible rush of closing a complex case, of finding a workaround, and of getting an all-5’s “bingo” (score) from a customer.  TAC is still here, but for those of us who worked in building K, its closure represents the end of an era.

8
by

The War on Expertise

Like many network engineers, I quickly fell in love with my field and worked hard to master it.  I got into networking when I was working in desktop support.  The behind-the-scenes stuff was way more interesting to me than the front lines.  Back in the late nineties, I bought a library of books to learn this field.  Perlman, Comer, and Stevens were the classics.  I rounded it out with blue-and-white Cisco Press books by Doyle, Peplnjak, Williamson, and many others.  I studied these books religiously, read through config guides, and practiced in the lab.

The network engineers on my team and I loved to debate the arcana of this mysterious field.  We always tried to one-up each other, learning new technologies, new protocols, and attaining new technical certifications.  I’ve worked with engineers who are smarter than I am, and better than I am, but that always motivated me to learn more.

I bring this up because I’ve had multiple conversations with multiple execs, for many years, in which they seem to decry the virtue of expertise.  Network engineers “revel in complexity”, they don’t realize their time has passed, the build networks that need “armies of CCIEs to maintain”, and they hate simplicity.  If only the pesky network engineers would get out of the way, the glorious MBAs could build us simple and elegant products, which is how the industry is going, don’t you know!

In short, our industry is suffering through a war on expertise.  Those arcana we love to master have put a target on our back.  If you want to learn those things, you must be reveling in complexity.  Go find something else to do, ChatGPT will replace you!

The first mistake in this line of thinking is the assumption that network engineers want to build networks that are complex.  We actually don’t.  A couple of anecdotes:

When I was working for a Gold partner, I was sent to help out an IT manager of a rather small company, only four sites.  She had contracted VPLS from two service providers, and asked me to implement a complex load balancing scheme she had conceived.  I begged her not to make me do it, but she insisted.  I ended up building a functional mess, a combination of PBR and EIGRP offset access-lists.  Man, was it ugly.  But it worked.  Then I got laid off from the partner, and a year later she was calling me, begging me to come back because nobody could figure out how it worked.  I didn’t want to build something that ugly and she didn’t need it.

Second anecdote.  My wife had to go in for a surgical procedure a few years ago.  We went to the best doctor in San Francisco.  When he got into the procedure he found that her anatomy is not conventional, and it was a very difficult procedure.  In the recovery room, he told us most doctors would have stopped.  My wife wanly smiled and said, “well, I’m sure you like a challenge.”  He looked back at her and said, “no, we like it when it’s easy.”

I think this is where the execs misunderstand expertise.  99.9% of the time, your airline flight could be handled by a low-time pilot who can work the automation systems.  But when the engines fail, you want Sully at the controls.  Just because some people understand complexity and study difficult concepts, it doesn’t follow that they want complexity.  When I administered networks, I wanted it to be easy.  But I was ready for when it was hard.

The war on expertise seems to me to be a war on the human spirit.  The CCIE exam, whatever you think of it, was a heck of a challenge, and passing it was one of the proudest days of my life.  Human beings want to learn, to grow, to push their limits, and to test themselves.  That’s why we spend hours in the lab.  We should encourage this behavior.  We should want people in our business who seek subject matter expertise and mastery.  We can make things simpler, fine, but we should still encourage expertise.

At the end of the day, networks are inherently complex.  A network is a large distributed system, connecting numerous devices running numerous operating systems over diverse transport mechanisms using a wide variety of protocols.  You can simplify the protocols a bit, but ultimately most simplification of networks is done one of two ways:  reducing the number of choices an administrator can configure, or abstracting and hiding the underlying complexity.  In the first case, you may close out necessary use cases.  In the case of abstraction, well, it works great until something breaks.  Then you need to call a network engineer.

I’m not in any way saying the new tools, from programmability to automation systems like Ansible, to “controllers” are unnecessary.  Far be it.  Any tool that makes an engineer’s job easier will be embraced by engineers.  I am saying that we need to stop blaming complexity on those who manage to understand it.

28
by

Cleared for nothing

I wrote this post on Feb 20, 2020, and I always thought it was an entertaining episode.  FBI Special Agent Elvis Chan, who features prominently in the post, has been in the news lately as he played a major role in the Twitter Files.  I will stay out of politics, except to note that Elvis was indeed a liaison to the business community, as seen here.

I was working at Juniper when the CIO asked me to apply for a government security clearance.  There were a number of hacking attempts on our network, and a security clearance would make me eligible for briefings from the government on the nature and scope of the threats against the United States’ networks.  Being one of the few US citizens in our department, and having a security background, it made sense.

I met with our “FSO”, the on-site liaison to the clearance-granting agency, in this case the Department of Defense.  I’ll call him Billy.  Billy pointed me to the government web site which housed the application, called “OPM”.  The OPM application was extensive, requiring me to input huge amounts of information about myself and my family.  It required a bit of work to track down some of the information, and when I printed the PDF copy of the application it totaled around eighty pages.

One day Billy called me into his office and told me I had been awarded a secret clearance.  He let me know that I could be subject to the death penalty if I divulged any classified information.  I signed some documents, and that was it. “Don’t I get a card for my wallet or anything?” I asked Billy.  He just smiled.

Shortly after getting my clearance, one of our other cleared employees brought me into a secure office in one of Juniper’s buildings where we could look at classified information.  He pulled a secured laptop out of a locked drawer, and a password out of a sealed envelope.  We began perusing classified information.  None of it was relevant to us, and none of it was particularly memorable.  For example, we read an article about several criminal gangs, the existence of which was unclassified.  The only classified information in the article happened to be the names of particular gangs.  They didn’t mean much to me, and I probably forgot them within a day or two.

One day I was invited to the San Francisco FBI office, to receive a classified briefing.  Billy had to fax the clearance over, because the DoD and FBI didn’t have an electronic way to exchange clearances.  I showed up, excited, to the federal building in San Francisco and proceeded up to the floor where the briefing was to take place.  Nobody was there.  I wandered around the white hallway with locked doors unable to make contact with anyone.  The elevator opened after a few minutes, and another equally confused attendee emerged.  We were wandering around for several minutes before someone showed up and told us to go to a different floor.

On the new floor a couple of young-looking FBI agents setup a table, checked our ID’s, and then took our cell phones.  The security did not seem very rigorous.  They then admitted us to the SCIF, or Sensitive Compartmented Information Facility.  The room we were led into was just a conference room, with a low ceiling and no windows.  Another young-looking FBI agent approached me, wearing a tie but no coat.  “Hi, I’m Elvis,” he said.

“I’m a special agent and the coordinator of the briefing today.  We’re very excited to have you here.”

We had a brief conversation about my job and role, and then I asked to use the bathroom.

“Go out the back door of the SCIF and hang a right, he said.”

I did this, and found myself walking with a wall on my right, and a row of waist-level cubicles on my left.  Nobody was in the the cubes, but paperwork was sitting on most of the desks. I wanted to peer at the paperwork as I walked by.  I have a clearance, I figured, so if I had a right to at least take a peek and see if the names of anyone I knew appeared.  Unfortunately, without pausing and staring, a chance I didn’t want to take, I couldn’t read anything.

I found the bathroom, and as I was participating in nature’s call, a couple of guys came in wearing ties but no sport coats.  They each had side-arms on their belts.  I wondered why these agents, who are basically office workers, needed to walk around armed.

As I came out of the bathroom, a female FBI agent was standing there, tapping her foot in anticipation of my emergence.  She looked like my school librarian.  Diminutive in stature, she had a side-arm that looked as big as she was.

“Are you FBI?” she asked pointedly.

“No,” I replied, thinking the answer was obvious.

She let out a long sigh, looking like a satisfied cop who has caught a perp.  “You can’t be here without an escort,” she scolded me.

“But Elvis told me I could!” was my retort.  I had a sudden realization that, in a large FBI office like San Francisco’s, it was entirely possible that not every FBI agent knew every other FBI agent, and that my host agent may have been entirely unknown to her.  Here I was, by myself, in the inner sanctum of an FBI office, explaining to an armed federal agent that I happened to be there because Elvis had sent me.

Fortunately, a glimmer of recognition flashed across her stern countenance.  “Oh, Elvis!” she said, exasperated.  “Come on,” she snapped, and led me back to the SCIF.

Back in the SCIF, the briefing began.  The first presenter was an FBI agent wearing a tie, with a coat this time.  Whatever he had learned at the FBI training center in Quantico, VA apparently did not include the fundamentals of haberdashery.  Anyone who buys a suit knows that you immediately have it tailored, as the pant legs are way too long.  Apparently this agent bought his cream-colored suit, with piping, and never sent it for alterations.  The trouser legs were so long he was actually walking on the bottom of his pant legs.  His presentation was no better than his tailoring.  Presenting on computer security, it was clear this was not somebody with even a basic knowledge of computing.

After him, two Homeland Security analysts presented.  They wore rumpled khakis with jacket and tie, and sported similar pyramid mustaches.  They presented on SCADA systems, a subject I could care less about.  Almost all of it was unclassified.

Shortly after my briefing, I learned that the OPM database had been hacked by the Chinese military.  All the personal information about myself and my family is in their hands now.  When I left Juniper, Cisco declined to renew my security clearance.

Some people hide that they have/had a clearance, as they can be targeted by foreign governments.  Personally, I don’t care.  What little classified information I saw, I can’t remember.  You could waterboard me and I wouldn’t be able to tell you a thing.

3
by

Plain Language

I don’t know about the rest of the world, but here in America our police speak in a sort of code language.  Instead of saying “he got out of the car and walked away,” the police will say, “the subject exited the vehicle and proceeded on foot.”  It’s not that their language is any clearer–in fact it’s less clear.  When talking to each other, the cops like to use language like this because it seems to elevate them and make them sound more professional.

A friend of mine told me a story of his friend who had always wanted to be a cop, but was too much of a screw-up to make it to the academy.  Despite his failure he remained a police buff, perhaps a cop in his own mind.  This fellow witnessed a crime and the local sheriff showed up.  This non-cop started to describe the crime to the sheriff as cops do:  “The subject exited the vehicle and proceeded to commit a 4-15…”  The deputy cut him off and shouted, “speak English, boy!”  The poor police-wannabe never lived it down.

Corporatist-types have a language like this too.  Attempting to sound smart and professional, they speak in an often-inaccessible code language replete with b-school buzz words.  I must admit, as long as I’ve been in the corporate world, I’ve frequently been confronted by language that I simply couldn’t understand.

“We collaborated with engineering and cross-functional product managers across multiple time zones to groom and prioritize backlog to ensure efficient program delivery.”  Huh?  “We need to build a motion that creates value at scale.”  What? “The adoption journey enables us to innovate continuously.”  Speak English, boy!

Then there was this gem, from an MBA describing a customer problem:  “The customer is not in the mindset of extracting value from the product.”  A lot of words there.  How about three words:  “It don’t work.”  Or, “customer hates it.”  Oh no.   If the MBA spoke that way, he’d sound like he learned nothing at his prestigious business school.  Professionals speak professionally, you see.  If he spoke like a normal human being, some people might suspect he actually doesn’t really know anything.  Although to be honest, that’s exactly what I was suspecting when he started talking about mindsets.

We can all fall into this trap, I’m afraid.  I refused, for years, to use “ask” and “spend” as nouns, because they’re not nouns.  (I remember an internal thread at Cisco years back in which someone said, “shouldn’t we productize that?”  The snarky response came back from an engineer, “no because at Cisco we can’t turn nouns into verbs.”)  Alas, I’ve surrendered to the progress of business-speak and have replaced “request” with “ask.”  Saving one syllable with a frequently-used word has certainly given me hours back to do other things, don’t you think?

Technical people can certainly fall into this and we have our own jargon.  Some of it is necessary.  Here is a snippet of a Cisco doc:  “To overcome the limitations of the flood-and-learn VXLAN as defined in RFC 7348, organizations can use Multiprotocol Border Gateway Protocol Ethernet Virtual Private Network (MP-BGP EVPN) as the control plane for VXLAN. ”  This is wordy, and it is jargony.  That said, I can’t think of a better way to say it.  This sort of language is unavoidable for network engineers.

What I don’t like is technical people adopting MBA-speak because they’re surrounded by it.  “Our latest release provides flexible options to operationalize your business intent.”  Oh dear, even if you get into some good technical meat, you’ve lost me already.  The simple secret for me in winning Cisco Live Hall of Fame for my speaking is simply to state things in plain, clear language.  Technical, yes, but clear.

I used to think I was stupid, sitting in meetings in the corporate world and not understanding what on Earth people were saying.  Then I learned that in many cases, the speakers didn’t understand what they were saying either.  In the event that they actually do, a few pointed questions can usually cut through the fog of fancy words.  I’m convinced many of the mistakes made in the corporate world would never happen if people actually spoke like normal people.

If you feel tempted to obscure your language to sound like you’re oh-so-smart, remember the advice of the deputy sheriff:  Speak English, boy!

8
by

Dear Diary

My blogging has been a little slow of late.  First there was Cisco Live.  Then there was the post-Cisco Live slump of not wanting to do anything.  Then there was the Cisco’s fiscal year-end crunch along with some very hot projects.  Then there was the departure of one of our key execs, and subsequent excitement.  Then one of my direct reports fell gravely ill, and there was both the stress of that along with the burden of picking up the management of his team.  Then there is simply the fact that, in order to blog, one has to come up with ideas.  And often I don’t have any.

So, let me get back into it with a bit of a diary entry.  Cisco Live was back in person for the first time since COVID began.  (And despite the rigid COVID protocols, tons of people got it, hmmm.)  Cisco Live, as I’ve said before, is not an easy show to put on.  It’s a massive effort, and we had not done it for two years.  Some of the key people who used to organize it left, and those of us who had done it before had some muscles atrophy.  It was not, shall we say, smooth.  But it happened, and seeing fellow network folks again in person made me realize many things.  Like the fact that I am the only male American network engineer who does not own a pair of cargo shorts.

It also made me remember the camaraderie of our industry, which is half the fun.  One of my hobbies is electrical work (not as dangerous as it sounds, usually), and there is a YouTube electrician who shot some videos at an electrician’s convention at Mandalay, the same convention center where we host Cisco Live.  I’m sure he hung out at the same places, felt the sense of camaraderie that only electricians have, etc.  There’s always a great feeling being a member of a club, whatever that club may be.  Speaking your own language, reminiscing on how things were in the past, and perhaps about how the youngsters never had to configure dialer maps or CSU/DSUs.  Our club, the people who gravitate towards network engineering, is a special bunch.  Perhaps that electrician can same the same thing.  Perhaps I’m a bit sentimental given one of my colleagues is ill.  All I can say is, I missed everyone and enjoyed being back in person.

For the first time in a while I failed to win a distinguished speaker award.  Cisco Live audiences humble the proud.  I delivered my session on the “CCIE in an SDN world”, but the old 90 minute session was crammed into 45 minutes because of the format change.  I couldn’t quite get the rhythm right.  Also, I had to deliver it at 8am.  I’m just not a morning person, and I’m zero for zero with DS awards delivering in that time slot.  I also had only 40 people.  I used to pack in 500.  It was a smaller show, but the audience was small nonetheless.  I was (finally) inducted into the Cisco Live Hall of Fame two years after I qualified, but perhaps I’m like an old movie star, washed up and waning in popularity.  I actually like how challenging the audiences are, however.  They keep you on your toes and force you to never get complacent.

Another question is whether the message resonates any more.  When I started doing this session, it was amidst the barrage of messaging from vendors (like us) that networking was changing forever, that AI and automation and intuitive networks would end network engineering as we know it.  My session basically said: well, it’s not quite the case.  You can automate, but good luck if you don’t understand what you automate.  Several years on, I wonder if the message is just so apparent that it’s not as needed as it was before.

Still, Cisco Live is my favorite place to present.  I do a ton of internal-facing presentations, many to executives, and each one is written, reviewed, and delivered by committee.  There’s no freedom in it at all.  The words and messages are tightly controlled.  What I love about Cisco Live is that, as an established speaker, I can pretty much present whatever I want.  I can build my own slides.  I don’t even need much of a review.

I’ll leave it there.  A rambling, diary post, perhaps not of a lot of interest.  At least it got me blogging again!

1
by

The type A conundrum

An old theory of personality holds that people fall into two types–A and B.  Put simply, Type A personalities are highly aggressive and competitive, whereas Type B are not.  We all have seen this broad difference in personalities.  Some people we encounter seem ready to walk over their own grandmothers to get ahead.  Like all stereotypes, this is a gross oversimplification, but there’s a lot of truth in it.

In the corporate world, type A personalities tend to rise to the top.  Why?  Because their very personality is aggressive and competitive.  They like to push, push, push for what they want and are willing to drive their agenda at any cost.  They frequently are talkers but rarely listeners.  They also judge people through their own lens.  If you’re an introvert, quiet, or deliberative, if you’re a listener instead of a talker, they think you are not “driven” and probably not worth listening to or promoting.

The question is:  does being type A make you right?  Does it make your opinions more valuable?  I cannot think of any reason why being aggressive and competitive makes you more likely to be correct about anything.  In fact, I think the opposite is true.  If you don’t listen well and are always pushing your own agenda, you’re less likely to consider the opinions of others, which means your decision-making is less well-rounded.

I’ve pointed out before, that quiet, deliberative people, type B’s, are often the ones you really want to listen to.  However, in the corporate world, they are often kicked to the curb.  “He never says anything in meetings,” the type A’s say.  Well, if you hired him maybe he’s actually a smart person but has a hard time contributing in a meeting with 20 people all talking fast.  Maybe he needs time to digest what he heard before providing recommendations.

Type A’s tend to be in positions of power not necessarily because they are smarter but because they fight for position in hierarchies.  This is not to say they are without value.  Their decisiveness and drive are very important to a healthy organization.  They can break indecision and move companies forward in ways type B’s cannot.

The key for both personality types is the old Greek maxim:  know thyself.  If you’re type A, you need to be careful not to be too aggressive.  Listen to your quieter colleagues, accept that they may have a different personality, and meet them where they are at.  Call on them in meetings, give them time to deliberate and come back to you.

For type B’s, you need to learn to speak out more.  You’re probably more respected than you realize, and when you do speak, you’re probably listened to.  Try to find forums that are more comfortable for you, like expressing your opinion in writing or 1:1’s.

Sadly, because of the cutthroat nature of the business world, I see little self-awareness and frequent domination of businesses by type A’s.  At the end they may get people to follow them, but if they’re leading you off a cliff, their drive may not be such a good thing.

1