Reflections

Goodbye, Building K

Posted by ccie14023 on July 28, 2023
Posted in: Reflections. Tagged: , , , . Leave a Comment

I haven’t written anything for a while, because of the simple fact that I had nothing to say.  The problem with being a writer is that sometimes you have nothing to write.  I also have a day job, and sometimes it can keep me quite busy.  Finally, an afternoon drive provided some inspiration.

There’s a funny thing about the buildings I work in–they all tend to be purchased by Google.  When I started at Juniper, I worked in their Ariba campus in Mountain View, several buildings they rented from that software company.  We were moved to Juniper’s (old) main campus, on Mathilda Drive, and the old Ariba buildings were bought and re-purposed by Google.  Then the Mathilda campus was bought by Google.

When I worked in TAC, from 2005-2007, I worked in building K on Tasman Drive in San Jose.  Back then, the meteoric growth of Cisco was measured by the size of its campus, which stretched all along Tasman, down Cisco way, and even extended into Milpitas.

Cisco’s campus has been going the opposite direction for a while now.  The letter buildings (on Tasman, West of Zanker Street) started closing before the COVID lockdowns changed everything.  Now a lot of buildings sit empty and will certainly be sold, including quite possibly the ones in Milpitas, where I work.

Building K closed, if not sometime during the lockdowns, shortly after.  I hadn’t driven by it in months, and when I did yesterday-lo and behold!-it was now a Google building!

What used to be building K

It’s funny how our memories can be so strongly evoked by places.  Building K was, for a long time, the home to Cisco TAC.  I vividly remember parking on Champion Drive, reviewing all of my technological notes before going in to be panel-interviewed by four tough TAC engineers.  I remember getting badged in the day I started, after passing the interview, and being told by my mentor that he wouldn’t be able to put me “on the queue” for three months, because I had so much to learn.

Two weeks later I was taking cases.  Not because I was a quick study, but because they needed a body.

I worked in High Touch Techical Support, dealing with Cisco’s largest customers.  The first team I was on was called ESO.  Nobody knew what it stood for.  The team specialzied in taking all route/switch cases for Cisco’s large financial customers like Goldman Sacks and JPMC.  Most of the cases involved the Cat 6k, although we supported a handful of other enterprise platforms.

When a priority 1 case came in, the Advanced Services Hotline (ASH) call center agents would call a special number that would cause all of the phones on the ESO team to play a special ring tone.  I grew to develop a visceral hatred of that ring tone.  Hearing it today would probably trigger PTSD.  I’d wait and wait for another TAC engineer (called CSEs) to answer it.  If nobody did, I’d swallow hard and grab the phone.

The first time I did it was a massive multicast meltdown disrputing operations on the NYSE trading floor.  I had just gotten my CCIE, but I had only worked previosly as a network engineer in a small environment.  Now I was dealing with a major outage, and it was the first time I had to handle real-world multicast.  Luckily, my mentor showed up after 20 minutes or so and helped me work the case.

My first boss in HTTS told me on the day I started, “at Cisco, if you don’t like your boss or your cubicle, wait three months.”  Three months later I had a new boss and a new cubicle.  The ESO team was broken up, and its engineers dispersed to other teams.  I was given a choice:  LAN Switch or Routing Protocols.  I chose the latter.

I joined the RP-LSA team as a still new TAC engineer.  The LSA stood for “Large Scale Architectures.”  The team was focused on service provider routing and platform issues.  Routing protocol cases were actually a minority of our workload.  We spent a lot of time dealing with platform issues on the GSR 12000-series router, the broadband aggregation 10000-series, and the 7500.  Many of the cases were crashes, others were ASIC issues.  I’d never even heard of the 12k and 10k, now I was expected to take cases and speak with authority.  I leaned on my team a lot in the early days.

Fortunately for me, these were service provider guys, and they knew little about enterprise networking or LAN switching.  With the breakup of the ESO team, the large financials were now coming into the RP-LSA queue.  And anyone who has worked in TAC can tell you, a routing protocols case is often not an RP case at all.  When the customer opens a case for flapping OSPF adjacencies, it’s usually just a symptom of a layer 2 issue.  The SP guys had no clue how to deal with these, but I did, so we ended up mutually educating each other.

In those days, most of the protocol cases were on Layer 3 MPLS.  I had never even heard of MPLS before I started there, but I did a one week online course (with lab), and started taking cases like a champ.  MPLS cases were often easily because it was new, but usually when a large service provider like AT&T, Orange, or Verizon opens a case on soemthing like BGP, it’s not because they misconfigured a route map.  They’ve looked at everything before opening the case, and so the CSE becomes essentially a middleman to coordinate the customer talking to developers.  In many cases the CSE is like a paramedic, stabilizing the patient before the doctor takes over to figure out what is wrong.  We often knew we were facing a bug, but our job was to find workarounds to bring the network back up so developers could find a fix.

I had my share of angry customers in those days, some even lividly angry.  But most customers were nice.  These were professional network engineers who understood that the machines we build don’t always act as we expect them to.  Nevertheless, TAC is a high-stress job.  It’s also relentless.  Close one case, and two more are waiting in the queue.  There is no break, no downtime.  The best thing about it was that when you went home, you were done.  If a call came in on an open case in your backlog, it would be routed to another engineer.  (Though sometimes they routed it back to you in the morning.)  In HTTS, we had the distinct disadvantage of having to work cases to resolution.  If the case came in at 5:55pm on Friday night, and your shift ended at 6pm, you might spend the next five hours in the office.  Backbone TAC engineers “followed the sun” and re-assigned cases as soon as their shift ended.

I make no secret of the fact that I hated the job.  My dream was to work at Cisco, but shortly after I started, I wanted out.  And yet the two years I spent in TAC are two of the most memorable of my career.  TAC was a crucible, a brutal environment dealing with nasty technical problems.  The fluff produced by marketeers has no place there.  There was no “actualize your business intent by optimizing and observing your network”-type nonsense.  Our emails were indeciperhable jumbles of acronyms and code names.  “The CEF adjacency table is not being programmed because the SNOOPY ASIC has a fault.”  (OK, I made that up…  but you get the point.)  This was not a place for the weak-minded.

When things got too sticky for me, I could call in escalation engineers.  I remember one case where four backbone TAC escalation engineers and one from HTTS took over my cube, peering at my screen and trying to figure out what was going on during a customer meltdown.

Building K was constructed with the brutalist-stlye of architecture so common in Silicon Valley.  One look at the concrete and glass and the non-descript offices and conference rooms is enough to drain one’s soul.  These buildings are pure function over form.  They are cheap to put up and operate, but emotionally crushing to work in.  There is no warmth, even on a winter day with the heat on.

Still, when I look at building K, or what’s become of it, I think of all the people I knew there.  I think of the battles fought and won, the cases taken and closed, the confrontational customers and the worthless responses from engineering, leaving us unable to close a case.  I think of the days I would approach that building in dread, not knowing what hell I would go through that day.  I also think of the incredible rush of closing a complex case, of finding a workaround, and of getting an all-5’s “bingo” (score) from a customer.  TAC is still here, but for those of us who worked in building K, its closure represents the end of an era.

7

The War on Expertise

Posted by ccie14023 on March 7, 2023
Posted in: Reflections. Tagged: , , , , . Leave a Comment

Like many network engineers, I quickly fell in love with my field and worked hard to master it.  I got into networking when I was working in desktop support.  The behind-the-scenes stuff was way more interesting to me than the front lines.  Back in the late nineties, I bought a library of books to learn this field.  Perlman, Comer, and Stevens were the classics.  I rounded it out with blue-and-white Cisco Press books by Doyle, Peplnjak, Williamson, and many others.  I studied these books religiously, read through config guides, and practiced in the lab.

The network engineers on my team and I loved to debate the arcana of this mysterious field.  We always tried to one-up each other, learning new technologies, new protocols, and attaining new technical certifications.  I’ve worked with engineers who are smarter than I am, and better than I am, but that always motivated me to learn more.

I bring this up because I’ve had multiple conversations with multiple execs, for many years, in which they seem to decry the virtue of expertise.  Network engineers “revel in complexity”, they don’t realize their time has passed, the build networks that need “armies of CCIEs to maintain”, and they hate simplicity.  If only the pesky network engineers would get out of the way, the glorious MBAs could build us simple and elegant products, which is how the industry is going, don’t you know!

In short, our industry is suffering through a war on expertise.  Those arcana we love to master have put a target on our back.  If you want to learn those things, you must be reveling in complexity.  Go find something else to do, ChatGPT will replace you!

The first mistake in this line of thinking is the assumption that network engineers want to build networks that are complex.  We actually don’t.  A couple of anecdotes:

When I was working for a Gold partner, I was sent to help out an IT manager of a rather small company, only four sites.  She had contracted VPLS from two service providers, and asked me to implement a complex load balancing scheme she had conceived.  I begged her not to make me do it, but she insisted.  I ended up building a functional mess, a combination of PBR and EIGRP offset access-lists.  Man, was it ugly.  But it worked.  Then I got laid off from the partner, and a year later she was calling me, begging me to come back because nobody could figure out how it worked.  I didn’t want to build something that ugly and she didn’t need it.

Second anecdote.  My wife had to go in for a surgical procedure a few years ago.  We went to the best doctor in San Francisco.  When he got into the procedure he found that her anatomy is not conventional, and it was a very difficult procedure.  In the recovery room, he told us most doctors would have stopped.  My wife wanly smiled and said, “well, I’m sure you like a challenge.”  He looked back at her and said, “no, we like it when it’s easy.”

I think this is where the execs misunderstand expertise.  99.9% of the time, your airline flight could be handled by a low-time pilot who can work the automation systems.  But when the engines fail, you want Sully at the controls.  Just because some people understand complexity and study difficult concepts, it doesn’t follow that they want complexity.  When I administered networks, I wanted it to be easy.  But I was ready for when it was hard.

The war on expertise seems to me to be a war on the human spirit.  The CCIE exam, whatever you think of it, was a heck of a challenge, and passing it was one of the proudest days of my life.  Human beings want to learn, to grow, to push their limits, and to test themselves.  That’s why we spend hours in the lab.  We should encourage this behavior.  We should want people in our business who seek subject matter expertise and mastery.  We can make things simpler, fine, but we should still encourage expertise.

At the end of the day, networks are inherently complex.  A network is a large distributed system, connecting numerous devices running numerous operating systems over diverse transport mechanisms using a wide variety of protocols.  You can simplify the protocols a bit, but ultimately most simplification of networks is done one of two ways:  reducing the number of choices an administrator can configure, or abstracting and hiding the underlying complexity.  In the first case, you may close out necessary use cases.  In the case of abstraction, well, it works great until something breaks.  Then you need to call a network engineer.

I’m not in any way saying the new tools, from programmability to automation systems like Ansible, to “controllers” are unnecessary.  Far be it.  Any tool that makes an engineer’s job easier will be embraced by engineers.  I am saying that we need to stop blaming complexity on those who manage to understand it.

25

Cleared for nothing

Posted by ccie14023 on December 22, 2022
Posted in: Reflections. Tagged: , , , , , , , . 1 comment

I wrote this post on Feb 20, 2020, and I always thought it was an entertaining episode.  FBI Special Agent Elvis Chan, who features prominently in the post, has been in the news lately as he played a major role in the Twitter Files.  I will stay out of politics, except to note that Elvis was indeed a liaison to the business community, as seen here.

I was working at Juniper when the CIO asked me to apply for a government security clearance.  There were a number of hacking attempts on our network, and a security clearance would make me eligible for briefings from the government on the nature and scope of the threats against the United States’ networks.  Being one of the few US citizens in our department, and having a security background, it made sense.

I met with our “FSO”, the on-site liaison to the clearance-granting agency, in this case the Department of Defense.  I’ll call him Billy.  Billy pointed me to the government web site which housed the application, called “OPM”.  The OPM application was extensive, requiring me to input huge amounts of information about myself and my family.  It required a bit of work to track down some of the information, and when I printed the PDF copy of the application it totaled around eighty pages.

One day Billy called me into his office and told me I had been awarded a secret clearance.  He let me know that I could be subject to the death penalty if I divulged any classified information.  I signed some documents, and that was it. “Don’t I get a card for my wallet or anything?” I asked Billy.  He just smiled.

Shortly after getting my clearance, one of our other cleared employees brought me into a secure office in one of Juniper’s buildings where we could look at classified information.  He pulled a secured laptop out of a locked drawer, and a password out of a sealed envelope.  We began perusing classified information.  None of it was relevant to us, and none of it was particularly memorable.  For example, we read an article about several criminal gangs, the existence of which was unclassified.  The only classified information in the article happened to be the names of particular gangs.  They didn’t mean much to me, and I probably forgot them within a day or two.

One day I was invited to the San Francisco FBI office, to receive a classified briefing.  Billy had to fax the clearance over, because the DoD and FBI didn’t have an electronic way to exchange clearances.  I showed up, excited, to the federal building in San Francisco and proceeded up to the floor where the briefing was to take place.  Nobody was there.  I wandered around the white hallway with locked doors unable to make contact with anyone.  The elevator opened after a few minutes, and another equally confused attendee emerged.  We were wandering around for several minutes before someone showed up and told us to go to a different floor.

On the new floor a couple of young-looking FBI agents setup a table, checked our ID’s, and then took our cell phones.  The security did not seem very rigorous.  They then admitted us to the SCIF, or Sensitive Compartmented Information Facility.  The room we were led into was just a conference room, with a low ceiling and no windows.  Another young-looking FBI agent approached me, wearing a tie but no coat.  “Hi, I’m Elvis,” he said.

“I’m a special agent and the coordinator of the briefing today.  We’re very excited to have you here.”

We had a brief conversation about my job and role, and then I asked to use the bathroom.

“Go out the back door of the SCIF and hang a right, he said.”

I did this, and found myself walking with a wall on my right, and a row of waist-level cubicles on my left.  Nobody was in the the cubes, but paperwork was sitting on most of the desks. I wanted to peer at the paperwork as I walked by.  I have a clearance, I figured, so if I had a right to at least take a peek and see if the names of anyone I knew appeared.  Unfortunately, without pausing and staring, a chance I didn’t want to take, I couldn’t read anything.

I found the bathroom, and as I was participating in nature’s call, a couple of guys came in wearing ties but no sport coats.  They each had side-arms on their belts.  I wondered why these agents, who are basically office workers, needed to walk around armed.

As I came out of the bathroom, a female FBI agent was standing there, tapping her foot in anticipation of my emergence.  She looked like my school librarian.  Diminutive in stature, she had a side-arm that looked as big as she was.

“Are you FBI?” she asked pointedly.

“No,” I replied, thinking the answer was obvious.

She let out a long sigh, looking like a satisfied cop who has caught a perp.  “You can’t be here without an escort,” she scolded me.

“But Elvis told me I could!” was my retort.  I had a sudden realization that, in a large FBI office like San Francisco’s, it was entirely possible that not every FBI agent knew every other FBI agent, and that my host agent may have been entirely unknown to her.  Here I was, by myself, in the inner sanctum of an FBI office, explaining to an armed federal agent that I happened to be there because Elvis had sent me.

Fortunately, a glimmer of recognition flashed across her stern countenance.  “Oh, Elvis!” she said, exasperated.  “Come on,” she snapped, and led me back to the SCIF.

Back in the SCIF, the briefing began.  The first presenter was an FBI agent wearing a tie, with a coat this time.  Whatever he had learned at the FBI training center in Quantico, VA apparently did not include the fundamentals of haberdashery.  Anyone who buys a suit knows that you immediately have it tailored, as the pant legs are way too long.  Apparently this agent bought his cream-colored suit, with piping, and never sent it for alterations.  The trouser legs were so long he was actually walking on the bottom of his pant legs.  His presentation was no better than his tailoring.  Presenting on computer security, it was clear this was not somebody with even a basic knowledge of computing.

After him, two Homeland Security analysts presented.  They wore rumpled khakis with jacket and tie, and sported similar pyramid mustaches.  They presented on SCADA systems, a subject I could care less about.  Almost all of it was unclassified.

Shortly after my briefing, I learned that the OPM database had been hacked by the Chinese military.  All the personal information about myself and my family is in their hands now.  When I left Juniper, Cisco declined to renew my security clearance.

Some people hide that they have/had a clearance, as they can be targeted by foreign governments.  Personally, I don’t care.  What little classified information I saw, I can’t remember.  You could waterboard me and I wouldn’t be able to tell you a thing.

3

Plain Language

Posted by ccie14023 on October 3, 2022
Posted in: Reflections. Tagged: , , , . Leave a Comment

I don’t know about the rest of the world, but here in America our police speak in a sort of code language.  Instead of saying “he got out of the car and walked away,” the police will say, “the subject exited the vehicle and proceeded on foot.”  It’s not that their language is any clearer–in fact it’s less clear.  When talking to each other, the cops like to use language like this because it seems to elevate them and make them sound more professional.

A friend of mine told me a story of his friend who had always wanted to be a cop, but was too much of a screw-up to make it to the academy.  Despite his failure he remained a police buff, perhaps a cop in his own mind.  This fellow witnessed a crime and the local sheriff showed up.  This non-cop started to describe the crime to the sheriff as cops do:  “The subject exited the vehicle and proceeded to commit a 4-15…”  The deputy cut him off and shouted, “speak English, boy!”  The poor police-wannabe never lived it down.

Corporatist-types have a language like this too.  Attempting to sound smart and professional, they speak in an often-inaccessible code language replete with b-school buzz words.  I must admit, as long as I’ve been in the corporate world, I’ve frequently been confronted by language that I simply couldn’t understand.

“We collaborated with engineering and cross-functional product managers across multiple time zones to groom and prioritize backlog to ensure efficient program delivery.”  Huh?  “We need to build a motion that creates value at scale.”  What? “The adoption journey enables us to innovate continuously.”  Speak English, boy!

Then there was this gem, from an MBA describing a customer problem:  “The customer is not in the mindset of extracting value from the product.”  A lot of words there.  How about three words:  “It don’t work.”  Or, “customer hates it.”  Oh no.   If the MBA spoke that way, he’d sound like he learned nothing at his prestigious business school.  Professionals speak professionally, you see.  If he spoke like a normal human being, some people might suspect he actually doesn’t really know anything.  Although to be honest, that’s exactly what I was suspecting when he started talking about mindsets.

We can all fall into this trap, I’m afraid.  I refused, for years, to use “ask” and “spend” as nouns, because they’re not nouns.  (I remember an internal thread at Cisco years back in which someone said, “shouldn’t we productize that?”  The snarky response came back from an engineer, “no because at Cisco we can’t turn nouns into verbs.”)  Alas, I’ve surrendered to the progress of business-speak and have replaced “request” with “ask.”  Saving one syllable with a frequently-used word has certainly given me hours back to do other things, don’t you think?

Technical people can certainly fall into this and we have our own jargon.  Some of it is necessary.  Here is a snippet of a Cisco doc:  “To overcome the limitations of the flood-and-learn VXLAN as defined in RFC 7348, organizations can use Multiprotocol Border Gateway Protocol Ethernet Virtual Private Network (MP-BGP EVPN) as the control plane for VXLAN. ”  This is wordy, and it is jargony.  That said, I can’t think of a better way to say it.  This sort of language is unavoidable for network engineers.

What I don’t like is technical people adopting MBA-speak because they’re surrounded by it.  “Our latest release provides flexible options to operationalize your business intent.”  Oh dear, even if you get into some good technical meat, you’ve lost me already.  The simple secret for me in winning Cisco Live Hall of Fame for my speaking is simply to state things in plain, clear language.  Technical, yes, but clear.

I used to think I was stupid, sitting in meetings in the corporate world and not understanding what on Earth people were saying.  Then I learned that in many cases, the speakers didn’t understand what they were saying either.  In the event that they actually do, a few pointed questions can usually cut through the fog of fancy words.  I’m convinced many of the mistakes made in the corporate world would never happen if people actually spoke like normal people.

If you feel tempted to obscure your language to sound like you’re oh-so-smart, remember the advice of the deputy sheriff:  Speak English, boy!

8

Dear Diary

Posted by ccie14023 on August 17, 2022
Posted in: Reflections. Tagged: , , , . Leave a Comment

My blogging has been a little slow of late.  First there was Cisco Live.  Then there was the post-Cisco Live slump of not wanting to do anything.  Then there was the Cisco’s fiscal year-end crunch along with some very hot projects.  Then there was the departure of one of our key execs, and subsequent excitement.  Then one of my direct reports fell gravely ill, and there was both the stress of that along with the burden of picking up the management of his team.  Then there is simply the fact that, in order to blog, one has to come up with ideas.  And often I don’t have any.

So, let me get back into it with a bit of a diary entry.  Cisco Live was back in person for the first time since COVID began.  (And despite the rigid COVID protocols, tons of people got it, hmmm.)  Cisco Live, as I’ve said before, is not an easy show to put on.  It’s a massive effort, and we had not done it for two years.  Some of the key people who used to organize it left, and those of us who had done it before had some muscles atrophy.  It was not, shall we say, smooth.  But it happened, and seeing fellow network folks again in person made me realize many things.  Like the fact that I am the only male American network engineer who does not own a pair of cargo shorts.

It also made me remember the camaraderie of our industry, which is half the fun.  One of my hobbies is electrical work (not as dangerous as it sounds, usually), and there is a YouTube electrician who shot some videos at an electrician’s convention at Mandalay, the same convention center where we host Cisco Live.  I’m sure he hung out at the same places, felt the sense of camaraderie that only electricians have, etc.  There’s always a great feeling being a member of a club, whatever that club may be.  Speaking your own language, reminiscing on how things were in the past, and perhaps about how the youngsters never had to configure dialer maps or CSU/DSUs.  Our club, the people who gravitate towards network engineering, is a special bunch.  Perhaps that electrician can same the same thing.  Perhaps I’m a bit sentimental given one of my colleagues is ill.  All I can say is, I missed everyone and enjoyed being back in person.

For the first time in a while I failed to win a distinguished speaker award.  Cisco Live audiences humble the proud.  I delivered my session on the “CCIE in an SDN world”, but the old 90 minute session was crammed into 45 minutes because of the format change.  I couldn’t quite get the rhythm right.  Also, I had to deliver it at 8am.  I’m just not a morning person, and I’m zero for zero with DS awards delivering in that time slot.  I also had only 40 people.  I used to pack in 500.  It was a smaller show, but the audience was small nonetheless.  I was (finally) inducted into the Cisco Live Hall of Fame two years after I qualified, but perhaps I’m like an old movie star, washed up and waning in popularity.  I actually like how challenging the audiences are, however.  They keep you on your toes and force you to never get complacent.

Another question is whether the message resonates any more.  When I started doing this session, it was amidst the barrage of messaging from vendors (like us) that networking was changing forever, that AI and automation and intuitive networks would end network engineering as we know it.  My session basically said: well, it’s not quite the case.  You can automate, but good luck if you don’t understand what you automate.  Several years on, I wonder if the message is just so apparent that it’s not as needed as it was before.

Still, Cisco Live is my favorite place to present.  I do a ton of internal-facing presentations, many to executives, and each one is written, reviewed, and delivered by committee.  There’s no freedom in it at all.  The words and messages are tightly controlled.  What I love about Cisco Live is that, as an established speaker, I can pretty much present whatever I want.  I can build my own slides.  I don’t even need much of a review.

I’ll leave it there.  A rambling, diary post, perhaps not of a lot of interest.  At least it got me blogging again!

1

The type A conundrum

Posted by ccie14023 on March 25, 2022
Posted in: Reflections. Tagged: , , , . 2 comments

An old theory of personality holds that people fall into two types–A and B.  Put simply, Type A personalities are highly aggressive and competitive, whereas Type B are not.  We all have seen this broad difference in personalities.  Some people we encounter seem ready to walk over their own grandmothers to get ahead.  Like all stereotypes, this is a gross oversimplification, but there’s a lot of truth in it.

In the corporate world, type A personalities tend to rise to the top.  Why?  Because their very personality is aggressive and competitive.  They like to push, push, push for what they want and are willing to drive their agenda at any cost.  They frequently are talkers but rarely listeners.  They also judge people through their own lens.  If you’re an introvert, quiet, or deliberative, if you’re a listener instead of a talker, they think you are not “driven” and probably not worth listening to or promoting.

The question is:  does being type A make you right?  Does it make your opinions more valuable?  I cannot think of any reason why being aggressive and competitive makes you more likely to be correct about anything.  In fact, I think the opposite is true.  If you don’t listen well and are always pushing your own agenda, you’re less likely to consider the opinions of others, which means your decision-making is less well-rounded.

I’ve pointed out before, that quiet, deliberative people, type B’s, are often the ones you really want to listen to.  However, in the corporate world, they are often kicked to the curb.  “He never says anything in meetings,” the type A’s say.  Well, if you hired him maybe he’s actually a smart person but has a hard time contributing in a meeting with 20 people all talking fast.  Maybe he needs time to digest what he heard before providing recommendations.

Type A’s tend to be in positions of power not necessarily because they are smarter but because they fight for position in hierarchies.  This is not to say they are without value.  Their decisiveness and drive are very important to a healthy organization.  They can break indecision and move companies forward in ways type B’s cannot.

The key for both personality types is the old Greek maxim:  know thyself.  If you’re type A, you need to be careful not to be too aggressive.  Listen to your quieter colleagues, accept that they may have a different personality, and meet them where they are at.  Call on them in meetings, give them time to deliberate and come back to you.

For type B’s, you need to learn to speak out more.  You’re probably more respected than you realize, and when you do speak, you’re probably listened to.  Try to find forums that are more comfortable for you, like expressing your opinion in writing or 1:1’s.

Sadly, because of the cutthroat nature of the business world, I see little self-awareness and frequent domination of businesses by type A’s.  At the end they may get people to follow them, but if they’re leading you off a cliff, their drive may not be such a good thing.

1

Can you spare a quarter?

Posted by ccie14023 on March 21, 2022
Posted in: Reflections. Tagged: , , , . 2 comments

I’ve been thinking about the corporate world, how it operates, and the effects of corporatism on our lives.  If you’re a network engineer and think this is boring, pay attention.  Corporate culture, the influence of Wall Street, and the rise of a non-skilled management class have direct impact on your work and personal life.  The products you use are heavily influenced by corporate culture.  Why vendors release certain products, when, and how, are all controlled by corporate culture.  When a company tries to sell you something that doesn’t work and doesn’t serve your needs, when the company discontinues support for a product you bought after crashing and burning with it, when companies force products down your throat with buzzword messaging that means nothing to you, corporate culture explains it.

If you work in a corporation, the culture creates politics which affect what projects you work on, your career trajectory, and how you interact with your team.  In your personal life, the food you eat and drugs you take are very much explained by corporate culture.

I wrote in a previous post about the lack of anything permanent in the corporate world.  Everything seems to be temporary, everything is always in flux.  Companies are afflicted by short-term thinking, and short-term thinking is killing everyone.

One way this manifests itself is quarter-by-quarter thinking.  We all know sales people are judged on a quarterly basis, but corporations in general are as well.  Publicly traded companies have to present results to analysts, and thus to investors, every single quarter.  The results are compared against the last quarter, against the same quarter the previous year, and against other companies in the industry.  The results have a huge impact on stock price, executive compensation, and even executives’ jobs.

The effect of this trickles down to all levels of a public company.  Business units are judged by the quarterly performance of their products.  This means product managers are judged by the quarter, much like sales people.  Product managers are not commissioned directly like sales people, but they live and die by quarterly numbers.  As a result, they want to do everything possible to ensure quarterly numbers shine.

Now, imagine you are a product manager.  You have a deal worth, $20 million on the line if you deliver specific features the customer wants.  You are going to do anything possible to win the deal, so your quarterly numbers look good.  Now it probably is the case that the $20 million customer’s feature requests are specific to their environment.  That is, adding the features will help that one customer, but probably very few others.  So, instead of trying to build a product that caters to a broad range of customers who might bring smaller deals, you end up building a product that caters to a narrow set of customers that make you look good in your quarterly business reviews.

Now this type of short-term thinking might be an obvious problem if you planned to spend twenty years at your company.  But instead you spend two years at a company, so you only have to pull this off for eight quarters.  You can put big happy numbers in your LinkedIn profile (“successfully drove record quarter of $100 million in sales!”) and then exit stage right to repeat the process elsewhere.  And the folks left-behind have to clean up the mess.  Keep in mind your success within the company is also being judged by non-technical MBAs who are looking to do the same thing you are.

The companies that do the best long-term are those that eschew short-term thinking.  Apple is a great example of this.  They’ve had some disasters, but have generally taken risks to build products with long-term appeal.  I often mention Zappos founder Tony Hsieh, who while he had serious personal problems, forsook short-term gain for long-term performance.  Even within a company, quarterly thinking can vary by business unit and leader.

At the end of the day, however, it’s Wall Street that encourages this.  Like any metric, execs end up chasing their stock price like a dog chasing its tail.  It doesn’t get you anywhere, however much progress you may think you are making.  Meanwhile, you may get rich, but you leave disaster in your wake.

1

The problem of permanence

Posted by ccie14023 on February 11, 2022
Posted in: Reflections. Tagged: , , , , . Leave a Comment

Some years ago, I worked for a company with a CEO who had a background in marketing.  It was 2010, and he decided to use his marketing skills to launch a huge new campaign called “Mission 10”.  Our goal:  to become the next $10 billion company.  At the time I think I revenue was less than five billion.  Slick slides were drawn up, pep rally company meetings were held, and everyone in the company began pivoting their work to fit the new agenda.  Anyone who has worked in the corporate world has been there more than once.  Suddenly every initiative had to have a “Mission 10” theme.

The problem?  Despite the rah-rah of our CEO, we never achieved even close to $10 billion in revenue.  In fact, that company is still below $5 billion last I checked.  The bigger problem?  The CEO moved three years after that, having never really achieved this or any other goal he set.  He later ended up CEO of an even larger and more famous company that has nothing to do with technology.  This is known as “failing upward”.

In light of the “great resignation” I’d like to write a little about permanence, or the lack thereof.  We live in a temporary world.  People pick up a job and stay for two or three years, and then move on.  This was true even before COVID.  I myself have several two-year stints on my resume.  The longest I’ve worked anywhere is six.

Three years is just long enough to kick off some major initiative and get out at the peak, just before the whole thing crashes.  The damage done by corporate executives pursuing this short-term strategy is massive.  It works like this.  An exciting new executive is hired on from a big company.  The new executive launches a new product, architecture, marketing campaign, acquisition, whatever.  Everyone rallies around it because, well he’s the boss, and because if you want funding for anything it needs to be tied to the boss’ initiative.  The new initiative (let’s say it’s a product) is pumped up with cash, the marketing engine kicks in, the company oversells the product, and then customers start snapping it up.  It doesn’t perform as expected.  Things start to crash.  Money dries up.  The executive exits.   And whoever decides to stay is left picking up the pieces of the mess that this guy created.

In Ancient Greece, you faced consequences for this sort of thing, usually exile, sometimes death.  While I’m not advocating the death penalty for corporate screw ups (although in some industries they do cost lives), what’s fascinating is that in the corporate world, the consequences are the opposite.  Said executive who just screwed up royally walks away with huge bonuses, lots of stock, has a nice sabbatical, and begins the cycle again somewhere else.

If you think executives are the only problem, think again.  It happens at every level of the corporate world.  When a junior IT guy messes up a new system and then bolts for another job, you have the same issue at a smaller scale.  He just doesn’t get the bonus and sabbatical.  As a leader of technical marketing engineers, we face all sorts of challenges when an experienced TME leaves and takes knowledge with him.  Features can be stalled when the people who were working on them leave.

In my grandfather’s era, and even my father’s, it was expected that you would start and end your career at the same company.  There was an expectation of permanence.  People were proud of their companies and how they were treated, and bragged about the excellent pension they’d receive when leaving.  Now, we spend three years and jump ship to boost our salary.

Companies, are of course, largely responsible.  Often they don’t create the sort of employment experience that anyone would want to tolerate for long.  People stopped being human beings and started becoming human “resources”.  Executives, under various pressures, began to see their workforce as mere “metrics” to be manipulated as they learned in their B-school classes.  Times are good?  Dial up the workforce.  Times are bad?  Lay off 3%.  People are just numbers on a slide to many execs, and the difficulties of terminating employment are a remote problem to be dealt with by line managers.  As a result, employment is not a long-term commitment but a short-term business transaction on both ends.

The temporary workforce has an interesting effect on longer-term employees as well.  Someone who has worked at the same company for 15 or 20 years sees executives and initiatives come and go, ebbing and flowing like the tide on a beach.  They often develop an apathy and callousness that makes their own work unproductive.  They tend to focus on the day-to-day instead of the long-term, and often dismissively ignore the plans of new leadership, figuring the leaders will just be replaced and the cycle will start over.  Thus, while they have a long-term career, they often have a short-term level of focus.

We all live in a temporary world now, and permanence is in short supply.  If you want to understand why companies build bad products, why executives start disastrous programs and leave, and why there never seem to be consequences, this is a huge part of it.  I don’t really have a solution I’m afraid. Some of the causes are:  greedy hedge-fund finance people who take over corporate boards, an undisciplined corporate press/media, the instant availability of information leading to a lack of deliberation, and the rise of a management class who are not actually experts in anything other than management itself.  We can all point fingers at ourselves for up and going when the going gets tough.

The Greek philosopher Heraclitus famously said that you cannot step in the same river twice.  He meant that, if you cross a river, each time you take a step the water you were originally standing in has passed on, and you’re in new water.  Thus, there’s really no river.  Sometimes the tech world, and the corporate world in general, feel like Heraclitus’ River.  Even if you stand in one place, everything just moves on.

5

Technical Confusion

Posted by ccie14023 on November 8, 2021
Posted in: Reflections. Tagged: , , , , . Leave a Comment

How often have you learned about a new technology, and couldn’t understand it?  How many trainings and presentations have you sat through that left you in a mental fog?  It amazes me how many technologies we are supposed to master in our industry, and how many we never do.

Let me give an example.  When I heard about “Cloud Computing” I could not, for the life of me, understand what it meant.  I went to meeting after meeting where we talked about “the Cloud” without any understanding of what it actually was.  I knew I used clouds a lot of Visio diagrams, but the MBA-types who were telling me we needed to migrate to the cloud would never be able to understand the Visio diagrams that network engineers make.  It seemed to involve using centralized computing resources, but I’d been doing this for years.  My first ISP accounts were shell accounts.  My email and other services were hosted on their computers.  Nothing was new about this.  In fact, Larry Ellison gave a hilarious talk in which he asked “What the hell is Cloud Computing?”

We all know the “cloud” has in fact made significant changes in how we engineer computing resources, but the truth is, the idea of centralized “compute” is not a new one.  (Side note:  I hate turning nouns into verbs.  “Compute”, “spend”, and “ask” are verbs, not nouns.  The MBAs who invent these terms apparently don’t have to study grammar.)  The scale is certainly different, but we all know that mainframes had both centralized computing and virtualization long before anyone said “cloud.”

SDN is another one.  I was told we needed SDN, but I couldn’t figure out what it meant.  I was a hard-core routing protocols guy.  BGP and OSPF are software.  Ergo, networks are already software defined.

Someone sent me a video from Nicira, later acquired by VMware.  The vague video described slicing networks into pools, or something like that.  I couldn’t understand what this meant.  Like a VLAN?  I finally found a document that described SDN as separation of the control plane from the data plane.  OK, but we already had been doing that in routers and switches for years?  Yes, but SDN was a centralized control plane.  Kind of like BGP route reflectors?  I couldn’t figure it out.  I spent some time getting OpenFlow up and running to try to understand it from the ground up.  What a waste that was.  Whatever SDN has become, it’s certainly not what it was originally defined to be.  And don’t get me started on SASE.

I used to think maybe I was stupid, but now I realized all of these things confused me because they were (a) confusing in themselves, or (b) so badly explained that nobody really understood them.  A little more detail:

  • Some technologies are simply vague marketing terms.  They don’t correspond to anything precise in reality.
  • Some technologies do correspond to reality, but they are simply bucket terms.  That is, the marketers took five, six, ten technologies, and slapped a new label on them.  In this case, you’re looking for some precise definition of term X and you realize term X refers to ten different things at once.
  • Sometimes new technologies are invented, and the inventors don’t want to cough up too much proprietary information.  So the produce vaguely worded marketing content that appeals to “analysts” with MBAs in marketing, but which technical people realize are meaningless.  Said “analysts” now run around creating hype (“You need software-defined cloud secure-access zero trust!”) and now we’re told we to implement it.
  • A lot of technical people are really bad explainers.  Sometimes there is a new technology which is clear and well-defined, but the people sent to explain it are completely incapable of explaining anything at all.

My point is, it’s ok to be confused.  A lot of times we’re in the room and everyone seems to be getting it, but we have no idea what is going on.  Chances are, nobody else really understands what is being said either.  Ask questions, drill down, and if you don’t understand something, chances are it’s hot air.  In a world where we prioritize talk over reality, there seems to be an abundance of that.

5

For the love of wiring

Posted by ccie14023 on September 10, 2021
Posted in: NetStalgia, Reflections. Tagged: , , , , , , , . 1 comment

We’ve moved into a wireless world, which is too bad for me because I love, more than anything, wiring.  I miss the days of good old Cat 3 cable, T1 lines, and ISDN BRIs.  I miss 66 blocks, punch down tools, cross-connect wire, and tone/probe kits.  And butt sets.  Especially butt sets.  Now I just have to add random wall receptacles around my house since, thank goodness, 120 volts cannot be delivered wirelessly.  But phone and network wiring was much more fun.

I first got interested in wiring when I was working at a small museum exhibit design and fabrication company in Marin, California.  One day, I had to have a new phone line installed, and I called in Pacific Bell to do the work.  Under the receptionist’s desk was a wall plate with two RJ-11 jacks, already in use.  “There aren’t any free jacks,” I told the phone guy.

“It doesn’t matter, as long as there is enough wire,” was his cryptic response.  I let him do his work, and when he was done I saw a little surface-mount RJ11 jack on the wall.  It had two blue and yellow wires running from it, going underneath the existing faceplate and somehow into the wall.  How on earth did he do that?  I remember thinking.

I waited until everyone went home.  I unscrewed the faceplate and found that the blue/yellow wire pair was spliced to the brown pair of wires in the cable that serviced the existing jacks.  The splice was a 3M UR-type, which looked like a piece of candy.  I was captivated.  How did he know where the brown wires went?  Where did the incoming phone line enter the building?  How did he connect them up?

The fun thing about the days before ubiquitous Internet was that you couldn’t get answers immediately.  When I found the 66-type punchdown blocks that formed our little MDF, I couldn’t Google the part number to figure out what they were.  Google didn’t exist.  I had no idea how to terminate a wire on one of them.  While thumbing through a Jensen Tools catalog our purchasing agent had I got an idea of how it worked.  There, I saw listed a “66 punch-down tool” and I had noticed the numbers “66” on the block.  OK, so they go together and the wire gets in the clip by “punching down.”

I didn’t have the money to afford a punch-down tool.  So I got a pair of pliers and a pair of forceps and started doing terminations myself, guiding the wire into the clip.  Sure, sparks were flying, but phone systems are robust, aren’t they?

It turns out not as robust as you might think.  Back in the day, phone lines provided by the phone company (POTS service) were indeed robust and could handle quite a bit of sparking and shorting.  But the Nitsuko (emphasis on the “suk”) system we used did not take kindly to my laissez-faire approach to wiring.  One day, while doing a move, I shorted out a couple terminals and all the phones in the building went out.  I came out of the closet with pliers in hand and everyone knew what happened.  This being before cell phones, business was done for the day.  Luckily we got our phone company out to replace the bad part, and they did it under warranty.  It was at this point I invested in a proper punch-down tool and learned how to wire correctly.

The author’s tools: Butt set, punch down tools, tone/probe set, and connectors

I saved up a lot of money and eventually got a test set, otherwise known as a “butt set”.  The butt set looks like an oversized telephone handset, and is the official sign of a telco wiring guy.  It also was my passport into telephone wiring closets–when a security guard sees you have a butt set, they just assume you’re a real phone guy.

I practiced my technique in my father’s 1910-era house.  The decades brought with them layers of phone wiring, including two 50-pair feeder cables and a 66-style punch-down block.  Why and how the feeder cables ended up in a house with 3 phone lines is a mystery to this day.  But I used the 66-block for practice and dissected the criss-crossing wires in his house, tracing them out with my trusty tone/probe set.

Wiring skills came in handy many times in my career as a network engineer.  I remember one customer I had in the late nineties who had ordered 8 Centrex lines from Pacific Bell.  The technician showed up to do the cross-connects, but he was not the sharpest knife in the drawer, did two of them, and left.  Using my butt set passport, I got access to the MDF, figured out the right punch-down block that fed to the customer’s suite, and ran the cross-connects myself.

When I worked for the San Francisco Chronicle, we actually had an in-house wiring tech.  Her name was Mona Lesa.  She couldn’t care less if I did my own wiring, as long as I adhered to her standards.  One day I had a frame relay T1 installed in our Sacramento bureau office, and once again, Pac Bell forgot to connect it to the suite.  The bureau was located in the Old Senator Office Building across from the state capitol.  I got the security guard to let me in to the MDF in the bowels of this historic building.  I figured out that one of the interconnects was in the office of a lobbying firm, and the receptionist dutifully let me in to do the wiring.  With my butt set I could clip into any of their phone lines and listen to their conversations without them knowing.  I didn’t but I was amazed that I could go anywhere to do cross-connects.

On another occasion, a customer of mine had two phone lines installed and had a weird problem.  If she picked up one line and dialed the other, she’d hear both ringing and a busy signal at the same time.  I found where the Pacific Bell guy had done the cross-connect, and realized he mixed the tip and ring wires for the two phone lines.  A couple punches and all fixed.

Phone wiring was always the perfect blend of mind and body to me.  To do it you need to work with your hands, but you also need to use your mind.  Some wiring closets were rats nests of 24-gauge wires color-coded with more colors than a tie-died shirt.  Finding that one pair you needed, and marrying it up to the other end was always a nice diversion from staring at a screen.

Alas, my tools mostly sit unused.  Regular POTS lines rarely exist now, and even they aren’t delivered on single copper pairs from the CO switch.  PBXs and key systems have gone the way of the dinosaur, replaced by voice-over-IP and now by cellular phones.  Nobody wants to be tied to wires anymore, and yet by un-encumbering ourselves in the name of freedom, we’ve paradoxically lost our freedom.  When you don’t need to be physically tied to a location for connectivity, you can never escape connectivity.  And that’s not entirely a good thing.

3