Networking Simplicity

Posted by ccie14023 on March 24, 2023
Posted in: NetStalgia. Tagged: , , , , . Leave a Comment

There’s a lot of talk about networking simplicity these days.  There’s been a lot of talk about networking simplicity, in fact, for as long as I can remember.  The drive to simplify networking has certainly been the catalyst for many new products, most (but not all) unsuccessful.  Sometimes we forget that networking has some inherent complexities (a large distributed system with multiple os’s, protocols, media types), but that much of the complexity can be attributed to humans and their choices.  IPv4 is a good example of this.

When I got into network engineering, I had assumed that network protocols were handed down from God and were immaculate in their perfection.  Reading Radia Perlman’s classic book Interconnections changed my understanding.  Aside from her ability to explain complex topics with utter clarity, Perlman also exposed the human side of protocol development.  Protocols are the result of committees, power politics, and the limitations of human personality.  Some protocols are obviously flawed.  Some flaws get fixed, but widely deployed protocols, like IPv4, are hard to fix.  Of course, v6 does remedy many of the problems of v4, but it’s still IP.

My vote for simplest protocol goes to AppleTalk.  When I was a young network guy, I mostly worked on Mac networks.  This was in the beige-box era before Jobs made Apple “cool” again.  The computers may have been lame, but Apple really had the best networking available in the 1990’s.  I’ve written about my love for LocalTalk, and its eminently flexible alternative PhoneNet in the past.  But the AppleTalk protocol suite was phenomenal as well.

N.B.  My description of AppleTalk protocol mechanics is largely from memory.  Even the Wikipedia article is a bit sparse on details.  So please don’t shoot me if I misremember something.

In the first place, you didn’t need to do anything to set up an AppleTalk network.  You just connected the computers together and switched either the printer or modem port into a network port.  Auto-configuration was flawless.  Without any DHCP server, AppleTalk devices figured out what network they were on, and acquired an address.  This was done by first probing for a router on the network, and then randomly grabbing an address.  The host then broadcast its address, and if another host was already using it, it would back off and try another one.  AppleTalk addresses consisted of a two byte network address which was equivalent to the “network” portion of an IP subnet, and a one-byte host address (equivalent to the “host” portion of an IP subnet.)  If this host portion of the address is only one byte, aren’t you limited to 255 (or so) addresses?  No!  AppleTalk (Phase 2) allowed aggregation of contiguous networks into “cable ranges”.  So I could have a cable range of 60001-60011, multiple networks on the same media, and now I could have 2530 end stations, at least in theory.

Routers did need some minimal configuration, and support for dynamic routing protocols was a bit light.  Once the router was up and running, it would create “zones” in the end-user’s computer in an application called “Chooser”.  They might see “1st floor”, “2nd floor”, “3rd floor”, for example, or “finance”, “HR”, “accounting”.  However you chose to divide things.  If they clicked on zone, they would see all of the AppleTalk file shares and printers.  You didn’t need to point end stations at their “default gateway”.  They simply discovered their router by broadcasting for it upon start up.

AppleTalk networks were a breeze to set up and simple to administer.  Were there downsides?  The biggest one was the chattiness of the protocols.  Auto-configuration was accomplished by using a lot of broadcast traffic, and in those days bandwidth was at a premium.  (I believe PhoneNet was around 200 Kbps or so.)  Still, I administered several large AppleTalk networks and was never able to quantify any performance hit from the broadcasts.  Like any network, it required at least some thinking to contain network (cable range) sizes.

AppleTalk was done away with as the Internet arose and IP became the dominant protocol.  For hosts on LocalTalk/PhoneNet networks, which did not support IP, we initially tunneled it over AppleTalk.  Ethernet-connected Macs had a native IP stack.  The worst thing about AppleTalk was the flaky protocol stack (called OpenTransport) in System 7.5, but this was a flaw in implementation, not protocol design.

I’ll end with my favorite Radia Perlman quote:  “We need more people in this industry who hate computers.”  If we did, more protocols might look like AppleTalk, and industry MBAs would need something else to talk about.

3

The death of CLI: A lesson from Swift

Posted by ccie14023 on March 13, 2023
Posted in: Perspectives. Tagged: , , , , , , , . 2 comments

Update:  From Fred, who was the guy referenced in the first paragraph:

Actually it was a white button with a router icon on it and “make cli great again”, I know this because it was me. It was June 2016. Needless to say in my view that did not age well.

When I attended Cisco Live sometime around the election of Donald Trump, there was a fellow walking around with a red hat with white lettering on it:  MAKE CLI GREAT AGAIN.  Ha!  I love Cisco Live.  These are my people.

I remember back when I worked at Juniper, one exec looked at me working on CLI and said, “you know that’s going to be gone soon.  It’ll all be GUI.”  That was 8 years ago…how’s that going?  When I work on CLI (and I still do!), or programming, my wife always says, “how can you stare at that cryptic black screen for hours?”  Hey, I’ve been doing it since I was a kid.

The black screen won’t go away, I’m afraid.  I’ve recently been learning iOS app development for fun (not profit).  It’s surprisingly hard given the number of successful app developers out there.  I may be too used to Python to program in Swift, and my hatred of object-oriented programming doesn’t help me when there is no way to avoid it in Swift.  Anyways, it took me about a week to sort out the different UI frameworks used in iOS.  There are basically three:

  • Storyboards.  Storyboards are a graphical design framework for UI layout.  Using storyboards, you drag and drop UI elements like buttons and textfields onto a miniature iPhone screen.
  • UIKit.  (Technically storyboards use UIKit, but I don’t know what else to call this.)  Most high-end app developers will delete the storyboard in their project and write the UI as code.  They actually type in code to tell iOS what UI elements they want, how to position them, and what to do in the event they are selected.  Positioning is fairly manual and is done relative to other UI elements.
  • SwiftUI.  Apple is pushing towards this model and will eventually deprecate the other two.  SwiftUI is also a UI-as-code model, but it’s declarative instead of imperative.  You tell SwiftUI what you want and roughly how you want to position things, and Swift does it for you.

Did you catch my point?  The GUI-based layout tool is going away in favor of UI-as-code!  The black screen always comes back!

The difference between computer people and non-computer-computer-people (many industry MBAs, analysts, etc.), is that computer people understand that text-based interaction is far more efficient, even if the learning curve is steeper.

Andrew Tanenbaum, author of the classic Computer Networks, typeset his massive work in troff.  Troff is a text-based typesetting tool where you enter input like this:

.ll 3i
.mk a
.ce
Preamble
.sp
We, the people of the United States, in order
to form a more perfect Union...

Why doesn’t he just use Word?  I’ll let Dr. Tanenbaum speak for himself:

All my typesetting is done using troff. I don’t have any need to see what the output will look like. I am quite convinced that troff will follow my instructions dutifully. If I give it the macro to insert a second-level heading, it will do that in the correct font and size, with the correct spacing, adding extra space to align facing pages down to the pixel if need be. Why should I worry about that? WYSIWYG is a step backwards. Human labor is used to do that which the computer can do better.  (Emphasis added.)

I myself am not quite enough of a cyborg to use troff (though I use vi), but I have used Latex with far better results than Word.  (Dr. Tanenbaum says “real authors use troff,” however.)

One of my more obscure interests (I have many) is Gregorian Chant.  Chant uses a musical notation which is markedly different from modern music notation, and occasionally I need to typeset it.  I use a tool called Gregorio, where I enter the chant like this:

(cb3) Ad(d)ór(f’)o(h) te(h’) de(h)vó(hi)te,(h.) (,) la(g)tens(f) Dé(e’)i(d)tas,(d.)

The letters in parentheses represent the different musical notes.  I once tried typesetting the chant graphically, and it was far more tedious than the above.  Why not enter what I want and let the typesetting system do the work?

Aside from the mere efficiency, text files can be easily version controlled and diff’d.  Try that with your GUI tool!

It’s very ironic that many of my customers who use controllers like DNAC or vManage are actually accessing the tool through APIs.  They bought a GUI tool, but they prefer the black screen.  The controller in this case becomes a point of aggregation for them, a system which at least does discovery and allows some level of abstraction.

The non-computer-computer-people look at SwiftUI, network device CLI, troff, Gregorio, APIs, and rend their garments, crying out to heaven, “why, oh why?!”  Some may even remember the days of text-based editing systems on their DOS machines, which they could never learn, and the great joy that WYSIWYG brought them.  It reminds me of a highly incompetent sales guy I worked with at the Gold partner back in the day.  He once saw me configuring a router and said:  “Wow, you still use DOS to configure routers!”

“It’s actually IOS CLI, not DOS.”

“That’s DOS!” he densely replied.  “I remember DOS.  I can’t believe you still use DOS!”

It’s funny that no matter how hard we try to get away from code, we always come back to it.  We’re hearing a lot about “low code” environments these days.  It tells you something when the first three Google hits on “low code” just come back to Gartner reports.  Gee, have we been down this path before?  Visual Basic was invented in 1991.  If low code is so great, why is Apple moving from storyboards to SwiftUI?

In my last post I wrote about the war on expertise.  This is one of the fronts in the war.  The non-computer-computer-people cannot understand the black screen, and are convinced they can eliminate it.  They learned about “innovation” in business school, and read case studies about Windows 95 and the end of DOS.  They read about how companies like Sun Microsystems went belly-up because they are not “disruptive.”  They did not, however, read about all the failed attempts to eliminate the black screen, spanning decades.  I believe it was George Santayana who said, “If you don’t remember computer history, you’re doomed to repeat it.”

4

The War on Expertise

Posted by ccie14023 on March 7, 2023
Posted in: Reflections. Tagged: , , , , . Leave a Comment

Like many network engineers, I quickly fell in love with my field and worked hard to master it.  I got into networking when I was working in desktop support.  The behind-the-scenes stuff was way more interesting to me than the front lines.  Back in the late nineties, I bought a library of books to learn this field.  Perlman, Comer, and Stevens were the classics.  I rounded it out with blue-and-white Cisco Press books by Doyle, Peplnjak, Williamson, and many others.  I studied these books religiously, read through config guides, and practiced in the lab.

The network engineers on my team and I loved to debate the arcana of this mysterious field.  We always tried to one-up each other, learning new technologies, new protocols, and attaining new technical certifications.  I’ve worked with engineers who are smarter than I am, and better than I am, but that always motivated me to learn more.

I bring this up because I’ve had multiple conversations with multiple execs, for many years, in which they seem to decry the virtue of expertise.  Network engineers “revel in complexity”, they don’t realize their time has passed, the build networks that need “armies of CCIEs to maintain”, and they hate simplicity.  If only the pesky network engineers would get out of the way, the glorious MBAs could build us simple and elegant products, which is how the industry is going, don’t you know!

In short, our industry is suffering through a war on expertise.  Those arcana we love to master have put a target on our back.  If you want to learn those things, you must be reveling in complexity.  Go find something else to do, ChatGPT will replace you!

The first mistake in this line of thinking is the assumption that network engineers want to build networks that are complex.  We actually don’t.  A couple of anecdotes:

When I was working for a Gold partner, I was sent to help out an IT manager of a rather small company, only four sites.  She had contracted VPLS from two service providers, and asked me to implement a complex load balancing scheme she had conceived.  I begged her not to make me do it, but she insisted.  I ended up building a functional mess, a combination of PBR and EIGRP offset access-lists.  Man, was it ugly.  But it worked.  Then I got laid off from the partner, and a year later she was calling me, begging me to come back because nobody could figure out how it worked.  I didn’t want to build something that ugly and she didn’t need it.

Second anecdote.  My wife had to go in for a surgical procedure a few years ago.  We went to the best doctor in San Francisco.  When he got into the procedure he found that her anatomy is not conventional, and it was a very difficult procedure.  In the recovery room, he told us most doctors would have stopped.  My wife wanly smiled and said, “well, I’m sure you like a challenge.”  He looked back at her and said, “no, we like it when it’s easy.”

I think this is where the execs misunderstand expertise.  99.9% of the time, your airline flight could be handled by a low-time pilot who can work the automation systems.  But when the engines fail, you want Sully at the controls.  Just because some people understand complexity and study difficult concepts, it doesn’t follow that they want complexity.  When I administered networks, I wanted it to be easy.  But I was ready for when it was hard.

The war on expertise seems to me to be a war on the human spirit.  The CCIE exam, whatever you think of it, was a heck of a challenge, and passing it was one of the proudest days of my life.  Human beings want to learn, to grow, to push their limits, and to test themselves.  That’s why we spend hours in the lab.  We should encourage this behavior.  We should want people in our business who seek subject matter expertise and mastery.  We can make things simpler, fine, but we should still encourage expertise.

At the end of the day, networks are inherently complex.  A network is a large distributed system, connecting numerous devices running numerous operating systems over diverse transport mechanisms using a wide variety of protocols.  You can simplify the protocols a bit, but ultimately most simplification of networks is done one of two ways:  reducing the number of choices an administrator can configure, or abstracting and hiding the underlying complexity.  In the first case, you may close out necessary use cases.  In the case of abstraction, well, it works great until something breaks.  Then you need to call a network engineer.

I’m not in any way saying the new tools, from programmability to automation systems like Ansible, to “controllers” are unnecessary.  Far be it.  Any tool that makes an engineer’s job easier will be embraced by engineers.  I am saying that we need to stop blaming complexity on those who manage to understand it.

24

When it’s time to go

Posted by ccie14023 on February 28, 2023
Posted in: NetStalgia. Tagged: , , , , , , . Leave a Comment

I’ve mentioned my first job as a network engineer several times on this blog.  I worked at the San Francisco Chronicle, the biggest newspaper in Northern California.   I was brought in to manage the network as a Cisco-certified engineer, having just passed a four-day CCNA bootcamp.  Right before the dot-bomb economic crash, network engineers were in short supply.

The Chronicle’s network had recently been completely re-engineered, and the vendor selected was Foundry Networks.  Foundry was an up-and-coming vendor famous for selling high-speed switches to internet service providers.  They weren’t known for selling into enterprises, but they had convinced the previous network manager to install their hardware in nearly all of the Chronicle’s wiring closets.

It didn’t go very well.  The network had become incredibly unstable.  No company wants an unstable network, but newspapers are a particularly high-pressure environment since they have tight deadlines in order to get the paper out every single day, without fail.  Management of the data network was taken away from the previous manager and assigned to the head of the telecom department.  The plan was to rip out the Foundry and replace it with Cisco.

Foundry, of course, had other ideas.  Their account manager, whom I’ll call Bill, was quite aggressive in trying to restore the good name of Foundry.  I’ll give him credit for his doomed mission.

We had several problems.  The first was that we had only a single core router.  The router had two management modules, but failover between them was not fast, and our reporters and advertising people used Tandem systems which were sensitive to even slight network outages.  Foundry was well known for their fast IP switches, but we used AppleTalk and IPX as well, and their protocol stacks were not well implemented.  The BigIron 8000 was prone to crashing and taking out a lot of our users.  We had only one because the previous manager had been trying to save money.

The second problem was not Foundry’s fault entirely, although I do blame the SE in part.  Nobody ever set the spanning tree bridge priority on the core box.  By default, STP selects the bridge with the lowest bridge identifier as root.  Since the BID is comprised of a user-configured priority and the MAC address, if no priority is configured, the oldest switch in the network becomes the root bridge, since MAC addresses and OUI’s are sequential.

It turned out our Windows guys had been hauling around an ancient Cabletron switch to multiplex switch ports when working on end users’ computers.  (This was before wireless).  They would plug in, the Cabletron would dutifully assume STP root, and the entire network would reconverge for 50 seconds, spanning tree roots not being sticky.  I remember once paying a bill at a nearby restaurant before we were finished and running with the other engineers back to the office, hoping to catch an outage in progress after our pagers went off.  Foundry’s logs were not very good and we didn’t know why the network kept going down.  Eventually I figured it out, I don’t remember how.

The third problem was that the Foundry FastIron switches we used in the wiring closets had bad optics.  The Molex optics Foundry had selected for its management modules were flaky, and so we had to replace every single one with modules using Finisar optics.  I remember Bill, our account manager, coming in for our middle-of-the-night maintenance window several weekends in a row, blades in tow, and helping us to swap out the cards.

All of these problems created a bad reputation for Foundry within the Chronicle.  I remember Bill walking out of the front door carrying a Foundry box with an RMA’d management module.  A non-technical employee, perhaps a reporter or advertising salesman, saw the box and shouted, “Hey, they’re getting rid of Foundry!”  People in the lobby started cheering.  Bill looked at me and said, “soon they’ll be cheering when I come into the building with a Foundry box.”

It never happened.  We ripped out Foundry and replaced everything with Cisco Catalyst 4k and 6k switches.

The fact of the matter is, had we added a second BigIron in the core, fixed the root bridge problem, and replaced all the faulty modules, we probably would have had a solid network.  But there often comes a point when a vendor has destroyed their reputation with a customer.  It takes a multitude of factors to reach this point, but there is definitely a point of no return.  Once that line is crossed, the customer will often allow cordial meetings, listen with sympathy to the account team and execs, and then go their separate way.

A few years later I was laid off from my job at a Gold Partner, and was interviewing with another Gold Partner.  The technical interviewer looked at my resume and said, “I see you worked at the San Francisco Chronicle.”

“Yes,” I said, “I was brought in to replace the Foundry network they had with Cisco.  The whole thing was a disaster, poorly designed and bad products.”

“I designed that network,” he replied, “when I worked for another partner.  I also installed it.”

I didn’t get the job.

4

Apple Certified…what??

Posted by ccie14023 on January 13, 2023
Posted in: NetStalgia. Tagged: , , , , , , . Leave a Comment

I’m thinking of doing some video blogging and kicking it off with a series with my thoughts on technical certifications.  Are they valuable or just a vendor racket?  Should you bother to invest time in them?  Why do the questions sometimes seem plain wrong?

Meanwhile, a little Netstalgia about the first technical certification I (almost) attempted:  The Apple Certified Server Engineer.

Back in the 1990’s, I worked for a small company doing desktop and network support.  When I say small, I mean small.  We had 60 employees and 30 of them had computers.  Still, it was where I first got into the computer industry, and I learned a surprising amount as networking was just starting to take off.

I administered a single AppleShare file server for the company, and I even set up my very first router, a Dayna Pathfinder.  I was looking for more, however, and I didn’t have much of a resume.  A year and a half of desktop support for 30 users was not impressing recruiters.  I felt I needed some sort of credential to prove my worth.

At the time Microsoft certifications, in particular the MCSE, were a hot commodity.  Apple decided to introduce its own program, the ACSE.  Bear in mind, this was back before Steve Jobs returned to Apple.  In the “beige-box” era of Apple, their products were not particularly popular, especially with corporations.  Nonetheless, I saw the ACSE as my ticket out of my pathetic little job.  I set to work on preparing for it.  If memory serves (and I can find little in the Wayback machine), the certification consisted of four exams covering AppleTalk networking, AppleShare file servers, and Backup.

Apple outsourced the certification development to a company called Network Frontiers, and its colorful leader, Dorian Cougias.  I had seen Dorian present at Macworld Expo once, and he clearly was very knowledgeable.  (He asked the room “what’s the difference between a switch and a bridge?” and then answered his own question.  “Marketing.”  Good answer.)  Dorian wrote all of the textbooks required for the program.  He may have known his stuff, but I found his writing style insufferable.  The books were written in an overly conversational tone, and laced with constant bad jokes.  (“To remove the jacketing of the cable you need a special tool…  I’d call it a ‘stripper’ but my mother is reading this.”  Ugh…)  A little levity in technical documentation is nice, but this got annoying fast.

This was in the era before Google, and despite my annoyance I did scour the books for scarce information on how networking actually worked.  I didn’t really study them, however, which you need to do if you want to pass a test.  I downloaded the practice exam on my Powerbook 140 laptop and fired it up.  I assumed with my day-to-day work and having read the book, I’d pass the sample exam and be ready for the real deal.

Instead, I scored 40%.  I used to be a bit dramatic back in my twenties, and went into a severe depression.  “40%???  I know this stuff!  I do it every day!  I read the book!  I’ll never get out of this stupid job!!!”  I had my first ocular migraine the next day.

In reality, it doesn’t matter how good or bad, easy or hard an exam is.  You’re not going to pass it on the first go without even studying.  And this was a practice exam.  I should have taken it four or five times, like I learned to do eventually studying Boson exams for my CCNP.

Instead, I gave up.  And, very shortly later, Apple cancelled the program due to a lack of interest.  Good thing I didn’t waste a lot of time on it.  Of course, I managed to get another job, and pass a few tests along the way.

I learned a few things about technical certifications from that.  In the first place, they can often involve learning a lot of knowledge that may not be practical.  Next, you can’t pass them without studying for them.  Also, that the value and long-term viability of the exams are largely up to the whims of the vendors.  And finally, don’t trust a certification when the author of the study materials thinks he’s Jerry Seinfeld.

 

3

Product Management

Posted by ccie14023 on January 11, 2023
Posted in: Perspectives. Tagged: , , , , . Leave a Comment

It’s impossible to count how many people at my college wanted to be “writers”.  So many early-twenty-somethings here in the US think they are going to spend their lives as screenwriters or novelists.  My colleagues from India tell me most people there want to be doctors or engineers, which tells you something about the decline of the United States.

Back in the mid-2000’s, a popular buddy-comedy came out about a novelist and an actor and their adventures in the “California wine country”.  The author of the film is an LA novelist.  The only people he knew, and the only characters he could create, were writers and actors.  I read that his first novel was about a screenwriter.  The movie was popular, but I found the characters utterly boring.  Who cares about a novelist and his romantic adventures?  Herman Melville spent years at sea, giving him the material to write Moby Dick.  Fyodor Dostoevsky wanted to be a writer from an early age, but he spent years in a prison camp followed by years of forced military service, to give him a view into nihilism and its effect on the human soul.  The point is, these great writers earned the right to talk about something, they didn’t just go to college and come out a genius with brilliant things to say.

I’ve been hearing a lot about “product management” lately.  I work in product management, in fact, and I’ve worked with product managers for many years.  However, I didn’t realize until recently that product management is the hot new field.  Everyone wants to major in PM in business school.  As one VP I know told me, “people want to be PMs because that’s where CEOs come from.”  Well, like 19-year-olds feeling entitled to be great novelists, b-school students are apparently expecting to become CEOs.  Somewhere missing in this sense of entitlement is that achievement has to be earned, and that is has to be earned by developing specific expertise.  A college student who wants to be a novelist thinks he or she simply deserves to be a novelist by virtue of his or her brilliance;  a b-school PM student apparently thinks the same way about being a CEO.

Back when I worked in TAC, one of my mentors was a TAC engineer who had previously been a product manager for GSR (12000-series) line cards.  He went back to TAC because he wanted to get into the new CRS-1 router and felt it was the best place to learn the new product quickly.  It made sense at the time, but it is inconceivable now that a PM would go to TAC.  The product manager career path is directed towards managing business, not technology, and it would be a step down for product managers to become technical again.

If you don’t work for a tech company, you may not know a lot about product management, but PMs are very important to the development of the products you use.  They decide what products are brought to market;  what features they will have;  they prioritize product roadmaps.  They are held accountable for the revenue (or lack thereof) for a product.

Imagine, now, that somebody with that responsibility for, say, a router has no direct experience as a network engineer, but instead has an MBA from Kellogg or Haas or Wharton.  They’ve studied product management as a discipline, but know nothing about the technology that they own.  Suppose this person has no particular interest in or passion for their field–they just want to succeed in business and be a CEO some day.  What do you think the roadmap will look like?  Do you think the product will take into account the needs of the customer?  When various technologists come to such a PM, will he be able to rationally sort through their competing proposals and select the correct technology?

To be clear, I am not criticizing any individual or my current employer here.  This problem extends industry-wide and explains why so many badly conceived products exist.  The problem of corporatism, which I’ve written about often, extends beyond product management too.  How often are decisions in IT departments made by business people who have little to no experience in the field they are responsible for?  I got into network engineering because I was fascinated by it and loved it.  I’m not the best engineer out there–I’ve worked with some brilliant people–but I do care about the industry and the products we make.  And most importantly, I care about network engineers because I’ve been one.

Corporatists believe generic management principles can be learned which apply to any business, and that they don’t really need domain-specific expertise.  They know business, so why would they?  True, there are some “business” specific tasks like finance that where generic business knowledge is really all that’s needed.  But the mistaken thinking that generic business knowledge qualifies one to be authoritative on technical topics doesn’t make sense.  This is how tech CEO’s end up CEO of coffee companies–it’s just business, right?

I don’t mean to denigrate product management as a discipline.  PMs have an important role to play, and product management is the art of making decisions between different alternatives with constrained resources.  I am saying this:  that if you want to become a product manager, spend the time to learn not just the business, but the actual thing you are product managing.  You’d be better off spending a couple years in TAC out of business school than going straight into PM.  Not that many CEO-aspiring PMs would ever do that, these days.

Now off to write my first novel.

2

Cleared for nothing

Posted by ccie14023 on December 22, 2022
Posted in: Reflections. Tagged: , , , , , , , . 1 comment

I wrote this post on Feb 20, 2020, and I always thought it was an entertaining episode.  FBI Special Agent Elvis Chan, who features prominently in the post, has been in the news lately as he played a major role in the Twitter Files.  I will stay out of politics, except to note that Elvis was indeed a liaison to the business community, as seen here.

I was working at Juniper when the CIO asked me to apply for a government security clearance.  There were a number of hacking attempts on our network, and a security clearance would make me eligible for briefings from the government on the nature and scope of the threats against the United States’ networks.  Being one of the few US citizens in our department, and having a security background, it made sense.

I met with our “FSO”, the on-site liaison to the clearance-granting agency, in this case the Department of Defense.  I’ll call him Billy.  Billy pointed me to the government web site which housed the application, called “OPM”.  The OPM application was extensive, requiring me to input huge amounts of information about myself and my family.  It required a bit of work to track down some of the information, and when I printed the PDF copy of the application it totaled around eighty pages.

One day Billy called me into his office and told me I had been awarded a secret clearance.  He let me know that I could be subject to the death penalty if I divulged any classified information.  I signed some documents, and that was it. “Don’t I get a card for my wallet or anything?” I asked Billy.  He just smiled.

Shortly after getting my clearance, one of our other cleared employees brought me into a secure office in one of Juniper’s buildings where we could look at classified information.  He pulled a secured laptop out of a locked drawer, and a password out of a sealed envelope.  We began perusing classified information.  None of it was relevant to us, and none of it was particularly memorable.  For example, we read an article about several criminal gangs, the existence of which was unclassified.  The only classified information in the article happened to be the names of particular gangs.  They didn’t mean much to me, and I probably forgot them within a day or two.

One day I was invited to the San Francisco FBI office, to receive a classified briefing.  Billy had to fax the clearance over, because the DoD and FBI didn’t have an electronic way to exchange clearances.  I showed up, excited, to the federal building in San Francisco and proceeded up to the floor where the briefing was to take place.  Nobody was there.  I wandered around the white hallway with locked doors unable to make contact with anyone.  The elevator opened after a few minutes, and another equally confused attendee emerged.  We were wandering around for several minutes before someone showed up and told us to go to a different floor.

On the new floor a couple of young-looking FBI agents setup a table, checked our ID’s, and then took our cell phones.  The security did not seem very rigorous.  They then admitted us to the SCIF, or Sensitive Compartmented Information Facility.  The room we were led into was just a conference room, with a low ceiling and no windows.  Another young-looking FBI agent approached me, wearing a tie but no coat.  “Hi, I’m Elvis,” he said.

“I’m a special agent and the coordinator of the briefing today.  We’re very excited to have you here.”

We had a brief conversation about my job and role, and then I asked to use the bathroom.

“Go out the back door of the SCIF and hang a right, he said.”

I did this, and found myself walking with a wall on my right, and a row of waist-level cubicles on my left.  Nobody was in the the cubes, but paperwork was sitting on most of the desks. I wanted to peer at the paperwork as I walked by.  I have a clearance, I figured, so if I had a right to at least take a peek and see if the names of anyone I knew appeared.  Unfortunately, without pausing and staring, a chance I didn’t want to take, I couldn’t read anything.

I found the bathroom, and as I was participating in nature’s call, a couple of guys came in wearing ties but no sport coats.  They each had side-arms on their belts.  I wondered why these agents, who are basically office workers, needed to walk around armed.

As I came out of the bathroom, a female FBI agent was standing there, tapping her foot in anticipation of my emergence.  She looked like my school librarian.  Diminutive in stature, she had a side-arm that looked as big as she was.

“Are you FBI?” she asked pointedly.

“No,” I replied, thinking the answer was obvious.

She let out a long sigh, looking like a satisfied cop who has caught a perp.  “You can’t be here without an escort,” she scolded me.

“But Elvis told me I could!” was my retort.  I had a sudden realization that, in a large FBI office like San Francisco’s, it was entirely possible that not every FBI agent knew every other FBI agent, and that my host agent may have been entirely unknown to her.  Here I was, by myself, in the inner sanctum of an FBI office, explaining to an armed federal agent that I happened to be there because Elvis had sent me.

Fortunately, a glimmer of recognition flashed across her stern countenance.  “Oh, Elvis!” she said, exasperated.  “Come on,” she snapped, and led me back to the SCIF.

Back in the SCIF, the briefing began.  The first presenter was an FBI agent wearing a tie, with a coat this time.  Whatever he had learned at the FBI training center in Quantico, VA apparently did not include the fundamentals of haberdashery.  Anyone who buys a suit knows that you immediately have it tailored, as the pant legs are way too long.  Apparently this agent bought his cream-colored suit, with piping, and never sent it for alterations.  The trouser legs were so long he was actually walking on the bottom of his pant legs.  His presentation was no better than his tailoring.  Presenting on computer security, it was clear this was not somebody with even a basic knowledge of computing.

After him, two Homeland Security analysts presented.  They wore rumpled khakis with jacket and tie, and sported similar pyramid mustaches.  They presented on SCADA systems, a subject I could care less about.  Almost all of it was unclassified.

Shortly after my briefing, I learned that the OPM database had been hacked by the Chinese military.  All the personal information about myself and my family is in their hands now.  When I left Juniper, Cisco declined to renew my security clearance.

Some people hide that they have/had a clearance, as they can be targeted by foreign governments.  Personally, I don’t care.  What little classified information I saw, I can’t remember.  You could waterboard me and I wouldn’t be able to tell you a thing.

3

On being a VOIP guy (not)

Posted by ccie14023 on December 22, 2022
Posted in: NetStalgia. Tagged: , , , . Leave a Comment

Well, the blog has been languishing for a while, as I’ve been extraordinarily busy with a new EVP, a round of layoffs, and many personal distractions.  Here’s a little Netstalgia piece, not really technical, for your enjoyment.

I’ve told a few stories about my years at the Cisco Gold Partner, where I did both pre- and post-sales roles.  The Cisco practice in the San Francisco office was new, so being the only Cisco guy required wearing a lot of hats.  That said, one day I wore a hat I didn’t expect or want.

At the end of every week I’d look at our calendar to figure out my schedule for the next week.  It was maintained by a project coordinator.  Some appointments I had put on the calendar myself, others were requested by account managers or customers directly.  One day as I looked at my calendar, I saw the following week booked.  “City of San Mateo,” it said.  I had no experience with this customer, so I called our project coordinator to figure out what the mystery job was.

“You’ll be placing phones,” she said.

“Placing them?” I asked, confused.  She told me we had sold a VOIP deal with San Mateo to replace all of their PBX-phones with Cisco IP phones.  The entire San Francisco office had been roped in to physically placing the phones on desks across the city.  Even our Citrix guy was going to be there.

I called my VP of services and complained.  “I have two CCIEs and you want me to run around for a week plugging in phones?”

“Just be glad you have billable hours,” he said.  Were we really that desperate?

It turns out, yes.  Myself, the office Citrix guy, and one or two other folks met in San Mateo city hall and divided up box after box of IP phones.  We had to do city hall and library, which were the easiest.  Then I ended up doing the police headquarters.  I remember putting phones on all the desks in the detective room, with concerned police officers looking on as I rooted around on my hands and knees for data jacks under their desks.  I had to move weapons (non-lethal), ballistic vests, and other police gear to find the ports.

I also had to do the fire department.  For a small city, San Mateo has a lot of fire stations.  It wasn’t always easy to park.  The first one I pulled up to in my BMW, loaded with phones, had no parking anywhere.  I found a notepad and a pencil in my car, scrawled out “OFFICIAL BUSINESS” on a sheet of lined paper, stuck it in my window, and parked on the sidewalk.  I used my pass at several fire stations, earning quizzical looks from firemen when I parked myself on the sidewalk in front of their station.

I learned an important lesson in leadership from this event.  If the VP had called a meeting the week before, he could have said the following:  “Look team, I know you’re all highly skilled and don’t want to do manual labor.  But we have a big deal here, it’s important to the company, and it’s all hands on deck.  I’ll be there myself with you placing phones.  Let’s get this done and I’ll buy you all a nice dinner at the end of the week.”  Had he said something like this, I think we would have rallied around him.  Instead, he just surreptitiously had it added to the calendar and copped an attitude when challenged.  He actually wasn’t a bad guy, but he missed on this one.

Anyways, plugging in phones is the closest I became to being a VOIP guy.

2

Computers with Brains

Posted by ccie14023 on October 25, 2022
Posted in: Perspectives. Tagged: , , , , , , . Leave a Comment

A couple of years back I purchased an AI-powered energy monitoring system for my home.  It clips on to the power mains and monitors amperage/wattage.  I can view an a graph showing energy usage over time, which is really quite helpful to keep tabs on my electricity consumption at a time when electricity is expensive.

The AI part identifies what devices are drawing power in my house.  Based simply on wattage patterns, so they claim, the app will tell me this device is a light, that device is an air conditioner, and so on.  An electric oven, for example, consumes so much power and switches itself on and off in such a pattern that AI can identify it.  The company has a large database of all of the sorts of products that can be plugged into an outlet, and it uses its database to figure out what you have connected.

So far my AI energy monitor has identified ten different heaters in my house.  That’s really cool, except for the fact that I have exactly one heater.  When the message popped up saying “We’ve identified a new device!  Heater #10!”, I must admit I wasn’t surprised.  It did raise an eyebrow, however, given that it was summer and over 100 degrees (38 C) outside.  At the very least, you’d think the algorithm could correlate location and weather data with its guesses.

Many “futurists” who lurk around Silicon Valley believe in a few years we’ll live for ever when we merge our brains with AI.  I’ve noticed that most of these “futurists” have no technological expertise at all.  Usually they’re journalists or marketing experts.  I, on the other hand, deal with technology every day, and it leaves me more than a little skeptical of the “AI” wave that’s been sweeping over the Valley for a few years.

Of course, once the “analysts” identify a trend, all of us vendors need to move on it.  (“SASE was hot last fall, but this season SSE is in!”)  A part of that involves labeling things with the latest buzzword even when they have nothing to do with it.  (Don’t get me started on “controllers”…)  One vendor has a tool that opens a TAC case after detecting a problem.  They call this something like “AI-driven issue resolution.”  Never mind that a human being gets the TAC case and has to troubleshoot it–this is the exact opposite of AI.  We can broaden the term to mean a computer doing anything on its own, in this case calling a human.  Hey, is there a better indicator of intelligence than asking for help?

Dynamic baselines are neat.  I remember finding the threshold altering capabilities in NMS tools useless back in the 90’s.  Do I set it at 50% of bandwidth?  60%?  80%?  Dynamic baselining determines the normal traffic (or whatever) level at a given time, and sets a variable threshold based on historical data.  It’s AI, I suppose, but it’s basically just pattern analysis.

True issue resolution is a remarkably harder problem.  I once sat in a product meeting where we had been asked to determine all of the different scenarios the tool we were developing would be able to troubleshoot.  Then we were to determine the steps the “AI” would take (i.e., what CLI to execute.)  We built slide after slide, racking our brains for all the ways networks fail and how we’d troubleshoot them.

The problem with this approach is that if you think of 100 ways networks fail, when a customer deploys the product it will fail in the 101st way.  Networks are large distributed systems, running multiple protocols, connecting multiple operating systems, with different media types and they have ways of failing, sometimes spectacularly, that nobody ever thinks about.  A human being can think adaptively and dynamically in a way that a computer cannot.  Troubleshooting an outage involves collecting data from multiple sources, and then thinking through the problem until a resolution is found.  How many times, when I was in TAC, did I grab two or three other engineers to sit around a whiteboard and debate what the problem could be?  Using our collective knowledge and experience, bouncing ideas off of one another, we would often come up with creative approaches to the problem at hand and solve it.  I just don’t see AI doing that.  So, maybe it’s a good thing it phones home for help.

I do see a role for AI and its analysis capabilities in providing troubleshooting information on common problems.  Also, data can be a problem for humans to process.  We’re inundated by numbers and often cannot easily find patterns in what we are presented.  AI-type tools can help to aggregate and analyze data from numerous sources in a single place.  So, I’m by no means saying we should be stuck in 1995 for our NMS tools.  But I don’t see AI tools replacing network operations teams any time soon, despite what may be sold.

And I certainly have no plans to live forever by fusing my brain with a computer.  We can leave that to science fiction writers, and their more respectable colleagues, the futurists.

2

Plain Language

Posted by ccie14023 on October 3, 2022
Posted in: Reflections. Tagged: , , , . Leave a Comment

I don’t know about the rest of the world, but here in America our police speak in a sort of code language.  Instead of saying “he got out of the car and walked away,” the police will say, “the subject exited the vehicle and proceeded on foot.”  It’s not that their language is any clearer–in fact it’s less clear.  When talking to each other, the cops like to use language like this because it seems to elevate them and make them sound more professional.

A friend of mine told me a story of his friend who had always wanted to be a cop, but was too much of a screw-up to make it to the academy.  Despite his failure he remained a police buff, perhaps a cop in his own mind.  This fellow witnessed a crime and the local sheriff showed up.  This non-cop started to describe the crime to the sheriff as cops do:  “The subject exited the vehicle and proceeded to commit a 4-15…”  The deputy cut him off and shouted, “speak English, boy!”  The poor police-wannabe never lived it down.

Corporatist-types have a language like this too.  Attempting to sound smart and professional, they speak in an often-inaccessible code language replete with b-school buzz words.  I must admit, as long as I’ve been in the corporate world, I’ve frequently been confronted by language that I simply couldn’t understand.

“We collaborated with engineering and cross-functional product managers across multiple time zones to groom and prioritize backlog to ensure efficient program delivery.”  Huh?  “We need to build a motion that creates value at scale.”  What? “The adoption journey enables us to innovate continuously.”  Speak English, boy!

Then there was this gem, from an MBA describing a customer problem:  “The customer is not in the mindset of extracting value from the product.”  A lot of words there.  How about three words:  “It don’t work.”  Or, “customer hates it.”  Oh no.   If the MBA spoke that way, he’d sound like he learned nothing at his prestigious business school.  Professionals speak professionally, you see.  If he spoke like a normal human being, some people might suspect he actually doesn’t really know anything.  Although to be honest, that’s exactly what I was suspecting when he started talking about mindsets.

We can all fall into this trap, I’m afraid.  I refused, for years, to use “ask” and “spend” as nouns, because they’re not nouns.  (I remember an internal thread at Cisco years back in which someone said, “shouldn’t we productize that?”  The snarky response came back from an engineer, “no because at Cisco we can’t turn nouns into verbs.”)  Alas, I’ve surrendered to the progress of business-speak and have replaced “request” with “ask.”  Saving one syllable with a frequently-used word has certainly given me hours back to do other things, don’t you think?

Technical people can certainly fall into this and we have our own jargon.  Some of it is necessary.  Here is a snippet of a Cisco doc:  “To overcome the limitations of the flood-and-learn VXLAN as defined in RFC 7348, organizations can use Multiprotocol Border Gateway Protocol Ethernet Virtual Private Network (MP-BGP EVPN) as the control plane for VXLAN. ”  This is wordy, and it is jargony.  That said, I can’t think of a better way to say it.  This sort of language is unavoidable for network engineers.

What I don’t like is technical people adopting MBA-speak because they’re surrounded by it.  “Our latest release provides flexible options to operationalize your business intent.”  Oh dear, even if you get into some good technical meat, you’ve lost me already.  The simple secret for me in winning Cisco Live Hall of Fame for my speaking is simply to state things in plain, clear language.  Technical, yes, but clear.

I used to think I was stupid, sitting in meetings in the corporate world and not understanding what on Earth people were saying.  Then I learned that in many cases, the speakers didn’t understand what they were saying either.  In the event that they actually do, a few pointed questions can usually cut through the fog of fancy words.  I’m convinced many of the mistakes made in the corporate world would never happen if people actually spoke like normal people.

If you feel tempted to obscure your language to sound like you’re oh-so-smart, remember the advice of the deputy sheriff:  Speak English, boy!

8