Category Archives: Perspectives

by

Topological Qubits

I’ve been very unconvinced on the reality and benefits of quantum computing.  Sure, a lot of people with fancy degrees from fancy places say it will work miracles.  Sure, they make really impressive machines with cooling units that look like they’ll beam you into the movie Tron.  Sure, Microsoft just released a new chip with a Hollywood-grade video, B-roll of high-end oscilloscopes included.  All of it is based on some very-theoretical theoretical physics, and I’m not sure that these machines will deliver what they promise.  Sure, I may be wrong.  I’m often wrong with technology predictions.

Satya Nadella, the CEO of Microsoft, announced their Majorana 1 chip, their leap forward in quantum computing, by saying “Imagine a chip that can fit in the palm of your hand yet is capable of solving problems that even all the computers on Earth today combined could not!”

Is it just me or does anyone else find that statement, I dunno…maybe…a little fucking concerning?

More computing power than ALL of the computers on Earth combined?  Do we ever stop to think if this sort of thing is really a good idea?

Oh sure, their video, with slick and well-spoken physicists, extols the ability of their topological qubits to invent medicines, develop new materials, run EV batteries forever, and all but solve world hunger.

On the other hand, nuclear physics gave us both nuclear power and nuclear weapons.  Science fiction movies have been warning about world domination by machines for decades.  If we unleash ChatGPT powered by more horsepower than all computers combined, what the hell is it going to invent?  Why would this be confined to materials scientists in a lab?  Wouldn’t the machine start doing whatever it wants?  Why wouldn’t it invent a fatal virus, unleash it, and rule the world itself?  Or at least, wouldn’t nefarious human beings try to use it to cook up a weapon that could hold the planet hostage, like in a James Bond movie?  Could it enable mass-scale spying and privacy invasion by governments?  Will it be smart enough to warn us of the negative consequences of the materials it invents, or will we be inundated by worse than the microplastics in our brains which weigh as much as a spoon?

I’ve been sick of the tech industry’s worship of technological progress for a long time.  All the hype assumes that technological progress is always and everywhere good.  But that’s been proven false, time and again.

Meanwhile, Nadella makes the asinine statement:  “When productivity rises, economies grow faster, benefiting every sector and every corner of the globe.”  I assume a computer that is more powerful than all computers put together will eliminate a hell of a lot of jobs.  Perhaps it might render human beings redundant.  Technological innovations always tend to replace human labor (except where it depends on third-world exploitation), wouldn’t a computer this ridiculously powerful destroy entire industries and career paths?

I’ve seen enough marketing to suspect Microsoft is exaggerating here.  They’re more than likely less interested in selling quantum computers, and more interested in selling quantum-ready products and services.  Anyways, I sure hope so.

 

by

Agentic AI for networking

As I’ve pointed out in several posts, and as you’ve certainly noticed, there is a teeny bit of hype surrounding AI these days. We’re told network engineers will be obsolete as our AI buddies take over our jobs. Want to roll out a network? No problem, your receptionist will do it for you while sipping a latte, pushing a button to call her AI agent. Some of us take a more realistic view. AI is a tool. An amazing one in many ways, but a tool nonetheless. And the big question is: how useful is it, really?

Most of the folks making the wild claims are marketers, “analysts”, and others who know little to nothing about networking. John Capobianco is certainly not one of these. John knows a lot about networking, and is a truly technical guy. (I’ve never met him in person, so he could be an AI agent, but I hope not.) John recently posted a detailed video about running an AI agent per device, and aggregating these up to a sort of agent-of-agents. His POC demonstrates that his AoA can go out and perform operations on multiple network devices. Cool.

That said, it brings me back to my early days in programmability, 2015 or so. One of the demos we used to run used Alexa. You could give voice commands to Alexa to go configure your routers, or collect data from them. Very cool! How many network engineers to date use Alexa to configure their networks? Approximately zero.

Of course, we weren’t really expecting to kick off a craze of Alexa-network-engineers. The demo was, as my peer Jason Frazier liked to say, about the “art of the possible.” Our model-driven interfaces made it that much easier to connect things in ways that weren’t previously possible.

As I mentioned in a recent post, programmability didn’t always click with network engineers. We would do a demo where we configured VLAN 100 using NETCONF. We just wrapped a giant block of XML in a giant block of Python, and–voilá–VLAN 100! The only problem was, every network engineer who saw the demo rolled his eyes and said, “I could do that with one line of CLI.”

Here’s the question for Capobianco: Is typing “Please configure an NTP server of 192.168.100.100 on all four devices” easier than, say, configuring an NTP server on all four devices? Or even using an Ansible script to do so? Is typing “What is the IP address of Ethernet 0/1 on R1 and R2” better than just going to R1 and R2 and looking at the “sh ip int brief” output?

We must also bear in mind that AI, still, has trouble with more complex tasks. In an earlier post I found it could not configure VRRP for IPv6 correctly. Even after providing it the operational output of the failed config, it still couldn’t provide the correct config. So, NTP servers are fine, but when we get into really complex stuff, will AI shine or fail?

I’ve been finding ChatGPT incredibly helpful working in my lab lately. I needed to do a few things on ProxMox, and it saved me a lot of searching through StackExchange. But if we want to go from useful tool to fully agentic network management, well, we have a long way to go. Right now the agentic demos feel a bit like the Alexa ones–the art of the possible, but not necessarily the probable.

3
by

VRRP, IPv6, and the “Why” Question

Ivan Pelpnjak has an interesting post (H/T to Ethan Banks) critiquing Cisco’s VRRP v3 CLI implementation for IPv6.  It got me thinking about a question I was told never to ask:  “Why?”

Never ask why?

Back when I really wanted to get into networking, I took a five-day bootcamp with Global Knowledge taught by a gentleman named Tony Marshall.  It was four-days of study, and then on day 5 you took the CCNA exam.  Tony was an awesome teacher, and I passed with flying colors.  However, if ever you asked Tony “Why?”, he would interrupt you before you finished your sentence and say, “never ask why!”  The way Cisco implemented it was the gospel truth handed down from above.  This is probably the correct approach for a four-day CCNA bootcamp, and producing automatons who never question Cisco’s decisions was also great for business.  In product management we call that “competitive advantage.”

Since I’ve been doing management stuff for eight years my CLI skills are a little rusty.  I’m going to walk through the VRRP problem in a lot more detail, largely for my own benefit, and then circle back to the “Why?” question in the broader sense.

VRRP and IPv6

Virtual Router Redundancy Protocol (VRRP) is a protocol used to provide default gateway redundancy, and is an IETF cousin of Cisco’s earlier Hot Standby Routing Protocol (HSRP).  VRRP allows two (or more) routers to share a single IP address, which one router actively responds to.  Should the router go down, the secondary router begins responding to that address.  If you’re reading this blog, you probably already know that.

VRRP is simple to configure.  I set up a VRRP group in my lab, using VRRP v3, which supports both IPv4 and IPv6 addresses:

s1(config-if)#vrrp 1 address-family ipv4
s1(config-if-vrrp)#address 10.1.1.100 primary
s1(config-if-vrrp)#priority 200

Here you can see I specify we’re using a v4 address, and I made 10.1.1.100 the primary address.  (You can have more.)  I also configured the priority value for this router (actually switch).  I did the same on the other router with a lower priority, and viola, we have a VIP responding to traffic.  From another router on this segment:

r1#p 10.1.1.100
.!!!!

 

How does ARP work in this case?  The MAC address used comes from a reserved block designated for VRRP.  The last octet is the VRRP group number.

r1#sh arp
Internet 10.1.1.100 0 0000.5e00.0101 ARPA Ethernet0/0

 

If this were VRRP group 2, then the MAC address would be 0000.5e00.0102.

So far so good, but how does it work for IPv6?

If I just attempt to configure a global IPv6 address with VRRP, a couple funky things happen.  First, I cannot add the “primary” keyword:

s1(config-if)#vrrp 1 address-family ipv6
s1(config-if-vrrp)#address 2001:db8:cafe:100::100/64
s1(config-if-vrrp)#address 2001:db8:cafe:100::100/64 pri?
% Unrecognized command

 

It will take the command without the “primary” keyword, but VRRP gets stuck in INIT:

Vlan111 - Group 1 - Address-Family IPv6
State is INIT (No Primary virtual IP address configured)

 

As Ivan points out, this is per the RFC.   The RFC requires that the source IPv6 address be a link-local address.  If you haven’t done a lot of v6, recall that each interface will have a globally routable IPv6 address as well as a link-local address, which, as the name implies, can only be used on the local link and is not routable beyond that.  Without getting into all the details, the link local address is assigned from the FE80::/10 block, and on Ethernet interfaces it is made unique by using the interface MAC address.

OK, the RFC says:

“…each router has its own link-local IPv6 address on the LAN interface and a link-local IPv6 address per VRID that is shared with the other routers that serve the same VRID.” (Emphasis added.)

So the router has its own address (assigned automatically when IPv6 comes up), but each virtual router ID requires a link-local address too.  Elsewhere, the RFC states the primary address must be the link-local address.  But why?

The RFC is explicit that this link-local address is the source-address for VRRP packets, so presumably this is to keep the VRRP packets local, and to prevent them from being routed outside the segment.

Fair enough.  Let’s add a link-local primary address:

s1(config-if)#vrrp 1 address-family ipv6
s1(config-if-vrrp)#address FE80::1 primary

And…it works!

Vlan111 - Group 1 - Address-Family IPv6
  State is MASTER

Easy, right?  One more step, perhaps, but it’s not too hard.  Except, as Ivan rightly points out, some vendors (like Arista) assign this link-local address automatically for you based on the group MAC address.  Aside from the fact that this makes it easier (no need to keep track of another VRRP address), it makes interoperability challenging.  If I want my IOS XE device to be in the same VRRP group as an Arista device, I need them both agreeing on the LLA.  (Although, to be honest, I’m not totally convinced on this use case.  Usually devices peering in VRRP groups will be identical hardware, even in mixed-vendor environments.  But I might be wrong.)

Let’s go with Ivan’s thought that this is poorly designed.  I have access to internal Cisco tools that Ivan doesn’t.  What are people saying?  Are we getting TAC cases on this?

It turns out, not many.  I found one case in Spanish, which I can read (even without ChatGPT).  The customer complained he configured VRRP “pero no hay ping”.  No “hay ping” because no hay link-local address.  The TAC engineer pointed the customer to a bug, so why isn’t it fixed?  Well, the bug (login may be required) is a documentation bug, requesting the link-local address be added to the VRRP docs.  They added this:

Device(config-if-vrrp)# address FE80::46B6:BEFF:FE50:DBB0 primary

 

…but nowhere do they tell you how they derived that address.  Meanwhile, on one of our internal mailers, an engineer asked about just using FE80::1 like I did.  The development engineer working on IPv6 VRRP replied:

Is the FE80::1 recommended?    <<< usage is fine AFAIK

 

Usage is fine, ok, but is this the best way to go about assigning link-local addresses for VRRP?

The “Why?” Question

Back to the larger question of why.  Cisco is not the only company to have built confusing CLI.  In Ivan’s very post, he even notes that Dell allows you to skip the link-local address altogether, in flagrant violation of the RFC.  And years ago I wrote pieces on utterly baffling Juniper CLI, such as the concept of and syntax for configuring RIB Groups.

Who designs the CLI?  A committee of experts who used to work at Apple, after thorough user experience research, you think?  More than likely, it was designed by the engineer charged with implementing the protocol.  This engineer is certainly not a network engineer and does not need to deal with operating a network.  Sure, he/she can configure routers a bit, but mainly for lab testing of his own features.  What makes sense to an engineer burning the oil at 3am in Bangalore might not make sense for the rest of us, but as long as the feature works, some way, some how, he can close his assignment and move on to the next thing.

But surely the executives at the vendor care, right?  Short answer:  they will care, and care a lot, if they meet with a giant customer and the giant customer screams at them that feature XYZ is impossible to configure.  Otherwise, it’s highly unlikely a senior vice president cares how VRRP is configured, and also unlikely he knows what it is.  Thus, the development of CLI can be a bit Lord of the Flies.  No adults in the room, so to speak.

What about filing an actual bug to get it fixed?  I could do so, for Ivan’s sake, but two problems.  First, the bug will be the lowest severity possible and engineering will be more focused on other things.  Second, if they do get to it, they won’t want to fix it for fear of breaking existing configs.  Years ago I brought up the problem with IOS XE’s model-driven management syntax.  To enable NETCONF you type “netconf-yang” at the global level. To enable RESTCONF you type “restconf”.  GNMI is “gnmi-yang”.  Couldn’t we have a code block called “model-based-management” (or something) and then you enable each of the features under that?  No, engineering told me, people are already using it the old way.  (Honestly, like, nobody was in 2016, but anwyays…)  Great idea, Jeff, but we can’t do it.

The TL;DR summary for you:  Tony Marshall was right.  Never ask: “Why?”

Postscript for those who think AI will replace network engineers:

I asked ChatGPT to provide me a configuration for VRRP v6, and provided it the global standby address.  ChatGPT gave me the configuration without a primary address.  Then, when I showed ChatGPT the “show vrrp” output which explicitly says “No primary virtual IP address configured”, it responded by telling me to:

  1. Make sure IPv6 is enabled globally. (!)
  2. Make sure the interface is up (!!)
  3. Reapply exactly the same config [without an LLA] (!!!)

So apparently we built CLI so confusing that even genius-level AI cannot figure it out.  Or, we got a ways to go before we get replaced by ChatGPT.  Just sayin’, “analysts.”

2
by

Are network engineers obsolete?

Continuing on the theme of AI:

I can tell you what the MBAs are saying.  Remember, the MBAs know more about network engineering than you, despite your training and experience, because, well, they’re MBAs!  The went to Stanford!  Or Kellogg!  Or San Jose State!

The MBAs are sure you’re going to be replaced by AI.  I’ve personally seen comments from “analysts” saying that’s the case.  We’re not going to need network engineers anymore!  So all that stuff you’re learning about BGP or OSPF or whatever.  Don’t learn it!  AI will do it for you.  Poof!  Away with network engineers!  After all, they speak in a weird language we didn’t learn in MBA school!  (But it’s ok to speak in our own weird language.)

I happen to live in a world called “reality”, a mythical fairy-tale land to B-school professors.  In “reality”, life is a bit more complicated than B-school types believe it to be.  And those of us who actually have a bit of experience building networks and making them work see AI for what it is:  a tool.  Could it replace some network engineers?  Yes.  All of them?  Count me a skeptic.  But then again, I don’t have an MBA.

I use a note-taking app called Obsidian.  This is becoming more important to me the older I get, as my memory seems to be functioning less optimally.  I was thinking I needed to see a doctor until I talked to other folks my age who have the same problem.  Despite billionaires’ attempts to live forever, the human body has a shelf life and the clock starts ticking the moment you’re born.

Obsidian is great because it stores notes in text files, you write your notes in markdown, and you can generate tags.  I’m terrible at filing things, so I hate programs like OneNote which place the burden on the user.  You have to create notebooks and tabs, and so forth.  With Obsidian, I just add tags where I need them and I can find anything later on.

Anyways, I had taken extensive notes on Hypershield after talking for quite a while with a TME who is working on it.  In Obsidian, I work off of a “daily note” which is like a scratch pad for the day, and if something is important enough, I can select it and extract it to a new note.  In this case, I already had a note on Hypershield and I wanted to extract the text to this exiting note.  I couldn’t find a way to do it.

Enter ChatGPT.  Obsidian is highly extensible, and ChatGPT wrote me a Javascript to extract and append to an existing note.  Fantastic!  I put it into a plug-in, selected the text, gave it the name of the note, and bam!  My text vanished, nowhere to be found.  Not on the daily note, not on the scratchpad, not anywhere in the directory with the text files.  Since the deletion happened from a script, and not a keyboard/menu command, I couldn’t undo.  The text was lost.

ChatGPT was quite apologetic about the mixup and offered several ways to recover the text, none of which worked.

Hallucinations and errors from GenAI are nothing new.  We’re all aware of the problem, and I pointed it out in another context early on in the GenAI days.  But this is a huge problem for the predicted future of networks being totally automated by AI.  Despite the name, AI is not intelligent.  It is a system which generates language, human or machine, by ingesting a lot of content and trying to predict what the correct output should be.  And it can be wildly wrong.

Sure, we can implement checks and balances.  It would have been wise to copy my text to the clipboard before running the script.  Still, whenever we automate, we risk runway automation, and in environments where the network is absolutely critical, trusting AI is going to be difficult.

I’ve heard “analysts” say that their customers are complaining they cannot find qualified network engineers.  In their incredible logic, then they reason, let’s replace them with AI!  Brilliant.  How about asking why it’s so hard to find network engineers?  Is it a great career choice when you are young and reading that network engineers will be replaced by AI?

I’ve written about the “war on expertise” which has been going on in society in general, and in our field in particular.  I got into networking (a) because it fascinated me and (b) because it was a very promising career field.  When you have an interest and a passion for something, acquiring knowledge in that area becomes an end in and of itself.  I remember reading an article about NBA basketball referees, and how, when they travel, they love to challenge each other on obscure rules.  It was like this with network engineers.  We were always trying to one-up each other, learn a new protocol or knob, figure out the mysteries of multicast, or new ways to design networks.  When I go to Cisco Live (and for the second year in a row I will not be going to CL Europe), these are the sort of people who show up.  They are hungry to learn, expand their skills, and up their game.

Now you go to a trade show and all you hear about is how you will be replaced by AI.  Why the hell am I going to devote time to mastering a field if I’ll be obsolete in a couple years?

I’m worried for the future because we’re headed down a concerning path.  Nobody will be entering network engineering anymore, and AI won’t be able to replace the humans who operate networks.  Then what?

It would be nice to see networking vendors (and I mean all of them) encouraging people to learn networking, showing people this is a viable and lucrative career path, and that AI is a tool that they might use to learn networking and augment their capabilities.  And it would be nice to see MBAs stick to finance.  Or can we at least outsource the job of the MBAs to AI?  Yeah, that’d be nice.

8
by

The Next Big Thing, TLDR version

At the last Cisco Live in June, I was asked by marketing to do a “center stage” presentation.  My days of getting normal sessions at Cisco Live seem to be over.  Perhaps I’m too far into the management track (although that’s changing) to impress the Cisco Live Session Group Managers.  Eager to speak again, I accepted the proposal.

The abstract was provided for me.  I don’t remember the title, but it was something about AI and the campus.  So, I did my best to craft a set of slides that would be interesting.  When I ran them by marketing, I was told I couldn’t use my own slides.  I had to use theirs.  One of my secrets to success at Cisco Live is that I always build my own slides.  Rarely do I use a single slide from someone else.

Still, I did my best to build a story that would work.  Then I was told I’d be co-presenting with another PM, and we’d also have a customer on stage with us for an Oprah-style panel interview.  Even with these constraints, I spent a lot of time in my hotel room in Vegas practicing to be sure I nailed it.

The center stage is on the show floor (World of Solutions), and presenters there are broadcast onto a series of TVs scattered around the Mandalay Bay convention center.  They walk around the stage like they’re performing King Lear, but nobody watches the TVs or can even hear them.  It’s very performative, but a part of trade shows.

We had a rehearsal with marketing people, stage managers, cameramen, audio technicians, and and army of other people.  On the day of, there were marketing people, stage managers, cameramen, audio technicians, and and army of other people.  There was also a lady there who did intros for all the speakers, to get the audience pumped up.  I’m sure she showed up in Las Vegas decades ago to be a showgirl or something, but now in her 40’s she was doing corporate gigs at Mandalay.  As I got mic’d up and ready to go, I looked out at my audience.  Of the 50 or so chairs, 5 were occupied.  Four of them were friends of the customer presenting.  I looked at the intro lady and said: “I hope you can handle the stage fright.”  She laughed.

I did my shtick, and it all went well enough.  At the very end, the one attendee who was not with the customer, who seemed to have shown up because it was a good spot for a nap, arose like Lazarus, raised his hand and asked:  “Could you guys please stop talking about AI at Cisco Live?”

If you’ve watched the Art of Network Engineering podcast, you probably are familiar with Lexie Cooper, one of the hosts.  I was on the podcast a while back and had a nice talk with her and Andy Lepteff.  The other day, Lexie showed up in my LinkedIn feed in clownish makeup and a bodysuit.  With the audio off, I looked at it and thought, “wow she’s really desperate for attention.”  Then I unmuted it.  I nearly died.

“Have you ever considered…using AI?”  she begins.

“Manage your network devices…with AI!”

“Manage your IOT stuff…with AI!”

“Design a PCB…with AI!”

“Automate your vegetable garden…with AI!”

“Ethernet cables?  Nope…AI!”

“Every vendor in the World of Solutions…with AI!”

…and so forth.

In a minute and thirteen seconds, Lexie captured the Zeitgeist of the current networking world perfectly and hilariously.  It seems that all of the protocols and technologies that make up the “Art of Network Engineering” have been single-handedly wiped away by AI.  Nobody talks about networking anymore, it’s all just AI.

Of course, those protocols and technologies are necessary for AI, for the Internet, and for the modern world to function.  Why do all vendors suddenly have a single-minded focus on AI, and seem to have stopped investing in actual networking technology?

It comes down to the culture of Silicon Valley, the corporate world dominated by Wall Street, and the quest for the “Next Big Thing”.  As network engineers we love acronyms, so I’ll coin a new one:  the NBT.  (With all due respect to NetBIOS over TCP.)

Technology executives are terrified of missing the NBT, and they spend their careers chasing the NBT.  It’s not entirely their fault.  If a technology company is not investing in the NBT, then the industry “analysts” will write somber reports criticizing the company and hurting the stock value.  Because the industry “analysts” have MBAs in topics like marketing and finance, they are experts at technology, and “analyzing” what networking companies should sell to network engineers.  In fact, because they are MBAs, they are experts in anything, really, and far more so than people who actually study and learn their specific fields.

There have indeed been some real NBTs.  Wireless is a good example.  When I started in networking, pretty much everything was hard wired.  Wireless was a major transformation in networking, and a new and different technology domain.  (I’m still not great at understanding it, admittedly.)  Mobile devices and smartphones radically changed the world, and nobody can argue that.

Cloud computing is an interesting one.  First of all, it was (and is) a marketing term.  It refers to several things, but in a broad definition we could say it refers to using someone else’s computing resources instead of your own.  In the case of SaaS, someone else is hosting the application and giving you access to it, whereas in the case of IaaS, they merely host the computing power and you manage the app.  Either way, it was not a new idea.  The idea of shared computing resources has been around since the advent of computing.  In the early days, all computing was done on shared systems.  At the dawn of the Internet, I got my email and other services through an ISP.  I telnetted into their system to check my email.  And in the mid-90’s, I worked at a company that offered a SaaS-based timecard service, before anyone even used the term “SaaS”.

Cloud computing in 1999

Still, we could say Cloud was an NBT.  I used to go to auctions during the dot-bomb of the early 2000’s, and even a small dotcom company had to purchase servers and network gear and host them in rented rack space in a colo.  AWS drastically changed that.

Of course, there have been many potential NBTs that turned out not to be.  The “Metaverse” was one of these.  After 2 years in COVID lockdown, nobody was interested in slapping on a VR headset and meeting their friends using a unicorn avatar floating around a fake version of Mars.

Watch out when an exec begins a presentation with this apocryphal Henry Ford quote:  “If I had asked people what they wanted, they’d have said faster horses.”

Aside from the fact Ford never said it, this quote is recited ad nauseam to inspire people to disruptive innovation.  Nobody ever seems to notice the obvious, however.  The automobile was popularized by Henry Ford over 110 years ago.  It hasn’t changed much since.  Sure, your Subaru is a lot different from a Model T, but the basic idea and design are the same.  The changes to automobiles–fuel injection systems, automatic transmissions–have been major, but nonetheless incremental improvements on the base design.  Once the NBT happened and spawned an industry, things reached a steady state.

From a corporate/investor perspective, this is problematic.  Stock prices are an indicator of future value, and investors demand “growth”.  (Hypothetical question:  is there an end-state to “growth?”  I.e., is a company ever done growing, and if so, when?  Related:  is there anything in nature which can grow indefinitely?)  Steady-state is not good for Wall Street.  So, execs need to go hunting for the NBT.

“Now wait,” many MBAs will correct me.  “The EV is a major disruptor in the automotive industry.”

Leaving aside the fact that EVs have existed in the past, and their questionable future, it just proves my point.  It took 100 years for the Tesla to exist.  But let’s circle back to that in a minute.

Recently I saw a LinkedIn post from a woman, Debbie Gomez, who is making a career change to become a network engineer.  She was joking about the contents of a woman’s purse, comparing it to the books she has in her car.  One of those books was Internet Routing Architectures by Sam Halabi.

When I was studying for my CCIE R/S in 2004, I used Halabi’s book.  It’s clearly visible in a picture of the stack of books I used to study for the infamous exam.  Debbie is studying the same content I was 20 years ago.

This is because, like the automobile, once networking was invented, change became incremental.  BGP hasn’t changed much because it can’t change much.  It’s run across multiple providers and multiple vendors, and it’s not easy to make changes.  Sure, it’s been extended since Halabi’s day, but it’s close enough to the original that his book is still totally relevant.

I’ve written in the past about how non-technical executives view the complexity of networking as a creation of engineers who “revel in complexity”.  In their view, the NBT in networking is to just have “simplicity”, where you don’t need all the fancy BGP, OSPF, EIGRP, ISIS, EVPN, VXLAN, STP stuff.  Just like the Tesla is so much simpler than a traditional car.

I recently started working on cars, because I always like to do things with my hands.  My 2011 BMW 328i is probably the wrong car to start working on.  It’s complex, and designed so that simple tasks require disassembling large parts of the engine.  I recently replaced the valve cover, successfully, but man was it a nightmare of carefully removing various parts.  To even get the thing out took about 30 minutes of me standing on the engine and my brother-in-law working it from the side.  If I learned one thing, it’s how complex a modern car is.

I have a Tesla as well.  There’s no question it’s simple.  There’s hardly an engine to speak of.  There is no gear shifting when you drive it.  You don’t even turn it on.  There is no maintenance required except for tires and brakes. The only fluid required is for the windshield washer.

Many technology executives feel this transformation needs to happen for networking as well.  The problem–they don’t seem to realize–is that the underlying complexity of networking, the protocols, cannot go away.  They exist for a reason.  Can they be improved?  Sure.  Can they be eliminated?  No.

That’s not to say much of the mess of networking cannot be improved.  Vendors have created a lot of that mess, and all are guilty to some degree.  We can distinguish unnecessary complexity from necessary complexity.  A lot of it is unnecessary, but even if you remove that, you’re left with the necessary complexity.

The only option for simplicity when you cannot really simplify, is to abstract.  That is, you hide the complexity.  It’s still there, but it’s easier to deal with.  Take a modern airplane.  It’s just as complex a machine, perhaps more so, than a plane built in the 1970s.  But the cockpit is throughly automated, and the systems throughly instrumented.  It’s much easier to manage than a 1970’s plane.  And yet, someone still needs to know how it all works.

This brings us back to our starting point, AI.

Why is AI driving Lexie to the point of putting own garish makeup and screaming into the camera?  Of course, everyone thinks it’s the NBT.  But is it?

We can easily understate the importance of GenAI and the significance of the technological advancement.  It’s nothing short of astounding.  ChatGPT makes a great search engine, but apart from that, it’s ability to interpret and generate language and code in creative ways is incredible.

Even though I worked on programmability, my knowledge of Python is pretty poor.  If there’s one programming language I feel absolutely comfortable in, it’s Applesoft BASIC from the 1980s.  I’ve found I can have ChatGPT explain some of the more challenging Python concepts by translating them to BASIC.  It’s crazy.  Computers haven’t been able to do anything like that before.

I’ve asked it to generate NETCONF code blocks for configuring IOS XE, with less success.  It gave me an operational data model to configure an IP address on an interface.  These errors can and will be corrected, however.

And yet, even if AI reaches the point of being able to configure and operate network devices, it will still be an abstraction layer.  I cannot fathom AI somehow doing away with networking.  At most, it would be like the automation systems on the plane, not like a Tesla.

I asked ChatGPT to design a networking system that does not use protocols.  It responded:  “Designing a data networking system that does not use protocols is a challenging idea because protocols are fundamental to networking—they define the rules for data exchange.”  It then dutifully attempted to frame out a protocol-free system, but the result was unimpressive, and the AI admitted that it would have a lot of problems.

I am among those working on AI projects at Cisco, both out of interest and out of necessity.  Working at a vendor, I’m caught up in the NBT just like we all are.  While I cannot talk about the specifics of any of the projects, I do see potential for its use beyond the current applications of AI.  (Mainly analyzing operational data.)

Is it really the NBT?  Is it really a “disruptor” on the level of wireless or smartphones?  Or are we tilting at windmills as with the Metaverse?

Time will tell.  But I’m sure Lexie will have plenty of content for more videos.

Meanwhile, keep reading Halabi.  We still need him.

4
by

Cisco Live, again and again

It’s funny that I remember a time when Cisco Live used to be a privilege, before it became a chore.  I’ve been to every Cisco Live US and every Cisco Live Europe since I started here in 2015.  I enjoy the show primarily because I enjoy meeting with fellow network engineers, and there is still a lot of energy every show.  But I’ve seen all the booths, the flashy keynotes, and the marketing sessions many times over.  I’ve eaten the bad food and stayed in a few bad hotels.

Speaking was one of my favorite activities.  The audiences are intimidating, and as someone who used to have a terrible fear of public speaking, there is always a bit of jitters until my session(s) are over.  But I enjoy communicating technical concepts in a clear and understandable way.  That’s why I got into speaking.  I just wish I was teaching a college course where I grade the students instead of the students grading me.

After I transitioned from being an individual contributor to a manager, I held onto my speaking slots for a while, but the last Cisco Live where I spoke was Las Vegas 2022.  I attended, but didn’t speak, at Cisco Live Amsterdam 2023 and Cisco Live Las Vegas 2023.  It was nice not to have to prepare much, but I missed the “speaker” tab on my badge.  It just wasn’t the same.  Yet this is what happens when you move into leadership.  The sessions at Cisco Live are meant to be technical, and the SGMs who control sessions want technical speakers, not manager-types.  When you lead a team, you also don’t want to take sessions from your staff.

In Vegas, I swore I would not return to Cisco Live without a speaker badge.  If I didn’t get a slot, I would skip CL for the first time since I came back here in 2015.  I applied for two sessions for Europe in Feb 2024, trying to cash in on my “Hall of Fame” status.  The SGMs want relevant topics and good speakers.  I didn’t get any emails and assumed I wasn’t going to Amsterdam.  Have I finally jumped the shark?

Then I got a funny email.  “Do you want your Amsterdam session considered for Las Vegas?”  Huh?  Only speakers get that email.  I checked the Cisco Live Speaker Resource Center, and lo and behold, I saw one DevNet session there.

The session I submitted is called “Time Machine! A 1980’s bulletin board system, revived”.  What??  I submitted this as a total long shot, nearly positive it would be denied.  As detailed in a couple posts on this blog, I ran a bulletin board system in the 1980’s.  I proposed to do a DevNet session on how BBS’s worked and how I had revived my own in an emulator.  And they accepted it!  Ha!  Now I have to come up with the content.  I remember in the abstract where it asks for “business value of this session” and I said something like “There is absolutely no business value to this session.”  I guess the SGMs have a sense of humor.

I guess I’m going to Amsterdam.  I know for a fact one of my three regular readers goes to Cisco Live Europe.  So I look forward to seeing you there.  Meanwhile, I realized at this point in my career I need to submit my sessions in the “IT Leadership” track, so I’m writing some Las Vegas abstracts for that.  Can’t keep an old dog down!

7
1
by

Elon Musk and remote work

In a recent article (paywall), Elon Musk has once again turned his wrath to remote workers.  Elon has a lot of good ideas, but also many bad ones, such as naming his child “X AE A-XII”.  This is certainly proof that we don’t need to take everything he says seriously.

Elon has said that those who want to work remote are “detached from reality” and give off “Marie Antoinette vibes” (I will forgive his apparent misunderstanding of history).  His argument, to the extent he even articulates it, seems to have two angles:

  • First, it is “morally wrong” to work remotely because so many people in the world cannot do their jobs from home.
  • Second, the productivity of remote workers is not the same.  (I’m extrapolating a bit here.)

I don’t think the first argument is terribly serious.  Just because some people (food service workers, factory workers, etc.) cannot work from home does not mean I should not.  Long before remote work, some people worked in clean offices while others worked in filthy coal mines.  We can debate the injustices of life, but I’m not convinced this disparity should guide office policies.

As to the second, well, as manager of many large teams, I can say that some of my most productive workers are fully remote, i.e., they work entirely from home.  I can think of two or three of my most respected and productive employees who have this arrangement.  Because they don’t live near a Cisco office, they have no choice.  I recently promoted one of them to prinicpal, a major accomplishment that is not given out lightly.  So, we cannot say that working from home automatically means lack of productivity.

On the other hand, I’ve had some very poor performers who worked from home.  This was particularly the case during the lockdowns.  I remember one engineer who seemed to be doing nothing, and when I checked her calendar it was empty.  She took a job elsewhere, finally.  But I, as any good boss should, was well aware of her lack of contribution and would have done something had she not taken the initiative herself.

Does being in the office guarantee productivity?  Not at all!  I can sit around and watch YouTube videos at work just the same as I can from home.  I remember a guy who sat near me many years ago, and had a rearview mirror on his monitor.  He was always playing Solitaire and every time I, or anyone else, walked by his desk he would glance in the mirror and minimize the game.  He wasn’t fooling anyone.

For me, the noise and distractions of the office often make productive work difficult.  Thankfully, post-lockdown, several Cisco buildings are virtually empty, and I decamp to one of them if I need to get actual work done.  Pre-COVID I used to head out to a coffee shop, put in earphones, and get productive work done there.  Open offices are the worst for this.  They make serious work nearly impossible.

Then there’s this…  Let me be open about it.  I never agreed with the lockdowns.  When they first implemented them, I wrote every congressman, city councilman, county supervisor, the health department, the governor, the president, and pretty much anyone I could think of with my opposition to what seemed to me a lunatic idea, and totally unneeded.  Now you can disagree with me vehemently, you can think I’m a jerk, and that’s fine.  But here’s the point:  Almost all the large corporations bought into it.  They could have fought these mandates, but they went along with them, shut their doors, and embraced remote work.  Many started marketing campaigns (and still have them) around hybrid work.  You cannot go 100% in for the thing and then make a 180 degree shift a few years later because you regret your decision.  The outcome of the lockdowns–a lot of people unwilling to return to the office–was entirely predictable.  I think corporations need to embrace the world they created, and live with the consequences of their choices.  Your workers want to be remote, let them be remote.  Sure, give them incentives to come into the office and be together.  Encourage them to do so.  But accept the reality of the new world.

Elon Musk reopened his factories mid-lockdown.  He may not know how to name a child, but I’ll at least give him points for consistency.

by

ChatGPT and the problem of authority

I’ve been reluctant to comment on the latest fad in our industry, generative AI, simply because everybody has weighed in on it.  I do also try to avoid commenting on subjects outside of my scope of authority.  Increasingly, though, people are coming to me at work and asking how we can incorporate this technology into our products, how our competitors are doing it, and what our AI strategy is.  So I guess I am an authority.

To be honest, I didn’t play with ChatGPT until this week.  When I first looked at it, it wanted my email address and phone number and I wasn’t sure I wanted to provide that to our new AI overlords.  So I passed on it.  Then Cisco release an internal-only version, which is supposedly anonymous, so I decided to try it out.

My first impression was, as they say, “meh.”  Obviously its ability to interpret and generate natural language are amazing.  Having it recite details of its data set in the style of Faulkner was cool.  But overall, the responses seemed like warmed-over search-engine results.  I asked it if AI is environmentally irresponsible since it will require so much computing power.  The response was middle-of-the-road, “no, AI is not environmentally irresponsible” but “we need to do more to protect the environment.”  Blah, blah.  Non-committal, playing both sides of the coin.  Like almost all of its answers.

Then I decided to dive a bit deeper into a subject I know well:  Ancient Greek.  How accurately would ChatGPT be on a relatively obscure subject (and yet one with thousands of years of data!)

Even if you have no interest, bear with me.  I asked ChatGPT if it knew the difference between the Ionic dialect of Herodotus and the more common dialect of classical Athens.  (Our version, at least, does not allow proper names so I had to refer to Herodotus somewhat elliptically.)  It assured me it did.  I asked it to write “the men are arriving at Athens” in the dialect of Herodotus.  It wrote, “Ἀφίκοντο οἱ ἄνδρες εἰς Ἀθήνας,” which is obviously wrong.  The first word there, “aphikonto“, would actually be “apikonto” in the dialect of Herodotus.  He was well known for dropping aspirations.  The version ChatGPT gave me would be the classical Attic version.

I let ChatGPT know it was wrong, and it dutifully apologized.  Then I asked it to summarize the differences in the dialects.  It said to me:

Herodotus and Ionic writers typically removed initial aspirations, while the Attic dialect retained them. For example, “Ἀφίκοντο” (Herodotus) vs. “ἔφικοντο” (Attic)

Uh, you don’t need to know the Greek alphabet to see it made exactly the same mistake, again.  It should have said that Herodotus would use “Ἀπίκοντο” (apikonto) whereas in Attic the word would be “Ἀφίκοντο” (aphikonto.)

Here’s a little more computer-person-friendly example.  I decided to ask about my other favorite subject, retro computing.  I asked, “What is the difference between the Apple IIGS ProDOS 8, ProDOS 16, and GS/OS operating systems?”  Among other things, ChatGPT responded that GS/OS ” introduced features such as a desktop metaphor, drop-down menus, and a hierarchical file system.”  Which happens to be wrong.  ProDOS 16 actually introduced those features.  (In fact, ProDOS 8 introduced the hierarchy.)  I pointed this out.  “You are correct, and I apologize for the oversight in my previous response. I misspoke in stating that GS/OS introduced [those] features.”  Hmmm.

I’m testing it and I know the right answers.   The problem would arise if I were trusting ChatGPT to provide me with a correct answer.  There have been plenty of examples of mistakes made by ChatGPT, such as adding a “permit any” to the top of access-lists.

The issue is, ChatGPT sounds authoritative when it responds.  Because it is a computer and speaking in natural language, we have a tendency to trust it.  And yet it has consistently proven it can be quite wrong on even simple subjects.  In fact, our own version has the caveat “Cisco Enterprise Chat AI may produce inaccurate information about people, places, or facts” at the bottom of the page, and I’m sure most implementations of ChatGPT carry a similar warning.

Search engines place the burden of determining truth or fiction upon the user.  I get hundreds or thousands of results, and I have to decide which is credible based on the authority of the source, how convincing it sounds, etc.  AI provides one answer.  It has done the work for you.  Sure, you can probe further, but it many cases you won’t even know the answer served back is not trustworthy.  For that reason, I see AI tools to be potentially very misleading and potentially harmful in some circumstances.

That aside, I do like the fact I can dialog with it in ancient Latin and Greek, even if it makes mistakes.  It’s a good way to kill time in boring meetings.

7
by

Some thoughts on “digital twins”

As I mentioned in my last post, I like modeling networks using tools like Cisco Modeling Labs or GNS3.  I recalled how, back in TAC, I had access to a Cisco-internal (at the time) tool called IOS on Unix, or IOU.  This enabled me to recreate customer environments in minutes, with no need to hunt down hardware.  Obviously IOU didn’t work for every case.  Often times, the issue the customer raised was very hardware specific, even when it was a “routing protocol” issue.  However, if I could avoid hardware, I would do the recreate virtually.

When I worked at Juniper (in IT), we did a huge project to refresh the WAN.  This was just before SD-WAN came about.  We sourced VPLS from two different service providers, and then ran our own layer 3 MPLS on top of it.  The VPLS just gave us layer 2 connectivity, like a giant switch.  We had two POPs in each region which acted as aggregation points for smaller sites.  For these sites we had CE routers deployed on prem, connecting to PE routers in the POPs.  This is a basic service provider configuration, with us as a service provider.  Larger sites had PE routers on site, with the campus core routers acting as CEs.

We got all the advantages of layer 3 MPLS (traffic engineering, segmentation via VRF) without the headaches (peering at layer 3 with your SP, yuck!)

As the “network architect” for IT, I needed a way to model and test changes to the network.  I used a tool called VMM, which was similar to IOU.  Using a text file I could define a topology of routers, and their interconnections.  Then I used a Python script to start it up.  I then had a fully functional network model running under a hypervisor, and I could test stuff out.

I never recreated the entire network–it wasn’t necessary.  I created a virtual version with two simulated POPs, a tier 1 site (PE on prem), and a tier 2 site (PE in POP).  I don’t fully remember the details, there may have been one or two other sites in my model.

For strictly testing routing issues assuming normally functioning devices, my VMM-based model was a dream.  Before we rolled out changes we could test them in my virtual lab.  We could apply the configuration exactly as entered into the real device, to see what effect there would be in the network.  I just didn’t have the cool marketing term “digital twin” as it didn’t exist yet.

I remember working on a project to roll out multicast on the WAN using Next Generation Multicast VPN (NGMVPN).  NGMVPN was (is?) a complex beast, and as I designed the network and sorted out things like RP placement, I used my virtual lab.  I even filed bugs against Juniper’s NGMVPN code, bugs I found while using my virtual devices.  I remember the night we did I pilot rollout to two sites.  Our Boston office dropped off the network entirely.  Luckily we had out-of-band access and rolled back the config.  I SSHd into my virtual lab, applied the config, and spend a short amount of time diagnosing the problem (a duplicate loopback address applied), and did so without the stress of troubleshooting a live network.

I’ve always been a bit skeptical of the network simulation/modeling approach.  This is where you have some software intellgence layer that tries to “think through” the consequences of applied changes.  The problem is the variability of networks.  So many things can happen in so many ways.  Actual devices running actual NOS code in a virtual environment will behave exactly the way real devices will, given their constraints.  (Such as:  not emulating the harware precisely, not emulating all the different interface types, etc.)  I may be entirely wrong on this one, I’ve spent virtually no time with these products.

The problems I was modeling were protocol issues amongst a friendly group of routers.  When you add in campus networking, the complexity increases quite dramatically.  Aside from wireless being in the mix, you also have hundreds, thousands of non-network devices like laptops, printers, and phones which often cause networks to behave unpredictably.  I don’t think our AI models are yet at the point where they can predict what comes with that complexity.

Of course, the problem you have is always the one you don’t predict.  In TAC, most of the cases I took were bugs.  Hardware and software behaves unexpectedly.  As in the NGMVPN case, if there is a bug in software that is strictly protocol related, you might catch it in an emulation.  But many bugs exist only on certain hardware platforms, or in versions of software that don’t run virtually, etc.

As for digital twins, I do think learning to use CML (of course I’m Cisco-centric) or similar tools is very worthwhile.  Preparing for major changes offline in a virtual environment is a fantastic way to prep for the real thing.  Don’t forget, though, that things never go as planned, and thank goodness for that, as it gives us all job security.

4
by

Back in the lab

We all have to make a decision, at some point in our career, about whether or not we get into the management track.  At Cisco, there is a very strong path for individual contributors (IC).  You can be come a principal (director-level), a distinguished (senior director-level), and a fellow (VP-level) as an IC, never having to manage a soul.  When I married my wife, I told her:  “Never expect me to get into management, I’m a technical guy and I love being a technical guy, and I have zero interest in managing people.”

Thus, I surprised myself back in 2016 when my boss asked me, out of the blue, to step into management and I said yes.  Partly it was my love of the Technical Marketing Engineer role, partly my desire to have some authority behind my ideas.  At one point my team grew to fifty TMEs.

All technical people know that, when you go that route, your technical skills will atrophy as you will have less and less hands-on experience.  This is very true.  In the first couple of years, I kept up my formidible lab, then over time it sat in Cisco building 23, unused and consuming OpEx.  I almost surrendered it numerous times.

Through attrition and corporate shenanigans, my team is considerably smaller (25 or so) and run by a very strong management team.  Last week, I decided to bring the lab back up.  I’ve been spending a lot of time sorting through servers and devices, figuring out which to scrap and which to keep.  (Many of my old servers require Flash to access the CIMC, which is not feasible going forward.)  I haven’t used ESXi in years, and finding out I can now access vSphere in a browser–from my Mac!!–was a pleasant surprise.  Getting CIMCs upgraded, ESXi installed, and a functional Ubuntu server was a bit of a pain, but this is oddly the sort of pain I miss.

I have several Cat 9k switches in my lab, but I installed Cisco Modeling Labs on one of my servers.  (The nice thing about working for Cisco is the license is free.)  I used VIRL many years ago, which I absolutely hated.  CML is quite slick.  It was simple to install, and within a short time I had a lab up and running with a CSR1kv, a Cat 8k, and a virtual Cat 9k.

When I was in TAC I discovered IOS on Unix, or IOU.  Back then, TAC agents were each given a Sun Sparc station, and I used mine almost exclusively to run IOU.  (I thought it was so cool back then to have a Sun box on my desk.  And those of you who remember them will know what I mean when I say I miss the keyboard.)  IOU allowed me to define a topology in a text file, and then spin up several virtual IOS devices on the Sparc station in that topology.  It only supported sinulated Ethernet links, but for pure routing protocols cases, IOU was more than adequate to recreate a customer environment.  In 15 minutes I could have my recreate up and running.  Other engineers would open a case to have a recreate built by our lab team, which could take days.  I never figured out why they wouldn’t use IOU.

When I left Cisco I had to resort to GNS3, which was a pretty helpful piece of software.  Then, when I went to Juniper I used Junosphere, or actually an internal version of it called VMM, to spin up topologies.  VMM was awesome.  Juniper produced a virtual version of its MX router that was so faithful to the real thing that I could pass the JNCIE Service Provider exam without ever having logged into a real one, at least until exam day.

It’ll be interesting to see what I can do on virtual 9ks in CML–I hear there are some limitations.  But I do plan to spend as much time as possible using the virtual version over the real thing.

One thing I think I lost sight of as I (slightly) climbed the corporate ladder was the necessity of technical leadership.  We have plenty of people managers and MBAs.  We need leaders who understand the technology, badly.  And while I have a lot of legacy knowledge in my mental database, it’s in need of refresh.  It’s hard to stay sharp technically when reading about new technologies in PowerPoint.

The other side of this is that, as engineers, we love the technology.  I love making stuff work.  My wife is not technical at all, and cannot understand why I get a thrill from five little exclamation points when a ping goes through.  I don’t love budgets and handling HR cases, although I’ve come to learn why I need to do those things.  I need to do them so my people can function optimally.  And I’m happy to do them for my team.

On the other hand, I’m glad to be in the frigid, loud, harsh lighting of a massive Cisco lab again. It’s very cool to have all this stuff.   Ain’t life grand!

5