Skip navigation

Layer 1 Friday

My readership is limited, so consider a post to be “viral” if I get more than 2 thumbs up at the bottom of a page.  (Incidentally, I’ve only ever gotten one thumbs down, for this post, but I don’t know why.)  My 2021 post For the Love of Wiring got 3 thumbs up (!) but actually did get a lot more hits than usual after Tom Hollingsworth linked to it from his own blog.  How about a little more layer 1?

After my initial foray into stringing Cat 3 cable around in various unwise ways, Category 5 quickly became the standard.  I hated Cat 5 cable.  Cat 3 had a small number of twists per foot (or meter, non-Americans, or metre, Brits!), so upon removing the jacketing of the cable it was quite easy to untwist it before punching it down.  Cat 5 is very twisted.  Not only are the pairs hard to untwist, but they remain kinked after untwisting, and they take a lot of work to smooth out.  (If you correctly terminate Cat 5, you shouldn’t have to untwist and smooth the wires, but I didn’t know that at first.)  I remember once, on my 10 Mbps Ethernet, running a speed test on Cat 3 cable and then being very disappointed when I saw no improvement running the same test over Cat 5.  (Doesn’t quite work that way, and for 10 Mbps, Cat 3 was more than adequate.)

I did a lot of research to learn how to run cable the correct way.  Mainly this means preserving the tight twists.  Cat 5 cable cannot be kinked or bent sharply, and the twists must be maintained up to the point of termination.  Not only did I use this information to run my own cable, but once I took a job at a computer consulting company, I oversaw many cabling projects and needed to inspect the work done by our vendors.  Voice cable did not have the stringent requirements of data, so often phone cabling experts would run the Cat 5 with tight bends and would untwist the wires several inches before punching down.

The consulting company used one such phone installer to do many of their jobs, often as a sub-contractor.  This was in the days before wireless, when every computer connected to the network, even laptops, had to be plugged in.  I remember one client, a small architectural firm in Berkley, where our installer ran a brand new, Cat 5 Ethernet network.  We showed up, installed a hub, Ethernet cards, etc., and got everyone online.

A week or so later we got called back.  Stations were dropping on and off the network.  I fought may way through Bay Area traffic back to the office to figure out what was going on.

With any layer 1 issue, replacing cables is a good first step.  As I unplugged one station from the wall jack, the entire jack and face plate fell off the wall.  Whoops.

Normally when a network jack is installed in an office building with sheetrock (drywall) walls, the installer cuts a fairly large opening in the sheetrock and then installs a “low voltage ring”.  This ring secures to the drywall from behind, and provides a place for the faceplate to screw into.  Then the Cat 5 cable is punched down on a small “keystone” jack, over which a cover is placed, and which then snaps into the faceplate.

Low voltage ring

Our clueless installer had not done this.  Instead he cut a hole in the drywall just small enough for the jack to fit through.  He never installed the low voltage ring, instead screwing the faceplate directly into the drywall.  He also never installed the cover on the contacts on the jack, so the contacts were covered with drywall powder.  Because screws don’t hold well in drywall, when I pulled the cable from the jack, the whole thing fell out.  I also found out that when he had installed the small office patch panel in their supply closet, he put the screws straight into the drywall as well.  Normally you would use a backboard, screw it into a stud, or at least use drywall anchors.  The patch panel fell off the wall too.

Keystone jack with cover

Needless to say, I wasn’t too happy and neither was the customer.  I hate taking the fall for something that’s not my fault, but the customer considered it our mistake.  I made the cabling vendor come out and redo the entire installation.  After that, I told the owner of our firm to never use that vendor again.

A major concern, even with good cabling vendors, was having people in the office around the cables before they were fully installed.  I remember one client where we had a reputable vendor install the cabling before everyone moved in.  They ran one really large bundle of Cat 5 on the floor, because the client was going to install a raised floor afterwards.  Unfortunately, it took them months to get the raised floor in, and the bundle of cable ran right outside of a row of offices.  People stepped on them going in and out of their offices.  One time I remember a guy in cowboy boots standing right on top of the bundle.  I asked him to move.  By the time the floor covered the cables, they had gone from a clean, round bundle, to totally flattened.  Oddly enough, I never had any problems with the wiring in the time I worked there.

When I worked at the San Francisco Chronicle, our cabling vendor was installing some new fiber optic cabling to some data center racks.  The data center also housed our operations team (NOC, more or less.)  There was one lady who worked there who was very nice, rather large, and a tad immature.  The vendor had laid the fiber out on the floor before routing it under the floor tiles.  We looked up and there was the woman, jumping up and down on the fiber and laughing hysterically.  “Is this good for the cables, is this good for the cables?!” she was saying.  When we explained the interior was made out of glass, she looked horrified and stopped, but it was too late.  It cost us a bit, but fortunately for the NOC lady, she was in a union and well protected.

Working on software now, I don’t have to worry about cabling very much anymore.  I touch racks so infrequently I still call SFPs “GBICs”.  I do think it’s good for network engineers to stay informed on layer 1.  As much as you may know about protocols, software defined networking, or automation systems, none of it will work if the wires aren’t right.

19

Some thoughts on “digital twins”

As I mentioned in my last post, I like modeling networks using tools like Cisco Modeling Labs or GNS3.  I recalled how, back in TAC, I had access to a Cisco-internal (at the time) tool called IOS on Unix, or IOU.  This enabled me to recreate customer environments in minutes, with no need to hunt down hardware.  Obviously IOU didn’t work for every case.  Often times, the issue the customer raised was very hardware specific, even when it was a “routing protocol” issue.  However, if I could avoid hardware, I would do the recreate virtually.

When I worked at Juniper (in IT), we did a huge project to refresh the WAN.  This was just before SD-WAN came about.  We sourced VPLS from two different service providers, and then ran our own layer 3 MPLS on top of it.  The VPLS just gave us layer 2 connectivity, like a giant switch.  We had two POPs in each region which acted as aggregation points for smaller sites.  For these sites we had CE routers deployed on prem, connecting to PE routers in the POPs.  This is a basic service provider configuration, with us as a service provider.  Larger sites had PE routers on site, with the campus core routers acting as CEs.

We got all the advantages of layer 3 MPLS (traffic engineering, segmentation via VRF) without the headaches (peering at layer 3 with your SP, yuck!)

As the “network architect” for IT, I needed a way to model and test changes to the network.  I used a tool called VMM, which was similar to IOU.  Using a text file I could define a topology of routers, and their interconnections.  Then I used a Python script to start it up.  I then had a fully functional network model running under a hypervisor, and I could test stuff out.

I never recreated the entire network–it wasn’t necessary.  I created a virtual version with two simulated POPs, a tier 1 site (PE on prem), and a tier 2 site (PE in POP).  I don’t fully remember the details, there may have been one or two other sites in my model.

For strictly testing routing issues assuming normally functioning devices, my VMM-based model was a dream.  Before we rolled out changes we could test them in my virtual lab.  We could apply the configuration exactly as entered into the real device, to see what effect there would be in the network.  I just didn’t have the cool marketing term “digital twin” as it didn’t exist yet.

I remember working on a project to roll out multicast on the WAN using Next Generation Multicast VPN (NGMVPN).  NGMVPN was (is?) a complex beast, and as I designed the network and sorted out things like RP placement, I used my virtual lab.  I even filed bugs against Juniper’s NGMVPN code, bugs I found while using my virtual devices.  I remember the night we did I pilot rollout to two sites.  Our Boston office dropped off the network entirely.  Luckily we had out-of-band access and rolled back the config.  I SSHd into my virtual lab, applied the config, and spend a short amount of time diagnosing the problem (a duplicate loopback address applied), and did so without the stress of troubleshooting a live network.

I’ve always been a bit skeptical of the network simulation/modeling approach.  This is where you have some software intellgence layer that tries to “think through” the consequences of applied changes.  The problem is the variability of networks.  So many things can happen in so many ways.  Actual devices running actual NOS code in a virtual environment will behave exactly the way real devices will, given their constraints.  (Such as:  not emulating the harware precisely, not emulating all the different interface types, etc.)  I may be entirely wrong on this one, I’ve spent virtually no time with these products.

The problems I was modeling were protocol issues amongst a friendly group of routers.  When you add in campus networking, the complexity increases quite dramatically.  Aside from wireless being in the mix, you also have hundreds, thousands of non-network devices like laptops, printers, and phones which often cause networks to behave unpredictably.  I don’t think our AI models are yet at the point where they can predict what comes with that complexity.

Of course, the problem you have is always the one you don’t predict.  In TAC, most of the cases I took were bugs.  Hardware and software behaves unexpectedly.  As in the NGMVPN case, if there is a bug in software that is strictly protocol related, you might catch it in an emulation.  But many bugs exist only on certain hardware platforms, or in versions of software that don’t run virtually, etc.

As for digital twins, I do think learning to use CML (of course I’m Cisco-centric) or similar tools is very worthwhile.  Preparing for major changes offline in a virtual environment is a fantastic way to prep for the real thing.  Don’t forget, though, that things never go as planned, and thank goodness for that, as it gives us all job security.

4

Back in the lab

We all have to make a decision, at some point in our career, about whether or not we get into the management track.  At Cisco, there is a very strong path for individual contributors (IC).  You can be come a principal (director-level), a distinguished (senior director-level), and a fellow (VP-level) as an IC, never having to manage a soul.  When I married my wife, I told her:  “Never expect me to get into management, I’m a technical guy and I love being a technical guy, and I have zero interest in managing people.”

Thus, I surprised myself back in 2016 when my boss asked me, out of the blue, to step into management and I said yes.  Partly it was my love of the Technical Marketing Engineer role, partly my desire to have some authority behind my ideas.  At one point my team grew to fifty TMEs.

All technical people know that, when you go that route, your technical skills will atrophy as you will have less and less hands-on experience.  This is very true.  In the first couple of years, I kept up my formidible lab, then over time it sat in Cisco building 23, unused and consuming OpEx.  I almost surrendered it numerous times.

Through attrition and corporate shenanigans, my team is considerably smaller (25 or so) and run by a very strong management team.  Last week, I decided to bring the lab back up.  I’ve been spending a lot of time sorting through servers and devices, figuring out which to scrap and which to keep.  (Many of my old servers require Flash to access the CIMC, which is not feasible going forward.)  I haven’t used ESXi in years, and finding out I can now access vSphere in a browser–from my Mac!!–was a pleasant surprise.  Getting CIMCs upgraded, ESXi installed, and a functional Ubuntu server was a bit of a pain, but this is oddly the sort of pain I miss.

I have several Cat 9k switches in my lab, but I installed Cisco Modeling Labs on one of my servers.  (The nice thing about working for Cisco is the license is free.)  I used VIRL many years ago, which I absolutely hated.  CML is quite slick.  It was simple to install, and within a short time I had a lab up and running with a CSR1kv, a Cat 8k, and a virtual Cat 9k.

When I was in TAC I discovered IOS on Unix, or IOU.  Back then, TAC agents were each given a Sun Sparc station, and I used mine almost exclusively to run IOU.  (I thought it was so cool back then to have a Sun box on my desk.  And those of you who remember them will know what I mean when I say I miss the keyboard.)  IOU allowed me to define a topology in a text file, and then spin up several virtual IOS devices on the Sparc station in that topology.  It only supported sinulated Ethernet links, but for pure routing protocols cases, IOU was more than adequate to recreate a customer environment.  In 15 minutes I could have my recreate up and running.  Other engineers would open a case to have a recreate built by our lab team, which could take days.  I never figured out why they wouldn’t use IOU.

When I left Cisco I had to resort to GNS3, which was a pretty helpful piece of software.  Then, when I went to Juniper I used Junosphere, or actually an internal version of it called VMM, to spin up topologies.  VMM was awesome.  Juniper produced a virtual version of its MX router that was so faithful to the real thing that I could pass the JNCIE Service Provider exam without ever having logged into a real one, at least until exam day.

It’ll be interesting to see what I can do on virtual 9ks in CML–I hear there are some limitations.  But I do plan to spend as much time as possible using the virtual version over the real thing.

One thing I think I lost sight of as I (slightly) climbed the corporate ladder was the necessity of technical leadership.  We have plenty of people managers and MBAs.  We need leaders who understand the technology, badly.  And while I have a lot of legacy knowledge in my mental database, it’s in need of refresh.  It’s hard to stay sharp technically when reading about new technologies in PowerPoint.

The other side of this is that, as engineers, we love the technology.  I love making stuff work.  My wife is not technical at all, and cannot understand why I get a thrill from five little exclamation points when a ping goes through.  I don’t love budgets and handling HR cases, although I’ve come to learn why I need to do those things.  I need to do them so my people can function optimally.  And I’m happy to do them for my team.

On the other hand, I’m glad to be in the frigid, loud, harsh lighting of a massive Cisco lab again. It’s very cool to have all this stuff.   Ain’t life grand!

5

Goodbye, Building K

I haven’t written anything for a while, because of the simple fact that I had nothing to say.  The problem with being a writer is that sometimes you have nothing to write.  I also have a day job, and sometimes it can keep me quite busy.  Finally, an afternoon drive provided some inspiration.

There’s a funny thing about the buildings I work in–they all tend to be purchased by Google.  When I started at Juniper, I worked in their Ariba campus in Mountain View, several buildings they rented from that software company.  We were moved to Juniper’s (old) main campus, on Mathilda Drive, and the old Ariba buildings were bought and re-purposed by Google.  Then the Mathilda campus was bought by Google.

When I worked in TAC, from 2005-2007, I worked in building K on Tasman Drive in San Jose.  Back then, the meteoric growth of Cisco was measured by the size of its campus, which stretched all along Tasman, down Cisco way, and even extended into Milpitas.

Cisco’s campus has been going the opposite direction for a while now.  The letter buildings (on Tasman, West of Zanker Street) started closing before the COVID lockdowns changed everything.  Now a lot of buildings sit empty and will certainly be sold, including quite possibly the ones in Milpitas, where I work.

Building K closed, if not sometime during the lockdowns, shortly after.  I hadn’t driven by it in months, and when I did yesterday-lo and behold!-it was now a Google building!

What used to be building K

It’s funny how our memories can be so strongly evoked by places.  Building K was, for a long time, the home to Cisco TAC.  I vividly remember parking on Champion Drive, reviewing all of my technological notes before going in to be panel-interviewed by four tough TAC engineers.  I remember getting badged in the day I started, after passing the interview, and being told by my mentor that he wouldn’t be able to put me “on the queue” for three months, because I had so much to learn.

Two weeks later I was taking cases.  Not because I was a quick study, but because they needed a body.

I worked in High Touch Techical Support, dealing with Cisco’s largest customers.  The first team I was on was called ESO.  Nobody knew what it stood for.  The team specialzied in taking all route/switch cases for Cisco’s large financial customers like Goldman Sacks and JPMC.  Most of the cases involved the Cat 6k, although we supported a handful of other enterprise platforms.

When a priority 1 case came in, the Advanced Services Hotline (ASH) call center agents would call a special number that would cause all of the phones on the ESO team to play a special ring tone.  I grew to develop a visceral hatred of that ring tone.  Hearing it today would probably trigger PTSD.  I’d wait and wait for another TAC engineer (called CSEs) to answer it.  If nobody did, I’d swallow hard and grab the phone.

The first time I did it was a massive multicast meltdown disrputing operations on the NYSE trading floor.  I had just gotten my CCIE, but I had only worked previosly as a network engineer in a small environment.  Now I was dealing with a major outage, and it was the first time I had to handle real-world multicast.  Luckily, my mentor showed up after 20 minutes or so and helped me work the case.

My first boss in HTTS told me on the day I started, “at Cisco, if you don’t like your boss or your cubicle, wait three months.”  Three months later I had a new boss and a new cubicle.  The ESO team was broken up, and its engineers dispersed to other teams.  I was given a choice:  LAN Switch or Routing Protocols.  I chose the latter.

I joined the RP-LSA team as a still new TAC engineer.  The LSA stood for “Large Scale Architectures.”  The team was focused on service provider routing and platform issues.  Routing protocol cases were actually a minority of our workload.  We spent a lot of time dealing with platform issues on the GSR 12000-series router, the broadband aggregation 10000-series, and the 7500.  Many of the cases were crashes, others were ASIC issues.  I’d never even heard of the 12k and 10k, now I was expected to take cases and speak with authority.  I leaned on my team a lot in the early days.

Fortunately for me, these were service provider guys, and they knew little about enterprise networking or LAN switching.  With the breakup of the ESO team, the large financials were now coming into the RP-LSA queue.  And anyone who has worked in TAC can tell you, a routing protocols case is often not an RP case at all.  When the customer opens a case for flapping OSPF adjacencies, it’s usually just a symptom of a layer 2 issue.  The SP guys had no clue how to deal with these, but I did, so we ended up mutually educating each other.

In those days, most of the protocol cases were on Layer 3 MPLS.  I had never even heard of MPLS before I started there, but I did a one week online course (with lab), and started taking cases like a champ.  MPLS cases were often easily because it was new, but usually when a large service provider like AT&T, Orange, or Verizon opens a case on soemthing like BGP, it’s not because they misconfigured a route map.  They’ve looked at everything before opening the case, and so the CSE becomes essentially a middleman to coordinate the customer talking to developers.  In many cases the CSE is like a paramedic, stabilizing the patient before the doctor takes over to figure out what is wrong.  We often knew we were facing a bug, but our job was to find workarounds to bring the network back up so developers could find a fix.

I had my share of angry customers in those days, some even lividly angry.  But most customers were nice.  These were professional network engineers who understood that the machines we build don’t always act as we expect them to.  Nevertheless, TAC is a high-stress job.  It’s also relentless.  Close one case, and two more are waiting in the queue.  There is no break, no downtime.  The best thing about it was that when you went home, you were done.  If a call came in on an open case in your backlog, it would be routed to another engineer.  (Though sometimes they routed it back to you in the morning.)  In HTTS, we had the distinct disadvantage of having to work cases to resolution.  If the case came in at 5:55pm on Friday night, and your shift ended at 6pm, you might spend the next five hours in the office.  Backbone TAC engineers “followed the sun” and re-assigned cases as soon as their shift ended.

I make no secret of the fact that I hated the job.  My dream was to work at Cisco, but shortly after I started, I wanted out.  And yet the two years I spent in TAC are two of the most memorable of my career.  TAC was a crucible, a brutal environment dealing with nasty technical problems.  The fluff produced by marketeers has no place there.  There was no “actualize your business intent by optimizing and observing your network”-type nonsense.  Our emails were indeciperhable jumbles of acronyms and code names.  “The CEF adjacency table is not being programmed because the SNOOPY ASIC has a fault.”  (OK, I made that up…  but you get the point.)  This was not a place for the weak-minded.

When things got too sticky for me, I could call in escalation engineers.  I remember one case where four backbone TAC escalation engineers and one from HTTS took over my cube, peering at my screen and trying to figure out what was going on during a customer meltdown.

Building K was constructed with the brutalist-stlye of architecture so common in Silicon Valley.  One look at the concrete and glass and the non-descript offices and conference rooms is enough to drain one’s soul.  These buildings are pure function over form.  They are cheap to put up and operate, but emotionally crushing to work in.  There is no warmth, even on a winter day with the heat on.

Still, when I look at building K, or what’s become of it, I think of all the people I knew there.  I think of the battles fought and won, the cases taken and closed, the confrontational customers and the worthless responses from engineering, leaving us unable to close a case.  I think of the days I would approach that building in dread, not knowing what hell I would go through that day.  I also think of the incredible rush of closing a complex case, of finding a workaround, and of getting an all-5’s “bingo” (score) from a customer.  TAC is still here, but for those of us who worked in building K, its closure represents the end of an era.

8

Networking Simplicity

There’s a lot of talk about networking simplicity these days.  There’s been a lot of talk about networking simplicity, in fact, for as long as I can remember.  The drive to simplify networking has certainly been the catalyst for many new products, most (but not all) unsuccessful.  Sometimes we forget that networking has some inherent complexities (a large distributed system with multiple os’s, protocols, media types), but that much of the complexity can be attributed to humans and their choices.  IPv4 is a good example of this.

When I got into network engineering, I had assumed that network protocols were handed down from God and were immaculate in their perfection.  Reading Radia Perlman’s classic book Interconnections changed my understanding.  Aside from her ability to explain complex topics with utter clarity, Perlman also exposed the human side of protocol development.  Protocols are the result of committees, power politics, and the limitations of human personality.  Some protocols are obviously flawed.  Some flaws get fixed, but widely deployed protocols, like IPv4, are hard to fix.  Of course, v6 does remedy many of the problems of v4, but it’s still IP.

My vote for simplest protocol goes to AppleTalk.  When I was a young network guy, I mostly worked on Mac networks.  This was in the beige-box era before Jobs made Apple “cool” again.  The computers may have been lame, but Apple really had the best networking available in the 1990’s.  I’ve written about my love for LocalTalk, and its eminently flexible alternative PhoneNet in the past.  But the AppleTalk protocol suite was phenomenal as well.

N.B.  My description of AppleTalk protocol mechanics is largely from memory.  Even the Wikipedia article is a bit sparse on details.  So please don’t shoot me if I misremember something.

In the first place, you didn’t need to do anything to set up an AppleTalk network.  You just connected the computers together and switched either the printer or modem port into a network port.  Auto-configuration was flawless.  Without any DHCP server, AppleTalk devices figured out what network they were on, and acquired an address.  This was done by first probing for a router on the network, and then randomly grabbing an address.  The host then broadcast its address, and if another host was already using it, it would back off and try another one.  AppleTalk addresses consisted of a two byte network address which was equivalent to the “network” portion of an IP subnet, and a one-byte host address (equivalent to the “host” portion of an IP subnet.)  If this host portion of the address is only one byte, aren’t you limited to 255 (or so) addresses?  No!  AppleTalk (Phase 2) allowed aggregation of contiguous networks into “cable ranges”.  So I could have a cable range of 60001-60011, multiple networks on the same media, and now I could have 2530 end stations, at least in theory.

Routers did need some minimal configuration, and support for dynamic routing protocols was a bit light.  Once the router was up and running, it would create “zones” in the end-user’s computer in an application called “Chooser”.  They might see “1st floor”, “2nd floor”, “3rd floor”, for example, or “finance”, “HR”, “accounting”.  However you chose to divide things.  If they clicked on zone, they would see all of the AppleTalk file shares and printers.  You didn’t need to point end stations at their “default gateway”.  They simply discovered their router by broadcasting for it upon start up.

AppleTalk networks were a breeze to set up and simple to administer.  Were there downsides?  The biggest one was the chattiness of the protocols.  Auto-configuration was accomplished by using a lot of broadcast traffic, and in those days bandwidth was at a premium.  (I believe PhoneNet was around 200 Kbps or so.)  Still, I administered several large AppleTalk networks and was never able to quantify any performance hit from the broadcasts.  Like any network, it required at least some thinking to contain network (cable range) sizes.

AppleTalk was done away with as the Internet arose and IP became the dominant protocol.  For hosts on LocalTalk/PhoneNet networks, which did not support IP, we initially tunneled it over AppleTalk.  Ethernet-connected Macs had a native IP stack.  The worst thing about AppleTalk was the flaky protocol stack (called OpenTransport) in System 7.5, but this was a flaw in implementation, not protocol design.

I’ll end with my favorite Radia Perlman quote:  “We need more people in this industry who hate computers.”  If we did, more protocols might look like AppleTalk, and industry MBAs would need something else to talk about.

5

The death of CLI: A lesson from Swift

Update:  From Fred, who was the guy referenced in the first paragraph:

Actually it was a white button with a router icon on it and “make cli great again”, I know this because it was me. It was June 2016. Needless to say in my view that did not age well.

When I attended Cisco Live sometime around the election of Donald Trump, there was a fellow walking around with a red hat with white lettering on it:  MAKE CLI GREAT AGAIN.  Ha!  I love Cisco Live.  These are my people.

I remember back when I worked at Juniper, one exec looked at me working on CLI and said, “you know that’s going to be gone soon.  It’ll all be GUI.”  That was 8 years ago…how’s that going?  When I work on CLI (and I still do!), or programming, my wife always says, “how can you stare at that cryptic black screen for hours?”  Hey, I’ve been doing it since I was a kid.

The black screen won’t go away, I’m afraid.  I’ve recently been learning iOS app development for fun (not profit).  It’s surprisingly hard given the number of successful app developers out there.  I may be too used to Python to program in Swift, and my hatred of object-oriented programming doesn’t help me when there is no way to avoid it in Swift.  Anyways, it took me about a week to sort out the different UI frameworks used in iOS.  There are basically three:

  • Storyboards.  Storyboards are a graphical design framework for UI layout.  Using storyboards, you drag and drop UI elements like buttons and textfields onto a miniature iPhone screen.
  • UIKit.  (Technically storyboards use UIKit, but I don’t know what else to call this.)  Most high-end app developers will delete the storyboard in their project and write the UI as code.  They actually type in code to tell iOS what UI elements they want, how to position them, and what to do in the event they are selected.  Positioning is fairly manual and is done relative to other UI elements.
  • SwiftUI.  Apple is pushing towards this model and will eventually deprecate the other two.  SwiftUI is also a UI-as-code model, but it’s declarative instead of imperative.  You tell SwiftUI what you want and roughly how you want to position things, and Swift does it for you.

Did you catch my point?  The GUI-based layout tool is going away in favor of UI-as-code!  The black screen always comes back!

The difference between computer people and non-computer-computer-people (many industry MBAs, analysts, etc.), is that computer people understand that text-based interaction is far more efficient, even if the learning curve is steeper.

Andrew Tanenbaum, author of the classic Computer Networks, typeset his massive work in troff.  Troff is a text-based typesetting tool where you enter input like this:

.ll 3i
.mk a
.ce
Preamble
.sp
We, the people of the United States, in order
to form a more perfect Union...

Why doesn’t he just use Word?  I’ll let Dr. Tanenbaum speak for himself:

All my typesetting is done using troff. I don’t have any need to see what the output will look like. I am quite convinced that troff will follow my instructions dutifully. If I give it the macro to insert a second-level heading, it will do that in the correct font and size, with the correct spacing, adding extra space to align facing pages down to the pixel if need be. Why should I worry about that? WYSIWYG is a step backwards. Human labor is used to do that which the computer can do better.  (Emphasis added.)

I myself am not quite enough of a cyborg to use troff (though I use vi), but I have used Latex with far better results than Word.  (Dr. Tanenbaum says “real authors use troff,” however.)

One of my more obscure interests (I have many) is Gregorian Chant.  Chant uses a musical notation which is markedly different from modern music notation, and occasionally I need to typeset it.  I use a tool called Gregorio, where I enter the chant like this:

(cb3) Ad(d)ór(f’)o(h) te(h’) de(h)vó(hi)te,(h.) (,) la(g)tens(f) Dé(e’)i(d)tas,(d.)

The letters in parentheses represent the different musical notes.  I once tried typesetting the chant graphically, and it was far more tedious than the above.  Why not enter what I want and let the typesetting system do the work?

Aside from the mere efficiency, text files can be easily version controlled and diff’d.  Try that with your GUI tool!

It’s very ironic that many of my customers who use controllers like DNAC or vManage are actually accessing the tool through APIs.  They bought a GUI tool, but they prefer the black screen.  The controller in this case becomes a point of aggregation for them, a system which at least does discovery and allows some level of abstraction.

The non-computer-computer-people look at SwiftUI, network device CLI, troff, Gregorio, APIs, and rend their garments, crying out to heaven, “why, oh why?!”  Some may even remember the days of text-based editing systems on their DOS machines, which they could never learn, and the great joy that WYSIWYG brought them.  It reminds me of a highly incompetent sales guy I worked with at the Gold partner back in the day.  He once saw me configuring a router and said:  “Wow, you still use DOS to configure routers!”

“It’s actually IOS CLI, not DOS.”

“That’s DOS!” he densely replied.  “I remember DOS.  I can’t believe you still use DOS!”

It’s funny that no matter how hard we try to get away from code, we always come back to it.  We’re hearing a lot about “low code” environments these days.  It tells you something when the first three Google hits on “low code” just come back to Gartner reports.  Gee, have we been down this path before?  Visual Basic was invented in 1991.  If low code is so great, why is Apple moving from storyboards to SwiftUI?

In my last post I wrote about the war on expertise.  This is one of the fronts in the war.  The non-computer-computer-people cannot understand the black screen, and are convinced they can eliminate it.  They learned about “innovation” in business school, and read case studies about Windows 95 and the end of DOS.  They read about how companies like Sun Microsystems went belly-up because they are not “disruptive.”  They did not, however, read about all the failed attempts to eliminate the black screen, spanning decades.  I believe it was George Santayana who said, “If you don’t remember computer history, you’re doomed to repeat it.”

6

The War on Expertise

Like many network engineers, I quickly fell in love with my field and worked hard to master it.  I got into networking when I was working in desktop support.  The behind-the-scenes stuff was way more interesting to me than the front lines.  Back in the late nineties, I bought a library of books to learn this field.  Perlman, Comer, and Stevens were the classics.  I rounded it out with blue-and-white Cisco Press books by Doyle, Peplnjak, Williamson, and many others.  I studied these books religiously, read through config guides, and practiced in the lab.

The network engineers on my team and I loved to debate the arcana of this mysterious field.  We always tried to one-up each other, learning new technologies, new protocols, and attaining new technical certifications.  I’ve worked with engineers who are smarter than I am, and better than I am, but that always motivated me to learn more.

I bring this up because I’ve had multiple conversations with multiple execs, for many years, in which they seem to decry the virtue of expertise.  Network engineers “revel in complexity”, they don’t realize their time has passed, the build networks that need “armies of CCIEs to maintain”, and they hate simplicity.  If only the pesky network engineers would get out of the way, the glorious MBAs could build us simple and elegant products, which is how the industry is going, don’t you know!

In short, our industry is suffering through a war on expertise.  Those arcana we love to master have put a target on our back.  If you want to learn those things, you must be reveling in complexity.  Go find something else to do, ChatGPT will replace you!

The first mistake in this line of thinking is the assumption that network engineers want to build networks that are complex.  We actually don’t.  A couple of anecdotes:

When I was working for a Gold partner, I was sent to help out an IT manager of a rather small company, only four sites.  She had contracted VPLS from two service providers, and asked me to implement a complex load balancing scheme she had conceived.  I begged her not to make me do it, but she insisted.  I ended up building a functional mess, a combination of PBR and EIGRP offset access-lists.  Man, was it ugly.  But it worked.  Then I got laid off from the partner, and a year later she was calling me, begging me to come back because nobody could figure out how it worked.  I didn’t want to build something that ugly and she didn’t need it.

Second anecdote.  My wife had to go in for a surgical procedure a few years ago.  We went to the best doctor in San Francisco.  When he got into the procedure he found that her anatomy is not conventional, and it was a very difficult procedure.  In the recovery room, he told us most doctors would have stopped.  My wife wanly smiled and said, “well, I’m sure you like a challenge.”  He looked back at her and said, “no, we like it when it’s easy.”

I think this is where the execs misunderstand expertise.  99.9% of the time, your airline flight could be handled by a low-time pilot who can work the automation systems.  But when the engines fail, you want Sully at the controls.  Just because some people understand complexity and study difficult concepts, it doesn’t follow that they want complexity.  When I administered networks, I wanted it to be easy.  But I was ready for when it was hard.

The war on expertise seems to me to be a war on the human spirit.  The CCIE exam, whatever you think of it, was a heck of a challenge, and passing it was one of the proudest days of my life.  Human beings want to learn, to grow, to push their limits, and to test themselves.  That’s why we spend hours in the lab.  We should encourage this behavior.  We should want people in our business who seek subject matter expertise and mastery.  We can make things simpler, fine, but we should still encourage expertise.

At the end of the day, networks are inherently complex.  A network is a large distributed system, connecting numerous devices running numerous operating systems over diverse transport mechanisms using a wide variety of protocols.  You can simplify the protocols a bit, but ultimately most simplification of networks is done one of two ways:  reducing the number of choices an administrator can configure, or abstracting and hiding the underlying complexity.  In the first case, you may close out necessary use cases.  In the case of abstraction, well, it works great until something breaks.  Then you need to call a network engineer.

I’m not in any way saying the new tools, from programmability to automation systems like Ansible, to “controllers” are unnecessary.  Far be it.  Any tool that makes an engineer’s job easier will be embraced by engineers.  I am saying that we need to stop blaming complexity on those who manage to understand it.

27

When it’s time to go

I’ve mentioned my first job as a network engineer several times on this blog.  I worked at the San Francisco Chronicle, the biggest newspaper in Northern California.   I was brought in to manage the network as a Cisco-certified engineer, having just passed a four-day CCNA bootcamp.  Right before the dot-bomb economic crash, network engineers were in short supply.

The Chronicle’s network had recently been completely re-engineered, and the vendor selected was Foundry Networks.  Foundry was an up-and-coming vendor famous for selling high-speed switches to internet service providers.  They weren’t known for selling into enterprises, but they had convinced the previous network manager to install their hardware in nearly all of the Chronicle’s wiring closets.

It didn’t go very well.  The network had become incredibly unstable.  No company wants an unstable network, but newspapers are a particularly high-pressure environment since they have tight deadlines in order to get the paper out every single day, without fail.  Management of the data network was taken away from the previous manager and assigned to the head of the telecom department.  The plan was to rip out the Foundry and replace it with Cisco.

Foundry, of course, had other ideas.  Their account manager, whom I’ll call Bill, was quite aggressive in trying to restore the good name of Foundry.  I’ll give him credit for his doomed mission.

We had several problems.  The first was that we had only a single core router.  The router had two management modules, but failover between them was not fast, and our reporters and advertising people used Tandem systems which were sensitive to even slight network outages.  Foundry was well known for their fast IP switches, but we used AppleTalk and IPX as well, and their protocol stacks were not well implemented.  The BigIron 8000 was prone to crashing and taking out a lot of our users.  We had only one because the previous manager had been trying to save money.

The second problem was not Foundry’s fault entirely, although I do blame the SE in part.  Nobody ever set the spanning tree bridge priority on the core box.  By default, STP selects the bridge with the lowest bridge identifier as root.  Since the BID is comprised of a user-configured priority and the MAC address, if no priority is configured, the oldest switch in the network becomes the root bridge, since MAC addresses and OUI’s are sequential.

It turned out our Windows guys had been hauling around an ancient Cabletron switch to multiplex switch ports when working on end users’ computers.  (This was before wireless).  They would plug in, the Cabletron would dutifully assume STP root, and the entire network would reconverge for 50 seconds, spanning tree roots not being sticky.  I remember once paying a bill at a nearby restaurant before we were finished and running with the other engineers back to the office, hoping to catch an outage in progress after our pagers went off.  Foundry’s logs were not very good and we didn’t know why the network kept going down.  Eventually I figured it out, I don’t remember how.

The third problem was that the Foundry FastIron switches we used in the wiring closets had bad optics.  The Molex optics Foundry had selected for its management modules were flaky, and so we had to replace every single one with modules using Finisar optics.  I remember Bill, our account manager, coming in for our middle-of-the-night maintenance window several weekends in a row, blades in tow, and helping us to swap out the cards.

All of these problems created a bad reputation for Foundry within the Chronicle.  I remember Bill walking out of the front door carrying a Foundry box with an RMA’d management module.  A non-technical employee, perhaps a reporter or advertising salesman, saw the box and shouted, “Hey, they’re getting rid of Foundry!”  People in the lobby started cheering.  Bill looked at me and said, “soon they’ll be cheering when I come into the building with a Foundry box.”

It never happened.  We ripped out Foundry and replaced everything with Cisco Catalyst 4k and 6k switches.

The fact of the matter is, had we added a second BigIron in the core, fixed the root bridge problem, and replaced all the faulty modules, we probably would have had a solid network.  But there often comes a point when a vendor has destroyed their reputation with a customer.  It takes a multitude of factors to reach this point, but there is definitely a point of no return.  Once that line is crossed, the customer will often allow cordial meetings, listen with sympathy to the account team and execs, and then go their separate way.

A few years later I was laid off from my job at a Gold Partner, and was interviewing with another Gold Partner.  The technical interviewer looked at my resume and said, “I see you worked at the San Francisco Chronicle.”

“Yes,” I said, “I was brought in to replace the Foundry network they had with Cisco.  The whole thing was a disaster, poorly designed and bad products.”

“I designed that network,” he replied, “when I worked for another partner.  I also installed it.”

I didn’t get the job.

6

Apple Certified…what??

I’m thinking of doing some video blogging and kicking it off with a series with my thoughts on technical certifications.  Are they valuable or just a vendor racket?  Should you bother to invest time in them?  Why do the questions sometimes seem plain wrong?

Meanwhile, a little Netstalgia about the first technical certification I (almost) attempted:  The Apple Certified Server Engineer.

Back in the 1990’s, I worked for a small company doing desktop and network support.  When I say small, I mean small.  We had 60 employees and 30 of them had computers.  Still, it was where I first got into the computer industry, and I learned a surprising amount as networking was just starting to take off.

I administered a single AppleShare file server for the company, and I even set up my very first router, a Dayna Pathfinder.  I was looking for more, however, and I didn’t have much of a resume.  A year and a half of desktop support for 30 users was not impressing recruiters.  I felt I needed some sort of credential to prove my worth.

At the time Microsoft certifications, in particular the MCSE, were a hot commodity.  Apple decided to introduce its own program, the ACSE.  Bear in mind, this was back before Steve Jobs returned to Apple.  In the “beige-box” era of Apple, their products were not particularly popular, especially with corporations.  Nonetheless, I saw the ACSE as my ticket out of my pathetic little job.  I set to work on preparing for it.  If memory serves (and I can find little in the Wayback machine), the certification consisted of four exams covering AppleTalk networking, AppleShare file servers, and Backup.

Apple outsourced the certification development to a company called Network Frontiers, and its colorful leader, Dorian Cougias.  I had seen Dorian present at Macworld Expo once, and he clearly was very knowledgeable.  (He asked the room “what’s the difference between a switch and a bridge?” and then answered his own question.  “Marketing.”  Good answer.)  Dorian wrote all of the textbooks required for the program.  He may have known his stuff, but I found his writing style insufferable.  The books were written in an overly conversational tone, and laced with constant bad jokes.  (“To remove the jacketing of the cable you need a special tool…  I’d call it a ‘stripper’ but my mother is reading this.”  Ugh…)  A little levity in technical documentation is nice, but this got annoying fast.

This was in the era before Google, and despite my annoyance I did scour the books for scarce information on how networking actually worked.  I didn’t really study them, however, which you need to do if you want to pass a test.  I downloaded the practice exam on my Powerbook 140 laptop and fired it up.  I assumed with my day-to-day work and having read the book, I’d pass the sample exam and be ready for the real deal.

Instead, I scored 40%.  I used to be a bit dramatic back in my twenties, and went into a severe depression.  “40%???  I know this stuff!  I do it every day!  I read the book!  I’ll never get out of this stupid job!!!”  I had my first ocular migraine the next day.

In reality, it doesn’t matter how good or bad, easy or hard an exam is.  You’re not going to pass it on the first go without even studying.  And this was a practice exam.  I should have taken it four or five times, like I learned to do eventually studying Boson exams for my CCNP.

Instead, I gave up.  And, very shortly later, Apple cancelled the program due to a lack of interest.  Good thing I didn’t waste a lot of time on it.  Of course, I managed to get another job, and pass a few tests along the way.

I learned a few things about technical certifications from that.  In the first place, they can often involve learning a lot of knowledge that may not be practical.  Next, you can’t pass them without studying for them.  Also, that the value and long-term viability of the exams are largely up to the whims of the vendors.  And finally, don’t trust a certification when the author of the study materials thinks he’s Jerry Seinfeld.

 

3

Product Management

It’s impossible to count how many people at my college wanted to be “writers”.  So many early-twenty-somethings here in the US think they are going to spend their lives as screenwriters or novelists.  My colleagues from India tell me most people there want to be doctors or engineers, which tells you something about the decline of the United States.

Back in the mid-2000’s, a popular buddy-comedy came out about a novelist and an actor and their adventures in the “California wine country”.  The author of the film is an LA novelist.  The only people he knew, and the only characters he could create, were writers and actors.  I read that his first novel was about a screenwriter.  The movie was popular, but I found the characters utterly boring.  Who cares about a novelist and his romantic adventures?  Herman Melville spent years at sea, giving him the material to write Moby Dick.  Fyodor Dostoevsky wanted to be a writer from an early age, but he spent years in a prison camp followed by years of forced military service, to give him a view into nihilism and its effect on the human soul.  The point is, these great writers earned the right to talk about something, they didn’t just go to college and come out a genius with brilliant things to say.

I’ve been hearing a lot about “product management” lately.  I work in product management, in fact, and I’ve worked with product managers for many years.  However, I didn’t realize until recently that product management is the hot new field.  Everyone wants to major in PM in business school.  As one VP I know told me, “people want to be PMs because that’s where CEOs come from.”  Well, like 19-year-olds feeling entitled to be great novelists, b-school students are apparently expecting to become CEOs.  Somewhere missing in this sense of entitlement is that achievement has to be earned, and that is has to be earned by developing specific expertise.  A college student who wants to be a novelist thinks he or she simply deserves to be a novelist by virtue of his or her brilliance;  a b-school PM student apparently thinks the same way about being a CEO.

Back when I worked in TAC, one of my mentors was a TAC engineer who had previously been a product manager for GSR (12000-series) line cards.  He went back to TAC because he wanted to get into the new CRS-1 router and felt it was the best place to learn the new product quickly.  It made sense at the time, but it is inconceivable now that a PM would go to TAC.  The product manager career path is directed towards managing business, not technology, and it would be a step down for product managers to become technical again.

If you don’t work for a tech company, you may not know a lot about product management, but PMs are very important to the development of the products you use.  They decide what products are brought to market;  what features they will have;  they prioritize product roadmaps.  They are held accountable for the revenue (or lack thereof) for a product.

Imagine, now, that somebody with that responsibility for, say, a router has no direct experience as a network engineer, but instead has an MBA from Kellogg or Haas or Wharton.  They’ve studied product management as a discipline, but know nothing about the technology that they own.  Suppose this person has no particular interest in or passion for their field–they just want to succeed in business and be a CEO some day.  What do you think the roadmap will look like?  Do you think the product will take into account the needs of the customer?  When various technologists come to such a PM, will he be able to rationally sort through their competing proposals and select the correct technology?

To be clear, I am not criticizing any individual or my current employer here.  This problem extends industry-wide and explains why so many badly conceived products exist.  The problem of corporatism, which I’ve written about often, extends beyond product management too.  How often are decisions in IT departments made by business people who have little to no experience in the field they are responsible for?  I got into network engineering because I was fascinated by it and loved it.  I’m not the best engineer out there–I’ve worked with some brilliant people–but I do care about the industry and the products we make.  And most importantly, I care about network engineers because I’ve been one.

Corporatists believe generic management principles can be learned which apply to any business, and that they don’t really need domain-specific expertise.  They know business, so why would they?  True, there are some “business” specific tasks like finance that where generic business knowledge is really all that’s needed.  But the mistaken thinking that generic business knowledge qualifies one to be authoritative on technical topics doesn’t make sense.  This is how tech CEO’s end up CEO of coffee companies–it’s just business, right?

I don’t mean to denigrate product management as a discipline.  PMs have an important role to play, and product management is the art of making decisions between different alternatives with constrained resources.  I am saying this:  that if you want to become a product manager, spend the time to learn not just the business, but the actual thing you are product managing.  You’d be better off spending a couple years in TAC out of business school than going straight into PM.  Not that many CEO-aspiring PMs would ever do that, these days.

Now off to write my first novel.

2