simplicity

All posts tagged simplicity

There’s a lot of talk about networking simplicity these days.  There’s been a lot of talk about networking simplicity, in fact, for as long as I can remember.  The drive to simplify networking has certainly been the catalyst for many new products, most (but not all) unsuccessful.  Sometimes we forget that networking has some inherent complexities (a large distributed system with multiple os’s, protocols, media types), but that much of the complexity can be attributed to humans and their choices.  IPv4 is a good example of this.

When I got into network engineering, I had assumed that network protocols were handed down from God and were immaculate in their perfection.  Reading Radia Perlman’s classic book Interconnections changed my understanding.  Aside from her ability to explain complex topics with utter clarity, Perlman also exposed the human side of protocol development.  Protocols are the result of committees, power politics, and the limitations of human personality.  Some protocols are obviously flawed.  Some flaws get fixed, but widely deployed protocols, like IPv4, are hard to fix.  Of course, v6 does remedy many of the problems of v4, but it’s still IP.

My vote for simplest protocol goes to AppleTalk.  When I was a young network guy, I mostly worked on Mac networks.  This was in the beige-box era before Jobs made Apple “cool” again.  The computers may have been lame, but Apple really had the best networking available in the 1990’s.  I’ve written about my love for LocalTalk, and its eminently flexible alternative PhoneNet in the past.  But the AppleTalk protocol suite was phenomenal as well.

N.B.  My description of AppleTalk protocol mechanics is largely from memory.  Even the Wikipedia article is a bit sparse on details.  So please don’t shoot me if I misremember something.

In the first place, you didn’t need to do anything to set up an AppleTalk network.  You just connected the computers together and switched either the printer or modem port into a network port.  Auto-configuration was flawless.  Without any DHCP server, AppleTalk devices figured out what network they were on, and acquired an address.  This was done by first probing for a router on the network, and then randomly grabbing an address.  The host then broadcast its address, and if another host was already using it, it would back off and try another one.  AppleTalk addresses consisted of a two byte network address which was equivalent to the “network” portion of an IP subnet, and a one-byte host address (equivalent to the “host” portion of an IP subnet.)  If this host portion of the address is only one byte, aren’t you limited to 255 (or so) addresses?  No!  AppleTalk (Phase 2) allowed aggregation of contiguous networks into “cable ranges”.  So I could have a cable range of 60001-60011, multiple networks on the same media, and now I could have 2530 end stations, at least in theory.

Routers did need some minimal configuration, and support for dynamic routing protocols was a bit light.  Once the router was up and running, it would create “zones” in the end-user’s computer in an application called “Chooser”.  They might see “1st floor”, “2nd floor”, “3rd floor”, for example, or “finance”, “HR”, “accounting”.  However you chose to divide things.  If they clicked on zone, they would see all of the AppleTalk file shares and printers.  You didn’t need to point end stations at their “default gateway”.  They simply discovered their router by broadcasting for it upon start up.

AppleTalk networks were a breeze to set up and simple to administer.  Were there downsides?  The biggest one was the chattiness of the protocols.  Auto-configuration was accomplished by using a lot of broadcast traffic, and in those days bandwidth was at a premium.  (I believe PhoneNet was around 200 Kbps or so.)  Still, I administered several large AppleTalk networks and was never able to quantify any performance hit from the broadcasts.  Like any network, it required at least some thinking to contain network (cable range) sizes.

AppleTalk was done away with as the Internet arose and IP became the dominant protocol.  For hosts on LocalTalk/PhoneNet networks, which did not support IP, we initially tunneled it over AppleTalk.  Ethernet-connected Macs had a native IP stack.  The worst thing about AppleTalk was the flaky protocol stack (called OpenTransport) in System 7.5, but this was a flaw in implementation, not protocol design.

I’ll end with my favorite Radia Perlman quote:  “We need more people in this industry who hate computers.”  If we did, more protocols might look like AppleTalk, and industry MBAs would need something else to talk about.

Like many network engineers, I quickly fell in love with my field and worked hard to master it.  I got into networking when I was working in desktop support.  The behind-the-scenes stuff was way more interesting to me than the front lines.  Back in the late nineties, I bought a library of books to learn this field.  Perlman, Comer, and Stevens were the classics.  I rounded it out with blue-and-white Cisco Press books by Doyle, Peplnjak, Williamson, and many others.  I studied these books religiously, read through config guides, and practiced in the lab.

The network engineers on my team and I loved to debate the arcana of this mysterious field.  We always tried to one-up each other, learning new technologies, new protocols, and attaining new technical certifications.  I’ve worked with engineers who are smarter than I am, and better than I am, but that always motivated me to learn more.

I bring this up because I’ve had multiple conversations with multiple execs, for many years, in which they seem to decry the virtue of expertise.  Network engineers “revel in complexity”, they don’t realize their time has passed, the build networks that need “armies of CCIEs to maintain”, and they hate simplicity.  If only the pesky network engineers would get out of the way, the glorious MBAs could build us simple and elegant products, which is how the industry is going, don’t you know!

In short, our industry is suffering through a war on expertise.  Those arcana we love to master have put a target on our back.  If you want to learn those things, you must be reveling in complexity.  Go find something else to do, ChatGPT will replace you!

The first mistake in this line of thinking is the assumption that network engineers want to build networks that are complex.  We actually don’t.  A couple of anecdotes:

When I was working for a Gold partner, I was sent to help out an IT manager of a rather small company, only four sites.  She had contracted VPLS from two service providers, and asked me to implement a complex load balancing scheme she had conceived.  I begged her not to make me do it, but she insisted.  I ended up building a functional mess, a combination of PBR and EIGRP offset access-lists.  Man, was it ugly.  But it worked.  Then I got laid off from the partner, and a year later she was calling me, begging me to come back because nobody could figure out how it worked.  I didn’t want to build something that ugly and she didn’t need it.

Second anecdote.  My wife had to go in for a surgical procedure a few years ago.  We went to the best doctor in San Francisco.  When he got into the procedure he found that her anatomy is not conventional, and it was a very difficult procedure.  In the recovery room, he told us most doctors would have stopped.  My wife wanly smiled and said, “well, I’m sure you like a challenge.”  He looked back at her and said, “no, we like it when it’s easy.”

I think this is where the execs misunderstand expertise.  99.9% of the time, your airline flight could be handled by a low-time pilot who can work the automation systems.  But when the engines fail, you want Sully at the controls.  Just because some people understand complexity and study difficult concepts, it doesn’t follow that they want complexity.  When I administered networks, I wanted it to be easy.  But I was ready for when it was hard.

The war on expertise seems to me to be a war on the human spirit.  The CCIE exam, whatever you think of it, was a heck of a challenge, and passing it was one of the proudest days of my life.  Human beings want to learn, to grow, to push their limits, and to test themselves.  That’s why we spend hours in the lab.  We should encourage this behavior.  We should want people in our business who seek subject matter expertise and mastery.  We can make things simpler, fine, but we should still encourage expertise.

At the end of the day, networks are inherently complex.  A network is a large distributed system, connecting numerous devices running numerous operating systems over diverse transport mechanisms using a wide variety of protocols.  You can simplify the protocols a bit, but ultimately most simplification of networks is done one of two ways:  reducing the number of choices an administrator can configure, or abstracting and hiding the underlying complexity.  In the first case, you may close out necessary use cases.  In the case of abstraction, well, it works great until something breaks.  Then you need to call a network engineer.

I’m not in any way saying the new tools, from programmability to automation systems like Ansible, to “controllers” are unnecessary.  Far be it.  Any tool that makes an engineer’s job easier will be embraced by engineers.  I am saying that we need to stop blaming complexity on those who manage to understand it.

26