Skip navigation

Blog Updates

Updates for my regular readership of two people plus 5000 spambots:

  • I stayed away from the blog for a long time because I made the mistake of sharing my last post, on HPE/Juniper, on LinkedIn.  Imagine the panic when people actually read it, including SVPs at Cisco.  I’m fairly open with my thoughts on this blog, and I don’t want to limit my career prospects.  In retrospect, I think I should embrace frankness and keep sharing blogs on LinkedIn.
  • I changed the WordPress theme for this blog.  I think this new theme is a bit more readable.  Hopefully you all feel the same.  The previous theme had a weird issue.  When I shared the post on LinkedIn, instead of the title in the auto-generated preview, it said “posts navigation”.  That looks cheesy.  Rather than try to hack the HTML and figure out why, I switched themes.  The new one doesn’t have that problem.
  • However, before I did this I attempted to share the latest blog on tech field day.  LinkedIn seems to have cached it with “posts navigation” as the title, so I’ll probably republish it with a different permalink, leaving the old one in place for anyone who might have been directed to it.  So if you see the article again, I’m not going crazy.
  • I hope to get back to doing some technical blogging.  I love the perspectives and reflections, but something can’t keep me out of the lab.  Look for a few blogs soon.  (And I have a great history of promising to do something for this blog and not delivering on it, hope that doesn’t happen.)
5

Feeling like a Puppet

Starting a new job is always a challenge.  It’s one thing to do an internal transfer within a company, like I did when moving from Technical Marketing Engineer to Product Management.  I even stayed in the same team.  But moving to a new company and new role is overwhelming and risky.

I experienced this when I went to Juniper.  I’d defined my entire existence in terms of Cisco.  When I worked at a Gold partner, I refused to touch anything–servers, desktops, etc.–that was not Cisco-branded.  I’d barely even touched a Juniper device.  Now I was expected to coherently articulate a network architecture when I didn’t even understand the products that were involved.  (One could argue network architecture should not be product-specific, but we all know better.)

Ironically enough, after six years at Juniper, going back to Cisco proved challenging.  I didn’t touch a single Cisco device in six years.  At the time I went to Juniper, Nexus products didn’t even exist.  Now I was expected to be doing technical marketing–at the principal level–for products I hadn’t even heard of.

My boss, the legend Carl Solder, assigned me to programmability.  I knew nothing about it.  During my interviews with Carl, I had mostly talked about controller-based architectures.  While I was at Juniper, Jeremy Pruitt, an automation expert, had managed to get Puppet working for managing Juniper devices.  He pitched it to me and I had no interest at all.  This was a server technology, not a networking technology.  I was a protocols guy.

Now I was assigned to YANG, NETCONF, Python scripting, Ansible, and (ha, ha) Puppet.  I was fortunate to end up on a team of low-ego engineers who helped me to learn this stuff and who were willing to share opportunities with me.  They are still friends to this day.

One of the opportunities they shared was a chance to present at Tech Field Day/Networking Field Day.  In fact, TFD is a high-stress presentation, and nobody really wanted to do it.  Thus, three months into my new job, knowing nothing about Puppet or Nexus, I was called upon to present Puppet management of Nexus to one of the most challenging audiences we have.  I think TFD delegates can be a bit nicer these days, but back then a few of them loved to eviscerate vendors, especially Cisco.  Oh, and Jason Edelman and Matt Oswalt, two automation experts, would be in the audience.  Also Terry Slattery, who is the first person ever to pass the CCIE lab.  And my boss Carl, one of the best public speakers at Cisco.  And it would be livestreamed and everyone (it seemed) in Cisco would be watching it.  Great.

A tough room to stand in front of

The only way out was through.  My team was very helpful in preparing me, and I was able to spend a lot of time with engineering.  Truth be told, it was just technology, and I’m a technology guy.  It was learnable.  Cisco’s marketing team was also very helpful, explaining possible pitfalls and traps in presenting to the TFD audience.  For example, if you put “Cisco Confidential” on your slides, they’ll make fun of you.  (It’s a default on our PowerPoint templates.)  They told us not to cite Gartner.  I made a fake slide with ridiculously large “Cisco Confidential” and “DO NOT SHARE”  and a magic quadrant and presented it at rehearsal.  The look of horror from the marketing team told me they didn’t get the obvious joke.

It’s a joke, really!

Puppet was particularly challenging because, unlike Ansible, it was agent-based.  (I’m not sure if this is the case anymore, I rarely hear about Puppet in networking contexts.)  This meant you had to bring up a guestshell and install the agent on any device that it managed.  To be honest, this is less than ideal.  It means that before you could use Puppet to manage the device, there was a fair amount of setup required.  This could be done through PoAP (Power-on Auto-Provisioning, Cisco’s version of ZTP for Nexus), but it required a lot more work than simply enabling a few CLIs, all that was needed for Ansible.

There was a non-technical challenge that faced me as well:  I had been working on overcoming a terrible fear of public speaking.  I had melted down in one presentation in my early years at Juniper, and had spent the last several years doing training, Toastmasters, and anything I could to get over this problem.  I had successfully delivered some major presentations at Juniper before I left, but this was the highest pressure I had ever been under.  Would I panic?  Would I crack?  Would I be shown the door only a few months into the new job?  Public speaking is risky, because when you fail, you fail in front of a lot of people.

I’m still here, of course, and aside from a couple minor stumbles, it went off quite well.  Watching the video eight years later, I do cringe a bit at some of my slides.  I’m not a genius when it comes to PowerPoint, but I’ve done a lot of work to improve the quality of my decks.  Given the time constraints, I had to borrow some slides and hack together others.  I do hear some of my “signature” lines taking shape.  In nearly all my presentations on automation, I joke about Notepad being the most common automation tool used by network engineers, and this must be the first place I used that line.  And while Jason and Terry did ask questions, they didn’t break me and I was able to answer them quite assuredly.

My demo was a bit simplistic.  I provisioned a few VLANs and interfaces, and pushed a patch to a Nexus 9k.  As I started to take over lead on programmability, I changed my approach to it.  For example, when I showed network engineers how to provision a VLAN with NETCONF, their eyes would glaze over upon seeing the massive block of Python and XML needed to make it work.  Why not just type one line of CLI?  I became convinced we needed to show the value of programmability by showing how we can use it beyond just simple tasks.  Configure a VLAN?  Meh.  Use Webex Teams (or Slack) to interact with a router, better.  We did some silly demos like using Alexa to configure switches, but the point was actually to show that programmability opens up possibilities that were not there before.  Nonetheless, when I did this TFD, it was still early days.

My TFD session also ended up generating a popular meme, although it wasn’t on my account.  Jason Edelman and Matt Oswalt were sitting next to each other.  Jason was dressed in a sportcoat, clean cut, with Windows PC, and no stickers on it.  Matt was sitting next to him with unkempt hair, using a Mac covered with stickers.  Someone screenshotted it and posted it to Reddit with the caption, “the two types of network engineers”.  It’s funny because Matt and Jason wrote a book on network automation together.  (In the photo, my boss Carl looks on in the background on the right.  Dead center in red is Lauren Friedman, with whom I worked many times as our TFD coordinator in marketing.  Stephen Foskett, one of the TFD organizers, is between Matt and Jason.)

The two types of network engineers

 

In our careers as network engineers, we’re often thrown into things we know nothing about and are expected to quickly turn around and become experts. It happened every week in TAC. Working in product marketing, we’re often on the cutting edge and constantly fighting to stay above water. I had only a few weeks to learn Puppet, get a demo up and running, and present it to a challenging audience. The most important thing is to not get cocky, to understand what you don’t know, and to spend the hands-on time to plug those gaps in your knowledge.

Every public speaker knows the feeling of relief when you finally finish and the stress unloads. As I closed my talk and walked to the back of the room, Carl, my new boss, looked at me and gave me a thumbs-up. At that point if you told me I’d win Hall of Fame at Cisco Live for talking about programmability, I’d have thought you were crazy.

As for Puppet, I never touched it again.

4

Why I am not an FBI Agent

Some years back I wrote a post poking fun at the Federal Bureau of Investigation, based on an experience I had at a briefing in their office.  The funny thing is–and this did not color my article– there was a point in my life when I badly wanted to be an FBI agent.

Back in the early 2000’s, I was a run-of-the-mill network engineer.  I liked what I was doing, but I also felt a lack of purpose and direction in life.  I also didn’t want to sit around in an office all day.  I think a lot of people in their twenties go through this, at least in America.  It’s why my nephew is trying his hardest to become an Air Force parajumper.  It’s why my brother wanted to be a Navy SEAL.  (He had to settle for infantry major.)  This seems to be a “first world” problem.  Coworkers from other countries tell me all they want is a good job and home, and a safe place to raise their kids.  American affluence leads to boredom, and boredom to a desire to do something more important.

At that time, the FBI had three tracks for joining as a special agent.  You could enter if you had a law degree, or an accounting degree.  If you didn’t have one of these, you could enter under the “diversified” qualification, meaning you had some other experience that might be relevant to the Bureau.  At some point in the early 2000’s, the FBI added CCNP and CCIE certifications, specifically, as qualifications equivalent to a law or accounting degree!

I’d actually had some interest in an FBI job back in college.  I remember a female student who was also quite interested, and at graduation we looked at each other and said “see you in Quantico“, the FBI’s training center.  (I looked her up for this article, and it turns out neither of us got there.)  I had parked my interest as I didn’t wind up getting a law degree, but it was rekindled when I saw they added Cisco certifications to the list.

How cool would it be to investigate and bust criminals?  How cool would it be to have a badge and credentials to flash at people?  How cool to have arrest powers?  I was bullied as a kid, so I always had respect for law enforcement officers, and a strong belief in the importance of law and order.  One night, walking home by myself, I remember some kids vandalizing cars and houses.  “We’re f**ing s**t up!” they shouted at me.  I didn’t have a cell phone on me, and I couldn’t do anything.  But if I had a badge and a gun, I could have put a stop to it!  (My guess is that actually, a lone FBI agent would call the cops and not try to singlehandedly stop a group of thugs.)

I bought several books on the FBI and the application process, and started reading up on it.  The X-Files was popular at the time, but I had a realistic understanding of what the FBI did and was well aware that I would not be driving around the country chasing space aliens.  My hope was to be a cybercrime investigator.  I remember telling my brother and some of my friends that this was my plan.

So what happened?  Well, I never even applied, for a few reasons.

Not for me!

First, I hate exercise.  I was always the last kid picked for sports teams.  I was always behind on our school’s annual fitness testing.  PE class was a constant source of embarrassment.  The FBI requires a challenging fitness test just to get in, let alone to pass their academy.  I ended up signing up at a local Karate studio to try to get in shape.  I never got beyond a white belt.  (I’m proud to have a white belt in Karate, Tae Kwon Do, and Judo!)  While I liked learning the techniques, they had an intense fitness class required as part of their program and I hated it.  How could I survive six months of that?

Second, I didn’t want to live in a dorm for the academy, and I really didn’t want to live with a roommate, even for a few months.  There were two videos on the FBI Academy at the time, one from Joan Lunden and another from CNN.  I paused them when they showed the dorm rooms, and I clearly saw two beds.  I had a horrible time with my college roommate, and I didn’t want to experience that again.  Granted, an FBI trainee was not likely to show up at 3am stoned and wake me up to chat, but still, the lack of personal space, the snoring…  I didn’t want it.

Actual FBI dorm room-no thanks!

Third, if I went to the FBI, I’d take a pay cut.  Even a few years into the IT world, as a Cisco-certified engineer, I was doing quite well for myself.  I wanted to serve a greater purpose, did I really want to get paid a lot less?

Next, FBI agents are required to go anywhere in the country the bureau assigns them.  They also have to move several times in their career.  All I wanted was to live in San Francisco, my home town.  My friends and family were there.  How could I deal with it if I were assigned to Peoria?  Or, horror of horrors, New York City??

Finally, I did not want my job to be tied to the polygraph examination.  I’d been fascinated by “lie detectors” since I built a rudimentary one from an electronics kit when I was a kid.  I had read A Tremor in the Blood by David Lykken, Ph.D.  I knew that the lie detector is anything but.  Sociopathic liars often pass it, and legitimate and honest people often get called out as liars.  If I did do the work to get in shape, set my sites on this accomplishment, and then failed a polygraph not because I was lying, but because it’s a stupid test, it’d be miserable.  And FBI agents can be routinely tested with polygraphs.  Imaging getting the job and then losing it due to a faulty polygraph!

Read this and you’ll never want to take a polygraph

A couple years ago, a former FBI special agent named Vincent Sellers released a book called “Eyes Pried Open:  Rookie FBI Agent.”  Sellers had a similar background, an IT guy who wanted to do something bigger.  (Although he was actually a strong runner.)  His book affirmed everything I had thought.  Even for him, the exercise was brutal, including knuckle pushups that left permanent scarring.

Vincent Sellers

Agent Sellers left the FBI after only two years.  He didn’t like the job!  The hours were brutal, and while it had a few exciting moments, it was not that rewarding for him.  Even if I made it through, which I wouldn’t have, I’d probably have hated it too.  Sellers went back to being an IT project manager.

I’ve been blessed to have an amazing career as a network engineer.  I’ve been an in-house engineer, a pre and post sales engineer at a partner, a TAC engineer, and now I work in product management.  I’ve been in telco CO’s, I’ve gotten to play with all kinds of interesting gear, I’ve presented at Cisco Live.  I suppose having a shiny badge to flash would be cool, but the novelty would wear off eventually.  And I’ve never done a single knuckle pushup.

I remember a negative review of the movie La La Land by James Bowman.  I appreciate Bowman because he basically doesn’t like any movie he sees.  He described the characters in La La Land as being successful, materially satisfied, with a lot of friends, and able to drink champagne unendingly without getting drunk.  Of them he says:  “The only thing they don’t have–and the only thing they really want–is fame.”  In other words, they have everything in life except that feeling of importance that comes from the recognition of other people.  My craving for an FBI badge was, to some degree, the same impulse.  Sure there were motives higher than the characters in the movie, but it was largely driven by an inflated sense of self-importance.

Virtually all tech companies (and corporations in general) have in recent years been moved to “create a sense of purpose” for their employees.  In furtherance of this, they create purpose statements of varying degrees of vacuousness, and often with no relevance to the real purpose of the company at all.  The flip side of this is the implication that the real purpose of the company is not meaningful–if it were, there would be no need to concoct purpose statements.  I can tell you that I find meaning and satisfaction in being a network engineer, in itself, and I hope most of my colleagues do as well.  Agent Sellers had his FBI badge for only two years, but I’ve had my Cisco badge for an aggregate total of 11 years, so there must be something to it.

4

HPE Buys Juniper

In 2007, I left Cisco after two brutal years in high-touch TAC.  I honestly hated the job, but it was an amazing learning experience.  I draw on my TAC experience every single day.  A buddy of mine got a job at a Gold Partner, offered to bring me in, and I jumped on the opportunity.  Things didn’t go so well, and in 2009, I was laid off and looking for a job again.  That’s when another buddy (buddies help!) called me and told me of an opportunity at Juniper.

I knew little about Juniper.  We had a Juniper SSL box in the network I used to manage, but the routers were mostly for service provider networks.  When I was at TAC, I had one case where a major outage was caused by misconfiguration of a Juniper BGP peer.  But otherwise, I didn’t know a thing.

The opportunity was to be the “network architect” for Juniper’s corporate network.  In other words, to work in internal IT at a network vendor.  It seemed like a good career move, but little did I know I would be thrust the corporate politics at the director-level instead of technical challenges.  I ended up spending six tumultuous years there, with several highlights:

  • My boss disappeared on medical leave on my very first day.
  • I was re-assigned to a Sr. Director who was an applications person and not knowledgeable in networking.  He viewed the network a bit like Col. Kendrick, the Marine, viewed the Navy in the movie A Few Good Men:  “Every time we gotta go some place to fight, you fellas always give us a ride.”
  • I proposed and got buy-off for a program to ensure we actually ran our own gear internally and to ensure we built solid network architectures.
  • I subsequently had the program taken away from me.
  • I found out a job posting with the identical title and JD to mine was listed on Juniper’s public site without my knowledge.
  • My manager was changed to a person two pay grades below me in another country without even informing me.  (Someone noticed it in the directory and told me.)
  • I quit in disgust, without any other job.
  • I was talked into staying.
  • After another year or misery, I was demoted two pay grades myself.
  • I focused on doing the best job I could ended up getting re-promoted to director and left on good terms.

Some of the above was my own fault, much of it was dysfunctional management, some of it was the stupidity we all know lurks in every good size company.  I actually bear Juniper no resentment at all.

I worked at Juniper in the pre-Mist days, and in the midst of the fiscal crisis that began in 2008.  We went from CEO Kevin Johnson’s rah-rah “Mission10” pep rallies that we would be the “next $10B company” (uh, no), to draconian OpEx cuts when a pump-and-dump “activist investor” took over our board.

At the time I was there, Juniper made some mistakes.  NetScreen firewalls had done well for us, but then we made the decision to kill the NetScreen in favor of the JunOS-based SRX.  This is the classic mistake of product management–replace a successful, popular product with a made-from-scratch product with no feature parity.  There were some good arguments to do SRX, but it was done abruptly which signalled EOL to NetScreen customers, and SRX didn’t even have a WebUI.

We also did QFabric while I was there.  We installed one of these beasts in a data center on campus.  I have no idea if they improved it, but the initial versions took a full day to upgrade.  Imagine taking a day-long outage on your data center just to do an upgrade!

Another product that didn’t work out was Space.  JunOS Space came out at the time when the iPhone was still new.  Juniper borrowed the idea.  Instead of building an NMS product, we’d build a platform, and then software developers could build apps on top of it.  Cisco might be able to get away with that approach, but Juniper didn’t have enough of the networking market to attract developers.

In addition, a bunch of other acquisitions fizzled out, including Trapeze, our WAN accelerator, our load balancer.

All that said, Juniper had some fine products when I worked there.  (And believe me, my current employer has had many failures too.)  I got my JNCIE-SP, working on MX routers, which were a really good platform.  I thought the EX switches were decent.  And the operating system was nicely done.  Funnily enough, I worked a solid year on the JNCIE and promptly went to Cisco.  I never renewed it and now it’s expired.

I left after meeting with a strategy VP and explaining our mission to use Juniper’s corporate network to demonstrate how to build an enterprise network to our customers.  She looked at me (and the CIO) and said, “Juniper is done with enterprise networking.  I’m not interested.”  I left after that.  In her defense, Mist was years off and she couldn’t have seen it coming.

She was right, in that Juniper certainly had a core SP market.  Juniper came about at the time when Cisco was still selling 7500’s and 12000’s to its service provider customers, dated platforms running a dated OS.  Juniper did such a nice job with their platform that Cisco had to turn around and build the CRS-1 and IOS-XR, both of which had, ehm, similarities to Juniper’s products.  Juniper really couldn’t crack the enterprise market while I was there.  The lack of a credible wireless solution was always a problem.  Obviously Mist changed the game for them.

Juniper always felt like a scrappy anti-Cisco when I was there, but it was fast becoming corporatized and taken over by the MBAs.  Many old-schoolers would tell me how different things were in the startup days.  It still always had the attitude of an anti-Cisco.  One of our engineers ALWAYS referred to Cisco devices as “Crisco boxes”, and when I announced I was returning to Cisco, a long-time IT guy called me an “asshole”.  A couple funny stories around this:

A customer came in to our office for training and looked in the window of one the data centers nearby.  He saw it was packed with Cisco gear and subsequently published a video on social media captioned “Juniper uses Cisco.”  He didn’t realize that we leased the building from another company called Ariba, and the data center was theirs, not ours.  In fact, we worked very hard to not run Cisco in our internal network.  Juniper subsequently asked Ariba to block out the window.

One time we solicited a proposal from one of our largest service provider customers to host a data center for us.  The SP came back to us with an architecture which was 100% Cisco.  Cisco switches, Cisco routers, Cisco firewalls.  I told the SP I would never deploy our DC on Cisco gear.  What if a major bug hit Cisco devices causing outages and our data center went down too?  What if we got hacked due to a Cisco PSIRT and it became public?

The SP didn’t care.  We were their customer, but they were also ours.  They used Cisco in their data center, and had no desire to re-tool for another vendor.  I escalated all the way to the CEO, who agreed with me, and the deal was scuttled.  Ironically, I used this story in my Cisco interviews when asked for an example of a time when I had taken a strong stand on something.

I work at Cisco now, and even ran the competitive team for a while.  Competition is healthy and makes us all better.  I actually value our competition.  Obviously my job is to win deals against them, but I have friends who work at Juniper and I have friends who work at HPE.  We’re all engineers doing our jobs, and I wish them no ill will.  I always respected Juniper, their engineering, and their scrappy attitude.  While I know some of this will be retained as they get absorbed into a large corporation, it’s definitely the end of an era, for the industry and for me.

39
1

Frozen Yellow Garden Hose

Ivan’s recent very interesting post on LAN Data Link addressing takes me back.  Specifically, footnote #1, referring to “ThickNet” Ethernet:  “The coaxial cable had to be bright yellow”.  In the US, at least, we also used to call the stuff “Frozen Yellow Garden Hose”, for obvious reasons.

The original Ethernet physical medium was rather interesting.  I’ve often talked about my love of layer 1, but I got into the business in the twisted-pair era.  I did do a bit of ThinNet networking, but only a little.  And I had one encounter with ThickNet.  Twisted pair is fun to work with and easy to manage.  Coax is not.

The original “ThickNet” Ethernet used, as the name implied, a fairly thick coaxial cable.  The cable had to be physically routed in a path which put it in proximity to the stations that would use it.  Connections to the cable were originally made with a “tap”, requiring physical piercing of the cable to reach the center conductor.  A drop cable then extended down from the tap to the station using it.  Messy, and inconvenient.

“ThinNet” used a substantially smaller cable, and it was far more flexible.  Think about something close to your TV cable coax, but even more flexible, and using BNC connectors.  As with ThickNet, the routing of the cable was tricky, but instead of physical taps we used T-style connectors to add stations into the network.  My very first Ethernet network consisted of a ThinNet segment tied into a 10Base-T twisted pair segment with an adapter.

I’ve written a few times about my first full-time networking job at the San Francisco Chronicle.  Back when I worked there, people actually read the paper and its printing and distribution was a big deal.  After the 9/11 attacks, you may remember several authorities and news agencies in the United States were mailed powdered anthrax.  These events prompted our management to set up a “disaster recovery” site, where we would have enough equipment to produce the newspaper in the event our main building was incapacitated.

For a reason I never understood, they chose a building on the same block.  Behind the Chronicle building were a number of alleys, and the company owned a building back there.  It made no sense at all.  If an earthquake hit, and the main building was destroyed, presumably the building across the street would be too.  If anthrax were mailed to the main building, I assume the entire block would be shut down.  But anyways…

We set up this empty building with computers and phones, and added a frame relay connection so it could connect to the WAN if the main building were out.  But we still wanted a high speed connection for file transfers and backups, so I was tasked with somehow getting a faster connection to the HQ.  I called our cabling contractor, who also resold several wireless building-to-building alternatives, but they were all expensive.  Then I made a discovery.

In the basement of this historic building was our MDF, the main phone equipment and wiring room.  There was a large distribution frame for phone wiring to the entire building, three or four racks of phone equipment, a desk that looked like it was made by prisoners, and a grumpy phone lady who sat in there.  One day I also noticed a pipe with a tag hanging off of it.  It had the address of the disaster recovery building in it, and a disconnected piece of frozen yellow garden hose hanging down.  Could it be?

I went outside and saw a conduit running from the top of the HQ building to the disaster recovery site.  At this time, ThickNet was not being used by anyone, but 10Mbps would be more than sufficient for this DR site that would probably never be fully staffed.

One of my fellow engineers (who had worked at the paper forever) had a lab packed full of all kinds of esoteric electronic equipment.  I went spelunking in the dingy room and found two ThickNet transceivers.  Thankfully the ThickNet was terminated on both ends with connectors, so I didn’t need to drill and tap.  I hooked up the transceivers, connected the RJ45 ports to a nearby switch on each end, and then tested the connection.  It worked!  I had no idea how many years that frozen yellow hose had been in place, but it took one problem off my plate pretty easily.

The “DR” building sat unused for several years, but with keys to the building it made a nice place to take a nap occasionally.  The site was decommissioned eventually.  I wouldn’t be surprised if the frozen yellow garden hose is still in the conduit to this day.

A lot more detail on ThickNet can be found at Matt’s Tech Pages here.

6

Cisco Live, again and again

It’s funny that I remember a time when Cisco Live used to be a privilege, before it became a chore.  I’ve been to every Cisco Live US and every Cisco Live Europe since I started here in 2015.  I enjoy the show primarily because I enjoy meeting with fellow network engineers, and there is still a lot of energy every show.  But I’ve seen all the booths, the flashy keynotes, and the marketing sessions many times over.  I’ve eaten the bad food and stayed in a few bad hotels.

Speaking was one of my favorite activities.  The audiences are intimidating, and as someone who used to have a terrible fear of public speaking, there is always a bit of jitters until my session(s) are over.  But I enjoy communicating technical concepts in a clear and understandable way.  That’s why I got into speaking.  I just wish I was teaching a college course where I grade the students instead of the students grading me.

After I transitioned from being an individual contributor to a manager, I held onto my speaking slots for a while, but the last Cisco Live where I spoke was Las Vegas 2022.  I attended, but didn’t speak, at Cisco Live Amsterdam 2023 and Cisco Live Las Vegas 2023.  It was nice not to have to prepare much, but I missed the “speaker” tab on my badge.  It just wasn’t the same.  Yet this is what happens when you move into leadership.  The sessions at Cisco Live are meant to be technical, and the SGMs who control sessions want technical speakers, not manager-types.  When you lead a team, you also don’t want to take sessions from your staff.

In Vegas, I swore I would not return to Cisco Live without a speaker badge.  If I didn’t get a slot, I would skip CL for the first time since I came back here in 2015.  I applied for two sessions for Europe in Feb 2024, trying to cash in on my “Hall of Fame” status.  The SGMs want relevant topics and good speakers.  I didn’t get any emails and assumed I wasn’t going to Amsterdam.  Have I finally jumped the shark?

Then I got a funny email.  “Do you want your Amsterdam session considered for Las Vegas?”  Huh?  Only speakers get that email.  I checked the Cisco Live Speaker Resource Center, and lo and behold, I saw one DevNet session there.

The session I submitted is called “Time Machine! A 1980’s bulletin board system, revived”.  What??  I submitted this as a total long shot, nearly positive it would be denied.  As detailed in a couple posts on this blog, I ran a bulletin board system in the 1980’s.  I proposed to do a DevNet session on how BBS’s worked and how I had revived my own in an emulator.  And they accepted it!  Ha!  Now I have to come up with the content.  I remember in the abstract where it asks for “business value of this session” and I said something like “There is absolutely no business value to this session.”  I guess the SGMs have a sense of humor.

I guess I’m going to Amsterdam.  I know for a fact one of my three regular readers goes to Cisco Live Europe.  So I look forward to seeing you there.  Meanwhile, I realized at this point in my career I need to submit my sessions in the “IT Leadership” track, so I’m writing some Las Vegas abstracts for that.  Can’t keep an old dog down!

7
1

Ancient Internet

The first company where I worked as a “systems administrator” had no Internet connectivity at all when I started.  By the time I left, I had installed an analog phone line which was shared amongst several users with modems for dial-up service.  The connectivity options in 1995 were limited, and very expensive.  Our company operated on a shoestring budget and could not afford the costly dedicated service offerings from our ISP.

When I moved to a “consulting” company, I finally had the opportunity to work with real dedicated Internet service.  For the customers I worked with, our main options were two:  ISDN and T1 lines.

ISDN stood for Integrated Services Digital Network.  It came in two major flavors, but we exclusively used the lower-end Basic Rate Interface (BRI).  ISDN was a digital phone line, and like its analog counterpart required dialing a phone number.  The BRI had two data channels which were 64 Kbps each, but we usually ran them together for a combined 128 Kbps.  At the time, this was quite speedy, more than double the speed of the modems we had, and our smaller customers loved ISDN.  Because it was a dial-up technology, however, per minute rates applied.  This meant the line would time out and disconnect periodically to save costs.  When it was down and an outbound packet arrived at the router with the ISDN interface, the router would dial up the SP again.  The connection time was much faster than analog modems, but it still added latency which was annoying.

I don’t know if other regions did it, but we had a local hack to get around this.  Understanding the hack requires a little background on the phone systems of the time.  Two kinds of business phone systems were typically in use:  PBXs and key systems.  With a key system, every phone had an extension number, but no direct dial into it.  If you wanted to speak to the person at extension 302, you dialed the main phone number for the business, and either asked the receptionist to connect you, or else an automated system did it for you.  For outbound dialing, the user would either lift the handset and select an unused line from the pool of lines available, or perhaps dial 9 for the key system to connect them to the next available line.  PBXs, on the other hand, were used by large companies, gave each user their own phone line and allowed inter-office extension-to-extension calling, as well as direct dial from the outside world.  If my extension was 3202, I would have a direct dial phone number of, say, 415-555-3202.

Some companies instead opted for the phone company to do their internal switching.  This was known as a Centrex service.  The phone company provided hard wired analog phone lines to the customer, but enabled extension-to-extension direct dial.  Thus, if I was at extension 3202 and I needed to dial extension 3203, I could pick up the phone and just dial the four digits.  The phone company took care of routing it.

What does this have to do with ISDN?  We used to order Centrex service for our customers in the same Centrex group as their ISP. Thus, the customer’s ISDN line became an “extension” of the ISP’s Centrex group.  Not only could the customer then dial the ISP with four digits (not a big deal when the router is doing the dialing), but there were no toll charges on Centrex lines.  We used to nail the line up so it would never disconnect, and if it did for an reason, it would auto-redial.  And then we had dedicated Internet service on a dial-up line!

T1’s (E1’s elsewhere) were 1.544 Mbps, blazing fast at the time.  Unlike the single-pair ISDN line, T1’s were delivered on four wires, two for TX and two for RX.  I won’t get into the details of line coding on T1’s, which we all studied as junior network engineers.  T1 lines were truly dedicated, and provided a point-to-point connection from customer to ISP.  They were distance-priced, but I worked in San Francisco which is a small city, so it wasn’t usually a factor.  Because 1.544 Mbps was expensive for some customers, we had the option of ordering fractional T1s, fewer channels at a slower speed, but still faster than ISDN.  In the early days we had to terminate the T1 on an external CSU/DSU device, and then run a serial cable to the router, but eventually the CSU/DSU came integrated on the router interface card.

When I worked at the San Francisco Chronicle, we were providing Internet service via a T1 line terminating on a 2500-series router.  (The same one disabled by paint roller in this story.)  1000 users on a single T1 was painfully slow, and we made the decision to upgrade to a DS3 (T3) which ran at 45Mbps.  The interface for DS3 used two BNC coax connections.  I remember being amazed that the phone company could deliver service over coax, but it turns out that service into the building used fiber optics.  Inside the building we ran coax.  The run of coax from the basement to our 2nd floor data center was expensive, but the result was phenomenal.  The DS3, which we terminated on a brand new 7200vxr, was vastly superior to the crawling T1, and the effort paid off with our users.

DSL was groundbreaking.  There was nothing consumer-grade before that, and small companies could not even afford Internet connectivity.  I was one of the first home adopters of DSL.  The freshly-trained phone guy showed up at my apartment and installed a splitter box in the basement.  This was needed because residential service was ADSL, which multiplexed digital service on an analog line.  Unlike ISDN, which converted analog phone signal to digital, ADSL left the analog in tact, adding the digital part of the signal onto the higher frequencies.  The splitter box took the incoming phone line from the street and peeled off the high frequencies, providing an analog signal for telephones.  It then passed through the analog/digital mix intact to the modem, which just ignored the analog frequencies.  The phone guy then sat down with his toolbelt and tried to configure TCP/IP on my computer.  He gave up because he had no idea what he was doing.  I told him to leave me the IP addresses and I’d do it myself.  Eventually the telco would just send you an small filter to plug into each analog phone jack yourself, and they could turn on the service without sending a phone guy to rewire things.  Once DSL in its various forms came out, the Internet was available to the masses.  Of course, cable modems came shortly after.

We take for granted instant connectivity from every location on portable devices.  Once upon a time,  connectivity was only available at certain locations, often requiring dialing a service provider.  There was a real excitement as new technologies emerged for making connectivity faster and easier.  Now, of course, we just expect things to work and get angry when they don’t.

4

Elon Musk and remote work

In a recent article (paywall), Elon Musk has once again turned his wrath to remote workers.  Elon has a lot of good ideas, but also many bad ones, such as naming his child “X AE A-XII”.  This is certainly proof that we don’t need to take everything he says seriously.

Elon has said that those who want to work remote are “detached from reality” and give off “Marie Antoinette vibes” (I will forgive his apparent misunderstanding of history).  His argument, to the extent he even articulates it, seems to have two angles:

  • First, it is “morally wrong” to work remotely because so many people in the world cannot do their jobs from home.
  • Second, the productivity of remote workers is not the same.  (I’m extrapolating a bit here.)

I don’t think the first argument is terribly serious.  Just because some people (food service workers, factory workers, etc.) cannot work from home does not mean I should not.  Long before remote work, some people worked in clean offices while others worked in filthy coal mines.  We can debate the injustices of life, but I’m not convinced this disparity should guide office policies.

As to the second, well, as manager of many large teams, I can say that some of my most productive workers are fully remote, i.e., they work entirely from home.  I can think of two or three of my most respected and productive employees who have this arrangement.  Because they don’t live near a Cisco office, they have no choice.  I recently promoted one of them to prinicpal, a major accomplishment that is not given out lightly.  So, we cannot say that working from home automatically means lack of productivity.

On the other hand, I’ve had some very poor performers who worked from home.  This was particularly the case during the lockdowns.  I remember one engineer who seemed to be doing nothing, and when I checked her calendar it was empty.  She took a job elsewhere, finally.  But I, as any good boss should, was well aware of her lack of contribution and would have done something had she not taken the initiative herself.

Does being in the office guarantee productivity?  Not at all!  I can sit around and watch YouTube videos at work just the same as I can from home.  I remember a guy who sat near me many years ago, and had a rearview mirror on his monitor.  He was always playing Solitaire and every time I, or anyone else, walked by his desk he would glance in the mirror and minimize the game.  He wasn’t fooling anyone.

For me, the noise and distractions of the office often make productive work difficult.  Thankfully, post-lockdown, several Cisco buildings are virtually empty, and I decamp to one of them if I need to get actual work done.  Pre-COVID I used to head out to a coffee shop, put in earphones, and get productive work done there.  Open offices are the worst for this.  They make serious work nearly impossible.

Then there’s this…  Let me be open about it.  I never agreed with the lockdowns.  When they first implemented them, I wrote every congressman, city councilman, county supervisor, the health department, the governor, the president, and pretty much anyone I could think of with my opposition to what seemed to me a lunatic idea, and totally unneeded.  Now you can disagree with me vehemently, you can think I’m a jerk, and that’s fine.  But here’s the point:  Almost all the large corporations bought into it.  They could have fought these mandates, but they went along with them, shut their doors, and embraced remote work.  Many started marketing campaigns (and still have them) around hybrid work.  You cannot go 100% in for the thing and then make a 180 degree shift a few years later because you regret your decision.  The outcome of the lockdowns–a lot of people unwilling to return to the office–was entirely predictable.  I think corporations need to embrace the world they created, and live with the consequences of their choices.  Your workers want to be remote, let them be remote.  Sure, give them incentives to come into the office and be together.  Encourage them to do so.  But accept the reality of the new world.

Elon Musk reopened his factories mid-lockdown.  He may not know how to name a child, but I’ll at least give him points for consistency.

Finally, a use for generative AI?

I asked our ChatGPT to create a dialog between two network engineers in the style of the absurdist playwright Eugene Ionesco, author of The Bald Soprano.  It’s not even close to the original, but I do think it captures our existence quite nicely.  “We are the clowns of the digital realm” is perhaps my favorite line:

Engineer 1: Good evening. Have you ever stopped to wonder why networks are like mazes?

Engineer 2: Ah, the eternal question! Networks are like mazes because, well, because mazes are like networks!

Engineer 1: But what purpose do they serve? These intricate paths leading us nowhere. It’s all so absurd!

Engineer 2: Absurd indeed! We tirelessly configure routers and switches, creating connections that resemble a tangled web. Yet, do we ever truly reach our destination?

Engineer 1: No, my friend, we simply pass through endless loops, like Sisyphus rolling the boulder up the hill, only to have it roll back down.

Engineer 2: And what about those cryptic error messages we encounter? They mock us, like whispers of an elusive truth we can never grasp.

Engineer 1: Indeed, they are the whispers of a madman, taunting us with their indecipherable codes. “Error 404: Page not found.” How fitting!

Engineer 2: Ah, the irony! We seek connectivity, yet we are trapped in a world of disconnections and network outages.

Engineer 1: Our lives revolve around troubleshooting, forever trapped in an endless cycle of pinging and tracert-ing.

Engineer 2: It’s a tragic comedy, my friend. We are the clowns of the digital realm, forever chasing packets that vanish into thin air.

Engineer 1: And yet, we persist, day in and day out, our minds slowly unraveling like a faulty Ethernet cable.

Engineer 2: We are but players in this grand theater of network engineering, destined to forever dance to the rhythm of broken connections and elusive solutions.

Engineer 1: So, my friend, let us raise our glasses to the absurdity of it all. To the networks that confound us and the mysteries we strive to unravel!

Engineer 2: Cheers to that, my comrade! May we forever find solace in the chaos and laughter in the face of the inexplicable!

(They clink their glasses together, lost in the absurdity of their network engineer existence.)

2

ChatGPT and the problem of authority

I’ve been reluctant to comment on the latest fad in our industry, generative AI, simply because everybody has weighed in on it.  I do also try to avoid commenting on subjects outside of my scope of authority.  Increasingly, though, people are coming to me at work and asking how we can incorporate this technology into our products, how our competitors are doing it, and what our AI strategy is.  So I guess I am an authority.

To be honest, I didn’t play with ChatGPT until this week.  When I first looked at it, it wanted my email address and phone number and I wasn’t sure I wanted to provide that to our new AI overlords.  So I passed on it.  Then Cisco release an internal-only version, which is supposedly anonymous, so I decided to try it out.

My first impression was, as they say, “meh.”  Obviously its ability to interpret and generate natural language are amazing.  Having it recite details of its data set in the style of Faulkner was cool.  But overall, the responses seemed like warmed-over search-engine results.  I asked it if AI is environmentally irresponsible since it will require so much computing power.  The response was middle-of-the-road, “no, AI is not environmentally irresponsible” but “we need to do more to protect the environment.”  Blah, blah.  Non-committal, playing both sides of the coin.  Like almost all of its answers.

Then I decided to dive a bit deeper into a subject I know well:  Ancient Greek.  How accurately would ChatGPT be on a relatively obscure subject (and yet one with thousands of years of data!)

Even if you have no interest, bear with me.  I asked ChatGPT if it knew the difference between the Ionic dialect of Herodotus and the more common dialect of classical Athens.  (Our version, at least, does not allow proper names so I had to refer to Herodotus somewhat elliptically.)  It assured me it did.  I asked it to write “the men are arriving at Athens” in the dialect of Herodotus.  It wrote, “Ἀφίκοντο οἱ ἄνδρες εἰς Ἀθήνας,” which is obviously wrong.  The first word there, “aphikonto“, would actually be “apikonto” in the dialect of Herodotus.  He was well known for dropping aspirations.  The version ChatGPT gave me would be the classical Attic version.

I let ChatGPT know it was wrong, and it dutifully apologized.  Then I asked it to summarize the differences in the dialects.  It said to me:

Herodotus and Ionic writers typically removed initial aspirations, while the Attic dialect retained them. For example, “Ἀφίκοντο” (Herodotus) vs. “ἔφικοντο” (Attic)

Uh, you don’t need to know the Greek alphabet to see it made exactly the same mistake, again.  It should have said that Herodotus would use “Ἀπίκοντο” (apikonto) whereas in Attic the word would be “Ἀφίκοντο” (aphikonto.)

Here’s a little more computer-person-friendly example.  I decided to ask about my other favorite subject, retro computing.  I asked, “What is the difference between the Apple IIGS ProDOS 8, ProDOS 16, and GS/OS operating systems?”  Among other things, ChatGPT responded that GS/OS ” introduced features such as a desktop metaphor, drop-down menus, and a hierarchical file system.”  Which happens to be wrong.  ProDOS 16 actually introduced those features.  (In fact, ProDOS 8 introduced the hierarchy.)  I pointed this out.  “You are correct, and I apologize for the oversight in my previous response. I misspoke in stating that GS/OS introduced [those] features.”  Hmmm.

I’m testing it and I know the right answers.   The problem would arise if I were trusting ChatGPT to provide me with a correct answer.  There have been plenty of examples of mistakes made by ChatGPT, such as adding a “permit any” to the top of access-lists.

The issue is, ChatGPT sounds authoritative when it responds.  Because it is a computer and speaking in natural language, we have a tendency to trust it.  And yet it has consistently proven it can be quite wrong on even simple subjects.  In fact, our own version has the caveat “Cisco Enterprise Chat AI may produce inaccurate information about people, places, or facts” at the bottom of the page, and I’m sure most implementations of ChatGPT carry a similar warning.

Search engines place the burden of determining truth or fiction upon the user.  I get hundreds or thousands of results, and I have to decide which is credible based on the authority of the source, how convincing it sounds, etc.  AI provides one answer.  It has done the work for you.  Sure, you can probe further, but it many cases you won’t even know the answer served back is not trustworthy.  For that reason, I see AI tools to be potentially very misleading and potentially harmful in some circumstances.

That aside, I do like the fact I can dialog with it in ancient Latin and Greek, even if it makes mistakes.  It’s a good way to kill time in boring meetings.

7