ccie

All posts tagged ccie

The War on Expertise

Posted by ccie14023 on March 7, 2023
Posted in: Reflections. Tagged: , , , , . Leave a Comment

Like many network engineers, I quickly fell in love with my field and worked hard to master it.  I got into networking when I was working in desktop support.  The behind-the-scenes stuff was way more interesting to me than the front lines.  Back in the late nineties, I bought a library of books to learn this field.  Perlman, Comer, and Stevens were the classics.  I rounded it out with blue-and-white Cisco Press books by Doyle, Peplnjak, Williamson, and many others.  I studied these books religiously, read through config guides, and practiced in the lab.

The network engineers on my team and I loved to debate the arcana of this mysterious field.  We always tried to one-up each other, learning new technologies, new protocols, and attaining new technical certifications.  I’ve worked with engineers who are smarter than I am, and better than I am, but that always motivated me to learn more.

I bring this up because I’ve had multiple conversations with multiple execs, for many years, in which they seem to decry the virtue of expertise.  Network engineers “revel in complexity”, they don’t realize their time is past, the build networks that need “armies of CCIEs to maintain”, and they hate simplicity.  If only the pesky network engineers would get out of the way, the glorious MBAs could build us simple and elegant products, which is how the industry is going, don’t you know!

In short, our industry is suffering through a war on expertise.  Those arcana we love to master have put a target on our back.  If you want to learn those things, you must be reveling in complexity.  Go find something else to do, ChatGPT will replace you!

The first mistake in this line of thinking is the assumption that network engineers want to build networks that are complex.  We actually don’t.  A couple of anecdotes:

When I was working for a Gold partner, I was sent to help out an IT manager of a rather small company, only four sites.  She had contracted VPLS from two service providers, and asked me to implement a complex load balancing scheme she had conceived.  I begged her not to make me do it, but she insisted.  I ended up building a functional mess, a combination of PBR and EIGRP offset access-lists.  Man, was it ugly.  But it worked.  Then I got laid off from the partner, and a year later she was calling me, begging me to come back because nobody could figure out how it worked.  I didn’t want to build something that ugly and she didn’t need it.

Second anecdote.  My wife had to go in for a surgical procedure a few years ago.  We went to the best doctor in San Francisco.  When he got into the procedure he found that her anatomy is not conventional, and it was a very difficult procedure.  In the recovery room, he told us most doctors would have stopped.  My wife wanly smiled and said, “well, I’m sure you like a challenge.”  He looked back at her and said, “no, we like it when it’s easy.”

I think this is where the execs misunderstand expertise.  99.9% of the time, your airline flight could be handled by a low-time pilot who can work the automation systems.  But when the engines fail, you want Sully at the controls.  Just because some people understand complexity and study difficult concepts, it doesn’t follow that they want complexity.  When I administered networks, I wanted it to be easy.  But I was ready for when it was hard.

The war on expertise seems to me to be a war on the human spirit.  The CCIE exam, whatever you think of it, was a heck of a challenge, and passing it was one of the proudest days of my life.  Human beings want to learn, to grow, to push their limits, and to test themselves.  That’s why we spend hours in the lab.  We should encourage this behavior.  We should want people in our business who seek subject matter expertise and mastery.  We can make things simpler, fine, but we should still encourage expertise.

At the end of the day, networks are inherently complex.  A network is a large distributed system, connecting numerous devices running numerous operating systems over diverse transport mechanisms using a wide variety of protocols.  You can simplify the protocols a bit, but ultimately most simplification of networks is done one of two ways:  reducing the number of choices an administrator can configure, or abstracting and hiding the underlying complexity.  In the first case, you may close out necessary use cases.  In the case of abstraction, well, it works great until something breaks.  Then you need to call a network engineer.

I’m not in any way saying the new tools, from programmability to automation systems like Ansible, to “controllers” are unnecessary.  Far be it.  Any tool that makes an engineer’s job easier will be embraced by engineers.  I am saying that we need to stop blaming complexity on those who manage to understand it.

19

On being a VOIP guy (not)

Posted by ccie14023 on December 22, 2022
Posted in: NetStalgia. Tagged: , , , . Leave a Comment

Well, the blog has been languishing for a while, as I’ve been extraordinarily busy with a new EVP, a round of layoffs, and many personal distractions.  Here’s a little Netstalgia piece, not really technical, for your enjoyment.

I’ve told a few stories about my years at the Cisco Gold Partner, where I did both pre- and post-sales roles.  The Cisco practice in the San Francisco office was new, so being the only Cisco guy required wearing a lot of hats.  That said, one day I wore a hat I didn’t expect or want.

At the end of every week I’d look at our calendar to figure out my schedule for the next week.  It was maintained by a project coordinator.  Some appointments I had put on the calendar myself, others were requested by account managers or customers directly.  One day as I looked at my calendar, I saw the following week booked.  “City of San Mateo,” it said.  I had no experience with this customer, so I called our project coordinator to figure out what the mystery job was.

“You’ll be placing phones,” she said.

“Placing them?” I asked, confused.  She told me we had sold a VOIP deal with San Mateo to replace all of their PBX-phones with Cisco IP phones.  The entire San Francisco office had been roped in to physically placing the phones on desks across the city.  Even our Citrix guy was going to be there.

I called my VP of services and complained.  “I have two CCIEs and you want me to run around for a week plugging in phones?”

“Just be glad you have billable hours,” he said.  Were we really that desperate?

It turns out, yes.  Myself, the office Citrix guy, and one or two other folks met in San Mateo city hall and divided up box after box of IP phones.  We had to do city hall and library, which were the easiest.  Then I ended up doing the police headquarters.  I remember putting phones on all the desks in the detective room, with concerned police officers looking on as I rooted around on my hands and knees for data jacks under their desks.  I had to move weapons (non-lethal), ballistic vests, and other police gear to find the ports.

I also had to do the fire department.  For a small city, San Mateo has a lot of fire stations.  It wasn’t always easy to park.  The first one I pulled up to in my BMW, loaded with phones, had no parking anywhere.  I found a notepad and a pencil in my car, scrawled out “OFFICIAL BUSINESS” on a sheet of lined paper, stuck it in my window, and parked on the sidewalk.  I used my pass at several fire stations, earning quizzical looks from firemen when I parked myself on the sidewalk in front of their station.

I learned an important lesson in leadership from this event.  If the VP had called a meeting the week before, he could have said the following:  “Look team, I know you’re all highly skilled and don’t want to do manual labor.  But we have a big deal here, it’s important to the company, and it’s all hands on deck.  I’ll be there myself with you placing phones.  Let’s get this done and I’ll buy you all a nice dinner at the end of the week.”  Had he said something like this, I think we would have rallied around him.  Instead, he just surreptitiously had it added to the calendar and copped an attitude when challenged.  He actually wasn’t a bad guy, but he missed on this one.

Anyways, plugging in phones is the closest I became to being a VOIP guy.

2

Back to Cisco Live

Posted by ccie14023 on June 10, 2022
Posted in: Perspectives. Tagged: , , . 3 comments

I haven’t posted in a while, for the simple reason that writing a blog is a challenge.  What the heck am I going to write about?  Sometimes ideas come easily, sometimes not.  Of course, I have a day job, and part of that day job involves Cisco Live, which is next week, in person, for the first time in two years.  Getting myself ready, as well as a coordinating with a team of almost fifty technical marketing engineers, does not leave a lot of free time.

For the last several in-person Cisco Lives, I did a two-hour breakout on programmability and scripting.  The meat of the presentation was NETCONF/RESTCONF/YANG, and how to use Python to configure/operate devices using those protocols.  I don’t really work on this anymore, and I have a very competent colleague who has taken over.  I kept delivering the session because I loved doing it.  But good things have to come to an end.  At the last in-person Cisco Live (Barcelona 2020), I had just wrapped up delivering the session for what I assumed would be the last time.  A couple of attendees approached me afterwards.  “We love your session, we come to it every year!” they told me.

I was surprised.  “But I deliver almost the same content every year,” I replied.  “I even use the same jokes.”

“Well, it’s our favorite session,” they said.

At that point I resolved to keep doing it, even if my experience was diminishing.  Then, COVID.

I had one other session which was also a lot of fun, called “The CCIE in an SDN world.”  Because it was in the certification track, I wasn’t taking a session away from my team by doing it.  There is a bit about the CCIE certification, its history, and its current form, but the thrust of it is this:  network engineers are still relevant, even today with SDN and APIs supposedly taking over everything.  There is so much marketing fluff around SDN and its offshoots, and while there may be good ideas in there (and a lot of bad ones), nevertheless we still need engineers who study who to manage and operate data networks, just like we did in the past.

I will be delivering that session.  I have 50 registered attendees, which is far cry from the 500 I used to pack in at the height of the programmability gig.  Being a Senior Director, you end up in limbo between keynotes (too junior) and breakouts (too senior).  But the cert guys were gracious enough to let me speak to my audience of 50.

Cisco Live is really the highlight of the TME role, and I’m happy to finally be back.  Let’s just hope I’m still over my stage fright, I haven’t had an audience in years!

5

Memories of Cisco Live

Posted by ccie14023 on February 25, 2021
Posted in: Reflections. Tagged: , , , , , , . 2 comments

The last Cisco Live I attended was in Barcelona in January 2020.  As I was in the airport heading home, I was reading news of a new virus emerging from China.  I looked with bemusement at a troop of high-school-age girls who all had surgical masks on.  Various authorities told us not to wear masks, saying they don’t do much to prevent viral spread at a large scale.  The girls kept pulling the masks on and off.  I thought back on my performance at Cisco Live, and looked forward to Cisco Live in Las Vegas in the summer.  Who knew that, a few months hence, everyone would be wearing masks and Cisco Live,  physically, would be indefinitely postponed.

For Technical Marketing Engineers (TMEs), Cisco Live (technically Cisco Live!) measures the seasons of our year like the crop cycle measures the seasons of a farmer’s year.  Four times annually a large portion of our team would hop on an airplane and depart for Europe, Cancun, Melbourne, or a US city.  Cancun and Melbourne were constant, but the European and US cities would change every couple of years.  In my time with Cisco, I have traveled to Cancun and Melbourne, Berlin, Barcelona, Las Vegas, Orlando, and San Diego to present and staff Cisco Live.

A trade show may just be a corporate event, but for those of us who devoted our career to that corporation’s products, it’s far more than a chance for a company to hawk its products.  The breakout sessions and labs are critical for staying up-to-date on a fast-moving industry, the keynotes are always too high-level but with entertaining productions, and the parties are a great chance to connect with other network engineers.  CL was fun for participants, exhausting for those of us staffing it, but still my favorite part of the job.

Cisco Live was originally called Networkers, and started in 1990.  For many years I badly wanted to go to this temporary Mecca of networking technology, but I worked for companies that would not pay the cost of a badge and the travel fees, a total of thousands of dollars.  Even when I first worked at Cisco, from 2005-2007, as a lowly TAC engineer I never had the opportunity to attend.  My first trip to CL came in 2007, when I was working for a Gold partner.  They sent several of us to the Anaheim show, and I remember well the thrill of walking into a CL for the first time.  I walked the show floor, talked to the booth staffers, and attended a lot of breakout sessions of varying quality.  I was quite excited to go to the CCIE party, but I’m not sure why I thought a party full of CCIEs would really be all that exciting.  I remember hanging out by myself for an hour or so before I gave up because I didn’t know anyone there.

The same partner sent me to Orlando in 2008 as well, just barely.  The recession was starting and we were short on cash.  My boss wanted me to share a badge with a colleague, and I didn’t like the idea of having to juggle time slots nor or trying to explain to security how my name could be “Nguyen”.  Thankfully, they ponied up the cash for a second badge.  I’m not a fan of loud music, so I generally don’t go to the CL party, but for Orlando they opened up Universal Studios for us and the aforementioned Nguyen and I, along with a couple others, had a great time on the rides and attending the Blue Man Group.  (OK, some loud music there, but it is an entertaining show.)

I attended CL once more before I came back to Cisco–in 2014, ironically, as an employee of Juniper.  Somehow I convinced my boss to give me a pass on the grounds of researching what Cisco IT was doing.  (They do present at Cisco Live.)  I remember sitting in just a few rows back at the keynote as John Chambers presented, amused I’d be bringing a report back to Juniper about what I’d heard.

 

My view of Chambers at CL 2014

It was actually at Cisco Live when I first got the idea to be a technical marketing engineer.  It’s a bit embarrassing, but I sat in a presentation given by a TME and thought, “I could do better than this guy.”  It took a few years, but I finally managed to get into tech marketing.

I became a Principal TME at Cisco in late 2015 and was told I’d be presenting at Cisco Live in Berlin in January, 2016!  Needless to say, I was thrilled to be given the opportunity, humbled, and more than a little nervous about standing up in front of an audience at the fabled event.

It’s been a sad year in so many ways.  After I came home from Barcelona in January 2020, I received another Distinguished Speaker award and knew I would be inducted into the Hall of Fame.  This was a dream of mine for years, but instead of standing up in front of my peers at Cisco Live Vegas to receive the award, it was mailed to me.  There would be no show floor, no breakouts, no CCIE parties.  The event would go virtual.  I must say, I am impressed with the CL team’s ability to pivot to a virtual format in so short a time.  Still, it was a sad year for those of us who organize the event, and those who were hoping to attend.

In the next couple posts, I thought I would offer a little behind-the-scenes look at how we put on CL, and look at a few events from the past.

CCIE Enterprise Infrastructure

Posted by ccie14023 on June 14, 2019
Posted in: Certifications. Tagged: , , , , , , , , , , . 5 comments

There were quite a few big announcements at Cisco Live this year.  One of the big ones was the overhaul of the certification program.  A number of new certifications were introduced (such as the DevNet CCNA/CCNP), and the existing ones were overhauled.  I wanted to do a post about this because I was involved with the certification program for quite a while on launching these.  I’m posting this on my personal blog, so my thoughts here are, of course, personal and not official.

First, the history.  Back when I was at Juniper, I had the opportunity to write questions for the service provider written exams.  It was a great experience, and I got thorough training from the cert program on how to properly write exam questions.  I don’t really remember how I got invited to do it, but it was a good opportunity, as a certified (certifiable?) individual, to give back to the program.  When I came to Cisco, I quickly connected with the cert program here, offering my services as a question writer. I had the training from Juniper, and was an active CCIE working on programmability.  It was a perfect fit, and a nice chance to recertify without taking the test, as writing/reviewing questions gets your CCIE renewed.

As I was managing a team within the business unit that was working on Software-Defined Access and programmability, it seemed logical for me to talk to the program about including those topics on the test.  I can assure you there was a lot of internal debate about this, as the CCIE exam is notoriously complex, and the point of our Intent-Based Networking products is simplicity.  One product manager even suggested a separate CCIE track for SD-Access, an idea I rejected immediately for that very reason.

Still, as I often point out here and elsewhere, SDN technologies do not mitigate the need for network engineers.  SDN products, all SDN products, are complex precisely because they are automated.  Automation enables us to build more complex things, in general.  You wouldn’t want to configure all the components of SD-Access by hand.  Still, we need engineers who understand what the automation tools are doing, and how to work with all the components which comprise a complex solution like SD-Access.  Network engineers aren’t going to disappear.

For this reason, we wanted SD-Access, SDWAN, and also device programmability (NETCONF/YANG, for example) to be on the lab.  We want to have engineers who know and understand these technologies, and the certification program is a fantastic way to help people to learn them.  I, and some members of my team, spent several months working with the CCIE program to build a new blueprint, which became the CCIE Enterprise Infrastructure.  The storied CCIE Routing and Switching will be no more.

At the end of the day, the CCIE exam has always adapted to changed in networking.  The R/S exam no longer has ISDN or IPX on it, nor should it.  Customers are looking for more automated solutions, and the exam is keeping pace.  If you’re studying for this exam, the new blueprint may be intimidating.  That said, CCIE exams have always been intimidating.  But think about this:  if you pass this exam, your resume will have skills on it that will make you incredibly marketable.

The new CCIE-EI (we always abbreviate stuff, right?) breaks down like this:

  • 60% is classic networking, the core routing protocols we all know and love.
  • 25% is SDx:  SD-Access and SD-WAN, primarily.
  • 15% is programmability.  NETCONF/YANG, controller APIs, Ansible, etc.

How do you study for this?  Like you study for anything.  Read about it and lab it.  There is quite a bit of material out there on all these subjects, but let me make some suggestions:

Programmability

You are not expected to be a programming expert for this section of the exam.  It’s not about seeing if you can write complex programs, but whether you know the basics well enough to execute some tasks via script/Ansible/etc instead of CLI.  DevNet is replete with examples of how to send NETCONF messages, or read data off a router or switch with programmable interfaces.  Download them, play with them, spend some time learning the fundamentals of Python, and relax about it.

  • Learn:  DevNet is a phenomenal resource.  Hank Preston, an evangelist for DevNet, has put out a wealth of material on programmability.  In addition, there is the book on IOS XE programmability I wrote with some colleagues.
  • Lab:  You can lab programmability stuff easily on your laptop.  Python and ncclient are free, as is Ansible.  If you have any sort of lab setup already, all you need to do is set up a Linux VM or install some tools on to your laptop.

Software-Defined

This is, as I said before, a tough one to test on.  After all, to add a device to an SD-Access fabric, you select it and click “Add to Fabric.”  What’s there to test?  Well, since these are new products you of course need to understand the components of SD-Access/SDWAN and how they interoperate.  How does policy work?  How do fabric domains talk to non-fabric domains?  There is plenty to study here.

  • Learn:  Again, we’ve written books on SD-Access and SD-WAN.  Also, we are moving a lot of documentation into Cisco Communities.
  • Lab:  Well, this is harder.  We’re working on getting SD-Access into the hands of learning partners, so you’ll have a place to get your hands on it.  We’re also working on virtualizing SD-Access as much as possible, to make it easier for people to run in labs.  I don’t have a timeframe on the latter, but hopefully the former we can do soon.

These are huge but exciting changes. I’ve been very lucky to have landed at a job where I am at the forefront of the changes in the industry, but this new exam will give others the opportunity to move themselves in that direction as well.  Happy labbing!

Cisco Live is over! Long Live Cisco Live!

Posted by ccie14023 on June 13, 2019
Posted in: Perspectives. Tagged: , , , , , , , , , , . 1 comment

I think it’s fair to say that all technical marketing engineers are excited for Cisco Live, and happy when it’s over.  Cisco Live is always a lot of fun–I heard one person say “it’s like a family reunion except I like everyone!”  It’s a great chance to see a lot of folks you don’t get to see very often, to discuss technology that you’re passionate about with other like minded people, to see and learn new things, and, for us TMEs, an opportunity to get up in front of a room full of hundreds of people and teach them something.  We all now wait anxiously for our scores, which are used to judge how well we did, and even whether we get invited back.

It always amazes me that it comes together at all.  In my last post, I mentioned all the work we do to pull together our sessions.  A lot of my TMEs did not do sessions, instead spending their Cisco Live on their feet at demo booths.  I’m also always amazed that World of Solutions comes together at all.  Here is a shot of what it looked like at 5:30 PM the night before it opened (at 10 AM.)  How the staff managed to clear out the garbage and get the booths together in that time I can’t imagine, but they did.

The WoS mess…

My boss, Carl Solder, got to do a demo in the main keynote.  There were something like 20,000 people in the room and the CEO was sitting there.  I think I would have been nervous, but Carl is ever-smooth and managed it without looking the least bit uncomfortable.

My boss (left) on the big stage!

The CCIE party was at the air and space museum, a great location for aviation lovers such as myself.  A highlight was seeing an actual Apollo capsule.  It seemed a lot smaller than I would have imagined.  I don’t think I would ever have gotten in that thing to go to the moon.  The party was also a great chance to see some of the legends of the CCIE program, such as Bruce Caslow, who wrote the fist major book on passing the CCIE exam, and Terry Slattery, the first person to actually pass it.

CCIE Party

I delivered two breakouts this year:  The CCIE in an SDN World, and Scripting the Catalyst.  The first one was a lot of fun because it was on Monday and the crowd was rowdy, but also because the changes to the program were just announced and folks were interested in knowing what was going on.  The second session was a bit more focused and deeper, but the audience was attentive and seemed to like it.  If you want to know what it feels like to be a Cisco Live presenter, see the photo below.

My view from the stage

I closed out my week with another interview with David Bombal, as well as the famous Network Chuck.  This was my first time meeting Chuck, who is a bit of a celebrity around Cisco Live and stands out because of his beard.  David and I had already done a two-part interview (part 1, part 2) when he was in San Jose visiting Cisco a couple months back.  We had a good chat about what is going on with the CCIE, and it should be out soon.

As I said, we love CL but we’re happy when it’s over.  This will be the first weekend in a long time I haven’t worked on CL slides.  I can relax, and then…Cisco Live Barcelona!

Interviewing #1: How I got my first networking job

Posted by ccie14023 on March 15, 2019
Posted in: NetStalgia, Technical Interviewing. Tagged: , , , , , , , . 4 comments

I’ve wanted to kick off a series for a while now on technical interviewing. Let me begin with a story.

My first job interview for a full network engineering role was at the San Francisco Chronicle in 2000. I had been working for five years in IT, mostly doing desktop and end-user support. I then decided to get a master’s degree in telecommunications management, which didn’t help at all, followed by a CCNA certification, which got me the interview.

My first interview was with the man who would be my boss. Henry was a manager who had almost no technical knowledge about networking, but I didn’t know that at the time. “Do you know Foundry switches at all?” Henry asked.

“No.” I was already worried.

“I doubted you would. That’s ok because we want to replace them all with Cisco and you know Cisco.” He pulled out a network diagram and handed it to me. “If you look at this, do you see a problem?” he asked.

I had never worked on a network larger than a couple switches, and now I was staring at a convoluted diagram depicting the network of the largest newspaper in Northern California. I was looking at subnet masks, link speeds, and hostnames, trying to find something wrong.

“I’m not sure,” I had to reply meekly.

He pointed at the main core switch for the network. There was only one, with no redundancy.  “There’s a huge single point of failure,” he said. I felt stupid missing the forest for the trees.

Henry brought me upstairs to interview with Tom, who was an on-site project management contractor from Lucent. I was extremely nervous–Lucent (later Avaya) was a big name in the industry and this guy worked for them! Henry left me with Tom. Tom pulled out a copy of the same diagram Henry was showing me earlier.

“Do you notice anything wrong with this?” he asked.

“Wow, that’s a huge single point of failure,” I replied.

He nodded his head in approval. “That’s right–very good.” He asked me a technical question about supernetting. I answered nervously, although it quickly became clear I knew more than he did.

The door flew open and another guy named Vincent walked in. He was the desktop support contractor, but again I didn’t know that. “Ask Jeff a few technical questions,” Tom said.

“Question number one,” said Vincent. “If you were running a network this size, would you subnet it?”

Now the answer seemed obviously to be “yes”, but I was trying to figure out if this is a trick. “Yes,” I answered, deciding to play it safe.

“Good! Next question: Can you route NetBios?” My desktop years were almost exclusively dedicated to Macs and I didn’t even know what NetBios was. I figured it was a 50/50 shot, and the way he asked it seemed to suggest the answer.

“No,” I said, trying to sound confident.

“He’s good,” said Vincent.

Next, the door flew open again, and in walked Bing. Bing was carrying some sort of network device with her. She handed it to me. “Is this a switch or a hub?” she asked. There was no obvious labeling on it, and as I turned the device over and over again in my hands, I had a sinking feeling.

“I don’t know,” I replied.

“Look at this,” Bing said. She pointed to a collision light. “Since there is one on each port, you can tell this is a switch.”

We don’t have collision lights on switches any more, but at the time we did and she had a valid point. Realizing this, I explained to her that a since a hub has a single collision domain, it would only have one collision light. I explained to her the concept of a collision domain, and how a switch worked versus a hub. It turns out she was a project manager for desktop support and she didn’t know any of that. Someone had just shown her the collision light thing and she thought it would be a good question.

“He’s good,” said Bing.

Tom had told me my next interview would be with a CCIE from Lucent. Now that was definitely intimidating. I knew of the reputation of CCIEs, and I didn’t expect to do well. The CCIE guy never showed up. As Tom was walking me to the elevator, however, we ran into him in the hallway. It turns out that Mike, who is still a friend of mine, and who later got three CCIE’s, had not passed the exam yet. We ended up talking about his home lab for a few minutes.

“He’s good,” said Mike. And a good thing too, as I’ve been in a couple of interviews with Mike and I’ve seen him grill people mercilessly.

I got a call with a job offer a few days later, and ended up working there five years.

For a while now I’ve had several posts in my drafts folder on the subject of technical interviewing. As you can see from the above story, interviews are often chaotic, disorganized, and conducted by unqualified people who have no plan. In the case of the San Francisco Chronicle, they made the right decision on me, and I don’t think anybody there would dispute that. I was thankful to begin my career in network engineering.

That said, I’ve had other interviews that didn’t go so well. Over the next few articles, I’d like to cover technical interviewing. Why do we interview people? How can we select good people from bad people? How worthwhile are the typical technical questions? Are gotchas worth throwing out “just to see how the candidate reacts”? Are interviews purely subjective or can we make them data-driven and objective?

I’ll throw out a few more anecdotes from my own experience to illustrate my points–feel free to comment with some of your own!

3

Two Years of Ten Years a CCIE

Posted by ccie14023 on May 25, 2018
Posted in: Reflections, Ten Years a CCIE. Tagged: , , , , , . 2 comments

Two years ago I published my Ten Years a CCIE series.  Actually, I had written the series a couple years before I published it, but as I say in my introduction to the series, I felt it was a bit self-indulgent an uninteresting, so I scrapped it for a while.  The original pieces were dictated, and I’ve been meaning to go back and clean up some of the grammatical errors or grating phrases, but haven’t had the time.  Not a lot of people have read it, nor did I expect many to read it, since I generally don’t advertise the blog in social media, or anywhere really.  But the feedback from the few who have read it has been positive, and I’m gratified for that.

Things have changed a lot since I got into networking in 1995, and since I passed my CCIE in 2004.  But it’s also amazing how much has stayed the same.  TCP/IP, and in fact IPv4, is still the heart of the network.  Knowledge of OSPF and BGP is still key.  For the most part, new controllers and programmable interfaces represent a different way of managing fundamentally the same thing.

The obvious reasons for this are that networks work and are hard to change.  The old protocols have been sufficient for passing data from point A to point B for a long time.  They’re not perfect but they are more than adequate.  They are hard to change because networks are heterogeneous.  There are so many types of different systems connecting to them, that if we wanted to fundamentally alter the building blocks of networks, we’d have to upgrade a lot of systems.  This is why IPv6 adoption is so slow.

Occasionally I poke around at TechExams.net to see what newer network engineers are thinking, and where they are struggling.  I’m probably the only director-level employee of Cisco who reads or comments on that message board.  I started reading it back when I was still at Juniper and studying for my JNCIE, but I’ve continued to read it because I like the insights I get from folks prepping for their certifications.  People are occasionally concerned that the new world of controllers and automation will make their jobs obsolete.

I built the first part of my career on CLI.  Now I’m building it on controllers and programmability.  In this industry, we have to adapt, but we don’t have to die.  Cars have changed drastically, with on-board computer systems and so forth, but we still need mechanics.  We still need good network engineers.

To be honest, I was getting tired of my career by the time I left Juniper and came to Cisco.  I was bored.  I thought of going back to school and getting a Ph.D. in classical languages, my other passion.  Getting married helped put an end to that idea (Ph.D.’s in ancient Greek make a lot less than network engineers) but when I came back to Cisco, I felt revitalized.  I started learning new things.  Networking was becoming fun again.

I wrote the “Ten Years a CCIE” series both for people who had passed the exam and wanted to have some fun remembering the experience, as well as for people struggling to pass it.  Some things change, as I said, but a lot remains the same.  I still think, closing in on 15 years since I took the exam, that it’s still worth it.  I still think it’s a fantastic way to launch a career.  The exam curriculum will adapt, as it always does, with new technologies, but it’s an amazing learning experience if you do it honestly, and you will be needed when you make it through.

The value of a CCIE

Posted by ccie14023 on November 2, 2016
Posted in: Ten Years a CCIE. Tagged: , , , . 10 comments

In the final post in my “Ten Years a CCIE” series, I take a look at the age-old question: Is a CCIE really worth it? I conclude the series with some thoughts on the value of this journey.

I’ve written this series of posts in the hope that others considering the pursuit of a CCIE  would have some idea of the process, the struggles, and the reward of passing this notorious exam.  I would like to finish this series with a final article examining the age-old question:  What is the value of a CCIE?  In other words, was it worth it?  All of the other articles in the series were written a couple of years ago, back when I worked for Juniper in IT, with only some slight revisions before publishing.  However, this article is being written now (late 2016), in very different circumstances.  I work at Cisco now, not Juniper.  I work in the switching business unit, not in IT.  I work in product management, and as a Principal Engineer have direct influence on the direction of our products.  I specifically work on programmability, automation, and SDN solutions.  As I write this, I see many potential CCIE candidates wondering if it is even worth pursuing a career in networking any more.  After all, won’t SDN and automation just eliminate their jobs? Wouldn’t they be better off learning Python?  Isn’t Cisco a company in its death throes, facing extinction at the hands of upstarts like Arista and Viptela?

Well, I can’t predict the future.  But I can look at the issues from the perspective of someone who has been in the industry a long time now, and at least set your mind at ease.  The short answer:  it’s worth it.  The longer answer requires us to examine the question from a few different angles.

Technical Knowledge

As I’ve pointed out in previous articles, during the CCIE preparation process, you will master a vast amount of material, if you approach the test honestly.  But, as anyone in this industry knows, the second you pass, a timer starts counting down the value of the material you learned.  On my R&S exam (2004), I studied DLSw+, ATM, ISDN, and Frame Relay.  On my Security (2008) exam, I studied the VPN 3k concentrator, PIX, and NAC Framework.  How much of that is valuable now?

True, I studied many things that are still in use today.  OSPF, BGP, EIGRP, and ISIS were all heavily tested on the R&S exam even back in 2004.  But how much of it do I remember?  I certainly have a good knowledge of each of them, but even after taking the JNCIE-SP exam less than two years ago, I find my knowledge of the details of routing protocols fading away.  If I were to take the JNCIE exam today, I would certainly fail.

Thus, some technologies I learned are obsolete, and some are not, but I have forgotten a lot.  However, there are also technologies that I never learned.  For example, ISE and Cisco TrustSec are huge topics that I am just starting to play with.  Despite my newness to these technologies, I do have the same right to call myself a “Security CCIE” as a guy who passed it yesterday, and knows those subjects cold!

So, then, what about the value of the technical knowledge?  Is it worth it?

First, it is because no matter what becomes obsolete, and no matter what you forget, the intensity of your study will guarantee you still have a core of knowledge that will be with you for a long time.  Fortunately, despite the rapidity of change our industry is supposedly undergoing, networking is conservative by nature.  The Internet is a large distributed system consisting of many systems running disparate operating systems and they have to work together.  Just ask the guys pushing IPv6.  The core of networking doesn’t change very much.  You will also learn that even new technologies, like VXLAN, build on concepts you already know.

Second, even obsolete knowledge is valuable.  Newer engineers often don’t realize how certain technologies developed, or why we do things the way that we do.  They don’t know the things that have been tried and which failed in the past.  Despite the premium our industry places on youth, old-timers do have important insight gleaned from playing with things like, say, DLSw+.  It’s important for us not to simply wax nostalgic (as I am here) about the glorious days of IPX, but to explain to younger engineers how these dinosaurs actually worked, so that they can understand what decisions protocol designers made and why, and why some technologies fell into disuse or were replaced.  And, millennials, it is your responsibility to sit up and listen, if not to seek out such information.

Technical “soft skills”

In addition to specific technical knowledge, CCIE preparation simply makes you better at configuring and troubleshooting in general.  The hours of frustration in the lab and the broken configs that have you pulling your hair out ultimately help you to learn what makes a network break, and how to go about systematically debugging it.  Certainly you can learn those skills elsewhere, but when you are faced with the challenge of building a lab in a short amount of time, you have to be able to think quickly on your feet.  Even troubleshooting a live network outage doesn’t quite compare the intensity of the CCIE crucible, the eight hour slog during which one mistake can cost you months of studying.  The exams where I took multiple attempts required me to seriously up my game between tries, and each time I’ve come out a stronger thinker.

Non-Technical Skills

Passing the CCIE exam honestly is a challenge, plain and simple.  It requires discipline, persistence, extreme attention to detail, resourcefulness, ability to read documentation carefully, and time management skills.  While you won’t pass the CCIE without having those skills to begin with, there is no question that the rigors of CCIE preparation will help to hone them.  You will be a better person all around if you submit yourself to the discipline of passing the exam honestly.

It’s a bit like studying the martial arts.  Most martial artists will tell you that, in addition to being able to deliver a mean axe kick to the head, they have achieved self-discipline and confidence from their practice of martial arts, and that they find these skills apply elsewhere in life.  It’s true of any rigorous pursuit, really, and definitely true of the CCIE.  Personally I have no doubt that the many hours of laborious study have made me a more detail-oriented and better engineer.

Employment value:  Employee’s perspective

In my own case, my CCIE certification was directly responsible for my getting hired at Cisco TAC in 2005.  It was absolutely essential for me to move to a VAR in 2007.  Believe it or not, it was critical in my getting a job at Juniper in 2009, and having two CCIE’s and a JNCIE helped open the door to be re-hired at Cisco in 2015.  Obviously, employers see value in it.

You still see CCIE certification listed as a requirement in many job descriptions.  Without one you are cut out of those positions.  Many employers see it as a key differentiator and qualification for a senior position in network engineering.  VARs are still required to have a number of CCIEs on staff, so for some positions they will only talk to someone who has a CCIE.

Rarely is it the only qualification, however, and rarely is it enough, on its own, to get you hired.  Shortly after I passed my second CCIE I got laid off from the VAR where I worked.  I ended up interviewing at Nexus IS (now Dimension Data), another VAR, a couple months later.  Having been out of work I was rusty, and the technical interviewer grilled me mercilessly.  I’ve stated in the past that I don’t find this sort of thing productive, and I was uncomfortable, and didn’t have a great performance.  Despite the fact that I had two CCIE’s, and despite the value of such credentials for a reseller, I got a call from HR in a few days telling me that they chose not to hire me.  (These things often work out in the end;  I am in a good place now, and I wouldn’t have made it here if I had stayed in the VAR world.)  The entire experience reinforces a point I made in my Cheaters post:  don’t think an ugly plaque and a number are a guarantee of employment.

That said, there is little doubt that a person with X skill-set and a CCIE is in a better place than a person with just X skill-set.  As I said above, the certification simply opens doors.  You cannot rely on it alone, but combined with good experience, the certification is invaluable.  I would never have made it this far without one.  Now it’s true that there are some very talented and knowledgeable engineers who don’t have, and in some cases disdain, the CCIE.  Many of these folks do quite well and have successful careers.  But again, add the famous number and you will always do better in the end.

Employment value:  Employer’s perspective

Let’s turn it around now, and look at it from the perspective of a hiring manager.  When I have the stack of resumes in front of me, how important is it for me to look for that famous CCIE number?

This is a much harder question to answer, and I’ve touched on some of the problems of the CCIE above.  If you are looking at somebody who has a CCIE #1xxx, you know you are getting someone with age and experience.  But if you’re looking for someone to do hands-on implementation work, will this be the right candidate?  Maybe, of course, if Mr. #1xxx has been doing a lot of that lately.  The point is, you can’t tell from the number alone.

Now let’s say you are looking at CCIE #50xxx.  Does her CCIE number mean she will be the perfect fit for the hands-on job?  Quite possibly, because she has a lot of recent hands-on experience.  Will she be unqualified for the management role because she lacks the experience of CCIE #1xxx?  Who knows?

The point is, from a hiring manager’s perspective, the CCIE is simply one piece of data in the overall picture of the candidate.  It tells you something, something important, but it’s not nearly enough to be sure you are picking the right person.  If you are hiring for a VAR and just need a number to get your Gold status, then the value of the CCIE jumps up a little higher.  If you are just hiring a network engineer for IT, you need to take a host of other factors into account.

At the end of the day, I hope that people don’t make hiring decisions based on that one criterion.  I had one job where we hired in a CCIE (against my advice) because the hiring manager believed anyone in possession of a CCIE number was a genius.  (See “The CCIE Mystique”, here.)  The guy was a disaster.  Did he pass by cheating, or was he just good at taking tests?  I don’t know.  I’ve also known many, many extremely smart and talented network engineers who never got their CCIE, including several who just couldn’t pass it.  Keep that in mind when you are getting overly impressed with the famous four letters.

SDN and the New World Order

I would like to conclude this post, and this series, with a few thoughts on the relevance of CCIE certification in the future.  Various people in the industry, most of whom hold MBA’s in finance or marketing, tell us that CLI is dead and that the Ciscos and Junipers will be replaced by cheap, “white-label” hardware.  Google and Facebook are managing thousands of network devices with a handful of staff, using scripting and automation.  A CCIE certification is a waste of time, according to this thinking, because Cisco is spiraling down into oblivion.  Soon, it will all be code.  Learn Python instead.

I’d like to present my thoughts with a caveat.  I’m not always a great technology prognosticator.  When a friend showed me a web browser for the first time in 1994, I told him this Web thing would never take off.  Oops.  However, I have a lot of experience in the industry, and right now my focus at Cisco is programmability and automation.  I think I have a right to an opinion here.

First of all, there is no question that interest in automation and programmability is increasing.  All of the vendors, including my employer, are devoting a lot of resources towards developing and increasing their programmability and automation capabilities.  I’ve spent my first year here at Cisco working on automating network devices with Puppet, Ansible, Python, and our own tools like DCNM.  We’ve been working hard to build out YANG data models for our features.  I barely touched Expect scripting before I came here!  Customers are interested in managing their networks more efficiently, sadly, sometimes with fewer people.

Let me look at this from a wider perspective.  As computing power increases, are humans redundant?  For example, as a pilot I know that it’s entirely possible to replace human pilots with computers.  Air Traffic Control could relay instructions digitally to the computers that now control most airplanes.  With ILS and precision approaches, airplanes could land themselves at most airports.  But would you get into an airplane that had no human pilots?

The problem is, even in the age of Watson, computers react predictably to predictable circumstances, but predictably badly to unpredictable circumstances.  Many of the alleged errors that are introduced with CLI will still be introduced with NETCONF when the operator puts in the wrong data.  Scripts and automation tools have to power to replicate errors across huge numbers of devices faster than CLI.

At the end of the day, you still need to know what it is that you’re automating.  Networks cannot go away.  If all the Cisco and Juniper boxes out there vanished suddenly, our digital world would come to a screeching halt.  We still need people who understand what switches, routers, and firewalls do, regardless of whether they are managing them with CLI or Python.  Look at the cockpit of a modern airplane.  The old dial gauges have been replaced by flat-panel displays.  Do you think pilots no longer need to study weather systems, aerodynamics, and engine operation?  Of course they do!  Just the same, we need network engineers, a lot of them, to make networks work correctly.

As to whether Cisco itself goes away, who knows?  Obviously returning here, I have a great deal of faith in the company and its ability to execute.  I think that a lot of the industry hype is just hype.  Sure, some of the largest network operators will displace our hardware with Open Compute boxes.  Sure, it will hurt us.  But we are still in most networks and I don’t see that changing for a long time.

Closing thoughts, and a postscript

I began this series with an article on what I called, “The CCIE Mystique.”  I don’t know if that quite exists the same way it did in 2001 when I started this journey, but I suspect it is still there a least a little bit.  Those of us who have been through it have had the veil pulled back a bit, and we see it for what it is:  a very hard, but ultimately very surmountable challenge.  A test that measures some, but not even close to all of the skill required to be a network engineer.  A ticket to a better career, but not a guarantee of one.

Being back at Cisco has been an amazing experience.  This is the company that started my career, and that has built an entire industry.  I’d be happy to finish things out here, but I have no idea in this age of mass layoffs whether that will be possible.

I had an interesting chat with the head of the CCIE route/switch lab curriculum not long ago.  I must say, that for whatever complaints I have about the program, I was quite impressed with him and his awareness of where the program is strong and where it is lacking.  I have no doubt that those guys work very hard on the CCIE program and it’s always easier to sit on the sidelines and complain.  Ten years ago, when I started this journey, I wouldn’t have believed I would be at Cisco, helping to develop and market products, with a lab stocked full of gear, chatting with a CCIE proctor.  Everyone’s path is different, and I certainly cannot promise my readers that investing in one certification will land them a pot of gold.  There were a lot of other factors in my career trajectory.  But at the end of a decade (actually 12 years as I write this), I can say that the time, money, and effort spent in pursuit of the elusive CCIE was well worth it.

As I close out this series I hope that my little autobiographical exclusion was not too self-indulgent.  I hope that those of you who are new to the industry or studying for the exam got some insight from the articles.  I hope that any other old CCIE’s who stopped by relived a few memories.  Thanks for reading the series, and good luck on your studies!

3

Why certifications can suck

Posted by ccie14023 on October 28, 2016
Posted in: Uncategorized. Tagged: , , , , . 1 comment

Ivan Peplnjak referenced a piece by Robert Graham called “Why cybersecurity certifications suck,” which I found amusing considering that I am doing some question writing right now.  I’ve certainly been a member of the chorus of complainers in the past, in various venues, but I have to say that my experience on the writing end of things has changed my perspective a bit.  I’m currently working on CCIE R/S questions, and previously worked on Juniper certifications.  Graham references a question which is pure trivia, and then attributes the problem to question writers who are “only one short step ahead of their students” because they “lack experience and deep knowledge.”

Writing and Writers

I can’t speak for every exam, but these statements are definitely incorrect for the Cisco and Juniper examinations I’ve worked on.  In both cases, I was required to prove myself to be a “subject matter expert” based on both credentials and work experience.  Obviously this does not mean I am a SME on every single topic on the exam, but in both cases I was also given the choice on which topics to write on.  I was never forced into writing questions on a topic that wasn’t familiar.

Writing exam questions is not easy.  First, let’s consider the motivation of the question writer.  Some are professionals in the training department of the company or organization whose exam they are writing for.  These people often produce high quality questions because they have some formal training in the subject and have done it a lot.  They are writing the questions simply because their job requires it of them.

Others, like myself, have a day job.  We do the question writing on the side.  We are given some training and extensive guidelines on how to write questions.  Usually there is some extra incentive for us.  At both Juniper and Cisco, I was able to recertify my existing certifications by writing questions, thus avoiding the dreaded recert exam.  Folks like us pose two problems:  (1) We are not experts at writing questions and (2) we have a motivation to get as many questions out the door as possible.  Given number 2, there can be an overwhelming desire to pick a topic, go dig through the relevant doc until you find footnote 7 on page 135, and drop a quick trivia question.

The hardest questions to write are ones that test thinking and not mere trivia.  I have spent hours on questions of that type.  They are my favorite to write, but often require drawing a diagram, setting up the scenario in the lab, collecting outputs, and pulling it all together into a question with right and wrong answers.  Given the difficulty, you can see why many writers skip right to the easy trivia, especially if they are in it to recertify.

Most exams are in multiple choice format these days, although there are several that use simulators and fill-in-the-blank style questions.  This is just a limitation of administering so many exams to so many people.  I wish we could do essay questions on the CCIE written, but it’s not happening.

Wrong Answers

Interestingly enough, the hardest part about writing multiple choice questions is coming up with the wrong answers.  That’s because they need to be really wrong, but also plausible.  Really wrong in that if the question says “pick the correct 2 out of 4”, it’s no good if 3 out of 4 are actually correct.  It’s very easy to accidentally write a wrong answer that is correct.  Plausible because they cannot be so obviously wrong that they give away the correct answer(s).  Let me take an example.

Which two cities are in Asia?  (Choose two.)

A.  Beijing
B.  Golden Retriever
C.  Singapore
D.  Banana

In this case, even if I didn’t know what cities were in Asia, I could easily find the right answer because I know B and D are not cities.  A better question would be:

Which two cities are in Asia?  (Choose two.)

A.  Beijing
B.  Warsaw
C.  Singapore
D.  Lisbon

However, the closer you make the wrong answers to reality, the more likely you will make one that is accidentally correct!

Writing Questions

So how do I write questions?  Usually I go through the topics assigned, and pick one that I am comfortable with.  Then I read through books, configuration guides, and other material on that subject until I think of a suitable question.  This could be factual, or it could be more of a scenario.  I generally write the question and then the right answer(s) and start working on the wrong answers.  After I’ve written it, I review it several times.  Every exam I have worked on required a reference document, so I always make sure to keep track of where I got the idea.  I try to put myself in the perspective of the examinee.  If I were studying this blueprint, where would I look for information?  How far into the document would I read?  At what point would I consider myself to have mastered the subject, and what knowledge would I have then?

Cheaters

It is very important for vendors such as Cisco to maintain the integrity of their exams.  Unfortunately, there is an army of professional cheaters who steal questions and post them verbatim on the Internet.  There are various techniques a vendor can use to combat this, but one of the best is simply to make the question pool so large that nobody can memorize it all.  This in turn puts relentless pressure on vendors to write as many questions as possible.  If you’ve ever used a PDF copy of a test to study, not only are you cheating yourself, you are also putting pressure on us to make more and more questions, which can decrease the question pool quality.

Amusingly enough, when I took my private pilot written exam back in 2005, we studied using the famous red Gleim book, which (legally) had all the questions and answers on the test.  When I took the test, I brought in my calculator and flight computer and never used them, because I had already seen the questions so many times!  I would just look at the question and think, “OK, the answer there is a heading of 270 degrees.”  The only reason I got less than one hundred percent was because I intentionally answered a few wrong, on the advice of my instructor.  Apparently if you show up to the oral portion of the flight exam with a 100% written score, the examiner might think you were arrogant and try to put you in your place with challenging questions.

 

I hope this article provided a little insight into why certification exams sometimes have bad questions.  Please know my intention is not to defend bad questions on a test.  Questions that are poorly written and which make it through Q/A are unfair to test takers and cause unneeded stress and even failures.  I have been on the other side many times, and have experienced the frustration of wrong or unfair exams.  As far as Cisco goes, a number of recent blogs have complained about the quality of the CCIE R/S question pool.  Having worked with the people responsible for it, I can say I have a lot of confidence in them and I think they are doing a good job administering it.

**  NOTE:  I will not answer any specific questions about the exam content I have written, so don’t even try!