ccie

All posts tagged ccie

The last Cisco Live I attended was in Barcelona in January 2020.  As I was in the airport heading home, I was reading news of a new virus emerging from China.  I looked with bemusement at a troop of high-school-age girls who all had surgical masks on.  Various authorities told us not to wear masks, saying they don’t do much to prevent viral spread at a large scale.  The girls kept pulling the masks on and off.  I thought back on my performance at Cisco Live, and looked forward to Cisco Live in Las Vegas in the summer.  Who knew that, a few months hence, everyone would be wearing masks and Cisco Live,  physically, would be indefinitely postponed.

For Technical Marketing Engineers (TMEs), Cisco Live (technically Cisco Live!) measures the seasons of our year like the crop cycle measures the seasons of a farmer’s year.  Four times annually a large portion of our team would hop on an airplane and depart for Europe, Cancun, Melbourne, or a US city.  Cancun and Melbourne were constant, but the European and US cities would change every couple of years.  In my time with Cisco, I have traveled to Cancun and Melbourne, Berlin, Barcelona, Las Vegas, Orlando, and San Diego to present and staff Cisco Live.

A trade show may just be a corporate event, but for those of us who devoted our career to that corporation’s products, it’s far more than a chance for a company to hawk its products.  The breakout sessions and labs are critical for staying up-to-date on a fast-moving industry, the keynotes are always too high-level but with entertaining productions, and the parties are a great chance to connect with other network engineers.  CL was fun for participants, exhausting for those of us staffing it, but still my favorite part of the job.

Cisco Live was originally called Networkers, and started in 1990.  For many years I badly wanted to go to this temporary Mecca of networking technology, but I worked for companies that would not pay the cost of a badge and the travel fees, a total of thousands of dollars.  Even when I first worked at Cisco, from 2005-2007, as a lowly TAC engineer I never had the opportunity to attend.  My first trip to CL came in 2007, when I was working for a Gold partner.  They sent several of us to the Anaheim show, and I remember well the thrill of walking into a CL for the first time.  I walked the show floor, talked to the booth staffers, and attended a lot of breakout sessions of varying quality.  I was quite excited to go to the CCIE party, but I’m not sure why I thought a party full of CCIEs would really be all that exciting.  I remember hanging out by myself for an hour or so before I gave up because I didn’t know anyone there.

The same partner sent me to Orlando in 2008 as well, just barely.  The recession was starting and we were short on cash.  My boss wanted me to share a badge with a colleague, and I didn’t like the idea of having to juggle time slots nor or trying to explain to security how my name could be “Nguyen”.  Thankfully, they ponied up the cash for a second badge.  I’m not a fan of loud music, so I generally don’t go to the CL party, but for Orlando they opened up Universal Studios for us and the aforementioned Nguyen and I, along with a couple others, had a great time on the rides and attending the Blue Man Group.  (OK, some loud music there, but it is an entertaining show.)

I attended CL once more before I came back to Cisco–in 2014, ironically, as an employee of Juniper.  Somehow I convinced my boss to give me a pass on the grounds of researching what Cisco IT was doing.  (They do present at Cisco Live.)  I remember sitting in just a few rows back at the keynote as John Chambers presented, amused I’d be bringing a report back to Juniper about what I’d heard.

 

My view of Chambers at CL 2014

It was actually at Cisco Live when I first got the idea to be a technical marketing engineer.  It’s a bit embarrassing, but I sat in a presentation given by a TME and thought, “I could do better than this guy.”  It took a few years, but I finally managed to get into tech marketing.

I became a Principal TME at Cisco in late 2015 and was told I’d be presenting at Cisco Live in Berlin in January, 2016!  Needless to say, I was thrilled to be given the opportunity, humbled, and more than a little nervous about standing up in front of an audience at the fabled event.

It’s been a sad year in so many ways.  After I came home from Barcelona in January 2020, I received another Distinguished Speaker award and knew I would be inducted into the Hall of Fame.  This was a dream of mine for years, but instead of standing up in front of my peers at Cisco Live Vegas to receive the award, it was mailed to me.  There would be no show floor, no breakouts, no CCIE parties.  The event would go virtual.  I must say, I am impressed with the CL team’s ability to pivot to a virtual format in so short a time.  Still, it was a sad year for those of us who organize the event, and those who were hoping to attend.

In the next couple posts, I thought I would offer a little behind-the-scenes look at how we put on CL, and look at a few events from the past.

There were quite a few big announcements at Cisco Live this year.  One of the big ones was the overhaul of the certification program.  A number of new certifications were introduced (such as the DevNet CCNA/CCNP), and the existing ones were overhauled.  I wanted to do a post about this because I was involved with the certification program for quite a while on launching these.  I’m posting this on my personal blog, so my thoughts here are, of course, personal and not official.

First, the history.  Back when I was at Juniper, I had the opportunity to write questions for the service provider written exams.  It was a great experience, and I got thorough training from the cert program on how to properly write exam questions.  I don’t really remember how I got invited to do it, but it was a good opportunity, as a certified (certifiable?) individual, to give back to the program.  When I came to Cisco, I quickly connected with the cert program here, offering my services as a question writer. I had the training from Juniper, and was an active CCIE working on programmability.  It was a perfect fit, and a nice chance to recertify without taking the test, as writing/reviewing questions gets your CCIE renewed.

As I was managing a team within the business unit that was working on Software-Defined Access and programmability, it seemed logical for me to talk to the program about including those topics on the test.  I can assure you there was a lot of internal debate about this, as the CCIE exam is notoriously complex, and the point of our Intent-Based Networking products is simplicity.  One product manager even suggested a separate CCIE track for SD-Access, an idea I rejected immediately for that very reason.

Still, as I often point out here and elsewhere, SDN technologies do not mitigate the need for network engineers.  SDN products, all SDN products, are complex precisely because they are automated.  Automation enables us to build more complex things, in general.  You wouldn’t want to configure all the components of SD-Access by hand.  Still, we need engineers who understand what the automation tools are doing, and how to work with all the components which comprise a complex solution like SD-Access.  Network engineers aren’t going to disappear.

For this reason, we wanted SD-Access, SDWAN, and also device programmability (NETCONF/YANG, for example) to be on the lab.  We want to have engineers who know and understand these technologies, and the certification program is a fantastic way to help people to learn them.  I, and some members of my team, spent several months working with the CCIE program to build a new blueprint, which became the CCIE Enterprise Infrastructure.  The storied CCIE Routing and Switching will be no more.

At the end of the day, the CCIE exam has always adapted to changed in networking.  The R/S exam no longer has ISDN or IPX on it, nor should it.  Customers are looking for more automated solutions, and the exam is keeping pace.  If you’re studying for this exam, the new blueprint may be intimidating.  That said, CCIE exams have always been intimidating.  But think about this:  if you pass this exam, your resume will have skills on it that will make you incredibly marketable.

The new CCIE-EI (we always abbreviate stuff, right?) breaks down like this:

  • 60% is classic networking, the core routing protocols we all know and love.
  • 25% is SDx:  SD-Access and SD-WAN, primarily.
  • 15% is programmability.  NETCONF/YANG, controller APIs, Ansible, etc.

How do you study for this?  Like you study for anything.  Read about it and lab it.  There is quite a bit of material out there on all these subjects, but let me make some suggestions:

Programmability

You are not expected to be a programming expert for this section of the exam.  It’s not about seeing if you can write complex programs, but whether you know the basics well enough to execute some tasks via script/Ansible/etc instead of CLI.  DevNet is replete with examples of how to send NETCONF messages, or read data off a router or switch with programmable interfaces.  Download them, play with them, spend some time learning the fundamentals of Python, and relax about it.

  • Learn:  DevNet is a phenomenal resource.  Hank Preston, an evangelist for DevNet, has put out a wealth of material on programmability.  In addition, there is the book on IOS XE programmability I wrote with some colleagues.
  • Lab:  You can lab programmability stuff easily on your laptop.  Python and ncclient are free, as is Ansible.  If you have any sort of lab setup already, all you need to do is set up a Linux VM or install some tools on to your laptop.

Software-Defined

This is, as I said before, a tough one to test on.  After all, to add a device to an SD-Access fabric, you select it and click “Add to Fabric.”  What’s there to test?  Well, since these are new products you of course need to understand the components of SD-Access/SDWAN and how they interoperate.  How does policy work?  How do fabric domains talk to non-fabric domains?  There is plenty to study here.

  • Learn:  Again, we’ve written books on SD-Access and SD-WAN.  Also, we are moving a lot of documentation into Cisco Communities.
  • Lab:  Well, this is harder.  We’re working on getting SD-Access into the hands of learning partners, so you’ll have a place to get your hands on it.  We’re also working on virtualizing SD-Access as much as possible, to make it easier for people to run in labs.  I don’t have a timeframe on the latter, but hopefully the former we can do soon.

These are huge but exciting changes. I’ve been very lucky to have landed at a job where I am at the forefront of the changes in the industry, but this new exam will give others the opportunity to move themselves in that direction as well.  Happy labbing!

I think it’s fair to say that all technical marketing engineers are excited for Cisco Live, and happy when it’s over.  Cisco Live is always a lot of fun–I heard one person say “it’s like a family reunion except I like everyone!”  It’s a great chance to see a lot of folks you don’t get to see very often, to discuss technology that you’re passionate about with other like minded people, to see and learn new things, and, for us TMEs, an opportunity to get up in front of a room full of hundreds of people and teach them something.  We all now wait anxiously for our scores, which are used to judge how well we did, and even whether we get invited back.

It always amazes me that it comes together at all.  In my last post, I mentioned all the work we do to pull together our sessions.  A lot of my TMEs did not do sessions, instead spending their Cisco Live on their feet at demo booths.  I’m also always amazed that World of Solutions comes together at all.  Here is a shot of what it looked like at 5:30 PM the night before it opened (at 10 AM.)  How the staff managed to clear out the garbage and get the booths together in that time I can’t imagine, but they did.

The WoS mess…

My boss, Carl Solder, got to do a demo in the main keynote.  There were something like 20,000 people in the room and the CEO was sitting there.  I think I would have been nervous, but Carl is ever-smooth and managed it without looking the least bit uncomfortable.

My boss (left) on the big stage!

The CCIE party was at the air and space museum, a great location for aviation lovers such as myself.  A highlight was seeing an actual Apollo capsule.  It seemed a lot smaller than I would have imagined.  I don’t think I would ever have gotten in that thing to go to the moon.  The party was also a great chance to see some of the legends of the CCIE program, such as Bruce Caslow, who wrote the fist major book on passing the CCIE exam, and Terry Slattery, the first person to actually pass it.

CCIE Party

I delivered two breakouts this year:  The CCIE in an SDN World, and Scripting the Catalyst.  The first one was a lot of fun because it was on Monday and the crowd was rowdy, but also because the changes to the program were just announced and folks were interested in knowing what was going on.  The second session was a bit more focused and deeper, but the audience was attentive and seemed to like it.  If you want to know what it feels like to be a Cisco Live presenter, see the photo below.

My view from the stage

I closed out my week with another interview with David Bombal, as well as the famous Network Chuck.  This was my first time meeting Chuck, who is a bit of a celebrity around Cisco Live and stands out because of his beard.  David and I had already done a two-part interview (part 1, part 2) when he was in San Jose visiting Cisco a couple months back.  We had a good chat about what is going on with the CCIE, and it should be out soon.

As I said, we love CL but we’re happy when it’s over.  This will be the first weekend in a long time I haven’t worked on CL slides.  I can relax, and then…Cisco Live Barcelona!

I’ve wanted to kick off a series for a while now on technical interviewing. Let me begin with a story.

My first job interview for a full network engineering role was at the San Francisco Chronicle in 2000. I had been working for five years in IT, mostly doing desktop and end-user support. I then decided to get a master’s degree in telecommunications management, which didn’t help at all, followed by a CCNA certification, which got me the interview.

My first interview was with the man who would be my boss. Henry was a manager who had almost no technical knowledge about networking, but I didn’t know that at the time. “Do you know Foundry switches at all?” Henry asked.

“No.” I was already worried.

“I doubted you would. That’s ok because we want to replace them all with Cisco and you know Cisco.” He pulled out a network diagram and handed it to me. “If you look at this, do you see a problem?” he asked.

I had never worked on a network larger than a couple switches, and now I was staring at a convoluted diagram depicting the network of the largest newspaper in Northern California. I was looking at subnet masks, link speeds, and hostnames, trying to find something wrong.

“I’m not sure,” I had to reply meekly.

He pointed at the main core switch for the network. There was only one, with no redundancy.  “There’s a huge single point of failure,” he said. I felt stupid missing the forest for the trees.

Henry brought me upstairs to interview with Tom, who was an on-site project management contractor from Lucent. I was extremely nervous–Lucent (later Avaya) was a big name in the industry and this guy worked for them! Henry left me with Tom. Tom pulled out a copy of the same diagram Henry was showing me earlier.

“Do you notice anything wrong with this?” he asked.

“Wow, that’s a huge single point of failure,” I replied.

He nodded his head in approval. “That’s right–very good.” He asked me a technical question about supernetting. I answered nervously, although it quickly became clear I knew more than he did.

The door flew open and another guy named Vincent walked in. He was the desktop support contractor, but again I didn’t know that. “Ask Jeff a few technical questions,” Tom said.

“Question number one,” said Vincent. “If you were running a network this size, would you subnet it?”

Now the answer seemed obviously to be “yes”, but I was trying to figure out if this is a trick. “Yes,” I answered, deciding to play it safe.

“Good! Next question: Can you route NetBios?” My desktop years were almost exclusively dedicated to Macs and I didn’t even know what NetBios was. I figured it was a 50/50 shot, and the way he asked it seemed to suggest the answer.

“No,” I said, trying to sound confident.

“He’s good,” said Vincent.

Next, the door flew open again, and in walked Bing. Bing was carrying some sort of network device with her. She handed it to me. “Is this a switch or a hub?” she asked. There was no obvious labeling on it, and as I turned the device over and over again in my hands, I had a sinking feeling.

“I don’t know,” I replied.

“Look at this,” Bing said. She pointed to a collision light. “Since there is one on each port, you can tell this is a switch.”

We don’t have collision lights on switches any more, but at the time we did and she had a valid point. Realizing this, I explained to her that a since a hub has a single collision domain, it would only have one collision light. I explained to her the concept of a collision domain, and how a switch worked versus a hub. It turns out she was a project manager for desktop support and she didn’t know any of that. Someone had just shown her the collision light thing and she thought it would be a good question.

“He’s good,” said Bing.

Tom had told me my next interview would be with a CCIE from Lucent. Now that was definitely intimidating. I knew of the reputation of CCIEs, and I didn’t expect to do well. The CCIE guy never showed up. As Tom was walking me to the elevator, however, we ran into him in the hallway. It turns out that Mike, who is still a friend of mine, and who later got three CCIE’s, had not passed the exam yet. We ended up talking about his home lab for a few minutes.

“He’s good,” said Mike. And a good thing too, as I’ve been in a couple of interviews with Mike and I’ve seen him grill people mercilessly.

I got a call with a job offer a few days later, and ended up working there five years.

For a while now I’ve had several posts in my drafts folder on the subject of technical interviewing. As you can see from the above story, interviews are often chaotic, disorganized, and conducted by unqualified people who have no plan. In the case of the San Francisco Chronicle, they made the right decision on me, and I don’t think anybody there would dispute that. I was thankful to begin my career in network engineering.

That said, I’ve had other interviews that didn’t go so well. Over the next few articles, I’d like to cover technical interviewing. Why do we interview people? How can we select good people from bad people? How worthwhile are the typical technical questions? Are gotchas worth throwing out “just to see how the candidate reacts”? Are interviews purely subjective or can we make them data-driven and objective?

I’ll throw out a few more anecdotes from my own experience to illustrate my points–feel free to comment with some of your own!

Two years ago I published my Ten Years a CCIE series.  Actually, I had written the series a couple years before I published it, but as I say in my introduction to the series, I felt it was a bit self-indulgent an uninteresting, so I scrapped it for a while.  The original pieces were dictated, and I’ve been meaning to go back and clean up some of the grammatical errors or grating phrases, but haven’t had the time.  Not a lot of people have read it, nor did I expect many to read it, since I generally don’t advertise the blog in social media, or anywhere really.  But the feedback from the few who have read it has been positive, and I’m gratified for that.

Things have changed a lot since I got into networking in 1995, and since I passed my CCIE in 2004.  But it’s also amazing how much has stayed the same.  TCP/IP, and in fact IPv4, is still the heart of the network.  Knowledge of OSPF and BGP is still key.  For the most part, new controllers and programmable interfaces represent a different way of managing fundamentally the same thing.

The obvious reasons for this are that networks work and are hard to change.  The old protocols have been sufficient for passing data from point A to point B for a long time.  They’re not perfect but they are more than adequate.  They are hard to change because networks are heterogeneous.  There are so many types of different systems connecting to them, that if we wanted to fundamentally alter the building blocks of networks, we’d have to upgrade a lot of systems.  This is why IPv6 adoption is so slow.

Occasionally I poke around at TechExams.net to see what newer network engineers are thinking, and where they are struggling.  I’m probably the only director-level employee of Cisco who reads or comments on that message board.  I started reading it back when I was still at Juniper and studying for my JNCIE, but I’ve continued to read it because I like the insights I get from folks prepping for their certifications.  People are occasionally concerned that the new world of controllers and automation will make their jobs obsolete.

I built the first part of my career on CLI.  Now I’m building it on controllers and programmability.  In this industry, we have to adapt, but we don’t have to die.  Cars have changed drastically, with on-board computer systems and so forth, but we still need mechanics.  We still need good network engineers.

To be honest, I was getting tired of my career by the time I left Juniper and came to Cisco.  I was bored.  I thought of going back to school and getting a Ph.D. in classical languages, my other passion.  Getting married helped put an end to that idea (Ph.D.’s in ancient Greek make a lot less than network engineers) but when I came back to Cisco, I felt revitalized.  I started learning new things.  Networking was becoming fun again.

I wrote the “Ten Years a CCIE” series both for people who had passed the exam and wanted to have some fun remembering the experience, as well as for people struggling to pass it.  Some things change, as I said, but a lot remains the same.  I still think, closing in on 15 years since I took the exam, that it’s still worth it.  I still think it’s a fantastic way to launch a career.  The exam curriculum will adapt, as it always does, with new technologies, but it’s an amazing learning experience if you do it honestly, and you will be needed when you make it through.

In the final post in my “Ten Years a CCIE” series, I take a look at the age-old question: Is a CCIE really worth it? I conclude the series with some thoughts on the value of this journey.

I’ve written this series of posts in the hope that others considering the pursuit of a CCIE  would have some idea of the process, the struggles, and the reward of passing this notorious exam.  I would like to finish this series with a final article examining the age-old question:  What is the value of a CCIE?  In other words, was it worth it?  All of the other articles in the series were written a couple of years ago, back when I worked for Juniper in IT, with only some slight revisions before publishing.  However, this article is being written now (late 2016), in very different circumstances.  I work at Cisco now, not Juniper.  I work in the switching business unit, not in IT.  I work in product management, and as a Principal Engineer have direct influence on the direction of our products.  I specifically work on programmability, automation, and SDN solutions.  As I write this, I see many potential CCIE candidates wondering if it is even worth pursuing a career in networking any more.  After all, won’t SDN and automation just eliminate their jobs? Wouldn’t they be better off learning Python?  Isn’t Cisco a company in its death throes, facing extinction at the hands of upstarts like Arista and Viptela?

Well, I can’t predict the future.  But I can look at the issues from the perspective of someone who has been in the industry a long time now, and at least set your mind at ease.  The short answer:  it’s worth it.  The longer answer requires us to examine the question from a few different angles.

Technical Knowledge

As I’ve pointed out in previous articles, during the CCIE preparation process, you will master a vast amount of material, if you approach the test honestly.  But, as anyone in this industry knows, the second you pass, a timer starts counting down the value of the material you learned.  On my R&S exam (2004), I studied DLSw+, ATM, ISDN, and Frame Relay.  On my Security (2008) exam, I studied the VPN 3k concentrator, PIX, and NAC Framework.  How much of that is valuable now?

True, I studied many things that are still in use today.  OSPF, BGP, EIGRP, and ISIS were all heavily tested on the R&S exam even back in 2004.  But how much of it do I remember?  I certainly have a good knowledge of each of them, but even after taking the JNCIE-SP exam less than two years ago, I find my knowledge of the details of routing protocols fading away.  If I were to take the JNCIE exam today, I would certainly fail.

Thus, some technologies I learned are obsolete, and some are not, but I have forgotten a lot.  However, there are also technologies that I never learned.  For example, ISE and Cisco TrustSec are huge topics that I am just starting to play with.  Despite my newness to these technologies, I do have the same right to call myself a “Security CCIE” as a guy who passed it yesterday, and knows those subjects cold!

So, then, what about the value of the technical knowledge?  Is it worth it?

First, it is because no matter what becomes obsolete, and no matter what you forget, the intensity of your study will guarantee you still have a core of knowledge that will be with you for a long time.  Fortunately, despite the rapidity of change our industry is supposedly undergoing, networking is conservative by nature.  The Internet is a large distributed system consisting of many systems running disparate operating systems and they have to work together.  Just ask the guys pushing IPv6.  The core of networking doesn’t change very much.  You will also learn that even new technologies, like VXLAN, build on concepts you already know.

Second, even obsolete knowledge is valuable.  Newer engineers often don’t realize how certain technologies developed, or why we do things the way that we do.  They don’t know the things that have been tried and which failed in the past.  Despite the premium our industry places on youth, old-timers do have important insight gleaned from playing with things like, say, DLSw+.  It’s important for us not to simply wax nostalgic (as I am here) about the glorious days of IPX, but to explain to younger engineers how these dinosaurs actually worked, so that they can understand what decisions protocol designers made and why, and why some technologies fell into disuse or were replaced.  And, millennials, it is your responsibility to sit up and listen, if not to seek out such information.

Technical “soft skills”

In addition to specific technical knowledge, CCIE preparation simply makes you better at configuring and troubleshooting in general.  The hours of frustration in the lab and the broken configs that have you pulling your hair out ultimately help you to learn what makes a network break, and how to go about systematically debugging it.  Certainly you can learn those skills elsewhere, but when you are faced with the challenge of building a lab in a short amount of time, you have to be able to think quickly on your feet.  Even troubleshooting a live network outage doesn’t quite compare the intensity of the CCIE crucible, the eight hour slog during which one mistake can cost you months of studying.  The exams where I took multiple attempts required me to seriously up my game between tries, and each time I’ve come out a stronger thinker.

Non-Technical Skills

Passing the CCIE exam honestly is a challenge, plain and simple.  It requires discipline, persistence, extreme attention to detail, resourcefulness, ability to read documentation carefully, and time management skills.  While you won’t pass the CCIE without having those skills to begin with, there is no question that the rigors of CCIE preparation will help to hone them.  You will be a better person all around if you submit yourself to the discipline of passing the exam honestly.

It’s a bit like studying the martial arts.  Most martial artists will tell you that, in addition to being able to deliver a mean axe kick to the head, they have achieved self-discipline and confidence from their practice of martial arts, and that they find these skills apply elsewhere in life.  It’s true of any rigorous pursuit, really, and definitely true of the CCIE.  Personally I have no doubt that the many hours of laborious study have made me a more detail-oriented and better engineer.

Employment value:  Employee’s perspective

In my own case, my CCIE certification was directly responsible for my getting hired at Cisco TAC in 2005.  It was absolutely essential for me to move to a VAR in 2007.  Believe it or not, it was critical in my getting a job at Juniper in 2009, and having two CCIE’s and a JNCIE helped open the door to be re-hired at Cisco in 2015.  Obviously, employers see value in it.

You still see CCIE certification listed as a requirement in many job descriptions.  Without one you are cut out of those positions.  Many employers see it as a key differentiator and qualification for a senior position in network engineering.  VARs are still required to have a number of CCIEs on staff, so for some positions they will only talk to someone who has a CCIE.

Rarely is it the only qualification, however, and rarely is it enough, on its own, to get you hired.  Shortly after I passed my second CCIE I got laid off from the VAR where I worked.  I ended up interviewing at Nexus IS (now Dimension Data), another VAR, a couple months later.  Having been out of work I was rusty, and the technical interviewer grilled me mercilessly.  I’ve stated in the past that I don’t find this sort of thing productive, and I was uncomfortable, and didn’t have a great performance.  Despite the fact that I had two CCIE’s, and despite the value of such credentials for a reseller, I got a call from HR in a few days telling me that they chose not to hire me.  (These things often work out in the end;  I am in a good place now, and I wouldn’t have made it here if I had stayed in the VAR world.)  The entire experience reinforces a point I made in my Cheaters post:  don’t think an ugly plaque and a number are a guarantee of employment.

That said, there is little doubt that a person with X skill-set and a CCIE is in a better place than a person with just X skill-set.  As I said above, the certification simply opens doors.  You cannot rely on it alone, but combined with good experience, the certification is invaluable.  I would never have made it this far without one.  Now it’s true that there are some very talented and knowledgeable engineers who don’t have, and in some cases disdain, the CCIE.  Many of these folks do quite well and have successful careers.  But again, add the famous number and you will always do better in the end.

Employment value:  Employer’s perspective

Let’s turn it around now, and look at it from the perspective of a hiring manager.  When I have the stack of resumes in front of me, how important is it for me to look for that famous CCIE number?

This is a much harder question to answer, and I’ve touched on some of the problems of the CCIE above.  If you are looking at somebody who has a CCIE #1xxx, you know you are getting someone with age and experience.  But if you’re looking for someone to do hands-on implementation work, will this be the right candidate?  Maybe, of course, if Mr. #1xxx has been doing a lot of that lately.  The point is, you can’t tell from the number alone.

Now let’s say you are looking at CCIE #50xxx.  Does her CCIE number mean she will be the perfect fit for the hands-on job?  Quite possibly, because she has a lot of recent hands-on experience.  Will she be unqualified for the management role because she lacks the experience of CCIE #1xxx?  Who knows?

The point is, from a hiring manager’s perspective, the CCIE is simply one piece of data in the overall picture of the candidate.  It tells you something, something important, but it’s not nearly enough to be sure you are picking the right person.  If you are hiring for a VAR and just need a number to get your Gold status, then the value of the CCIE jumps up a little higher.  If you are just hiring a network engineer for IT, you need to take a host of other factors into account.

At the end of the day, I hope that people don’t make hiring decisions based on that one criterion.  I had one job where we hired in a CCIE (against my advice) because the hiring manager believed anyone in possession of a CCIE number was a genius.  (See “The CCIE Mystique”, here.)  The guy was a disaster.  Did he pass by cheating, or was he just good at taking tests?  I don’t know.  I’ve also known many, many extremely smart and talented network engineers who never got their CCIE, including several who just couldn’t pass it.  Keep that in mind when you are getting overly impressed with the famous four letters.

SDN and the New World Order

I would like to conclude this post, and this series, with a few thoughts on the relevance of CCIE certification in the future.  Various people in the industry, most of whom hold MBA’s in finance or marketing, tell us that CLI is dead and that the Ciscos and Junipers will be replaced by cheap, “white-label” hardware.  Google and Facebook are managing thousands of network devices with a handful of staff, using scripting and automation.  A CCIE certification is a waste of time, according to this thinking, because Cisco is spiraling down into oblivion.  Soon, it will all be code.  Learn Python instead.

I’d like to present my thoughts with a caveat.  I’m not always a great technology prognosticator.  When a friend showed me a web browser for the first time in 1994, I told him this Web thing would never take off.  Oops.  However, I have a lot of experience in the industry, and right now my focus at Cisco is programmability and automation.  I think I have a right to an opinion here.

First of all, there is no question that interest in automation and programmability is increasing.  All of the vendors, including my employer, are devoting a lot of resources towards developing and increasing their programmability and automation capabilities.  I’ve spent my first year here at Cisco working on automating network devices with Puppet, Ansible, Python, and our own tools like DCNM.  We’ve been working hard to build out YANG data models for our features.  I barely touched Expect scripting before I came here!  Customers are interested in managing their networks more efficiently, sadly, sometimes with fewer people.

Let me look at this from a wider perspective.  As computing power increases, are humans redundant?  For example, as a pilot I know that it’s entirely possible to replace human pilots with computers.  Air Traffic Control could relay instructions digitally to the computers that now control most airplanes.  With ILS and precision approaches, airplanes could land themselves at most airports.  But would you get into an airplane that had no human pilots?

The problem is, even in the age of Watson, computers react predictably to predictable circumstances, but predictably badly to unpredictable circumstances.  Many of the alleged errors that are introduced with CLI will still be introduced with NETCONF when the operator puts in the wrong data.  Scripts and automation tools have to power to replicate errors across huge numbers of devices faster than CLI.

At the end of the day, you still need to know what it is that you’re automating.  Networks cannot go away.  If all the Cisco and Juniper boxes out there vanished suddenly, our digital world would come to a screeching halt.  We still need people who understand what switches, routers, and firewalls do, regardless of whether they are managing them with CLI or Python.  Look at the cockpit of a modern airplane.  The old dial gauges have been replaced by flat-panel displays.  Do you think pilots no longer need to study weather systems, aerodynamics, and engine operation?  Of course they do!  Just the same, we need network engineers, a lot of them, to make networks work correctly.

As to whether Cisco itself goes away, who knows?  Obviously returning here, I have a great deal of faith in the company and its ability to execute.  I think that a lot of the industry hype is just hype.  Sure, some of the largest network operators will displace our hardware with Open Compute boxes.  Sure, it will hurt us.  But we are still in most networks and I don’t see that changing for a long time.

Closing thoughts, and a postscript

I began this series with an article on what I called, “The CCIE Mystique.”  I don’t know if that quite exists the same way it did in 2001 when I started this journey, but I suspect it is still there a least a little bit.  Those of us who have been through it have had the veil pulled back a bit, and we see it for what it is:  a very hard, but ultimately very surmountable challenge.  A test that measures some, but not even close to all of the skill required to be a network engineer.  A ticket to a better career, but not a guarantee of one.

Being back at Cisco has been an amazing experience.  This is the company that started my career, and that has built an entire industry.  I’d be happy to finish things out here, but I have no idea in this age of mass layoffs whether that will be possible.

I had an interesting chat with the head of the CCIE route/switch lab curriculum not long ago.  I must say, that for whatever complaints I have about the program, I was quite impressed with him and his awareness of where the program is strong and where it is lacking.  I have no doubt that those guys work very hard on the CCIE program and it’s always easier to sit on the sidelines and complain.  Ten years ago, when I started this journey, I wouldn’t have believed I would be at Cisco, helping to develop and market products, with a lab stocked full of gear, chatting with a CCIE proctor.  Everyone’s path is different, and I certainly cannot promise my readers that investing in one certification will land them a pot of gold.  There were a lot of other factors in my career trajectory.  But at the end of a decade (actually 12 years as I write this), I can say that the time, money, and effort spent in pursuit of the elusive CCIE was well worth it.

As I close out this series I hope that my little autobiographical exclusion was not too self-indulgent.  I hope that those of you who are new to the industry or studying for the exam got some insight from the articles.  I hope that any other old CCIE’s who stopped by relived a few memories.  Thanks for reading the series, and good luck on your studies!

Ivan Peplnjak referenced a piece by Robert Graham called “Why cybersecurity certifications suck,” which I found amusing considering that I am doing some question writing right now.  I’ve certainly been a member of the chorus of complainers in the past, in various venues, but I have to say that my experience on the writing end of things has changed my perspective a bit.  I’m currently working on CCIE R/S questions, and previously worked on Juniper certifications.  Graham references a question which is pure trivia, and then attributes the problem to question writers who are “only one short step ahead of their students” because they “lack experience and deep knowledge.”

Writing and Writers

I can’t speak for every exam, but these statements are definitely incorrect for the Cisco and Juniper examinations I’ve worked on.  In both cases, I was required to prove myself to be a “subject matter expert” based on both credentials and work experience.  Obviously this does not mean I am a SME on every single topic on the exam, but in both cases I was also given the choice on which topics to write on.  I was never forced into writing questions on a topic that wasn’t familiar.

Writing exam questions is not easy.  First, let’s consider the motivation of the question writer.  Some are professionals in the training department of the company or organization whose exam they are writing for.  These people often produce high quality questions because they have some formal training in the subject and have done it a lot.  They are writing the questions simply because their job requires it of them.

Others, like myself, have a day job.  We do the question writing on the side.  We are given some training and extensive guidelines on how to write questions.  Usually there is some extra incentive for us.  At both Juniper and Cisco, I was able to recertify my existing certifications by writing questions, thus avoiding the dreaded recert exam.  Folks like us pose two problems:  (1) We are not experts at writing questions and (2) we have a motivation to get as many questions out the door as possible.  Given number 2, there can be an overwhelming desire to pick a topic, go dig through the relevant doc until you find footnote 7 on page 135, and drop a quick trivia question.

The hardest questions to write are ones that test thinking and not mere trivia.  I have spent hours on questions of that type.  They are my favorite to write, but often require drawing a diagram, setting up the scenario in the lab, collecting outputs, and pulling it all together into a question with right and wrong answers.  Given the difficulty, you can see why many writers skip right to the easy trivia, especially if they are in it to recertify.

Most exams are in multiple choice format these days, although there are several that use simulators and fill-in-the-blank style questions.  This is just a limitation of administering so many exams to so many people.  I wish we could do essay questions on the CCIE written, but it’s not happening.

Wrong Answers

Interestingly enough, the hardest part about writing multiple choice questions is coming up with the wrong answers.  That’s because they need to be really wrong, but also plausible.  Really wrong in that if the question says “pick the correct 2 out of 4”, it’s no good if 3 out of 4 are actually correct.  It’s very easy to accidentally write a wrong answer that is correct.  Plausible because they cannot be so obviously wrong that they give away the correct answer(s).  Let me take an example.

Which two cities are in Asia?  (Choose two.)

A.  Beijing
B.  Golden Retriever
C.  Singapore
D.  Banana

In this case, even if I didn’t know what cities were in Asia, I could easily find the right answer because I know B and D are not cities.  A better question would be:

Which two cities are in Asia?  (Choose two.)

A.  Beijing
B.  Warsaw
C.  Singapore
D.  Lisbon

However, the closer you make the wrong answers to reality, the more likely you will make one that is accidentally correct!

Writing Questions

So how do I write questions?  Usually I go through the topics assigned, and pick one that I am comfortable with.  Then I read through books, configuration guides, and other material on that subject until I think of a suitable question.  This could be factual, or it could be more of a scenario.  I generally write the question and then the right answer(s) and start working on the wrong answers.  After I’ve written it, I review it several times.  Every exam I have worked on required a reference document, so I always make sure to keep track of where I got the idea.  I try to put myself in the perspective of the examinee.  If I were studying this blueprint, where would I look for information?  How far into the document would I read?  At what point would I consider myself to have mastered the subject, and what knowledge would I have then?

Cheaters

It is very important for vendors such as Cisco to maintain the integrity of their exams.  Unfortunately, there is an army of professional cheaters who steal questions and post them verbatim on the Internet.  There are various techniques a vendor can use to combat this, but one of the best is simply to make the question pool so large that nobody can memorize it all.  This in turn puts relentless pressure on vendors to write as many questions as possible.  If you’ve ever used a PDF copy of a test to study, not only are you cheating yourself, you are also putting pressure on us to make more and more questions, which can decrease the question pool quality.

Amusingly enough, when I took my private pilot written exam back in 2005, we studied using the famous red Gleim book, which (legally) had all the questions and answers on the test.  When I took the test, I brought in my calculator and flight computer and never used them, because I had already seen the questions so many times!  I would just look at the question and think, “OK, the answer there is a heading of 270 degrees.”  The only reason I got less than one hundred percent was because I intentionally answered a few wrong, on the advice of my instructor.  Apparently if you show up to the oral portion of the flight exam with a 100% written score, the examiner might think you were arrogant and try to put you in your place with challenging questions.

 

I hope this article provided a little insight into why certification exams sometimes have bad questions.  Please know my intention is not to defend bad questions on a test.  Questions that are poorly written and which make it through Q/A are unfair to test takers and cause unneeded stress and even failures.  I have been on the other side many times, and have experienced the frustration of wrong or unfair exams.  As far as Cisco goes, a number of recent blogs have complained about the quality of the CCIE R/S question pool.  Having worked with the people responsible for it, I can say I have a lot of confidence in them and I think they are doing a good job administering it.

**  NOTE:  I will not answer any specific questions about the exam content I have written, so don’t even try!

 

Note:  This article was originally posted in 2016.  Since that time, the CCIE program has changed the process for earning a CCIE, and the separate written exam is no longer used.  This means that the problem of people claiming to be a “CCIE” when they have only passed the written exam is no longer the case.  I’m leaving the article as is for now, but will modify it in the future when I have time, to reflect the new circumstances.  Regardless, you should never claim you have a certification when you have only passed a part of the requirements.  (ccie14023, Feb 2020)

In this article in the “Ten Years a CCIE” series, I look at the question of cheating.  Is it possible to cheat on the CCIE exam?  And what does cheating do to the value of the certification?

Yes, you can cheat on the CCIE

Shortly after I passed my Security exam I spoke with the first CCIE to pass the Voice exam. He took a beta version while he worked at Cisco. I commented that I valued my CCIE so much because it was simply impossible to cheat at the exam. It wasn’t a written exam; you couldn’t just walk in knowing the answers; you had to think on your feet. He laughed at me and explained my ignorance.

A lot of people cheat on the CCIE lab exam, he said. Either they work in groups and share the contents of the exams  they’ve seen, or else they get copies of the exam from unscrupulous vendors on the Internet. Then, having seen the actual exam, and having researched the difficult problems, and configured it several times in their lab, they can pass with ease.

I was quite shocked to hear this. I had always studied alone, and when I started down the CCIE road, I didn’t just want to pass the exams, I wanted to beat them. I didn’t just want the CCIE, I wanted the CCIE mystique. I was flabbergasted that people would want the certification without the work. Of course there is a great appeal to gaining something so valuable with minimal effort, but how are you going to make it through a job interview?

The stupidity of cheating

I had encountered rampant cheating in graduate school. This was at the dawn of the Internet, and I saw that many of my fellow students ripped off entire papers from the Internet. We used to send our papers to each other via email, and occasionally I would paste a snippet into AltaVista (Google not being available yet), and often I would hit upon the original work that they’d stolen. Leaving aside the ethical issues of stealing someone else’s work, or ripping off the questions and answers for an exam, there is a practical downside to cheating. You are claiming a credential that you haven’t earned. I remember conducting a job interview of a girl with a Masters degree from the same program as myself. I asked her the subject of one of her papers, and it had to do with routing protocols. She couldn’t answer even the most basic questions about the content of her paper. It was obvious that she cheated her way through the program. And she looked like a complete fool claiming to be a “master” of a subject about which she knew nothing.

Sometimes engineers I know roll their eyes when they hear I have a CCIE. They have encountered one of my fellow “experts” only to find that he seemed hardly an expert at all. Since I know that it’s possible to cheat on this exam, I’m convinced that many of these so-called CCIE’s cheated on their exams. They look like fools as did the girl with her Masters degree.

I’ll talk more about the value of the certification and later post, but one thing to keep in mind is that there’s great value in the study process. There’s great value in learning. And if you study for the test as you are supposed to study for it, you’re guaranteed to learn a lot.

A blurry ethical line?

I, like almost everybody these days, passed my exams using material from legitimate vendors, primarily Internetwork Expert and IPExpert.  (The latter has closed their doors.)  These vendors provide quite a lot of material, but their signature product is a book of sample exams designed to prepare you for the real thing.  This brings up a question.  Presumably some of the scenarios covered by the “legitimate” vendors are scenarios that might come up on the real lab.  After all, how many ways are there to configure BGP?

Interestingly enough, any exam has to provide a certain amount of information to test-takers beforehand.  The CCIE exams have detailed blueprints which guide candidates in their studies.  It would be impossible to take and pass an exam without such advance information.  With merely a blueprint in hand, it would be possible to construct some kind of sample exam, but do vendors simply build them off the blueprint?  Or do they get information from candidates and use them to build their tests?

The ethical lines can be blurry, but one thing is for certain:  studying for a CCIE exam using an actual copy of a real test is blatant cheating and disgraceful behavior.

Cheating on the written exam

Cheating is also rampant on the written exam. This is even the case among CCIE’s who are recertifying, perhaps especially the case. As I mentioned in my recertification post, taking an exam every two years, especially a hard one, is a big hassle. Many CCIE’s get lazy about the process. There are vendors who will sell verbatim copies of the tests. There is still, of course, some work involved. Someone with a copy of the test has to actually memorize all the answers. But it is far easier than going into a test blind.

My last re-certification was quite painful, and yet I refuse to use any sort of brain dump.  Instead, I built an Anki database of questions.  It wasn’t perfect, and it took a couple of failures for me to build a database that had sufficient coverage.

False CCIEs

Another way to “cheat” is simply to falsely claim CCIE status.  Anyone with a CCO account can verify if somebody actually has a CCIE, and whether they are active, but oftentimes employers just don’t bother to check.  When I was at Cisco HTTS, we were very close to hiring someone for a CCIE-requiring position, when I ran his name through the tool.  His CCIE had been revoked because he hadn’t recertified.  He was clearly embarrassed, and had simply been too busy to recertify.  While I can empathize with that, the fact was that he did not have a CCIE and would need to take both written and lab to get it back.  We didn’t hire him, but it amazed me that nobody had bothered to check early in the hiring process.

There is also a large group of imposters who have passed the written, and somehow think this qualifies them to put “CCIE” on their resume.  I recently saw a poster on a LinkedIn group who gave herself a CCIE (Wrt) title.  I also remember one candidate who put “CCIE Routing/Switching” in huge, bold letters on his resume, with “written” in a tiny font right next to it.  Well, I have news for you.  There is no CCIE written certification.  You either have a CCIE or you don’t.  Pass the lab before you put CCIE anything on your resume.  If you are looking at an employer that is willing to sponsor you for the lab, then by all means, tell them you passed the written.  But don’t claim CCIE status without a number.

What is the value?

We all know that there are a number of CCIE’s out there who should not have the certification.  It reflects poorly on the CCIE community.  There is no question that whatever value the CCIE has is diminished by those who obtained their credential through fraud.  If you are frustrated with the exam and thinking of hunting for a brain dump, remember this:  if you can’t pass the exam, you have no right to call yourself a CCIE.

In my next and final article in the series, The Value of a CCIE, I will take a look at the value of the credential.  Ten years later, do I think it was worth it?  Would I recommend someone take the CCIE exam now?  What do I think the future is for network experts in the world of SDN and automation?

Note:  This article was written in 2016 and has not been modified.  A number of changes have been made to the CCIE program which have dramatically improved the re-certification process.  Continuing education is now an option, as I suggest in this article.  The re-certification frequency has been reduced.  I may modify this article in the future, but I am leaving it as is for now for historical purposes.  However, please note the description of the process is no longer accurate at all.  (ccie14023, Sept 2021)

 

In this installment of “Ten Years a CCIE,” I look at what you have to do to stay certified, and the difficulty of maintaining your credential.

Passing your CCIE gives you a great feeling of accomplishment, and also a sense of relief.  You’ve spent months studying and late nights configuring scenarios in the lab.  Maybe you took the exam multiple times, and had to experience the letdown of knowing that, instead of being finished, you had more months of studying ahead.  So, you’ve finally passed, and it’s all over, right?

No, unfortunately.  You have a CCIE, but if you want to keep it, you have to worry about hitting the books again every two years.  All CCIE’s have to re-certify, a biennial ritual that becomes harder as the years go by.

Here’s how it works.  Before two years after your lab date, you have to re-certify your CCIE by passing a CCIE written exam.  You can take any written exam, just as long as it is a CCIE written.  For example, if you passed Routing and Switching, you could recertify by taking the Data Center written exam.  This has the advantage of simultaneously qualifying you for another lab exam, if you are so inclined.  If you have more than one CCIE, you can recertify all of them by taking any CCIE written.  For example, if you have Routing/Switching, ISP Dial, and Collaboration CCIEs, you could recertify all of them at once by taking the Wireless written.  This holds true even though ISP Dial is no longer a valid certification.  Even if you only have a certification that no longer exists (such as ISP Dial or SNA IP), you can maintain active CCIE status by passing any written exam.

If you don’t pass a written exam, at the two year mark your certification becomes suspended.  You can no longer use your CCIE number in your signature or claim to be a CCIE.  You can still pass the recert exam within a year, but if a year elapses after you go suspended, you lose your CCIEs, all of them, and have to retake both written and lab for any CCIE you hold.  Needless to say, you don’t want that to happen.

recert

What you want to see when you verify your CCIE…

(For comparison, my JNCIE-SP expires every three years, and I have to take the JNCIP-SP exam to recertify.  If I had a JNCIE-ENT as well, I would have to take both exams to recertify.)

If you just passed your lab exam and you feel super-confident, you may think you don’t have to worry about a measly written exam in two years.  However, any CCIE will tell you the recertification ritual is onerous and a huge waste of time.  As your career advances, you will often find yourself doing less and less CLI, and you might in fact work less with Cisco products.  In my case, re-certifying became especially painful during my six years at Juniper.

It would be less of a burden if the exams were better written.  The last time I took the written, there was a question that was flat out wrong, and many that were just obscure.

I first wrote this entry in 2014, and I am now re-writing it two years later.  When I first wrote it, I was working on my recert and in a state of extreme annoyance, came up with a couple of sample questions intended to mimic the actual exam:

When is the MSDP ConnectRetry timer used?
a.  When the MSDP peer with the highest IP address transitions from the INACTIVE to the CONNECTING state.
b.  When the MSDP peer with the lowest IP address transitions from the CONNECTING to the ESTABLISHED state.
c.  When the MSDP peer with the lowest IP address transitions from the INACTIVE to CONNECTING state.
d.  When the MSDP peer with the highest IP address transitions from the CONNECTING to the ESTABLISHED state.

What is the RSVP message type for a PathTear message?
a. 4
b. 0
c. 5
d. 3

What does the “ipv6 mld limit 100″ command do?
a.  Limits the number of hosts that multicast listener discovery can discover to 100
b.  Limits the hosts permitted by MLD to those contained in ACL 100
c.  Limits the number of MLD states to 100 on a per-interface basis.
d.  Limits the number of MLD states to 100 globally.

At the time I wrote them, these questions were technically within the blueprint topics for the Routing and Switching written exam, but they are obviously rather stupid questions.  The R&S blueprint is so huge that it is essentially impossible to know all of the subjects it covers.  Nevertheless, I was encountering questions of roughly this level of obscurity on the exam.

The purpose of recertification

Why do we have to recertify?  Obviously, the main reason is to ensure CCIE’s stay current in the field.  I passed routing/switching back in 2004, and a lot has changed in 12 years.  It’s important that people who come to me for expertise believe that I actually have relevant knowledge.

We have to ask a question though:  how well do you stay up-to-date taking a written exam every two years?  And why can you keep your credential when you re-certified in a different track?

For example, if someone acquired a CCIE Security certification back in 2002, but re-certified for 14 years using the routing/switching written, why is that engineer qualified to continue calling himself a “CCIE Security”?  He probably knows nothing of modern security technologies.  Juniper requires JCNIE’s to recertify in each track they have certified, so a triple JNCIE has to take three separate exams.  While this is painful (and kept me to one JNCIE), it makes more sense.

I think an even more reasonable approach is to allow continuing education in lieu of a test.  This is the requirement for CISSPs, lawyers, and even doctors, and it makes a lot of sense.  I never remember much from the recert exams, but a couple days of training would be a great way to get current.

I do think Cisco was smart to introduce the Emeritus option.  CCIE Emeritus allows CCIE’s who have hit the 10 year mark to pay a fee to keep their number in a non-active status indefinitely, with the option to recertify.  Many CCIEs reach a point where they don’t deal with day-to-day CLI configuration, and find the exams harder and less relevant to their careers.  Several of my friends have chosen this option.  I almost did when I worked at Juniper, but I am thankfully still current.

By the way, the answer to all of the above questions is ‘C’.

In my next article, Cheaters, I look at the question of whether people cheat on the CCIE exam, and the effect it has on the value of the certification.

In this article in my “Ten Years a CCIE” series, I look at passing the Security exam in 2008.  I get to experience the agony of failure for the first time, and have to re-tool my strategy.

Goodbye to Cisco

I worked two long years at Cisco. Two very long years. I learned so much there but it was a brutal job. The relentless flood of new and challenging cases grew tiresome.  When my aforementioned sushi eating CCIE friend called me in 2007 and invited me to come join him at a Gold partner I couldn’t say no.  Cisco sells much of its gear through value added resellers (VARs), also known as partners.  These partners are assigned different levels depending on the amount of business they do, and Gold is the highest.

Working at a gold partner with a CCIE was quite enjoyable. Gold partners need CCIE’s and so they have a lot of incentive to make you happy. My boss suggested that I get a second CCIE, this time in voice. I started to buy material for the voice exam, when my VP showed up in the office, fired my boss, and told me to start studying for the security exam. (His firing of my boss had nothing to do with my CCIE exam, but it certainly made me stand up and listen to what he was asking.) So, having really not started on voice, I switched immediately to security.

I had already passed the security written back at Cisco partly to qualify for the lab exam, and partly to re-certify my existing CCIE, so it was straight to the lab exam for me. The equipment list was a big challenge. At that time, you needed two ASA’s, one PIX, a VPN 3000 series concentrator, and IDS device, six routers and two switches, and some sort of Windows server running Cisco secure ACS. I still had my old lab equipment from before, but I was missing everything else. I had one ASA 5505 from work, but no other security devices. I decided that the cost was too prohibitive for me to set up my own lab. I was going to have to use rack rentals. That was my first big mistake.

I decided to approach the exam in exactly the same way I approached the routing and switching exam. I studied the various subjects on the blueprint individually, and then started doing full labs from the Internetwork Expert workbook. As great as IE’s workbook was for routing and switching, in 2008 it really wasn’t very good for security. I have a lot of respect for the Bryans, and I’m sure it’s come a long way, but at that time it just wasn’t enough.

Attempt number one

When I showed up at the familiar CCIE lab, I didn’t feel well prepared, because I wasn’t. The lab was a disaster. I only managed to complete about a third of the exam. While configuring DMVPN, all of my routers locked up and crashed. I called the proctor over, and when he saw that the console ports were locked up, he started to accuse me of having made a configuration error. I explained that I hadn’t touched the console configuration, and just then we both saw bus errors appear on the console sessions followed by reloads. It was obvious then that I was not at fault. I had heard that if routers crash during CCIE exam, the proctor will give you your time back. However, the proctor admonished me to save my configs frequently, and refused to give me any time back. I had probably lost 15 minutes. I would have fought it, except that I was already so far behind on the exam,  I knew it would make no difference. Still, to this day am a bit angry at that proctor. As I left the exam room I looked at him and said “don’t even bother grading this.” He looked at me and said, “Oh, I’m sure you’re exaggerating.” I looked at him and told him I hadn’t completed two thirds of the exam.” Oh!” He exclaimed.” Well… Don’t wait six months for your next exam!”

… It was six months before my next attempt.

The author's 2008 CCIE security lab. The laptop ran Windows Server in a VM for ACS. An ASA 5505 is visible on top of the drawers.

The author’s 2008 CCIE security lab. The laptop ran Windows Server in a VM for ACS. An ASA 5505 is visible on top of the drawers.

Changes to my approach

I knew I had to revise my strategy. Something wasn’t working. The first thing I fixed was the lab situation. When I did Routing and Switching, I knew that I needed my own lab at home. Using remote rack rentals for security just didn’t give me enough time in the lab. I managed to get a hold of the PIX from a friend who was decommissioning it. I bought myself an ASA 5510, which, at $2500, was the most expensive piece of hardware I had. I really needed two of them, in order to cluster them, but I had to make do with the mismatched pair of the 5510 and 5505. As with the Routing and Switching exam, I knew I could use remote rack rentals to fill in for the equipment that I didn’t have. The ASA 5505 was adequate for basically everything except clustering. It had almost all of the capabilities of the 5510, but the configuration of VLANs was slightly different.  I also managed to acquire an IDS, and VPN 3000 series concentrator. I borrowed a laptop from work and got a Windows server license and managed to install Cisco Secure ACS. I ended up with a very complete lab.

I realized that a big part of my problem was that IPSec configurations are long, complicated, and counterintuitive. IPSec is the core of the CCIE security exam, and you need to know it as well as BGP and OSPF on the routing and switching exam. I made a series of diagrams which depicted each of the constituent configuration elements for the various IPSec technologies as blocks, which were then connected together by arrows. For example, for basic IPSec configuration, I would have one block representing the IKE configuration, and another representing the IPSec policy. I would draw an arrow to show how they were connected, labeling the arrow with the command used to connect them. Before I was trying to memorize these configurations. Now I was able to visualize them.

Visualizing complex configurations helps make them easy to understand and remember

Visualizing complex configurations helps make them easy to understand and remember

I also completely abandoned using the IE workbook. It just wasn’t ready at that point. Instead I invented my own VPN challenge lab. It had every kind of VPN on it: IPSec on ASA, IPSec on PIX, IPSec on VPN 3K, client IPSec on all of those platforms, L2TP, PPTP, DMVPN, SSL. I worked this lab over and over again until I could configure all of these automatically, and I made sure I configured between disparate platforms.

I felt good but not 100% prepared when I went to take my second attempt. I failed, but my score was much higher than before. I continued preparing for another month or so before taking my third attempt. I was so ready for my third attempt, that I completed the lab shortly after lunch. As I was coming out of the bathroom, I ran into Ted the proctor (not his real name), in the hallway. I had seen Ted on my second attempt and he told me he was attending a bluegrass Festival in San Francisco. I spent a good 15 minutes talking to Ted about the festival in the hallway, and I think at that point Ted realized that I was feeling pretty confident. Most people don’t spend 15 minutes shooting the breeze in the middle of the CCIE exam.

Interestingly enough, while Ted had been the most helpful Proctor on the Routing and Switching exam, he was of almost no help at all on the security exam. I’m not sure if he had changed in the intervening four years, or if he simply wasn’t as familiar with the security exams that I took. Either way, be prepared to make difficult decisions on your own in the lab, without the help of the proctors. Of all the questions I asked them, only once did I get a useful answer. I realize that their job is not to give away the test, but often the test is poorly written and I think that they need to be more helpful in explaining the exam.
Passing Routing and Switching was exciting; passing Security was a relief. I had almost given up after my disastrous first attempt. And I’m glad that I passed it when I did. As with the Routing and Switching exam, I passed Security in November. And as with the Routing and Switching exam, Cisco was changing the test at the beginning of the new year. The VPN 3000, PIX, NAC framework, and several other technologies were being removed. Of course, they never removed technologies without adding some as well. Had I failed my third attempt, it’s likely I would never have tried again.

In summary:

  • Having “always-on” access to a lab is critical!  Remote rack rental is good to fill in for a few things you might be missing, but don’t rely on it.
  • You may have to spend some more money than you want to acquiring gear, but it pays off.
  • The way you pass one CCIE exam is not necessarily the way you pass another exam.  You have to spend some time looking at the topics you will be covering, figuring out the best way to reach the point of automatic configuration of the technologies.
  • Sometimes, the study material from the vendors just won’t cut it.
  • Proctors aren’t always nice, and don’t always do what you thought they were supposed to.

I will cover the question of lab blueprint changes in a later article on the value of a CCIE, but it’s worth noting that for both my routing/switching and security exams, a blueprint change happened immediately after my passing.  I spent a lot of time studying, for example, the VPN 3000 concentrator which was already obsolete.  Still, I would have the same credential as a guy who passed the exam with the new technologies a couple months later.

Also worth noting:  I passed all of my expert exams (2 CCIEs and a JNCIE) in November.

In the next article in the series, Recertification Pain, I look at the biennial penance we all inherit for passing our CCIEs–the dreaded recertification.  I give my thoughts on improving the process, not that anybody is listening.