Skip navigation

Category Archives: Ten Years a CCIE

Two Years of Ten Years a CCIE

Two years ago I published my Ten Years a CCIE series.  Actually, I had written the series a couple years before I published it, but as I say in my introduction to the series, I felt it was a bit self-indulgent an uninteresting, so I scrapped it for a while.  The original pieces were dictated, and I’ve been meaning to go back and clean up some of the grammatical errors or grating phrases, but haven’t had the time.  Not a lot of people have read it, nor did I expect many to read it, since I generally don’t advertise the blog in social media, or anywhere really.  But the feedback from the few who have read it has been positive, and I’m gratified for that.

Things have changed a lot since I got into networking in 1995, and since I passed my CCIE in 2004.  But it’s also amazing how much has stayed the same.  TCP/IP, and in fact IPv4, is still the heart of the network.  Knowledge of OSPF and BGP is still key.  For the most part, new controllers and programmable interfaces represent a different way of managing fundamentally the same thing.

The obvious reasons for this are that networks work and are hard to change.  The old protocols have been sufficient for passing data from point A to point B for a long time.  They’re not perfect but they are more than adequate.  They are hard to change because networks are heterogeneous.  There are so many types of different systems connecting to them, that if we wanted to fundamentally alter the building blocks of networks, we’d have to upgrade a lot of systems.  This is why IPv6 adoption is so slow.

Occasionally I poke around at TechExams.net to see what newer network engineers are thinking, and where they are struggling.  I’m probably the only director-level employee of Cisco who reads or comments on that message board.  I started reading it back when I was still at Juniper and studying for my JNCIE, but I’ve continued to read it because I like the insights I get from folks prepping for their certifications.  People are occasionally concerned that the new world of controllers and automation will make their jobs obsolete.

I built the first part of my career on CLI.  Now I’m building it on controllers and programmability.  In this industry, we have to adapt, but we don’t have to die.  Cars have changed drastically, with on-board computer systems and so forth, but we still need mechanics.  We still need good network engineers.

To be honest, I was getting tired of my career by the time I left Juniper and came to Cisco.  I was bored.  I thought of going back to school and getting a Ph.D. in classical languages, my other passion.  Getting married helped put an end to that idea (Ph.D.’s in ancient Greek make a lot less than network engineers) but when I came back to Cisco, I felt revitalized.  I started learning new things.  Networking was becoming fun again.

I wrote the “Ten Years a CCIE” series both for people who had passed the exam and wanted to have some fun remembering the experience, as well as for people struggling to pass it.  Some things change, as I said, but a lot remains the same.  I still think, closing in on 15 years since I took the exam, that it’s still worth it.  I still think it’s a fantastic way to launch a career.  The exam curriculum will adapt, as it always does, with new technologies, but it’s an amazing learning experience if you do it honestly, and you will be needed when you make it through.

The value of a CCIE

In the final post in my “Ten Years a CCIE” series, I take a look at the age-old question: Is a CCIE really worth it? I conclude the series with some thoughts on the value of this journey.

I’ve written this series of posts in the hope that others considering the pursuit of a CCIE  would have some idea of the process, the struggles, and the reward of passing this notorious exam.  I would like to finish this series with a final article examining the age-old question:  What is the value of a CCIE?  In other words, was it worth it?  All of the other articles in the series were written a couple of years ago, back when I worked for Juniper in IT, with only some slight revisions before publishing.  However, this article is being written now (late 2016), in very different circumstances.  I work at Cisco now, not Juniper.  I work in the switching business unit, not in IT.  I work in product management, and as a Principal Engineer have direct influence on the direction of our products.  I specifically work on programmability, automation, and SDN solutions.  As I write this, I see many potential CCIE candidates wondering if it is even worth pursuing a career in networking any more.  After all, won’t SDN and automation just eliminate their jobs? Wouldn’t they be better off learning Python?  Isn’t Cisco a company in its death throes, facing extinction at the hands of upstarts like Arista and Viptela?

Well, I can’t predict the future.  But I can look at the issues from the perspective of someone who has been in the industry a long time now, and at least set your mind at ease.  The short answer:  it’s worth it.  The longer answer requires us to examine the question from a few different angles.

Technical Knowledge

As I’ve pointed out in previous articles, during the CCIE preparation process, you will master a vast amount of material, if you approach the test honestly.  But, as anyone in this industry knows, the second you pass, a timer starts counting down the value of the material you learned.  On my R&S exam (2004), I studied DLSw+, ATM, ISDN, and Frame Relay.  On my Security (2008) exam, I studied the VPN 3k concentrator, PIX, and NAC Framework.  How much of that is valuable now?

True, I studied many things that are still in use today.  OSPF, BGP, EIGRP, and ISIS were all heavily tested on the R&S exam even back in 2004.  But how much of it do I remember?  I certainly have a good knowledge of each of them, but even after taking the JNCIE-SP exam less than two years ago, I find my knowledge of the details of routing protocols fading away.  If I were to take the JNCIE exam today, I would certainly fail.

Thus, some technologies I learned are obsolete, and some are not, but I have forgotten a lot.  However, there are also technologies that I never learned.  For example, ISE and Cisco TrustSec are huge topics that I am just starting to play with.  Despite my newness to these technologies, I do have the same right to call myself a “Security CCIE” as a guy who passed it yesterday, and knows those subjects cold!

So, then, what about the value of the technical knowledge?  Is it worth it?

First, it is because no matter what becomes obsolete, and no matter what you forget, the intensity of your study will guarantee you still have a core of knowledge that will be with you for a long time.  Fortunately, despite the rapidity of change our industry is supposedly undergoing, networking is conservative by nature.  The Internet is a large distributed system consisting of many systems running disparate operating systems and they have to work together.  Just ask the guys pushing IPv6.  The core of networking doesn’t change very much.  You will also learn that even new technologies, like VXLAN, build on concepts you already know.

Second, even obsolete knowledge is valuable.  Newer engineers often don’t realize how certain technologies developed, or why we do things the way that we do.  They don’t know the things that have been tried and which failed in the past.  Despite the premium our industry places on youth, old-timers do have important insight gleaned from playing with things like, say, DLSw+.  It’s important for us not to simply wax nostalgic (as I am here) about the glorious days of IPX, but to explain to younger engineers how these dinosaurs actually worked, so that they can understand what decisions protocol designers made and why, and why some technologies fell into disuse or were replaced.  And, millennials, it is your responsibility to sit up and listen, if not to seek out such information.

Technical “soft skills”

In addition to specific technical knowledge, CCIE preparation simply makes you better at configuring and troubleshooting in general.  The hours of frustration in the lab and the broken configs that have you pulling your hair out ultimately help you to learn what makes a network break, and how to go about systematically debugging it.  Certainly you can learn those skills elsewhere, but when you are faced with the challenge of building a lab in a short amount of time, you have to be able to think quickly on your feet.  Even troubleshooting a live network outage doesn’t quite compare the intensity of the CCIE crucible, the eight hour slog during which one mistake can cost you months of studying.  The exams where I took multiple attempts required me to seriously up my game between tries, and each time I’ve come out a stronger thinker.

Non-Technical Skills

Passing the CCIE exam honestly is a challenge, plain and simple.  It requires discipline, persistence, extreme attention to detail, resourcefulness, ability to read documentation carefully, and time management skills.  While you won’t pass the CCIE without having those skills to begin with, there is no question that the rigors of CCIE preparation will help to hone them.  You will be a better person all around if you submit yourself to the discipline of passing the exam honestly.

It’s a bit like studying the martial arts.  Most martial artists will tell you that, in addition to being able to deliver a mean axe kick to the head, they have achieved self-discipline and confidence from their practice of martial arts, and that they find these skills apply elsewhere in life.  It’s true of any rigorous pursuit, really, and definitely true of the CCIE.  Personally I have no doubt that the many hours of laborious study have made me a more detail-oriented and better engineer.

Employment value:  Employee’s perspective

In my own case, my CCIE certification was directly responsible for my getting hired at Cisco TAC in 2005.  It was absolutely essential for me to move to a VAR in 2007.  Believe it or not, it was critical in my getting a job at Juniper in 2009, and having two CCIE’s and a JNCIE helped open the door to be re-hired at Cisco in 2015.  Obviously, employers see value in it.

You still see CCIE certification listed as a requirement in many job descriptions.  Without one you are cut out of those positions.  Many employers see it as a key differentiator and qualification for a senior position in network engineering.  VARs are still required to have a number of CCIEs on staff, so for some positions they will only talk to someone who has a CCIE.

Rarely is it the only qualification, however, and rarely is it enough, on its own, to get you hired.  Shortly after I passed my second CCIE I got laid off from the VAR where I worked.  I ended up interviewing at Nexus IS (now Dimension Data), another VAR, a couple months later.  Having been out of work I was rusty, and the technical interviewer grilled me mercilessly.  I’ve stated in the past that I don’t find this sort of thing productive, and I was uncomfortable, and didn’t have a great performance.  Despite the fact that I had two CCIE’s, and despite the value of such credentials for a reseller, I got a call from HR in a few days telling me that they chose not to hire me.  (These things often work out in the end;  I am in a good place now, and I wouldn’t have made it here if I had stayed in the VAR world.)  The entire experience reinforces a point I made in my Cheaters post:  don’t think an ugly plaque and a number are a guarantee of employment.

That said, there is little doubt that a person with X skill-set and a CCIE is in a better place than a person with just X skill-set.  As I said above, the certification simply opens doors.  You cannot rely on it alone, but combined with good experience, the certification is invaluable.  I would never have made it this far without one.  Now it’s true that there are some very talented and knowledgeable engineers who don’t have, and in some cases disdain, the CCIE.  Many of these folks do quite well and have successful careers.  But again, add the famous number and you will always do better in the end.

Employment value:  Employer’s perspective

Let’s turn it around now, and look at it from the perspective of a hiring manager.  When I have the stack of resumes in front of me, how important is it for me to look for that famous CCIE number?

This is a much harder question to answer, and I’ve touched on some of the problems of the CCIE above.  If you are looking at somebody who has a CCIE #1xxx, you know you are getting someone with age and experience.  But if you’re looking for someone to do hands-on implementation work, will this be the right candidate?  Maybe, of course, if Mr. #1xxx has been doing a lot of that lately.  The point is, you can’t tell from the number alone.

Now let’s say you are looking at CCIE #50xxx.  Does her CCIE number mean she will be the perfect fit for the hands-on job?  Quite possibly, because she has a lot of recent hands-on experience.  Will she be unqualified for the management role because she lacks the experience of CCIE #1xxx?  Who knows?

The point is, from a hiring manager’s perspective, the CCIE is simply one piece of data in the overall picture of the candidate.  It tells you something, something important, but it’s not nearly enough to be sure you are picking the right person.  If you are hiring for a VAR and just need a number to get your Gold status, then the value of the CCIE jumps up a little higher.  If you are just hiring a network engineer for IT, you need to take a host of other factors into account.

At the end of the day, I hope that people don’t make hiring decisions based on that one criterion.  I had one job where we hired in a CCIE (against my advice) because the hiring manager believed anyone in possession of a CCIE number was a genius.  (See “The CCIE Mystique”, here.)  The guy was a disaster.  Did he pass by cheating, or was he just good at taking tests?  I don’t know.  I’ve also known many, many extremely smart and talented network engineers who never got their CCIE, including several who just couldn’t pass it.  Keep that in mind when you are getting overly impressed with the famous four letters.

SDN and the New World Order

I would like to conclude this post, and this series, with a few thoughts on the relevance of CCIE certification in the future.  Various people in the industry, most of whom hold MBA’s in finance or marketing, tell us that CLI is dead and that the Ciscos and Junipers will be replaced by cheap, “white-label” hardware.  Google and Facebook are managing thousands of network devices with a handful of staff, using scripting and automation.  A CCIE certification is a waste of time, according to this thinking, because Cisco is spiraling down into oblivion.  Soon, it will all be code.  Learn Python instead.

I’d like to present my thoughts with a caveat.  I’m not always a great technology prognosticator.  When a friend showed me a web browser for the first time in 1994, I told him this Web thing would never take off.  Oops.  However, I have a lot of experience in the industry, and right now my focus at Cisco is programmability and automation.  I think I have a right to an opinion here.

First of all, there is no question that interest in automation and programmability is increasing.  All of the vendors, including my employer, are devoting a lot of resources towards developing and increasing their programmability and automation capabilities.  I’ve spent my first year here at Cisco working on automating network devices with Puppet, Ansible, Python, and our own tools like DCNM.  We’ve been working hard to build out YANG data models for our features.  I barely touched Expect scripting before I came here!  Customers are interested in managing their networks more efficiently, sadly, sometimes with fewer people.

Let me look at this from a wider perspective.  As computing power increases, are humans redundant?  For example, as a pilot I know that it’s entirely possible to replace human pilots with computers.  Air Traffic Control could relay instructions digitally to the computers that now control most airplanes.  With ILS and precision approaches, airplanes could land themselves at most airports.  But would you get into an airplane that had no human pilots?

The problem is, even in the age of Watson, computers react predictably to predictable circumstances, but predictably badly to unpredictable circumstances.  Many of the alleged errors that are introduced with CLI will still be introduced with NETCONF when the operator puts in the wrong data.  Scripts and automation tools have to power to replicate errors across huge numbers of devices faster than CLI.

At the end of the day, you still need to know what it is that you’re automating.  Networks cannot go away.  If all the Cisco and Juniper boxes out there vanished suddenly, our digital world would come to a screeching halt.  We still need people who understand what switches, routers, and firewalls do, regardless of whether they are managing them with CLI or Python.  Look at the cockpit of a modern airplane.  The old dial gauges have been replaced by flat-panel displays.  Do you think pilots no longer need to study weather systems, aerodynamics, and engine operation?  Of course they do!  Just the same, we need network engineers, a lot of them, to make networks work correctly.

As to whether Cisco itself goes away, who knows?  Obviously returning here, I have a great deal of faith in the company and its ability to execute.  I think that a lot of the industry hype is just hype.  Sure, some of the largest network operators will displace our hardware with Open Compute boxes.  Sure, it will hurt us.  But we are still in most networks and I don’t see that changing for a long time.

Closing thoughts, and a postscript

I began this series with an article on what I called, “The CCIE Mystique.”  I don’t know if that quite exists the same way it did in 2001 when I started this journey, but I suspect it is still there a least a little bit.  Those of us who have been through it have had the veil pulled back a bit, and we see it for what it is:  a very hard, but ultimately very surmountable challenge.  A test that measures some, but not even close to all of the skill required to be a network engineer.  A ticket to a better career, but not a guarantee of one.

Being back at Cisco has been an amazing experience.  This is the company that started my career, and that has built an entire industry.  I’d be happy to finish things out here, but I have no idea in this age of mass layoffs whether that will be possible.

I had an interesting chat with the head of the CCIE route/switch lab curriculum not long ago.  I must say, that for whatever complaints I have about the program, I was quite impressed with him and his awareness of where the program is strong and where it is lacking.  I have no doubt that those guys work very hard on the CCIE program and it’s always easier to sit on the sidelines and complain.  Ten years ago, when I started this journey, I wouldn’t have believed I would be at Cisco, helping to develop and market products, with a lab stocked full of gear, chatting with a CCIE proctor.  Everyone’s path is different, and I certainly cannot promise my readers that investing in one certification will land them a pot of gold.  There were a lot of other factors in my career trajectory.  But at the end of a decade (actually 12 years as I write this), I can say that the time, money, and effort spent in pursuit of the elusive CCIE was well worth it.

As I close out this series I hope that my little autobiographical exclusion was not too self-indulgent.  I hope that those of you who are new to the industry or studying for the exam got some insight from the articles.  I hope that any other old CCIE’s who stopped by relived a few memories.  Thanks for reading the series, and good luck on your studies!

3

Cheaters

Note:  This article was originally posted in 2016.  Since that time, the CCIE program has changed the process for earning a CCIE, and the separate written exam is no longer used.  This means that the problem of people claiming to be a “CCIE” when they have only passed the written exam is no longer the case.  I’m leaving the article as is for now, but will modify it in the future when I have time, to reflect the new circumstances.  Regardless, you should never claim you have a certification when you have only passed a part of the requirements.  (ccie14023, Feb 2020)

In this article in the “Ten Years a CCIE” series, I look at the question of cheating.  Is it possible to cheat on the CCIE exam?  And what does cheating do to the value of the certification?

Yes, you can cheat on the CCIE

Shortly after I passed my Security exam I spoke with the first CCIE to pass the Voice exam. He took a beta version while he worked at Cisco. I commented that I valued my CCIE so much because it was simply impossible to cheat at the exam. It wasn’t a written exam; you couldn’t just walk in knowing the answers; you had to think on your feet. He laughed at me and explained my ignorance.

A lot of people cheat on the CCIE lab exam, he said. Either they work in groups and share the contents of the exams  they’ve seen, or else they get copies of the exam from unscrupulous vendors on the Internet. Then, having seen the actual exam, and having researched the difficult problems, and configured it several times in their lab, they can pass with ease.

I was quite shocked to hear this. I had always studied alone, and when I started down the CCIE road, I didn’t just want to pass the exams, I wanted to beat them. I didn’t just want the CCIE, I wanted the CCIE mystique. I was flabbergasted that people would want the certification without the work. Of course there is a great appeal to gaining something so valuable with minimal effort, but how are you going to make it through a job interview?

The stupidity of cheating

I had encountered rampant cheating in graduate school. This was at the dawn of the Internet, and I saw that many of my fellow students ripped off entire papers from the Internet. We used to send our papers to each other via email, and occasionally I would paste a snippet into AltaVista (Google not being available yet), and often I would hit upon the original work that they’d stolen. Leaving aside the ethical issues of stealing someone else’s work, or ripping off the questions and answers for an exam, there is a practical downside to cheating. You are claiming a credential that you haven’t earned. I remember conducting a job interview of a girl with a Masters degree from the same program as myself. I asked her the subject of one of her papers, and it had to do with routing protocols. She couldn’t answer even the most basic questions about the content of her paper. It was obvious that she cheated her way through the program. And she looked like a complete fool claiming to be a “master” of a subject about which she knew nothing.

Sometimes engineers I know roll their eyes when they hear I have a CCIE. They have encountered one of my fellow “experts” only to find that he seemed hardly an expert at all. Since I know that it’s possible to cheat on this exam, I’m convinced that many of these so-called CCIE’s cheated on their exams. They look like fools as did the girl with her Masters degree.

I’ll talk more about the value of the certification and later post, but one thing to keep in mind is that there’s great value in the study process. There’s great value in learning. And if you study for the test as you are supposed to study for it, you’re guaranteed to learn a lot.

A blurry ethical line?

I, like almost everybody these days, passed my exams using material from legitimate vendors, primarily Internetwork Expert and IPExpert.  (The latter has closed their doors.)  These vendors provide quite a lot of material, but their signature product is a book of sample exams designed to prepare you for the real thing.  This brings up a question.  Presumably some of the scenarios covered by the “legitimate” vendors are scenarios that might come up on the real lab.  After all, how many ways are there to configure BGP?

Interestingly enough, any exam has to provide a certain amount of information to test-takers beforehand.  The CCIE exams have detailed blueprints which guide candidates in their studies.  It would be impossible to take and pass an exam without such advance information.  With merely a blueprint in hand, it would be possible to construct some kind of sample exam, but do vendors simply build them off the blueprint?  Or do they get information from candidates and use them to build their tests?

The ethical lines can be blurry, but one thing is for certain:  studying for a CCIE exam using an actual copy of a real test is blatant cheating and disgraceful behavior.

Cheating on the written exam

Cheating is also rampant on the written exam. This is even the case among CCIE’s who are recertifying, perhaps especially the case. As I mentioned in my recertification post, taking an exam every two years, especially a hard one, is a big hassle. Many CCIE’s get lazy about the process. There are vendors who will sell verbatim copies of the tests. There is still, of course, some work involved. Someone with a copy of the test has to actually memorize all the answers. But it is far easier than going into a test blind.

My last re-certification was quite painful, and yet I refuse to use any sort of brain dump.  Instead, I built an Anki database of questions.  It wasn’t perfect, and it took a couple of failures for me to build a database that had sufficient coverage.

False CCIEs

Another way to “cheat” is simply to falsely claim CCIE status.  Anyone with a CCO account can verify if somebody actually has a CCIE, and whether they are active, but oftentimes employers just don’t bother to check.  When I was at Cisco HTTS, we were very close to hiring someone for a CCIE-requiring position, when I ran his name through the tool.  His CCIE had been revoked because he hadn’t recertified.  He was clearly embarrassed, and had simply been too busy to recertify.  While I can empathize with that, the fact was that he did not have a CCIE and would need to take both written and lab to get it back.  We didn’t hire him, but it amazed me that nobody had bothered to check early in the hiring process.

There is also a large group of imposters who have passed the written, and somehow think this qualifies them to put “CCIE” on their resume.  I recently saw a poster on a LinkedIn group who gave herself a CCIE (Wrt) title.  I also remember one candidate who put “CCIE Routing/Switching” in huge, bold letters on his resume, with “written” in a tiny font right next to it.  Well, I have news for you.  There is no CCIE written certification.  You either have a CCIE or you don’t.  Pass the lab before you put CCIE anything on your resume.  If you are looking at an employer that is willing to sponsor you for the lab, then by all means, tell them you passed the written.  But don’t claim CCIE status without a number.

What is the value?

We all know that there are a number of CCIE’s out there who should not have the certification.  It reflects poorly on the CCIE community.  There is no question that whatever value the CCIE has is diminished by those who obtained their credential through fraud.  If you are frustrated with the exam and thinking of hunting for a brain dump, remember this:  if you can’t pass the exam, you have no right to call yourself a CCIE.

In my next and final article in the series, The Value of a CCIE, I will take a look at the value of the credential.  Ten years later, do I think it was worth it?  Would I recommend someone take the CCIE exam now?  What do I think the future is for network experts in the world of SDN and automation?

1

Recertification Pain

Note:  This article was written in 2016 and has not been modified.  A number of changes have been made to the CCIE program which have dramatically improved the re-certification process.  Continuing education is now an option, as I suggest in this article.  The re-certification frequency has been reduced.  I may modify this article in the future, but I am leaving it as is for now for historical purposes.  However, please note the description of the process is no longer accurate at all.  (ccie14023, Sept 2021)

 

In this installment of “Ten Years a CCIE,” I look at what you have to do to stay certified, and the difficulty of maintaining your credential.

Passing your CCIE gives you a great feeling of accomplishment, and also a sense of relief.  You’ve spent months studying and late nights configuring scenarios in the lab.  Maybe you took the exam multiple times, and had to experience the letdown of knowing that, instead of being finished, you had more months of studying ahead.  So, you’ve finally passed, and it’s all over, right?

No, unfortunately.  You have a CCIE, but if you want to keep it, you have to worry about hitting the books again every two years.  All CCIE’s have to re-certify, a biennial ritual that becomes harder as the years go by.

Here’s how it works.  Before two years after your lab date, you have to re-certify your CCIE by passing a CCIE written exam.  You can take any written exam, just as long as it is a CCIE written.  For example, if you passed Routing and Switching, you could recertify by taking the Data Center written exam.  This has the advantage of simultaneously qualifying you for another lab exam, if you are so inclined.  If you have more than one CCIE, you can recertify all of them by taking any CCIE written.  For example, if you have Routing/Switching, ISP Dial, and Collaboration CCIEs, you could recertify all of them at once by taking the Wireless written.  This holds true even though ISP Dial is no longer a valid certification.  Even if you only have a certification that no longer exists (such as ISP Dial or SNA IP), you can maintain active CCIE status by passing any written exam.

If you don’t pass a written exam, at the two year mark your certification becomes suspended.  You can no longer use your CCIE number in your signature or claim to be a CCIE.  You can still pass the recert exam within a year, but if a year elapses after you go suspended, you lose your CCIEs, all of them, and have to retake both written and lab for any CCIE you hold.  Needless to say, you don’t want that to happen.

recert

What you want to see when you verify your CCIE…

(For comparison, my JNCIE-SP expires every three years, and I have to take the JNCIP-SP exam to recertify.  If I had a JNCIE-ENT as well, I would have to take both exams to recertify.)

If you just passed your lab exam and you feel super-confident, you may think you don’t have to worry about a measly written exam in two years.  However, any CCIE will tell you the recertification ritual is onerous and a huge waste of time.  As your career advances, you will often find yourself doing less and less CLI, and you might in fact work less with Cisco products.  In my case, re-certifying became especially painful during my six years at Juniper.

It would be less of a burden if the exams were better written.  The last time I took the written, there was a question that was flat out wrong, and many that were just obscure.

I first wrote this entry in 2014, and I am now re-writing it two years later.  When I first wrote it, I was working on my recert and in a state of extreme annoyance, came up with a couple of sample questions intended to mimic the actual exam:

When is the MSDP ConnectRetry timer used?
a.  When the MSDP peer with the highest IP address transitions from the INACTIVE to the CONNECTING state.
b.  When the MSDP peer with the lowest IP address transitions from the CONNECTING to the ESTABLISHED state.
c.  When the MSDP peer with the lowest IP address transitions from the INACTIVE to CONNECTING state.
d.  When the MSDP peer with the highest IP address transitions from the CONNECTING to the ESTABLISHED state.

What is the RSVP message type for a PathTear message?
a. 4
b. 0
c. 5
d. 3

What does the “ipv6 mld limit 100″ command do?
a.  Limits the number of hosts that multicast listener discovery can discover to 100
b.  Limits the hosts permitted by MLD to those contained in ACL 100
c.  Limits the number of MLD states to 100 on a per-interface basis.
d.  Limits the number of MLD states to 100 globally.

At the time I wrote them, these questions were technically within the blueprint topics for the Routing and Switching written exam, but they are obviously rather stupid questions.  The R&S blueprint is so huge that it is essentially impossible to know all of the subjects it covers.  Nevertheless, I was encountering questions of roughly this level of obscurity on the exam.

The purpose of recertification

Why do we have to recertify?  Obviously, the main reason is to ensure CCIE’s stay current in the field.  I passed routing/switching back in 2004, and a lot has changed in 12 years.  It’s important that people who come to me for expertise believe that I actually have relevant knowledge.

We have to ask a question though:  how well do you stay up-to-date taking a written exam every two years?  And why can you keep your credential when you re-certified in a different track?

For example, if someone acquired a CCIE Security certification back in 2002, but re-certified for 14 years using the routing/switching written, why is that engineer qualified to continue calling himself a “CCIE Security”?  He probably knows nothing of modern security technologies.  Juniper requires JCNIE’s to recertify in each track they have certified, so a triple JNCIE has to take three separate exams.  While this is painful (and kept me to one JNCIE), it makes more sense.

I think an even more reasonable approach is to allow continuing education in lieu of a test.  This is the requirement for CISSPs, lawyers, and even doctors, and it makes a lot of sense.  I never remember much from the recert exams, but a couple days of training would be a great way to get current.

I do think Cisco was smart to introduce the Emeritus option.  CCIE Emeritus allows CCIE’s who have hit the 10 year mark to pay a fee to keep their number in a non-active status indefinitely, with the option to recertify.  Many CCIEs reach a point where they don’t deal with day-to-day CLI configuration, and find the exams harder and less relevant to their careers.  Several of my friends have chosen this option.  I almost did when I worked at Juniper, but I am thankfully still current.

By the way, the answer to all of the above questions is ‘C’.

In my next article, Cheaters, I look at the question of whether people cheat on the CCIE exam, and the effect it has on the value of the certification.

Multiple CCIE’s, multiple attempts

In this article in my “Ten Years a CCIE” series, I look at passing the Security exam in 2008.  I get to experience the agony of failure for the first time, and have to re-tool my strategy.

Goodbye to Cisco

I worked two long years at Cisco. Two very long years. I learned so much there but it was a brutal job. The relentless flood of new and challenging cases grew tiresome.  When my aforementioned sushi eating CCIE friend called me in 2007 and invited me to come join him at a Gold partner I couldn’t say no.  Cisco sells much of its gear through value added resellers (VARs), also known as partners.  These partners are assigned different levels depending on the amount of business they do, and Gold is the highest.

Working at a gold partner with a CCIE was quite enjoyable. Gold partners need CCIE’s and so they have a lot of incentive to make you happy. My boss suggested that I get a second CCIE, this time in voice. I started to buy material for the voice exam, when my VP showed up in the office, fired my boss, and told me to start studying for the security exam. (His firing of my boss had nothing to do with my CCIE exam, but it certainly made me stand up and listen to what he was asking.) So, having really not started on voice, I switched immediately to security.

I had already passed the security written back at Cisco partly to qualify for the lab exam, and partly to re-certify my existing CCIE, so it was straight to the lab exam for me. The equipment list was a big challenge. At that time, you needed two ASA’s, one PIX, a VPN 3000 series concentrator, and IDS device, six routers and two switches, and some sort of Windows server running Cisco secure ACS. I still had my old lab equipment from before, but I was missing everything else. I had one ASA 5505 from work, but no other security devices. I decided that the cost was too prohibitive for me to set up my own lab. I was going to have to use rack rentals. That was my first big mistake.

I decided to approach the exam in exactly the same way I approached the routing and switching exam. I studied the various subjects on the blueprint individually, and then started doing full labs from the Internetwork Expert workbook. As great as IE’s workbook was for routing and switching, in 2008 it really wasn’t very good for security. I have a lot of respect for the Bryans, and I’m sure it’s come a long way, but at that time it just wasn’t enough.

Attempt number one

When I showed up at the familiar CCIE lab, I didn’t feel well prepared, because I wasn’t. The lab was a disaster. I only managed to complete about a third of the exam. While configuring DMVPN, all of my routers locked up and crashed. I called the proctor over, and when he saw that the console ports were locked up, he started to accuse me of having made a configuration error. I explained that I hadn’t touched the console configuration, and just then we both saw bus errors appear on the console sessions followed by reloads. It was obvious then that I was not at fault. I had heard that if routers crash during CCIE exam, the proctor will give you your time back. However, the proctor admonished me to save my configs frequently, and refused to give me any time back. I had probably lost 15 minutes. I would have fought it, except that I was already so far behind on the exam,  I knew it would make no difference. Still, to this day am a bit angry at that proctor. As I left the exam room I looked at him and said “don’t even bother grading this.” He looked at me and said, “Oh, I’m sure you’re exaggerating.” I looked at him and told him I hadn’t completed two thirds of the exam.” Oh!” He exclaimed.” Well… Don’t wait six months for your next exam!”

… It was six months before my next attempt.

The author's 2008 CCIE security lab. The laptop ran Windows Server in a VM for ACS. An ASA 5505 is visible on top of the drawers.

The author’s 2008 CCIE security lab. The laptop ran Windows Server in a VM for ACS. An ASA 5505 is visible on top of the drawers.

Changes to my approach

I knew I had to revise my strategy. Something wasn’t working. The first thing I fixed was the lab situation. When I did Routing and Switching, I knew that I needed my own lab at home. Using remote rack rentals for security just didn’t give me enough time in the lab. I managed to get a hold of the PIX from a friend who was decommissioning it. I bought myself an ASA 5510, which, at $2500, was the most expensive piece of hardware I had. I really needed two of them, in order to cluster them, but I had to make do with the mismatched pair of the 5510 and 5505. As with the Routing and Switching exam, I knew I could use remote rack rentals to fill in for the equipment that I didn’t have. The ASA 5505 was adequate for basically everything except clustering. It had almost all of the capabilities of the 5510, but the configuration of VLANs was slightly different.  I also managed to acquire an IDS, and VPN 3000 series concentrator. I borrowed a laptop from work and got a Windows server license and managed to install Cisco Secure ACS. I ended up with a very complete lab.

I realized that a big part of my problem was that IPSec configurations are long, complicated, and counterintuitive. IPSec is the core of the CCIE security exam, and you need to know it as well as BGP and OSPF on the routing and switching exam. I made a series of diagrams which depicted each of the constituent configuration elements for the various IPSec technologies as blocks, which were then connected together by arrows. For example, for basic IPSec configuration, I would have one block representing the IKE configuration, and another representing the IPSec policy. I would draw an arrow to show how they were connected, labeling the arrow with the command used to connect them. Before I was trying to memorize these configurations. Now I was able to visualize them.

Visualizing complex configurations helps make them easy to understand and remember

Visualizing complex configurations helps make them easy to understand and remember

I also completely abandoned using the IE workbook. It just wasn’t ready at that point. Instead I invented my own VPN challenge lab. It had every kind of VPN on it: IPSec on ASA, IPSec on PIX, IPSec on VPN 3K, client IPSec on all of those platforms, L2TP, PPTP, DMVPN, SSL. I worked this lab over and over again until I could configure all of these automatically, and I made sure I configured between disparate platforms.

I felt good but not 100% prepared when I went to take my second attempt. I failed, but my score was much higher than before. I continued preparing for another month or so before taking my third attempt. I was so ready for my third attempt, that I completed the lab shortly after lunch. As I was coming out of the bathroom, I ran into Ted the proctor (not his real name), in the hallway. I had seen Ted on my second attempt and he told me he was attending a bluegrass Festival in San Francisco. I spent a good 15 minutes talking to Ted about the festival in the hallway, and I think at that point Ted realized that I was feeling pretty confident. Most people don’t spend 15 minutes shooting the breeze in the middle of the CCIE exam.

Interestingly enough, while Ted had been the most helpful Proctor on the Routing and Switching exam, he was of almost no help at all on the security exam. I’m not sure if he had changed in the intervening four years, or if he simply wasn’t as familiar with the security exams that I took. Either way, be prepared to make difficult decisions on your own in the lab, without the help of the proctors. Of all the questions I asked them, only once did I get a useful answer. I realize that their job is not to give away the test, but often the test is poorly written and I think that they need to be more helpful in explaining the exam.
Passing Routing and Switching was exciting; passing Security was a relief. I had almost given up after my disastrous first attempt. And I’m glad that I passed it when I did. As with the Routing and Switching exam, I passed Security in November. And as with the Routing and Switching exam, Cisco was changing the test at the beginning of the new year. The VPN 3000, PIX, NAC framework, and several other technologies were being removed. Of course, they never removed technologies without adding some as well. Had I failed my third attempt, it’s likely I would never have tried again.

In summary:

  • Having “always-on” access to a lab is critical!  Remote rack rental is good to fill in for a few things you might be missing, but don’t rely on it.
  • You may have to spend some more money than you want to acquiring gear, but it pays off.
  • The way you pass one CCIE exam is not necessarily the way you pass another exam.  You have to spend some time looking at the topics you will be covering, figuring out the best way to reach the point of automatic configuration of the technologies.
  • Sometimes, the study material from the vendors just won’t cut it.
  • Proctors aren’t always nice, and don’t always do what you thought they were supposed to.

I will cover the question of lab blueprint changes in a later article on the value of a CCIE, but it’s worth noting that for both my routing/switching and security exams, a blueprint change happened immediately after my passing.  I spent a lot of time studying, for example, the VPN 3000 concentrator which was already obsolete.  Still, I would have the same credential as a guy who passed the exam with the new technologies a couple months later.

Also worth noting:  I passed all of my expert exams (2 CCIEs and a JNCIE) in November.

In the next article in the series, Recertification Pain, I look at the biennial penance we all inherit for passing our CCIEs–the dreaded recertification.  I give my thoughts on improving the process, not that anybody is listening.

1

A CCIE Goes Home to Cisco

In this article in my “Ten Years a CCIE” series, I describe my experience going to work at Cisco as a CCIE.  Unlike many Cisco-employed CCIE’s, I earned my certification outside of Cisco.

A CCIE leads to a job at Cisco

I returned to my old job at the Chronicle and had my business cards reprinted with my CCIE number. I loved handing it out, particularly at meetings with telephone companies and Internet service providers whose salespeople were likely to know what such a certification meant.  I remember one such sales person, duly impressed, saying “wow, on that test you can be forced to configure any feature on any Cisco product…I don’t know how anyone could prepare for that!”  (Uh, right.  See “The CCIE Mystique“.)

At that time, the most popular forum for aspiring CCIE’s was an email distribution list called groupstudy.com. I had been a subscriber to this mailing list, but prior to passing my exam, I didn’t feel adequate to post anything there. However, once I passed, I began posting regularly, beginning with a summary of my test preparation process. One day I got an email from a mysterious CCIE who told me that I sounded like I knew what I was talking about, asking me if I wanted to interview for a job. I thought his name and number sounded familiar, and when I got home I confirm my suspicions by digging through my bookshelf. He was the author of one of my books about Catalyst Quality of Service.

A brutal interview

It turns out he was a manager at Cisco High Touch Technical Support, a group of TAC engineers who specialized in high profile customers. I scheduled an interview right away.

This interview was by far the most difficult I’ve had in my career. They brought me into a room with four CCIE’s, two of them double, all of them sharp. Each one of them had a different specialty. One of them was a security guy, another one was an expert on multicast, another was an expert on switching. When it came to Kumar, the voice guy, I figured I was scot-free. After all, I didn’t claim to know anything about voice over IP. Kumar looked over my resume, and then he looked up at me. “I see you have ISDN on your resume,” Kumar said. And then he began to grill me on ISDN. Darn, I should have thought of that!  Thankfully, I was well prepared.

Despite one or two mistakes in the interview, I got hired on and began my new job as a customer support engineer at HTTS. My first few months were in a group called ESO, which supported large enterprises and was very focused on Catalyst switching.  I won’t go into the details of the job here, but you can see my many TAC tales if you are interested.

Cisco's San Jose campus

Cisco’s San Jose campus

Little purple stickers everywhere!

One thing I quickly noticed when I got to Cisco was that a lot of the people in my department had a nickel-sized purple dot on their ID badges and cubicle nameplates. I found out that these purple dots were actually stickers with the CCIE logo. Cisco employees who had their CCIE’s stuck these purple dots on their badges and nameplates to show it off. Many of the CCIE’s who had passed multiple exams actually placed multiple dots on their badges and nameplates. I wanted one quite badly. The problem was, sheets of these purple stickers were sent out only to the early CCIE’s, and by the time I had passed, Cisco was no longer providing the sheets of stickers. I suppose I could’ve had some printed out, but I asked around looking for a CCIE who was generous to give me one of his dots. They were in scarce supply, however, and nobody was willing to part with one. It was just another way newer CCIE’s were getting jipped.

The real CCIE logo

The real CCIE logo

Even though the exam had now switched to the one day format, you still didn’t meet too many CCIE’s outside of Cisco. It was thus quite a shock when I went to Cisco and saw purple dots everywhere. It seemed like fully half of the people I was working with in my new job had CCIE’s. And many of them had low numbers, in the 2000’s and even in the 1000’s. I was quite relieved to find that they all treated me with total respect; nobody ever challenged me on account of my one-day CCIE. Still, I always had (and always will have) a great deal of respect for those people who passed the test when it was a two day test, and the cottage industry devoted to minting CCIE’s had not yet come into existence.

CCIE challenges customer

Customers were another story. I remember one BGP case in particular. I looked at the customer’s configuration and immediately realized that it was a simple matter of misconfiguration. I fired it up in my lab reproduced his configuration and proved to him that it was indeed a configuration error on his part. I wrote it all up in an email and proudly signed it with my CCIE number. Within a half an hour I got a call from the customer and one of his colleagues on the line. They proceeded to grill me rapidly on BGP asking me all sorts of questions that weren’t relevant to their case and stumping me several times. At that point I realized that when many people see you are a CCIE, they take it as a challenge. In some cases they failed the test themselves, or else they’ve met stupid CCIE’s in the past and they feel themselves to be on a mission to discredit all CCIE’s. After that episode, I removed my CCIE number from my email signature. I gained a feeling of self-importance after I passed my exam, but working among so many people with the same certification, and dealing with such intelligent customers, I realized that the CCIE didn’t always carry the prestige I thought it did.  The mystique diminished even further.

Incidentally, I became friends with all of the guys who interviewed me, and I was on the interview team myself during my tenure at Cisco. One extremely sharp CCIE we hired told me our interview was so tough he had to “hit the bottle” afterwards. It was considered a rite of passage at TAC to go through a tough interview, but I have gotten a lot nicer in my interview style now, having been on the receiving end of a few grillings.

The value of a CCIE

One of the later posts in this series will examine the question of the value of a CCIE certification.  After all, this is one of the most common questions I see in forums dedicated to certification.  However, my experience getting hired into Cisco (the first time) has some lessons.

  • The immediate reason I got hired was because of my experience and willingness to go out of my way helping others to get their CCIE on Groupstudy.  However, I would not have gotten the position without a CCIE, so clearly it proved its value there.
  • Once you are at Cisco, although people commonly display their stickers and plaques, having a CCIE certification will not necessarily distinguish you.
  • There are many CCIE’s who have made a bad impression on others, whether they are only book-knowledgeable, or even cheaters.  Often people challenge you when you have a CCIE, instead of respecting you.

In the next article in the series, Multiple CCIEs, Multiple Attempts, I describe passing the CCIE Security exam.  I talk about my experience suffering the agony of defeat for the first time, and how I eventually conquered that test.

Room of horrors: Inside the CCIE lab

In this article in my “10 Years a CCIE” series, I take you inside the infamous CCIE lab, where countless candidates have sweated out the devious challenges concocted by the CCIE exam authors.

Planning travel

I was fortunate at the time I took the lab exam in that I lived in San Francisco, very close to the San Jose test site. However, knowing the unpredictability of Bay Area traffic, and also knowing that the exam was very early in the morning (I am not a morning person) I decided it would be best to book a hotel room close to the test site. I even went so far as the book 2 nights in a hotel room, figuring that on the morning of the exam I wouldn’t want to deal with checking out of the hotel. This was perhaps excessive, but it made me relax and even for the well-prepared candidate your mental state is important.

The hotel on Great America Parkway where I stayed

The hotel on Great America parkway where I stayed

I often see test advice which says to get a good rest, to eat well, to not drink alcohol, etc., before your exam. Frankly, I always feel insulted when I see this advice. Of course I’m not going to get drunk the night before the exam. I don’t think anybody needs someone to tell them this. However, I do recommend planning your travel arrangements carefully to reduce stress on the day of the exam. I even drove the straight shot down Tasman the night before to check out the building where the exam was, just so that I would know my way. Read More »

1

How to pass the CCIE lab exam in one attempt

In this post in the Ten Years a CCIE series, I go over my preparations for the CCIE Routing and Switching exam, and what I did to pass in one attempt.

The first months…

I passed my CCIE Routing and Switching Lab in one attempt, so I think my approach can be considered effective. At least, it was for the exam at the time. I decided to spend my first several months of study diving deep into each of the exam topics on the blueprint. I was determined to focus on core technologies such as BGP and OSPF and to minimize the amount of time spent on ancillary topics such as DLSw. Because you have access to the documentation CD in the lab, you don’t need to know absolutely everything. However, you do not want to spend a long time trying to figure out how to configure core tasks which you should be able to do automatically.

I didn’t work from a particular manual or outline these first few months. Instead I would pick a topic, say BGP. I would go through all of the examples I could find in the books that I had, Jeff Doyle’s books being the most helpful. I would set up the examples from the books in my lab to see if they work as described. Then I performed free-form experimentation. I tried different things; I indulged my curiosity; I came up with new ways to test the protocols and tried to break them. I introduced loops where there weren’t loops in the examples I had. I saw what happened if I ran the protocol over ISDN instead of Frame Relay. And I made very sure that everything I learned I recorded in my notes. For every subject I kept two note files. The first file contained general, conceptual notes. The second file was a list of commands that I thought were important and I needed to remember. These files grew over time, and I studied them thoroughly before attempting the lab.

I had also acquired practice labs from three different sources. I had IP Expert’s lab book; I also had Internetwork Experts’ lab book; and finally, I had the Cisco press official lab book, which was written by a CCIE proctor. I found that this last book’s labs most closely resembled the real thing in terms of how the labs were written and how the diagrams were drawn. Still, as I studied I quickly came to favor the Internetwork Expert book for its thoroughness and accuracy. At that time, they were still relatively new, but the quality of their material was the best.

Closing in on test day…

In the last couple of months before the exam, I shifted my strategy. Instead of focusing on individual topics I spent my time working the practice labs in the IE book. At first I worked them slowly and methodically. I didn’t do them on a timer, and I didn’t rush through them. If it took me 24 hours to work through lab then it took me 24 hours. My main interest was in covering the material, understanding it thoroughly, and in documenting my learnings. I knew so many people who started giving themselves timed exams when they weren’t ready for them. Yes, it’s important to have a strategy and to understand clock management, but it’s far more important to understand the material thoroughly. The best time management strategy is knowing the material so well you can configure most of it on auto-pilot.

Every time I completed the lab I graded myself using IE’s answer key. I used to say that I was my own worst enemy. I never gave myself a pass on the slightest discrepancy between my solution and IE solution. Every single mistake that I wrote I listed out in a document, and in the last few weeks before the exam I reread that document several times every day. Constantly reviewing the mistakes I had made reinforced my own errors in my mind.  I also found that in my note documents that I was highlighting certain important points or gotchas with the capital words “BE SURE”. I created another document that I called my “BE SURE” list. I also reviewed this list several times a day in the last few weeks before the exam. Reviewing both my mistakes as well as my “BE SURE” list so frequently was quite effective in helping me remember my mistakes and important notes.

A snippet of my BE SURE list

A snippet of my BE SURE list

When I was studying for my CCIE exam Cisco press had just released two handy books. These books covered all of the commands in IOS at that time for BGP and OSPF. Not only did they describe the commands but they had examples of their use as well. In the last few days before the exam I would review the table of contents of these books which listed all the commands by name. I did this every night in bed. If I was able to accurately describe the command, I would cross it off.  Some commands that I couldn’t remember I saw night after night, until they were so familiar I had no problem using them.  Doing this every night helped me to commit fully to memory all of the different BGP and OSPF commands that make up the core of the CCIE lab exam.

I also took the CCIE Lab Boot Camp from Internetwork Expert just a few weeks before I took the actual lab exam. This was a wonderful experience. I was able to take the course from home, using IE’s Java-based virtual environment. Because most of the work and the class consisted of full, eight-hour timed labs, there was no need to travel to a classroom. And, because the eight hour exams were administered on Internetwork Expert’s own racks of equipment, there was no problem with not having a full CCIE lab at home. We had a small amount of lecture each day, followed by the eight hour lab, which was then graded each night. In the morning we were given our results. I was told that people scoring over 80% generally passed the CCIE lab exam, and I was scoring higher with no problem. The Brians gave me some great advice and particularly fixed some problems that I had in configuring multicast.

At the end of the boot camp Brian Dennis, the grumpier of the Brians, gave what I would charitably call a pep talk. He told us that a test is just a test, that we should get some of the classic books on networking and study them thoroughly, and that we should know our subject, not simply pass the test.  “You meet some CCIEs and wonder, how did this guy pass the test?” Brian said.

In November 2004 the time came to take the test. I had no idea if I was ready. A good friend of mine who passed shortly before spent four hours with me in a sushi restaurant grilling me on every possible subject that could be on the exam. They closed the restaurant on us.  For my final preparation, I studied all of the new features in IOS which they were now using in the CCIE lab. I also studied the documentation CD thoroughly so that I would have no trouble navigating it in the lab.

Passing the test

If you’re working on the CCIE exam, why should you care what someone did to prepare for it ten years ago?  Well, as I’ve said, it is a different test now.  My advice on learning ISDN dial maps isn’t going to help you.  However, there are some general principles here that you should pay attention to.

  1. Figure out the core topics and learn them well.  Cold.  On every expert exam, there are some core topics and some ancillary topics.  You cannot know everything.  Figure out the core topics and drill them over, and over, and over again.  You need to be able to configure them without thinking.
  2. Make things harder than they have to be.  As I said, break things intentionally.  Introduce problems.  Ask questions.  Don’t just run the scenarios you bought with your labs.
  3. Be your own worst enemy.  Remember, the CCIE exam is not just about doing what they tell you, but doing exactly what they tell you.  When you grade yourself, read and re-read the tasks.  Make absolutely sure that you have accurately and completely fulfilled the requirements.
  4. Document your mistakes.  Review things you have done wrong, and keep reviewing them.

In the next post in the series,  Room of Horrors, I describe the CCIE lab experience.  I talk about what it was like to enter the infamous lab in Cisco Building C, and take the challenging exam.

In those days, you had to build a lab…

In this article in my series, Ten Years a CCIE, I discuss the challenge of building and maintaining a lab, and the question of building versus not building a physical lab.

Acquiring Gear

As I mentioned in a previous post, GNS3 was not available at the time I took the CCIE exam. This meant that the CCIE candidate had to acquire hardware — quite a lot. This was one way that the number of CCIE’s was limited, even after the institution of the one-day format. I acquired equipment by various means; mostly by purchasing old routers off eBay. Some I was able to borrow from work. Some I bought directly from coworkers.

My lab consisted mostly of 2500 series routers. These were adequate to test most of the routing protocol features needed for the exam. However, some of the newer or more advanced features required more advanced routers, and for these I used 2600 series routers. These were essential for testing Voice over IP, as a 2500 did not support the hardware (FXS/FXO cards) required to do dial peers. In 2004, Frame Relay was still a big part of the CCIE exam, so I decided that a Frame Relay switch was absolutely necessary. For this I acquired a 2500 series router with a lot of serial ports. It was fortunate that I didn’t need to purchase a WAN emulator of some sort, since Cisco included Frame Relay switching capability in its IOS software and WAN emulators were ridiculously expensive.

Early on I determined that a terminal server was necessary. I started out thinking it wasn’t, but moving your console cable around from router to router gets old very quickly. Luckily, a 2500 series router with a lot of serial ports, designed primarily for modems, made an excellent terminal server. Instead of having to console individually to each router, I could simply telnet to the terminal server on a particular TCP port number to access the individual devices.

Ethernet switching was a bit more difficult. At that time, the CCIE equipment list had Catalyst 3550 switches on it. The 3550s were quite expensive and I couldn’t afford any. Luckily, we had a spare at work which I borrowed, but only one. This meant that for spanning tree and trunking configuration, I had to configure one end only and hope that the configuration would work if there were a switch on the other end.  Building a physical lab often requires compromises, since it can be impossible to acquire or afford all the equipment.

ISDN and ATM

ISDN was a big part of the CCIE exam in the early 2000’s, even though it was becoming obsolete. It was also considered one of the more difficult subjects because of the myriad of ways of configuring it. I felt that it necessary to have some way to configure ISDN in my home lab. ISDN simulators were available, but were extremely expensive, and out of my price range. Some other CCIE candidates suggested to me that I purchase ISDN service for my apartment from the telephone company. This seemed to me like a huge hassle and it was also quite expensive. Luckily at just the time I started studying for this exam a small company came out with an inexpensive ISDN simulator. As I recall, it was only a few hundred bucks. I bought one, and for the most part it worked great. The one problem was that the on/off switch was labeled backwards. Easily fixed with a sticker.

The author's 2004 lab. Not neat! Note the but sets for VoIP testing, the 3000-series lying vertically on the right, and the small black ISDN sim on the table, center-front.

The author’s 2004 lab. Not neat! Note the butt sets for VoIP testing, the 3000-series lying vertically on the right, and the small black ISDN sim on the table, center-front.

I had no way of simulating ATM. Doing ATM required a Lightstream switch and expensive interface cards. As I mentioned earlier, ATM was a very small part of the exam. It simply didn’t seem worth it to spend a huge amount of money to configure a single interface.

Why a home lab?

Just because I didn’t have certain technologies available to me doesn’t mean I didn’t study them. At that time, there were a number of companies offering CCIE rack rentals (an example is here.)  These were either small entrepreneurs or some of the larger CCIE test prep companies, who had set up full-blown CCIE labs and rented out time on them. There were two or three that I used regularly to fill in on technologies I didn’t have, including ATM.

Why did I spend thousands of dollars on lab equipment for my home lab when I could have simply rented time from a rack rental company? This question will come up again when I discuss the CCIE security exam. Working with the remote labs was a pain. Every time you started up, you had to load your configurations back into the lab because someone else had been using it in the interim. When the time came to shut down your lab, you had to back everything up. A huge chunk of your paid-for time slot had to be devoted to config management.  In second place, you could only access the lab during certain scheduled slots that you had purchased. I felt then, and I was right, that being held to someone else’s schedule would be a major impediment to serious study. I therefore decided to build my own lab, and use rack rentals only to fill in technologies I didn’t have. This meant that whenever I had spare time I could flip on the equipment and study. I even set up remote access to my lab and left it running during the daytime, so that I could study in my spare moments at work. Any time I wanted, I could access my lab and the configurations I was working on would be right where I left them.  This was essential to my passing the exam.

The CCIE lab had six routers on it that time, and connected to your lab were three backbone routers. These backbone routers out of your control, but were simply route injectors. I had enough routers to simulate the core of the lab, but I did not have enough routers to simulate all of the backbone. I therefore had to get creative. I used my frame relay switch as well as my term server as backbone routers as well. Thus, they were doing double duty. The configuration was relatively easy with the FR switch since it was only doing layer 2 in the lab, but it got tricky with the term server, since I couldn’t change its IP address. The IP address of the term server was the address I telnetted to to access my lab.  So, I often ended up changing IP addresses in the scenarios I had purchased, so that I could use the access server as a backbone router. I also used an ancient 3000-series router running IOS 9 as a BB router. (IOS 12.1 was current at that time!)  It was hard to configure but it could do OSPF and BGP.

Maintaining a home lab was a lot of work. I had to re-cable it a number of times until I finally got the stable topology that I liked. I often had to deal with bent pins which were so common with the serial cables of the time. I had to do IOS upgrades and the occasional memory upgrade when I needed an IOS but didn’t have the memory to support it. Sometimes, this work distracted from the work of studying. However, all of it was valuable experience. I don’t think I would trade it for GNS3 any day. GNS3 certainly adds an element of convenience, and allows the student to focus on the lab and not on extraneous tasks. And, it certainly has some challenges of its own. But I do think that dealing with the physical hardware, upgrading it, cabling it, etc. — helped me to grow as a network engineer, and also improve my general troubleshooting skills. Of course, as I mentioned before, the expense and difficulty of working with physical equipment prevented a number of people from attempting the lab, thus keeping the number of CCIE’s lower than in the GNS3 era.

Modern candidates for R/S may be able to use GNS3, but candidates for the other exams, such as data center, will go through the same struggles as myself.  Data Center is particularly bad because the equipment required is so expensive.  It is almost impossible to procure it on your own.  Rack rental is one option, but is fraught with problems, as I mentioned above.  The only other option is to try to get your employer to help you build and pay for a lab.  Despite the many changes in the exam, some things remain the same I suppose!

My next article, How to Pass the CCIE Lab Exam in One Attempt, reviews my study and preparation for the Routing and Switching lab exam.

Routing and Switching: An exam in flux

In the second article in the “Ten Years a CCIE” series, I discuss the Routing and Switching written exam, and the changes to the CCIE exam in the early 2000s.

Passing the written

As was most common in the early 2000’s I attempted my Routing and Switching exam first. Having passed the CCNP exams, my first order of business was to study for the CCIE written exam. Even though I had heard the CCIE written exam was quite easy, I approached it much like I approached a graduate school research project. I didn’t care how easy it was, I wanted to learn the subject material well. I also didn’t want to fail. I had not yet failed a Cisco exam, and I made a pledge with myself: the first Cisco exam I will allow myself to fail will be the CCIE lab. Read More »