Skip navigation

When I was still a new engineer, a fellow customer support engineer (CSE) asked a favor of me. I’ll call him Andy.

“I’m going on PTO, could you cover a case for me? I’ve filed a bug and while I’m gone there will be a conference call. Just jump on it and tell them that the bug has been filed an engineering is working on it.” The case was with one of our largest service provider clients. I won’t say which, but they were a household name.

When you’re new and want to make a good impression, you jump on chances like this. It was a simple request and would prove I’m a team player. Of course I accepted the case and went about my business with the conference call on my calendar for the next week.

Before I got on the call I took a brief look at the case notes and the DDTS (what Cisco calls a bug.) Everything seemed to be in order. The bug was filed and in engineering’s hands. Nothing to do but hop on the call and report that the bug was filed and we were working on it.

I dialed the bridge and after I gave my name the automated conference bridge said “there are 20 other parties in the conference.” Uh oh. Why did they need so many?

After I joined, someone asked for introductions. As they went around the call, there were a few engineers, several VP’s, and multiple senior directors. Double uh oh.

“Jeff is calling from Cisco,” the leader of the call said. “He is here to report on the P1 outage we had last week affecting multiple customers. I’m happy to tell you that Cisco has been working diligently on the problem and is here to report their findings and their solution. Cisco, take it away.”

I felt my heart in my throat. I cleared my voice, and sheepishly said: “Uh, we’ve, uh, filed a bug for your problem and, uh, engineering is looking into it.”

It was dead silence, followed by a VP chiming in: “That’s it?”

I was then chewed out thoroughly for not doing enough and wasting everyone’s time.

When Andy got back he grabbed the case back from me. “How’d the call go?” he asked.

I told him how it went horribly, how they were expecting more than I delivered, and how I took a beating for him.

Andy just smiled. Welcome to TAC.

My job as a customer support engineer (CSE) at TAC was the most quantified I’ve ever had.  Every aspect of our job performance was tracked and measured.  We live in the era of big data, and while numbers can be helpful, they can also mislead.  In TAC, there were many examples of that.

Take, for example, our customer satisfaction rating, known as a “bingo” score.  Every time a customer filled out a survey at the end of a TAC case, the engineer was notified and the bingo score recorded and averaged with all his previous scores.  While this would seem to be an effective measure of an engineer’s performance, it often wasn’t.

In TAC, we often ended up taking cases that were “requeues.”  These were cases that were previously worked by another engineer.  Imagine you got a requeue of a case that another CSE had handled terribly.  You close the case quickly, but the customer is still angry at the first CSE, so he gives a low bingo score.  That score was credited against the CSE who closed the case, so even though you took care of it, you got stuck with the low numbers.

This also happened with create-to-close numbers.  We were measured on how quickly we closed cases.  Imagine another CSE had been sitting on a case for six months doing nothing.  The customer requeues it, and you end up with the case, closing it immediately.  You end up with a six month create-to-close number even though it wasn’t your fault.

Even worse, if you think about it, the create-to-close number discouraged engineers from taking hard cases.  Easy cases close quickly, but hard ones stay open while recreates are done and bugs are filed.  The engineers who took the hardest cases and were very skilled often had terrible create-to-close numbers.

The bottom line is that you need more than data to understand a person.  Most things in life don’t lend themselves to easy quantification.  Numbers always need to be in context.  Corporate managers are obsessed with quantification, and the Google’s of the world are helping to drive our number-love even further.  Meanwhile, reducing people to numbers is a great way to treat them less humanly.

Introduction

I’ve been side-tracked for a while doing personal articles, so I thought it would be a good time to get back to some technical explanations.  Seeing that I work for Cisco now, I thought it would be a good time to cover some Cisco technology.  My focus here has been on programmability and automation.  Some of this work has involved using tools like Puppet and Ansible to configure switches, as well as Python and NETCONF.  I also recently had a chance to present BRKNMS-2002 at Cisco Live in Las Vegas, on LAN management with Data Center Network Manager 10.  It was my first Cisco Live breakout, and of course I had a few problems, from projector issues to live demo failures.  Ah well.  But for those of you who don’t have access to the CL library, or the time to watch the breakout, I thought I’d cover an important DCNM concept for you here on my blog.

Read More »

When I first started at TAC, I wasn’t allowed to take cases by myself.  If I grabbed a case, I had to get an experienced engineer to help me out.  One day I grabbed a case on a Catalyst 6k power supply, and asked Veena (not her real name) to help me on the case.

We got the customer on the phone.  He was an engineer at a New York financial institution, and sounded like he came from Brooklyn.  I lived in Williamsburg for a while with my mom back in the 1980’s before it was cool, and I know the accent.  He explained that he had a new 6k, and it wasn’t recognizing the power supply he had bought for it.  All of the modules had a “power denied” message on them.

I put the customer on speaker phone in my cube and Veena looked at the case notes.  As was often the case in TAC, we put the customer on mute while discussing the issues.  Veena thought it was a bad connection between the power supply and the switch.

“Here’s what I want you to do,” Veena said to the customer, un-muting the phone.  “I used to work in the BU, and a lot of times these power supplies don’t connect to the backplane.  You need to put it in hard.  Pull the power supply out and slam it in to the chassis.  I want to hear it crack!”

The customer seemed surprised.  “You want me to do what?!” he bristled.

“Slam it in!  Just slam it in as hard as you can!  We saw this in the BU all the time!”

“Hey lady,” he responded, “we paid a couple hundred grand for this box and I don’t want to break it.”

“It’s ok,” she said, “it’ll be fine.  I want to hear the crack!”

“Well, ok,” he said with resignation.  He put the phone down and we heard him shuffle off to the switch.  Meanwhile Veena looked at me and said “Pull up the release notes.”  I pulled up the notes, and we saw that the power supply wasn’t supported in his version of Catalyst OS.

Meanwhile in the background:  CRACK!!!

The customer came back on the line.  “Lady, I slammed that power supply into the chassis as hard as I could.  I think I broke something on it, and it still doesn’t work!”

“Yes,” Veena replied.  “We’ve discovered that your software doesn’t support the power supply and you will need to do an upgrade…”

Note:  This article was originally posted in 2016.  Since that time, the CCIE program has changed the process for earning a CCIE, and the separate written exam is no longer used.  This means that the problem of people claiming to be a “CCIE” when they have only passed the written exam is no longer the case.  I’m leaving the article as is for now, but will modify it in the future when I have time, to reflect the new circumstances.  Regardless, you should never claim you have a certification when you have only passed a part of the requirements.  (ccie14023, Feb 2020)

In this article in the “Ten Years a CCIE” series, I look at the question of cheating.  Is it possible to cheat on the CCIE exam?  And what does cheating do to the value of the certification?

Yes, you can cheat on the CCIE

Shortly after I passed my Security exam I spoke with the first CCIE to pass the Voice exam. He took a beta version while he worked at Cisco. I commented that I valued my CCIE so much because it was simply impossible to cheat at the exam. It wasn’t a written exam; you couldn’t just walk in knowing the answers; you had to think on your feet. He laughed at me and explained my ignorance.

A lot of people cheat on the CCIE lab exam, he said. Either they work in groups and share the contents of the exams  they’ve seen, or else they get copies of the exam from unscrupulous vendors on the Internet. Then, having seen the actual exam, and having researched the difficult problems, and configured it several times in their lab, they can pass with ease.

I was quite shocked to hear this. I had always studied alone, and when I started down the CCIE road, I didn’t just want to pass the exams, I wanted to beat them. I didn’t just want the CCIE, I wanted the CCIE mystique. I was flabbergasted that people would want the certification without the work. Of course there is a great appeal to gaining something so valuable with minimal effort, but how are you going to make it through a job interview?

The stupidity of cheating

I had encountered rampant cheating in graduate school. This was at the dawn of the Internet, and I saw that many of my fellow students ripped off entire papers from the Internet. We used to send our papers to each other via email, and occasionally I would paste a snippet into AltaVista (Google not being available yet), and often I would hit upon the original work that they’d stolen. Leaving aside the ethical issues of stealing someone else’s work, or ripping off the questions and answers for an exam, there is a practical downside to cheating. You are claiming a credential that you haven’t earned. I remember conducting a job interview of a girl with a Masters degree from the same program as myself. I asked her the subject of one of her papers, and it had to do with routing protocols. She couldn’t answer even the most basic questions about the content of her paper. It was obvious that she cheated her way through the program. And she looked like a complete fool claiming to be a “master” of a subject about which she knew nothing.

Sometimes engineers I know roll their eyes when they hear I have a CCIE. They have encountered one of my fellow “experts” only to find that he seemed hardly an expert at all. Since I know that it’s possible to cheat on this exam, I’m convinced that many of these so-called CCIE’s cheated on their exams. They look like fools as did the girl with her Masters degree.

I’ll talk more about the value of the certification and later post, but one thing to keep in mind is that there’s great value in the study process. There’s great value in learning. And if you study for the test as you are supposed to study for it, you’re guaranteed to learn a lot.

A blurry ethical line?

I, like almost everybody these days, passed my exams using material from legitimate vendors, primarily Internetwork Expert and IPExpert.  (The latter has closed their doors.)  These vendors provide quite a lot of material, but their signature product is a book of sample exams designed to prepare you for the real thing.  This brings up a question.  Presumably some of the scenarios covered by the “legitimate” vendors are scenarios that might come up on the real lab.  After all, how many ways are there to configure BGP?

Interestingly enough, any exam has to provide a certain amount of information to test-takers beforehand.  The CCIE exams have detailed blueprints which guide candidates in their studies.  It would be impossible to take and pass an exam without such advance information.  With merely a blueprint in hand, it would be possible to construct some kind of sample exam, but do vendors simply build them off the blueprint?  Or do they get information from candidates and use them to build their tests?

The ethical lines can be blurry, but one thing is for certain:  studying for a CCIE exam using an actual copy of a real test is blatant cheating and disgraceful behavior.

Cheating on the written exam

Cheating is also rampant on the written exam. This is even the case among CCIE’s who are recertifying, perhaps especially the case. As I mentioned in my recertification post, taking an exam every two years, especially a hard one, is a big hassle. Many CCIE’s get lazy about the process. There are vendors who will sell verbatim copies of the tests. There is still, of course, some work involved. Someone with a copy of the test has to actually memorize all the answers. But it is far easier than going into a test blind.

My last re-certification was quite painful, and yet I refuse to use any sort of brain dump.  Instead, I built an Anki database of questions.  It wasn’t perfect, and it took a couple of failures for me to build a database that had sufficient coverage.

False CCIEs

Another way to “cheat” is simply to falsely claim CCIE status.  Anyone with a CCO account can verify if somebody actually has a CCIE, and whether they are active, but oftentimes employers just don’t bother to check.  When I was at Cisco HTTS, we were very close to hiring someone for a CCIE-requiring position, when I ran his name through the tool.  His CCIE had been revoked because he hadn’t recertified.  He was clearly embarrassed, and had simply been too busy to recertify.  While I can empathize with that, the fact was that he did not have a CCIE and would need to take both written and lab to get it back.  We didn’t hire him, but it amazed me that nobody had bothered to check early in the hiring process.

There is also a large group of imposters who have passed the written, and somehow think this qualifies them to put “CCIE” on their resume.  I recently saw a poster on a LinkedIn group who gave herself a CCIE (Wrt) title.  I also remember one candidate who put “CCIE Routing/Switching” in huge, bold letters on his resume, with “written” in a tiny font right next to it.  Well, I have news for you.  There is no CCIE written certification.  You either have a CCIE or you don’t.  Pass the lab before you put CCIE anything on your resume.  If you are looking at an employer that is willing to sponsor you for the lab, then by all means, tell them you passed the written.  But don’t claim CCIE status without a number.

What is the value?

We all know that there are a number of CCIE’s out there who should not have the certification.  It reflects poorly on the CCIE community.  There is no question that whatever value the CCIE has is diminished by those who obtained their credential through fraud.  If you are frustrated with the exam and thinking of hunting for a brain dump, remember this:  if you can’t pass the exam, you have no right to call yourself a CCIE.

In my next and final article in the series, The Value of a CCIE, I will take a look at the value of the credential.  Ten years later, do I think it was worth it?  Would I recommend someone take the CCIE exam now?  What do I think the future is for network experts in the world of SDN and automation?

Note:  This article was written in 2016 and has not been modified.  A number of changes have been made to the CCIE program which have dramatically improved the re-certification process.  Continuing education is now an option, as I suggest in this article.  The re-certification frequency has been reduced.  I may modify this article in the future, but I am leaving it as is for now for historical purposes.  However, please note the description of the process is no longer accurate at all.  (ccie14023, Sept 2021)

 

In this installment of “Ten Years a CCIE,” I look at what you have to do to stay certified, and the difficulty of maintaining your credential.

Passing your CCIE gives you a great feeling of accomplishment, and also a sense of relief.  You’ve spent months studying and late nights configuring scenarios in the lab.  Maybe you took the exam multiple times, and had to experience the letdown of knowing that, instead of being finished, you had more months of studying ahead.  So, you’ve finally passed, and it’s all over, right?

No, unfortunately.  You have a CCIE, but if you want to keep it, you have to worry about hitting the books again every two years.  All CCIE’s have to re-certify, a biennial ritual that becomes harder as the years go by.

Here’s how it works.  Before two years after your lab date, you have to re-certify your CCIE by passing a CCIE written exam.  You can take any written exam, just as long as it is a CCIE written.  For example, if you passed Routing and Switching, you could recertify by taking the Data Center written exam.  This has the advantage of simultaneously qualifying you for another lab exam, if you are so inclined.  If you have more than one CCIE, you can recertify all of them by taking any CCIE written.  For example, if you have Routing/Switching, ISP Dial, and Collaboration CCIEs, you could recertify all of them at once by taking the Wireless written.  This holds true even though ISP Dial is no longer a valid certification.  Even if you only have a certification that no longer exists (such as ISP Dial or SNA IP), you can maintain active CCIE status by passing any written exam.

If you don’t pass a written exam, at the two year mark your certification becomes suspended.  You can no longer use your CCIE number in your signature or claim to be a CCIE.  You can still pass the recert exam within a year, but if a year elapses after you go suspended, you lose your CCIEs, all of them, and have to retake both written and lab for any CCIE you hold.  Needless to say, you don’t want that to happen.

recert

What you want to see when you verify your CCIE…

(For comparison, my JNCIE-SP expires every three years, and I have to take the JNCIP-SP exam to recertify.  If I had a JNCIE-ENT as well, I would have to take both exams to recertify.)

If you just passed your lab exam and you feel super-confident, you may think you don’t have to worry about a measly written exam in two years.  However, any CCIE will tell you the recertification ritual is onerous and a huge waste of time.  As your career advances, you will often find yourself doing less and less CLI, and you might in fact work less with Cisco products.  In my case, re-certifying became especially painful during my six years at Juniper.

It would be less of a burden if the exams were better written.  The last time I took the written, there was a question that was flat out wrong, and many that were just obscure.

I first wrote this entry in 2014, and I am now re-writing it two years later.  When I first wrote it, I was working on my recert and in a state of extreme annoyance, came up with a couple of sample questions intended to mimic the actual exam:

When is the MSDP ConnectRetry timer used?
a.  When the MSDP peer with the highest IP address transitions from the INACTIVE to the CONNECTING state.
b.  When the MSDP peer with the lowest IP address transitions from the CONNECTING to the ESTABLISHED state.
c.  When the MSDP peer with the lowest IP address transitions from the INACTIVE to CONNECTING state.
d.  When the MSDP peer with the highest IP address transitions from the CONNECTING to the ESTABLISHED state.

What is the RSVP message type for a PathTear message?
a. 4
b. 0
c. 5
d. 3

What does the “ipv6 mld limit 100″ command do?
a.  Limits the number of hosts that multicast listener discovery can discover to 100
b.  Limits the hosts permitted by MLD to those contained in ACL 100
c.  Limits the number of MLD states to 100 on a per-interface basis.
d.  Limits the number of MLD states to 100 globally.

At the time I wrote them, these questions were technically within the blueprint topics for the Routing and Switching written exam, but they are obviously rather stupid questions.  The R&S blueprint is so huge that it is essentially impossible to know all of the subjects it covers.  Nevertheless, I was encountering questions of roughly this level of obscurity on the exam.

The purpose of recertification

Why do we have to recertify?  Obviously, the main reason is to ensure CCIE’s stay current in the field.  I passed routing/switching back in 2004, and a lot has changed in 12 years.  It’s important that people who come to me for expertise believe that I actually have relevant knowledge.

We have to ask a question though:  how well do you stay up-to-date taking a written exam every two years?  And why can you keep your credential when you re-certified in a different track?

For example, if someone acquired a CCIE Security certification back in 2002, but re-certified for 14 years using the routing/switching written, why is that engineer qualified to continue calling himself a “CCIE Security”?  He probably knows nothing of modern security technologies.  Juniper requires JCNIE’s to recertify in each track they have certified, so a triple JNCIE has to take three separate exams.  While this is painful (and kept me to one JNCIE), it makes more sense.

I think an even more reasonable approach is to allow continuing education in lieu of a test.  This is the requirement for CISSPs, lawyers, and even doctors, and it makes a lot of sense.  I never remember much from the recert exams, but a couple days of training would be a great way to get current.

I do think Cisco was smart to introduce the Emeritus option.  CCIE Emeritus allows CCIE’s who have hit the 10 year mark to pay a fee to keep their number in a non-active status indefinitely, with the option to recertify.  Many CCIEs reach a point where they don’t deal with day-to-day CLI configuration, and find the exams harder and less relevant to their careers.  Several of my friends have chosen this option.  I almost did when I worked at Juniper, but I am thankfully still current.

By the way, the answer to all of the above questions is ‘C’.

In my next article, Cheaters, I look at the question of whether people cheat on the CCIE exam, and the effect it has on the value of the certification.

I’ve come back to Cisco recently, and I think I can say that I haven’t worked this hard since the last time I was at Cisco.  I remember my first manager at TAC telling me in an interview that “Cisco loves workaholics.”  In an attempt to get more organized, I’ve been taking a second crack at using OmniFocus and the GTD methodology.  To be honest, I haven’t had much luck with these systems in the past.  I usually end up entering a bunch of tasks into the system, and then quickly get behind on crossing them off.  I find that the tasks I really want to do, or need to do, I would do without the system, and the ones that I am putting off I keep putting off anyway.  I have so much to do now, however, that I need to track things more efficiently and I am hoping OmniFocus is the solution. Read More »

In this article in my “Ten Years a CCIE” series, I look at passing the Security exam in 2008.  I get to experience the agony of failure for the first time, and have to re-tool my strategy.

Goodbye to Cisco

I worked two long years at Cisco. Two very long years. I learned so much there but it was a brutal job. The relentless flood of new and challenging cases grew tiresome.  When my aforementioned sushi eating CCIE friend called me in 2007 and invited me to come join him at a Gold partner I couldn’t say no.  Cisco sells much of its gear through value added resellers (VARs), also known as partners.  These partners are assigned different levels depending on the amount of business they do, and Gold is the highest.

Working at a gold partner with a CCIE was quite enjoyable. Gold partners need CCIE’s and so they have a lot of incentive to make you happy. My boss suggested that I get a second CCIE, this time in voice. I started to buy material for the voice exam, when my VP showed up in the office, fired my boss, and told me to start studying for the security exam. (His firing of my boss had nothing to do with my CCIE exam, but it certainly made me stand up and listen to what he was asking.) So, having really not started on voice, I switched immediately to security.

I had already passed the security written back at Cisco partly to qualify for the lab exam, and partly to re-certify my existing CCIE, so it was straight to the lab exam for me. The equipment list was a big challenge. At that time, you needed two ASA’s, one PIX, a VPN 3000 series concentrator, and IDS device, six routers and two switches, and some sort of Windows server running Cisco secure ACS. I still had my old lab equipment from before, but I was missing everything else. I had one ASA 5505 from work, but no other security devices. I decided that the cost was too prohibitive for me to set up my own lab. I was going to have to use rack rentals. That was my first big mistake.

I decided to approach the exam in exactly the same way I approached the routing and switching exam. I studied the various subjects on the blueprint individually, and then started doing full labs from the Internetwork Expert workbook. As great as IE’s workbook was for routing and switching, in 2008 it really wasn’t very good for security. I have a lot of respect for the Bryans, and I’m sure it’s come a long way, but at that time it just wasn’t enough.

Attempt number one

When I showed up at the familiar CCIE lab, I didn’t feel well prepared, because I wasn’t. The lab was a disaster. I only managed to complete about a third of the exam. While configuring DMVPN, all of my routers locked up and crashed. I called the proctor over, and when he saw that the console ports were locked up, he started to accuse me of having made a configuration error. I explained that I hadn’t touched the console configuration, and just then we both saw bus errors appear on the console sessions followed by reloads. It was obvious then that I was not at fault. I had heard that if routers crash during CCIE exam, the proctor will give you your time back. However, the proctor admonished me to save my configs frequently, and refused to give me any time back. I had probably lost 15 minutes. I would have fought it, except that I was already so far behind on the exam,  I knew it would make no difference. Still, to this day am a bit angry at that proctor. As I left the exam room I looked at him and said “don’t even bother grading this.” He looked at me and said, “Oh, I’m sure you’re exaggerating.” I looked at him and told him I hadn’t completed two thirds of the exam.” Oh!” He exclaimed.” Well… Don’t wait six months for your next exam!”

… It was six months before my next attempt.

The author's 2008 CCIE security lab. The laptop ran Windows Server in a VM for ACS. An ASA 5505 is visible on top of the drawers.

The author’s 2008 CCIE security lab. The laptop ran Windows Server in a VM for ACS. An ASA 5505 is visible on top of the drawers.

Changes to my approach

I knew I had to revise my strategy. Something wasn’t working. The first thing I fixed was the lab situation. When I did Routing and Switching, I knew that I needed my own lab at home. Using remote rack rentals for security just didn’t give me enough time in the lab. I managed to get a hold of the PIX from a friend who was decommissioning it. I bought myself an ASA 5510, which, at $2500, was the most expensive piece of hardware I had. I really needed two of them, in order to cluster them, but I had to make do with the mismatched pair of the 5510 and 5505. As with the Routing and Switching exam, I knew I could use remote rack rentals to fill in for the equipment that I didn’t have. The ASA 5505 was adequate for basically everything except clustering. It had almost all of the capabilities of the 5510, but the configuration of VLANs was slightly different.  I also managed to acquire an IDS, and VPN 3000 series concentrator. I borrowed a laptop from work and got a Windows server license and managed to install Cisco Secure ACS. I ended up with a very complete lab.

I realized that a big part of my problem was that IPSec configurations are long, complicated, and counterintuitive. IPSec is the core of the CCIE security exam, and you need to know it as well as BGP and OSPF on the routing and switching exam. I made a series of diagrams which depicted each of the constituent configuration elements for the various IPSec technologies as blocks, which were then connected together by arrows. For example, for basic IPSec configuration, I would have one block representing the IKE configuration, and another representing the IPSec policy. I would draw an arrow to show how they were connected, labeling the arrow with the command used to connect them. Before I was trying to memorize these configurations. Now I was able to visualize them.

Visualizing complex configurations helps make them easy to understand and remember

Visualizing complex configurations helps make them easy to understand and remember

I also completely abandoned using the IE workbook. It just wasn’t ready at that point. Instead I invented my own VPN challenge lab. It had every kind of VPN on it: IPSec on ASA, IPSec on PIX, IPSec on VPN 3K, client IPSec on all of those platforms, L2TP, PPTP, DMVPN, SSL. I worked this lab over and over again until I could configure all of these automatically, and I made sure I configured between disparate platforms.

I felt good but not 100% prepared when I went to take my second attempt. I failed, but my score was much higher than before. I continued preparing for another month or so before taking my third attempt. I was so ready for my third attempt, that I completed the lab shortly after lunch. As I was coming out of the bathroom, I ran into Ted the proctor (not his real name), in the hallway. I had seen Ted on my second attempt and he told me he was attending a bluegrass Festival in San Francisco. I spent a good 15 minutes talking to Ted about the festival in the hallway, and I think at that point Ted realized that I was feeling pretty confident. Most people don’t spend 15 minutes shooting the breeze in the middle of the CCIE exam.

Interestingly enough, while Ted had been the most helpful Proctor on the Routing and Switching exam, he was of almost no help at all on the security exam. I’m not sure if he had changed in the intervening four years, or if he simply wasn’t as familiar with the security exams that I took. Either way, be prepared to make difficult decisions on your own in the lab, without the help of the proctors. Of all the questions I asked them, only once did I get a useful answer. I realize that their job is not to give away the test, but often the test is poorly written and I think that they need to be more helpful in explaining the exam.
Passing Routing and Switching was exciting; passing Security was a relief. I had almost given up after my disastrous first attempt. And I’m glad that I passed it when I did. As with the Routing and Switching exam, I passed Security in November. And as with the Routing and Switching exam, Cisco was changing the test at the beginning of the new year. The VPN 3000, PIX, NAC framework, and several other technologies were being removed. Of course, they never removed technologies without adding some as well. Had I failed my third attempt, it’s likely I would never have tried again.

In summary:

  • Having “always-on” access to a lab is critical!  Remote rack rental is good to fill in for a few things you might be missing, but don’t rely on it.
  • You may have to spend some more money than you want to acquiring gear, but it pays off.
  • The way you pass one CCIE exam is not necessarily the way you pass another exam.  You have to spend some time looking at the topics you will be covering, figuring out the best way to reach the point of automatic configuration of the technologies.
  • Sometimes, the study material from the vendors just won’t cut it.
  • Proctors aren’t always nice, and don’t always do what you thought they were supposed to.

I will cover the question of lab blueprint changes in a later article on the value of a CCIE, but it’s worth noting that for both my routing/switching and security exams, a blueprint change happened immediately after my passing.  I spent a lot of time studying, for example, the VPN 3000 concentrator which was already obsolete.  Still, I would have the same credential as a guy who passed the exam with the new technologies a couple months later.

Also worth noting:  I passed all of my expert exams (2 CCIEs and a JNCIE) in November.

In the next article in the series, Recertification Pain, I look at the biennial penance we all inherit for passing our CCIEs–the dreaded recertification.  I give my thoughts on improving the process, not that anybody is listening.

In this article in my “Ten Years a CCIE” series, I describe my experience going to work at Cisco as a CCIE.  Unlike many Cisco-employed CCIE’s, I earned my certification outside of Cisco.

A CCIE leads to a job at Cisco

I returned to my old job at the Chronicle and had my business cards reprinted with my CCIE number. I loved handing it out, particularly at meetings with telephone companies and Internet service providers whose salespeople were likely to know what such a certification meant.  I remember one such sales person, duly impressed, saying “wow, on that test you can be forced to configure any feature on any Cisco product…I don’t know how anyone could prepare for that!”  (Uh, right.  See “The CCIE Mystique“.)

At that time, the most popular forum for aspiring CCIE’s was an email distribution list called groupstudy.com. I had been a subscriber to this mailing list, but prior to passing my exam, I didn’t feel adequate to post anything there. However, once I passed, I began posting regularly, beginning with a summary of my test preparation process. One day I got an email from a mysterious CCIE who told me that I sounded like I knew what I was talking about, asking me if I wanted to interview for a job. I thought his name and number sounded familiar, and when I got home I confirm my suspicions by digging through my bookshelf. He was the author of one of my books about Catalyst Quality of Service.

A brutal interview

It turns out he was a manager at Cisco High Touch Technical Support, a group of TAC engineers who specialized in high profile customers. I scheduled an interview right away.

This interview was by far the most difficult I’ve had in my career. They brought me into a room with four CCIE’s, two of them double, all of them sharp. Each one of them had a different specialty. One of them was a security guy, another one was an expert on multicast, another was an expert on switching. When it came to Kumar, the voice guy, I figured I was scot-free. After all, I didn’t claim to know anything about voice over IP. Kumar looked over my resume, and then he looked up at me. “I see you have ISDN on your resume,” Kumar said. And then he began to grill me on ISDN. Darn, I should have thought of that!  Thankfully, I was well prepared.

Despite one or two mistakes in the interview, I got hired on and began my new job as a customer support engineer at HTTS. My first few months were in a group called ESO, which supported large enterprises and was very focused on Catalyst switching.  I won’t go into the details of the job here, but you can see my many TAC tales if you are interested.

Cisco's San Jose campus

Cisco’s San Jose campus

Little purple stickers everywhere!

One thing I quickly noticed when I got to Cisco was that a lot of the people in my department had a nickel-sized purple dot on their ID badges and cubicle nameplates. I found out that these purple dots were actually stickers with the CCIE logo. Cisco employees who had their CCIE’s stuck these purple dots on their badges and nameplates to show it off. Many of the CCIE’s who had passed multiple exams actually placed multiple dots on their badges and nameplates. I wanted one quite badly. The problem was, sheets of these purple stickers were sent out only to the early CCIE’s, and by the time I had passed, Cisco was no longer providing the sheets of stickers. I suppose I could’ve had some printed out, but I asked around looking for a CCIE who was generous to give me one of his dots. They were in scarce supply, however, and nobody was willing to part with one. It was just another way newer CCIE’s were getting jipped.

The real CCIE logo

The real CCIE logo

Even though the exam had now switched to the one day format, you still didn’t meet too many CCIE’s outside of Cisco. It was thus quite a shock when I went to Cisco and saw purple dots everywhere. It seemed like fully half of the people I was working with in my new job had CCIE’s. And many of them had low numbers, in the 2000’s and even in the 1000’s. I was quite relieved to find that they all treated me with total respect; nobody ever challenged me on account of my one-day CCIE. Still, I always had (and always will have) a great deal of respect for those people who passed the test when it was a two day test, and the cottage industry devoted to minting CCIE’s had not yet come into existence.

CCIE challenges customer

Customers were another story. I remember one BGP case in particular. I looked at the customer’s configuration and immediately realized that it was a simple matter of misconfiguration. I fired it up in my lab reproduced his configuration and proved to him that it was indeed a configuration error on his part. I wrote it all up in an email and proudly signed it with my CCIE number. Within a half an hour I got a call from the customer and one of his colleagues on the line. They proceeded to grill me rapidly on BGP asking me all sorts of questions that weren’t relevant to their case and stumping me several times. At that point I realized that when many people see you are a CCIE, they take it as a challenge. In some cases they failed the test themselves, or else they’ve met stupid CCIE’s in the past and they feel themselves to be on a mission to discredit all CCIE’s. After that episode, I removed my CCIE number from my email signature. I gained a feeling of self-importance after I passed my exam, but working among so many people with the same certification, and dealing with such intelligent customers, I realized that the CCIE didn’t always carry the prestige I thought it did.  The mystique diminished even further.

Incidentally, I became friends with all of the guys who interviewed me, and I was on the interview team myself during my tenure at Cisco. One extremely sharp CCIE we hired told me our interview was so tough he had to “hit the bottle” afterwards. It was considered a rite of passage at TAC to go through a tough interview, but I have gotten a lot nicer in my interview style now, having been on the receiving end of a few grillings.

The value of a CCIE

One of the later posts in this series will examine the question of the value of a CCIE certification.  After all, this is one of the most common questions I see in forums dedicated to certification.  However, my experience getting hired into Cisco (the first time) has some lessons.

  • The immediate reason I got hired was because of my experience and willingness to go out of my way helping others to get their CCIE on Groupstudy.  However, I would not have gotten the position without a CCIE, so clearly it proved its value there.
  • Once you are at Cisco, although people commonly display their stickers and plaques, having a CCIE certification will not necessarily distinguish you.
  • There are many CCIE’s who have made a bad impression on others, whether they are only book-knowledgeable, or even cheaters.  Often people challenge you when you have a CCIE, instead of respecting you.

In the next article in the series, Multiple CCIEs, Multiple Attempts, I describe passing the CCIE Security exam.  I talk about my experience suffering the agony of defeat for the first time, and how I eventually conquered that test.

In this article in my “10 Years a CCIE” series, I take you inside the infamous CCIE lab, where countless candidates have sweated out the devious challenges concocted by the CCIE exam authors.

Planning travel

I was fortunate at the time I took the lab exam in that I lived in San Francisco, very close to the San Jose test site. However, knowing the unpredictability of Bay Area traffic, and also knowing that the exam was very early in the morning (I am not a morning person) I decided it would be best to book a hotel room close to the test site. I even went so far as the book 2 nights in a hotel room, figuring that on the morning of the exam I wouldn’t want to deal with checking out of the hotel. This was perhaps excessive, but it made me relax and even for the well-prepared candidate your mental state is important.

The hotel on Great America Parkway where I stayed

The hotel on Great America parkway where I stayed

I often see test advice which says to get a good rest, to eat well, to not drink alcohol, etc., before your exam. Frankly, I always feel insulted when I see this advice. Of course I’m not going to get drunk the night before the exam. I don’t think anybody needs someone to tell them this. However, I do recommend planning your travel arrangements carefully to reduce stress on the day of the exam. I even drove the straight shot down Tasman the night before to check out the building where the exam was, just so that I would know my way. Read More »