Category Archives: Reflections

by

Cisco Live US 2018

Cisco Live Orlando has wrapped up, at least for me, and I can relax until Cisco Live Europe in January.  I never realized how much work goes into Cisco Live until I became a TME.  Building labs, working on slides, preparing demos, and arranging customer meetings is a months-long process and always a scramble at the end.  It’s a great show, and I can say that having attended as a customer.  It’s more fun and less work to be an attendee, but for technical marketing engineers, it’s still a blast and the highlight of the year.

Orlando had a special significance for me because it was at CL Orlando in 2007 that I decided I really wanted to be a TME.  I attended several breakouts and thought that I’d love to be up in front of the room, teaching folks how about technology.  The only problem:  I was terrified of public speaking.

It took years of trainings, including many as a Toastmaster, before I became comfortable in front of an audience.  That’s a story for another time.  It also took years before the right job opened up, and there were a couple near moves into technical marketing that didn’t work out.  I have to say, I’m glad I have this job and love (almost) every minute of it.

Still, getting up in front of a bunch of your (rather smart) peer network engineers and claiming some sort of expertise is nerve-wracking.  Wanting to do well in front of an audience can lead to frustration.  My main breakout session, BRKCRS-2451, Scripting Catalyst Switches, won me two distinguished speaker awards in a row.  This year, however, the scores are looking quite a bit lower.

It didn’t help that the start time was 8am.  I’m not a morning person, and 8am in Orlando was 5am for me.  The old neurons just weren’t firing for the first 30-45 minutes of the presentation, and in front of 400 people that just isn’t good.

A dose of humility is a good thing, though.  I know TMEs who would kill for my “disappointing” score, so it wasn’t that bad.  And the comments were quite helpful, in fact, and make clear what people are looking for and where they didn’t think I delivered.

I structured BRKCRS-2451 as a journey through developing a script on IOS XE.  The session begins with a demo of a fairly simple script, which pulls some data down from a switch and then formats it and sends it to a Webex Teams (formerly Spark) room.  Then, I break down the script starting with installing Python, and some of the tools needed, like Git and Virtual Environments.  Then I move on to YANG/NETCONF, talk about REST, and then wrap it up by showing how it all fits together to build the script I demoed.

It was a winning formula for a while, but I’m suspecting network engineers have up-leveled their programmability skills in the last year or so.  When I used to explain what GitHub was, network engineers usually were relieved to have it explained to them.  Now I think they all know.

I have a few ideas for making the session more relevant.  Still, it was a great experience talking to 400 people, meeting customers around the show floor and halls, and visiting some of my colleagues’ sessions.  Hopefully my attendees got something out of the session, and I look forward to the next Cisco Live.

by

Two Years of Ten Years a CCIE

Two years ago I published my Ten Years a CCIE series.  Actually, I had written the series a couple years before I published it, but as I say in my introduction to the series, I felt it was a bit self-indulgent an uninteresting, so I scrapped it for a while.  The original pieces were dictated, and I’ve been meaning to go back and clean up some of the grammatical errors or grating phrases, but haven’t had the time.  Not a lot of people have read it, nor did I expect many to read it, since I generally don’t advertise the blog in social media, or anywhere really.  But the feedback from the few who have read it has been positive, and I’m gratified for that.

Things have changed a lot since I got into networking in 1995, and since I passed my CCIE in 2004.  But it’s also amazing how much has stayed the same.  TCP/IP, and in fact IPv4, is still the heart of the network.  Knowledge of OSPF and BGP is still key.  For the most part, new controllers and programmable interfaces represent a different way of managing fundamentally the same thing.

The obvious reasons for this are that networks work and are hard to change.  The old protocols have been sufficient for passing data from point A to point B for a long time.  They’re not perfect but they are more than adequate.  They are hard to change because networks are heterogeneous.  There are so many types of different systems connecting to them, that if we wanted to fundamentally alter the building blocks of networks, we’d have to upgrade a lot of systems.  This is why IPv6 adoption is so slow.

Occasionally I poke around at TechExams.net to see what newer network engineers are thinking, and where they are struggling.  I’m probably the only director-level employee of Cisco who reads or comments on that message board.  I started reading it back when I was still at Juniper and studying for my JNCIE, but I’ve continued to read it because I like the insights I get from folks prepping for their certifications.  People are occasionally concerned that the new world of controllers and automation will make their jobs obsolete.

I built the first part of my career on CLI.  Now I’m building it on controllers and programmability.  In this industry, we have to adapt, but we don’t have to die.  Cars have changed drastically, with on-board computer systems and so forth, but we still need mechanics.  We still need good network engineers.

To be honest, I was getting tired of my career by the time I left Juniper and came to Cisco.  I was bored.  I thought of going back to school and getting a Ph.D. in classical languages, my other passion.  Getting married helped put an end to that idea (Ph.D.’s in ancient Greek make a lot less than network engineers) but when I came back to Cisco, I felt revitalized.  I started learning new things.  Networking was becoming fun again.

I wrote the “Ten Years a CCIE” series both for people who had passed the exam and wanted to have some fun remembering the experience, as well as for people struggling to pass it.  Some things change, as I said, but a lot remains the same.  I still think, closing in on 15 years since I took the exam, that it’s still worth it.  I still think it’s a fantastic way to launch a career.  The exam curriculum will adapt, as it always does, with new technologies, but it’s an amazing learning experience if you do it honestly, and you will be needed when you make it through.

by

Book Sprint

[et_pb_section admin_label=”section”][et_pb_row admin_label=”row”][et_pb_column type=”4_4″][et_pb_text admin_label=”Text”]

I’m somewhat recovered from an exhausting week.  I spent last week with a team of 10 others locked up in building 4 at Cisco writing a book using the book sprint methodology.

Several of the TMEs who report to me got together and wrote a book on Software-Defined Access earlier this year.  The PDF version of that book is available here.  Then, just over a month ago, some TMEs (including one member of my team) got together and wrote a book on the Catalyst 9000-series, available here.  Both of these were also produced with the book sprint methodology, and the quality is surprisingly good.

These books are written with the help of the Book Sprint company.  They send a facilitator who guides the team through writing a book from scratch in a week.  There is no preparation beforehand, and almost no work after the week is over.

The week begins with everyone writing their ideas on post-its, and then organizing them into the basic structure of the book.  By the second half of day one, we were assembled into to small teams to outline our sections.  After outlining the section, the sub-teams then break down and individuals start writing the book.

By the end of Tuesday, the book is written, but it doesn’t end there.  On Wednesday the entire book is reviewed by teams different from the ones that wrote it, and then on Thursday it is reviewed again.  Friday the entire book is reviewed by a sub-team to iron out the English and ensure the voice is consistent throughout.  While all this is going on, editors and illustrators are working on the book in the background.

As I mentioned, it’s exhausting.  We worked until midnight on Thursday and 10pm on Friday.  But we got it done and we’ll have some copies printed up for Cisco Live in Orlando in June.

I can’t say I agree with the approach of every part of the book, but that’s the idea.  It’s a team effort.  It’s not my book, nor the book of any other team member.  It’s our book.  I tend to write in a more conversational tone that works for blogs but is not as good for books.  I think that my occasionally excessive wordiness helps to draw the reader along, and gives them space to digest what I’m saying.  So, it was occasionally painful to see my prose hacked apart by other authors.  Still, at the end of the day, the process works and the result was good.

For any readers who might be attending CL Orlando, I’ll be happy to sign a copy for you.  For those who aren’t, when we have the PDF finalized I’ll link it on the blog.

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]

by

Where I’ve been, and what a TME is

Jesse, a recent commentor, asked why I haven’t been posting much lately.  In fact, my last post was August of 2017.  Well, there are several reasons I don’t post much these days.  In part, I’m not convinced anyone is reading.  It’s nice to see a comment now and again to realize it’s not just spambots looking at SZ.

The other major reason was a job change.  I moved to Cisco over two years ago, and I came in as an individual contributor (IC).  I liked to joke that I had never been so busy since…the last time I worked at Cisco.  However, as an IC, I had no idea how easy I had it.

Someone got the crazy idea to make me a manager.  So now, not only do I have the Principal Technical Marketing Engineer title, I also manage a team of 10 TMEs.  The team happens to be driving Software-Defined Access, currently Cisco’s flagship product.  So, the time for blogging is a bit limited.  I’m still working on programmability in my spare time, and I’m continuing to do Cisco Live sessions at least twice a year.  My hair is turning white and I don’t think it’s just my age.

That said, I cannot image a better job or place to be than this job at this time.  It’s an exciting company to work for, and an exciting time.  The team that reports to me includes some of the smartest and hardest working TMEs in Cisco.  These guys are legendary.  (For me “guys” is gender-neutral, for those of you who worry about such things.)  And my boss is considered by many to be one of the best who ever did the TME job.

A quick primer on exactly what a TME does, for those who don’t know:  We are (usually) attached to a business unit within Cisco, and we are really an interface between sales and engineering.  We also work directly with customers, but generally when sales pulls us in.  TMEs are technical (the “T” in “TME”) so they are expected to know their product/technology in detail.  They are, however marketers (the “M” in “TME”) so they need to be able to explain what their product does.

On the inbound side, we learn the requirements for products from sales and customers and communicate those requirements to engineering.  We work closely with Product Managers (PMs) to develop Product Requirement Documents (PRDs) and meet regularly with engineering to ensure that they are building their products in a way that satisfies marketing requirements.

On the outbound side, we develop collateral, which could be white papers, videos, slide decks, etc.  (We do not write the documentation you see on CCO, but we certainly review it.)  We present to our sales team in twice-a-year events, explaining the latest developments in our products and collecting their feedback on what we could do better.  We travel on site to meet with customers in support of sales, or else meet the customers here, at the Executive Briefing Center (EBC).  The most enjoyable part of the year, for most of us, is Cisco Live, our major trade show.

We have four CL events each year:  Europe, South America, Australia, and US.  The US event is the largest of all.  I generally attend both Europe (we were in Barcelona this year) and the US event (Orlando this time).  These events are a blast, but I never realized how much hard work goes into planning the event and developing the content.  It’s also stressful.  I’ve been fortunate to win distinguished speaker two events in a row, which means I was rated in the top 10%.  However, standing in front of an audience of hundreds is always a bit nerve-wracking, and getting ready requires a ton of preparation.  Still, it’s a great opportunity to meet with customers and have a good time.

The pace of work for TMEs is relentless.  I used to say TAC was relentless, because the second you close a case, you take another.  Well, with two SEVTs (sales events) and four Cisco Lives to prepare for, plus a constant and never-ending series of product/software releases…well, it never stops here either.

So that’s why the blogs have fallen away.  I do think I can find 10-15 minutes to post updates at least every week, so I’m going to try to do it.  I wouldn’t mind actually writing the series on programmability I started.  I’d like to clean up and revise the 10 years a CCIE series.  I also have another TAC tale to write up, one of my all-time favorites, so look for that soon.

And to Jesse:  thanks for getting me going!

by

Cheaters

Note:  This article was originally posted in 2016.  Since that time, the CCIE program has changed the process for earning a CCIE, and the separate written exam is no longer used.  This means that the problem of people claiming to be a “CCIE” when they have only passed the written exam is no longer the case.  I’m leaving the article as is for now, but will modify it in the future when I have time, to reflect the new circumstances.  Regardless, you should never claim you have a certification when you have only passed a part of the requirements.  (ccie14023, Feb 2020)

In this article in the “Ten Years a CCIE” series, I look at the question of cheating.  Is it possible to cheat on the CCIE exam?  And what does cheating do to the value of the certification?

Yes, you can cheat on the CCIE

Shortly after I passed my Security exam I spoke with the first CCIE to pass the Voice exam. He took a beta version while he worked at Cisco. I commented that I valued my CCIE so much because it was simply impossible to cheat at the exam. It wasn’t a written exam; you couldn’t just walk in knowing the answers; you had to think on your feet. He laughed at me and explained my ignorance.

A lot of people cheat on the CCIE lab exam, he said. Either they work in groups and share the contents of the exams  they’ve seen, or else they get copies of the exam from unscrupulous vendors on the Internet. Then, having seen the actual exam, and having researched the difficult problems, and configured it several times in their lab, they can pass with ease.

I was quite shocked to hear this. I had always studied alone, and when I started down the CCIE road, I didn’t just want to pass the exams, I wanted to beat them. I didn’t just want the CCIE, I wanted the CCIE mystique. I was flabbergasted that people would want the certification without the work. Of course there is a great appeal to gaining something so valuable with minimal effort, but how are you going to make it through a job interview?

The stupidity of cheating

I had encountered rampant cheating in graduate school. This was at the dawn of the Internet, and I saw that many of my fellow students ripped off entire papers from the Internet. We used to send our papers to each other via email, and occasionally I would paste a snippet into AltaVista (Google not being available yet), and often I would hit upon the original work that they’d stolen. Leaving aside the ethical issues of stealing someone else’s work, or ripping off the questions and answers for an exam, there is a practical downside to cheating. You are claiming a credential that you haven’t earned. I remember conducting a job interview of a girl with a Masters degree from the same program as myself. I asked her the subject of one of her papers, and it had to do with routing protocols. She couldn’t answer even the most basic questions about the content of her paper. It was obvious that she cheated her way through the program. And she looked like a complete fool claiming to be a “master” of a subject about which she knew nothing.

Sometimes engineers I know roll their eyes when they hear I have a CCIE. They have encountered one of my fellow “experts” only to find that he seemed hardly an expert at all. Since I know that it’s possible to cheat on this exam, I’m convinced that many of these so-called CCIE’s cheated on their exams. They look like fools as did the girl with her Masters degree.

I’ll talk more about the value of the certification and later post, but one thing to keep in mind is that there’s great value in the study process. There’s great value in learning. And if you study for the test as you are supposed to study for it, you’re guaranteed to learn a lot.

A blurry ethical line?

I, like almost everybody these days, passed my exams using material from legitimate vendors, primarily Internetwork Expert and IPExpert.  (The latter has closed their doors.)  These vendors provide quite a lot of material, but their signature product is a book of sample exams designed to prepare you for the real thing.  This brings up a question.  Presumably some of the scenarios covered by the “legitimate” vendors are scenarios that might come up on the real lab.  After all, how many ways are there to configure BGP?

Interestingly enough, any exam has to provide a certain amount of information to test-takers beforehand.  The CCIE exams have detailed blueprints which guide candidates in their studies.  It would be impossible to take and pass an exam without such advance information.  With merely a blueprint in hand, it would be possible to construct some kind of sample exam, but do vendors simply build them off the blueprint?  Or do they get information from candidates and use them to build their tests?

The ethical lines can be blurry, but one thing is for certain:  studying for a CCIE exam using an actual copy of a real test is blatant cheating and disgraceful behavior.

Cheating on the written exam

Cheating is also rampant on the written exam. This is even the case among CCIE’s who are recertifying, perhaps especially the case. As I mentioned in my recertification post, taking an exam every two years, especially a hard one, is a big hassle. Many CCIE’s get lazy about the process. There are vendors who will sell verbatim copies of the tests. There is still, of course, some work involved. Someone with a copy of the test has to actually memorize all the answers. But it is far easier than going into a test blind.

My last re-certification was quite painful, and yet I refuse to use any sort of brain dump.  Instead, I built an Anki database of questions.  It wasn’t perfect, and it took a couple of failures for me to build a database that had sufficient coverage.

False CCIEs

Another way to “cheat” is simply to falsely claim CCIE status.  Anyone with a CCO account can verify if somebody actually has a CCIE, and whether they are active, but oftentimes employers just don’t bother to check.  When I was at Cisco HTTS, we were very close to hiring someone for a CCIE-requiring position, when I ran his name through the tool.  His CCIE had been revoked because he hadn’t recertified.  He was clearly embarrassed, and had simply been too busy to recertify.  While I can empathize with that, the fact was that he did not have a CCIE and would need to take both written and lab to get it back.  We didn’t hire him, but it amazed me that nobody had bothered to check early in the hiring process.

There is also a large group of imposters who have passed the written, and somehow think this qualifies them to put “CCIE” on their resume.  I recently saw a poster on a LinkedIn group who gave herself a CCIE (Wrt) title.  I also remember one candidate who put “CCIE Routing/Switching” in huge, bold letters on his resume, with “written” in a tiny font right next to it.  Well, I have news for you.  There is no CCIE written certification.  You either have a CCIE or you don’t.  Pass the lab before you put CCIE anything on your resume.  If you are looking at an employer that is willing to sponsor you for the lab, then by all means, tell them you passed the written.  But don’t claim CCIE status without a number.

What is the value?

We all know that there are a number of CCIE’s out there who should not have the certification.  It reflects poorly on the CCIE community.  There is no question that whatever value the CCIE has is diminished by those who obtained their credential through fraud.  If you are frustrated with the exam and thinking of hunting for a brain dump, remember this:  if you can’t pass the exam, you have no right to call yourself a CCIE.

In my next and final article in the series, The Value of a CCIE, I will take a look at the value of the credential.  Ten years later, do I think it was worth it?  Would I recommend someone take the CCIE exam now?  What do I think the future is for network experts in the world of SDN and automation?

1
by

Recertification Pain

Note:  This article was written in 2016 and has not been modified.  A number of changes have been made to the CCIE program which have dramatically improved the re-certification process.  Continuing education is now an option, as I suggest in this article.  The re-certification frequency has been reduced.  I may modify this article in the future, but I am leaving it as is for now for historical purposes.  However, please note the description of the process is no longer accurate at all.  (ccie14023, Sept 2021)

 

In this installment of “Ten Years a CCIE,” I look at what you have to do to stay certified, and the difficulty of maintaining your credential.

Passing your CCIE gives you a great feeling of accomplishment, and also a sense of relief.  You’ve spent months studying and late nights configuring scenarios in the lab.  Maybe you took the exam multiple times, and had to experience the letdown of knowing that, instead of being finished, you had more months of studying ahead.  So, you’ve finally passed, and it’s all over, right?

No, unfortunately.  You have a CCIE, but if you want to keep it, you have to worry about hitting the books again every two years.  All CCIE’s have to re-certify, a biennial ritual that becomes harder as the years go by.

Here’s how it works.  Before two years after your lab date, you have to re-certify your CCIE by passing a CCIE written exam.  You can take any written exam, just as long as it is a CCIE written.  For example, if you passed Routing and Switching, you could recertify by taking the Data Center written exam.  This has the advantage of simultaneously qualifying you for another lab exam, if you are so inclined.  If you have more than one CCIE, you can recertify all of them by taking any CCIE written.  For example, if you have Routing/Switching, ISP Dial, and Collaboration CCIEs, you could recertify all of them at once by taking the Wireless written.  This holds true even though ISP Dial is no longer a valid certification.  Even if you only have a certification that no longer exists (such as ISP Dial or SNA IP), you can maintain active CCIE status by passing any written exam.

If you don’t pass a written exam, at the two year mark your certification becomes suspended.  You can no longer use your CCIE number in your signature or claim to be a CCIE.  You can still pass the recert exam within a year, but if a year elapses after you go suspended, you lose your CCIEs, all of them, and have to retake both written and lab for any CCIE you hold.  Needless to say, you don’t want that to happen.

recert

What you want to see when you verify your CCIE…

(For comparison, my JNCIE-SP expires every three years, and I have to take the JNCIP-SP exam to recertify.  If I had a JNCIE-ENT as well, I would have to take both exams to recertify.)

If you just passed your lab exam and you feel super-confident, you may think you don’t have to worry about a measly written exam in two years.  However, any CCIE will tell you the recertification ritual is onerous and a huge waste of time.  As your career advances, you will often find yourself doing less and less CLI, and you might in fact work less with Cisco products.  In my case, re-certifying became especially painful during my six years at Juniper.

It would be less of a burden if the exams were better written.  The last time I took the written, there was a question that was flat out wrong, and many that were just obscure.

I first wrote this entry in 2014, and I am now re-writing it two years later.  When I first wrote it, I was working on my recert and in a state of extreme annoyance, came up with a couple of sample questions intended to mimic the actual exam:

When is the MSDP ConnectRetry timer used?
a.  When the MSDP peer with the highest IP address transitions from the INACTIVE to the CONNECTING state.
b.  When the MSDP peer with the lowest IP address transitions from the CONNECTING to the ESTABLISHED state.
c.  When the MSDP peer with the lowest IP address transitions from the INACTIVE to CONNECTING state.
d.  When the MSDP peer with the highest IP address transitions from the CONNECTING to the ESTABLISHED state.

What is the RSVP message type for a PathTear message?
a. 4
b. 0
c. 5
d. 3

What does the “ipv6 mld limit 100″ command do?
a.  Limits the number of hosts that multicast listener discovery can discover to 100
b.  Limits the hosts permitted by MLD to those contained in ACL 100
c.  Limits the number of MLD states to 100 on a per-interface basis.
d.  Limits the number of MLD states to 100 globally.

At the time I wrote them, these questions were technically within the blueprint topics for the Routing and Switching written exam, but they are obviously rather stupid questions.  The R&S blueprint is so huge that it is essentially impossible to know all of the subjects it covers.  Nevertheless, I was encountering questions of roughly this level of obscurity on the exam.

The purpose of recertification

Why do we have to recertify?  Obviously, the main reason is to ensure CCIE’s stay current in the field.  I passed routing/switching back in 2004, and a lot has changed in 12 years.  It’s important that people who come to me for expertise believe that I actually have relevant knowledge.

We have to ask a question though:  how well do you stay up-to-date taking a written exam every two years?  And why can you keep your credential when you re-certified in a different track?

For example, if someone acquired a CCIE Security certification back in 2002, but re-certified for 14 years using the routing/switching written, why is that engineer qualified to continue calling himself a “CCIE Security”?  He probably knows nothing of modern security technologies.  Juniper requires JCNIE’s to recertify in each track they have certified, so a triple JNCIE has to take three separate exams.  While this is painful (and kept me to one JNCIE), it makes more sense.

I think an even more reasonable approach is to allow continuing education in lieu of a test.  This is the requirement for CISSPs, lawyers, and even doctors, and it makes a lot of sense.  I never remember much from the recert exams, but a couple days of training would be a great way to get current.

I do think Cisco was smart to introduce the Emeritus option.  CCIE Emeritus allows CCIE’s who have hit the 10 year mark to pay a fee to keep their number in a non-active status indefinitely, with the option to recertify.  Many CCIEs reach a point where they don’t deal with day-to-day CLI configuration, and find the exams harder and less relevant to their careers.  Several of my friends have chosen this option.  I almost did when I worked at Juniper, but I am thankfully still current.

By the way, the answer to all of the above questions is ‘C’.

In my next article, Cheaters, I look at the question of whether people cheat on the CCIE exam, and the effect it has on the value of the certification.

by

Multiple CCIE’s, multiple attempts

In this article in my “Ten Years a CCIE” series, I look at passing the Security exam in 2008.  I get to experience the agony of failure for the first time, and have to re-tool my strategy.

Goodbye to Cisco

I worked two long years at Cisco. Two very long years. I learned so much there but it was a brutal job. The relentless flood of new and challenging cases grew tiresome.  When my aforementioned sushi eating CCIE friend called me in 2007 and invited me to come join him at a Gold partner I couldn’t say no.  Cisco sells much of its gear through value added resellers (VARs), also known as partners.  These partners are assigned different levels depending on the amount of business they do, and Gold is the highest.

Working at a gold partner with a CCIE was quite enjoyable. Gold partners need CCIE’s and so they have a lot of incentive to make you happy. My boss suggested that I get a second CCIE, this time in voice. I started to buy material for the voice exam, when my VP showed up in the office, fired my boss, and told me to start studying for the security exam. (His firing of my boss had nothing to do with my CCIE exam, but it certainly made me stand up and listen to what he was asking.) So, having really not started on voice, I switched immediately to security.

I had already passed the security written back at Cisco partly to qualify for the lab exam, and partly to re-certify my existing CCIE, so it was straight to the lab exam for me. The equipment list was a big challenge. At that time, you needed two ASA’s, one PIX, a VPN 3000 series concentrator, and IDS device, six routers and two switches, and some sort of Windows server running Cisco secure ACS. I still had my old lab equipment from before, but I was missing everything else. I had one ASA 5505 from work, but no other security devices. I decided that the cost was too prohibitive for me to set up my own lab. I was going to have to use rack rentals. That was my first big mistake.

I decided to approach the exam in exactly the same way I approached the routing and switching exam. I studied the various subjects on the blueprint individually, and then started doing full labs from the Internetwork Expert workbook. As great as IE’s workbook was for routing and switching, in 2008 it really wasn’t very good for security. I have a lot of respect for the Bryans, and I’m sure it’s come a long way, but at that time it just wasn’t enough.

Attempt number one

When I showed up at the familiar CCIE lab, I didn’t feel well prepared, because I wasn’t. The lab was a disaster. I only managed to complete about a third of the exam. While configuring DMVPN, all of my routers locked up and crashed. I called the proctor over, and when he saw that the console ports were locked up, he started to accuse me of having made a configuration error. I explained that I hadn’t touched the console configuration, and just then we both saw bus errors appear on the console sessions followed by reloads. It was obvious then that I was not at fault. I had heard that if routers crash during CCIE exam, the proctor will give you your time back. However, the proctor admonished me to save my configs frequently, and refused to give me any time back. I had probably lost 15 minutes. I would have fought it, except that I was already so far behind on the exam,  I knew it would make no difference. Still, to this day am a bit angry at that proctor. As I left the exam room I looked at him and said “don’t even bother grading this.” He looked at me and said, “Oh, I’m sure you’re exaggerating.” I looked at him and told him I hadn’t completed two thirds of the exam.” Oh!” He exclaimed.” Well… Don’t wait six months for your next exam!”

… It was six months before my next attempt.

The author's 2008 CCIE security lab. The laptop ran Windows Server in a VM for ACS. An ASA 5505 is visible on top of the drawers.

The author’s 2008 CCIE security lab. The laptop ran Windows Server in a VM for ACS. An ASA 5505 is visible on top of the drawers.

Changes to my approach

I knew I had to revise my strategy. Something wasn’t working. The first thing I fixed was the lab situation. When I did Routing and Switching, I knew that I needed my own lab at home. Using remote rack rentals for security just didn’t give me enough time in the lab. I managed to get a hold of the PIX from a friend who was decommissioning it. I bought myself an ASA 5510, which, at $2500, was the most expensive piece of hardware I had. I really needed two of them, in order to cluster them, but I had to make do with the mismatched pair of the 5510 and 5505. As with the Routing and Switching exam, I knew I could use remote rack rentals to fill in for the equipment that I didn’t have. The ASA 5505 was adequate for basically everything except clustering. It had almost all of the capabilities of the 5510, but the configuration of VLANs was slightly different.  I also managed to acquire an IDS, and VPN 3000 series concentrator. I borrowed a laptop from work and got a Windows server license and managed to install Cisco Secure ACS. I ended up with a very complete lab.

I realized that a big part of my problem was that IPSec configurations are long, complicated, and counterintuitive. IPSec is the core of the CCIE security exam, and you need to know it as well as BGP and OSPF on the routing and switching exam. I made a series of diagrams which depicted each of the constituent configuration elements for the various IPSec technologies as blocks, which were then connected together by arrows. For example, for basic IPSec configuration, I would have one block representing the IKE configuration, and another representing the IPSec policy. I would draw an arrow to show how they were connected, labeling the arrow with the command used to connect them. Before I was trying to memorize these configurations. Now I was able to visualize them.

Visualizing complex configurations helps make them easy to understand and remember

Visualizing complex configurations helps make them easy to understand and remember

I also completely abandoned using the IE workbook. It just wasn’t ready at that point. Instead I invented my own VPN challenge lab. It had every kind of VPN on it: IPSec on ASA, IPSec on PIX, IPSec on VPN 3K, client IPSec on all of those platforms, L2TP, PPTP, DMVPN, SSL. I worked this lab over and over again until I could configure all of these automatically, and I made sure I configured between disparate platforms.

I felt good but not 100% prepared when I went to take my second attempt. I failed, but my score was much higher than before. I continued preparing for another month or so before taking my third attempt. I was so ready for my third attempt, that I completed the lab shortly after lunch. As I was coming out of the bathroom, I ran into Ted the proctor (not his real name), in the hallway. I had seen Ted on my second attempt and he told me he was attending a bluegrass Festival in San Francisco. I spent a good 15 minutes talking to Ted about the festival in the hallway, and I think at that point Ted realized that I was feeling pretty confident. Most people don’t spend 15 minutes shooting the breeze in the middle of the CCIE exam.

Interestingly enough, while Ted had been the most helpful Proctor on the Routing and Switching exam, he was of almost no help at all on the security exam. I’m not sure if he had changed in the intervening four years, or if he simply wasn’t as familiar with the security exams that I took. Either way, be prepared to make difficult decisions on your own in the lab, without the help of the proctors. Of all the questions I asked them, only once did I get a useful answer. I realize that their job is not to give away the test, but often the test is poorly written and I think that they need to be more helpful in explaining the exam.
Passing Routing and Switching was exciting; passing Security was a relief. I had almost given up after my disastrous first attempt. And I’m glad that I passed it when I did. As with the Routing and Switching exam, I passed Security in November. And as with the Routing and Switching exam, Cisco was changing the test at the beginning of the new year. The VPN 3000, PIX, NAC framework, and several other technologies were being removed. Of course, they never removed technologies without adding some as well. Had I failed my third attempt, it’s likely I would never have tried again.

In summary:

  • Having “always-on” access to a lab is critical!  Remote rack rental is good to fill in for a few things you might be missing, but don’t rely on it.
  • You may have to spend some more money than you want to acquiring gear, but it pays off.
  • The way you pass one CCIE exam is not necessarily the way you pass another exam.  You have to spend some time looking at the topics you will be covering, figuring out the best way to reach the point of automatic configuration of the technologies.
  • Sometimes, the study material from the vendors just won’t cut it.
  • Proctors aren’t always nice, and don’t always do what you thought they were supposed to.

I will cover the question of lab blueprint changes in a later article on the value of a CCIE, but it’s worth noting that for both my routing/switching and security exams, a blueprint change happened immediately after my passing.  I spent a lot of time studying, for example, the VPN 3000 concentrator which was already obsolete.  Still, I would have the same credential as a guy who passed the exam with the new technologies a couple months later.

Also worth noting:  I passed all of my expert exams (2 CCIEs and a JNCIE) in November.

In the next article in the series, Recertification Pain, I look at the biennial penance we all inherit for passing our CCIEs–the dreaded recertification.  I give my thoughts on improving the process, not that anybody is listening.

1
by

A CCIE Goes Home to Cisco

In this article in my “Ten Years a CCIE” series, I describe my experience going to work at Cisco as a CCIE.  Unlike many Cisco-employed CCIE’s, I earned my certification outside of Cisco.

A CCIE leads to a job at Cisco

I returned to my old job at the Chronicle and had my business cards reprinted with my CCIE number. I loved handing it out, particularly at meetings with telephone companies and Internet service providers whose salespeople were likely to know what such a certification meant.  I remember one such sales person, duly impressed, saying “wow, on that test you can be forced to configure any feature on any Cisco product…I don’t know how anyone could prepare for that!”  (Uh, right.  See “The CCIE Mystique“.)

At that time, the most popular forum for aspiring CCIE’s was an email distribution list called groupstudy.com. I had been a subscriber to this mailing list, but prior to passing my exam, I didn’t feel adequate to post anything there. However, once I passed, I began posting regularly, beginning with a summary of my test preparation process. One day I got an email from a mysterious CCIE who told me that I sounded like I knew what I was talking about, asking me if I wanted to interview for a job. I thought his name and number sounded familiar, and when I got home I confirm my suspicions by digging through my bookshelf. He was the author of one of my books about Catalyst Quality of Service.

A brutal interview

It turns out he was a manager at Cisco High Touch Technical Support, a group of TAC engineers who specialized in high profile customers. I scheduled an interview right away.

This interview was by far the most difficult I’ve had in my career. They brought me into a room with four CCIE’s, two of them double, all of them sharp. Each one of them had a different specialty. One of them was a security guy, another one was an expert on multicast, another was an expert on switching. When it came to Kumar, the voice guy, I figured I was scot-free. After all, I didn’t claim to know anything about voice over IP. Kumar looked over my resume, and then he looked up at me. “I see you have ISDN on your resume,” Kumar said. And then he began to grill me on ISDN. Darn, I should have thought of that!  Thankfully, I was well prepared.

Despite one or two mistakes in the interview, I got hired on and began my new job as a customer support engineer at HTTS. My first few months were in a group called ESO, which supported large enterprises and was very focused on Catalyst switching.  I won’t go into the details of the job here, but you can see my many TAC tales if you are interested.

Cisco's San Jose campus

Cisco’s San Jose campus

Little purple stickers everywhere!

One thing I quickly noticed when I got to Cisco was that a lot of the people in my department had a nickel-sized purple dot on their ID badges and cubicle nameplates. I found out that these purple dots were actually stickers with the CCIE logo. Cisco employees who had their CCIE’s stuck these purple dots on their badges and nameplates to show it off. Many of the CCIE’s who had passed multiple exams actually placed multiple dots on their badges and nameplates. I wanted one quite badly. The problem was, sheets of these purple stickers were sent out only to the early CCIE’s, and by the time I had passed, Cisco was no longer providing the sheets of stickers. I suppose I could’ve had some printed out, but I asked around looking for a CCIE who was generous to give me one of his dots. They were in scarce supply, however, and nobody was willing to part with one. It was just another way newer CCIE’s were getting jipped.

The real CCIE logo

The real CCIE logo

Even though the exam had now switched to the one day format, you still didn’t meet too many CCIE’s outside of Cisco. It was thus quite a shock when I went to Cisco and saw purple dots everywhere. It seemed like fully half of the people I was working with in my new job had CCIE’s. And many of them had low numbers, in the 2000’s and even in the 1000’s. I was quite relieved to find that they all treated me with total respect; nobody ever challenged me on account of my one-day CCIE. Still, I always had (and always will have) a great deal of respect for those people who passed the test when it was a two day test, and the cottage industry devoted to minting CCIE’s had not yet come into existence.

CCIE challenges customer

Customers were another story. I remember one BGP case in particular. I looked at the customer’s configuration and immediately realized that it was a simple matter of misconfiguration. I fired it up in my lab reproduced his configuration and proved to him that it was indeed a configuration error on his part. I wrote it all up in an email and proudly signed it with my CCIE number. Within a half an hour I got a call from the customer and one of his colleagues on the line. They proceeded to grill me rapidly on BGP asking me all sorts of questions that weren’t relevant to their case and stumping me several times. At that point I realized that when many people see you are a CCIE, they take it as a challenge. In some cases they failed the test themselves, or else they’ve met stupid CCIE’s in the past and they feel themselves to be on a mission to discredit all CCIE’s. After that episode, I removed my CCIE number from my email signature. I gained a feeling of self-importance after I passed my exam, but working among so many people with the same certification, and dealing with such intelligent customers, I realized that the CCIE didn’t always carry the prestige I thought it did.  The mystique diminished even further.

Incidentally, I became friends with all of the guys who interviewed me, and I was on the interview team myself during my tenure at Cisco. One extremely sharp CCIE we hired told me our interview was so tough he had to “hit the bottle” afterwards. It was considered a rite of passage at TAC to go through a tough interview, but I have gotten a lot nicer in my interview style now, having been on the receiving end of a few grillings.

The value of a CCIE

One of the later posts in this series will examine the question of the value of a CCIE certification.  After all, this is one of the most common questions I see in forums dedicated to certification.  However, my experience getting hired into Cisco (the first time) has some lessons.

  • The immediate reason I got hired was because of my experience and willingness to go out of my way helping others to get their CCIE on Groupstudy.  However, I would not have gotten the position without a CCIE, so clearly it proved its value there.
  • Once you are at Cisco, although people commonly display their stickers and plaques, having a CCIE certification will not necessarily distinguish you.
  • There are many CCIE’s who have made a bad impression on others, whether they are only book-knowledgeable, or even cheaters.  Often people challenge you when you have a CCIE, instead of respecting you.

In the next article in the series, Multiple CCIEs, Multiple Attempts, I describe passing the CCIE Security exam.  I talk about my experience suffering the agony of defeat for the first time, and how I eventually conquered that test.

by

Room of horrors: Inside the CCIE lab

In this article in my “10 Years a CCIE” series, I take you inside the infamous CCIE lab, where countless candidates have sweated out the devious challenges concocted by the CCIE exam authors.

Planning travel

I was fortunate at the time I took the lab exam in that I lived in San Francisco, very close to the San Jose test site. However, knowing the unpredictability of Bay Area traffic, and also knowing that the exam was very early in the morning (I am not a morning person) I decided it would be best to book a hotel room close to the test site. I even went so far as the book 2 nights in a hotel room, figuring that on the morning of the exam I wouldn’t want to deal with checking out of the hotel. This was perhaps excessive, but it made me relax and even for the well-prepared candidate your mental state is important.

The hotel on Great America Parkway where I stayed

The hotel on Great America parkway where I stayed

I often see test advice which says to get a good rest, to eat well, to not drink alcohol, etc., before your exam. Frankly, I always feel insulted when I see this advice. Of course I’m not going to get drunk the night before the exam. I don’t think anybody needs someone to tell them this. However, I do recommend planning your travel arrangements carefully to reduce stress on the day of the exam. I even drove the straight shot down Tasman the night before to check out the building where the exam was, just so that I would know my way. Continue reading →

1
by

How to pass the CCIE lab exam in one attempt

In this post in the Ten Years a CCIE series, I go over my preparations for the CCIE Routing and Switching exam, and what I did to pass in one attempt.

The first months…

I passed my CCIE Routing and Switching Lab in one attempt, so I think my approach can be considered effective. At least, it was for the exam at the time. I decided to spend my first several months of study diving deep into each of the exam topics on the blueprint. I was determined to focus on core technologies such as BGP and OSPF and to minimize the amount of time spent on ancillary topics such as DLSw. Because you have access to the documentation CD in the lab, you don’t need to know absolutely everything. However, you do not want to spend a long time trying to figure out how to configure core tasks which you should be able to do automatically.

I didn’t work from a particular manual or outline these first few months. Instead I would pick a topic, say BGP. I would go through all of the examples I could find in the books that I had, Jeff Doyle’s books being the most helpful. I would set up the examples from the books in my lab to see if they work as described. Then I performed free-form experimentation. I tried different things; I indulged my curiosity; I came up with new ways to test the protocols and tried to break them. I introduced loops where there weren’t loops in the examples I had. I saw what happened if I ran the protocol over ISDN instead of Frame Relay. And I made very sure that everything I learned I recorded in my notes. For every subject I kept two note files. The first file contained general, conceptual notes. The second file was a list of commands that I thought were important and I needed to remember. These files grew over time, and I studied them thoroughly before attempting the lab.

I had also acquired practice labs from three different sources. I had IP Expert’s lab book; I also had Internetwork Experts’ lab book; and finally, I had the Cisco press official lab book, which was written by a CCIE proctor. I found that this last book’s labs most closely resembled the real thing in terms of how the labs were written and how the diagrams were drawn. Still, as I studied I quickly came to favor the Internetwork Expert book for its thoroughness and accuracy. At that time, they were still relatively new, but the quality of their material was the best.

Closing in on test day…

In the last couple of months before the exam, I shifted my strategy. Instead of focusing on individual topics I spent my time working the practice labs in the IE book. At first I worked them slowly and methodically. I didn’t do them on a timer, and I didn’t rush through them. If it took me 24 hours to work through lab then it took me 24 hours. My main interest was in covering the material, understanding it thoroughly, and in documenting my learnings. I knew so many people who started giving themselves timed exams when they weren’t ready for them. Yes, it’s important to have a strategy and to understand clock management, but it’s far more important to understand the material thoroughly. The best time management strategy is knowing the material so well you can configure most of it on auto-pilot.

Every time I completed the lab I graded myself using IE’s answer key. I used to say that I was my own worst enemy. I never gave myself a pass on the slightest discrepancy between my solution and IE solution. Every single mistake that I wrote I listed out in a document, and in the last few weeks before the exam I reread that document several times every day. Constantly reviewing the mistakes I had made reinforced my own errors in my mind.  I also found that in my note documents that I was highlighting certain important points or gotchas with the capital words “BE SURE”. I created another document that I called my “BE SURE” list. I also reviewed this list several times a day in the last few weeks before the exam. Reviewing both my mistakes as well as my “BE SURE” list so frequently was quite effective in helping me remember my mistakes and important notes.

A snippet of my BE SURE list

A snippet of my BE SURE list

When I was studying for my CCIE exam Cisco press had just released two handy books. These books covered all of the commands in IOS at that time for BGP and OSPF. Not only did they describe the commands but they had examples of their use as well. In the last few days before the exam I would review the table of contents of these books which listed all the commands by name. I did this every night in bed. If I was able to accurately describe the command, I would cross it off.  Some commands that I couldn’t remember I saw night after night, until they were so familiar I had no problem using them.  Doing this every night helped me to commit fully to memory all of the different BGP and OSPF commands that make up the core of the CCIE lab exam.

I also took the CCIE Lab Boot Camp from Internetwork Expert just a few weeks before I took the actual lab exam. This was a wonderful experience. I was able to take the course from home, using IE’s Java-based virtual environment. Because most of the work and the class consisted of full, eight-hour timed labs, there was no need to travel to a classroom. And, because the eight hour exams were administered on Internetwork Expert’s own racks of equipment, there was no problem with not having a full CCIE lab at home. We had a small amount of lecture each day, followed by the eight hour lab, which was then graded each night. In the morning we were given our results. I was told that people scoring over 80% generally passed the CCIE lab exam, and I was scoring higher with no problem. The Brians gave me some great advice and particularly fixed some problems that I had in configuring multicast.

At the end of the boot camp Brian Dennis, the grumpier of the Brians, gave what I would charitably call a pep talk. He told us that a test is just a test, that we should get some of the classic books on networking and study them thoroughly, and that we should know our subject, not simply pass the test.  “You meet some CCIEs and wonder, how did this guy pass the test?” Brian said.

In November 2004 the time came to take the test. I had no idea if I was ready. A good friend of mine who passed shortly before spent four hours with me in a sushi restaurant grilling me on every possible subject that could be on the exam. They closed the restaurant on us.  For my final preparation, I studied all of the new features in IOS which they were now using in the CCIE lab. I also studied the documentation CD thoroughly so that I would have no trouble navigating it in the lab.

Passing the test

If you’re working on the CCIE exam, why should you care what someone did to prepare for it ten years ago?  Well, as I’ve said, it is a different test now.  My advice on learning ISDN dial maps isn’t going to help you.  However, there are some general principles here that you should pay attention to.

  1. Figure out the core topics and learn them well.  Cold.  On every expert exam, there are some core topics and some ancillary topics.  You cannot know everything.  Figure out the core topics and drill them over, and over, and over again.  You need to be able to configure them without thinking.
  2. Make things harder than they have to be.  As I said, break things intentionally.  Introduce problems.  Ask questions.  Don’t just run the scenarios you bought with your labs.
  3. Be your own worst enemy.  Remember, the CCIE exam is not just about doing what they tell you, but doing exactly what they tell you.  When you grade yourself, read and re-read the tasks.  Make absolutely sure that you have accurately and completely fulfilled the requirements.
  4. Document your mistakes.  Review things you have done wrong, and keep reviewing them.

In the next post in the series,  Room of Horrors, I describe the CCIE lab experience.  I talk about what it was like to enter the infamous lab in Cisco Building C, and take the challenging exam.