Tag Archives: cisco

by

TAC Tales #7: “Just slam it in!”

When I first started at TAC, I wasn’t allowed to take cases by myself.  If I grabbed a case, I had to get an experienced engineer to help me out.  One day I grabbed a case on a Catalyst 6k power supply, and asked Veena (not her real name) to help me on the case.

We got the customer on the phone.  He was an engineer at a New York financial institution, and sounded like he came from Brooklyn.  I lived in Williamsburg for a while with my mom back in the 1980’s before it was cool, and I know the accent.  He explained that he had a new 6k, and it wasn’t recognizing the power supply he had bought for it.  All of the modules had a “power denied” message on them.

I put the customer on speaker phone in my cube and Veena looked at the case notes.  As was often the case in TAC, we put the customer on mute while discussing the issues.  Veena thought it was a bad connection between the power supply and the switch.

“Here’s what I want you to do,” Veena said to the customer, un-muting the phone.  “I used to work in the BU, and a lot of times these power supplies don’t connect to the backplane.  You need to put it in hard.  Pull the power supply out and slam it in to the chassis.  I want to hear it crack!”

The customer seemed surprised.  “You want me to do what?!” he bristled.

“Slam it in!  Just slam it in as hard as you can!  We saw this in the BU all the time!”

“Hey lady,” he responded, “we paid a couple hundred grand for this box and I don’t want to break it.”

“It’s ok,” she said, “it’ll be fine.  I want to hear the crack!”

“Well, ok,” he said with resignation.  He put the phone down and we heard him shuffle off to the switch.  Meanwhile Veena looked at me and said “Pull up the release notes.”  I pulled up the notes, and we saw that the power supply wasn’t supported in his version of Catalyst OS.

Meanwhile in the background:  CRACK!!!

The customer came back on the line.  “Lady, I slammed that power supply into the chassis as hard as I could.  I think I broke something on it, and it still doesn’t work!”

“Yes,” Veena replied.  “We’ve discovered that your software doesn’t support the power supply and you will need to do an upgrade…”

2
by

Recertification Pain

Note:  This article was written in 2016 and has not been modified.  A number of changes have been made to the CCIE program which have dramatically improved the re-certification process.  Continuing education is now an option, as I suggest in this article.  The re-certification frequency has been reduced.  I may modify this article in the future, but I am leaving it as is for now for historical purposes.  However, please note the description of the process is no longer accurate at all.  (ccie14023, Sept 2021)

 

In this installment of “Ten Years a CCIE,” I look at what you have to do to stay certified, and the difficulty of maintaining your credential.

Passing your CCIE gives you a great feeling of accomplishment, and also a sense of relief.  You’ve spent months studying and late nights configuring scenarios in the lab.  Maybe you took the exam multiple times, and had to experience the letdown of knowing that, instead of being finished, you had more months of studying ahead.  So, you’ve finally passed, and it’s all over, right?

No, unfortunately.  You have a CCIE, but if you want to keep it, you have to worry about hitting the books again every two years.  All CCIE’s have to re-certify, a biennial ritual that becomes harder as the years go by.

Here’s how it works.  Before two years after your lab date, you have to re-certify your CCIE by passing a CCIE written exam.  You can take any written exam, just as long as it is a CCIE written.  For example, if you passed Routing and Switching, you could recertify by taking the Data Center written exam.  This has the advantage of simultaneously qualifying you for another lab exam, if you are so inclined.  If you have more than one CCIE, you can recertify all of them by taking any CCIE written.  For example, if you have Routing/Switching, ISP Dial, and Collaboration CCIEs, you could recertify all of them at once by taking the Wireless written.  This holds true even though ISP Dial is no longer a valid certification.  Even if you only have a certification that no longer exists (such as ISP Dial or SNA IP), you can maintain active CCIE status by passing any written exam.

If you don’t pass a written exam, at the two year mark your certification becomes suspended.  You can no longer use your CCIE number in your signature or claim to be a CCIE.  You can still pass the recert exam within a year, but if a year elapses after you go suspended, you lose your CCIEs, all of them, and have to retake both written and lab for any CCIE you hold.  Needless to say, you don’t want that to happen.

recert

What you want to see when you verify your CCIE…

(For comparison, my JNCIE-SP expires every three years, and I have to take the JNCIP-SP exam to recertify.  If I had a JNCIE-ENT as well, I would have to take both exams to recertify.)

If you just passed your lab exam and you feel super-confident, you may think you don’t have to worry about a measly written exam in two years.  However, any CCIE will tell you the recertification ritual is onerous and a huge waste of time.  As your career advances, you will often find yourself doing less and less CLI, and you might in fact work less with Cisco products.  In my case, re-certifying became especially painful during my six years at Juniper.

It would be less of a burden if the exams were better written.  The last time I took the written, there was a question that was flat out wrong, and many that were just obscure.

I first wrote this entry in 2014, and I am now re-writing it two years later.  When I first wrote it, I was working on my recert and in a state of extreme annoyance, came up with a couple of sample questions intended to mimic the actual exam:

When is the MSDP ConnectRetry timer used?
a.  When the MSDP peer with the highest IP address transitions from the INACTIVE to the CONNECTING state.
b.  When the MSDP peer with the lowest IP address transitions from the CONNECTING to the ESTABLISHED state.
c.  When the MSDP peer with the lowest IP address transitions from the INACTIVE to CONNECTING state.
d.  When the MSDP peer with the highest IP address transitions from the CONNECTING to the ESTABLISHED state.

What is the RSVP message type for a PathTear message?
a. 4
b. 0
c. 5
d. 3

What does the “ipv6 mld limit 100″ command do?
a.  Limits the number of hosts that multicast listener discovery can discover to 100
b.  Limits the hosts permitted by MLD to those contained in ACL 100
c.  Limits the number of MLD states to 100 on a per-interface basis.
d.  Limits the number of MLD states to 100 globally.

At the time I wrote them, these questions were technically within the blueprint topics for the Routing and Switching written exam, but they are obviously rather stupid questions.  The R&S blueprint is so huge that it is essentially impossible to know all of the subjects it covers.  Nevertheless, I was encountering questions of roughly this level of obscurity on the exam.

The purpose of recertification

Why do we have to recertify?  Obviously, the main reason is to ensure CCIE’s stay current in the field.  I passed routing/switching back in 2004, and a lot has changed in 12 years.  It’s important that people who come to me for expertise believe that I actually have relevant knowledge.

We have to ask a question though:  how well do you stay up-to-date taking a written exam every two years?  And why can you keep your credential when you re-certified in a different track?

For example, if someone acquired a CCIE Security certification back in 2002, but re-certified for 14 years using the routing/switching written, why is that engineer qualified to continue calling himself a “CCIE Security”?  He probably knows nothing of modern security technologies.  Juniper requires JCNIE’s to recertify in each track they have certified, so a triple JNCIE has to take three separate exams.  While this is painful (and kept me to one JNCIE), it makes more sense.

I think an even more reasonable approach is to allow continuing education in lieu of a test.  This is the requirement for CISSPs, lawyers, and even doctors, and it makes a lot of sense.  I never remember much from the recert exams, but a couple days of training would be a great way to get current.

I do think Cisco was smart to introduce the Emeritus option.  CCIE Emeritus allows CCIE’s who have hit the 10 year mark to pay a fee to keep their number in a non-active status indefinitely, with the option to recertify.  Many CCIEs reach a point where they don’t deal with day-to-day CLI configuration, and find the exams harder and less relevant to their careers.  Several of my friends have chosen this option.  I almost did when I worked at Juniper, but I am thankfully still current.

By the way, the answer to all of the above questions is ‘C’.

In my next article, Cheaters, I look at the question of whether people cheat on the CCIE exam, and the effect it has on the value of the certification.

by

TAC Tales #6: The best organization tool ever

I’ve come back to Cisco recently, and I think I can say that I haven’t worked this hard since the last time I was at Cisco.  I remember my first manager at TAC telling me in an interview that “Cisco loves workaholics.”  In an attempt to get more organized, I’ve been taking a second crack at using OmniFocus and the GTD methodology.  To be honest, I haven’t had much luck with these systems in the past.  I usually end up entering a bunch of tasks into the system, and then quickly get behind on crossing them off.  I find that the tasks I really want to do, or need to do, I would do without the system, and the ones that I am putting off I keep putting off anyway.  I have so much to do now, however, that I need to track things more efficiently and I am hoping OmniFocus is the solution. Continue reading →

4
by

A CCIE Goes Home to Cisco

In this article in my “Ten Years a CCIE” series, I describe my experience going to work at Cisco as a CCIE.  Unlike many Cisco-employed CCIE’s, I earned my certification outside of Cisco.

A CCIE leads to a job at Cisco

I returned to my old job at the Chronicle and had my business cards reprinted with my CCIE number. I loved handing it out, particularly at meetings with telephone companies and Internet service providers whose salespeople were likely to know what such a certification meant.  I remember one such sales person, duly impressed, saying “wow, on that test you can be forced to configure any feature on any Cisco product…I don’t know how anyone could prepare for that!”  (Uh, right.  See “The CCIE Mystique“.)

At that time, the most popular forum for aspiring CCIE’s was an email distribution list called groupstudy.com. I had been a subscriber to this mailing list, but prior to passing my exam, I didn’t feel adequate to post anything there. However, once I passed, I began posting regularly, beginning with a summary of my test preparation process. One day I got an email from a mysterious CCIE who told me that I sounded like I knew what I was talking about, asking me if I wanted to interview for a job. I thought his name and number sounded familiar, and when I got home I confirm my suspicions by digging through my bookshelf. He was the author of one of my books about Catalyst Quality of Service.

A brutal interview

It turns out he was a manager at Cisco High Touch Technical Support, a group of TAC engineers who specialized in high profile customers. I scheduled an interview right away.

This interview was by far the most difficult I’ve had in my career. They brought me into a room with four CCIE’s, two of them double, all of them sharp. Each one of them had a different specialty. One of them was a security guy, another one was an expert on multicast, another was an expert on switching. When it came to Kumar, the voice guy, I figured I was scot-free. After all, I didn’t claim to know anything about voice over IP. Kumar looked over my resume, and then he looked up at me. “I see you have ISDN on your resume,” Kumar said. And then he began to grill me on ISDN. Darn, I should have thought of that!  Thankfully, I was well prepared.

Despite one or two mistakes in the interview, I got hired on and began my new job as a customer support engineer at HTTS. My first few months were in a group called ESO, which supported large enterprises and was very focused on Catalyst switching.  I won’t go into the details of the job here, but you can see my many TAC tales if you are interested.

Cisco's San Jose campus

Cisco’s San Jose campus

Little purple stickers everywhere!

One thing I quickly noticed when I got to Cisco was that a lot of the people in my department had a nickel-sized purple dot on their ID badges and cubicle nameplates. I found out that these purple dots were actually stickers with the CCIE logo. Cisco employees who had their CCIE’s stuck these purple dots on their badges and nameplates to show it off. Many of the CCIE’s who had passed multiple exams actually placed multiple dots on their badges and nameplates. I wanted one quite badly. The problem was, sheets of these purple stickers were sent out only to the early CCIE’s, and by the time I had passed, Cisco was no longer providing the sheets of stickers. I suppose I could’ve had some printed out, but I asked around looking for a CCIE who was generous to give me one of his dots. They were in scarce supply, however, and nobody was willing to part with one. It was just another way newer CCIE’s were getting jipped.

The real CCIE logo

The real CCIE logo

Even though the exam had now switched to the one day format, you still didn’t meet too many CCIE’s outside of Cisco. It was thus quite a shock when I went to Cisco and saw purple dots everywhere. It seemed like fully half of the people I was working with in my new job had CCIE’s. And many of them had low numbers, in the 2000’s and even in the 1000’s. I was quite relieved to find that they all treated me with total respect; nobody ever challenged me on account of my one-day CCIE. Still, I always had (and always will have) a great deal of respect for those people who passed the test when it was a two day test, and the cottage industry devoted to minting CCIE’s had not yet come into existence.

CCIE challenges customer

Customers were another story. I remember one BGP case in particular. I looked at the customer’s configuration and immediately realized that it was a simple matter of misconfiguration. I fired it up in my lab reproduced his configuration and proved to him that it was indeed a configuration error on his part. I wrote it all up in an email and proudly signed it with my CCIE number. Within a half an hour I got a call from the customer and one of his colleagues on the line. They proceeded to grill me rapidly on BGP asking me all sorts of questions that weren’t relevant to their case and stumping me several times. At that point I realized that when many people see you are a CCIE, they take it as a challenge. In some cases they failed the test themselves, or else they’ve met stupid CCIE’s in the past and they feel themselves to be on a mission to discredit all CCIE’s. After that episode, I removed my CCIE number from my email signature. I gained a feeling of self-importance after I passed my exam, but working among so many people with the same certification, and dealing with such intelligent customers, I realized that the CCIE didn’t always carry the prestige I thought it did.  The mystique diminished even further.

Incidentally, I became friends with all of the guys who interviewed me, and I was on the interview team myself during my tenure at Cisco. One extremely sharp CCIE we hired told me our interview was so tough he had to “hit the bottle” afterwards. It was considered a rite of passage at TAC to go through a tough interview, but I have gotten a lot nicer in my interview style now, having been on the receiving end of a few grillings.

The value of a CCIE

One of the later posts in this series will examine the question of the value of a CCIE certification.  After all, this is one of the most common questions I see in forums dedicated to certification.  However, my experience getting hired into Cisco (the first time) has some lessons.

  • The immediate reason I got hired was because of my experience and willingness to go out of my way helping others to get their CCIE on Groupstudy.  However, I would not have gotten the position without a CCIE, so clearly it proved its value there.
  • Once you are at Cisco, although people commonly display their stickers and plaques, having a CCIE certification will not necessarily distinguish you.
  • There are many CCIE’s who have made a bad impression on others, whether they are only book-knowledgeable, or even cheaters.  Often people challenge you when you have a CCIE, instead of respecting you.

In the next article in the series, Multiple CCIEs, Multiple Attempts, I describe passing the CCIE Security exam.  I talk about my experience suffering the agony of defeat for the first time, and how I eventually conquered that test.

by

The CCIE Mystique

In my first article in the “Ten Years a CCIE” series, I discuss the mystique of the CCIE certification which made me want to attempt the test.

Learning about the CCIE

My first vague awareness of the CCIE certification came in 1999 while I was a Master’s student in Telecommunications Management at Golden Gate University in San Francisco. A family friend was staying at my father’s house, an instructor in the PhD program in telecommunications at the University of Idaho. I was excited to meet him because I was thinking of pursuing further graduate studies, but I was a bit surprised when I told him of my ambitions to be a network engineer, and of my coursework at GGU. He told me not to waste my time in a graduate program if I wanted to be a network engineer. “You should get a Cisco certification instead,” he said, “they’re gold.” A disappointing comment, seeing that I was in my second year of the Masters program, but I kept it in mind and completed my degree. Continue reading →

2
by

10 years a CCIE

When I approached my tenth anniversary of first passing the CCIE routing/switching exam (November 2004-2014), I had the idea to post some short reflections on the exam, its value, and my personal experience being a CCIE. I hoped that, although the nature of the exam has changed quite a bit over the last 10 years, these reflections would provide some useful information to those who are preparing to take it. I also hoped that the historical information will prove useful and/or entertaining to newer candidates, but that it also will be a nice walk down memory lane for older CCIE’s.

It turned out that short became long. I was surprised by how much there was to say. I wrote these pieces in 2014, and having done so, decided they were a bit self-indulgent and uninteresting.  I shelved them in my drafts folder.  Now, two years later, I have re-read them and decided that perhaps they have the value I had initially hoped for.

So, I have decided to go ahead and publish my “Ten Years a CCIE” series.  I won’t publish them at once, but one at a time over the next months.  I hope that you find them interesting and a little entertaining.  Good luck to those of you who are starting your own CCIE journey, and I look forward to reading your stories ten years from now.

Ten Years A CCIE, by Jeff McLaughlin #14023

The CCIE Mystique  –  Published 2/3/16
Routing and Switching:  An exam in flux – Published 2/23/16
In those days, you had to build a lab – Published 3/6/16
How to pass the CCIE lab exam in one attempt – Published 3/10/16
Room of horrors:  Inside the CCIE lab – Published 3/15/16
A CCIE goes home to Cisco – Published 3/28/16
Multiple CCIE’s, multiple attempts – Published 4/4/16
Recertification pain – Published 5/2/16
Cheaters – Published 5/14/16
The value of a CCIE – Published 11/2/16

by

Tac Tales #5: MWAM

New Year’s resolutions are made to be broken, and I haven’t been keeping up with my resolution to do more blog posts.  Now that I am back at Cisco, I am focusing on programmability and automation, and I do have a lot to say.  However, in honor of my return to Cisco, I thought I would post a new Tac Tales entry.  There is a moral to this story.

One day my boss came to me and said my team would be supporting the MWAM module in the Cat 6K.  I had done a lot of Cat 6K work at that point, but I had never even heard of an MWAM, and failed to see why cases on it would be sent to the routing protocols team.  My boss didn’t seem too concerned with my objections, and said, “Just go watch the VoD.”  VoD = Video on Demand.  So, I did.  I watched the VoD, and it started out by telling me how many processors were on the card and of what kind;  what types of buses were used to transmit data;  what kind of memory it had;  and how it interfaced with the Catalyst and its backplane.  Never did the video ever tell me what the card actually did.  I had no idea why one would buy an MWAM or what one would do with it.  I hoped a case wouldn’t come in on the card, and when it did I immediately escalated to engineering because I had no idea what to do.  Fortunately I only ever had one case on the MWAM.  (And the fun thing about coming back to Cisco after 10 years is that I can go look up all these cases I remember and read my notes.  Very cool!)

What is the moral, you ask?  Well, as a Technical Marketing Engineer, a big part of my role is communicating technical concepts clearly to others.  How often have you bought a book or looked at a web page to learn some new protocol, only to find that the description of it begins with packet header formats or state machines?  Fine, but tell me what it actually does before you tell me how it works.  Imagine if you went out into the jungle and encountered someone who has never seen a car.  You wouldn’t start explaining it to him by saying “it uses an internal combustion engine which has a four-stroke cycle of intake, compression, power, exhaust.”  You’d say, “it has wheels and takes me places very fast.”  Now in defense of the MWAM VoD guy, he may have designed his video for people who already knew what the card was.  But often I have found that people make this assumption, and when I backtrack and start at the beginning when explaining something, often people say, “you know, I’ve always been afraid to ask about that, but thanks for explaining it.”

Meanwhile, my second try at Cisco is much more fun than the last.  And thankfully, no MWAMs.  TAC was an great experience and period of growth, but it’s a not a fun job.

1
by

Goodbye Juniper, Hello Cisco

I feel a bit of guilt for letting this blog languish for a while. I can see from the response to my articles explaining confusing Juniper features that my work had some benefit outside my own edification, and so I hate to leave articles unfinished which might have been helpful. In addition, WordPress is not easy to maintain and I keep losing notifications of comments, which means that when I am not logging in, I miss the opportunity to respond to kind words and questions.

As it is, my work explaining Juniper to the masses will have to be put on hold, as I have left Juniper after six years and returned to my old employer Cisco! I worked at Juniper longer than I had anywhere else, and it’s amazing to consider that I just closed the door on half a decade. But, I even after attaining my JNCIE I always felt like a Cisco guy at heart, and so here I am again. A few random thoughts then:

1. I interviewed for a number of jobs, and now that I am hired I can say that I really hate interviewing. My interviews at Cisco were very fair and reasonable. Just for the heck of it I did a phone screen with Google and completely bombed it. I’m not ashamed to admit that. I’m not supposed to reveal their questions, and I won’t, but they were mostly basic questions about TCP functionality, and MAC/ARP stuff, and it’s amazing how you forget some of the basics over the years. I wasn’t really interested in working there so I did no preparation, and in fact the recruiter warned me to brush up on basics. I just figured my work and blog show that I am at least somewhat technical. I plan to write some posts on the art of technical interviewing, but I was certainly underwhelmed by Google’s screening process, as I’m sure they were by my performance. I really wanted the Cisco job, and what a difference attitude makes! (Oh, and I completely munged an MPLS FRR/Node & Link protection question, less than a year after passing the JNCIE-SP. Uh, whoops.)

2. I bear Juniper no ill will. It was an interesting six years. When I came on board, during the Kevin Johnson years, it was all rah-rah pep talks about how we were going to be the next $10 billion company (errr, no…) followed by a plethora of product disasters. Killing off Netscreen gave the firewall market to Palo Alto, Fortinet, and amazingly resuscitated Checkpoint. Junos Space was a disaster, and Pulse slightly less so. QFabric was not a bad idea, but was far too complex. You needed to buy a professional services contract with the product, because it was too complex to install by itself. And yet it supposedly simplified the data center? There was a fiasco with our load balancer product. And then came the activist investors with their Integrated Operating Plan. I will permanently loathe activist investors. Juniper was hurting and they just magnified the hurt. There’s nothing worse than a bunch of generic business-types who wouldn’t know a router if they saw one trying to tell a router company how to do its business. They thought they could apply the same formula you learn in B-school to any company no matter what it manufactures or does. Then we had the CEO revolving door.

Despite all of this, as I said, I like Juniper. I did ok there, and there are a lot of people I respect working there. Rami Rahim is a good choice for CEO. I left for personal reasons. They still have some good products and good ideas, and I think competition is always good for the marketplace. For the sake of my friends there, I hope Juniper does well.

3. If you read my bio, you will see that I was THE network architect for Juniper IT, meaning I covered everything. This included (in theory at least) campus LAN, WAN, data center, wireless, network security, etc. I did something in all of these spaces. It was a broad level of knowledge, but not deep. That’s why I did my JNCIE-SP–I was hungering to go deep on something. My new job at Cisco is principal technical strategy engineer for data center. This is an opportunity to go deep but not as broad, and I’m happy to be doing that. The data center space is where it’s at these days, and I can’t wait to get deeper into it.

4. Coming back to Cisco after an eight year hiatus was bizarre. It was cool to pull up all my old bugs and postings to internal aliases to see what I was doing back then. Heck, I actually sounded like I knew a thing or two. I was thrilled to find out I am on the same team as Tim Stevenson, whose work as a Cat 6K TME I admired when I worked in TAC. Just for fun I walked though my old building and floor (K, floor 2) and nearly fell over when I saw that it looked identical. I mean, not only the cubes, but there were these giant signs for the different teams (e.g. “HTTS AT&T TEAM”) which were still hanging there as though the intervening eight years had never happened.

Unfortunately, I have to leave a few in progress articles in the dustbin. First, I shouldn’t really be promoting Juniper now that I am working for Cisco. And second, I’ve lost access to VMM, the internal Juniper tool I used to spin up VM versions of Juniper routers. However, I hope to start posting on Cisco topics now that I have access to that gear. Cisco’s products are generally better documented than Juniper’s, but I promise to fill any gaps I might find. And I will leave my previous articles up in hopes that they will benefit future engineers who struggle with Junos.

Onwards!