In this article in my “Ten Years a CCIE” series, I look at passing the Security exam in 2008.  I get to experience the agony of failure for the first time, and have to re-tool my strategy.

Goodbye to Cisco

I worked two long years at Cisco. Two very long years. I learned so much there but it was a brutal job. The relentless flood of new and challenging cases grew tiresome.  When my aforementioned sushi eating CCIE friend called me in 2007 and invited me to come join him at a Gold partner I couldn’t say no.  Cisco sells much of its gear through value added resellers (VARs), also known as partners.  These partners are assigned different levels depending on the amount of business they do, and Gold is the highest.

Working at a gold partner with a CCIE was quite enjoyable. Gold partners need CCIE’s and so they have a lot of incentive to make you happy. My boss suggested that I get a second CCIE, this time in voice. I started to buy material for the voice exam, when my VP showed up in the office, fired my boss, and told me to start studying for the security exam. (His firing of my boss had nothing to do with my CCIE exam, but it certainly made me stand up and listen to what he was asking.) So, having really not started on voice, I switched immediately to security.

I had already passed the security written back at Cisco partly to qualify for the lab exam, and partly to re-certify my existing CCIE, so it was straight to the lab exam for me. The equipment list was a big challenge. At that time, you needed two ASA’s, one PIX, a VPN 3000 series concentrator, and IDS device, six routers and two switches, and some sort of Windows server running Cisco secure ACS. I still had my old lab equipment from before, but I was missing everything else. I had one ASA 5505 from work, but no other security devices. I decided that the cost was too prohibitive for me to set up my own lab. I was going to have to use rack rentals. That was my first big mistake.

I decided to approach the exam in exactly the same way I approached the routing and switching exam. I studied the various subjects on the blueprint individually, and then started doing full labs from the Internetwork Expert workbook. As great as IE’s workbook was for routing and switching, in 2008 it really wasn’t very good for security. I have a lot of respect for the Bryans, and I’m sure it’s come a long way, but at that time it just wasn’t enough.

Attempt number one

When I showed up at the familiar CCIE lab, I didn’t feel well prepared, because I wasn’t. The lab was a disaster. I only managed to complete about a third of the exam. While configuring DMVPN, all of my routers locked up and crashed. I called the proctor over, and when he saw that the console ports were locked up, he started to accuse me of having made a configuration error. I explained that I hadn’t touched the console configuration, and just then we both saw bus errors appear on the console sessions followed by reloads. It was obvious then that I was not at fault. I had heard that if routers crash during CCIE exam, the proctor will give you your time back. However, the proctor admonished me to save my configs frequently, and refused to give me any time back. I had probably lost 15 minutes. I would have fought it, except that I was already so far behind on the exam,  I knew it would make no difference. Still, to this day am a bit angry at that proctor. As I left the exam room I looked at him and said “don’t even bother grading this.” He looked at me and said, “Oh, I’m sure you’re exaggerating.” I looked at him and told him I hadn’t completed two thirds of the exam.” Oh!” He exclaimed.” Well… Don’t wait six months for your next exam!”

… It was six months before my next attempt.

The author’s 2008 CCIE security lab. The laptop ran Windows Server in a VM for ACS. An ASA 5505 is visible on top of the drawers.

Changes to my approach

I knew I had to revise my strategy. Something wasn’t working. The first thing I fixed was the lab situation. When I did Routing and Switching, I knew that I needed my own lab at home. Using remote rack rentals for security just didn’t give me enough time in the lab. I managed to get a hold of the PIX from a friend who was decommissioning it. I bought myself an ASA 5510, which, at $2500, was the most expensive piece of hardware I had. I really needed two of them, in order to cluster them, but I had to make do with the mismatched pair of the 5510 and 5505. As with the Routing and Switching exam, I knew I could use remote rack rentals to fill in for the equipment that I didn’t have. The ASA 5505 was adequate for basically everything except clustering. It had almost all of the capabilities of the 5510, but the configuration of VLANs was slightly different.  I also managed to acquire an IDS, and VPN 3000 series concentrator. I borrowed a laptop from work and got a Windows server license and managed to install Cisco Secure ACS. I ended up with a very complete lab.

I realized that a big part of my problem was that IPSec configurations are long, complicated, and counterintuitive. IPSec is the core of the CCIE security exam, and you need to know it as well as BGP and OSPF on the routing and switching exam. I made a series of diagrams which depicted each of the constituent configuration elements for the various IPSec technologies as blocks, which were then connected together by arrows. For example, for basic IPSec configuration, I would have one block representing the IKE configuration, and another representing the IPSec policy. I would draw an arrow to show how they were connected, labeling the arrow with the command used to connect them. Before I was trying to memorize these configurations. Now I was able to visualize them.

Visualizing complex configurations helps make them easy to understand and remember

I also completely abandoned using the IE workbook. It just wasn’t ready at that point. Instead I invented my own VPN challenge lab. It had every kind of VPN on it: IPSec on ASA, IPSec on PIX, IPSec on VPN 3K, client IPSec on all of those platforms, L2TP, PPTP, DMVPN, SSL. I worked this lab over and over again until I could configure all of these automatically, and I made sure I configured between disparate platforms.

I felt good but not 100% prepared when I went to take my second attempt. I failed, but my score was much higher than before. I continued preparing for another month or so before taking my third attempt. I was so ready for my third attempt, that I completed the lab shortly after lunch. As I was coming out of the bathroom, I ran into Ted the proctor (not his real name), in the hallway. I had seen Ted on my second attempt and he told me he was attending a bluegrass Festival in San Francisco. I spent a good 15 minutes talking to Ted about the festival in the hallway, and I think at that point Ted realized that I was feeling pretty confident. Most people don’t spend 15 minutes shooting the breeze in the middle of the CCIE exam.

Interestingly enough, while Ted had been the most helpful Proctor on the Routing and Switching exam, he was of almost no help at all on the security exam. I’m not sure if he had changed in the intervening four years, or if he simply wasn’t as familiar with the security exams that I took. Either way, be prepared to make difficult decisions on your own in the lab, without the help of the proctors. Of all the questions I asked them, only once did I get a useful answer. I realize that their job is not to give away the test, but often the test is poorly written and I think that they need to be more helpful in explaining the exam.
Passing Routing and Switching was exciting; passing Security was a relief. I had almost given up after my disastrous first attempt. And I’m glad that I passed it when I did. As with the Routing and Switching exam, I passed Security in November. And as with the Routing and Switching exam, Cisco was changing the test at the beginning of the new year. The VPN 3000, PIX, NAC framework, and several other technologies were being removed. Of course, they never removed technologies without adding some as well. Had I failed my third attempt, it’s likely I would never have tried again.

In summary:

• Having “always-on” access to a lab is critical!  Remote rack rental is good to fill in for a few things you might be missing, but don’t rely on it.
• You may have to spend some more money than you want to acquiring gear, but it pays off.
• The way you pass one CCIE exam is not necessarily the way you pass another exam.  You have to spend some time looking at the topics you will be covering, figuring out the best way to reach the point of automatic configuration of the technologies.
• Sometimes, the study material from the vendors just won’t cut it.
• Proctors aren’t always nice, and don’t always do what you thought they were supposed to.

I will cover the question of lab blueprint changes in a later article on the value of a CCIE, but it’s worth noting that for both my routing/switching and security exams, a blueprint change happened immediately after my passing.  I spent a lot of time studying, for example, the VPN 3000 concentrator which was already obsolete.  Still, I would have the same credential as a guy who passed the exam with the new technologies a couple months later.

Also worth noting:  I passed all of my expert exams (2 CCIEs and a JNCIE) in November.

In the next article in the series, Recertification Pain, I look at the biennial penance we all inherit for passing our CCIEs–the dreaded recertification.  I give my thoughts on improving the process, not that anybody is listening.