Skip navigation

How to pass the CCIE lab exam in one attempt

In this post in the Ten Years a CCIE series, I go over my preparations for the CCIE Routing and Switching exam, and what I did to pass in one attempt.

The first months…

I passed my CCIE Routing and Switching Lab in one attempt, so I think my approach can be considered effective. At least, it was for the exam at the time. I decided to spend my first several months of study diving deep into each of the exam topics on the blueprint. I was determined to focus on core technologies such as BGP and OSPF and to minimize the amount of time spent on ancillary topics such as DLSw. Because you have access to the documentation CD in the lab, you don’t need to know absolutely everything. However, you do not want to spend a long time trying to figure out how to configure core tasks which you should be able to do automatically.

I didn’t work from a particular manual or outline these first few months. Instead I would pick a topic, say BGP. I would go through all of the examples I could find in the books that I had, Jeff Doyle’s books being the most helpful. I would set up the examples from the books in my lab to see if they work as described. Then I performed free-form experimentation. I tried different things; I indulged my curiosity; I came up with new ways to test the protocols and tried to break them. I introduced loops where there weren’t loops in the examples I had. I saw what happened if I ran the protocol over ISDN instead of Frame Relay. And I made very sure that everything I learned I recorded in my notes. For every subject I kept two note files. The first file contained general, conceptual notes. The second file was a list of commands that I thought were important and I needed to remember. These files grew over time, and I studied them thoroughly before attempting the lab.

I had also acquired practice labs from three different sources. I had IP Expert’s lab book; I also had Internetwork Experts’ lab book; and finally, I had the Cisco press official lab book, which was written by a CCIE proctor. I found that this last book’s labs most closely resembled the real thing in terms of how the labs were written and how the diagrams were drawn. Still, as I studied I quickly came to favor the Internetwork Expert book for its thoroughness and accuracy. At that time, they were still relatively new, but the quality of their material was the best.

Closing in on test day…

In the last couple of months before the exam, I shifted my strategy. Instead of focusing on individual topics I spent my time working the practice labs in the IE book. At first I worked them slowly and methodically. I didn’t do them on a timer, and I didn’t rush through them. If it took me 24 hours to work through lab then it took me 24 hours. My main interest was in covering the material, understanding it thoroughly, and in documenting my learnings. I knew so many people who started giving themselves timed exams when they weren’t ready for them. Yes, it’s important to have a strategy and to understand clock management, but it’s far more important to understand the material thoroughly. The best time management strategy is knowing the material so well you can configure most of it on auto-pilot.

Every time I completed the lab I graded myself using IE’s answer key. I used to say that I was my own worst enemy. I never gave myself a pass on the slightest discrepancy between my solution and IE solution. Every single mistake that I wrote I listed out in a document, and in the last few weeks before the exam I reread that document several times every day. Constantly reviewing the mistakes I had made reinforced my own errors in my mind.  I also found that in my note documents that I was highlighting certain important points or gotchas with the capital words “BE SURE”. I created another document that I called my “BE SURE” list. I also reviewed this list several times a day in the last few weeks before the exam. Reviewing both my mistakes as well as my “BE SURE” list so frequently was quite effective in helping me remember my mistakes and important notes.

A snippet of my BE SURE list

A snippet of my BE SURE list

When I was studying for my CCIE exam Cisco press had just released two handy books. These books covered all of the commands in IOS at that time for BGP and OSPF. Not only did they describe the commands but they had examples of their use as well. In the last few days before the exam I would review the table of contents of these books which listed all the commands by name. I did this every night in bed. If I was able to accurately describe the command, I would cross it off.  Some commands that I couldn’t remember I saw night after night, until they were so familiar I had no problem using them.  Doing this every night helped me to commit fully to memory all of the different BGP and OSPF commands that make up the core of the CCIE lab exam.

I also took the CCIE Lab Boot Camp from Internetwork Expert just a few weeks before I took the actual lab exam. This was a wonderful experience. I was able to take the course from home, using IE’s Java-based virtual environment. Because most of the work and the class consisted of full, eight-hour timed labs, there was no need to travel to a classroom. And, because the eight hour exams were administered on Internetwork Expert’s own racks of equipment, there was no problem with not having a full CCIE lab at home. We had a small amount of lecture each day, followed by the eight hour lab, which was then graded each night. In the morning we were given our results. I was told that people scoring over 80% generally passed the CCIE lab exam, and I was scoring higher with no problem. The Brians gave me some great advice and particularly fixed some problems that I had in configuring multicast.

At the end of the boot camp Brian Dennis, the grumpier of the Brians, gave what I would charitably call a pep talk. He told us that a test is just a test, that we should get some of the classic books on networking and study them thoroughly, and that we should know our subject, not simply pass the test.  “You meet some CCIEs and wonder, how did this guy pass the test?” Brian said.

In November 2004 the time came to take the test. I had no idea if I was ready. A good friend of mine who passed shortly before spent four hours with me in a sushi restaurant grilling me on every possible subject that could be on the exam. They closed the restaurant on us.  For my final preparation, I studied all of the new features in IOS which they were now using in the CCIE lab. I also studied the documentation CD thoroughly so that I would have no trouble navigating it in the lab.

Passing the test

If you’re working on the CCIE exam, why should you care what someone did to prepare for it ten years ago?  Well, as I’ve said, it is a different test now.  My advice on learning ISDN dial maps isn’t going to help you.  However, there are some general principles here that you should pay attention to.

  1. Figure out the core topics and learn them well.  Cold.  On every expert exam, there are some core topics and some ancillary topics.  You cannot know everything.  Figure out the core topics and drill them over, and over, and over again.  You need to be able to configure them without thinking.
  2. Make things harder than they have to be.  As I said, break things intentionally.  Introduce problems.  Ask questions.  Don’t just run the scenarios you bought with your labs.
  3. Be your own worst enemy.  Remember, the CCIE exam is not just about doing what they tell you, but doing exactly what they tell you.  When you grade yourself, read and re-read the tasks.  Make absolutely sure that you have accurately and completely fulfilled the requirements.
  4. Document your mistakes.  Review things you have done wrong, and keep reviewing them.

In the next post in the series,  Room of Horrors, I describe the CCIE lab experience.  I talk about what it was like to enter the infamous lab in Cisco Building C, and take the challenging exam.

In those days, you had to build a lab…

In this article in my series, Ten Years a CCIE, I discuss the challenge of building and maintaining a lab, and the question of building versus not building a physical lab.

Acquiring Gear

As I mentioned in a previous post, GNS3 was not available at the time I took the CCIE exam. This meant that the CCIE candidate had to acquire hardware — quite a lot. This was one way that the number of CCIE’s was limited, even after the institution of the one-day format. I acquired equipment by various means; mostly by purchasing old routers off eBay. Some I was able to borrow from work. Some I bought directly from coworkers.

My lab consisted mostly of 2500 series routers. These were adequate to test most of the routing protocol features needed for the exam. However, some of the newer or more advanced features required more advanced routers, and for these I used 2600 series routers. These were essential for testing Voice over IP, as a 2500 did not support the hardware (FXS/FXO cards) required to do dial peers. In 2004, Frame Relay was still a big part of the CCIE exam, so I decided that a Frame Relay switch was absolutely necessary. For this I acquired a 2500 series router with a lot of serial ports. It was fortunate that I didn’t need to purchase a WAN emulator of some sort, since Cisco included Frame Relay switching capability in its IOS software and WAN emulators were ridiculously expensive.

Early on I determined that a terminal server was necessary. I started out thinking it wasn’t, but moving your console cable around from router to router gets old very quickly. Luckily, a 2500 series router with a lot of serial ports, designed primarily for modems, made an excellent terminal server. Instead of having to console individually to each router, I could simply telnet to the terminal server on a particular TCP port number to access the individual devices.

Ethernet switching was a bit more difficult. At that time, the CCIE equipment list had Catalyst 3550 switches on it. The 3550s were quite expensive and I couldn’t afford any. Luckily, we had a spare at work which I borrowed, but only one. This meant that for spanning tree and trunking configuration, I had to configure one end only and hope that the configuration would work if there were a switch on the other end.  Building a physical lab often requires compromises, since it can be impossible to acquire or afford all the equipment.

ISDN and ATM

ISDN was a big part of the CCIE exam in the early 2000’s, even though it was becoming obsolete. It was also considered one of the more difficult subjects because of the myriad of ways of configuring it. I felt that it necessary to have some way to configure ISDN in my home lab. ISDN simulators were available, but were extremely expensive, and out of my price range. Some other CCIE candidates suggested to me that I purchase ISDN service for my apartment from the telephone company. This seemed to me like a huge hassle and it was also quite expensive. Luckily at just the time I started studying for this exam a small company came out with an inexpensive ISDN simulator. As I recall, it was only a few hundred bucks. I bought one, and for the most part it worked great. The one problem was that the on/off switch was labeled backwards. Easily fixed with a sticker.

The author's 2004 lab. Not neat! Note the but sets for VoIP testing, the 3000-series lying vertically on the right, and the small black ISDN sim on the table, center-front.

The author’s 2004 lab. Not neat! Note the butt sets for VoIP testing, the 3000-series lying vertically on the right, and the small black ISDN sim on the table, center-front.

I had no way of simulating ATM. Doing ATM required a Lightstream switch and expensive interface cards. As I mentioned earlier, ATM was a very small part of the exam. It simply didn’t seem worth it to spend a huge amount of money to configure a single interface.

Why a home lab?

Just because I didn’t have certain technologies available to me doesn’t mean I didn’t study them. At that time, there were a number of companies offering CCIE rack rentals (an example is here.)  These were either small entrepreneurs or some of the larger CCIE test prep companies, who had set up full-blown CCIE labs and rented out time on them. There were two or three that I used regularly to fill in on technologies I didn’t have, including ATM.

Why did I spend thousands of dollars on lab equipment for my home lab when I could have simply rented time from a rack rental company? This question will come up again when I discuss the CCIE security exam. Working with the remote labs was a pain. Every time you started up, you had to load your configurations back into the lab because someone else had been using it in the interim. When the time came to shut down your lab, you had to back everything up. A huge chunk of your paid-for time slot had to be devoted to config management.  In second place, you could only access the lab during certain scheduled slots that you had purchased. I felt then, and I was right, that being held to someone else’s schedule would be a major impediment to serious study. I therefore decided to build my own lab, and use rack rentals only to fill in technologies I didn’t have. This meant that whenever I had spare time I could flip on the equipment and study. I even set up remote access to my lab and left it running during the daytime, so that I could study in my spare moments at work. Any time I wanted, I could access my lab and the configurations I was working on would be right where I left them.  This was essential to my passing the exam.

The CCIE lab had six routers on it that time, and connected to your lab were three backbone routers. These backbone routers out of your control, but were simply route injectors. I had enough routers to simulate the core of the lab, but I did not have enough routers to simulate all of the backbone. I therefore had to get creative. I used my frame relay switch as well as my term server as backbone routers as well. Thus, they were doing double duty. The configuration was relatively easy with the FR switch since it was only doing layer 2 in the lab, but it got tricky with the term server, since I couldn’t change its IP address. The IP address of the term server was the address I telnetted to to access my lab.  So, I often ended up changing IP addresses in the scenarios I had purchased, so that I could use the access server as a backbone router. I also used an ancient 3000-series router running IOS 9 as a BB router. (IOS 12.1 was current at that time!)  It was hard to configure but it could do OSPF and BGP.

Maintaining a home lab was a lot of work. I had to re-cable it a number of times until I finally got the stable topology that I liked. I often had to deal with bent pins which were so common with the serial cables of the time. I had to do IOS upgrades and the occasional memory upgrade when I needed an IOS but didn’t have the memory to support it. Sometimes, this work distracted from the work of studying. However, all of it was valuable experience. I don’t think I would trade it for GNS3 any day. GNS3 certainly adds an element of convenience, and allows the student to focus on the lab and not on extraneous tasks. And, it certainly has some challenges of its own. But I do think that dealing with the physical hardware, upgrading it, cabling it, etc. — helped me to grow as a network engineer, and also improve my general troubleshooting skills. Of course, as I mentioned before, the expense and difficulty of working with physical equipment prevented a number of people from attempting the lab, thus keeping the number of CCIE’s lower than in the GNS3 era.

Modern candidates for R/S may be able to use GNS3, but candidates for the other exams, such as data center, will go through the same struggles as myself.  Data Center is particularly bad because the equipment required is so expensive.  It is almost impossible to procure it on your own.  Rack rental is one option, but is fraught with problems, as I mentioned above.  The only other option is to try to get your employer to help you build and pay for a lab.  Despite the many changes in the exam, some things remain the same I suppose!

My next article, How to Pass the CCIE Lab Exam in One Attempt, reviews my study and preparation for the Routing and Switching lab exam.

Routing and Switching: An exam in flux

In the second article in the “Ten Years a CCIE” series, I discuss the Routing and Switching written exam, and the changes to the CCIE exam in the early 2000s.

Passing the written

As was most common in the early 2000’s I attempted my Routing and Switching exam first. Having passed the CCNP exams, my first order of business was to study for the CCIE written exam. Even though I had heard the CCIE written exam was quite easy, I approached it much like I approached a graduate school research project. I didn’t care how easy it was, I wanted to learn the subject material well. I also didn’t want to fail. I had not yet failed a Cisco exam, and I made a pledge with myself: the first Cisco exam I will allow myself to fail will be the CCIE lab. Read More »

The CCIE Mystique

In my first article in the “Ten Years a CCIE” series, I discuss the mystique of the CCIE certification which made me want to attempt the test.

Learning about the CCIE

My first vague awareness of the CCIE certification came in 1999 while I was a Master’s student in Telecommunications Management at Golden Gate University in San Francisco. A family friend was staying at my father’s house, an instructor in the PhD program in telecommunications at the University of Idaho. I was excited to meet him because I was thinking of pursuing further graduate studies, but I was a bit surprised when I told him of my ambitions to be a network engineer, and of my coursework at GGU. He told me not to waste my time in a graduate program if I wanted to be a network engineer. “You should get a Cisco certification instead,” he said, “they’re gold.” A disappointing comment, seeing that I was in my second year of the Masters program, but I kept it in mind and completed my degree. Read More »

1

10 years a CCIE

When I approached my tenth anniversary of first passing the CCIE routing/switching exam (November 2004-2014), I had the idea to post some short reflections on the exam, its value, and my personal experience being a CCIE. I hoped that, although the nature of the exam has changed quite a bit over the last 10 years, these reflections would provide some useful information to those who are preparing to take it. I also hoped that the historical information will prove useful and/or entertaining to newer candidates, but that it also will be a nice walk down memory lane for older CCIE’s.

It turned out that short became long. I was surprised by how much there was to say. I wrote these pieces in 2014, and having done so, decided they were a bit self-indulgent and uninteresting.  I shelved them in my drafts folder.  Now, two years later, I have re-read them and decided that perhaps they have the value I had initially hoped for.

So, I have decided to go ahead and publish my “Ten Years a CCIE” series.  I won’t publish them at once, but one at a time over the next months.  I hope that you find them interesting and a little entertaining.  Good luck to those of you who are starting your own CCIE journey, and I look forward to reading your stories ten years from now.

Ten Years A CCIE, by Jeff McLaughlin #14023

The CCIE Mystique  –  Published 2/3/16
Routing and Switching:  An exam in flux – Published 2/23/16
In those days, you had to build a lab – Published 3/6/16
How to pass the CCIE lab exam in one attempt – Published 3/10/16
Room of horrors:  Inside the CCIE lab – Published 3/15/16
A CCIE goes home to Cisco – Published 3/28/16
Multiple CCIE’s, multiple attempts – Published 4/4/16
Recertification pain – Published 5/2/16
Cheaters – Published 5/14/16
The value of a CCIE – Published 11/2/16

Tac Tales #5: MWAM

New Year’s resolutions are made to be broken, and I haven’t been keeping up with my resolution to do more blog posts.  Now that I am back at Cisco, I am focusing on programmability and automation, and I do have a lot to say.  However, in honor of my return to Cisco, I thought I would post a new Tac Tales entry.  There is a moral to this story.

One day my boss came to me and said my team would be supporting the MWAM module in the Cat 6K.  I had done a lot of Cat 6K work at that point, but I had never even heard of an MWAM, and failed to see why cases on it would be sent to the routing protocols team.  My boss didn’t seem too concerned with my objections, and said, “Just go watch the VoD.”  VoD = Video on Demand.  So, I did.  I watched the VoD, and it started out by telling me how many processors were on the card and of what kind;  what types of buses were used to transmit data;  what kind of memory it had;  and how it interfaced with the Catalyst and its backplane.  Never did the video ever tell me what the card actually did.  I had no idea why one would buy an MWAM or what one would do with it.  I hoped a case wouldn’t come in on the card, and when it did I immediately escalated to engineering because I had no idea what to do.  Fortunately I only ever had one case on the MWAM.  (And the fun thing about coming back to Cisco after 10 years is that I can go look up all these cases I remember and read my notes.  Very cool!)

What is the moral, you ask?  Well, as a Technical Marketing Engineer, a big part of my role is communicating technical concepts clearly to others.  How often have you bought a book or looked at a web page to learn some new protocol, only to find that the description of it begins with packet header formats or state machines?  Fine, but tell me what it actually does before you tell me how it works.  Imagine if you went out into the jungle and encountered someone who has never seen a car.  You wouldn’t start explaining it to him by saying “it uses an internal combustion engine which has a four-stroke cycle of intake, compression, power, exhaust.”  You’d say, “it has wheels and takes me places very fast.”  Now in defense of the MWAM VoD guy, he may have designed his video for people who already knew what the card was.  But often I have found that people make this assumption, and when I backtrack and start at the beginning when explaining something, often people say, “you know, I’ve always been afraid to ask about that, but thanks for explaining it.”

Meanwhile, my second try at Cisco is much more fun than the last.  And thankfully, no MWAMs.  TAC was an great experience and period of growth, but it’s a not a fun job.

1

Goodbye Juniper, Hello Cisco

I feel a bit of guilt for letting this blog languish for a while. I can see from the response to my articles explaining confusing Juniper features that my work had some benefit outside my own edification, and so I hate to leave articles unfinished which might have been helpful. In addition, WordPress is not easy to maintain and I keep losing notifications of comments, which means that when I am not logging in, I miss the opportunity to respond to kind words and questions.

As it is, my work explaining Juniper to the masses will have to be put on hold, as I have left Juniper after six years and returned to my old employer Cisco! I worked at Juniper longer than I had anywhere else, and it’s amazing to consider that I just closed the door on half a decade. But, I even after attaining my JNCIE I always felt like a Cisco guy at heart, and so here I am again. A few random thoughts then:

1. I interviewed for a number of jobs, and now that I am hired I can say that I really hate interviewing. My interviews at Cisco were very fair and reasonable. Just for the heck of it I did a phone screen with Google and completely bombed it. I’m not ashamed to admit that. I’m not supposed to reveal their questions, and I won’t, but they were mostly basic questions about TCP functionality, and MAC/ARP stuff, and it’s amazing how you forget some of the basics over the years. I wasn’t really interested in working there so I did no preparation, and in fact the recruiter warned me to brush up on basics. I just figured my work and blog show that I am at least somewhat technical. I plan to write some posts on the art of technical interviewing, but I was certainly underwhelmed by Google’s screening process, as I’m sure they were by my performance. I really wanted the Cisco job, and what a difference attitude makes! (Oh, and I completely munged an MPLS FRR/Node & Link protection question, less than a year after passing the JNCIE-SP. Uh, whoops.)

2. I bear Juniper no ill will. It was an interesting six years. When I came on board, during the Kevin Johnson years, it was all rah-rah pep talks about how we were going to be the next $10 billion company (errr, no…) followed by a plethora of product disasters. Killing off Netscreen gave the firewall market to Palo Alto, Fortinet, and amazingly resuscitated Checkpoint. Junos Space was a disaster, and Pulse slightly less so. QFabric was not a bad idea, but was far too complex. You needed to buy a professional services contract with the product, because it was too complex to install by itself. And yet it supposedly simplified the data center? There was a fiasco with our load balancer product. And then came the activist investors with their Integrated Operating Plan. I will permanently loathe activist investors. Juniper was hurting and they just magnified the hurt. There’s nothing worse than a bunch of generic business-types who wouldn’t know a router if they saw one trying to tell a router company how to do its business. They thought they could apply the same formula you learn in B-school to any company no matter what it manufactures or does. Then we had the CEO revolving door.

Despite all of this, as I said, I like Juniper. I did ok there, and there are a lot of people I respect working there. Rami Rahim is a good choice for CEO. I left for personal reasons. They still have some good products and good ideas, and I think competition is always good for the marketplace. For the sake of my friends there, I hope Juniper does well.

3. If you read my bio, you will see that I was THE network architect for Juniper IT, meaning I covered everything. This included (in theory at least) campus LAN, WAN, data center, wireless, network security, etc. I did something in all of these spaces. It was a broad level of knowledge, but not deep. That’s why I did my JNCIE-SP–I was hungering to go deep on something. My new job at Cisco is principal technical strategy engineer for data center. This is an opportunity to go deep but not as broad, and I’m happy to be doing that. The data center space is where it’s at these days, and I can’t wait to get deeper into it.

4. Coming back to Cisco after an eight year hiatus was bizarre. It was cool to pull up all my old bugs and postings to internal aliases to see what I was doing back then. Heck, I actually sounded like I knew a thing or two. I was thrilled to find out I am on the same team as Tim Stevenson, whose work as a Cat 6K TME I admired when I worked in TAC. Just for fun I walked though my old building and floor (K, floor 2) and nearly fell over when I saw that it looked identical. I mean, not only the cubes, but there were these giant signs for the different teams (e.g. “HTTS AT&T TEAM”) which were still hanging there as though the intervening eight years had never happened.

Unfortunately, I have to leave a few in progress articles in the dustbin. First, I shouldn’t really be promoting Juniper now that I am working for Cisco. And second, I’ve lost access to VMM, the internal Juniper tool I used to spin up VM versions of Juniper routers. However, I hope to start posting on Cisco topics now that I have access to that gear. Cisco’s products are generally better documented than Juniper’s, but I promise to fill any gaps I might find. And I will leave my previous articles up in hopes that they will benefit future engineers who struggle with Junos.

Onwards!

TAC Tales #4: Airline Outage

I don’t advertise this blog so I’m always amazed that people even find it. I figured the least-read articles on this blog were my “TAC Tales,” but someone recently commented that they wanted to see more… Well, I’m happy to oblige.

The recent events at United reminded me of a case where operations were down for one of the major airlines at Miami International Airport. It didn’t directly impact flight operations, but ticketing and baggage handling systems were down. Naturally, it was a P1 and so I dialed into the conference bridge.

This airline had four Cat 6500’s acting as their core devices for the network. The four switches had vastly disparate configurations, both hardware and software. I seem to recall one of them was running a Supe 1 module, which was even old in 2007 when I took the case. There was a different software version on each of them.

EIGRP was acting funny. As a TAC engineer in the routing protocols team, I absolutely hated EIGRP. EIGRP Stuck-In-Active was my nightmare case. It was always such a pain to track down the source, and meanwhile you’d have peers resetting all over the place. OSPF doesn’t do that, nor ISIS. I once got in a debate on an internal Cisco alias with some EIGRP guys. Granted, I had insulted their life’s work, but I stated that EIGRP was fast, but unreliable and prone to meltdown. Their retort was that properly designed EIGRP networks do not melt down. Great, but when are networks ever properly designed? They are so often slapped together haphazardly, grow organically, and overall need to be resilient when even when unplanned. Of course, those of us in design and architecture positions do our best to build highly available networks, but you don’t want to be running a protocol that flips out when a route at some far end of the network disappears. Anyhow…

The adjacencies on all four boxes were resetting constantly. It was totally unstable. Every five minutes or so, some manager from the airline would hop on the bridge to tell us that they were using handwritten tickets and baggage tags, that lines at the ticket counters were going out the door, etc, etc. Because that really helps me to concentrate. I tried to troubleshoot the way TAC engineers are trained to troubleshoot: collect logs, search for bugs in the relevant software, look for configuration issues. With routing adjacency flaps on switches, always check for STP issues. I couldn’t figure it out.

Finally some high-level engineer for the airline got on the phone and took over like a five-star general. He had his ops team systematically shut down and reset the switches, one at a time. The instability stopped. Wish I’d thought of that.

The standards for a routing protocol like OSPF are written by slow-moving committees, and hence don’t change much. These committees often have members from multiple competing vendors who disagree on exactly what should be done, and even when they do agree, nothing happens fast in IETF committees. Conversely, Cisco owns EIGRP, and they can change it as much as they want. Even their internal committees are nowhere near as bureaucratic as IETF. This means that there can be significant changes in the EIGRP code between IOS releases, much more so than for OSPF, and it is thus vital to keep code revisions amongst participating routers fairly close.

In this case, the consulting engineers for the airline helped them to standardize the hardware and software revisions. They never re-opened the case.

Hub and Spoke with BGP

In my previous post, we saw the theory behind hub-and-spoke VPN. We saw how H/S involves multiple VRFs with cross-importation between them, and we traced the basic flow of a route advertised from one spoke to another.
Next, we are going to look at two options for configuring H/S VPNs. In this post, I will cover using BGP as the PE-CE routing protocol without independent route reflectors. In my next post, I will cover OSPF. Finally, I will return to BGP and examine the issues that come up when we use independent route reflectors with hub and spoke VPN. Read More »

1

Hub and Spoke VPN Part 1

One of the JNCIE-SP exam objectives I found difficult was hub and spoke VPN. Conceptually it’s not easy, and as is often the case, the documentation is only somewhat helpful. This series of posts is designed to walk you through the concepts of hub and spoke VPN, as well as its basic configuration using BGP, and then OSPF as the PE-CE protocol. Finally I will talk about route reflector issues when using H/S VPN. It is an important topic to master for your JNCIE exam, and if properly explained you will find it’s not as difficult as it seems. This article assumes you are familiar with basic configuration of Layer 3 MPLS VPN, including vrf import and export policies.

Note: I recommend that you only use vrf import/export policies for the JNCIE lab and avoid the vrf-target command for layer 3 VPN.

We’ll take a simple example topology with three sites.  Normally, if these sites are configured in a Layer 3 MPLS VPN, site 1 and site 2 are able to talk directly over the MLPS network. That is, the traffic between site 1 and site 2 is never seen by the hub site.
hs1
This is, of course desirable. Routing the site-to-site traffic through a central site could produce significant delays, especially if the geographic distance between the sites is large. It also would consume bandwidth unnecessarily on the hub interfaces. By default MPLS VPNs avoid this behavior.

Hub and spoke VPN changes this model and actually routes the spoke-to-spoke traffic through the hub site.
hs2
Given the above paragraph, why would anybody want to do this? Well, I would give you the caveat that for expert-level certification exams, you should not spend too much time asking why. Nevertheless, one reason you might use this configuration is to apply some sort of monitoring to the spoke-to-spoke traffic.

You will notice that the hub site has two interfaces. Whether physical or logical, the hub site will need to have two interfaces.

We’re going to blow the picture up a bit, and put some actual routers in to flesh out the picture, and then we are going to look at how the control plane works.

hs-route

Site 1 CE advertises a route (172.255.255.9) to its PE. The PE router then advertises the route via MBGP to the hub PE. The hub PE then advertises it over the top link to the hub CE. The hub CE learns the route and then re-advertises it back to the same PE it learned it from. If you haven’t seen h/s VPN before this may confuse you, but here comes the trick: the two interfaces that connect the hub CE to the hub PE are in different VRFs. Therefore, the hub PE will have a copy of the 172.255.255.9 route in both VRFs. This is the part which makes it a bit confusing.

The hub PE, having re-learned the route from the hub CE, then advertises it out to the site 2 PE, which sends it on its way to the site 2 CE. Since site 2’s route came from the hub PE via the hub CE, the route actually is pointing along the path through the hub site. As a result of this hocus-pocus, the data plane will forward through the hub site. To achieve this we will need to define specific route export/import policies on the PEs and do some cross-VRF importing, as illustrated below.

hs-communities

The spoke PEs do not learn routes directly from each other, but instead only learn routes that have been re-advertised by the hub site. Therefore, they will only import routes into their VRF that have the hub VRF target. They will not, repeat, will not import routes from the spoke VRF target. This may be counterintuitive because they are not importing routes with their “own” VRF target. However, this is how we achieve the cross-VRF importation. Even though they do not import routes with their own target, they do export with it. All routes exported from the spoke sites have the spoke target.

The hub PE is even more complex. It has two VRFs. The spoke VRF imports routes that have the spoke target, but it doesn’t export anything. That’s because we want to force the spoke sites to pull from the hub VRF. No routes are exported from the spoke. Meanwhile the hub instance does not import anything at all. It pulls no routes from MBGP or the other PEs. Instead, any routes in its table (aside from interface routes) it learns from the hub CE. Unlike the spoke, however, it does export routes, which are tagged with the hub VRF target. It is this target that the spokes accept.

In my next article you will see how we configure a null policy to achieve these goals. In the meantime, a little memory trick helped me to configure this. On the hub router, the spoke instance imports but does not export. I named my import policy “spoke-in” which sounds a bit like Spokane, the city. It may not be the greatest memory trick, but you can remember all the rest of the hub PE policies if you remember this. Remember, each VRF has a “null” policy, and they are the opposite of each other. So, if the spoke has an import policy (“spoke-in”), its export policy must be null, and therefore the import policy for the hub must be null and it must have an export policy (“hub-out”). This will become clearer in future articles when I explain the configuration if it’s not clear now.

To summarize this article:

  • Hub and Spoke MPLS VPN routes traffic through a hub site instead of directly between spokes.
  • To achieve this, the control plane (i.e. routing) also follows a hub and spoke model.
  • The spoke routers import and export from different hub PE VRFs.
  • The hub PE has two VRFs, one for sending routes to the hub CE and one for receiving them.

Look forward to the next article which will explain how to configure this with BGP as the PE-CE routing protocol.