In this article in my series, Ten Years a CCIE, I discuss the challenge of building and maintaining a lab, and the question of building versus not building a physical lab.

Acquiring Gear

As I mentioned in a previous post, GNS3 was not available at the time I took the CCIE exam. This meant that the CCIE candidate had to acquire hardware — quite a lot. This was one way that the number of CCIE’s was limited, even after the institution of the one-day format. I acquired equipment by various means; mostly by purchasing old routers off eBay. Some I was able to borrow from work. Some I bought directly from coworkers.

My lab consisted mostly of 2500 series routers. These were adequate to test most of the routing protocol features needed for the exam. However, some of the newer or more advanced features required more advanced routers, and for these I used 2600 series routers. These were essential for testing Voice over IP, as a 2500 did not support the hardware (FXS/FXO cards) required to do dial peers. In 2004, Frame Relay was still a big part of the CCIE exam, so I decided that a Frame Relay switch was absolutely necessary. For this I acquired a 2500 series router with a lot of serial ports. It was fortunate that I didn’t need to purchase a WAN emulator of some sort, since Cisco included Frame Relay switching capability in its IOS software and WAN emulators were ridiculously expensive.

Early on I determined that a terminal server was necessary. I started out thinking it wasn’t, but moving your console cable around from router to router gets old very quickly. Luckily, a 2500 series router with a lot of serial ports, designed primarily for modems, made an excellent terminal server. Instead of having to console individually to each router, I could simply telnet to the terminal server on a particular TCP port number to access the individual devices.

Ethernet switching was a bit more difficult. At that time, the CCIE equipment list had Catalyst 3550 switches on it. The 3550s were quite expensive and I couldn’t afford any. Luckily, we had a spare at work which I borrowed, but only one. This meant that for spanning tree and trunking configuration, I had to configure one end only and hope that the configuration would work if there were a switch on the other end.  Building a physical lab often requires compromises, since it can be impossible to acquire or afford all the equipment.

ISDN and ATM

ISDN was a big part of the CCIE exam in the early 2000’s, even though it was becoming obsolete. It was also considered one of the more difficult subjects because of the myriad of ways of configuring it. I felt that it necessary to have some way to configure ISDN in my home lab. ISDN simulators were available, but were extremely expensive, and out of my price range. Some other CCIE candidates suggested to me that I purchase ISDN service for my apartment from the telephone company. This seemed to me like a huge hassle and it was also quite expensive. Luckily at just the time I started studying for this exam a small company came out with an inexpensive ISDN simulator. As I recall, it was only a few hundred bucks. I bought one, and for the most part it worked great. The one problem was that the on/off switch was labeled backwards. Easily fixed with a sticker.

The author’s 2004 lab. Not neat! Note the butt sets for VoIP testing, the 3000-series lying vertically on the right, and the small black ISDN sim on the table, center-front.

I had no way of simulating ATM. Doing ATM required a Lightstream switch and expensive interface cards. As I mentioned earlier, ATM was a very small part of the exam. It simply didn’t seem worth it to spend a huge amount of money to configure a single interface.

Why a home lab?

Just because I didn’t have certain technologies available to me doesn’t mean I didn’t study them. At that time, there were a number of companies offering CCIE rack rentals (an example is here.)  These were either small entrepreneurs or some of the larger CCIE test prep companies, who had set up full-blown CCIE labs and rented out time on them. There were two or three that I used regularly to fill in on technologies I didn’t have, including ATM.

Why did I spend thousands of dollars on lab equipment for my home lab when I could have simply rented time from a rack rental company? This question will come up again when I discuss the CCIE security exam. Working with the remote labs was a pain. Every time you started up, you had to load your configurations back into the lab because someone else had been using it in the interim. When the time came to shut down your lab, you had to back everything up. A huge chunk of your paid-for time slot had to be devoted to config management.  In second place, you could only access the lab during certain scheduled slots that you had purchased. I felt then, and I was right, that being held to someone else’s schedule would be a major impediment to serious study. I therefore decided to build my own lab, and use rack rentals only to fill in technologies I didn’t have. This meant that whenever I had spare time I could flip on the equipment and study. I even set up remote access to my lab and left it running during the daytime, so that I could study in my spare moments at work. Any time I wanted, I could access my lab and the configurations I was working on would be right where I left them.  This was essential to my passing the exam.

The CCIE lab had six routers on it that time, and connected to your lab were three backbone routers. These backbone routers out of your control, but were simply route injectors. I had enough routers to simulate the core of the lab, but I did not have enough routers to simulate all of the backbone. I therefore had to get creative. I used my frame relay switch as well as my term server as backbone routers as well. Thus, they were doing double duty. The configuration was relatively easy with the FR switch since it was only doing layer 2 in the lab, but it got tricky with the term server, since I couldn’t change its IP address. The IP address of the term server was the address I telnetted to to access my lab.  So, I often ended up changing IP addresses in the scenarios I had purchased, so that I could use the access server as a backbone router. I also used an ancient 3000-series router running IOS 9 as a BB router. (IOS 12.1 was current at that time!)  It was hard to configure but it could do OSPF and BGP.

Maintaining a home lab was a lot of work. I had to re-cable it a number of times until I finally got the stable topology that I liked. I often had to deal with bent pins which were so common with the serial cables of the time. I had to do IOS upgrades and the occasional memory upgrade when I needed an IOS but didn’t have the memory to support it. Sometimes, this work distracted from the work of studying. However, all of it was valuable experience. I don’t think I would trade it for GNS3 any day. GNS3 certainly adds an element of convenience, and allows the student to focus on the lab and not on extraneous tasks. And, it certainly has some challenges of its own. But I do think that dealing with the physical hardware, upgrading it, cabling it, etc. — helped me to grow as a network engineer, and also improve my general troubleshooting skills. Of course, as I mentioned before, the expense and difficulty of working with physical equipment prevented a number of people from attempting the lab, thus keeping the number of CCIE’s lower than in the GNS3 era.

Modern candidates for R/S may be able to use GNS3, but candidates for the other exams, such as data center, will go through the same struggles as myself.  Data Center is particularly bad because the equipment required is so expensive.  It is almost impossible to procure it on your own.  Rack rental is one option, but is fraught with problems, as I mentioned above.  The only other option is to try to get your employer to help you build and pay for a lab.  Despite the many changes in the exam, some things remain the same I suppose!

My next article, How to Pass the CCIE Lab Exam in One Attempt, reviews my study and preparation for the Routing and Switching lab exam.