Skip navigation

Tag Archives: isdn

Ancient Internet

The first company where I worked as a “systems administrator” had no Internet connectivity at all when I started.  By the time I left, I had installed an analog phone line which was shared amongst several users with modems for dial-up service.  The connectivity options in 1995 were limited, and very expensive.  Our company operated on a shoestring budget and could not afford the costly dedicated service offerings from our ISP.

When I moved to a “consulting” company, I finally had the opportunity to work with real dedicated Internet service.  For the customers I worked with, our main options were two:  ISDN and T1 lines.

ISDN stood for Integrated Services Digital Network.  It came in two major flavors, but we exclusively used the lower-end Basic Rate Interface (BRI).  ISDN was a digital phone line, and like its analog counterpart required dialing a phone number.  The BRI had two data channels which were 64 Kbps each, but we usually ran them together for a combined 128 Kbps.  At the time, this was quite speedy, more than double the speed of the modems we had, and our smaller customers loved ISDN.  Because it was a dial-up technology, however, per minute rates applied.  This meant the line would time out and disconnect periodically to save costs.  When it was down and an outbound packet arrived at the router with the ISDN interface, the router would dial up the SP again.  The connection time was much faster than analog modems, but it still added latency which was annoying.

I don’t know if other regions did it, but we had a local hack to get around this.  Understanding the hack requires a little background on the phone systems of the time.  Two kinds of business phone systems were typically in use:  PBXs and key systems.  With a key system, every phone had an extension number, but no direct dial into it.  If you wanted to speak to the person at extension 302, you dialed the main phone number for the business, and either asked the receptionist to connect you, or else an automated system did it for you.  For outbound dialing, the user would either lift the handset and select an unused line from the pool of lines available, or perhaps dial 9 for the key system to connect them to the next available line.  PBXs, on the other hand, were used by large companies, gave each user their own phone line and allowed inter-office extension-to-extension calling, as well as direct dial from the outside world.  If my extension was 3202, I would have a direct dial phone number of, say, 415-555-3202.

Some companies instead opted for the phone company to do their internal switching.  This was known as a Centrex service.  The phone company provided hard wired analog phone lines to the customer, but enabled extension-to-extension direct dial.  Thus, if I was at extension 3202 and I needed to dial extension 3203, I could pick up the phone and just dial the four digits.  The phone company took care of routing it.

What does this have to do with ISDN?  We used to order Centrex service for our customers in the same Centrex group as their ISP. Thus, the customer’s ISDN line became an “extension” of the ISP’s Centrex group.  Not only could the customer then dial the ISP with four digits (not a big deal when the router is doing the dialing), but there were no toll charges on Centrex lines.  We used to nail the line up so it would never disconnect, and if it did for an reason, it would auto-redial.  And then we had dedicated Internet service on a dial-up line!

T1’s (E1’s elsewhere) were 1.544 Mbps, blazing fast at the time.  Unlike the single-pair ISDN line, T1’s were delivered on four wires, two for TX and two for RX.  I won’t get into the details of line coding on T1’s, which we all studied as junior network engineers.  T1 lines were truly dedicated, and provided a point-to-point connection from customer to ISP.  They were distance-priced, but I worked in San Francisco which is a small city, so it wasn’t usually a factor.  Because 1.544 Mbps was expensive for some customers, we had the option of ordering fractional T1s, fewer channels at a slower speed, but still faster than ISDN.  In the early days we had to terminate the T1 on an external CSU/DSU device, and then run a serial cable to the router, but eventually the CSU/DSU came integrated on the router interface card.

When I worked at the San Francisco Chronicle, we were providing Internet service via a T1 line terminating on a 2500-series router.  (The same one disabled by paint roller in this story.)  1000 users on a single T1 was painfully slow, and we made the decision to upgrade to a DS3 (T3) which ran at 45Mbps.  The interface for DS3 used two BNC coax connections.  I remember being amazed that the phone company could deliver service over coax, but it turns out that service into the building used fiber optics.  Inside the building we ran coax.  The run of coax from the basement to our 2nd floor data center was expensive, but the result was phenomenal.  The DS3, which we terminated on a brand new 7200vxr, was vastly superior to the crawling T1, and the effort paid off with our users.

DSL was groundbreaking.  There was nothing consumer-grade before that, and small companies could not even afford Internet connectivity.  I was one of the first home adopters of DSL.  The freshly-trained phone guy showed up at my apartment and installed a splitter box in the basement.  This was needed because residential service was ADSL, which multiplexed digital service on an analog line.  Unlike ISDN, which converted analog phone signal to digital, ADSL left the analog in tact, adding the digital part of the signal onto the higher frequencies.  The splitter box took the incoming phone line from the street and peeled off the high frequencies, providing an analog signal for telephones.  It then passed through the analog/digital mix intact to the modem, which just ignored the analog frequencies.  The phone guy then sat down with his toolbelt and tried to configure TCP/IP on my computer.  He gave up because he had no idea what he was doing.  I told him to leave me the IP addresses and I’d do it myself.  Eventually the telco would just send you an small filter to plug into each analog phone jack yourself, and they could turn on the service without sending a phone guy to rewire things.  Once DSL in its various forms came out, the Internet was available to the masses.  Of course, cable modems came shortly after.

We take for granted instant connectivity from every location on portable devices.  Once upon a time,  connectivity was only available at certain locations, often requiring dialing a service provider.  There was a real excitement as new technologies emerged for making connectivity faster and easier.  Now, of course, we just expect things to work and get angry when they don’t.

4

How not to do Internet Connectivity

My first IT job was at a small company in Novato, California, that designed and built museum exhibits.  At the time most companies either designed the exhibits or built them, but ours was the only one that did both.  You could separate the services, and just do one or the other, but our end-to-end model was the best offering because the fabricators and designers were in the same building and could collaborate easily.  The odd thing about separating the functions was that we could lose a bid to design a project, but win the bid to build it, and hence end up having to work closely with a competitor to deliver their vision.

A museum exhibit we designed and built

The company was small–only 60 employees.  Half of them were fabricators who did not have computers, whereas the other half were designers and office staff who did.  My original job was to be a “gopher” (or go-fer), who goes for stuff.  If someone needed paint, screws, a nail gun, fumigation of a stuffed tiger, whatever, I’d get in the truck and take care of it.  However, they quickly realized I was skilled with computers and they asked me to take over as their IT guy.  (Note to newbies:  When this happens, especially at a small company, people often don’t forget you had the old job.  One day I might be fixing a computer, then the next day I’d be hauling the stuffed tiger.)

This was in the mid-1990’s, so let me give you an idea of how Internet connectivity worked:  it didn’t.  We had none when I started.  We had a company-internal network using LocalTalk (which I described in a previous post), so users could share files, but they had no way to access the Internet at all.  We had an internal-only email system called SnapMail, but it had no ability to do SNMP or connect beyond our little company.

The users started complaining about this, and I had to brainstorm what to do when we had virtually no operating budget at all.  I pulled out the yellow pages and looked under “I”, and found a local ISP.  I called them, and the told me I could use Frame Relay, a T1, or ISDN.  I had no idea what they were talking about.  The sales person faxed me a technical description of these technologies, and I still had no idea what they were talking about.  At this point I didn’t know the phone company could deliver anything other than, well, a phone line.  I wasn’t at the point where I needed to hear about framing formats and B8ZS line encoding.

We decided we could afford neither the ongoing expense, nor the hardware, so we came up with a really bad solution.  We ordered modems for three of the computers in the office:  the receptionist, the CEO, and the science researcher.  For those of you too young to remember, modems allow you to interface computers using an ordinary phone line.  We ordered a single phone line (all we could afford).  When one of them wanted to use the Internet, they would run around the office to check with the other two if the line was free.

A circa-1990’s Global Village modem

The reason we gave the receptionist a modem is amusing.  Our dial-up ISP allowed us to create public email addresses for all of our employees.  However, they all dumped into one mailbox.  The receptionist would dial in in the morning, download all the emails, and copy and paste them into the internal email system.  If somebody wanted to reply, the would send it to the receptionist via SnapMail and she would dial up, paste it into the administrator account, and send it.  Brilliant.

Needless to say, customer satisfaction was not high, even in those days.  Sick of trying to run IT with no money, I bailed for a computer consulting company in San Francisco and started installing the aforementioned T1s and ISDN lines for customers, with actual routers.

If ever you’re annoyed with slow Wi-Fi, be glad you aren’t living in the 1990’s.

TAC Tales #14: Stuck in Active

Everyone who’s worked in TAC can tell you their nightmare case–the type of case that, when they see it in the queue, makes them want to run away, take an unexpected lunch break, and hope some other engineer grabs it.  The nightmare case is the case you know you’ll get stuck on for hours, on a conference bridge, escalating to other engineers, trying to find a solution to an impossible problem.  For some it’s unexplained packet loss.  For others, it’s multicast.  For me, it was EIGRP Stuck-in-Active (SIA).

Some customer support engineers (CSEs) thought SIA cases were easy.  Not me.  A number of times I had a network in total meltdown due to SIA with no clue as to where the problem was.  Often the solution required a significant redesign of the network.

As a review, EIGRP is more-or-less a distance-vector routing protocol, which uses an algorithm called DUAL to achieve better performance than a traditional DV protocol like RIP.  I don’t want to get into all the fun CCIE questions on the protocol details, but what matters for this article is how querying works.  When an EIGRP neighbor loses a route, it sets the route as “Active” and then queries its neighbors as to where the route went.  Then, if the neighbors don’t have it, they set it active and query their neighbors.  If those neighbors don’t have the route active, they of course mark it active and query their neighbors.  And so forth.

It should be obvious from this process that in a large network, the queries can multiply quite quickly.  If a router has a lot of neighbors, and its neighbors have a lot of neighbors, the queries multiply exponentially, and can get out of control.  Meanwhile, when a router sets a route active, it sets a timer.  If it doesn’t get a reply before the timer expires, then the router marks the route “Stuck In Active”, and resets the entire EIGRP adjacency.  In a large network with a lot of neighbors, even if the route is present, the time lag between sending a query and getting a response can be so long that the route gets reset before the response makes it to the original querying router.

I’ve ironed out some of the details here, since obviously an EIGRP router can lose a route entirely without going SIA.  For details, see this article.  The main point to remember is that the SIA route happens when the querying route just doesn’t get a response back.

Back in my TAC days, I of course wasn’t happy to see an SIA drop in the queue.  I waited to see if one of my colleagues would take the case and alleviate the burden, but the case turned blue after 20 minutes, meaning someone had to take it.  Darn.

Now I can show my age, because the customer had adjacencies resetting on Token Ring interfaces.  I asked the customer for a topology diagram, some debugs, and to check whether there was packet loss across the network.  Sometimes, if packets are getting dropped, the query responses don’t make it back to the original router, causing SIA.  The logs from the resets looked like this:

rtr1 - 172.16.109.118 - TokenRing1/0
Sep 1 16:58:06: %DUAL-3-SIA: Route 172.16.161.58/32 stuck-in-active state in IP-EIGRP(0) 55555. Cleaning up
Sep 1 16:58:06: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 55555: Neighbor 172.16.109.124 (TokenRing1/0) is down: stuck in active
Sep 1 16:58:07: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 55555: Neighbor 172.16.109.124 (TokenRing1/0) is up: new adjacency

This is typical of SIA.  The adjacency flapped, but the logs showed no particular reason why.

I thought back to my first troubleshooting experience as a network engineer.  I had brought up a new branch office but it couldn’t talk back to HQ.  Mike, my friend and mentor, showed up and started pinging hop-by-hop until he found a missing route.  “That’s how I learned it,” he said, “just go one hop a time.”  The big clue I had in the SIA case was the missing route:  172.16.161.58/32.  I started tracing it back, hop-by-hop.

I found that the route originated from a router on the edge of the customer network, which had an ISDN PRI connected.  (Showing my age again!)  They had a number of smaller offices that would dial into the ISDN on-demand, and then drop off.  ISDN had per-minute charges and thus, in this pre-VPN era, it was common to setup ISDN in on-demand mode.  ISDN was a digital dial-up technology with very short call setup times.  I discovered that, as these calls were going up and down, the router was generating /32 peer routes for the neighbors and injecting them into EIGRP.  They had a poorly designed network with a huge query domain size, and so as these dial peers were going up and down, routers on the opposite side of the network were going into active on the route and not getting responses back.

They were advertising a /16 for the entire 172.16.x.x network, so sending a /32 per dial peer was totally unnecessary.  I recommended they enable “no peer neighbor-route” on the PRI to suppress the /32’s and the SIAs went away.

I hate to bite the hand that feeds me, but even though I work at Cisco I can say I really never liked EIGRP.  EIGRP is fast, and if the network is designed well, it works fine.  However, networks often grow organically, and the larger the domain, the more unstable EIGRP becomes.  I’ve never seen this sort of problem with OSPF or ISIS.  Fortunately, this case ended up being much less problematic than I expected, but often these cases were far nastier.  Oftentimes it was nearly impossible to find the route causing the problem and why it was going crazy.  Anyhow it’s always good to relive a case with both Token Ring and ISDN for a double case of nostalgia.

A CCIE Goes Home to Cisco

In this article in my “Ten Years a CCIE” series, I describe my experience going to work at Cisco as a CCIE.  Unlike many Cisco-employed CCIE’s, I earned my certification outside of Cisco.

A CCIE leads to a job at Cisco

I returned to my old job at the Chronicle and had my business cards reprinted with my CCIE number. I loved handing it out, particularly at meetings with telephone companies and Internet service providers whose salespeople were likely to know what such a certification meant.  I remember one such sales person, duly impressed, saying “wow, on that test you can be forced to configure any feature on any Cisco product…I don’t know how anyone could prepare for that!”  (Uh, right.  See “The CCIE Mystique“.)

At that time, the most popular forum for aspiring CCIE’s was an email distribution list called groupstudy.com. I had been a subscriber to this mailing list, but prior to passing my exam, I didn’t feel adequate to post anything there. However, once I passed, I began posting regularly, beginning with a summary of my test preparation process. One day I got an email from a mysterious CCIE who told me that I sounded like I knew what I was talking about, asking me if I wanted to interview for a job. I thought his name and number sounded familiar, and when I got home I confirm my suspicions by digging through my bookshelf. He was the author of one of my books about Catalyst Quality of Service.

A brutal interview

It turns out he was a manager at Cisco High Touch Technical Support, a group of TAC engineers who specialized in high profile customers. I scheduled an interview right away.

This interview was by far the most difficult I’ve had in my career. They brought me into a room with four CCIE’s, two of them double, all of them sharp. Each one of them had a different specialty. One of them was a security guy, another one was an expert on multicast, another was an expert on switching. When it came to Kumar, the voice guy, I figured I was scot-free. After all, I didn’t claim to know anything about voice over IP. Kumar looked over my resume, and then he looked up at me. “I see you have ISDN on your resume,” Kumar said. And then he began to grill me on ISDN. Darn, I should have thought of that!  Thankfully, I was well prepared.

Despite one or two mistakes in the interview, I got hired on and began my new job as a customer support engineer at HTTS. My first few months were in a group called ESO, which supported large enterprises and was very focused on Catalyst switching.  I won’t go into the details of the job here, but you can see my many TAC tales if you are interested.

Cisco's San Jose campus

Cisco’s San Jose campus

Little purple stickers everywhere!

One thing I quickly noticed when I got to Cisco was that a lot of the people in my department had a nickel-sized purple dot on their ID badges and cubicle nameplates. I found out that these purple dots were actually stickers with the CCIE logo. Cisco employees who had their CCIE’s stuck these purple dots on their badges and nameplates to show it off. Many of the CCIE’s who had passed multiple exams actually placed multiple dots on their badges and nameplates. I wanted one quite badly. The problem was, sheets of these purple stickers were sent out only to the early CCIE’s, and by the time I had passed, Cisco was no longer providing the sheets of stickers. I suppose I could’ve had some printed out, but I asked around looking for a CCIE who was generous to give me one of his dots. They were in scarce supply, however, and nobody was willing to part with one. It was just another way newer CCIE’s were getting jipped.

The real CCIE logo

The real CCIE logo

Even though the exam had now switched to the one day format, you still didn’t meet too many CCIE’s outside of Cisco. It was thus quite a shock when I went to Cisco and saw purple dots everywhere. It seemed like fully half of the people I was working with in my new job had CCIE’s. And many of them had low numbers, in the 2000’s and even in the 1000’s. I was quite relieved to find that they all treated me with total respect; nobody ever challenged me on account of my one-day CCIE. Still, I always had (and always will have) a great deal of respect for those people who passed the test when it was a two day test, and the cottage industry devoted to minting CCIE’s had not yet come into existence.

CCIE challenges customer

Customers were another story. I remember one BGP case in particular. I looked at the customer’s configuration and immediately realized that it was a simple matter of misconfiguration. I fired it up in my lab reproduced his configuration and proved to him that it was indeed a configuration error on his part. I wrote it all up in an email and proudly signed it with my CCIE number. Within a half an hour I got a call from the customer and one of his colleagues on the line. They proceeded to grill me rapidly on BGP asking me all sorts of questions that weren’t relevant to their case and stumping me several times. At that point I realized that when many people see you are a CCIE, they take it as a challenge. In some cases they failed the test themselves, or else they’ve met stupid CCIE’s in the past and they feel themselves to be on a mission to discredit all CCIE’s. After that episode, I removed my CCIE number from my email signature. I gained a feeling of self-importance after I passed my exam, but working among so many people with the same certification, and dealing with such intelligent customers, I realized that the CCIE didn’t always carry the prestige I thought it did.  The mystique diminished even further.

Incidentally, I became friends with all of the guys who interviewed me, and I was on the interview team myself during my tenure at Cisco. One extremely sharp CCIE we hired told me our interview was so tough he had to “hit the bottle” afterwards. It was considered a rite of passage at TAC to go through a tough interview, but I have gotten a lot nicer in my interview style now, having been on the receiving end of a few grillings.

The value of a CCIE

One of the later posts in this series will examine the question of the value of a CCIE certification.  After all, this is one of the most common questions I see in forums dedicated to certification.  However, my experience getting hired into Cisco (the first time) has some lessons.

  • The immediate reason I got hired was because of my experience and willingness to go out of my way helping others to get their CCIE on Groupstudy.  However, I would not have gotten the position without a CCIE, so clearly it proved its value there.
  • Once you are at Cisco, although people commonly display their stickers and plaques, having a CCIE certification will not necessarily distinguish you.
  • There are many CCIE’s who have made a bad impression on others, whether they are only book-knowledgeable, or even cheaters.  Often people challenge you when you have a CCIE, instead of respecting you.

In the next article in the series, Multiple CCIEs, Multiple Attempts, I describe passing the CCIE Security exam.  I talk about my experience suffering the agony of defeat for the first time, and how I eventually conquered that test.

In those days, you had to build a lab…

In this article in my series, Ten Years a CCIE, I discuss the challenge of building and maintaining a lab, and the question of building versus not building a physical lab.

Acquiring Gear

As I mentioned in a previous post, GNS3 was not available at the time I took the CCIE exam. This meant that the CCIE candidate had to acquire hardware — quite a lot. This was one way that the number of CCIE’s was limited, even after the institution of the one-day format. I acquired equipment by various means; mostly by purchasing old routers off eBay. Some I was able to borrow from work. Some I bought directly from coworkers.

My lab consisted mostly of 2500 series routers. These were adequate to test most of the routing protocol features needed for the exam. However, some of the newer or more advanced features required more advanced routers, and for these I used 2600 series routers. These were essential for testing Voice over IP, as a 2500 did not support the hardware (FXS/FXO cards) required to do dial peers. In 2004, Frame Relay was still a big part of the CCIE exam, so I decided that a Frame Relay switch was absolutely necessary. For this I acquired a 2500 series router with a lot of serial ports. It was fortunate that I didn’t need to purchase a WAN emulator of some sort, since Cisco included Frame Relay switching capability in its IOS software and WAN emulators were ridiculously expensive.

Early on I determined that a terminal server was necessary. I started out thinking it wasn’t, but moving your console cable around from router to router gets old very quickly. Luckily, a 2500 series router with a lot of serial ports, designed primarily for modems, made an excellent terminal server. Instead of having to console individually to each router, I could simply telnet to the terminal server on a particular TCP port number to access the individual devices.

Ethernet switching was a bit more difficult. At that time, the CCIE equipment list had Catalyst 3550 switches on it. The 3550s were quite expensive and I couldn’t afford any. Luckily, we had a spare at work which I borrowed, but only one. This meant that for spanning tree and trunking configuration, I had to configure one end only and hope that the configuration would work if there were a switch on the other end.  Building a physical lab often requires compromises, since it can be impossible to acquire or afford all the equipment.

ISDN and ATM

ISDN was a big part of the CCIE exam in the early 2000’s, even though it was becoming obsolete. It was also considered one of the more difficult subjects because of the myriad of ways of configuring it. I felt that it necessary to have some way to configure ISDN in my home lab. ISDN simulators were available, but were extremely expensive, and out of my price range. Some other CCIE candidates suggested to me that I purchase ISDN service for my apartment from the telephone company. This seemed to me like a huge hassle and it was also quite expensive. Luckily at just the time I started studying for this exam a small company came out with an inexpensive ISDN simulator. As I recall, it was only a few hundred bucks. I bought one, and for the most part it worked great. The one problem was that the on/off switch was labeled backwards. Easily fixed with a sticker.

The author's 2004 lab. Not neat! Note the but sets for VoIP testing, the 3000-series lying vertically on the right, and the small black ISDN sim on the table, center-front.

The author’s 2004 lab. Not neat! Note the butt sets for VoIP testing, the 3000-series lying vertically on the right, and the small black ISDN sim on the table, center-front.

I had no way of simulating ATM. Doing ATM required a Lightstream switch and expensive interface cards. As I mentioned earlier, ATM was a very small part of the exam. It simply didn’t seem worth it to spend a huge amount of money to configure a single interface.

Why a home lab?

Just because I didn’t have certain technologies available to me doesn’t mean I didn’t study them. At that time, there were a number of companies offering CCIE rack rentals (an example is here.)  These were either small entrepreneurs or some of the larger CCIE test prep companies, who had set up full-blown CCIE labs and rented out time on them. There were two or three that I used regularly to fill in on technologies I didn’t have, including ATM.

Why did I spend thousands of dollars on lab equipment for my home lab when I could have simply rented time from a rack rental company? This question will come up again when I discuss the CCIE security exam. Working with the remote labs was a pain. Every time you started up, you had to load your configurations back into the lab because someone else had been using it in the interim. When the time came to shut down your lab, you had to back everything up. A huge chunk of your paid-for time slot had to be devoted to config management.  In second place, you could only access the lab during certain scheduled slots that you had purchased. I felt then, and I was right, that being held to someone else’s schedule would be a major impediment to serious study. I therefore decided to build my own lab, and use rack rentals only to fill in technologies I didn’t have. This meant that whenever I had spare time I could flip on the equipment and study. I even set up remote access to my lab and left it running during the daytime, so that I could study in my spare moments at work. Any time I wanted, I could access my lab and the configurations I was working on would be right where I left them.  This was essential to my passing the exam.

The CCIE lab had six routers on it that time, and connected to your lab were three backbone routers. These backbone routers out of your control, but were simply route injectors. I had enough routers to simulate the core of the lab, but I did not have enough routers to simulate all of the backbone. I therefore had to get creative. I used my frame relay switch as well as my term server as backbone routers as well. Thus, they were doing double duty. The configuration was relatively easy with the FR switch since it was only doing layer 2 in the lab, but it got tricky with the term server, since I couldn’t change its IP address. The IP address of the term server was the address I telnetted to to access my lab.  So, I often ended up changing IP addresses in the scenarios I had purchased, so that I could use the access server as a backbone router. I also used an ancient 3000-series router running IOS 9 as a BB router. (IOS 12.1 was current at that time!)  It was hard to configure but it could do OSPF and BGP.

Maintaining a home lab was a lot of work. I had to re-cable it a number of times until I finally got the stable topology that I liked. I often had to deal with bent pins which were so common with the serial cables of the time. I had to do IOS upgrades and the occasional memory upgrade when I needed an IOS but didn’t have the memory to support it. Sometimes, this work distracted from the work of studying. However, all of it was valuable experience. I don’t think I would trade it for GNS3 any day. GNS3 certainly adds an element of convenience, and allows the student to focus on the lab and not on extraneous tasks. And, it certainly has some challenges of its own. But I do think that dealing with the physical hardware, upgrading it, cabling it, etc. — helped me to grow as a network engineer, and also improve my general troubleshooting skills. Of course, as I mentioned before, the expense and difficulty of working with physical equipment prevented a number of people from attempting the lab, thus keeping the number of CCIE’s lower than in the GNS3 era.

Modern candidates for R/S may be able to use GNS3, but candidates for the other exams, such as data center, will go through the same struggles as myself.  Data Center is particularly bad because the equipment required is so expensive.  It is almost impossible to procure it on your own.  Rack rental is one option, but is fraught with problems, as I mentioned above.  The only other option is to try to get your employer to help you build and pay for a lab.  Despite the many changes in the exam, some things remain the same I suppose!

My next article, How to Pass the CCIE Lab Exam in One Attempt, reviews my study and preparation for the Routing and Switching lab exam.