Skip navigation

Tag Archives: catalyst

Programmability for Network Engineers

Since I finished my series of articles on the CCIE, I thought I would kick off a new series on my current area of focus:  network programmability.  The past year at Cisco, programmability and automation have been my focus, first on Nexus and now on Catalyst switches.  I did do a two-part post on DCNM, a product which I am no longer covering, but it’s well worth a read if you are interested in learning the value of automation.

One thing I’ve noticed about this topic is that many of the people working on and explaining programmability have a background in software engineering.  I, on the other hand, approach the subject from the perspective of a network engineer.  I did do some programming when I was younger, in Pascal (showing my age here) and C.  I also did a tiny bit of C++ but not enough to really get comfortable with object-oriented programming.  Regardless, I left programming (now known as “coding”) behind for a long time, and the field has advanced in the meantime.  Because of this, when I explain these concepts I don’t bring the assumptions of a professional software engineer, but assume you, the reader, know nothing either.

Thus, it seems logical that in starting out this series, I need to explain what exactly programmability means in the context of network engineering, and what it means to do something programmatically.

Programmability simply means the capacity for a network device to be configured and managed by a computer program, as opposed to being configured and managed directly by humans.  This is a broad definition, but technically using an interface like Expect (really just CLI) or SNMP qualifies as a type of programmability.  Thus, we can qualify this by saying that programmability in today’s world includes the design of interfaces that are optimized for machine-to-machine control.

To manage a network device programmatically really just means using a computer program to control that network device.  However, when we consider a computer program, it has certain characteristics over and above simply controlling a device.  Essential to programming is the design of control structures that make decisions based on certain pieces of information.

Thus, we could use NETCONF to simply push configuration to a router or switch, but this isn’t the most optimal reason to use it.  It would be a far more effective use of NETCONF if we read some piece of data from the device (say interface errors) and took an action based on that data (say, shutting the interface down when the counters got too high.)  The other advantage of programmability is the ability to tie together multiple systems.  For example, we could read a device inventory out of APIC-EM, and then push config to devices based on the device type.  In other words, the decision-making capability of programmability is most important.

Network programmability encompasses a number of technologies:

  • Day 0 technologies to bring network devices up with an appropriate software version and configuration, with little to no human intervention.  Examples:  ZTP, PoAP, PnP.
  • Technologies to push and read configuration and operational data from devices.  Examples:  SNMP, NETCONF.
  • Automation systems such as Puppet, Chef, and Ansible, which are not strictly programming languages, but allow for configuration of numerous devices based on the role of the device.
  • The use of external programming languages, such as Ruby and Python, to interact with network devices.
  • The use of on-box programming technologies, such as on-box Python and EEM, to control network devices.

In this series of articles we will cover all of these topics as well as the mysteries of data models, YANG, YAML, JSON, XML, etc., all within the context of network engineering.  I know when I first encountered YANG and data models, I was quite confused and I hope I clear up some of this confusion.

1
1

TAC Tales #7: “Just slam it in!”

When I first started at TAC, I wasn’t allowed to take cases by myself.  If I grabbed a case, I had to get an experienced engineer to help me out.  One day I grabbed a case on a Catalyst 6k power supply, and asked Veena (not her real name) to help me on the case.

We got the customer on the phone.  He was an engineer at a New York financial institution, and sounded like he came from Brooklyn.  I lived in Williamsburg for a while with my mom back in the 1980’s before it was cool, and I know the accent.  He explained that he had a new 6k, and it wasn’t recognizing the power supply he had bought for it.  All of the modules had a “power denied” message on them.

I put the customer on speaker phone in my cube and Veena looked at the case notes.  As was often the case in TAC, we put the customer on mute while discussing the issues.  Veena thought it was a bad connection between the power supply and the switch.

“Here’s what I want you to do,” Veena said to the customer, un-muting the phone.  “I used to work in the BU, and a lot of times these power supplies don’t connect to the backplane.  You need to put it in hard.  Pull the power supply out and slam it in to the chassis.  I want to hear it crack!”

The customer seemed surprised.  “You want me to do what?!” he bristled.

“Slam it in!  Just slam it in as hard as you can!  We saw this in the BU all the time!”

“Hey lady,” he responded, “we paid a couple hundred grand for this box and I don’t want to break it.”

“It’s ok,” she said, “it’ll be fine.  I want to hear the crack!”

“Well, ok,” he said with resignation.  He put the phone down and we heard him shuffle off to the switch.  Meanwhile Veena looked at me and said “Pull up the release notes.”  I pulled up the notes, and we saw that the power supply wasn’t supported in his version of Catalyst OS.

Meanwhile in the background:  CRACK!!!

The customer came back on the line.  “Lady, I slammed that power supply into the chassis as hard as I could.  I think I broke something on it, and it still doesn’t work!”

“Yes,” Veena replied.  “We’ve discovered that your software doesn’t support the power supply and you will need to do an upgrade…”

2

In those days, you had to build a lab…

In this article in my series, Ten Years a CCIE, I discuss the challenge of building and maintaining a lab, and the question of building versus not building a physical lab.

Acquiring Gear

As I mentioned in a previous post, GNS3 was not available at the time I took the CCIE exam. This meant that the CCIE candidate had to acquire hardware — quite a lot. This was one way that the number of CCIE’s was limited, even after the institution of the one-day format. I acquired equipment by various means; mostly by purchasing old routers off eBay. Some I was able to borrow from work. Some I bought directly from coworkers.

My lab consisted mostly of 2500 series routers. These were adequate to test most of the routing protocol features needed for the exam. However, some of the newer or more advanced features required more advanced routers, and for these I used 2600 series routers. These were essential for testing Voice over IP, as a 2500 did not support the hardware (FXS/FXO cards) required to do dial peers. In 2004, Frame Relay was still a big part of the CCIE exam, so I decided that a Frame Relay switch was absolutely necessary. For this I acquired a 2500 series router with a lot of serial ports. It was fortunate that I didn’t need to purchase a WAN emulator of some sort, since Cisco included Frame Relay switching capability in its IOS software and WAN emulators were ridiculously expensive.

Early on I determined that a terminal server was necessary. I started out thinking it wasn’t, but moving your console cable around from router to router gets old very quickly. Luckily, a 2500 series router with a lot of serial ports, designed primarily for modems, made an excellent terminal server. Instead of having to console individually to each router, I could simply telnet to the terminal server on a particular TCP port number to access the individual devices.

Ethernet switching was a bit more difficult. At that time, the CCIE equipment list had Catalyst 3550 switches on it. The 3550s were quite expensive and I couldn’t afford any. Luckily, we had a spare at work which I borrowed, but only one. This meant that for spanning tree and trunking configuration, I had to configure one end only and hope that the configuration would work if there were a switch on the other end.  Building a physical lab often requires compromises, since it can be impossible to acquire or afford all the equipment.

ISDN and ATM

ISDN was a big part of the CCIE exam in the early 2000’s, even though it was becoming obsolete. It was also considered one of the more difficult subjects because of the myriad of ways of configuring it. I felt that it necessary to have some way to configure ISDN in my home lab. ISDN simulators were available, but were extremely expensive, and out of my price range. Some other CCIE candidates suggested to me that I purchase ISDN service for my apartment from the telephone company. This seemed to me like a huge hassle and it was also quite expensive. Luckily at just the time I started studying for this exam a small company came out with an inexpensive ISDN simulator. As I recall, it was only a few hundred bucks. I bought one, and for the most part it worked great. The one problem was that the on/off switch was labeled backwards. Easily fixed with a sticker.

The author's 2004 lab. Not neat! Note the but sets for VoIP testing, the 3000-series lying vertically on the right, and the small black ISDN sim on the table, center-front.

The author’s 2004 lab. Not neat! Note the butt sets for VoIP testing, the 3000-series lying vertically on the right, and the small black ISDN sim on the table, center-front.

I had no way of simulating ATM. Doing ATM required a Lightstream switch and expensive interface cards. As I mentioned earlier, ATM was a very small part of the exam. It simply didn’t seem worth it to spend a huge amount of money to configure a single interface.

Why a home lab?

Just because I didn’t have certain technologies available to me doesn’t mean I didn’t study them. At that time, there were a number of companies offering CCIE rack rentals (an example is here.)  These were either small entrepreneurs or some of the larger CCIE test prep companies, who had set up full-blown CCIE labs and rented out time on them. There were two or three that I used regularly to fill in on technologies I didn’t have, including ATM.

Why did I spend thousands of dollars on lab equipment for my home lab when I could have simply rented time from a rack rental company? This question will come up again when I discuss the CCIE security exam. Working with the remote labs was a pain. Every time you started up, you had to load your configurations back into the lab because someone else had been using it in the interim. When the time came to shut down your lab, you had to back everything up. A huge chunk of your paid-for time slot had to be devoted to config management.  In second place, you could only access the lab during certain scheduled slots that you had purchased. I felt then, and I was right, that being held to someone else’s schedule would be a major impediment to serious study. I therefore decided to build my own lab, and use rack rentals only to fill in technologies I didn’t have. This meant that whenever I had spare time I could flip on the equipment and study. I even set up remote access to my lab and left it running during the daytime, so that I could study in my spare moments at work. Any time I wanted, I could access my lab and the configurations I was working on would be right where I left them.  This was essential to my passing the exam.

The CCIE lab had six routers on it that time, and connected to your lab were three backbone routers. These backbone routers out of your control, but were simply route injectors. I had enough routers to simulate the core of the lab, but I did not have enough routers to simulate all of the backbone. I therefore had to get creative. I used my frame relay switch as well as my term server as backbone routers as well. Thus, they were doing double duty. The configuration was relatively easy with the FR switch since it was only doing layer 2 in the lab, but it got tricky with the term server, since I couldn’t change its IP address. The IP address of the term server was the address I telnetted to to access my lab.  So, I often ended up changing IP addresses in the scenarios I had purchased, so that I could use the access server as a backbone router. I also used an ancient 3000-series router running IOS 9 as a BB router. (IOS 12.1 was current at that time!)  It was hard to configure but it could do OSPF and BGP.

Maintaining a home lab was a lot of work. I had to re-cable it a number of times until I finally got the stable topology that I liked. I often had to deal with bent pins which were so common with the serial cables of the time. I had to do IOS upgrades and the occasional memory upgrade when I needed an IOS but didn’t have the memory to support it. Sometimes, this work distracted from the work of studying. However, all of it was valuable experience. I don’t think I would trade it for GNS3 any day. GNS3 certainly adds an element of convenience, and allows the student to focus on the lab and not on extraneous tasks. And, it certainly has some challenges of its own. But I do think that dealing with the physical hardware, upgrading it, cabling it, etc. — helped me to grow as a network engineer, and also improve my general troubleshooting skills. Of course, as I mentioned before, the expense and difficulty of working with physical equipment prevented a number of people from attempting the lab, thus keeping the number of CCIE’s lower than in the GNS3 era.

Modern candidates for R/S may be able to use GNS3, but candidates for the other exams, such as data center, will go through the same struggles as myself.  Data Center is particularly bad because the equipment required is so expensive.  It is almost impossible to procure it on your own.  Rack rental is one option, but is fraught with problems, as I mentioned above.  The only other option is to try to get your employer to help you build and pay for a lab.  Despite the many changes in the exam, some things remain the same I suppose!

My next article, How to Pass the CCIE Lab Exam in One Attempt, reviews my study and preparation for the Routing and Switching lab exam.

The CCIE Mystique

In my first article in the “Ten Years a CCIE” series, I discuss the mystique of the CCIE certification which made me want to attempt the test.

Learning about the CCIE

My first vague awareness of the CCIE certification came in 1999 while I was a Master’s student in Telecommunications Management at Golden Gate University in San Francisco. A family friend was staying at my father’s house, an instructor in the PhD program in telecommunications at the University of Idaho. I was excited to meet him because I was thinking of pursuing further graduate studies, but I was a bit surprised when I told him of my ambitions to be a network engineer, and of my coursework at GGU. He told me not to waste my time in a graduate program if I wanted to be a network engineer. “You should get a Cisco certification instead,” he said, “they’re gold.” A disappointing comment, seeing that I was in my second year of the Masters program, but I kept it in mind and completed my degree. Read More »

1