All network engineers have had the experience of thinking: “Oh no, this is going to get me fired.” I remember Mike Flanagan, my first boss in TAC, telling me he once brought down an ISP POP in a remote location, requiring a truck roll to correct it. (Mike is also the guy who told me “if you don’t like your boss or your cubicle at Cisco, wait three months.” Three months later I had a new boss and a new cubicle.) In TAC, I remember once crashing a 7500 by issuing a show tech while consoled into a line card. The 7500 was already experiencing data bus congestion, and it locked the thing up. Neither Mike nor I got fired.
Actually, the first time I thought I’d get fired was early into my first job as a network engineer at the local newspaper. I had driven out to a circulation site, where they used to receive giant shipments of papers and then divide them up amongst different delivery trucks. I had gone out there later in the day to do something or other, I don’t really remember now. When I opened the wiring closet, I was shocked by the condition of the 2500-series router they had. It was caked, CAKED in dust, giant dust bunnies were blocking the vent hole and it was overall filthy.
I dutifully disconnected it, popped the lid, and blew it out, then wiped down the exterior surfaces. Then I reconnected the CSU/DSU, which is what we used to connect to a T1 line back then. The CSU/DSU had a serial connection for the router.
On the other serial port was something I had never seen before. It was a beige box about the size of a home router, with no lights that appeared to be non-functional. It looked to be as old as time. I called back to Stan at our HQ, a salty old newspaper IT guy, to ask if I could remove this as a part of my cleanup. “Sure,” said Stan, “if it has no lights on it, might as well get rid of it.”
Newspaper circulation sites tend to work through the night and early morning, so it was early the next morning when I got the call. Apparently the mystery box was used by the IBM mainframe to send subscriber lists out to the circulation office so they knew where to route the papers. A whole lot of people would not be getting their morning paper that day. Someone had to drive to my house to pick it up and then drive up to Marin to reconnect it. The manager of the mainframe group sent the CIO an email calling what I did an “irresponsible act”. (The mainframe guy used to manage the network before it was taken away from him, so there was already bad blood.) Somehow I managed to survive. In that rather toxic environment, revenge was necessary. I later discovered that, under the mainframe guy’s watch, he had purchased two DS3’s for dial-up access, but the second one was sitting on the Ascend dial-in box unconfigured. We had paid thousands of dollars over several years. My email called it an “irresponsible act”.
Mainframes were (and still are) a mystery to me. They have their own operating systems, their own connectivity systems, their own vocabulary, and their own IT people. They’re still around, but they’ve become a lot more integrated into mainstream computing, running Unix-like operating systems and allowing direct network connectivity. Back in the days I was dealing with them, they were still an island.
System/360, VTAM, TN3270, ESCON, FEP, zVM, SNA… Learning networking required mastering an alphabet soup, but mainframes took it too a new level. In five years of working on mainframe connectivity I only managed to scratch the surface, never feeling confident in this strange world.
(My inability to master mainframes is made all the more embarrassing after watching this video of 18-year-old Connor Krukosky purchasing a one-ton IBM z890 mainframe, and getting it operational in his basement. Trenching the yard to fit the thing through the basement door was priceless.)
In the early days of mainframes 3270 terminals, with black screens and green letters, were the primary means of interfacing with the system. In the MDF of our newspaper, there were a lot of single twisted-pair cross-connects which I believe were for this purpose. These could be terminated on a sort of converter that had a BNC coax on the other end. When I went to college, the administrators would use these terminals to input our course schedules into the mainframe while we selected them.
By the time I worked at the newspaper, the business people who worked on the mainframe interacted using TN3270 software on their PCs. This, of course, meant that the mainframe needed some sort of TCP/IP stack, and connection to a local area network. Eventually you could just plug the Ethernet cable in to the back of the mainframe, but in my day, not so! Instead, the network connection had to be “channel-attached”, the “channel” being a massive parallel-processing architecture used by mainframes to talk to storage and other facilities.
In our case, the channel attachment went to a 7513 router with a “Channel Interface Processor”, or CIP card. The CIP was as mysterious as the mainframe. It was a 7500 line card with its own unique configuration, which looked nothing like IOS config. It had the distinct disadvantage of having a config that neither the networking guys (too mainframey) nor the mainframe guys (too networky) understood.
A comically large cable was used to connect devices to the channel. This “bus-and-tag” cable had, as the name implies, a data bus and a “tag” channel which was used for out-of-band signaling. The thick gray cable ran under the floor tiles across the data center and plugged into the CIP. The TN3270 sessions actually terminated on the CIP card, which front-ended the sessions for the mainframe. Because the sessions were heavily connection-oriented, they were quite sensitive to network blips.
We also had no backup for the CIP card, since we had only one 7513 router. The network was poorly designed. The 7513 was our main WAN router, terminating the frame relay connections from our remote sites. ISDN circuits provided the redundancy for the FR, but the dial backup terminated on the same 7513. Whoops. If the 7513 went down, the WAN and the mainframe went down. This also meant that if we had to do maintenance on the 7513 for the WAN side, we’d take down the mainframe and vice versa. (Decisions like this are why the mainframe manager lost ownership of the network.) I had inked a plan to purchase a 7200 router for redundancy, since it supported CIP-like card called a “CPA”. However, since nobody really understood how the CIP worked, nobody could really figure out how to make channel redundancy work, so it never happened.
When I worked in TAC, I took a few cases on CIP cards. I learned then that there were basically two people who knew how they worked, and they both worked in Cisco’s mainframe TAC. They did CIP cards for a living, and knew everything about them, but they didn’t know much else. In fact, when I was a customer, my CIP cases were handled by these same two people. Now in TAC, I was IM’ing them and requeing my cases to them.
One good thing came out of my time working on mainframe networking. The aforementioned beige boxes connected to the 2500-series routers? They used DLSw+ to transport their SNA traffic. (SNA was the IBM networking protocol.) When I took the CCIE in 2004, DLSw+ was still a topic, and I was one of the few candidates who had hands-on experience with it.
I know mainframes are still around, but I haven’t touched mainframe networking in 20 years. I’m happy to leave that to Connor Krukosky, wherever he is now.