NetStalgia

Two articles (here and here) in my Netstalgia series covered the old bulletin board system (BBS) I used to operate back in the late 1980’s.  It wasn’t much by today’s standards, but I thoroughly enjoyed my time as a Sysop (systems operator).  How the BBS died is a lesson in product management.

My BBS ran on an Apple IIGS with a 2400 baud modem and two external 30MB hard drives (the Apple II series did not support internal hard drives.)  Hard drives were ridiculously expensive back then, and I had acquired the cheapest hard drives I could buy, manufactured by a company called Chinook.  I never knew anybody else who had Chinook hard drives, probably for good reason.  I had some of the files backed up on floppy disks, but there really wasn’t a good way to back up 60 megs of data without another hard drive.

One day I had the BBS shut down for some reason or other, and I went to turn it back on.  When I flipped the switch on Chinook #1, the disk didn’t spin up.  It simply clicked.  Not knowing what to do, I decided to call tech support.  I had lost the manual, however, so I had to do what we did before the Internet:  I called information.  By dialing 411 on my phone, I was connected with an operator who helped me to hunt down the number.

A 30MB Chinook HD

I dialed the number for Chinook.  A nice midwestern, older sounding man answered the phone.  He patiently listened while I explained my conundrum, and then said to me: “This is the Chinook fencing company.  You’re looking for Chinook, a computer company, it sounds like.”  I went back to information and got the right number.

Explaining my situation yet again, this time I got an answer.  “I want you to pick up the front of the hard drive and drop it on the table,” said the tech support guy.  I did it, and voila!  The hard drive spun up.  Despite my tender age of 16, I somehow suspected this was, as we say in the corporate world, “an unsustainable operating model.”

Luckily I rarely shut the hard drive down, but when I did I needed to drop it on the table to get it going again.  Chinook #2 started to have the same problem.  One day I flipped the switch on Chinook #1 and heard a metal-on-metal grinding noise.  And thus, my career as a Sysop ended.  All for the better I suppose, as the Internet was just around the corner.

I still have the Chinook hard drives, in the vain hope that I could crack them and recover some data some day.  I once called DriveSavers to see if they could do it, but the request to recover data on 1980’s Apple II crashed hard drives was just too weird for them.  Their proposal was expensive and not likely to succeed.

Three years ago, when I moved into my new neighborhood, we had a block party, and I ended up sitting next to an older fellow who had been a long-time product manager for Apple.  He provided a wealth of interesting stories about the Apple II line, and the history of many of the computers I got my start on so many decades ago.  I mentioned to him the Chinook problem, and to my surprise he knew Chinook.  Chinook actually repackaged a particular model Seagate HD, which was notorious for locking up and needing physical force to unstick the head.  My neighbor told me that this hard drive was included in the original prototypes of the Mac SE, over the objections of the technical product managers.  The business-types who were running things wanted the drive, either because it was cheap or because they had an agreement with Seagate (I don’t really recall).

Finally one of the technical PMs built a version of the SE which had a pinball plunger attached to the front of the built in HD.  Great idea!  When the hard drive got stuck, just pull back the plunger and let it rip!  He showed it to management and they decided to pick a different hard drive.  Good for them, the SE was to be a very popular Mac and the pinball plunger might have prevented that.  Anyways, as I had learned, the plunger wouldn’t work for very long.

Sun Ultra 10

It was four o’clock in the early hours of one Sunday morning in 2001.  I had been up all night sitting in our data center at the San Francisco Chronicle with our Unix guy.  He was handing off responsibility for managing the firewalls to the network team, and he was walking me through the setup.  He’d been trying all night to get failover to work between the two firewalls, and so far nothing was going right.

We were using Checkpoint which was running on Solaris.  Despite my desire to be Cisco-only, I was interested in security and happy to be managing the firewalls.  Still, looking at the setup our Unix guy had conceived, my enthusiasm was waning.

He drew a complex diagram on a piece of paper, showing the two Solaris servers.  There was no automatic failover, so any failure required manual intervention.  He has two levels of failover.  First, he was using RAID to duplicate the main hard disk over to a secondary hard disk.  If the main disk failed, we’d need to edit some text files with vi to somehow bring the Sparc Ultra 10 up on the second drive.  If the Ultra 10 failed entirely, we would have to edit some text files on the second Ultra 10 to bring it up with the configuration of the first.  With Unix guys, it’s always about editing text files in vi.

Aside from being cumbersome, it didn’t work.  We’d been at it for hours, and whatever disk targets he changed in whatever files, failover wasn’t happening. At the newspaper, we had until 5am Sunday to do our work, after which everything had to be back on line.  And we were getting concerned it wouldn’t come back at all.

Finally the Unix guy did manage to get the firewall booted up and running again.  On Monday I called Checkpoint and asked how we could get off Solaris.  They made a product called SecurePlatform, which installed a hardened Linux and Checkpoint all with one installer.  I ordered it at once, along with two IBM servers.

The software worked as promised, and I brought up a new system, imported our rules, and did interface and box failover with no problem.  I told the Unix guy to decommission his Ultra 10s.  He was furious that there was a *nix system on the network his team wasn’t managing.  I told him it was an appliance and there was no customization allowed.  The new system worked flawlessly and I didn’t even have to touch vi.

Network engineers are used to relatively simple devices that just work.  Routers and switches can be upgraded with a single image, and device and OS-level management is mostly under the hood.  While a lot of network engineers like Linux or Unix and have to work with these operating systems, at the end of the day when we want to do our job, we want systems that install and upgrade quickly, and fail over seamlessly.  As networking vendors move more into “software”, we need to keep that in mind.

When I worked at the San Francisco Chronicle, I started a project to bring Internet connectivity to a number of sites that had only limited mainframe circuits.  To do this I decided to get DSL lines and run IPSec over them, a relatively new way of doing things for the time.  It was a lot cheaper than the Frame Relay we used at larger sites.

After setting up connectivity at one of our sites, the local office manager called me.  Web pages, he said, were only loading partially.  Some of the text and none of the images would show up.

Everyone blamed the network for everything, so I punted him to desktop support.  I could ping across the tunnel, I could send traffic just fine, the latency was minimal, and nothing was obviously wrong.  The network is usually up or down, but web pages don’t partially load when everything else is working.  Degraded service might cause the pages to load slowly, but not partially.

The desktop guys told me it was my problem.  We had a constant battle, as nine times out of ten they blamed the network, and nine times out of ten it was not the network.  The office manager was getting angry, so I decided I would do some investigation on site and prove to the desktop guys that they were wrong.

I went to the office and fired up my laptop.  Pages were partially loading for me too.  Hmmm.  I did what every network engineer does and fired up a packet sniffer.

I could see the TCP handshake succeeding, and the browser requests and data exchange.  It looked normal, but why wasn’t the browser displaying the images?  I tried another browser and saw the same thing.

As I examined the sniffs, something hit me.  All the packets were being sourced with the Do not Fragment (DF) bit set in the header.  Could it be that the IPSec/GRE headers were causing the packets to be large enough to require fragmentation?  And why was Windows setting the DF bit anyways?

As I wasn’t a desktop guy, I left the latter question alone.  I jumped on the router and built a routing policy which cleared the DF bit on incoming packets.  The pages started loading fine.  I left the policy in place and hoped that there would not be any unanticipated consequences.  I never saw any.

Sometimes, it is, indeed, the network.

I’ve mentioned in the past how my first job in IT (starting in 1995) was as a “systems administrator” for a small company in Marin County, California.  The company designed and built museum exhibits, and its team of around 60 employees was split between fabricators, who built the exhibits, and office workers.  Some of the office workers did administrative work, while others were designers.  So, I was managing a network of around 30 computers, all Macs.

When I got to the company, the computers were networked using LocalTalk, a LAN technology from Apple, and specifically the PhoneNet variation.  PhoneNet was a product from Farallon Networks which enabled you to send the LocalTalk signal down a single pair of ordinary telephone wire.  The common practice was to use an extra pair of phone wires in the same cable that carried the user’s phone line.  In my first Netstalgia piece, I mentioned that my PhoneNet network was entirely passive, and ran into a lot of challenges as a result.

PhoneNet was also slow, and our designers had to transfer large files.  I decided to set up a separate Ethernet network for them.  All I knew about Ethernet was that it was faster, and that the higher-end PowerPC’s used by the designers supported it.  These computers had an AAUI port, a modification of the AUI port commonly in use for Ethernet connectivity at the time.  An AUI port required a transceiver to connect it to the Ethernet network.  Why?  Because we had Thicknet and Thinnet coaxial Ethernet, 10Base-T twisted pair and fiber optic Ethernet as well.  The universal AUI port (and Apple’s AAUI equivalent) gave you a choice of medium.

I didn’t really know how to make this work, and Google was not available at the time.  I had heard that you needed a “hub”, but I wasn’t sure exactly why or what the hub did.  The MacWarehouse catalogs I used to receive at the time advertised a product called an Etherwave, from the same company that made the PhoneNet transceiver.  The Etherwave allowed daisy-chaining of a twisted-pair Ethernet network.  I don’t know why, but this seemed easier and cheaper to me that buying a hub.  It was neither.

Farallon Etherwave Adapter

I bought a bunch of Etherwave adapters, got a ladder, and spend a night running Cat 3 cable in the suspended ceiling, and crimping RJ45 cubes.  Finally, I daisy chained everything together, and switched the computers to the new Ethernet network.  It worked very well–file transfers were screaming!

The designers loved it, but there was a flaw.  The Ethernet network was not connected at all to the LocalTalk network.  The LocalTalk network was where email, printing, and many other services resided.  Their computers had connections to both, but they had to go to a control panel and switch between one or the other.  That meant, if they wanted to do a file transfer, the two designers would have to shout to each other to switch networks, at which point they could do it peer-to-peer.

There was another problem.  Apple’s networking software, called OpenTransport, was notoriously buggy.  The switches between Ethernet and LocalTalk resulted in frequent crashes and reboots.  The initial thrill was wearing off.

I searched through catalogs and primitive websites looking for a solution.  I learned that I could buy a device called a router to connect the Ethernet and LocalTalk into a single unified network.  I desperately looked for the cheapest one.  My go-to vendor, Farallon, made a router but it was way too expensive.  Finally, I found a cheap router called a PathFinder manufactured by Dayna systems.

I went to my boss, the VP of operations.  I showed her the price (maybe $800?) and she balked.  This company ran a tight ship, and she said we couldn’t afford it.

I went back to our head designer and asked her to keep a post-in on her computer for a day, with a tally mark each time she had to reboot due to the Ethernet to LocalTalk switching.  Then we timed how long it took her to reboot.  I went back to my boss and showed her how much time was being lost each day to a single designer.  Her left hand flew over her the buttons on the calculator on her desk, then she looked up at me.  “Buy the router,” she said.

The PathFinder did indeed fix the problem.  And so my first Ethernet network, as well as my first experience configuring a router, came at a company in 1995 with 30 Macs, and I’ve spent decades working with both technologies since those days.

As a part of my job at Cisco I’ve been looking into Zscaler and their offerings.  It started me thinking back to the early days of remote access, and I figured it would make a good topic for Netstalgia.

I wrote in the past about how bulletin board systems (BBSs) work, and in another article I resurrected my old BBS in an Apple II emulator.  In a nutshell, a computer with a BBS set up had a modem on it and users dialed in using their own modem over dial-up phone lines.  I’m not sure how many readers are young and don’t remember modems, and how many are dinosaurs like me, but as a reminder, modems connect computers to phone lines.  One modem is set to answer any call that comes in, and waits.  Then another user with a modem inputs the phone number of the other end into his software.  His modem dials out, the phone rings, and the other modem answers with a carrier tone.  Then the dialer responds and after some negotiation on the line, a connection is established and data is sent.

Now in my first job, at a small company in Marin California in the mid-1990’s, we had one computer set up as a dedicated remote access server.  It had a single modem with a single phone line, and ran Apple Remote Access server, since we were a Mac shop.  We only had one user with a laptop, the CEO, so when he traveled he would dial-in and be able to access basic functions like email and our file server.  There was no Internet access back then.

When I moved on to a consulting company, I did a few more industrial set ups.  Usually these involved remote access servers that were comprised of a bunch of modems and a LAN port.  The remote access server would accept a bunch of phone lines and then provide TCP/IP or AppleTalk connectivity to the network.  By this time users had Internet connectivity.  The Shiva LanRover is one example of this sort of device.

Shiva LANRover

When I worked at the San Francisco Chronicle, we had an Ascend Max which served this purpose.  The Max had two DS3 lines plugged into it.  It was the first time I had seen a DS3, and I remember being excited to learn the phone company could deliver a circuit over coax.  (It actually entered the building on fiber and went over coax from the MPOE.)  The DS3 was an ISDN PRI, with 24 dial-in phone lines multiplexed over a single digital circuit.  It took me months to find someone who had the password to the Max, and when I finally got in I found out that the second DS3 was unconfigured.  Users had been complaining about busy signals and all I had to do was change a few menu settings.

Ascend Max

Remote access dial-up was heavily used at the Chronicle.  Reporters filed their stories via modem.  VPN was just coming out, and I decided to replace the dial-up with VPN + dial-up.  A company called Fiberlink provided a dialer with a vast database of local Internet dial-up lines from a variety of carriers they contracted with.  Our users would pick a local phone line and then dial into it.  They then launched our Nortel VPN client to establish connectivity.  This saved us a fortune on 800-number charges, but our users hated it.  As a good senior guy, I did the initial design and left implementation to a junior guy.  I’m amazed he still talks to me.  (And he’s not junior anymore!)

Despite being a long-time Cisco guy, I never touched the Cisco remote access stuff.  I did use 2500-series routers with serial ports as terminal servers in the lab, but I never connected modems to them.  Still, when I passed my CCNP, one exam covered remote access and I needed to know a lot about modems.

Nowadays I rarely log into VPN.  Most systems I need to access can authenticate through our Zero Trust/SSO system without the need for a connection to Cisco’s network.  We’ve come a long way since the days of dial-up.  And while I said I missed wiring in another post, I sure don’t miss modem tones!

 

When I worked for the Gold partner I generally serviced clients in the San Francisco Bay Area, but because we were a national partner I was occasionally called to other locations around the country.  Being a double CCIE who had worked in TAC, I had a unique skill set among our engineers, which was often demanded by other field offices.

One day my boss called me and told me he needed my help with a customer out of Des Plaines, Illinois.  The company was a manufacturer of fuses.  They were experiencing a network meltdown and needed troubleshooting help.  Great, I thought, I left TAC and came here precisely to get out of this sort of thing.  I liked doing sales calls and new installations, not fixing buggy messes.

I was assigned to the customer on a Monday and was immediately pulled into what we often call a “shit show”.  (Pardon the language.)  The customer had a large international MPLS network with VPN backup.  Several of the sites were experiencing performance issues.  Sites were unable to perform manufacturing and the previous CIO had been fired.  The interim CIO was an ex-military person who seemed to think he was George S. Patton.  He was scheduling calls from early in the morning until late at night, status updates, live troubleshooting sessions, and pow wows with TAC.

Meanwhile, I was starting to feel ill.  Not because of the case, just sick to my stomach.  I didn’t think much of it at first, but it started to go downhill fast.  Luckily I was working from home due to the crazy hours.

But not for long.  The CIO had set up a troubleshooting session in the middle of the night Saturday, into Sunday morning.  He got my boss and me on the phone and insisted I come to Des Plains that weekend, in person.  We argued every way we could that I could be just as productive remotely, but Patton was having none of it.  “If this is your best guy,” he said to my boss, “you need to have him on a plane and out here in person.  Otherwise we can take our business somewhere else.”  Not only was it the weekend, and not only did I feel ill, it was also Memorial Day weekend.  My brother, who actually was (and is) in the Army was paying a rare visit to the Bay Area.  There was no sympathy from the customer, and soon I was booking my ticket to Illinois.  The local account executive booked me a car with GPS and promised to meet me at the airport.  Keep in mind, this was before smartphones, and so you needed to rent a car with a built-in GPS unit if you wanted to get around without maps.

I had a miserable flight and was starting to feel more sick.  There’s nothing worse than being sick to your stomach on a plane.  Being forced to stay in your seat and long lines for tiny bathrooms make for torture.  I ate nothing, and arrived at Chicago airport late.  The account executive was nowhere to be found, and the car he had arranged did not have GPS.  The rental car company didn’t have any GPS-equipped cars, so they provided me with a map and directions.

I drove through Chicago to Des Plaines.  Realizing I needed to eat something, I found a McDonalds, the only thing open at that time, and managed to choke down a half of a quarter pounder.  My stomach felt like burning acid.  I continued my drive, through a bad part of the city.  I was on the right road, but I needed to keep pulling over to check the address.  A couple times I pulled over, swarms of what I assume were drug dealers would approach.  I’d pull out just as they got to the car.

Eventually I made it to the customer site, and met the general.  I was shown in to a conference room with a raised floor right next to the data center, and we began troubleshooting.

It was nothing I couldn’t have aided with remotely.  Basically, the customer had scoped circuits that were too small for the volume of traffic they were carrying.  They were also having degradation problems on the MPLS.   Some of the sites were performing better on VPN backup circuits, so we were switching them to backups.  We performed tests with the telco.  We also looked into an issue with their core Catalyst 6k switch.  When they had done a circuit switch earlier in the week, all traffic on the network had stopped, according to the customer.  The customer had reloaded the core device and traffic came back.  Because there was no crash or crashdump file, and nothing in the logs, I could not explain this event.  It was a smorgasbord of issues, mostly due to bad design and a little due to bad luck.

The troubleshooting window was supposed to end at 2am, but we worked until 6am.  I had a flight to catch and hadn’t slept all night.  The customer wanted me to stick around but I told him to stuff it and left.  I checked in to the hotel room I had booked, slept one hour, checked out, and got on my plane.

On the flight back, I was seated in the middle seat between a two very large people.  I figured out that they were married, but they only spoke Spanish.  I used my rudimentary Spanish to extract myself.  “Yo, a la ventana.  Su esposa, aquí,” I suggested.  They liked the idea.  I moved to the window seat.  When the plane took off, they spread out a massive feast on the tray tables.  I ate a single muffin from Starbucks, one bite at a time, until I landed and went home.

I never determined if it was a norovirus or food poisoning, but I lost twenty pounds in a week.  The customer realized they needed to invest in new circuits, which had a 12-16 week turnaround time.  I think the new CIO got fired as well.  And the account executive never invited me back.  I only wish he had shown up at the airport as I might have thrown up on him.

We’ve moved into a wireless world, which is too bad for me because I love, more than anything, wiring.  I miss the days of good old Cat 3 cable, T1 lines, and ISDN BRIs.  I miss 66 blocks, punch down tools, cross-connect wire, and tone/probe kits.  And butt sets.  Especially butt sets.  Now I just have to add random wall receptacles around my house since, thank goodness, 120 volts cannot be delivered wirelessly.  But phone and network wiring was much more fun.

I first got interested in wiring when I was working at a small museum exhibit design and fabrication company in Marin, California.  One day, I had to have a new phone line installed, and I called in Pacific Bell to do the work.  Under the receptionist’s desk was a wall plate with two RJ-11 jacks, already in use.  “There aren’t any free jacks,” I told the phone guy.

“It doesn’t matter, as long as there is enough wire,” was his cryptic response.  I let him do his work, and when he was done I saw a little surface-mount RJ11 jack on the wall.  It had two blue and yellow wires running from it, going underneath the existing faceplate and somehow into the wall.  How on earth did he do that?  I remember thinking.

I waited until everyone went home.  I unscrewed the faceplate and found that the blue/yellow wire pair was spliced to the brown pair of wires in the cable that serviced the existing jacks.  The splice was a 3M UR-type, which looked like a piece of candy.  I was captivated.  How did he know where the brown wires went?  Where did the incoming phone line enter the building?  How did he connect them up?

The fun thing about the days before ubiquitous Internet was that you couldn’t get answers immediately.  When I found the 66-type punchdown blocks that formed our little MDF, I couldn’t Google the part number to figure out what they were.  Google didn’t exist.  I had no idea how to terminate a wire on one of them.  While thumbing through a Jensen Tools catalog our purchasing agent had I got an idea of how it worked.  There, I saw listed a “66 punch-down tool” and I had noticed the numbers “66” on the block.  OK, so they go together and the wire gets in the clip by “punching down.”

I didn’t have the money to afford a punch-down tool.  So I got a pair of pliers and a pair of forceps and started doing terminations myself, guiding the wire into the clip.  Sure, sparks were flying, but phone systems are robust, aren’t they?

It turns out not as robust as you might think.  Back in the day, phone lines provided by the phone company (POTS service) were indeed robust and could handle quite a bit of sparking and shorting.  But the Nitsuko (emphasis on the “suk”) system we used did not take kindly to my laissez-faire approach to wiring.  One day, while doing a move, I shorted out a couple terminals and all the phones in the building went out.  I came out of the closet with pliers in hand and everyone knew what happened.  This being before cell phones, business was done for the day.  Luckily we got our phone company out to replace the bad part, and they did it under warranty.  It was at this point I invested in a proper punch-down tool and learned how to wire correctly.

The author’s tools: Butt set, punch down tools, tone/probe set, and connectors

I saved up a lot of money and eventually got a test set, otherwise known as a “butt set”.  The butt set looks like an oversized telephone handset, and is the official sign of a telco wiring guy.  It also was my passport into telephone wiring closets–when a security guard sees you have a butt set, they just assume you’re a real phone guy.

I practiced my technique in my father’s 1910-era house.  The decades brought with them layers of phone wiring, including two 50-pair feeder cables and a 66-style punch-down block.  Why and how the feeder cables ended up in a house with 3 phone lines is a mystery to this day.  But I used the 66-block for practice and dissected the criss-crossing wires in his house, tracing them out with my trusty tone/probe set.

Wiring skills came in handy many times in my career as a network engineer.  I remember one customer I had in the late nineties who had ordered 8 Centrex lines from Pacific Bell.  The technician showed up to do the cross-connects, but he was not the sharpest knife in the drawer, did two of them, and left.  Using my butt set passport, I got access to the MDF, figured out the right punch-down block that fed to the customer’s suite, and ran the cross-connects myself.

When I worked for the San Francisco Chronicle, we actually had an in-house wiring tech.  Her name was Mona Lesa.  She couldn’t care less if I did my own wiring, as long as I adhered to her standards.  One day I had a frame relay T1 installed in our Sacramento bureau office, and once again, Pac Bell forgot to connect it to the suite.  The bureau was located in the Old Senator Office Building across from the state capitol.  I got the security guard to let me in to the MDF in the bowels of this historic building.  I figured out that one of the interconnects was in the office of a lobbying firm, and the receptionist dutifully let me in to do the wiring.  With my butt set I could clip into any of their phone lines and listen to their conversations without them knowing.  I didn’t but I was amazed that I could go anywhere to do cross-connects.

On another occasion, a customer of mine had two phone lines installed and had a weird problem.  If she picked up one line and dialed the other, she’d hear both ringing and a busy signal at the same time.  I found where the Pacific Bell guy had done the cross-connect, and realized he mixed the tip and ring wires for the two phone lines.  A couple punches and all fixed.

Phone wiring was always the perfect blend of mind and body to me.  To do it you need to work with your hands, but you also need to use your mind.  Some wiring closets were rats nests of 24-gauge wires color-coded with more colors than a tie-died shirt.  Finding that one pair you needed, and marrying it up to the other end was always a nice diversion from staring at a screen.

Alas, my tools mostly sit unused.  Regular POTS lines rarely exist now, and even they aren’t delivered on single copper pairs from the CO switch.  PBXs and key systems have gone the way of the dinosaur, replaced by voice-over-IP and now by cellular phones.  Nobody wants to be tied to wires anymore, and yet by un-encumbering ourselves in the name of freedom, we’ve paradoxically lost our freedom.  When you don’t need to be physically tied to a location for connectivity, you can never escape connectivity.  And that’s not entirely a good thing.

I’ve written before about my years at the San Francisco Chronicle, my first job which was exclusively network engineering.  It was an interesting environment, as this was back in the years before the Internet totally displaced newspapers.  We had printing plants to support, active newsrooms with reporters and photographers, and a massive circulation operation.

When I first arrived, our Internet was provided by a T1.  At the time, the 1.5 Mbps was considered pretty fast, but with over 1000 users depending on the T1, it was slowing to a crawl.  The T1 was terminated on a good old-fashioned Cisco 2500-series router.  Later I upgraded our service to a T3 and terminated it on a 7204VXR, which led to a dramatic speed improvement.  The 2500 sat in an open, free-standing rack in the corner of our data center.

One day I noticed that the Internet was down.  Even though this was the early 2000’s, Internet connectivity was already critical, especially at a newspaper.  The problem quickly escalated to the CIO and I scrambled into action.  I could reach as far as the firewall, but there was no connectivity beyond that.

Sometimes the best thing to do is to physically check on a problem.  The networking team was in the basement and our data center was on the second floor.  I sprinted up the stairs and badged in to the data center.  One of our mainframe operators, who worked in the data center grabbed me and asked “hey, do you know the Internet is…?”

“Yeah, yeah,” I said and ran to the rack with our T1 router and DMZ switches.

I saw a gentleman in white painters clothes with a paint roller in the corner.  He was painting the wall right along side the rack.  There was only a few inches of clearance between the rack and the wall where he was rolling paint.  On the side of the rack, held in place with plastic zip ties, was a power strip with all the rack’s hardware plugged in.  He’d hit it with the roller and knocked out the power.

Network engineers love to solve complex problems, but when people are yelling at you it’s a relief when the problem is simple.  I flipped the switch on the power strip and Internet was restored as soon as the router booted up.  Someone had obviously placed the power strip on the wall-side of the rack figuring it would minimize the chance of it getting bumped.  They had inadvertently created the problem they were trying to solve.  I told the painter to stay away from my rack.

Networks are complex entities, but often the problems we face involve bad splices in holes in the ground, physical obstructions to WiFi signals, loose connections, and rogue paint rollers.

 

I thought I’d take  break from Cisco Live to relive some memories in another Netstalgia.

Working in product management at a Cisco business unit, we are constantly talking about the latest and greatest, the cutting edge of technology.  It’s easy to forget how many customers out there are nowhere near the cutting edge.  They’re not near any edge at all.  When I worked at a Gold partner, I got to see all sorts of customer networks in a variety of states.

I remember one customer who ended up spending a lot of money with my company.  They were (and are) a major grocery chain in the United States.  They had reached a point when their network was grinding to a halt, and they needed help.  They had two overworked network engineers running things, and I remember being amused that their company policy required them to wear ties in the office.  This was not a financial company in San Francisco, but a discount grocery chain in a very relaxed part of the East Bay.  Anyways, they had hosts dropping off the network, performance problems, and their mainframe kept losing contact with its default gateway.

Walking in to these kinds of situations, you’re never sure what you might find.  Oftentimes the problems aren’t clear and the customer gets frustrated with their hired gun.  In this case, the very affable in-house engineers were thrilled to have experienced help.  They explained to me that the entire network, a large corporate office and countless stores, were on a single /8 network.  Only the on-site data center had a separate subnet.  Even the remote sites were in the /8!!

It got worse.  The stores were connected to HQ with IPSec VPN, but the hardware VPN devices they were made by a company that no longer existed.  The devices kept failing, and one of the network engineers had a stock of them he had purchased on eBay.  He amazingly was using his electronics skills to perform component-level repairs on the devices, cannibalizing parts from the eBay stash, which enabled him to stretch the stash longer than if he had simply swapped them.

My favorite was the data center.  The mainframe was sending constant pings to its default gateway, which would occasionally drop packets, in which case the mainframe would declare the gateway dead.  I found out that the default gateway was none other than a 2500-series router.

An old 2503 router

Even in 2009, this router was ancient history.  It had an old AUI connector on it which was nearly falling out.  In their 100 Mbps environment, they were limited to 10 Mbps.  I seem to recall it was doing router-on-a-stick on that interface, hairpinning traffic, but I don’t think the 2500 could do subinterfaces, so I may be wrong.  Anyways, the poor little 2500 was being slammed by traffic and dropping packets from time-to-time.

The ubiquitous CentreCOM AUI 10Base-T transceiver

I spent months at the client.  We designed a subnet scheme, renumbered their network, installed ASAs for IPSec, cut over all the stores, and put some high-end switches in the data center.  They were a grateful client, never complained, and I was able to make a genuine improvement in the lives of their users.  Unlike that other client I wrote about before.

I have a lot of bad memories of working for that partner, but one of the most interesting things was walking into so many different customers’ worlds, seeing what they dealt with every day, the mistakes they had made in building their networks, and helping them out by fixing those mistakes.

My first IT job was at a small company in Novato, California, that designed and built museum exhibits.  At the time most companies either designed the exhibits or built them, but ours was the only one that did both.  You could separate the services, and just do one or the other, but our end-to-end model was the best offering because the fabricators and designers were in the same building and could collaborate easily.  The odd thing about separating the functions was that we could lose a bid to design a project, but win the bid to build it, and hence end up having to work closely with a competitor to deliver their vision.

A museum exhibit we designed and built

The company was small–only 60 employees.  Half of them were fabricators who did not have computers, whereas the other half were designers and office staff who did.  My original job was to be a “gopher” (or go-fer), who goes for stuff.  If someone needed paint, screws, a nail gun, fumigation of a stuffed tiger, whatever, I’d get in the truck and take care of it.  However, they quickly realized I was skilled with computers and they asked me to take over as their IT guy.  (Note to newbies:  When this happens, especially at a small company, people often don’t forget you had the old job.  One day I might be fixing a computer, then the next day I’d be hauling the stuffed tiger.)

This was in the mid-1990’s, so let me give you an idea of how Internet connectivity worked:  it didn’t.  We had none when I started.  We had a company-internal network using LocalTalk (which I described in a previous post), so users could share files, but they had no way to access the Internet at all.  We had an internal-only email system called SnapMail, but it had no ability to do SNMP or connect beyond our little company.

The users started complaining about this, and I had to brainstorm what to do when we had virtually no operating budget at all.  I pulled out the yellow pages and looked under “I”, and found a local ISP.  I called them, and the told me I could use Frame Relay, a T1, or ISDN.  I had no idea what they were talking about.  The sales person faxed me a technical description of these technologies, and I still had no idea what they were talking about.  At this point I didn’t know the phone company could deliver anything other than, well, a phone line.  I wasn’t at the point where I needed to hear about framing formats and B8ZS line encoding.

We decided we could afford neither the ongoing expense, nor the hardware, so we came up with a really bad solution.  We ordered modems for three of the computers in the office:  the receptionist, the CEO, and the science researcher.  For those of you too young to remember, modems allow you to interface computers using an ordinary phone line.  We ordered a single phone line (all we could afford).  When one of them wanted to use the Internet, they would run around the office to check with the other two if the line was free.

A circa-1990’s Global Village modem

The reason we gave the receptionist a modem is amusing.  Our dial-up ISP allowed us to create public email addresses for all of our employees.  However, they all dumped into one mailbox.  The receptionist would dial in in the morning, download all the emails, and copy and paste them into the internal email system.  If somebody wanted to reply, the would send it to the receptionist via SnapMail and she would dial up, paste it into the administrator account, and send it.  Brilliant.

Needless to say, customer satisfaction was not high, even in those days.  Sick of trying to run IT with no money, I bailed for a computer consulting company in San Francisco and started installing the aforementioned T1s and ISDN lines for customers, with actual routers.

If ever you’re annoyed with slow Wi-Fi, be glad you aren’t living in the 1990’s.