Category Archives: NetStalgia

by

It is the network

When I worked at the San Francisco Chronicle, I started a project to bring Internet connectivity to a number of sites that had only limited mainframe circuits.  To do this I decided to get DSL lines and run IPSec over them, a relatively new way of doing things for the time.  It was a lot cheaper than the Frame Relay we used at larger sites.

After setting up connectivity at one of our sites, the local office manager called me.  Web pages, he said, were only loading partially.  Some of the text and none of the images would show up.

Everyone blamed the network for everything, so I punted him to desktop support.  I could ping across the tunnel, I could send traffic just fine, the latency was minimal, and nothing was obviously wrong.  The network is usually up or down, but web pages don’t partially load when everything else is working.  Degraded service might cause the pages to load slowly, but not partially.

The desktop guys told me it was my problem.  We had a constant battle, as nine times out of ten they blamed the network, and nine times out of ten it was not the network.  The office manager was getting angry, so I decided I would do some investigation on site and prove to the desktop guys that they were wrong.

I went to the office and fired up my laptop.  Pages were partially loading for me too.  Hmmm.  I did what every network engineer does and fired up a packet sniffer.

I could see the TCP handshake succeeding, and the browser requests and data exchange.  It looked normal, but why wasn’t the browser displaying the images?  I tried another browser and saw the same thing.

As I examined the sniffs, something hit me.  All the packets were being sourced with the Do not Fragment (DF) bit set in the header.  Could it be that the IPSec/GRE headers were causing the packets to be large enough to require fragmentation?  And why was Windows setting the DF bit anyways?

As I wasn’t a desktop guy, I left the latter question alone.  I jumped on the router and built a routing policy which cleared the DF bit on incoming packets.  The pages started loading fine.  I left the policy in place and hoped that there would not be any unanticipated consequences.  I never saw any.

Sometimes, it is, indeed, the network.

5
by

My first Ethernet Network and Router

I’ve mentioned in the past how my first job in IT (starting in 1995) was as a “systems administrator” for a small company in Marin County, California.  The company designed and built museum exhibits, and its team of around 60 employees was split between fabricators, who built the exhibits, and office workers.  Some of the office workers did administrative work, while others were designers.  So, I was managing a network of around 30 computers, all Macs.

When I got to the company, the computers were networked using LocalTalk, a LAN technology from Apple, and specifically the PhoneNet variation.  PhoneNet was a product from Farallon Networks which enabled you to send the LocalTalk signal down a single pair of ordinary telephone wire.  The common practice was to use an extra pair of phone wires in the same cable that carried the user’s phone line.  In my first Netstalgia piece, I mentioned that my PhoneNet network was entirely passive, and ran into a lot of challenges as a result.

PhoneNet was also slow, and our designers had to transfer large files.  I decided to set up a separate Ethernet network for them.  All I knew about Ethernet was that it was faster, and that the higher-end PowerPC’s used by the designers supported it.  These computers had an AAUI port, a modification of the AUI port commonly in use for Ethernet connectivity at the time.  An AUI port required a transceiver to connect it to the Ethernet network.  Why?  Because we had Thicknet and Thinnet coaxial Ethernet, 10Base-T twisted pair and fiber optic Ethernet as well.  The universal AUI port (and Apple’s AAUI equivalent) gave you a choice of medium.

I didn’t really know how to make this work, and Google was not available at the time.  I had heard that you needed a “hub”, but I wasn’t sure exactly why or what the hub did.  The MacWarehouse catalogs I used to receive at the time advertised a product called an Etherwave, from the same company that made the PhoneNet transceiver.  The Etherwave allowed daisy-chaining of a twisted-pair Ethernet network.  I don’t know why, but this seemed easier and cheaper to me that buying a hub.  It was neither.

Farallon Etherwave Adapter

I bought a bunch of Etherwave adapters, got a ladder, and spend a night running Cat 3 cable in the suspended ceiling, and crimping RJ45 cubes.  Finally, I daisy chained everything together, and switched the computers to the new Ethernet network.  It worked very well–file transfers were screaming!

The designers loved it, but there was a flaw.  The Ethernet network was not connected at all to the LocalTalk network.  The LocalTalk network was where email, printing, and many other services resided.  Their computers had connections to both, but they had to go to a control panel and switch between one or the other.  That meant, if they wanted to do a file transfer, the two designers would have to shout to each other to switch networks, at which point they could do it peer-to-peer.

There was another problem.  Apple’s networking software, called OpenTransport, was notoriously buggy.  The switches between Ethernet and LocalTalk resulted in frequent crashes and reboots.  The initial thrill was wearing off.

I searched through catalogs and primitive websites looking for a solution.  I learned that I could buy a device called a router to connect the Ethernet and LocalTalk into a single unified network.  I desperately looked for the cheapest one.  My go-to vendor, Farallon, made a router but it was way too expensive.  Finally, I found a cheap router called a PathFinder manufactured by Dayna systems.

I went to my boss, the VP of operations.  I showed her the price (maybe $800?) and she balked.  This company ran a tight ship, and she said we couldn’t afford it.

I went back to our head designer and asked her to keep a post-in on her computer for a day, with a tally mark each time she had to reboot due to the Ethernet to LocalTalk switching.  Then we timed how long it took her to reboot.  I went back to my boss and showed her how much time was being lost each day to a single designer.  Her left hand flew over her the buttons on the calculator on her desk, then she looked up at me.  “Buy the router,” she said.

The PathFinder did indeed fix the problem.  And so my first Ethernet network, as well as my first experience configuring a router, came at a company in 1995 with 30 Macs, and I’ve spent decades working with both technologies since those days.

1
by

Remote Access

As a part of my job at Cisco I’ve been looking into Zscaler and their offerings.  It started me thinking back to the early days of remote access, and I figured it would make a good topic for Netstalgia.

I wrote in the past about how bulletin board systems (BBSs) work, and in another article I resurrected my old BBS in an Apple II emulator.  In a nutshell, a computer with a BBS set up had a modem on it and users dialed in using their own modem over dial-up phone lines.  I’m not sure how many readers are young and don’t remember modems, and how many are dinosaurs like me, but as a reminder, modems connect computers to phone lines.  One modem is set to answer any call that comes in, and waits.  Then another user with a modem inputs the phone number of the other end into his software.  His modem dials out, the phone rings, and the other modem answers with a carrier tone.  Then the dialer responds and after some negotiation on the line, a connection is established and data is sent.

Now in my first job, at a small company in Marin California in the mid-1990’s, we had one computer set up as a dedicated remote access server.  It had a single modem with a single phone line, and ran Apple Remote Access server, since we were a Mac shop.  We only had one user with a laptop, the CEO, so when he traveled he would dial-in and be able to access basic functions like email and our file server.  There was no Internet access back then.

When I moved on to a consulting company, I did a few more industrial set ups.  Usually these involved remote access servers that were comprised of a bunch of modems and a LAN port.  The remote access server would accept a bunch of phone lines and then provide TCP/IP or AppleTalk connectivity to the network.  By this time users had Internet connectivity.  The Shiva LanRover is one example of this sort of device.

Shiva LANRover

When I worked at the San Francisco Chronicle, we had an Ascend Max which served this purpose.  The Max had two DS3 lines plugged into it.  It was the first time I had seen a DS3, and I remember being excited to learn the phone company could deliver a circuit over coax.  (It actually entered the building on fiber and went over coax from the MPOE.)  The DS3 was an ISDN PRI, with 24 dial-in phone lines multiplexed over a single digital circuit.  It took me months to find someone who had the password to the Max, and when I finally got in I found out that the second DS3 was unconfigured.  Users had been complaining about busy signals and all I had to do was change a few menu settings.

Ascend Max

Remote access dial-up was heavily used at the Chronicle.  Reporters filed their stories via modem.  VPN was just coming out, and I decided to replace the dial-up with VPN + dial-up.  A company called Fiberlink provided a dialer with a vast database of local Internet dial-up lines from a variety of carriers they contracted with.  Our users would pick a local phone line and then dial into it.  They then launched our Nortel VPN client to establish connectivity.  This saved us a fortune on 800-number charges, but our users hated it.  As a good senior guy, I did the initial design and left implementation to a junior guy.  I’m amazed he still talks to me.  (And he’s not junior anymore!)

Despite being a long-time Cisco guy, I never touched the Cisco remote access stuff.  I did use 2500-series routers with serial ports as terminal servers in the lab, but I never connected modems to them.  Still, when I passed my CCNP, one exam covered remote access and I needed to know a lot about modems.

Nowadays I rarely log into VPN.  Most systems I need to access can authenticate through our Zero Trust/SSO system without the need for a connection to Cisco’s network.  We’ve come a long way since the days of dial-up.  And while I said I missed wiring in another post, I sure don’t miss modem tones!

 

2
by

A nauseating outage

When I worked for the Gold partner I generally serviced clients in the San Francisco Bay Area, but because we were a national partner I was occasionally called to other locations around the country.  Being a double CCIE who had worked in TAC, I had a unique skill set among our engineers, which was often demanded by other field offices.

One day my boss called me and told me he needed my help with a customer out of Des Plaines, Illinois.  The company was a manufacturer of fuses.  They were experiencing a network meltdown and needed troubleshooting help.  Great, I thought, I left TAC and came here precisely to get out of this sort of thing.  I liked doing sales calls and new installations, not fixing buggy messes.

I was assigned to the customer on a Monday and was immediately pulled into what we often call a “shit show”.  (Pardon the language.)  The customer had a large international MPLS network with VPN backup.  Several of the sites were experiencing performance issues.  Sites were unable to perform manufacturing and the previous CIO had been fired.  The interim CIO was an ex-military person who seemed to think he was George S. Patton.  He was scheduling calls from early in the morning until late at night, status updates, live troubleshooting sessions, and pow wows with TAC.

Meanwhile, I was starting to feel ill.  Not because of the case, just sick to my stomach.  I didn’t think much of it at first, but it started to go downhill fast.  Luckily I was working from home due to the crazy hours.

But not for long.  The CIO had set up a troubleshooting session in the middle of the night Saturday, into Sunday morning.  He got my boss and me on the phone and insisted I come to Des Plains that weekend, in person.  We argued every way we could that I could be just as productive remotely, but Patton was having none of it.  “If this is your best guy,” he said to my boss, “you need to have him on a plane and out here in person.  Otherwise we can take our business somewhere else.”  Not only was it the weekend, and not only did I feel ill, it was also Memorial Day weekend.  My brother, who actually was (and is) in the Army was paying a rare visit to the Bay Area.  There was no sympathy from the customer, and soon I was booking my ticket to Illinois.  The local account executive booked me a car with GPS and promised to meet me at the airport.  Keep in mind, this was before smartphones, and so you needed to rent a car with a built-in GPS unit if you wanted to get around without maps.

I had a miserable flight and was starting to feel more sick.  There’s nothing worse than being sick to your stomach on a plane.  Being forced to stay in your seat and long lines for tiny bathrooms make for torture.  I ate nothing, and arrived at Chicago airport late.  The account executive was nowhere to be found, and the car he had arranged did not have GPS.  The rental car company didn’t have any GPS-equipped cars, so they provided me with a map and directions.

I drove through Chicago to Des Plaines.  Realizing I needed to eat something, I found a McDonalds, the only thing open at that time, and managed to choke down a half of a quarter pounder.  My stomach felt like burning acid.  I continued my drive, through a bad part of the city.  I was on the right road, but I needed to keep pulling over to check the address.  A couple times I pulled over, swarms of what I assume were drug dealers would approach.  I’d pull out just as they got to the car.

Eventually I made it to the customer site, and met the general.  I was shown in to a conference room with a raised floor right next to the data center, and we began troubleshooting.

It was nothing I couldn’t have aided with remotely.  Basically, the customer had scoped circuits that were too small for the volume of traffic they were carrying.  They were also having degradation problems on the MPLS.   Some of the sites were performing better on VPN backup circuits, so we were switching them to backups.  We performed tests with the telco.  We also looked into an issue with their core Catalyst 6k switch.  When they had done a circuit switch earlier in the week, all traffic on the network had stopped, according to the customer.  The customer had reloaded the core device and traffic came back.  Because there was no crash or crashdump file, and nothing in the logs, I could not explain this event.  It was a smorgasbord of issues, mostly due to bad design and a little due to bad luck.

The troubleshooting window was supposed to end at 2am, but we worked until 6am.  I had a flight to catch and hadn’t slept all night.  The customer wanted me to stick around but I told him to stuff it and left.  I checked in to the hotel room I had booked, slept one hour, checked out, and got on my plane.

On the flight back, I was seated in the middle seat between a two very large people.  I figured out that they were married, but they only spoke Spanish.  I used my rudimentary Spanish to extract myself.  “Yo, a la ventana.  Su esposa, aquí,” I suggested.  They liked the idea.  I moved to the window seat.  When the plane took off, they spread out a massive feast on the tray tables.  I ate a single muffin from Starbucks, one bite at a time, until I landed and went home.

I never determined if it was a norovirus or food poisoning, but I lost twenty pounds in a week.  The customer realized they needed to invest in new circuits, which had a 12-16 week turnaround time.  I think the new CIO got fired as well.  And the account executive never invited me back.  I only wish he had shown up at the airport as I might have thrown up on him.

5
by

For the love of wiring

We’ve moved into a wireless world, which is too bad for me because I love, more than anything, wiring.  I miss the days of good old Cat 3 cable, T1 lines, and ISDN BRIs.  I miss 66 blocks, punch down tools, cross-connect wire, and tone/probe kits.  And butt sets.  Especially butt sets.  Now I just have to add random wall receptacles around my house since, thank goodness, 120 volts cannot be delivered wirelessly.  But phone and network wiring was much more fun.

I first got interested in wiring when I was working at a small museum exhibit design and fabrication company in Marin, California.  One day, I had to have a new phone line installed, and I called in Pacific Bell to do the work.  Under the receptionist’s desk was a wall plate with two RJ-11 jacks, already in use.  “There aren’t any free jacks,” I told the phone guy.

“It doesn’t matter, as long as there is enough wire,” was his cryptic response.  I let him do his work, and when he was done I saw a little surface-mount RJ11 jack on the wall.  It had two blue and yellow wires running from it, going underneath the existing faceplate and somehow into the wall.  How on earth did he do that?  I remember thinking.

I waited until everyone went home.  I unscrewed the faceplate and found that the blue/yellow wire pair was spliced to the brown pair of wires in the cable that serviced the existing jacks.  The splice was a 3M UR-type, which looked like a piece of candy.  I was captivated.  How did he know where the brown wires went?  Where did the incoming phone line enter the building?  How did he connect them up?

The fun thing about the days before ubiquitous Internet was that you couldn’t get answers immediately.  When I found the 66-type punchdown blocks that formed our little MDF, I couldn’t Google the part number to figure out what they were.  Google didn’t exist.  I had no idea how to terminate a wire on one of them.  While thumbing through a Jensen Tools catalog our purchasing agent had I got an idea of how it worked.  There, I saw listed a “66 punch-down tool” and I had noticed the numbers “66” on the block.  OK, so they go together and the wire gets in the clip by “punching down.”

I didn’t have the money to afford a punch-down tool.  So I got a pair of pliers and a pair of forceps and started doing terminations myself, guiding the wire into the clip.  Sure, sparks were flying, but phone systems are robust, aren’t they?

It turns out not as robust as you might think.  Back in the day, phone lines provided by the phone company (POTS service) were indeed robust and could handle quite a bit of sparking and shorting.  But the Nitsuko (emphasis on the “suk”) system we used did not take kindly to my laissez-faire approach to wiring.  One day, while doing a move, I shorted out a couple terminals and all the phones in the building went out.  I came out of the closet with pliers in hand and everyone knew what happened.  This being before cell phones, business was done for the day.  Luckily we got our phone company out to replace the bad part, and they did it under warranty.  It was at this point I invested in a proper punch-down tool and learned how to wire correctly.

The author’s tools: Butt set, punch down tools, tone/probe set, and connectors

I saved up a lot of money and eventually got a test set, otherwise known as a “butt set”.  The butt set looks like an oversized telephone handset, and is the official sign of a telco wiring guy.  It also was my passport into telephone wiring closets–when a security guard sees you have a butt set, they just assume you’re a real phone guy.

I practiced my technique in my father’s 1910-era house.  The decades brought with them layers of phone wiring, including two 50-pair feeder cables and a 66-style punch-down block.  Why and how the feeder cables ended up in a house with 3 phone lines is a mystery to this day.  But I used the 66-block for practice and dissected the criss-crossing wires in his house, tracing them out with my trusty tone/probe set.

Wiring skills came in handy many times in my career as a network engineer.  I remember one customer I had in the late nineties who had ordered 8 Centrex lines from Pacific Bell.  The technician showed up to do the cross-connects, but he was not the sharpest knife in the drawer, did two of them, and left.  Using my butt set passport, I got access to the MDF, figured out the right punch-down block that fed to the customer’s suite, and ran the cross-connects myself.

When I worked for the San Francisco Chronicle, we actually had an in-house wiring tech.  Her name was Mona Lesa.  She couldn’t care less if I did my own wiring, as long as I adhered to her standards.  One day I had a frame relay T1 installed in our Sacramento bureau office, and once again, Pac Bell forgot to connect it to the suite.  The bureau was located in the Old Senator Office Building across from the state capitol.  I got the security guard to let me in to the MDF in the bowels of this historic building.  I figured out that one of the interconnects was in the office of a lobbying firm, and the receptionist dutifully let me in to do the wiring.  With my butt set I could clip into any of their phone lines and listen to their conversations without them knowing.  I didn’t but I was amazed that I could go anywhere to do cross-connects.

On another occasion, a customer of mine had two phone lines installed and had a weird problem.  If she picked up one line and dialed the other, she’d hear both ringing and a busy signal at the same time.  I found where the Pacific Bell guy had done the cross-connect, and realized he mixed the tip and ring wires for the two phone lines.  A couple punches and all fixed.

Phone wiring was always the perfect blend of mind and body to me.  To do it you need to work with your hands, but you also need to use your mind.  Some wiring closets were rats nests of 24-gauge wires color-coded with more colors than a tie-died shirt.  Finding that one pair you needed, and marrying it up to the other end was always a nice diversion from staring at a screen.

Alas, my tools mostly sit unused.  Regular POTS lines rarely exist now, and even they aren’t delivered on single copper pairs from the CO switch.  PBXs and key systems have gone the way of the dinosaur, replaced by voice-over-IP and now by cellular phones.  Nobody wants to be tied to wires anymore, and yet by un-encumbering ourselves in the name of freedom, we’ve paradoxically lost our freedom.  When you don’t need to be physically tied to a location for connectivity, you can never escape connectivity.  And that’s not entirely a good thing.

3
by

Painted into a corner

I’ve written before about my years at the San Francisco Chronicle, my first job which was exclusively network engineering.  It was an interesting environment, as this was back in the years before the Internet totally displaced newspapers.  We had printing plants to support, active newsrooms with reporters and photographers, and a massive circulation operation.

When I first arrived, our Internet was provided by a T1.  At the time, the 1.5 Mbps was considered pretty fast, but with over 1000 users depending on the T1, it was slowing to a crawl.  The T1 was terminated on a good old-fashioned Cisco 2500-series router.  Later I upgraded our service to a T3 and terminated it on a 7204VXR, which led to a dramatic speed improvement.  The 2500 sat in an open, free-standing rack in the corner of our data center.

One day I noticed that the Internet was down.  Even though this was the early 2000’s, Internet connectivity was already critical, especially at a newspaper.  The problem quickly escalated to the CIO and I scrambled into action.  I could reach as far as the firewall, but there was no connectivity beyond that.

Sometimes the best thing to do is to physically check on a problem.  The networking team was in the basement and our data center was on the second floor.  I sprinted up the stairs and badged in to the data center.  One of our mainframe operators, who worked in the data center grabbed me and asked “hey, do you know the Internet is…?”

“Yeah, yeah,” I said and ran to the rack with our T1 router and DMZ switches.

I saw a gentleman in white painters clothes with a paint roller in the corner.  He was painting the wall right along side the rack.  There was only a few inches of clearance between the rack and the wall where he was rolling paint.  On the side of the rack, held in place with plastic zip ties, was a power strip with all the rack’s hardware plugged in.  He’d hit it with the roller and knocked out the power.

Network engineers love to solve complex problems, but when people are yelling at you it’s a relief when the problem is simple.  I flipped the switch on the power strip and Internet was restored as soon as the router booted up.  Someone had obviously placed the power strip on the wall-side of the rack figuring it would minimize the chance of it getting bumped.  They had inadvertently created the problem they were trying to solve.  I told the painter to stay away from my rack.

Networks are complex entities, but often the problems we face involve bad splices in holes in the ground, physical obstructions to WiFi signals, loose connections, and rogue paint rollers.

 

3
by

One big /8

I thought I’d take  break from Cisco Live to relive some memories in another Netstalgia.

Working in product management at a Cisco business unit, we are constantly talking about the latest and greatest, the cutting edge of technology.  It’s easy to forget how many customers out there are nowhere near the cutting edge.  They’re not near any edge at all.  When I worked at a Gold partner, I got to see all sorts of customer networks in a variety of states.

I remember one customer who ended up spending a lot of money with my company.  They were (and are) a major grocery chain in the United States.  They had reached a point when their network was grinding to a halt, and they needed help.  They had two overworked network engineers running things, and I remember being amused that their company policy required them to wear ties in the office.  This was not a financial company in San Francisco, but a discount grocery chain in a very relaxed part of the East Bay.  Anyways, they had hosts dropping off the network, performance problems, and their mainframe kept losing contact with its default gateway.

Walking in to these kinds of situations, you’re never sure what you might find.  Oftentimes the problems aren’t clear and the customer gets frustrated with their hired gun.  In this case, the very affable in-house engineers were thrilled to have experienced help.  They explained to me that the entire network, a large corporate office and countless stores, were on a single /8 network.  Only the on-site data center had a separate subnet.  Even the remote sites were in the /8!!

It got worse.  The stores were connected to HQ with IPSec VPN, but the hardware VPN devices they were made by a company that no longer existed.  The devices kept failing, and one of the network engineers had a stock of them he had purchased on eBay.  He amazingly was using his electronics skills to perform component-level repairs on the devices, cannibalizing parts from the eBay stash, which enabled him to stretch the stash longer than if he had simply swapped them.

My favorite was the data center.  The mainframe was sending constant pings to its default gateway, which would occasionally drop packets, in which case the mainframe would declare the gateway dead.  I found out that the default gateway was none other than a 2500-series router.

An old 2503 router

Even in 2009, this router was ancient history.  It had an old AUI connector on it which was nearly falling out.  In their 100 Mbps environment, they were limited to 10 Mbps.  I seem to recall it was doing router-on-a-stick on that interface, hairpinning traffic, but I don’t think the 2500 could do subinterfaces, so I may be wrong.  Anyways, the poor little 2500 was being slammed by traffic and dropping packets from time-to-time.

The ubiquitous CentreCOM AUI 10Base-T transceiver

I spent months at the client.  We designed a subnet scheme, renumbered their network, installed ASAs for IPSec, cut over all the stores, and put some high-end switches in the data center.  They were a grateful client, never complained, and I was able to make a genuine improvement in the lives of their users.  Unlike that other client I wrote about before.

I have a lot of bad memories of working for that partner, but one of the most interesting things was walking into so many different customers’ worlds, seeing what they dealt with every day, the mistakes they had made in building their networks, and helping them out by fixing those mistakes.

1
by

How not to do Internet Connectivity

My first IT job was at a small company in Novato, California, that designed and built museum exhibits.  At the time most companies either designed the exhibits or built them, but ours was the only one that did both.  You could separate the services, and just do one or the other, but our end-to-end model was the best offering because the fabricators and designers were in the same building and could collaborate easily.  The odd thing about separating the functions was that we could lose a bid to design a project, but win the bid to build it, and hence end up having to work closely with a competitor to deliver their vision.

A museum exhibit we designed and built

The company was small–only 60 employees.  Half of them were fabricators who did not have computers, whereas the other half were designers and office staff who did.  My original job was to be a “gopher” (or go-fer), who goes for stuff.  If someone needed paint, screws, a nail gun, fumigation of a stuffed tiger, whatever, I’d get in the truck and take care of it.  However, they quickly realized I was skilled with computers and they asked me to take over as their IT guy.  (Note to newbies:  When this happens, especially at a small company, people often don’t forget you had the old job.  One day I might be fixing a computer, then the next day I’d be hauling the stuffed tiger.)

This was in the mid-1990’s, so let me give you an idea of how Internet connectivity worked:  it didn’t.  We had none when I started.  We had a company-internal network using LocalTalk (which I described in a previous post), so users could share files, but they had no way to access the Internet at all.  We had an internal-only email system called SnapMail, but it had no ability to do SNMP or connect beyond our little company.

The users started complaining about this, and I had to brainstorm what to do when we had virtually no operating budget at all.  I pulled out the yellow pages and looked under “I”, and found a local ISP.  I called them, and the told me I could use Frame Relay, a T1, or ISDN.  I had no idea what they were talking about.  The sales person faxed me a technical description of these technologies, and I still had no idea what they were talking about.  At this point I didn’t know the phone company could deliver anything other than, well, a phone line.  I wasn’t at the point where I needed to hear about framing formats and B8ZS line encoding.

We decided we could afford neither the ongoing expense, nor the hardware, so we came up with a really bad solution.  We ordered modems for three of the computers in the office:  the receptionist, the CEO, and the science researcher.  For those of you too young to remember, modems allow you to interface computers using an ordinary phone line.  We ordered a single phone line (all we could afford).  When one of them wanted to use the Internet, they would run around the office to check with the other two if the line was free.

A circa-1990’s Global Village modem

The reason we gave the receptionist a modem is amusing.  Our dial-up ISP allowed us to create public email addresses for all of our employees.  However, they all dumped into one mailbox.  The receptionist would dial in in the morning, download all the emails, and copy and paste them into the internal email system.  If somebody wanted to reply, the would send it to the receptionist via SnapMail and she would dial up, paste it into the administrator account, and send it.  Brilliant.

Needless to say, customer satisfaction was not high, even in those days.  Sick of trying to run IT with no money, I bailed for a computer consulting company in San Francisco and started installing the aforementioned T1s and ISDN lines for customers, with actual routers.

If ever you’re annoyed with slow Wi-Fi, be glad you aren’t living in the 1990’s.

by

Netstalgia: Bad Timing

After I left TAC I worked for two years at a Gold Partner in San Francisco.  One of my first customers there was one of my most difficult, and it all came down to timing.

I was dispatched to perform a network assessment of a small real-estate SaaS company in the SF East Bay.  Having just spent two years in TAC, I had no idea how to perform a network assessment, and unfortunately nobody at the partner was helping me.  I had been told they had a dedicated laptop loaded with tools just for this purpose, but nobody could locate it.  I started downloading tools on my own, but I couldn’t find a good, free network analysis tool.  Another engineer recommended a product called “The Dude” from MikroTik, and since it was easy to install I decided to use it.  I needed to leave it collecting data for a few days, and since nobody had provided me an assessment laptop I had to leave my own computer there.  I distinctly remember the client asking me what tool I was using to collect data, and sheepishly answering “Uh, it’s called The Dude.”  He looked at me skeptically.  (Despite the name, the tool was actually quite decent.)

Without any format or examples for an assessment, I looked at bandwidth utilization, device security, IOS versions, and a host of other configuration items.  The network was very simple.  It was a small company with a handful of switches in their main office, and a T1 line connecting them to a satellite office in LA.  They used a Cisco VoIP system for phones, and the satellite phones connected over the T1 back to the main campus.  I wrote up a document with recommendations and presented it to the customer.  Almost everything was minor, and they agreed to have me come back in and make a few upgrades.

One item I noted in the assessment was that the clocks on the routers and switches were set incorrectly.  The clock has absolutely nothing to do with the operation of the device, but having just come from TAC I knew how important device clocks are.  If there is a network-wide incident, one of the first things we look at is the logging messages across the network, and without an accurate device clock we cannot properly compare log messages across multiple network devices.  We need to know if the log message on this router happened at the same time as that other log message on that switch, but if the clocks are set to some random time, it is  difficult or impossible.

I proceeded to make my changes, including synchronizing the clocks to NTP and doing a few IOS upgrades.  Then I closed out the work order and moved on to other clients.  I thought I was done, but I sure wasn’t.

We started getting calls from the customer complaining that the T1 between the two offices wasn’t working right.  I came over and found that it had come back up, but soon they were calling again.  And again.  I had used up all the hours on our contract, and my VP was not keen on me providing services for free.  But our client insisted the problem began as a result of my work, and I had to fix it.  Nothing I had done was major, but I went back and reverted to the old IOS (in case of a bug) and reverted to saved configs (which I had kept.)

With the changes rolled back, the problem kept happening.  The LA office was not only losing its Internet connectivity, but was dealing with repeated voice outages.  The temperature of the client was hot.  I opened a TAC case and had an RMA sent out for the WIC, the card that the T1 connected to.  I replaced it and the problem persisted.  At this point I was insisting it could not have been my fault, since I rolled back the changes, but the customer didn’t see it that way and I don’t blame them.

The customer called up their SBC (now ATT) rep, basically a sales person, to complain as well.  He told her a consultant had been working on his network, and she asked what had been changed.  He said “the clock on the router” and she immediately flagged that as the problem.  Sadly, the rep mistook the router clock, which has no effect on operations, with the T1 clocking, which does.  I never touched the T1 clocking.  I knew the sales rep as she had been my sales rep years before at the San Francisco Chronicle, and I knew she was a non-technical sales person who had no idea what she was talking about.  Alas, she had planted the seeds in the customer’s mind that I had messed everything up by touching the clock.  I pleaded my two CCIE’s and two years of TAC experience to try to persuade this customer that the router clock has zero, nada, zilch to do with the T1, to no avail.

The customer then, being sick of us, hired another consultant who got on the phone with SBC.  It turns out there was an issue with the line encoding on the T1, which SBC fixed and the problem went away.  The new consultant looked like a hero, and the next we heard from the client was a letter from a lawyer.  They were demanding their money back.

It’s funny, I’ve never really had another charge of technical incompetence leveled at me.  In this case I hadn’t done anything wrong at all, but the telco messed up the T1 line around the same time as I made my changes.  So I guess in more than one way, you could say it was a matter of bad timing.

1
by

Y2K

In 1998 I left my job as a computer “consultant” to pursue a master’s degree in Telecommunications Management.  I was stuck in my job, tired of troubleshooting people’s email clients and installing Word on their desktops, and was looking for a way to make a leap into bigger and better things.  That did happen–although not how I expected–but meanwhile for two years I needed to support myself while achieving my degree.  I took the easy path and stole a client from my previous employer.  This was at the height of the dotcom boom, and he was frankly too busy to even notice.  For a couple years I worked part-time at my advertising agency client, setting up computers, managing the servers, running the fairly simple network they had implemented.  It was a good deal for both of us, as the office manager had responsibility for IT and little inclination to work on technology.

Anyone who was around back then will remember the “Y2K” scare.  As we approached the year 2000, someone realized that many computers had been storing the year part of dates with only the last two digits.  For example, a program would store “98” instead of “1998.”  This meant that, as we moved into the new millennium, the year 2001 would be interpreted by these systems to be “1901”.  A legitimate problem, to be sure.  Some software, such as banking programs running on mainframes, could act unexpectedly and potentially lead to serious consequences.

However, the panic that followed was completely out of proportion to the threat.  Instead of focusing on the limited systems that would likely experience problems, a paranoia built up fueled by newspeople who had no idea what they were talking about.  People became concerned that, on January 1, 2000, our entire society would melt down and we would descend into chaos as the financial system melted down, stoplights stopped working, water and power systems crashed, and millions prepared to regress to living like cavemen.  A brilliant ad from Nike from just before the millennium captured the overall belief of what would happen come New Year’s Day.  (Sadly, the ad looks more realistic for 2020 than it did for 2000).

The ad agency I was working was told by their parent company that every single piece of electronic equipment they had needed to be Y2K-certified.  The downside of capitalism is that armies of opportunists arise (sound familiar?) to take advantage of social panics like these.  In fact, for such opportunists, exacerbating the panic is in their best interests.  Thus were spawned legions of “Y2K consultants”, non-technical business-types telling us that even our smoke alarms needed to be Y2K-certified and receive a literal sticker of approval.

When my office manager boss told me I needed to certify every piece of equipment I controlled my response was as follows:  “It doesn’t matter whether a network switch has the Y2K problem or not.  Most of the network gear was built recently and doesn’t even have this issue.  But if the vendor did make the mistake of storing the date with two digits instead of four, the worst that will happen is the time stamp on the log will be off.  Everything will still work.  So, rather than bill out hours and hours of me doing the certification, I’m going to do nothing.  And if I’m wrong, you can personally sue me for negligence.”

My boss didn’t care much about corporate HQ and took my advice.  New Year’s Eve, 2000, came and went.  A lot of people, expecting the worst, stayed home.  I went out and partied, well…like it was 1999.  And nothing happened.  Of course, the Y2k consultants patted themselves on the back for a job well done.  If the army of MBA’s hadn’t saved the day with their stickers, life would have ground to a halt.  I, on the other hand, knew the reality.  A panic was whipped up by the media, a bunch of opportunists swooped in to make a buck of the crisis, and helped to whip it up further, and life would have gone on either way.