When I worked for the Gold partner I generally serviced clients in the San Francisco Bay Area, but because we were a national partner I was occasionally called to other locations around the country. Being a double CCIE who had worked in TAC, I had a unique skill set among our engineers, which was often demanded by other field offices.
One day my boss called me and told me he needed my help with a customer out of Des Plaines, Illinois. The company was a manufacturer of fuses. They were experiencing a network meltdown and needed troubleshooting help. Great, I thought, I left TAC and came here precisely to get out of this sort of thing. I liked doing sales calls and new installations, not fixing buggy messes.
I was assigned to the customer on a Monday and was immediately pulled into what we often call a “shit show”. (Pardon the language.) The customer had a large international MPLS network with VPN backup. Several of the sites were experiencing performance issues. Sites were unable to perform manufacturing and the previous CIO had been fired. The interim CIO was an ex-military person who seemed to think he was George S. Patton. He was scheduling calls from early in the morning until late at night, status updates, live troubleshooting sessions, and pow wows with TAC.
Meanwhile, I was starting to feel ill. Not because of the case, just sick to my stomach. I didn’t think much of it at first, but it started to go downhill fast. Luckily I was working from home due to the crazy hours.
But not for long. The CIO had set up a troubleshooting session in the middle of the night Saturday, into Sunday morning. He got my boss and me on the phone and insisted I come to Des Plains that weekend, in person. We argued every way we could that I could be just as productive remotely, but Patton was having none of it. “If this is your best guy,” he said to my boss, “you need to have him on a plane and out here in person. Otherwise we can take our business somewhere else.” Not only was it the weekend, and not only did I feel ill, it was also Memorial Day weekend. My brother, who actually was (and is) in the Army was paying a rare visit to the Bay Area. There was no sympathy from the customer, and soon I was booking my ticket to Illinois. The local account executive booked me a car with GPS and promised to meet me at the airport. Keep in mind, this was before smartphones, and so you needed to rent a car with a built-in GPS unit if you wanted to get around without maps.
I had a miserable flight and was starting to feel more sick. There’s nothing worse than being sick to your stomach on a plane. Being forced to stay in your seat and long lines for tiny bathrooms make for torture. I ate nothing, and arrived at Chicago airport late. The account executive was nowhere to be found, and the car he had arranged did not have GPS. The rental car company didn’t have any GPS-equipped cars, so they provided me with a map and directions.
I drove through Chicago to Des Plaines. Realizing I needed to eat something, I found a McDonalds, the only thing open at that time, and managed to choke down a half of a quarter pounder. My stomach felt like burning acid. I continued my drive, through a bad part of the city. I was on the right road, but I needed to keep pulling over to check the address. A couple times I pulled over, swarms of what I assume were drug dealers would approach. I’d pull out just as they got to the car.
Eventually I made it to the customer site, and met the general. I was shown in to a conference room with a raised floor right next to the data center, and we began troubleshooting.
It was nothing I couldn’t have aided with remotely. Basically, the customer had scoped circuits that were too small for the volume of traffic they were carrying. They were also having degradation problems on the MPLS. Some of the sites were performing better on VPN backup circuits, so we were switching them to backups. We performed tests with the telco. We also looked into an issue with their core Catalyst 6k switch. When they had done a circuit switch earlier in the week, all traffic on the network had stopped, according to the customer. The customer had reloaded the core device and traffic came back. Because there was no crash or crashdump file, and nothing in the logs, I could not explain this event. It was a smorgasbord of issues, mostly due to bad design and a little due to bad luck.
The troubleshooting window was supposed to end at 2am, but we worked until 6am. I had a flight to catch and hadn’t slept all night. The customer wanted me to stick around but I told him to stuff it and left. I checked in to the hotel room I had booked, slept one hour, checked out, and got on my plane.
On the flight back, I was seated in the middle seat between a two very large people. I figured out that they were married, but they only spoke Spanish. I used my rudimentary Spanish to extract myself. “Yo, a la ventana. Su esposa, aquí,” I suggested. They liked the idea. I moved to the window seat. When the plane took off, they spread out a massive feast on the tray tables. I ate a single muffin from Starbucks, one bite at a time, until I landed and went home.
I never determined if it was a norovirus or food poisoning, but I lost twenty pounds in a week. The customer realized they needed to invest in new circuits, which had a 12-16 week turnaround time. I think the new CIO got fired as well. And the account executive never invited me back. I only wish he had shown up at the airport as I might have thrown up on him.