interviewing

All posts tagged interviewing

In the last article on technical interviewing, I told the story of how I got my first networking job.  The interview was chaotic and unorganized, and resulted in me getting the job and being quite successful.  In this post, I’d like to start with a very basic question:  Why is it that we interview job candidates in first place?

This may seem like an obvious question, but if you think about it face-to-face interviewing is not necessarily the best way to assess a candidate for a networking position.  To evaluate their technical credentials, why don’t we administer a test? Or, force network engineering candidates to configure a small network? (Some places do!)  What exactly is it that we hope to achieve by sitting down for an hour and talking to this person face-to-face?

Interviewing is fundamentally a subjective process.  Even when an interviewer attempts to bring objectivity to the interview by, say, asking right/wrong questions, interviews are just not structured as objective tests.  The interviewer feedback is usually derived from gut reactions and feelings as much as it is from any objective criteria.  The interviewer has a narrow window into the candidate’s personality and achievements, and frequently an interviewer will make an incorrect assessment in either direction:

  • By turning down a candidate who is qualified for the job.  When I worked at TAC, I remember declining a candidate who didn’t answer some questions about OSPF correctly.  Because he was a friend of a TAC engineer, he got a second chance and did better in his second interview.  He got hired and was quite successful.
  • By hiring a candidate who is unqualified for the job.  This happens all the time.  We pass people through interviews who end up being terrible at the job.  Sometimes we just assess their personality wrong and they end up being complete jerks.  Sometimes, they knew enough technical material to skate through the interview.

Having interviewed hundreds of people in my career, I think I’m a very good judge of people.  I was on the interview team for TAC, and everyone we hired was a successful engineer.  Every TME I’ve hired as a manager has been top notch.  That said, it’s tricky to assess someone in such a short amount of time. As the interviewee, you need to remember that you only have an hour or so to convince this person you are any good, and one misplaced comment could torpedo you unfairly.

I remember when I interviewed for the TME job here at Cisco.  I did really well, and had one final interview with the SVP at the time.  He was very personable, and I felt at ease with him.  He asked me for my proudest accomplishment in my career.  I mentioned how I had hated TAC when I started, but I managed to persevere and left TAC well respected and successful.  He looked at my quizzically.  I realized it was a stupid answer.  I was interviewing for a director-level position.  He wanted to hear some initiative and drive, not that I stuck it out at a crappy job.  I should have told him about how I started the Juniper on Juniper project, for example.  Luckily I got through but that one answer gave him an impression that took me down a bit.

When you are interviewing, you really need to think about the impression you create.  You need empathy.  You need to feel how your interviewer feels, or at least be self-aware enough to know the impression you are creating.  That’s because this is a subjective process.

I remember a couple of years back I was interviewing a candidate for an open position.I asked him why he was interested in the job. The candidate proceeded to give me a depressing account of how bad things were in his current job.”It’s miserable here,” he said.  “Nobody’s going anywhere in his job.  I don’t like the team they’re not motivated.”  And so forth.  He claimed he had programming capabilities and so I asked him what his favorite programming language.”I hate them all,” he said. I actually think that he was technically fairly competent but in my opinion working with this guy would’ve been such a downer that I didn’t hire him.

In my next article I’ll take a look at different things hiring managers and interviewers are looking for in a candidate, and how they assess them in an interview.

 

I’ve wanted to kick off a series for a while now on technical interviewing. Let me begin with a story.

My first job interview for a full network engineering role was at the San Francisco Chronicle in 2000. I had been working for five years in IT, mostly doing desktop and end-user support. I then decided to get a master’s degree in telecommunications management, which didn’t help at all, followed by a CCNA certification, which got me the interview.

My first interview was with the man who would be my boss. Henry was a manager who had almost no technical knowledge about networking, but I didn’t know that at the time. “Do you know Foundry switches at all?” Henry asked.

“No.” I was already worried.

“I doubted you would. That’s ok because we want to replace them all with Cisco and you know Cisco.” He pulled out a network diagram and handed it to me. “If you look at this, do you see a problem?” he asked.

I had never worked on a network larger than a couple switches, and now I was staring at a convoluted diagram depicting the network of the largest newspaper in Northern California. I was looking at subnet masks, link speeds, and hostnames, trying to find something wrong.

“I’m not sure,” I had to reply meekly.

He pointed at the main core switch for the network. There was only one, with no redundancy.  “There’s a huge single point of failure,” he said. I felt stupid missing the forest for the trees.

Henry brought me upstairs to interview with Tom, who was an on-site project management contractor from Lucent. I was extremely nervous–Lucent (later Avaya) was a big name in the industry and this guy worked for them! Henry left me with Tom. Tom pulled out a copy of the same diagram Henry was showing me earlier.

“Do you notice anything wrong with this?” he asked.

“Wow, that’s a huge single point of failure,” I replied.

He nodded his head in approval. “That’s right–very good.” He asked me a technical question about supernetting. I answered nervously, although it quickly became clear I knew more than he did.

The door flew open and another guy named Vincent walked in. He was the desktop support contractor, but again I didn’t know that. “Ask Jeff a few technical questions,” Tom said.

“Question number one,” said Vincent. “If you were running a network this size, would you subnet it?”

Now the answer seemed obviously to be “yes”, but I was trying to figure out if this is a trick. “Yes,” I answered, deciding to play it safe.

“Good! Next question: Can you route NetBios?” My desktop years were almost exclusively dedicated to Macs and I didn’t even know what NetBios was. I figured it was a 50/50 shot, and the way he asked it seemed to suggest the answer.

“No,” I said, trying to sound confident.

“He’s good,” said Vincent.

Next, the door flew open again, and in walked Bing. Bing was carrying some sort of network device with her. She handed it to me. “Is this a switch or a hub?” she asked. There was no obvious labeling on it, and as I turned the device over and over again in my hands, I had a sinking feeling.

“I don’t know,” I replied.

“Look at this,” Bing said. She pointed to a collision light. “Since there is one on each port, you can tell this is a switch.”

We don’t have collision lights on switches any more, but at the time we did and she had a valid point. Realizing this, I explained to her that a since a hub has a single collision domain, it would only have one collision light. I explained to her the concept of a collision domain, and how a switch worked versus a hub. It turns out she was a project manager for desktop support and she didn’t know any of that. Someone had just shown her the collision light thing and she thought it would be a good question.

“He’s good,” said Bing.

Tom had told me my next interview would be with a CCIE from Lucent. Now that was definitely intimidating. I knew of the reputation of CCIEs, and I didn’t expect to do well. The CCIE guy never showed up. As Tom was walking me to the elevator, however, we ran into him in the hallway. It turns out that Mike, who is still a friend of mine, and who later got three CCIE’s, had not passed the exam yet. We ended up talking about his home lab for a few minutes.

“He’s good,” said Mike. And a good thing too, as I’ve been in a couple of interviews with Mike and I’ve seen him grill people mercilessly.

I got a call with a job offer a few days later, and ended up working there five years.

For a while now I’ve had several posts in my drafts folder on the subject of technical interviewing. As you can see from the above story, interviews are often chaotic, disorganized, and conducted by unqualified people who have no plan. In the case of the San Francisco Chronicle, they made the right decision on me, and I don’t think anybody there would dispute that. I was thankful to begin my career in network engineering.

That said, I’ve had other interviews that didn’t go so well. Over the next few articles, I’d like to cover technical interviewing. Why do we interview people? How can we select good people from bad people? How worthwhile are the typical technical questions? Are gotchas worth throwing out “just to see how the candidate reacts”? Are interviews purely subjective or can we make them data-driven and objective?

I’ll throw out a few more anecdotes from my own experience to illustrate my points–feel free to comment with some of your own!

I worked for two years at a Cisco Gold Partner.  The first year was great.  We were trying to start up a Cisco practice in San Francisco (they were primarily a Citrix partner before), so my buddy and I wined and dined Cisco channel account managers trying to impress them with our CCIE’s and get them to steer business our way.  Eventually, the 2009 financial crisis hit and business started to dry up.  The jobs became fewer and less interesting.  I had two CCIE’s and at one point, I drove out to Mare Island near San Francisco to install a single switch for a customer whose entire network consisted of–a single switch.  I always recommend people not to stay in jobs like this too long, as it hurts your prospects for future employment.

Potential Employer:  “So what kind of jobs have you done lately?”

You:  “Uh, I installed one switch at a customer.”

Anyhow, we had one other customer that managed to keep me surprisingly busy, considering their network was quite small as well.  They were a local builder, and with three small offices connected together with ASAs and VPN tunnels.  The owner was filthy rich and also paranoid about security, which meant I was out there a lot changing passwords, tightening up ACLs, and cleaning up the mess the last network engineer had left.

The owner had a ranch near Wilits, CA which was reputed to be the size of the city of Concord, CA.  He also had two jets to take him to his private landing strip at his ranch.  Being a pilot myself, the prospect of a trip in a small jet to his ranch made me wish for some sort of network problems up there.  However, there wasn’t much up there for me to work on.  He had a single ASA 5505 connected to satellite uplink which he primarily used to connect to the cameras (which he had everywhere) at the ranch.

One day, my contact at the builder told me the cameras weren’t reachable.  Yes!  Finally a trip in the jet.  We set a date and I spent my time wondering whether I’d get the Lear or the Citation.

Unfortunately, when the day rolled around, the weather was hideous.  A Lear jet can handle most any weather, but the little airstrip had no instrument approaches.  Instead, my contact gave me an alternative:  I was to drive up there with her in-house cabling contractor (I’ll call him “Tim”) to do the job.  (I never understood why a business this small had an in-house cabling contractor.  As far as I knew he didn’t work on the actual construction projects associated with the company.)  Now from San Francisco, the drive to Willits is about 2.5 hours.  However, the ranch was near Willits.  After driving 2.5 hours to Willits, we had another hour drive over dirt roads to the middle of nowhere.

The cabling contractor was exactly the sort of person with whom I have nothing in common, and spending 3.5 hours in a car with him, in the era before smartphones are a handy distraction, was painful.  Tim loved fishtailing his truck as we drove on dirt roads on the side of a mountain.  I think he also liked just scaring the white collar guy.  It worked.

We arrived at the ranch and Tim opened up the back of his pickup.  “Can you give me a hand here?” he asked.  In the bed of his truck were several large carpet rolls and piles of dry cleaning.  I grabbed one end of a carpet roll and began the backbreaking work.  My company was billing me out at $250/hour to haul some lady’s dry-cleaning into her ranch.

The ASA itself was located in a pole in the middle of the property, which had a satellite dish on top.  I was amazed the ASA 5505 even functioned out there, given that the external temperature could reach over 100 degrees Fahrenheit.  The metal box housing the ASA was like an oven.  I consoled into it and immediately saw a problem.  Latency on the link was over one second round-trip.  There was no way he was going to get real-time video streaming with this slow satellite uplink.  I reported my findings to Tim and, after eating lunch with the ranch hands, we hopped back in the truck.  Tim put on a song called “You piss me off, f*cking jerk” while we drove.  I guess he didn’t like me.

When I mentor people, I often tell them you have to know the right time to quit a job.  There were several signs in this story that it was time for a change.  With two CCIEs, installing a single switch or working on a single ASA 5505 was not really a good use of my skills.  Neither was moving in carpet rolls and dresses for $250/hour.  Luckily I had enough big jobs at the partner that I managed to get through my interviews at Juniper without trouble.

Meanwhile, a few years later I read about the FBI raiding the builder who was my customer.  I guess he had good reasons for cameras.

 

In this article in my “Ten Years a CCIE” series, I describe my experience going to work at Cisco as a CCIE.  Unlike many Cisco-employed CCIE’s, I earned my certification outside of Cisco.

A CCIE leads to a job at Cisco

I returned to my old job at the Chronicle and had my business cards reprinted with my CCIE number. I loved handing it out, particularly at meetings with telephone companies and Internet service providers whose salespeople were likely to know what such a certification meant.  I remember one such sales person, duly impressed, saying “wow, on that test you can be forced to configure any feature on any Cisco product…I don’t know how anyone could prepare for that!”  (Uh, right.  See “The CCIE Mystique“.)

At that time, the most popular forum for aspiring CCIE’s was an email distribution list called groupstudy.com. I had been a subscriber to this mailing list, but prior to passing my exam, I didn’t feel adequate to post anything there. However, once I passed, I began posting regularly, beginning with a summary of my test preparation process. One day I got an email from a mysterious CCIE who told me that I sounded like I knew what I was talking about, asking me if I wanted to interview for a job. I thought his name and number sounded familiar, and when I got home I confirm my suspicions by digging through my bookshelf. He was the author of one of my books about Catalyst Quality of Service.

A brutal interview

It turns out he was a manager at Cisco High Touch Technical Support, a group of TAC engineers who specialized in high profile customers. I scheduled an interview right away.

This interview was by far the most difficult I’ve had in my career. They brought me into a room with four CCIE’s, two of them double, all of them sharp. Each one of them had a different specialty. One of them was a security guy, another one was an expert on multicast, another was an expert on switching. When it came to Kumar, the voice guy, I figured I was scot-free. After all, I didn’t claim to know anything about voice over IP. Kumar looked over my resume, and then he looked up at me. “I see you have ISDN on your resume,” Kumar said. And then he began to grill me on ISDN. Darn, I should have thought of that!  Thankfully, I was well prepared.

Despite one or two mistakes in the interview, I got hired on and began my new job as a customer support engineer at HTTS. My first few months were in a group called ESO, which supported large enterprises and was very focused on Catalyst switching.  I won’t go into the details of the job here, but you can see my many TAC tales if you are interested.

Cisco's San Jose campus

Cisco’s San Jose campus

Little purple stickers everywhere!

One thing I quickly noticed when I got to Cisco was that a lot of the people in my department had a nickel-sized purple dot on their ID badges and cubicle nameplates. I found out that these purple dots were actually stickers with the CCIE logo. Cisco employees who had their CCIE’s stuck these purple dots on their badges and nameplates to show it off. Many of the CCIE’s who had passed multiple exams actually placed multiple dots on their badges and nameplates. I wanted one quite badly. The problem was, sheets of these purple stickers were sent out only to the early CCIE’s, and by the time I had passed, Cisco was no longer providing the sheets of stickers. I suppose I could’ve had some printed out, but I asked around looking for a CCIE who was generous to give me one of his dots. They were in scarce supply, however, and nobody was willing to part with one. It was just another way newer CCIE’s were getting jipped.

The real CCIE logo

The real CCIE logo

Even though the exam had now switched to the one day format, you still didn’t meet too many CCIE’s outside of Cisco. It was thus quite a shock when I went to Cisco and saw purple dots everywhere. It seemed like fully half of the people I was working with in my new job had CCIE’s. And many of them had low numbers, in the 2000’s and even in the 1000’s. I was quite relieved to find that they all treated me with total respect; nobody ever challenged me on account of my one-day CCIE. Still, I always had (and always will have) a great deal of respect for those people who passed the test when it was a two day test, and the cottage industry devoted to minting CCIE’s had not yet come into existence.

CCIE challenges customer

Customers were another story. I remember one BGP case in particular. I looked at the customer’s configuration and immediately realized that it was a simple matter of misconfiguration. I fired it up in my lab reproduced his configuration and proved to him that it was indeed a configuration error on his part. I wrote it all up in an email and proudly signed it with my CCIE number. Within a half an hour I got a call from the customer and one of his colleagues on the line. They proceeded to grill me rapidly on BGP asking me all sorts of questions that weren’t relevant to their case and stumping me several times. At that point I realized that when many people see you are a CCIE, they take it as a challenge. In some cases they failed the test themselves, or else they’ve met stupid CCIE’s in the past and they feel themselves to be on a mission to discredit all CCIE’s. After that episode, I removed my CCIE number from my email signature. I gained a feeling of self-importance after I passed my exam, but working among so many people with the same certification, and dealing with such intelligent customers, I realized that the CCIE didn’t always carry the prestige I thought it did.  The mystique diminished even further.

Incidentally, I became friends with all of the guys who interviewed me, and I was on the interview team myself during my tenure at Cisco. One extremely sharp CCIE we hired told me our interview was so tough he had to “hit the bottle” afterwards. It was considered a rite of passage at TAC to go through a tough interview, but I have gotten a lot nicer in my interview style now, having been on the receiving end of a few grillings.

The value of a CCIE

One of the later posts in this series will examine the question of the value of a CCIE certification.  After all, this is one of the most common questions I see in forums dedicated to certification.  However, my experience getting hired into Cisco (the first time) has some lessons.

  • The immediate reason I got hired was because of my experience and willingness to go out of my way helping others to get their CCIE on Groupstudy.  However, I would not have gotten the position without a CCIE, so clearly it proved its value there.
  • Once you are at Cisco, although people commonly display their stickers and plaques, having a CCIE certification will not necessarily distinguish you.
  • There are many CCIE’s who have made a bad impression on others, whether they are only book-knowledgeable, or even cheaters.  Often people challenge you when you have a CCIE, instead of respecting you.

In the next article in the series, Multiple CCIEs, Multiple Attempts, I describe passing the CCIE Security exam.  I talk about my experience suffering the agony of defeat for the first time, and how I eventually conquered that test.