Skip navigation

Category Archives: Uncategorized

Art of Network Engineering

I’ve been a little busy lately, and frankly sometimes topics to write about come fast and furiously, sometimes they don’t come at all.

Meanwhile, I had a chance to sit down with the team over at the Art of Network Engineering podcast.  I was really enjoying their interviews with engineers around the industry and I not so shamelessly asked if I could be on.  I had a great time telling my story, and bantering with two of the AONE team.  They do great work evangelizing the industry.

Unfortunately my high-end mic didn’t pick up for some reason and my audio was not so good.  But it’s not terrible and hopefully the content is better than the sound.

5

Interviewing #3: Technical Assessment

We all have weaknesses, and one of mine is that I’m good at starting things and bad at finishing them.  Two years ago (gasp) I had started writing a series about technical interviewing.  I wrote two posts (here and here) on the subject and never finished.  A recent commenter asked for me to keep writing on the subject, so here goes.

Whenever we, as hiring managers, post a job, we have a specific idea in mind of what the ideal candidate would look like.  We are looking for a combination of skills, personality, experience, and credentials.  The fact of the matter is, we can create an ideal candidate in our head, but that person does not exist.  We will never hire this fictional person, so we have to look for the closest match.  We need to decide which of these skills, credentials, etc., we are willing to compromise on, and which are non-negotiable.  Historically, the process for deciding the match used two pieces of input:  the resume (or CV) and the interview.  The resume provides a rough view of how close the candidate is to the ideal, whereas the interview allows the hiring team to explore more deeply the closeness of the match.  I said “historically” because I’ve seen some companies using newer techniques, such as administering programming tests.  I’ll leave these newer techniques aside to focus on the interview.

The interview is an attempt to evaluate the candidate’s “fit” by asking questions of the candidate.  The largest part of this effort is trying to understand if the candidate’s skills fit with the skills we believe are required for the job.  A secondary part of the interview is assessing the candidate’s personality and “soft” skills.

How do we assess technical skill?  Let’s say the job we are hiring for is a low-level network engineer.  We need someone who has basic skills in configuring Cisco switches, routers, and wireless.  We want someone with a basic understanding of TCP/IP, IOS-XE, NX-OS, routing protocols like EIGRP and OSPF.  Often all these will be listed at the top of the candidate’s resume.  But anyone can put anything in the alphabet soup.  The interview is our chance to determine whether the alphabet soup is true.  (Hint:  DO NOT put anything in your resume you are not comfortable talking about.)  There are generally two ways of doing this.  First, we can assess the candidate’s hands-on experience with these technologies.  We can use questions like:

  • Did you work with EIGRP in your last job?  If so what did you do with it?
  • How much hands-on experience do you have with NXOS?  Tell me the environment where you configured it and what specifically you did.

In this case, I’m trying to understand whether you’ve worked with the technologies and how much you have worked with them.  You might claim experience with NXOS, but you only looked at “show interface” outputs in your last job.

The other way I can assess technical skill is by asking theoretical questions about the technologies in question.  For EIGRP, for example:

  • Can you tell me all the different components of the EIGRP metric and which are the defaults?
  • What is an EIGRP SIA condition and how would you go about resolving it?

Any skill is a combination of theory and practice.  If you don’t understand the theory behind a wing stall in an airplane, you won’t be able to resolve it, but if you don’t go out and do stall practice in the airplane, you also won’t be able to resolve it.  Most interviewers will use a combination of these techniques to assess your competency.

Another technique is to provide a theoretical scenario.  I can get up on the whiteboard and draw a bunch of OSPF areas with different types and ask you how routing would work between them.

As an interviewer, I now need to think about your answers and how they align with what I’m expecting.  What if you know a lot of EIGRP theory, but you’ve never configured it in the real world?  What if you know a lot about switching and routing but nothing about wireless?  And, most importantly, how do you compare to the other candidates I’ve interviewed?  The other strong candidate might know a ton about wireless and nothing about routing.  I also have to consider whether I want a “ready-made” engineer who can just walk in and start working, and how willing I am to provide ramp time to the hire.  If you know you’re stuff in routing and switching, but don’t understand wireless, I might assume your overall technical competency will enable you to learn wireless with relative ease.

How effective are these techniques?  Well, if I remember to continue with this series, I’ll discuss it in the next article.

3

Theme Changes

I started this blog in February 2013.  Amazingly, I’m closing in on ten years.  It’s certainly turned out differently than I expected, and I haven’t written as much as I’d liked.

In 2016, I hired a company to update the blog and design a new theme.  Maintaining your own WordPress blog is a lot of work, and it was worthwhile to have an expert fix things up.  I was fairly happy with the results, although there are a few glitches in the theme that annoy me.  The tile layout was never my favorite, and it gets really uneven depending on whether I choose a featured photo or not.

Anyways, with the WP updates over the years, my theme is breaking down.  On the editing page, many buttons don’t work at all.  I contacted the original company, which still exists, but I haven’t been able to get a response.

Over the next few weeks, I’m going to experiment with the built-in WP themes to see if there is one I like.  Expect the look and feel to change from time to time as I explore options.  Hopefully it won’t throw off the spambots that seem to be my primary readership!

Moscone Microwave

My first full-time networking job was at the San Francisco Chronicle.  Now there isn’t much to the Chronicle anymore, but in the early 2000’s the newspaper was still going strong.  It was the beginning of the decline, but most people still took their local newspaper as their primary source of news.  Being a network engineer at a major metropolitan newspaper was fascinating.  It is a massive operation to print and distribute a newspaper every single day, and you can never, ever, miss.  There is no slippage of production deadlines.  It has to be out every day, and every day you start all over, with a blank page.

As the lead network engineer, I touched everything from editorial (the news and photography content of the paper) to advertising, pre-press, production systems, and circulation.  Every one of these was critical.  If editorial content didn’t make it through, there was nothing to go into the paper.  If advertising didn’t make it in, we didn’t earn revenue.  If pre-press or production had problems, the paper wasn’t printed.  If circulation wasn’t working, nobody could get their paper.

The Chronicle owned and operated three printing plants in the Bay Area.  One was on Army Street in San Francisco, while the other two were in Union City and Richmond in the East Bay.  The main office was on Fifth and Mission in downtown SF, so the paper was prepared in San Francisco and then sent to the plants via microwave.  That’s where I came in.

Our microwave system used a dish on the clock tower of our building.  From 5th and Mission we sent a signal up to Roundtop Mountain in the East Bay hills. At Roundtop we leased space in a little concrete bunker that was used for various kinds of radio communication including cellular.  From Roundtop we bounced the signal back to the three printing plants.

Chronicle building with the microwave visible on the clock tower

The microwave presented itself to us as T1 lines.  I had the T1 lines connected to dual routers at the main site and each of the plants.  In addition to the microwave, we had two additional backup T1’s to each plant which were landlines from different carriers with diverse paths into the buildings.  We kept the microwave and the first T1 plugged into the routers, with the third one on manual standby in case we needed it.  You don’t take chances with production in a newspaper, and we had triple redundancy on everything.  I used OSPF for redundancy between the microwave and #1 backup circuit on the routers, and HSRP for gateway redundancy.  With only four sites it was a simple enough topology and it never gave me much trouble.

Until, that is, the day when I got a call from our operations center that the primary circuits were all down.  We were running on backups.  I immediately called up the production systems engineer who managed the microwave and told him his circuits were down.  “Impossible!” he said, “that microwave is five-nines reliable.  Check your router!”  I tried a few of the usual:  shut/no shut the interface, changing the line encoding, etc.  No go.  He wanted me to start swapping hardware, which was a big deal in a live newspaper environment, and seemed pointless.  If it was hardware, why would all of the circuits be down?

We bickered a bit before I moved to have the tertiary backup circuits swapped in so we had automatic failover while we worked on the microwave.  I got out our old T-berd tester to see if I could find any indication of the problem.  Then the systems engineer called:  “We need to meet at the clock tower, I’ve found the problem,” he said.  It’s always a relief to hear that when finger pointing is going around.

T-berd T1 Tester

I showed up at the entrance to the tower and followed the systems guy up a rusty ladder mounted to the wall.  Up in the tower there were bird droppings and as I climbed higher I fought the urge to look down.  I’ve never much liked heights and being out of shape and relying on my own strength to keep from falling several stories onto concrete was not promising.  Once I got to the top there was a large separation between the ladder and the floor, and I fought the urge to panic as I flung my leg way over to climb onto the concrete flooring.  From there we went outside and I saw the problem right away.

If you’ve ever been to a convention in San Francisco, chances are it took place in the Moscone Center.  In the early 2000’s, the city decided to expand Moscone by building a new Moscone Center West on 4th and Howard streets.  And from up on the clock tower it was plain as day:  they had built a cooling tower on the roof right in the path of our microwave beam.  I looked at the systems guy and said, “Well, I guess you could make popcorn in that cooling tower.  Anyways, there goes your five nines.”

We hastily called meetings together to decide what to do.  Sue the city?  Call the FCC?  Find another building to bounce the microwave off of?  Those were long term solutions but we had an immediate problem.  Two circuits might seem like enough, but they were telco circuits and not as reliable as the microwave was, at least when its path wasn’t blocked.

Getting the city to cut the cooling tower off Moscone West was a non-starter, especially when it was the newspaper asking, a newspaper that made its money being critical of city officials.  So, we decided to lease roof space from another building and add an additional repeater.  However, this was a long process.  We needed to negotiate with the landlord, replan the radio deployment, license it and obtain permits, add the new repeater, and re-point the old dish to the new building.  That last item was not as simple as it sounded, since this wasn’t a DirecTV dish.  It was welded to the tower, so we needed to hire ironworkers to cut it off and re-position it.

Meantime, we ordered T1’s from downtown SF up to Roundtop to bypass the segment that wasn’t working.  We’d go hard wire to Roundtop, the microwave the rest of the way.  This was not, by any means, an ideal solution, nor was it an overnight solution, but we could at least get some redundancy faster than it would take to add the repeater.  I’m glad we did because shortly after the microwave went down we started having terrible problems with the landlines and needed the triple redundancy.

If you drive by Fifth and Mission now, the microwave dish is gone from the clock tower.  The Chronicle, a shadow of its former self, no longer operates its own printing plants, and has a circulation far smaller than it did in 2004, when I left.  As I said in my last post, it’s great to have a sense of purpose when you work in IT.  It wasn’t about fixing a microwave but about getting that paper in the hands of our readers.  I’m thankful I got to be a part of that for a few years, even if it cost me some vertigo and sleepless nights.

TAC Tales #13: All Zeros

A common approach for TAC engineers and customers working on a tough case is to just “throw hardware at it.”  Sometimes this can be laziness:  why troubleshoot a complex problem when you can send an RMA, swap out a line card, and hope it works?  Other times it’s a legitimate step in a complex process of elimination.  RMA the card and if the problem still happens, well, you’ve eliminated the card as one source of the problem.

Hence, it was not an uncommon event the day that I got a P1 case from a major service provider, requeued (reassigned) after multiple RMAs.   The customer had a 12000-series GSR, top of the line back then, and was frustrated because ISIS wasn’t working.

“We just upgraded the GRP to a PRP to speed the router up,” he said, “but now it’s taking 4 hours for ISIS to converge.  Why did we pay all this money on a new route processor when it just slowed our box way down?!”

The GSR router is a chassis-type router, with multiple line cards with ports of different types, a fabric interconnecting them, and a management module (route processor, or RP) acting as the brains of the device.  The original RP was called a GRP, but Cisco had released an improved version called the PRP.

The GSR 12000-series

The customer seemed to think the new PRP had performance issues, but this didn’t make sense.  Performance issues might cause some small delays or possibly packet loss for packets destined to the RP, but not delays of four hours.  Something else was amiss.  I asked the customer to send me the ISIS database, and it was full of LSPs like this:

#sh isis database

IS-IS Level-2 Link State Database
LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime
0651.8412.7001.00-00  0x00000000   0x0000        193               0/0/0

ISIS routers periodically send CSNPs, or Complete Sequence Number PDUs, which contain a list of all the link state packets (LSPs) in the router database.  In this case, the GSR was directly attached to a Juniper router which was its sole ISIS adjacency.  It was receiving the entire ISIS database from this router.  Normally an ISIS database entry looks like this:

#sh isis database

IS-IS Level-2 Link State Database
LSPID                 LSP Seq Num  LSP Checksum  LSP Holdtime
bb1-sjc.00-00         0x0000041E   0xF97D        65365             0/0/0

Note that instead of a router ID, we actually have a router name.  Note also that we have a sequence number and a checksum for each LSP.  As the previous output shows, something was wrong with the LSPs we were receiving.  Not only was the name not resolving, the sequence and checksum were zero.  How can we possibly have an LSP which has no sequence number at all?

Even weirder was that as I refreshed the ISIS outputs, the LSPs started resolving, suddenly popping up with names and non-zero sequences and checksums.  I stayed on the phone with the customer for several hours, before finally every LSP was resolved, and the customer had full reachability.  “Don’t do anything to the router until I get back to you,” I said before hanging up.  If only he had listened.

I was about to pack up for the day and I got called by our hotline.  The customer had called in and escalated to a P1 after reloading the router.  The entire link state database was zero’d out again, and the network was down.  He only had a short maintenance window in which to work, and now he had an outage.  It was 6pm.  I knew I wasn’t going home for a while.

Whatever was happening was well beyond my ISIS expertise.  Even in the routing protocols team, it was hard to find deep knowledge of ISIS.  I needed an expert, and Abe Martey, who sat across from me, literally wrote the book on ISIS.  The Cisco Press book, that is.  The only issue:  Abe had decided to take PTO that week.  Of course.  I pinged a protocols escalation engineer, one of our best BGP guys.  He didn’t want anything to do with it.  Finally I reached out to the duty manager and asked for help.  I also emailed our internal mailers for ISIS, but after 6pm I wasn’t too optimistic.

Why were we seeing what appeared to be invalid LSPs?  How could an LSP even have a zero checksum or sequence number?  Why did they seem to clear out, and why so slowly?  Did the upgrade to the PRP have anything to do with it?  Was it hardware?  A bug?  As a TAC engineer, you have to consider every single possibility, from A to Z.

The duty manager finally got Sanjeev, an “ISIS expert” from Australia on the call.  The customer may not realize this while a case is being handled, but if it’s complex and high priority, there is often a flurry of instant messaging going on behind the scenes.  We had a chat room up, and as the “expert” listened to the description of the problem and looked at the notes, he typed in the window:  “This is way over my head.”  Great, so much for expertise.  Our conversation was getting heated with the customer, as his frustration with the lack of progress escalated.  The so-called expert asked him to run a command, which another TAC engineer suggested.

“Fantastic,” said the customer, “Sanjeev wants us to run a command.  Sanjeev, tell us, why do you want to run this command?  What’s it going to do?”

“Uh, I’m not sure,” said Sanjeev, “I’ll have to get back to you on that.”

Not a good answer.

By 8:30 PM we also had a senior routing protocols engineer in the chat window.  He seemed to think it was a hardware issue and was scraping the error counters on the line cards. The dedicated Advanced Services NCE for the account also signed on and was looking at the errors. It’s a painful feeling knowing you and the customer are stranded, but we honestly had no idea what to do.  Because the other end of the problem was a Juniper router, JTAC came on board as well.  We may have been competitors, but we were professionals and put it aside to best help the customer.

Looking at the chat transcript, which I saved, is painful.  One person suggests physically cleaning the fiber connection.  Another thinks it’s memory corruption.  Another believes it is packet corruption.  We schedule a circuit test with the customer to look for transmission errors.

All the while, the 0x0000 LSPs are re-populating with legitimate information, until, by 9pm, the ISIS database was fully converged and routing was working again.  “This time,” I said, “DO NOT touch the router.”  The customer agreed.  I headed home at 9:12pm, secretly hoping they would reload the router so the case would get requeued to night shift and taken off my hands.

In the morning we got on our scheduled update call with the customer.  I was tired, and not happy to make the call.  We had gotten nowhere in the night, and had not gotten helpful responses to our emails.  I wasn’t sure what I was going to say.  I was surprised to hear the customer in a chipper mood.  “I’m happy to report Juniper has reproduced the problem in their lab and has identified the problem.”

There was a little bit of wounded pride knowing they found the fix before we did, but also a sense of relief to know I could close the case.

It turns out that the customer, around the same time they installed the PRP, had attempted to normalize the configs between the Juniper and Cisco devices.  They had mistakenly configured a timer called the “LSP pacing interval” on the Juniper side.  This controls the rate at which the Juniper box sends out LSPs.  They had thought they were configuring the same timer as the LSP refresh interval on the Cisco side, but they were two different things.  By cranking it way up, they ensured that the hundreds of LSPs in the database would trickle in, taking hours to converge.

Why the 0x0000 entries then?  It turns out that in the initial exchange, the ISIS routers share with each other what LSPs they have, without sending the full LSP.  Thus, in Cisco ISIS databases, the 0x0000 entry acts as a placeholder until complete LSP data is received.  Normally this period is short and you don’t see the entry.  We probably would have found the person who knew that eventually, but we didn’t find him that night and our database of cases, newsgroup postings, and bugs turned up nothing to point us in the right direction.

I touched a couple thousand cases in my time at TAC, but this case I remember even 10 years later because of the seeming complexity, the simplicity of the resolution, the weirdness of the symptoms, and the distractors like the PRP upgrade.  Often a major outage sends you in a lot of directions and down many rat holes.  I don’t think we could have done much differently, since the config error was totally invisible to us.  Anyway, if Juniper and Cisco can work together to solve a customer issue, maybe we should have hope for world peace.

3

Why certifications can suck

Ivan Peplnjak referenced a piece by Robert Graham called “Why cybersecurity certifications suck,” which I found amusing considering that I am doing some question writing right now.  I’ve certainly been a member of the chorus of complainers in the past, in various venues, but I have to say that my experience on the writing end of things has changed my perspective a bit.  I’m currently working on CCIE R/S questions, and previously worked on Juniper certifications.  Graham references a question which is pure trivia, and then attributes the problem to question writers who are “only one short step ahead of their students” because they “lack experience and deep knowledge.”

Writing and Writers

I can’t speak for every exam, but these statements are definitely incorrect for the Cisco and Juniper examinations I’ve worked on.  In both cases, I was required to prove myself to be a “subject matter expert” based on both credentials and work experience.  Obviously this does not mean I am a SME on every single topic on the exam, but in both cases I was also given the choice on which topics to write on.  I was never forced into writing questions on a topic that wasn’t familiar.

Writing exam questions is not easy.  First, let’s consider the motivation of the question writer.  Some are professionals in the training department of the company or organization whose exam they are writing for.  These people often produce high quality questions because they have some formal training in the subject and have done it a lot.  They are writing the questions simply because their job requires it of them.

Others, like myself, have a day job.  We do the question writing on the side.  We are given some training and extensive guidelines on how to write questions.  Usually there is some extra incentive for us.  At both Juniper and Cisco, I was able to recertify my existing certifications by writing questions, thus avoiding the dreaded recert exam.  Folks like us pose two problems:  (1) We are not experts at writing questions and (2) we have a motivation to get as many questions out the door as possible.  Given number 2, there can be an overwhelming desire to pick a topic, go dig through the relevant doc until you find footnote 7 on page 135, and drop a quick trivia question.

The hardest questions to write are ones that test thinking and not mere trivia.  I have spent hours on questions of that type.  They are my favorite to write, but often require drawing a diagram, setting up the scenario in the lab, collecting outputs, and pulling it all together into a question with right and wrong answers.  Given the difficulty, you can see why many writers skip right to the easy trivia, especially if they are in it to recertify.

Most exams are in multiple choice format these days, although there are several that use simulators and fill-in-the-blank style questions.  This is just a limitation of administering so many exams to so many people.  I wish we could do essay questions on the CCIE written, but it’s not happening.

Wrong Answers

Interestingly enough, the hardest part about writing multiple choice questions is coming up with the wrong answers.  That’s because they need to be really wrong, but also plausible.  Really wrong in that if the question says “pick the correct 2 out of 4”, it’s no good if 3 out of 4 are actually correct.  It’s very easy to accidentally write a wrong answer that is correct.  Plausible because they cannot be so obviously wrong that they give away the correct answer(s).  Let me take an example.

Which two cities are in Asia?  (Choose two.)

A.  Beijing
B.  Golden Retriever
C.  Singapore
D.  Banana

In this case, even if I didn’t know what cities were in Asia, I could easily find the right answer because I know B and D are not cities.  A better question would be:

Which two cities are in Asia?  (Choose two.)

A.  Beijing
B.  Warsaw
C.  Singapore
D.  Lisbon

However, the closer you make the wrong answers to reality, the more likely you will make one that is accidentally correct!

Writing Questions

So how do I write questions?  Usually I go through the topics assigned, and pick one that I am comfortable with.  Then I read through books, configuration guides, and other material on that subject until I think of a suitable question.  This could be factual, or it could be more of a scenario.  I generally write the question and then the right answer(s) and start working on the wrong answers.  After I’ve written it, I review it several times.  Every exam I have worked on required a reference document, so I always make sure to keep track of where I got the idea.  I try to put myself in the perspective of the examinee.  If I were studying this blueprint, where would I look for information?  How far into the document would I read?  At what point would I consider myself to have mastered the subject, and what knowledge would I have then?

Cheaters

It is very important for vendors such as Cisco to maintain the integrity of their exams.  Unfortunately, there is an army of professional cheaters who steal questions and post them verbatim on the Internet.  There are various techniques a vendor can use to combat this, but one of the best is simply to make the question pool so large that nobody can memorize it all.  This in turn puts relentless pressure on vendors to write as many questions as possible.  If you’ve ever used a PDF copy of a test to study, not only are you cheating yourself, you are also putting pressure on us to make more and more questions, which can decrease the question pool quality.

Amusingly enough, when I took my private pilot written exam back in 2005, we studied using the famous red Gleim book, which (legally) had all the questions and answers on the test.  When I took the test, I brought in my calculator and flight computer and never used them, because I had already seen the questions so many times!  I would just look at the question and think, “OK, the answer there is a heading of 270 degrees.”  The only reason I got less than one hundred percent was because I intentionally answered a few wrong, on the advice of my instructor.  Apparently if you show up to the oral portion of the flight exam with a 100% written score, the examiner might think you were arrogant and try to put you in your place with challenging questions.

 

I hope this article provided a little insight into why certification exams sometimes have bad questions.  Please know my intention is not to defend bad questions on a test.  Questions that are poorly written and which make it through Q/A are unfair to test takers and cause unneeded stress and even failures.  I have been on the other side many times, and have experienced the frustration of wrong or unfair exams.  As far as Cisco goes, a number of recent blogs have complained about the quality of the CCIE R/S question pool.  Having worked with the people responsible for it, I can say I have a lot of confidence in them and I think they are doing a good job administering it.

**  NOTE:  I will not answer any specific questions about the exam content I have written, so don’t even try!

 

Goodbye Juniper, Hello Cisco

I feel a bit of guilt for letting this blog languish for a while. I can see from the response to my articles explaining confusing Juniper features that my work had some benefit outside my own edification, and so I hate to leave articles unfinished which might have been helpful. In addition, WordPress is not easy to maintain and I keep losing notifications of comments, which means that when I am not logging in, I miss the opportunity to respond to kind words and questions.

As it is, my work explaining Juniper to the masses will have to be put on hold, as I have left Juniper after six years and returned to my old employer Cisco! I worked at Juniper longer than I had anywhere else, and it’s amazing to consider that I just closed the door on half a decade. But, I even after attaining my JNCIE I always felt like a Cisco guy at heart, and so here I am again. A few random thoughts then:

1. I interviewed for a number of jobs, and now that I am hired I can say that I really hate interviewing. My interviews at Cisco were very fair and reasonable. Just for the heck of it I did a phone screen with Google and completely bombed it. I’m not ashamed to admit that. I’m not supposed to reveal their questions, and I won’t, but they were mostly basic questions about TCP functionality, and MAC/ARP stuff, and it’s amazing how you forget some of the basics over the years. I wasn’t really interested in working there so I did no preparation, and in fact the recruiter warned me to brush up on basics. I just figured my work and blog show that I am at least somewhat technical. I plan to write some posts on the art of technical interviewing, but I was certainly underwhelmed by Google’s screening process, as I’m sure they were by my performance. I really wanted the Cisco job, and what a difference attitude makes! (Oh, and I completely munged an MPLS FRR/Node & Link protection question, less than a year after passing the JNCIE-SP. Uh, whoops.)

2. I bear Juniper no ill will. It was an interesting six years. When I came on board, during the Kevin Johnson years, it was all rah-rah pep talks about how we were going to be the next $10 billion company (errr, no…) followed by a plethora of product disasters. Killing off Netscreen gave the firewall market to Palo Alto, Fortinet, and amazingly resuscitated Checkpoint. Junos Space was a disaster, and Pulse slightly less so. QFabric was not a bad idea, but was far too complex. You needed to buy a professional services contract with the product, because it was too complex to install by itself. And yet it supposedly simplified the data center? There was a fiasco with our load balancer product. And then came the activist investors with their Integrated Operating Plan. I will permanently loathe activist investors. Juniper was hurting and they just magnified the hurt. There’s nothing worse than a bunch of generic business-types who wouldn’t know a router if they saw one trying to tell a router company how to do its business. They thought they could apply the same formula you learn in B-school to any company no matter what it manufactures or does. Then we had the CEO revolving door.

Despite all of this, as I said, I like Juniper. I did ok there, and there are a lot of people I respect working there. Rami Rahim is a good choice for CEO. I left for personal reasons. They still have some good products and good ideas, and I think competition is always good for the marketplace. For the sake of my friends there, I hope Juniper does well.

3. If you read my bio, you will see that I was THE network architect for Juniper IT, meaning I covered everything. This included (in theory at least) campus LAN, WAN, data center, wireless, network security, etc. I did something in all of these spaces. It was a broad level of knowledge, but not deep. That’s why I did my JNCIE-SP–I was hungering to go deep on something. My new job at Cisco is principal technical strategy engineer for data center. This is an opportunity to go deep but not as broad, and I’m happy to be doing that. The data center space is where it’s at these days, and I can’t wait to get deeper into it.

4. Coming back to Cisco after an eight year hiatus was bizarre. It was cool to pull up all my old bugs and postings to internal aliases to see what I was doing back then. Heck, I actually sounded like I knew a thing or two. I was thrilled to find out I am on the same team as Tim Stevenson, whose work as a Cat 6K TME I admired when I worked in TAC. Just for fun I walked though my old building and floor (K, floor 2) and nearly fell over when I saw that it looked identical. I mean, not only the cubes, but there were these giant signs for the different teams (e.g. “HTTS AT&T TEAM”) which were still hanging there as though the intervening eight years had never happened.

Unfortunately, I have to leave a few in progress articles in the dustbin. First, I shouldn’t really be promoting Juniper now that I am working for Cisco. And second, I’ve lost access to VMM, the internal Juniper tool I used to spin up VM versions of Juniper routers. However, I hope to start posting on Cisco topics now that I have access to that gear. Cisco’s products are generally better documented than Juniper’s, but I promise to fill any gaps I might find. And I will leave my previous articles up in hopes that they will benefit future engineers who struggle with Junos.

Onwards!

The joy of being an “expert”

Ah, the joys of being an “expert.”  I had forgotten what happens after you pass an exam like the JNCIE.  One of my colleagues starts grilling me on various topics for which I am unprepared, since they weren’t covered on the exam.  MX architecture, MC-LAG, MX virtual chassis, etc.  Be careful what you wish for.  The second you put “expert” on your title some people will take it as a challenge.  Of course, being in a very non-hands-on position, I am not an expert in many of the things day-to-day engineers know quite well.  I certainly know the topics on the test.  Then again, having obtained a CCIE 10 years ago, I know better than to start parading myself around as an expert on anything.  The JNCIE number goes on the blog and the bottom of my LinkedIn, but I am not putting it in my email signature.  Meanwhile, time to start reading Doug Hanks’ book on MX architecture!

Blogging on hold

For the handful of people who come across this blog and have posted comments, thank you very much for the kind words. This blog is on hold for a bit while I finish up my JNCIE-SP, which I am taking in a couple weeks. I’ve come across a lot of excellent blog post topics from my studying, so I hope to get back into the swing of things soon. Keep an eye out, especially if you are new to Juniper or struggling with some of the less-documented commands.

Welcome!

This is my first post on this blog which I created some time ago and have left dormant.  Give that there are about twice as many blogs as people, it would seem best to start out with a statement of my purpose and intent.

Before that, a little background:  I currently work for Juniper Networks, although I don’t claim to speak for them.  I am responsible for network architecture within IT, which gives a unique perspective since I am both a customer and a vendor.  I’ve been working in this industry for over 15 years, although my history with computers goes back farther than that.  I hold dual CCIE certifications in Routing/Switching and Security, and an M.S. in Telecommunications Management.  Non-technical credentials:  I hold an FAA Private Pilot’s certificate, I have studied and taught Ancient Greek and Latin.

My hope is to provide several types of articles here.   I specialize in communicating technical concepts in simple and direct language, so I will be breaking down difficult technical subjects for my readers, focusing particularly on subjects that frustrate me.  (Don’t get me started on MSTP).  I will also provide frank commentary on the industry, its trends, and on training and certification.  I will also augment this with stories from my years as a network engineer that hopefully will keep things entertaining.  Finally I hope my language and humanities experience will lend some interesting color to this site so that it is not just another tech blog.