Perspectives

Inside Cisco Live

Posted by ccie14023 on June 2, 2019
Posted in: Perspectives. Tagged: , , . 3 comments

 

While I’m thinking about another TAC Tale, I’m quite busy working on slides for Cisco Live.  I figured this makes for another interesting “inside Cisco” post, since most people who have been to the show don’t know much about how it comes together.  A couple years back I asked a customer if I could schedule a meeting with him after Cisco Live, since I was working on slides.  “I thought the Cisco Live people made the slides and you just showed up and presented them!” he said.  Wow, I wish that was the case.  With hundreds of sessions I’m not sure how the CL team could accomplish that, but it would sure be nice for me.  Unfortunately, that’s not the case.

If you haven’t been, Cisco Live is a large trade show for network engineers which happens four times globally: in Europe, Australia, the US, and Mexico.  The US event is the largest, but Europe is rather large as well.  Australia and Mexico are smaller but still draw a good crowd.  The Europe and US shows move around.  The last two years Europe was in Barcelona, as it will be next year, but it was in Berlin two years before that.  The US show is in San Diego this year, was in Orlando last year, and was in Las Vegas for two years before that.  Australia is always in Melbourne, and Mexico is always in Cancun.  I went to Cisco Live US twice when I worked for a partner, and I’ve been to every event at least once since I’ve worked at Cisco as a TME.

The show has an number of attractions.  There is a large show floor with booths from Cisco and partners.  There are executive and celebrity keynotes.  The deepest content is delivered in sessions–labs, techtorials, and breakout sessions which can have between 20 and several hundred attendees.  The sessions are divided into different tracks:  collaboration, security, certification, routing and switching, etc., so attendees can focus on one or more areas.

Most CL sessions are delivered by technical marketing engineers like myself, who work in a business unit, day in and day out, with their given product.  As far as I know anyone in Cisco can submit a session, so some are delivered by people in sales, IT, CX (TAC or AS), and other organizations.  Some are even delivered by partners and customers.

Six months before a given event, a “call for papers” goes out.  I’m always amused that they pulled this term from academia, as the “papers” are mostly powerpoints and not exactly academic.  If you want to do a session, you need to figure out what you want to present and then write up an abstract, which contains not only the description, but also explains why the session is relevant to attendees, what they can hope to get out of it, and what the prerequisites are.  Each track has a group of technical experts who manage it, called “Session Group Managers”, or SGMs.  They come from anywhere in the business, but have the technical expertise to review the abstracts and sessions to ensure they are relevant and well-delivered.  For about a year, the SGM for the track I usually presented actually reported to me.  They have a tough job, because they receive a large number of applications for sessions, far more than the slots they have.  They look at the topic, quality of the abstract, quality of the speaker, available slots, and other factors in figuring out which sessions get the green light.

Once you have an approved session, you can start making slides.  Other than a standard template, there is not much guidance on how to build a deck for Cisco Live.  My old SGM liked to review each new presentation live, although some SGMs don’t.  Most of us end up making our slides quite close to the event, partly because we are busy, but also because we want to have the latest and most current info in our decks.  It’s actually hard to write up a session abstract six months before the event.  Things change rapidly in our industry, and often your original plan for a session gets derailed by changes in the product or organization.  More than once I’ve had a TME on my team presenting on a topic he is no longer working on!  One of my TMEs was presenting on Nexus switches several months after our team switched to Catalyst only.

At Cisco Live you may run into the “speaker ready room.”  It’s a space for speakers to work on slides, supplied by coffee and food, but there is also a small army of graphic design experts in there who will review the speakers’ slides one last time before they are presented. They won’t comment on your design choices, but simply review them to ensure they are consistent with the template formatting.  We’re required to submit our final deck 24 hours before our session, which gives the CL staff time to post the slides for the attendees.

Standing up in front  of a room full of engineers is never easy, especially when they are grading you.  If you rate in the top 10% of speakers, you win a “Distinguished Speaker” award.  If you score below 4.2 you need to take remedial speaker training.  If your score is low more than a couple times, the SGMs might ask you not to come back.  Customers pay a lot of money to come to CL and we don’t want them disappointed.  For a presenter, being scored, and the high stakes associated with the number you receive, makes a CL presentation even more stressful.  One thing I’ve had to accept is that some people just won’t like me.  I’ve won distinguished speaker before, but I’ve had some sessions with less-than-stellar comments too.

The stress aside, CL is one of the most rewarding things we do.  Most of the audience is friendly and wants to learn.It’s a fun event, and we make great contacts with others who are passionate about their field.  For my readers who are not Cisco TMEs (most I suspect), I hope you have a chance to experience Cisco Live at least once in your career.  Now you know the amount of work that goes into it.

I am not a coder!

Posted by ccie14023 on February 13, 2019
Posted in: Perspectives. Tagged: , , , , , . 6 comments

I recently replied to a comment that I think warrants a full blog post.

I’ve been here at Cisco working on programmability for a few years.  Brian Turner wrote in to say, essentially:  Hang on!  I became a network engineer precisely because I don’t want to be a coder!  I tried programming and hated it!  Now you’re telling me to become a programmer!

As I said in my reply, I have a lot of sympathy for him.  It reminds me of a story.

Back when I was at Juniper, I met with the IT department’s head of automation to discuss using some of his tools for network automation.  Jeremy was an expert in all things Puppet and Ansible, and a rather enthusiastic promoter of these tools on the server/app side of the house.  He had also managed to get Puppet running on a Junos device.  I was meeting with him because, frankly, the wind seemed to be blowing in his direction.  That said, I did not share his enthusiasm.  He told me about a server guy he had worked with, Stephane.  When Jeremy proposed to Stephane that he should use automation tools to make his life easier, Stephane vehemently rejected the idea, and the meeting ended with Stephane banging his fists on the table and shouting “I am not a coder!”

Flash forward a couple years and Stephane ended up the head of automation for a major company.  Apparently he finally bought into the idea.

Frankly I had no desire to become a coder either.  When I interviewed at Cisco, most of my discussions were around the controllers I was working with at the time, data center fabrics, etc.  When I arrived, my new boss assigned me as his Principal TME for programmability.  I never claimed to be an expert in this area.  Two months later I was presenting to Tech Field Day, and experienced automation guys like Jason Edelman and Matt Oswalt on how to run Puppet on a Nexus switch.  Three years later and I’m known as a NETCONF/YANG guy.  I’d barely heard of them when I started.

As I replied to Brian, Cisco doesn’t want him or anyone to learn Python or YANG or whatever.  Think about it from my perspective in product management.  Implementing YANG models for all of IOS XE is a massive undertaking.  Engineering devoted a huge amount of effort to pull this off.  Huge.  Mandating YANG models for their ongoing development burns cycles.  Product marketing and engineering would never prioritize this unless we thought there was a high probability someone would use it.  In other words, we don’t want people to use it so much as customers want us to develop it.  We have demand for programmable interfaces for network devices, and hence we’ve delivered on it.   My job as a TME is not to push NETCONF/YANG on anyone, but to provide the enablement to make it easier for someone to use this technology if they themselves want to.

As I often say in my presentations, the why is important.  Why do some customers demand these interfaces?  Well, because they know Notepad is a horrible automation tool, and it’s what 90% of network engineers use.  If you want to configure 50 switches, you’re going to configure one, paste the config into Notepad, tweak a few values, and then paste it into the next switch.  Do this 48 more times and tell me if this is the best use of your time as a highly skilled network engineer.  You can write a script to do this and save yourself a lot of trouble.  Or use Ansible to do it.  Or Cisco DNAC.  Whatever you want.  But if you want any of these tools to work efficiently, you need a machine interface, which CLI is not.  If you don’t believe me, try writing a script to do regular expression-based parsing of CLI outputs.  It’s a lot easier with YANG.

The point is not for network engineers to become programmers.  The point is to add some tools to your toolbox to help you focus on what you do well.  One weekend spent with a Python course and one more weekend with a DevNet course on YANG will give you a tool you can use to make your life easier.  That’s it.  Some customers may take it a lot further, of course, and go way into CI/CD workflows and that’s fine.  If you want to do 95% of your work in CLI and write a few scripts to do the other 5%, that’s fine.  If you want to use Cisco DNAC to do almost everything, knock yourself out.  It’s about what works best for you, as a network engineer.

I often point out how lousy my code quality is.  I’m sometimes ashamed to show the code for some of the scripts I’ve written.  I’m not a coder!  That’s a point I often make.  I don’t want to be a full-time software developer.  I’m a network engineer.  So for Brian and all the other CCIE’s out there, keep doing what you do best, but don’t close yourself off to some additional tools that will make your life easier.

Do we hate network engineers?

Posted by ccie14023 on October 3, 2018
Posted in: Perspectives. Tagged: , , , , , . Leave a Comment

I was doing well on the blog for a few months but lately fell behind.  With (now) 12 people reporting to me, and three major areas of responsibility (SD-Access, Assurance, and Programmability), it’s not easy to find time to write up a blog post.   I have about five drafts needing work but I cannot seem to find the will to finish them.  Sometimes, however, it just takes a spark to get me going. That spark came in my inbox from Ivan Peplnjak.  I like Ivan’s blog posts, which, while often not favorable to Cisco, are nonetheless fair and balanced and raise some very important points.

“Why Is Every SDN Vendor Bashing Networking Engineers?” asks Ivan in the form email I received.  “[T]he vendors know they wouldn’t be able to sell their latest concoctions to people who actually understand how networking works and why some architectures have no chance of ever working in real life,” answers Ivan.  “The only way to sell the warez is to try to convince everyone else how to get rid of the pesky ossified CLI jockeys.”

Now I work for a vendor, and since I deal with the aforementioned products, I guess I am an SDN vendor.  That would seem to qualify me to speak on this subject.  (With, of course, the usual disclaimer that the opinions here are my own and do not represent Cisco officially.)

Selling Concoctions

I must admit, I do want to sell our products.  Everyone at Cisco should want our products to sell.  Just about all of us have a personal, financial stake in the matter, whether we have stock grants or ESPP.  We would be insane not to want people to buy our products.  I, and most of my co-workers, are driven by far more than finance, however.  We all want to know that our work means something, and that we are coming up with innovative solutions to problems.  Otherwise, why show up in the office every day?

We operate in a highly competitive environment, which means if we are not constantly innovating and coming up with better ways to do things, we will all suffer.  You can complain about the macroeconomic system, and believe me, I’m not a Randian, objectivist believer in unbridled capitalism.  But, at the end of the day, a public company needs to create the perception of future value in the eyes of the stock market, and that’s a motivating factor for all of us.

These things being said, I’ve been in product management for a few years now and I have never heard anyone, ever, talk about trying to put one over on our customers.  I’m not saying that’s what Ivan means here, but it’s an accusation I’ve heard before.  In the first place, our customers are network engineers who are quite smart.  If ever I’ve presented to my customer and was not crystal clear on what I was talking about and what advantage it would bring the customer, they will let me know it.  We’re constantly trying to find ways to do things better and make our customers’ lives easier.  As somebody who worked in IT for more years than product management, I’m very interested in this subject.  There were a lot of things that were frustrating and I want to fix things that used to annoy me.  You can argue about whether we’ve come up with the right ideas, but I hope nobody questions our motivations.

CLI Jockeys

Do I bash CLI jockeys in order to sell my products?  I should hope not, given that most of my customers are CLI jockeys, as I am myself!  I have two CCIEs and a JNCIE.  I spent a couple years in routing protocols TAC and many years in IT.  I spent a long time learning my trade and I have a lot of respect for those who have put the time and effort into learning it as well.  It’s not easy.

However, I don’t operate under the delusion that network engineers do a good job of configuring and managing CLI.  When I was at Juniper, I had designed a new NGMVPN system for our WAN.  I handed it off to the implementation team with some sample configs and asked them to come back to me with their plan.  I think we were touching about 20 devices the first go around.  The engineer came back with 20 Word documents.  He took my sample config and copied and pasted it into Word, and then modified the config in a separate Word doc for each CE/PE he was touching.  CLI itself isn’t a problem, but how we manage it.  This is where programmability and automation tools come in.  At the very least Ansible templating would have made this easier.  Software-Defined Networking (a very loose term, for what it’s worth), is not about replacing ossified CLI jockeys but getting them to focus on what they should be doing (network engineering) and avoiding what they should not (pasting stuff in Word docs.)

SD-Access takes this quite a bit further than Ansible, NETCONF, and other device-level tools.  Rather than saying “I want this device to be a LISP MS/MR” and so forth, you just say “I want this device to be a control plane node” and the system figures out what you need.  Theoretically we could change from LISP to some other protocol and the end-user shouldn’t even notice.  The idea here is somewhat like a fly-by-wire system.  When a pilot operated the controls of an airplane, they used to be directly coupled to the control surfaces via hydraulics.  Now, the pilot is operating what is essentially a joystick, providing control inputs to a computer, which then computes the best way to move the control surfaces given the conditions.  This is then relayed to servo motors in the wings, tail, etc.  The complexity of a fly-by-wire system is much higher than an old hydraulic system, but the complexity is hidden from the pilot in order to provide a better experience.  Likewise, with SD-Access, we’ve made the details more complex in order to deliver a better experience (TrustSec, layer 3 routed backbone, etc.) while hiding the complexity from the user.  It’s a different approach, for sure, but the idea is to allow engineers to focus on the right problems, like how to design their network, and not worry so much about configuration.

A New Era?

I’ve written extensively (see, for example, here, and here) about the role for CLI-jockey network engineers in the future.  When airplanes switched from the old dials and gauges to sleek, modern computerized (glass) cockpits, I’m sure some old timers threw up their hands, retired, and got their old Piper Super Cubs out of the hanger to do some “real” flying.  But most adapted, and in the end, saw how the new automation systems helped them do their jobs better.  That’s an era I’m looking forward to.  And as I always, always say, the pilots who fly the new cockpits still need to understand weather systems, engines, navigation, etc.  We still need network engineers who know how networks operate.

Meanwhile, I won’t bash any CLI jockeys and I hope nobody else here does either.

Network Engineering Jobs

Posted by ccie14023 on July 20, 2018
Posted in: Perspectives. Tagged: , , , , , , , , . 3 comments

I’ve been in this industry a while now, and I’ve done a lot of jobs.  Certainly not every job, but a lot.  My first full time network engineering job came in 2000, but I was doing some networking for a few years before that.

I often see younger network engineers posting in public forums asking about the pros and cons of different job roles.  I’ve learned over the years that you have to take people’s advice with a grain of salt.  Jobs in one category may have some common characteristics, but a huge amount is dependent on the company, your manager, and the people you work with.  However, we all have a natural tendency to try to figure out the situation at a potential job in advance, and the experience of others can be quite helpful in sizing up a role.  Therefore, I’ve decided to post a summary of the jobs I’ve done, and the advantages/disadvantages of each.

IT Network Engineer

Summary:
This is an in-house engineer at a corporation, government agency, educational institution, or really any entity that runs a network.  The typical job tasks vary depending on level, but they usually involve a large amount of day-to-day network management.  This can be responding to complaints about network performance, patching in network connectivity (less so these days because of wireless), upgrading and maintaining devices, working with carriers, etc.  Larger scale projects could be turning up new buildings and sites, planning for adding new functionality (e.g. multicast), etc.

Pros:

  • Stable and predictable work environment.  You show up at the same place and know the people, unlike consulting.
  • You know the network.  You’re not showing up a new place trying to figure out what’s going on.
  • It can be a great chance to learn if the company is growing and funds new projects

Cons:

  • You only get to see one network and one way of doing things.
  • IT is a cost center, so there is a constant desire to cut personnel/expenses.
  • Automation is reducing the type of on-site work that was once a staple for these engineers.
  • Your fellow employees often hate IT and blame you for everything.
  • Occasionally uncomfortable hours due to maintenance windows.

Key Takeaway:
I often tell people that if you want to do an in-house IT job, try to find an interesting place to work.  Being an IT guy at a law firm can be kind of boring.  Being an IT guy at the Pentagon could be quite interesting.  I worked for a major metropolitan newspaper for five years (when there was such a thing) and it was fascinating to see how newspapers work.  Smaller companies can be better in that you often get to touch more technologies, but the work can be less interesting.  Larger companies can pigeonhole you into a particular area.  You might work only on the WAN and never touch the campus or data center side of things, for example.

Technical Support Engineer

Summary:
Work at a vendor like Cisco or Juniper taking cases when things go wrong.  Troubleshoot problems, recreate them in the lab, file bugs, find solutions for customers.  Help resolve outages.  See my TAC Tales for the gory details.

Pros:

  • Great way to get a vast amount of experience by taking lots of tough cases
  • Huge support organization to help you through trouble
  • Short-term work for the most part–when you close a case you’re done with it and move on to something new
  • Usually works on a shift schedule, with predictable hours.  Maintenance windows can often be handed off.

Cons:

  • Nearly every call involves someone who is unhappy.
  • Complex and annoying technical problems.  Your job is 100% troubleshooting and it gets old.
  • Usually a high case volume which means a mountain of work.

Key Takeway:
Technical Support is a tough and demanding environment, but a great way to get exposure to a constant stream of major technical issues.  Some people actually like tech support and make a career out of it, but most I’ve known can burn out after a while.  I wouldn’t trade my TAC years for anything despite the difficulties, as it was an incredible learning experience for me.

Sales Engineer

Summary:
I’ve only filled this role at a partner, so I cannot speak directly to the experience inside a company like Cisco (although I constantly work with Cisco SE’s).  This is a pre-sales technical role, generally partnered with a less-technical account manager.  SE’s ultimately are responsible for generating sales, but act as a consultant or adviser to the customer to ensure they are selling something that fits.  SE’s do initial architecture of a given solution, work on developing the Bill of Materials (BoM), and in the case of partners, help to write the Statement of Work (SoW) for deployment.  SE’s are often involved in deployment of the solutions they sell but it is not their primary job.

Pros:

  • Architectural work is often very rewarding;  great chance to partner with customer and build networks.
  • Often allows working on a broad range of technologies and customers.
  • Because it involves sales, usually good training on the latest technologies.
  • Unlike pure sales (account managers in Cisco lingo), a large amount of compensation is salary so better financial stability.
  • Often very lucrative.

Cons:

  • Like any account-based job, success/enjoyability is highly dependent on the account(s) you are assigned to.
  • Compensation tied to sales, so while there are good opportunities to make money, there is also a chance to lose a lot of discretionary income.
  • Often take the hit for poor product quality from the company whose products you are selling.
  • Because it is a pre-sales role, often don’t get as much hands-on as post-sales engineers.
  • For some products, building BoM’s can be tedious.
  • Sales pressure.  Your superiors have numbers to make and if you’re not seen to be helping, you could be in trouble.

Key Takeaway:
Pre-sales at a partner or vendor can be a well-paying and enjoyable job.  Working on architecture is rewarding and interesting, and a great chance to stay current on the latest technologies.  However, like any sales/account-based job, the financial and career success of SE’s is highly dependent on the customers they are assigned to and the quality of the sales team they are working with.  Generally SE’s don’t do technical support, but often can get pulled into late-night calls if a solution they sell doesn’t work.  SEs are often the face of the company and can take a lot of hits for things that they sell which don’t work as expected.  Overall I enjoyed being a partner SE for the most part, although the partner I worked for had some problems.

Post-Sales/Advanced Services

Summary:
I’m including both partner post-sales, which I have done, and advanced services at a vendor like Cisco, which are similar.  A post-sales engineer is responsible for deploying a solution once the customer has purchased it, and oftentimes the AS/deployment piece is a part of the sale.  Occasionally these engineers are used for non-project-based work, more so at partners.  In this case, the engineer might be called to be on site to do some regular maintenance, fill in for a vacationing engineer, etc.

Pros:

  • Hands-on network engineering.  This is what we all signed up for, right?  Getting into real networks, setting stuff up, and making things happen.
  • Unlike IT network engineers, this job is more deployment focused so you don’t have to spend as much time on day-to-day administrative tasks.
  • Unlike sales, the designs you work on are lower-level and more detailed, so again, this is a great nuts-and-bolts engineering role.

Cons:

  • As with sales, the quality and enjoyability is highly dependent on the customers you end up with.
  • You can get into some nasty deployment scenarios with very unhappy customers.
  • Often these engagements are short-term, so less of a chance to learn a customer/network.  Often it is get in, do the deployment, and move on to the next one.
  • Can involve a lot of travel.
  • Frequently end up assisting technical support with deployments you have done.
  • Can have odd hours.
  • Often left scrambling when sales messed up the BoM and didn’t order the right gear or parts.

Key Takeway:
I definitely enjoyed many of my post-sales deployments at the VAR.  Being on-site, and doing a live deployment with a customer is great.  I remember one time when I did a total network refresh and  VoIP deployment up at St. Helena Unified School District in Napa, CA.  It was a small school district, but over a week in the summer we went building-by-building replacing the switches and routers and setting up the new system.  The customer was totally easygoing, gave us 100% free reign to do it how we wanted, was understanding of complications, and was satisfied with the result.  Plus, I enjoyed spending a week up in Napa, eating well and loving the peace. However, I also had some nightmare customers who micromanaged me or where things just went south.  It’s definitely a great job to gain experience on a variety of live customer networks.

Technical Marketing Engineer

Summary:
I’m currently a Principal TME and a manager of TMEs.  This is definitely my favorite job in the industry.  I give more details in my post on what a TME does, but generally we work in a business unit of a vendor, on a specific product or product family, both guiding the requirements for the product and doing outbound work to explain the product to others, via white papers, videos, presentations, etc.

Pros:

  • Working on the product side allows a network engineer to actually guide products and see the results.  It’s exciting to see a new feature, CLI, etc., added to a product because you drove it.
  • Get to attend at least several trade shows a year.  Everyone likes getting a free pass to a conference like Cisco Live, but to be a part of making it happen is exhilarating.
  • Great career visibility.  Because the nature of the job requires producing content related to your product, you have an excellent body of work to showcase when you decide to move on.
  • Revenue side.  I didn’t mention this in the sales write-up, but it’s true there too.  Being close to revenue is generally more fun than being in a cost center like IT, because you can usually spend more money.  This means getting new stuff for your lab, etc.
  • Working with products before they are ever released to the public is a lot of fun too.
  • Mostly you don’t work on production networks so not as many maintenance windows and late nights as IT or AS.

Cons:

  • Relentless pace of work.  New software releases are constantly coming;  as soon as one trade show wraps up it’s time to prepare for the next one.  I often say TME work is as relentless as TAC.
  • Can be heavy on the travel.  That could, of course, be a good thing but it gets old.
  • Difficulty of influencing engineering without them reporting to you.  Often it’s a fight to get your ideas implemented when people don’t agree.
  • If you don’t like getting up in front of an audience, or writing documents, this job may not be for you.
  • For new products, often the TMEs are the only resources who are customer facing with a knowledge of the product, so you can end up working IT/AS-type hours anyways.  Less an issue with established/stable products.

Key Takeaway:
As I said, I love this job but it is a frenetic pace.  Most of the posts I manage to squeeze in on the blog are done in five minute intervals over a course of weeks.  But I have to say, I like being a TME more than anything else I’ve done.  Being on the product side is fascinating, especially if you have been on the consumer side.  Going to shows is a lot of fun.  If you like to teach and explain, and mess around with new things in your lab, this is for you.

It’s not a comprehensive list of the jobs you can do as a network engineer, but it covers some of the main ones.  I’m certainly interested in your thoughts on other jobs you’ve done, or if you’ve done one of the above, whether you agree with my assessment.  Please drop a comment–I don’t require registration but do require an email address just to keep spam down.

Cisco IOS XE Programmability Book

Posted by ccie14023 on June 28, 2018
Posted in: Perspectives. Tagged: , , . Leave a Comment

In a previous post I had mentioned I co-authored a book on IOS XE Programmability with some colleagues of mine.  For those who are interested, the book is available here.

The book is not a comprehensive how-to, but a summary of the IOS XE features along with a few samples.  It should provide a good overview of the capabilities of IOS XE.  For those who were on my CCIE webinar, it should be more than adequate to get you up to speed on CCIE written programmability topics.

As with any technical book, there could be some errata, so please feel free to pass them along and I can get them corrected in the next edition.

On Being a Dinosaur

Posted by ccie14023 on June 24, 2018
Posted in: Perspectives, Reflections. Tagged: , , , , , . 5 comments

An old networking friend whom I mentored for his CCIE a long time ago wrote me an email:  I’ve been a CCIE for 10 years now, he said, and I’m feeling like a dinosaur.  Everyone wants people who know AWS and automation and they don’t want old-school CLI guys.

It takes me back to a moment in my career that has always stuck with me.  I was in my early twenties at my first job as a full-time network engineer.  I was working at the San Francisco Chronicle, at the time (early 2000’s) a large newspaper with a wide circulation.  The company had a large newsroom, a huge advertising call center, three printing plants, and numerous circulation offices across the bay area.  We had IP, IPX, AppleTalk and SNA on the network, typical of the multi-protocol environments of the time.

My colleague Tony and I were up in the MIS area on the second floor of the old Chronicle building on 5th and Mission St. in downtown San Francisco.  The area we were in contained armies of mainframe programmers, looking at the black screens of COBOL code that were the backbone of the newspaper systems in those days.  Most of the programmers were in their fifties, with gray hair and beards.  Tony and I were young, and TCP/IP networking was new to these guys.

I was telling Tony how I always wanted to be technical.  I loved CLI, and it was good at it.  I was working on my first CCIE.  I was at the top of my game, and if any weird problem cropped up on our network I dove in and got it fixed, no matter how hard.  As I explained to Tony, this was all I wanted to do in my career, to be a CLI guy, working with Cisco routers and switches.

Tony gestured at the mainframe programmers, sitting in their cubes typing their COBOL.  “Is this what you want to be when you’re in your fifties,” he said under his breath, “a dinosaur?  Do you just want to be typing obscure code into systems that are probably going to be one step away from being shut down?  How long do you think these guys will have their jobs anyways?”

Well, I haven’t been to the Chronicle in a while but those jobs are almost certainly gone.  Fortunately for the COBOL guys, they’re all retirement age anyways.

We live in a world and an industry that worships the young and the new.  If you’re in your twenties, and totally current on the latest DevOps tools, be warned:  someday you’ll be in your forties and people will think DevOps is for dinosaurs.  The tech industry is under constant pressure to innovate, and innovating usually means getting machines to do things people used to do.  This is why some tech titans are pushing for universal basic income.  They realize that their innovations eliminate jobs at such a rate that people won’t be able to afford to live anymore.  I think it’s a terrible idea, but that’s a subject for another post.  The point is, in this industry, when you think you’ve mastered something and are relevant, be ready:  your obsolescence commeth.

This is an inversion of the natural respect for age and experience we’ve had throughout human history.  I don’t say this as a 40-something feeling some bitterness for the changes to his industry;  in fact, I actually had this thought when I was much younger.  In the West, at least,  in the 1960’s there developed a sense that, to paraphrase Hunter Thompson, old is evil.  This was of course born from legitimately bad things that were perpetuated by previous generations, but it’s interesting to see how the attitude has taken hold in every aspect of our culture.  If you look at medieval guilds, the idea was that the young spent years going through apprentice and journeyman stages before being considered a master of their craft.  This system is still in place in many trades that do not experience innovation at the rate of our industry, and there is a lot to be said for it.  The older members of the trade get security and the younger get experience.

I’ve written a bit about the relevance of the CCIE, and of networking skills in general, in the new age.  Are we becoming the COBOL programmers of the early 2000’s?  Is investing in networking skills about the same as studying mainframe programming back then, a waste of cycles on dying systems?

I’ve made the point many times on this blog that I don’t think that’s (yet) the case.  At the end of the day, we still need to move packets around, and we’re still doing it in much the same way as we did in 1995.  Most of the protocols are the same, and even the newer ones like VXLAN are not that different from the old ones.  Silicon improves, speeds increase, but fundamentally we’re still doing the same thing.  What changing is how we’re managing those systems, and as I say in my presentations, that’s not a bad thing.  Using Notepad to copy/paste across a large number of devices is not a good use of network engineers’ time.  Automating can indeed help us to do things better and focus on what matters.

I’ve often used the example of airline pilots.  A modern airplane cockpit looks totally different from a cockpit in the 1980’s or even 1990’s.  The old dials and switches have been replaced by LCD panels and much greater automation.  And yet we still have pilots, and the pilot today still needs to understand engine systems, weather, aerodynamics, and navigation.  What’s changed is how that pilot interacts with the machine.  As a pilot myself, I can tell you how much better a glass cockpit is than the old dials.  I get better information presented in a much more useful way and don’t have to waste my time on unnecessary tasks.  This is how network automation should work.

When I raised this point to some customer execs at a recent briefing, one of them said that the pilots could be eliminated since automation is so good now.  I’m skeptical we will ever reach that level of automation, despite the futurists’ love of making such predictions.  The pilots aren’t there for the 99% of the time when things work as expected, but for the 1% when they don’t, and it will be a long time, if ever, before AI can make judgement calls like a human can.  And in order to make those 1% of calls, the pilots need to be flying the 99% of the time when it’s routine, so they know what to do.

So, are we dinosaurs?  Are we the COBOL programmers of the late 2010’s, ready to be eliminated in the next wave of layoffs?  I don’t think so, but we have to adapt.  We need to learn the glass cockpit.  We need to stay on top of developments, and learn how those developments help us to manage the systems we know well how to manage.  Mainframes and operating systems will come and go, but interconnecting those systems will still be relevant for a long time.

Meanwhile, an SVP at Cisco told me he saw someone with a ballcap at Cisco Live:  “Make CLI Great Again”.  Gotta love that.  Some dinosaurs don’t want to go extinct.

Where I’ve been, and what a TME is

Posted by ccie14023 on April 24, 2018
Posted in: Perspectives, Reflections. Tagged: , , , , , . 7 comments

Jesse, a recent commentor, asked why I haven’t been posting much lately.  In fact, my last post was August of 2017.  Well, there are several reasons I don’t post much these days.  In part, I’m not convinced anyone is reading.  It’s nice to see a comment now and again to realize it’s not just spambots looking at SZ.

The other major reason was a job change.  I moved to Cisco over two years ago, and I came in as an individual contributor (IC).  I liked to joke that I had never been so busy since…the last time I worked at Cisco.  However, as an IC, I had no idea how easy I had it.

Someone got the crazy idea to make me a manager.  So now, not only do I have the Principal Technical Marketing Engineer title, I also manage a team of 10 TMEs.  The team happens to be driving Software-Defined Access, currently Cisco’s flagship product.  So, the time for blogging is a bit limited.  I’m still working on programmability in my spare time, and I’m continuing to do Cisco Live sessions at least twice a year.  My hair is turning white and I don’t think it’s just my age.

That said, I cannot image a better job or place to be than this job at this time.  It’s an exciting company to work for, and an exciting time.  The team that reports to me includes some of the smartest and hardest working TMEs in Cisco.  These guys are legendary.  (For me “guys” is gender-neutral, for those of you who worry about such things.)  And my boss is considered by many to be one of the best who ever did the TME job.

A quick primer on exactly what a TME does, for those who don’t know:  We are (usually) attached to a business unit within Cisco, and we are really an interface between sales and engineering.  We also work directly with customers, but generally when sales pulls us in.  TMEs are technical (the “T” in “TME”) so they are expected to know their product/technology in detail.  They are, however marketers (the “M” in “TME”) so they need to be able to explain what their product does.

On the inbound side, we learn the requirements for products from sales and customers and communicate those requirements to engineering.  We work closely with Product Managers (PMs) to develop Product Requirement Documents (PRDs) and meet regularly with engineering to ensure that they are building their products in a way that satisfies marketing requirements.

On the outbound side, we develop collateral, which could be white papers, videos, slide decks, etc.  (We do not write the documentation you see on CCO, but we certainly review it.)  We present to our sales team in twice-a-year events, explaining the latest developments in our products and collecting their feedback on what we could do better.  We travel on site to meet with customers in support of sales, or else meet the customers here, at the Executive Briefing Center (EBC).  The most enjoyable part of the year, for most of us, is Cisco Live, our major trade show.

We have four CL events each year:  Europe, South America, Australia, and US.  The US event is the largest of all.  I generally attend both Europe (we were in Barcelona this year) and the US event (Orlando this time).  These events are a blast, but I never realized how much hard work goes into planning the event and developing the content.  It’s also stressful.  I’ve been fortunate to win distinguished speaker two events in a row, which means I was rated in the top 10%.  However, standing in front of an audience of hundreds is always a bit nerve-wracking, and getting ready requires a ton of preparation.  Still, it’s a great opportunity to meet with customers and have a good time.

The pace of work for TMEs is relentless.  I used to say TAC was relentless, because the second you close a case, you take another.  Well, with two SEVTs (sales events) and four Cisco Lives to prepare for, plus a constant and never-ending series of product/software releases…well, it never stops here either.

So that’s why the blogs have fallen away.  I do think I can find 10-15 minutes to post updates at least every week, so I’m going to try to do it.  I wouldn’t mind actually writing the series on programmability I started.  I’d like to clean up and revise the 10 years a CCIE series.  I also have another TAC tale to write up, one of my all-time favorites, so look for that soon.

And to Jesse:  thanks for getting me going!

In Praise of Vendor Lock-In

Posted by ccie14023 on July 17, 2017
Posted in: Perspectives. Tagged: , , , , , , , , . 1 comment

There is one really nice thing about having a blog whose readership consists mainly of car insurance spambots:  I don’t have to feel guilty when I don’t post anything for a while.  I had started a series on programmability, but I managed to get sidetracked by the inevitable runup to Cisco Live that consumes Cisco TME’s, and so that thread got a bit neglected.

Meanwhile, an old article by the great Ivan Pepelnjak got me out of post-CL recuperation and back onto the blog.  Ivan’s article talks about how vendor lock-in is inevitable.  Thank you, Ivan.  Allow me to go further, and write a paean in praise of vendor lock-in.  Now this might seem predicable given that I work at Cisco, and previously worked at Juniper.  Of course, lock-in is very good for the vendor who gets the lock.  However, I also spent many years in IT, and also worked at a partner, and I can say from experience that I prefer to manage single vendor networks.  At least, as single vendor as is possible in a modern network.  Two stories will help to illustrate this.

In my first full-fledged network engineer job, I managed the network for a large metropolitan newspaper (back when such a thing existed.)  The previous network team had installed a bunch of Foundry gear.  They also had a fair amount of Cisco.  It was all first generation, and the network was totally unstable.  Foundry actually had some decent hardware, but their early focus was IP.  We were running a typical 1990’s multi-protocol network, with AppleTalk, IPX, SNA, and a few other things thrown in.  The AppleTalk/IPX stack on the Foundry was particularly bad, and when it interacted with Cisco devices we had a real mess.

We ended up tossing the Foundry and going 100% Cisco.  We managed to stabilize the network, and now we were dealing with a single vendor instead of two.  This made support and maintenance contract management far easier.

Second story:  When I worked for the partner, I had to do a complete retrofit of the network for a small school district.  They had a ton of old HP, and were upgrading their PBX to a Cisco VoIP solution.  This was in the late 2000’s.  I did the data network, and my partner did the voice setup.  The customer didn’t have enough money to replace all their switches, so a couple of classrooms were left with HP.

Well, guess what.  In all the Cisco-based rooms, I plugged in the phones and they came up fine.  The computers hanging off the internal phone switch port also came up fine, on the correct data VLAN.  But on the classrooms with the HP switches, I spent hours trying to get the phones and switches working together.

There is a point here which is obvious, but needs restating.  If Cisco makes the switches, the routers, the firewalls, and the phones, the chances of them all working together is much higher than if several vendors are in the mix.  Even with Cisco’s internal BU structure, it is far easier to call a meeting with different departments within a company than to fix problems that occur between vendors.  Working on Software Defined-Access, I learned very quickly how well we can pull together a team from different product groups, since our product involves switching (my BU), ISE, wireless, and APIC-EM.

As I mentioned above, the other advantage is easier management of the non-technical side of things.  Managing support contracts, and simply having one throat to choke when things go wrong are big advantages of a single-vendor environment.

All this being said, from a programmability perspective we are committed to open standards.  We realize that many customers want a multi-vendor environment and tools like OpenConfig with which to manage it.  Despite Cisco’s reputation, we’re here to make our customers happy and not force them into anything.  From my point of view, however, if I ever go back to managing a network I hope it is a single-vendor network and not a Fraken-network.

Meanwhile, if you’d like to hear my podcast with Ivan, click here.