Ivan Peplnjak referenced a piece by Robert Graham called “Why cybersecurity certifications suck,” which I found amusing considering that I am doing some question writing right now. I’ve certainly been a member of the chorus of complainers in the past, in various venues, but I have to say that my experience on the writing end of things has changed my perspective a bit. I’m currently working on CCIE R/S questions, and previously worked on Juniper certifications. Graham references a question which is pure trivia, and then attributes the problem to question writers who are “only one short step ahead of their students” because they “lack experience and deep knowledge.”
Writing and Writers
I can’t speak for every exam, but these statements are definitely incorrect for the Cisco and Juniper examinations I’ve worked on. In both cases, I was required to prove myself to be a “subject matter expert” based on both credentials and work experience. Obviously this does not mean I am a SME on every single topic on the exam, but in both cases I was also given the choice on which topics to write on. I was never forced into writing questions on a topic that wasn’t familiar.
Writing exam questions is not easy. First, let’s consider the motivation of the question writer. Some are professionals in the training department of the company or organization whose exam they are writing for. These people often produce high quality questions because they have some formal training in the subject and have done it a lot. They are writing the questions simply because their job requires it of them.
Others, like myself, have a day job. We do the question writing on the side. We are given some training and extensive guidelines on how to write questions. Usually there is some extra incentive for us. At both Juniper and Cisco, I was able to recertify my existing certifications by writing questions, thus avoiding the dreaded recert exam. Folks like us pose two problems: (1) We are not experts at writing questions and (2) we have a motivation to get as many questions out the door as possible. Given number 2, there can be an overwhelming desire to pick a topic, go dig through the relevant doc until you find footnote 7 on page 135, and drop a quick trivia question.
The hardest questions to write are ones that test thinking and not mere trivia. I have spent hours on questions of that type. They are my favorite to write, but often require drawing a diagram, setting up the scenario in the lab, collecting outputs, and pulling it all together into a question with right and wrong answers. Given the difficulty, you can see why many writers skip right to the easy trivia, especially if they are in it to recertify.
Most exams are in multiple choice format these days, although there are several that use simulators and fill-in-the-blank style questions. This is just a limitation of administering so many exams to so many people. I wish we could do essay questions on the CCIE written, but it’s not happening.
Wrong Answers
Interestingly enough, the hardest part about writing multiple choice questions is coming up with the wrong answers. That’s because they need to be really wrong, but also plausible. Really wrong in that if the question says “pick the correct 2 out of 4”, it’s no good if 3 out of 4 are actually correct. It’s very easy to accidentally write a wrong answer that is correct. Plausible because they cannot be so obviously wrong that they give away the correct answer(s). Let me take an example.
Which two cities are in Asia? (Choose two.)
A. Beijing
B. Golden Retriever
C. Singapore
D. Banana
In this case, even if I didn’t know what cities were in Asia, I could easily find the right answer because I know B and D are not cities. A better question would be:
Which two cities are in Asia? (Choose two.)
A. Beijing
B. Warsaw
C. Singapore
D. Lisbon
However, the closer you make the wrong answers to reality, the more likely you will make one that is accidentally correct!
Writing Questions
So how do I write questions? Usually I go through the topics assigned, and pick one that I am comfortable with. Then I read through books, configuration guides, and other material on that subject until I think of a suitable question. This could be factual, or it could be more of a scenario. I generally write the question and then the right answer(s) and start working on the wrong answers. After I’ve written it, I review it several times. Every exam I have worked on required a reference document, so I always make sure to keep track of where I got the idea. I try to put myself in the perspective of the examinee. If I were studying this blueprint, where would I look for information? How far into the document would I read? At what point would I consider myself to have mastered the subject, and what knowledge would I have then?
Cheaters
It is very important for vendors such as Cisco to maintain the integrity of their exams. Unfortunately, there is an army of professional cheaters who steal questions and post them verbatim on the Internet. There are various techniques a vendor can use to combat this, but one of the best is simply to make the question pool so large that nobody can memorize it all. This in turn puts relentless pressure on vendors to write as many questions as possible. If you’ve ever used a PDF copy of a test to study, not only are you cheating yourself, you are also putting pressure on us to make more and more questions, which can decrease the question pool quality.
Amusingly enough, when I took my private pilot written exam back in 2005, we studied using the famous red Gleim book, which (legally) had all the questions and answers on the test. When I took the test, I brought in my calculator and flight computer and never used them, because I had already seen the questions so many times! I would just look at the question and think, “OK, the answer there is a heading of 270 degrees.” The only reason I got less than one hundred percent was because I intentionally answered a few wrong, on the advice of my instructor. Apparently if you show up to the oral portion of the flight exam with a 100% written score, the examiner might think you were arrogant and try to put you in your place with challenging questions.
I hope this article provided a little insight into why certification exams sometimes have bad questions. Please know my intention is not to defend bad questions on a test. Questions that are poorly written and which make it through Q/A are unfair to test takers and cause unneeded stress and even failures. I have been on the other side many times, and have experienced the frustration of wrong or unfair exams. As far as Cisco goes, a number of recent blogs have complained about the quality of the CCIE R/S question pool. Having worked with the people responsible for it, I can say I have a lot of confidence in them and I think they are doing a good job administering it.
** NOTE: I will not answer any specific questions about the exam content I have written, so don’t even try!