My CCIE Journey – Chapter 3

Last time, we covered the CCIE R/S Written exam. This time, I’ll explain how I approached preparation for the CCIE R/S Lab Exam. In this post, I’ll cover how I got started with the lab exam preparation, what worked and what didn’t, and end with my first lab attempt. Fair warning, this post will be long.

I think it’s important to examine my situation at the time, because as I’ve tried to drive home with every post, tackling this exam is almost impossible without adequate time, support and money.

SitRep

After passing the Written exam, I evaluated my situation. My wife was ready to let me study on weekends, and I was ready to commit to early mornings. This meant my new study schedule would be 4:30AM – 7:30AM on Mon-Fri, and 8AM – 4PM on weekends. For those keeping score, that gave me roughly 31 hours a week to study. Truthfully, it was the rare week where I could get all 31 hours. Usually, it was closer to 26-28. I knew that this would severely cut into family time, as my weekends would largely disappear, and in order to wake up in time to start so early, I would fall asleep around 8:30 or 9:00 PM, often immediately after getting the kids to bed. Some nights, I’d fall asleep on my daughter’s bed while sitting with them, waking up some time later to blearily stumble off to my own bed. As I got nearer to scheduling my lab exam, I was granted some time at work to study also, which put me over the 35 hour mark. I was basically working two full-time jobs.

Starting Out

Since I had adequate study time, I had to get the study materials together. There were lots of options, but I focused on using INE at first. Cisco provided me an account for the INE All-Access Pass, which gave me access to the CCIE R/S v5 Advanced Technology videos. These videos covered the entire blueprint in digestible sections and served as the bulk of my focus for my early studies. I tackled each video in turn and the corresponding ATC labs in the accompanying workbook, working hard to establish a base level of knowledge for the technologies I wasn’t already familiar with, and refreshing my memory on technologies I knew well.

A long, long time later, I completed my first (and only complete) run-through of both the ATC videos and workbook. This served to give me a base level of expertise in all the blueprint technologies. From this point, I started to focus my time on technologies that I didn’t already have extensive work experience with, and others that I had experience with but were so massive that there was no way to get work experience with all of the nerd knobs (ie, BGP).

I allocated a week to deep dive with each technology exclusively, and after that week, instead of moving onto the next technology, I instead did a repetition round-up of part of the blueprint. This strategy used the concept of spaced repetition, as I used for the written. By changing up technology exposure, I was building a solid floor instead of trying to play Diner Dash (or Overcooked) and concentrating on a whack-a-mole approach.

This required a lot of time, as spaced repetition requires, well…space. However, the blueprint is so massive that I never found myself waiting around for time to pass. It wove together more like a quilt, with lots of overlap. I don’t want to focus on hours or months spent at this approach, because it sends the wrong message that time is a token system – put in hours, get understanding and expertise. It’s not like that at all, it’s different for everyone and can’t be boiled down to something so simple as, “I spent 300 hours on BGP, and if you do, that’s enough”. Having said that, this is where I spent the most amount of time on my journey – not the initial run-through of the videos and blueprint, but on the weaving of a strong foundation of expertise that would be there when I needed it.

An accurate depiction of my CCIE study efforts.

Labbing Struggles and Wins

I should point out here that I struggled endlessly with my choice of labbing options. I started with GNS3 and had access to Cisco IOL images for Layer 2 and Layer 3. This served me well for about 85% of my studies. I built the INE ATC modular topology (10 routers, 4 switches) and spent some time up front setting up the ability to fast-switch between ATC labs. IOL images have a unix file system that is persistent and so I would load the configurations for a particular lab onto all devices, then save the running configuration to that file system with a name like basic.bgp.routing.cfg. This allowed me to later script the fast-switch when moving between labs using the configure replace command. If I did a 10-minute lab on BGP prefix injection, and was ready to move on to the next lab about BGP add-path, I could simply do configure replace unix:bgp.add-path.cfg at the CLI of each device, and within less than a minute my next lab was ready to go. Credit for this innovation goes to a good friend of mine, Mike Doe who had done the same for his CCIE studies.

I was able to use GNS3 with no issues for almost the entirety of the CCIE v5 ATC Workbook, including the INE Foundation Labs. The Foundation Labs start to introduce the task-based format of the real CCIE lab exam, but still only consist of 10 routers and 4 switches. Once it was time to move on to the Full-Scale labs, which consist of 20 routers and 4 switches, my GNS3 began to have issues. It may not have been a GNS3 issue. It could have been small bugs in IOL, combined with resource starvation with my laptop. I know I struggled endlessly with problems, and it got to the point that I was wasting valuable lab time troubleshooting it rather than labbing. The worst cases were when I was trying to do one of the troubleshooting labs, wasting endless amounts of time trying to verify a problem, only to find that the issue I was experiencing wasn’t part of the lab at all! Talk about hard mode!

I tried a few other non-starters like WebIOL, which proved to be buggier, and I switched IOL images to try and see if the images were the issue. Neither worked. I couldn’t update GNS3 at the time, either, because I was building several customer environments within GNS3, actively testing changes with customers on a schedule, and couldn’t take the risk that an upgrade would break anything. At this point, I left GNS3 where it was and went with EVE-NG. I liked the web interface for EVE and the ease of use, but I had similar issues with EVE as with GNS3 on the largest topologies, and so I decided that enough was enough. I couldn’t slow down my studies to troubleshoot small issues, I was losing ground.

Cisco allocated a certain number of INE rack rental tokens to me and I was back to study. Rack rentals were somewhat painful to use compared to the ease of running my own environment, but they worked every time, as needed, and I could focus on studying for my CCIE. I used rack rentals for the rest of my studies, except to stop and run through small parts of the ATC workbook that I was still having trouble with or needing reinforcement on.

RouterGods

I need to devote a section just to RouterGods, as their impact on my studies (and later, my life) can not be understated. Early on in my studies I had tried to form a group at work to study for the CCIE, and that had fallen apart. I joined the INE Spark (now Webex Teams) room, but that was not much help either. I had been going it alone for a good long time and wasn’t sure of anything, when Katherine McNamara showed up in the INE room to talk to people. Later, I would find out that she devoted time to this often, looking in the INE room for people who were actually serious about their studies, and telling them to sign up for a more focused study group, RouterGods.

So, basically, she’s a CCIE Valkyrie, descending from Asgard to the battlefields below to choose worthy einherjar to ascend to CCIE Study Valhalla. She’s awesome for spending her time doing it and I credit her, and RouterGods, with a large part of the reason I was able to find great study group partners and friends. RouterGods members were incredibly accepting, very helpful in answering questions and suggesting study habits, materials, etc.

Bringing it Together: Nearing the First Lab Attempt

I studied for the lab exam from July 2017 until January 2018, when I took my first attempt. Honestly, my timeline might have been aggressive, I think people who can’t afford to put 30+ hours a week to study, and didn’t already have years of experience to leverage would take twice as long. Around November, I started throwing around the idea of taking the exam as a first-try, but I felt like I needed better prep. I had only used INE materials up until this point. I hadn’t attended any bootcamps or other training besides the INE All Access Pass. I hadn’t looked at Cisco 360 labs or Micronics at this point. I would later come to regret not exploring either of those, but for now, I felt pretty good. I thought I would go in and clean up the lab, or at least get close. After talking to my manager, Cisco paid for INE’s Online Graded Practice Lab. This is a $500 practice test administered by INE that is supposed to be as close to the real lab as you can get without booking the real thing. I feel like INE has a really good product with the Graded Lab. Dave Smith is an excellent instructor/proctor and INE has some very challenging practice labs. The practice lab is done via remote VIRL rack, and it’s suggested to use SecureCRT for the practice exam to facilitate the quick-access that the real exam has. SecureCRT has an auto-login script that can be set up, so that when you connect to the remote VIRL rack, it can automatically log you into the device you need. In my case, I wasn’t aware of this ahead of time and had no time to go install SecureCRT on lab day, so I used Putty.

The Graded Practice Exam has a Troubleshoot portion and a Configuration portion, but there was no Diagnostic portion when I took it. It was a full 8 hour exam, though. For those on the east coast, it starts later than you think to accommodate those on the west coast. I think the initial setup for lab day started at 10AM and took an hour, so the lab itself didn’t start until 11AM. I finished at 7PM that night feeling absolutely exhausted. It was a real eye-opener in terms of the stamina required to just make it through the exam, to say nothing of the technical aptitude required to pass it.

I felt pretty good coming out of that practice exam, though. I felt like I didn’t pass, but I thought I’d get close.

I failed. But I didn’t just fail. I failed hard. It wasn’t even close. And this was the first time that I was supposed to learn a very important lesson, that I didn’t end up learning until after my first attempt:

Attention. To. Detail.

It’s not enough to make the packets flow. You have to do it exactly like the task says. “But, Tim, isn’t that obvious?”, comes the cry from the sidelines. The answer is yes and no. When I speak of attention to detail, I mean every last detail. If there are five bullet points in a task, and it’s worth three points, the slightest misstep on any of them loses all points.

Most of the errors on this score report were very small missteps. The CCIE Lab exam can be death by a thousand cuts.

Re-Evaluation

After this crushing result, I seriously re-evaluated my readiness to take the exam and went back to the drawing board. I did download the configs and build it to re-run on my own, along with the lab workbook.

I also started working on my own material. I built small labs to see what I could build, and then started making task lists similar to the real lab exam. My first try was only 10 routers, it was an MPLS PE-CE lab to set up traffic between disparate corporate networks, but it taught me a lot. Not so much about the technology, but about how to think like someone creating a lab exam. How to think about pairing tasks, structuring wording, etc, and this would be very helpful moving forward. It was a different angle of attacking the same problem – passing the lab exam, and I recommend it. I also stepped up my stamina game. Now, I was doing Full-Scale, 5 hour labs almost exclusively whenever I had the block of time. The lab exam is also about physical endurance and I knew I needed to work on that.

In December, I knew it was time. I was circling the same material, I felt a lot better about my performance and I wasn’t getting as tired or distracted on long labs. I knew that any further improvements I could get would be minor. It was time to try the real thing, and see how I did.