How to validate PBR on Nexus 7K

On the Nexus 7k, when looking for ACL hits I was told to use the following command –

statistics per-entry

This would allow you to see matches in the ACL for traffic that has been routed via PBR. To date I have not seen this working, so I went searching for another command to try and validate my PBR ACL.

On the Nexus 7K, software version 7.2(0)D1(1) I found this command –

show system internal access-list vlan xxxx input entries detail

This shows entries in hardware when traffic has been subject to PBR. There are many different add-ons to this command, allowing you to see QoS statistics as well. I haven’t come across documentation on this command so I am only taking an educated guess on what it is doing.

If you look at the following output, you will see where a [0] is this has never been used, and any entries that have [xxxxxx] have been successfully PBR’d.

The first entry and final entries is added automatically with the implicit allow in the route map.


Tcam 1 resource usage:
Label_b = 0x207
Bank 0
IPv4 Class
Netflow profile: 0
Netflow deny profile: 0
[Index] Entry [Stats]
[001a:0010:0010] prec 1 permit-routed ip [9753]
[001b:0011:0011] prec 1 redirect(0x62)-routed ip x.x.x.x/22 x.x.x.x/16 [537353]
[001c:0012:0012] prec 1 redirect(0x62)-routed ip x.x.x.x/22 x.x.x.x/16 [12423838]
[001d:0013:0013] prec 1 redirect(0x62)-routed ip x.x.x.x/22 x.x.x.x/16 [582]
[001e:0014:0014] prec 1 redirect(0x62)-routed ip x.x.x.x/22 x.x.x.x/16 [8326043]
[001f:0015:0015] prec 1 redirect(0x62)-routed ip x.x.x.x/22 x.x.x.x/16  [3145]
[0020:0016:0016] prec 1 redirect(0x62)-routed ip x.x.x.x/22 x.x.x.x/16 [0]
[0021:0017:0017] prec 1 redirect(0x62)-routed ip x.x.x.x/22 x.x.x.x/16 [0]
[0022:0018:0018] prec 1 redirect(0x62)-routed ip x.x.x.x/22 x.x.x.x/16 [0]
[0023:0019:0019] prec 1 redirect(0x62)-routed ip x.x.x.x/22 x.x.x.x/16 [223736]
[0024:001a:001a] prec 1 permit-routed ip [302952013]
[0025:001b:001b] prec 1 permit-routed ip [0]

Hope this helps in your troubleshooting.



Port Channel Load Balancing

Found a handy way to check on the 3850 what port is being used to forward traffic over a layer 2 Port Channel. Recently we found some APs dropping off randomly, only a couple and it was linked to a Bug with a LACP port being suspended for a millisecond and then re-bundling.The worst part was no other messages. This is across a VSS core.

Seems that when the port goes suspended, the APs request a DHCP address again and the DHCP server fills up with BAD_ADDRESS entries. I believe that is related to this –

(Not sure if i need permission to post this guys blog?)

So, why would only a bunch of APs lose connectivity and not all? Only some of the APs were using the faulty port to communicate to the core based on the source mac address load balancing algorithm.

When entering this command, the switch will show you which interface is used –

show platform etherchannel 1 load-balance mac <src mac>















4am Phone Call – The answer is C.

Most IT related exams are multiple choice. I do remember my teacher once saying “When you get a call at 4 am in the morning, the correct answer isn’t C”.

Many years ago I was working for a company in Victoria, Australia. We had engaged an integrator to deploy a new WAN. The technology was GetVPN. This technology allows any site to talk to any other site via encrypted tunnels. These tunnels are not really tunnels but security associations between a source and destination. This technology is designed to be used over a MPLS network, a network which we call any to any connectivity. Traffic doesn’t need to pass through other sites, it can go straight to the destination.

This technology also relies on an underlying routing protocol to provide connectivity to all sites before the encryption takes place. I won’t go into any more detail, but I will try to capture what happened that morning and the intense pressure we felt.

I don’t recall the date or day, but I do recall the time.

4 am.

My colleague was on call, and he was the lucky one to receive the first call. All sites down, over 120 sites to be exact. This was not a normal outage, this must have been a change or something surely? What would take down such a network?

Now, maybe the configuration that was deployed was standard (actual Cisco configs from the configuration guide…..profile id 1234 lol) maybe, we had an underlying carrier issue? Nope. Sites were actually dropping in and out and getting the customer to reboot the router brought it back online…only for a few minutes.

The call came to me, I was the ‘one’ who had the best knowledge of GetVPN in the team. My colleague had worked out it was a GetVPN problem, an encryption problem. Yikes! That sounded technical and difficult. I spent many hours reading and learning about GetVPN when it was deployed at our workplace, but I still was no expert. See this technology relies on two very special routers, known as key servers. These guys are the backbone of the network, coordinating encryption keys to be handed out to every node. Depending on a configured time, they refresh and everyone is using the same key. If you have some nodes using one key and another using an older key, guess what happens? It’s like a Chinese person talking to Indian person, they can’t understand each others language.

We tried many things that morning, while I laid in bed on the phone, we rebooted key servers, got the customers to reboot any routers they could. Still nothing. That was about 5:30am. That was the time I decided we have to go into the office before everyone in the IT office gets there.

I got in the car and headed straight for work.

I was frantic in the car, still on the phone trying to work out what the hell happened!

We needed more information, we needed data to troubleshoot. Although we could not access the routers remotely to gather this data.

Arriving at work, it hit us. We have a lab. A sweet sweet lab and it was in our office. We could troubleshoot from here! We spent about 30 mins trying to debug and find the cause of the issue, but by this time it just all started to look the same. Managers started to come in, and demanded answers. They were not harsh, they were helpful but the entire WAN was down. We had to give constant updates…we are with TAC. Sites had to go to manual processes with absolutely no connection to the Data Centres. Phones didn’t work. No email no nothing. Imagine sitting at your house and you phone and internet was down and how annoyed you would be. Now multiple it by 120 sites and maybe at least 10 people per small site and 200 + at 5 large sites across the country. Is the correct answer C yet?

The next step after gathering all the logs, was Cisco TAC. This is technical support, from Cisco themselves. The experts.

I made the call and we got a guy from Texas. He was a GetVPN expert.

He was able to connect to my PC via Webex and found our first problem. Encryption was broken and when this happens you need to make sure that certain protocols in GetVPN are not encrypted, ever. This is so you can build the underlying connectivity using a routing protocol and also in case of an encryption problem you can still manage the routers.

Routing updates, ping and SSH should not be encrypted. SSH is already encrypted anyway. We modified this on the key servers and suddenly we had SSH access to all routers. More troubleshooting continued.

He found the problem, thank god.

Colleague performed a change on all remote routers a few days before. It was to update SSH keys for remote routers. Although by accident it included the key servers. It took the current generated keys used for both SSH and GetVPN encryption and removed them. During the morning connectivity was lost between the two key servers and they both became master. Then connectivity restored, but it was too late. The remote routers were still using the old keys from one key server and connectivity was lost. (as you can imagine it was even more technical than this, but this is all I remember).

Did you ask about Change Control? Yeah it was followed for the SSH key generation, but to be honest even with all my reading I still knew jack shit about GetVPN. The only way I really learned was when it broke.

So….make sure you lab things. Make sure you get your hands dirty, even if it is a virtual lab. It is the only way you will learn anything in life to be honest. Don’t be afraid to break things in the lab, watch what it does when it breaks and what it does when you restore it.

Don’t ever be afraid to ask for help, you will always learn something. Don’t ever give up either, if it has been broken then it can be fixed and don’t memorize the answers A,B,C & D because the question hasn’t been written yet in the real world!

So, the correct answer was not C, not all of the above or even phone a friend! It was when you deep down in the shit, escalate and ask for help. No one can be an expert at everything!


Original posted on my personal blog in 2011.

Cisco Champion 2019

Late last year I was pondering my New Years resolutions. How can I achieve my goals for the new year? How can I make my dreams come true, when suddenly I realised I didn’t have any new goals. Since being in the industry I have been wanting to go for my CCIE, but every time I think about it, I doubt myself. I still feel like I don’t have enough experience, although I have been in the industry for 10 years.

I think there is a big difference between 10 years experience and 10 years of experience, not including the first two years where you were trying to understand how you can take a packet capture and see a QoS (what’s QoS?) marking of a voice call and then play that voice call back whenever you want. Amazing…and how do you tell that phone call to go first when it reaches a point of congestion? What I am trying to say is the CCIE is an expert level certification and I really want a solid 10 years of expert level experience.

My latest mantra I have been using is basically from the movie “Yes Man” with Jim Carrey. It is about a man who says “no” to everything and I have found that not only does it hold you back from experiences, it actually can give you anxiety. It’s one of those illnesses that has been around since we were cavemen but only now do we talk about it freely. It is designed to help us in the time of the dinosaurs or the Megalodon and can really cripple us as we try to achieve greatness in the modern world where threats only exist in your mind.

So the mantra, is say Yes! Say yes to everything! No matter what it is…within reason of course (nothing illegal folks!) and this is where the Cisco Champions program enters my life. For many years not only have I wanted to be a CCIE I have also wanted to be a teacher. I unfortunately did not go to university, so i guess the correct title would be instructor. The Cisco Champion program allows me to blog, learn and educate people across the globe as I work in my career as a network engineer.

I have seen a few people who I watch on YouTube be selected that have been blogging and learning for many years have been selected. So when I got the email, the little monster spoke into my ear, no you’re not good enough. I spoke with my wife and pondered a few days and then the mantra kicked in…say Yes! no matter what and here I am!

So I am very grateful to be selected as a Cisco Champion for 2019. I will be blogging a lot more this year as I continue my studies and work in my full-time job. I have a lot of interesting projects coming up this year, including more cloud technologies and Nexus/ACI work so I am excited to learn and try to pass on this information to other people in the industry.

So for now, this is just the beginning to a long, knowledgeable and exciting New Year. I also better get a new Cisco Polo, overindulged over Christmas and she may not fit 🙂






Network Engineers don’t wear suits?

I remember a 21-year-old CCIE telling me this. “Network Engineers don’t wear suits!” He had come out of a boot camp and had the world at his feet. The CCIE is the most sort after certification by business. It stands for Cisco Certified Internetwork Expert.

Although, instead of configuring routers and logging onto devices all across the world he was in customer meetings. Gathering requirements and trying to design networks that would allow better productivity and lower costs. His technical knowledge was in his head, but see they only looking at his face, his suit.

This is not what I was prepared for when I entered the world of networking as well. All I cared about was not having enough technical knowledge! I didn’t know the ins and outs of a 6500 switch or configuring MPLS! But I can speak to people. I can relate to people, I’m a people person!

Or so I thought…..

I had a bad experience once….

It put me off the suit….

It was a meeting where I wasn’t prepared and the good news I was giving the customer that nothing needed to be done, I thought it meant I really had not been doing anything the whole time I was there. I should have realised it was good news and put that spin on it. The meeting started, and my mouth opened. A few words came out something something ‘nothing is needed in your network’. Then….


I took a breath and closed my mouth. It wouldn’t open back up. I froze. How could I freeze? I was a new engineer worried about BGP , routing tables and not knowing enough technical information!,

I can talk to people. I can relate!

I got it wrong.

People skills are probably the most important part of a network engineers job.

That experience put me off for sometime. A long time actually. It wasn’t until I read the book ‘How to win friends and influence people’ when I started to learn the best way to deal with people. The book Who moved my cheese also made a big difference. Those books transformed my outlook and my life.

9 years has passed since I met that 21-year-old CCIE and 8 years have passed since that meeting.

The chapter I should have started 8 years ago starts today.

I should have worn the suit first.


Network Engineer Interview.

Recently I have been struggling with my career choices. I am a little confused about what I really want to do for the rest of my working life. Originally my first goal was just to wear a Cisco T-shirt to work! I now do that as often as I can to remind myself of my first goal.

My second goal was to be a ‘Core Network Engineer or Network Design Engineer’. I am currently working towards those now.

My brothers friend of mine recently contacted me to do an interview for her students at a TAFE in Victoria. I was more than happy to reply and help out upcoming students. This made me think back to when I was in TAFE. My teachers were great and really pushed us to get the best score possible. I was even asked by my teachers to apply for a Network Scholarship, although it wasn’t meant to be.

I did pass with pretty good marks at the Cisco Networking Academy and even though my worklife has been completely different than TAFE I have learnt some valuable lessons. Maybe one day, I could be a Cisco Networking Academy teacher?

Below is the interview I did for the friend. I hope it does help the students in some way.

  1. What kind of qualifications do you have?
  • Advanced Diploma of Computer Systems engineering
  • CCNA
  • CCDA
  • CCNP
  • CCDP
  1. How many years have you been in this industry?
  • I started in 2006 part time while still at TAFE, so about 9 years.
  1. What does a day in your job look like?
  • My current job is network support for mining companies. Typical day, if not on call is arriving at work and checking my sites I have been assigned. We use some monitoring systems which I can quickly check to see if any links or devices are down. Once that is complete I have a few projects I am working on, so I may have to check on them. Day to day I would be either making changes to switches or wireless infrastructure. Although to perform changes, we need to follow a change control procedure. We can only make on the fly changes if it is an emergency.

I usually have to speak with IS managers on the mine sites to get approvals to perform my changes. The work comes in via incidents (service desk) or by my manager. It could be a new VLAN to be pushed across the wireless network, a IPSec tunnel from our office to other remote offices or configuring an autonomous AP for wireless access in a piece of mining equipment. I might also receive small requests to update access control lists or adding new subnets to routers. Some bigger projects may be taking over management of existing switches and cleaning up configurations.

Usually when a major outage happens we need to do the troubleshooting, check power and comms to WAN router then go from there. Most incidents are related to power in the mining industry or unauthorised changes.

  1. What’s the worst network attack that you’ve ever seen?
  • I worked for a small ISP a year ago. We had a few different Internet pipes and we supplied either a secure internet (firewall in front of customer) or a non-secure (direct pipe to the internet). The customer would then need to provide own firewall. One night I was on call and started to receive some alerts from multiple customers. The firewall was being attacked, millions of half open TCP sessions where being created within the firewall and the firewall could no process the information. This caused the firewall to drop its routing neighborship with our PE router (Provider Edge) and took out every customers internet for about an hour. Manual intervention was required to black hole the traffic, direct it to Null0 (destination that doesn’t exist) which stopped the traffic flow. The final fix, was to install a IDS (intrusion detection system) to automatically detect this type of attack and block the traffic before it made its way to firewall. One other attack I saw is Crypto-locker. Comes in an email and encrypts peoples hard drives so they cannot open it without paying a ransom to the hackers themselves to unlock.

If you want to see people trying to hack networks right now, the Norse corporation has deployed Honeypot servers (devices that look legitimate but are not) and they monitor attacks on this website –

  1. How much do you get paid per year?
  • 9 years ago when I started full time I got $25 dollars an hour while still at TAFE. I didn’t have my CCNA or any experience. As a contractor you could make $50 – $80 an hour as a CCNA with a couple of years’ experience back then as well. These days contracting pays the best, but there is no guarantee or work stability. Check out the current Hays Salary Centre for current rates depending on years’ experience. It also depends on the work you will be doing. Design & consulting pay the best.
  1. How do you set up a physical hardware firewall device on a network?
  • This really depends on the network design itself. Best practice is to have dual firewalls. You can deploy them in the active/passive configuration which means one is doing the work and the other is sitting there ready to take over. The active/active configuration is where traffic is shared or load balanced across the hardware. This is also dependent on the hardware itself as it needs to be able to support this configuration.
  1. What is the most difficult task that you have dealt with?
  • I had to install and troubleshoot a new wireless network. Unfortunately there was no wireless site survey done, they just installed Wireless AP’s where they thought it would be best. It took me a long time to try and get the network running and stable. They also had voice over the wireless which was not taken into consideration as well. Most enterprise wireless networks are controlled by a centralised device that can automatically change power and channels. Although if the RF environment has not been mapped out correctly, you are going to run into problems. Interference, rouge wireless AP’s and incorrect settings all played havoc with this network. I recommend always following the best practice guide lines when deploying a wireless network.
  1. If a client reports a wireless dead zone in their building what do you do to find the extent of it and then how would you fix it?
  • If the wireless is being monitored, a check of AP’s in the area and if they are all online is the first step. The next step would be to go to the area affected and use a wireless scanner (network Stumbler) to measure the signal strength of the AP in the dead spot. This will determine if the power level on the AP is high enough to service the area or with visual inspection you may find something that is causing interference. It could be the physical environment, a rouge AP or another wireless device. Depending on the wireless frequency you may have another AP in the area trying to use the same channel, a scan will pick this up.
  1. What is the most extreme problem you have come across?
  • Anything to do with entire WAN networks going down. Usually the WAN once deployed should be stable with redundancy in the design. The WAN is critical for major companies that access resources in the data centre. I was on call once and got a call at 4am in the morning. Half of our WAN started to fail, people could not access the data centre or internet, and out of 120 sites only half where working. It took 4 hours to fix this problem. It was caused by a change that previous night to all routers in the organization regarding SSH key generation and how the WAN communicated with encryption. We had to escalate to Cisco TAC (Cisco’s Technical Support Team) and an engineer from Texas found the issue and rectified. It was the worse outage I have ever had a phone call about.
  1. What is the most common problem you come across?
  • Problems like incorrect VLANs or duplex issues. People are in the wrong network, or the cabling is ruined causing major packet loss.
  1. How do you set up a wireless repeater/extender to get better signal in a room?
  • Ha! I have never actually used a wireless repeater! People use them in their homes, but enterprise access points are a lot more powerful regarding antennas and coverage. Usually in the industry it has been determined beforehand during the design process. If someone came up to me and asked me to deploy one, I would check the Cisco site and follow the instructions.


Step Up.

I been doing the same role for almost 9 years. Its a support role, so helping people and fixing problems.

I like it because I like to solve problems, I like to troubleshoot. I like to come across something that has never been seen before, discover and gather data and then try and resolve the issue.

I used to work as a Network Engineer for an integrator. That role was more projects and deploying networks. I left that role to go back to support because I wasn’t really ready to design networks and be a consultant. I don’t know much about business to be honest and that is a major driver in network design. You need to take business requirements and turn them into technical solutions.

I was in a support role and we had an integrator come in and he was doing some voice work for us. We got to chatting and I told him my history of working in projects and going back to support. He said to me, I can’t believe you would go back to support! At the time though I was happy.

So, its been nearly 9 years. I have done on call for about 7 years of that. On call isn’t my favourite thing to be honest. I feel like I have done my on call now. I feel like its time to move on. I feel like its time to step up.

About 7 months ago, I was in a hole. The kind of hole that you grow a moustache for in November. I wasn’t motivated and my career goals I had forgotten. I wasn’t in a good spot. This was also the second time I was here.

You get to a stage when you have exhausted all though processes, you have analysed everything and there is nowhere to go. That is when you need to go and seek help.

I did…

It was a tough road, as it always is and I am still on that road. But now, I have my motivation back. My goals are energised, I sit here trying to work out how I can achieve them. What can I do right now to realise my career goals.

I’m hungry god damn it. I want to work again. I want to learn again. I want to push myself again.

I want to complete my goals and I want to be the person I was destined to become.

I’m not just going to step up…no.

I am going to stand up and never sit down again.


Cisco Certified Father.

I write this to let off some steam, I write this to take a break and motivate myself.

Recently, as in 7 days ago I became a Father for the first time. It has brightened my life and my direction more than I could ever imagine. In the last 3 months of the pregnancy I have been studying to sit the CCIE Routing & Switching Written Exam. I need to re-certify my current Cisco certs and wanted to attempt this exam after failing it a few years back.

I enjoy sitting new exams so I can learn new topics and new skills. My ultimate goal is to sit the CCIE Lab to become CCIE certified. I will attempt this maybe in the next year or so I think.

I have until July to pass this exam, so I have time up my sleeve, although I am on four weeks leave and trying to study while looking after the new born. It is proving to be difficult, so I wanted to explain my past and now present study tactics I use.

During the last 3 months, I watched a CBT nuggets every morning on the train to work. On the way home I would read a CCIE Exam Cert guide. I used to watch videos for one week, books for one week and then labs for on week but this isn’t working this time.

I am pretty tired and was planning 2 hours a day during the week for study, currently I’m getting about 1 at the moment. Its not going to cut it and the exam is at the end of the month.

My new plan, I am doing the Boson Ex Sim CCIE written practice exams. These exams are really tough and I can’t seem to break through the 70% mark yet. If i do fail, Boson will pay for the resit, as they guarantee a pass. At this stage its looking likely they will fund a resit. I find myself wondering, do I know enough and these exams are too hard or I am way off?

I decided to go back to my old T-SHOOT simulator through Boson, to check my knowledge. I did the first four simulators and got them right. I have never had an issue troubleshooting complex networks.

So here I am, 3 weeks away from the exam and I am going to have to change things up.

To motivate myself, I will put my new plan and some advice I have heard once before here. It may help other people when sitting exams and also re-motivate me.

  1. Always read every possible answer before just diving into what you know is right
  2. Remove the answer you know for sure are incorrect, leaving better odds on a guess
  3. When clicking start on the computer at the exam, write down everything you can from memory. Especially subnet masks and wildcard masks. Also binary and hex tables so you don’t waste time on them.
  4. Take your time and don’t panic. Try and use real world examples when you were troubleshooting a protocol in the work place to help your logic.
  5. Never give up, learn your weaknesses and address them.

My current study plan is now –

  1. Small study till the end of this weekend (practice questions, chapters here and there)
  2. Starting Monday, One Video, One Lab and One full practice exam. After the practice exam, right down the troubling topics and use them the next day.
  3. For Monday I will be studying NBAR, IPv6 NAT and SDN.
  4. Never give up, learn your weaknesses and address them.

Hopefully Freddie (my son) can get some sleep and so can I to prepare for next weeks study. I am really one week behind and it was crazy to think I can study while looking after a newborn! But I am dedicated and I really want my CCIE. I want to really push myself to become one of the top engineers so I can one day either work for Cisco or become a Cisco teacher.



Note: Originally published on LinkedIn on March 10th, 2017.

Third times a charm.

Unfortunately I was unable to pass on my second attempt at the CCIE written. It was tougher than the previous version I attempted three years ago. I used the Boson Exam Simulator and I personally found it useless. I was able to get a refund on the software but it didn’t really cover the $580 for the CCIE written.

All is not lost, the study I have done has prepared me for the CCNP route exam which I will attempt shortly to re-certify my current Cisco certifications.

When I was in TAFE many years ago all I wanted was to wear a Cisco polo. It might sound stupid but every time I wear one it reminds me of the goal I set out to achieve. This new goal is still there, I will master it one day but for now I need to get back into the books and labs. I won’t attempt it again until I mastered every subject. The CCIE I do not take for granted, some people have said it’s the PhD of Networking and if I am to get the that level it will take every ounce of who I am.

The long term goal is to specialise in a technology, to truly become an expert. Once I have decided on a technology and that goal is achieved I would like to spend my days teaching new networking and IT students.

So for now, it’s back to the books I go.



Note: Originally posted on LinkedIn on May 2nd, 2017