Rise of the Introverts

Many people out in the world have the potential to be professional guitarists, professional actors, professional dancers, the list goes on. Unfortunately for some people the natural talent still is subject to the crushing power and internal dialogue of the mind. The mind is our greatest asset and our greatest enemy. It sometimes acts like a magician, creating illusions and scenarios that may never happen in the real world. Sometimes it’s like a personal trainer, pushing you harder than you can ever go.

If you were in a band, would you be the front man, or the rhythm guitarist in the back? Some people just don’t want to use their talents and prefer to keep them as hobbies and just work in a job they fall into. Some people avoid uncomfortable situations and find an easy way out, they take jobs they are overqualified for and live in the world of comfort and happiness. Some people want success, they want to use their talents and fulfill their potential but they are held back by their own thoughts.

If you continue to avoid the scenarios that bring you distress and anxiety, it only gets worse as you worry everyday about something that hasn’t even happened.

Some people in the corporate world are impacted by this as well. Afraid of public speaking, meetings or social interaction. Some people are even afraid to talk on the phone.

The work from home scenario has changed the dynamic for me and maybe many other introverts. Some of the triggers have not been avoided but they have been shifted. If you don’t like speaking in public, you still have to but now it is in the comfort of your own home. Will this lead to introverts’ hidden talents such as leadership skills, conflict management and negotiation skills start to flourish now that they are in the comfortable cocoon of a home office? No longer are they controlled by the physical flight and fight response when standing in front of management and colleagues, exposing themselves to gazing eyes and our internal thoughts of “Am I good enough?”, “Am I speaking too fast?” or “Do I look like an idiot?”.

Will you or your fellow introverted co workers start to reveal their full potential during the new work from home paradigm? Have you already noticed some usually quiet people starting to speak up?

With this new found confidence, and practice of Zoom calls and Google Meets when we return to the office will the introverts now rise? Will they become what they always were so afraid of becoming? I think yes, because practice makes perfect. When you have a presentation or a meeting, you get prepared, you practice so what better way to practice than from home and live on the internet!

Technology has opened up new ways to communicate, I spend most of my days either on a video chat or a chat room talking, troubleshooting and fixing issues all across the world. I speak with senior leadership, vendors, colleagues, customers all from a simple chat room or video call. Five years ago, I avoided going into a meeting room with so many people. I would look for ways out, or just pray they didn’t ask me to speak out aloud.

The people I speak with daily now have no idea how tall I am, they have know idea how wide I am (LOL) they don’t judge me by what I look like, what I wear or how I stand they judge me but what I say and do. They judge me by my content and that is all an introvert wants. They don’t want to be put up in front of everyone like a statue to admire, they just want to contribute and be the quiet achiever.

Granted some people like the sink or swim approach, but with mental health at the forefront we need to find the triggers and ask each of our employees, if they are willing to share, what they can and cannot do. We then guide them, teach them and help nurture their talents in a safe environment. Not everyone has the stomach for this manufactured and down right unnatural environment we call the corporate world.

In this work from home world, the introverts will rise. Now they can achieve their potential, in a safe environment equipped with the confidence of experience when we finally return face to face.

Last but not least, this is usually a technical blog so if you are working on SDWAN and using Internet transports, make sure your ISP is giving you the best possible path internationally to avoid latency and packet loss. Australia to Japan via the US ain’t the best path :-).

~Brad.

This disclaimer informs readers that the views, thoughts, and opinions expressed in the text belong solely to the author, and not necessarily to the author’s employer, organization, committee or other group or individual.

Excuse the latency of my reply.

I have been very slack with my blogs this year; I have been busy trying to learn SD WAN as fast as possible and even going in depth with Wireshark and TCP. I am on both sides of the network spectrum, working with cutting edge and then rediscovering a protocol that has existed since the decade of my birth and still in use today.

I have had many challenging operational issues lately, and still searching for root causes in some of them. I hope to share some great stories once I get to the bottom of them.

To try and kick me into gear in the new year I have also decided to apply for the Cisco Champion program in 2021. I was last in this program in 2019 and won the Best Newcomer for my blog. I am hoping to learn and share what I can when if I am selected.

If I am not, then I just need to set some time each week to blog and stick to it.

In a non-technical path, my current manager in my region has got me onto a good book which I am also being a bit slack on reading, it is called the 7 Habits of Highly Effective people. I am about three quarters of the way through it but like most informational and self-help books it needs a few reads to become embedded in the mind. The last self-help I obsessed over and still think about, especially this year is Who Moved My Cheese. It is very simple but very effective in dealing with unforeseen changes and preparing for change.

I will try and post before the end of the year.

Until then, peace.

~Brad.

Dreams & Nightmares

It’s been a few months since I have done a proper blog entry. Many things have changed in my career, my life and in all our lives. The nightmare we all live at the moment is one of a pandemic.

Many, many years ago, my dream was to live and work in New York City as a Network Engineer. This dream, still partially alive but is now also partially fulfilled. 

As you can see and so can the world, we can’t travel anywhere due to Covid 19. 

But, virtually we can travel anywhere. I can connect to devices across hundreds of countries, scientists can control rovers on the surface of Mars and we can even stay in touch with Voyager 1 & 2 beyond our Solar System. My boss is 15000km from me, but I talk to him across the internet like he is two suburbs away. The world is more connected than ever, how lucky we are to live in a time of such great connectivity and be so segmented physically. 

I was lucky enough to be offered a role in a Global Team, my managers are now in the US. So, technically although I am not in the US my managers are and my team is all across the world. This is as good as it gets for now, but this may also be the future. 

I now have been given a chance to do what I have wanted to do, specialise in a technology.There are thousands of Network Engineers across the world being the glue to all other departments. The fabric to intertwine and connect people’s knowledge together to solve a problem. There will always be a need for a network person to join an Incident Management call, help with explaining a traffic flow or finding the location of an IP address. Every now and then we become overwhelmed with the amount we need to know, or we may have a spark that wants us to specialise in a field. 

My field of interest in this new role is SD-WAN. 

For the uninitiated, it is a new WAN technology which allows greater control and also is more an application based network, than a traditional network. Instead of configuring, managing and troubleshooting each router in the network, we can now separate the planes of the router and stretch them across a fabric. 

The control plane is now segmented and centralized. The Data Plane stays on the router and passes traffic based on control plane policies and the management plane is a GUI and suite of protocols used to bring this all together. 

Another way to kind of think about this is a Cisco 6500 switch. It’s a modular chassis and contains hardware modules that slide into a chassis. The backplane of the chassis can be seen as the management plane (allow modules to speak to each other and share information), the supervisors can be seen as the control plane (routing protocols) and the modules that do hardware forwarding (switching of traffic) can be seen as the data plane. 

Obviously you can’t run a 6500 with no supervisors at all, but was hoping this might give you an idea of how it has been separated.  

SDWAN routers can operate without any connection to the control plane, they will retain the last known configuration and routing information. 

For now, I am still using the CLi  for SD-WAN at this stage, although the possibilities of APIs, automation and using the centralized vManage is very exciting. 

The goal of CCIE is still front of mind, but by the time I actually sit it I am not even sure what the topics will be on it. Luckily for me, I am at the forefront of new technologies right now and working on them daily. This is also a nightmare scenario, so much to learn and not enough time in the day at work and at home with kids. 

So, I have my latest Cisco Press SD-WAN book and a new motivation and purpose.

For now, I just need to catch up. 

~Brad. 

vEdge Tools

I am new to the vEdge and spending most my working life on the command line, I find it hard not to use it when troubleshooting. SDWAN and other technologies seem to be pushing for the self healing network, the network that can fix everything itself with little or no network knowledge for administrators.

Although this is great for routing protocols to re-converge or traffic engineering that automatically re-routes traffic, sometimes you still need to send a telnet packet on port 5568 for that Dev guy in Finance.

On the vEdge, you do have access to some installed tools. For one troubleshooting event I had recently I had to test if TACACS packets were getting to the TACACS server. I can’t use the telnet command, but on the vEdge if you type the following some nice little tools are presented.

# tools ?

Possible completions:

ike-debug

ip-route

iperf

minicom

netstat

nping

ss

stun-client

vtysh

So, for todays blog we will look into nping. These tools have been around for a long time in the linux world, so they are not new but usually you have to install them on laptops or machines to use. Here on the vEdge you have direct access to the tools.

To craft a TACACS packet, using nping, I did the following –

(Don’t forget the quotation marks!)

tools nping vpn <xxx> options “–tcp -p 49” 10.x.x.x

xxx = VPN number of source interface being used

–tcp = use TCP

-p = use Port 49

With this command I was able to send a ping to the application port and see TCP Sent and RCV packets. It contains TTL, IPLength, seq numbers and window size, including the mss.

Check out the list of all options for nping here –

https://www.cisco.com/c/en/us/td/docs/routers/sdwan/command/sdwan-cr-book/operational-cmd.html#wp5736741660

~Brad

The NeverEnding Default Route….Part 2.

So, after the IOS upgrade the route is now stable! I have taken a new packet capture and the default route is still being sent to us, but now our router won’t recalculate and enter it into table.

I have informed Cisco and waiting to see if they have an explanation. My theory now is, the ISP sending the default is some type of soft reconfiguration or route refresh issue. On our side it must have been the same, where it was constantly rechecking and entering  due to a bug.

Until Cisco get back to me, I can’t do anymore testing as I don’t have such a lab.

~Brad.

The NeverEnding Default Route….

The NeverEnding Story….whoa…whoa…whoa!

Loved that movie as a kid.

Moving on and speaking about moving on I have been made redundant at my current workplace. I have one week remaining and ever since we installed a new Internet link, I been looking at a very peculiar default route issue.

The ISP, as proven by Wireshark is sending me a update packet for the default route every two seconds. This causes our router to re-install the route into our routing table, repeatedly. The initial troubleshooting began with changing the advertisement interval of BGP to 60 seconds. So now, the default route update packet comes in every 60 seconds.

Pretty obvious it’s a carrier issue.

I spoke with colleagues, Cisco TAC, posted on message boards and everyone is of the opinion that nothing on our router could ask our BGP neighbor to send us the default every 60 seconds.

Last night, the carrier showed me a lab they had built with the exact hardware in use. It doesn’t occur in the lab. I found out the IOS they are using for our device is newer, and the config is a little light. They are only advertising a default, where the real link is advertising a full BGP table and we are filtering.

Still, got me very puzzled. I did ask for a packet capture on their side, to see if they send an update packet to other customers but they decided the lab was a better avenue to explore.

So, tonight a colleague of mine is updating the memory and upgrading the IOS.

Will the IOS fix the issue?

Will the IOS upgrade stop the BGP peer from sending default route updates?

Will Batman and Robin escape from the Joker trap?

Stay tuned.

~Brad.

Cisco Community Post –

https://community.cisco.com/t5/routing/bgp-no-keepalives-and-two-second-updates/td-p/4048584

 

Powershell, the new Telnet!

As a network engineer, you usually need to prove that ports are open across the network. One troubleshooting step when working with TCP is using Telnet and accessing the remote host with the destination port. This is a quick way to ensure end to end connectivity is allowed, and you can safely update the ticket and send it back to the user for further application layer testing.

The issue is that a lot of servers and hosts don’t run telnet anymore, as it is a insecure protocol, clear text and not best practice. Usually the application owner will then request Telnet be installed so they can do testing and then probably leave it installed. Eventually a security scan will detect it and it will be removed.

I have found, that you don’t need to install Telnet on your servers. If you have access to the Powershell terminal, you can do the exact same test and it also gives you a better response. The Telnet test usually gives you a blank command line window, which isn’t very user friendly. The powershell test gives you better feedback and can be customized to ensure you can test your TCP protocols.

Once you have access to the powershell terminal, you can use the following command –

Test-NetConnection x.x.x.x -Port y

x.x.x.x = IP address or domain name

y = Port Number

The response is as follows –

PS C:\Users\ciscoworkerbee> Test-NetConnection google.com -Port 443

ComputerName : google.com
RemoteAddress : 216.58.196.142
RemotePort : 443
InterfaceAlias : Ethernet 2
SourceAddress : 10.10.10.10
PingSucceeded : True
PingReplyDetails (RTT) : 1 ms
TcpTestSucceeded : True

Here is the Microsoft page for more information.

https://docs.microsoft.com/en-us/powershell/module/nettcpip/test-netconnection?view=win10-ps

Happy troubleshooting.

~Brad.

 

 

Real World vs Exams – 4500 Chassis

First, you don’t know what device you are on in the Cisco exam. It’s just a simulator and testing your knowledge of the command line, not the specific hardware.  I recently had a module failure in a Cisco 4500 chassis, and part of the troubleshooting reminded me of doing a Cisco exam.

Let me explain.

The year was 2020, the month was January and work had just started back for the year. I had logged in, checking emails and the incident queue when I get a message appear on my google chat window. In the real world, people don’t log tickets they just remember you helped them long ago and reach out directly. It doesn’t matter how many times you say Service Desk they just come straight to you. I won’t say no or ‘ log a ticket’ because someone is in need and this person was in the IT department, so if you do him or her a favour then you will get one back. That is the way it should work.

The messages said wireless connectivity was impacted and some desk ports were not working. We also got some messages from our monitoring that some ports had gone offline.

I still had connectivity to the site and after logging in I found the following in the logs –

Feb 3 09:31:06 AEST: %C4K_HWPORTMAN-3-SUPERPORTMACLINKDOWN: Superport Mac link down on Superport 20 on slot 7.

Feb 3 09:31:06 AEST: %C4K_HWPORTMAN-3-SUPERPORTMACLINKDOWN: Superport Mac link down on Superport 21 on slot 7.

Feb 3 09:31:06 AEST: %C4K_HWPORTMAN-3-SUPERPORTMACLINKDOWN: Superport Mac link down on Superport 32 on slot 7.

Feb 3 09:31:06 AEST: %C4K_HWPORTMAN-3-SUPERPORTMACLINKDOWN: Superport Mac link down on Superport 33 on slot 7.

I took a ‘show tech’ and then I power cycled the module, which you can do on a 4500 chassis.

The module came back up and connectivity was restored, for about 12 hours. I was on call that night and got a message about midnight that ports had failed again. I reset the module and then went back to sleep.

By this time, I had logged a case to Cisco to check for bugs. That morning it happened again, although this time when I reset the module it never came back.

So far, this is all real world. TAC case logged, new module now shipped and then we do a swap out (under change control of course).

The new module did not work.

Not the best message to see –

Feb 5 14:55:03 AEST: %C4K_CHASSIS-3-LINECARDSEEPROMREADFAILED: Failed to read module 7’s serial eeprom, try reinserting module

It is now day two and site has a workaround for people connected to this module. The bugs and cases online point to a possible chassis fault. A new module and new chassis were sent to site.

Replacing the chassis is not an easy task, but we had remote hands to do this for us as this site was about 1000 kms away from me.

To ensure nothing was wrong with the second module, we first swapped a working module, so module 1 to module 7. Still the same message that it failed to read module 7.

It was chassis replacement time.

It took them about an hour to replace the chassis and plug all cables in (the cabling was very neat) and this is where it started to feel more like a Cisco exam.

The site had the 4500 connected to two 3750 switches, with EIGRP neighbors for connectivity. Two / 30 Layer 3 interconnects were provisioned between the devices for load balancing and I had connectivity to the 3750 via BGP.

I was on the 3750 switch, and it had been enough time for the chassis to power up and EIGRP neighbors to form, but I saw nothing.

So, check Layer 1 and it’s all good, ports are up. Check layer 2, CDP neighbors and I can see the device, its hostname and IP addressing. Configuration was loaded but I had no EIGRP?

I had connectivity from local site to the 4500, but I could not access the 4500 from the WAN. The default route was not in the 4500, as it was coming from BGP, redistributed to EIGRP and no floating static was used on the local site.

Time to ssh directly to the layer 3 interconnect IP and see what is going on. I was able to login with local credentials and found that EIGRP config was not even present in the configuration. Before I continue troubleshooting, I needed to get the site operational, so I deployed a floating static (static route with high AD) and it had to be higher than external EIGRP AD of 170.

On the 3750 I deployed a static route to the site summary address; this was then redistributed into BGP so the WAN could get there. So, for now connectivity restored and all modules were online.

So, why didn’t EIGRP show up in the configuration?

I found that due to the chassis replacement, the serial change causes the licence to fall back to the default licence. It had no layer 3 routing protocols supported so it didn’t load the EIGRP config. The licence file is linked to chassis serial and I needed a new one!

Once I had the new licence, I applied it rebooted during an outage window. I applied the EIGRP config and the external EIGRP default route was placed into the routing table.

So, in summary the real world is very physical, you must be aware of the hardware and how it operates.  The exam world will teach you the commands and theory, but it is real world experience that really makes you a network engineer and can set you apart from others.

In this example, not only did I apply some commands I had to deal with onsite engineers, troubleshoot hardware, perform diagnostic testing, liaise with third parties for equipment, organise change windows, log changes, seek approvals, copy current config and status, check and test after replacing equipment, apply licences and document.

There is not enough time in an exam to do all that 🙂

~Brad.

New Study Begins – Interested?

The first new study of the year will be Python. I have found this course below via Reddit which is free and directed at network engineers, which is handy for me.

This should really start the ball rolling for the CCIE Lab exam, as you are expected to automate during the lab.

If you are looking for something free and in IT why not give it a go?

~Brad.