SD-WAN

My recent trip to Darwin exposed me to my first ever SD-WAN router. The Viptela vEdge 100, a small branch office router. Although this wasn’t a full SD-WAN deployment yet, I was able to upgrade the software and deploy some initial config so the experts could jump on and get it going.

Because SD-WAN is actually using Internet links, once your public IP is assigned and your device ‘calls home’ it is accessible.

The device is running in a type of hybrid mode so to speak, with a connection to vManage  and vSmart devices, and some static IPSec tunnels to some firewalls for non SD-WAN connectivity.

Was a very good learning experience, with the first being the USB cable, which isn’t your standard Serial to USB. It was a specific USB cable which require a specific driver. Also, the power supply and connectors don’t seem to be ones you can pick up at a local Jaycar.

The commands, a little different and the terminology as well. I am currently working through some documents to work out what a TLOC is and a Color. The VPN, seems to be a VRF so that’s not too difficult.

Then there was the routing table and the mysterious OMP protocol. It has a AD of 250, so don’t forget that for the new CCNP! Briefly looking at it, its route selection is similar to BGP.

These little routers run pretty hot as well. They have no internal fan it seems.

After playing with the command line a little and assisting with troubleshooting, the basics of routing and IPSec are still in play here, it is more the way the router becomes part of the SD-WAN fabric that I need to understand.

I also was looking at what I thought was the BGP table, but was wrong so don’t forget show bgp routes for that.

Finally, I just found out that it relies on BFD and can’t be turned off, so this allows extremely fast failovers within the Fabric.

I will have more once I get a lab up and running. Site is now live and can’t be messing around with a production network!

~Brad.

 

 

 

 

 

 

 

Cisco SG300 – Helper

Just a quick post for today, wrapping up in Darwin and will blog a bit more about it next week.

It took a lot of hours and a lot of people to get the site working, so plenty to blog about.

The first issue I found when the site came online though was I couldn’t a DHCP address. When I pre-configured the switch in the office I applied the ip helper address to the VLAN interfaces and I hit a limit, which I though was kind of weird. So I couldn’t apply helpers to all my VLANs.

After some research I found you could apply a global command. I applied it then sent the switch to site.

So, when troubleshooting on site I looked at the helper command and noticed that it automatically adds the ports it does the helper function for and it didn’t include DHCP ports 67,68.

If you are unaware of what a helper is, it forwards packets that usually are destined for a host within the same VLAN like a DHCP server. This allows a switch or router to respond to the broadcast and forward on your behalf to a device off of the local VLAN.

After checking the Cisco site again, I found that I was using the wrong command. The SG300 actually uses the DHCP relay command. It must be enabled globally, with the DHCP server addresses and then assigned to each VLAN with the command –

ip dhcp relay enable.

Hooray, I could get an IP address from a remote DHCP server.

One small caveat, if you wanted to configure a local DHCP scope on the switch as well, You can’t when relay is enabled, it’s either all DHCP local on switch or remote DHCP.

So, in closing some good learnings, always check the documentation and if you need more functionality this switch isn’t for you.

~Brad.

Birds of Prey…flying high!

I am currently at 38,561ft above sea level over the Northern Territory enroute to Darwin. My ping is about 580 ms to our Data Centre and I am working with a few colleagues on some tasks for a deployment this week.

I have Internet, entertainment & Facebook but Qantas still won’t update the headphones from 20 years ago? My headset also doesn’t work, so I ended up watching an episode of Quantum Leap on my iPhone, streamed from 9 Now.

I fly in a great mechanical machine, as one would say if a time traveler from many decades ago arrived today.

This week is amount doing whatever it takes to complete a project. It’s a long story and has many technical challenges. I am not even sure if the site is 100% ready for me but it has already been delayed sometime. I am half way through a design document, and also still working on parts of it right now. It is not the way I like to do things, but this is a good lesson in perfectionism.

I do want everything perfect, but the world just isn’t like that. I wanted control, but I have a team I work with and I rely on them as much as they rely on me. In the end, as many networks have been deployed, it will work. This is the reality of working in the IT world. It is not an exam, it is not a lab with the right and wrong questions. SO the priority is to get it working as quick as possible, deploy best practices, secure it and document.

We started with a firewall and two internet connections. Now we deploying SD-WAN on the fly, well SD-WAN routers but not true SD-WAN as it isn’t ready yet.

It’s gonna be a hell of a week.

I also got CCTV, Wifi and other goodies and have to get it working in three days. Yippee.

Worst case scenario is I get it all connected side by side, and cut over on Wednesday.

Keep an eye out for some good troubleshooting and tips after we get through this week.

 

~Brad.

Attention ASA Users, lend me your virtual ears.

The week before Christmas, someone on the Internet decided it was a good idea to start rebooting externally accessible Cisco ASA firewalls. Not great for me, office was closing and I was on call.

I spent a lot of hours fixing it, working with multiple parties and vendors.

Turns out, this vulnerability is quite old. This was my first actual real experience detecting and mitigating an attack and it was actually quite fun. Kind of felt like the nerd guy in the movies that seems to be able to connect to security systems and bypass it in seconds.

This is how it all went down (the process, and the firewall).

Early Friday morning I saw four ASAs reboot, 2 per DC. Thought that was kind of weird, as usually you suspect power or someone is in the cabinet knocking cables. Asked the question and all was good. Very unlikely people were in the two DCs at the same time, accidently knocking the same cables.

I only had two other theories, a time based bug or a network attack.

Either way, both of those theories would mean that many other people might be impacted, so I decided to log a ticket to Cisco and then posted on both Reddit & the Cisco Support Community.

https://community.cisco.com/t5/firewalls/multiple-asa-reloads/m-p/4002229#M183484

Well, some people started to speak up…saying they started to see this as well. The ASAs were rebooting every hour in the afternoon like clockwork. Because there was a large gap between the first reboot and the afternoon, I suspect an attack instead of time based.

Cisco responded with a security vulnerability and I had the joy of deciding what software to upgrade to without impacting our setup and not testing. In the end, I decided it was best to go to a supported release, not the latest maintenance. We upgraded and ‘fixed the glitch’ (for you Office Space fans).

https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180606-asaftd

Security teams are now scanning the public IPs seen and trying to locate the attackers.

Highlighted once again the importance of upgrading software for not only bugs, but security vulnerabilities. There is thousands of attacks happening all over the world everyday, and with the current situation in the Middle East, you can expect some new ones in the coming weeks. War doesn’t just occur in the physical world, the virtual world is at risk as well.

So lend your eyes, to your monitoring systems.

~Brad.

 

Winner Winner Chicken Dinner!

The Senator Palpatine meme, ‘a surprise, to be sure, but a welcome one’ comes to mind.

This is a nice way to start my first day back at work, so I better learn something new today and start blogging!

Thank you to everyone who voted and reads my blog.

Full list of winners is here –

https://www.cisco.com/c/en/us/training-events/events-webinars/influencer-hub/blog-awards.html

~Brad.

2019

This will be my last post as a Cisco Champion for 2019. I enjoyed being part of the program, although due to the time difference with the US I missed out on a lot of stuff. Most of the presentations were in the middle of the night and as a father of two young kids, I couldn’t make it work.

Hopefully in a couple more years I can re-join, or maybe one day I could fulfil my dream or working at Cisco itself! My original goal when I started in the Networking field was to work as a Network Engineer in New York. I have been to the United States a few times on holidays and always wanted to return for work someday.

At this stage, I would be happy to even attend a conference in the US, as my family commitments have changed since I have a wife and two children and, moving them all to the US isn’t in the best interests of the family unit.

This brings me to next year’s goals.

Currently I am pondering the following two fields –

I have been a ‘Network Engineer’ for many years, and I really wanted to achieve the role of Senior Network Engineer. I think this would be a great introduction into achieving the long term of goal of a Network Architect. Even to work within a group of architects would be enough for me now. Unfortunately, there isn’t one where I work. We have Enterprise Architect and Cloud Solutions Architects that work at mostly Layer 7 and 8 (People).

Every Network Role I have had, I have managed firewalls. Firewall Rules & VPNs mostly. I have always enjoyed security and what it stands for. It is very black and white, either its secure or not. Either you trust it, or you don’t. I like these boundaries and it helps create process and procedures, which creates accountability and what is right and wrong.

Security is exploding, it’s the next field that we don’t have enough people in, and I think would be very welcoming to a person with a networking background.

I will continue to blog as well, hopefully with more teaching topics. I really do enjoy learning something new and then telling others about it. I like to share my information and want others to succeed.

I haven’t switched off just yet for 2019, I am on call a little over the break. But when I do get a chance, I will ponder next year’s goals and resolutions.

Interestingly, in Feb 2020 my certifications which I just renewed will change as Cisco have revamped their entire certification stream. The CCDA & CCDP is being absorbed, so I won’t have as many. I will gain the new CCNA & CCNP certifications though.

I don’t think it is quantity though, it’s all about quality!

Merry Xmas & Happy New Year!

~Brad.

 

 

 

 

 

CCNP/DP Recertified – now what?

Ok, another recertification out of the way. It will be another 3 years before I need to do it again, but things are changing.

The new CCIE, is very different to the one that I wanted to get when I first started in Networking. I was pretty keen on attempting the CCIE written again next year, as I have failed it twice so far. Now I have no idea what to study and where to begin.

What was Routing & Switching CCIE has now disappeared it seems. So, I cannot re-sit that written exam again and have a case of third time lucky.

The streams are now –

  • CCIE Enterprise Infrastructure
  • CCIE Enterprise Wireless
  • CCIE Data Center
  • CCIE Security
  • CCIE Service Provider
  • CCIE Collaboration

I have already removed Collaboration, I don’t like Voice and Video deployments. I like working on the networking, the QoS, and the Multicast if needed but I don’t like administering the backend and working on phone systems.

I love Service Provider, but I don’t work for one anymore and it’s a limited space in Brisbane. It is my favourite of all the CCIE, but I don’t have enough experience in the core of a Service Provider. They are heavily automated now as well. Usually when you go to work for a SP, you get pimped out as a consultant and selling stuff. Sales, is not me either.  Data Centre is awesome as well, but most companies are sending to the cloud…which is someone else’s Data Center! I don’t see many cloud companies in Brisbane either!

I have to be really careful in what I select, I want to do something I have had the most experience in. Although my experience has been so wide spread lately. One minute I am deploying Multicast, the next I have to start learning Viptela SD-WAN CLI to help cutover a site, or lab a Palo Alto IPSec Tunnel.

The two topics that have been constant in every workplace is Wireless & Security. I have deployed Wireless in both Mesh, Standalone & Controller based setups. I have never had formal training on firewalls but I have touched ASA, FWSM, Palo Alto, Juniper and Checkpoint.

Security and Wireless, is not going anywhere. That I am certain of, as we advance our digital footprint, there is someone that is following those footprints…waiting for the right moment to exploit or take your information.

We are all mobile, so wireless is here for good. It’s always been here since the invention of the television, and as we talk to our distance spacecraft that land on Mars or our automated vehicle, we can’t have a cable attached.

Same kind of goes for Routing & Switching. We must route IPv6, existing IPv4 and send traffic the quickest path within the carrier networks. Switching is changing, spanning tree is disappearing but still the same at the enterprise edge for now.

The biggest change for me, is automation and programmability. I don’t know about you but I find coding incredibly boring. I can’t be creative with it like some other things I like to enjoy. The typing, the logic it’s just one giant maths equation to me.

But, we must change to grow so I will begin with this. It is something new and maybe once I get the basics, I could be creative. The problem is I have to start at the beginning again. The programming I did in school, I don’t even know how I passed. I still have dreams to this day that I haven’t finished my programming assignments and fail the class and my course.

It seems that Python is the way to go for networking, so that’s where I will start. It also seems to be all through the new CCIE exams, so that’s a step in the right direction.

And  when I say start from the beginning I mean, the beginning.

sys.exit(~Brad.)

 

 

 

 

 

 

 

 

 

 

Cisco CLI Analyzer

Have you used this? If not…download it now.

Before you log a TAC case and be told to update the software (90% of the time lol) log onto your Cisco equipment with the CLI analyzer.

It gives you a powerful set of Cisco TAC tools right at your fingertips. It also helps with understanding commands, best practices, bugs and saving configurations.

https://cway.cisco.com/docs/cisco-cli-analyzer/2.0/New_Features.htm

In the words of Maury Finkle, founder of Finkle Fixtures, Biggest Lighting Fixture Chain in the Southland, do it.

Do it.

~Brad.

 

Passed on what you have learned….

The one problem we have in the IT world is documentation. No one seems to like doing it. Unless you specifically pay for a design document and as-built, you are going to get a document that usually has a lot of cut & paste from official vendor documentation.

Some of the design decisions were not captured, the implementation may have been rushed or not scoped properly and this leads to a ‘just make it work’ scenario.

But at what cost? The poor support team? The customer that paid top dollar and didn’t get what they paid for?

I guarantee that when you get a tradesperson in to build you a kitchen or a house, you would be checking every corner, every wall to make sure it is what you paid for. Why does this not occur as much as it should in the IT world? Or maybe upper management believe it is happening, but at the coal face it is not.

For me, this isn’t right.

Until we have a system that will scan the network and automatically build an as built document (once I learn python, I will try!) we are stuck with people retaining information in their heads and not documenting designs.

This seems to occur more in the enterprise space as well, due to external vendors being used and scopes not including such vigorous documentation.

Some believe this slows down the process and the build as technology must move at the speed of business these days. This is all achievable if you just make a few extra steps at the start of the project.

Steps like a clear scope of works, measurable and provable outcomes from each technology and then allow time in the budget and project for these to occur.

So, my philosophy is as follows and really should be common sense and the basics of an operational network. Its based on a document I encountered a few years ago. I have made it high level, for easy reading.

1. Customer provides a clear and concise document outlining what they require.

2. Customer provides a document that highlights each technology, what they expect and the expected performance of this technology.

3. Project team prepares a POC or high level design document, POCs can be dangerous, by the time it works the customer expects it to work the next day and suddenly you have a POC GONE LIVE.

4. Project team stages design, tests design and documents.

5. Customer checks design and documentation.

6. Build begins, with testing documentation. Each section built, tested to specifications and then signed off.

7. Build completed, as built created from design. Highlights any design changes during the deployment. The real world may have introduced these depending on how close your testing environment was.

8. Diagrams and support documentation added to as built.

9. Project completed, and changes going forward to be added to as built and version updated.

This along with the following –

1. Transparency to customer and management

2. Handover session with support

Is a great start in building a new or upgrading an existing network.

It’s not the end of the project that matters or the completion that is the most critical part of ‘getting it done’ it’s the beginning.

This is where all those problems that will be coming can be removed.

~Brad.