Author Archives: sr71rocks

Run the Race – See Others as God Does

We had our first performance of our Easter program at church this evening…it was a wonderful time of worship and remembering Christ’s sacrifice on the cross. It got me thinking…there are so many things wrong in this world…politics, race, religion, just to name a few. If the Christian church could just move past all of that and see people as God sees them, and then act accordingly, the changed lives as a result would be incredible!!

So can we all do that? Please?

I’m going to really try…

Office Phones Down on Monday Morning – DHCP Issue

If you are a Network Engineer, then having your phone ring early on a Monday morning is never a good thing. And that’s what happened this morning. Seems that none of our Cisco phones were working at the Corporate office…yep, none, as in 150 employees. (And it always seems to be the big offices…why does nothing go down at a little remote office that only has 3 employees?)

After talking to a couple of users (via cell phone of course), I realized that all of the phones were trying to get an IP address but were unable to.  (Phones were displaying:  Configuring IP). Well that is very interesting…and I recalled that the DHCP server was just replaced this past Friday. (Grrrrrr….how do you mess up a DHCP server?)

Well, it’s easy to mess things up if you don’t use the same IP address on the new server as the old server. Remember, DHCP is a layer two broadcast mechanism…the device booting up sends out a DHCP Request packet destined for all F’s, and the server responds.  Now, if the server is NOT on the same local network, then you need an IP helper statement such as this to properly forward the DHCP request…

interface Vlan11
description VoIP VLAN for Corporate office
ip address 10.11.2.1 255.255.255.0
ip helper-address 10.10.2.10

As you can see the switch will take any DHCP requests and forward as a unicast packet to the DHCP server. However, the new DHCP server had a different IP address. Why? I don’t know…however the server guy did fix the issue quickly, which was much appreciated. He simply changed the IP address of the new server to that of the old server…and boom, all of the phones started registering. You may be wondering about all of the PC’s on the network…they were working just fine as they were on the same local network as the DHCP server.

I then was able to finish shaving and get ready for the day. You just have to love Mondays…

GLIBC getaddrinfo Vulnerability in Linux Systems

A pair of Google researchers recently released a vulnerability report on the GLIBC function “getaddrinfo”, which if exploited, could crash the system or even give a hacker command line control. (Yikes!!) MANY Linux systems are vulnerable to this so please patch your systems quickly. GLIBC packages affected by this are versions 2.9 and newer…2.9 was released way back in 2008, so you can see that the size of the affected systems is huge! Note…”getaddrinfo” is used by systems in resolving DNS names to an IP address. Talk about important…right??

To see what version of GLIBC you are using, simply run the command:  ldd –version

Here is an example from one of my test Linux systems at work…

Example of affected GLIBC package

Example of affected GLIBC package

As you can see, this system is running version 2.10.1 of GLIBC and needs to be patched. For CentOS (which I’m running), you can obtain a more detailed listing about GLIBC this way…

GLIBC info from my CentOS system

GLIBC info from my CentOS system

Most Linux distributors have patches ready to fix the issue, so running the appropriate update commands should take care of things. For CentOS, just run “yum update” and it will grab the fix and apply it…a reboot of your system will be required.

Related links…
Google announcement
SANS InfoSec Post

Cisco Security Advisory – IKE Vulnerability in ASA Code (CRITICAL)

Cisco ASA (via cisco.com)

Cisco ASA (via cisco.com)

Cisco released a critical security advisory today concerning an IKE vulnerability within the ASA software OS…and let me tell you, this will affect a LOT of people! If you are running one of the affected software versions (and I am), then you will want to update your ASA appliance very soon. I’ll have mine updated in the next couple of days.

Don’t delay. Once you read the advisory, you will know why!

Packet Capture on Both Sides of a Conversation

Greetings everyone!! Yes, it’s been a while…sorry about that. Been busy with life…I’ll just leave it at that.

So at work, for the last couple of days, the Citrix admin has been having an issue with users at one of our larger remote sites…seems they are intermittently unable to connect to a Citrix server after being redirected by the Citrix license server. He brought me into the problem this morning once he realized the problem was only occurring at this one site. Very interesting!

First step was to capture some of the traffic and see what’s going on. I have a Linux server at the Data Center running Snort, watching most all of the traffic into and out of the Data Center, so this comes in very handy when I need to capture some traffic. I started a TCPDUMP on the server, specifying the interface in question that has the traffic from the remote site, and dumped it into a file…the command was…

tcpdump -vv -nn -s 0 -i eth3 -w /root/pcapfiles/citrix_issue.pcap host 10.10.21.223

The Citrix admin was remoted onto a PC at the site, and he attempted to connect into Citrix. After I captured the data, I moved the file to my laptop and opened it up with Wireshark…I saw the 3 way handshake (SYN – SYN/ACK – ACK), and then some data going back and forth, but the session never started up and it timed out. Weird.

I also captured a Wireshark session from my own laptop so I could see how it was supposed to work. I still could not see what the issue was…very strange.

I then decided that I needed to see packet captures from both sides of the conversation. So I remoted into the PC and installed Wireshark on it…I then started up Wireshark on the PC and TCPDUMP on my Linux server, and then tried the Citrix client again. After it timed out, I opened up both PCAP files in Wireshark and examined them side by side, packet by packet. On the PC, I saw this…(PC = 10.10.21.223, and Citrix server = 10.12.1.122)…

Packet capture on PC side showing initial SYN packet

Packet capture on PC side showing initial SYN packet

I found the matching packet on the capture from the Data Center…

Packet capture from Citrix server side showing full handshake

Packet capture from Citrix server side showing full handshake

Say what????? I see a completed 3-way handshake…including the SYN/ACK from the server back to the PC (which was not in the PC capture), and another packet from the PC to the server completing the handshake…also not in the PC capture. VERY bizarre!!!

Then it hit me…Riverbed!! The only way a handshake could be completed in this manner was from another device sitting in-between…and we do have that. We have Riverbed devices sitting at the Data Center and at most large remotes sites handling WAN optimization. Since no other sites were reporting any issues, then the Riverbed at this remote site must be “kinked”. So we restarted the Citrix application optimization process on the Riverbed, and that fixed everything!

VERY cool…and very interesting. This took some time to figure out, but once I got visibility at both ends of the conversation, the answer was easy. Remember…Wireshark is your friend.

1984 Has Arrived, Though a Bit Late

If you want to read a novel that paints a bleak picture of our future, then read Nineteen Eighty-Four. It was written by George Orwell way back in 1949, and basically describes a future society that controls all thought and expression, in favor of the ruling party. Government surveillance of the population runs rampant, with little or no privacy.

I remember when 1984 arrived thinking how glad I was that this vision of the future was wrong. Unfortunately though, it’s starting to look like certain aspects of the novel are in fact becoming real. Just look at the NSA and it’s surveillance program (which it just ended…maybe), or all the talk about installing back-doors into applications so that governments can track “terrorists”. Of course they won’t track us, right??

So today I read an article at ComputerWorld authored by Darlene Storm…

LA’s plan to scan license plates and send Dear Prostitute-seeking John letters

Incredible!! Just driving through the area could trigger this letter. Yes, we need to target the John’s and do what we can to shut down the sex-slave industry, but I’m not sure this is a good solution. Plus, our society is already a lot closer to 1984 than some may care to think…with all of the license plate scanning, facial recognition, and related Internet tracking that is in effect now, our perceived level of privacy is much smaller than we think.

So, we need to ask (if it’s not already too late), where do we draw the line?

Cisco VIRL – 20 Node IOSv Test

Greetings!! So I’ve been playing with the latest Cisco VIRL release (v1.0.0), and let me say for the record, I like it!! Of course I’m still relatively new at VIRL and have much to learn, but I am very impressed by the latest release. AND I’m very pleased with the new bare-metal VIRL installation I have!

If you recall, several months ago, I installed VIRL on my new desktop system (Quad core i7 processor, 32 GB of RAM, fast SSD and storage drives, Windows 8.1 and VMware Workstation 11). VIRL ran very well on that system, which is to be expected, but there were times when my PC just behaved a bit strangely…a bit of pausing, some hiccups…you know what I mean. Of course I was running other power hungry programs at the same time…Photoshop and Lightroom. This installation is a bit more complex…you have a physical PC, running Windows 8.1 for an operating system, running VMware 11, which runs a VM (Virtual Machine), which is running Linux, which then runs it’s own VM that runs the network simulation (routers, switches, etc). Whew…it’s complicated just typing all of that out!

So…I decided to upgrade my Dell 2950 PowerEdge server a bit. I added RAM and another processor, so it now has:  dual Quad-core Zeon processors (3 Ghz); 16 GB RAM; and fast 15K RPM drives. Let me tell you…this thing screams. Yes, it has less RAM than my PC, but now it will be dedicated to just Linux and the network simulation…nothing else. The new setup is much simpler now…a physical server, running Linux, which runs a VM for the network simulation.

How does it work? NICE!! I setup a test 20 node router simulation and cranked it up…it took about 2.5 minutes for all nodes to go ACTIVE, and another 1.5 minutes for BGP and OSPF convergence to complete. Here’s the topology for the test…

20 Node VIRL Simulation

20 Node VIRL Simulation

And here are the resources used…

Resources used for 20 node test

Resources used for 20 node test

Everything is looking very good here. Of course, I am using just IOSv nodes (routers)…they are the least CPU and RAM intensive. If I were using other node types (NX-OSv, ASAv, CSR1000v, etc) then I would be running out of resources sooner and would have to balance the number and type of nodes I could run in a simulation. What will help is to upgrade my RAM to 32 GB, which I will do early next year.

Do you like what you see? Then take a look at VIRL…I think you will be very pleased.

Run the Race – Living a Life of Generosity

In this world of always wanting more (especially here in America), I found this article to be an inspiration to my soul…

Alan Barnhart Limits His Salary

It seems that no matter how much you make, you always need a bit more. And a lot of us Christians are in the same boat as the general public. Just think what our impact on the world around us could be if we lived within our means, and used the surplus to bless those in need…the impact would be earth shattering!

Not sure where to start? Take a look at the materials available from Dave Ramsey…it’s a good starting point if you really want to live within your means and to use your finances wisely.