SixTimesEight

there should be a public service announcement before any node conversions

August 17, 2005 – 8:18 pm

One of our clients in Destin, a large multi-office escrow and title company was without internet, email and their VPN connection to their other offices today.

We didn’t know this until the technicians from the cable company called us from the client’s office. Apparently they had been out there sometime late last week working on some things and had broken something at that point. The office had been without broadband ever since, but rather than call us, they called the cable company back out (since they don’t charge).

So the cable technician called us to ask about some things at the office that didn’t make sense to them. They have one cable modem which sends its traffic to a D-Link switch. From the D-Link there’s one cable to a Linksys router and another that goes to a Netscreen security appliance. From the router there is one cable out to a 24 port switch which feeds all the PCs and networked printers. The Netscreen is not a throughput device. There is no in port or out port. It acts as a client on the network just as any PC would. But that didn’t make any sense to the cable guy. Try as I might, I couldn’t explain it to him in terms he could grasp. I finally had to tell him to ignore it.

“It’s been there for years and it’s not the problem, trust me.”

So he goes on to tell me that they’re going to hook up a laptop directly to the modem and try to connect. If they can connect, then there’s no problem on their side, it must be our network configuration that’s not working. He calls back 5 minutes later to tell me that they connected just fine, it’s out of his hands now and he’s going to the office next door which is also having problems.

So I head on down there as fast as I could to try and catch him while he’s still there. I had a feeling that when they were out last week and broke it, that they managed to screw up the static IP which this office has had for ever. The static IP, which is bound to the VPN settings, the remotely hosted MX record (for email) and the remotely hosted A Record (for domain purposes).

I get there and he’s already moved on. I’m almost sure he heard in my voice that I was coming down there to show him up, so he took off ASAP. So I just call the cable company to have them tell me what IP they have listed for the office. My hunch was right, as it had indeed changed. I ask them to change it back and the guy says he can’t.

“They’ve been doing some network restructuring in that area and that IP is no longer available.”

The City of Destin is rapidly expanding and the cable company had to reorganize and add some nodes out here to keep up with the demand, and in the process, we got moved to another IP segment. That’s not really a big deal, we can change those settings. But it would have been nice to know it before hand. This is probably what the cable guys were doing out here last week. Which would have been perfect timing had we known about it. We could have changed all the IP settings and the weekend would have provided the necessary time for it all to propagate. Instead, this poor company will be without email until Friday if they’re lucky.

Oddly enough, Sprint had called our office earlier in the day and basically begged us to switch to their DSL service. But given the amount of billable hours we can attribute to DSL, that will never happen. Even with crap like this and even if it is half the price.