IT BS thread

Happy Pi Day

 

I have launched into many the profanity-laden tirade whenever those numbers appear.

 

My favorite is when the user that knows just enough to be dangerous tells me it has to be a problem with the internet because his computer has an IP address, 169.254.x.x. However it is a useful troubles shooting clue, because I know where not to look.

 

They pay you enough to afford booze? Where do I apply?

 

That is probably the best disaster recovery plan I've ever seen. A previous gig at a natural gas pipeline one of the directors expected people to abandon their families to come in and restore the systems. I told him if the "big one" hits he is fourth on my list, just behind family, friends, and neighbors. If they want fast restorations they need to build a backup facility in another city. To my complete and utter amazement they agreed. We backed up the corporate office (tornados) and they backed up our office (earth quakes).

 

We are so redundant it's insane. We bridge two major transmission lines (separate plants), so one plant can go dark and we still have power. If the other plant fails, we have a locomotive engine for a generator, plus something like 80 hours worth of fuel.

Then there are the redundant circuits - 3 10-gig lines from separate vendors...












...2 of which got cut by some fucktard highway worked in a backhoe who can't read "do not dig" warnings.

We did get to sue Century Link for lying about running their own fiber and not subbing it out to AT&T, which was what they did. They are no longer our service provider.

 

I've never worked anywhere where they had a DR plan that I believe would actually work. Always redundant and backup power (backup power had to be used once, but the facility was otherwise fine), circuits, etc. But once you get past restoring the Tier 0 and 1 services, I don't think anyone knows what all the dependencies between systems are. It would be quite the circus. I guess the individual plans have a bit of value, but I've never seen anybody who has an overall plan.

 

We have multiple linked datacenters just in North America with full DRS and HA. It's pretty darn comprehensive. I used to contract with the company, but when I came on as a full admin and got to learn all the inner workings, it was a pretty "woah" moment.

 

Yeah, it's not the hardware or infrastructure I am worried about. Infrastructure can't fix poor application architecture. Applications have to be designed to take advantage of those capabilities in the right way, and sadly most are not. Even if they are, there are always little dependencies on other systems that are left out of the plan. Most DR tests I have been a part of only look at one application at a time -- rarely is loss of an entire facility simulated. When it is, teams know ahead of time that they will be pulling the plug. But I guess it is like the wise Tyson said, everyone has a plan until they get punched in the mouth.

 

I like to think we've done well there. Failovers have been implemented in ways that if a server falls over, the backup picks right up. And with redundant datacenters, there are barely miliseconds of downtime at the most in cause of a failure -- maybe some latency, but shit still works. Not all of the systems have been implmeneted in that way, and unfortunately 3am Oh My God calls do happen. Without going into details on an internet forum, we've been pretty hardcore about high availability, becaue in many cases, lives may be at risk.

 

I think for most business disaster recovery is backup, cross fingers, done. As soon as you start talking about budgets for redundancy of people, hardware and facilities the conversation gets very quiet.

 

We don't have much flexibility here. I only have 1-2 potential fiber providers to any of my buildings. The state ran fiber to all public buildings a while back and awarded the operation contract to some Canadian company that ended up suing the state because they were promised more customers than they got and were operating the network at a loss. They won and now some other company took it over. Fortunately, it's been really solid - only two very brief outages in the ~8 years we've been on it.

Charter has fiber going past about half of my buildings.

Verizon doesn't have fiber to serve businesses or homes on the poles, but has some COs with fiber to them. They quoted running it from the COs to our buildings but the price was really high.

 

Tell me you were in military IT back in 1999 without telling me it was 1999.

I remember when we lost our Exchange servers and domain controller to NOSC control in 2000. There was a near open rebellion because our base only had a 9MB pipe, and we felt it. We went up to 42 MB the next year, but still....we HATED losing our Exchange servers.

The upside was that AMC let us get rid of all Novell servers. Eliminating IPX traffic sped the network up quite a bit.

 

Sorry, couldn't help myself.

We are migrating to Azure and M365 and what the hell were we thinking??

 

You're going to hate it. Frustration squared.

 

Our overlords mandated it. Apparently GSuite wasn't good enough. Whatever. I didn't have a problem with Google at all, but I'm not paid to have opinions. At least it's not Lotus Freaking Notes.

 

Whoa. That sets the way-back machine a few decades.