IT BS thread

I had this happen before when I worked for a charter school organization here in los angeles. So much fun.
Multiple times actually. Once took out the site we ran our data center out of and the second just took out one of the schools.

 

i hit a power line to a light pole. i had called for locates and still found the power line. it cost me $1200. I hate to see what that bill would be. I'm guessing somewhere north of $100k

 

What the hell? Same shit just happened to us. Contractor hit two major fiber lines with one stroke of his bucket. Guess what was in those lines, yes, line 1 our primary and line 2 our backup. So much for different paths to the internet. They had to bore 1,500 feet of new line to get us and thousands of other users back on line. Now we are in the market for a second internet provider.

But, it did make for a quiet 2 days, no internet, no phone, no email.

 

Previous place i worked had 240 stands of fiber between their 2 datacenters that are about 2 miles apart. They took 2 different paths (they own all the land in between the 2) that are about 1/4 mile apart so that someone can't cut both lines with one swoop of the bucket.

 

Previous place I worked bored an 8" hole in the side of the basement data center and trenched a completely different path to different CO. Problem was they cut the sprinkler lines. When they went off in the middle of the night, the trench channeled all of the water into that 8' hole. Oh yeah, that was a mess!

I do love my job security!

 

That is just funny.

 

We should do funny IT stories.

Back in the mainframe days my company decided to get rid of their IBM mainframe dumb terminals and replace them with PC's running an emulation software. My job was to build out the network and replace the terminals with PC's. Because people don't like their cheese to be moved, and I was the face they saw when their beloved terminal was replaced, I wasn't the most popular guy. So, every time something went wrong it was "my" fault.

One nice spring Saturday I get a call, "YOUR network has crashed!" (it was a 7/24/365 day operation and I was more or less permanently on call) I remoted in and my network was fine, but the connection to the mainframe was down and there was no way to trouble shoot that remotely. When I got to the data center everything looked fine. The mainframe was up, the network was up, but the connection between the two, which was a single network cable, was down.

(For those that have never lived the joy of mainframe computer support they were fussy beasts that required very specific cooling, humidity and a raised floor. (this will be important in a minute) This data center wasn't too big, maybe 2,000 square feet, and it had 3 massive 8'x12'x3' chiller/humidifiers in it. They weren't known for their reliability and that's why they had three, so one could be down for repair.)

I couldn't for the life of me figure out what was wrong, it should be working. It was working when I left the previous evening. I was the only one that dared touch it and I sure wasn't going to change something that would ruin my Saturday. The users were breathing fire down my neck, saying the cable has gone bad, and in my head I'm thinking cable doesn't just go bad. Just to humor them I pulled up a tile where the connection between the two was made. Well . . . cable does go bad . . . if you soak it in three feet of water. One of the drains on the chiller had clogged and created a 2,000 square foot wadding pool. It was a glorious sight! Plug strips glowing away, network and mainframe hardware drawing untold voltage, data and connection cables all under 3 feet of water. And, the best part was the water alarm floating upside down on top of it all!

 

I'll play too. Setting the wayback machine to Desert Storm...

We had a crapload of systems we fought the air war on from Z-150s to early Sun workstations. Well, as much as the Saudi's tried to make Riyadh a genteel place it's still in the middle of the desert. The sand there is of the consistency of dust, and it's omnipresent. You can fight it, but the best you can hope for is a draw. You will NEVER win.

After a while, some of our critical (and expensive) battle damage imaging systems (Sun Ultras) were acting up. Our maintenance folks couldn't figure out what was wrong with the computers. They'd shut them down, open them up, inspect everything - nothing was wrong or broken. It was frustrating.

One day I'm sitting dealing with comms to and from some aircraft carriers, and the maintenance guy is trying to get the imaging box to work (again). "I can't for the life me figure out what's wrong!" he'd complain.

"Well, did you gravity check it?" I asked. A gravity check, staple of combat comm (USAF) and signal corps (Army) is where a piece of very expensive equipment is checked for how well it works with gravity. The joke goes that if something doesn't work, you pick it up 12 inches and drop it. If it still doesn't work you pick it up 3 feet and drop it. If it doesn't work after that - it's broke. It's a well established joke - taught to me by my leaders as I would later pass it down.

Anyway, I'm focused on keeping comms up to the USS Kitty Hawk (they had a gimpy satcom antenna) when I hear a loud WHAM! I turn, and he'd picked up this $30,000 workstation (early Sun systems were not cheap) and dropped it about 18 inches.

"WHAT THE FUCK ARE YOU DOING?" I started, but was cut off when the stupid thing beeped, went through BIOS checks, and fucking booted successfully.

We looked at each other like we were witnessing some supernatural phenomenon.

TURNS OUT...That Saudi dust is so fine it can get into the tightest spaces between unsoldered connections. Fix action and regular recurring maintenance on these things was to open 'em up, pull and clean all card and accessible pinned connections, and put 'em back together. No more issues after that.

No more gravity checks, either.

 

What, yours didn't auto swap over? We have ours set up so 5 secs, it switches.

 

We had that very same issue about two years ago.
Switched one to a different provider for different path.

 

They did cut over. Then we needed our backup back up because if we go down it makes the national news.

 

https://youtu.be/bKgf5PaBzyg

 

People are too dumb to think a change isn't really a change... Dude couldn't access certain locations on 1 subnet but could on another. I noticed someone changed the VLAN on the switch. Didn't help the dude kept referring one of the subnet octets as a "VLAN" Kept confusing the shit outta me

Haha ol Mr McAfee gives zero fucks about McAfee now, yet it's still widely used. It's hilarious

 

I might know someone that's gone on a few wild goose chases because of this sort of thing.

 

I think I still have the install disc for McAfee 6.0 somewhere

 

Probably a stupid question but what "program" do websites use to stream radio?

I ask because I work for the gov and they have all streaming radio sites blocked. However, I can at least go to individual stations pages. Sadly whatever program/plugin needed for it is just not there. (http://www.wmal.com/listenlive/). That's an example.
I've read I could use a proxy but worried trying that would also be blocked (or get in trouble for trying it)

 

My personal cell phone and a physical radio because all streaming services outside of YouTube are blocked. Also a govvie.

I have found, however, that YouTube has streaming music channels that are really good. I'm personally a fan of the following, and highly recommend:
https://www.youtube.com/user/Chillhopdotcom
https://www.youtube.com/channel/UCi8wqezBudeAiTdKOX571ug
https://www.youtube.com/channel/UCZyyXrEF2WCZbI653PFNBbA
https://www.youtube.com/user/tripmastermonkee
https://www.youtube.com/channel/UCSsz5GO1rQjzp1RND7QtEjg (LOVE their mixes)
https://www.youtube.com/user/fozzium

And I've ripped this and play it in the truck at high volume in traffic. It's fun when my traffic neighbors bob their heads along with the beat.
https://youtu.be/TA9LVzuC7z4

 

Thanks for the reply. It would be like heaven if I could have my phone, sadly I cannot. A radio would also be nice but radio signals are blocked

I upgraded to YouTube premium so I could at least listen to nerdy lectures and documentaries. If I could figure out how to dl the podcasts and covert to a video, I could upload to my YouTube acct....

 

WHILE AT HOME...

Google "YouTube to MP3 converter" and play around with the various options. You don't have to download programs anymore - there are active websites that do the processing for you.

Also, a lot of these podcasts are also on YouTube. My favorite is "Stuff You Missed in History Class" and it's on YouTube too.