sunnuntai 30. heinäkuuta 2017

"Your internet isn't working"


Few weeks ago I got a call from our office internet operator, informing us that our internet modem had gone down, they couldn't contact it. I wasn't at office, and since we run web service there too, I had of course go to have a look. Fortunately this happened when I wasn't on the other side of globe at least.

At the office I quickly noticed that pretty much all computers have gone down, causing the office to be eerily quiet (I'd love an quiet server that doesn't sound like friggin' 747 taking off on full load!). The mains fuse for critical systems fhad tripped! Switch that fuse back and everything starts waking up.

Our power supply for computers isn't exactly highly redundant. Single mains fuse feeding UPS that can take a load for good a while. Now was first time that it hadn't been enough. So when all the critical computers and services are back up, I get back to what I was doing, which at that time was heading out towards frisbee park with the kid.

Next day at office, I start going through logs and start noticing some remaining issues. One network storage was inaccessible. With a quick glance it indeed was - it was still off. Curious, those should come back up pretty quickly when power is restored.

Some more research and it seems that its internal power supply had died. No sign of life of any kind (specifically +5v standby; without it nothing of course works anyway). Not a first power supply that has gone down on me (in a bad way), and very likely not last one either.

Fortunately this wasn't critical storage, but "only" secondary backup storage that I had just month or so earlier retired from its main storage role and moved to its current role. I don't remember its exact age, but I'd guess it was some 10 years old now. Running pretty much 24/7 of course. I have to say timing was very lucky - few months earlier and I'd be far less happy about the situation.

Now I'm trying to make a decision here. Find a replacement power supply - which might be a bit difficult, due to less than common form factor - or retire storage device and its disks now. At the moment I'm leaning more on the latter option, especially since disks, too, have never been replaced during its lifetime. It's very likely near end of its useful life anyway.

There's no high need to replace it immediately either, as this was secondary backup. My primary one, of course, is off-side on third-party server and updated constantly and automatically... With data encrypted at our end, of course. You don't really think I'd be stupid enough to allow any data out of office unless it's encrypted with keys held only by me?



Ei kommentteja:

Lähetä kommentti