bonifartius 𒂼𒄄 @bonifartius@social.cleverlibre.org

idk what I should call this still, feel free to suggest a tag or a blog name or something
anyway, time for a dayjob story

so we have a small DC, just a rack or two, no biggie
computation/network-heavy workload, most of the file IO is just logs
as such, the biggest priority in terms of filesystems is making sure it does NOT crash
application servers are kind of whatever, so they just run ext4
so what if like 5 of them die, we have a bunch more
gateways/databases are mdadm raid1-lvm-ext4 (again, low IO, remember) - we don't want too many of them to go down, or else who knows what might happen to quorum etc (this was overengineered, happened before I was with the company, etc)

so, one day, there's this HUGE thunderstorm in the city
half of everything goes down
1/4 of the city blackout
but not the DC, DC is shared, has backup generators, and so on

. . . suddenly everything goes dark, both at the office and in the DC (at least from what I can see)
"I'm sure it's fine"
we go back home and check up on it tomorrow
. . . the DC is not back up

I go there, and hook up the various machines
all of the application servers, without fail, have the root fs fail to mount
I fsck one just to see
e2fsck literally refuses to operate on it
"wow wild", I say, and move on to the gateways

all of them. all of them have degraded raid
so ok, I separate them out and see if either copy looks ok
they're both corrupted
same deal as with the gateways

spent the next week or so manually doing the job of e2fsck and trying to recover as many files as possible
things seem to just be corrupted at absolute random
I'm essentially restoring db files and /bin/cat at the same rate
the backup host is on-site and there's an external drive that stuff got pushed to (no one wanted to pay for offsite storage besides the office, and there's just not enough space there to back EVERYTHING up; though maybe there would have been if we only backed up the "important" data, rather than 50 copies of /bin/cat)
the backup drive is obviously just as dead
the external drive is also completely fried - doesn't even spin up (this was during the spinning rust external drive days) - it was fine before!

I finally ask them what the hell happened
a FUCKING LIGHTNING BOLT
HIT THE TOP OF THE BUILDING
SENDING AN ELECTRIC SHOCK THROUGH ALL THE SERVERS
BLOWING PAST THE SHARED UPS
and we just had sufficiently old power supplies that they failed to sudoku themselves to save the grid
and ofc it was all just plugged into regular power strips - it's behind a DC level power protector after all

however, some filesystems did survive!
the fat32 EFI partition (efi was still new so only one machine actually had that)
one xfs partition
and the experimental btrfs partition that I wanted to move the backups to

the lessons:
1. use LOGICAL backups of the data that's IMPORTANT, restore time is whatever
2. ext4 will eat your data
3. lightning strikes will eat your data
4. xfs/btrfs/fat32 can survive acts of god