I used to work at a car dealership. One day I had to use a bay in a different building because my usual workplace was occupied. The other building had a lift that I hadn’t used before.
Anyways, I drove the car onto the lift, got out and placed the arms of the lift under the jacking points like I had done a thousand times before. I raised the lift a little and checked if the placement was still correct. It looked good, so I raised the car to a medium height. When I looked again, I realized that this lift had a central platform that was also raised and was set about 20 centimeters higher than the four arms that usually lift the car.
This 90.000 Euro SUV was basically balancing on a 180x50cm piece of metal right in the center. I managed to lower it down safely but my pulse goes up just thinking about that day.
Several years ago, when I was more just the unofficial office geek, our email was acting up. Though we had Internet access as normal. At the time, email (Exchange) was hosted on-prem on our server. Anything server related, I’d contact our MSP to handle it. Which usually meant they’d simply reboot the server. Easy enough, but I was kinda afraid and hesitant to touch the server unless the MSP explicitly asked/told me to do something.
I reported it to our MSP, expecting a quick response, but nothing. Not even acknowledgment of the issue. This was already going on for like an hour, so I decided to take matters into my own hands. I went to the server, turned on the monitor…and it was black. Well, shit. Couldn’t even do a proper shutdown. So I emailed again, waited a bit, and again no response.
Well, if the server was being unresponsive, I figured a hard shutdown and reboot would be fine. I knew that’s what the MSP would (ask me to) do. What difference was them telling me to do it versus just me doing it on my own? I was going to fix email! I was going to be the hero! So I did it.
Server booted up, but after getting past the BIOS and other checks…it went back to black screen again. No Windows login. That’s not so terrible, since that was the status quo. Except now, people were also saying Internet all of a sudden stopped working. Oh shit.
Little did I know that the sever was acting as our DNS. So I essentially took down everything: email, Internet, even some server access (network drives, DBs). I was in a cold sweat now since we were pretty much dead in the water. I of course reached out AGAIN to the MSP, but AGAIN nothing. Wtf…
So I told my co-workers and bosses, expecting to get in some trouble for making things worse. Surprisingly, no one cared. A couple people decided to go home and work. Some people took super long lunches or chitchatted. Our receptionist was playing games on her computer. Our CEO had his feet up on his desk and was scrolling Facebook on his phone. Another C-suite decided to call it an early day.
Eventually, at basically the end of the day, the MSP reached out. They sent some remote commands to the server and it all started working again. Apparently, they were dealing with an actual catastrophe elsewhere: one of their clients’ offices had burned down so they were focused on BCDR over there all day.
So yeah, I took down our server for half a day. And no one cared, except me.
Two nights ago I had a random meeting with the CEO, who I have a really good relationship with, added to my calendar. Thought nothing of it.
I entered the zoom call and said ‘so am I getting fired?’
The answer was yes.
Awkward silence ensued for a minute until they started telling me about the severance package.
Side note: I can try to negotiate that severance a bit right?
Definitely negotiate that severance. What a shit deal
How do you negotiate severance? Don’t you have zero leverage in that situation?
Some severance packages will have a non disparagement clause in it, or they’ll say you can’t recruit people to xyz competitor for a number of years. You can then say “yes I can do that, but if and only if you give me 20% extra of my estimated salary”
deleted by creator
Urgh yeah I had one of those. A “small quick meeting” that makes you think they just want an informal update. Nope, its the getting fired talk. Still, turned out to be a blessing.
Mine sucks because it’s the best job I’ve ever had. Planned on staying as long as they’d keep me (just under 5 years it turns out) and had no plans at all to even poke around at other roles.
The silver lining is I’ll prob get a nice pay increase since I’ve been pretty underpaid at this place as it’s an NPO.
Alt tabbed once too many times, clicked drop database, clicked yes. Realized what I’d done and panicked.
Deleted the user db for the east coast auth server for the game America’s Army: Operations. Thankfully it was the secondary so we just redid replication.
Older gentleman walked into the lobby of our office. None of us knew who he was or had seen him before. He looked confused and lost. Someone went over to ask if they could help him. He tried to but didn’t respond. Then fell over. Hit his head on a table on the way down. Was dead before the pandemics arrived.
We were all in shock. Poor guy was starting into a stroke when he walked in. Maybe even walked into our office to try getting help. But it was already too late.
before the pandemics arrived
I know this was a typo and you meant to write paramedics, but all I could think first thing I read this was “what a lucky bastard”
“Run, you fools.”
This fucked me up when I learned and finally accepted it, but it’s actually “Fly, you fools!”
An isolated shingle spit nature reserve. We’d lost mains power in a storm some while back and were running on a generator. Fuel deliveries were hard to arrange. We’d finally got one. We were pretty much running on fumes and another storm was coming in. We really needed this delivery.
To collect the fuel, I had to take the Unimog along a dump track and across 5 miles of loose shingle - including one low causeway stretch through a lagoon that was prone to wash out during storms. We’d rebuilt it a LOT over the years. On the way up, there was plenty of water around there, but it was still solid.
I get up to the top ok and get the tank full - 2000L of red diesel - but the wind is pretty strong by the time I have. Half way back, I drop down off the seawall and reach the causeway section. The water is just about topping over. If I don’t go immediately, I won’t get through at all and we will be out of fuel for days - maybe weeks. So I put my foot down and get through that section only to find that 200 meters on, another section already has washed out. Oh shit.
I back up a little but sure enough the first section has also washed through now. I now have the vehicle and a full load of fuel marooned on a short section of causeway that is slowly washing out. Oh double shit. Probably more than double. Calling it in on the radio, everyone else agrees and starts preparing for a pollution incident.
In the end I find the firmest spot that I can in that short stretch and leave the Moggie there. Picking my route and my moment carefully I can get off that ‘island’ on foot - no hope with the truck - BUT due to the layout of the lagoons only to the seaward ridge, where the waves are now crashing over into the lagoon with alarming force. I then spend one of the longest half-hours I can remember freezing cold and drenched, scrambling yard by yard along the back side of that ridge and flattening myself and hoping each time a big wave hits.
The firm bit of causeway survived and there was no washed away Unimog or pollution in the end - and I didn’t drown either - but much more by luck than judgement.
These days I am in a position where I am responsible for writing risk assessments and methods statements for procedures like this. It was another world back then.
My first week on a new job I ran a DELETE query without (accidentally) selecting the WHERE clause. In Prod. I thought I was going to get fired on the spot, but my boss was a complete bro about it, and helped with data restore personally.
Everyone at that company was great both professionally and personally. It’s the highlight of my 30+ year career.
That’s the employer’s fault for making it so easy to connect to prod with read-write permissions. Not your fault.
Oh there was plenty of blame to go around. I wasn’t exactly fresh out of school either. I had “extensive experience with SQL Server” on my resume by then.
At my last job I was given write permissions to production and I asked for read only credentials instead, I know my own stupidity
+1
We have read only access.
Also transactions are good ideas.
Also my database tool (the one built into pycharm) warns and requires you to hit submit a second time if you try a delete or update without a where. Discovered this on local where I really did want to update every record, but it’s a good setting.
Look at mister fancy pants over here with a database tool. Back in my time we had to use Query Analyzer uphill both ways.
My first salaried job was also my first proper IT job and I was a “junior technician” … the only other member of IT staff was my supervisor who had been a secretary that got a 1 week sysadmin course and knew very little.
The server room was a complete rat’s nest and I resolved to sort it out. It was all going very well until I tripped over the loose SCSI 3 cable between the AIX server and it’s raid array. While it was in use.
It took me 2 days to restore everything from tape. My supervisor was completely useless.
A few months later I was “made redundant”, leaving behind me everything working perfectly and a super tidy server room. I got calls from the company asking for help for the following 6 months, which I politely declined.
It’s always fun when a job calls you up after you’ve been fired to ask how to do the things they didn’t know you were doing
Yep, I remember in one job I was at for 8 years a manager 2 levels up complemented me for sorting out the networking for a re-arrange of our own office … I was gobsmacked because I’d been managing a whole network and server upgrade for a client that involved well over 1000 users at the time yet an hour of fiddling with wires under desks was the only thing that got his attention.
One job I was fired from and rehired within the day, after they quickly realised that I was their only Android developer and they couldn’t build an app with just hopes and wishes. They fired me again later, which they quickly regretted since I was the only one with the signing key (meaning they couldn’t update the app).
Man fuck those guys. Not a sysadmin myself, but from what I hear the position is criminally underappreciated. Why is it so hard for people to understand that if things aren’t breaking, it means people are doing their job correctly?
Yeah, I got laid off twice more before switching careers. Both times they wanted me to come back and fix stuff after letting me go.
It goes hand in hand with the “if someone works hard, they should be given more work as a reward” line of thinking.
That’s when you offer consulting and tell them your hourly rate!
I didn’t have them over a barrel, they were just being lazy and trying to exploit me further for free.
no work related but am overfill sink with water changer because forgot to remove drain cover
is flood
am get in trouble also ._.
Now that i think about my first job was fucking wild.
My buddy was in a forklift taking some stock down and i was spotting, basically just hanging out and making sure no one got in the way. A few minutes after the normal time it’d take he thinks something is wrong and calls me to take a look (from afar) to see how fucked we are; the answer was very, the pallet was barely holding together at all, but i couldn’t see a damn thing from my position. Before i could get back to spotting we heard a loud crack and the world went still, i imagine for much longer by him, and not a second later we had hundreds of pounds of foul smelling mulch everywhere.
I had a lot more there too; babysitting an old man that looked on the verge of death with no management anywhere to be found, moving hundreds of pounds at a time by hand, dealing with the best conspiracy theorist ever.
I’ve been bored everywhere else I’ve ever worked.
what do you mean, the best conspiracy theorist ??
Turns out the world is flat. And under a dome. And Jesus is on top of the dome. And aliens have visited us in the dome. And so much more.
All those times someone accidentally printed a hundred blank pages out of the printer because they went to the very bottom of Microsoft Excel while messing around and unknowingly inserted the space bar in the bottom bar before printing.
I spent a decade working in insolvency.
When we were going into a business that had failed the question was “Are the idiots, criminals or both?”
One highlight:
A boat sales / marine business goes bust. When we arrive with the paper work and seize the place there are about a dozen new boats on the lot worth several million. We change the locks on the gates.
Arrive the next day, the gates have been busted open and several million in boats are now missing. We look up the addresses of the owners (one of them lives on acreage) and drive to their property…from the road we can see the boats stashed there. Really smart guys.
So we call the police. Someone inside notices use there and decides to flee with one of the boats, it is huge but they think they can get away.
We then have the slowest car chase in history as we calmly follow this guy towing a boat on a trailer down the road while talking to the cops to meet us.
I worked as a software dev, I had an intern to supervise. Everything was going well until he somehow got behind my back, tried some stuff on his own, and wiped out the whole database. Luckily we had backups, but I had to scramble to get it back and running as fast and possible, as I received dozens of calls or complaints almost instantly. For some reason, I was not allowed to modify permissions and accounts, and both me and the intern had superadmin privileges.
I broke the home page of a big tech (FAANG) company.
I added a call to an API created by another team. I did an initial test with 2% of production traffic + 50% of employee traffic, and it worked fine. After a day or two, I rolled out to 100% of users, and it broke the home page. It was broken for around 3 minutes until the deployment oncall found the killswitch I put in the code and turned it off. They noticed the issue quicker than I did.
What I didn’t realise was that only some of the methods of this class had Memcache caching. The method I was calling did not. It turns out it was running a database query on a DB with a single shard and only 4 replicas, that wasn’t designed for production traffic. As soon as my code rolled out to 100% of users. the DBs immediately fell over from tens of thousands of simultaneous connections.
Always use feature flags for risky work! It would have been broken for a lot longer if I didn’t add one and they had to re-deploy the site. The site was continuously pushed all day, but building and deploying could take 45+ mins.
I work on a SOC team and were really trying to hammer the usage of feature flags into our devs.
Always use feature flags for risky work! It would have been broken for a lot longer if I didn’t add one and they had to re-deploy the site. The site was continuously pushed all day, but building and deploying could take 45+ mins
This reminds me of the old saying: everyone has a test environment. Some people are lucky enough to have a separate production environment, too.
deleted by creator
