I loved the article! I love listening & telling "how I messed up" stories. They always have the fun & learning part.
"I felt physically ill, and helpless." I know the feeling :) The feeling of receding blood flow from your limbs and it gushing to your head. That scary moment after you mess up something big and before you know you have to fix it.
"For those of you not working in startups - please don’t judge 😅"
I won't! Meta had something similar back in the day. I remember a new hire from MSFT being in awe when I hot-fixed an issue and ran it from my dev server.
You are right, that would have saved me. I've known about that option, but honestly I was just to cocky to use it that time. I've been much more careful since...
this is a soft delete. reverting could be done in seconds with;
UPDATE orders SET is_deleted = false
it is true that previously deleted orders will come up. But no one will notice the system is down. previously deleted orders can be get from backups and everything will be OK in less than an hour without system going down.
I simplified in the article, the orders are in fact internal ‘missons’, that our operations team needed to execute (I work at an Agtech company, and each such mission is a drone flight above a field).
Due to some bug, there were duplicate missions created, and it messed up the systems and confusing the operations team.
In case I would have revived everything, it would have created ton of complications with them, and it was better to stop them for an hour then risk flying wrong missions.
The new format was great, Anton and the takeaways were insightful too. Helped me realize how I act afterwards is a delicate balance too and I could probably do things better when I’ve been in similar situations
I loved the article! I love listening & telling "how I messed up" stories. They always have the fun & learning part.
"I felt physically ill, and helpless." I know the feeling :) The feeling of receding blood flow from your limbs and it gushing to your head. That scary moment after you mess up something big and before you know you have to fix it.
"For those of you not working in startups - please don’t judge 😅"
I won't! Meta had something similar back in the day. I remember a new hire from MSFT being in awe when I hot-fixed an issue and ran it from my dev server.
"That scary moment after you mess up something big and before you know you have to fix it." Exactly... That's a scary feeling :)
Thank you!
The article is both entertaining and educational. Been there, done that.
FWIW, we are building a tool called Bytebase (bytebase.com) that is supposed to prevent this happening.
Thanks :)
Cool, I’ll take a look.
https://stackoverflow.com/questions/76162823/dbeaver-issue-with-query-and-new-line
oops
Yeah :/
I disabled it afterwards, but it's a damn strange default
I don’t see you mention transactions anywhere in your post. Would using them have made a difference?
You are right, that would have saved me. I've known about that option, but honestly I was just to cocky to use it that time. I've been much more careful since...
To be honest I found this really funny but I was able to take some valuable lessons from it. Thank you!
In retrospect, I find it funny too 😅
But turns out such small and stupid mistakes are common 🙃
this is a soft delete. reverting could be done in seconds with;
UPDATE orders SET is_deleted = false
it is true that previously deleted orders will come up. But no one will notice the system is down. previously deleted orders can be get from backups and everything will be OK in less than an hour without system going down.
I simplified in the article, the orders are in fact internal ‘missons’, that our operations team needed to execute (I work at an Agtech company, and each such mission is a drone flight above a field).
Due to some bug, there were duplicate missions created, and it messed up the systems and confusing the operations team.
In case I would have revived everything, it would have created ton of complications with them, and it was better to stop them for an hour then risk flying wrong missions.
The new format was great, Anton and the takeaways were insightful too. Helped me realize how I act afterwards is a delicate balance too and I could probably do things better when I’ve been in similar situations
Thank for the feedback Jordan! I think I'll try to do it once in a while, to spice things up.
It's easy for me in hindsight, when you are in such a moment... Usually the last thing that crosses your mind is how you should act :)