And survived to talk about it
I loved the article! I love listening & telling "how I messed up" stories. They always have the fun & learning part.
"I felt physically ill, and helpless." I know the feeling :) The feeling of receding blood flow from your limbs and it gushing to your head. That scary moment after you mess up something big and before you know you have to fix it.
"For those of you not working in startups - please don’t judge 😅"
I won't! Meta had something similar back in the day. I remember a new hire from MSFT being in awe when I hot-fixed an issue and ran it from my dev server.
The article is both entertaining and educational. Been there, done that.
FWIW, we are building a tool called Bytebase (bytebase.com) that is supposed to prevent this happening.
I don’t see you mention transactions anywhere in your post. Would using them have made a difference?
To be honest I found this really funny but I was able to take some valuable lessons from it. Thank you!
this is a soft delete. reverting could be done in seconds with;
UPDATE orders SET is_deleted = false
it is true that previously deleted orders will come up. But no one will notice the system is down. previously deleted orders can be get from backups and everything will be OK in less than an hour without system going down.
The new format was great, Anton and the takeaways were insightful too. Helped me realize how I act afterwards is a delicate balance too and I could probably do things better when I’ve been in similar situations