I loved the article! I love listening & telling "how I messed up" stories. They always have the fun & learning part.

"I felt physically ill, and helpless." I know the feeling :) The feeling of receding blood flow from your limbs and it gushing to your head. That scary moment after you mess up something big and before you know you have to fix it.

"For those of you not working in startups - please don’t judge πŸ˜…"

I won't! Meta had something similar back in the day. I remember a new hire from MSFT being in awe when I hot-fixed an issue and ran it from my dev server.

Expand full comment
Jan 8Β·edited Jan 8Liked by Anton Zaides

The article is both entertaining and educational. Been there, done that.

FWIW, we are building a tool called Bytebase (bytebase.com) that is supposed to prevent this happening.

Expand full comment
Jan 1Liked by Anton Zaides

I don’t see you mention transactions anywhere in your post. Would using them have made a difference?

Expand full comment
Sep 26, 2023Liked by Anton Zaides

To be honest I found this really funny but I was able to take some valuable lessons from it. Thank you!

Expand full comment

this is a soft delete. reverting could be done in seconds with;

UPDATE orders SET is_deleted = false

it is true that previously deleted orders will come up. But no one will notice the system is down. previously deleted orders can be get from backups and everything will be OK in less than an hour without system going down.

Expand full comment

The new format was great, Anton and the takeaways were insightful too. Helped me realize how I act afterwards is a delicate balance too and I could probably do things better when I’ve been in similar situations

Expand full comment