The ridiculous policy that forced engineers to come to the office at 2 AM twice a year
And how it may affect you
Twice a year, at 2:00 AM sharp, a group of engineers arrive at the office of Corporation Y. The poor engineers sit in front of their computers for 60 long minutes, turn them off, and go home.
Can you guess the cause?
First, let me tell you a bit about ‘Corporation Y’, where I worked.
Y is a government corporation with hundreds of software developers. Each engineering department in Y serves a different domain and can choose its own technological stack. Some use NodeJS, some C#, and some Python. Some React, and some Vue. 🤮
When it comes to Databases, there are fewer options. Y is closed off from the public web and runs on a separate network. As managing an on-prem infrastructure (like k8s clusters, VMs, and Databases) is a lot of work, it was decided to create a cross-organizational department, that will manage it all, and provide an IaaS solution (Infrastructure-as-a-Service).
Basically - an internal cloud provider. Great idea, right? 🤦♂️
I led the Database-as-a-Service (DBaaS) team, and we provided managed PostgreSQL and MongoDB solutions.
Back to our riddle. Twice a year, one of my team members (and one from each team in the department) had to be in the office at 2:00 AM.
Hint: this has to do with developers mismanaging dates…
It’s all about economics
..and Daylight Saving Time.
The concept is simple: clocks are set forward by one hour in the spring to extend evening daylight, and set back again in the fall to return to ‘standard’ time.
It was first proposed by Franklin in 1784 as a satirical suggestion to reduce candle usage. 100 years later, the New Zealander George Vernon proposed it again, so he would have more time to catch butterflies in the summer.
The first actual implementation was during World War I, by Germany and its allies. The thought was that extending daylight hours would reduce the need for artificial lighting in the evenings, saving fuel for the war effort.
Ok, enough chit-chat, back to tech.
How does this policy, which existed even before computers, affects us today?
How is DST achieved
There are 2 types of time changes. For simplicity, I’ll call them the ‘easy’ and the ‘hard’.
The easy
In the spring, clocks are set forward by one hour, usually at 2:00 AM local standard time, which becomes 3:00 AM local daylight time.
In terms of technology, you’ll have a ‘missing’ hour. There will be no activity in your systems from 2:00 AM to 3:00 AM. It’s not that bad, as it’s easily explainable.
The hard
During the fall, the process is reversed. Clocks are set back by one hour at 2:00 AM local daylight time, which becomes 1:00 AM local standard time. Here starts the mess, as a ‘duplicate’ hour occurs.
Imagine that you write software for a bank, and store the timestamps of transactions. Let’s see what happens when George tries to withdraw $10,000:
1:45 AM: George wins $10,000 in a casino, and deposits the money.
2:00 AM: The clock is shifted 1 AM.
1:15 AM: George reconsiders and withdraws the $10,000 to continue playing.If we judge based on those logs (and a VERY poorly written software), George’s withdrawal at 1:15 AM might be considered an overdraft, as it happened before he deposited the money. Sorting those logs by the timestamp will create a total mess.
So what the hell happened in Corporation Y?
As you remember - there were no real standards across the organization, and each development team did pretty much whatever they wanted.
Of course, many of them didn't consider Daylight Saving Time… So we had a bunch of systems that didn’t correctly manage the timestamps, storing them in local time instead of UTC (which doesn’t have Daylight Saving Time).
The information was sensitive to time, so it created a huge mess.
Instead of handling it, it was decided in some departments to just shut down all the applications for that ‘double-hour’…
There were also new applications popping up now and then, and as the knowledge sharing was so ‘great’, they too didn’t properly handle the cases. So every other year there was an urgent need at 2:00 AM to restore the DB to the state it was before the clock change (and this had to be done by my team, as we didn’t have self-service capabilities for point-in-time recovery).
Big corporations love rules and processes. So instead of formalizing the correct way to handle DST, they formalized the fact that for each clock switch (even if was the ‘easy’ one!), a developer from each infrastructure team had to be on-prem to solve potential issues… 🤷♂️
Some optimism
While I enjoyed bashing Corporation Y, the double-hour problem is not that easy to solve. Eventually, they got over it.
5 Points to consider
This old StackOverflow post covers dealing with DST pretty well. It depends a lot on your technology, business domain, and customer needs.
Few points to think about in your systems:
1. Storing the time
You should probably store the time based on a unified standard that is not affected by daylight savings.
If you are using PostgreSQL - use the timestamptz (vs the timestamp) column. timestamptz is always stored as UTC, so Postgres and your application know the exact moment in time it represents.
2. Displaying the time during the double hour
There are a few ways to solve it, and it depends on the role of timestamps in your application. If your application significantly relies on time (like scheduling or calendar applications), consider notifying users about how the time change will be handled.
You can also just add the timezone near the time, so it’ll change from 1:59 AM EDT (Eastern Daylight Time) to 1:00 AM EST (Eastern Standard Time), for example.
3. Scheduling during the double/missing hour
When scheduling jobs or performing time-based calculations, be mindful of the double hour (and missing hour) - things might be repeated or skipped.
4. UTC is NOT the answer to everything
If you need a task to run at 7 AM for each user, in the user’s timezone. Let’s say all your users are on UTC+3 time, and you schedule the task for 4 AM. Once a clock change occurs, the time of the task will change.
Sometimes you do need the data in the user’s local time.
5. Governments do weird things
For example:
Standard time in the Netherlands was exactly 19 minutes and 32.13 seconds ahead of UTC by law from 1909-05-01 through 1937-06-30.
Less weird but still annoying, governments change the timing of the clock changes, and you need to make sure you support it. And each goverement has a different vision of the right way to do it…
Dealing with stupid organizational policies
3 takeaways if you have similar absurd situations:
Fight them
There is nothing more demotivating than stupidity. Don’t accept any norms because ‘that’s how it always happened’. For example in this case - why would we need to be in the office during the ‘easy’ clock switch?
Take part in the load
Until you are able to change it, you have to follow those rules yourself, otherwise it’s even more demotivating for the team. In the above case, I made sure to take a couple of those shifts myself.
Don’t make the issue bigger than it is
Yeah, it’s stupid, and it sucks, but in a team of 5 engineers, it was 1 hour every 2.5 years… It’s more of a sad joke than a real inconvenience.
Final Words
If you have some horror stories about DST, I would love to hear them :)
If you don’t want to become Corporation Y, I suggest reading some engineering newsletters!
and are 2 of my favorites:
DST is a pain. The graphs always look annoying during the harder time change. Thanks for sharing!
Thanks for sharing… I feel like every time i need to work with date & time, it is an annoying step in the development. What kind of dateTime string does the other interface send me, is it documented, which timezone is set for the timestamp etc.
So yeah - DST is another level of annoyance😅