The funny thing is that we complain that feature flags can become messy and complex, so we create a system to control that mess, which in turn introduces more complexity (Hodgson 4 categories).
Or maybe everything started that way and under the hood of every infrastructure is an orchestrated mess? 🤔
It’s an addiction for sure and gives us that reassurance that we can turn a feature off without redeploying or re-releasing /hot fixing.
I’m would say that FF are very handy during experimentation and sampling specific features across a group set of customers.
That said, there are many redundant FF on our current code base which requires some house keeping and FF that have become ‘Business as usual’ thus, dead code.
Static analysis tools don’t like them either with increasing amount of cyclomatic complexity.
Good read! I completely agree a lot of feature flags can make for messy code. And your observation on clean up is spot on.
We only use feature flags for big releases or features we think about releasing to gauge as an experiment before releasing to all. Bug fixes, minor design updates, etc are never under feature flags. As you mentioned, part of the epic for the feature has a ticket to clean up feature flag once it’s out to all users so that’s part of our definition of done which has been super helpful!
Great read, Anton! I believe that the number of feature flags should be monitored for each release and kept under control, ideally enforced in the pipeline.
This is an excellent article, Anton! Feature flags are everywhere, but the distinction you made between them is essential: feature, experimental, and permission toggles.
The biggest cost to these flags is, indeed, the dead code and what comes with it.
Being part of a refactoring project that lasted a year, I caught myself more than once spinning my wheels on a component that was used in a component that was behind a feature flag that never was enabled and was essentially canceled.
It requires a strong engineering culture to cut these ends.
In my previous team, we managed to remove release toggles with tickets. We always had a release meeting to celebrate the release and turn the toggle on. After that, someone used to take the task to remove it. It was more about a cultural mindset than relying on the ticket itself. Everyone was aligned, including the PM. It was something expected as output. As we kept our backlog clean most of the time, it was easy to see if something was pending.
I also saw codebases with combinations of feature toggles, which were chaotic. Seeing things like "if (featureA && featureB)..." made me crazy. Thinking about refactoring something like this that was managed in an external tool was just insane.
Engineers are the owners of the technical solutions, and feature toggles is just one thing among many others that can become unsustainable. Many engineers struggle to advocate for solutions that make sense before the damage is too big because they often don't know the business enough to buy people in. The ones I saw succeeding, were what we are calling product engineers.
It is easy, you just need to wait for the planets to align 😂
Kidding, it took a while but everyone developed the mindset "if we haven't touched it for so long or the risk to forget exists, it isn't so important to keep it on the team backlog"
Fantastic writeup. Thank you for diving so deep into this subject. Feature flags are great, but need to be managed. They are not magic and just adding them nilly-willy causes issues. If you manage them, they are a super power.
One sticking point: The flags that are cited in the various Knight Capital Group collapse writeups were order flags, not feature flags. Despite sharing the word "flag" they are not similar things. Order flags are a way of describing the type of an order. The KCG example is not a good example of feature flags gone wrong.
Key for me has always been setting a hard date for removal of feature flag or its wrapped feature. No in between no what ifs, we will wrap the flag but we set a hard date of removal before we put the flag in code. Lot of push back but end up getting the date and able to hold people accountable.
If there is a tool they never gave it to us 🤣. We set feature flag removal days usually every other month and if the feature flags job was complete we removed it if it’s defined complete date matched up. It was written down agreement between devs and business.
So discipline, takes a little more to get but on and really hard to get business to commit to a date.
Great article Anton!
I really enjoyed reading it and learned a bunch!
The funny thing is that we complain that feature flags can become messy and complex, so we create a system to control that mess, which in turn introduces more complexity (Hodgson 4 categories).
Or maybe everything started that way and under the hood of every infrastructure is an orchestrated mess? 🤔
I think the system is mainly a thought excerise - I don’t think we should have 4 different systems under the hood, especially in small organizations.
Great article with useful insights. Cleaning up feature flags can be automated. Here is a demo. Get in touch at gitar.co for early access!
https://www.youtube.com/watch?v=QWdwfqF8o_I
As what others are saying, this hits home.
It’s an addiction for sure and gives us that reassurance that we can turn a feature off without redeploying or re-releasing /hot fixing.
I’m would say that FF are very handy during experimentation and sampling specific features across a group set of customers.
That said, there are many redundant FF on our current code base which requires some house keeping and FF that have become ‘Business as usual’ thus, dead code.
Static analysis tools don’t like them either with increasing amount of cyclomatic complexity.
I guess it’s a ‘use but don’t abuse’ thing.
'Use but don't abuse' is a good summary :)
Good read! I completely agree a lot of feature flags can make for messy code. And your observation on clean up is spot on.
We only use feature flags for big releases or features we think about releasing to gauge as an experiment before releasing to all. Bug fixes, minor design updates, etc are never under feature flags. As you mentioned, part of the epic for the feature has a ticket to clean up feature flag once it’s out to all users so that’s part of our definition of done which has been super helpful!
That sounds like a healthy culture :)
Great read, Anton! I believe that the number of feature flags should be monitored for each release and kept under control, ideally enforced in the pipeline.
Thank you Bogdan!
I agree, I still haven’t seen a good automatic solution though.
This is an excellent article, Anton! Feature flags are everywhere, but the distinction you made between them is essential: feature, experimental, and permission toggles.
The biggest cost to these flags is, indeed, the dead code and what comes with it.
Being part of a refactoring project that lasted a year, I caught myself more than once spinning my wheels on a component that was used in a component that was behind a feature flag that never was enabled and was essentially canceled.
It requires a strong engineering culture to cut these ends.
I was not sure how common was the scenario you mentioned, but it seems we are not alone :)
It requires good processes, which come from good culture.
Great topic Anton!
In my previous team, we managed to remove release toggles with tickets. We always had a release meeting to celebrate the release and turn the toggle on. After that, someone used to take the task to remove it. It was more about a cultural mindset than relying on the ticket itself. Everyone was aligned, including the PM. It was something expected as output. As we kept our backlog clean most of the time, it was easy to see if something was pending.
I also saw codebases with combinations of feature toggles, which were chaotic. Seeing things like "if (featureA && featureB)..." made me crazy. Thinking about refactoring something like this that was managed in an external tool was just insane.
Engineers are the owners of the technical solutions, and feature toggles is just one thing among many others that can become unsustainable. Many engineers struggle to advocate for solutions that make sense before the damage is too big because they often don't know the business enough to buy people in. The ones I saw succeeding, were what we are calling product engineers.
“We kept our backlog clean” 😱 - teach me how to do that 😂
It is easy, you just need to wait for the planets to align 😂
Kidding, it took a while but everyone developed the mindset "if we haven't touched it for so long or the risk to forget exists, it isn't so important to keep it on the team backlog"
Fantastic writeup. Thank you for diving so deep into this subject. Feature flags are great, but need to be managed. They are not magic and just adding them nilly-willy causes issues. If you manage them, they are a super power.
One sticking point: The flags that are cited in the various Knight Capital Group collapse writeups were order flags, not feature flags. Despite sharing the word "flag" they are not similar things. Order flags are a way of describing the type of an order. The KCG example is not a good example of feature flags gone wrong.
Thank you Krystian. You are right, the story is not 100% related, just a very good one :)
I think it shows the dangers of have ‘latent’ code, which somehow relates to the topic 🙃
This article hits home 😅. On each point, I was like, “Yep… 😭”.
It’s something you need to take an active role and process for managing.
Thanks for the article, Anton
It took me a while to understand we are not alone in this suffering :)
Key for me has always been setting a hard date for removal of feature flag or its wrapped feature. No in between no what ifs, we will wrap the flag but we set a hard date of removal before we put the flag in code. Lot of push back but end up getting the date and able to hold people accountable.
Is there a way you enforce it except discipline? Some automatic tool?
If there is a tool they never gave it to us 🤣. We set feature flag removal days usually every other month and if the feature flags job was complete we removed it if it’s defined complete date matched up. It was written down agreement between devs and business.
So discipline, takes a little more to get but on and really hard to get business to commit to a date.
The only reason toggles get out of hand is the same age-old problem that ALL messy codebases have - a lack of discipline.