ZeroFlucs Reduces Infrastructure Cost by 60% with Feature Flag-Triggered Optimization

• Over 1,400 releases per month; 10+ releases per day per developer
• 98.8% of deployments are successful - not requiring a rollback
• 3-minute average build and deploy pipeline including testing, benchmarking, and Kubernetes rollout of over 100 services
• 50-60% drop in infrastructure cost within 1 week

About ZeroFlucs

ZeroFlucs, a startup located in Brisbane, Queensland, provides technology that enables bookmakers and wagering service providers to offer same-game betting. In a highly price-sensitive, risk-averse, and regulated market, the ZeroFlucs technology is enabling customers to double their margins. By running low-cost simulations with ZeroFlucs, customers are able to quantify the probability of a bet succeeding and use it to set the price for the transaction.

Challenge: Minimizing Cost to Customers to Survive in a Price-Sensitive Industry

Steve Gray, CEO and Founder of ZeroFlucs, knew that keeping the time taken to generate predictions low was critically important to his business – both to maximize profits for bookmakers, as well as provide engaging products for end users. “Numbers are our game. We use numbers in everything we do,” Gray said. The more simulations performed of a given match, the more accurate the pricing offered to customers becomes, but with each simulation, comes a cost. The challenge then was to minimize the cost while maximizing the accuracy of the analysis.

In his own words, automation and efficiency became the focus of the company and its culture right from the start. He wanted to “come out strong out of the gate with automation and a continuous integration and delivery (CI/CD) culture. All the successful teams I ever worked in had that culture first. They’ve been able to ship software continuously, reliably, and safely.”

The more the ZeroFlucs team could focus on being automation-first, the better they’d be able to meet their customer needs. The amount of processing required in their system directly impacts the cost their customers pay, especially when low bet simulation costs were a primary value proposition. In a price-sensitive and risk-averse market, it was important to show a demonstrable improvement in margins to get customers to even look at the product of such a young company.

Driving the cost per simulation as low as possible was the top priority for ZeroFlucs. According to Gray, “coming in as a new product that’s more expensive is a death knell in our industry. Cost and margins are absolutely the biggest things.” 

Solution: Leveraging Feature Flags to Run Betting Simulations as Efficiently as Possible

The first step was to build a massively parallel system that could run full-match simulations as quickly as possible. However, the team had yet to analyze the optimum configuration that would minimize the cost of running simulations, which would in turn enable them to run more simulations and boost the accuracy of the probabilities they output. What backend configuration would make the most sense to minimize cost and maximize performance?

As it would turn out, the answer would be feature flags. Gray and his team had 12 configuration points – such as the balance between storage and RAM usage and the number of servers to use to compute in parallel – and about four options at each configuration point, netting out to over 16 million possible permutations of configuration. 

Initially, the team was testing these configurations manually. And for each configuration, the application had to be redeployed, which took 5-10 minutes each time. Even with an automated script, this would have taken the ZeroFlucs team months to test every alternative in sequence. This extended time would directly impact their ability to provide value to their customers and maintain the viability of the company.

Gray then had the idea to leverage feature flags. Instead of sequentially deploying and testing one permutation at a time, they could put each of the configurations behind a feature flag and write a script that would test all those permutations live while in production.

The team was already familiar with LaunchDarkly, but it was cost-prohibitive for the sports betting market and also took away from the value ZeroFlucs offered to the market. When they found out that Harness, which they already used for CI/CD, offered the Feature Flags solution, it was a no-brainer to work with Harness again. 

“Feature Flags was a very easy and economical way to continue leveraging that investment with Harness,” Gray said. “It’s part of a holistic solution.”

Results: Passing on 50-60% Reduction in Costs to the Customer

Steve and the team were already leveraging Harness CI/CD for their deployments. After using Drone until it could not be scaled further, ZeroFlucs turned to Harness CI/CD. With 30 services, the effort associated with the upkeep of the scripts and pipelines took resources away from building the software. So, the team upgraded from Drone to Harness CI/CD to build and streamline those templates, resulting in:

• Over 100 services that are maintained by 10 deployment templates 
• Over 1,400 releases per month
• 98.8% of deployments are successful - not requiring a rollback
• Three-minute average build and deploy pipeline including testing, benchmarking, and Kubernetes deployment of the new versions
• 2x the number of services in less than 9 months

‍

With their success using Harness, it only made sense to extend their use of the Harness platform and implement Feature Flags. After all, this was a mission-critical part of the puzzle to minimize cost and maximize performance. 

Once Feature Flags was implemented and the necessary configurations were set, it became effortless for ZeroFlucs to test large numbers of permutations and find the most optimal configuration between cost and performance. Since implementing feature flags, ZeroFlucs has optimized their system for where they’re at, and they’re confident they can continue to optimize as they scale using the same feature flags. Today, they’re evaluating flags over one billion times per day.

Each developer was already releasing more than 10 times a day with Harness CI/CD, and when they expanded into Harness Feature Flags, ZeroFlucs gained the following benefits:

• From 5-10 mins per configuration test to 5-10 seconds
• Optimized their infrastructure within 1 week instead of months
• 50-60% drop in infrastructure cost
• 50,000 simulations run per second due to the reduced cost profile
• No performance impact - running over 1 billion flag evaluations per day
• 30-50% price advantage over competition

‍“With Harness, we have decreased infrastructure cost by 60% with no performance impact as we run over 1 billion flag evaluations per day. This allows us to maintain a price advantage over competitors!” - Steve Gray, CEO and Founder, ZeroFlucs 

According to Gray, they’ve only barely scratched the surface of what they can do with Harness. In fact, Gray and his team continue to use Harness Feature Flags in creative ways. As one example, the ZeroFlucs team was struggling to show off the intelligence in their system in a live demo to prospective customers because the technology worked so fast – normally, that’s a good thing. By strategically placing feature flags at key inflection points in the product, they were able to artificially slow down their demo so that prospects could see the technology at work and be properly wowed by what they were seeing.

By leveraging Harness CI/CD and Harness Feature Flags, ZeroFlucs has created a cost and performance-efficient system that will continue to be competitive as the business continues to mature. In a price-sensitive industry, being able to beat out the competition by 30-50% in price is sure to drive the growth ZeroFlucs is looking for. 

“We have never gotten an objection on price,” says Steve.

The Race for First Place

ZeroFlucs is just getting started leveraging Harness. In the near future, they’re looking to implement Feature Flags in more traditional ways, including using it to increase the velocity of their software delivery process. 

Gray’s vision is “to get human hands out of the release process as much as possible. Ideally, we want it all to go through a predefined and automated process.” With this automation-first culture, ZeroFlucs wants to leverage Harness to automate the rollout of new features and changes to their algorithm. They want to be able to create a release template that will ensure any changes meet their strict release criteria. 

While a lot of the work to date has been focused on optimisation of raw costs, another key area is managing the risks associated with changing a live, customer-facing system. By using Harness Feature Flags to incrementally roll out changes to users and evaluate performance versus the current solution, as ZeroFlucs evolves its models and algorithms they’ll be able to minimize the risk of regressions and ensure that customers get a consistent, profitable experience.

With Harness CI/CD speeding up and automating deployments and Harness Feature Flags optimizing their infrastructure, Gray and ZeroFlucs are turning their eyes to customer acquisition and how to leverage more of the Harness platform. By continuing to automate and create efficiencies, ZeroFlucs is ready to take on the sports betting market and completely disrupt the status quo.

“The biggest benefit of Harness is that essentially everything is automated and it's simple,” said Gray. “It's a cohesive package of tools, and all of it is usable, available, and approachable. My 12-year-old can use Harness to trigger a build pipeline!"