Entur Improves Deployment Frequency from Twice a Week to 14 Times a Day, Rollback Time from Two Hours to Five Minutes

• Increased deployment frequency from twice per week to 14 times per day
• Decreased change failure rate from 20% to 5%
• Reduced rollback time from two hours to five minutes

About

Entur’s mission is to create a more sustainable path for public transport in Norway. For this to happen, they must create new and unified solutions for travelers across multiple train network operators. To meet this need, Entur provides a solution to gather, enhance, and share data for public transport to help simplify travel across Norway.

Challenge: Stabilizing Pre-Production Environments for Scale

As the number of development teams nearly doubled (12 teams became 20 teams), Entur recognized their growth as an inflection point - an opportunity to reimagine how they deliver software. Each dev team had a different methodology for how they deliver and operate deployments. Therefore, a new, more unified approach was required to accommodate their immediate and future growth if they desired the ability to accelerate and become more efficient. 

Entur’s current state included three major environments: development, staging, and production – either deploying to production manually or employing CircleCI for continuous integration (CI) and continuous delivery (CD). While they were happy with their CI process, they had reached the velocity limitations of their custom scripted CircleCI pipelines in terms of deployment, and they were drowning in the volume of maintenance required to keep them up and running. According to Tommy Bø, DevOps Engineer at Entur, “These CircleCI pipelines were configured using thousands of YAML lines, making them a nightmare to maintain and update.”

The team needed to reduce the amount of maintenance required to keep their DevOps toolchains functioning and standardize on a deployment method for Entur that would deliver the kind of velocity gains they would need as their business continued to scale. They were deploying software on a bi-weekly basis and were experiencing a 20% change failure rate. Failure observed in a production deployment required a two-hour manual roll-back. These metrics weren’t even close to where they needed to be to enable growth for their business. 

Additionally, Entur developers were spending far too much time maintaining DevOps tools. They were also concerned about deployment pipeline stability when they did need to use those tools to deploy and were operating with the worry of unintentionally causing a major outage – far from an ideal developer experience. 

The pain developers were feeling trying to scale velocity in their current software development lifecycle (SDLC) was clear. Bø explained that “whenever you have to manually deploy, you have to configure every environment correctly. That wasn’t feasible as we were scaling.”

Feature management was also a functionality that Entur desired to leverage in their SDLC, but were making use of an in-house developed system that wasn’t meeting their needs. As new team members joined and new features were being developed, the process of controlling the flag state of newly developed features proved to be inefficient.

Christian Nyvoll, Systems Developer at Entur, remarked that “the way we were doing feature toggling slowed down our deployment process. We wanted to be able to pick up a new task and start doing it without thinking about what other team members are doing.”

The desire to alleviate these challenges led Entur to distill down their requirements for a new SDLC solution that was able to automate and execute on:

• Rolling deployments
• Canary deployments
• Instant rollbacks on failures
• Infrastructure provisioning through the deployment process

Bø shared that “we didn’t want a tool that we would have to manage ourselves, so that developers could focus on their domain and on our core business.”

Solution: Intuitive, Self-Service CD Tooling

Entur chose Harness to simplify a complex continuous delivery (CD) process, reduce the amount of DevOps tools maintenance, reduce complexity in feature management, and provide developers with self-service tooling. According to Bø, “We chose Harness for the compelling user interface and intuitive workflows. It’s a modern, state-of-the-art solution for software delivery that is very suitable for cloud services and microservices architecture applications.”

Nyvoll added: “Everyone has been pleased with the usability of Harness and how easy it is to actually just click in the UI. No need to learn commands. You just click the service that you want to deploy for or roll back for and the version that you want to use.”

Result: Higher Deployment Velocity, More Control in Production

Harness CD gave Entur’s developers self-service deployment capabilities without worrying about breaking their DevOps toolchain in the process. “Our deployment velocity increased from twice per week to more than 14 times per day with Harness. The standardized and automated process that we now employ makes it easier to deliver smaller features more often and with lower risks. In fact, our change failure rate decreased from 20% to 5%,” said Bø. Even if a deployment failure occurred, Entur could roll-back by just clicking a button. In under 5 minutes, the failing deployment would be rolled back via automation instead of making use of the thousands of lines of YAML that comprised their previous rollback solution.

Using the role-based access control (RBAC) capability within Harness, Entur can segment permissions and privileges by team, thereby decreasing the risk of accidentally making changes to other people’s systems. Furthermore, templatized pipelines have let the team increase control of the approval process as well as more automated deployments. Bø shared: “It's easier to track who's responsible for deployments using Harness. We can trace changes, not only internally within the team, but also across teams. This has a lot of value, especially for the support team that’s trying to identify the cause of an incident.”

As for resolving issues caused by feature toggles, Nyvoll and his team now leverage Harness Feature Flags (FF) to reduce blockers when testing features, to get features out the door quicker, and to gain better control when feature errors arise. With both Harness CD and Harness FF, the Entur team is now delivering more features, more frequently, as well as remedying production issues in minutes instead of hours.

Nyvoll observed, “Now that we have Harness Feature Flags established in our team, we don’t need to worry about everyone else during deployment. This makes it much easier to deal with complexities stemming from the increasing number of applications and services. And then when something is going wrong in production or in a staging environment, we have the ability to turn it off.”

Entur is not only seeing improvements in deployment velocity and reductions in the time it takes to resolve issues, but also seeing an overall reduction in stress levels given the significant improvement in developer experience with Harness. Making the shift to Harness has equipped Entur’s development teams with the software delivery platform they require to serve the transportation needs of Norway today and in the future.