Citi Improves Software Delivery Performance, Reduces Toil With Harness CD

About

Citi is a preeminent banking partner for institutions with cross-border needs, a global leader in wealth management and a valued personal bank in its home market of the United States. Citi does business in nearly 160 countries and jurisdictions, providing corporations, governments, investors, institutions and individuals with a broad range of financial products and services.

Challenge: Implement Continuous Delivery at Enterprise Scale While Maintaining a Focus on Risk and Controls

Citi’s next-generation software delivery platform provides an integrated experience across all stages of software delivery with a user base of over 20,000 people across the bank’s software engineering, site reliability engineering (SRE), product, and risk and control functions. The platform brings together all the tools and services involved in software delivery with the aim of improving performance, consistency, and maintenance across the enterprise. And all this while operating in the unique regulatory and risk management environment that comes with being one of the biggest banks in the world.

As Stefanos Piperoglou, Technical Program Manager for the platform, explained, the deployment solution in place at Citi was not meeting their requirements:

• It did not provide continuous delivery / continuous deployment (CD) capabilities
• It required manual configuration and a centralized operations team
• It was not cloud-native and did not play well with modern microservice architectures
• It was difficult to provide a clear audit trail of the hundreds of thousands of production releases that are executed annually

After evaluating several other solutions, Citi landed on Harness Continuous Delivery, which met all of its current needs and enabled its ambitions to host the majority of its software in the public cloud.

Solution: Enter Harness CD

With the adoption of Harness Continuous Delivery, the platform team was able to automate the continuous integration and continuous delivery (CI/CD) process end to end, helping deliver on its goals – and more.

Move Fast, Deploy Often, Roll Back Quickly

Improving Citi’s core DORA outcomes – Deployment Frequency, Lead Time for Changes, Change Failure Rate and Time to Restore Service – is at the core of the platform’s mission. Harness CD allows Citi to construct CD pipelines that go through all necessary deployment, testing, audit logging, and change management stages after every code change is reviewed and accepted. It allows ops teams to quickly roll back any change that causes an issue in production. And Harness CD’s integration with Citi’s other observability tools allows CD pipelines to alert ops teams or even automatically roll back changes if they cause any unexpected behaviour in production systems.

“Harness CD lets us release each change within minutes of a pull request being merged. Running all the necessary tests and scans during the pipeline gives us the confidence to do this. Using Harness’s implementations of deployment strategies such as Canary and Blue/Green allows us to verify deployments while minimizing the blast radius of any problems, and the UX allows us to quickly roll back any change with minimal fuss as soon as we detect a problem,” Piperoglou said.

Flexibility with Control

Development teams at Citi operate in an environment of much higher scrutiny and control than is typical in many other industries. The high standards set by financial services regulators in the various jurisdictions in which Citi operates, and the unique security posture required by such a high-profile target, mean that delivering software to production involves an extensive collection of controls. Application teams are frequently subject to internal and external audits, security threat modelling exercises, and penetration tests. It is critical to keep all of this process in place, but equally important not to affect developer productivity or slow down deliveries.

“Harness CD’s template system and OPA framework lets us give application teams the flexibility to design CD pipelines in a way that fits the requirements of the businesses they serve while being sure they are following all required and recommended practices. It lets users enforce policy and promote standardisation without restricting what engineers can achieve,” said Michael Freeman, lead software engineer for the platform.

Cloud-Native, but not Cloud-Only

Citi has been adopting cloud patterns such as containers and Kubernetes on-premises for several years, and is also ramping up the migration of many of its applications to various public cloud providers. Meanwhile, many applications will continue to be hosted either on Citi’s premises, or on High-Frequency Trading (HFT) platforms that need dedicated, physical infrastructure for the foreseeable future. It was critical that whatever CD solution Citi adopted would cater to both on- and off-premises use cases and allow for a smooth, gradual migration to the cloud while maintaining consistency for users on their journey.

Result: Improved Software Delivery Performance, Reduced Toil

With Citi’s previous deployment solution, teams were releasing code with a cadence that usually ranged from once per week to once every several months. “The developer was the pipeline. They’d sit in front of a screen and click to start deployment for each environment promotion, while manually entering change management details and attesting that all necessary tests and controls had been completed. That process was pure toil and took hours, sometimes days,” said Piperoglou. “Harness CD allowed us to automate all of this and go from build to running in production in under 7 minutes, with most of our users performing production deployments several times a day.”

Moving to the Cloud with Confidence

Operating in a highly regulated environment and with a backdrop of increasingly frequent and sophisticated attacks on the software supply chain, Citi’s top-speed innovation is enabled by Harness’s industry experience. “Working with Harness’s solutions team has helped us validate our ideas and understand what our peers in financial services as well as other industries are doing in this space. Building on top of a widely adopted platform that has been proven successful gives you an extra layer of confidence and provides valuable input into the design process,” said Freeman.

No Looking Back

With the Harness Continuous Delivery solution, Citi has given its application developers and operations teams control, ease of use, and unmatched speed for its deployment pipeline while eliminating toil, reducing risk, and increasing security. Above all, it has allowed Citi to deliver new and innovative solutions that help its clients achieve their goals and prosper today and in the future.