Ancestry Adds Consistency and Governance to Cut Downtime and Systems Onboarding Effort

Harness Impact

• Centralized governance to drive deployment consistency and best practices
• 78% reduction in systems onboarding effort
• 50% decrease in deployment caused production outages

About

For more than 30 years, Ancestry has been using cutting-edge technologies, ongoing innovation in DNA science, digitizing historical records, and automated research to help customers create a more complete family portrait. It has invested over $300 million to build the world’s largest collection of family history records and continuously launch new products and services for journeys of personal discovery. Now, with over 10,000 terabytes of data, $1 billion in revenue, and 1,400 employees, Ancestry has become a global leader in family history and consumer genomics.

Challenges

Ancestry built its massive business by helping people uncover information about their family histories. But running such a popular online business at scale for more than three million paying subscribers and over 125 million family trees across a growing portfolio of brands, products, and services requires an equally massive development effort. As Ancestry ramped up its development team with new hires while maintaining 30-plus years of development history, its deployment processes became inconsistent and difficult to govern. That put undue strain on the company’s architecture team, which was charged with standardizing and adding best practices to its deployment processes. 

“We’ve been growing our engineering base, onboarding new developers, and creating new teams for new products,” explained Ken Angell, Principal Architect at Ancestry. “Ancestry has also been around for a long time and our development processes were left up to the developers. With everyone focused on new products, we didn’t have the time to guide teams on processes and best practices.” 

With most of its developers already familiar with Jenkins, Ancestry managed delivery and integration using Jenkins as a CI solution extended to CD. But every team had a different process for building and shipping code. Without any consolidated governance, the company had little consistency in how products were developed, when code was pushed live, if quality checks were in place, or if a canary was deployed before going live. It left Ancestry with 80 to 100 distinct Jenkins instances, requiring additional cost to be stood up and maintained.

“We were acting like we had an advanced CI/CD culture with the capability to deploy multiple times a day,” said Russ Barnett, Chief Architect at Ancestry. “The truth is, we didn’t have the infrastructure to support that for all of our teams.”

“It was up to us, the architecture team, to drive standards and best practices, but we had no controls in place to do so,” added Angell. “In an effort to deliver valuable features to Ancestry’s customers, teams lacked the tools to deliver that value quickly without occasionally causing temporary disruptions in service. As a company, we wanted to deliver value quickly and maintain the highest quality service.”

With outages hitting the company’s bottom line, executives pushed Angell’s team to strive for three-nines availability, or just eight hours and 46 minutes of downtime per year. But, to hit 99.9% uptime, its deployment processes had to change. 

“We had the standard processes and best practices already documented, but we had no ability to govern it,” Angell said.

Ancestry decided to leverage the expertise of the company’s software engineers to identify best practices and spread those throughout Ancestry. However, doing this manually and team-by-team, per CI pipeline, became unmanageable and Angell had no assurance that any requested change had actually been implemented. For DevOps, it was also a time- and energy-intensive process since each change required constant follow-up that overstretched the team’s bandwidth.

Solution

Realizing the development time for a custom solution would be measured in years, not months, Ancestry stepped back to examine its requirements. The team started looking for a third-party solution that would be much faster to deploy and, looking forward, could manage both CI and CD on the same platform. Any solution would also need to standardize deployment and integration processes without hampering developers’ ability to quickly ideate, innovate, and maintain new products.

“We considered each vendor’s governance capabilities, security model, breadth of coverage of different deployment types, and ability to configure with code,” Angell explained. “When it came down to it, Harness met our requirements and was one of few with an Elastic Container Service (ECS) process out of the box.”

Partnering with different stakeholders (Security, Ops, Platform, etc.) allowed Angell to get a better view of the challenges and potential solutions. Once it selected Harness, Ancestry was able to onboard 350 systems in the first year and an 85% reduction in effort when compared to Jenkins. Ancestry also expected full adoption across all systems by the end of year two.

Results

Harness gave Ancestry a new standard for pipelines that were easy to adopt, change, and govern. That oversight helped to quickly ensure canary deployments became the default Ancestry deployment strategy. Even more, Ancestry’s centralized testing tool’s adoption rate rose to 97%.

“Once teams are using a Harness pipeline, it’s easy to add additional mandatory steps,” said Barnett. “Our teams were initially skeptical of getting rid of their Jenkins pipelines for Harness, but now they’re big fans of the Harness user interface and the ease it brings to the deployment process.”

Its prior Jenkins approach required slow, manual changes to each of its 80 pipelines, but Harness lets Ancestry now make a single change that applies to all pipelines. The increase in pipeline governance and consistent usage of best practices enabled by Harness also resulted in a 50% decrease in overall production outages and significantly reduced outages related to deployment issues. Harness further helped Ancestry increase developer productivity with a 3x increase in deployment velocity.

“Harness now empowers Ancestry to implement new features once and then automatically extend those across every pipeline, representing an 80-to-1 reduction in developer effort.” - Ken Angell, Principal Architect at Ancestry.

Now, as Ancestry continues to bolster its CD governance and compliance efforts, it’s looking to expand its use of Harness up the DevOps process into CI. Ancestry is also taking advantage of Harness options for cloud hosting, zero touch maintenance, and automatic backups, all of which give Angell and team more time to build a robust, consistent, and governed CI/CD process.

“I’m going to focus on improving availability and our ability to govern change management in production environments,” concluded Angell. “We’re letting Harness take care of the infrastructure.”