Carvana Cuts Infrastructure Provisioning Time by 92%, Saves $1.4 Million With Granular Cloud Cost Management

Harness Impact

• 92% decrease in infrastructure resource provisioning time
• $1.4M annualized savings in cloud spend
• Granular cost insights for cloud infrastructure 
• 4 months to deploy automated provisioning

About

Founded in 2012, Carvana is on a mission to change the way people buy and sell used cars. By focusing on customers, technology, and innovation, Carvana offers an intuitive and convenient online car buying, selling, and financing experience. Its customers can search and browse more than 70,000 vehicles online or visit one of its more than 30 automated car vending machines. By cutting out dealership overhead, Carvana saves customers thousands of dollars per purchase and provides a better car buying experience.

Challenges

Carvana’s online experience and logistical efficiency is driven by the company’s proprietary technology created by its innovative development team. Supporting that development is the company’s eight-person infrastructure team, which has been using a script-based process to provision new instances. When a request for new infrastructure came along, the team created a custom image and deployed each one manually: an engineer would run scripts to build the machine, run scripts to install software, and run scripts to join the domain. The process took 2 to 3 hours for each of the 5 images requested per week. 

“We manage three different cloud environments: AWS, GCP, and Azure,” explained Will Adams, Manager, Cloud Support at Carvana. “We do a lot of build outs so we knew we needed to automate.” 

As Carvana’s business grew in the years leading up to and during the pandemic, this manual provisioning process threatened to put the brakes on growth. Unfortunately, with up to 15 hours per week devoted to manual deployments, the team just didn’t have the time to focus on automation. This put the infrastructure team under added pressure. But then, as car shortages and rising interest rates caused a drop in car sales in 2022, Carvana had to move from speeding infrastructure deployments to quickly trimming excessive infrastructure costs. 

Solution

First, Carvana’s infrastructure team set out to automate their highly manual provisioning processes with a project they termed TURBO, The Universal Request to Build Orchestrator. The team was using Jira to accept incoming infrastructure requests and wanted to add Terraform to automate the provisioning. It initially considered using a third-party to develop a custom solution, but soon realized that would be a painful, slow process and an ongoing maintenance nightmare. Then they found Harness. 

“Harness was open enough to orchestrate Carvana’s custom build process,” Adams explained. “We wouldn’t have to wait for something to be built. Harness could orchestrate our capabilities right away and get our infrastructure automation up and running as quickly as possible.”

After a quick proof of concept, Carvana had an automated provisioning process in just 4 months. With Harness doing the orchestration, engineers could make a provisioning request in Jira and virtual machines would be automatically built using Terraform. It was just in time, too. As Carvana’s business boomed in 2021—the company went from 7,000 to 21,000 employees—the team had to provision thousands of new machines for those now working from home due to COVID restrictions.

“We just have to approve a build and it happens,” added Richard Taylor, Senior Cloud Infrastructure Engineer at Carvana. “Automation became crucial for us not to become paralyzed by sheer growth.”

As the post-pandemic economy shifted, however, vehicles were in short supply and rising interest rates further dampened the used car market. Carvana pivoted to now focusing on costs. For the infrastructure team, however, with instances spread across cloud environments and managing teams, the view was murky at best. 

“We needed to understand our cloud costs because we were getting a lot of questions,” said Adams. “Harness Cloud Cost Management helped a lot. The anomaly detection saved a lot of time for me by pinpointing what was happening and how our bill changed month to month. The product is pretty awesome.”

Cloud Cost Management also helps Carvana answer cost-related questions related to its different vendors, services, and environments by providing monthly or even daily insights upon request. It’s also looking to expand the use of Intelligent Cloud AutoStopping to power down more systems when they’re not in use and to expand its use deeper into Azure and GCP.

“We understand now, when we run this environment for five days, this is what the cost is. So now I'm going to run it for three days or I'm going to power it off when I don't need it. Cloud Cost Management helps us immensely.” - Will Adams, Manager, Cloud Support at Carvana

Results

Harness helped Carvana decrease manual effort and increase scalability during a huge growth period. Then, as Carvana needed to cut costs, Harness helped Carvana build on those automation gains while providing deeper and more granular insights into its cloud infrastructure costs. 

Harness orchestration has helped Carvana cut 92% from its infrastructure provisioning time. What used to take typically 2 to 3 hours and as long as 4 hours can now be completed automatically in as little as 20 minutes. At 5 instances per week, Carvana can redirect approximately 700 annual hours of expensive infrastructure engineering resources.

Harness Cloud Cost Management has also helped Carvana cut a significant portion of their cloud spend, with still more to go. 

“We're currently on a downward trend for our GCP spend and, for the last three months, we've been able to cut the bill down as much as we can,” said Adams. “We’ve saved about $120,000 per month, which is $1.4 million per year.”

Carvana expects even more savings as it expands its use of Intelligent Cloud AutoStopping to power down unused instances and even incorporate it into its automated provisioning system. The team’s goal is to automatically add AutoStopping when a new machine is built so it can optimize costs from day one.

“Cloud Cost Management was something we didn't know we needed at the time,” said Adams. “Now, especially with other departments using it, we can't live without it.”

Harness has helped Carvana remove the roadblocks associated with infrastructure provisioning. When the company needs to spin up new instances, it can now do so quickly and with little manual effort. And, it has granular cloud cost insights so it can provide and maintain instances without breaking the bank. 

“We are all-in on Harness because the tool works very well for every cloud environment and we're able to identify everything, which is amazing,“ concluded Adams. “Having control of our billing is key with this product.”