Meltwater Runs 1200 Pipelines Per Day With Harness CI

About

Meltwater helps companies make better, more informed decisions based on insights from the outside. We believe that business strategy will be increasingly shaped by insights from online data. Organizations will look outside, beyond their internal reporting systems to a world of data that is constantly growing and changing. Their customers use these insights to make timely decisions based on real-time analysis.

If it ain’t broke… Well, actually it’s broke.

Meltwater was losing developers’ confidence. The Jenkins instances used to patch together Continuous Integration pipelines lacked autoscaling, meaning developers waited in a line to complete their builds. Meltwater used multiple Jenkins instances, so no one wanted to spend the months of maintenance it would take to fix the issue.

“It is such a hassle to update Jenkins that you end up avoiding it for months and years.” - Jim Sheldon | Principle Software Engineer at Meltwater

Only 15% of Meltwater’s developers were using the corporate-issued Jenkins pipelines. Teams began building their own pipelines using TravisCI and CirlceCI. Even with developers using custom pipelines, only 60% of Meltwater’s repos were integrated into a CI pipeline.

“Teams were running away from Jenkins as fast as possible.” - Jim Sheldon | Principle Software Engineer at Meltwater

Developers were happy using one-off CI tools until Meltwater decided to migrate to Kubernetes. Travis CI and CircleCI became security risks because they required Meltwater to externally expose their APIs. Meltwater needed to find an on-premise CI solution.

Simplicity = Adoption

Meltwater chose Harness CI to be the base of their Kubernetes migration. Since moving to Harness CI, adoption rates have increased 3x to 222 developers, which is 50% of the organization. Those developers run 1200 pipelines per day. Harness CI has also increased Meltwater’s repo automation from 60% to 75%.

Harness CI’s simplicity offered a low learning curve and has since become the defacto language to share best CI practices. Harness CI lets developers create both simple and complex pipelines.

“Harness CI has handled everything we’ve thrown at it, it can be as complex or simple as we need it to be.” - Jim Sheldon | Principle Software Engineer at Meltwater

Harness CI’s autoscaling feature ensures there are always enough instances to meet demand, while significantly reducing cloud infrastructure cost by spinning down agents when demand decreases. Harness CI plugins become another pipeline step and can be set up instantaneously, without requiring coordination between developers and build administrators.

“Harness CI is extremely open and flexible. There’s nothing holding you back from building and deploying the way you want. Jenkins was extremely rigid and wouldn’t have allowed us to grow the way we have.” - Jim Sheldon | Principle Software Engineer at Meltwater

And if something ever does come up, Meltwater has one of the best open-source communities to reference. For example, when Docker imposed new rate limits, there was a Harness CI solution posted on a forum within a couple of days. For other CI tools, Docker rate limits are still an issue.