On-Demand Resources in a Multi-Cloud Environment – CI/CD Solution for OpenStack’s Zuul Project

About Zuul

Zuul is an open-source program that drives continuous integration, delivery, and deployment systems with a focus on project gating and interrelated projects. The OpenStack foundation recognized Zuul as an independent, openly governed project in early 2019. 

As opposed to previous generations of CI systems, Zuul does gating, which is that the patch is tested both pre-merge as well in the state it will be post-merge. This takes into consideration any changes that have been made to the project since the last time it was merged, especially critical in projects with multiple contributors. It also handles cross-project dependencies, comments on job changes, and highlights syntax problems. 

Zuul was initially developed for the OpenStack project to be able to manage the cross-project dependencies between all the micro-services and projects that OpenStack is composed of.  

Nowadays, Zuul is used by enterprise development teams with hundreds of simultaneous projects. Some companies cited (by the Zuul team) as currently using it are BMW, GoDaddy, Volvo, Ansible, OpenLab, and Wikimedia.

The Need

The Zuul open-source project is an extremely complex global system and needs to have resources available that can be continuously spun up and destroyed to handle the testing protocols and requirements of the totality of Open Infra projects in active development, like Starlingx, OpenDev, X, and Airship alongside OpenStack.

The Zuul project uses a network of multiple cloud providers to spin up these resources. One of the problems they constantly run into with clouds in their ecosystem is lag and inconsistencies that lead to longer wait times to get the test jobs done. So there was also a specific need to have the server instance available in the shortest period of time.

The Solution

Zuul’s multi-cloud infrastructure is a resilient system where the different clouds act as available resources for Zuul to appropriately assign workloads. 

InMotion Hosting, a Silver founding member of the Open Infrastructure Foundation, recently released an on-demand private cloud solution on OpenStack and wanted to show support to the open-source technologies being developed by the Open Infra Foundation. An opportunity presented itself to work closely with the OpenInfra team to contribute back while also testing the new cloud product.

Yuriy Shyyan, a lead engineer on InMotion Hosting’s team met with key members of the Zuul Project and helped provision and set up Flex Metal Cloud, InMotion Hosting’s on-demand private cloud solution, as a cloud resource for their CI/CD needs. This process was completed just over the course of a few hours. And was all done virtually with no physical changes needed to the hardware. 

On the Flex Metal Cloud side, the cloud was first provisioned and upgraded to the latest stable version of OpenStack (at that time Victoria), SSL and TLS were enabled,  and the keys were handed over to the Zuul team. As part of the deployment for the anticipated workload intensity, a few optimizations were made, including enabling instance provisioning randomization, adjusted overcommit ratios, and system reserved resources.

The only thing that the Zuul team had to do was to create the Zuul OpenStack Projects and the project users in Horizon, which is OpenStack’s UI Dashboard to create and manage cloud resources. Once they added the Flex Metal Cloud to their NodePool configuration as a resource with the previously created authentication, the existing system was able to instantly connect and upload various project artifacts, cloud images, and CI requirements. This allowed Zuul to nearly instantly be able to delegate CI workloads onto the Flex Metal Cloud.

Within minutes, images were populated on the cloud and the cloud was also added automatically to the Zuul Project’s Grafana.

Optimized and Customized Hardware Configuration

To validate the Flex Metal Cloud Core as a suitable resource pool, a hyper-converged cluster of 6 servers including the Cloud Core – Standard and three additional Large Computes were selected for the cloud composition to satisfy the Zuul explosive workload requirements. One of the other known requirements was that each of Zuul’s Continuous Integration testing instances requires at least 8vCPU/8GB RAM to process complex CI workflows. 

Of the six servers included in their cloud:

  • Each server of the three servers in the Cloud Core-Standard comes with: Intel(R) Xeon(R) D-2141I CPU @ 2.20GHz, 16 Cores/Threads, 128 GB RAM and 3.2 TB NVMe
  • Each of the three Large Computes comes with: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 40 Cores/Threads, 512 GB RAM and 3.2 TB NVMe

With optimizations, this configuration gave them access to about 50 VMs. In the first week of implementation, the Zuul project used these 50 VMs to create and destroy on average about 164 VMs per day, which approximately maps out to about 5000 VMs in the first month.  

Faster Deployment than public cloud

This Cloud Core configuration comes with a 17TB NVMe, which helped speed up deployment times, to where VMs were being spun up in under 30 seconds on average. In some cases, this is 10X faster than what is being reported by Zuul from some public cloud vendors who are also providing infrastructure to the Zuul Project. 

In the Cloud Core deployed by InMotion Hosting, cloud storage is highly available as the system keeps 3 copies of the data on 3 separate servers. Flex Metal Clouds also support LVM provisioning directly to NVMe, which is ideal for high IO throughput and workloads that don’t require data replication. Directly interacting with LVM reduces the overhead of various software processing the IO flow.

Ready out of the box

Flex Metal Cloud came with the default networking, security, and governance needed by the Zuul team. Since the Zuul team was using automation tools, technologies, and processes that are OpenStack native, the integration with their multi-cloud infrastructure was seamless. The only changes made prior to integration were to upgrade to the latest stable version of OpenStack and optimize the cloud for the desired workload.

Infrastructure as Code

Nodepool is an infrastructure as code(IaC) inventory system for cloud-provided resources that supports the launch of single-use tenant nodes through various cloud providers and delegates requests to the pre-defined clouds supporting the launch. 

Flex Metal Clouds are validated against rigorous integration testing and OpenStack’s native API capabilities mean that with just the addition of a few lines of code, Flex Metal Cloud resources are made immediately and reliably available to the Zuul project.

All the Zuul team had to do was make minimal code changes to get InMotion Hosting’s Cloud added as a resource, and then the automation powered by Zuul tested and validated the new YAML infrastructure as code addition.

Instant On-Demand Resources

In the test PoC implementation, the Standard Cloud Core was used to validate cloud configuration, optimization, and utility. In the production deployment, three Large Computes were added in less than an hour to the original Standard Cloud Core in order to scale up resource availability and increase the number of instances available to Zuul. The on-demand nature of Flex Metal Cloud means these resources were immediately available for the Zuul Project to utilize after the addition. 

Dedicated Technical Lead

One of the top engineers on the Flex Metal Cloud team was assigned to assist Zuul with getting the project implemented successfully. Our technical engineers are OpenStack enthusiasts and users themselves and are very invested in the success of Open Infra Projects.

“I enjoyed working with Clark Boylan and Jeremy Stanley. Zuul is incredibly and impressively complex. It is nice to connect with infrastructure experts and share our experiences with automation and large scale systems.”

Yuriy Shyyan, Lead Engineer on Flex Metal Cloud

Continuous Monitoring and Optimization

Throughout the various stages of this deployment, the Flex Metal Cloud technical team continuously reviewed the performance of the cloud set up for the Zuul project. In order to scale up and optimize the delivery of desired performance and instance target, the team determined that some changes to the IP allocation, overcommit ratios, placement randomization would allow for the project to fully utilize the resources available in this Cloud Core configuration. These additional IPs increased the number of VMs available to be provisioned directly on an external network for use by the project.

Giving back to open source

InMotion Hosting is a Silver Founding Member of the Open Infrastructure Foundation and its origin story and growth has a close connection with open-source technologies and platforms. InMotion Hosting is proud to be part of a thriving group of companies that are actively building on OpenStack and was thrilled to get the opportunity to donate infrastructure to the Zuul Project as a way of giving back to the open-source community.

Solution Synopsis

This specific implementation for the Zuul project highlights the usage of Flex Metal Cloud as part of a multi-cloud and hybrid cloud solution that is always available as a reliable cloud resource, with VMs that can be continuously used and destroyed as needed.

This type of architecture would be valuable for companies with solutions running temporary workloads while also needing the ability to instantly scale resources.

About Flex Metal Cloud

Flex Metal Cloud makes it easy for companies and enterprises to move to or integrate an on-demand private cloud platform as part of their multi-cloud strategy. 

Flex Metal Cloud provides the additional benefits of in-built privacy, security, and governance for sensitive data and workload processing. The cloud administrator has full ownership over the resource capacity of the chosen architecture and the hardware component specifications are completely transparent. 

Pricing is also shown up-front with no unexpected licensing costs, or any other hidden costs, allowing for reliable IT infrastructure budget forecasting, critical in multi-cloud scenarios.

Our no-risk, no-strings-attached “Test for Free” model gives eligible candidates instant access to almost $1300 worth of cloud infrastructure to build out a proof of concept cloud. Cloud deployment times are under 45 minutes. Teams can be easily added to the platform for free to help test and manage the cloud.

Talk to our Cloud Solutions Team to get a demo of this innovative on-demand private cloud platform and get started with your free PoC Cloud today!

Case Study Author: Sash Ghosh | Technical Lead on Project: Yuriy Shyyan

Zuul: https://zuul-ci.org/ 

Flex Metal Cloud:https://www.inmotionhosting.com/flex

Was this article helpful? Let us know!