About Zuul
Zuul is an open source program that drives continuous integration, delivery, and deployment systems with a focus on project gating and interrelated projects. The OpenStack foundation recognized Zuul as an independent, openly governed project in early 2019.
As opposed to previous generations of CI systems, Zuul does gating, which is that the patch is tested both pre-merge as well in the state it will be post-merge. This takes into consideration any changes that have been made to the project since the last time it was merged, especially critical in projects with multiple contributors. It also handles cross-project dependencies, comments on job changes, and highlights syntax problems.
Zuul was initially developed for the OpenStack project to be able to manage the cross-project dependencies between all the micro-services and projects that OpenStack is composed of.
Nowadays, Zuul is used by enterprise development teams with hundreds of simultaneous projects. Some companies cited (by the Zuul team) as currently using it are BMW, GoDaddy, Volvo, Ansible, OpenLab, and Wikimedia.
The Need
The Zuul open source project is an extremely complex global system and needs to have resources available that can be continuously spun up and destroyed to handle the testing protocols and requirements of the totality of Open Infra projects in active development, like Starlingx, OpenDev, X, and Airship alongside OpenStack.
The Zuul project uses a network of multiple cloud providers to spin up these resources. One of the problems they constantly run into with clouds in their ecosystem is lag and inconsistencies that lead to longer wait times to get the test jobs done. So there was also a specific need to have the server instance available in the shortest period of time.
The Solution
Zuul’s multi-cloud infrastructure is a resilient system where the different clouds act as available resources for Zuul to appropriately assign workloads.
OpenMetal, a cloud infrastructure company incubated out of InMotion Hosting, had released a private On-Demand OpenStack cloud and wanted to show support to the open source technologies being developed by the OpenInfra Foundation. OpenMetal is a Silver Member of the Open Infrastructure Foundation. But, this was an opportunity to work closely with the OpenInfra team and contribute back, while also testing the new cloud product.
Yuriy Shyyan, a lead engineer on the OpenMetal team met with key members of the Zuul Project and helped provision and set up an on-demand, Hosted Private Cloud on OpenStack, as a cloud resource for their CI/CD needs. This process was completed just over the course of a few hours. And was all done virtually with no physical changes needed to the hardware.
On the OpenMetal side, the cloud was first provisioned and upgraded to the latest stable version of OpenStack (at that time Victoria), SSL and TLS were enabled, and the keys were handed over to the Zuul team. As part of the deployment for the anticipated workload intensity, a few optimizations were made, including enabling instance provisioning randomization, adjusted overcommit ratios, and system reserved resources.
The only thing that the Zuul team had to do was to create the Zuul OpenStack Projects and the project users in Horizon, which is OpenStack’s UI Dashboard to create and manage cloud resources. Once they added the OpenMetal Cloud to their NodePool configuration as a resource with the previously created authentication, the existing system was able to instantly connect and upload various project artifacts, cloud images, and CI requirements. This allowed Zuul to nearly instantly be able to delegate CI workloads onto the OpenMetal Cloud.
Within minutes, images were populated on the cloud and the cloud was also added automatically to the Zuul Project’s Grafana.
Optimized and Customized Hardware Configuration
To validate the OpenMetal Cloud Core as a suitable resource pool, a hyper-converged cluster of 6 servers including the Cloud Core – Standard and three additional Large Computes were selected for the cloud composition to satisfy the Zuul explosive workload requirements. One of the other known requirements was that each of Zuul’s Continuous Integration testing instances requires at least 8vCPU/8GB RAM to process complex CI workflows.
Of the six servers included in their cloud:
- Each server of the three servers in the Cloud Core-Standard comes with: Intel(R) Xeon(R) D-2141I CPU @ 2.20GHz, 16 Cores/Threads, 128 GB RAM and 3.2 TB NVMe
- Each of the three Large Computes comes with: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz, 40 Cores/Threads, 512 GB RAM and 3.2 TB NVMe
With optimizations, this configuration gave them access to about 50 VMs. In the first week of implementation, the Zuul project used these 50 VMs to create and destroy on average about 164 VMs per day, which approximately maps out to about 5000 VMs in the first month.
Faster Deployment than public cloud
This Cloud Core configuration comes with a 17TB NVMe, which helped speed up deployment times, to where VMs were being spun up in under 30 seconds on average. In some cases, this is 10X faster than what is being reported by Zuul from some public cloud vendors who are also providing infrastructure to the Zuul Project.
In the Cloud Core deployed by OpenMetal, cloud storage is highly available as the system keeps 3 copies of the data on 3 separate servers. OpenMetal Clouds also support LVM provisioning directly to NVMe, which is ideal for high IO throughput and workloads that don’t require data replication. Directly interacting with LVM reduces the overhead of various software processing the IO flow.
Ready out of the box
OpenMetal Cloud came with the default networking, security, and governance needed by the Zuul team. Since the Zuul team was using automation tools, technologies, and processes that are OpenStack native, the integration with their multi-cloud infrastructure was seamless. The only changes made prior to integration were to upgrade to the latest stable version of OpenStack and optimize the cloud for the desired workload.
Infrastructure as Code
Nodepool is an infrastructure as code(IaC) inventory system for cloud-provided resources that supports the launch of single-use tenant nodes through various cloud providers and delegates requests to the pre-defined clouds supporting the launch.
OpenMetal Clouds are validated against rigorous integration testing and OpenStack’s native API capabilities mean that with just the addition of a few lines of code, OpenMetal Cloud resources are made immediately and reliably available to the Zuul project.
All the Zuul team had to do was make minimal code changes to get the OpenMetal Cloud added as a resource, and then the automation powered by Zuul tested and validated the new YAML infrastructure as code addition.
Instant On-Demand Resources
In the test PoC implementation, the Standard Cloud Core was used to validate cloud configuration, optimization, and utility. In the production deployment, three Large Computes were added in less than an hour to the original Standard Cloud Core in order to scale up resource availability and increase the number of instances available to Zuul. The on-demand nature of OpenMetal Cloud means these resources were immediately available for the Zuul Project to utilize after the addition.
Dedicated Technical Lead
One of the top engineers on the OpenMetal Cloud team was assigned to assist Zuul with getting the project implemented successfully. Our technical engineers are OpenStack enthusiasts and users themselves and are very invested in the success of Open Infra Projects.
“I enjoyed working with Clark Boylan and Jeremy Stanley. Zuul is incredibly and impressively complex. It is nice to connect with infrastructure experts and share our experiences with automation and large scale systems.”
Yuriy Shyyan, Lead Engineer on OpenMetal Cloud
Continuous Monitoring and Optimization
Throughout the various stages of this deployment, the OpenMetal Cloud technical team continuously reviewed the performance of the cloud set up for the Zuul project. In order to scale up and optimize the delivery of desired performance and instance target, the team determined that some changes to the IP allocation, overcommit ratios, placement randomization would allow for the project to fully utilize the resources available in this Cloud Core configuration. These additional IPs increased the number of VMs available to be provisioned directly on an external network for use by the project.
Giving back to open source
OpenMetal is a Silver Founding Member of the Open Infrastructure Foundation and its origin story and growth has a close connection with open source technologies and platforms. OpenMetal is proud to be part of a thriving group of companies that are actively building on OpenStack and was thrilled to get the opportunity to donate infrastructure to the Zuul Project as a way of giving back to the open source community.
Solution Synopsis
This specific implementation for the Zuul project highlights the usage of OpenMetal Cloud as part of a multi-cloud and hybrid cloud solution that is always available as a reliable cloud resource, with VMs that can be continuously used and destroyed as needed.
This type of architecture would be valuable for companies with solutions running temporary workloads while also needing the ability to instantly scale resources.
About OpenMetal Cloud
OpenMetal Cloud makes it easy for companies and enterprises to move to or integrate an on-demand private cloud platform as part of their multi-cloud strategy.
OpenMetal Cloud provides the additional benefits of in-built privacy, security, and governance for sensitive data and workload processing. The cloud administrator has full ownership over the resource capacity of the chosen architecture and the hardware component specifications are completely transparent.
OpenMetal Cloud Core Pricing is also shown up-front with no unexpected licensing costs, or any other hidden costs, allowing for reliable IT infrastructure budget forecasting, critical in multi-cloud scenarios.
Our no-risk, no-strings-attached “Trial” model gives eligible candidates instant access to trial clouds. Cloud deployment times are under 45 seconds. Organizational cloud trials allow you to easily add Teams to the platform to help test and manage the cloud. Request a cloud trial. You can also apply to receive around $1300 worth of cloud infrastructure to build out a custom proof of concept (PoC) cloud.
Talk to our Cloud Team to get a demo of this innovative on-demand private cloud platform and get started with a PoC Cloud today!
Case Study Author: Sash Ghosh | Technical Lead on Project: Yuriy Shyyan
Zuul: https://zuul-ci.org/
OpenMetal Cloud: https://openmetal.io/