History
The DevOps movement inspired large, enterprise organizations with agile practices to allow developers to make quick changes, but it was very difficult to get full benefit, as their legacy development process was not designed for supporting short software development delivery cycles and frequent production releases. In order to release software into production quickly but reliably and – most importantly – repeatably, the DevOps movement developed the Continuous Integration/Continuous Delivery (CI/CD) methodology. Jez Humble and David Farley covered these principles and practices in their book “Continuous Delivery: Reliable Software Releases through Build.” Cloud-native data center technology, adopted by enterprises in 2000, offers the flexibility that paves the path forward to realize new services for mobile operators, but data center technologies introduce a level of operational complexity in the RAN. Implementation of this data center inspired approach is the next step in evolving legacy RAN to fully automated Open RAN.
Why
VNFs are software versions of network services that used to reside on dedicated network appliances. Functions like vEPCs have been deployed as Virtual Machines (VMs) on COTS infrastructure for over a decade now. VMs are an abstraction of physical hardware, turning one server into many servers. The hypervisor allows multiple VMs (i.e. EPC, IMS) to run on a single machine. Each VM includes a full copy of an operating system, making them slow to boot; it also includes one or more apps, necessary binaries and libraries — taking up tens of GBs of storage.
The fact that something like vEPC is implemented as a monolithic software application on top of a VM may explain some of the performance issues. The RAN is the most transaction-intensive and time-sensitive area of the network and any performance issues will create havoc on the user experience. This is why Open RAN, which consists of many different components from different vendors, needs to adapt to a data center approach to develop, run and optimize software.
But Open RAN brings new problems, such as time and complexity of integrations and software upgrades between different vendors. The actual development of a suite of interoperability tests amongst Open RAN vendors is the main challenge. A special set of tests are needed to standardize the certification of software, whether it is hosted on cloud and/or bare-metal infrastructure. A second consideration is the diversity of RAN splits that can be implemented using different Open RAN vendors as a testing process, and cases for Split 7 will be different than for Split 8.
Current CI/CD frameworks have typically been used for IT and enterprise apps. As RAN has hardware components like radios that will remain physical, the suite of tests for CI/CD needs to take that into consideration, as the hardware and software in Open RAN will be coming from different vendors. The integration, software upgrades, and the lifecycle management of many vendors will be challenging. Vendors will need to adapt to a new testing model – wherein they are not testing their own products in silos, but rather under an overall CI/CD umbrella. This is especially important in test environments before going into field deployments. This approach will help with creating CI/CD blueprints for future deployments.
The main takeaway: as the Open RAN ecosystem evolves, the number of software upgrades will increase, and so will the importance of including these upgrades in the CI/CD environment that is specifically developed for Open RAN across many hardware and software vendors.
What
To optimize performance, in the enterprise example, software implementation went from monolithic, self-contained applications running on dedicated servers to a new model that was built on Webscale models. It eventually evolved to microservices. A microservice is decomposition of an application into a multitude of separate parts, each one of them running in a lightweight “container”-like environment – for example, Docker or rkt or Linux LXD. VMs (with a whole OS to boot) are simply too heavyweight to host microservices. By deconstructing a RAN service into microservices, it’s easy to address any performance issue by spinning up multiple instances of the RAN microservice that might be creating a performance issue.
Open RAN software doesn’t have to be just *one* big monolithic service like a vEPC, as different RAN function components can be implemented as separate microservices and can be scaled up in any way required to optimize the RAN function’s performance.
A microservices architecture also has several other very important benefits. One of them is the ability to continuously innovate by embracing an agile DevOps model. A mobile operator can push out any RAN upgrades without taking down the entire site or sites, as testing a microservice involves a very few test cases; testing an entire monolithic (though virtualized) application takes many days of testing.
Source: Parallel Wireless
How
Implementing these four data center principles will help make Open RAN be responsive and high performing. Principle 1 ensures that each RAN function component is deployed as a microservice. Principle 2 advocates for a very responsive container environment that should be used and not legacy based infrastructure. Principle 3 promotes a DevOps method to be adopted for the software to be delivered frequently. And lastly, principle 4 highlights that using CI/CD is key for automation: 1000 sites can be upgraded whenever desired.
Each microservice can be deployed, upgraded, scaled, and restarted independently of other microservices in the RAN application, using an automated system, enabling frequent updates to live applications without impacting end-user experience. Under the hood, a microservice is hosted in a container and exposing APIs to access the service it offers. A particular microservice runs in a container. Then Kubernetes orchestrates the dynamic instantiation of several of those containers in pods, according to performance, resilience and security requirements.
A container image is a lightweight, standalone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, environment. These capabilities are key for the ever-changing RAN environment. Multiple containers can run on the same machine and share the OS kernel with other containers. Containers take up less space than VMs (container images are typically tens of MBs in size), and containers can start almost instantly. Containers offer both efficiency and speed compared with standard virtual machines. Using operating system level virtualization, a single OS instance is dynamically divided among one or more isolated containers, each with a unique writable file system and resource quota.
DevOps is a set of practices developed for IT that combines software development (Dev) and information-technology operations (Ops) which aims to shorten the systems development lifecycle and provide continuous delivery with high software quality.
CI/CD is a set of practices that enables how software, in our case RAN software, gets delivered. Continuous Integration is the practice of merging all developers working code to a shared mainline several times a day. Every merge is validated before merging to uncover any issues.
CD stands for Continuous Delivery/Deployment, which is the practice in which software teams produce reliable software in short cycles that can be released (delivered) at any time, which can then be deployed. As a result, any software can be released much faster, more frequently and reliably into production networks. The benefit of CD is more predictable deployments that can happen on demand without waiting for an “official” upgrade cycle – this is now thing of the past.
Any RAN software in a CD environment is always ready to be released, since every single change is delivered to a joint staging environment using complete automation and the feedback loop to ensure that any RAN application can be deployed into a live network with the push of a button when it is needed based on the business goals for a particular set of RAN sites based on geography or other factors.
Source: Pivotal.io
By including a CI/CD approach as a part of RAN deployments, RAN software changes can be delivered in a faster and automated way. As a result, mobile operators now can bring a webscale application approach into wireless network architectures. CI/CD helps to solve the challenges of multi-node complex RAN environments with added Kubernetes virtualization approach, where a mini-data center can be brought to the edge and be properly managed and upgraded when needed.
Operators can use x86 powered virtualized infrastructure.
The main takeaway: The potentially complex integration of DevOps toolsets to enable a microservices-based flexible and agile environment should not scare mobile operators that want to decompose monolithic VNFs into a microservices-architecture. The result can deliver constant innovation, optimal RAN performance for the subscribers, and increased agility for an MNO.
When
In this blog written two years ago, Rakuten took a page from Humble’s and Farley’s book by explaining how mobile operators can implement CI/CD principles and processes and what pitfalls to avoid when implementing CI/CD.
Source: Rakuten
Rakuten believes that in order to achieve successful CI/CD, organizational culture needs to be changed saying, “to be realistic thought, it would be obvious true that it is pretty much tough or highly costly that let more than thousands developers across more than hundred development teams align to the same software development cycle based on single idea without authorized suppressing, especially for large organization like Rakuten.” They advise “to do small start with the deployment in the feedback loop related stages in CD pipeline, which somehow works to many developers to get them on a same compliance. As for CI, for the time being, it wouldn’t be big deal let developers do on their own even it is unique way until the thought of CD sink into developers, since most of the issues in CI should be related to developer’s matter.”
Rakuten targets to empower their operational teams with open CI/CD applications and interfaces to help them react faster to their customer demands and continually enhance network resiliency while introducing new services.
IpT
Telefonica, as a part of IpT initiative, has advocated on the importance of opening RAN interfaces to open up the RAN vendor ecosystem. The implemented architecture in IpT has already shown an unprecedented potential in terms of minimizing the cost of operations while integrating different Open RAN vendors with multiple core networks and the reduced time to release new RAN features in the field. IpT embraces DevOps and automation best practices while adopting a microservice-based architecture. IpT is not only the largest Open RAN deployment today, but also the world’s largest implementation of a CI/CD solution for Open RAN in a telco cloud deployment. Telefonica and their supply chain vendors must work together to make the shift from the current siloed deployment to a DevOps cloud-centric approach.
TIP
Telefonica hosts one of TIP’s interoperability testing labs focused on implementing a CI/CD framework for OpenRAN at its R&D labs in Madrid, Spain. One of the fundamental architecture principles is aligning development and operational responsibilities across all hardware and software vendors to drive collaboration and innovation. Therefore, this platform is open by default, containerized, and scalable, and fully automated.
The main takeaway: CI/CD is showing great momentum in Open RAN environments by delivering faster, more efficient automated deployments. By embracing CI/CD, mobile operators embrace greater collaboration between different ecosystem members, which fosters innovation. This approach will help to minimize the risk through frequent delivery of new features and new optimizations, while increasing efficiency via automation that leads to faster new services introduction.
Summary
The integration of many new tools might seem challenging when mobile operators first start deploying microservices-based DevOps CI/CD driven network implementations. As MNOs get ready for Phase 2 in their Open RAN journeys, that will include containers and microservices. Agile DevOps will simplify automation by providing validated stack templates for containers to host microservices. These will be automated to be secure, reliable, maintained, and supported to allow MNOs to easily define their own architecture and make Open RAN easier and more cost-effective to deploy and maintain.
