I'm excited to announce that we have joined the Industry Consortium for the Platforms for Advanced Wireless Research (PAWR) program to both contribute our ONAP expertise and to learn about cutting edge 5G research. The PAWR program is a critical organization for the US government's research initiatives in the field of advanced wireless technologies including 5G networks and beyond.

The PAWR Project Office (PPO) manages the $100 million public-private partnership and oversees the research platforms. The PPO is co-led by US Ignite and Northeastern University. It collaborates closely with the National Science Foundation, the wireless research community, local communities, and industry, in part through the Industry Consortium, in the design, development, deployment, and initial operations of the research platforms. PAWR has one testbed in Salt Lake City that's generally available, and two under construction — COSMOS in New York City and AERPAW in Research Triangle, North Carolina.

The Linux Foundation open source Open Network Automation Platform (ONAP) project is a popular automation software used by PAWR projects. As the leading ONAP distribution vendor, this is the main rationale for us joining PAWR. In addition to working on orchestrating, managing, and automating 5G Core and RAN projects, we are also looking forward to some advanced work around network slicing, O-RAN integration, AI/ML analytics, and management of edge computing applications — all using ONAP. We will contribute and learn at the same time!

If you are involved with PAWR and wish to collaborate with us, feel free to contact us. In the meantime, please join me on May 19th for a Linux Foundation webinar titled, "Integrating ONAP with a 5G Cloud Native Network". We will share the latest progress and a demo on the community based 5G Cloud Native Network + ONAP demo that is being developed in the OPNFV community.

ONAP Components SSL certificate expiry root cause analysis and approaches to resolve it

Recently, ONAP Dublin has been failing due to SSL certificate expiration. This blog discusses the issue, looks at how the community is addressing it, and then offers some possible alternative solutions.

ONAP SSL Certificate expiry issue

‍SSL certificates, generated at the time of building the Docker image, are generally bundled together by ONAP components. These build time SSL certificates (with private & public keys) have default one (1) year validity from the date and time that they are generated. And it is this validity that breaks any and all  running ONAP deployments past the expiration date. Past this date, most components in Dublin throw the following exception.

‍javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

‍Few instances, demonstrating this today are:

• Policy portal console web app fails to authenticate with the Portal application
• DMaaP certificate expiry impacts A&AI, SO and other components

Why can’t we overwrite these SSL certificate files in ONAP?

‍
Overwriting these SSL certificates sounds like an easy fix. However, while we can overwrite these files in case of a simple docker container, we can not do the same  in case of a K8s cluster. This is primarily because of how ConfigMap/Secrets and Volume mount works in the K8s cluster infrastructure. Simply put, K8s does not support binary file mounting or overwriting of the file using ConfiMap.

Below  are the 2 file formats adapted by ONAP components which indicates  the key reason for bundling of these files within Docker image.

• Java Key Store (JKS) format is a binary Java Key Store (JKS) format (Till Java 8)
• PKCS#12 or PFX format is a binary format for storing the server certificates  (From Java 9)

‍

Frankfurt Approach: Deployment time keystore file generation

‍

Adapted by DCAE components, ONAP Frankfurt generates keystore files at the time of deployment with 1 year certificate validity. Consequently, one runs into issues especially when one wants to  run a component for more than a year. In which case, the only way out is to redeploy the affected components.

Does ONAP support importing enterprise specific SSL certificates?

‍
At present there is no direct solution, since all SSL certificates use ONAP domain name
as a key alias and this requires additional work for each customer.

Potential Solutions

‍
K8s persistent volumes
We can consider leveraging K8s persistent volumes over NFS share. For each ONAP application, we can pre-create the NFS share path and appropriately mount it with the local application path.

For example, ONAP policy PODs store the SSL keystore under a local folder /opt/app/policy/etc/ssl,. We should mount it to an NFS share /onap-ssl-cert-store/policy/ssl to avoid the certificate expiry issue.

The flip side of this approach is that it opens up security holes because if someone were to gain unauthorized access to this NFS share, it would compromise the entire ONAP system.

Base64 encoded ConfigMaps/Secrets
In this approach, we can base64 encode the SSL keystore files and mount it as a K8s volume. The base64 data can be decoded to generate the keystore file at the time of deployment:

• Identify and overwrite the Docker entry point scripts to decode the base64 ConfigMap/Secrets volume.
• Install base64 decoding CLI or scripts, in case the docker image does not have one already.

Build docker container images and host them under separate tagsIn this approach, we need to set up a CI/CD pipe-line to build the target K8s service Docker images to inject the SSL keystore with custom or customer provided keys.

Pros of this approach are:

• Will protect and secure ONAP components by locally hosting a Docker registry within the private network.
• Make code security scanning and auditing more transparent and compliant with enterprise wide security principals

The only caveat with this approach is that the Docker image maintainability becomes very challenging when we want to roll out upstream fixes.

Conclusion

‍
Addressing ONAP SSL certificates is one part of the puzzle and having an enterprise ready ONAP deployment to get early adopters to put ONAP into production. We at Aarna.ml are working actively on this topic to greatly simplify ONAP enterprise adoption in addition to topics such as backup, restore and upgrade capability.

We are pleased to announce that Aarna.ml, a 5G/EdgeOps company, has raised financing via an angel round. The lead investor is Subba Reddy, a noted angel investor and technologist who also led seed rounds for i2Chain (cybersecurity) and Xaptum (IoT).

‍

This funding will let us 1) build our 5G/edge product, based on the Linux Foundation open source ONAP4K8s, that will orchestrate, manage, monitor, and automate cloud native 5G and edge computing workloads across multiple Kubernetes edge clouds, 2) complete a public 5G PoC, and 3) complete a private LTE/5G PoC. This round will also instill confidence within our customers and partners.

While the coronavirus is a global tragedy, it is going to provide 5G with a big boost as telemedicine and emergency response become hot topics. 5G with its high bandwidth, ultra-reliable low-latency, and IoT support promises to revolutionize healthcare through AR/VR, edge IoT analytics, faster communication, and even futuristic trends like remote surgery. Similarly, emergency response is expected to be revolutionized by 5G through accurate IoT data, analytics, drone control, AR/VR etc. And our Aarna.ml ONAP Distribution will help accelerate the move to 5G.

The three work streams listed above will help us get to the next milestone. I will summarize the goals and expected outcomes of these activities in subsequent blogs.

There was a Linux Foundation Edge Akraino face-to-face TSC and 2020 planning meeting last week held at Facebook in Menlo Park, CA. It was a fascinating meeting where I got to see the state-of-the-art of edge computing; here are my key takeaways:

1) The dominance of Kubernetes (K8s): I think it might be safe to say that K8s will be the default NFVI on the edge. Most blueprints (the term used by Akraino to signify projects) were based on K8s. The few that were based on OpenStack used a containerized version of OpenStack deployed on K8s. So essentially it is safe to say that K8s will dominate edge environments.

2) Focus on a small footprint: Edge environments could be quite constrained. There were several blueprints focusing on very small environments such as on-prem. Specifically, see slide presentations on the ELIOT (Edge Lightweight and IOT), uMEC (Micro MEC), KubeEdge, and others.

3) Centralized infra controller: With 100s to 100s of thousand edge locations, it is impractical to install K8s on each edge manually. Almost every blueprint had some notion of a centralized infrastructure controller that would automagically install K8s on a given edge cloud. See IEC and ICN blueprints as example.

4) Diversity of applications: Another interesting aspect was the diversity of applications. Apps ranged from LTE, 5G, SD-WAN, AR/VR, gaming, V2X, IoT, and more. The edge is indeed going to a lot more than just network functions.

5) Framework wars: From general frameworks, e.g. TARS to specialized frameworks, e.g. Kubeflow for AI/ML apps, I expect numerous edge application frameworks to jockey for position. I expect full blown edge application framework wars to emerge in earnest in late 2020. These wars will be critical to establish control points, similar to IOS and Android for phones.

6) Interest in dataplane acceleration: RAN and UPF network functions are expected to put heavy demands on datapath computation. For this reason, hardware acceleration is expected to play an important role on the edge. Dataplane acceleration will come in numerous flavors ranging from software techniques, SmartNICs, FPGAs, GPUs, inference engines, and more.

Overall, I truly enjoyed the event and believe Akraino will play a key role in moving the industry forward. Aarna is involved with the Akraino ICN Blueprint helping with ONAP4K8s from an orchestration, lifecycle management, and automation point of view. Contact us if you want to learn more.

‍

I was recently interviewed by AvidThink on the top 5 NFV Trends in 2020.  You can watch my interview or read a summary here:

1. Disaggregation in earnest: We have heard about the trend where customers will buy  hardware from one vendor, virtualization software from another, VNFs from yet another set of vendors, so on and so forth. But this has largely not materialized. With 5G/edge, I am definitely seeing serious interest in disaggregation. From our stand point, ONAP is well suited to address this trend from an orchestration, management, and automation point of view as it is vendor agnostic.

2. Cloud native: The conventional wisdom is that the move from VNFs to CNFs (cloud native network functions) will take a long time—perhaps 3-5 years. I am seeing something quite surprising, some customers are betting the farm on CNFs and cloud native applications for 5G/edge in 2020! ONAP definitely has the lead on K8s and CNF support, so I find this trend very promising.

3. Open source: Several operators are starting to adopt an open source "first" methodology. In other words,  technologists at these companies need to justify to the management team why they want to use proprietary software instead of open source. Open source is not for the faint of heart though, and I am hoping more and more customers will understand how to engage effectively with open source projects such as ONAP.

4. Widespread adoption of automation: Automation is now new. A lot of aspects of SDN/NFV have been already been automated. However, we are yet to see widespread adoption of automation frameworks. With 5G/edge around the corner, it will not be viable for humans to manage fault and performance management issues. For this reason, I feel automation using frameworks such as ONAP DCAE +AI/ML will be adopted starting in 2020.

5. Convergence of management and orchestration layers. There are so many layers of management and orchestration, e.g. LSO (MEF Lifecycle Service Orchestration), NFVO for VNFs, NFVO for CNFs, MEAO (Multi-Access Edge Computing Application Orchestration), and SDN Orchestration. There just isn't enough room for so many different platforms. My prediction is that all of these platforms will get collapsed into one framework such as ONAP. Of course there will still be cloud orchestration using OpenStack or Kubernetes, but the rest should get consolidated.

Curious to know how ONAP, more specifically the ONAP4K8s profile, can help with CNFs, 5G, and edge? Come see us as ONES in Los Angeles in April where we will have several demos/presentations on these topics. Set up a meeting with us (scroll down to the contact us form).

If you are a VNF vendor (or for that matter PNF/CNF), establishing interop with ONAP can be stressful. Sure, there is excellent documentation to help you (VNF Requirements project) and easy-to-use tooling to establish baseline compliance. But given our busy lives, it can be a bit difficult to figure out where to start. You might have questions such as:

• What is ONAP?
• Why should you bother about interop with ONAP?
• What is the OPNFV Verification Program (OVP)? Why should I care?
• How does OVP help with interop testing with ONAP?
• How do I get started?

A recent webinar titled "How the OPNFV Verification Program (OVP) Can Boost VNF Interoperability" featuring yours truly as one of the panelists answered these questions. But that was not a hands-on experience. To help address the above questions in a 100% hands-on manner and to help you get started on your OVP VNF interop journey, the ONAP community is organizing an OVP VNF hacking track at the upcoming free LFN Developer and Testing event in Prague this January.

There will be ONAP and OPNFV environments available, experts at your beck-and-call, and guidance to get you testing your VNF against ONAP. Although there is a formal test plan, you can do as little or as much as you want (this format is inspired by ETSI Plugtests). It's a judgement free zone and no vendor names will be associated with results in the post-event report. CNF/PNF vendors are welcome too, though there is no equivalent test plan for you.

‍

Our (Aarna.ml) engineers will be at the event to help out. So what are you waiting for? Ask your engineers to sign up for this free event and experience the beautiful city of Prague in the dead of winter (the average temperature is a balmy -2C, so should not be too bad).

#ONAP #NFV