Recently we blogged about optimizing O-RAN performance and management using  O-RAN SMO and a new O-RAN SMO performance dashboard now available in AMCOP. 

Another key function of the O-RAN Service Management and Orchestration (SMO) framework is Trace Management. The O-RAN SMO provides a set of trace management functions that enable network operators to collect, store, and analyze trace data from different network elements.

The O-RAN SMO's trace management capabilities include real-time trace collection and analysis, as well as long-term trace storage and retrieval. The trace data collected from different network elements can be used for a variety of purposes, including network troubleshooting, performance optimization, and security analysis.

The O-RAN SMO's trace management functions are designed to support the O-RAN Alliance's open and standardized interfaces and protocols. This ensures that trace data can be collected from different vendors' equipment and analyzed in a consistent and interoperable way.

The O-RAN SMO's trace management capabilities also support advanced analytics and machine learning techniques. This enables network operators to detect and analyze patterns in the trace data, identify performance issues, and predict potential problems before they occur.

Trace Data can be reported from the Network Function (NF) to the SMO via trace files or via a streaming interface. Currently, Aarna’s SMO offering supports file-based trace records that are available to the SMO and streaming support will be available soon. Trace management utilizes O-RAN file management standards to collect trace files from NF, and notifications can be transmitted through NETCONF or VES.

Let us take a closer look at the high-level workflow involved in using AMCOP SMO (Service Management and Orchestration) to enhance trace collection processes.

High-level Workflow:

1. User Initiates Trace Job Creation:

The workflow begins with a user leveraging AMCOP SMO to create one or more trace jobs. These trace jobs specify the criteria and parameters for collecting trace data from the RAN elements.

2. User Requests Trace Collection:

Once the trace jobs are configured, the user sends an RPC (Remote Procedure Call) to initiate the trace collection process. This RPC serves as the trigger for data collection.

3. RAN Element Starts Trace Collection:

Upon receiving the RPC, the RAN element responsible for trace collection commences its operations. It starts collecting trace data based on the predefined job parameters.

4. Notification to AMCOP SMO:

As the RAN element collects trace data, it communicates with the Management Service (Mns) Consumer, which is typically the AMCOP SMO component. This communication is achieved through a notification event known as "notifyFileReady."

5. File Generation and Notification:

The "notifyFileReady" event indicates that a Trace Measurement file has been successfully generated and is ready for upload. This notification acts as a signal for further actions in the workflow.

6. SMO Notifies VES Collector:

AMCOP SMO, upon receiving the notification, takes the next step by notifying the VES (Virtual Event Stream) Collector. The VES Collector is responsible for gathering events and data from various network elements.

7. DFC Polls VES Collector:

The Data File Collector (DFC), an integral part of the process, continuously polls the VES Collector. This polling aims to retrieve information about the generated trace files and their locations.

8. Trace File Upload to DFC:

Armed with the file information obtained from the VES Collector, the DFC proceeds to upload the trace files from the RAN element to its storage location within the DFC pod.

9. Storage Location in DFC Pod:

The trace files are stored in a specific directory structure within the DFC pod. This structure typically follows the pattern: /tmp/onap_datafile/<RAN Device name>/<file path from notification>. This organization simplifies data management and retrieval.

10. Optional SFTP Upload:

If a Secure File Transfer Protocol (SFTP) uploader is configured, the DFC can upload the trace files to an external SFTP server. This step enhances data redundancy and accessibility.

11. Data Retention Decision:

The DFC may decide whether to retain or delete the trace files after successful upload based on a predefined variable, typically referred to as "delete_datafile."

12. User-Controlled Trace Collection Termination:

To maintain flexibility, users have the ability to stop trace collection jobs at any time by sending a stop trace job RPC. This user-initiated action allows for the management of trace collection processes.

Overall, the O-RAN SMO's trace management functions are critical for ensuring the reliability, availability, and security of O-RAN networks. By providing real-time trace analysis, long-term trace storage, and advanced analytics capabilities, the O-RAN SMO enables operators to quickly troubleshoot network issues, optimize network performance, and enhance network security.

See the Aarna.ml AMCOP SMO User Guide for more information.
‍

I’ll be at the upcoming Fyuz event in Madrid, Oct 9-10, talking about O-RAN, Private 5G, the Nephio project, and more. Contact us to meet up at the show. 

It's not often when a proof of concept crosses domains between wired and wireless but that's exactly what Aarna .ml recent integration with CableLabs does.When devices are registered inside the same 5G Core, we can apply a routing intelligence that crosses domains and provides a value added service to the user.

The demonstration illustrated below shows how a user with home internet and mobile service from the same operator can allocate bandwidth dynamically and automatically. Once the mobile device connects to the home Internet Wi-Fi access point and is detected to be also registered with the 5G Core, the CSML – built on the Edge Multi-Cluster Orchestration (EMCO) project and used in Aarna's AMCOP – pushes a higher-bandwidth DOCSIS service flow via the PCMM controller and CMTS to the user’s cable modem with a boost in cable bandwidth for that mobile device.

The user benefits from receiving the extra bandwidth automatically and the operator benefits from zero-touch provisioning that avoids any manual configuration.

This is just one example of the power of software defined networks and the orchestration of applications and services.

We invite you to read the case study and contact us with any questions and ideas for you network orchestration needs.

‍

Performance Management is a key function of the O-RAN Service Management and Orchestration (SMO) framework. The O-RAN SMO provides a set of performance management functions that enable network operators to monitor, measure, and optimize the performance of their O-RAN networks.

The O-RAN SMO provides a rich set of performance metrics that cover various aspects of the network, including radio access, transport, and core network performance. These metrics are collected in real-time from different network elements and can be used for real-time monitoring, analysis, and troubleshooting. 

We recently developed a new SMO Performance Dashboard and integrated it with AMCOP, learn more in my previous blog post. 

The O-RAN SMO also provides advanced analytics capabilities that enable network operators to detect and predict performance issues before they occur. This helps operators to proactively address performance issues, minimize network downtime, and improve overall network quality.

In addition to real-time monitoring and analytics, the O-RAN SMO also provides historical performance analysis capabilities. This enables network operators to analyze performance trends over time, identify areas for improvement, and optimize network performance over the long term.

Overall, the O-RAN SMO's performance management capabilities are critical for ensuring the reliability, availability, and quality of O-RAN networks. By providing real-time monitoring, predictive analytics, and historical analysis, the O-RAN SMO enables operators to optimize network performance, reduce downtime, and improve customer satisfaction.

Aarna.ml offers the number one open source and vendor neutral O-RAN SMO as part of Aarna.ml Multi Cluster Orchestration Platform (AMCOP). Learn more about O-RAN SMO and contact us for a free consultation. 

‍

Aarna.ml is growing fast and we are thrilled to introduce and welcome eight new, exceptionally talented staffers to the Aarna family. Each individual brings their unique skills, experiences, and enthusiasm to our dynamic team, propelling us further on our mission to help enterprises and network operators unlock previously unimagined new services, drastically slash operational costs, and improve time to market. Let's take a closer look at the remarkable individuals who have joined us:

1. Subash Saravanan: Crafting Excellence in Automation and Infrastructure

With a diverse skill set encompassing Python programming, networking (CCNA), shell scripting, RDBMS, and Ansible, Subash Saravanan is a valuable addition to our team. His ability to handle automation, containerization, and infrastructure management with ease enhances our capabilities to deliver innovative solutions. Subash holds a B.E. in Computer Science and Engineering from Govt College of Engineering, Bargur, Krishnagiri.

2. Sowmiya Jayakumar: Mastering Java and Problem Solving

Sowmiya's passion for Java programming and problem-solving makes her a perfect fit for our team. Her expertise extends to software tools like Git, Eclipse, IntelliJ IDEA, and Visual Studio Code. With proficiency in Java, Python, and Groovy Script, Sowmiya brings a fresh perspective to our projects. She holds a Bachelor's degree in Electronics and Communication Engineering from the University College of Engineering, Arni.

3. Thilagawathi Sambath: Navigating Networking and Python with Precision

Thilagawathi brings a solid understanding of networking and proficiency in Python and Core Java to our AES project. His knowledge and expertise are integral to our ongoing innovations. Thilagawathi graduated from the University College of Engineering in Arni-Tiruvannamalai, Tamil Nadu, with a degree in electronics and communication engineering.

4. Priyanka Shalini Jayaprakash: Adept Developer with a Technological Flair

Priyanka is a talented software developer with proficiency in Python, Java fundamentals, and networking. Her dedication to her craft is evident through her accomplishments, including a bachelor's degree in ECE from the University College of Engineering in Arni. Her skills will undoubtedly contribute to the success of our projects.

5. Durga Devi Selvaraj: Navigating Engineering with Expertise

Durga is a graduate of the University College of Engineering Arani with a degree in Electronic and Communication Engineering. Her proficiency in networking fundamentals, Java, Groovy scripts, and Python enhances our capabilities and furthers our commitment to excellence.

6. Tamilselvan Mani: Bridging Networking and Programming Proficiency

Tamilselvan brings a comprehensive skill set to our team, including networking basics, Python, Linux, Shell scripting, and more. With a background in Electronics and Communication Engineering from University College of Engineering Arni, his expertise contributes to our collective excellence.

7. Mohan Raj Dhandapani: Solving Challenges with Technical Proficiency

Mohan’s proficiency in Python, Shell scripts, Linux, and network basics is an asset to our team. His problem-solving abilities and knowledge of Flask framework add depth to our capabilities. Mohan holds a degree in Electronics and Communication Engineering from University College of Engineering Arni.

8. Gurpreet Singh: Adding Marketing Acumen to Our Team

Gurpreet joins us with extensive experience in marketing, advertising, and sales, acquired during his tenure at Amazon. His strategic insights and brand engagement expertise contribute to our dynamic team. Gurpreet holds an alumnus status at the Institute of Management Technology, Ghaziabad.

At Aarna.ml, our team is our greatest strength. With the addition of these talented individuals, we're excited to continue pushing boundaries, innovating, and delivering exceptional solutions. Welcome aboard! Together, we'll create a future defined by excellence and innovation.

‍

It’s been a little over a year since Nephio was first announced and the progress the community has made since that time has been inspiring. In my role on the Nephio TSC, I’ve had the chance to collaborate across the industry and build a community around this promising new approach to intent-based network orchestration.

Nephio is declarative and intent driven (an intent driven system monitors the current state and continuously reconciles it with the intended state) and presents a new approach to the industry for orchestration and management of network services, edge computing applications, and the underlying infrastructure. What's most exciting to me is using Nehio to overcome the following challenges for edge workload orchestration and management. 

• Scale: Edge deployments today are growing exponentially in size with tens of thousands of edge sites, tens of infra providers, with hundreds of workloads.
• Infrastructure Dependency: Workloads have specific requirements which means that infra needs to be orchestrated first to meet the requirements of the workloads. 
• Heterogeneity: Deployments today include a wide variety of edge clusters, storage, compute, K8s flavors,  HW, SW, Storage, services and application.  

Nephio takes on these challenges by using Kubernetes as a general purpose control plane. It constantly reconciles the current state with the intended state which is required for scaling large and complex networks. It is suitable for both infrastructure and workloads for on-demand infrastructure and clouds. And because it’s heterogeneous, it can handle both public and private clouds, multi-vendor environments, and third-party network functions and cloud native applications.

I’m also excited that Nephio uses the O2 interface (as defined in O-RAN architecture) as one of its stated use cases. In the recent LF Networking Developer & Testing Forum, I presented a session covering how O2 IMS (Infrastructure Management Service) can be achieved on Nephio architecture and how Nephio is influencing the evolution of radio networks with an O-RAN-O2 IMS K8s profile. With its simplicity and cloud native Kubernetes approach, I believe Nephio is a perfect fit for the management and orchestration of cloud native RAN workloads and I’m looking forward to further collaboration in this area. 

Lastly, I’m very proud of the way the community came together to achieve consensus on the scope, produce the release artifacts and documentation with many new features and capabilities, and deliver R1 for the industry.

If you’ve been on the sidelines with Nephio, I encourage you to get involved in the community by learning more on the Nephio wiki. Feel free to contact me as well about any questions or ideas for Nephio you may have and I’d be happy to reserve some time to discuss them.

Those in Bangalore, India are invited to a Nephio R1 Overview and Collaboration meetup on Thursday, September 28th. Learn more and register.

‍

O-RAN Service Management and Orchestration (SMO) is a component that addresses the orchestration, management, and automation aspects of O-RAN and is critical for supporting and managing disaggregated, multi-vendor RAN environments. Aarna.ml has developed the industry leading open source O-RAN SMO that is included in AMCOP.

Overview

A performance dashboard is a critical tool within the O-RAN SMO ecosystem that offers insights into the network's performance and key performance indicators (KPIs). The usability of a performance dashboard in O-RAN SMO can be quite beneficial for various stakeholders:

• Network operators: The performance dashboard offers historical and real-time statistics on network performance, enabling operators to keep an eye on the network's health, spot possible problems or bottlenecks, and take the necessary steps to improve user experience and optimize network performance.

• Service Assurance Teams: With the use of the dashboard, service assurance teams are able to monitor the quality of services provided via the O-RAN network, ensuring that service-level agreements (SLAs) are met and locating any service-related problems that require quick attention.

• Network Engineers: Network engineers may undertake in-depth analysis, diagnose issues, and fine-tune network configurations to increase overall network efficiency with the aid of performance data and visualizations on the dashboard.

• Capacity Planning: The dashboard aids in capacity planning for future network development and resource allocation by analyzing historical performance trends. That enables network operators to better anticipate and meet rising traffic demands.

• Decision Makers and Executives: The dashboard's high-level summary and KPIs make it simpler for executives and decision makers to comprehend the overall performance and health of the O-RAN network.

• Proactive Maintenance: The dashboard enables proactive maintenance with real-time data, enabling operators to identify anomalies early and address any problems before they have an impact on end users.

• Vendor Performance Evaluation: The dashboard can be used by operators who work with numerous suppliers to compare the performance of each vendor's machinery, assisting in vendor selection and guaranteeing contractual compliance.

The actual usability of a performance dashboard in O-RAN SMO will depend on its layout, user interface, the breadth of the performance indicators displayed, and the connection with other management and orchestration components. 

Getting Started

Aarna.ml AMCOP O-RAN SMO supports performance dashboard view in Grafana. There are two types of dashboards – one predefined and one user defined. The SMO will launch Grafana at deployment with some predefined dashboards added. In the new Performance App of SMO, these dashboards will be visible, but users cannot do anything from the UI. User defined dashboards can be added or deleted from the UI.

One can query, visualize, alert on, and analyze your metrics, logs, and traces using Grafana open source software, regardless of where they are kept. You can use the tools provided by Grafana OSS to create meaningful graphs and visualizations from the data in your time-series database (TSDB).

Grafana is accessible at http://<AMCOP_IP>:30080 and Prometheus is accessible at http://<AMCOP_IP>:30900/graph. Note: Please swap in the AMCOP_IP address to get the relevant results for your VM deployment.

Credentials to access the Grafana is: 

• username/password:-  admin/admin

To create a dashboard in Grafana, one has to follow the steps below:

• After selecting the dashboard option, there is a list of available dashboards. In the right-side corner there is the new button where one can choose the option. 

• To import an existing dashboard, one can choose any of  the following methods: 1. Import Dashboard from File, 2. Import Dashboard from Grafana.com, 3. Import via Panel json.

To add a new dashboard one needs to add the following data:

1. Add Visualization - Select a data source and then query and visualize your data with charts, statistics and tables or create lists, markdowns, and other widgets. 
2. Add row - Group your visualization into expandable sections.
3. Import Library Panel - Import visualizations that are shared with other dashboards.

After adding all the data, click on the Apply button on the right hand side to save the dashboard. 

To add matrices in Grafana, follow these steps:

Note: To send data, Users can use Aarna’s simulator to send PM data.

•  Run the simulator image (In this case we are using CU simulator)

• sudo docker run -d -p62300:830 -p61301:22 amcopnightly/amcop-cu-image:v0.1

• pick any one of the attached XML.gz files and copy it to the /tmp folder of the docker container that runs the simulator using below commands

• sudo docker ps | grep amcop-cu | awk '{print $1}'
• sudo docker cp A20230318.0115+0000-20230318.0120+0000_5_VCUDEVICE.xml.gz <container_id>:/tmp/
• sudo docker exec -it <container_id> bash -c "chown -R netconf:netconf /tmp/A20230318.0115+0000-20230318.0120+0000_5_VCUDEVICE.xml.gz"
• sudo docker exec -it <container_id> bash -c "ls -l /tmp"

• Follow the below steps to push a ves notification to get the file inside DFC

• Get ves collector IP

• kubectl get svc -n amcop-system | grep ves | awk '{print $3}'

• Modify the below  curl call to send ves notification

curl -i -X POST -H "Content-Type:application/json" -d '{

"event": {

"commonEventHeader": {

"version": "4.0.1",

"vesEventListenerVersion": "7.0.1",

"domain": "notification",

"eventName": "Noti_RnNode-Ericsson_FileReady",

"eventId": "FileReady_1797490e-10ae-4d48-9ea7-3d7d790b25e1",

"lastEpochMicrosec": 8745745764578,

"priority": "Normal",

"reportingEntityName": "otenb5309",

"sequence": 0,

"sourceName": "oteNB5309",

"startEpochMicrosec": 8745745764578,

"timeZoneOffset": "UTC+05.30"

},

"notificationFields": {

"changeIdentifier": "PM_MEAS_FILES",

"changeType": "FileReady",

"notificationFieldsVersion": "2.0",

"arrayOfNamedHashMap": [{

"name": "A20230318.0115+0000-20230318.0120+0000_5_VCUDEVICE.xml.gz“,

"hashMap": {

"location": 

"sftp://netconf:netconf@<vm_ip>:61301/tmp/A20230318.0115+0000-20230318.0120+0000_5_VCUDEVICE.xml.gz",

"compression": "gzip",

"fileFormatType": "org.3GPP.28.532#measData",

"fileFormatVersion": "V7"

}

}]

}

}

}

}' 'http://<ves_collector_ip>:8080/eventListener/v7'

Go to the Grafana dashboard and select the metrics from the dropdown and click apply. 

To get embedded URL from grafana dashboard, follow these steps:

• Open any dashboard and click on the panel name. A drop down will come. 
• Click on the share option. 
• One modal will come and then click on the embed option and copy the highlighted link. 

More on Performance App support in AMCOP SMO:

• Navigate to the performance tab from the left-hand side menu. 
• Find the list of dashboards with a dropdown to choose the dashboards to view, add, and delete. 
• Click on the dashboard to view the list of user defined dashboards. 
• Select or unselect the dashboards needed.lick on the Apply button to apply the changes. 
• On the right-hand side corner, there will be two buttons to add new and delete existing dashboards.
• Click on the add new button and fill in the details (dashboard or panel name and dashboard embed URL) to add a new dashboard. 
• Click on the delete button to delete one or multiple dashboards.

AMCOP O-RAN SMO

Aarna.ml AMCOP contains the number one open source SMO in the market for orchestrating and managing O-RAN network functions. It is a cloud native application that enables network operators and vendors to manage multi-vendor RAN environments and select best-of-breed network functions for validation and interoperability testing. It can be utilized in both stand-alone and non-stand-alone vRAN contexts as well as O-RAN environments. Learn more. 

We encourage to try out these performance dashboards and let us know what you find out or if you have any questions.

‍