Main information results and proposed solution are not described here until the submission in the Conference
An Applied Computing master's dissertation repository developed by Felipe Grings.
Link to Master's defense: https://youtu.be/m2SN5c9PoEo
NASP Demo: https://youtu.be/HqKeamq2XcM?si=P9CoZbDYnN28ek5Y
Master's thesis: https://track.authorhub.elsevier.com/?uuid=57423768-0494-4c9f-901f-7a68e21c897f
The 5th generation of mobile telecommunications (5G) is rapidly adopted worldwide, ac- celerating the demands for highly flexible private networks. In this context, 5G has mobile net- work slicing as one of its main features, where the 3rd Generation Partnership Project (3GPP) defines three main use cases: massive Internet of Things (mIoT), enhanced Mobile BroadBand (eMBB), and Ultra Reliable Low Latency Communications (URLLC), along with their man- agement functions. Moreover, the European Telecommunications Standards Institute (ETSI) defines standards for Zero-touch network & Service Management (ZSM) without human inter- vention. However, the technical documents of these institutes fail to define End-to-End (E2E) management and integration among different domains and subnet instances. This work presents a network slice as a service platform (NASP) agnostically to 3GPP and non-3GPP networks. A NASP architecture is based on the main components, namely: (i) onboard requests for new slices at the business level, fulfilling the translation for definitions of physical instances, distri- butions, and interfaces among domains; (ii) hierarchy orchestrator working among management functions; and (iii) communication interfaces with network controllers. These configurations are based on the technical documents of entities such as 3GPP, ETSI, and O-RAN, following the study of overlapping designs and gaps among the different views. The NASP prototype was developed based on the proposed architecture, bringing implementations and solutions for an agnostic platform and provider of an end-to-end Network Slice as a Service. The tests were analyzed using two use cases (3GPP and Non-3GPP) with four different scenarios, i.e., mIoT, URLLC, 3GPP Shared, and Non-3GPP. The results pointed out the platform’s adaptabil- ity in serving different requests received by the Communication Service Management Function. Moreover, the evaluation showed the time to create a Network Slice Instance, where 68% is dedicated to the Core configuration. The tests also presented a 93% reduction in data session establishment time comparing the URLLC and Shared scenarios. Finally, the study presents the cost variation for operating the platform with the orchestration of 5 and 10 slices, presenting a variation of 112% between Edge and Central.
The project presents the architecture for a platform for NSaaS, considering the end-to-end NS, mapping the definitions of international entities and their relationship, and proposing tech- niques to supply the gaps among the technical documents. For this, several technical documents from entities such as 3GPP, ETSI, GSMA, and O-RAN Alliance were analyzed, where the most mature and convergent definitions were merged, forming the main blocks of this project. There- fore, the architecture maps the orchestration and management of a multi-slice network working with slicing as a service using the management definitions found in 3GPP. The main purposes can be defined in five topics: (i) translation of business templates to define instantiated network subslices; (ii) use of a hierarchy of responsibilities for the deployment of slices; (iii) definition of communication interfaces between management functions and control agents; (iv) Appli- cation of CI/CD concept and IaC strategies for integration between management and physical instances, and finally; (v) telemetry and observability flow.
NASP manages three physical and distributed different domains, RAN, Transport, and Core, described by 3GPP and ETSI reference architectures as shown in Figure 4. The NASP architec- ture has four major components: (i) Onboard; (ii) Orchestration; (iii) Quality Assurance Closed Loops; (iv) Interfaces (RN1, TN1, CN1) to integrate with the controller as shown at the top of Figure 4. The NASP architecture describes the integration and components required on an NSaaS platform to onboard, deploy, and control the entire lifecycle of a network slice through all domains from a mobile network. Furthermore, the NASP architecture proposes a communi- cation flow and interfaces to instantiate all infrastructure necessary to establish an E2E slice, as seen in the lower part of Figure 4. The architecture also proposed the responsibilities for each domain and decision level in each closed loop to pursue SLAs set at the onboard process. In this way, the architecture is based on the responsibilities for each domain and at the decision level defined in the integration process. Each macro functionality of the NASP architecture as a service is described in the following.
The NASP prototype uses Open Source tools to control different domains and manage the infrastructure. The work uses Docker and Container Runtime Interface (CRI) as the container run-time for virtualization and containerization, and for container orchestration, we are choos- ing Kubernetes running over Ubuntu 20.04 machines. Moreover, we define to work with Open Network Operating System (ONOS) for SDN infrastructure controlling a Mininet virtual envi- ronment. We also use OpenAir interface Radio Access modules for 3GPP use cases for RAN control, and the Core network runs over the Free5gc project.
Figure 11 shows SMO functions are containerized NVFs running over a Kubernetes con- tainer orchestrator exposing Hypertext Transfer Protocol (HTTP) services over Ingress Con- troller and Load Balancer provided by Kubernetes. Core NSSMF function interface with Core controller will use Kube-API-server API provided by Kubernetes over HTTP Representational State Transfer (REST) API. Kubernetes is a complete REST Service-based Architecture, so once authentication keys are changed, all infrastructure management is available in an M2M communication. Transport NSSMF function interface with WAN SDN Controller is projected to use ONOS REST API provided by the ONOS project controlling a Mininet virtualized in- frastructure. The API contains manipulation interfaces for devices, network topology, hosts, groups, components, and applications, where once the infrastructure is deployed, NASP throw ONOS can request intents depending on Slice requirements. Finally, RAN NSSMF interfaces with Kube-API-server and My5G-RANTester. Templates, instance descriptions, and Infrastruc- ture codes are designed to be stored as documents in a document-oriented database, in this case, MongoDB. Infrastructure codes are using Helm manifests as a majority. Sometimes, the use of Helm cannot be stored as a manifest. In that case, raw YAMLs were projected to be used as infrastructure descriptors. Tiers distinguish templates and Instance descriptions as tags on document descriptions. For monitoring, alerting, and observability, the project was based on Prometheus as a metric collector, Istio as traffic tracing, and Grafana to expose all the data and configure alerting webhooks.
The NG-RAN simulation initially is integrated with the My5G-RANTester, on the entire RAN disaggregation protocol stack. First, however, nodes must be available for instantiation and able to request at the RAN Controller interface. Next, the node is virtualized in a container for the prototype to orchestrate Kubernetes. Therefore, according to the characteristics of Ku- bernetes, initially, the multi-tasking infrastructure or O-Cloud (from the O-RAN architecture) is developed in the cluster format and subdivided into three distinct computational resources (CRs). Finally, the Kubernetes platform validates the prototype once the orchestrator is cus- tomized within the scaling function and the multi-tasking infrastructure’s global vision
This repository descries steps for innstalling the infrastructure and elements require to make up 6G Network Slice Platform (K8s Infrastructure + 5G Core + N3IWF + UEs Testers + NASP). The instalation process involves X diferrent major steps (Kubernetes Infrastructure + NASP + Run UEs). This project requires at least 2 linux machines.
Kubernetes Infrastructure: 2x
- SO: Ubuntu 22.04 (LTS) x64
- Uname -r >= 5.4.0-122-generic
- Memory: +8GB RAM
- Disk: 80GB
The steps described in this repository were performed based on a network topology similar to the one shown in the figure below:
The installaltion step for each component are described below. Attention: All installation steps deescribed bellow should be executed with root privileges(SUDO SU).
For Each Machine Get acces to any Ubuntu 22.04, with specified resources, and prepare the enviroment to the Platform installation
Install dwarves make and gcc
sudo apt install -y dwarves make gcc curlCopy modules to BPF linux kernel
cp /sys/kernel/btf/vmlinux /usr/lib/modules/`uname -r`/build/Get GTP5G
git clone -b v0.8.2 https://github.com/free5gc/gtp5g.git
cd gtp5g && make clean && make && make installFor Kubernetes Master Machine (ETCD Installed)
Get acces to any Ubuntu 22.04, with specified resources, that you want to be a Cluster Master (etcd installed). Access via terminal the machine where the 5GC will be executed and Run:
Install python3-pip annd python3-venv
apt install -y python3-pip python3-venvGet Kubespray github:
git clone https://github.com/kubernetes-sigs/kubespray.gitGenerate RSA KEY and paste to each .authozired_keys file in each machine
ssh-keygen -t rsa -C "$HOSTNAME" -f "$HOME/.ssh/id_rsa" -P ""
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_rsa.pubPrepare Kubespray packages
VENVDIR=kubespray-venv
KUBESPRAYDIR=kubespray
python3 -m venv $VENVDIR
source $VENVDIR/bin/activate
cd $KUBESPRAYDIR
pip install -U -r requirements.txtCreate Kubespray Inventory
cp -rfp inventory/sample inventory/myclusterDeclare Cluster IPS. ATTENTION. Space separated
declare -a IPS=(IP_1 IP_2 IP_3)
# should look like
# declare -a IPS=(10.116.0.3 10.116.0.2)Create Inventory Configs
CONFIG_FILE=inventory/mycluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}Enable Multus:
sed -e 's/kube_network_plugin_multus: false/kube_network_plugin_multus: true/' -i roles/kubespray-defaults/defaults/main.yaml ; grep plugin_multus roles/kubespray-defaults/defaults/main.yamlEnable IP Forward
sed -e 's/calico_allow_ip_forwarding: false/calico_allow_ip_forwarding: true/' -i roles/network_plugin/calico/defaults/main.yml ; grep forw roles/network_plugin/calico/defaults/main.ymlRUN ANSIBLE*
ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.ymlThe Excepted output should be like this:
Install Kubectl
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-archive-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install kubectlConnect to any machine and execute
kubectl get nodesAnd the output should be like this::
Access via terminal a machine who has access to Kubernetes Cluster CTL. Check Kubernetes version:
sudo kubectl --version
Install HELM
curl -fsSL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bashCreate Monitoring Namespace
kubectl create namespace monitoringInstall Observability Helm Charts: Install Prometheus
helm install prometheus prometheus-community/kube-prometheus-stack -n monitoring
Install Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm create ns grafana
helm install grafana -n monitoring
kubectl expose service grafana --port=3000 --type=NodePort --target-port=3000 --name=grafana-exposed
Configure Prometheus Data Sorce
Add Network Dashboard to Grafana
https://grafana.com/grafana/dashboards/12125-kubernetes-networking-namespace-pods/
Install Loki:
helm repo add loki https://grafana.github.io/helm-charts
helm create ns loki
helm install loki -n monitoring
Install Linkerd:
# To add the repo for Linkerd stable releases:
helm repo add linkerd https://helm.linkerd.io/stable
# To add the repo for Linkerd edge releases:
helm repo add linkerd-edge https://helm.linkerd.io/edge
helm install linkerd-crds linkerd/linkerd-crds \
-n linkerd --create-namespace
helm install linkerd-control-plane \
-n linkerd \
--set-file identityTrustAnchorsPEM=ca.crt \
--set-file identity.issuer.tls.crtPEM=issuer.crt \
--set-file identity.issuer.tls.keyPEM=issuer.key \
linkerd/linkerd-control-plane
In our local machine install python3, pip and poetry:
sudo apt install -y python3 python3-pip poetry
Clone this repository:
git clone https://github.com/fhgrings/NASP.git
Access NASP Python application:
cd NASP/nasp
Install and Update python packages:
poetry install --all
Run the following command to execute NASP Application with poetry environment:
poetry run flask run --debug
Copy .kube/config file from one of the kubernetes Machines and paste on
./NASP/config/kubeconfigThen, access it from your browser:
Download the example infrastructure require from the repository:
https://github.com/fhgrings/NASP/tree/main/helm_charts
Open the main values.yaml on Helm repo root directory
Update input values from required on infrastructure choosen and save the file:
global:
name: free5gc
userPlaneArchitecture: single # possible values are "single" and "ulcl"
nrf:
service:
name: nrf-nnrf
type: ClusterIP
port: "8000"
nodePort: "30800"
nodeSelector:
sbi:
scheme: http
amf:
service:
name: amf-n2
port: 38412
targetport: 31412
protocol: SCTP
type: NodePort
n2:Compress the entire folder
Upload the ZIP file to http://localhost:5000/nsst using the Add Subnet Slice Template button:
To run the UE copy the file from https://github.com/fhgrings/NASP/tree/main/helm_charts/rantester
Compress and import on NSST tab.
To Deploy it Create an NST using Free5GC as Core and RAN_Tester as RAN and UE. After create, press Deploy NST.
After a few seconds the slice will appear as an instance on Instances Tab
To check where UE is located use the command
kubectl get pods -A -o wide | grep -e rantester -e NAMEIt should return something like:
To connect to the UE and run the network load tests use:
kubectl exec -it ran-rantester-0 -n {GIVEN_NAMESPACE} -- bashAfter connected run
ping -I uetun1 www.google.com
The Dashboard Tabs should start to show the metrics data related to the instance
Skipping BTF generation for /root/gtp5g/gtp5g.ko due to unavailability of vmlinux
/tools/bpf/resolve_btfids/resolve_btfids: not found
You should be using Ubuntu 20.04+. Debian distros do not work properly with Free5gc GTP5G
AMF Logs
Protocol Not Supported
Double Check SMTP properties in your CNI configs
XMFR Operation not permitted
N3IWF Container is not runnning with Privileged permissions. Double check security Properties in Pod definition Double check Container Runtime Permissions
The 5GC network functions are initialized through a ./run.sh. If, eventually, the connection with the terminal that initialized the .sh is closed improperly, the functions will continue running as loose processes on the execution stack. When trying to initialize the 5GC again, errors related to connection ports that are already in use will be triggered.
To solve the problem, just run ./force_kill.sh, is located in the same directory as the run.sh script. When executing ./force_kill.sh all active processes related to the execution of the 5GC functions will be terminated and a new execution can be performed.
For a UE to be able to use the 5GC functionalities, it must be duly registered in the database. This record is performed through a Rest API provided by the 5GC. If eventually the terminal used to initialize the API is closed improperly, port 5000 will remain in use, preventing a new execution. In this case, the process must be terminated using kill -9 as described below:
- Find the PID of the process that is using port 50000:
sudo netstat -nlp | grep :5000 - Kill the process through the PID::
kill -9 <<PID-Nº>>
In experimental scenarios, where the UE-non3GPP + 5GC connection procedures are repeated several times, it may happen that MongoDB persists inconsistent information. In these cases, it is interesting to delete the database and register the UEs again. To resolve the issue, perform the following steps:
- Terminate the execution of UE-non3GPP, N3IWF and 5GC.
- Access through a new terminal on the machine where 5GC was installed.
- Type in terminal
mongoto connect MongoDB CLI. - In MongoDB CLI type
use free5gcto connect to free5gc base. - Type
db.dropDatabase()to completely delete the 5gc database.
After deleting the database, initialize 5GC, initialize API Server, initialize N3IWF, register UE-non3GPP and test Proto6G components.