docs/installation-guide.rst

.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. SPDX-License-Identifier: CC-BY-4.0
.. Copyright (C) 2019 Wind River Systems, Inc.


Installation Guide
==================

.. contents::
   :depth: 3
   :local:

Abstract
--------

This document describes how to install O-RAN INF image, example configuration for better
real time performance, and example deployment of Kubernetes cluster and plugins. 

The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
and container technology.

Version history

+--------------------+--------------------+--------------------+--------------------+
| **Date**           | **Ver.**           | **Author**         | **Comment**        |
|                    |                    |                    |                    |
+--------------------+--------------------+--------------------+--------------------+
| 2019-11-02         | 1.0.0              | Jackie Huang       | Initail version    |
|                    |                    |                    |                    |
+--------------------+--------------------+--------------------+--------------------+
|                    |                    |                    |                    |
|                    |                    |                    |                    |
+--------------------+--------------------+--------------------+--------------------+
|                    |                    |                    |                    |
|                    |                    |                    |                    |
+--------------------+--------------------+--------------------+--------------------+


Preface
-------

Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.


Hardware Requirements
---------------------

Following minimum hardware requirements must be met for installation of O-RAN INF image:

+--------------------+----------------------------------------------------+
| **HW Aspect**      | **Requirement**                                    |
|                    |                                                    |
+--------------------+----------------------------------------------------+
| **# of servers**   | 1                                                  |
+--------------------+----------------------------------------------------+
| **CPU**            | 2                                                  |
|                    |                                                    |
+--------------------+----------------------------------------------------+
| **RAM**            | 4G                                                 |
|                    |                                                    |
+--------------------+----------------------------------------------------+
| **Disk**           | 20G                                                |
|                    |                                                    |
+--------------------+----------------------------------------------------+
| **NICs**           | 1                                                  |
|                    |                                                    |
+--------------------+----------------------------------------------------+



Software Installation and Deployment
------------------------------------

1. Installation from the O-RAN INF ISO image
````````````````````````````````````````````

- Please see the README.md file for how to build the image.
- The Image is a live ISO image with CLI installer: oran-image-inf-host-intel-x86-64.iso

1.1 Burn the image to the USB device
''''''''''''''''''''''''''''''''''''

- Assume the the usb device is /dev/sdX here

::

  $ sudo dd if=/path/to/oran-image-inf-host-intel-x86-64.iso of=/dev/sdX bs=1M

1.2 Insert the USB device in the target to be booted.
'''''''''''''''''''''''''''''''''''''''''''''''''''''

1.3 Reboot the target from the USB device.
''''''''''''''''''''''''''''''''''''''''''

1.4 Select "Graphics console install" or "Serial console install" and press ENTER
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

1.5 Select the hard disk and press ENTER
''''''''''''''''''''''''''''''''''''''''

Notes: In this installer, you can only select which hard disk to install, the whole disk will be used and partitioned automatically.

- e.g. insert "sda" and press ENTER

1.6 Remove the USB device and press ENTER to reboot
'''''''''''''''''''''''''''''''''''''''''''''''''''

2. Configuration for better real time performance
`````````````````````````````````````````````````

Notes: Some of the tuning options are machine specific or depend on use cases,
like the hugepages, isolcpus, rcu_nocbs, kthread_cpus, irqaffinity, nohz_full and
so on, please do not just copy and past.

- Edit the grub.cfg with the following example tuning options

::

  # Notes: the grub.cfg file path is different for legacy and UEFI mode
  #   For legacy mode: /boot/grub/grub.cfg
  #   For UEFI mode: /boot/EFI/BOOT/grub.cfg

  grub_cfg="/boot/grub/grub.cfg"
  #grub_cfg="/boot/EFI/BOOT/grub.cfg"

  # In this example, 1-16 cores are isolated for real time processes
  root@intel-x86-64:~# rt_tuning="crashkernel=auto biosdevname=0 iommu=pt usbcore.autosuspend=-1 nmi_watchdog=0 softlockup_panic=0 intel_iommu=on cgroup_enable=memory skew_tick=1 hugepagesz=1G hugepages=4 default_hugepagesz=1G isolcpus=1-16 rcu_nocbs=1-16 kthread_cpus=0 irqaffinity=0 nohz=on nohz_full=1-16 intel_idle.max_cstate=0 processor.max_cstate=1 intel_pstate=disable nosoftlockup idle=poll mce=ignore_ce"

  # optional to add the console setting
  root@intel-x86-64:~# console="console=ttyS0,115200"

  root@intel-x86-64:~# sed -i "/linux / s/$/ $console $rt_tuning/" $grub_cfg


- Reboot the target

::

  root@intel-x86-64:~# reboot

3. Kubernetes cluster and plugins deployment instructions (All-in-one)
``````````````````````````````````````````````````````````````````````
This instruction will show you how to deploy kubernetes cluster and plugins in an all-in-one example scenario after the above installation.

3.1 Change the hostname (Optional)
''''''''''''''''''''''''''''''''''

::

  # Assuming the hostname is oran-aio, ip address is <aio_host_ip>
  # please DO NOT copy and paste, use your actaul hostname and ip address
  root@intel-x86-64:~# echo oran-aio > /etc/hostname
  root@intel-x86-64:~# export AIO_HOST_IP="<aio_host_ip>"
  root@intel-x86-64:~# echo "$AIO_HOST_IP oran-aio" >> /etc/hosts

3.2 Disable swap for Kubernetes
'''''''''''''''''''''''''''''''

::

  root@intel-x86-64:~# sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
  root@intel-x86-64:~# systemctl mask dev-sda4.swap

3.3 Set the proxy for docker (Optional)
'''''''''''''''''''''''''''''''''''''''

- If you are under a firewall, you may need to set the proxy for docker to pull images

::

  root@intel-x86-64:~# HTTP_PROXY="http://<your_proxy_server_ip>:<port>"
  root@intel-x86-64:~# mkdir /etc/systemd/system/docker.service.d/
  root@intel-x86-64:~# cat << EOF > /etc/systemd/system/docker.service.d/http-proxy.conf
  [Service]
  Environment="HTTP_PROXY=$HTTP_PROXY" "NO_PROXY=localhost,127.0.0.1,localaddress,.localdomain.com,$AIO_HOST_IP,10.244.0.0/16"
  EOF

3.4 Reboot the target
'''''''''''''''''''''

::

  root@intel-x86-64:~# reboot

3.5 Initialize kubernetes cluster master
''''''''''''''''''''''''''''''''''''''''

::

  root@oran-aio:~# kubeadm init --kubernetes-version v1.16.2 --pod-network-cidr=10.244.0.0/16
  root@oran-aio:~# mkdir -p $HOME/.kube
  root@oran-aio:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  root@oran-aio:~# chown $(id -u):$(id -g) $HOME/.kube/config

3.6 Make the master also works as a worker node
'''''''''''''''''''''''''''''''''''''''''''''''

::

  root@oran-aio:~# kubectl taint nodes oran-aio node-role.kubernetes.io/master-

3.7 Deploy flannel
''''''''''''''''''

::

  root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/flannel/kube-flannel.yml

Check that the aio node is ready after flannel is successfully deployed and running

::

  root@oran-aio:~# kubectl get pods --all-namespaces |grep flannel
  kube-system   kube-flannel-ds-amd64-bwt52        1/1     Running   0          3m24s

  root@oran-aio:~# kubectl get nodes
  NAME       STATUS   ROLES    AGE     VERSION
  oran-aio   Ready    master   3m17s   v1.15.2-dirty

3.8 Deploy kubernetes dashboard
'''''''''''''''''''''''''''''''

Deploy kubernetes dashboard

::

  root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/kubernetes-dashboard/kubernetes-dashboard-admin.rbac.yaml
  root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/kubernetes-dashboard/kubernetes-dashboard.yaml

Verify that the dashboard is up and running

::

  # Check the pod for dashboard
  root@oran-aio:~# kubectl get pods --all-namespaces |grep dashboard
  kube-system   kubernetes-dashboard-5b67bf4d5f-ghg4f   1/1     Running   0          64s

Access the dashboard UI in a web browser with the https url, port number is 30443.

- For detail usage, please refer to `Doc for dashboard`_

.. _`Doc for dashboard`: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

3.9 Deploy Multus-CNI
'''''''''''''''''''''

::

  root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/multus-cni/multus-daemonset.yml

Verify that the multus-cni is up and running

::

  root@oran-aio:~# kubectl get pods --all-namespaces | grep -i multus
  kube-system   kube-multus-ds-amd64-hjpk4              1/1     Running   0          7m34s

- For further validating, please refer to the `Multus-CNI quick start`_

.. _`Multus-CNI quick start`: https://github.com/intel/multus-cni/blob/master/doc/quickstart.md

3.10 Deploy NFD (node-feature-discovery)
''''''''''''''''''''''''''''''''''''''''

::

  root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/node-feature-discovery/nfd-master.yaml
  root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/node-feature-discovery/nfd-worker-daemonset.yaml

Verify that nfd-master and nfd-worker are up and running

::

  root@oran-aio:~# kubectl get pods --all-namespaces |grep nfd
  default       nfd-master-7v75k                        1/1     Running   0          91s
  default       nfd-worker-xn797                        1/1     Running   0          24s

Verify that the node is labeled by nfd:

::

  root@oran-aio:~# kubectl describe nodes|grep feature.node.kubernetes
                     feature.node.kubernetes.io/cpu-cpuid.AESNI=true
                     feature.node.kubernetes.io/cpu-cpuid.AVX=true
                     feature.node.kubernetes.io/cpu-cpuid.AVX2=true
                     (...snip...)

3.11 Deploy SRIOV CNI
'''''''''''''''''''''

Provision VF drivers and devices


Enumerate  PF Devices

::

  root@oran-aio:~/dpdk-18.08/usertools# lspci -D |grep 82599
  0000:04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  0000:04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Correlate the PF device to eth interfaces and bring them up

::

  root@oran-aio:~# ethtool -i eth4  |grep bus-info
  bus-info: 0000:04:00.0
  root@oran-aio:~# ethtool -i eth5  |grep bus-info
  bus-info: 0000:04:00.1
  root@oran-aio:~# ifconfig eth4 up
  root@oran-aio:~# ifconfig eth5 up

Load VF Driver modules

::

  root@oran-aio:~# modprobe ixgbevf
  root@oran-aio:~# modprobe uio
  root@oran-aio:~# modprobe igb-uio
  root@oran-aio:~# modprobe vfio
  root@oran-aio:~# modprobe vfio-pci
  root@oran-aio:~# lsmod |grep ixgbevf
  ixgbevf                61440  0
  root@oran-aio:~# lsmod |grep vfio
  vfio_pci               40960  0
  vfio_virqfd            16384  1 vfio_pci
  vfio_iommu_type1       24576  0
  vfio                   24576  2 vfio_iommu_type1,vfio_pci
  irqbypass              16384  2 vfio_pci,kvm


Bind VF drivers to VF devices

::

  root@oran-aio:~# cat /sys/bus/pci/devices/0000\:04\:00.0/sriov_totalvfs
  root@oran-aio:~# cat /sys/bus/pci/devices/0000\:04\:00.1/sriov_totalvfs
  root@oran-aio:~# cat /sys/bus/pci/devices/0000\:04\:00.0/sriov_numvfs
  root@oran-aio:~# cat /sys/bus/pci/devices/0000\:04\:00.1/sriov_numvfs
  root@oran-aio:~# echo 8 > /sys/bus/pci/devices/0000\:04\:00.0/sriov_numvfs
  root@oran-aio:~# echo 8 > /sys/bus/pci/devices/0000\:04\:00.1/sriov_numvfs
  
  root@oran-aio:~# lspci -D |grep 82599
  0000:04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  0000:04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  0000:04:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  0000:04:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  
  root@oran-aio:~# dpdk-devbind -b vfio-pci 0000:04:11.0 0000:04:11.1 0000:04:11.2 0000:04:11.3 0000:04:11.4 0000:04:11.5 0000:04:11.6 0000:04:11.7
  
  root@oran-aio:~# dpdk-devbind --status-dev net
  
  Network devices using DPDK-compatible driver
  ============================================
  0000:04:11.0 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf,igb_uio
  0000:04:11.1 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf,igb_uio
  0000:04:11.2 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf,igb_uio
  0000:04:11.3 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf,igb_uio
  0000:04:11.4 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf,igb_uio
  0000:04:11.5 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf,igb_uio
  0000:04:11.6 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf,igb_uio
  0000:04:11.7 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf,igb_uio
  
  Network devices using kernel driver
  ===================================
  0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=eth4 drv=ixgbe unused=igb_uio,vfio-pci
  0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=eth5 drv=ixgbe unused=igb_uio,vfio-pci
  0000:04:10.0 '82599 Ethernet Controller Virtual Function 10ed' if=eth6 drv=ixgbevf unused=igb_uio,vfio-pci
  0000:04:10.1 '82599 Ethernet Controller Virtual Function 10ed' if=eth14 drv=ixgbevf unused=igb_uio,vfio-pci
  0000:04:10.2 '82599 Ethernet Controller Virtual Function 10ed' if=eth7 drv=ixgbevf unused=igb_uio,vfio-pci
  0000:04:10.3 '82599 Ethernet Controller Virtual Function 10ed' if=eth15 drv=ixgbevf unused=igb_uio,vfio-pci
  0000:04:10.4 '82599 Ethernet Controller Virtual Function 10ed' if=eth8 drv=ixgbevf unused=igb_uio,vfio-pci
  0000:04:10.5 '82599 Ethernet Controller Virtual Function 10ed' if=eth16 drv=ixgbevf unused=igb_uio,vfio-pci
  0000:04:10.6 '82599 Ethernet Controller Virtual Function 10ed' if= drv=ixgbevf unused=igb_uio,vfio-pci
  0000:04:10.7 '82599 Ethernet Controller Virtual Function 10ed' if=eth17 drv=ixgbevf unused=igb_uio,vfio-pci


Build SRIOV CNI

::

  root@oran-aio:~# HTTP_PROXY="http://<your_proxy_server_ip>:<port>"
  
  root@oran-aio:~# wget https://dl.google.com/go/go1.14.1.linux-amd64.tar.gz
  root@oran-aio:~# tar -zxvf go1.14.1.linux-amd64.tar.gz
  root@oran-aio:~# PATH=$PATH:/root/go/bin/
  root@oran-aio:~# git clone https://github.com/intel/sriov-cni
  root@oran-aio:~# cd sriov-cni
  root@oran-aio:~/sriov-cni# make
  root@oran-aio:~/sriov-cni# cp build/sriov /opt/cni/bin
  
  root@oran-aio:~# cd ~/
  root@oran-aio:~# git clone https://github.com/intel/sriov-network-device-plugin
  root@oran-aio:~# cd sriov-network-device-plugin
  root@oran-aio:~/sriov-network-device-plugin# git fetch origin pull/196/head:fpgadp
  root@oran-aio:~/sriov-network-device-plugin# git checkout fpgadp
  root@oran-aio:~/sriov-network-device-plugin# make image
  root@oran-aio:~/sriov-network-device-plugin# docker images |grep sriov-device-plugin
  nfvpe/sriov-device-plugin                             latest              f4e6bbefad67        5 minutes ago       25.5MB


Deploy SRIOV CNI

::

  root@oran-aio:~/sriov-network-device-plugin# cat <<EOF> deployments/sriovdp_configMap.yaml
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: sriovdp-config
    namespace: kube-system
  data:
    config.json: |
      {
          "resourceList": [{
                  "resourceName": "intel_sriov_netdevice",
                  "selectors": {
                      "vendors": ["8086"],
                      "devices": ["154c", "10ed"],
                      "drivers": ["i40evf", "ixgbevf"]
                  }
              },
              {
                  "resourceName": "intel_sriov_dpdk",
                  "selectors": {
                      "vendors": ["8086"],
                      "devices": ["154c", "10ed"],
                      "drivers": ["vfio-pci"]
                  }
              },
              {
                  "resourceName": "mlnx_sriov_rdma",
                  "isRdma": true,
                  "selectors": {
                      "vendors": ["15b3"],
                      "devices": ["1018"],
                      "drivers": ["mlx5_ib"]
                  }
              }
          ]
      }
  EOF
  
  root@oran-aio:~/sriov-network-device-plugin# kubectl create -f deployments/sriovdp_configMap.yaml
  root@oran-aio:~/sriov-network-device-plugin# kubectl create -f deployments/k8s-v1.16/sriovdp-daemonset.yaml

  root@oran-aio:~/sriov-network-device-plugin# kubectl get pods --all-namespaces |grep kube-sriov-device-plugin
  kube-system   kube-sriov-device-plugin-amd64-6lm8n   1/1     Running   0          12m
  
  root@oran-aio:~/sriov-network-device-plugin# kubectl -n kube-system logs kube-sriov-device-plugin-amd64-6lm8n
  I0327 02:14:46.488409   14488 manager.go:115] Creating new ResourcePool: intel_sriov_netdevice
  I0327 02:14:46.488427   14488 factory.go:144] device added: [pciAddr: 0000:04:10.0, vendor: 8086, device: 10ed, driver: ixgbevf]
  I0327 02:14:46.488439   14488 factory.go:144] device added: [pciAddr: 0000:04:10.1, vendor: 8086, device: 10ed, driver: ixgbevf]
  I0327 02:14:46.488446   14488 factory.go:144] device added: [pciAddr: 0000:04:10.2, vendor: 8086, device: 10ed, driver: ixgbevf]
  I0327 02:14:46.488459   14488 factory.go:144] device added: [pciAddr: 0000:04:10.3, vendor: 8086, device: 10ed, driver: ixgbevf]
  I0327 02:14:46.488467   14488 factory.go:144] device added: [pciAddr: 0000:04:10.4, vendor: 8086, device: 10ed, driver: ixgbevf]
  I0327 02:14:46.488473   14488 factory.go:144] device added: [pciAddr: 0000:04:10.5, vendor: 8086, device: 10ed, driver: ixgbevf]
  I0327 02:14:46.488479   14488 factory.go:144] device added: [pciAddr: 0000:04:10.6, vendor: 8086, device: 10ed, driver: ixgbevf]
  I0327 02:14:46.488485   14488 factory.go:144] device added: [pciAddr: 0000:04:10.7, vendor: 8086, device: 10ed, driver: ixgbevf]
  I0327 02:14:46.488502   14488 manager.go:128] New resource server is created for intel_sriov_netdevice ResourcePool
  I0327 02:14:46.488511   14488 manager.go:114]
  I0327 02:14:46.488516   14488 manager.go:115] Creating new ResourcePool: intel_sriov_dpdk
  I0327 02:14:46.488529   14488 factory.go:144] device added: [pciAddr: 0000:04:11.0, vendor: 8086, device: 10ed, driver: vfio-pci]
  I0327 02:14:46.488538   14488 factory.go:144] device added: [pciAddr: 0000:04:11.1, vendor: 8086, device: 10ed, driver: vfio-pci]
  I0327 02:14:46.488545   14488 factory.go:144] device added: [pciAddr: 0000:04:11.2, vendor: 8086, device: 10ed, driver: vfio-pci]
  I0327 02:14:46.488551   14488 factory.go:144] device added: [pciAddr: 0000:04:11.3, vendor: 8086, device: 10ed, driver: vfio-pci]
  I0327 02:14:46.488562   14488 factory.go:144] device added: [pciAddr: 0000:04:11.4, vendor: 8086, device: 10ed, driver: vfio-pci]
  I0327 02:14:46.488569   14488 factory.go:144] device added: [pciAddr: 0000:04:11.5, vendor: 8086, device: 10ed, driver: vfio-pci]
  I0327 02:14:46.488575   14488 factory.go:144] device added: [pciAddr: 0000:04:11.6, vendor: 8086, device: 10ed, driver: vfio-pci]
  I0327 02:14:46.488581   14488 factory.go:144] device added: [pciAddr: 0000:04:11.7, vendor: 8086, device: 10ed, driver: vfio-pci]
  I0327 02:14:46.488591   14488 manager.go:128] New resource server is created for intel_sriov_dpdk ResourcePool


Test intel_sriov_netdeivce

::

  root@oran-aio:~/sriov-network-device-plugin# cat <<EOF> deployments/sriov-crd.yaml
  apiVersion: "k8s.cni.cncf.io/v1"
  kind: NetworkAttachmentDefinition
  metadata:
    name: sriov-net1
    annotations:
      k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_netdevice
  spec:
    config: '{
    "type": "sriov",
    "cniVersion": "0.3.1",
    "name": "sriov-network",
    "vlan": 100,
    "ipam": {
      "type": "host-local",
      "subnet": "10.56.217.0/24",
      "routes": [{
        "dst": "0.0.0.0/0"
      }],
      "gateway": "10.56.217.1"
    }
  }'
  EOF
  
  root@oran-aio:~/sriov-network-device-plugin# kubectl create -f deployments/sriov-crd.yaml
  root@oran-aio:~/sriov-network-device-plugin# kubectl create -f deployments/pod-tc1.yaml
  root@oran-aio:~/sriov-network-device-plugin# kubectl get pods  |grep testpod1
  root@oran-aio:~/sriov-network-device-plugin# ip link |grep 'vlan 100'
    vf 3 MAC a6:01:0a:34:39:e1, vlan 100, spoof checking on, link-state auto, trust off, query_rss off
   
  root@oran-aio:~/sriov-network-device-plugin# kubectl exec -it testpod1 -- ip addr show |grep a6:01:0a:34:39:e1 -C 2
    valid_lft forever preferred_lft forever
  21: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a6:01:0a:34:39:e1 brd ff:ff:ff:ff:ff:ff
    inet 10.56.217.3/24 brd 10.56.217.255 scope global net1
       valid_lft forever preferred_lft forever


Test intel_sriov_dpdk

::

  root@oran-aio:~/sriov-network-device-plugin# cat <<EOF> deployments/sriovdpdk-crd.yaml
  apiVersion: "k8s.cni.cncf.io/v1"
  kind: NetworkAttachmentDefinition
  metadata:
    name: sriov1-vfio
    annotations:
      k8s.v1.cni.cncf.io/resourceName: intel.com/intel_sriov_dpdk
  spec:
    config: '{
    "type": "sriov",
    "cniVersion": "0.3.1",
    "vlan": 101,
    "name": "sriov1-vfio"
  }'
  EOF
  
  root@oran-aio:~/sriov-network-device-plugin# cat <<EOF> deployments/dpdk-1g.yaml
  apiVersion: v1
  kind: Pod
  metadata:
    name: dpdk-1g
    annotations:
      k8s.v1.cni.cncf.io/networks: '[
         {"name": "sriov1-vfio"},
         {"name": "sriov1-vfio"}
      ]'
  spec:
    restartPolicy: Never
    containers:
    - name: dpdk-1g
      image: centos/tools
      imagePullPolicy: IfNotPresent
      volumeMounts:
      - mountPath: /mnt/huge-2048
        name: hugepage
      - name: lib-modules
        mountPath: /lib/modules
      - name: src
        mountPath: /usr/src
      command: ["/bin/bash", "-ec", "sleep infinity"]
      securityContext:
        privileged: true
        capabilities:
          add:
          - ALL
      resources:
        requests:
          memory: 4Gi
          hugepages-1Gi: 4Gi
          intel.com/intel_sriov_dpdk: '2'
        limits:
          memory: 4Gi
          hugepages-1Gi: 4Gi
          intel.com/intel_sriov_dpdk: '2'
    imagePullSecrets:
    - name: admin-registry-secret
    volumes:
    - name: hugepage
      emptyDir:
        medium: HugePages
    - name: lib-modules
      hostPath:
        path: /lib/modules
    - name: src
      hostPath:
        path: /usr/src
    imagePullSecrets:
    - name: admin-registry-secret
  EOF
  
  root@oran-aio:~/sriov-network-device-plugin# kubectl create -f deployments/sriovdpdk-crd.yaml
  root@oran-aio:~/sriov-network-device-plugin# kubectl create -f deployments/dpdk-1g.yaml
  
  root@oran-aio:~/sriov-network-device-plugin# root@oran-aio:~/sriov-network-device-plugin# kubectl get pods | grep dpdk
  dpdk-1g    1/1     Running   0          13s
  
  root@oran-aio:~/sriov-network-device-plugin# ip link |grep 101
    vf 7 MAC 00:00:00:00:00:00, vlan 101, spoof checking on, link-state auto, trust off, query_rss off
    vf 6 MAC 00:00:00:00:00:00, vlan 101, spoof checking on, link-state auto, trust off, query_rss off


Now test with dpdk

::

  ### build following package and copy to target server: bitbake bison;bitbake kernel-devsrc
  root@oran-aio:~/sriov-network-device-plugin# rpm -ivh ~/bison-3.0.4-r0.corei7_64.rpm
  root@oran-aio:~/sriov-network-device-plugin# rpm -ivh ~/kernel-devsrc-1.0-r0.intel_x86_64.rpm

  root@oran-aio:~/sriov-network-device-plugin# kubectl exec -it $(kubectl get pods -o wide | grep dpdk | awk '{ print $1 }') -- /bin/bash
  [root@dpdk-1g /]# export |grep INTEL
    declare -x PCIDEVICE_INTEL_COM_INTEL_SRIOV_DPDK="0000:04:11.6,0000:04:11.5"
  
  [root@dpdk-1g /]# yum -y install wget ncurses-devel unzip libpcap-devel ncurses-devel libedit-devel pciutils lua-devel
  
  [root@dpdk-1g /]# cd /opt
  [root@dpdk-1g /]# wget https://fast.dpdk.org/rel/dpdk-18.08.tar.xz
  [root@dpdk-1g /]# tar xf dpdk-18.08.tar.xz
  [root@dpdk-1g /]# cd dpdk-18.08/
  [root@dpdk-1g /]# sed -i 's/CONFIG_RTE_EAL_IGB_UIO=y/CONFIG_RTE_EAL_IGB_UIO=n/g' config/common_linuxapp
  [root@dpdk-1g /]# sed -i 's/CONFIG_RTE_LIBRTE_KNI=y/CONFIG_RTE_LIBRTE_KNI=n/g' config/common_linuxapp
  [root@dpdk-1g /]# sed -i 's/CONFIG_RTE_KNI_KMOD=y/CONFIG_RTE_KNI_KMOD=n/g' config/common_linuxapp
  [root@dpdk-1g /]# export RTE_SDK=/opt/dpdk-18.08
  [root@dpdk-1g /]# export RTE_TARGET=x86_64-native-linuxapp-gcc
  [root@dpdk-1g /]# export RTE_BIND=$RTE_SDK/usertools/dpdk-devbind.py
  [root@dpdk-1g /]# make install T=$RTE_TARGET
  [root@dpdk-1g /]# cd examples/helloworld
  [root@dpdk-1g /]# make
  [root@dpdk-1g /]# NR_hugepages=2
  [root@dpdk-1g /]# ./build/helloworld -l 1-4 -n 4 -m $NR_hugepages
  ...
      hello from core 2
      hello from core 3
      hello from core 4
      hello from core 1



3.12 Deploy CMK (CPU-Manager-for-Kubernetes)
''''''''''''''''''''''''''''''''''''''''''''

Build the CMK docker image

::

  root@oran-aio:~# cd /opt/kubernetes_plugins/cpu-manager-for-kubernetes/
  root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# make

Verify that the cmk docker images is built successfully

::

  root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# docker images|grep cmk
  cmk          v1.3.1              3fec5f753b05        44 minutes ago      765MB

Edit the template yaml file for your deployment:
  - The template file is: /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-cluster-init-pod-template.yaml
  - The options you may need to change:

::

  # You can change the value for the following env:
  env:
  - name: HOST_LIST
    # Change this to modify the the host list to be initialized
    value: "oran-aio"
  - name: NUM_EXCLUSIVE_CORES
    # Change this to modify the value passed to `--num-exclusive-cores` flag
    value: "4"
  - name: NUM_SHARED_CORES
    # Change this to modify the value passed to `--num-shared-cores` flag
    value: "1"
  - name: CMK_IMG
    # Change his ONLY if you built the docker images with a different tag name
    value: "cmk:v1.3.1"

Or you can also refer to `CMK operator manual`_

.. _`CMK operator manual`: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/master/docs/operator.md


Depoly CMK from yaml files

::

  root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-rbac-rules.yaml
  root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-serviceaccount.yaml
  root@oran-aio:~# kubectl apply -f /etc/kubernetes/plugins/cpu-manager-for-kubernetes/cmk-cluster-init-pod-template.yaml

Verify that the cmk cluster init completed and the pods for nodereport and webhook deployment are up and running

::

  root@oran-aio:/opt/kubernetes_plugins/cpu-manager-for-kubernetes# kubectl get pods --all-namespaces |grep cmk
  default       cmk-cluster-init-pod                         0/1     Completed   0          11m
  default       cmk-init-install-discover-pod-oran-aio       0/2     Completed   0          10m
  default       cmk-reconcile-nodereport-ds-oran-aio-qbdqb   2/2     Running     0          10m
  default       cmk-webhook-deployment-6f9dd7dfb6-2lj2p      1/1     Running     0          10m

- For detail usage, please refer to `CMK user manual`_

.. _`CMK user manual`: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/master/docs/user.md

References
----------

- `Flannel`_
- `Doc for dashboard`_
- `Multus-CNI quick start`_
- `CMK operator manual`_
- `CMK user manual`_

.. _`Flannel`: https://github.com/coreos/flannel/blob/master/README.md