Gitiles
Code Review Sign In
gerrit.nordix.org / oransc / pti / rtp / 6a5f10c55b8883a86e05b101c57713b1f9ce238b / . / docs / installation-guide.rst
blob: 7b6bc22878b9f1ba40466ddd427581649b24c616 [file] [log] [blame]
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. SPDX-License-Identifier: CC-BY-4.0
.. Copyright (C) 2019 Wind River Systems, Inc.
Installation Guide
==================
.. contents::
:depth: 3
:local:
Abstract
--------
This document describes how to install O-RAN INF image, example configuration for better
real time performance, and example deployment of Kubernetes cluster and plugins.
The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
and container technology.
Preface
-------
Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
Hardware Requirements
---------------------
Following minimum hardware requirements must be met for installation of O-RAN INF image:
+--------------------+----------------------------------------------------+
| **HW Aspect** | **Requirement** |
| | |
+--------------------+----------------------------------------------------+
| **# of servers** | 1 |
+--------------------+----------------------------------------------------+
| **CPU** | 4 |
| | |
+--------------------+----------------------------------------------------+
| **RAM** | 16G |
| | |
+--------------------+----------------------------------------------------+
| **Disk** | 500G * 2 |
| | |
+--------------------+----------------------------------------------------+
| **NICs** | 2 |
| | |
+--------------------+----------------------------------------------------+
If want to enable the HA mode which is available in Cherry release, 2 servers are needed.
ORAN INF Cherry test on Dell R740
=================================
1. Installation for the first server from the O-RAN INF ISO image
-----------------------------------------------------------------
- Please see the README.md file for how to build the image.
- The Image is a live ISO image with CLI installer:
inf-image-aio-installer-intel-corei7-64.iso
1.1 Burn the image to the USB device
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Assume the the usb device is /dev/sdX here
::
$ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
1.2 Install the first Dell-R740 server (controller-0)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Reboot the target from the USB device.
- Select “All-in-one Graphics console” or “All-in-one Serial console
install” and press ENTER
- Start the auto installation
- It will reboot aotumatically after installation
2. Configuration and initialize the bootstrap
---------------------------------------------
2.1 First Login with sysadmin/sysadmin and change password
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2.2 Set OAM network before bootstrap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
export OAM_DEV=eno3
export CONTROLLER0_OAM_CIDR=128.224.210.110/24
export DEFAULT_OAM_GATEWAY=128.224.210.1
sudo ip address add $CONTROLLER0_OAM_CIDR dev $OAM_DEV
sudo ip link set up dev $OAM_DEV
sudo ip route add default via $DEFAULT_OAM_GATEWAY dev $OAM_DEV
2.3 Login the server through SSH with sysadmin
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2.4 Prepare the localhost.yml for bootstrap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
cat << EOF > localhost.yml
system_mode: duplex
management_subnet: 192.168.18.0/24
management_start_address: 192.168.18.2
management_end_address: 192.168.18.50
management_gateway_address: 192.168.18.1
external_oam_subnet: 128.224.210.0/24
external_oam_gateway_address: 128.224.210.1
external_oam_floating_address: 128.224.210.110
external_oam_node_0_address: 128.224.210.111
external_oam_node_1_address: 128.224.210.112
EOF
2.5 Run the ansible bootstrap
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
After the bootstrap successfully finish, it will show as following:
::
PLAY RECAP *************************************************************************************************************
localhost : ok=257 changed=151 unreachable=0 failed=0 skipped=214 rescued=0 ignored=0
2.6 Congiure controller-0
~~~~~~~~~~~~~~~~~~~~~~~~~
Acquire admin credentials:
::
controller-0:~$ source /etc/platform/openrc
[sysadmin@controller-0 ~(keystone_admin)]$
Configure the OAM and MGMT interfaces of controller-0 and specify the
attached networks:
::
OAM_IF=eno3
MGMT_IF=eno1
system host-if-modify controller-0 lo -c none
IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
for UUID in $IFNET_UUIDS; do
system interface-network-remove ${UUID}
done
system host-if-modify controller-0 $OAM_IF -n oam0
system host-if-modify controller-0 $MGMT_IF -n pxeboot0
system host-if-modify controller-0 oam0 -c platform
system interface-network-assign controller-0 oam0 oam
system host-if-modify controller-0 pxeboot0 -c platform
system interface-network-assign controller-0 pxeboot0 pxeboot
system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
system interface-network-assign controller-0 mgmt0 mgmt
system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
system interface-network-assign controller-0 cluster0 cluster-host
Example output:
::
[sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
[sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 lo -c none
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | lo |
| iftype | virtual |
| ports | [] |
| imac | 00:00:00:00:00:00 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 08c95952-892b-40b5-b17a-7d2ad46e725c |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2020-11-17T00:30:45.265032+00:00 |
| updated_at | 2020-11-17T01:03:39.031612+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
[sysadmin@controller-0 ~(keystone_admin)]$ for UUID in $IFNET_UUIDS; do
> system interface-network-remove ${UUID}
> done
Deleted Interface Network: 0bf11f1b-4fc6-4e97-b896-3d6393a3744e
Deleted Interface Network: a62d95f6-ad4e-4779-bfc0-6a885067f8d8
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -n oam0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | oam0 |
| iftype | ethernet |
| ports | [u'eno3'] |
| imac | 24:6e:96:5d:0c:b2 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2020-11-17T00:28:32.365863+00:00 |
| updated_at | 2020-11-17T01:03:45.090904+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $MGMT_IF -n pxeboot0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | pxeboot0 |
| iftype | ethernet |
| ports | [u'eno1'] |
| imac | 24:6e:96:5d:0c:92 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2020-11-17T00:28:32.612230+00:00 |
| updated_at | 2020-11-17T01:03:47.341003+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 oam0 -c platform
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | oam0 |
| iftype | ethernet |
| ports | [u'eno3'] |
| imac | 24:6e:96:5d:0c:b2 |
| imtu | 1500 |
| ifclass | platform |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2020-11-17T00:28:32.365863+00:00 |
| updated_at | 2020-11-17T01:03:49.368879+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 oam0 oam
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-0 |
| uuid | 3c8bd181-d3f3-4e14-8e89-75a3432db1e4 |
| ifname | oam0 |
| network_name | oam |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 pxeboot0 -c platform
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | pxeboot0 |
| iftype | ethernet |
| ports | [u'eno1'] |
| imac | 24:6e:96:5d:0c:92 |
| imtu | 1500 |
| ifclass | platform |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2020-11-17T00:28:32.612230+00:00 |
| updated_at | 2020-11-17T01:03:53.143795+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 pxeboot0 pxeboot
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-0 |
| uuid | 6c55622d-2da4-4f4e-ab5e-f8e06e03af7c |
| ifname | pxeboot0 |
| network_name | pxeboot |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | mgmt0 |
| iftype | vlan |
| ports | [] |
| imac | 24:6e:96:5d:0c:92 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | 18 |
| uses | [u'pxeboot0'] |
| used_by | [] |
| created_at | 2020-11-17T01:03:57.303000+00:00 |
| updated_at | None |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 mgmt0 mgmt
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-0 |
| uuid | 2e93ef03-e9ee-457a-8667-05b52b7109a5 |
| ifname | mgmt0 |
| network_name | mgmt |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | cluster0 |
| iftype | vlan |
| ports | [] |
| imac | 24:6e:96:5d:0c:92 |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vlan_id | 19 |
| uses | [u'pxeboot0'] |
| used_by | [] |
| created_at | 2020-11-17T01:04:02.613518+00:00 |
| updated_at | None |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 cluster0 cluster-host
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-0 |
| uuid | fb8b6be6-1618-4662-b063-b1e8d340aa48 |
| ifname | cluster0 |
| network_name | cluster-host |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
| 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
| 23b5e923-1e53-4e70-a975-542d8380b7f2 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
| 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
| d8a048fa-67ef-43ac-8166-671be93caa30 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
Configure NTP Servers for network time synchronization:
::
system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
Output
::
[sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| uuid | 3206cf01-c64a-457e-ac66-b8224c9684c3 |
| ntpservers | 0.pool.ntp.org,1.pool.ntp.org |
| isystem_uuid | cc79b616-d24e-4432-a953-85c9b242cb3a |
| created_at | 2020-11-17T00:27:23.529571+00:00 |
| updated_at | None |
+--------------+--------------------------------------+
Add an OSD on controller-0 for Ceph:
::
system host-disk-list controller-0
system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
system host-stor-list controller-0
Output
::
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
| uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
| | de | num | type | gib | gib | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
| 8e2a719a-fa5a-4c25-89af-70a23fb7b238 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00c66a07604fa8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
| | | | | 75 | | | | |
| | | | | | | | | |
| 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 00c6b9139b76a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 0053be63c794a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| 4879b381-8e9f-48f3-84e2-f9c6a94bbfe0 | /dev/sdd | 2096 | HDD | 1117. | 0.0 | Undetermined | 0065482503bca8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
+------------------+-------------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------------+
| osdid | 0 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
| journal_size_gib | 1024 |
| journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
| journal_node | /dev/sdb2 |
| uuid | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| idisk_uuid | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 |
| tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
| tier_name | storage |
| created_at | 2020-11-17T01:05:04.063823+00:00 |
| updated_at | None |
+------------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
+------------------+-------------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------------+
| osdid | 1 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
| journal_size_gib | 1024 |
| journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
| journal_node | /dev/sdc2 |
| uuid | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
| ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| idisk_uuid | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 |
| tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
| tier_name | storage |
| created_at | 2020-11-17T01:05:06.939798+00:00 |
| updated_at | None |
+------------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| 0816c72f-a4f0-49ea-9a95-0f02c880717c | osd | 0 | configuring-on-unlock | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
| 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | osd | 1 | configuring-on-unlock | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
2.7 Unlock controller-0
~~~~~~~~~~~~~~~~~~~~~~~
::
system host-unlock controller-0
Output:
::
[sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
+-----------------------+-------------------------------------------------+
| Property | Value |
+-----------------------+-------------------------------------------------+
| action | none |
| administrative | locked |
| availability | online |
| bm_ip | None |
| bm_type | none |
| bm_username | None |
| boot_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
| capabilities | {u'stor_function': u'monitor'} |
| clock_synchronization | ntp |
| config_applied | 6aa15fb4-8cb3-494e-b94e-95f85b560f22 |
| config_status | None |
| config_target | c6ae9b2d-a3c4-4751-a79e-5487ba81ed82 |
| console | ttyS0,115200 |
| created_at | 2020-11-17T00:28:01.983673+00:00 |
| hostname | controller-0 |
| id | 1 |
| install_output | graphical |
| install_state | None |
| install_state_info | None |
| inv_state | inventoried |
| invprovision | provisioning |
| location | {} |
| mgmt_ip | 192.168.18.3 |
| mgmt_mac | 24:6e:96:5d:0c:92 |
| operational | disabled |
| personality | controller |
| reserved | False |
| rootfs_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
| serialid | None |
| software_load | 19.12 |
| subfunction_avail | online |
| subfunction_oper | disabled |
| subfunctions | controller,worker,lowlatency |
| task | Unlocking |
| tboot | false |
| ttys_dcd | None |
| updated_at | 2020-11-17T01:05:07.015414+00:00 |
| uptime | 3496 |
| uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
| vim_progress_status | None |
+-----------------------+-------------------------------------------------+
Controller-0 will reboot to apply configuration changes and come into
service. This can take 5-10 minutes, depending on the performance of the
host machine.
Once the controller comes back up, check the status of controller-0
::
controller-0:~$ source /etc/platform/openrc
[sysadmin@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
2. Installation for the second server (controller-1)
----------------------------------------------------
2.1 Power on the controller-1 server and force it to network boot
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2.2 As controller-1 boots, a message appears on its console instructing you to configure the personality of the node
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2.3 On the console of controller-0, list hosts to see newly discovered controller-1 host (hostname=None)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
[sysadmin@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | degraded |
| 2 | None | None | locked | disabled | offline |
+----+--------------+-------------+----------------+-------------+--------------+
2.4 Using the host id, set the personality of this host to 'controller’:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
[sysadmin@controller-0 ~(keystone_admin)]$ system host-update 2 personality=controller
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| action | none |
| administrative | locked |
| availability | offline |
| bm_ip | None |
| bm_type | None |
| bm_username | None |
| boot_device | /dev/sda |
| capabilities | {} |
| clock_synchronization | ntp |
| config_applied | None |
| config_status | None |
| config_target | None |
| console | ttyS0,115200 |
| created_at | 2020-11-17T10:17:44.387813+00:00 |
| hostname | controller-1 |
| id | 2 |
| install_output | text |
| install_state | None |
| install_state_info | None |
| inv_state | None |
| invprovision | None |
| location | {} |
| mgmt_ip | 192.168.18.4 |
| mgmt_mac | 24:6e:96:5d:38:ee |
| operational | disabled |
| personality | controller |
| reserved | False |
| rootfs_device | /dev/sda |
| serialid | None |
| software_load | 19.12 |
| subfunction_avail | not-installed |
| subfunction_oper | disabled |
| subfunctions | controller,worker,lowlatency |
| task | None |
| tboot | false |
| ttys_dcd | None |
| updated_at | None |
| uptime | 0 |
| uuid | f069381d-9743-49cc-bf8b-eb4bd3972203 |
| vim_progress_status | None |
+-----------------------+--------------------------------------+
2.5 Wait for the software installation on controller-1 to complete, for controller-1 to reboot, and for controller-1 to show as locked/disabled/online in 'system host-list'.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This can take 5-10 minutes, depending on the performance of the host
machine.
::
[root@controller-0 hieradata(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | locked | disabled | online |
+----+--------------+-------------+----------------+-------------+--------------+
2.6 Configure controller-1
~~~~~~~~~~~~~~~~~~~~~~~~~~
::
OAM_IF=eno3
MGMT_IF=eno1
system host-if-modify controller-1 $OAM_IF -n oam0
system host-if-modify controller-1 oam0 -c platform
system interface-network-assign controller-1 oam0 oam
system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
system interface-network-assign controller-1 cluster0 cluster-host
system host-if-list controller-1
system host-disk-list controller-1
system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
system host-stor-list controller-1
Output:
::
[sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
[sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 $OAM_IF -n oam0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | oam0 |
| iftype | ethernet |
| ports | [u'eno3'] |
| imac | 24:6e:96:5d:39:0e |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2020-11-22T14:01:32.365863+00:00 |
| updated_at | 2020-11-22T15:04:45.090904+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 oam0 -c platform
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | oam0 |
| iftype | ethernet |
| ports | [u'eno3'] |
| imac | 24:6e:96:5d:39:0e |
| imtu | 1500 |
| ifclass | platform |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vlan_id | None |
| uses | [] |
| used_by | [] |
| created_at | 2020-11-22T14:05:16.052229+00:00 |
| updated_at | 2020-11-22T15:08:35.324634+00:00 |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 oam0 oam
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-1 |
| uuid | f2e7f088-0dd0-4adc-8348-4e3cef23bc47 |
| ifname | oam0 |
| network_name | oam |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | cluster0 |
| iftype | vlan |
| ports | [] |
| imac | 24:6e:96:5d:38:ee |
| imtu | 1500 |
| ifclass | None |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | b6783682-b2aa-4135-90d2-676e1db41ae8 |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vlan_id | 19 |
| uses | [u'pxeboot0'] |
| used_by | [] |
| created_at | 2020-11-22T15:08:43.932209+00:00 |
| updated_at | None |
| sriov_numvfs | 0 |
| sriov_vf_driver | None |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 cluster0 cluster-host
+--------------+--------------------------------------+
| Property | Value |
+--------------+--------------------------------------+
| hostname | controller-1 |
| uuid | 8fc64805-b54b-45a4-b88a-e13b236abfe8 |
| ifname | cluster0 |
| network_name | cluster-host |
+--------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
| b6783682-b2aa-4135-90d2-676e1db41ae8 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
| b8921960-fde5-44c3-960d-2aebf42ea400 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
| c103275b-2b75-4568-865f-ac6be32ecb2d | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
| c2473511-d0d6-445d-9739-4d43dc029de9 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
+--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
| uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
| | de | num | type | gib | gib | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
| 5b8fade4-b048-48fa-b906-9dcbdbed8e96 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00cbd97f3e36ccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
| | | | | 75 | | | | |
| | | | | | | | | |
| 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 006d0e977b5fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 003a2377ac7fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| 774c3cd0-1178-4145-9573-f0d6dee2ba06 | /dev/sdd | 2096 | HDD | 1117. | 1117.247 | Undetermined | 00d7093ef0adccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
| | | | | 25 | | | | |
| | | | | | | | | |
| 00361302-8d55-4730-855c-b0098c73ab7e | /dev/sde | 2112 | SSD | 223. | 223.568 | N/A | PHDW730104QM240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-1 |
| | | | | 57 | | | | |
| | | | | | | | | |
| 7ce735e6-920f-4424-a890-a7a7f48d7632 | /dev/sdf | 2128 | SSD | 223. | 223.568 | N/A | PHDW730104LL240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-2 |
| | | | | 57 | | | | |
| | | | | | | | | |
+--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
+------------------+-------------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------------+
| osdid | 2 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
| journal_size_gib | 1024 |
| journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
| journal_node | /dev/sdb2 |
| uuid | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| idisk_uuid | 1a3f0a36-5961-42e5-a271-e71db1c25d42 |
| tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
| tier_name | storage |
| created_at | 2020-11-22T15:11:55.641193+00:00 |
| updated_at | None |
+------------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
+------------------+-------------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------------+
| osdid | 3 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
| journal_size_gib | 1024 |
| journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
| journal_node | /dev/sdc2 |
| uuid | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
| ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| idisk_uuid | eddd732f-2cea-49b3-86db-b722c0b1a1ae |
| tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
| tier_name | storage |
| created_at | 2020-11-22T15:12:04.274839+00:00 |
| updated_at | None |
+------------------+-------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-1
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
| 54a218d8-0466-4366-9ef0-3ec5a952fde7 | osd | 2 | configuring-on-unlock | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
| 5be88c7a-3a94-4b97-9da5-b247bb89406c | osd | 3 | configuring-on-unlock | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
+--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
2.7 Unlock controller-1
~~~~~~~~~~~~~~~~~~~~~~~
Unlock controller-1 in order to bring it into service:
::
[sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-1
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| action | none |
| administrative | locked |
| availability | online |
| bm_ip | None |
| bm_type | None |
| bm_username | None |
| boot_device | /dev/sda |
| capabilities | {u'stor_function': u'monitor'} |
| clock_synchronization | ntp |
| config_applied | None |
| config_status | Config out-of-date |
| config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
| console | ttyS0,115200 |
| created_at | 2020-11-22T12:58:11.630526+00:00 |
| hostname | controller-1 |
| id | 3 |
| install_output | text |
| install_state | None |
| install_state_info | None |
| inv_state | inventoried |
| invprovision | unprovisioned |
| location | {} |
| mgmt_ip | 192.168.18.4 |
| mgmt_mac | 24:6e:96:5d:38:ee |
| operational | disabled |
| personality | controller |
| reserved | False |
| rootfs_device | /dev/sda |
| serialid | None |
| software_load | 19.12 |
| subfunction_avail | online |
| subfunction_oper | disabled |
| subfunctions | controller,worker,lowlatency |
| task | Unlocking |
| tboot | false |
| ttys_dcd | None |
| updated_at | 2020-11-22T15:13:09.716324+00:00 |
| uptime | 752 |
| uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vim_progress_status | None |
+-----------------------+--------------------------------------+
Controller-1 will reboot in order to apply configuration changes and
come into service. This can take 5-10 minutes, depending on the
performance of the host machine.
::
[root@controller-0 hieradata(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-show controller-1
+-----------------------+-----------------------------------------------------------------------+
| Property | Value |
+-----------------------+-----------------------------------------------------------------------+
| action | none |
| administrative | unlocked |
| availability | available |
| bm_ip | None |
| bm_type | None |
| bm_username | None |
| boot_device | /dev/sda |
| capabilities | {u'stor_function': u'monitor', u'Personality': u'Controller-Standby'} |
| clock_synchronization | ntp |
| config_applied | 9747e0ce-2319-409d-b75c-2475bc5065ac |
| config_status | None |
| config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
| console | ttyS0,115200 |
| created_at | 2020-11-22T12:58:11.630526+00:00 |
| hostname | controller-1 |
| id | 2 |
| install_output | text |
| install_state | None |
| install_state_info | None |
| inv_state | inventoried |
| invprovision | provisioned |
| location | {} |
| mgmt_ip | 192.168.18.4 |
| mgmt_mac | 24:6e:96:5d:38:ee |
| operational | enabled |
| personality | controller |
| reserved | False |
| rootfs_device | /dev/sda |
| serialid | None |
| software_load | 19.12 |
| subfunction_avail | available |
| subfunction_oper | enabled |
| subfunctions | controller,worker,lowlatency |
| task | |
| tboot | false |
| ttys_dcd | None |
| updated_at | 2020-11-22T23:59:07.787759+00:00 |
| uptime | 31008 |
| uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
| vim_progress_status | services-enabled |
+-----------------------+-----------------------------------------------------------------------+
3. Simple use case for sriov
````````````````````````````
3.1 After controller-0 is rebooted and up running, download the DPDK
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
::
[sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
[sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
Password:
--2020-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10251680 (9.8M) [application/octet-stream]
Saving to: ‘dpdk-17.11.10.tar.xz’
dpdk-17.11.10.tar.xz 100%
[========================================================================================>] 9.78M
1.48MB/s in 6.8s
2020-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
sudo tar xvf dpdk-17.11.10.tar.xz
sudo ln -s dpdk-stable-17.11.10 dpdk-stable
3.2 Prepare the yaml file for the network assignment container
''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
The following the exmaple of the yaml file:
::
[sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
> apiVersion: "k8s.cni.cncf.io/v1"
> kind: NetworkAttachmentDefinition
> metadata:
> name: sriov-data-dpdk-0
> annotations:
> k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
> spec:
> config: '{
> "type": "sriov",
> "name": "sriov-data-dpdk-0"
> }'
>
> ---
> apiVersion: "k8s.cni.cncf.io/v1"
> kind: NetworkAttachmentDefinition
> metadata:
> name: sriov-data-dpdk-1
> annotations:
> k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
> spec:
> config: '{
> "type": "sriov",
> "name": "sriov-data-dpdk-1"
> }'
> EOF
3.3 Run the network assignent container for the 2 VFs
'''''''''''''''''''''''''''''''''''''''''''''''''''''
::
[sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
3.4 Prepare the VF container yaml file
''''''''''''''''''''''''''''''''''''''
::
[sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
> apiVersion: v1
> kind: Pod
metadata:
> metadata:
> name: pod-with-dpdk-vfs-0
> annotations:
> k8s.v1.cni.cncf.io/networks: '[
> { "name": "sriov-data-dpdk-0" },
{ "name": "sriov-data-dpdk-1" }
> { "name": "sriov-data-dpdk-1" }
> ]'
> spec:
> restartPolicy: Never
> containers:
> - name: pod-with-dpdk-vfs-0
> image: wrsnfv/ubuntu-dpdk-build:v0.3
> env:
> - name: RTE_SDK
> value: "/usr/src/dpdk"
> command:
> - sleep
> - infinity
> stdin: true
> tty: true
> securityContext:
> privileged: true
> capabilities:
> add:
> - ALL
> resources:
> requests:
> cpu: 4
> memory: 4Gi
> intel.com/pci_sriov_net_physnet0: '1'
> intel.com/pci_sriov_net_physnet1: '1'
> limits:
> cpu: 4
> hugepages-1Gi: 2Gi
> memory: 4Gi
> intel.com/pci_sriov_net_physnet0: '1'
> intel.com/pci_sriov_net_physnet1: '1'
> volumeMounts:
> - mountPath: /mnt/huge-1048576kB
> name: hugepage
> - name: dpdk-volume
> mountPath: /usr/src/dpdk
> - name: lib-volume
> mountPath: /lib/modules
> - name: src-volume
> mountPath: /usr/src/
> volumes:
> - name: hugepage
> emptyDir:
> medium: HugePages
> - name: dpdk-volume
> hostPath:
> path: /opt/dpdk-stable/
> - name: lib-volume
> hostPath:
> path: /lib/modules
> - name: src-volume
> hostPath:
> path: /usr/src/
> EOF
3.5 Run the VF container
''''''''''''''''''''''''
Start the VF container.
::
[sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
pod/pod-with-dpdk-vfs-0 created
[sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
NAME READY STATUS RESTARTS AGE
pod-with-dpdk-vfs-0 1/1 Running 0 6m40s
Login the VF container
::
kubectl exec -it pod-with-dpdk-vfs-0 -- bash
Build the DPDK
::
cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
HOSTCC scripts/kconfig/confdata.o
HOSTCC scripts/kconfig/expr.o
HOSTCC scripts/kconfig/symbol.o
HOSTCC scripts/kconfig/preprocess.o
HOSTCC scripts/kconfig/zconf.lex.o
HOSTCC scripts/kconfig/zconf.tab.o
HOSTLD scripts/kconfig/conf
scripts/kconfig/conf --syncconfig Kconfig
HOSTCC arch/x86/tools/relocs_32.o
HOSTCC arch/x86/tools/relocs_64.o
HOSTCC arch/x86/tools/relocs_common.o
HOSTLD arch/x86/tools/relocs
HOSTCC scripts/genksyms/genksyms.o
YACC scripts/genksyms/parse.tab.c
HOSTCC scripts/genksyms/parse.tab.o
LEX scripts/genksyms/lex.lex.c
YACC scripts/genksyms/parse.tab.h
HOSTCC scripts/genksyms/lex.lex.o
HOSTLD scripts/genksyms/genksyms
HOSTCC scripts/bin2c
HOSTCC scripts/kallsyms
HOSTCC scripts/conmakehash
HOSTCC scripts/recordmcount
HOSTCC scripts/sortextable
HOSTCC scripts/asn1_compiler
HOSTCC scripts/sign-file
HOSTCC scripts/extract-cert
CC scripts/mod/empty.o
HOSTCC scripts/mod/mk_elfconfig
MKELF scripts/mod/elfconfig.h
HOSTCC scripts/mod/modpost.o
CC scripts/mod/devicetable-offsets.s
UPD scripts/mod/devicetable-offsets.h
HOSTCC scripts/mod/file2alias.o
HOSTCC scripts/mod/sumversion.o
HOSTLD scripts/mod/modpost
CC kernel/bounds.s
CC arch/x86/kernel/asm-offsets.s
CALL scripts/checksyscalls.sh
Build the test_pmd application
::
cd $RTE_SDK
./usertools/dpdk-setup.sh
Option: 14
CC config.o
CC iofwd.o
CC macfwd.o
CC macswap.o
CC flowgen.o
CC rxonly.o
CC txonly.o
CC csumonly.o
CC icmpecho.o
CC tm.o
LD testpmd
INSTALL-APP testpmd
INSTALL-MAP testpmd.map
== Build app/proc_info
CC main.o
LD dpdk-procinfo
INSTALL-APP dpdk-procinfo
INSTALL-MAP dpdk-procinfo.map
== Build app/pdump
CC main.o
LD dpdk-pdump
INSTALL-APP dpdk-pdump
INSTALL-MAP dpdk-pdump.map
== Build app/test-crypto-perf
CC main.o
CC cperf_ops.o
CC cperf_options_parsing.o
CC cperf_test_vectors.o
CC cperf_test_throughput.o
CC cperf_test_latency.o
CC cperf_test_pmd_cyclecount.o
CC cperf_test_verify.o
CC cperf_test_vector_parsing.o
CC cperf_test_common.o
LD dpdk-test-crypto-perf
INSTALL-APP dpdk-test-crypto-perf
INSTALL-MAP dpdk-test-crypto-perf.map
== Build app/test-eventdev
CC evt_main.o
CC evt_options.o
CC evt_test.o
CC parser.o
CC test_order_common.o
CC test_order_queue.o
CC test_order_atq.o
CC test_perf_common.o
CC test_perf_queue.o
CC test_perf_atq.o
LD dpdk-test-eventdev
INSTALL-APP dpdk-test-eventdev
INSTALL-MAP dpdk-test-eventdev.map
Build complete [x86_64-native-linuxapp-gcc]
Installation cannot run with T defined and DESTDIR undefined
------------------------------------------------------------------------------
RTE_TARGET exported as x86_64-native-linuxapp-gcc
------------------------------------------------------------------------------
Press enter to continue ...
Check the VF PCI information:
::
root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
::
[root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
[root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
[root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
[root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
Get back to pod and run the test_pmd
::
cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
taskset -p --cpu-list 1
pid 1's current affinity list: 2,3,18,19
./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
peer=1,"9e:fd:e6:dd:c1:01"
EAL: Detected 32 lcore(s)
EAL: No free hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:05:11.0 on NUMA socket 0
EAL: probe driver: 8086:10ed net_ixgbe_vf
EAL: using IOMMU type 1 (Type 1)
EAL: PCI device 0000:05:11.1 on NUMA socket 0
EAL: probe driver: 8086:10ed net_ixgbe_vf
Auto-start selected
Ports to start sending a burst of packets first
Warning: lsc_interrupt needs to be off when using tx_first. Disabling.
USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
Configuring Port 0 (socket 0)
Port 0: 9E:FD:E6:DD:C1:01
Configuring Port 1 (socket 0)
Port 1: 9E:FD:E6:DD:C1:02
Checking link statuses...
Port0 Link Up. speed 10000 Mbps- full-duplex
Port1 Link Up. speed 10000 Mbps- full-duplex
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
disabled
Logical Core 3 (socket 0) forwards packets on 2 streams:
RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
io packet forwarding packets/burst=32
nb forwarding cores=1 - nb forwarding ports=2
port 0:
CRC stripping enabled
RX queues=1 - RX desc=128 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
TX queues=1 - TX desc=512 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX RS bit threshold=32 - TXQ flags=0xf01
port 1:
CRC stripping enabled
RX queues=1 - RX desc=128 - RX free threshold=32
RX threshold registers: pthresh=8 hthresh=8 wthresh=0
TX queues=1 - TX desc=512 - TX free threshold=32
TX threshold registers: pthresh=32 hthresh=0 wthresh=0
TX RS bit threshold=32 - TXQ flags=0xf01
Port statistics ====================================
######################## NIC statistics for port 0 ########################
RX-packets: 56 RX-missed: 0 RX-bytes: 4096
RX-errors: 0
RX-nombuf: 0
TX-packets: 64 TX-errors: 0 TX-bytes: 4096
Throughput (since last show)
Rx-pps: 0
Tx-pps: 0
############################################################################
######################## NIC statistics for port 1 ########################
RX-packets: 432 RX-missed: 0 RX-bytes: 27712
RX-errors: 0
RX-nombuf: 0
TX-packets: 461 TX-errors: 0 TX-bytes: 30080
Throughput (since last show)
Rx-pps: 0
Tx-pps: 0
############################################################################
Port statistics ====================================
######################## NIC statistics for port 0 ########################
RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344
RX-errors: 0
RX-nombuf: 0
TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376
Throughput (since last show)
Rx-pps: 7068409
Tx-pps: 7091206
############################################################################
References
----------
- `StarlingX`_
.. _`StarlingX`: https://docs.starlingx.io/