Gitiles
Code Review Sign In
gerrit.nordix.org / oransc / pti / rtp / b7bf0fac172585c8efb9107fc5d2638373390868 / . / docs / installation-guide.rst
blob: 7356c5a5b87e1a0eac0a9ee883799ce610fe8b24 [file] [log] [blame]
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]1.. This work is licensed under a Creative Commons Attribution 4.0 International License.
2.. SPDX-License-Identifier: CC-BY-4.0
3.. Copyright (C) 2019 Wind River Systems, Inc.
4
5
6Installation Guide
7==================
8
9.. contents::
10 :depth: 3
11 :local:
12
13Abstract
14--------
15
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]16This document describes how to install O-RAN INF image, example configuration (All-in-one Duplex)
17for better real time performance, and example deployment of Kubernetes cluster and plugins.
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]18
19The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
20and container technology.
21
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]22
23Preface
24-------
25
26Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
27
28
29Hardware Requirements
30---------------------
31
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]32Following minimum hardware requirements must be met for installation of O-RAN INF image with AIO-DX:
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]33
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]34+-------------------+----------------------------------------------------------------------------+
35| **HW Aspect** | **Requirement** |
36| | |
37+-------------------+----------------------------------------------------------------------------+
38| **# of servers** | 2 |
39+-------------------+----------------------------------------------------------------------------+
40| **CPU** | Dual-CPU Intel® Xeon® E5 26xx family (SandyBridge) 8 cores/socket |
41| | or |
42| | Single-CPU Intel® Xeon® D-15xx family, 8 cores (low-power/low-cost option) |
43+-------------------+----------------------------------------------------------------------------+
44| **RAM** | 32G |
45| | |
46+-------------------+----------------------------------------------------------------------------+
47| **Disk** | * Disk 1: 500G(It's better to be SSD) |
48| | * Disk 2: 1 or more 500 GB for Ceph OSD |
49+-------------------+----------------------------------------------------------------------------+
50| **NICs** | * OAM: 1x1GE |
51| | * Data: 1 or more x 10GE (optional) |
52+-------------------+----------------------------------------------------------------------------+
53| **BIOS settings** | * Hyper-Threading technology enabled |
54| | * Virtualization technology enabled |
55| | * VT for directed I/O enabled |
56| | * CPU power and performance policy set to performance |
57| | * CPU C state control disabled |
58| | * Plug & play BMC detection disabled |
59+-------------------+----------------------------------------------------------------------------+
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]60
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]61ORAN INF E Release tested on HP ProLiant DL380p Gen8
62====================================================
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]63
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]641. Installation for the first server from the O-RAN INF ISO image
65-----------------------------------------------------------------
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]66
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]67- Please see the README.md file for how to build the image.
68- The Image is a live ISO image with CLI installer:
69 inf-image-aio-installer-intel-corei7-64.iso
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]70
711.1 Burn the image to the USB device
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]72~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]73
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]74- Assume the the usb device is /dev/sdX here
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]75
76::
77
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]78 $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]79
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]801.2 Install the first server (controller-0)
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]81~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]82
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]83- Reboot the target from the USB device.
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]84
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]85- Select “All-in-one Graphics console” or “All-in-one Serial console
86 install” and press ENTER
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]87
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]88- Start the auto installation
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]89
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]90- It will reboot aotumatically after installation
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]91
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]922. Configuration and initialize the bootstrap
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]93---------------------------------------------
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]94
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]952.1 First Login with sysadmin/sysadmin and change password
96~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
97
982.2 Set OAM network before bootstrap
99~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]100
101::
102
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]103 export OAM_DEV=eno3
104 export CONTROLLER0_OAM_CIDR=128.224.210.110/24
105 export DEFAULT_OAM_GATEWAY=128.224.210.1
106 sudo ip address add $CONTROLLER0_OAM_CIDR dev $OAM_DEV
107 sudo ip link set up dev $OAM_DEV
108 sudo ip route add default via $DEFAULT_OAM_GATEWAY dev $OAM_DEV
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]109
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]1102.3 Login the server through SSH with sysadmin
111~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]112
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]1132.4 Prepare the localhost.yml for bootstrap
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]114~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]115
116::
117
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]118 cat << EOF > localhost.yml
119 system_mode: duplex
120 management_subnet: 192.168.18.0/24
121 management_start_address: 192.168.18.2
122 management_end_address: 192.168.18.50
123 management_gateway_address: 192.168.18.1
124 external_oam_subnet: 128.224.210.0/24
125 external_oam_gateway_address: 128.224.210.1
126 external_oam_floating_address: 128.224.210.110
127 external_oam_node_0_address: 128.224.210.111
128 external_oam_node_1_address: 128.224.210.112
129 EOF
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]130
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]1312.5 Run the ansible bootstrap
132~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]133
134::
135
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]136 ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]137
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]138After the bootstrap successfully finish, it will show as following:
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]139
140::
141
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]142 PLAY RECAP *************************************************************************************************************
143 localhost : ok=257 changed=151 unreachable=0 failed=0 skipped=214 rescued=0 ignored=0
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]144
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]1452.6 Congiure controller-0
146~~~~~~~~~~~~~~~~~~~~~~~~~
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]147
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]148Acquire admin credentials:
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]149
150::
151
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]152 controller-0:~$ source /etc/platform/openrc
153 [sysadmin@controller-0 ~(keystone_admin)]$
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]154
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]155Configure the OAM and MGMT interfaces of controller-0 and specify the
156attached networks:
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]157
158::
159
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]160 OAM_IF=eno3
161 MGMT_IF=eno1
162 system host-if-modify controller-0 lo -c none
163 IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
164 for UUID in $IFNET_UUIDS; do
165 system interface-network-remove ${UUID}
166 done
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame]167
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]168 system host-if-modify controller-0 $OAM_IF -n oam0
169 system host-if-modify controller-0 $MGMT_IF -n pxeboot0
170
171 system host-if-modify controller-0 oam0 -c platform
172 system interface-network-assign controller-0 oam0 oam
173
174 system host-if-modify controller-0 pxeboot0 -c platform
175 system interface-network-assign controller-0 pxeboot0 pxeboot
176
177 system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
178 system interface-network-assign controller-0 mgmt0 mgmt
179
180 system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
181 system interface-network-assign controller-0 cluster0 cluster-host
182
183Example output:
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]184
185::
186
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]187 [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
188 [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
189 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 lo -c none
190 +-----------------+--------------------------------------+
191 | Property | Value |
192 +-----------------+--------------------------------------+
193 | ifname | lo |
194 | iftype | virtual |
195 | ports | [] |
196 | imac | 00:00:00:00:00:00 |
197 | imtu | 1500 |
198 | ifclass | None |
199 | aemode | None |
200 | schedpolicy | None |
201 | txhashpolicy | None |
202 | uuid | 08c95952-892b-40b5-b17a-7d2ad46e725c |
203 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
204 | vlan_id | None |
205 | uses | [] |
206 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]207 | created_at | 2021-11-17T00:30:45.265032+00:00 |
208 | updated_at | 2021-11-17T01:03:39.031612+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]209 | sriov_numvfs | 0 |
210 | sriov_vf_driver | None |
211 +-----------------+--------------------------------------+
212 [sysadmin@controller-0 ~(keystone_admin)]$ IFNET_UUIDS=$(system interface-network-list controller-0 | awk '{if ($6=="lo") print $4;}')
213 [sysadmin@controller-0 ~(keystone_admin)]$ for UUID in $IFNET_UUIDS; do
214 > system interface-network-remove ${UUID}
215 > done
216 Deleted Interface Network: 0bf11f1b-4fc6-4e97-b896-3d6393a3744e
217 Deleted Interface Network: a62d95f6-ad4e-4779-bfc0-6a885067f8d8
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]218
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]219 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -n oam0
220 +-----------------+--------------------------------------+
221 | Property | Value |
222 +-----------------+--------------------------------------+
223 | ifname | oam0 |
224 | iftype | ethernet |
225 | ports | [u'eno3'] |
226 | imac | 24:6e:96:5d:0c:b2 |
227 | imtu | 1500 |
228 | ifclass | None |
229 | aemode | None |
230 | schedpolicy | None |
231 | txhashpolicy | None |
232 | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
233 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
234 | vlan_id | None |
235 | uses | [] |
236 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]237 | created_at | 2021-11-17T00:28:32.365863+00:00 |
238 | updated_at | 2021-11-17T01:03:45.090904+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]239 | sriov_numvfs | 0 |
240 | sriov_vf_driver | None |
241 | accelerated | [True] |
242 +-----------------+--------------------------------------+
243 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $MGMT_IF -n pxeboot0
244 +-----------------+--------------------------------------+
245 | Property | Value |
246 +-----------------+--------------------------------------+
247 | ifname | pxeboot0 |
248 | iftype | ethernet |
249 | ports | [u'eno1'] |
250 | imac | 24:6e:96:5d:0c:92 |
251 | imtu | 1500 |
252 | ifclass | None |
253 | aemode | None |
254 | schedpolicy | None |
255 | txhashpolicy | None |
256 | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
257 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
258 | vlan_id | None |
259 | uses | [] |
260 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]261 | created_at | 2021-11-17T00:28:32.612230+00:00 |
262 | updated_at | 2021-11-17T01:03:47.341003+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]263 | sriov_numvfs | 0 |
264 | sriov_vf_driver | None |
265 | accelerated | [True] |
266 +-----------------+--------------------------------------+
267 [sysadmin@controller-0 ~(keystone_admin)]$
268 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 oam0 -c platform
269 +-----------------+--------------------------------------+
270 | Property | Value |
271 +-----------------+--------------------------------------+
272 | ifname | oam0 |
273 | iftype | ethernet |
274 | ports | [u'eno3'] |
275 | imac | 24:6e:96:5d:0c:b2 |
276 | imtu | 1500 |
277 | ifclass | platform |
278 | aemode | None |
279 | schedpolicy | None |
280 | txhashpolicy | None |
281 | uuid | d8a048fa-67ef-43ac-8166-671be93caa30 |
282 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
283 | vlan_id | None |
284 | uses | [] |
285 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]286 | created_at | 2021-11-17T00:28:32.365863+00:00 |
287 | updated_at | 2021-11-17T01:03:49.368879+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]288 | sriov_numvfs | 0 |
289 | sriov_vf_driver | None |
290 | accelerated | [True] |
291 +-----------------+--------------------------------------+
292 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 oam0 oam
293 +--------------+--------------------------------------+
294 | Property | Value |
295 +--------------+--------------------------------------+
296 | hostname | controller-0 |
297 | uuid | 3c8bd181-d3f3-4e14-8e89-75a3432db1e4 |
298 | ifname | oam0 |
299 | network_name | oam |
300 +--------------+--------------------------------------+
301 [sysadmin@controller-0 ~(keystone_admin)]$
302 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 pxeboot0 -c platform
303 +-----------------+--------------------------------------+
304 | Property | Value |
305 +-----------------+--------------------------------------+
306 | ifname | pxeboot0 |
307 | iftype | ethernet |
308 | ports | [u'eno1'] |
309 | imac | 24:6e:96:5d:0c:92 |
310 | imtu | 1500 |
311 | ifclass | platform |
312 | aemode | None |
313 | schedpolicy | None |
314 | txhashpolicy | None |
315 | uuid | 23b5e923-1e53-4e70-a975-542d8380b7f2 |
316 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
317 | vlan_id | None |
318 | uses | [] |
319 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]320 | created_at | 2021-11-17T00:28:32.612230+00:00 |
321 | updated_at | 2021-11-17T01:03:53.143795+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]322 | sriov_numvfs | 0 |
323 | sriov_vf_driver | None |
324 | accelerated | [True] |
325 +-----------------+--------------------------------------+
326 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 pxeboot0 pxeboot
327 +--------------+--------------------------------------+
328 | Property | Value |
329 +--------------+--------------------------------------+
330 | hostname | controller-0 |
331 | uuid | 6c55622d-2da4-4f4e-ab5e-f8e06e03af7c |
332 | ifname | pxeboot0 |
333 | network_name | pxeboot |
334 +--------------+--------------------------------------+
335 [sysadmin@controller-0 ~(keystone_admin)]$
336 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 18 controller-0 mgmt0 vlan pxeboot0
337 +-----------------+--------------------------------------+
338 | Property | Value |
339 +-----------------+--------------------------------------+
340 | ifname | mgmt0 |
341 | iftype | vlan |
342 | ports | [] |
343 | imac | 24:6e:96:5d:0c:92 |
344 | imtu | 1500 |
345 | ifclass | None |
346 | aemode | None |
347 | schedpolicy | None |
348 | txhashpolicy | None |
349 | uuid | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 |
350 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
351 | vlan_id | 18 |
352 | uses | [u'pxeboot0'] |
353 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]354 | created_at | 2021-11-17T01:03:57.303000+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]355 | updated_at | None |
356 | sriov_numvfs | 0 |
357 | sriov_vf_driver | None |
358 | accelerated | [True] |
359 +-----------------+--------------------------------------+
360 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 mgmt0 mgmt
361 +--------------+--------------------------------------+
362 | Property | Value |
363 +--------------+--------------------------------------+
364 | hostname | controller-0 |
365 | uuid | 2e93ef03-e9ee-457a-8667-05b52b7109a5 |
366 | ifname | mgmt0 |
367 | network_name | mgmt |
368 +--------------+--------------------------------------+
369 [sysadmin@controller-0 ~(keystone_admin)]$
370 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-0 cluster0 vlan pxeboot0
371 +-----------------+--------------------------------------+
372 | Property | Value |
373 +-----------------+--------------------------------------+
374 | ifname | cluster0 |
375 | iftype | vlan |
376 | ports | [] |
377 | imac | 24:6e:96:5d:0c:92 |
378 | imtu | 1500 |
379 | ifclass | None |
380 | aemode | None |
381 | schedpolicy | None |
382 | txhashpolicy | None |
383 | uuid | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 |
384 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
385 | vlan_id | 19 |
386 | uses | [u'pxeboot0'] |
387 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]388 | created_at | 2021-11-17T01:04:02.613518+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]389 | updated_at | None |
390 | sriov_numvfs | 0 |
391 | sriov_vf_driver | None |
392 | accelerated | [True] |
393 +-----------------+--------------------------------------+
394 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 cluster0 cluster-host
395 +--------------+--------------------------------------+
396 | Property | Value |
397 +--------------+--------------------------------------+
398 | hostname | controller-0 |
399 | uuid | fb8b6be6-1618-4662-b063-b1e8d340aa48 |
400 | ifname | cluster0 |
401 | network_name | cluster-host |
402 +--------------+--------------------------------------+
403 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0
404 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
405 | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
406 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
407 | 119bdb85-1e24-44ff-b527-fe8f167b0ad3 | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
408 | 23b5e923-1e53-4e70-a975-542d8380b7f2 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
409 | 6a620c8e-4f7b-4f74-a9f4-2a91d3ae9756 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
410 | d8a048fa-67ef-43ac-8166-671be93caa30 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
411 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
412
413Configure NTP Servers for network time synchronization:
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]414
415::
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]416
417 system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
418
419Output
420
421::
422
423 [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
424 +--------------+--------------------------------------+
425 | Property | Value |
426 +--------------+--------------------------------------+
427 | uuid | 3206cf01-c64a-457e-ac66-b8224c9684c3 |
428 | ntpservers | 0.pool.ntp.org,1.pool.ntp.org |
429 | isystem_uuid | cc79b616-d24e-4432-a953-85c9b242cb3a |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]430 | created_at | 2021-11-17T00:27:23.529571+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]431 | updated_at | None |
432 +--------------+--------------------------------------+
433
434Add an OSD on controller-0 for Ceph:
435
436::
437
438 system host-disk-list controller-0
439 system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
440 system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
441 system host-stor-list controller-0
442
443Output
444
445::
446
447 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
448 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
449 | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
450 | | de | num | type | gib | gib | | | |
451 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
452 | 8e2a719a-fa5a-4c25-89af-70a23fb7b238 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00c66a07604fa8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
453 | | | | | 75 | | | | |
454 | | | | | | | | | |
455 | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 00c6b9139b76a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
456 | | | | | 25 | | | | |
457 | | | | | | | | | |
458 | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 0053be63c794a8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
459 | | | | | 25 | | | | |
460 | | | | | | | | | |
461 | 4879b381-8e9f-48f3-84e2-f9c6a94bbfe0 | /dev/sdd | 2096 | HDD | 1117. | 0.0 | Undetermined | 0065482503bca8de2500151b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
462 | | | | | 25 | | | | |
463 | | | | | | | | | |
464 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
465 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
466 +------------------+-------------------------------------------------------+
467 | Property | Value |
468 +------------------+-------------------------------------------------------+
469 | osdid | 0 |
470 | function | osd |
471 | state | configuring-on-unlock |
472 | journal_location | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
473 | journal_size_gib | 1024 |
474 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
475 | journal_node | /dev/sdb2 |
476 | uuid | 0816c72f-a4f0-49ea-9a95-0f02c880717c |
477 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
478 | idisk_uuid | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 |
479 | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
480 | tier_name | storage |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]481 | created_at | 2021-11-17T01:05:04.063823+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]482 | updated_at | None |
483 +------------------+-------------------------------------------------------+
484 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-0 {}
485 +------------------+-------------------------------------------------------+
486 | Property | Value |
487 +------------------+-------------------------------------------------------+
488 | osdid | 1 |
489 | function | osd |
490 | state | configuring-on-unlock |
491 | journal_location | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
492 | journal_size_gib | 1024 |
493 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
494 | journal_node | /dev/sdc2 |
495 | uuid | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 |
496 | ihost_uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
497 | idisk_uuid | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 |
498 | tier_uuid | 3af8c893-9dd4-40af-afc6-30bb79048448 |
499 | tier_name | storage |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]500 | created_at | 2021-11-17T01:05:06.939798+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]501 | updated_at | None |
502 +------------------+-------------------------------------------------------+
503 [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
504 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
505 | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
506 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
507 | 0816c72f-a4f0-49ea-9a95-0f02c880717c | osd | 0 | configuring-on-unlock | 61b6f262-a51f-4310-aeac-373b1c1bbbc2 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
508 | 7a0b3727-0e3f-4582-9415-56e44bb8f1e5 | osd | 1 | configuring-on-unlock | 81a7f4f9-dd3a-49b5-80d9-e1953aa43c79 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
509 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
510
5112.7 Unlock controller-0
512~~~~~~~~~~~~~~~~~~~~~~~
513
514::
515
516 system host-unlock controller-0
517
518Output:
519
520::
521
522 [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
523 +-----------------------+-------------------------------------------------+
524 | Property | Value |
525 +-----------------------+-------------------------------------------------+
526 | action | none |
527 | administrative | locked |
528 | availability | online |
529 | bm_ip | None |
530 | bm_type | none |
531 | bm_username | None |
532 | boot_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
533 | capabilities | {u'stor_function': u'monitor'} |
534 | clock_synchronization | ntp |
535 | config_applied | 6aa15fb4-8cb3-494e-b94e-95f85b560f22 |
536 | config_status | None |
537 | config_target | c6ae9b2d-a3c4-4751-a79e-5487ba81ed82 |
538 | console | ttyS0,115200 |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]539 | created_at | 2021-11-17T00:28:01.983673+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]540 | hostname | controller-0 |
541 | id | 1 |
542 | install_output | graphical |
543 | install_state | None |
544 | install_state_info | None |
545 | inv_state | inventoried |
546 | invprovision | provisioning |
547 | location | {} |
548 | mgmt_ip | 192.168.18.3 |
549 | mgmt_mac | 24:6e:96:5d:0c:92 |
550 | operational | disabled |
551 | personality | controller |
552 | reserved | False |
553 | rootfs_device | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
554 | serialid | None |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]555 | software_load | 21.05 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]556 | subfunction_avail | online |
557 | subfunction_oper | disabled |
558 | subfunctions | controller,worker,lowlatency |
559 | task | Unlocking |
560 | tboot | false |
561 | ttys_dcd | None |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]562 | updated_at | 2021-11-17T01:05:07.015414+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]563 | uptime | 3496 |
564 | uuid | 16afe3a2-ba50-46b8-9fd7-09010059e8b9 |
565 | vim_progress_status | None |
566 +-----------------------+-------------------------------------------------+
567
568Controller-0 will reboot to apply configuration changes and come into
569service. This can take 5-10 minutes, depending on the performance of the
570host machine.
571
572Once the controller comes back up, check the status of controller-0
573
574::
575
576 controller-0:~$ source /etc/platform/openrc
577 [sysadmin@controller-0 ~(keystone_admin)]$ system host-list
578 +----+--------------+-------------+----------------+-------------+--------------+
579 | id | hostname | personality | administrative | operational | availability |
580 +----+--------------+-------------+----------------+-------------+--------------+
581 | 1 | controller-0 | controller | unlocked | enabled | available |
582 +----+--------------+-------------+----------------+-------------+--------------+
583
5842. Installation for the second server (controller-1)
585----------------------------------------------------
586
5872.1 Power on the controller-1 server and force it to network boot
588~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
589
5902.2 As controller-1 boots, a message appears on its console instructing you to configure the personality of the node
591~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
592
5932.3 On the console of controller-0, list hosts to see newly discovered controller-1 host (hostname=None)
594~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
595
596::
597
598 [sysadmin@controller-0 ~(keystone_admin)]$ system host-list
599 +----+--------------+-------------+----------------+-------------+--------------+
600 | id | hostname | personality | administrative | operational | availability |
601 +----+--------------+-------------+----------------+-------------+--------------+
602 | 1 | controller-0 | controller | unlocked | enabled | degraded |
603 | 2 | None | None | locked | disabled | offline |
604 +----+--------------+-------------+----------------+-------------+--------------+
605
6062.4 Using the host id, set the personality of this host to 'controller’:
607~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
608
609::
610
611 [sysadmin@controller-0 ~(keystone_admin)]$ system host-update 2 personality=controller
612 +-----------------------+--------------------------------------+
613 | Property | Value |
614 +-----------------------+--------------------------------------+
615 | action | none |
616 | administrative | locked |
617 | availability | offline |
618 | bm_ip | None |
619 | bm_type | None |
620 | bm_username | None |
621 | boot_device | /dev/sda |
622 | capabilities | {} |
623 | clock_synchronization | ntp |
624 | config_applied | None |
625 | config_status | None |
626 | config_target | None |
627 | console | ttyS0,115200 |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]628 | created_at | 2021-11-17T10:17:44.387813+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]629 | hostname | controller-1 |
630 | id | 2 |
631 | install_output | text |
632 | install_state | None |
633 | install_state_info | None |
634 | inv_state | None |
635 | invprovision | None |
636 | location | {} |
637 | mgmt_ip | 192.168.18.4 |
638 | mgmt_mac | 24:6e:96:5d:38:ee |
639 | operational | disabled |
640 | personality | controller |
641 | reserved | False |
642 | rootfs_device | /dev/sda |
643 | serialid | None |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]644 | software_load | 21.05 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]645 | subfunction_avail | not-installed |
646 | subfunction_oper | disabled |
647 | subfunctions | controller,worker,lowlatency |
648 | task | None |
649 | tboot | false |
650 | ttys_dcd | None |
651 | updated_at | None |
652 | uptime | 0 |
653 | uuid | f069381d-9743-49cc-bf8b-eb4bd3972203 |
654 | vim_progress_status | None |
655 +-----------------------+--------------------------------------+
656
6572.5 Wait for the software installation on controller-1 to complete, for controller-1 to reboot, and for controller-1 to show as locked/disabled/online in 'system host-list'.
658~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
659
660This can take 5-10 minutes, depending on the performance of the host
661machine.
662
663::
664
665 [root@controller-0 hieradata(keystone_admin)]$ system host-list
666 +----+--------------+-------------+----------------+-------------+--------------+
667 | id | hostname | personality | administrative | operational | availability |
668 +----+--------------+-------------+----------------+-------------+--------------+
669 | 1 | controller-0 | controller | unlocked | enabled | available |
670 | 2 | controller-1 | controller | locked | disabled | online |
671 +----+--------------+-------------+----------------+-------------+--------------+
672
6732.6 Configure controller-1
674~~~~~~~~~~~~~~~~~~~~~~~~~~
675
676::
677
678 OAM_IF=eno3
679 MGMT_IF=eno1
680 system host-if-modify controller-1 $OAM_IF -n oam0
681 system host-if-modify controller-1 oam0 -c platform
682 system interface-network-assign controller-1 oam0 oam
683
684 system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
685 system interface-network-assign controller-1 cluster0 cluster-host
686
687 system host-if-list controller-1
688
689 system host-disk-list controller-1
690 system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
691 system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
692 system host-stor-list controller-1
693
694Output:
695
696::
697
698 [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno3
699 [sysadmin@controller-0 ~(keystone_admin)]$ MGMT_IF=eno1
700 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 $OAM_IF -n oam0
701 +-----------------+--------------------------------------+
702 | Property | Value |
703 +-----------------+--------------------------------------+
704 | ifname | oam0 |
705 | iftype | ethernet |
706 | ports | [u'eno3'] |
707 | imac | 24:6e:96:5d:39:0e |
708 | imtu | 1500 |
709 | ifclass | None |
710 | aemode | None |
711 | schedpolicy | None |
712 | txhashpolicy | None |
713 | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
714 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
715 | vlan_id | None |
716 | uses | [] |
717 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]718 | created_at | 2021-11-22T14:01:32.365863+00:00 |
719 | updated_at | 2021-11-22T15:04:45.090904+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]720 | sriov_numvfs | 0 |
721 | sriov_vf_driver | None |
722 | accelerated | [True] |
723 +-----------------+--------------------------------------+
724 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-1 oam0 -c platform
725 +-----------------+--------------------------------------+
726 | Property | Value |
727 +-----------------+--------------------------------------+
728 | ifname | oam0 |
729 | iftype | ethernet |
730 | ports | [u'eno3'] |
731 | imac | 24:6e:96:5d:39:0e |
732 | imtu | 1500 |
733 | ifclass | platform |
734 | aemode | None |
735 | schedpolicy | None |
736 | txhashpolicy | None |
737 | uuid | c2473511-d0d6-445d-9739-4d43dc029de9 |
738 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
739 | vlan_id | None |
740 | uses | [] |
741 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]742 | created_at | 2021-11-22T14:05:16.052229+00:00 |
743 | updated_at | 2021-11-22T15:08:35.324634+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]744 | sriov_numvfs | 0 |
745 | sriov_vf_driver | None |
746 | accelerated | [True] |
747 +-----------------+--------------------------------------+
748 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 oam0 oam
749 +--------------+--------------------------------------+
750 | Property | Value |
751 +--------------+--------------------------------------+
752 | hostname | controller-1 |
753 | uuid | f2e7f088-0dd0-4adc-8348-4e3cef23bc47 |
754 | ifname | oam0 |
755 | network_name | oam |
756 +--------------+--------------------------------------+
757 [sysadmin@controller-0 ~(keystone_admin)]$
758
759 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-add -V 19 controller-1 cluster0 vlan pxeboot0
760 +-----------------+--------------------------------------+
761 | Property | Value |
762 +-----------------+--------------------------------------+
763 | ifname | cluster0 |
764 | iftype | vlan |
765 | ports | [] |
766 | imac | 24:6e:96:5d:38:ee |
767 | imtu | 1500 |
768 | ifclass | None |
769 | aemode | None |
770 | schedpolicy | None |
771 | txhashpolicy | None |
772 | uuid | b6783682-b2aa-4135-90d2-676e1db41ae8 |
773 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
774 | vlan_id | 19 |
775 | uses | [u'pxeboot0'] |
776 | used_by | [] |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]777 | created_at | 2021-11-22T15:08:43.932209+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]778 | updated_at | None |
779 | sriov_numvfs | 0 |
780 | sriov_vf_driver | None |
781 | accelerated | [True] |
782 +-----------------+--------------------------------------+
783
784 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-1 cluster0 cluster-host
785 +--------------+--------------------------------------+
786 | Property | Value |
787 +--------------+--------------------------------------+
788 | hostname | controller-1 |
789 | uuid | 8fc64805-b54b-45a4-b88a-e13b236abfe8 |
790 | ifname | cluster0 |
791 | network_name | cluster-host |
792 +--------------+--------------------------------------+
793 [sysadmin@controller-0 ~(keystone_admin)]$
794 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1
795 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
796 | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes |
797 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
798 | b6783682-b2aa-4135-90d2-676e1db41ae8 | cluster0 | platform | vlan | 19 | [] | [u'pxeboot0'] | [] | MTU=1500 |
799 | b8921960-fde5-44c3-960d-2aebf42ea400 | pxeboot0 | platform | ethernet | None | [u'eno1'] | [] | [u'mgmt0', u'cluster0'] | MTU=1500 |
800 | c103275b-2b75-4568-865f-ac6be32ecb2d | mgmt0 | platform | vlan | 18 | [] | [u'pxeboot0'] | [] | MTU=1500 |
801 | c2473511-d0d6-445d-9739-4d43dc029de9 | oam0 | platform | ethernet | None | [u'eno3'] | [] | [] | MTU=1500 |
802 +--------------------------------------+----------+----------+----------+---------+-----------+---------------+-------------------------+------------+
803 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1
804 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
805 | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_id | device_path |
806 | | de | num | type | gib | gib | | | |
807 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
808 | 5b8fade4-b048-48fa-b906-9dcbdbed8e96 | /dev/sda | 2048 | HDD | 893. | 644.726 | Undetermined | 00cbd97f3e36ccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:0:0 |
809 | | | | | 75 | | | | |
810 | | | | | | | | | |
811 | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/sdb | 2064 | HDD | 1117. | 1117.247 | Undetermined | 006d0e977b5fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0 |
812 | | | | | 25 | | | | |
813 | | | | | | | | | |
814 | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/sdc | 2080 | HDD | 1117. | 1117.247 | Undetermined | 003a2377ac7fccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0 |
815 | | | | | 25 | | | | |
816 | | | | | | | | | |
817 | 774c3cd0-1178-4145-9573-f0d6dee2ba06 | /dev/sdd | 2096 | HDD | 1117. | 1117.247 | Undetermined | 00d7093ef0adccfa2500561b14604609 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:3:0 |
818 | | | | | 25 | | | | |
819 | | | | | | | | | |
820 | 00361302-8d55-4730-855c-b0098c73ab7e | /dev/sde | 2112 | SSD | 223. | 223.568 | N/A | PHDW730104QM240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-1 |
821 | | | | | 57 | | | | |
822 | | | | | | | | | |
823 | 7ce735e6-920f-4424-a890-a7a7f48d7632 | /dev/sdf | 2128 | SSD | 223. | 223.568 | N/A | PHDW730104LL240E | /dev/disk/by-path/pci-0000:d8:00.0-ata-2 |
824 | | | | | 57 | | | | |
825 | | | | | | | | | |
826 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+----------------------------------+-------------------------------------------------+
827 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-1 {}
828 +------------------+-------------------------------------------------------+
829 | Property | Value |
830 +------------------+-------------------------------------------------------+
831 | osdid | 2 |
832 | function | osd |
833 | state | configuring-on-unlock |
834 | journal_location | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
835 | journal_size_gib | 1024 |
836 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 |
837 | journal_node | /dev/sdb2 |
838 | uuid | 54a218d8-0466-4366-9ef0-3ec5a952fde7 |
839 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
840 | idisk_uuid | 1a3f0a36-5961-42e5-a271-e71db1c25d42 |
841 | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
842 | tier_name | storage |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]843 | created_at | 2021-11-22T15:11:55.641193+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]844 | updated_at | None |
845 +------------------+-------------------------------------------------------+
846
847 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-1 | awk '/\/dev\/sdc/{print $2}' | xargs -i system host-stor-add controller-1 {}
848 +------------------+-------------------------------------------------------+
849 | Property | Value |
850 +------------------+-------------------------------------------------------+
851 | osdid | 3 |
852 | function | osd |
853 | state | configuring-on-unlock |
854 | journal_location | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
855 | journal_size_gib | 1024 |
856 | journal_path | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 |
857 | journal_node | /dev/sdc2 |
858 | uuid | 5be88c7a-3a94-4b97-9da5-b247bb89406c |
859 | ihost_uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
860 | idisk_uuid | eddd732f-2cea-49b3-86db-b722c0b1a1ae |
861 | tier_uuid | 06b4740e-29db-4896-9600-03ee40fe0d6c |
862 | tier_name | storage |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]863 | created_at | 2021-11-22T15:12:04.274839+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]864 | updated_at | None |
865 +------------------+-------------------------------------------------------+
866
867 [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-1
868 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
869 | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
870 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
871 | 54a218d8-0466-4366-9ef0-3ec5a952fde7 | osd | 2 | configuring-on-unlock | 1a3f0a36-5961-42e5-a271-e71db1c25d42 | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:1:0-part2 | /dev/sdb2 | 1 | storage |
872 | 5be88c7a-3a94-4b97-9da5-b247bb89406c | osd | 3 | configuring-on-unlock | eddd732f-2cea-49b3-86db-b722c0b1a1ae | /dev/disk/by-path/pci-0000:86:00.0-scsi-0:2:2:0-part2 | /dev/sdc2 | 1 | storage |
873 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
874
8752.7 Unlock controller-1
876~~~~~~~~~~~~~~~~~~~~~~~
877
878Unlock controller-1 in order to bring it into service:
879
880::
881
882 [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-1
883 +-----------------------+--------------------------------------+
884 | Property | Value |
885 +-----------------------+--------------------------------------+
886 | action | none |
887 | administrative | locked |
888 | availability | online |
889 | bm_ip | None |
890 | bm_type | None |
891 | bm_username | None |
892 | boot_device | /dev/sda |
893 | capabilities | {u'stor_function': u'monitor'} |
894 | clock_synchronization | ntp |
895 | config_applied | None |
896 | config_status | Config out-of-date |
897 | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
898 | console | ttyS0,115200 |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]899 | created_at | 2021-11-22T12:58:11.630526+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]900 | hostname | controller-1 |
901 | id | 3 |
902 | install_output | text |
903 | install_state | None |
904 | install_state_info | None |
905 | inv_state | inventoried |
906 | invprovision | unprovisioned |
907 | location | {} |
908 | mgmt_ip | 192.168.18.4 |
909 | mgmt_mac | 24:6e:96:5d:38:ee |
910 | operational | disabled |
911 | personality | controller |
912 | reserved | False |
913 | rootfs_device | /dev/sda |
914 | serialid | None |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]915 | software_load | 21.05 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]916 | subfunction_avail | online |
917 | subfunction_oper | disabled |
918 | subfunctions | controller,worker,lowlatency |
919 | task | Unlocking |
920 | tboot | false |
921 | ttys_dcd | None |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]922 | updated_at | 2021-11-22T15:13:09.716324+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]923 | uptime | 752 |
924 | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
925 | vim_progress_status | None |
926 +-----------------------+--------------------------------------+
927
928Controller-1 will reboot in order to apply configuration changes and
929come into service. This can take 5-10 minutes, depending on the
930performance of the host machine.
931
932::
933
934 [root@controller-0 hieradata(keystone_admin)]$ system host-list
935 +----+--------------+-------------+----------------+-------------+--------------+
936 | id | hostname | personality | administrative | operational | availability |
937 +----+--------------+-------------+----------------+-------------+--------------+
938 | 1 | controller-0 | controller | unlocked | enabled | available |
939 | 2 | controller-1 | controller | unlocked | enabled | available |
940 +----+--------------+-------------+----------------+-------------+--------------+
941
942 [sysadmin@controller-0 ~(keystone_admin)]$ system host-show controller-1
943 +-----------------------+-----------------------------------------------------------------------+
944 | Property | Value |
945 +-----------------------+-----------------------------------------------------------------------+
946 | action | none |
947 | administrative | unlocked |
948 | availability | available |
949 | bm_ip | None |
950 | bm_type | None |
951 | bm_username | None |
952 | boot_device | /dev/sda |
953 | capabilities | {u'stor_function': u'monitor', u'Personality': u'Controller-Standby'} |
954 | clock_synchronization | ntp |
955 | config_applied | 9747e0ce-2319-409d-b75c-2475bc5065ac |
956 | config_status | None |
957 | config_target | 9747e0ce-2319-409d-b75c-2475bc5065ac |
958 | console | ttyS0,115200 |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]959 | created_at | 2021-11-22T12:58:11.630526+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]960 | hostname | controller-1 |
961 | id | 2 |
962 | install_output | text |
963 | install_state | None |
964 | install_state_info | None |
965 | inv_state | inventoried |
966 | invprovision | provisioned |
967 | location | {} |
968 | mgmt_ip | 192.168.18.4 |
969 | mgmt_mac | 24:6e:96:5d:38:ee |
970 | operational | enabled |
971 | personality | controller |
972 | reserved | False |
973 | rootfs_device | /dev/sda |
974 | serialid | None |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]975 | software_load | 21.05 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]976 | subfunction_avail | available |
977 | subfunction_oper | enabled |
978 | subfunctions | controller,worker,lowlatency |
979 | task | |
980 | tboot | false |
981 | ttys_dcd | None |
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]982 | updated_at | 2021-11-22T23:59:07.787759+00:00 |
Xiaohua Zhang3ff08af2020-11-23 09:19:57 -0500[diff] [blame]983 | uptime | 31008 |
984 | uuid | 63c930c7-2195-4d5a-870c-be610fd6b4fc |
985 | vim_progress_status | services-enabled |
986 +-----------------------+-----------------------------------------------------------------------+
987
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]988
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]9893. Simple use case for sriov
990````````````````````````````
991
9923.1 After controller-0 is rebooted and up running, download the DPDK
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame]993''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]994
995::
996
997 [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
998 [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
999 Password:
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]1000 --2021-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1001 Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
1002 Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]1003
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1004 HTTP request sent, awaiting response... 200 OK
1005 Length: 10251680 (9.8M) [application/octet-stream]
1006 Saving to: ‘dpdk-17.11.10.tar.xz’
1007
1008 dpdk-17.11.10.tar.xz 100%
1009 [========================================================================================>] 9.78M
1010 1.48MB/s in 6.8s
1011
Jackie Huang25577232021-12-03 14:30:37 +0800[diff] [blame]1012 2021-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1013
1014 sudo tar xvf dpdk-17.11.10.tar.xz
1015
1016 sudo ln -s dpdk-stable-17.11.10 dpdk-stable
1017
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame]10183.2 Prepare the yaml file for the network assignment container
1019''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1020
1021The following the exmaple of the yaml file:
1022
1023::
1024
1025 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
1026 > apiVersion: "k8s.cni.cncf.io/v1"
1027 > kind: NetworkAttachmentDefinition
1028 > metadata:
1029 > name: sriov-data-dpdk-0
1030 > annotations:
1031 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
1032 > spec:
1033 > config: '{
1034 > "type": "sriov",
1035 > "name": "sriov-data-dpdk-0"
1036 > }'
1037 >
1038 > ---
1039 > apiVersion: "k8s.cni.cncf.io/v1"
1040 > kind: NetworkAttachmentDefinition
1041 > metadata:
1042 > name: sriov-data-dpdk-1
1043 > annotations:
1044 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
1045 > spec:
1046 > config: '{
1047 > "type": "sriov",
1048 > "name": "sriov-data-dpdk-1"
1049 > }'
1050 > EOF
1051
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame]10523.3 Run the network assignent container for the 2 VFs
1053'''''''''''''''''''''''''''''''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1054
1055::
1056
1057 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
1058 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
1059 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
1060
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame]10613.4 Prepare the VF container yaml file
1062''''''''''''''''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1063
1064::
1065
1066 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
1067 > apiVersion: v1
1068 > kind: Pod
1069 metadata:
1070 > metadata:
1071 > name: pod-with-dpdk-vfs-0
1072 > annotations:
1073 > k8s.v1.cni.cncf.io/networks: '[
1074 > { "name": "sriov-data-dpdk-0" },
1075 { "name": "sriov-data-dpdk-1" }
1076 > { "name": "sriov-data-dpdk-1" }
1077 > ]'
1078 > spec:
1079 > restartPolicy: Never
1080 > containers:
1081 > - name: pod-with-dpdk-vfs-0
1082 > image: wrsnfv/ubuntu-dpdk-build:v0.3
1083 > env:
1084 > - name: RTE_SDK
1085 > value: "/usr/src/dpdk"
1086 > command:
1087 > - sleep
1088 > - infinity
1089 > stdin: true
1090 > tty: true
1091 > securityContext:
1092 > privileged: true
1093 > capabilities:
1094 > add:
1095 > - ALL
1096 > resources:
1097 > requests:
1098 > cpu: 4
1099 > memory: 4Gi
1100 > intel.com/pci_sriov_net_physnet0: '1'
1101 > intel.com/pci_sriov_net_physnet1: '1'
1102 > limits:
1103 > cpu: 4
1104 > hugepages-1Gi: 2Gi
1105 > memory: 4Gi
1106 > intel.com/pci_sriov_net_physnet0: '1'
1107 > intel.com/pci_sriov_net_physnet1: '1'
1108 > volumeMounts:
1109 > - mountPath: /mnt/huge-1048576kB
1110 > name: hugepage
1111 > - name: dpdk-volume
1112 > mountPath: /usr/src/dpdk
1113 > - name: lib-volume
1114 > mountPath: /lib/modules
1115 > - name: src-volume
1116 > mountPath: /usr/src/
1117 > volumes:
1118 > - name: hugepage
1119 > emptyDir:
1120 > medium: HugePages
1121 > - name: dpdk-volume
1122 > hostPath:
1123 > path: /opt/dpdk-stable/
1124 > - name: lib-volume
1125 > hostPath:
1126 > path: /lib/modules
1127 > - name: src-volume
1128 > hostPath:
1129 > path: /usr/src/
1130 > EOF
1131
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame]11323.5 Run the VF container
1133''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1134
1135Start the VF container.
1136
1137::
1138
1139 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
1140 pod/pod-with-dpdk-vfs-0 created
1141
1142 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
1143 NAME READY STATUS RESTARTS AGE
1144 pod-with-dpdk-vfs-0 1/1 Running 0 6m40s
1145
1146Login the VF container
1147
1148::
1149
1150 kubectl exec -it pod-with-dpdk-vfs-0 -- bash
1151
1152Build the DPDK
1153
1154::
1155
1156 cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
1157
1158 root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
1159 HOSTCC scripts/basic/fixdep
1160 HOSTCC scripts/kconfig/conf.o
1161 HOSTCC scripts/kconfig/confdata.o
1162 HOSTCC scripts/kconfig/expr.o
1163 HOSTCC scripts/kconfig/symbol.o
1164 HOSTCC scripts/kconfig/preprocess.o
1165 HOSTCC scripts/kconfig/zconf.lex.o
1166 HOSTCC scripts/kconfig/zconf.tab.o
1167 HOSTLD scripts/kconfig/conf
1168 scripts/kconfig/conf --syncconfig Kconfig
1169 HOSTCC arch/x86/tools/relocs_32.o
1170 HOSTCC arch/x86/tools/relocs_64.o
1171 HOSTCC arch/x86/tools/relocs_common.o
1172 HOSTLD arch/x86/tools/relocs
1173 HOSTCC scripts/genksyms/genksyms.o
1174 YACC scripts/genksyms/parse.tab.c
1175 HOSTCC scripts/genksyms/parse.tab.o
1176 LEX scripts/genksyms/lex.lex.c
1177 YACC scripts/genksyms/parse.tab.h
1178 HOSTCC scripts/genksyms/lex.lex.o
1179 HOSTLD scripts/genksyms/genksyms
1180 HOSTCC scripts/bin2c
1181 HOSTCC scripts/kallsyms
1182 HOSTCC scripts/conmakehash
1183 HOSTCC scripts/recordmcount
1184 HOSTCC scripts/sortextable
1185 HOSTCC scripts/asn1_compiler
1186 HOSTCC scripts/sign-file
1187 HOSTCC scripts/extract-cert
1188 CC scripts/mod/empty.o
1189 HOSTCC scripts/mod/mk_elfconfig
1190 MKELF scripts/mod/elfconfig.h
1191 HOSTCC scripts/mod/modpost.o
1192 CC scripts/mod/devicetable-offsets.s
1193 UPD scripts/mod/devicetable-offsets.h
1194 HOSTCC scripts/mod/file2alias.o
1195 HOSTCC scripts/mod/sumversion.o
1196 HOSTLD scripts/mod/modpost
1197 CC kernel/bounds.s
1198 CC arch/x86/kernel/asm-offsets.s
1199 CALL scripts/checksyscalls.sh
1200
1201Build the test_pmd application
1202
1203::
1204
1205 cd $RTE_SDK
1206 ./usertools/dpdk-setup.sh
1207 Option: 14
1208 CC config.o
1209 CC iofwd.o
1210 CC macfwd.o
1211 CC macswap.o
1212 CC flowgen.o
1213 CC rxonly.o
1214 CC txonly.o
1215 CC csumonly.o
1216 CC icmpecho.o
1217 CC tm.o
1218 LD testpmd
1219 INSTALL-APP testpmd
1220 INSTALL-MAP testpmd.map
1221 == Build app/proc_info
1222 CC main.o
1223 LD dpdk-procinfo
1224 INSTALL-APP dpdk-procinfo
1225 INSTALL-MAP dpdk-procinfo.map
1226 == Build app/pdump
1227 CC main.o
1228 LD dpdk-pdump
1229 INSTALL-APP dpdk-pdump
1230 INSTALL-MAP dpdk-pdump.map
1231 == Build app/test-crypto-perf
1232 CC main.o
1233 CC cperf_ops.o
1234 CC cperf_options_parsing.o
1235 CC cperf_test_vectors.o
1236 CC cperf_test_throughput.o
1237 CC cperf_test_latency.o
1238 CC cperf_test_pmd_cyclecount.o
1239 CC cperf_test_verify.o
1240 CC cperf_test_vector_parsing.o
1241 CC cperf_test_common.o
1242 LD dpdk-test-crypto-perf
1243 INSTALL-APP dpdk-test-crypto-perf
1244 INSTALL-MAP dpdk-test-crypto-perf.map
1245 == Build app/test-eventdev
1246 CC evt_main.o
1247 CC evt_options.o
1248 CC evt_test.o
1249 CC parser.o
1250 CC test_order_common.o
1251 CC test_order_queue.o
1252 CC test_order_atq.o
1253 CC test_perf_common.o
1254 CC test_perf_queue.o
1255 CC test_perf_atq.o
1256 LD dpdk-test-eventdev
1257 INSTALL-APP dpdk-test-eventdev
1258 INSTALL-MAP dpdk-test-eventdev.map
1259 Build complete [x86_64-native-linuxapp-gcc]
1260 Installation cannot run with T defined and DESTDIR undefined
1261 ------------------------------------------------------------------------------
1262 RTE_TARGET exported as x86_64-native-linuxapp-gcc
1263 ------------------------------------------------------------------------------
1264
1265 Press enter to continue ...
1266
1267Check the VF PCI information:
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame]1268
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1269::
1270
1271 root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
1272 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
1273 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
1274
1275Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame]1276
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1277::
1278
1279 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
1280 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
1281 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
1282 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
1283 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
1284 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
1285 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
1286
1287 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
1288 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
1289 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
1290 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
1291 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
1292 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
1293 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
1294
1295 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
1296 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
1297
1298
1299Get back to pod and run the test_pmd
1300
1301::
1302
1303 cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
1304
1305 taskset -p --cpu-list 1
1306 pid 1's current affinity list: 2,3,18,19
1307
1308 ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
1309 start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
1310 peer=1,"9e:fd:e6:dd:c1:01"
1311 EAL: Detected 32 lcore(s)
1312 EAL: No free hugepages reported in hugepages-2048kB
1313 EAL: Probing VFIO support...
1314 EAL: VFIO support initialized
1315 EAL: PCI device 0000:05:11.0 on NUMA socket 0
1316 EAL: probe driver: 8086:10ed net_ixgbe_vf
1317 EAL: using IOMMU type 1 (Type 1)
1318 EAL: PCI device 0000:05:11.1 on NUMA socket 0
1319 EAL: probe driver: 8086:10ed net_ixgbe_vf
1320 Auto-start selected
1321 Ports to start sending a burst of packets first
1322 Warning: lsc_interrupt needs to be off when using tx_first. Disabling.
1323 USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
1324 Configuring Port 0 (socket 0)
1325 Port 0: 9E:FD:E6:DD:C1:01
1326 Configuring Port 1 (socket 0)
1327 Port 1: 9E:FD:E6:DD:C1:02
1328 Checking link statuses...
1329 Port0 Link Up. speed 10000 Mbps- full-duplex
1330 Port1 Link Up. speed 10000 Mbps- full-duplex
1331 Done
1332 No commandline core given, start packet forwarding
1333 io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
1334 disabled
1335 Logical Core 3 (socket 0) forwards packets on 2 streams:
1336 RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
1337 RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
1338
1339 io packet forwarding packets/burst=32
1340 nb forwarding cores=1 - nb forwarding ports=2
1341 port 0:
1342 CRC stripping enabled
1343 RX queues=1 - RX desc=128 - RX free threshold=32
1344 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1345 TX queues=1 - TX desc=512 - TX free threshold=32
1346 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1347 TX RS bit threshold=32 - TXQ flags=0xf01
1348 port 1:
1349 CRC stripping enabled
1350 RX queues=1 - RX desc=128 - RX free threshold=32
1351 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1352 TX queues=1 - TX desc=512 - TX free threshold=32
1353 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1354 TX RS bit threshold=32 - TXQ flags=0xf01
1355
1356 Port statistics ====================================
1357 ######################## NIC statistics for port 0 ########################
1358 RX-packets: 56 RX-missed: 0 RX-bytes: 4096
1359 RX-errors: 0
1360 RX-nombuf: 0
1361 TX-packets: 64 TX-errors: 0 TX-bytes: 4096
1362
1363 Throughput (since last show)
1364 Rx-pps: 0
1365 Tx-pps: 0
1366 ############################################################################
1367
1368 ######################## NIC statistics for port 1 ########################
1369 RX-packets: 432 RX-missed: 0 RX-bytes: 27712
1370 RX-errors: 0
1371 RX-nombuf: 0
1372 TX-packets: 461 TX-errors: 0 TX-bytes: 30080
1373
1374 Throughput (since last show)
1375 Rx-pps: 0
1376 Tx-pps: 0
1377 ############################################################################
1378
1379 Port statistics ====================================
1380 ######################## NIC statistics for port 0 ########################
1381 RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344
1382 RX-errors: 0
1383 RX-nombuf: 0
1384 TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376
1385
1386 Throughput (since last show)
1387 Rx-pps: 7068409
1388 Tx-pps: 7091206
1389 ############################################################################
1390
1391
1392
1393
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]1394
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]1395
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]1396References
1397----------
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]1398
1399- `StarlingX`_
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]1400
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]1401.. _`StarlingX`: https://docs.starlingx.io/
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame]1402