Gitiles
Code Review Sign In
gerrit.nordix.org / oransc / pti / rtp / d6dfaa5a49b8c9c97553d2903a1d8222cb6c5ab7 / . / docs / installation-guide.rst
blob: ae524a944cbbb844d549d9a63b400622f7d6d433 [file] [log] [blame]
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]1.. This work is licensed under a Creative Commons Attribution 4.0 International License.
2.. SPDX-License-Identifier: CC-BY-4.0
3.. Copyright (C) 2019 Wind River Systems, Inc.
4
5
6Installation Guide
7==================
8
9.. contents::
10 :depth: 3
11 :local:
12
13Abstract
14--------
15
16This document describes how to install O-RAN INF image, example configuration for better
17real time performance, and example deployment of Kubernetes cluster and plugins.
18
19The audience of this document is assumed to have basic knowledge in Yocto/Open-Embedded Linux
20and container technology.
21
22Version history
23
24+--------------------+--------------------+--------------------+--------------------+
25| **Date** | **Ver.** | **Author** | **Comment** |
26| | | | |
27+--------------------+--------------------+--------------------+--------------------+
28| 2019-11-02 | 1.0.0 | Jackie Huang | Initail version |
29| | | | |
30+--------------------+--------------------+--------------------+--------------------+
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]31| 2020-06-03 | 2.0.0 (Bronze) | Xiaohua Zhang | Bronze version |
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]32| | | | |
33+--------------------+--------------------+--------------------+--------------------+
34| | | | |
35| | | | |
36+--------------------+--------------------+--------------------+--------------------+
37
38
39Preface
40-------
41
42Before starting the installation and deployment of O-RAN INF, you need to download the ISO image or build from source as described in developer-guide.
43
44
45Hardware Requirements
46---------------------
47
48Following minimum hardware requirements must be met for installation of O-RAN INF image:
49
50+--------------------+----------------------------------------------------+
51| **HW Aspect** | **Requirement** |
52| | |
53+--------------------+----------------------------------------------------+
54| **# of servers** | 1 |
55+--------------------+----------------------------------------------------+
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]56| **CPU** | 4 |
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]57| | |
58+--------------------+----------------------------------------------------+
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]59| **RAM** | 16G |
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]60| | |
61+--------------------+----------------------------------------------------+
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]62| **Disk** | 500G * 2 |
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]63| | |
64+--------------------+----------------------------------------------------+
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]65| **NICs** | 2 |
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]66| | |
67+--------------------+----------------------------------------------------+
68
69
70
71Software Installation and Deployment
72------------------------------------
73
741. Installation from the O-RAN INF ISO image
75````````````````````````````````````````````
76
77- Please see the README.md file for how to build the image.
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]78- The Image is a live ISO image with CLI installer: inf-image-aio-installer-intel-corei7-64.iso
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]79
801.1 Burn the image to the USB device
81''''''''''''''''''''''''''''''''''''
82
83- Assume the the usb device is /dev/sdX here
84
85::
86
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]87 $ sudo dd if=/path/to/inf-image-aio-installer-intel-corei7-64.iso of=/dev/sdX bs=1M
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]88
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]891.2 Install the server
90''''''''''''''''''''''
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]91
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]92- Reboot the target from the USB device.
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]93
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]94- Select "Graphics console install" or "Serial console install" and press ENTER
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]95
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]96- Remove the USB device and press ENTER to reboot
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]97
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]98
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]992. Configuration and initialize the bootstrap
100`````````````````````````````````````````````
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]101
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]1022.1 First Login
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]103'''''''''''''''
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]104Example:
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]105
106::
107
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]108 ifconfig eno1 128.224.180.14/24 up
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]109
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]110 ip route add default via 128.224.180.1
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]111
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]112 ping 8.8.8.8
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]113
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]1142.3 Login the server through SSH
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]115''''''''''''''''''''''''''''''''
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]116
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]1172.4 Prepare the localhost.yml for bootstrap
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]118'''''''''''''''''''''''''''''''''''''''''''
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]119
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]120Example:
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]121
122::
123
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]124 system_mode: simplex
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]125
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]126 external_oam_subnet: 128.224.180.0/24
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]127
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]128 external_oam_gateway_address: 128.224.180.1
129
130 external_oam_floating_address: 128.224.180.14
131
132 admin_username: admin
133
134 admin_password: Wind123!
135
136 ansible_become_pass: Wind123!
137
138 docker_http_proxy: http://xxx.xxx.xxx.xxx:8080
139
140 docker_https_proxy: https://xxx.xxx.xxx.xxx:8080
141
1422.5 Run the bootstrap
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]143'''''''''''''''''''''
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]144
145Use the following command:
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]146
147::
148
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]149 ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap.yml -vvv
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]150
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]151After the bootstrap successfully finish, it will show as following:
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]152
153::
154
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]155 PLAY RECAP ******************************************************************************************************************************************
156 localhost : ok=256 changed=135 unreachable=0 failed=0 skipped=220 rescued=0 ignored=0
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]157
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]1582.6 Configure the OAM network
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]159'''''''''''''''''''''''''''''
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]160
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]161Example:
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]162
163::
164
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]165 controller-0:~$ source /etc/platform/openrc
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]166
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]167 [sysadmin@controller-0 ~(keystone_admin)]$ OAM_IF=eno1
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]168
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]169 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify controller-0 $OAM_IF -c platform
170
171 +-----------------+--------------------------------------+
172 | Property | Value |
173 +-----------------+--------------------------------------+
174 | ifname | eno1 |
175 +-----------------+--------------------------------------+
176 | iftype | ethernet |
177 +-----------------+--------------------------------------+
178 | ports | [u'eno1'] |
179 +-----------------+--------------------------------------+
180 | imac | 3c:a8:2a:1d:31:d0 |
181 +-----------------+--------------------------------------+
182 | imtu | 1500 |
183 +-----------------+--------------------------------------+
184 | ifclass | platform |
185 +-----------------+--------------------------------------+
186 | aemode | None |
187 +-----------------+--------------------------------------+
188 | schedpolicy | None |
189 +-----------------+--------------------------------------+
190 | txhashpolicy | None |
191 +-----------------+--------------------------------------+
192 | uuid | 95a32d13-01fc-43d2-90ab-801970fb3859 |
193 +-----------------+--------------------------------------+
194 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
195 +-----------------+--------------------------------------+
196 | vlan_id | None |
197 +-----------------+--------------------------------------+
198 | uses | [] |
199 +-----------------+--------------------------------------+
200 | used_by | [] |
201 +-----------------+--------------------------------------+
202 | created_at | 2020-06-03T08:21:17.526303+00:00 |
203 +-----------------+--------------------------------------+
204 | updated_at | 2020-06-03T10:02:54.682501+00:00 |
205 +-----------------+--------------------------------------+
206 | sriov_numvfs | 0 |
207 +-----------------+--------------------------------------+
208 | sriov_vf_driver | None |
209 +-----------------+--------------------------------------+
210 | accelerated | [False] |
211 +-----------------+--------------------------------------+
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]212
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]213 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-assign controller-0 $OAM_IF oam
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]214
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]215 +--------------+--------------------------------------+
216 | Property | Value |
217 +--------------+--------------------------------------+
218 | hostname | controller-0 |
219 +--------------+--------------------------------------+
220 | uuid | 32592cc3-9d66-4961-a51e-23686da00290 |
221 +--------------+--------------------------------------+
222 | ifname | eno1 |
223 +--------------+--------------------------------------+
224 | network_name | oam |
225 +--------------+--------------------------------------+
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]226
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]227
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]228 [sysadmin@controller-0 ~(keystone_admin)]$
229 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]230
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]231 +--------------+--------------------------------------+--------+--------------+
232 | hostname | uuid | ifname | network_name |
233 +--------------+--------------------------------------+--------+--------------+
234 | controller-0 | 2478f4c0-a7b4-4656-aa2a-97a0e8891955 | lo | mgmt |
235 +--------------+--------------------------------------+--------+--------------+
236 | controller-0 | 32592cc3-9d66-4961-a51e-23686da00290 | eno1 | oam |
237 +--------------+--------------------------------------+--------+--------------+
238 | controller-0 | 43721e4d-4f52-4b4b-bd02-0efa00ef2124 | lo | cluster-host |
239 +--------------+--------------------------------------+--------+--------------+
240
2412.7 Configure the NTP server
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]242''''''''''''''''''''''''''''
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]243
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]244Example:
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]245
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]246::
247
248 [sysadmin@controller-0 ~(keystone_admin)]$
249 [sysadmin@controller-0 ~(keystone_admin)]$ system ntp-modify ntpservers=0.pool.ntp.org,1.pool.ntp.org
250
251 +--------------+--------------------------------------+
252 | Property | Value |
253 +--------------+--------------------------------------+
254 | uuid | 8fbe5712-6ba0-4e61-b174-ed60f152f616 |
255 +--------------+--------------------------------------+
256 | ntpservers | 0.pool.ntp.org,1.pool.ntp.org |
257 +--------------+--------------------------------------+
258 | isystem_uuid | 802ee0be-3728-4df4-b9f7-1533c67fd960 |
259 +--------------+--------------------------------------+
260 | created_at | 2020-06-03T08:19:47.812249+00:00 |
261 +--------------+--------------------------------------+
262 | updated_at | None |
263 +--------------+--------------------------------------+
264
2652.8 Configure the Storage by using the second hard drive as ceph backend
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]266''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]267
268Example:
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]269
270::
271
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]272 [sysadmin@controller-0 ~(keystone_admin)]$
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]273
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]274 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]275
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]276 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
277 | uuid | device_no | device_ | device_ | size_ | available_ | rpm | serial_ | device_path |
278 | | de | num | type | gib | gib | | id | |
279 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
280 | 3884f906-cb9b-4052-bcb8-8be70e599b9f | /dev/sda | 2048 | HDD | 1117. | 868.759 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: |
281 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
282 | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:0 |
283 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
284 | | | | | | | | | |
285 | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/sdb | 2064 | HDD | 1117. | 0.0 | Undetermined | PDNLH0B | /dev/disk/by-path/pci-0000: |
286 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
287 | | | | | 782 | | | RH8W4QQ | 03:00.0-scsi-0:1:0:1 |
288 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
289 | | | | | | | | | |
290 +--------------------------------------+-----------+---------+---------+-------+------------+--------------+---------+-----------------------------+
291
292 [sysadmin@controller-0 ~(keystone_admin)]$ system host-disk-list controller-0 | awk '/\/dev\/sdb/{print $2}' | xargs -i system host-stor-add controller-0 {}
293
294 +------------------+-------------------------------------------------------+
295 | Property | Value |
296 +------------------+-------------------------------------------------------+
297 | osdid | 0 |
298 +------------------+-------------------------------------------------------+
299 | function | osd |
300 +------------------+-------------------------------------------------------+
301 | state | configuring-on-unlock |
302 +------------------+-------------------------------------------------------+
303 | journal_location | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf |
304 +------------------+-------------------------------------------------------+
305 | journal_size_gib | 1024 |
306 +------------------+-------------------------------------------------------+
307 | journal_path | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 |
308 +------------------+-------------------------------------------------------+
309 | journal_node | /dev/sdb2 |
310 +------------------+-------------------------------------------------------+
311 | uuid | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf |
312 +------------------+-------------------------------------------------------+
313 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
314 +------------------+-------------------------------------------------------+
315 | idisk_uuid | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 |
316 +------------------+-------------------------------------------------------+
317 | tier_uuid | db6eabf9-9922-4d8e-9740-c0f9878e692d |
318 +------------------+-------------------------------------------------------+
319 | tier_name | storage |
320 +------------------+-------------------------------------------------------+
321 | created_at | 2020-06-03T10:03:55.645744+00:00 |
322 +------------------+-------------------------------------------------------+
323 | updated_at | None |
324 +------------------+-------------------------------------------------------+
325
326 [sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-list controller-0
327
328 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
329 | uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
330 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
331 | 71a9d508-06f6-4f5a-84f2-4723e70b1ecf | osd | 0 | configuring-on-unlock | 4e6c4b16-11e6-40b7-a2ff-751b14b8f0e9 | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:1-part2 | /dev/sdb2 | 1 | storage |
332 +--------------------------------------+----------+-------+-----------------------+--------------------------------------+-------------------------------------------------------+--------------+------------------+-----------+
333
3342.9 Configure the SR-IOV enabled network
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]335''''''''''''''''''''''''''''''''''''''''
336
337It needs :
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]338 - assign the label of SR-IOV
339 - change the CPU policy to static
340 - change the hugepage setting
341 - create the data network
342 - assign the SR-IOV interface to data network
343
344Example:
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]345
346::
347
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]348 [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]349
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]350 [sysadmin@controller-0 ~(keystone_admin)]$
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]351
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]352 [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER sriovdp=enabled
353
354 +-------------+--------------------------------------+
355 | Property | Value |
356 +-------------+--------------------------------------+
357 | uuid | 7101010d-1897-4cc8-8c87-7dffaebf848b |
358 +-------------+--------------------------------------+
359 | host_uuid | 08c23843-afa4-4b80-a285-be126365276e |
360 +-------------+--------------------------------------+
361 | label_key | sriovdp |
362 +-------------+--------------------------------------+
363 | label_value | enabled |
364 +-------------+--------------------------------------+
365
366 [sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign $WORKER kube-cpu-mgr-policy=static
367
368 +-------------+--------------------------------------+
369 | Property | Value |
370 +-------------+--------------------------------------+
371 | uuid | 4936f32b-6939-4115-b643-846bd50e7738 |
372 +-------------+--------------------------------------+
373 | host_uuid | 08c23843-afa4-4b80-a285-be126365276e |
374 +-------------+--------------------------------------+
375 | label_key | kube-cpu-mgr-policy |
376 +-------------+--------------------------------------+
377 | label_value | static |
378 +-------------+--------------------------------------+
379
380
381 [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 0
382
383 +-------------------------------------+--------------------------------------+
384 | Property | Value |
385 +-------------------------------------+--------------------------------------+
386 | Memory: Usable Total (MiB) | 0 |
387 +-------------------------------------+--------------------------------------+
388 | Platform (MiB) | 7000 |
389 +-------------------------------------+--------------------------------------+
390 | Available (MiB) | 0 |
391 +-------------------------------------+--------------------------------------+
392 | Huge Pages Configured | True |
393 +-------------------------------------+--------------------------------------+
394 | vSwitch Huge Pages: Size (MiB) | 1024 |
395 +-------------------------------------+--------------------------------------+
396 | Total | 0 |
397 +-------------------------------------+--------------------------------------+
398 | Available | 0 |
399 +-------------------------------------+--------------------------------------+
400 | Required | None |
401 +-------------------------------------+--------------------------------------+
402 | Application Pages (4K): Total | None |
403 +-------------------------------------+--------------------------------------+
404 | Application Huge Pages (2M): Total | 0 |
405 +-------------------------------------+--------------------------------------+
406 | Available | 0 |
407 +-------------------------------------+--------------------------------------+
408 | Application Huge Pages (1G): Total | 0 |
409 +-------------------------------------+--------------------------------------+
410 | Total Pending | 7 |
411 +-------------------------------------+--------------------------------------+
412 | Available | 0 |
413 +-------------------------------------+--------------------------------------+
414 | uuid | 15702f44-2281-41f3-8c83-a1e9a300ad01 |
415 +-------------------------------------+--------------------------------------+
416 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
417 +-------------------------------------+--------------------------------------+
418 | inode_uuid | d52648bc-dfa6-4214-afa3-5d2148ee4180 |
419 +-------------------------------------+--------------------------------------+
420 | created_at | 2020-06-03T08:21:27.468357+00:00 |
421 +-------------------------------------+--------------------------------------+
422 | updated_at | 2020-06-03T10:10:49.159080+00:00 |
423 +-------------------------------------+--------------------------------------+
424
425
426 [sysadmin@controller-0 ~(keystone_admin)]$ system host-memory-modify $WORKER -1G 7 1
427
428 +-------------------------------------+--------------------------------------+
429 | Property | Value |
430 +-------------------------------------+--------------------------------------+
431 | Memory: Usable Total (MiB) | 0 |
432 +-------------------------------------+--------------------------------------+
433 | Platform (MiB) | 1000 |
434 +-------------------------------------+--------------------------------------+
435 | Available (MiB) | 0 |
436 +-------------------------------------+--------------------------------------+
437 | Huge Pages Configured | True |
438 +-------------------------------------+--------------------------------------+
439 | vSwitch Huge Pages: Size (MiB) | 1024 |
440 +-------------------------------------+--------------------------------------+
441 | Total | 0 |
442 +-------------------------------------+--------------------------------------+
443 | Available | 0 |
444 +-------------------------------------+--------------------------------------+
445 | Required | None |
446 +-------------------------------------+--------------------------------------+
447 | Application Pages (4K): Total | None |
448 +-------------------------------------+--------------------------------------+
449 | Application Huge Pages (2M): Total | 0 |
450 +-------------------------------------+--------------------------------------+
451 | Available | 0 |
452 +-------------------------------------+--------------------------------------+
453 | Application Huge Pages (1G): Total | 0 |
454 +-------------------------------------+--------------------------------------+
455 | Total Pending | 7 |
456 +-------------------------------------+--------------------------------------+
457 | Available | 0 |
458 +-------------------------------------+--------------------------------------+
459 | uuid | 544580b9-0f2a-41a1-b4e3-90fdf5c8f7e0 |
460 +-------------------------------------+--------------------------------------+
461 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
462 +-------------------------------------+--------------------------------------+
463 | inode_uuid | e02c3c41-03b7-4f49-8244-bca851fa2d2f |
464 +-------------------------------------+--------------------------------------+
465 | created_at | 2020-06-03T08:21:27.566279+00:00 |
466 +-------------------------------------+--------------------------------------+
467 | updated_at | 2020-06-03T10:10:49.220907+00:00 |
468 +-------------------------------------+--------------------------------------+
469
470
471 [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET0='physnet0'
472
473 [sysadmin@controller-0 ~(keystone_admin)]$ PHYSNET1='physnet1'
474
475 [sysadmin@controller-0 ~(keystone_admin)]$ WORKER=controller-0
476
477 [sysadmin@controller-0 ~(keystone_admin)]$
478
479 [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET0} flat
480
481 +--------------+--------------------------------------+
482 | Property | Value |
483 +--------------+--------------------------------------+
484 | id | 1 |
485 +--------------+--------------------------------------+
486 | uuid | 0ffe0b0a-35ff-45b2-9a62-63a494d2d18b |
487 +--------------+--------------------------------------+
488 | name | physnet0 |
489 +--------------+--------------------------------------+
490 | network_type | flat |
491 +--------------+--------------------------------------+
492 | mtu | 1500 |
493 +--------------+--------------------------------------+
494 | description | None |
495 +--------------+--------------------------------------+
496
497 [sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-add ${PHYSNET1} flat
498
499 +--------------+--------------------------------------+
500 | Property | Value |
501 +--------------+--------------------------------------+
502 | id | 2 |
503 +--------------+--------------------------------------+
504 | uuid | 899ca8ba-6800-4d62-aed4-3770878da738 |
505 +--------------+--------------------------------------+
506 | name | physnet1 |
507 +--------------+--------------------------------------+
508 | network_type | flat |
509 +--------------+--------------------------------------+
510 | mtu | 1500 |
511 +--------------+--------------------------------------+
512 | description | None |
513 +--------------+--------------------------------------+
514
515
516 [sysadmin@controller-0 ~(keystone_admin)]$ system host-port-list controller-0
517
518 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
519 | uuid | name | type | pci address | device | processor | accelerated | device type |
520 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
521 | 7007dfb1-ea55-4355-9d27-3ed4c6696283 | eno1 | ethernet | 0000:02:00.0 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
522 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
523 | 51f62ecf-7897-43e3-beb8-46ae5550ef05 | eno2 | ethernet | 0000:02:00.1 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
524 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
525 | e987725e-ab45-4151-b53d-94ba21706eb4 | eno3 | ethernet | 0000:02:00.2 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
526 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
527 | f4a8115a-ca0b-4336-a634-2cad77c5a88c | eno4 | ethernet | 0000:02:00.3 | 0 | 0 | False | NetXtreme BCM5719 Gigabit Ethernet PCIe [1657] |
528 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
529 | 88965882-c816-42e8-ba0e-d3ff6f9f553e | eno49 | ethernet | 0000:04:00.0 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] |
530 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
531 | 98e2cedc-9ef8-4ca0-b3e0-db3a2bd0273a | eno50 | ethernet | 0000:04:00.1 | 0 | 0 | False | OneConnect NIC (Skyhawk) [0720] |
532 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
533 | e75d7aa6-7832-4ec4-9dd4-ea8300773fd8 | ens2f0 | ethernet | 0000:05:00.0 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] |
534 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
535 | 8ac4ffca-4a61-4b47-8e17-171f2e6cea66 | ens2f1 | ethernet | 0000:05:00.1 | 0 | 0 | True | 82599ES 10-Gigabit SFI/SFP+ Network Connection [10fb] |
536 +--------------------------------------+--------+----------+--------------+--------+-----------+-------------+-------------------------------------------------------+
537
538
539 [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f0
540
541 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov0 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF
542
543 +-----------------+--------------------------------------+
544 | Property | Value |
545 +-----------------+--------------------------------------+
546 | ifname | sriov0 |
547 +-----------------+--------------------------------------+
548 | iftype | ethernet |
549 +-----------------+--------------------------------------+
550 | ports | [u'ens2f0'] |
551 +-----------------+--------------------------------------+
552 | imac | 00:1b:21:b3:46:2c |
553 +-----------------+--------------------------------------+
554 | imtu | 1500 |
555 +-----------------+--------------------------------------+
556 | ifclass | pci-sriov |
557 +-----------------+--------------------------------------+
558 | aemode | None |
559 +-----------------+--------------------------------------+
560 | schedpolicy | None |
561 +-----------------+--------------------------------------+
562 | txhashpolicy | None |
563 +-----------------+--------------------------------------+
564 | uuid | 9450b1ae-d1a1-4ecb-811a-fa3cb1cdf362 |
565 +-----------------+--------------------------------------+
566 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
567 +-----------------+--------------------------------------+
568 | vlan_id | None |
569 +-----------------+--------------------------------------+
570 | uses | [] |
571 +-----------------+--------------------------------------+
572 | used_by | [] |
573 +-----------------+--------------------------------------+
574 | created_at | 2020-06-03T12:34:11.633254+00:00 |
575 +-----------------+--------------------------------------+
576 | updated_at | 2020-06-04T02:01:28.168127+00:00 |
577 +-----------------+--------------------------------------+
578 | sriov_numvfs | 6 |
579 +-----------------+--------------------------------------+
580 | sriov_vf_driver | vfio |
581 +-----------------+--------------------------------------+
582 | accelerated | [True] |
583 +-----------------+--------------------------------------+
584
585 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov0 ${PHYSNET0}
586
587 +------------------+--------------------------------------+
588 | Property | Value |
589 +------------------+--------------------------------------+
590 | hostname | controller-0 |
591 +------------------+--------------------------------------+
592 | uuid | 1d35219d-3258-42e7-8f0e-41d865710d58 |
593 +------------------+--------------------------------------+
594 | ifname | sriov0 |
595 +------------------+--------------------------------------+
596 | datanetwork_name | physnet0 |
597 +------------------+--------------------------------------+
598
599
600
601 [sysadmin@controller-0 ~(keystone_admin)]$ INTL_IF=ens2f1
602
603 [sysadmin@controller-0 ~(keystone_admin)]$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov -N 6 --vf-driver=vfio $WORKER $INTL_IF
604
605 +-----------------+--------------------------------------+
606 | Property | Value |
607 +-----------------+--------------------------------------+
608 | ifname | sriov1 |
609 +-----------------+--------------------------------------+
610 | iftype | ethernet |
611 +-----------------+--------------------------------------+
612 | ports | [u'ens2f1'] |
613 +-----------------+--------------------------------------+
614 | imac | 00:1b:21:b3:46:2d |
615 +-----------------+--------------------------------------+
616 | imtu | 1500 |
617 +-----------------+--------------------------------------+
618 | ifclass | pci-sriov |
619 +-----------------+--------------------------------------+
620 | aemode | None |
621 +-----------------+--------------------------------------+
622 | schedpolicy | None |
623 +-----------------+--------------------------------------+
624 | txhashpolicy | None |
625 +-----------------+--------------------------------------+
626 | uuid | 13d5ee99-0399-4b6b-a359-0f96660afdc4 |
627 | ihost_uuid | 08c23843-afa4-4b80-a285-be126365276e |
628 | vlan_id | None |
629 | uses | [] |
630 | used_by | [] |
631 | created_at | 2020-06-03T12:34:12.072624+00:00 |
632 | updated_at | 2020-06-04T02:03:51.405322+00:00 |
633 | sriov_numvfs | 6 |
634 | sriov_vf_driver | vfio |
635 | accelerated | [True] |
636 +-----------------+--------------------------------------+
637
638 [sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-assign ${WORKER} sriov1 ${PHYSNET1}
639
640 +------------------+--------------------------------------+
641 | Property | Value |
642 +------------------+--------------------------------------+
643 | hostname | controller-0 |
644 +------------------+--------------------------------------+
645 | uuid | a87c6a5f-0c53-4ebf-9a32-38366700426f |
646 +------------------+--------------------------------------+
647 | ifname | sriov1 |
648 +------------------+--------------------------------------+
649 | datanetwork_name | physnet1 |
650 +------------------+--------------------------------------+
651
6522.10 Unlock the server
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]653''''''''''''''''''''''
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]654
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]655Example:
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]656
657::
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]658
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]659 [sysadmin@controller-0 ~(keystone_admin)]$ system host-unlock controller-0
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]660
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]661 +-----------------------+-------------------------------------------------+
662 | Property | Value |
663 +-----------------------+-------------------------------------------------+
664 | action | none |
665 +-----------------------+-------------------------------------------------+
666 | administrative | locked |
667 +-----------------------+-------------------------------------------------+
668 | availability | online |
669 +-----------------------+-------------------------------------------------+
670 | bm_ip | None |
671 +-----------------------+-------------------------------------------------+
672 | bm_type | none |
673 +-----------------------+-------------------------------------------------+
674 | bm_username | None |
675 +-----------------------+-------------------------------------------------+
676 | boot_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 |
677 +-----------------------+-------------------------------------------------+
678 | capabilities | {u'stor_function': u'monitor'} |
679 +-----------------------+-------------------------------------------------+
680 | clock_synchronization | ntp |
681 +-----------------------+-------------------------------------------------+
682 | config_applied | 3daf20c7-c032-4aa2-839d-93da92051bea |
683 +-----------------------+-------------------------------------------------+
684 | config_status | Config out-of-date |
685 +-----------------------+-------------------------------------------------+
686 | config_target | bdaf20c7-c032-4aa2-839d-93da92051bea |
687 +-----------------------+-------------------------------------------------+
688 | console | tty0 |
689 +-----------------------+-------------------------------------------------+
690 | created_at | 2020-06-03T08:20:59.372602+00:00 |
691 +-----------------------+-------------------------------------------------+
692 | hostname | controller-0 |
693 +-----------------------+-------------------------------------------------+
694 | id | 1 |
695 +-----------------------+-------------------------------------------------+
696 | install_output | graphical |
697 +-----------------------+-------------------------------------------------+
698 | install_state | None |
699 +-----------------------+-------------------------------------------------+
700 | install_state_info | None |
701 +-----------------------+-------------------------------------------------+
702 | inv_state | inventoried |
703 +-----------------------+-------------------------------------------------+
704 | invprovision | provisioning |
705 +-----------------------+-------------------------------------------------+
706 | location | {} |
707 +-----------------------+-------------------------------------------------+
708 | mgmt_ip | 192.168.204.2 |
709 +-----------------------+-------------------------------------------------+
710 | mgmt_mac | 00:00:00:00:00:00 |
711 +-----------------------+-------------------------------------------------+
712 | operational | disabled |
713 +-----------------------+-------------------------------------------------+
714 | personality | controller |
715 +-----------------------+-------------------------------------------------+
716 | reserved | False |
717 +-----------------------+-------------------------------------------------+
718 | rootfs_device | /dev/disk/by-path/pci-0000:03:00.0-scsi-0:1:0:0 |
719 +-----------------------+-------------------------------------------------+
720 | serialid | None |
721 +-----------------------+-------------------------------------------------+
722 | software_load | 19.12 |
723 +-----------------------+-------------------------------------------------+
724 | subfunction_avail | online |
725 +-----------------------+-------------------------------------------------+
726 | subfunction_oper | disabled |
727 +-----------------------+-------------------------------------------------+
728 | subfunctions | controller,worker |
729 +-----------------------+-------------------------------------------------+
730 | task | Unlocking |
731 +-----------------------+-------------------------------------------------+
732 | tboot | false |
733 +-----------------------+-------------------------------------------------+
734 | ttys_dcd | None |
735 +-----------------------+-------------------------------------------------+
736 | updated_at | 2020-06-04T02:04:59.514854+00:00 |
737 +-----------------------+-------------------------------------------------+
738 | uptime | 49475 |
739 +-----------------------+-------------------------------------------------+
740 | uuid | 08c23843-afa4-4b80-a285-be126365276e |
741 +-----------------------+-------------------------------------------------+
742 | vim_progress_status | None |
743 +-----------------------+-------------------------------------------------+
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]744
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]7453. Simple use case for sriov
746````````````````````````````
747
7483.1 After controller-0 is rebooted and up running, download the DPDK
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]749''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]750
751::
752
753 [sysadmin@controller-0 ~(keystone_admin)]$ cd /opt
754 [sysadmin@controller-0 opt(keystone_admin)]$ sudo wget https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
755 Password:
756 --2020-06-04 02:35:30-- https://fast.dpdk.org/rel/dpdk-17.11.10.tar.xz
757 Resolving fast.dpdk.org... 151.101.2.49, 151.101.66.49, 151.101.130.49, ...
758 Connecting to fast.dpdk.org|151.101.2.49|:443... connected.
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]759
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]760 HTTP request sent, awaiting response... 200 OK
761 Length: 10251680 (9.8M) [application/octet-stream]
762 Saving to: ‘dpdk-17.11.10.tar.xz’
763
764 dpdk-17.11.10.tar.xz 100%
765 [========================================================================================>] 9.78M
766 1.48MB/s in 6.8s
767
768 2020-06-04 02:35:40 (1.43 MB/s) - ‘dpdk-17.11.10.tar.xz’ saved [10251680/10251680]
769
770 sudo tar xvf dpdk-17.11.10.tar.xz
771
772 sudo ln -s dpdk-stable-17.11.10 dpdk-stable
773
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]7743.2 Prepare the yaml file for the network assignment container
775''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]776
777The following the exmaple of the yaml file:
778
779::
780
781 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > netdef-data-dpdk.yaml
782 > apiVersion: "k8s.cni.cncf.io/v1"
783 > kind: NetworkAttachmentDefinition
784 > metadata:
785 > name: sriov-data-dpdk-0
786 > annotations:
787 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet0
788 > spec:
789 > config: '{
790 > "type": "sriov",
791 > "name": "sriov-data-dpdk-0"
792 > }'
793 >
794 > ---
795 > apiVersion: "k8s.cni.cncf.io/v1"
796 > kind: NetworkAttachmentDefinition
797 > metadata:
798 > name: sriov-data-dpdk-1
799 > annotations:
800 > k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_physnet1
801 > spec:
802 > config: '{
803 > "type": "sriov",
804 > "name": "sriov-data-dpdk-1"
805 > }'
806 > EOF
807
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]8083.3 Run the network assignent container for the 2 VFs
809'''''''''''''''''''''''''''''''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]810
811::
812
813 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f netdef-data-dpdk.yaml
814 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-0 created
815 networkattachmentdefinition.k8s.cni.cncf.io/sriov-data-dpdk-1 created
816
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]8173.4 Prepare the VF container yaml file
818''''''''''''''''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]819
820::
821
822 [sysadmin@controller-0 sriov(keystone_admin)]$ cat <<EOF > pod-with-dpdk-vfs-0.yaml
823 > apiVersion: v1
824 > kind: Pod
825 metadata:
826 > metadata:
827 > name: pod-with-dpdk-vfs-0
828 > annotations:
829 > k8s.v1.cni.cncf.io/networks: '[
830 > { "name": "sriov-data-dpdk-0" },
831 { "name": "sriov-data-dpdk-1" }
832 > { "name": "sriov-data-dpdk-1" }
833 > ]'
834 > spec:
835 > restartPolicy: Never
836 > containers:
837 > - name: pod-with-dpdk-vfs-0
838 > image: wrsnfv/ubuntu-dpdk-build:v0.3
839 > env:
840 > - name: RTE_SDK
841 > value: "/usr/src/dpdk"
842 > command:
843 > - sleep
844 > - infinity
845 > stdin: true
846 > tty: true
847 > securityContext:
848 > privileged: true
849 > capabilities:
850 > add:
851 > - ALL
852 > resources:
853 > requests:
854 > cpu: 4
855 > memory: 4Gi
856 > intel.com/pci_sriov_net_physnet0: '1'
857 > intel.com/pci_sriov_net_physnet1: '1'
858 > limits:
859 > cpu: 4
860 > hugepages-1Gi: 2Gi
861 > memory: 4Gi
862 > intel.com/pci_sriov_net_physnet0: '1'
863 > intel.com/pci_sriov_net_physnet1: '1'
864 > volumeMounts:
865 > - mountPath: /mnt/huge-1048576kB
866 > name: hugepage
867 > - name: dpdk-volume
868 > mountPath: /usr/src/dpdk
869 > - name: lib-volume
870 > mountPath: /lib/modules
871 > - name: src-volume
872 > mountPath: /usr/src/
873 > volumes:
874 > - name: hugepage
875 > emptyDir:
876 > medium: HugePages
877 > - name: dpdk-volume
878 > hostPath:
879 > path: /opt/dpdk-stable/
880 > - name: lib-volume
881 > hostPath:
882 > path: /lib/modules
883 > - name: src-volume
884 > hostPath:
885 > path: /usr/src/
886 > EOF
887
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]8883.5 Run the VF container
889''''''''''''''''''''''''
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]890
891Start the VF container.
892
893::
894
895 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl create -f pod-with-dpdk-vfs-0.yaml
896 pod/pod-with-dpdk-vfs-0 created
897
898 [sysadmin@controller-0 sriov(keystone_admin)]$ kubectl get pod
899 NAME READY STATUS RESTARTS AGE
900 pod-with-dpdk-vfs-0 1/1 Running 0 6m40s
901
902Login the VF container
903
904::
905
906 kubectl exec -it pod-with-dpdk-vfs-0 -- bash
907
908Build the DPDK
909
910::
911
912 cd /lib/modules/5.0.19-rt11-yocto-preempt-rt/build
913
914 root@pod-with-dpdk-vfs-0:/lib/modules/5.0.19-rt11-yocto-preempt-rt/build# make prepare
915 HOSTCC scripts/basic/fixdep
916 HOSTCC scripts/kconfig/conf.o
917 HOSTCC scripts/kconfig/confdata.o
918 HOSTCC scripts/kconfig/expr.o
919 HOSTCC scripts/kconfig/symbol.o
920 HOSTCC scripts/kconfig/preprocess.o
921 HOSTCC scripts/kconfig/zconf.lex.o
922 HOSTCC scripts/kconfig/zconf.tab.o
923 HOSTLD scripts/kconfig/conf
924 scripts/kconfig/conf --syncconfig Kconfig
925 HOSTCC arch/x86/tools/relocs_32.o
926 HOSTCC arch/x86/tools/relocs_64.o
927 HOSTCC arch/x86/tools/relocs_common.o
928 HOSTLD arch/x86/tools/relocs
929 HOSTCC scripts/genksyms/genksyms.o
930 YACC scripts/genksyms/parse.tab.c
931 HOSTCC scripts/genksyms/parse.tab.o
932 LEX scripts/genksyms/lex.lex.c
933 YACC scripts/genksyms/parse.tab.h
934 HOSTCC scripts/genksyms/lex.lex.o
935 HOSTLD scripts/genksyms/genksyms
936 HOSTCC scripts/bin2c
937 HOSTCC scripts/kallsyms
938 HOSTCC scripts/conmakehash
939 HOSTCC scripts/recordmcount
940 HOSTCC scripts/sortextable
941 HOSTCC scripts/asn1_compiler
942 HOSTCC scripts/sign-file
943 HOSTCC scripts/extract-cert
944 CC scripts/mod/empty.o
945 HOSTCC scripts/mod/mk_elfconfig
946 MKELF scripts/mod/elfconfig.h
947 HOSTCC scripts/mod/modpost.o
948 CC scripts/mod/devicetable-offsets.s
949 UPD scripts/mod/devicetable-offsets.h
950 HOSTCC scripts/mod/file2alias.o
951 HOSTCC scripts/mod/sumversion.o
952 HOSTLD scripts/mod/modpost
953 CC kernel/bounds.s
954 CC arch/x86/kernel/asm-offsets.s
955 CALL scripts/checksyscalls.sh
956
957Build the test_pmd application
958
959::
960
961 cd $RTE_SDK
962 ./usertools/dpdk-setup.sh
963 Option: 14
964 CC config.o
965 CC iofwd.o
966 CC macfwd.o
967 CC macswap.o
968 CC flowgen.o
969 CC rxonly.o
970 CC txonly.o
971 CC csumonly.o
972 CC icmpecho.o
973 CC tm.o
974 LD testpmd
975 INSTALL-APP testpmd
976 INSTALL-MAP testpmd.map
977 == Build app/proc_info
978 CC main.o
979 LD dpdk-procinfo
980 INSTALL-APP dpdk-procinfo
981 INSTALL-MAP dpdk-procinfo.map
982 == Build app/pdump
983 CC main.o
984 LD dpdk-pdump
985 INSTALL-APP dpdk-pdump
986 INSTALL-MAP dpdk-pdump.map
987 == Build app/test-crypto-perf
988 CC main.o
989 CC cperf_ops.o
990 CC cperf_options_parsing.o
991 CC cperf_test_vectors.o
992 CC cperf_test_throughput.o
993 CC cperf_test_latency.o
994 CC cperf_test_pmd_cyclecount.o
995 CC cperf_test_verify.o
996 CC cperf_test_vector_parsing.o
997 CC cperf_test_common.o
998 LD dpdk-test-crypto-perf
999 INSTALL-APP dpdk-test-crypto-perf
1000 INSTALL-MAP dpdk-test-crypto-perf.map
1001 == Build app/test-eventdev
1002 CC evt_main.o
1003 CC evt_options.o
1004 CC evt_test.o
1005 CC parser.o
1006 CC test_order_common.o
1007 CC test_order_queue.o
1008 CC test_order_atq.o
1009 CC test_perf_common.o
1010 CC test_perf_queue.o
1011 CC test_perf_atq.o
1012 LD dpdk-test-eventdev
1013 INSTALL-APP dpdk-test-eventdev
1014 INSTALL-MAP dpdk-test-eventdev.map
1015 Build complete [x86_64-native-linuxapp-gcc]
1016 Installation cannot run with T defined and DESTDIR undefined
1017 ------------------------------------------------------------------------------
1018 RTE_TARGET exported as x86_64-native-linuxapp-gcc
1019 ------------------------------------------------------------------------------
1020
1021 Press enter to continue ...
1022
1023Check the VF PCI information:
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]1024
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1025::
1026
1027 root@pod-with-dpdk-vfs-0:/usr/src/dpdk# printenv | grep PCIDEVICE_INTEL_COM
1028 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET1=0000:05:11.1
1029 PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_PHYSNET0=0000:05:11.0
1030
1031Exit from pod back to host to find which VFs are assigned to this pod by check the pci address:
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]1032
Xiaohua Zhang8ff63e92020-06-01 23:24:41 -0400[diff] [blame]1033::
1034
1035 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f0/device/virtfn*
1036 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn0 -> ../0000:05:10.0
1037 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn1 -> ../0000:05:10.2
1038 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn2 -> ../0000:05:10.4
1039 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn3 -> ../0000:05:10.6
1040 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn4 -> ../0000:05:11.0
1041 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f0/device/virtfn5 -> ../0000:05:11.2
1042
1043 [root@controller-0 sysadmin(keystone_admin)]# ls -l /sys/class/net/ens2f1/device/virtfn*
1044 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn0 -> ../0000:05:10.1
1045 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn1 -> ../0000:05:10.3
1046 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn2 -> ../0000:05:10.5
1047 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn3 -> ../0000:05:10.7
1048 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn4 -> ../0000:05:11.1
1049 lrwxrwxrwx 1 root root 0 Jun 4 02:12 /sys/class/net/ens2f1/device/virtfn5 -> ../0000:05:11.3
1050
1051 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f0 vf 4 mac 9e:fd:e6:dd:c1:01
1052 [root@controller-0 sysadmin(keystone_admin)]# sudo ip link set ens2f1 vf 4 mac 9e:fd:e6:dd:c1:02
1053
1054
1055Get back to pod and run the test_pmd
1056
1057::
1058
1059 cd $RTE_SDK/x86_64-native-linuxapp-gcc/app/
1060
1061 taskset -p --cpu-list 1
1062 pid 1's current affinity list: 2,3,18,19
1063
1064 ./testpmd --socket-mem 1024,1024 -l 2,3 -w 0000:05:11.0 -w 0000:05:11.1 --file-prefix=testpmd_ -- --auto-
1065 start --tx-first --stats-period 1 --disable-hw-vlan --eth-peer=0,"9e:fd:e6:dd:c1:02" --eth-
1066 peer=1,"9e:fd:e6:dd:c1:01"
1067 EAL: Detected 32 lcore(s)
1068 EAL: No free hugepages reported in hugepages-2048kB
1069 EAL: Probing VFIO support...
1070 EAL: VFIO support initialized
1071 EAL: PCI device 0000:05:11.0 on NUMA socket 0
1072 EAL: probe driver: 8086:10ed net_ixgbe_vf
1073 EAL: using IOMMU type 1 (Type 1)
1074 EAL: PCI device 0000:05:11.1 on NUMA socket 0
1075 EAL: probe driver: 8086:10ed net_ixgbe_vf
1076 Auto-start selected
1077 Ports to start sending a burst of packets first
1078 Warning: lsc_interrupt needs to be off when using tx_first. Disabling.
1079 USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=155456, size=2176, socket=0
1080 Configuring Port 0 (socket 0)
1081 Port 0: 9E:FD:E6:DD:C1:01
1082 Configuring Port 1 (socket 0)
1083 Port 1: 9E:FD:E6:DD:C1:02
1084 Checking link statuses...
1085 Port0 Link Up. speed 10000 Mbps- full-duplex
1086 Port1 Link Up. speed 10000 Mbps- full-duplex
1087 Done
1088 No commandline core given, start packet forwarding
1089 io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP over anonymous pages
1090 disabled
1091 Logical Core 3 (socket 0) forwards packets on 2 streams:
1092 RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:01
1093 RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=9E:FD:E6:DD:C1:02
1094
1095 io packet forwarding packets/burst=32
1096 nb forwarding cores=1 - nb forwarding ports=2
1097 port 0:
1098 CRC stripping enabled
1099 RX queues=1 - RX desc=128 - RX free threshold=32
1100 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1101 TX queues=1 - TX desc=512 - TX free threshold=32
1102 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1103 TX RS bit threshold=32 - TXQ flags=0xf01
1104 port 1:
1105 CRC stripping enabled
1106 RX queues=1 - RX desc=128 - RX free threshold=32
1107 RX threshold registers: pthresh=8 hthresh=8 wthresh=0
1108 TX queues=1 - TX desc=512 - TX free threshold=32
1109 TX threshold registers: pthresh=32 hthresh=0 wthresh=0
1110 TX RS bit threshold=32 - TXQ flags=0xf01
1111
1112 Port statistics ====================================
1113 ######################## NIC statistics for port 0 ########################
1114 RX-packets: 56 RX-missed: 0 RX-bytes: 4096
1115 RX-errors: 0
1116 RX-nombuf: 0
1117 TX-packets: 64 TX-errors: 0 TX-bytes: 4096
1118
1119 Throughput (since last show)
1120 Rx-pps: 0
1121 Tx-pps: 0
1122 ############################################################################
1123
1124 ######################## NIC statistics for port 1 ########################
1125 RX-packets: 432 RX-missed: 0 RX-bytes: 27712
1126 RX-errors: 0
1127 RX-nombuf: 0
1128 TX-packets: 461 TX-errors: 0 TX-bytes: 30080
1129
1130 Throughput (since last show)
1131 Rx-pps: 0
1132 Tx-pps: 0
1133 ############################################################################
1134
1135 Port statistics ====================================
1136 ######################## NIC statistics for port 0 ########################
1137 RX-packets: 14124641 RX-missed: 0 RX-bytes: 903977344
1138 RX-errors: 0
1139 RX-nombuf: 0
1140 TX-packets: 14170205 TX-errors: 0 TX-bytes: 906893376
1141
1142 Throughput (since last show)
1143 Rx-pps: 7068409
1144 Tx-pps: 7091206
1145 ############################################################################
1146
1147
1148
1149
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]1150
Bin Yange71c66a2020-03-27 04:03:43 +0000[diff] [blame]1151
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]1152References
1153----------
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]1154
1155- `StarlingX`_
Jackie Huang799759c2019-11-13 15:45:51 +0800[diff] [blame]1156
Xiaohua Zhang4cf6f2d2020-06-01 20:34:21 -0400[diff] [blame]1157.. _`StarlingX`: https://docs.starlingx.io/
Xiaohua Zhangd6dfaa52020-06-02 08:20:25 -0400[diff] [blame^]1158