blob: 2359b0cb84c4109dcc270a42e163a969df0125ba [file] [log] [blame]
Petr Ospalýbe81ab02019-02-14 21:30:31 +01001.. This work is licensed under a Creative Commons Attribution 4.0 International License.
2.. http://creativecommons.org/licenses/by/4.0
Bartek Grzybowskid549d822021-03-22 13:12:01 +01003.. Copyright 2021 Samsung Electronics Co., Ltd.
Petr Ospalýbe81ab02019-02-14 21:30:31 +01004
Bartek Grzybowskid549d822021-03-22 13:12:01 +01005Offline Installer - Installation Guide
6======================================
Petr Ospalýbe81ab02019-02-14 21:30:31 +01007
Bartek Grzybowski2936ef12021-03-23 14:15:33 +01008This document describes offline installation procedure for `OOM ONAP`_, which is done by the ansible based `Offline installer`_.
Petr Ospalýbe81ab02019-02-14 21:30:31 +01009
Bartek Grzybowski553a9202021-03-23 15:09:11 +010010Before you begin the installation process you should prepare the offline installation packages. Please refer to the `Build Guide`_ for instructions on how to create them.
Petr Ospalýbe81ab02019-02-14 21:30:31 +010011
12-----
13
Petr Ospalýbe81ab02019-02-14 21:30:31 +010014Part 1. Prerequisites
15---------------------
16
Bartek Grzybowski553a9202021-03-23 15:09:11 +010017ONAP platform has certain software requirements - see `Software requirements`_ and minimum hardware recommendations: ``224 GB RAM``, ``112 vCPUs`` and ``160GB`` of storage (see `Hardware requirements`_). The minimum count of nodes should not drop below three - otherwise you may have to do some tweaking to make it work, which is not covered here (there is a pod count limit for a single kubernetes node - you can read more about it in this `discussion <https://lists.onap.org/g/onap-discuss/topic/oom_110_kubernetes_pod/25213556>`_).
Petr Ospalýbe81ab02019-02-14 21:30:31 +010018
Petr Ospalýbe81ab02019-02-14 21:30:31 +010019Kubernetes cluster
20~~~~~~~~~~~~~~~~~~
21
22The four nodes/VMs will be running these services:
23
24- **infra-node**::
25
26 - nexus
27 - nginx proxy
28 - dns
Michal Ptacek26278df2019-07-09 10:46:26 +000029 - kubernetes-etcd
30 - kubernetes-control-plane
Bartek Grzybowski553a9202021-03-23 15:09:11 +010031 - chartmuseum (if using helm v3)
Michal Ptacek26278df2019-07-09 10:46:26 +000032
Bartek Grzybowskia6ced382021-03-24 14:30:39 +010033**NOTE:** kubernetes-* control plane can be colocated directly with k8s nodes and not necessarily on infra node.
Petr Ospalýbe81ab02019-02-14 21:30:31 +010034
35- **kubernetes node 1-3**::
36
Michal Ptacek26278df2019-07-09 10:46:26 +000037 - kubernetes worker
Petr Ospalýbe81ab02019-02-14 21:30:31 +010038
Bartek Grzybowski553a9202021-03-23 15:09:11 +010039You don't need to care about these services now - that is the responsibility of the installer (described below). Just start four VMs as seen in below table (or according to your needs as we hinted above):
Petr Ospalýbe81ab02019-02-14 21:30:31 +010040
41.. _Overview table of the kubernetes cluster:
42
43Kubernetes cluster overview
44^^^^^^^^^^^^^^^^^^^^^^^^^^^
45
Bartek Grzybowski553a9202021-03-23 15:09:11 +010046.. note:: Offline installer leverages `RKE`_ to provision kubernetes cluster. If you'd like to use different k8s installation method please exclude ``rke.yml`` ansible playbook from execution and provide your own.
Michal Ptacek26278df2019-07-09 10:46:26 +000047
Tomáš Levora2a355bb2019-10-10 14:04:08 +020048=================== ================== ==================== ============== ============ ===============
49KUBERNETES NODE OS NETWORK CPU RAM STORAGE
50=================== ================== ==================== ============== ============ ===============
Bartek Grzybowski632891d2021-09-24 12:25:33 +020051**infra-node** RHEL/CentOS 7.9 ``10.8.8.100/24`` ``8 vCPUs`` ``8 GB`` ``100 GB``
52**kube-node1** RHEL/CentOS 7.9 ``10.8.8.101/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB``
53**kube-node2** RHEL/CentOS 7.9 ``10.8.8.102/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB``
54**kube-node3** RHEL/CentOS 7.9 ``10.8.8.103/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB``
Tomáš Levora2a355bb2019-10-10 14:04:08 +020055SUM ``56 vCPUs`` ``176+ GB`` ``400 GB``
56=========================================================== ============== ============ ===============
Petr Ospalýbe81ab02019-02-14 21:30:31 +010057
Bartek Grzybowski632891d2021-09-24 12:25:33 +020058As of now, the offline installer supports only **RHEL 7.x** and **CentOS 7.9** distributions, with at least *@core* and *@base* package groups installed including *Mandatory* and *Default* package sets. So, your VMs should be preinstalled with this operating system - the hypervisor and platform can be of your choosing.
Petr Ospalýbe81ab02019-02-14 21:30:31 +010059
Bartek Grzybowski553a9202021-03-23 15:09:11 +010060We will expect from now on that you installed four VMs and they are connected to the shared network. All VMs must be reachable from *install-server* (below), which can be the hypervisor, *infra-node* or completely different host. But in either of these cases the *install-server* must be able to connect over ssh to all of these nodes.
Petr Ospalýbe81ab02019-02-14 21:30:31 +010061
Petr Ospalýbe81ab02019-02-14 21:30:31 +010062Install-server
63~~~~~~~~~~~~~~
64
65We will use distinct *install-server* and keep it separate from the four-node cluster. But if you wish so, you can use *infra-node* for this goal (if you use the default ``'chroot'`` option of the installer), but in that case double the size of the storage requirement!
66
67Prerequisites for the *install-server*:
68
69- packages described in `Build Guide`_
70- extra ``100 GB`` storage (to have space where to store these packages)
71- installed ``'chroot'`` and/or ``'docker'`` system commands
72- network connection to the nodes - especially functioning ssh client
73
74Our *install-server* will have ip: ``10.8.8.4``.
75
76**NOTE:** All the subsequent commands below, are executed from within this *install-server*.
77
78-----
79
Bartek Grzybowskia6ced382021-03-24 14:30:39 +010080Part 2. Configuration
81---------------------
Petr Ospalýbe81ab02019-02-14 21:30:31 +010082
Bartek Grzybowskia6ced382021-03-24 14:30:39 +010083All commands and setups described in this chapter *MUST* be run on the *install-server*. It's assumed here that all commands are run as ``root`` which is of course not necessary - you can use a regular user account. The ssh/ansible connection to the nodes will also expect that we are connecting as ``root`` - you need to elevate privileges to be able to install on them. Although it can be achieved by other means (sudo), we decided here to keep instructions simple.
Petr Ospalýbe81ab02019-02-14 21:30:31 +010084
Petr Ospalýbe81ab02019-02-14 21:30:31 +010085Installer packages
86~~~~~~~~~~~~~~~~~~
87
Bartek Grzybowskia6ced382021-03-24 14:30:39 +010088At this point you should have the installer packages already prepared (see `Build Guide`_):
Petr Ospalýbe81ab02019-02-14 21:30:31 +010089
Michal Ptacek26278df2019-07-09 10:46:26 +000090- sw_package.tar
91- resources_package.tar
92- aux_package.tar
Petr Ospalýbe81ab02019-02-14 21:30:31 +010093
Michal Ptacek26278df2019-07-09 10:46:26 +000094**NOTE:** ``'aux_package.tar'`` is optional and if you don't have use for it, you can ignore it.
Petr Ospalýbe81ab02019-02-14 21:30:31 +010095
Bartek Grzybowskia6ced382021-03-24 14:30:39 +010096Copy above packages to the ``/data`` directory on the *install-server* and then unpack the ``'sw_package.tar'`` to your home directory:
97
98::
Petr Ospalýbe81ab02019-02-14 21:30:31 +010099
100 $ mkdir ~/onap-offline-installer
Michal Ptacek26278df2019-07-09 10:46:26 +0000101 $ tar -C ~/onap-offline-installer -xf /data/sw_package.tar
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100102
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100103Application directory
104~~~~~~~~~~~~~~~~~~~~~
105
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100106Change the current directory to ``'ansible'``::
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100107
108 $ cd ~/onap-offline-installer/ansible
109
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100110You can see multiple files and directories inside - those are the *offline-installer* ansible playbooks.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100111
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100112If you created the ``'sw_package.tar'`` package according to the *Build Guide* then at least the following files should be present:
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100113
Denis Kasanica7702f22019-11-14 12:35:46 +0100114- ``application/application_configuration.yml``
115- ``inventory/hosts.yml``
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100116
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100117Following paragraphs describe fine-tuning of ``'inventory.yml'`` to reflect your VMs setup and ``'application_configuration.yml'`` to setup the provisioner itself.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100118
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100119hosts.yml
120~~~~~~~~~
121
122We need to setup the ``'hosts.yml'`` first, the template looks like this::
123
124 ---
125 # This group contains hosts with all resources (binaries, packages, etc.)
126 # in tarball.
127 all:
128 vars:
129 # this key is supposed to be generated during setup.yml playbook execution
130 # change it just when you have better one working for all nodes
131 ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
132 ansible_ssh_common_args: '-o StrictHostKeyChecking=no'
133
134 children:
135 resources:
136 hosts:
137 resource-host:
138 ansible_host: 10.8.8.5
139
140 # This is group of hosts where nexus, nginx, dns and all other required
141 # services are running.
142 infrastructure:
143 hosts:
144 infrastructure-server:
145 ansible_host: 10.8.8.13
146 #IP used for communication between infra and kubernetes nodes, must be specified.
147 cluster_ip: 10.8.8.13
148
149 # This is group of hosts which are/will be part of Kubernetes cluster.
150 kubernetes:
Michal Zeganf9ef0c12019-06-25 11:05:05 +0200151 children:
152 # This is a group of hosts containing kubernetes worker nodes.
153 kubernetes-node:
154 hosts:
155 kubernetes-node-1:
156 ansible_host: 10.8.8.19
157 #ip of the node that it uses for communication with k8s cluster.
158 cluster_ip: 10.8.8.19
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100159 # External ip of the node, used for access from outside of the cluster.
160 # Can be set to some kind of floating or public ip.
161 # If not set, cluster_ip is used for this purpose.
162 # external_ip: x.x.x.x
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100163
Michal Zeganf9ef0c12019-06-25 11:05:05 +0200164 # Group of hosts containing etcd cluster nodes.
165 # Defaults to infra.
166 kubernetes-etcd:
167 hosts:
168 infrastructure-server
169
170 # This is a group of hosts that are to be used as kubernetes control plane nodes.
171 # This means they host kubernetes api server, controller manager and scheduler.
172 # This example uses infra for this purpose, however note that any
173 # other host could be used including kubernetes nodes.
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100174 # cluster_ip needs to be set for hosts used as control planes, external_ip can also be used.
Michal Zeganf9ef0c12019-06-25 11:05:05 +0200175 kubernetes-control-plane:
176 hosts:
177 infrastructure-server
Bartek Grzybowskicf6797c2019-05-22 14:53:31 +0200178
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100179 nfs-server:
180 hosts:
181 kubernetes-node-1
182
183There is some ssh configuration under the ``'vars'`` section - we will deal with ssh setup a little bit later in the `SSH authentication`_.
184
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100185First you need to set the ip addresses and add a couple of kubernetes nodes to match your four-node cluster:
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100186
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100187- Under the ``'resource-host'`` set the ``'ansible_host'`` address to the ip of the host where the packages are stored - it must be reachable by ssh from the *install-server* (for ansible to run playbooks on it) **AND** *infra-node* (to extract resource data from *resource-host* to *infra-node* over ssh). In our scenario the *resource-host* is the same as the *install-server*: ``'10.8.8.4'``
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100188- Similarly, set the ``'ansible_host'`` to the address of the *infra-node* under the ``'infrastructure-server'``.
189- Copy the whole ``'kubernetes-node-1'`` subsection and paste it twice directly after. Change the numbers to ``'kubernetes-node-2'`` and ``'kubernetes-node-3'`` respectively and fix the addresses in the ``'ansible_host'`` variables again to match *kube-node1*, *kube-node2* and *kube-node3*.
190
191As you can see, there is another ``'cluster_ip'`` variable for each node - this serve as a designated node address in the kubernetes cluster. Make it the same as the respective ``'ansible_host'``.
192
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100193**NOTE:** In our simple setup we have only one interface per node, but that does not need to be a case for some other deployments - especially if we start to deal with a production usage. Basically, an ``'ansible_host'`` is an entry point for the *install-server's* ansible (*offline-installer*), but the kubernetes cluster can be communicating on a separate network to which *install-server* has no access. That is why we have this distinctive variable, so we can tell the installer that there is a different network, where we want to run the kubernetes traffic and what address each node has on such a network.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100194
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100195After applying all described changes, the ``'hosts.yml'`` should look similar to this::
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100196
197 ---
198 # This group contains hosts with all resources (binaries, packages, etc.)
199 # in tarball.
200 all:
201 vars:
202 # this key is supposed to be generated during setup.yml playbook execution
203 # change it just when you have better one working for all nodes
204 ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
205 ansible_ssh_common_args: '-o StrictHostKeyChecking=no'
206
207 children:
208 resources:
209 hosts:
210 resource-host:
211 ansible_host: 10.8.8.4
212
213 # This is group of hosts where nexus, nginx, dns and all other required
214 # services are running.
215 infrastructure:
216 hosts:
217 infrastructure-server:
Denis Kasanica7702f22019-11-14 12:35:46 +0100218 ansible_host: 10.8.8.100
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100219 #IP used for communication between infra and kubernetes nodes, must be specified.
220 cluster_ip: 10.8.8.100
221
222 # This is group of hosts which are/will be part of Kubernetes cluster.
223 kubernetes:
Michal Zeganf9ef0c12019-06-25 11:05:05 +0200224 children:
225 # This is a group of hosts containing kubernetes worker nodes.
226 kubernetes-node:
227 hosts:
228 kubernetes-node-1:
229 ansible_host: 10.8.8.101
230 #ip of the node that it uses for communication with k8s cluster.
231 cluster_ip: 10.8.8.101
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100232 # External ip of the node, used for access from outside of the cluster.
233 # Can be set to some kind of floating or public ip.
234 # If not set, cluster_ip is used for this purpose.
235 # external_ip: x.x.x.x
Michal Zeganf9ef0c12019-06-25 11:05:05 +0200236 kubernetes-node-2:
237 ansible_host: 10.8.8.102
238 #ip of the node that it uses for communication with k8s cluster.
239 cluster_ip: 10.8.8.102
240 kubernetes-node-3:
241 ansible_host: 10.8.8.103
242 #ip of the node that it uses for communication with k8s cluster.
243 cluster_ip: 10.8.8.103
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100244
Michal Zeganf9ef0c12019-06-25 11:05:05 +0200245 # Group of hosts containing etcd cluster nodes.
246 # Defaults to infra.
247 kubernetes-etcd:
248 hosts:
249 infrastructure-server
250
251 # This is a group of hosts that are to be used as kubernetes control plane nodes.
252 # This means they host kubernetes api server, controller manager and scheduler.
253 # This example uses infra for this purpose, however note that any
254 # other host could be used including kubernetes nodes.
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100255 # cluster_ip needs to be set for hosts used as control planes, external_ip can also be used.
Michal Zeganf9ef0c12019-06-25 11:05:05 +0200256 kubernetes-control-plane:
257 hosts:
258 infrastructure-server
Bartek Grzybowskicf6797c2019-05-22 14:53:31 +0200259
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100260 nfs-server:
261 hosts:
262 kubernetes-node-1
263
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100264application_configuration.yml
265~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
266
267Here, we will be interested in the following variables:
268
269- ``resources_dir``
270- ``resources_filename``
271- ``aux_resources_filename``
272- ``app_data_path``
273- ``aux_data_path``
274- ``app_name``
Bartek Grzybowski30b2cbf2019-03-26 16:10:10 +0100275- ``timesync``
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100276
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100277``'resource_dir'``, ``'resources_filename'`` and ``'aux_resources_filename'`` must correspond to the file paths on the *resource-host* (``'resource-host'`` in ``hosts.yml``), which in our case is the *install-server* host.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100278
Michal Ptacek26278df2019-07-09 10:46:26 +0000279The ``'resource_dir'`` should be set to ``'/data'``, ``'resources_filename'`` to ``'resources_package.tar'`` and ``'aux_resources_filename'`` to ``'aux_package.tar'``. The values should be the same as are in the `Installer packages`_ section.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100280
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100281``'app_data_path'`` is the absolute path on the *infra-node* to where the package ``'resources_package.tar'`` will be extracted and similarly ``'aux_data_path'`` is another absolute path for ``'aux_package.tar'``. Both paths are fully arbitrary, but they should point to the filesystem with enough disk space - the storage requirements are described in `Overview table of the kubernetes cluster`_.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100282
283**NOTE:** As we mentioned in `Installer packages`_ - the auxiliary package is not mandatory and we will not utilize it in here either.
284
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100285The ``'app_name'`` variable should be short and descriptive. We will set it simply to ``onap``.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100286
Bartek Grzybowski30b2cbf2019-03-26 16:10:10 +0100287The ``'timesync'`` variable is optional and controls synchronisation of the system clock on hosts. It should be configured only if a custom NTP server is available and needed. Such a time authority should be on a host reachable from all installation nodes. If this setting is not provided then the default behavior is to setup NTP daemon on infra-node and sync all kube-nodes' time with it.
288
289If you wish to provide your own NTP servers configure their IPs as follows::
290
291 timesync:
292 servers:
293 - <ip address of NTP_1>
294 - <...>
295 - <ip address of NTP_N>
296
297Another time adjustment related variables are ``'timesync.slewclock'`` and ``'timesync.timezone'`` .
298First one can have value of ``'true'`` or ``'false'`` (default). It controls whether (in case of big time difference compared to server) time should be adjusted gradually by slowing down or speeding up the clock as required (``'true'``) or in one step (``'false'``)::
299
300 timesync:
301 slewclock: true
302
303Second one controls time zone setting on host. It's value should be time zone name according to tz database names with ``'Universal'`` being the default one::
304
305 timesync.
306 timezone: UTC
307
308``'timesync.servers'``, ``'timesync.slewclock'`` and ``'timesync.timezone'`` settings can be used independently.
309
310Final configuration can resemble the following::
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100311
312 resources_dir: /data
Denis Kasanica7702f22019-11-14 12:35:46 +0100313 resources_filename: resources_package.tar
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100314 app_data_path: /opt/onap
315 app_name: onap
Bartek Grzybowski30b2cbf2019-03-26 16:10:10 +0100316 timesync:
317 servers:
318 - 192.168.0.1
319 - 192.168.0.2
320 slewclock: true
321 timezone: UTC
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100322
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100323Helm chart values overrides
324^^^^^^^^^^^^^^^^^^^^^^^^^^^
Michal Zegana579c982019-04-02 15:33:30 +0200325
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100326OOM charts are coming with all ONAP components disabled, this setting is also prepackaged within our sw_package.tar. Luckily there are multiple ways supported how to override this setting. It's also necessary for setting-up VIM specific entries and basically to configure any stuff with non default values.
Michal Ptacek26278df2019-07-09 10:46:26 +0000327
328First option is to use ``overrides`` key in ``application_configuration.yml``.
Michal Zegana579c982019-04-02 15:33:30 +0200329These settings will override helm values originally stored in ``values.yaml`` files in helm chart directories.
330
331For example, the following lines could be appended to ``application_configuration.yml`` to set up managed openstack credentials for onap's so component::
332
333 overrides:
334 so:
335 config:
336 openStackUserName: "os_user"
337 openStackRegion: "region_name"
338 openStackKeyStoneUrl: "keystone_url"
339 openStackEncryptedPasswordHere: "encrypted_password"
340
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100341In addition or alternatively to that one can configure ``helm_override_files`` variable in ``'application_configuration.yml'`` and mention all files with helm chart values there, e.g.:
342
343::
344
345 helm_override_files:
346 - "/path/to/values1.yaml"
347 - "/path/to/values2.yaml"
Michal Ptacek26278df2019-07-09 10:46:26 +0000348
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100349SSH authentication
350~~~~~~~~~~~~~~~~~~
351
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100352Finally you need to setup password-less login from *install-server* to the nodes.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100353
Bartek Grzybowskia6ced382021-03-24 14:30:39 +0100354You can use the ansible playbook ``'setup.yml'`` for that purpose::
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100355
Denis Kasanica7702f22019-11-14 12:35:46 +0100356 $ ./run_playbook.sh -i inventory/hosts.yml setup.yml -u root --ask-pass
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100357
358You will be asked for password per each node and the playbook will generate a unprotected ssh key-pair ``'~/.ssh/offline_ssh_key'``, which will be distributed to the nodes.
359
360Another option is to generate a ssh key-pair manually. We strongly advise you to protect it with a passphrase, but for simplicity we will showcase generating of a private key without any such protection::
361
362 $ ssh-keygen -N "" -f ~/.ssh/identity
363
364The next step will be to distribute the public key to these nodes and from that point no password is needed::
365
366 $ for ip in 100 101 102 103 ; do ssh-copy-id -i ~/.ssh/identity.pub root@10.8.8.${ip} ; done
367
368This command behaves almost identically to the ``'setup.yml'`` playbook.
369
370If you generated the ssh key manually then you can now run the ``'setup.yml'`` playbook like this and achieve the same result as in the first execution::
371
Denis Kasanica7702f22019-11-14 12:35:46 +0100372 $ ./run_playbook.sh -i inventory/hosts.yml setup.yml
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100373
374This time it should not ask you for any password - of course this is very redundant, because you just distributed two ssh keys for no good reason.
375
376We can finally edit and finish the configuration of the ``'hosts.yml'``:
377
378- if you used the ``'setup.yml'`` playbook then you can just leave this line as it is::
379
380 ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
381
382- if you created a ssh key manually then change it like this::
383
384 ansible_ssh_private_key_file: /root/.ssh/identity
385
386-----
387
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100388Part 3. Installation
389--------------------
390
391We should have the configuration complete and be ready to start the installation. The installation is done via ansible playbooks, which are run either inside a **chroot** environment (default) or from the **docker** container. If for some reason you want to run playbooks from the docker instead of chroot then you cannot use *infra-node* or any other *kube-node* as the *install-server* - otherwise you risk that installation will fail due to restarting of the docker service.
392
Bartek Grzybowskicf2c37a2021-03-24 14:58:08 +0100393``'sw_package.tar'`` should contain ``'ansible_chroot.tgz'`` file inside the ``'docker'`` directory. Detailed instructions on how to create it manually and to get more info about the scripts dealing with docker and chroot, go to `Appendix 1. Ansible execution/bootstrap`_.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100394
395We will use the default chroot option so we don't need any docker service to be running.
396
Bartek Grzybowskicf2c37a2021-03-24 14:58:08 +0100397Commence the installation process by running following command::
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100398
Denis Kasanica7702f22019-11-14 12:35:46 +0100399 $ ./run_playbook.sh -i inventory/hosts.yml -e @application/application_configuration.yml site.yml
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100400
Bartek Grzybowskicf2c37a2021-03-24 14:58:08 +0100401This will take a while so be patient. The whole provisioning process is idempotent so you may safely re-run it if required.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100402
Bartek Grzybowski9d4c15d2021-12-09 13:29:37 +0100403``'site.yml'`` playbook will run following playbooks in the given order:
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100404
Bartek Grzybowskicf2c37a2021-03-24 14:58:08 +0100405- ``resources.yml``
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100406- ``infrastructure.yml``
Bartek Grzybowskicf6797c2019-05-22 14:53:31 +0200407- ``rke.yml``
Bartek Grzybowski9d4c15d2021-12-09 13:29:37 +0100408- ``nfs.yml``
409- ``kube_prometheus.yml``
410- ``cert_manager.yml``
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100411- ``application.yml``
412
Michal Ptacekc424cff2019-03-06 16:25:43 +0000413----
414
Bartek Grzybowski32bf2fb2019-05-30 11:52:40 +0200415Part 4. Post-installation and troubleshooting
416---------------------------------------------
Michal Ptacekc424cff2019-03-06 16:25:43 +0000417
Bartek Grzybowskieb100462021-03-24 15:11:55 +0100418After all of the playbooks are run successfully the ONAP kubernetes application will be still deploying and it might take some time until all pods are up and running. You can monitor your newly created kubernetes cluster with this command::
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100419
Denis Kasanica7702f22019-11-14 12:35:46 +0100420 $ ssh -i ~/.ssh/offline_ssh_key root@10.8.8.100 # tailor this command to connect to your infra-node
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100421 $ watch -d -n 5 'kubectl get pods --all-namespaces'
422
Bartek Grzybowski32bf2fb2019-05-30 11:52:40 +0200423Alternatively you can monitor progress with ``helm_deployment_status.py`` script located in offline-installer directory. Transfer it to infra-node and run::
Milan Verespej1a230472019-03-20 13:51:40 +0100424
425 $ python helm_deployment_status.py -n <namespace_name> # namespace defaults to onap
426
Bartek Grzybowski32bf2fb2019-05-30 11:52:40 +0200427To automatically verify functionality with healthchecks after deployment becomes ready or after timeout period expires, append ``-hp`` switch followed by the full path to the healthcheck script and ``--health-mode`` optional switch with appropriate mode supported by that script (``health`` by default, ``--help`` displays available modes)::
Milan Verespej1a230472019-03-20 13:51:40 +0100428
Bartek Grzybowski32bf2fb2019-05-30 11:52:40 +0200429 $ python helm_deployment_status.py -hp <app_data_path>/<app_name>/helm_charts/robot/ete-k8s.sh --health-mode <healthcheck mode>
Milan Verespej1a230472019-03-20 13:51:40 +0100430
Bartek Grzybowski32bf2fb2019-05-30 11:52:40 +0200431It is strongly recommended to tailor ``helm_deployment_status.py`` to your needs since default values might not be what you'd expect. The defaults can be displayed with ``--help`` switch.
Michal Ptacekc424cff2019-03-06 16:25:43 +0000432
Bartek Grzybowskieb100462021-03-24 15:11:55 +0100433Final result of installation varies based on number of k8s nodes used and distribution of pods. In successful deployments all jobs should be in successful state. This can be verified with:
434
435::
Michal Ptacekc424cff2019-03-06 16:25:43 +0000436
437 $ kubectl get jobs -n <namespace>
438
439If some of the job is hanging in some wrong end-state like ``'BackoffLimitExceeded'`` manual intervention is required to heal this and make also dependent jobs passing. More details about particular job state can be obtained using ::
440
441 $ kubectl describe job -n <namespace> <job_name>
442
443If manual intervention is required, one can remove failing job and retry helm install command directly, which will not launch full deployment but rather check current state of the system and rebuild parts which are not up & running. Exact commands are as follows ::
444
445 $ kubectl delete job -n <namespace> <job_name>
446 $ helm deploy <env_name> <helm_chart_name> --namespace <namespace_name>
447
448 E.g. helm deploy dev local/onap --namespace onap
449
450Once all pods are properly deployed and in running state, one can verify functionality e.g. by running onap healthchecks ::
451
452 $ cd <app_data_path>/<app_name>/helm_charts/robot
453 $ ./ete-k8s.sh onap health
454
Bartek Grzybowskieb100462021-03-24 15:11:55 +0100455You can install ``screen`` and ``jq`` packages to aid troubleshooting. Those can be installed from resources directory.
Jan Benedikt1b3915e2019-11-13 04:13:34 -0500456
Bartek Grzybowskieb100462021-03-24 15:11:55 +0100457Screen is a terminal multiplexer and allows running multiple virtual terminal sessions as well as keep active SSH connections even when terminal is closed.
Jan Benedikt1b3915e2019-11-13 04:13:34 -0500458
459Jq can be used for editing json data format as output of kubectl. For example jq was used to troubleshoot `SDNC-739 (UEB - Listener in Crashloopback) <https://jira.onap.org/browse/SDNC-739/>`_ ::
460
461 $ kubectl -n onap get job onap-sdc-sdc-be-config-backend -o json | jq "del(.spec.selector)" | jq "del(.spec.template.metadata.labels)" | kubectl -n onap replace --force -f -
Michal Ptacekc424cff2019-03-06 16:25:43 +0000462
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100463-----
464
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100465Appendix 1. Ansible execution/bootstrap
466---------------------------------------
467
468There are two ways how to easily run the installer's ansible playbooks:
469
470- If you already have or can install a docker then you can build the provided ``'Dockerfile'`` for the ansible and run playbooks in the docker container.
471- Another way to deploy ansible is via chroot environment which is bundled together within this directory.
472
473(Re)build docker image and/or chroot archive
474~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
475
Bartek Grzybowski13141272021-03-24 15:29:13 +0100476Inside the ``'ansible/docker'`` directory you'll find the ``'Dockerfile'`` and ``'build_ansible_image.sh'`` script. You can run ``'build_ansible_image.sh'`` script on some machine with the internet connectivity and it will download all required packages needed for building the ansible docker image and for exporting it into a flat chroot environment.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100477
Bartek Grzybowski13141272021-03-24 15:29:13 +0100478Built image is exported into ``'ansible_chroot.tgz'`` archive in the same (``'ansible/docker'``) directory.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100479
480This script has two optional arguments:
481
482#. ansible version
483#. docker image name
484
485**Note:** if optional arguments are not used, docker image name will be set to ``'ansible'`` by default.
486
487Launching ansible playbook using chroot environment
488~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
489
490This is the default and preferred way of running ansible playbooks in an offline environment as there is no dependency on docker to be installed on the system. Chroot environment is already provided by included archive ``'ansible_chroot.tgz'``.
491
Bartek Grzybowski13141272021-03-24 15:29:13 +0100492It should be available in the ``'ansible/docker'`` directory as the end-result of the packaging script or after manual run of the ``'build_ansible_image.sh'`` script referenced above.
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100493
494All playbooks can be executed via ``'./run_playbook.sh'`` wrapper script.
495
496To get more info about the way how the ``'./run_playbook.sh'`` wrapper script should be used, run::
497
498 $ ./run_playbook.sh
499
500The main purpose of this wrapper script is to provide the ansible framework to a machine where it was bootstrapped without need of installing additional packages. The user can run this to display ``'ansible-playbook'`` command help::
501
502 $ ./run_playbook.sh --help
503
504Developers notes
505~~~~~~~~~~~~~~~~
506
507* There are two scripts which work in tandem for creating and running chroot
508* First one can convert docker image into chroot directory
509* Second script will automate chrooting (necessary steps for chroot to work and cleanup)
510* Both of them have help - just run::
511
Bartek Grzybowski13141272021-03-24 15:29:13 +0100512 $ cd ansible/docker
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100513 $ ./create_docker_chroot.sh help
514 $ ./run_chroot.sh help
515
516Example usage::
517
518 $ sudo su
Bartek Grzybowski13141272021-03-24 15:29:13 +0100519 $ ansible/docker/create_docker_chroot.sh convert some_docker_image ./new_name_for_chroot
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100520 $ cat ./new_name_for_chroot/README.md
Bartek Grzybowski13141272021-03-24 15:29:13 +0100521 $ ansible/docker/run_chroot.sh execute ./new_name_for_chroot cat /etc/os-release 2>/dev/null
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100522
523Launching ansible playbook using docker container (ALTERNATIVE APPROACH)
524~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
525
526This option is here just to keep support for the older method which relies on a running docker service. For the offline deployment use the chroot option as indicated above.
527
528You will not need ``'ansible_chroot.tgz'`` archive anymore, but the new requirement is a prebuilt docker image of ansible (based on the provided ``'Dockerfile'``). It should be available in your local docker repository (otherwise the default name ``'ansible'`` may fetch unwanted image from default registry!).
529
530To trigger this functionality and to run ``'ansible-playbook'`` inside a docker container instead of the chroot environment, you must first set the ``ANSIBLE_DOCKER_IMAGE`` variable. The value must be a name of the built ansible docker image.
531
532Usage is basically the same as with the default chroot way - the only difference is the existence of the environment variable::
533
534 $ ANSIBLE_DOCKER_IMAGE=ansible ./run_playbook.sh --help
535
536-----
537
Bartek Grzybowski40b4d152021-04-01 15:05:00 +0200538Appendix 2. Running Kubernetes Dashboard
539----------------------------------------
540
541Kubernetes Dashboard is a web-based, general purpose user interface for managing a k8s cluster.
542
543Some of its capabilities are:
544
545* workloads/services management (troubleshooting, scaling, editing, restarting pods)
546* deploying new workloads/applications to the cluster
547* managing the cluster itself
548
549Dashboard also provides information on the state of the cluster resources and on any errors that may have occurred.
550
551Kubernetes Dashboard itself is a kubernetes application. For user convenience the Offline platform has it already pre-installed:
552
553::
554
555 $ kubectl -n kubernetes-dashboard get deployment
556 NAME READY UP-TO-DATE AVAILABLE AGE
557 dashboard-metrics-scraper 1/1 1 1 76m
558 kubernetes-dashboard 1/1 1 1 76m
559
560Accessing the dashboard
561~~~~~~~~~~~~~~~~~~~~~~~
562
563There are multiple ways to access the application's web UI. Here we'll assume usage of local port forwarding on a box where you have access to a browser since the dashboard in Offline platform is exposed via a node port by default.
564
565First get the node port number that the dashboard service is exposed on:
566
567::
568
569 $ kubectl -n kubernetes-dashboard get svc kubernetes-dashboard -o custom-columns=PORTS:.spec.ports[].nodePort
570 PORTS
571 30825
572
573Now establish an ssh session to the infra node from your box from which you'll be accessing the dashboard:
574
575::
576
577 $ ssh -L 8080:127.0.0.1:30825 root@<infra host ip>
578
579Point your browser at https://localhost:8080/ and you should see the login page:
580
581.. image:: images/kubernetes-dashboard-signin.png
582 :alt: Kubernetes Dashboard signin
583
584Here, we'll leverage the Bearer Token to log in. Offline platform comes with dashboard admin user already created, we just need to extract its token. On the infra node issue following command:
585
586::
587
588 $ kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"
589
590It will return the token string on stdout. Copy-paste it into the sign-in form, selecting the "Token" option first. Upon successful login you'll be presented the cluster resources from ``default`` namespace. In the drop down box at the top select the namespace into which you installed the Onap application (namespace name equals the value of ``app_name`` variable from offline-installer setup) and you should see the cluster resources for Onap:
591
592.. image:: images/kubernetes-dashboard-main.png
593 :alt: Kubernetes Dashboard main page
594
595For additional information concerning the Kubernetes Dashboard please refer to the `official documentation <https://github.com/kubernetes/dashboard/tree/master/docs>`_.
596
597-----
598
Bartek Grzybowski8aa5c682021-06-21 16:04:38 +0200599Appendix 3. Running kube-prometheus stack
600-----------------------------------------
601
602`Kube-prometheus stack`_ is a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the `Prometheus Operator`_.
603
604The Stack is not deployed by default in Offline ONAP Platform, but all artifacts which it requires are downloaded by relevant scripts in the package build phase (see `Build Guide`_).
605
606Setup (optional)
607~~~~~~~~~~~~~~~~
608
609Kube-prometheus stack itself is a Kubernetes native application provisioned using Helm Charts. As such it can be configured using Helm values. Offline Installer provides a handy way for passing those values to the helm installation process.
610
611Any values for the Stack should be defined as subkeys of **kube_prometheus_helm_values** variable in **application_configuration.yml**. For instance, in order to override the default Grafana password, insert below structure into application_configuration.yml::
612
613 kube_prometheus_helm_values:
614 grafana:
615 adminPassword: <password>
616
617Another example - to set custom storage size for Prometheus tsdb::
618
619
620 kube_prometheus_helm_values:
621 prometheus:
622 prometheusSpec:
623 storageSpec:
624 volumeClaimTemplate:
625 spec:
626 resources:
627 requests:
628 storage: 6Gi
629
630A comprehensive list of Helm values for the Stack can be obtained on the `Kube-prometheus stack`_ project site, in the `values.yaml`_ file. Additional values for the Grafana can be checked on the `Grafana`_ project site in the *charts/grafana/values.yaml* file.
631
632Installation
633~~~~~~~~~~~~
634
635In order to actually install this tool it's required to set the following variable in application_configuration.yml::
636
637 kube_prometheus_stack_enabled: true
638
639After the Offline Platform installation process is complete, the Stack will be deployed into its own kubernetes and helm namespace **kube-prometheus**.
640
641Accessing Grafana dashboard
642~~~~~~~~~~~~~~~~~~~~~~~~~~~
643
644The most straightforward way to access the Grafana UI is by leveraging the *port-forward* k8s facility. Issue following command on the Infra host::
645
646 kubectl -n kube-prometheus port-forward --address 0.0.0.0 svc/kube-prometheus-stack-grafana 8081:80
647
648Then navigate to http://<infra IP>:8081 to access the UI:
649
650.. image:: images/grafana-signin.png
651 :alt: Grafana Login page
652
Bartek Grzybowski9d4c15d2021-12-09 13:29:37 +0100653Default username is *admin* and the default password is *grafana*.
Bartek Grzybowski8aa5c682021-06-21 16:04:38 +0200654
655In the left pane navigate to *Dashboards -> Manage* to see the various pre-defined dashboards that come bundled with kube-prometheus stack. There is also the *Custom* folder which holds few additional dashes defined by the Offline Installer authors:
656
657.. image:: images/grafana-dashboards.png
658 :alt: Grafana dashboards
659
Bartek Grzybowski327b7dc2021-06-24 11:29:58 +0200660Alternative way of accessing the UI is by leveraging the NodePort type service which exposes Grafana UI on the Infra host public port directly. To do so get the port number first::
661
662 kubectl -n kube-prometheus get service/kube-prometheus-stack-grafana -o custom-columns=PORTS:.spec.ports[].nodePort
663
664Then navigate to http://<infra IP>:<nodePort> to access the UI.
665
Petr Ospalýbe81ab02019-02-14 21:30:31 +0100666.. _Build Guide: ./BuildGuide.rst
Bartek Grzybowski553a9202021-03-23 15:09:11 +0100667.. _Software requirements: https://docs.onap.org/projects/onap-oom/en/latest/oom_cloud_setup_guide.html#software-requirements
668.. _Hardware requirements: https://docs.onap.org/projects/onap-oom/en/latest/oom_cloud_setup_guide.html#minimum-hardware-configuration
Bartek Grzybowski2936ef12021-03-23 14:15:33 +0100669.. _OOM ONAP: https://docs.onap.org/projects/onap-oom/en/latest/index.html
670.. _Offline installer: https://gerrit.onap.org/r/q/oom/offline-installer
Bartek Grzybowski553a9202021-03-23 15:09:11 +0100671.. _RKE: https://rancher.com/products/rke/
Bartek Grzybowski8aa5c682021-06-21 16:04:38 +0200672.. _Kube-prometheus stack: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
673.. _Prometheus Operator: https://github.com/prometheus-operator/prometheus-operator
674.. _values.yaml: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml
675.. _Grafana: https://github.com/grafana/helm-charts