Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 1 | .. This work is licensed under a Creative Commons Attribution 4.0 International License. |
| 2 | .. http://creativecommons.org/licenses/by/4.0 |
Bartek Grzybowski | d549d82 | 2021-03-22 13:12:01 +0100 | [diff] [blame] | 3 | .. Copyright 2021 Samsung Electronics Co., Ltd. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 4 | |
Bartek Grzybowski | d549d82 | 2021-03-22 13:12:01 +0100 | [diff] [blame] | 5 | Offline Installer - Installation Guide |
| 6 | ====================================== |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 7 | |
Bartek Grzybowski | 2936ef1 | 2021-03-23 14:15:33 +0100 | [diff] [blame] | 8 | This document describes offline installation procedure for `OOM ONAP`_, which is done by the ansible based `Offline installer`_. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 9 | |
Bartek Grzybowski | 553a920 | 2021-03-23 15:09:11 +0100 | [diff] [blame] | 10 | Before you begin the installation process you should prepare the offline installation packages. Please refer to the `Build Guide`_ for instructions on how to create them. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 11 | |
| 12 | ----- |
| 13 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 14 | Part 1. Prerequisites |
| 15 | --------------------- |
| 16 | |
Bartek Grzybowski | 553a920 | 2021-03-23 15:09:11 +0100 | [diff] [blame] | 17 | ONAP platform has certain software requirements - see `Software requirements`_ and minimum hardware recommendations: ``224 GB RAM``, ``112 vCPUs`` and ``160GB`` of storage (see `Hardware requirements`_). The minimum count of nodes should not drop below three - otherwise you may have to do some tweaking to make it work, which is not covered here (there is a pod count limit for a single kubernetes node - you can read more about it in this `discussion <https://lists.onap.org/g/onap-discuss/topic/oom_110_kubernetes_pod/25213556>`_). |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 18 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 19 | Kubernetes cluster |
| 20 | ~~~~~~~~~~~~~~~~~~ |
| 21 | |
| 22 | The four nodes/VMs will be running these services: |
| 23 | |
| 24 | - **infra-node**:: |
| 25 | |
| 26 | - nexus |
| 27 | - nginx proxy |
| 28 | - dns |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 29 | - kubernetes-etcd |
| 30 | - kubernetes-control-plane |
Bartek Grzybowski | 553a920 | 2021-03-23 15:09:11 +0100 | [diff] [blame] | 31 | - chartmuseum (if using helm v3) |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 32 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 33 | **NOTE:** kubernetes-* control plane can be colocated directly with k8s nodes and not necessarily on infra node. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 34 | |
| 35 | - **kubernetes node 1-3**:: |
| 36 | |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 37 | - kubernetes worker |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 38 | |
Bartek Grzybowski | 553a920 | 2021-03-23 15:09:11 +0100 | [diff] [blame] | 39 | You don't need to care about these services now - that is the responsibility of the installer (described below). Just start four VMs as seen in below table (or according to your needs as we hinted above): |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 40 | |
| 41 | .. _Overview table of the kubernetes cluster: |
| 42 | |
| 43 | Kubernetes cluster overview |
| 44 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 45 | |
Bartek Grzybowski | 553a920 | 2021-03-23 15:09:11 +0100 | [diff] [blame] | 46 | .. note:: Offline installer leverages `RKE`_ to provision kubernetes cluster. If you'd like to use different k8s installation method please exclude ``rke.yml`` ansible playbook from execution and provide your own. |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 47 | |
Tomáš Levora | 2a355bb | 2019-10-10 14:04:08 +0200 | [diff] [blame] | 48 | =================== ================== ==================== ============== ============ =============== |
| 49 | KUBERNETES NODE OS NETWORK CPU RAM STORAGE |
| 50 | =================== ================== ==================== ============== ============ =============== |
Bartek Grzybowski | 632891d | 2021-09-24 12:25:33 +0200 | [diff] [blame] | 51 | **infra-node** RHEL/CentOS 7.9 ``10.8.8.100/24`` ``8 vCPUs`` ``8 GB`` ``100 GB`` |
| 52 | **kube-node1** RHEL/CentOS 7.9 ``10.8.8.101/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB`` |
| 53 | **kube-node2** RHEL/CentOS 7.9 ``10.8.8.102/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB`` |
| 54 | **kube-node3** RHEL/CentOS 7.9 ``10.8.8.103/24`` ``16 vCPUs`` ``56+ GB`` ``100 GB`` |
Tomáš Levora | 2a355bb | 2019-10-10 14:04:08 +0200 | [diff] [blame] | 55 | SUM ``56 vCPUs`` ``176+ GB`` ``400 GB`` |
| 56 | =========================================================== ============== ============ =============== |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 57 | |
Bartek Grzybowski | 632891d | 2021-09-24 12:25:33 +0200 | [diff] [blame] | 58 | As of now, the offline installer supports only **RHEL 7.x** and **CentOS 7.9** distributions, with at least *@core* and *@base* package groups installed including *Mandatory* and *Default* package sets. So, your VMs should be preinstalled with this operating system - the hypervisor and platform can be of your choosing. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 59 | |
Bartek Grzybowski | 553a920 | 2021-03-23 15:09:11 +0100 | [diff] [blame] | 60 | We will expect from now on that you installed four VMs and they are connected to the shared network. All VMs must be reachable from *install-server* (below), which can be the hypervisor, *infra-node* or completely different host. But in either of these cases the *install-server* must be able to connect over ssh to all of these nodes. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 61 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 62 | Install-server |
| 63 | ~~~~~~~~~~~~~~ |
| 64 | |
| 65 | We will use distinct *install-server* and keep it separate from the four-node cluster. But if you wish so, you can use *infra-node* for this goal (if you use the default ``'chroot'`` option of the installer), but in that case double the size of the storage requirement! |
| 66 | |
| 67 | Prerequisites for the *install-server*: |
| 68 | |
| 69 | - packages described in `Build Guide`_ |
| 70 | - extra ``100 GB`` storage (to have space where to store these packages) |
| 71 | - installed ``'chroot'`` and/or ``'docker'`` system commands |
| 72 | - network connection to the nodes - especially functioning ssh client |
| 73 | |
| 74 | Our *install-server* will have ip: ``10.8.8.4``. |
| 75 | |
| 76 | **NOTE:** All the subsequent commands below, are executed from within this *install-server*. |
| 77 | |
| 78 | ----- |
| 79 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 80 | Part 2. Configuration |
| 81 | --------------------- |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 82 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 83 | All commands and setups described in this chapter *MUST* be run on the *install-server*. It's assumed here that all commands are run as ``root`` which is of course not necessary - you can use a regular user account. The ssh/ansible connection to the nodes will also expect that we are connecting as ``root`` - you need to elevate privileges to be able to install on them. Although it can be achieved by other means (sudo), we decided here to keep instructions simple. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 84 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 85 | Installer packages |
| 86 | ~~~~~~~~~~~~~~~~~~ |
| 87 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 88 | At this point you should have the installer packages already prepared (see `Build Guide`_): |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 89 | |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 90 | - sw_package.tar |
| 91 | - resources_package.tar |
| 92 | - aux_package.tar |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 93 | |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 94 | **NOTE:** ``'aux_package.tar'`` is optional and if you don't have use for it, you can ignore it. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 95 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 96 | Copy above packages to the ``/data`` directory on the *install-server* and then unpack the ``'sw_package.tar'`` to your home directory: |
| 97 | |
| 98 | :: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 99 | |
| 100 | $ mkdir ~/onap-offline-installer |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 101 | $ tar -C ~/onap-offline-installer -xf /data/sw_package.tar |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 102 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 103 | Application directory |
| 104 | ~~~~~~~~~~~~~~~~~~~~~ |
| 105 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 106 | Change the current directory to ``'ansible'``:: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 107 | |
| 108 | $ cd ~/onap-offline-installer/ansible |
| 109 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 110 | You can see multiple files and directories inside - those are the *offline-installer* ansible playbooks. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 111 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 112 | If you created the ``'sw_package.tar'`` package according to the *Build Guide* then at least the following files should be present: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 113 | |
Denis Kasanic | a7702f2 | 2019-11-14 12:35:46 +0100 | [diff] [blame] | 114 | - ``application/application_configuration.yml`` |
| 115 | - ``inventory/hosts.yml`` |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 116 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 117 | Following paragraphs describe fine-tuning of ``'inventory.yml'`` to reflect your VMs setup and ``'application_configuration.yml'`` to setup the provisioner itself. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 118 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 119 | hosts.yml |
| 120 | ~~~~~~~~~ |
| 121 | |
| 122 | We need to setup the ``'hosts.yml'`` first, the template looks like this:: |
| 123 | |
| 124 | --- |
| 125 | # This group contains hosts with all resources (binaries, packages, etc.) |
| 126 | # in tarball. |
| 127 | all: |
| 128 | vars: |
| 129 | # this key is supposed to be generated during setup.yml playbook execution |
| 130 | # change it just when you have better one working for all nodes |
| 131 | ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key |
| 132 | ansible_ssh_common_args: '-o StrictHostKeyChecking=no' |
| 133 | |
| 134 | children: |
| 135 | resources: |
| 136 | hosts: |
| 137 | resource-host: |
| 138 | ansible_host: 10.8.8.5 |
| 139 | |
| 140 | # This is group of hosts where nexus, nginx, dns and all other required |
| 141 | # services are running. |
| 142 | infrastructure: |
| 143 | hosts: |
| 144 | infrastructure-server: |
| 145 | ansible_host: 10.8.8.13 |
| 146 | #IP used for communication between infra and kubernetes nodes, must be specified. |
| 147 | cluster_ip: 10.8.8.13 |
| 148 | |
| 149 | # This is group of hosts which are/will be part of Kubernetes cluster. |
| 150 | kubernetes: |
Michal Zegan | f9ef0c1 | 2019-06-25 11:05:05 +0200 | [diff] [blame] | 151 | children: |
| 152 | # This is a group of hosts containing kubernetes worker nodes. |
| 153 | kubernetes-node: |
| 154 | hosts: |
| 155 | kubernetes-node-1: |
| 156 | ansible_host: 10.8.8.19 |
| 157 | #ip of the node that it uses for communication with k8s cluster. |
| 158 | cluster_ip: 10.8.8.19 |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 159 | # External ip of the node, used for access from outside of the cluster. |
| 160 | # Can be set to some kind of floating or public ip. |
| 161 | # If not set, cluster_ip is used for this purpose. |
| 162 | # external_ip: x.x.x.x |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 163 | |
Michal Zegan | f9ef0c1 | 2019-06-25 11:05:05 +0200 | [diff] [blame] | 164 | # Group of hosts containing etcd cluster nodes. |
| 165 | # Defaults to infra. |
| 166 | kubernetes-etcd: |
| 167 | hosts: |
| 168 | infrastructure-server |
| 169 | |
| 170 | # This is a group of hosts that are to be used as kubernetes control plane nodes. |
| 171 | # This means they host kubernetes api server, controller manager and scheduler. |
| 172 | # This example uses infra for this purpose, however note that any |
| 173 | # other host could be used including kubernetes nodes. |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 174 | # cluster_ip needs to be set for hosts used as control planes, external_ip can also be used. |
Michal Zegan | f9ef0c1 | 2019-06-25 11:05:05 +0200 | [diff] [blame] | 175 | kubernetes-control-plane: |
| 176 | hosts: |
| 177 | infrastructure-server |
Bartek Grzybowski | cf6797c | 2019-05-22 14:53:31 +0200 | [diff] [blame] | 178 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 179 | nfs-server: |
| 180 | hosts: |
| 181 | kubernetes-node-1 |
| 182 | |
| 183 | There is some ssh configuration under the ``'vars'`` section - we will deal with ssh setup a little bit later in the `SSH authentication`_. |
| 184 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 185 | First you need to set the ip addresses and add a couple of kubernetes nodes to match your four-node cluster: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 186 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 187 | - Under the ``'resource-host'`` set the ``'ansible_host'`` address to the ip of the host where the packages are stored - it must be reachable by ssh from the *install-server* (for ansible to run playbooks on it) **AND** *infra-node* (to extract resource data from *resource-host* to *infra-node* over ssh). In our scenario the *resource-host* is the same as the *install-server*: ``'10.8.8.4'`` |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 188 | - Similarly, set the ``'ansible_host'`` to the address of the *infra-node* under the ``'infrastructure-server'``. |
| 189 | - Copy the whole ``'kubernetes-node-1'`` subsection and paste it twice directly after. Change the numbers to ``'kubernetes-node-2'`` and ``'kubernetes-node-3'`` respectively and fix the addresses in the ``'ansible_host'`` variables again to match *kube-node1*, *kube-node2* and *kube-node3*. |
| 190 | |
| 191 | As you can see, there is another ``'cluster_ip'`` variable for each node - this serve as a designated node address in the kubernetes cluster. Make it the same as the respective ``'ansible_host'``. |
| 192 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 193 | **NOTE:** In our simple setup we have only one interface per node, but that does not need to be a case for some other deployments - especially if we start to deal with a production usage. Basically, an ``'ansible_host'`` is an entry point for the *install-server's* ansible (*offline-installer*), but the kubernetes cluster can be communicating on a separate network to which *install-server* has no access. That is why we have this distinctive variable, so we can tell the installer that there is a different network, where we want to run the kubernetes traffic and what address each node has on such a network. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 194 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 195 | After applying all described changes, the ``'hosts.yml'`` should look similar to this:: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 196 | |
| 197 | --- |
| 198 | # This group contains hosts with all resources (binaries, packages, etc.) |
| 199 | # in tarball. |
| 200 | all: |
| 201 | vars: |
| 202 | # this key is supposed to be generated during setup.yml playbook execution |
| 203 | # change it just when you have better one working for all nodes |
| 204 | ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key |
| 205 | ansible_ssh_common_args: '-o StrictHostKeyChecking=no' |
| 206 | |
| 207 | children: |
| 208 | resources: |
| 209 | hosts: |
| 210 | resource-host: |
| 211 | ansible_host: 10.8.8.4 |
| 212 | |
| 213 | # This is group of hosts where nexus, nginx, dns and all other required |
| 214 | # services are running. |
| 215 | infrastructure: |
| 216 | hosts: |
| 217 | infrastructure-server: |
Denis Kasanic | a7702f2 | 2019-11-14 12:35:46 +0100 | [diff] [blame] | 218 | ansible_host: 10.8.8.100 |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 219 | #IP used for communication between infra and kubernetes nodes, must be specified. |
| 220 | cluster_ip: 10.8.8.100 |
| 221 | |
| 222 | # This is group of hosts which are/will be part of Kubernetes cluster. |
| 223 | kubernetes: |
Michal Zegan | f9ef0c1 | 2019-06-25 11:05:05 +0200 | [diff] [blame] | 224 | children: |
| 225 | # This is a group of hosts containing kubernetes worker nodes. |
| 226 | kubernetes-node: |
| 227 | hosts: |
| 228 | kubernetes-node-1: |
| 229 | ansible_host: 10.8.8.101 |
| 230 | #ip of the node that it uses for communication with k8s cluster. |
| 231 | cluster_ip: 10.8.8.101 |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 232 | # External ip of the node, used for access from outside of the cluster. |
| 233 | # Can be set to some kind of floating or public ip. |
| 234 | # If not set, cluster_ip is used for this purpose. |
| 235 | # external_ip: x.x.x.x |
Michal Zegan | f9ef0c1 | 2019-06-25 11:05:05 +0200 | [diff] [blame] | 236 | kubernetes-node-2: |
| 237 | ansible_host: 10.8.8.102 |
| 238 | #ip of the node that it uses for communication with k8s cluster. |
| 239 | cluster_ip: 10.8.8.102 |
| 240 | kubernetes-node-3: |
| 241 | ansible_host: 10.8.8.103 |
| 242 | #ip of the node that it uses for communication with k8s cluster. |
| 243 | cluster_ip: 10.8.8.103 |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 244 | |
Michal Zegan | f9ef0c1 | 2019-06-25 11:05:05 +0200 | [diff] [blame] | 245 | # Group of hosts containing etcd cluster nodes. |
| 246 | # Defaults to infra. |
| 247 | kubernetes-etcd: |
| 248 | hosts: |
| 249 | infrastructure-server |
| 250 | |
| 251 | # This is a group of hosts that are to be used as kubernetes control plane nodes. |
| 252 | # This means they host kubernetes api server, controller manager and scheduler. |
| 253 | # This example uses infra for this purpose, however note that any |
| 254 | # other host could be used including kubernetes nodes. |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 255 | # cluster_ip needs to be set for hosts used as control planes, external_ip can also be used. |
Michal Zegan | f9ef0c1 | 2019-06-25 11:05:05 +0200 | [diff] [blame] | 256 | kubernetes-control-plane: |
| 257 | hosts: |
| 258 | infrastructure-server |
Bartek Grzybowski | cf6797c | 2019-05-22 14:53:31 +0200 | [diff] [blame] | 259 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 260 | nfs-server: |
| 261 | hosts: |
| 262 | kubernetes-node-1 |
| 263 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 264 | application_configuration.yml |
| 265 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 266 | |
| 267 | Here, we will be interested in the following variables: |
| 268 | |
| 269 | - ``resources_dir`` |
| 270 | - ``resources_filename`` |
| 271 | - ``aux_resources_filename`` |
| 272 | - ``app_data_path`` |
| 273 | - ``aux_data_path`` |
| 274 | - ``app_name`` |
Bartek Grzybowski | 30b2cbf | 2019-03-26 16:10:10 +0100 | [diff] [blame] | 275 | - ``timesync`` |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 276 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 277 | ``'resource_dir'``, ``'resources_filename'`` and ``'aux_resources_filename'`` must correspond to the file paths on the *resource-host* (``'resource-host'`` in ``hosts.yml``), which in our case is the *install-server* host. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 278 | |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 279 | The ``'resource_dir'`` should be set to ``'/data'``, ``'resources_filename'`` to ``'resources_package.tar'`` and ``'aux_resources_filename'`` to ``'aux_package.tar'``. The values should be the same as are in the `Installer packages`_ section. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 280 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 281 | ``'app_data_path'`` is the absolute path on the *infra-node* to where the package ``'resources_package.tar'`` will be extracted and similarly ``'aux_data_path'`` is another absolute path for ``'aux_package.tar'``. Both paths are fully arbitrary, but they should point to the filesystem with enough disk space - the storage requirements are described in `Overview table of the kubernetes cluster`_. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 282 | |
| 283 | **NOTE:** As we mentioned in `Installer packages`_ - the auxiliary package is not mandatory and we will not utilize it in here either. |
| 284 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 285 | The ``'app_name'`` variable should be short and descriptive. We will set it simply to ``onap``. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 286 | |
Bartek Grzybowski | 30b2cbf | 2019-03-26 16:10:10 +0100 | [diff] [blame] | 287 | The ``'timesync'`` variable is optional and controls synchronisation of the system clock on hosts. It should be configured only if a custom NTP server is available and needed. Such a time authority should be on a host reachable from all installation nodes. If this setting is not provided then the default behavior is to setup NTP daemon on infra-node and sync all kube-nodes' time with it. |
| 288 | |
| 289 | If you wish to provide your own NTP servers configure their IPs as follows:: |
| 290 | |
| 291 | timesync: |
| 292 | servers: |
| 293 | - <ip address of NTP_1> |
| 294 | - <...> |
| 295 | - <ip address of NTP_N> |
| 296 | |
| 297 | Another time adjustment related variables are ``'timesync.slewclock'`` and ``'timesync.timezone'`` . |
| 298 | First one can have value of ``'true'`` or ``'false'`` (default). It controls whether (in case of big time difference compared to server) time should be adjusted gradually by slowing down or speeding up the clock as required (``'true'``) or in one step (``'false'``):: |
| 299 | |
| 300 | timesync: |
| 301 | slewclock: true |
| 302 | |
| 303 | Second one controls time zone setting on host. It's value should be time zone name according to tz database names with ``'Universal'`` being the default one:: |
| 304 | |
| 305 | timesync. |
| 306 | timezone: UTC |
| 307 | |
| 308 | ``'timesync.servers'``, ``'timesync.slewclock'`` and ``'timesync.timezone'`` settings can be used independently. |
| 309 | |
| 310 | Final configuration can resemble the following:: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 311 | |
| 312 | resources_dir: /data |
Denis Kasanic | a7702f2 | 2019-11-14 12:35:46 +0100 | [diff] [blame] | 313 | resources_filename: resources_package.tar |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 314 | app_data_path: /opt/onap |
| 315 | app_name: onap |
Bartek Grzybowski | 30b2cbf | 2019-03-26 16:10:10 +0100 | [diff] [blame] | 316 | timesync: |
| 317 | servers: |
| 318 | - 192.168.0.1 |
| 319 | - 192.168.0.2 |
| 320 | slewclock: true |
| 321 | timezone: UTC |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 322 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 323 | Helm chart values overrides |
| 324 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
Michal Zegan | a579c98 | 2019-04-02 15:33:30 +0200 | [diff] [blame] | 325 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 326 | OOM charts are coming with all ONAP components disabled, this setting is also prepackaged within our sw_package.tar. Luckily there are multiple ways supported how to override this setting. It's also necessary for setting-up VIM specific entries and basically to configure any stuff with non default values. |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 327 | |
| 328 | First option is to use ``overrides`` key in ``application_configuration.yml``. |
Michal Zegan | a579c98 | 2019-04-02 15:33:30 +0200 | [diff] [blame] | 329 | These settings will override helm values originally stored in ``values.yaml`` files in helm chart directories. |
| 330 | |
| 331 | For example, the following lines could be appended to ``application_configuration.yml`` to set up managed openstack credentials for onap's so component:: |
| 332 | |
| 333 | overrides: |
| 334 | so: |
| 335 | config: |
| 336 | openStackUserName: "os_user" |
| 337 | openStackRegion: "region_name" |
| 338 | openStackKeyStoneUrl: "keystone_url" |
| 339 | openStackEncryptedPasswordHere: "encrypted_password" |
| 340 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 341 | In addition or alternatively to that one can configure ``helm_override_files`` variable in ``'application_configuration.yml'`` and mention all files with helm chart values there, e.g.: |
| 342 | |
| 343 | :: |
| 344 | |
| 345 | helm_override_files: |
| 346 | - "/path/to/values1.yaml" |
| 347 | - "/path/to/values2.yaml" |
Michal Ptacek | 26278df | 2019-07-09 10:46:26 +0000 | [diff] [blame] | 348 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 349 | SSH authentication |
| 350 | ~~~~~~~~~~~~~~~~~~ |
| 351 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 352 | Finally you need to setup password-less login from *install-server* to the nodes. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 353 | |
Bartek Grzybowski | a6ced38 | 2021-03-24 14:30:39 +0100 | [diff] [blame] | 354 | You can use the ansible playbook ``'setup.yml'`` for that purpose:: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 355 | |
Denis Kasanic | a7702f2 | 2019-11-14 12:35:46 +0100 | [diff] [blame] | 356 | $ ./run_playbook.sh -i inventory/hosts.yml setup.yml -u root --ask-pass |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 357 | |
| 358 | You will be asked for password per each node and the playbook will generate a unprotected ssh key-pair ``'~/.ssh/offline_ssh_key'``, which will be distributed to the nodes. |
| 359 | |
| 360 | Another option is to generate a ssh key-pair manually. We strongly advise you to protect it with a passphrase, but for simplicity we will showcase generating of a private key without any such protection:: |
| 361 | |
| 362 | $ ssh-keygen -N "" -f ~/.ssh/identity |
| 363 | |
| 364 | The next step will be to distribute the public key to these nodes and from that point no password is needed:: |
| 365 | |
| 366 | $ for ip in 100 101 102 103 ; do ssh-copy-id -i ~/.ssh/identity.pub root@10.8.8.${ip} ; done |
| 367 | |
| 368 | This command behaves almost identically to the ``'setup.yml'`` playbook. |
| 369 | |
| 370 | If you generated the ssh key manually then you can now run the ``'setup.yml'`` playbook like this and achieve the same result as in the first execution:: |
| 371 | |
Denis Kasanic | a7702f2 | 2019-11-14 12:35:46 +0100 | [diff] [blame] | 372 | $ ./run_playbook.sh -i inventory/hosts.yml setup.yml |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 373 | |
| 374 | This time it should not ask you for any password - of course this is very redundant, because you just distributed two ssh keys for no good reason. |
| 375 | |
| 376 | We can finally edit and finish the configuration of the ``'hosts.yml'``: |
| 377 | |
| 378 | - if you used the ``'setup.yml'`` playbook then you can just leave this line as it is:: |
| 379 | |
| 380 | ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key |
| 381 | |
| 382 | - if you created a ssh key manually then change it like this:: |
| 383 | |
| 384 | ansible_ssh_private_key_file: /root/.ssh/identity |
| 385 | |
| 386 | ----- |
| 387 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 388 | Part 3. Installation |
| 389 | -------------------- |
| 390 | |
| 391 | We should have the configuration complete and be ready to start the installation. The installation is done via ansible playbooks, which are run either inside a **chroot** environment (default) or from the **docker** container. If for some reason you want to run playbooks from the docker instead of chroot then you cannot use *infra-node* or any other *kube-node* as the *install-server* - otherwise you risk that installation will fail due to restarting of the docker service. |
| 392 | |
Bartek Grzybowski | cf2c37a | 2021-03-24 14:58:08 +0100 | [diff] [blame] | 393 | ``'sw_package.tar'`` should contain ``'ansible_chroot.tgz'`` file inside the ``'docker'`` directory. Detailed instructions on how to create it manually and to get more info about the scripts dealing with docker and chroot, go to `Appendix 1. Ansible execution/bootstrap`_. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 394 | |
| 395 | We will use the default chroot option so we don't need any docker service to be running. |
| 396 | |
Bartek Grzybowski | cf2c37a | 2021-03-24 14:58:08 +0100 | [diff] [blame] | 397 | Commence the installation process by running following command:: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 398 | |
Denis Kasanic | a7702f2 | 2019-11-14 12:35:46 +0100 | [diff] [blame] | 399 | $ ./run_playbook.sh -i inventory/hosts.yml -e @application/application_configuration.yml site.yml |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 400 | |
Bartek Grzybowski | cf2c37a | 2021-03-24 14:58:08 +0100 | [diff] [blame] | 401 | This will take a while so be patient. The whole provisioning process is idempotent so you may safely re-run it if required. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 402 | |
Bartek Grzybowski | 9d4c15d | 2021-12-09 13:29:37 +0100 | [diff] [blame] | 403 | ``'site.yml'`` playbook will run following playbooks in the given order: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 404 | |
Bartek Grzybowski | cf2c37a | 2021-03-24 14:58:08 +0100 | [diff] [blame] | 405 | - ``resources.yml`` |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 406 | - ``infrastructure.yml`` |
Bartek Grzybowski | cf6797c | 2019-05-22 14:53:31 +0200 | [diff] [blame] | 407 | - ``rke.yml`` |
Bartek Grzybowski | 9d4c15d | 2021-12-09 13:29:37 +0100 | [diff] [blame] | 408 | - ``nfs.yml`` |
| 409 | - ``kube_prometheus.yml`` |
| 410 | - ``cert_manager.yml`` |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 411 | - ``application.yml`` |
| 412 | |
Michal Ptacek | c424cff | 2019-03-06 16:25:43 +0000 | [diff] [blame] | 413 | ---- |
| 414 | |
Bartek Grzybowski | 32bf2fb | 2019-05-30 11:52:40 +0200 | [diff] [blame] | 415 | Part 4. Post-installation and troubleshooting |
| 416 | --------------------------------------------- |
Michal Ptacek | c424cff | 2019-03-06 16:25:43 +0000 | [diff] [blame] | 417 | |
Bartek Grzybowski | eb10046 | 2021-03-24 15:11:55 +0100 | [diff] [blame] | 418 | After all of the playbooks are run successfully the ONAP kubernetes application will be still deploying and it might take some time until all pods are up and running. You can monitor your newly created kubernetes cluster with this command:: |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 419 | |
Denis Kasanic | a7702f2 | 2019-11-14 12:35:46 +0100 | [diff] [blame] | 420 | $ ssh -i ~/.ssh/offline_ssh_key root@10.8.8.100 # tailor this command to connect to your infra-node |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 421 | $ watch -d -n 5 'kubectl get pods --all-namespaces' |
| 422 | |
Bartek Grzybowski | 32bf2fb | 2019-05-30 11:52:40 +0200 | [diff] [blame] | 423 | Alternatively you can monitor progress with ``helm_deployment_status.py`` script located in offline-installer directory. Transfer it to infra-node and run:: |
Milan Verespej | 1a23047 | 2019-03-20 13:51:40 +0100 | [diff] [blame] | 424 | |
| 425 | $ python helm_deployment_status.py -n <namespace_name> # namespace defaults to onap |
| 426 | |
Bartek Grzybowski | 32bf2fb | 2019-05-30 11:52:40 +0200 | [diff] [blame] | 427 | To automatically verify functionality with healthchecks after deployment becomes ready or after timeout period expires, append ``-hp`` switch followed by the full path to the healthcheck script and ``--health-mode`` optional switch with appropriate mode supported by that script (``health`` by default, ``--help`` displays available modes):: |
Milan Verespej | 1a23047 | 2019-03-20 13:51:40 +0100 | [diff] [blame] | 428 | |
Bartek Grzybowski | 32bf2fb | 2019-05-30 11:52:40 +0200 | [diff] [blame] | 429 | $ python helm_deployment_status.py -hp <app_data_path>/<app_name>/helm_charts/robot/ete-k8s.sh --health-mode <healthcheck mode> |
Milan Verespej | 1a23047 | 2019-03-20 13:51:40 +0100 | [diff] [blame] | 430 | |
Bartek Grzybowski | 32bf2fb | 2019-05-30 11:52:40 +0200 | [diff] [blame] | 431 | It is strongly recommended to tailor ``helm_deployment_status.py`` to your needs since default values might not be what you'd expect. The defaults can be displayed with ``--help`` switch. |
Michal Ptacek | c424cff | 2019-03-06 16:25:43 +0000 | [diff] [blame] | 432 | |
Bartek Grzybowski | eb10046 | 2021-03-24 15:11:55 +0100 | [diff] [blame] | 433 | Final result of installation varies based on number of k8s nodes used and distribution of pods. In successful deployments all jobs should be in successful state. This can be verified with: |
| 434 | |
| 435 | :: |
Michal Ptacek | c424cff | 2019-03-06 16:25:43 +0000 | [diff] [blame] | 436 | |
| 437 | $ kubectl get jobs -n <namespace> |
| 438 | |
| 439 | If some of the job is hanging in some wrong end-state like ``'BackoffLimitExceeded'`` manual intervention is required to heal this and make also dependent jobs passing. More details about particular job state can be obtained using :: |
| 440 | |
| 441 | $ kubectl describe job -n <namespace> <job_name> |
| 442 | |
| 443 | If manual intervention is required, one can remove failing job and retry helm install command directly, which will not launch full deployment but rather check current state of the system and rebuild parts which are not up & running. Exact commands are as follows :: |
| 444 | |
| 445 | $ kubectl delete job -n <namespace> <job_name> |
| 446 | $ helm deploy <env_name> <helm_chart_name> --namespace <namespace_name> |
| 447 | |
| 448 | E.g. helm deploy dev local/onap --namespace onap |
| 449 | |
| 450 | Once all pods are properly deployed and in running state, one can verify functionality e.g. by running onap healthchecks :: |
| 451 | |
| 452 | $ cd <app_data_path>/<app_name>/helm_charts/robot |
| 453 | $ ./ete-k8s.sh onap health |
| 454 | |
Bartek Grzybowski | eb10046 | 2021-03-24 15:11:55 +0100 | [diff] [blame] | 455 | You can install ``screen`` and ``jq`` packages to aid troubleshooting. Those can be installed from resources directory. |
Jan Benedikt | 1b3915e | 2019-11-13 04:13:34 -0500 | [diff] [blame] | 456 | |
Bartek Grzybowski | eb10046 | 2021-03-24 15:11:55 +0100 | [diff] [blame] | 457 | Screen is a terminal multiplexer and allows running multiple virtual terminal sessions as well as keep active SSH connections even when terminal is closed. |
Jan Benedikt | 1b3915e | 2019-11-13 04:13:34 -0500 | [diff] [blame] | 458 | |
| 459 | Jq can be used for editing json data format as output of kubectl. For example jq was used to troubleshoot `SDNC-739 (UEB - Listener in Crashloopback) <https://jira.onap.org/browse/SDNC-739/>`_ :: |
| 460 | |
| 461 | $ kubectl -n onap get job onap-sdc-sdc-be-config-backend -o json | jq "del(.spec.selector)" | jq "del(.spec.template.metadata.labels)" | kubectl -n onap replace --force -f - |
Michal Ptacek | c424cff | 2019-03-06 16:25:43 +0000 | [diff] [blame] | 462 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 463 | ----- |
| 464 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 465 | Appendix 1. Ansible execution/bootstrap |
| 466 | --------------------------------------- |
| 467 | |
| 468 | There are two ways how to easily run the installer's ansible playbooks: |
| 469 | |
| 470 | - If you already have or can install a docker then you can build the provided ``'Dockerfile'`` for the ansible and run playbooks in the docker container. |
| 471 | - Another way to deploy ansible is via chroot environment which is bundled together within this directory. |
| 472 | |
| 473 | (Re)build docker image and/or chroot archive |
| 474 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 475 | |
Bartek Grzybowski | 1314127 | 2021-03-24 15:29:13 +0100 | [diff] [blame] | 476 | Inside the ``'ansible/docker'`` directory you'll find the ``'Dockerfile'`` and ``'build_ansible_image.sh'`` script. You can run ``'build_ansible_image.sh'`` script on some machine with the internet connectivity and it will download all required packages needed for building the ansible docker image and for exporting it into a flat chroot environment. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 477 | |
Bartek Grzybowski | 1314127 | 2021-03-24 15:29:13 +0100 | [diff] [blame] | 478 | Built image is exported into ``'ansible_chroot.tgz'`` archive in the same (``'ansible/docker'``) directory. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 479 | |
| 480 | This script has two optional arguments: |
| 481 | |
| 482 | #. ansible version |
| 483 | #. docker image name |
| 484 | |
| 485 | **Note:** if optional arguments are not used, docker image name will be set to ``'ansible'`` by default. |
| 486 | |
| 487 | Launching ansible playbook using chroot environment |
| 488 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 489 | |
| 490 | This is the default and preferred way of running ansible playbooks in an offline environment as there is no dependency on docker to be installed on the system. Chroot environment is already provided by included archive ``'ansible_chroot.tgz'``. |
| 491 | |
Bartek Grzybowski | 1314127 | 2021-03-24 15:29:13 +0100 | [diff] [blame] | 492 | It should be available in the ``'ansible/docker'`` directory as the end-result of the packaging script or after manual run of the ``'build_ansible_image.sh'`` script referenced above. |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 493 | |
| 494 | All playbooks can be executed via ``'./run_playbook.sh'`` wrapper script. |
| 495 | |
| 496 | To get more info about the way how the ``'./run_playbook.sh'`` wrapper script should be used, run:: |
| 497 | |
| 498 | $ ./run_playbook.sh |
| 499 | |
| 500 | The main purpose of this wrapper script is to provide the ansible framework to a machine where it was bootstrapped without need of installing additional packages. The user can run this to display ``'ansible-playbook'`` command help:: |
| 501 | |
| 502 | $ ./run_playbook.sh --help |
| 503 | |
| 504 | Developers notes |
| 505 | ~~~~~~~~~~~~~~~~ |
| 506 | |
| 507 | * There are two scripts which work in tandem for creating and running chroot |
| 508 | * First one can convert docker image into chroot directory |
| 509 | * Second script will automate chrooting (necessary steps for chroot to work and cleanup) |
| 510 | * Both of them have help - just run:: |
| 511 | |
Bartek Grzybowski | 1314127 | 2021-03-24 15:29:13 +0100 | [diff] [blame] | 512 | $ cd ansible/docker |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 513 | $ ./create_docker_chroot.sh help |
| 514 | $ ./run_chroot.sh help |
| 515 | |
| 516 | Example usage:: |
| 517 | |
| 518 | $ sudo su |
Bartek Grzybowski | 1314127 | 2021-03-24 15:29:13 +0100 | [diff] [blame] | 519 | $ ansible/docker/create_docker_chroot.sh convert some_docker_image ./new_name_for_chroot |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 520 | $ cat ./new_name_for_chroot/README.md |
Bartek Grzybowski | 1314127 | 2021-03-24 15:29:13 +0100 | [diff] [blame] | 521 | $ ansible/docker/run_chroot.sh execute ./new_name_for_chroot cat /etc/os-release 2>/dev/null |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 522 | |
| 523 | Launching ansible playbook using docker container (ALTERNATIVE APPROACH) |
| 524 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 525 | |
| 526 | This option is here just to keep support for the older method which relies on a running docker service. For the offline deployment use the chroot option as indicated above. |
| 527 | |
| 528 | You will not need ``'ansible_chroot.tgz'`` archive anymore, but the new requirement is a prebuilt docker image of ansible (based on the provided ``'Dockerfile'``). It should be available in your local docker repository (otherwise the default name ``'ansible'`` may fetch unwanted image from default registry!). |
| 529 | |
| 530 | To trigger this functionality and to run ``'ansible-playbook'`` inside a docker container instead of the chroot environment, you must first set the ``ANSIBLE_DOCKER_IMAGE`` variable. The value must be a name of the built ansible docker image. |
| 531 | |
| 532 | Usage is basically the same as with the default chroot way - the only difference is the existence of the environment variable:: |
| 533 | |
| 534 | $ ANSIBLE_DOCKER_IMAGE=ansible ./run_playbook.sh --help |
| 535 | |
| 536 | ----- |
| 537 | |
Bartek Grzybowski | 40b4d15 | 2021-04-01 15:05:00 +0200 | [diff] [blame] | 538 | Appendix 2. Running Kubernetes Dashboard |
| 539 | ---------------------------------------- |
| 540 | |
| 541 | Kubernetes Dashboard is a web-based, general purpose user interface for managing a k8s cluster. |
| 542 | |
| 543 | Some of its capabilities are: |
| 544 | |
| 545 | * workloads/services management (troubleshooting, scaling, editing, restarting pods) |
| 546 | * deploying new workloads/applications to the cluster |
| 547 | * managing the cluster itself |
| 548 | |
| 549 | Dashboard also provides information on the state of the cluster resources and on any errors that may have occurred. |
| 550 | |
| 551 | Kubernetes Dashboard itself is a kubernetes application. For user convenience the Offline platform has it already pre-installed: |
| 552 | |
| 553 | :: |
| 554 | |
| 555 | $ kubectl -n kubernetes-dashboard get deployment |
| 556 | NAME READY UP-TO-DATE AVAILABLE AGE |
| 557 | dashboard-metrics-scraper 1/1 1 1 76m |
| 558 | kubernetes-dashboard 1/1 1 1 76m |
| 559 | |
| 560 | Accessing the dashboard |
| 561 | ~~~~~~~~~~~~~~~~~~~~~~~ |
| 562 | |
| 563 | There are multiple ways to access the application's web UI. Here we'll assume usage of local port forwarding on a box where you have access to a browser since the dashboard in Offline platform is exposed via a node port by default. |
| 564 | |
| 565 | First get the node port number that the dashboard service is exposed on: |
| 566 | |
| 567 | :: |
| 568 | |
| 569 | $ kubectl -n kubernetes-dashboard get svc kubernetes-dashboard -o custom-columns=PORTS:.spec.ports[].nodePort |
| 570 | PORTS |
| 571 | 30825 |
| 572 | |
| 573 | Now establish an ssh session to the infra node from your box from which you'll be accessing the dashboard: |
| 574 | |
| 575 | :: |
| 576 | |
| 577 | $ ssh -L 8080:127.0.0.1:30825 root@<infra host ip> |
| 578 | |
| 579 | Point your browser at https://localhost:8080/ and you should see the login page: |
| 580 | |
| 581 | .. image:: images/kubernetes-dashboard-signin.png |
| 582 | :alt: Kubernetes Dashboard signin |
| 583 | |
| 584 | Here, we'll leverage the Bearer Token to log in. Offline platform comes with dashboard admin user already created, we just need to extract its token. On the infra node issue following command: |
| 585 | |
| 586 | :: |
| 587 | |
| 588 | $ kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}" |
| 589 | |
| 590 | It will return the token string on stdout. Copy-paste it into the sign-in form, selecting the "Token" option first. Upon successful login you'll be presented the cluster resources from ``default`` namespace. In the drop down box at the top select the namespace into which you installed the Onap application (namespace name equals the value of ``app_name`` variable from offline-installer setup) and you should see the cluster resources for Onap: |
| 591 | |
| 592 | .. image:: images/kubernetes-dashboard-main.png |
| 593 | :alt: Kubernetes Dashboard main page |
| 594 | |
| 595 | For additional information concerning the Kubernetes Dashboard please refer to the `official documentation <https://github.com/kubernetes/dashboard/tree/master/docs>`_. |
| 596 | |
| 597 | ----- |
| 598 | |
Bartek Grzybowski | 8aa5c68 | 2021-06-21 16:04:38 +0200 | [diff] [blame] | 599 | Appendix 3. Running kube-prometheus stack |
| 600 | ----------------------------------------- |
| 601 | |
| 602 | `Kube-prometheus stack`_ is a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the `Prometheus Operator`_. |
| 603 | |
| 604 | The Stack is not deployed by default in Offline ONAP Platform, but all artifacts which it requires are downloaded by relevant scripts in the package build phase (see `Build Guide`_). |
| 605 | |
| 606 | Setup (optional) |
| 607 | ~~~~~~~~~~~~~~~~ |
| 608 | |
| 609 | Kube-prometheus stack itself is a Kubernetes native application provisioned using Helm Charts. As such it can be configured using Helm values. Offline Installer provides a handy way for passing those values to the helm installation process. |
| 610 | |
| 611 | Any values for the Stack should be defined as subkeys of **kube_prometheus_helm_values** variable in **application_configuration.yml**. For instance, in order to override the default Grafana password, insert below structure into application_configuration.yml:: |
| 612 | |
| 613 | kube_prometheus_helm_values: |
| 614 | grafana: |
| 615 | adminPassword: <password> |
| 616 | |
| 617 | Another example - to set custom storage size for Prometheus tsdb:: |
| 618 | |
| 619 | |
| 620 | kube_prometheus_helm_values: |
| 621 | prometheus: |
| 622 | prometheusSpec: |
| 623 | storageSpec: |
| 624 | volumeClaimTemplate: |
| 625 | spec: |
| 626 | resources: |
| 627 | requests: |
| 628 | storage: 6Gi |
| 629 | |
| 630 | A comprehensive list of Helm values for the Stack can be obtained on the `Kube-prometheus stack`_ project site, in the `values.yaml`_ file. Additional values for the Grafana can be checked on the `Grafana`_ project site in the *charts/grafana/values.yaml* file. |
| 631 | |
| 632 | Installation |
| 633 | ~~~~~~~~~~~~ |
| 634 | |
| 635 | In order to actually install this tool it's required to set the following variable in application_configuration.yml:: |
| 636 | |
| 637 | kube_prometheus_stack_enabled: true |
| 638 | |
| 639 | After the Offline Platform installation process is complete, the Stack will be deployed into its own kubernetes and helm namespace **kube-prometheus**. |
| 640 | |
Bartek Grzybowski | fd62ba6 | 2022-02-18 12:01:07 +0100 | [diff] [blame^] | 641 | ONAP Services Monitoring |
| 642 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
| 643 | |
| 644 | Some ONAP services export application metrics which can be scraped by Prometheus by leveraging the ServiceMonitor objects. Offline Platform provides a curated set of Grafana panels for monitoring ONAP's mariadb-galera chart. To enable mariadb-galera monitoring provide the following helm values in ``application_configuration.yml``:: |
| 645 | |
| 646 | overrides: |
| 647 | mariadb-galera: |
| 648 | metrics: |
| 649 | serviceMonitor: |
| 650 | enabled: true |
| 651 | basicAuth: |
| 652 | enabled: false |
| 653 | |
| 654 | To access the Galera/MariaDB dashboard navigate to *Dashboards -> Manage -> ONAP -> Galera/MariaDB* in Grafana UI. |
| 655 | |
Bartek Grzybowski | 8aa5c68 | 2021-06-21 16:04:38 +0200 | [diff] [blame] | 656 | Accessing Grafana dashboard |
| 657 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 658 | |
| 659 | The most straightforward way to access the Grafana UI is by leveraging the *port-forward* k8s facility. Issue following command on the Infra host:: |
| 660 | |
| 661 | kubectl -n kube-prometheus port-forward --address 0.0.0.0 svc/kube-prometheus-stack-grafana 8081:80 |
| 662 | |
| 663 | Then navigate to http://<infra IP>:8081 to access the UI: |
| 664 | |
| 665 | .. image:: images/grafana-signin.png |
| 666 | :alt: Grafana Login page |
| 667 | |
Bartek Grzybowski | 9d4c15d | 2021-12-09 13:29:37 +0100 | [diff] [blame] | 668 | Default username is *admin* and the default password is *grafana*. |
Bartek Grzybowski | 8aa5c68 | 2021-06-21 16:04:38 +0200 | [diff] [blame] | 669 | |
| 670 | In the left pane navigate to *Dashboards -> Manage* to see the various pre-defined dashboards that come bundled with kube-prometheus stack. There is also the *Custom* folder which holds few additional dashes defined by the Offline Installer authors: |
| 671 | |
| 672 | .. image:: images/grafana-dashboards.png |
| 673 | :alt: Grafana dashboards |
| 674 | |
Bartek Grzybowski | 327b7dc | 2021-06-24 11:29:58 +0200 | [diff] [blame] | 675 | Alternative way of accessing the UI is by leveraging the NodePort type service which exposes Grafana UI on the Infra host public port directly. To do so get the port number first:: |
| 676 | |
| 677 | kubectl -n kube-prometheus get service/kube-prometheus-stack-grafana -o custom-columns=PORTS:.spec.ports[].nodePort |
| 678 | |
| 679 | Then navigate to http://<infra IP>:<nodePort> to access the UI. |
| 680 | |
Petr Ospalý | be81ab0 | 2019-02-14 21:30:31 +0100 | [diff] [blame] | 681 | .. _Build Guide: ./BuildGuide.rst |
Bartek Grzybowski | 553a920 | 2021-03-23 15:09:11 +0100 | [diff] [blame] | 682 | .. _Software requirements: https://docs.onap.org/projects/onap-oom/en/latest/oom_cloud_setup_guide.html#software-requirements |
| 683 | .. _Hardware requirements: https://docs.onap.org/projects/onap-oom/en/latest/oom_cloud_setup_guide.html#minimum-hardware-configuration |
Bartek Grzybowski | 2936ef1 | 2021-03-23 14:15:33 +0100 | [diff] [blame] | 684 | .. _OOM ONAP: https://docs.onap.org/projects/onap-oom/en/latest/index.html |
| 685 | .. _Offline installer: https://gerrit.onap.org/r/q/oom/offline-installer |
Bartek Grzybowski | 553a920 | 2021-03-23 15:09:11 +0100 | [diff] [blame] | 686 | .. _RKE: https://rancher.com/products/rke/ |
Bartek Grzybowski | 8aa5c68 | 2021-06-21 16:04:38 +0200 | [diff] [blame] | 687 | .. _Kube-prometheus stack: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack |
| 688 | .. _Prometheus Operator: https://github.com/prometheus-operator/prometheus-operator |
| 689 | .. _values.yaml: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml |
| 690 | .. _Grafana: https://github.com/grafana/helm-charts |