blob: f34ee03e27ed3d14c9ac8f493dd15e2e2e94b872 [file] [log] [blame]
Petr Ospalýbe81ab02019-02-14 21:30:31 +01001.. This work is licensed under a Creative Commons Attribution 4.0 International License.
2.. http://creativecommons.org/licenses/by/4.0
3.. Copyright 2019 Samsung Electronics Co., Ltd.
4
5.. _oooi_installguide:
6
7OOM ONAP Offline Installer - Installation Guide
8===============================================
9
10This document describes the correct offline installation procedure for `OOM ONAP`_, which is done by the ansible based `offline-installer <https://gerrit.onap.org/r/#/admin/projects/oom/offline-installer>`_.
11
12Before you dive into the installation you should prepare the offline installer itself - the installer consists of at least two packages/resources. You can read about it in the `Build Guide`_, which provides the instructions for creating them.
13
14This current version of the *Installation Guide* supports `Casablanca release`_.
15
16-----
17
18.. _oooi_installguide_preparations:
19
20Part 1. Prerequisites
21---------------------
22
23OOM ONAP deployment has certain hardware resource requirements - `Casablanca requirements`_:
24
25- 14 VM (1 Rancher, 13 K8s nodes) - 8 vCPU - 16 GB RAM
26- 160 GB Storage
27
28That means the full deployment footprint is about ``224 GB RAM`` and ``112 vCPUs``. We will not follow strictly this setup due to such demanding resource consumption and so we will deploy our installation across four nodes (VMs) instead of fourteen. Our simplified setup is definitively not supported or recommended - you are free to diverge - you can follow the official guidelines or make completely different layout, but the minimal count of nodes should not drop below three - otherwise you may have to do some tweaking to make it work, which is not covered here (there is a pod count limit for a single kubernetes node - you can read more about it in this `discussion <https://lists.onap.org/g/onap-discuss/topic/oom_110_kubernetes_pod/25213556>`_).
29
30.. _oooi_installguide_preparations_k8s_cluster:
31
32Kubernetes cluster
33~~~~~~~~~~~~~~~~~~
34
35The four nodes/VMs will be running these services:
36
37- **infra-node**::
38
39 - nexus
40 - nginx proxy
41 - dns
42 - rancher server
43
44- **kubernetes node 1-3**::
45
46 - rancher agent
47
48You don't need to care about these services now - that is the responsibility of the installer (described below). Just start four VMs as seen in this table (or according to your needs as we hinted above):
49
50.. _Overview table of the kubernetes cluster:
51
52Kubernetes cluster overview
53^^^^^^^^^^^^^^^^^^^^^^^^^^^
54
55=================== ========= ==================== ============== ============ ===============
56KUBERNETES NODE OS NETWORK CPU RAM STORAGE
57=================== ========= ==================== ============== ============ ===============
58**infra-node** RHEL 7 ``10.8.8.100/24`` ``8 vCPUs`` ``8 GB`` ``100 GB``
59**kube-node1** RHEL 7 ``10.8.8.101/24`` ``16 vCPUs`` ``48+ GB`` ``100 GB``
60**kube-node2** RHEL 7 ``10.8.8.102/24`` ``16 vCPUs`` ``48+ GB`` ``100 GB``
61**kube-node3** RHEL 7 ``10.8.8.103/24`` ``16 vCPUs`` ``48+ GB`` ``100 GB``
62SUM ``56 vCPUs`` ``152+ GB`` ``400 GB``
63================================================== ============== ============ ===============
64
65Unfortunately, the offline installer supports only **RHEL 7.x** distribution as of now. So, your VMs should be preinstalled with this operating system - the hypervisor and platform can be of your choosing. It is also worth knowing that the exact RHEL version (major and minor number - 7.6 for example) should match for the package build procedure and the target installation. That means: if you are building packages on RHEL 7.6 release your VMs should be RHEL 7.6 too.
66
67We will expect from now on that you installed four VMs and they are connected to the shared network. All VMs must be reachable from our *install-server* (below), which can be the hypervisor, *infra-node* or completely different machine. But in either of these cases the *install-server* must be able to connect over ssh to all of these nodes.
68
69.. _oooi_installguide_preparations_installserver:
70
71Install-server
72~~~~~~~~~~~~~~
73
74We will use distinct *install-server* and keep it separate from the four-node cluster. But if you wish so, you can use *infra-node* for this goal (if you use the default ``'chroot'`` option of the installer), but in that case double the size of the storage requirement!
75
76Prerequisites for the *install-server*:
77
78- packages described in `Build Guide`_
79- extra ``100 GB`` storage (to have space where to store these packages)
80- installed ``'chroot'`` and/or ``'docker'`` system commands
81- network connection to the nodes - especially functioning ssh client
82
83Our *install-server* will have ip: ``10.8.8.4``.
84
85**NOTE:** All the subsequent commands below, are executed from within this *install-server*.
86
87-----
88
89.. _oooi_installguide_config:
90
91Part 2. Preparation and configuration
92-------------------------------------
93
94We *MUST* do all the following instructions from the *install-server* and also we will be running them as a user ``root``. But that is not necessary - you can without any problem pick and use a regular user. The ssh/ansible connection to the nodes will also expect that we are connecting as a ``root`` - you need to elevate privileges to be able to install on them. Although it can be achieved by other means (sudo), we decided here to keep instructions simple.
95
96.. _oooi_installguide_config_packages:
97
98Installer packages
99~~~~~~~~~~~~~~~~~~
100
101As was stated above you must have prepared the installer packages (names will differ - check out the `Build Guide`_):
102
103- offline-onap-3.0.1-resources.tar
104- offline-onap-3.0.1-aux-resources.tar
105- offline-onap-3.0.1-sw.tar
106
107**NOTE:** ``'offline-onap-3.0.1-aux-resources.tar'`` is optional and if you don't have use for it, you can ignore it.
108
109We will store them in the ``/data`` directory on the *install-server* and then we will unpack the ``'sw'`` package to your home directory for example::
110
111 $ mkdir ~/onap-offline-installer
112 $ tar -C ~/onap-offline-installer -xf /data/offline-onap-3.0.1-sw.tar
113
114.. _oooi_installguide_config_app:
115
116Application directory
117~~~~~~~~~~~~~~~~~~~~~
118
119Change the current directory to the ``'ansible'``::
120
121 $ cd ~/onap-offline-installer/ansible
122
123You can see multiple files and directories inside - this is the *offline-installer*. It is implemented as a set of ansible playbooks.
124
125If you created the ``'sw'`` package according to the *Build Guide* then you should had have the ``'application'`` directory populated with at least the following files:
126
127- ``application_configuration.yml``
128- ``hosts.yml``
129
130**NOTE:** The following paragraph describes a way how to create or fine-tune your own ``'application_configuration.yml'`` - we are discouraging you from executing this step. The recommended way is to use the packaged files inside the ``'application'`` directory.
131
132**NOT RECOMMENDED:** If for some reason you don't have these files inside the ``'application'`` directory or you simply want to do things the hard way then you can recreate them from their templates. It is better to keep the originals (templates) intact - so we will copy them to the ``'application'`` directory::
133
134 $ cp ../config/application_configuration.yml application/
135 $ cp inventory/hosts.yml application/
136
137.. _oooi_installguide_config_hosts:
138
139hosts.yml
140~~~~~~~~~
141
142We need to setup the ``'hosts.yml'`` first, the template looks like this::
143
144 ---
145 # This group contains hosts with all resources (binaries, packages, etc.)
146 # in tarball.
147 all:
148 vars:
149 # this key is supposed to be generated during setup.yml playbook execution
150 # change it just when you have better one working for all nodes
151 ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
152 ansible_ssh_common_args: '-o StrictHostKeyChecking=no'
153
154 children:
155 resources:
156 hosts:
157 resource-host:
158 ansible_host: 10.8.8.5
159
160 # This is group of hosts where nexus, nginx, dns and all other required
161 # services are running.
162 infrastructure:
163 hosts:
164 infrastructure-server:
165 ansible_host: 10.8.8.13
166 #IP used for communication between infra and kubernetes nodes, must be specified.
167 cluster_ip: 10.8.8.13
168
169 # This is group of hosts which are/will be part of Kubernetes cluster.
170 kubernetes:
171 hosts:
172 kubernetes-node-1:
173 ansible_host: 10.8.8.19
174 #ip of the node that it uses for communication with k8s cluster.
175 cluster_ip: 10.8.8.19
176
177 nfs-server:
178 hosts:
179 kubernetes-node-1
180
181There is some ssh configuration under the ``'vars'`` section - we will deal with ssh setup a little bit later in the `SSH authentication`_.
182
183We need to first correct the ip addresses and add a couple of kubernetes nodes to match our four-node cluster:
184
185- Under the ``'resource-host'`` set the ``'ansible_host'`` address to the ip of your server, where the packages are stored - it must be reachable by ssh from the *install-server* (for ansible to run playbooks on it) **AND** *infra-node* (to extract resource data from *resource-host* to *infra-node* over ssh). In our scenario the *resource-host* is the same as the *install-server*: ``'10.8.8.4'``
186- Similarly, set the ``'ansible_host'`` to the address of the *infra-node* under the ``'infrastructure-server'``.
187- Copy the whole ``'kubernetes-node-1'`` subsection and paste it twice directly after. Change the numbers to ``'kubernetes-node-2'`` and ``'kubernetes-node-3'`` respectively and fix the addresses in the ``'ansible_host'`` variables again to match *kube-node1*, *kube-node2* and *kube-node3*.
188
189As you can see, there is another ``'cluster_ip'`` variable for each node - this serve as a designated node address in the kubernetes cluster. Make it the same as the respective ``'ansible_host'``.
190
191**NOTE:** In our simple setup we have only one interface per node, but that does not need to be a case for some other deployment - especially if we start to deal with a production usage. Basically, an ``'ansible_host'`` is an entry point for the *install-server's* ansible (*offline-installer*), but the kubernetes cluster can be communicating on a separate network to which *install-server* has no access. That is why we have this distinctive variable, so we can tell the installer that there is a different network, where we want to run the kubernetes traffic and what address each node has on such a network.
192
193After all the changes, the ``'hosts.yml'`` should look similar to this::
194
195 ---
196 # This group contains hosts with all resources (binaries, packages, etc.)
197 # in tarball.
198 all:
199 vars:
200 # this key is supposed to be generated during setup.yml playbook execution
201 # change it just when you have better one working for all nodes
202 ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
203 ansible_ssh_common_args: '-o StrictHostKeyChecking=no'
204
205 children:
206 resources:
207 hosts:
208 resource-host:
209 ansible_host: 10.8.8.4
210
211 # This is group of hosts where nexus, nginx, dns and all other required
212 # services are running.
213 infrastructure:
214 hosts:
215 infrastructure-server:
216 ansible_host: 10.8.8.100
217 #IP used for communication between infra and kubernetes nodes, must be specified.
218 cluster_ip: 10.8.8.100
219
220 # This is group of hosts which are/will be part of Kubernetes cluster.
221 kubernetes:
222 hosts:
223 kubernetes-node-1:
224 ansible_host: 10.8.8.101
225 #ip of the node that it uses for communication with k8s cluster.
226 cluster_ip: 10.8.8.101
227 kubernetes-node-2:
228 ansible_host: 10.8.8.102
229 #ip of the node that it uses for communication with k8s cluster.
230 cluster_ip: 10.8.8.102
231 kubernetes-node-3:
232 ansible_host: 10.8.8.103
233 #ip of the node that it uses for communication with k8s cluster.
234 cluster_ip: 10.8.8.103
235
236 nfs-server:
237 hosts:
238 kubernetes-node-1
239
240.. _oooi_installguide_config_appconfig:
241
242application_configuration.yml
243~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
244
245Here, we will be interested in the following variables:
246
247- ``resources_dir``
248- ``resources_filename``
249- ``aux_resources_filename``
250- ``app_data_path``
251- ``aux_data_path``
252- ``app_name``
253
254``'resource_dir'``, ``'resources_filename'`` and ``'aux_resources_filename'`` must correspond to the file paths on the *resource-host* (variable ``'resource_host'``), which is in our case the *install-server*.
255
256The ``'resource_dir'`` should be set to ``'/data'``, ``'resources_filename'`` to ``'offline-onap-3.0.1-resources.tar'`` and ``'aux_resources_filename'`` to ``'offline-onap-3.0.1-aux-resources.tar'``. The values should be the same as are in the `Installer packages`_ section.
257
258``'app_data_path'`` is the absolute path on the *infra-node* to where the package ``'offline-onap-3.0.1-resources.tar'`` will be extracted and similarly ``'aux_data_path'`` is another absolute path for ``'offline-onap-3.0.1-aux-resources.tar'``. Both the paths are fully arbitrary, but they should point to the filesystem with enough space - the storage requirement in `Overview table of the kubernetes cluster`_.
259
260**NOTE:** As we mentioned in `Installer packages`_ - the auxiliary package is not mandatory and we will not utilize it in here either.
261
262The last variable ``'app_name'`` should be short and descriptive. We will set it simply to: ``onap``.
263
264It can look all together something like this::
265
266 resources_dir: /data
267 resources_filename: offline-onap-3.0.1-resources.tar
268 app_data_path: /opt/onap
269 app_name: onap
270
271.. _oooi_installguide_config_ssh:
272
273SSH authentication
274~~~~~~~~~~~~~~~~~~
275
276We are almost finished with the configuration and we are close to start the installation, but we need to setup password-less login from *install-server* to the nodes.
277
278You can use the ansible playbook ``'setup.yml'`` like this::
279
280 $ ./run_playbook.sh -i application/hosts.yml setup.yml -u root --ask-pass
281
282You will be asked for password per each node and the playbook will generate a unprotected ssh key-pair ``'~/.ssh/offline_ssh_key'``, which will be distributed to the nodes.
283
284Another option is to generate a ssh key-pair manually. We strongly advise you to protect it with a passphrase, but for simplicity we will showcase generating of a private key without any such protection::
285
286 $ ssh-keygen -N "" -f ~/.ssh/identity
287
288The next step will be to distribute the public key to these nodes and from that point no password is needed::
289
290 $ for ip in 100 101 102 103 ; do ssh-copy-id -i ~/.ssh/identity.pub root@10.8.8.${ip} ; done
291
292This command behaves almost identically to the ``'setup.yml'`` playbook.
293
294If you generated the ssh key manually then you can now run the ``'setup.yml'`` playbook like this and achieve the same result as in the first execution::
295
296 $ ./run_playbook.sh -i application/hosts.yml setup.yml
297
298This time it should not ask you for any password - of course this is very redundant, because you just distributed two ssh keys for no good reason.
299
300We can finally edit and finish the configuration of the ``'hosts.yml'``:
301
302- if you used the ``'setup.yml'`` playbook then you can just leave this line as it is::
303
304 ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
305
306- if you created a ssh key manually then change it like this::
307
308 ansible_ssh_private_key_file: /root/.ssh/identity
309
310-----
311
312.. _oooi_installguide_install:
313
314Part 3. Installation
315--------------------
316
317We should have the configuration complete and be ready to start the installation. The installation is done via ansible playbooks, which are run either inside a **chroot** environment (default) or from the **docker** container. If for some reason you want to run playbooks from the docker instead of chroot then you cannot use *infra-node* or any other *kube-node* as the *install-server* - otherwise you risk that installation will fail due to restarting of the docker service.
318
319If you built your ``'sw'`` package well then there should be the file ``'ansible_chroot.tgz'`` inside the ``'docker'`` directory. If not then you must create it - to learn how to do that and to get more info about the scripts dealing with docker and chroot, go to `Appendix 1. Ansible execution/bootstrap`_
320
321We will use the default chroot option so we don't need any docker service to be running.
322
323Installation is actually very straightforward now::
324
325 $ ./run_playbook.sh -i application/hosts.yml -e @application/application_configuration.yml site.yml
326
327This will take a while so be patient.
328
329``'site.yml'`` playbook actually runs in the order the following playbooks:
330
331- ``upload_resources.yml``
332- ``infrastructure.yml``
333- ``rancher_kubernetes.yml``
334- ``application.yml``
335
336After all the playbooks are finished, it will still take a lot of time until all pods will be up and running. You can monitor your newly created kubernetes cluster for example like this::
337
338 $ ssh -i ~/.ssh/offline_ssh_key root@10.8.8.4 # tailor this command to connect to your infra-node
339 $ watch -d -n 5 'kubectl get pods --all-namespaces'
340
341-----
342
343.. _oooi_installguide_appendix1:
344
345Appendix 1. Ansible execution/bootstrap
346---------------------------------------
347
348There are two ways how to easily run the installer's ansible playbooks:
349
350- If you already have or can install a docker then you can build the provided ``'Dockerfile'`` for the ansible and run playbooks in the docker container.
351- Another way to deploy ansible is via chroot environment which is bundled together within this directory.
352
353(Re)build docker image and/or chroot archive
354~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
355
356Inside the ``'docker'`` directory is the ``'Dockerfile'`` and ``'build_ansible_image.sh'`` script. You can run ``'build_ansible_image.sh'`` script on some machine with the internet connectivity and it will download all required packages needed for building the ansible docker image and for exporting it into a flat chroot environment.
357
358Built image is exported into ``'ansible_chroot.tgz'`` archive in the same (``'docker'``) directory.
359
360This script has two optional arguments:
361
362#. ansible version
363#. docker image name
364
365**Note:** if optional arguments are not used, docker image name will be set to ``'ansible'`` by default.
366
367Launching ansible playbook using chroot environment
368~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
369
370This is the default and preferred way of running ansible playbooks in an offline environment as there is no dependency on docker to be installed on the system. Chroot environment is already provided by included archive ``'ansible_chroot.tgz'``.
371
372It should be available in the ``'docker'`` directory as the end-result of the packaging script or after manual run of the ``'build_ansible_image.sh'`` script referenced above.
373
374All playbooks can be executed via ``'./run_playbook.sh'`` wrapper script.
375
376To get more info about the way how the ``'./run_playbook.sh'`` wrapper script should be used, run::
377
378 $ ./run_playbook.sh
379
380The main purpose of this wrapper script is to provide the ansible framework to a machine where it was bootstrapped without need of installing additional packages. The user can run this to display ``'ansible-playbook'`` command help::
381
382 $ ./run_playbook.sh --help
383
384Developers notes
385~~~~~~~~~~~~~~~~
386
387* There are two scripts which work in tandem for creating and running chroot
388* First one can convert docker image into chroot directory
389* Second script will automate chrooting (necessary steps for chroot to work and cleanup)
390* Both of them have help - just run::
391
392 $ cd docker
393 $ ./create_docker_chroot.sh help
394 $ ./run_chroot.sh help
395
396Example usage::
397
398 $ sudo su
399 $ docker/create_docker_chroot.sh convert some_docker_image ./new_name_for_chroot
400 $ cat ./new_name_for_chroot/README.md
401 $ docker/run_chroot.sh execute ./new_name_for_chroot cat /etc/os-release 2>/dev/null
402
403Launching ansible playbook using docker container (ALTERNATIVE APPROACH)
404~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
405
406This option is here just to keep support for the older method which relies on a running docker service. For the offline deployment use the chroot option as indicated above.
407
408You will not need ``'ansible_chroot.tgz'`` archive anymore, but the new requirement is a prebuilt docker image of ansible (based on the provided ``'Dockerfile'``). It should be available in your local docker repository (otherwise the default name ``'ansible'`` may fetch unwanted image from default registry!).
409
410To trigger this functionality and to run ``'ansible-playbook'`` inside a docker container instead of the chroot environment, you must first set the ``ANSIBLE_DOCKER_IMAGE`` variable. The value must be a name of the built ansible docker image.
411
412Usage is basically the same as with the default chroot way - the only difference is the existence of the environment variable::
413
414 $ ANSIBLE_DOCKER_IMAGE=ansible ./run_playbook.sh --help
415
416-----
417
418.. _Build Guide: ./BuildGuide.rst
419.. _Casablanca requirements: https://onap.readthedocs.io/en/casablanca/guides/onap-developer/settingup/index.html#installing-onap
420.. _Casablanca release: https://docs.onap.org/en/casablanca/release/
421.. _OOM ONAP: https://wiki.onap.org/display/DW/ONAP+Operations+Manager+Project