Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 1 | .. This work is licensed under a Creative Commons Attribution 4.0 |
| 2 | .. International License. |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 3 | .. http://creativecommons.org/licenses/by/4.0 |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 4 | .. Copyright 2018-2020 Amdocs, Bell Canada, Orange, Samsung |
Jessica Wagantall | afb1ead | 2020-04-09 12:39:10 -0700 | [diff] [blame] | 5 | .. _oom_user_guide: |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 6 | |
| 7 | .. Links |
| 8 | .. _Curated applications for Kubernetes: https://github.com/kubernetes/charts |
| 9 | .. _Services: https://kubernetes.io/docs/concepts/services-networking/service/ |
| 10 | .. _ReplicaSet: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/ |
| 11 | .. _StatefulSet: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ |
| 12 | .. _Helm Documentation: https://docs.helm.sh/helm/ |
| 13 | .. _Helm: https://docs.helm.sh/ |
| 14 | .. _Kubernetes: https://Kubernetes.io/ |
Eric Debeau | c4e405f | 2020-12-07 14:49:52 +0100 | [diff] [blame] | 15 | .. _Kubernetes LoadBalancer: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 16 | .. _user-guide-label: |
| 17 | |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 18 | OOM User Guide helm3 (experimental) |
| 19 | ################################### |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 20 | |
| 21 | The ONAP Operations Manager (OOM) provide the ability to manage the entire |
| 22 | life-cycle of an ONAP installation, from the initial deployment to final |
| 23 | decommissioning. This guide provides instructions for users of ONAP to |
| 24 | use the Kubernetes_/Helm_ system as a complete ONAP management system. |
| 25 | |
| 26 | This guide provides many examples of Helm command line operations. For a |
| 27 | complete description of these commands please refer to the `Helm |
| 28 | Documentation`_. |
| 29 | |
| 30 | .. figure:: oomLogoV2-medium.png |
| 31 | :align: right |
| 32 | |
| 33 | The following sections describe the life-cycle operations: |
| 34 | |
| 35 | - Deploy_ - with built-in component dependency management |
| 36 | - Configure_ - unified configuration across all ONAP components |
| 37 | - Monitor_ - real-time health monitoring feeding to a Consul UI and Kubernetes |
| 38 | - Heal_- failed ONAP containers are recreated automatically |
| 39 | - Scale_ - cluster ONAP services to enable seamless scaling |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 40 | - Upgrade_ - change-out containers or configuration with little or no service |
| 41 | impact |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 42 | - Delete_ - cleanup individual containers or entire deployments |
| 43 | |
| 44 | .. figure:: oomLogoV2-Deploy.png |
| 45 | :align: right |
| 46 | |
| 47 | Deploy |
| 48 | ====== |
| 49 | |
| 50 | The OOM team with assistance from the ONAP project teams, have built a |
| 51 | comprehensive set of Helm charts, yaml files very similar to TOSCA files, that |
| 52 | describe the composition of each of the ONAP components and the relationship |
| 53 | within and between components. Using this model Helm is able to deploy all of |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 54 | ONAP with a few simple commands. |
| 55 | |
| 56 | Pre-requisites |
| 57 | -------------- |
Piotr Marcinkiewicz | 2ca8a47 | 2021-04-29 13:29:02 +0200 | [diff] [blame] | 58 | Your environment must have the Kubernetes `kubectl` with Cert-Manager |
| 59 | and Helm setup as a one time activity. |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 60 | |
| 61 | Install Kubectl |
| 62 | ~~~~~~~~~~~~~~~ |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 63 | Enter the following to install kubectl (on Ubuntu, there are slight differences |
| 64 | on other O/Ss), the Kubernetes command line interface used to manage a |
| 65 | Kubernetes cluster:: |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 66 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 67 | > curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.15.11/bin/linux/amd64/kubectl |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 68 | > chmod +x ./kubectl |
| 69 | > sudo mv ./kubectl /usr/local/bin/kubectl |
| 70 | > mkdir ~/.kube |
| 71 | |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 72 | Paste kubectl config from Rancher (see the :ref:`cloud-setup-guide-label` for |
| 73 | alternative Kubernetes environment setups) into the `~/.kube/config` file. |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 74 | |
| 75 | Verify that the Kubernetes config is correct:: |
| 76 | |
| 77 | > kubectl get pods --all-namespaces |
| 78 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 79 | At this point you should see Kubernetes pods running. |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 80 | |
Piotr Marcinkiewicz | 2ca8a47 | 2021-04-29 13:29:02 +0200 | [diff] [blame] | 81 | Install Cert-Manager |
| 82 | ~~~~~~~~~~~~~~~~~~~~ |
| 83 | Details on how to install Cert-Manager can be found |
| 84 | :doc:`here <oom_setup_paas>`. |
| 85 | |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 86 | Install Helm |
| 87 | ~~~~~~~~~~~~ |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 88 | Helm is used by OOM for package and configuration management. To install Helm, |
| 89 | enter the following:: |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 90 | |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 91 | > wget https://get.helm.sh/helm-v3.5.2-linux-amd64.tar.gz |
| 92 | > tar -zxvf helm-v3.5.2-linux-amd64.tar.gz |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 93 | > sudo mv linux-amd64/helm /usr/local/bin/helm |
| 94 | |
| 95 | Verify the Helm version with:: |
| 96 | |
| 97 | > helm version |
| 98 | |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 99 | Install the Helm Repo |
| 100 | --------------------- |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 101 | Once kubectl and Helm are setup, one needs to setup a local Helm server to |
| 102 | server up the ONAP charts:: |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 103 | |
| 104 | > helm install osn/onap |
| 105 | |
| 106 | .. note:: |
| 107 | The osn repo is not currently available so creation of a local repository is |
| 108 | required. |
| 109 | |
| 110 | Helm is able to use charts served up from a repository and comes setup with a |
| 111 | default CNCF provided `Curated applications for Kubernetes`_ repository called |
| 112 | stable which should be removed to avoid confusion:: |
| 113 | |
| 114 | > helm repo remove stable |
| 115 | |
| 116 | .. To setup the Open Source Networking Nexus repository for helm enter:: |
| 117 | .. > helm repo add osn 'https://nexus3.onap.org:10001/helm/helm-repo-in-nexus/master/' |
| 118 | |
| 119 | To prepare your system for an installation of ONAP, you'll need to:: |
| 120 | |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 121 | > git clone -b guilin --recurse-submodules -j2 http://gerrit.onap.org/r/oom |
Roger Maitland | bb8adda | 2018-04-05 16:18:11 -0400 | [diff] [blame] | 122 | > cd oom/kubernetes |
| 123 | |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 124 | |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 125 | To install a local Helm server:: |
| 126 | |
| 127 | > curl -LO https://s3.amazonaws.com/chartmuseum/release/latest/bin/linux/amd64/chartmuseum |
| 128 | > chmod +x ./chartmuseum |
| 129 | > mv ./chartmuseum /usr/local/bin |
| 130 | |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 131 | To setup a local Helm server to server up the ONAP charts:: |
| 132 | |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 133 | > mkdir -p ~/helm3-storage |
| 134 | > chartmuseum --storage local --storage-local-rootdir ~/helm3-storage -port 8879 & |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 135 | |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 136 | Note the port number that is listed and use it in the Helm repo add as |
| 137 | follows:: |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 138 | |
| 139 | > helm repo add local http://127.0.0.1:8879 |
| 140 | |
| 141 | To get a list of all of the available Helm chart repositories:: |
| 142 | |
| 143 | > helm repo list |
| 144 | NAME URL |
| 145 | local http://127.0.0.1:8879 |
| 146 | |
Roger Maitland | 9e5067c | 2018-03-27 10:57:08 -0400 | [diff] [blame] | 147 | Then build your local Helm repository:: |
| 148 | |
Jakub Latusek | db52a6d | 2020-10-15 15:02:47 +0200 | [diff] [blame] | 149 | > make SKIP_LINT=TRUE [HELM_BIN=<HELM_PATH>] all |
| 150 | |
| 151 | `HELM_BIN` |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 152 | Sets the helm binary to be used. The default value use helm from PATH |
Roger Maitland | 9e5067c | 2018-03-27 10:57:08 -0400 | [diff] [blame] | 153 | |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 154 | The Helm search command reads through all of the repositories configured on the |
| 155 | system, and looks for matches:: |
| 156 | |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 157 | > helm search repo local |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 158 | NAME VERSION DESCRIPTION |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 159 | local/appc 2.0.0 Application Controller |
| 160 | local/clamp 2.0.0 ONAP Clamp |
| 161 | local/common 2.0.0 Common templates for inclusion in other charts |
| 162 | local/onap 2.0.0 Open Network Automation Platform (ONAP) |
| 163 | local/robot 2.0.0 A helm Chart for kubernetes-ONAP Robot |
| 164 | local/so 2.0.0 ONAP Service Orchestrator |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 165 | |
| 166 | In any case, setup of the Helm repository is a one time activity. |
| 167 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 168 | Next, install Helm Plugins required to deploy the ONAP release:: |
Pawel Wieczorek | 1d4b96f | 2019-01-23 16:46:56 +0100 | [diff] [blame] | 169 | |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 170 | > cp -R ~/oom/kubernetes/helm/plugins/ ~/.local/share/helm/plugins |
Pawel Wieczorek | 1d4b96f | 2019-01-23 16:46:56 +0100 | [diff] [blame] | 171 | |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 172 | Once the repo is setup, installation of ONAP can be done with a single |
| 173 | command:: |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 174 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 175 | > helm deploy development local/onap --namespace onap --set global.masterPassword=password |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 176 | |
| 177 | This will install ONAP from a local repository in a 'development' Helm release. |
| 178 | As described below, to override the default configuration values provided by |
| 179 | OOM, an environment file can be provided on the command line as follows:: |
| 180 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 181 | |
| 182 | |
| 183 | > helm deploy development local/onap --namespace onap -f overrides.yaml --set global.masterPassword=password |
| 184 | |
| 185 | .. note:: |
| 186 | Refer the Configure_ section on how to update overrides.yaml and values.yaml |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 187 | |
| 188 | To get a summary of the status of all of the pods (containers) running in your |
| 189 | deployment:: |
| 190 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 191 | > kubectl get pods --namespace onap -o=wide |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 192 | |
| 193 | .. note:: |
| 194 | The Kubernetes namespace concept allows for multiple instances of a component |
| 195 | (such as all of ONAP) to co-exist with other components in the same |
| 196 | Kubernetes cluster by isolating them entirely. Namespaces share only the |
| 197 | hosts that form the cluster thus providing isolation between production and |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 198 | development systems as an example. |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 199 | |
| 200 | .. note:: |
Roger Maitland | d96413f | 2018-04-09 10:06:07 -0400 | [diff] [blame] | 201 | The Helm `--name` option refers to a release name and not a Kubernetes namespace. |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 202 | |
| 203 | |
| 204 | To install a specific version of a single ONAP component (`so` in this example) |
Pawel Wieczorek | 1d4b96f | 2019-01-23 16:46:56 +0100 | [diff] [blame] | 205 | with the given release name enter:: |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 206 | |
Sylvain Desbureaux | c47ce73 | 2021-10-12 15:15:10 +0200 | [diff] [blame^] | 207 | > helm deploy so onap/so --version 9.0.0 --set global.masterPassword=password --set global.flavor=unlimited --namespace onap |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 208 | |
| 209 | .. note:: |
| 210 | The dependent components should be installed for component being installed |
| 211 | |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 212 | |
| 213 | To display details of a specific resource or group of resources type:: |
| 214 | |
| 215 | > kubectl describe pod so-1071802958-6twbl |
| 216 | |
| 217 | where the pod identifier refers to the auto-generated pod identifier. |
| 218 | |
| 219 | .. figure:: oomLogoV2-Configure.png |
| 220 | :align: right |
| 221 | |
| 222 | Configure |
| 223 | ========= |
| 224 | |
| 225 | Each project within ONAP has its own configuration data generally consisting |
| 226 | of: environment variables, configuration files, and database initial values. |
| 227 | Many technologies are used across the projects resulting in significant |
| 228 | operational complexity and an inability to apply global parameters across the |
| 229 | entire ONAP deployment. OOM solves this problem by introducing a common |
| 230 | configuration technology, Helm charts, that provide a hierarchical |
Gildas Lanilis | 64d17ae | 2018-05-18 16:58:05 -0700 | [diff] [blame] | 231 | configuration with the ability to override values with higher |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 232 | level charts or command line options. |
| 233 | |
| 234 | The structure of the configuration of ONAP is shown in the following diagram. |
| 235 | Note that key/value pairs of a parent will always take precedence over those |
| 236 | of a child. Also note that values set on the command line have the highest |
| 237 | precedence of all. |
| 238 | |
| 239 | .. graphviz:: |
| 240 | |
| 241 | digraph config { |
| 242 | { |
| 243 | node [shape=folder] |
| 244 | oValues [label="values.yaml"] |
| 245 | demo [label="onap-demo.yaml"] |
| 246 | prod [label="onap-production.yaml"] |
| 247 | oReq [label="requirements.yaml"] |
| 248 | soValues [label="values.yaml"] |
| 249 | soReq [label="requirements.yaml"] |
| 250 | mdValues [label="values.yaml"] |
| 251 | } |
| 252 | { |
| 253 | oResources [label="resources"] |
| 254 | } |
| 255 | onap -> oResources |
| 256 | onap -> oValues |
| 257 | oResources -> environments |
| 258 | oResources -> oReq |
| 259 | oReq -> so |
| 260 | environments -> demo |
| 261 | environments -> prod |
| 262 | so -> soValues |
| 263 | so -> soReq |
| 264 | so -> charts |
| 265 | charts -> mariadb |
| 266 | mariadb -> mdValues |
| 267 | |
| 268 | } |
| 269 | |
| 270 | The top level onap/values.yaml file contains the values required to be set |
| 271 | before deploying ONAP. Here is the contents of this file: |
| 272 | |
Pawel Wieczorek | a1903d6 | 2019-11-14 14:19:59 +0100 | [diff] [blame] | 273 | .. include:: ../kubernetes/onap/values.yaml |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 274 | :code: yaml |
| 275 | |
| 276 | One may wish to create a value file that is specific to a given deployment such |
| 277 | that it can be differentiated from other deployments. For example, a |
| 278 | onap-development.yaml file may create a minimal environment for development |
| 279 | while onap-production.yaml might describe a production deployment that operates |
| 280 | independently of the developer version. |
| 281 | |
| 282 | For example, if the production OpenStack instance was different from a |
| 283 | developer's instance, the onap-production.yaml file may contain a different |
| 284 | value for the vnfDeployment/openstack/oam_network_cidr key as shown below. |
| 285 | |
| 286 | .. code-block:: yaml |
| 287 | |
| 288 | nsPrefix: onap |
| 289 | nodePortPrefix: 302 |
| 290 | apps: consul msb mso message-router sdnc vid robot portal policy appc aai |
| 291 | sdc dcaegen2 log cli multicloud clamp vnfsdk aaf kube2msb |
| 292 | dataRootDir: /dockerdata-nfs |
| 293 | |
| 294 | # docker repositories |
| 295 | repository: |
| 296 | onap: nexus3.onap.org:10001 |
| 297 | oom: oomk8s |
| 298 | aai: aaionap |
| 299 | filebeat: docker.elastic.co |
| 300 | |
| 301 | image: |
| 302 | pullPolicy: Never |
| 303 | |
| 304 | # vnf deployment environment |
| 305 | vnfDeployment: |
| 306 | openstack: |
| 307 | ubuntu_14_image: "Ubuntu_14.04.5_LTS" |
| 308 | public_net_id: "e8f51956-00dd-4425-af36-045716781ffc" |
| 309 | oam_network_id: "d4769dfb-c9e4-4f72-b3d6-1d18f4ac4ee6" |
| 310 | oam_subnet_id: "191f7580-acf6-4c2b-8ec0-ba7d99b3bc4e" |
| 311 | oam_network_cidr: "192.168.30.0/24" |
| 312 | <...> |
| 313 | |
| 314 | |
| 315 | To deploy ONAP with this environment file, enter:: |
| 316 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 317 | > helm deploy local/onap -n onap -f onap/resources/environments/onap-production.yaml --set global.masterPassword=password |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 318 | |
| 319 | .. include:: environments_onap_demo.yaml |
| 320 | :code: yaml |
| 321 | |
| 322 | When deploying all of ONAP a requirements.yaml file control which and what |
| 323 | version of the ONAP components are included. Here is an excerpt of this |
| 324 | file: |
| 325 | |
| 326 | .. code-block:: yaml |
| 327 | |
| 328 | # Referencing a named repo called 'local'. |
| 329 | # Can add this repo by running commands like: |
| 330 | # > helm serve |
| 331 | # > helm repo add local http://127.0.0.1:8879 |
| 332 | dependencies: |
| 333 | <...> |
| 334 | - name: so |
Sylvain Desbureaux | c47ce73 | 2021-10-12 15:15:10 +0200 | [diff] [blame^] | 335 | version: ~9.0.0 |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 336 | repository: '@local' |
| 337 | condition: so.enabled |
| 338 | <...> |
| 339 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 340 | The ~ operator in the `so` version value indicates that the latest "8.X.X" |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 341 | version of `so` shall be used thus allowing the chart to allow for minor |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 342 | upgrades that don't impact the so API; hence, version 8.0.1 will be installed |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 343 | in this case. |
| 344 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 345 | The onap/resources/environment/dev.yaml (see the excerpt below) enables |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 346 | for fine grained control on what components are included as part of this |
| 347 | deployment. By changing this `so` line to `enabled: false` the `so` component |
| 348 | will not be deployed. If this change is part of an upgrade the existing `so` |
| 349 | component will be shut down. Other `so` parameters and even `so` child values |
| 350 | can be modified, for example the `so`'s `liveness` probe could be disabled |
| 351 | (which is not recommended as this change would disable auto-healing of `so`). |
| 352 | |
| 353 | .. code-block:: yaml |
| 354 | |
| 355 | ################################################################# |
| 356 | # Global configuration overrides. |
| 357 | # |
| 358 | # These overrides will affect all helm charts (ie. applications) |
| 359 | # that are listed below and are 'enabled'. |
| 360 | ################################################################# |
| 361 | global: |
| 362 | <...> |
| 363 | |
| 364 | ################################################################# |
| 365 | # Enable/disable and configure helm charts (ie. applications) |
| 366 | # to customize the ONAP deployment. |
| 367 | ################################################################# |
| 368 | aaf: |
| 369 | enabled: false |
| 370 | <...> |
| 371 | so: # Service Orchestrator |
| 372 | enabled: true |
| 373 | |
| 374 | replicaCount: 1 |
| 375 | |
| 376 | liveness: |
| 377 | # necessary to disable liveness probe when setting breakpoints |
| 378 | # in debugger so K8s doesn't restart unresponsive container |
| 379 | enabled: true |
| 380 | |
| 381 | <...> |
| 382 | |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 383 | Accessing the ONAP Portal using OOM and a Kubernetes Cluster |
| 384 | ------------------------------------------------------------ |
| 385 | |
| 386 | The ONAP deployment created by OOM operates in a private IP network that isn't |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 387 | publicly accessible (i.e. OpenStack VMs with private internal network) which |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 388 | blocks access to the ONAP Portal. To enable direct access to this Portal from a |
| 389 | user's own environment (a laptop etc.) the portal application's port 8989 is |
| 390 | exposed through a `Kubernetes LoadBalancer`_ object. |
| 391 | |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 392 | Typically, to be able to access the Kubernetes nodes publicly a public address |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 393 | is assigned. In OpenStack this is a floating IP address. |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 394 | |
| 395 | When the `portal-app` chart is deployed a Kubernetes service is created that |
| 396 | instantiates a load balancer. The LB chooses the private interface of one of |
| 397 | the nodes as in the example below (10.0.0.4 is private to the K8s cluster only). |
| 398 | Then to be able to access the portal on port 8989 from outside the K8s & |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 399 | OpenStack environment, the user needs to assign/get the floating IP address that |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 400 | corresponds to the private IP as follows:: |
| 401 | |
| 402 | > kubectl -n onap get services|grep "portal-app" |
| 403 | portal-app LoadBalancer 10.43.142.201 10.0.0.4 8989:30215/TCP,8006:30213/TCP,8010:30214/TCP 1d app=portal-app,release=dev |
| 404 | |
| 405 | |
| 406 | In this example, use the 10.0.0.4 private address as a key find the |
| 407 | corresponding public address which in this example is 10.12.6.155. If you're |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 408 | using OpenStack you'll do the lookup with the horizon GUI or the OpenStack CLI |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 409 | for your tenant (openstack server list). That IP is then used in your |
| 410 | `/etc/hosts` to map the fixed DNS aliases required by the ONAP Portal as shown |
| 411 | below:: |
| 412 | |
| 413 | 10.12.6.155 portal.api.simpledemo.onap.org |
| 414 | 10.12.6.155 vid.api.simpledemo.onap.org |
| 415 | 10.12.6.155 sdc.api.fe.simpledemo.onap.org |
andreasgeissler | 4a618ba | 2018-11-30 14:20:46 +0000 | [diff] [blame] | 416 | 10.12.6.155 sdc.workflow.plugin.simpledemo.onap.org |
| 417 | 10.12.6.155 sdc.dcae.plugin.simpledemo.onap.org |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 418 | 10.12.6.155 portal-sdk.simpledemo.onap.org |
| 419 | 10.12.6.155 policy.api.simpledemo.onap.org |
| 420 | 10.12.6.155 aai.api.sparky.simpledemo.onap.org |
| 421 | 10.12.6.155 cli.api.simpledemo.onap.org |
| 422 | 10.12.6.155 msb.api.discovery.simpledemo.onap.org |
andreasgeissler | 4a618ba | 2018-11-30 14:20:46 +0000 | [diff] [blame] | 423 | 10.12.6.155 msb.api.simpledemo.onap.org |
| 424 | 10.12.6.155 clamp.api.simpledemo.onap.org |
| 425 | 10.12.6.155 so.api.simpledemo.onap.org |
Sylvain Desbureaux | 1037d75 | 2020-04-20 14:17:16 +0200 | [diff] [blame] | 426 | 10.12.6.155 sdc.workflow.plugin.simpledemo.onap.org |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 427 | |
| 428 | Ensure you've disabled any proxy settings the browser you are using to access |
andreasgeissler | 4a618ba | 2018-11-30 14:20:46 +0000 | [diff] [blame] | 429 | the portal and then simply access now the new ssl-encrypted URL: |
Eric Debeau | c4e405f | 2020-12-07 14:49:52 +0100 | [diff] [blame] | 430 | ``https://portal.api.simpledemo.onap.org:30225/ONAPPORTAL/login.htm`` |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 431 | |
andreasgeissler | 4a618ba | 2018-11-30 14:20:46 +0000 | [diff] [blame] | 432 | .. note:: |
| 433 | Using the HTTPS based Portal URL the Browser needs to be configured to accept |
| 434 | unsecure credentials. |
| 435 | Additionally when opening an Application inside the Portal, the Browser |
| 436 | might block the content, which requires to disable the blocking and reloading |
| 437 | of the page |
| 438 | |
| 439 | .. note:: |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 440 | Besides the ONAP Portal the Components can deliver additional user interfaces, |
andreasgeissler | 4a618ba | 2018-11-30 14:20:46 +0000 | [diff] [blame] | 441 | please check the Component specific documentation. |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 442 | |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 443 | .. note:: |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 444 | |
Hector Anapan-Lavalle | 55547da | 2018-07-26 13:33:17 -0400 | [diff] [blame] | 445 | | Alternatives Considered: |
| 446 | |
guillaume.lambert | f3319a8 | 2021-09-26 21:37:50 +0200 | [diff] [blame] | 447 | - Kubernetes port forwarding was considered but discarded as it would |
| 448 | require the end user to run a script that opens up port forwarding tunnels |
| 449 | to each of the pods that provides a portal application widget. |
Hector Anapan-Lavalle | 55547da | 2018-07-26 13:33:17 -0400 | [diff] [blame] | 450 | |
| 451 | - Reverting to a VNC server similar to what was deployed in the Amsterdam |
guillaume.lambert | f3319a8 | 2021-09-26 21:37:50 +0200 | [diff] [blame] | 452 | release was also considered but there were many issues with resolution, |
| 453 | lack of volume mount, /etc/hosts dynamic update, file upload that were |
| 454 | a tall order to solve in time for the Beijing release. |
Hector Anapan-Lavalle | 55547da | 2018-07-26 13:33:17 -0400 | [diff] [blame] | 455 | |
| 456 | Observations: |
| 457 | |
guillaume.lambert | f3319a8 | 2021-09-26 21:37:50 +0200 | [diff] [blame] | 458 | - If you are not using floating IPs in your Kubernetes deployment and |
| 459 | directly attaching a public IP address (i.e. by using your public provider |
| 460 | network) to your K8S Node VMs' network interface, then the output of |
| 461 | 'kubectl -n onap get services | grep "portal-app"' |
Hector Anapan-Lavalle | 55547da | 2018-07-26 13:33:17 -0400 | [diff] [blame] | 462 | will show your public IP instead of the private network's IP. Therefore, |
guillaume.lambert | f3319a8 | 2021-09-26 21:37:50 +0200 | [diff] [blame] | 463 | you can grab this public IP directly (as compared to trying to find the |
| 464 | floating IP first) and map this IP in /etc/hosts. |
Roger Maitland | da22158 | 2018-05-10 13:43:58 -0400 | [diff] [blame] | 465 | |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 466 | .. figure:: oomLogoV2-Monitor.png |
| 467 | :align: right |
| 468 | |
| 469 | Monitor |
| 470 | ======= |
| 471 | |
| 472 | All highly available systems include at least one facility to monitor the |
| 473 | health of components within the system. Such health monitors are often used as |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 474 | inputs to distributed coordination systems (such as etcd, Zookeeper, or Consul) |
| 475 | and monitoring systems (such as Nagios or Zabbix). OOM provides two mechanisms |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 476 | to monitor the real-time health of an ONAP deployment: |
| 477 | |
| 478 | - a Consul GUI for a human operator or downstream monitoring systems and |
| 479 | Kubernetes liveness probes that enable automatic healing of failed |
| 480 | containers, and |
| 481 | - a set of liveness probes which feed into the Kubernetes manager which |
| 482 | are described in the Heal section. |
| 483 | |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 484 | Within ONAP, Consul is the monitoring system of choice and deployed by OOM in |
| 485 | two parts: |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 486 | |
| 487 | - a three-way, centralized Consul server cluster is deployed as a highly |
Gildas Lanilis | 64d17ae | 2018-05-18 16:58:05 -0700 | [diff] [blame] | 488 | available monitor of all of the ONAP components, and |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 489 | - a number of Consul agents. |
| 490 | |
| 491 | The Consul server provides a user interface that allows a user to graphically |
| 492 | view the current health status of all of the ONAP components for which agents |
| 493 | have been created - a sample from the ONAP Integration labs follows: |
| 494 | |
| 495 | .. figure:: consulHealth.png |
| 496 | :align: center |
| 497 | |
Eric Debeau | c4e405f | 2020-12-07 14:49:52 +0100 | [diff] [blame] | 498 | To see the real-time health of a deployment go to: ``http://<kubernetes IP>:30270/ui/`` |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 499 | where a GUI much like the following will be found: |
| 500 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 501 | .. note:: |
| 502 | If Consul GUI is not accessible, you can refer this |
| 503 | `kubectl port-forward <https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/>`_ method to access an application |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 504 | |
| 505 | .. figure:: oomLogoV2-Heal.png |
| 506 | :align: right |
| 507 | |
| 508 | Heal |
| 509 | ==== |
| 510 | |
| 511 | The ONAP deployment is defined by Helm charts as mentioned earlier. These Helm |
| 512 | charts are also used to implement automatic recoverability of ONAP components |
| 513 | when individual components fail. Once ONAP is deployed, a "liveness" probe |
| 514 | starts checking the health of the components after a specified startup time. |
| 515 | |
| 516 | Should a liveness probe indicate a failed container it will be terminated and a |
| 517 | replacement will be started in its place - containers are ephemeral. Should the |
| 518 | deployment specification indicate that there are one or more dependencies to |
| 519 | this container or component (for example a dependency on a database) the |
| 520 | dependency will be satisfied before the replacement container/component is |
| 521 | started. This mechanism ensures that, after a failure, all of the ONAP |
| 522 | components restart successfully. |
| 523 | |
| 524 | To test healing, the following command can be used to delete a pod:: |
| 525 | |
| 526 | > kubectl delete pod [pod name] -n [pod namespace] |
| 527 | |
| 528 | One could then use the following command to monitor the pods and observe the |
| 529 | pod being terminated and the service being automatically healed with the |
| 530 | creation of a replacement pod:: |
| 531 | |
| 532 | > kubectl get pods --all-namespaces -o=wide |
| 533 | |
| 534 | .. figure:: oomLogoV2-Scale.png |
| 535 | :align: right |
| 536 | |
| 537 | Scale |
| 538 | ===== |
| 539 | |
| 540 | Many of the ONAP components are horizontally scalable which allows them to |
| 541 | adapt to expected offered load. During the Beijing release scaling is static, |
| 542 | that is during deployment or upgrade a cluster size is defined and this cluster |
| 543 | will be maintained even in the presence of faults. The parameter that controls |
| 544 | the cluster size of a given component is found in the values.yaml file for that |
| 545 | component. Here is an excerpt that shows this parameter: |
| 546 | |
| 547 | .. code-block:: yaml |
| 548 | |
| 549 | # default number of instances |
| 550 | replicaCount: 1 |
| 551 | |
| 552 | In order to change the size of a cluster, an operator could use a helm upgrade |
| 553 | (described in detail in the next section) as follows:: |
| 554 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 555 | > helm upgrade [RELEASE] [CHART] [flags] |
| 556 | |
| 557 | The RELEASE argument can be obtained from the following command:: |
| 558 | |
| 559 | > helm list |
| 560 | |
| 561 | Below is the example for the same:: |
| 562 | |
| 563 | > helm list |
| 564 | NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE |
Sylvain Desbureaux | c47ce73 | 2021-10-12 15:15:10 +0200 | [diff] [blame^] | 565 | dev 1 Wed Oct 14 13:49:52 2020 DEPLOYED onap-9.0.0 Istanbul onap |
| 566 | dev-cassandra 5 Thu Oct 15 14:45:34 2020 DEPLOYED cassandra-9.0.0 onap |
| 567 | dev-contrib 1 Wed Oct 14 13:52:53 2020 DEPLOYED contrib-9.0.0 onap |
| 568 | dev-mariadb-galera 1 Wed Oct 14 13:55:56 2020 DEPLOYED mariadb-galera-9.0.0 onap |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 569 | |
| 570 | Here the Name column shows the RELEASE NAME, In our case we want to try the |
| 571 | scale operation on cassandra, thus the RELEASE NAME would be dev-cassandra. |
| 572 | |
guillaume.lambert | b90e8fc | 2021-09-26 21:28:50 +0200 | [diff] [blame] | 573 | Now we need to obtain the chart name for cassandra. Use the below |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 574 | command to get the chart name:: |
| 575 | |
| 576 | > helm search cassandra |
| 577 | |
| 578 | Below is the example for the same:: |
| 579 | |
| 580 | > helm search cassandra |
| 581 | NAME CHART VERSION APP VERSION DESCRIPTION |
Sylvain Desbureaux | c47ce73 | 2021-10-12 15:15:10 +0200 | [diff] [blame^] | 582 | local/cassandra 9.0.0 ONAP cassandra |
| 583 | local/portal-cassandra 9.0.0 Portal cassandra |
| 584 | local/aaf-cass 9.0.0 ONAP AAF cassandra |
| 585 | local/sdc-cs 9.0.0 ONAP Service Design and Creation Cassandra |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 586 | |
| 587 | Here the Name column shows the chart name. As we want to try the scale |
guillaume.lambert | b90e8fc | 2021-09-26 21:28:50 +0200 | [diff] [blame] | 588 | operation for cassandra, thus the corresponding chart name is local/cassandra |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 589 | |
| 590 | |
| 591 | Now we have both the command's arguments, thus we can perform the |
guillaume.lambert | b90e8fc | 2021-09-26 21:28:50 +0200 | [diff] [blame] | 592 | scale operation for cassandra as follows:: |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 593 | |
| 594 | > helm upgrade dev-cassandra local/cassandra --set replicaCount=3 |
| 595 | |
guillaume.lambert | b90e8fc | 2021-09-26 21:28:50 +0200 | [diff] [blame] | 596 | Using this command we can scale up or scale down the cassandra db instances. |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 597 | |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 598 | |
| 599 | The ONAP components use Kubernetes provided facilities to build clustered, |
| 600 | highly available systems including: Services_ with load-balancers, ReplicaSet_, |
| 601 | and StatefulSet_. Some of the open-source projects used by the ONAP components |
| 602 | directly support clustered configurations, for example ODL and MariaDB Galera. |
| 603 | |
| 604 | The Kubernetes Services_ abstraction to provide a consistent access point for |
| 605 | each of the ONAP components, independent of the pod or container architecture |
| 606 | of that component. For example, SDN-C uses OpenDaylight clustering with a |
| 607 | default cluster size of three but uses a Kubernetes service to and change the |
| 608 | number of pods in this abstract this cluster from the other ONAP components |
| 609 | such that the cluster could change size and this change is isolated from the |
| 610 | other ONAP components by the load-balancer implemented in the ODL service |
| 611 | abstraction. |
| 612 | |
| 613 | A ReplicaSet_ is a construct that is used to describe the desired state of the |
| 614 | cluster. For example 'replicas: 3' indicates to Kubernetes that a cluster of 3 |
| 615 | instances is the desired state. Should one of the members of the cluster fail, |
| 616 | a new member will be automatically started to replace it. |
| 617 | |
| 618 | Some of the ONAP components many need a more deterministic deployment; for |
| 619 | example to enable intra-cluster communication. For these applications the |
| 620 | component can be deployed as a Kubernetes StatefulSet_ which will maintain a |
| 621 | persistent identifier for the pods and thus a stable network id for the pods. |
| 622 | For example: the pod names might be web-0, web-1, web-{N-1} for N 'web' pods |
| 623 | with corresponding DNS entries such that intra service communication is simple |
| 624 | even if the pods are physically distributed across multiple nodes. An example |
| 625 | of how these capabilities can be used is described in the Running Consul on |
| 626 | Kubernetes tutorial. |
| 627 | |
| 628 | .. figure:: oomLogoV2-Upgrade.png |
| 629 | :align: right |
| 630 | |
| 631 | Upgrade |
| 632 | ======= |
| 633 | |
| 634 | Helm has built-in capabilities to enable the upgrade of pods without causing a |
| 635 | loss of the service being provided by that pod or pods (if configured as a |
| 636 | cluster). As described in the OOM Developer's Guide, ONAP components provide |
| 637 | an abstracted 'service' end point with the pods or containers providing this |
| 638 | service hidden from other ONAP components by a load balancer. This capability |
| 639 | is used during upgrades to allow a pod with a new image to be added to the |
| 640 | service before removing the pod with the old image. This 'make before break' |
| 641 | capability ensures minimal downtime. |
| 642 | |
| 643 | Prior to doing an upgrade, determine of the status of the deployed charts:: |
| 644 | |
| 645 | > helm list |
| 646 | NAME REVISION UPDATED STATUS CHART NAMESPACE |
Sylvain Desbureaux | c47ce73 | 2021-10-12 15:15:10 +0200 | [diff] [blame^] | 647 | so 1 Mon Feb 5 10:05:22 2020 DEPLOYED so-9.0.0 onap |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 648 | |
| 649 | When upgrading a cluster a parameter controls the minimum size of the cluster |
| 650 | during the upgrade while another parameter controls the maximum number of nodes |
| 651 | in the cluster. For example, SNDC configured as a 3-way ODL cluster might |
| 652 | require that during the upgrade no fewer than 2 pods are available at all times |
| 653 | to provide service while no more than 5 pods are ever deployed across the two |
| 654 | versions at any one time to avoid depleting the cluster of resources. In this |
| 655 | scenario, the SDNC cluster would start with 3 old pods then Kubernetes may add |
| 656 | a new pod (3 old, 1 new), delete one old (2 old, 1 new), add two new pods (2 |
| 657 | old, 3 new) and finally delete the 2 old pods (3 new). During this sequence |
| 658 | the constraints of the minimum of two pods and maximum of five would be |
| 659 | maintained while providing service the whole time. |
| 660 | |
| 661 | Initiation of an upgrade is triggered by changes in the Helm charts. For |
| 662 | example, if the image specified for one of the pods in the SDNC deployment |
| 663 | specification were to change (i.e. point to a new Docker image in the nexus3 |
| 664 | repository - commonly through the change of a deployment variable), the |
| 665 | sequence of events described in the previous paragraph would be initiated. |
| 666 | |
| 667 | For example, to upgrade a container by changing configuration, specifically an |
| 668 | environment value:: |
| 669 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 670 | > helm upgrade so onap/so --version 8.0.1 --set enableDebug=true |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 671 | |
| 672 | Issuing this command will result in the appropriate container being stopped by |
| 673 | Kubernetes and replaced with a new container with the new environment value. |
| 674 | |
| 675 | To upgrade a component to a new version with a new configuration file enter:: |
| 676 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 677 | > helm upgrade so onap/so --version 8.0.1 -f environments/demo.yaml |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 678 | |
| 679 | To fetch release history enter:: |
| 680 | |
| 681 | > helm history so |
| 682 | REVISION UPDATED STATUS CHART DESCRIPTION |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 683 | 1 Mon Feb 5 10:05:22 2020 SUPERSEDED so-8.0.0 Install complete |
Sylvain Desbureaux | c47ce73 | 2021-10-12 15:15:10 +0200 | [diff] [blame^] | 684 | 2 Mon Feb 5 10:10:55 2020 DEPLOYED so-9.0.0 Upgrade complete |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 685 | |
| 686 | Unfortunately, not all upgrades are successful. In recognition of this the |
| 687 | lineup of pods within an ONAP deployment is tagged such that an administrator |
| 688 | may force the ONAP deployment back to the previously tagged configuration or to |
| 689 | a specific configuration, say to jump back two steps if an incompatibility |
| 690 | between two ONAP components is discovered after the two individual upgrades |
| 691 | succeeded. |
| 692 | |
| 693 | This rollback functionality gives the administrator confidence that in the |
| 694 | unfortunate circumstance of a failed upgrade the system can be rapidly brought |
| 695 | back to a known good state. This process of rolling upgrades while under |
| 696 | service is illustrated in this short YouTube video showing a Zero Downtime |
| 697 | Upgrade of a web application while under a 10 million transaction per second |
| 698 | load. |
| 699 | |
| 700 | For example, to roll-back back to previous system revision enter:: |
| 701 | |
| 702 | > helm rollback so 1 |
| 703 | |
| 704 | > helm history so |
| 705 | REVISION UPDATED STATUS CHART DESCRIPTION |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 706 | 1 Mon Feb 5 10:05:22 2020 SUPERSEDED so-8.0.0 Install complete |
Sylvain Desbureaux | c47ce73 | 2021-10-12 15:15:10 +0200 | [diff] [blame^] | 707 | 2 Mon Feb 5 10:10:55 2020 SUPERSEDED so-9.0.0 Upgrade complete |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 708 | 3 Mon Feb 5 10:14:32 2020 DEPLOYED so-8.0.0 Rollback to 1 |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 709 | |
| 710 | .. note:: |
| 711 | |
| 712 | The description field can be overridden to document actions taken or include |
| 713 | tracking numbers. |
| 714 | |
| 715 | Many of the ONAP components contain their own databases which are used to |
| 716 | record configuration or state information. The schemas of these databases may |
| 717 | change from version to version in such a way that data stored within the |
| 718 | database needs to be migrated between versions. If such a migration script is |
| 719 | available it can be invoked during the upgrade (or rollback) by Container |
| 720 | Lifecycle Hooks. Two such hooks are available, PostStart and PreStop, which |
| 721 | containers can access by registering a handler against one or both. Note that |
| 722 | it is the responsibility of the ONAP component owners to implement the hook |
| 723 | handlers - which could be a shell script or a call to a specific container HTTP |
| 724 | endpoint - following the guidelines listed on the Kubernetes site. Lifecycle |
| 725 | hooks are not restricted to database migration or even upgrades but can be used |
| 726 | anywhere specific operations need to be taken during lifecycle operations. |
| 727 | |
| 728 | OOM uses Helm K8S package manager to deploy ONAP components. Each component is |
| 729 | arranged in a packaging format called a chart - a collection of files that |
| 730 | describe a set of k8s resources. Helm allows for rolling upgrades of the ONAP |
| 731 | component deployed. To upgrade a component Helm release you will need an |
| 732 | updated Helm chart. The chart might have modified, deleted or added values, |
| 733 | deployment yamls, and more. To get the release name use:: |
| 734 | |
| 735 | > helm ls |
| 736 | |
| 737 | To easily upgrade the release use:: |
| 738 | |
| 739 | > helm upgrade [RELEASE] [CHART] |
| 740 | |
| 741 | To roll back to a previous release version use:: |
| 742 | |
| 743 | > helm rollback [flags] [RELEASE] [REVISION] |
| 744 | |
| 745 | For example, to upgrade the onap-so helm release to the latest SO container |
| 746 | release v1.1.2: |
| 747 | |
| 748 | - Edit so values.yaml which is part of the chart |
| 749 | - Change "so: nexus3.onap.org:10001/openecomp/so:v1.1.1" to |
| 750 | "so: nexus3.onap.org:10001/openecomp/so:v1.1.2" |
| 751 | - From the chart location run:: |
| 752 | |
| 753 | > helm upgrade onap-so |
| 754 | |
| 755 | The previous so pod will be terminated and a new so pod with an updated so |
| 756 | container will be created. |
| 757 | |
| 758 | .. figure:: oomLogoV2-Delete.png |
| 759 | :align: right |
| 760 | |
| 761 | Delete |
| 762 | ====== |
| 763 | |
| 764 | Existing deployments can be partially or fully removed once they are no longer |
| 765 | needed. To minimize errors it is recommended that before deleting components |
| 766 | from a running deployment the operator perform a 'dry-run' to display exactly |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 767 | what will happen with a given command prior to actually deleting anything. |
| 768 | For example:: |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 769 | |
Sylvain Desbureaux | 5e19e24 | 2020-03-02 14:41:48 +0100 | [diff] [blame] | 770 | > helm undeploy onap --dry-run |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 771 | |
Sylvain Desbureaux | 5e19e24 | 2020-03-02 14:41:48 +0100 | [diff] [blame] | 772 | will display the outcome of deleting the 'onap' release from the |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 773 | deployment. |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 774 | To completely delete a release and remove it from the internal store enter:: |
| 775 | |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 776 | > helm undeploy onap |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 777 | |
ramagp | 060a51c | 2020-10-29 04:24:16 +0000 | [diff] [blame] | 778 | Once complete undeploy is done then delete the namespace as well |
| 779 | using following command:: |
| 780 | |
| 781 | > kubectl delete namespace <name of namespace> |
| 782 | |
| 783 | .. note:: |
| 784 | You need to provide the namespace name which you used during deployment, |
| 785 | below is the example:: |
| 786 | |
| 787 | > kubectl delete namespace onap |
| 788 | |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 789 | One can also remove individual components from a deployment by changing the |
| 790 | ONAP configuration values. For example, to remove `so` from a running |
| 791 | deployment enter:: |
| 792 | |
Sylvain Desbureaux | 557628a | 2021-03-24 14:59:16 +0100 | [diff] [blame] | 793 | > helm undeploy onap-so |
Roger Maitland | 953b5f1 | 2018-03-22 15:24:04 -0400 | [diff] [blame] | 794 | |
| 795 | will remove `so` as the configuration indicates it's no longer part of the |
| 796 | deployment. This might be useful if a one wanted to replace just `so` by |
Sylvain Desbureaux | 983c755 | 2019-01-28 13:59:43 +0100 | [diff] [blame] | 797 | installing a custom version. |