Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 1 | .. This work is licensed under a Creative Commons Attribution 4.0 |
| 2 | .. International License. |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 3 | .. http://creativecommons.org/licenses/by/4.0 |
| 4 | .. Copyright 2020, Samsung Electronics |
| 5 | |
| 6 | .. Links |
| 7 | .. _HELM Best Practices Guide: https://docs.helm.sh/chart_best_practices/#requirements |
| 8 | .. _kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/ |
| 9 | .. _Kubernetes documentation for emptyDir: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir |
| 10 | .. _metallb Metal Load Balancer installation: https://metallb.universe.tf/installation/ |
| 11 | .. _http://cd.onap.info:30223/mso/logging/debug: http://cd.onap.info:30223/mso/logging/debug |
| 12 | .. _Onboarding and Distributing a Vendor Software Product: https://wiki.onap.org/pages/viewpage.action?pageId=1018474 |
| 13 | .. _README.md: https://gerrit.onap.org/r/gitweb?p=oom.git;a=blob;f=kubernetes/README.md |
| 14 | |
| 15 | .. figure:: oomLogoV2-medium.png |
| 16 | :align: right |
| 17 | |
Sylvain Desbureaux | ec07bca | 2020-12-09 10:13:42 +0100 | [diff] [blame^] | 18 | .. _oom_setup_ingress_controller: |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 19 | |
| 20 | |
| 21 | Ingress controller setup on HA Kubernetes Cluster |
| 22 | ################################################# |
| 23 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 24 | This guide provides instruction how to setup experimental ingress controller |
| 25 | feature. For this, we are hosting our cluster on OpenStack VMs and using the |
| 26 | Rancher Kubernetes Engine (RKE) to deploy and manage our Kubernetes Cluster and |
| 27 | ingress controller |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 28 | |
| 29 | .. contents:: |
| 30 | :depth: 1 |
| 31 | :local: |
| 32 | .. |
| 33 | |
| 34 | The result at the end of this tutorial will be: |
| 35 | |
| 36 | #. Customization of the cluster.yaml file for ingress controller support |
| 37 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 38 | #. Installation and configuration test DNS server for ingress host resolution |
| 39 | on testing machines |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 40 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 41 | #. Installation and configuration MLB (Metal Load Balancer) required for |
| 42 | exposing ingress service |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 43 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 44 | #. Installation and configuration NGINX ingress controller |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 45 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 46 | #. Additional info how to deploy ONAP with services exposed via Ingress |
| 47 | controller |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 48 | |
| 49 | Customize cluster.yml file |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 50 | ========================== |
| 51 | Before setup cluster for ingress purposes DNS cluster IP and ingress provider |
| 52 | should be configured and following: |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 53 | |
| 54 | .. code-block:: yaml |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 55 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 56 | --- |
| 57 | <...> |
| 58 | restore: |
| 59 | restore: false |
| 60 | snapshot_name: "" |
| 61 | ingress: |
| 62 | provider: none |
| 63 | dns: |
| 64 | provider: coredns |
| 65 | upstreamnameservers: |
| 66 | - <custer_dns_ip>:31555 |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 67 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 68 | Where the <cluster_dns_ip> should be set to the same IP as the CONTROLPANE |
| 69 | node. |
| 70 | |
| 71 | For external load balancer purposes, minimum one of the worker node should be |
| 72 | configured with external IP address accessible outside the cluster. It can be |
| 73 | done using the following example node configuration: |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 74 | |
| 75 | .. code-block:: yaml |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 76 | |
| 77 | --- |
| 78 | <...> |
| 79 | - address: <external_ip> |
| 80 | internal_address: <internal_ip> |
| 81 | port: "22" |
| 82 | role: |
| 83 | - worker |
| 84 | hostname_override: "onap-worker-0" |
| 85 | user: ubuntu |
| 86 | ssh_key_path: "~/.ssh/id_rsa" |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 87 | <...> |
| 88 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 89 | Where the <external_ip> is external worker node IP address, and <internal_ip> |
| 90 | is internal node IP address if it is required. |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 91 | |
| 92 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 93 | DNS server configuration and installation |
| 94 | ========================================= |
| 95 | DNS server deployed on the Kubernetes cluster makes it easy to use services |
| 96 | exposed through ingress controller because it resolves all subdomain related to |
| 97 | the ONAP cluster to the load balancer IP. Testing ONAP cluster requires a lot |
| 98 | of entries on the target machines in the /etc/hosts. Adding many entries into |
| 99 | the configuration files on testing machines is quite problematic and error |
| 100 | prone. The better wait is to create central DNS server with entries for all |
| 101 | virtual host pointed to simpledemo.onap.org and add custom DNS server as a |
| 102 | target DNS server for testing machines and/or as external DNS for Kubernetes |
| 103 | cluster. |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 104 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 105 | DNS server has automatic installation and configuration script, so installation |
| 106 | is quite easy:: |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 107 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 108 | > cd kubernetes/contrib/dns-server-for-vhost-ingress-testing |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 109 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 110 | > ./deploy\_dns.sh |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 111 | |
| 112 | After DNS deploy you need to setup DNS entry on the target testing machine. |
| 113 | Because DNS listen on non standard port configuration require iptables rules |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 114 | on the target machine. Please follow the configuration proposed by the deploy |
| 115 | scripts. |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 116 | Example output depends on the IP address and example output looks like bellow:: |
| 117 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 118 | DNS server already deployed: |
| 119 | 1. You can add the DNS server to the target machine using following commands: |
| 120 | sudo iptables -t nat -A OUTPUT -p tcp -d 192.168.211.211 --dport 53 -j DNAT --to-destination 10.10.13.14:31555 |
| 121 | sudo iptables -t nat -A OUTPUT -p udp -d 192.168.211.211 --dport 53 -j DNAT --to-destination 10.10.13.14:31555 |
| 122 | sudo sysctl -w net.ipv4.conf.all.route_localnet=1 |
| 123 | sudo sysctl -w net.ipv4.ip_forward=1 |
| 124 | 2. Update /etc/resolv.conf file with nameserver 192.168.211.211 entry on your target machine |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 125 | |
| 126 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 127 | MetalLB Load Balancer installation and configuration |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 128 | ==================================================== |
| 129 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 130 | By default pure Kubernetes cluster requires external load balancer if we want |
| 131 | to expose external port using LoadBalancer settings. For this purpose MetalLB |
| 132 | can be used. Before installing the MetalLB you need to ensure that at least one |
| 133 | worker has assigned IP accessible outside the cluster. |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 134 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 135 | MetalLB Load balancer can be easily installed using automatic install script:: |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 136 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 137 | > cd kubernetes/contrib/metallb-loadbalancer-inst |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 138 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 139 | > ./install-metallb-on-cluster.sh |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 140 | |
| 141 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 142 | Configuration Ngninx ingress controller |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 143 | ======================================= |
| 144 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 145 | After installation DNS server and ingress controller we can install and |
| 146 | configure ingress controller. |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 147 | It can be done using the following commands:: |
| 148 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 149 | > cd kubernetes/contrib/ingress-nginx-post-inst |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 150 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 151 | > kubectl apply -f nginx_ingress_cluster_config.yaml |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 152 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 153 | > kubectl apply -f nginx_ingress_enable_optional_load_balacer_service.yaml |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 154 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 155 | After deploy NGINX ingress controller you can ensure that the ingress port is |
| 156 | exposed as load balancer service with external IP address:: |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 157 | |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 158 | > kubectl get svc -n ingress-nginx |
| 159 | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE |
| 160 | default-http-backend ClusterIP 10.10.10.10 <none> 80/TCP 25h |
| 161 | ingress-nginx LoadBalancer 10.10.10.11 10.12.13.14 80:31308/TCP,443:30314/TCP 24h |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 162 | |
| 163 | |
| 164 | ONAP with ingress exposed services |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 165 | ================================== |
| 166 | If you want to deploy onap with services exposed through ingress controller you |
| 167 | can use full onap deploy script:: |
| 168 | |
| 169 | > onap/resources/overrides/onap-all-ingress-nginx-vhost.yaml |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 170 | |
| 171 | Ingress also can be enabled on any onap setup override using following code: |
| 172 | |
| 173 | .. code-block:: yaml |
Eric Debeau | 993b77b | 2020-08-19 15:30:00 +0200 | [diff] [blame] | 174 | |
| 175 | --- |
| 176 | <...> |
| 177 | #ingress virtualhost based configuration |
| 178 | global: |
| 179 | <...> |
| 180 | ingress: |
| 181 | enabled: true |
Lucjan Bryndza | 6f23615 | 2020-05-06 14:08:52 +0000 | [diff] [blame] | 182 | |