blob: 29dac92c95eab86f3a9745a8ace69e0ef6f812b8 [file] [log] [blame]
Eric Debeau993b77b2020-08-19 15:30:00 +02001.. This work is licensed under a Creative Commons Attribution 4.0
2.. International License.
Lucjan Bryndza6f236152020-05-06 14:08:52 +00003.. http://creativecommons.org/licenses/by/4.0
4.. Copyright 2020, Samsung Electronics
5
6.. Links
7.. _HELM Best Practices Guide: https://docs.helm.sh/chart_best_practices/#requirements
8.. _kubectl Cheat Sheet: https://kubernetes.io/docs/reference/kubectl/cheatsheet/
9.. _Kubernetes documentation for emptyDir: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
10.. _metallb Metal Load Balancer installation: https://metallb.universe.tf/installation/
11.. _http://cd.onap.info:30223/mso/logging/debug: http://cd.onap.info:30223/mso/logging/debug
12.. _Onboarding and Distributing a Vendor Software Product: https://wiki.onap.org/pages/viewpage.action?pageId=1018474
13.. _README.md: https://gerrit.onap.org/r/gitweb?p=oom.git;a=blob;f=kubernetes/README.md
14
15.. figure:: oomLogoV2-medium.png
16 :align: right
17
Sylvain Desbureauxec07bca2020-12-09 10:13:42 +010018.. _oom_setup_ingress_controller:
Lucjan Bryndza6f236152020-05-06 14:08:52 +000019
20
21Ingress controller setup on HA Kubernetes Cluster
22#################################################
23
Eric Debeau993b77b2020-08-19 15:30:00 +020024This guide provides instruction how to setup experimental ingress controller
25feature. For this, we are hosting our cluster on OpenStack VMs and using the
26Rancher Kubernetes Engine (RKE) to deploy and manage our Kubernetes Cluster and
27ingress controller
Lucjan Bryndza6f236152020-05-06 14:08:52 +000028
29.. contents::
30 :depth: 1
31 :local:
32..
33
34The result at the end of this tutorial will be:
35
36#. Customization of the cluster.yaml file for ingress controller support
37
Eric Debeau993b77b2020-08-19 15:30:00 +020038#. Installation and configuration test DNS server for ingress host resolution
39 on testing machines
Lucjan Bryndza6f236152020-05-06 14:08:52 +000040
Eric Debeau993b77b2020-08-19 15:30:00 +020041#. Installation and configuration MLB (Metal Load Balancer) required for
42 exposing ingress service
Lucjan Bryndza6f236152020-05-06 14:08:52 +000043
Eric Debeau993b77b2020-08-19 15:30:00 +020044#. Installation and configuration NGINX ingress controller
Lucjan Bryndza6f236152020-05-06 14:08:52 +000045
Eric Debeau993b77b2020-08-19 15:30:00 +020046#. Additional info how to deploy ONAP with services exposed via Ingress
47 controller
Lucjan Bryndza6f236152020-05-06 14:08:52 +000048
49Customize cluster.yml file
Eric Debeau993b77b2020-08-19 15:30:00 +020050==========================
51Before setup cluster for ingress purposes DNS cluster IP and ingress provider
52should be configured and following:
Lucjan Bryndza6f236152020-05-06 14:08:52 +000053
54.. code-block:: yaml
Lucjan Bryndza6f236152020-05-06 14:08:52 +000055
Eric Debeau993b77b2020-08-19 15:30:00 +020056 ---
57 <...>
58 restore:
59 restore: false
60 snapshot_name: ""
61 ingress:
62 provider: none
63 dns:
64 provider: coredns
65 upstreamnameservers:
66 - <custer_dns_ip>:31555
Lucjan Bryndza6f236152020-05-06 14:08:52 +000067
Eric Debeau993b77b2020-08-19 15:30:00 +020068Where the <cluster_dns_ip> should be set to the same IP as the CONTROLPANE
69node.
70
71For external load balancer purposes, minimum one of the worker node should be
72configured with external IP address accessible outside the cluster. It can be
73done using the following example node configuration:
Lucjan Bryndza6f236152020-05-06 14:08:52 +000074
75.. code-block:: yaml
Eric Debeau993b77b2020-08-19 15:30:00 +020076
77 ---
78 <...>
79 - address: <external_ip>
80 internal_address: <internal_ip>
81 port: "22"
82 role:
83 - worker
84 hostname_override: "onap-worker-0"
85 user: ubuntu
86 ssh_key_path: "~/.ssh/id_rsa"
Lucjan Bryndza6f236152020-05-06 14:08:52 +000087 <...>
88
Eric Debeau993b77b2020-08-19 15:30:00 +020089Where the <external_ip> is external worker node IP address, and <internal_ip>
90is internal node IP address if it is required.
Lucjan Bryndza6f236152020-05-06 14:08:52 +000091
92
Eric Debeau993b77b2020-08-19 15:30:00 +020093DNS server configuration and installation
94=========================================
95DNS server deployed on the Kubernetes cluster makes it easy to use services
96exposed through ingress controller because it resolves all subdomain related to
97the ONAP cluster to the load balancer IP. Testing ONAP cluster requires a lot
98of entries on the target machines in the /etc/hosts. Adding many entries into
99the configuration files on testing machines is quite problematic and error
100prone. The better wait is to create central DNS server with entries for all
101virtual host pointed to simpledemo.onap.org and add custom DNS server as a
102target DNS server for testing machines and/or as external DNS for Kubernetes
103cluster.
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000104
Eric Debeau993b77b2020-08-19 15:30:00 +0200105DNS server has automatic installation and configuration script, so installation
106is quite easy::
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000107
Eric Debeau993b77b2020-08-19 15:30:00 +0200108 > cd kubernetes/contrib/dns-server-for-vhost-ingress-testing
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000109
Eric Debeau993b77b2020-08-19 15:30:00 +0200110 > ./deploy\_dns.sh
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000111
112After DNS deploy you need to setup DNS entry on the target testing machine.
113Because DNS listen on non standard port configuration require iptables rules
Eric Debeau993b77b2020-08-19 15:30:00 +0200114on the target machine. Please follow the configuration proposed by the deploy
115scripts.
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000116Example output depends on the IP address and example output looks like bellow::
117
Eric Debeau993b77b2020-08-19 15:30:00 +0200118 DNS server already deployed:
119 1. You can add the DNS server to the target machine using following commands:
120 sudo iptables -t nat -A OUTPUT -p tcp -d 192.168.211.211 --dport 53 -j DNAT --to-destination 10.10.13.14:31555
121 sudo iptables -t nat -A OUTPUT -p udp -d 192.168.211.211 --dport 53 -j DNAT --to-destination 10.10.13.14:31555
122 sudo sysctl -w net.ipv4.conf.all.route_localnet=1
123 sudo sysctl -w net.ipv4.ip_forward=1
124 2. Update /etc/resolv.conf file with nameserver 192.168.211.211 entry on your target machine
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000125
126
Eric Debeau993b77b2020-08-19 15:30:00 +0200127MetalLB Load Balancer installation and configuration
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000128====================================================
129
Eric Debeau993b77b2020-08-19 15:30:00 +0200130By default pure Kubernetes cluster requires external load balancer if we want
131to expose external port using LoadBalancer settings. For this purpose MetalLB
132can be used. Before installing the MetalLB you need to ensure that at least one
133worker has assigned IP accessible outside the cluster.
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000134
Eric Debeau993b77b2020-08-19 15:30:00 +0200135MetalLB Load balancer can be easily installed using automatic install script::
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000136
Eric Debeau993b77b2020-08-19 15:30:00 +0200137 > cd kubernetes/contrib/metallb-loadbalancer-inst
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000138
Eric Debeau993b77b2020-08-19 15:30:00 +0200139 > ./install-metallb-on-cluster.sh
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000140
141
Eric Debeau993b77b2020-08-19 15:30:00 +0200142Configuration Ngninx ingress controller
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000143=======================================
144
Eric Debeau993b77b2020-08-19 15:30:00 +0200145After installation DNS server and ingress controller we can install and
146configure ingress controller.
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000147It can be done using the following commands::
148
Eric Debeau993b77b2020-08-19 15:30:00 +0200149 > cd kubernetes/contrib/ingress-nginx-post-inst
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000150
Eric Debeau993b77b2020-08-19 15:30:00 +0200151 > kubectl apply -f nginx_ingress_cluster_config.yaml
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000152
Eric Debeau993b77b2020-08-19 15:30:00 +0200153 > kubectl apply -f nginx_ingress_enable_optional_load_balacer_service.yaml
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000154
Eric Debeau993b77b2020-08-19 15:30:00 +0200155After deploy NGINX ingress controller you can ensure that the ingress port is
156exposed as load balancer service with external IP address::
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000157
Eric Debeau993b77b2020-08-19 15:30:00 +0200158 > kubectl get svc -n ingress-nginx
159 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
160 default-http-backend ClusterIP 10.10.10.10 <none> 80/TCP 25h
161 ingress-nginx LoadBalancer 10.10.10.11 10.12.13.14 80:31308/TCP,443:30314/TCP 24h
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000162
163
164ONAP with ingress exposed services
Eric Debeau993b77b2020-08-19 15:30:00 +0200165==================================
166If you want to deploy onap with services exposed through ingress controller you
167can use full onap deploy script::
168
169 > onap/resources/overrides/onap-all-ingress-nginx-vhost.yaml
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000170
171Ingress also can be enabled on any onap setup override using following code:
172
173.. code-block:: yaml
Eric Debeau993b77b2020-08-19 15:30:00 +0200174
175 ---
176 <...>
177 #ingress virtualhost based configuration
178 global:
179 <...>
180 ingress:
181 enabled: true
Lucjan Bryndza6f236152020-05-06 14:08:52 +0000182