blob: 1db411c6b9ef58a0495cb66b2df8b735a3b45735 [file] [log] [blame]
RossCab1335d2020-05-05 12:12:54 +01001.. This work is licensed under a
2.. Creative Commons Attribution 4.0 International License.
3.. http://creativecommons.org/licenses/by/4.0
4
5.. _distribution-s3p-label:
6
7Policy Distribution component
8#############################
9
gregory.hayes6765cb42020-10-21 16:32:59 +00001072h Stability and 4h Performance Tests of Distribution
11++++++++++++++++++++++++++++++++++++++++++++++++++++++
RossCab1335d2020-05-05 12:12:54 +010012
gregory.hayes6765cb42020-10-21 16:32:59 +000013VM Details
14----------
RossCab1335d2020-05-05 12:12:54 +010015
gregory.hayes6765cb42020-10-21 16:32:59 +000016The stability and performance tests are performed on VM's running in the OpenStack cloud environment in the ONAP integration lab. There are two separate VMs, one for running backend policy services which policy distribution needs, and the other for the policy distribution service itself and Jmeter.
RossCab1335d2020-05-05 12:12:54 +010017
18**OpenStack environment details**
19
gregory.hayes6765cb42020-10-21 16:32:59 +000020- Version: Windriver Titanium
RossCab1335d2020-05-05 12:12:54 +010021
gregory.hayes6765cb42020-10-21 16:32:59 +000022**Policy Backend VM details (VM1)**
RossCab1335d2020-05-05 12:12:54 +010023
gregory.hayes6765cb42020-10-21 16:32:59 +000024- OS: Ubuntu 18.04.5 LTS
25- CPU: 8 core, Intel Xeon E3-12xx v2 (Ivy Bridge), 2693.668 MHz, 16384 kB cache
26- RAM: 32 GB
27- HardDisk: 200 GB
28- Docker version 19.03.8, build afacb8b7f0
29- Java: openjdk 11.0.8 2020-07-14
RossCab1335d2020-05-05 12:12:54 +010030
gregory.hayes6765cb42020-10-21 16:32:59 +000031**JMeter and Distribution VM details (VM2)**
RossCab1335d2020-05-05 12:12:54 +010032
gregory.hayes6765cb42020-10-21 16:32:59 +000033- OS: Ubuntu 18.04.5 LTS
34- CPU: 8 core, Intel Xeon E3-12xx v2 (Ivy Bridge), 2693.668 MHz, 16384 kB cache
35- RAM: 32 GB
36- HardDisk: 200 GB
37- Docker version 19.03.8, build afacb8b7f0
38- Java: openjdk 11.0.8 2020-07-14
39- JMeter: 5.1.1
RossCab1335d2020-05-05 12:12:54 +010040
RossCab1335d2020-05-05 12:12:54 +010041
gregory.hayes6765cb42020-10-21 16:32:59 +000042VM1 & VM2: Common Setup
43-----------------------
44Make sure to execute below commands on both VM1 & VM2
RossCab1335d2020-05-05 12:12:54 +010045
46Update the ubuntu software installer
47
48.. code-block:: bash
49
gregory.hayes6765cb42020-10-21 16:32:59 +000050 sudo apt update
RossCab1335d2020-05-05 12:12:54 +010051
gregory.hayes6765cb42020-10-21 16:32:59 +000052Install Java
RossCab1335d2020-05-05 12:12:54 +010053
54.. code-block:: bash
55
gregory.hayes6765cb42020-10-21 16:32:59 +000056 sudo apt install -y openjdk-11-jdk
RossCab1335d2020-05-05 12:12:54 +010057
58Ensure that the Java version that is executing is OpenJDK version 11
59
gregory.hayes6765cb42020-10-21 16:32:59 +000060.. code-block:: bash
61
62 $ java --version
63 openjdk 11.0.8 2020-07-14
64 OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1)
65 OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1, mixed mode, sharing)
66
67Install Docker
RossCab1335d2020-05-05 12:12:54 +010068
69.. code-block:: bash
70
gregory.hayes6765cb42020-10-21 16:32:59 +000071 # Add docker repository
72 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
73 sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
74 sudo apt update
75
76 # Check available docker versions (if necessary)
77 apt-cache policy docker-ce
78
79 # Install docker
80 sudo apt install -y docker-ce=5:19.03.8~3-0~ubuntu-bionic docker-ce-cli=5:19.03.8~3-0~ubuntu-bionic containerd.io
RossCab1335d2020-05-05 12:12:54 +010081
82Change the permissions of the Docker socket file
83
84.. code-block:: bash
85
gregory.hayes6765cb42020-10-21 16:32:59 +000086 sudo chmod 666 /var/run/docker.sock
RossCab1335d2020-05-05 12:12:54 +010087
88Check the status of the Docker service and ensure it is running correctly
89
90.. code-block:: bash
91
gregory.hayes6765cb42020-10-21 16:32:59 +000092 $ systemctl status --no-pager docker
93 docker.service - Docker Application Container Engine
94 Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
95 Active: active (running) since Wed 2020-10-14 13:59:40 UTC; 1 weeks 0 days ago
96 # ... (truncated for brevity)
97
RossCab1335d2020-05-05 12:12:54 +010098 $ docker ps
gregory.hayes6765cb42020-10-21 16:32:59 +000099 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
RossCab1335d2020-05-05 12:12:54 +0100100
gregory.hayes6765cb42020-10-21 16:32:59 +0000101Clone the policy-distribution repo to access the test scripts
RossCab1335d2020-05-05 12:12:54 +0100102
gregory.hayes6765cb42020-10-21 16:32:59 +0000103.. code-block:: bash
RossCab1335d2020-05-05 12:12:54 +0100104
gregory.hayes6765cb42020-10-21 16:32:59 +0000105 git clone https://gerrit.onap.org/r/policy/distribution
106
107
108VM1 Only: Install Simulators, Policy-PAP, Policy-API and MariaDB
109----------------------------------------------------------------
110
111Modify the setup_components.sh script located at:
112
113- ~/distribution/testsuites/stability/src/main/resources/simulatorsetup/setup_components.sh
114
115Ensure the correct docker image versions are specified - e.g. for Guilin-RC0
116
117- nexus3.onap.org:10001/onap/policy-api:2.3.2
118- nexus3.onap.org:10001/onap/policy-pap:2.3.2
119
120Run the setup_components.sh script to start the test support components:
121
122.. code-block:: bash
123
124 ~/distribution/testsuites/stability/src/main/resources/distributionsetup/setup_distribution.sh
RossCab1335d2020-05-05 12:12:54 +0100125
126After installation, ensure the following docker containers are up and running:
127
128.. code-block:: bash
129
gregory.hayes6765cb42020-10-21 16:32:59 +0000130 $ docker ps
131 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
132 a187cb0ff08a nexus3.onap.org:10001/onap/policy-pap:2.3.2 "bash ./policy-pap.sh" 4 days ago Up 4 days 0.0.0.0:7000->6969/tcp policy-pap
133 2f7632fe90c3 nexus3.onap.org:10001/onap/policy-api:2.3.2 "bash ./policy-api.sh" 4 days ago Up 4 days 0.0.0.0:6969->6969/tcp policy-api
134 70fa27d6d992 pdp/simulator:latest "bash pdp-sim.sh" 4 days ago Up 4 days pdp-simulator
135 3c9ff28ba050 dmaap/simulator:latest "bash dmaap-sim.sh" 4 days ago Up 4 days 0.0.0.0:3904->3904/tcp message-router
136 60cfcf8cfe65 mariadb:10.2.14 "docker-entrypoint.s…" 4 days ago Up 4 days 0.0.0.0:3306->3306/tcp mariadb
RossCab1335d2020-05-05 12:12:54 +0100137
RossCab1335d2020-05-05 12:12:54 +0100138
gregory.hayes6765cb42020-10-21 16:32:59 +0000139VM2 Only: Install Distribution
140------------------------------
RossCab1335d2020-05-05 12:12:54 +0100141
gregory.hayes6765cb42020-10-21 16:32:59 +0000142Modify the setup_distribution.sh script located at:
RossCab1335d2020-05-05 12:12:54 +0100143
gregory.hayes6765cb42020-10-21 16:32:59 +0000144- ~/distribution/testsuites/stability/src/main/resources/distributionsetup/setup_distribution.sh
145
146Ensure the correct docker image version is specified - e.g. for Guilin-RC0:
147
148- nexus3.onap.org:10001/onap/policy-distribution:2.4.2
149
150Run the setup_distribution.sh script to install the distribution service, provide the IP of VM1 (twice) as the arguments to the script:
151
RossCab1335d2020-05-05 12:12:54 +0100152.. code-block:: bash
153
gregory.hayes6765cb42020-10-21 16:32:59 +0000154 ~/distribution/testsuites/stability/src/main/resources/distributionsetup/setup_distribution.sh <vm1-ipaddr> <vm1-ipaddr>
RossCab1335d2020-05-05 12:12:54 +0100155
156Ensure the distribution container is running.
157
gregory.hayes6765cb42020-10-21 16:32:59 +0000158.. code-block:: bash
RossCab1335d2020-05-05 12:12:54 +0100159
gregory.hayes6765cb42020-10-21 16:32:59 +0000160 $ docker ps
161 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
162 9a8db2bad156 nexus3.onap.org:10001/onap/policy-distribution:2.4.2 "bash ./policy-dist.…" 29 hours ago Up 29 hours 0.0.0.0:6969->6969/tcp, 0.0.0.0:9090->9090/tcp policy-distribution
163
164
165VM2 Only: Install JMeter
166------------------------
167
168Download and install JMeter
RossCab1335d2020-05-05 12:12:54 +0100169
170.. code-block:: bash
171
gregory.hayes6765cb42020-10-21 16:32:59 +0000172 # Install required packages
173 sudo apt install -y wget unzip
RossCab1335d2020-05-05 12:12:54 +0100174
gregory.hayes6765cb42020-10-21 16:32:59 +0000175 # Install JMeter
176 mkdir -p jmeter
177 wget https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-5.1.1.zip
178 unzip -qd jmeter apache-jmeter-5.1.1.zip
179 rm apache-jmeter-5.1.1.zip
180
181
182VM2 Only: Install & configure visualVM
183--------------------------------------
184
185VisualVM needs to be installed in the virtual machine running Distrbution (VM2). It will be used to monitor CPU, Memory and GC for Distribution while the stability tests are running.
RossCab1335d2020-05-05 12:12:54 +0100186
187.. code-block:: bash
188
gregory.hayes6765cb42020-10-21 16:32:59 +0000189 sudo apt install -y visualvm
RossCab1335d2020-05-05 12:12:54 +0100190
191Run these commands to configure permissions
192
193.. code-block:: bash
194
gregory.hayes6765cb42020-10-21 16:32:59 +0000195 # Create Java security policy file for VisualVM
196 sudo cat > /usr/lib/jvm/java-11-openjdk-amd64/bin/visualvm.policy << EOF
197 grant codebase "jrt:/jdk.jstatd" {
198 permission java.security.AllPermission;
RossCab1335d2020-05-05 12:12:54 +0100199 };
gregory.hayes6765cb42020-10-21 16:32:59 +0000200 grant codebase "jrt:/jdk.internal.jvmstat" {
201 permission java.security.AllPermission;
202 };
203 EOF
RossCab1335d2020-05-05 12:12:54 +0100204
gregory.hayes6765cb42020-10-21 16:32:59 +0000205 # Set globally accessable permissions on policy file
206 sudo chmod 777 /usr/lib/jvm/java-11-openjdk-amd64/bin/visualvm.policy
207
208Run the following command to start jstatd using port 1111
RossCab1335d2020-05-05 12:12:54 +0100209
210.. code-block:: bash
211
gregory.hayes6765cb42020-10-21 16:32:59 +0000212 /usr/lib/jvm/java-11-openjdk-amd64/bin/jstatd -p 1111 -J-Djava.security.policy=/usr/lib/jvm/java-11-openjdk-amd64/bin/visualvm.policy &
RossCab1335d2020-05-05 12:12:54 +0100213
gregory.hayes6765cb42020-10-21 16:32:59 +0000214Run visualVM to connect to localhost:9090
RossCab1335d2020-05-05 12:12:54 +0100215
216.. code-block:: bash
217
gregory.hayes6765cb42020-10-21 16:32:59 +0000218 visualvm &
RossCab1335d2020-05-05 12:12:54 +0100219
220This will load up the visualVM GUI
221
222Connect to Distribution JMX Port.
223
224 1. Right click on "Local" in the left panel of the screen and select "Add JMX Connection"
225 2. Enter the Port 9090. this is the JMX port exposed by the dsitribution container
226 3. Double click on the newly added nodes under "Local" to start monitoring CPU, Memory & GC.
227
gregory.hayes6765cb42020-10-21 16:32:59 +0000228Example Screenshot of visualVM
RossCab1335d2020-05-05 12:12:54 +0100229
230.. image:: images/distribution-s3p-vvm-sample.png
231
RossCab1335d2020-05-05 12:12:54 +0100232
gregory.hayes6765cb42020-10-21 16:32:59 +0000233Stability Test of Policy Distribution
234+++++++++++++++++++++++++++++++++++++
235
236Introduction
237------------
238
239The 72 hour Stability Test for policy distribution has the goal of introducing a steady flow of transactions initiated from a test client server running JMeter. The policy distribution is configured with a special FileSystemReception plugin to monitor a local directory for newly added csar files to be processed by itself. The input CSAR will be added/removed by the test client(JMeter) and the result will be pulled from the backend(PAP and PolicyAPI) by the test client (JMeter).
240
241The test will be performed in an environment where Jmeter will continuously add/remove a test csar into the special directory where policy distribuion is monitoring and will then get the processed results from PAP and PolicyAPI to verify the successful deployment of the policy. The policy will then be undeployed and the test will loop continuously until 72 hours have elapsed.
242
243
244Test Plan Sequence
245------------------
246
247The 72h stability test will run the following steps sequentially in a single threaded loop.
RossCab1335d2020-05-05 12:12:54 +0100248
249- **Delete Old CSAR** - Checks if CSAR already exists in the watched directory, if so it deletes it
250- **Add CSAR** - Adds CSAR to the directory that distribution is watching
251- **Get Healthcheck** - Ensures Healthcheck is returning 200 OK
252- **Get Statistics** - Ensures Statistics is returning 200 OK
253- **CheckPDPGroupQuery** - Checks that PDPGroupQuery contains the deployed policy
254- **CheckPolicyDeployed** - Checks that the policy is deployed
255- **Undeploy Policy** - Undeploys the policy
256- **Delete Policy** - Deletes the Policy for the next loop
257- **Check PDP Group for Deletion** - Ensures the policy has been removed and does not exist
258
259The following steps can be used to configure the parameters of the test plan.
260
261- **HTTP Authorization Manager** - used to store user/password authentication details.
262- **HTTP Header Manager** - used to store headers which will be used for making HTTP requests.
263- **User Defined Variables** - used to store following user defined parameters.
264
265========== ===============================================
266 **Name** **Description**
267========== ===============================================
268 PAP_HOST IP Address or host name of PAP component
269 PAP_PORT Port number of PAP for making REST API calls
270 API_HOST IP Address or host name of API component
271 API_PORT Port number of API for making REST API calls
272 DURATION Duration of Test
273========== ===============================================
274
275Screenshot of Distribution stability test plan
276
277.. image:: images/distribution-s3p-testplan.png
278
gregory.hayes6765cb42020-10-21 16:32:59 +0000279
RossCab1335d2020-05-05 12:12:54 +0100280Running the Test Plan
281---------------------
282
gregory.hayes6765cb42020-10-21 16:32:59 +0000283Edit the /tmp folder permissions to allow the testplan to insert the CSAR into the /tmp/policydistribution/distributionmount folder
RossCab1335d2020-05-05 12:12:54 +0100284
285.. code-block:: bash
286
gregory.hayes6765cb42020-10-21 16:32:59 +0000287 sudo mkdir -p /tmp/policydistribution/distributionmount
288 sudo chmod -R a+trwx /tmp
RossCab1335d2020-05-05 12:12:54 +0100289
gregory.hayes6765cb42020-10-21 16:32:59 +0000290From the apache JMeter folder run the test for 72h, pointing it towards the stability.jmx file inside the testplans folder and specifying a logfile to collect the results
RossCab1335d2020-05-05 12:12:54 +0100291
292.. code-block:: bash
293
gregory.hayes6765cb42020-10-21 16:32:59 +0000294 ~/jmeter/apache-jmeter-5.1.1/bin/jmeter -n -t ~/distribution/testsuites/stability/src/main/resources/testplans/stability.jmx -Jduration=259200 -l ~/20201016-1715-distr-stability.jtl &
295
RossCab1335d2020-05-05 12:12:54 +0100296
297Test Results
298------------
299
300**Summary**
301
gregory.hayes6765cb42020-10-21 16:32:59 +0000302- Stability test plan was triggered for 72 hours.
303- No errors were reported
RossCab1335d2020-05-05 12:12:54 +0100304
305**Test Statistics**
306
gregory.hayes6765cb42020-10-21 16:32:59 +0000307.. csv-table:: Stability Results - Summary Report
308 :file: csv/20201016-1715-distr-stability-summary.csv
309 :header-rows: 1
RossCab1335d2020-05-05 12:12:54 +0100310
gregory.hayes6765cb42020-10-21 16:32:59 +0000311.. csv-table:: Stability Results - Aggregate Report
312 :file: csv/20201016-1715-distr-stability-aggregate.csv
313 :header-rows: 1
RossCab1335d2020-05-05 12:12:54 +0100314
gregory.hayes6765cb42020-10-21 16:32:59 +0000315**VisualVM Screenshots**
RossCab1335d2020-05-05 12:12:54 +0100316
gregory.hayes6765cb42020-10-21 16:32:59 +0000317.. image:: images/20201016-1715-distr-stability-20201018T2040-monitor.png
318.. image:: images/20201016-1715-distr-stability-20201018T2040-threads.png
RossCab1335d2020-05-05 12:12:54 +0100319
RossCac964ed2020-05-19 12:07:33 +0100320
321Performance Test of Policy Distribution
322+++++++++++++++++++++++++++++++++++++++
323
324Introduction
325------------
326
gregory.hayes6765cb42020-10-21 16:32:59 +0000327The 4h Performance Test of Policy Distribution has the goal of testing the min/avg/max processing time and rest call throughput for all the requests when the number of requests are large enough to saturate the resource and find the bottleneck.
328
329It also tests that distribution can handle multiple policy CSARs and that these are deployed within 30 seconds consistently.
330
RossCac964ed2020-05-19 12:07:33 +0100331
332Setup Details
333-------------
334
335The performance test is based on the same setup as the distribution stability tests.
336
gregory.hayes6765cb42020-10-21 16:32:59 +0000337
338Test Plan Sequence
339------------------
RossCac964ed2020-05-19 12:07:33 +0100340
341Performance test plan is different from the stability test plan.
RossCac964ed2020-05-19 12:07:33 +0100342
gregory.hayes6765cb42020-10-21 16:32:59 +0000343- Instead of handling one policy csar at a time, multiple csar's are deployed within the watched folder at the exact same time.
344- We expect all policies from these csar's to be deployed within 30 seconds.
345- There are also multithreaded tests running towards the healtchcheck and statistics endpoints of the distribution service.
RossCac964ed2020-05-19 12:07:33 +0100346
gregory.hayes6765cb42020-10-21 16:32:59 +0000347
348Running the Test Plan
349---------------------
350
351Edit the /tmp folder permissions to allow the Testplan to insert the CSAR into the /tmp/policydistribution/distributionmount folder.
RossCac964ed2020-05-19 12:07:33 +0100352
353.. code-block:: bash
354
gregory.hayes6765cb42020-10-21 16:32:59 +0000355 sudo mkdir -p /tmp/policydistribution/distributionmount
356 sudo chmod -R a+trwx /tmp
RossCac964ed2020-05-19 12:07:33 +0100357
gregory.hayes6765cb42020-10-21 16:32:59 +0000358From the apache JMeter folder run the test for 4h, pointing it towards the performance.jmx file inside the testplans folder and specifying a logfile to collect the results
RossCac964ed2020-05-19 12:07:33 +0100359
360.. code-block:: bash
361
gregory.hayes6765cb42020-10-21 16:32:59 +0000362 ~/jmeter/apache-jmeter-5.1.1/bin/jmeter -n -t ~/distribution/testsuites/performance/src/main/resources/testplans/performance.jmx -Jduration=14400 -l ~/20201020-1730-distr-performance.jtl &
RossCac964ed2020-05-19 12:07:33 +0100363
364Test Results
365------------
366
367**Summary**
368
gregory.hayes6765cb42020-10-21 16:32:59 +0000369- Performance test plan was triggered for 4 hours.
370- No errors were reported
RossCac964ed2020-05-19 12:07:33 +0100371
372**Test Statistics**
373
gregory.hayes6765cb42020-10-21 16:32:59 +0000374.. csv-table:: Performance Results - Summary Report
375 :file: csv/20201020-1730-distr-performance-summary.csv
376 :header-rows: 1
RossCac964ed2020-05-19 12:07:33 +0100377
gregory.hayes6765cb42020-10-21 16:32:59 +0000378.. csv-table:: Performance Results - Aggregate Report
379 :file: csv/20201020-1730-distr-performance-aggregate.csv
380 :header-rows: 1
RossCac964ed2020-05-19 12:07:33 +0100381
gregory.hayes6765cb42020-10-21 16:32:59 +0000382**VisualVM Screenshots**
383
384.. image:: images/20201020-1730-distr-performance-20201020T2025-monitor.png
385.. image:: images/20201020-1730-distr-performance-20201020T2025-threads.png