docs/development/devtools/xacml-s3p.rst

.. This work is licensed under a
.. Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0

.. _xacml-s3p-label:

.. toctree::
   :maxdepth: 2

Policy XACML PDP component
##########################

Both the Performance and the Stability tests were executed by performing requests
against the Policy RESTful APIs residing on the XACML PDP installed in the windriver
lab.  This was running on a kubernetes pod having the following configuration:

- 16GB RAM
- 8 VCPU
- 160GB Disk

Both tests were run via jmeter, which was installed on a separate VM so-as not
to impact the performance of the XACML-PDP being tested.

Performance Test of Policy XACML PDP
************************************

Summary
=======

The Performance test was executed, and the result analyzed, via:

.. code-block:: bash

    jmeter -Jduration=1200 -Jusers=10 \
        -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \
        -Jxacml_port=31104 -Jpap_port=32425 -Japi_port=30709 \
        -n -t perf.jmx

    ./result.sh

Note: the ports listed above correspond to port 6969 of the respective components.

The performance test, perf.jmx, runs the following, all in parallel:

- Healthcheck, 10 simultaneous threads
- Statistics, 10 simultaneous threads
- Decisions, 10 simultaneous threads, each running the following in sequence:

   - Monitoring Decision
   - Monitoring Decision, abbreviated
   - Naming Decision
   - Optimization Decision
   - Default Guard Decision (always "Permit")
   - Frequency Limiter Guard Decision
   - Min/Max Guard Decision

When the script starts up, it uses policy-api to create, and policy-pap to deploy,
the policies that are needed by the test.  It assumes that the "naming" policy has
already been created and deployed.  Once the test completes, it undeploys and deletes
the policies that it previously created.

Results
=======

The test was run for 20 minutes at a time, for different numbers of users (i.e.,
threads), with the following results:

.. csv-table::
   :header: "Number of Users", "Throughput (requests/second)", "Average Latency (ms)"

   10, 6064, 4.1
   20, 6495, 7.2
   40, 6457, 12.2
   80, 5803, 21.3


Stability Test of Policy XACML PDP
************************************

Summary
=======

The Stability test was run with the same pods/VMs and uses the same jmeter script as the
performance test, except that it was run for 72 hours instead of 20 minutes.  In
addition, it was run in the background via "nohup", to prevent it from being interrupted:

.. code-block:: bash

    nohup jmeter -Jduration=259200 \
        -Jxacml_ip=$ip -Jpap_ip=$ip -Japi_ip=$ip \
        -Jxacml_port=31104 -Jpap_port=32425 -Japi_port=30709 \
        -n -t perf.jmx &

The memory and CPU usage can be monitored by running "top" on the xacml pod.  By taking
a snapshot before the test is started, and again when it completes, the total CPU used
by all of the requests can be computed.

Results
=======

The final output of the jmeter script is found in the nohup.out file:

.. image:: images/xacml-s3p-jmeter.png

The final memory and CPU from "top":

.. image:: images/xacml-s3p-top.png

The through-put reported by jmeter was 4849 requests/second, with 0 errors.  In addition,
the memory usage observed via "top" indicated that the virtual memory and resident set
sizes remained virtually unchanged through-out the test.

Unfortunately, the initial CPU usage was not recorded, so the CPU time reported in
the "top" screen-shot includes XACML-PDP start-up time as well as requests that were
executed before the stability test was started.  Nevertheless, even including that, we find:

.. code-block:: bash

    13,166 CPU minutes * 60sec/min * 1000ms/sec / 1,256,834,239 requests = 0.63ms/request