.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0


How to use Fault Management
===========================

This user guide shows how to create subscription on DMS-ETSI for FM and how
DMS-ETSI set alert rules in monitoring tool (prometheus).

.. contents::
   :depth: 3
   :local:


Prerequisites
-------------

#. To create PaaS environment for FM that can be integrated with INF O2 service,
   the following software need to be installed.

   * OpenStack
   * Prometheus
   * Alert Manager
   * Node Exporter
   * Kube-state-metrics
   * Notification server

#. In PaaS environment containerized VNF should be installed, deployed and
   instantiated according to the steps below.

   https://docs.openstack.org/tacker/latest/user/v2/cnf/deployment_using_helm/index.html


Abbreviations
-------------

* FM           : Fault Management
* DMS-ETSI     : Deployment Management Services of ETSI like tacker
* NF FM        : NF FM may be human in this version or a component part of SMO


Procedure
---------

#. Create FM subscription

   NF FM sends a request to DMS-ETSI to create FM subscription. In this FM
   subscription, multiple filter conditions can be set for fault like compute,
   storage, network etc mentioned in create subscription sample file
   sample_param_file.json.

   Follow the below steps to create a subscription for fault:

   * Confirm "ID" of the instantiated VNF by executing below command.

      .. code-block:: console

          $ openstack vnflcm list --os-tacker-api-version 2
          +--------------------------------------+-------------------+---------------------+--------------+----------------------+------------------+--------------------------------------+
          | ID                                   | VNF Instance Name | Instantiation State | VNF Provider | VNF Software Version | VNF Product Name | VNFD ID                              |
          +--------------------------------------+-------------------+---------------------+--------------+----------------------+------------------+--------------------------------------+
          | d2e61392-14dc-4b23-8d33-a19456de65c4 |                   | INSTANTIATED        | Company      | 1.0                  | Sample VNF       | b1bb0ce7-ebca-4fa7-95ed-4840d70a1177 |
          +--------------------------------------+-------------------+---------------------+--------------+----------------------+------------------+--------------------------------------+

   * Change the following values in subscription sample file
     sample_param_file.json to the actual values confirmed
     from above and save the file.

      * "vnfdIds"              : Set the value of "VNFD ID"
      * "vnfProvider"          : Set the string of "VNF Provider"
      * "vnfProductName"       : Set the string of "VNF Product Name"
      * "vnfSoftwareVersion"   : Set the value of "VNF Software Version"
      * "vnfInstanceIds"       : Set the value of "ID"

     The content of the subscription sample sample_param_file.json is as follows:

     .. code-block:: console

         {
             "filter": {
                 "vnfInstanceSubscriptionFilter": {
                     "vnfdIds": [
                         "b1bb0ce7-ebca-4fa7-95ed-4840d70a1177"
                     ],
                     "vnfProductsFromProviders": [
                         {
                             "vnfProvider": "Company",
                             "vnfProducts": [
                                 {
                                     "vnfProductName": "Sample VNF",
                                     "versions": [
                                         {
                                             "vnfSoftwareVersion": 1.0,
                                             "vnfdVersions": [1.0, 2.0]
                                         }
                                     ]
                                 }
                             ]
                         }
                     ],
                     "vnfInstanceIds": [
                         "d2e61392-14dc-4b23-8d33-a19456de65c4"
                     ]
                 },
                 "notificationTypes": [
                     "AlarmNotification",
                     "AlarmClearedNotification",
                     "AlarmListRebuiltNotification"
                 ],
                 "faultyResourceTypes": [
                     "COMPUTE",
                     "STORAGE",
                    "NETWORK"
                 ],
                 "perceivedSeverities": [
                     "CRITICAL",
                     "MAJOR",
                     "MINOR",
                     "WARNING",
                     "INDETERMINATE",
                     "CLEARED"
                 ],
                 "eventTypes": [
                     "EQUIPMENT_ALARM",
                     "COMMUNICATIONS_ALARM",
                     "PROCESSING_ERROR_ALARM",
                     "ENVIRONMENTAL_ALARM",
                     "QOS_ALARM"
                 ],
                 "probableCauses": [
                     "The server cannot be connected."
                 ]
             },
             "callbackUri": "http://10.0.0.194:5000/your-callback-endpoint",
             "authentication": {
                 "authType": [
                     "BASIC"
                 ],
                 "paramsBasic": {
                     "userName": "nfv_user",
                     "password": "devstack"
                 }
             }
         }

   * Execute below command to create FM subscription.

     .. code-block:: console

         $ openstack vnffm sub create sample_param_file.json --os-tacker-api-version 2

   * Verify FM subscription by executing following command.

     .. code-block:: console

         $ openstack vnffm sub list --os-tacker-api-version 2
         +--------------------------------------+-----------------------------------------------+
         | ID                                   | Callback Uri                                  |
         +--------------------------------------+-----------------------------------------------+
         | 724b6752-b782-48e8-a8bb-a20a0fdb8d9f | http://10.0.0.194:5000/your-callback-endpoint |
         +--------------------------------------+-----------------------------------------------+


#. Create alert rules on Monitoring tool

   * Prometheus configuration has two files.

      #. deployment.yaml which contains all the configurations to discover pods
         and services running in the Kubernetes cluster dynamically. No need to
         change in deployment.yaml

      #. configmap.yaml which contains all the alert rules for sending alerts
         to the Alert manager.

         The content of the sample configmap.yaml is as follow:

         .. code-block:: console

             apiVersion: v1
             kind: ConfigMap
             metadata:
               name: prometheus-config
               namespace: monitoring
             data:
               prometheus.rules: |-
                 groups:
                 - name: example
                   rules:
                   - alert: KubePodCrashLooping
                     annotations:
                       probable_cause: The server cannot be connected.
                       fault_type: Server Down
                       fault_details: fault details
                     expr: |
                       increase(kube_pod_container_status_restarts_total[10m]) > 0
                     for: 1m
                     labels:
                       receiver_type: tacker
                       function_type: vnffm
                       vnf_instance_id: 8c93a232-92fb-461a-a5b4-60efa2dd5f81
                       pod: vdu2-798d577c96-6t42j
                       perceived_severity: CRITICAL
                       event_type: EQUIPMENT_ALARM

   * After add/delete/modify alert rule in sample configmap.yaml, perform
     following steps to make it effective.

      #. Delete old Prometheus ConfigMap

         .. code-block:: console

             $ kubectl delete -f configmap.yaml

      #. Delete old Prometheus Deployment File

         .. code-block:: console

             $ kubectl delete -f deployment.yaml

      #. Delete Prometheus Service

         .. code-block:: console

             $ kubectl delete -f service.yaml

      #. Create Prometheus ConfigMap with updated ConfigMap

         .. code-block:: console

             $ kubectl apply -f configmap.yaml

      #. Create Prometheus Deployment File

         .. code-block:: console

             $ kubectl apply -f deployment.yaml

      #. Create Prometheus Service

         .. code-block:: console

             $ kubectl apply -f service.yaml


Requirements
------------

#. Receiving Notification

   * The NF FM sends a create subscription request to the DMS-ETSI.

   * After sending the create subscription request, DMS-ETSI will send a
     GET request to the callback_uri in NF FM to verify its correctness.
     NF FM should receive this request and then return HTTP 204 to DMS-ETSI.

#. Sending Heal Request

   * When a fault occurs in a CNF and matches subscribed alarm condition,
     DMS-ETSI will send an Alarm Notification to NF FM.

   * NF FM should receive the notification, get VNF/VNFC information
     (vnfInstanceId, vnfcInstanceId) from it, and then send Heal CNF
     request to NF-LCM, which further sends heal request to DMS-ETSI.


References
----------

.. [1]  ETSI NFV-SOL CNF Auto Healing with Prometheus via FM Interfaces [1]_.
        https://docs.openstack.org/tacker/zed/user/etsi_cnf_auto_healing_fm.html

.. [2]  ORAN-SC Wiki [2]_.
        https://wiki.o-ran-sc.org/download/attachments/35881444/SMO%20Functions%20v4.docx?api=v2

.. [3]  Tacker API Document for Receiving Notification [3]_.
        https://docs.openstack.org/api-ref/nfv-orchestration/v2/vnflcm.html#create-a-new-subscription-v2

.. [4]  Tacker API Document for Heal Request [4]_.
        https://docs.openstack.org/api-ref/nfv-orchestration/v2/vnflcm.html#heal-a-vnf-instance-v2